Behavior Prior Representation learning for Offline Reinforcement Learning
Authors: Hongyu Zang
Xin Li
Jie Yu
Chen Liu
Riashat Islam
Remi Tachet des Combes
Romain Laroche
Venue: ICLR 2023 poster (2023)
Keywords: offline reinforcement learning
representation learning

Abstract: Offline reinforcement learning (RL) struggles in environments with rich and noisy inputs, where the agent only has access to a fixed dataset without environment interactions. Past works have proposed common workarounds based on the pre-training of state representations, followed by policy training. In this work, we introduce a simple, yet effective approach for learning state representations. Our method, Behavior Prior Representation (BPR), learns state representations with an easy-to-integrate objective based on behavior cloning of the dataset: we first learn a state representation by mimicking actions from the dataset, and then train a policy on top of the fixed representation, using any off-the-shelf Offline RL algorithm. Theoretically, we prove that BPR carries out performance guarantees when integrated into algorithms that have either policy improvement guarantees (conservative algorithms) or produce lower bounds of the policy values (pessimistic algorithms). Empirically, we show that BPR combined with existing state-of-the-art Offline RL algorithms leads to significant improvements across several offline control benchmarks. The code is available at \url{https://github.com/bit1029public/offline_bpr}

Published as a conference paper at ICLR 2023
BEHAVIOR PRIOR REPRESENTATION LEARNING FOR OFFLINE REINFORCEMENT LEARNING
Hongyu Zang', Xin Li"; Jie Yu', Chen Liu', Riahsat Islam?, Rémi Tachet des Combes', Romain Laroche
Beijing Institute of Technology, China 2 Mila, Quebec Al Instiute, Canada (zanghyu, xinli, yujie, chenliu] @bit. edu. riashat is ma www.glll.c......mmmmmm......m. (remi. tachet, r romain. laroche ) @ com
ABSTRACT
Offline reinforcement learning (RL) struggles in environments with rich and noisy inputs, where the agent only has access to a fixed dataset without envi- ronment interactions. Past works have proposed common workarounds based on the pre-training of state representations, followed by policy training. In this work, introduce simple, yet effective approach for learning state represen- tations. Our method, Behavior Prior Representation (BPR), learns state repre- sentations with an dataset: we first learn state representation by mimicking actions from the dataset, and then train a policy on top of the fixed representation, using any off-the- shelf Offline RL. algorithm. Theoretically, we prove that BPR carries out per- formance guarantees when integrated into algorithms that have either policy im- provement guarantees (conservative algorithms) or produce lower bounds of the policy values (pessimistic algorithms). Empirically, we show that BPR com- bined with existing state-of-the-e Offline RL algorithms leads to significant im- provements across several offline control benchmarks. The code is available at https::/ / thub. com t 02 029publ ieliblll///e ofooffllff offffinnnbriio np_.pr.
objective based on behavior cloning of the
INTRODUCTION
Offline Reinforcement Learning (Offline RL) is one of the most promising data-driven ways of optimizing sequential decision-making. Offline RI. differs from the typical settings of Deep Rein- forcement Learning (DRL) in that the agent is trained by some arbitrary process, and does not interact with the environment during learning (Lange et al., 2012; Levine al., 2020). Consequently, it benefits the scenarios where online exploration is chal- lenging and/or unsafe, especially for application domains such as healthcare (Wang et al., 2018; Gottesman al., 2019; Satija al., 2021) and autonomous driving (Bojarski al., 2016; Yurtsever et al., 2020). A common baseline of Offline RL is Behavior Cloning (BC) (Pomerleau, 1991). BC performs maximum--lkellhooe training on collected set of demonstrations, essentially mimicking the behavior policy to produce predictions (actions) conditioned on observations. While BC can only achieve proficient policies when dealing with expert demonstrations, Offline RL goes beyond the goal of simply imitating and aims train policy that improves over the behavior one. De- spite promising results, Offline RL algorithms still suffer from two main issues: i) difficulty dealing with limited high- high-dimensional data, especially visual observations with continuous action space (Lu et al., 2022); ii) implicit under- padaeeraratttiizzaaioon of value networks exacerbated by highly re- useu data, that is, an expressive value network implicitly behaves as paraee-arameterrize one when trained using bootstrapping (Kumar et al., 2021
fixed dataset that was previously collected
In this paper, we focus state representation learning for Offline RL. to mitigate the above issues: projecting the high-dimensional observations to low-dimensional space can lead to a better per- formance given limited data in the Offline RL scenario. Moreover, disentangling representation learning from policy training (or value function learning), referred to as pre-training the state rep- resentations, can potentially mitigate the "implicit under- paleeeerraattiiiiioonnnn phenomenon associated with the emergence of low-rank features in the value network (Wang et al., 2022). In contrast to
"oorrsponndecce to Xin Li. This work was partially supported by NSFC under Grant 62276024 and 92270125.
Work done while Microsoft Research Montreal.

---

Published as a conference paper at ICLR 2023
previous work that pre-train state representations by specifying the required properties, e.g., maxi- mizing the diversity of states encountered by the agent (Liu & Abbeel, 2021; Eysenbach ct.,, 20 2019), exploring the attentive knowledge sub-trajectory (Yang & Nachum, 2021), or capturing temporal information about the environment (Schwarzer et al., 2021a), we consider using the hehavior policy to learn generic state representations instead of specifying specific properties.
Many existing Offline RL. methods regularize the policy to be close to the behavior policy (Fujimoto al., 2019; Laroche et al., 2019b; Kumar et al., 2019) or constrain the learned value function of OOD actions not to be overestimated (Kumar et al., 2020; Kostrikov et al., 2021). Beyond these use, the behavior policy is often ignored, as it does not directly provide information on the environment. However, the choice of behavior has huge impact on the Offline RL task. As shown by recent theoretical work (Xiao et al., 2022; Foster et al., 2022), under an agnostic baseline, the Offline RI, task is intractable (near optimality is exponential in the state space size), but it becomes tractable with a well-designed behavior (e.g. the optimal policy or policy trained online). This impact indicates that the information collected from the behavior policy might deserve more attention.
To this end, we propose Behavior Prior Rep- resentation (BPR), a state representation learn- ing method tailored to Offline RL settings (Fig- ure 1). BPR learns state representations implic- itly by enforcing them to be predictive of the ac- tion performed by the behavior policy, normal- ized to be on the unit sphere. Then, the learned encoder is frozen and utilized train down- stream policy with any Offline RL algorithms. Intuitively, to be predictive of the normalized actions, BPR encourages the encoder to ignore the rask-irrelevant information while maintain- ing the task-specific knowledge relative to the behavior, which we posit is efficient for learn- ing state representation. Theoretically, we prove that BPR carries out performance guaran- tees when combined with conservative pes- simistic Offline RL algorithms. While an uninformative behavioral policy may lead to bad repre- sentations and therefore degraded performance, we note that such scenario may be predicted from the empirical returns of the dataset. Furthermore, since the learning procedure of BPR does not in- volve value functions or bootstrapping methods like Temporal-Differenn the "implicit under-paree paramcterization phenomenon. We prove this empirically by utilizing effective dimensions measurement to evaluate feature compactness in the value network's penultimate layer. The key contributions of our work are summarized below:
Behavior Cloning
Offine delmel
Slate
Pollcy mi....
Trained pollcy
(5.ol
Action
Mixinize
MSE
Predicted cled adtion
Behavioral Prior Representation
Action
Normelization
Mrtmize MSF
Nor melizeiinn
(1a)
Offine dataset D/ tuple
State
Enzoder
Prodictor
Predicted
Ma.., (r)
Offiing RL algonthm
Trained
1uple
pollcy
Figure I:
Illustration of Behavior Prior Represen- tations and comparison with Behavior Cloning.
can naturally mitigate
We propose a simple, yet effective method for state representation learning in Offline RL, re- lying on the behavior cloning of actions; and find that this approach is effective across several offline benchmarks, including raw state and pixel-based ones. Our approach can be combined to any existing Offline RL pipeline with minimal changes.
Behavior prior representation (BPR) is theoretically grounded: we show, under usual assump- tions, that policy improvement guarantees from offline RL algorithms are retained through the BPR, at the only expense of an additive behavior cloning error term.
We provide extensive empirical studies, comparing BPR to several state representation objec- tives for Offline RL, and show that it outperforms the baselines across wide range of tasks.
2 RELATED WORK
Offline RL with behavior regularization. Although to the best our knowledge, we are the first to leverage behavior cloning (BC) to learn state representation in Offline RL, we remark that combining Offline RL with behavior regularization has been considered previously by many works. A common way of combining BC with RL is to utilize it as a reference for policy optimization with baseline methods, such as natural policy gradient (Rajeswaran et al., 2018), DDPG (Nair et al., 2018; Goecks et al., 2020), BCQ (Fujimoto et al., 2019), SPIBB (Laroche al., 2019a; Nadjahi et al.,
2

---

Published as a conference paper at ICLR 2023
2019; Simão et al., 2020; Satija et al., 2021; Brandfonbrener et al., 2022), CQL (Kumar et al., 2020), and TD3 (Fujimoto & Gu, 2021). Other previous works include learning adaptive behavior policies that biased towards higher-rewards trajectories (Cang et al., 2021), and pretraining behavior policies to guide downstream policies training (Zhang et al., 2021b). However, all these approaches neglect the potential of using the behavior policy guide representation learning. In contrast, we investigate state representation learning via a BC-style approach and show that the downstream policy can be greatly boosted in that regime.
Representation learning in Offline RL. Pretraining representation has been recently studied in Offline RI. settings, where several studies presented its effectiveness (Arora et al., 2020; Schwarzer et al., 202 1a; Nachum & Yang, 2021a). Some typical auxiliary tasks for pretraining state represen- tations include capturing the dynamical (Nachum & Yang, 2021b) and temporal (Schwarzer et al., 2021a) information of the environment, exploring the attentive knowledge & Nachum, 2021), improving policy performance by applying data augmentations techniques to the pixel-based inputs (Chen et al., 2021; Lu et al., 2022)... While driven by various motivations, most of these methods can be thought of as including different inductive biases via the specific design of the target properties of the representation. In this paper, the inductive bias we enforce is that the representation should allow matching the behavior action normalized to the unit hypersphere, we demonstrate its effectiveness on both raw-state and visual-observation inputs tasks. Additional related works are discussed in appendix B.
sub-trajectory (Yang
3 PRELIMINARIES
Offiine RL We consider the standard Markov decision process (MDP) framework, in which the environment given by a tuple M AS, T, p. r, 7), with state space S, action space A, transi- tion function T that decides the next state s' ~ [- a). initial state distribution p. reward function r(s, a) bounded by Rmaxs and a discount factor 7 E [0, 1). The agent in state s E selects an action E A according to its policy, mapping states to a probability distribution over actions:
<--ss). We make use of the state value function (s) = E,M IEi= 7'r (s,, a,) | so = s] to describe the long term discounted reward of policy starting state In Offline RL, we are given fixed dataset of environment interactions that include N transition samples, i.e., D = {si, as 8/, r N We assume that the dataset D is generated i.i.d. from distribution ju(s, a) that specifies the ef- fective behavior policy Ta (als) = μ(ss, a)/ >n puss, a). and denote by ds*(ss its discounted state occupancy density. Following Rashidinejad al. (2021), the goal of Offline RL is minimize the suboptimality of with respect to the optimal policy π² given a dataset D:
SubOpt(z) ED= [J (x*) - [(]] ED= Exo~p [V* (so) V (so)ll.
(1)
where [(T) = Esorp [V* (so) is the performance of the policy π.
State Representation Learning In this paper, our objective is find a state representation fune- tion &= S + 2 that maps each state S to its representation = d(s) E Z. The desired representations should provide necessary and useful information to summarize the task-relevant knowledge and facilitate policy learning. To investigate the capacity of state representation learning for mitigating the "implicit under- pardaerrratttiiiiiioonn phenomenon, following Lylc et al. (2022) and Kumar al. (2021a), we measure the compactness of the feature in the penultimate layer of the value network by utilizing the Effective Dimension, where we refer to the output of the penultimate layer of the state-action value network as the feature matrix V E R|S|| xd 1.
Definition 1. Effective Dimension Let ED(M) denote the eigenvalues of a square matrix M, D be the offline dataset, and e be a fixed hyperparameter. Then, the effective dimension of Vi is defined as
D(), =
-Eo [Ko
ED (DDIS--1AA)11 (V)TT// 1
is the expected number of eigenvalues of T To that are larger than S||AAe. By studying this quantity, we explicitly observe the usefulness of the state representation objective in mitigating the "implicit
problem in value networks (see Section 6.3 for the results).
Notably, when we characterize the state by its corresponding state representation. the dimension of the feature matrix changes accordingly as TR [2] We defer detailed discussion to Appendix D.
3

---

Published as a conference paper at ICLR 2023
4 BEHAVIOR PRIOR REPRESENTATION
Given an offline dataset D consisting of (s, pairs, our goal is to learn an encoder ((8) that pro- duces state representations
((s) allowing efficient and successful downstream policy learning. During pre-training, the BPR network is comprised of two connected components: an encoder dø and a predictor fw- The encoder maps the state to the representation space, while the predictor projects the representations onto the unit sphere in dimension AA]. In state s, the BPR network out- puts a representation 2œ = (o(s), and a prediction y f2o). We then éz-normalize the prediction y and the action to // lylle and a a/ l/llall2 and minimize the mean squared error to train the state representation (or equivalently maximize the cosine similarity between y and a):
[|ž B(s,a)-D [2
(fo(do())),
C = (s,a)> ll
(2- 2-2- 2-
(3)
( |all
where we set the action from the pair (s. a) as the target. During the training process, stochastic optimization is performed to minimize L with respect to & and w. Note that although the encoder and predictor are updated together through this optimization procedure, only the encoder dee used in the downstream task. At the end of the training, we keep the encoder deg fixed and build the downstream Offline RL agents on top of the state representation that BPR learnt.
Implementation details BPR method does not rely on any specific architecture its encoder network. Depending on the nature of given inputs, it can either be a convolutional neural network (CNN) for visual observation inputs, state inputs. The representation 2q that corresponds to the output of the encoder is then projected to the action space. In this paper, the projection done by an MLP consisting of two linear layers followed by rectified linear units (ReL.) (Nair & Hinton, 2010), and a final linear layer followed by a tanh activation layer.
multi-layer perceptron (MLP) for physically meaningful
5 THEORETICAL ANALYSIS
For simplicity, we consider the optimization problem of BPR without the normalization term as:
1 n Σ (1(((())) ail1: ò DE
min
where (si; is i.i.d. sample from the offline dataset (of size n). BPR consists in learning representation e impacting the policy search. In other words, with BPR, the function class for the policy consists in UPr = {TT.S.. s.t. Fo with x--ls) (db((s) Vs}. where Sa is the neural model parameterized with w characterizing the policy on top of embedding ée(.). Like any representation learning technique, the potential benefits are (i) enhancing the signal-to-noise ratio and (ii) reducing the size of the policy search. In this section, we develop an analysis showing that the potential harm of using BPR upper bounded by an error cæ that we control in Section 5.3. To our knowledge, this is the first representation learning technique for Offline RL with such guarantees.
Idealized assumptions : Letting å denote an estimate of a quantity x computed using D, we start by considering the following idealized assumptions:
Assumption 1. (1.1) Access to the true behavior: F = T3- (1.2) Access to the true performance of
policies: (() = (T) for all policies x. (1.3) The embedding & allows to represent the behavior policy estimate: (())) (T(aas) TaPa- (1.4) The Offline RL algorithm performs perfect
optimization on top of Φ: JUER = maxxellore (T).
Theorem 1. Under idealized Assumption 1, BPR returns policy that improves over the behavior policy: JuPn 2 [(m)).
Its proof immediate: under perfect estimation and optimization, the fact that 3 E TRRR guarantees policy improvement. The above assumptions are stringent, and we propose to relax them in two different ways: for conservative algorithms that derive safe policy improvement guarantees (Petrik et al., 2016; Fujimoto et al., 2019; Laroche et al., 2019b; Simão et al., 2020), and for pessimistic
"For readability, we defer rigorous proofs of theorems to the Appendix F.
4

---

Published as a conference paper at ICLR 2023
algorithms that derive value function lower bound (Kumar al., 2020; Buckman et al., 2021). We note that the policy improvement is constrained by the set ΠBx over which the policy search is conducted. Appendix G.6 empirically shows that the method will fail to improve over the behavioral policy in the extreme case where the behavioral policy uniform and therefore uninformative. The BPR efficiency therefore strongly relies the assumption that the behavioral policy provides beneficial inductive bias.
5.1 SAFE POLICY IMPROVEMENT
Conservative algorithms constrain the policy search to the set of policies Пę for which the true policy improvement Ax,a, = J (TT))(TT)) can be safely estimated from AT.r [T) ( (TT)):
vx E Ip,, A
A CA.
(5)
worth noting that the estimated behavior policy " necessarily belongs to pp, since trivially g 0. Combined with BPR, conservative algorithms optimize on the policy set Hh IPR, and we consider the following relaxed assumptions, corresponding respectively to Assumptions 1.1 and 1.2:
Å g Argig
Assumption 2. (2.1) Access to an accurate estimate n of the true behavior Tz. [(T)) I(T))) S EB. (2.2) Access to an accurate estimate Da,g of the true policy improvement D for all policies π 1h Tupx: Aa,r - AR,TA S Cs.
Theorem 2. Under Assumption 2, BPR returns policy Tark with the following performance bounds: Jurn 2 [[TT(( + Mare.
EB-
Assumption 2.1 amounts to behavior cloning (BC) guarantees. It generally assumed as being an easier task than policy optimization, as BC is supervised learning. Theorem 4 provides bounds on Eg. Assumption 2.2 reflects the policy improvement objective. For instance, the SPIBB principle guarantees with high probability in finite MDPs (Laroche et al., 2019b; Nadjahi et al., 2019; Simão et al., 2020). It is important to notice that Assumption 2.2 forbids, in theory, the use of the BPR representation to estimate the value directly. But in practice, we show that the representation learned by BPR is still useful estimate the value function. We provide some experimental results in Appendix G.7 and a counterexample in Appendix F. Additionally, it is worth noting that ensuring that À T is positive is necessary to obtain approximate policy improvement guarantees. To do so, it suffices to fall back to g when the optimization of Thex does not lead to positive expected improvement Å Agaas
5.2 VALUE FUNCTION LOWER BOUND
In this section, we propose an alternative to Assumption 2.2 following the performance lower bound principle derived in pessimistic algorithms:
Assumption 3. (3.1) Access to a lower bound of the performance J. (T) S [(T) for all policies, which is accurate for zz J. (TT) 2 I (TT) 6..
In order to provide performance improvement bounds, we make use of the lower bound value gap A n.ZS =](()) (
Theorem 3. Under Assumption 2.1 and 3.1, BPR returns policy Tane with the following perfor- mance bounds: Jnnn 2 I (TTA) + A 1p
Assumption 3.1 is satisfied by the Offline RL algorithm that is used in combination with BPR. For instance, CQL (Kumar et al., 2020; Yu et al., 2021) relies on the computation of a lower bound the value function of the considered policies. More generally, pessimistic algorithms (Petrik et al., 2016; Yu et al., 2020; Kidambi et al., 2020; Jin et al., 2021; Buckman et al., 2021) have for principle to add an uncertainty- based penalty to the reward function, in order to control its risk to be overestimated. is important to notice that, like for Assumption 2.2, this assumption would forbid (in theory) the of the BPR representation to estimate the value.
5

---

Published as a conference paper at ICLR 2023
Both approaches (safe policy improvement and value function lower bounds) suffer the same cost eg. in comparison to their guarantees without BPR. The next subsection controls this quantity.
5.3 PERFORMANCE BOUND WITH BPR OBJECTIVE
In this section, we are concerned with the statistical property of the emoor between the estimated behavior based on the representation and the true behavior. To this end, we have the upper bound of the Eg as:
Theorem 4. With probability least 6, for any 8 E (0,
c C.K 1 Σ ( - TB ((())ss)))) + +2V2K Rad( +
2
(6)
where n is the size of the dataset, Rod( () is the Rademacher complexity of o' function class 4 is the behavior policy over the dataset, C is constant, and K
Rass
Equation 6 the exact optimization problem BPR in Equation 4 multi- plied by constant C' K, where the action a, is sampled from the offline dataset pairs (st, a;) whose behavior policy is Ty. and the predictor plays the similar role as क. The second and the third are both irrelevant the specific representation, or the estimated behavior. This indicates that the potential harm of utilizing BPR can be reduced as the representation training procedure goes on.
Note that the first
EXPERIMENTS
BPR be easily combined with any existing Offline RL pipeline. Our experimental results be broken down into the following sections
Does BPR outperform baseline Offline RL algorithms on standard raw-state inputs bench- ?othe?other representation objectives?
Is BPR effective in learning policies high-dimensional pixel-based inputs? Can BPR improve the robustness of the representation when the input contains complex distractors?
Can BPR successfully improve the effective dimension the feature in the value network?
6.1 EFFECTIVENESS OF BEHAVIOR REPRESENTATIONS IN D4RL OFFLINE BENCHMARK
(pretrained
CQL
halfcheetah
Cumulaaiive Returns
medium expert
hopper medium sxpart
halchhettah medium expart
26
Timestegs
Figure 2: Performance comparison on D4RL dataset. The x-axis indicates the number of environ- ment steps. The vertical axis reports the normalized cumulative returns. We train our 6 algorithms (EDAC, TD3+BC, CQL, and their BPR variants) on 6 seeds, and evaluate each one every 5,000 environment steps by computing the average return over 10 evaluation episodes. Lighter lines cor- respond to baselines, darker ones to versions pretrained with BPR. The black dotted line shows the time when representation pre-training ends and policy training process begins.
Due the page limit, provide additional experiment results Appendix G.

---

Published as a conference paper at ICLR 2023
Performance Comparison in D4RL Benchmark Experimental Setup: We analyze our proposed method BPR on the D4RL benchmark (Fu et al., 2020) of OpenAl gym MuJoCo tasks (Todorov et al., 2012) which includes variety of datasets that have been commonly used in the Offline RL community. We evaluate our method by integrating it into three Offline RI. methods: TD3+BC (Fu- jimoto & Gu, 2021), CQL (Kumar al., 2020), and EDAC (An et al., 2021). We consider three environments: halfcheetah, hopper and walker2d, and two datasets per task: expert and medium- expert. Instead of passing the raw state as input the value and policy networks in the baseline methods, we first pretrain the encoder during 100k timesteps, then freeze it, pass the raw state through the frozen encoder to obtain representation, and use that as the input of the Offline RL algorithm. Further details on the experiment setup are included in appendix G.
Analysis: Figure shows the performance of all models on the D4RI, tasks. We observe that pre- training the encoder with BPR leads to faster convergence for all Offline RI. algorithms, and can improve policy performance. Notably, since different Offlin RL. algorithms share the same encoder network architecture, the pretrained BPR encoder can be reused across them, which substantially amortizes the pretraining cost.
Performance Comparison among Differ- ent Representation Objectives Experimen- tal Setup: We follow the pre-training and fine- tuning paradigm of Yang & Nachum (2021), where we first pre-train the encoder for 100k timesteps, and then fine-tune BRAC (Wu et al., 2019) agent on downstream tasks. We conduct performance comparison by training the encoder with different objectives, including BPR and several other state-of-the-an represen- tation objectives*.
Sourter
0.25
Contrwative riveri te Supel
00.00
0.10
0.00
160
malized
Figure 3: Bootstrapping distributions for uncer- tainty in IQM measurements with 5000 resamples
Analysis: We follow the performance criterion from Schwarzer et al. (2021b); Agarwal et al.
(2021), and use the inter-quartile mean (IQM) normalized score, which is calculated overruns rather than tasks, and the percentile bootstrap confidence intervals, to compare the performance of the different objectives. The detailed results are provided in appendix normalized return and 95% bootstrap confidence intervals for all methods, the numerical values can be found in Table 1. The performance gain of BPR over the two most competitive baselines, Fourier and ACL, is statistically significant(p 0.05).
Figure presents the IQM
6.2 EFFECTIVENESS OF BEHAVIOR REPRESENTATIONS VISUAL OFFLINE RL BENCHMARK
Funoredd NOX
Ras
2116
cumulative Beturns
85
Timesteps
Figure 4: Performance comparison of BPR with several other baselines DMC environments (left). Run time comparison for all representation objectives rumning on cheetah run _en_eppert task with 100K timesteps, the vertical axis indicates the wall-clock time in seconds (right).
We the codehase from Yang & Nachum (2021) in these experiments: google wwwrrch///aoo wewgrhhhgggogeeeeeorooeeeee wew.ooggeereeerrteeeeoo htw.arrtttteeeeeeeeeee//t///
httpa://githbbe
7

---

Published as a conference paper at ICLR 2023
Table I: Interquartile mean, median, and mean normalized scores, evaluated over all six runs for each of 16 DMC tasks. Confidence intervals computed by percentile bootstrap with 5000 resamples.
Method
IQM
95% CT
Median
95% CI
Mean
95% CI
BPR
89,99
(85.55, 93.97)
87.04
(77.04.
91.53)
81.70
(77.38,
86,21)
ACL
70.79
(66.09, 75.53)
67.50
(57.73.
77.28)
69.91
(65.98, 74.05)
Fourier
(77.86, 86.17)
(73.25, 88.99)
72.61
(68.01, 76.51)
Forward
18.59
(16.35, 20.90)
9,26
(6.68, 14,48)
29.68
(27.25, 32.20)
Contrastive
44.87
(38.36, 51.52)
41.48
(35.82, 54.39)
43.56
(39.18, 47.98)
Inverse
42.79
(34.97, 50.83)
40,33
(33.38, 47.47)
44.58
(39.20, 49,99)
Super
62.62
(56.81, 67.91)
58,91
(48 66, 68.79)
60.72
(57.08,64.53)
Performance comparison in V-D4RL benchmark
Experimental Setup We evaluate our method with five representation objectives that are consid- ered as state-of-the-art methods benchmarking suite for Offline RL from visual observations of DMControl suite (DMC) tasks (Lu et al., 2022). We note that not all methods have been shown to be effective for visual-based continuous control problems. The baselines include: (i) Temporal contrastive learning methods such as DRIML (Mazoure et al., 2020) and HOMER (Misra et al., 2020); (ii) spatial contrastive approach, CURL (Laskin al., 2020); (iii) one-step inverse action prediction model, Inverse model (Pathak et al., 2017); and (iv) Representation module which com- bines self predictive, conl-conditioned RL and inverse model, namely SGI (Schwarzer et al., 202 We evaluate all representation objectives by integrating the pre-trained encoder from each into an Offline RL method DrQ+BC (Lu et al., 2022), which combines data augmentation techniques with the TD3+BC method (TD3 with regularizing pehavioral-cloniee term to the policy loss).
Analysis As shown Figure 4, BPR consistently improves over the other state-of-the-eee algo- rithms, except for SGI in the molker_walk, task, while SGI is the most time-consuming representation objective of all methods. We evaluate the wall-clock training time of each representation objective for the 100K time steps. Our approach compares favorably against all previous methods. In particu- lar, BPR is 2.5x faster than the most competitive SGI method. This suggests that the BPR objective improve sample efficiency more effectively than the other representation objectives.
Robustness to Visual Distractions Experimental Setup: To test the policy robustness brought by BPR, we use three levels of distractors: easy, medium, and hard, to evaluate the performance of the model, following Lu et al. (2022). Each distraction represents a shift in the data distribution, where task-irrelevant visual factors are added (i.e., backgrounds, agent colors, and camera posi- tions). Those factors of variation disturb the model training (see the demonstrations provided in Appendix G.8). We train the baseline agent, DrQ+BC, and its variant using fixed encoder pre- trained by BPR, on the three levels of the distraction of the cheetah- theetah-run-meedee mun-mediuu--eppr dataset with million data points. In the experiment, the agent is evaluated on three different test environments: i) Original Env, the evaluation environment is the original environment i.e., without any distractors; fii Distractor Train, the evaluation environment has the same distraction configuration as the train- ing dataset, and the configuration is fixed during the evaluation procedure; iii) Distractor Test, the specific distractions the evaluation environment changes over evaluation, while the level of the distractions remains the same.
Table 2: Cumulative return from easy to hard. We compare DrQ+BC (before -) to DrQ+BC with an encoder pretrained with BPR (after -+). The final performance is averaged over 6 seeds (highest result for each task under- lined).
dataset with visual distractions ranging
Level of Distractions
Eval. Return Dis. Train
Original
Dis. Test
easy medium
45,50 37.02) - 113.64( 24.83)
425.15(# 65.67) + 710.11(# 56.70)
45.16(= 24.96) + 113.27 26.48) 14.70 5.93) 27.53 11.62) 18.88( = 11.32) - 16.72 6.88)
37,35(+ 25.66) 104. 300 22.95)
1 99.66) 702.00 90.44)
hard
13.49( 10.66)
64.79 + 33.47)
497.41(1 103.85) 608 98( 1: 64.64)
Analysis: The final performances are shown in Table 2. As can be seen. BPR improves the policy performance in most cases, except for the task with combination of the hard level and the distractor
8

---

Published as a conference paper at ICLR 2023
test setting where the averaged performance of the agent with BPR is close to the without BPR but has a lower variance. Besides, the different evaluation environment corresponds to different degrees of the distribution shift from the training dataset. It is therefore not surprising to that the agent performs better in Distractor train environment than the others. This experiment shows that much better (more robust) policies be obtained against distractions with the help of BPR, indicating that BPR objective can somehow disentangle sources of distractions through training, facilitating the learning of general features.
6.3 EFFECTIVE DIMENSION OF THE VALUE NETWORK
Experimental Setup: We investigate whether utilizing
LBL
COunith excoler
Colppettaies 8PR)
the fixed encoder learned from BPR can alleviate the phenomenon of the value net-
Effecctve dimentoon
work. To this end, we propose to use the effective di- mension (defined in Definition 1) as a way to evaluate the compactness of features represented the value network. We first sample batch of state-action pairs from the of- fline dataset and then compute the effective dimension of the output of the penultimate layer of the value network. In this experiment, we evaluate the effective dimension of the value network of the CQL agent Halfeheetah- medium-expert--2 task in D4RI. datasets, which operates on the physical meaning raw state inputs. For compari- son, we also develop variant of CQL. whose input is still the raw state, but shared encoder head is applied both the value network and policy network and is trained with the critic loss along the policy learning procedure.
Comdative futums
Timestps
105
Figure Effective dimension of the penul- timate layer of the value network (top) and performance (bottom) over the training on the
task.
Analysis:
Figure 5 illustrates that over the course of training, BPR objective successfully improves the effective dimensions of the state representation, meanwhile achieving better performance com- pared to the agent that is without encoder and the agent whose encoder is trained with the critic loss. Notably, when training the encoder via the critic loss along the policy learning procedure, the effective dimension increases significantly carly in training, suggesting that even without any specific representation loss, the encoder can still disentangle some similar states. When the training goes on, the agent will observe more diverse states over time as the policy improves, which induces a mismatch between the learnt representations and the newly observed states, leading to a decline in the effective dimension. In contrast, with fixed encoder learned using the BPR objective, CQL produces more effective dimensions than its vanilla version, leading to better performance, which indicates that BPR is capable of providing compact information for downstream Offline RI. training.
7 DISCUSSION
Limitations and Future Work Although this work has shown effectiveness in learning repre- sentations that are robust to visual distractions, it still struggles when the distribution of evaluation environments shifts too much from the training environment, this suggests that further improve- ment on the generalization abilities of the representation is required. Another limitation related to the nature of the approach is that if the behavioral policy is uninformative, then approach will likely result in a policy that has the same performance as the behavior but without any improvement. However, it is straightforward to leverage information about the behavior (either priori or from statistics on the returns in the dataset) to decide when to use BPR. From these perspectives, further development of a more theoretically grounded representation objective might be needed.
Conclusion Offline RL algorithms typically suffer from two main issues: one is the difficulty of learning from high-dimensional visual input data, the other is the implicit under-parameetrizztion phenomenon that can lead to low-rank features of the value network, further resulting in low perfor- mance. In this work, propose simple yet effective method for state representation learning in an Offline RL setting which can be combined with any existing Offline RL pipeline with minimal changes.

---

Published as a conference paper at ICLR 2023
8 REPRODUCIBILITY STATEMENT
To ensure the reproducibility of all empirical results, we provide the code base in the supplementary material, which contains: training scripts, and requirements for the virtual environment. All datasets and code platforms (PyTorch and Tensorflow) we use are public. To rebuild the architecture of BPR model and plug BPR in any Offline RL algorithms, the readers are suggested check the descriptions in Section 4, especially the Implementation details paragraph. All proof' are stated in Appendix E with explanations and underlying assumptions. We also provide the pseudocode of the pretraining process and the co-training process in Algorithm 1 and 2 in Appendix.
All training details are specified Section and Appendix G. In the experiment on d4rl tasks, all representation objectives use the same encoder architecture, i.e. with 4-layer MLP activated by ReLU, followed by another linear layer activated by Tanh, where the final output feature dimension of the encoder is 256. Besides, all representation objectives follow the same optimizer settings, pre-training data, and the number of pre-training epochs. In the experiment on v-d4rl tasks, all representation objectives use the same encoder architecture, i.e., with four convolutional layers acti- vated by ReLU, followed by a linear layer normalized by LayerNorm and activated by Tanh, where the final output feature dimension of the encoder is 256. We also provide the pseudocode for each visual representation objective in Algorithm 4-7 in Appendix.
REFERENCES
Alekh Agarwal, Yuda Song, Wen Sun, Kaiwen Wang. Mengdi Wang, and Xuezhou Zhang. Provable benefits of representational transfer in reinforcement learning. CoRR, 0te/22005.157777 2022.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C. Courville, and Marc G. Belle- mare. Deep reinforcement learning at the edge of the statistical precipice. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Sys- tems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 29304-29320, 2021.
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty -baeed offline rein- forcement learning with diversified g-ensemble. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 202 December 6-14, 2021, virtual, pp. 7436-7447, 2021.
David Andre and Stuart J. Russell. State abstraction for programmable reinforcement learning agents. In Proceedings of the Eighteenth National Conference Artificial Intelligence and Four- teenth Conference on Innovative Applications of Artificial Intelligence, July 28- August 1, 2002, Edmonton, Alberta, Canada, pp. 119-125. AAAI Press / The MIT Press, 2002.
Sanjeev Arora, Simon S. Du, Sham M. Kakade, Yuping Luo, and Nikunj Saunshi. Provable rep- resentation learning for imitation learning via bi-level optimization. In Proceedings of the 37th International Conference Machine Learning, ICMI 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 367-376. PMLR, 2020.
Kavosh Asadi, David Abel, and Michael L. Littman. Learning state abstractions for transfer in continuous control. CoRR, abs/2002.05518. 2020.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba. End to end learning for self-driving cars. CoRR, hbbs/164400736. 2016.
David Brandfonbrener, Rémi Tachet des Combes, and Romain Laroche. Deep-SPIBB: Scaling up safe policy improvement for offline reinforcement learning, Proceedings of the 5th Reinforcement Learning and Decision Making (RLDM), 2022.
Jacob Buckman, Carles Gelada, and Mare Bellemare. The importance of pessimism in fixed- dataset policy optimization. In Proceedings of the 10th International Conference Learning Representations (ICLR), 2021.
Emmanuel J. Candès and Benjamin Recht. Exact matrix completion via convex optimization. Found. Comput, Math., (6)7777-77, 2009.
10

---

Published as a conference paper at ICLR 2023
Catherine Cang, Aravind Rajeswaran, Pieter Abbeel, and Michael Laskin. Behavioral priors and dynamics models: Improving performance and domain transfer in offline RL. CoRR, abs/2106.00 091110000099,,,... 19, 2021.
Pablo Samuel Castro, Tyler Kastner, Prakash Panangaden, and Mark Rowland. MICo: Improved representations via sampling-based state similarity for markov decision processes. In Advances in Neural Information Processing Systems, 2021.
Xin Chen, Sam Toyer, Cody Wild, Scott Emmons, Ian Fischer, Kuang-Huei Lee, Neel Alex, Steven H Wang, Ping Luo, Stuart Russell, Pieter Abbeel, and Rohin Shah. An empirical investi- gation of representation learning for imitation. Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Truck (Round 2), 2021.
Gheorghe Comanici, Prakash Panangaden, and Doina Precup. On-the-fly algorithms for bisimula- tion metrics. Ninth International Conference on Quantitative Evaluation of Systems, QEST 2012, London, United Kingdom, September 17-20, 2012, pp. 94-103. IEEE Computer Society, 2012.
Petros Drineas, Michael W. Mahoney, and S. Muthukrishnan. Sampling algorithms for la regression and applications. In Proceedings of the Seventeenth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA 2006, Miami, Florida, USA, January 22-26, 2006, pp. 1127-1136. ACM Press, 2006.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without reward function. In 7th International Conference on Learning Repre- sentations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Norm Ferns, Pablo Samuel Castro, Doina Precup, and Prakash Panangaden. Methods for computing state similarity in markov decision processes. In UAI '06, Proceedings of the 22nd Conference in Uncertainty in Artificial Intelligence, Cambridge, MA, USA, July 13-16, 2006. AUAI Press, 2006.
Dylan Foster, Akshay Krishnamurthy, David Simchi-Levi, and Yunzong Xu. Offline reinforcement learning: Fundamental barriers for value function approximation. 2022.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: datasets for deep data-driven reinforcement learning. CoRR, abs/2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Informa- tion Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 20142, 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beuch, California, USA, volume 97 of Proceedings of Machine Learning Research, pp.
PMLR, 2019.
Vinicius G. Goecks, Gregory M. Gremillion, Vernon J. Lawhern, John Valasek, and Nicholas R. Waytowich. Integrating behavior cloning and reinforcement learning for improved performance in dense and sparse reward environments. In Proceedings of the 19th International Conference on Autonomous Agents and Multiagent Systems, AAMAS '20, Auckland, New Zealand, May 9-13, 2020, pp. 465-473. International Foundation for Autonomous Agents and Multiagent Systems, 2020.
O Gottesman, F Johansson, M Komorowski, A Faisal, D Sontag. F Doshi-Velez, and LA Celi. Guidelines for reinforcement learning healthcare. Nature medicine, 25:16-18, 2019.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent A new approach to self-supervised learning. In Advances in Neural Information Processing Sys- tems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 1-1 2020, viriual, 2020.
11

---

Published as a conference paper at ICLR 2023
Danijar Hafner, Timothy P. Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In Proceedings of the 36th International Conference Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, Cali- fornia, USA, volume 97 of Proceedings of Machine Learning Research, pp. 2565. PMLR, 2019.
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for Offline RL? In Proceedings of the 38th International Conference on Machine Learning (ICML), pp. 5084-5096. PMLR. 2021.
Nicholas K. Jong and Peter Stone. State abstraction discovery from irrelevant state variables. In Leslie Pack Kaelbling and Alessandro Saffiotti (eds.), IJCAI-05, Proceedings of the Nineteenth International Joint Conference on Artificial Intelligence, Edinburgh, Scotland, UK, July 30 Au- gust 5, 2005, pp. 752-757, Professional Book Center, 2005.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL: Model- based offline reinforcement learning. In Proceedings of the 34th Annual Conference on Neural Information Processing Systems (NeurIPS), volume 33, pp. 21810-21823, 2020.
Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In Proceedings of the 38th International Conference on Machine Learning, ICMI. 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 5774-5783, PMLR, 2021.
Ramnandan Krishnamurthy, Aravind S. Lakshminarayanan, Peeyush Kumar, and Balaraman Ravin- dran. Hierarchical reinforcement learning using spatio-temporal abstractions and deep neural networks. CoRR, abs/16055005351 2016.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Sys- tems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 11761-11771, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit underr-parametterizaioon inhibits data-efficient deep reinforcement learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.nett 2021a.
Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron C. Courville, George Tucker, and Sergey Levine. DR3: value- based deep reinforcement learning requires explicit regularization. CoRR, abs/2112.047166 2021b.
Charline Le Lan, Stephen Tu, Adam Oberman, Rishabh Agarwal, and Marc G. Bellemare. On the generalization of representations in reinforcement learning- In Gustau Camps- Valls, Francisco J. R. Ruiz, and Isabel Valera (eds.), International Conference on Artificial Intelligence and Statis- fics, AISTATS 2022, 28-30 March 2022, Virtual Event, volume 151 of Proceedings of Machine Learning Research, pp. 4132 4157. PMLR, 2022.
Sascha Lange, Thomas Gabel, and Martin A. Riedmiller. Batch reinforcement learning. In Rein- forcement Learning, volume 12 of Adaptation, Learning, and Optimization, pp. 45-73. Springer, 2012.
Romain Laroche, Paul Trichelair, and Remi Tachet des Combes. Safe policy improvement with baseline bootstrapping. In Proceedings of the 36th International Conference on Machine Learn- ing. ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 3652-3661. PMLR, 2019a.
Romain Laroche, Paul Trichelair, and Rémi Tachet des Combes. Safe policy improvement with baseline bootstrapping. In Proceedings of the 36th International Conference on Machine Learning (ICML), 2019b.
12

---

Published as a conference paper at ICLR 2023
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: contrastive unsupervised representa- tions for reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 5639-5650. PMLR, 2020.
Alex X. Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. CoRR, abs/2005.01645 6433, 2020.
Lihong Li, Thomas J. Walsh, and Michael L. Littman. Towards unified theory of state abstraction for mdps. In International Symposium on Artificial Intelligence and Mathematics, &f&Math 2006, Fort Lauderdale, Florida, USA, January 4-6, 2006, 2006.
Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin. Percy Liang, and Jennifer Wort- man Vaughan (eds.), Advances in Neural Information Processing Systems 34: Anmual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 18459-18473, 2021.
Cong Lu, Philip J. Ball, Tim G. J. Rudner, Jack Parker-Holder, Michael A. Osborne, and Yee Whye Teh. Challenges and opportunities in offline reinforcement learning from visual observations. CoRR, abs/2206.04779, 2022.
Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in rein- forcement learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
Michael W. Mahoney, Petros Drineas, Malik Magdon-Ismail, and David P. Woodruff. Fast approx- imation of matrix coherence and statistical leverage. In Proceedings of the 29th International Conference on Machine Learning, ICML 2012, Edinburgh, Scotland, UK, June 26 July 1, 2012. ieml.ce/ / Omnipress, 2012.
Shie Mannor, Ishai Menache, Amit Hoze, and Uri Klein. Dynamic abstraction in reinforcement learning via clustering. In Machine Learning, Proceedings of the twenty-firt International Con- ference (ICMI. 2004), Bansf, Alberta, Canada, July 4-8, 2004, volume 69 of ACM International Conference Proceeding Series. ACM, 2004.
Bogdan Mazoure, Remi Tachet des Combes, Thang Doan, Philip Bachman, and R. Devon Hjelm. Deep reinforcement and infomax learning. In Advances in Neural Information Processing Sys- tems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Dipendra Misra, Mikael Henaff, Akshay Krishnamurthy, and John Langford. Kinematic state ab- straction and provably efficient rich-observation reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 6961-6971. PMLR, 2020.
Aditya Modi, Jinglin Chen, Akshay Krishnamurthy, Nan Jiang, and Alekh Agarwal. Model-free representation learning and exploration in low-rank mdps. CoRR, abs/2102.07035, 2021.
Mehryar Mohri and Ameet Talwalkar. Can matrix coherence be efficiently and accurately esti- mated? In Geoffrey J. Gordon, David B. Dunson, and Miroslav Dudík (eds.), Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, AISTATS 201 Fort Lauderdale, USA, April I 11-13, 201 I, volume 15 of JMLR Proceedings, pp. 534-542. JMLR.org. 2011.
Ofir Nachum and Mengjiao Yang. Provable representation learning for imitation with contrastive fourier features. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6- 6-14, 2021, virtual, pp. 30100-301 12, 2021a.
13

---

Published as a conference paper at ICLR 2023
Ofir Nachum and Mengjiao Yang. Provable representation learning for imitation with contrastive fourier features. CoRR, abs/21055.12777 2021b.
Kimia Nadjahi, Romain Laroche, and Rémi Tachet des Combes. Safe policy improvement with soft baseline bootstrapping. In Proceedings of the 17th European Conference Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD), 2019.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Over- coming exploration in reinforcement learning with demonstrations. In 2018 IEFE International Conference on Robotics and Automation, ICRA 2018, Brisbane, Australia, May 21-25, 2018, pp. 6292-6299. IEEE, 2018.
Vinod Nair and Geoffrey F. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (ICMI-10), June 21-24, 2010, Haifa, Israel, pp. 807-814. Omnipress, 2010.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driver exploration by self-supervised prediction. In Proceedings of the 34th International Conference on Machine Learning, ICMI. 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 2778-2788 PMLR, 2017.
Marek Petrik, Mohammad Ghavamzadeh, and Yinlam Chow. Safe policy improvement by minimiz- ing robust baseline regret. In Proceedings of the 29th Advances in Neural Information Processing Systems (NIPS), 2016.
Dean Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural Compur., 3:88 1991.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. In Robotics: Science and Systems XIV, Carnegie Mellon University, Pittsburgh, Pennsylvania, USA, June 26-30, 2018, 2018
Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. In Advances in Neural In- formation Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 11702-11716, 2021.
Harsh Satija, Philip S. Thomas, Joelle Pineau, and Romain Laroche. Multi-objective spibb: Seldo- nian offinn policy improvement with safety constraints in finite mdps. In Proceedings of the 35th Advances in Neural Information Processing Systems (NeurIPS), October 2021.
Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, R. Devon Hjelm, Philip Bachman, and Aaron C. Courville. Pretraining representations for data- efficient reinforcement learning. In Advances in Neural Information Processing Systems 34: An- nual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. (6686-1269,) 2021a.
Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, R. Devon Hjelm, Philip Bachman, and Aaron C. Courville. Pretraining representations for data- efficient reinforcement learning. CoRR, abs/2106.04499, 2021b.
Kajetan Schweighofer, Markus Hofmarcher,
Dinu, Philipp Renz, Angela Bitto- Nemling, Vihang P. Patil, and Sepp Hochreiter. Understanding the effects of dataset characteristics on offline reinforcement learning. CoRR, abs/2111.0477 14, 2021.
Evan Shelhamer, Parsa Mahmoudich, Max Argus, and Trevor Darrell. Loss is its own reward: Self-supervision for reinforcement learning. In 5th International Conference on Learning Repre- sentations, ICLR 2017, Toulon, France, April 24-26, 20/7 Workshop Track Proceedings. Open- Review.net, 2017.
Thiago D. Simão, Romain Laroche, and Rémi Tachet des Combes. Safe policy improvement with estimated baseline bootstrapping. In Proceedings of the 19th International Conference on Au- tonomous A Agents and Multi-Agent Systems (AAMAS), 2020.
14

---

Published as a conference paper at ICLR 2023
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 9870-9879. PMLR, 2021.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2012, pp. 5026 5033. IEEE, 2012.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. CoRR, abs/1 807 00774, 2018.
Han Wang, Erfan Miahi, Martha White, Marlos C. Machado, Zaheer Abbas, Raksha Kumaraswamy, Vincent Liu, and Adam White. Investigating the properties of neural network representations in reinforcement learning. CoRR, abs/2203.159555 2022.
Lu Wang, Wei Zhang, Xiaofeng He, and Hongyuan Zha. Supervised reinforcement learning with recurrent neural network for dynamic treatment recommendation. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19-23, 2018, pp. 2447-2456. ACM, 2018.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity the hypersphere. In Proceedings of the 37th International Conference on Machine Learning, ICMI. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 9929-9939. PMLR, 2020.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. CoRR, abs/1911.11361, 2019.
Chenjun Xiao, Ilbin Lee, Bo Dai, Dale Schuurmans, and Csaba Szepesvari. The curse of passive data collection in batch reinforcement learning. In Proceedings of the 25th International Conference on Artificial Intelligence and Statistics (AISTATS), 2022.
Xin Xu, Zhenhua Huang, Daniel Graves, and Witold Pedrycz. A clustering-based graph laplacian framework for value function approximation in reinforcement learning. IEEE Trans. Cybern., 44: 2613-2625, 2014.
Mengjiao Yang and Ofir Nachum. Representation matters: Offline pretraining for sequential deci- sion making. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 11784-11794, PMLR, 2021.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with proto- typical representations. In Proceedings of the 38th International Conference on Machine Learn- ing, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 11920-11931 PMLR, 2021a.
Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improv- ing sample efficiency in model-free reinforcement learning from images. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Appli- cations of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pp. 10674-10681. AAAI Press. 2021b.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based offline policy optimization. In Proceedings of the 34th Annual Conference on Neural Information Processing Systems (NeurIPS), volume 33, pp- 14142, 2020.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. COMBO: Conservative offline model-based policy optimization. In Proceedings of the 35th An- nual Conference Neural Information Processing Systems (NeurIPS), volume 34, 2021.
15

---

Published as a conference paper at ICLR 2023
Ekim Yurtsever, Jacob Lambert, Alexander Carballo, and Kazuya Takeda. A survey of autonomous driving: Common practices and emerging technologies. IEEE Access, 8:58443-5888 58469, 2020.
Hongyu Zang, Xin Li, and Mingzhong Wang. Simsr: Simple distance-based state representations for deep reinforcement learning. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovalive Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 March 1, 2022, pp. 8997-9005. AAAI Press, 2022.
Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. In 9th International Conference Learning Representations. ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021a.
Chi Zhang, Sanmukh R. Kuppannagari, and Viktor K. Prasanna. BRAC+: improved behavior regu- larized actor critic for offline reinforcement learning. In Asian Conference Machine Learning, 2021 17-19 November 2021, Virtual Event, volume 157 of Proceedings of Machine Learn- ing Research, pp. 204-219. PMLR, 2021b.
16

---

Published as a conference paper at ICLR 2023
Appendix
CONTENTS
Notation
18
Additional Related Work
18
C Comparison
18
C.I Behavior Cloning
18
C.2 -r-irrelevance abstraction
18
C,3 Fourier
20
C.4 ACL
20
Effective dimension and feature rank
20
Proofs
22
E.1 Useful technical results
22
E.2 Theoretical analysis
22
E.3 Performance bound with BPR objective
23
Counterexample
25
G All Experimental Results
26
G.1 Effectiveness of BPR in D4RL benchmark
26
G.2 Results comparison with other representation baselines in d4rl benchmark
28
G.3 Pre-training v.s. Co-training
29
G.4 Performance comparison in v-d4rl benchmark
30
G.5 Performance comparison with representation objectives in V-D4RL Benchmark
G.6 Performance on the datasets collected by the random policy
32
G.7 Representation as the input of Value network
32
G.8 Visual D4RL Benchmark with Distractors
34
17

---

Published as a conference paper at ICLR 2023
A
NOTATION
Table 3 summarizes our notation.
Table 3: Table of Notation.
Notation
Meaning
Notation
Mcaning
state encoder network
A
action space state representation space
f
predictor network
Z
P
initial state distribution
M,
dimension of state space dimension of action space transition probability function
effective dimension
na
7
discounted factor state action reward the feature dimension
T
8
D
offline dataset
V
feature matrix
r
Tg(a(s)
behavior policy of offline dataset the estimated behavior policy discounted state occupancy density state value function the performance of policy the lower bound of the performance
d
Ta(als)
y
state prediction state representation
(s)
2
(*(s)
the numbers of datapoints state space
[(T)
S
I.
B ADDITIONAL RELATED WORK
Representation learning in Online RL State representation learning lies the heart of the em- pirical successes of deep RL. Traditionally, state representation learning has been framed as learning state abstractions aggregations (Andre Russell, 2002; Ferns et al., 2006; Mannor et al., 2004; Comanici et al., 2012), and most of these methods aim to reduce the original state space size and to minimize the system complexity. Recent studies present promising results in learning robust repre- sentations that can then be used to accelerate policy learning in pixel-based observation spaces. Lee et al. (2020); Yarats et al. (2021b) propose to train deep auto-encoders with a reconstruction loss to learn compact representations, Shelhamer et al. (2017); Hafner et al. (2019) learn state representa- tions from predictive losses to maintain trajectory consistency, van den Oord et al. (2018); Laskin al. (2020); Stooke et al. (2021) contrastive losses auxiliary tasks improve performance, Xu et al. (2014); Krishnamurthy et al. (2016); Yarats et al. (2021a) develop state clustering meth- ods to improve sample-cfficiency, and finally Zhang et al. (2021a); Castro al. (2021); Zang et al. (2022) measure state distance by bisimulation similarity to shape state representations.
C COMPARISON
C.1 BEHAVIOR CLONING
One would consider BPR similar to the behavior cloning method that imitates the behavior policy of the data since they both learn a projection from state to action. Despite resembling behavior cloning in form, we emphasize that the goal of BPR is not approximate the exact behavior policy. Instead, is a two-stage process that reutilizes the learned encoder on downstream tasks to accelerate policy learning, while discarding the initial projection layer. Additionally, this allows us to apply ly-oormalization to the prediction and action rather than using their true values. This normalization has been shown to be important for representation learning (Wang & Isola, 2020; Grill al., 2020; Zang et al., 2022). The difference is illustrated in Figure 6.
C.2 π -RRELEVANCE ABSTRACTION
A π -irrelevance abstraction dz. (Jong & Stone, 2005; Li et al., 2006) is such that every abstract class has an action a" that is optimal for all the states in that class that is o (s)) = oz- (82) implies max, (82, which attempt preserve
that Q" (81, a*) max, Q*1 a) and (82,
18

---

Published as a conference paper at ICLR 2023
Behavior Cloning
Offline datasel
State
Policy natwork
Trained policy
(s.a)
tuple
Action
Minimize
MSE
Pradicted
action
Behavioral Prior Representation
Minimize
Action
Normalization
Normalization
MSE
(s,a)
Offline dataset D
tuple
State
Encoder
Predictor
Predicted
action
as. (r)
Offline RL
Irained
tuple
algorithm
policy
Figure 6: summary of the BPR algorithm shows its difference from Behavior Cloning. BPR focus on learning state representation to benefit the downstream Offline RL algorithms instead of concentrating on learning the behavior policy.
the optimal action In ei al. (2006), -rrreleevance abstraction was applied Q-learning and was proven that the induced value function could not converge to the optimal in the ground MDP. While comparing with BPR, the biggest difference is that: in theory, as described in Assumption 2.2, we use the BPR representation only for the policy search, while et al. (2006) assumes that the value function should be estimated for learning reasonable value in representation space. As consequence, Li et al. (2006) requires much stronger assumptions for the policy abstractions, such as bisimulation invariance for the optimal value (policy). As an example, consider an MDP as shown in Figure 7 (an example is taken from the MDP investigated by Li al. (2006)): for x ---irrleevance abstraction, induces an abstract MDP M =< 5, A, P. R. ? 2, where we apply the q learning on the induced M. Under this abstract MDP, since S and S₂ are accidentally aggregated to one state abstraction, the abstract reward R and the abstract transition P will be estimated mistakenly, therefore the q value cannot be updated to the optimal. Notably, the value estimation we perform on the original input space, which belongs to the ground MDP state space, differs from the value estimation the representation space that raises issues that can only be alleviated with the drastic assumptions in Li et al. (2006).
0.5
S
a1
a2
So
S
2
O
S₂
Figure 7: The solid and dashed lines represent actions and 2, respectively. The corresponding graph shows the value function in the aggregated (middle) state for each of its actions. Φx- yields an optimal policy for M that is suboptimal in M, while BPR still find the optimal policy in M by policy search.
"To keep the notation aligned well with ei al. (2006), we will replace T² with n², and replace M with M in this section.
19

---

Published as a conference paper at ICLR 2023
C.3 FOURIER
Nachum & Yang (2021a) developed a representation objective that combines contrastive learning with the linear dynamic model (i.e., learnable function whose input is the pair of next state and action (&', a), the output is a representation of the current state), where the contrastive objective is approximated by leveraging the Random Fourier Features. Though effective, this method assumes the underlying dynamic model is linear, while the BPR objective does not show reliance on such an assumption. Since the BPR objective can be oblivious to the reward function and transition model, it can be applied to a wild range of domains where the reward or the transition could be hard to approximate. On the other hand, without modeling the reward function or transition dynamics, the BPR objective is slightly theoretically weaker (see the counterexample in the Appendix). A combination of BPR and the methods like Fourier should be further investigated in future work, which will provide more theoretically grounded yet still simple approach.
C.4 ACL
Yang & Nachum (2021) introduced Attentive Contrastive Learning (ACL) to learn state representa- tion, which uses the transformer- baaso architecture skeleton, where subset of trajeccoossie are randomly masked to make predictions. Similar to the Fourier approach described above, ACL also utilizes a contrastive learning objective to update its feature mapping, while further reconstruct- ing the action and the reward to stabilize the training. BPR, on the other hand, can be seen as a representation objective that, with only a reconstruction module for predicting actions, is far more module that requires sequential prediction. Another potential ben- efit of the BPR objective is that it can possibly integrate into other kinds of representation objectives due to its independence from the encoder architecture.
simple than the transformer-batt
D EFFECTIVE DIMENSION AND FEATURE RANK
The desirable state representation should not only be able to guide a good policy learning procedure, but also be compact enough to provide concise yet effective information. We focus on measuring the "compactness" the state representation to show that BPR suffices for the agent to benefit from learning representation with auxiliary representation loss, which can alleviate the "implicit under- parameterization" phenomenon.
Assumption 4. (Feature reachability) Denote Amin(A) as the smallest eigenvalue of positive- definite matrix A. With a mapping function ø², we assume that there exists E R+ such that,
sup Amin (Ez=at)()) [00(s) (s)Ø² (s) (1) 2
Assumption 4 posits that in MDP M, for each latent state in representation space Rd. there exists a policy that reaches with a non-zer zenzeero probability. This ensures the optimal state representation should have no redundant dimensions, which is reasonable assumption for compact latent space. While a similar assumption is commonly made in previous work (Agarwal et al., 2022; Modi et al., 2021), seldom of them consider leveraging it to measure the feature effectiveness. Based on this assumption, we define the following measurement, inspired by the Feature rank defined in Lyle al. (2022):
Definition 2. Effective Dimension Let ED(M) denote the multiset of eigenvalues of square matrix M. Then the effective dimension of V is defined to be
(VISI V (V w) | 1
<(V, e,e) = ED OOEED
-ED
(8)
where V E (S)AA) is the feature matrix with respect to the output of the penultimate layer of the state-action value network.
Lemma 5. Let X,, C [RSSIAA be a set of n state-action pairs in RIS||AA sampled from a fixed distribution d (s, a), and the corresponding feature matrix being Un a consistent estimator can be conducted as:
((V,D,e) FOEDD ((t.
(
Vn Vn )) c
20

---

Published as a conference paper at ICLR 2023
Proof. The following proof mimics the derivation in Lyle (2022). Recall that
(+2
1 Un
v =
(x;)))))
(10)
The following property of the expected value holds
ED (M))) = E
(11)
Then, consider each element the matrix M:
(+( = My = (vb(())
th (mx) b/y (ox) 3 Mi
(12)
Since we have convergence for any Mig. we get convergence of the resulting matrix to M. And since the eigenvalues are continuous functions of M, the eigenvalues of M., converge to those of M almost surely.
Intuitively, possesses similar equivalent form of feature reachability. Empirically, we e as 0.01 where only higher eigenvalues are considered, eliminating the distraction caused by small noise. With the effective dimensions measurement, we can show that BPR can stabilize the training procedure, and provides a higher effective feature dimension for the value network, thus helping to alleviate the "implicit under-paramettrizationn" phenomenon. Notably, Equation can also be used to measure the effective dimension of the state representation, i.e., C(Φ, D, e), when we sub- stitute the feature matrix V by the representation matrix T [|| where 2 is the latent state representation space.
Connection to the effective dimension proposed in Lan et al. (2022) To develop measure- ment the effective dimension of state representation, Lan et al. (2022) borrow the concept of j--coherecce (Candès & Recht, 2009; Mohri & Talwalkar, 2011), which is related to the statistical leverage scores (Drineas et al., 2006; Mahoney et al., 2012).
Definition Given an arbitrary n matrix with n d and rank(A) = r, let denote the x d matrix consisting of the d left singular vectors of A, and let U(i) denote the i-th row of the matrix U as row vector. Then the statistical leverage scores given by ([ for i {1...., the coherence is: U max (1) iE(1.-.,, .n)
(13)
i.e. the largest statistical leverage score. When we let Py be the orthogonal projection onto and (ex) the canonical basis, we can define p -coherence as:
(U) max (E(I....n)
|Poe;112 = max |UUTe,11" iE(I.". ..)
(14)
max iE(l....)
|U eill=122 == max xE(1,...m)
[1
Following the above definition, Lan et al. (2022) defined the effective dimension as: Definition 4 (Effective dimension in Lan et al. (2022)). Let E S a feature matrix. The effective dimension of (vis-a-vis the standard basis (e:)) defined as the quantity
( S max (P
(15)
where Ps is the orthogonal projector onto the column space of C is the number of the state.
Note that for any feature matrix, the smallest der can be the rank of the state space, achieved, for example, if & is spanned by vectors whose entries all have magnitude 1/VS, meaning that the fea- ture matrix is perfectly learned. The largest possible value for der is S. which would correspond any subspace that contains standard basis element, meaning that the state space is full-rank.
21

---

Published as a conference paper at ICLR 2023
As comparison, effective dimension in this paper and the one in Lan et al. (2022) are both devel- oped from the Singular Value Decomposition (SVD) that was generally utilized for dimensionality reduction', the difference is that compute the number of the eigenvalue of the Gram matrix of the feature matrix as the measurement, while Lan et al. (2022) compute the largest leverage score of the feature matrix that considered the effective dimension in Lan et al. (2022) can be seen as variation of Equation 8 with O, which also indicates that the small disturbance component of the feature matrix cannot be discarded in Equation 15 without prior knowledge of the rank of the state space.
the left singular vectors of the state space. In some sense,
E PROOFS
E. USEFUL TECHNICAL RESULTS
We begin this section by introducing Rademacher complexity, which can be used derive data- dependent upper bounds on the learnability of function classes.
Definition 5. Let be any set, X1, X.. ,nn be i.i.d. random variables with values in X. We have G a class of functions X + R. The quality
1
Rad (g) = Exy.a,
sup REG
Σ og (X₂)
(16)
called Rademacher complexity of the class G on the sample X X2, .,,) X", where is the Rademacher random variables that are drawn uniformly at random from {+]}.
The corresponding vectorized version of Rademacher complexity is:
Rg) == Exy,03
sup gEg n
ji (X²);
(17)
/=1 1=1
where G is class of functions g : R
E.2 THEORETICAL ANALYSIS
Assumption 1. (1.1) Access to the true behavior: F3 Tg. (1.2) Access to the true performance of
policies: ( = J (T) for all policies (1.3) The embedding & allows to represent the behavior
policy estimate: TT(alo(s)) xg(ass) E IgeR- (1.4) The algorithm performs perfect optimization:
Jam max J (T²..
Theorem 2. Under idealized Assumption 1.1-1.4, BPR returns a policy that improves over the behavior policy: Jurk Z I (Tg).
Proof.
Jnma = arg max I (T) r
(Assumption 1.4)
max J(x)
(Assumption 1.2)
rell..
2 ((T
(Assumption 1.3)
2 (TT)
(Assumption 1.1)
which concludes the proof.
Assumption 2. (2.1) Access to an accurate estimate 3 of the true behavior B: [(x)) - - <
Eg. (2.2) Access to accurate estimate An,z of the true policy improvement aR.ag for all policies
E 1p O Hgps: Ân, AR. < c..
'the singular values of the feature matrix are the eigenvalues of M in Equation 12 when we only consider the state feature
22

---

Published as a conference paper at ICLR 2023
Theorem 3. Under Assumption 2, BPR returns policy ThPR with the following performance
bounds: Jura 2 [ (TT) + Mars. Ea Eg.
Proof.
JAPZ = I (TTeK)
(by definition) (Assumption 2.2)
2 (TTz) + Nomn cs
2 [TT) + A TuniT CB
(Assumption 2.1)
which concludes the proof.
Assumption 3. (3.1) access to lower bound of the performance (x) S [T) for all policies,
which is tight for B: J. (Tg) 2 [(TT) -
Theorem 4. Under Assumptions 2.1 and 3.1, BPR returns policy Tare with the following perfor-
mance bounds: Jara 2 I (TT)) + 7 c_ C3.
Proof.
JaPa
I BPRR
(by definition) (Assumption 3.1) (by definition)
J (Tz)
+ A Ta
2 I (TT))
A A E
(Assumption 3.1)
2 I (TT)) + A A Hunn
(Assumption 2.1)
-
which concludes the proof.
E.3 PERFORMANCE BOUND WITH BPR OBJECTIVE
BPR solve the optimization problem:
min ni
- aill D.
(18)
where (st, a;) is i.i.d. sample from the offline dataset with the size of
The difference between the estimated behavior policy and the true behavior policy is:
E [((())) (TT) Eera [V (s) - V (s)].
(19)
Lemma 5. Consider fixed dataset 'D and the reward bounded by Pmmar then the difference
between T3 and will be bounded as:
EE ((T)))((TT)(D)))))
< 2Rmax
(1 (1-7)² E [Drvv (Ta())lIT(ss) [D]
(20)
Proof. Recall that the total variation distance (TVD) between the estimated policy Th and the be- havior policy Fy for a given state is defined as:
DTv(Tg(s), (s()) sup .E.A
Ta(als) Te(a|s) = Σ ( -T(a|s))
(21)
We expand the expectation in the value bound:
Ex-p ( - V *s))
Σ p(s) Σ a (a 8) - 29 (a s) (((s,))+ + TT. v (s)))
p(s)
xz(a|
- To(a |s|
R(s,a)+ + + TTs,,,, - aa (s'))
(22)
23

---

Published as a conference paper at ICLR 2023
Apply the upper bound of the value function Rmax/ /11 7) the above equation:
E ((TT)
(TT))
Boup [V*
(s) VTT
Rmax
B Banp
Σ
TT(a 8) TT(a
[ 6s)
(23)
2
Bann Drv (Ta(S)lTT(sS) [D]
1-y
2Rmax
TE
Bandtr (DVV (TE(S)ITT(s))]
(1 7)2
For simplicity, we denote Exnd"H
Drvv (T
as E Drv (T)(5))(T(()) |D]
This lemma tells us the suboptimality of an arbitrary policy is upper-bounded by the TVD between the optimal policy and itself over the dataset D.
Consider we learn the representation mapping & S + Z, and we learn policy Rs 2 + A based on the fixed mapping 6, instead of learning the policy π : :S + A. The following theorem will provide upper bound of the suboptimality of the policy TØ?
Theorem 5. With probability at least for any d E (0, 1):
EB = - - (
D < CK-
TT(((ss) - TB d(sii)
(24)
2 1
+ 2V2K. Rad( + K
where n is the size of the dataset, Rad is the Rademacher complexity, Ty is the behavior policy over the dataset, C is constant, K Ras..
Proof. The following proves build on techniques provided by Asadi et al. (2020). First, for Vo, we have:
E Ey TT(. (s) ---a-- (
1 n
Σ
n& (. 8;) - -xa( ((())))] (
sup 6E
E Exnats
TTa (2(--1))) TT(. (()))
|
THe (: 8:) - T (. (b(s)))
(25)
Lemma 6. (Asadi et al., 2020) The expected value of E can be bounded as:
E
[ S
2v Rad(
(26)
where Rad is the Rademacher complexity.
Given the fact that [E (S1..... Si... Si....a. 8n) E (s1,..., S.....
sn) S 2 and apply McDiarmid's in-
equality, vg 1), we have:
2 In z
Pr
E E
- 8
(27)
Combining Equation 26 and Equation 27 with Equation 25, we get the following holds with proba- bility at least 1-8 8:
2 1
Ep [ TT(- - TT(. r
To (.1 st) TB ( +2 (a) +
5
n
(28)
NNte that will abuse the notation of ie. 7 (db()), and ((-1(()) for readability and simplicity.
24

---

Published as a conference paper at ICLR 2023
Now from the left-hand side, with the help of Lemma 5, get:
E (TTg) [ < 2
EE Div (Ta(+)lIT(ss)) P] 7)2
(29)
Rya
Ep [T( - T 1
And from the right-hand side, have:
2 J
T3 (.)s.) TT ( (6:1)))))// + 22 Rad())) +
(30)
n S
21n
Σ
Th 8:) - T (- (-1((s))) 2-22 Rad(
2
With combining Equation 29 and Equation 30 together, can conclude the proof:
c/ = E [(TT)) (TTD)
D. S C
Rmax
TT(-1a)) TB ( ((ss)
(31)
Raax
Ranx 2 1
2v2
1 Rad(
+ 1 7
with probability at least 8.
Note that the second term and the last term do not depend on the exact form of the representation. Minimizing the first term decrease the difference between the learned policy and the behavior policy, where the first term itself aligns with the optimization problem of BPR (Equation 4). indicat- ing that with BPR objective, we can improve the performance of the policy. And the potential harm of using BPR is bounded by the errоr Eg that we can control.
F COUNTEREXAMPLE
We consider the following dataset composed of four trajectories T1, 72, 73, and 74 in deterministic MDP (1so, 81). {ao, )1 po()) I,p.r, 1):
==
(so, oo, sy. 0), (so, ao sy, 0), (so: 01 $1 0), (s1, ao. sy, sy, 1), (so, 01
01, $1 0), (81, a1 sy,
where the final state sy denotes the termination of the trajectory. Then, BPR may collapse 8o and 81 to single embedding where the estimated behavior policy is uniform:
Nz(cul2) = TT(01 (z) TT(aolso) =....
(32)
Then, have:
1
[(Fz)) 4
(33)
Jara (T( 1)
(34)
Jara 1) 0
(35)
2
Jora = 0.5) =
x
+
x
x Japn(T((ol2)) = 0.5)
(36)
2
3
2
3
1
+
x JaPR (T (apl 22) = 0.5)
(37)
6
3
3
1
(38)
2
4
As a conclusion, with a BPR value, a greedy algorithm would converge to <(aol2) = 1, which has the performance of in the environment MDP m, and is therefore not policy improvement.
25

---

Published as a conference paper at ICLR 2023
Algorithm BPR Pretraining Pseudocode, PyTorch-like
encoder: mlp, encoder network
predictor: mip, prediction
def pretzain (encoder, predictor, zeplay_buffer, batch_ steration -
for iteration < 1e5:
state, action, -
(batch_size) # sample batch of
cuples from replay buffer
state encoder (state)
prediction predictor (state) pradiction normal iza
dim=1, p=2). 4 12 normal iza
action normslize fact ion, dim-l, p=2) #12 pormslize
encoder_loss MSK (prediction, action) loss for encoder eneader_lode
haw.ddr__oossshccwaeeeeeeee backward
update (encoder,, iteration +-
G ALL EXPERIMENTAL RESULTS
G.1 EFFECTIVENESS OF BPR IN D4RL BENCHMARK
Experimental Setup We analyze our proposed method BPR on the D4RL benchmark (Fu al., 2020) of OpenAl gym MuJoCo tasks (Todorov et al., 2012) which includes variety of dataset do- mains that have been commonly used in the Offline RL community. We evaluate our method by integrating it into three Offline RL methods, including: i)TD3+BC agent (Fujimoto & Gu, 2021), which one of existing state-of-the-art Offline RL approaches combining Behavior Cloning with TD3 together and has good balancing the simplicity and efficiency; ii) CQL agent (Ku- mar al., 2020), which learns conservative Q-function such that the expected value of policy under this Q-function fower-bounde its true value; iii) EDAC agent (An et al., 2021), which is an umcertainty-basee mnsemble-diversffie Offline RL algorithm. We consider three simulated tasks from the Mujoco control domain with D4RL benchmark halfcheetah, hopper, and walker2d) and consider the expert and medium-expert datasets. For our experiments, the baseline methods take the raw
the input of both the value network and policy network, while we also pre-train the representations with 100k timesteps and then freeze the encoder and take it as the input during the Offline RL optimization; and follow the standard evaluation procedure as in D4RL benchmark. We provide the pseudocode of the pretraining process in Algorithm 1, and the training curve that without the accounts for the pre-training steps (100K timesteps) BPR (Figure 8).
EDAC (pretrained
TD3+BC (pretrained
COL (pretrained by
EDAC expert
TD3+9C hopper_expee expert
CQL
halfcheetah, halfcheetah_expert expert
Cumulative Returns
24
24
09
04
expert
hopper medium expert
halfcheetah mepeut_expert
6
68
13
69
62
64
Timesteps
Figure 8: Performance comparison on D4RL dataset.
26

---

Published as a conference paper at ICLR 2023
Forward
Contrastive
Super
ant expport
medium, expert
ant medium, replay
art mediuu
06
165
64
12
walker22, expert
walker2d medium expert
walker2d, medium replay
walkrr2d medium
Cumulative Returns
22
25
10
14
10
02
04
04
50
hopper expert
hopper medium expert
hopper medium replay
hopper medium
halfchestah
halfcheetah
halfcheetah medium
halfcheetah
Timesteps
Figure 9: Performance comparison D4RL mujoco dataset.
ant-expert-v2
antemeedlum-xxpprtevz
antemediun-reellyyy2
ant-medium-v2
Con
-00
heepchee halicheetall-medlumeeeertty
hallcheetth-mmdduurreaaaaa
halfchectal-medium-zz
ropper-expeert-22
Nepper-meedummeeppeer
hopper- madpeer-eedlmmmmmm reolayyy
mopper-medlun--2
Walker20 v2
walker2d medium expertiv2
walkerr2d medium r2
walkcrr2d medium V2
Touree
100
200
Figure 10: Performance comparison with different representation objectives on D4RL. Mujoco dataset 100K timestep. x-axis shows the average normalized score the final 10 evaluations and seeds. Blue dotted lines show the average normalized return without pretraining.
27

---

Published as a conference paper at ICLR 2023
G.2 RESULTS COMPARISON WITH OTHER REPRESENTATION BASELINES IN D4RL BENCHMARK
Experimental Setup The purpose of this experiment is to demonstrate that, when compared to the state-of-the-ait representation objectives, BPR still has competitive advantage. In this experi- ment, we follow the veetrain-finetum paradigm suggested by Yang & Nachum (2021), i... we first pretrain state representations of 100k time steps, then the learned encoder is fixed and applied to the downstream task, which performs the BRAC (Wu al., 2019) agent for 100K steps. Same as the configuration from Yang & Nachum (2021), we also fix the regularization strengths and policy learning rates in BRAC for all domains. And since representation is more valuable in informative data, our experiment does not include interaction pairs collected by a random policy, but rather has interests in the dataset collected from a learned (or at least partially learned) agent. The experiment is still conducted the D4RL mujoco tasks, with 16 tasks in total included. In this setting, the pretraining datasets and downstream datasets are the same, determined by a single choice of task. We compare method to several published leading representation RL algorithms, which include:
ACL (Yang & Nachum, 2021) applies contrastive learning the wansformer-basee archi- tecture: (1) take a sub-trajectory St.t+kr at:+k Te.c+k. (2) randomly mask a subset of these, (3) pass the masked sequence into transformer, and then (4) for each masked input state, apply a contrastive loss between its representation (s)) and the transformer output at its sequential position.
Fourier (Nachum& & Yang, 2021a) is implemented as contrastive learning where the transi- tion dynamics are approximated by an implicit linear model with representations given by random Fourier features.
Forward model (Pathak et al., 2017) uses the state representation and the action to predict the reward and next state given a sub-trajectory, where the transition probability is de- fined entropy-based model, i.e., given sub-trajectory Ta.+1, use (1s)), az predict $t+1, Tt-
Inverse model (Pathak et al., 2017) uses the current state representation and the next state representation predict the current action, i.e., given a sub-trajectory Th-++1, of (((c++1) to predict az.
Contrastive (Yang & Nachum, 2021) learns state representation with contrastive objective. Given sub-trajectory Tss+1, contrastive loss is applied between (sx,,, (($++1) as:
- (s142) W (st) + log E, [exp { {6(5) W o (s:)
(39)
Super (Yang & Nachum, 2021) a combination of the forward model and the backward model, which has a self- self-predictive module and inverse module to learn dynamical informa- tion.
In this experiment, all methods use the same encoder architecture, i.e., with 4-layer MLP activated by ReLU, followed by another linear layer activated by Tanh, where the final output feature dimension of the encoder 256. Besides, all methods follow the same optimizer settings, pre-training data, and the number of pre-training epochs.
Analysis Figure 9 and Figure 10 provide the performance comparison results for BPR and other representation objectives D4RL mujoco dataset. We find that BPR outperforms or at least is competitive with the previous state-of-the-art methods in the majority of environments. Notably, the variance on expert and medium datasets is much smaller than that on mecdium-expent and medium- replay datasets, especially for BPR, which indicates that the learned representation is more stable when the dataset from the same level of the agent. The underlying reason may be that the behavior policy of the mixed dataset could differ even in the same mini-batch in the training stage, which will limit the ability of the neural network to learn the information related to (near) optimal decision- making. This suggests that the representation quality could be impacted more by the characteristics of the dataset, than the representation objectives, which confirms the observation of Schweighofer et al. (2021). We also notice that even with the pretrained encoder learned from BPR, there still re- mains large performance gap between the BRAC agent and the baseline TD3+BC agent, indicating that despite the representation objective being helpful for Offline RI, choosing the suitable offline
28

---

Published as a conference paper at ICLR 2023
agent for different tasks is still essential for improving the performance, which may also partially solve the aforementioned high variance issue of the different dataset. Furthermore, to understand
the performance difference of different representation objectives comprehensively, we additionally use uncertainty- comparisons for all these methods.
G.3 PRE-TRAINING V.S. CO-TRAINING
TD3+
TD3+BCIco-training with BPR)
TD3+BCCpretraining with BPR)
walkerr2d expert
hopper expert
halfchestah, expert
333
100
Cumulative Returns
102
43-
24
02
3.8
35
05
64
0.0
13
03
82
04
35
10
warker medium, expert
hopper medium, expert
halfcheetah medium meeiuumeexpert
139
1:00
30
82
10
Timesteps
Figure 11: Performance comparison on D4RL mujoco dataset. The horizontal axis indicates the
number of environment steps. The vertical axis indicates the normalized cumulative returns. We trained 6 different instances of each algorithm with different random seeds, with each instance per- forming an evaluation every 5,000 environment steps by computing the average episode return 10 evaluation episodes. The solid lines represent the mean over the 6 trials.
Experimental Setup There are two design choices of representation learning in the experiment: i) learn state representations via pretraining, then freeze the encoder and apply it to the downstream tasks, ii) learn state representations simultaneously with the policy, where both the representation and the policy will be learned choulder-by-shoute different designs of choices, and what the difference between the latter one and the one that addi- tionally applies BC to policy learning. To illustrate both questions, here, evaluate both designs the state-of-the-ar TD3+BC backbone. For the pretraining one, we still set the pretraining stage as 100,000 time steps. For the co-training one, we train the state encoder simultaneously with the value network and the policy network, where the corresponding pscudocode is listed in Algorithm 2. For reference, we also provide the pseudocode of the baseline TD3+BC method Algorithm 3. Con- cretely. the differences between BPR and the behavior cloning in TD3+BC are listed as follows: i) they use different projection (or actor) heads, ii) the gradient of the policy loss is non-visine non-visible for the encoder, but the gradient of the encoder loss can pass through and update both the encoder layer and the projection layer; iii) BPR require the 12-normalization for the representation and the action, which is known to be effective for representation learning, while lz-oomaalization is not good fit for action in BC.
One can wonder how BPR performs with the
Analysis Figure 11 shows the performance of three models on D4RL tasks, indicating that BPR objective can improve sample efficiency in most cases. BPR consistently outperforms the baseline method, TD3+BC, for the majority of tasks and has a considerably lower variance, demonstrating that our approach has the advantage of convergence rate and stability. On the walker2d tasks, the baseline method performs better than the model with BPRs, we consider the possible reason could be that the baseline is sufficient to solve this task, therefore adding extra auxiliary losses may, in turn, damage the performance. Meanwhile, this finding is consistent with our hypothesis that the representation objective becomes more valuable as the difficulty of the task increases. Another impressive result is that compared to co-training BPR objective with the policy, BPR yields more performance gains and faster convergence under the pretrain-finet finetune paradigm, illustrating that learning policy from "good" fixed encoder might be more suitable for Offline RI..
29

---

Published as a conference paper at ICLR 2023
Algorithm 2 TD3+BC Co-training with BPR Pseudocode, PyTorch-like
mlp, poy natwork critic: mip, value network
actor:
encoder: mip, encoder network
predictor: mlp, prediction network
Ereq: update Irequency of policy network
iteration
for as,
loader: load a minibatch tuple with samples
pred predictor (encoder
nerm_pred normalize ipred_s, dim-1) 12-normalize norm_a = normall le dim=1) 12-normaliza
[_encuder MSE (norm_pred, aorm_a) MSE loss for enceder l_encader. iLencoder.backwardyy
updace (encader, predictor)
Adam update
enc_a, enc_ns - sy (ancoder (encodez (ns)) $ sg is acop_gradient target_v
semmansgg/riitee (enc_ns,
i_value XSE (critic (enc_s,a), target_v) MSE loss for value I_value, backkarr...kkrr.))
updacalcrition Adam update
if iteration freq - 0:
actor(eno_s)
paplicy -Imbbav0 (ene pi). mean ()+ (pi, F loss for policy E.policy.backwere update (astor)
propapgee
updace
iteration +-
Algorithm TD3+BC Pseudocode, PyTorch-like
actor: mlp, policy network
critic: mlp, value natwork
frog: update frequency of policy metwork
iteration
for n5, in loader: F load a minibatch tuple with samples gamma-sy
target_v i_value MSE (critic (ha,a), zazget_v) MSE losa for value l_value.ba backward
(m),
updateleritic) Adam update it iperation $ freq -- C: actor(ns)
1_policy -imbbax0 (es, pil pil.mean + MSE (pi,a) loss for policy backward)) brop-proeagate
updato (astor) #t Adam updace
Steration +-
G.4 PERFORMANCE COMPARISON V-D4RL BENCHMARK
We evaluate our method on a benchmarking suite for Offline RL from vi- sual observations of DMControl suite (DMC) tasks (Lu et al., 2022). We consider three domains of environments (walker-walk, humanoid-walk, and cheetah-run) and two different datasets per vironment (medium-exper and expert). We include two baselines from Lu et al. (2022) in this experiment:
Experimental Setup:
DrQ+BC: combining data augmentation techniques with the TD3+BC method, which ap- plies TD3 the offline setting with a regularizing behavioral-cloning term to the policy loss. The policy objective is: argmaxE (s.a)- [XQ(s, (s)) (I(() -a?
DrQ+CQL: adding the regularization in CQL to the Q-function of an actor-critic approach, which is DrQ-v2.
To investigate if BPR learns valuable representation, we pretrain the encoder by using the BPR objective for 100k time steps, followed by fine-tuning stage where we keep the encoder fixed and apply two baselines based on the learned encoder.
Analysis: Figure/ 12 shows the performance of the four models on V-D4RL. We emphasize that these tasks are more challenging than their conventional D4RL counterparts. Neither DrQ+BC DrQ+CQL solve them without further modifications. By integrating the frozen encoder learned from BPR, both of them obtain significant performance gains. Note that DrQ+CQL typi- cally requires longer training time achieve similar performance DrQ+BC. Nevertheless, the
30

---

Published as a conference paper at ICLR 2023
substantial improvement of the modified variant of DrQ+CQL suggests that BPR can be effective in improving sample efficiency, decrease computational cost, and reach better final performance.
G.5 PERFORMANCE COMPARISON WITH REPRESENTATION OBJECTIVES IN V-D4RL BENCHIMAR
DrQ BC (protrained BPR)
DrQ+BC
DrQ (pretrained BPR)
DrQ+CQL
cheetah expert
humanoid walk expert
walker walk expert
Cumulative Returns
02
34
36
00
60
32
66
3.8
10
33
02
06
60
cheetah run medium expert
humancid walk medium expert
walker walk medium expert
100
90
02
34
Timesteps
Figure 12: Performance comparison on v-d4rl benchmark, the results are averaged over 10 evalua- tions and 6 seeds.
Experimental Setup We evaluate method with five other representation objectives that considered as state-of-the-art methods on a benchmarking suite for Offline RL from visual observa- tions of DMControl suite (DMC) tasks (Lu et al., 2022). The baselines include:
DRIML (Mazoure et al., 2020) and HOMER (Misra al., 2020) (Time Contrastive meth- ods) learn representations which can discriminate between adjacent observations in roll- out and pairs of random observations.
CURL (Laskin et al., 2020) (Augmentation Contrastive method) learns representation that is invariant class of data augmentations while being different across random example pairs.
Inverse model (Pathak et al., 2017) (One-Step Inverse Models) predict the action taken conditioned the previous and resulting observations.
SGI (Schwarzer al., 2021a): A combination of three kinds of techniques (Self-Prediin Predictive Representations + Inverse modelling goal-conditioned RL). As SGI was not tested on the Deepmind Control suite (continuous-action space) setting, modified its code to allow it to be fairly compared to BPR. Specifically, for the inverse model, we follow the same architecture the above Inverse model; for the self-predictive representation, we design simple MLP network acting as an inverse model to capture the dynamic information; for the coldiinddt parts, we utilize the same architecture of FiLM module in DRIML, apply DrQ+BC as the backbone, and define the potential- based reward of goal-conditioned RL to pretrain the state representations.
In this experiment, all methods use the same encoder architecture, i.e., with four convolutional layers activated by ReLU, followed by linear layer normalized by LayerNorm and activated Tanh, where the final output feature dimension of the encoder is 256. Besides, all methods follow the same optimizer settings, image augmentation, pre- pre-training data, and the number of pre-train pre-training epochs. We provide pscudocode for each method in Algorithm 4-7.
'https://gi thub. com/mi 1a ae--iii//ss
31

---

Published as a conference paper at ICLR 2023
Algorithm 4 DRIML(HOMER) Pretraining Pseudocode, PyTorch-like
encoder: CNN, encoder network driml_ net: mip, def (eecoaii.tenzoderr,, drimi_net, replay_buffer, batch_ zemp-0.1): steration -
block residual black
for iteration < 1e5:
state, action, state_k_step,
butfer, (batch_size)
state_k_stt
the step future ebservation from the ourrent state, where x=1 in HOMER and k-5 in DRIML
state encoder (statee state_k encoder kstats_
u,t, t_tpk drimi_ (state, state_k, action)
outer_prod torch.einsum( ik, jk->i1', uLt, b_tsk)
scares - wofttxxxxxxxtrrrr.oooooo
temp, -2) temp is she temporature
mask (yccheeyesscres..eaee
acores - (mask
encoder_lose -szores loss for encoder
encoder www.:_loe...ccaaae......
update (eniooer,, driml_net) icaration
Algorithm CURL Pretraining Pseudocode, PyTorch-like
encoder: CNN, encoder network 4 W: macrix-wise parametera
pretrain (encoder, W, replay_buffer, batch_size, temp-0.11: itrration = iteration
state, action, next_state
replay SarFlla_bufffr.co bafferrrr.sal (batch_size)
obs augment istate.flost (1)'e augment
pes augment (torch. wwwee (state) float ()) augment
encoder lobs)
2_pos encoder (pas, ema-True) using
encoder
logits torch.matmsl [z_a, corch.matmul (K_param, z_pos.T)
logits logits torch.max (logits, [0] None]
labels torch. slogitsssaape [0]] [0])
encoder_loss wnn.CrrasRnnrrpyl..sss () (logits, 1sbels)
engader_lose bacckard los..backwaard
update (encoder, [_param) icaration +-
G.6 PERFORMANCE ON THE DATASETS COLLECTED BY THE RANDOM POLICY
should be noted that in this work, we assume the inductive bias of the behavior policy is efficient. In particular, a bad behavioral policy may lead BPR to encode an undesirable bias and therefore deteriorate the performance of the algorithm is paired with. To verify this, we run BPR on data collected using random policy and report the results in Figures 13 and 14. We see that the gains provided by BPR on datasets where the behavior policy is informative vanish.
Super
Cumulative Returss
Timestaps
Figure 13: Performance comparison on D4RL mujoco dataset where the data is collected by random policy.
G.7 REPRESENTATION AS THE INPUT OF VALUE NETWORK
Experimental Setup As we stated in Section 5, Assumption 2 forbids theory, the use of the BPR representation to estimate the value. In this experiment, the purpose show that the utilization
32

---

Published as a conference paper at ICLR 2023
Algorithm 6 Inverse model Pretraining Pseudocode, PyTorch-like
encoder: CNN, encoder network
backward_ned mip, inverse model
def pretzain (encoder, back reglay_bufter, batch_size: steration -
for iteration < 1e5:
scate, action, next_atate, state encoder (state)
replay_buffer wtt.:pbuutrrrrralalee...aee.. (batch__see))
next_ encoder (next_atate) statel
state_cat torch. (Istate, next_atate), dim -1)
act actior_has hat = backward net (obs_cat)
norch.normisee cormisction satien dies-1, 2=2, keepdim-Zrue) backward_erres backward_error.. mean
encoder_los,
eneader_lose hws.loss.bbee w.s.ss.ckaardll
update (encoder, R_ paraa))) iteration +-
Algorithm 7 SGI Pretraining Pseudocode, PyToorhh-like
encoder: CMN, encoder network
predictor: miy, predition head for forward model
forward net: mlp, forward model
backward_net: mip, inverse model
4 goa goal_conditioner: mip, FiIM block
for goal -conditionnd fake_critid critic for goal-conditioned RL
fake_actor:
det (encoddeer predictor, forward_net, backward_net, gpal_conditioner, fake_actor, fake_critic, replay_buffer, batch_sixe):
iteration -
for iteration 1e5:
state, actien, stxt_state, inp_state state
replay (batch_size)
inp_next_state next_stace
state (state)
next_state = target_ encoder mext__satee & updated with EMA from encoder goal targst_encader (goal) updated with EMA from encader
sa_est = torch.cat (state, action), dim = -1)
next_state_han predictor (self. fowwd.orwaadddddddd..oooo.....d. net net (sa_sat))
state_cat torch.cat ((state, next_atated dim - -1) action_hat backward_net (state_cat)
forward torch. backward_errer normacioomo action_hat, dine- 1 2_3 goal_canditiannet (state, goal)
next_state_hatt din, dim--1,p-2- keepdim-True)
with terch.no_sradll
gusl_reward exp distancu (next_ state, goall distainn (state, geal) next_action fake actor sample mext_statee target_01, target_02 - fake_ critic_tateee target (next_ next_action) target_V min(targtt__1 target_02) target_C goal_reward + (diacount target_V)
Q1, Q2 fake_criticca.. (2_9, action) loke_eritic_lss_ MSE (Q1, target_0) MSB target_2)
encoder_loss fake_critic__
forward_error. backward_error,
encoder_loss lostr_loss.baar baw.k/awaa..bb update pradcccer,,rediiosss
forward_nct, backward_act, goal_ coul_conditiener, fake_critic)
state encoder (ing_ettae
lake_acttorsamee (state)
Q1, fake_critie critic(state, pred_action)
Q morch.minf01.. Q2)
actor_policy_impe wwt.iiimroveee___uee loss -Q.mean
bcter_be_lose loss 1oss (pred_action, action]
fake_actor_lass eeter_policy_jipreeeee
lambda actor actor_be_ losss
fake_actor_less less
bactor_lass.bbecaard) backkard
update (fuke_actor) iceration +-
of taking BPR representation the input of the value network is acceptable empirically. We still
pretrain the encoder and then fix it, while developing two variants of the architecture afterward: i) the learned encoder is only used for training the policy network, while the value network takes the
33

---

Published as a conference paper at ICLR 2023
CARL
OPENE
HONER
Cumsiation Hasaum
Figure 14: Performance comparison on Visual-D4RL mujoco dataset where the data is collected by a random policy.
raw state as its input; ii) the learned encoder is used as the input of both the policy network and the value network.
Analysis As shown in Figure 15, the performance of both variants are almost equivalent, which means that o(s) good enough to learn the state-action value empirically. Notably, when taking ((s) as the input, the agent will have a lower variance along the training procedure in 2 of tasks, which is the empirical evidence of that ((s) as the input of value network is an acceptable choice.
#15) as input of V
as input of V
walker2d, export
hopper_exe expertt
halfchestah
303
Cumulative Returns
wal ker2d medium, 1
hepper_mediune medium, exedit
halfcheetah halichectah_medium medium expert
100
Timesteps
Figure 15: Performance comparison D4RL mujoco dataset, (88) v.s. input of value network.
G.8 VISUAL D4RL BENCHMARK WITH DISTRACTORS
Easy
Medium
Hard
Figure 16: Pixel observations on visual d4rl benchmark with distractors.
We use three levels of distractions (i.e., easy, medium, hard) to evaluate the performance of the model (See Figure 16. Each distraction represents shift in the data distribution, where add task-irrelevant visual factors (i.c., backgrounds, agent colors, and camera positions) that vary with environments to disturb the model training.
34


Reviews:
[{"review_id": "tIAbHAwE0jS", "reviewer": "Reviewer_yZFP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper lies along the line of work exploring how to combine BC and offline RL. It takes the perspective of using BC as a state representation learning method, proposing to train an intermediate representation in the offline RL agent via BC (either as pre-training or co-training). Experimental results show that the proposed method outperforms other state representation learning methods, and offline RL from scratch, on the visual D4RL benchmark in “expert’ and “medium expert” modes.\n\nContributions:\nProposing BC as a state representation learning algorithm for offline RL\nTheoretical analysis from a representation learning perspective\nEmpirical comparison to other state representation learning algorithms with visual inputs", "summary_of_the_review": "I am concerned about the theoretical results regarding learnability of the optimal policy using this representation, and particularly how this might depend on the performance of the behavior policy. I would like to see this tested empirically as well by evaluating on the “random” data collection setting in D4RL. It is still a valuable contribution if the method only works well with certain data collection policies - however, this should be explained and tested empirically. \n\nI am currently unconvinced about the correctness of the claims without assumptions on the behavior policy (see Questions above). This is reflected in my current scores, which I will revise upward if my concerns are addressed. \n\n\n------ Update 12/9/22 --------\n\nThanks to the authors for the response clarifying the theoretical contributions. I also appreciate the changes the authors made to the paper to state explicitly the dependence of the quality of the learned state representation on the performance of the behavior policy, and to add experiments demonstrating this dependence. \n\nThe results in this submission are about the relative performance of the learned policy to the behavior policy, and so the sufficiency of the learned representations for representing the optimal policy (as explored in Li et al. ) is a moot point (as the authors point out). \n\nAs a side note, I do not understand the authors’ explanation: “Instead, we relax their strong requirements by allowing the value estimation in the ground MDP.” How is value estimation being done in the ground MDP when you run the offline RL algorithm in the “abstract” MDP defined by the learned state representation. In the updated appendix of the paper (page 18), it says that in contrast to Li et al., you “only change the state space,” leaving transition and reward functions unchanged. How can this be? \n\nMy concerns for the theoretical section of the paper have been addressed and I have adjusted my score for the paper accordingly. \n\nHowever, I continue to think the limitation of the method to high-performing behavior policies is a significant limitation in the context of offline RL, which is often predicated on the existence of suboptimal data (indeed, when expert data is available, BC methods have been shown to be quite competitive with offline RL). In the \"random\" setting D4RL experiments shown in Figure 13, BPR is the worst-performing method. I also think the novelty algorithmically is rather low, consisting of a few tweaks from representation learning algorithms applied to BC. Therefore most of the contribution of the paper is carried in the empirical results that the method works better than other forms of combining BC and offline RL. I think this is a valuable enough contribution and timely given the strong interest in this topic, so I cautiously recommend acceptance. \n\nI encourage the authors to further clarify the wording in the theoretical sections to impart the conclusions more clearly, and also to de-clutter the appendix (e.g., by removing pseudo-code) to make it easier to find results (such as the \"random\" D4RL experiment illustrating a failure case of the method).\n", "strength_and_weaknesses": "Questions and Comments\n\n1) I am struggling to square Theorem 2 in this work with Theorem 3.4 in Li et al. 2006. If the data collection policy Beta is the optimal policy, then does a representation that satisfies Assumption 1.3 correspond to one that is \\pi*-irrelevant in Li et al? I would appreciate if the authors could explain if and how the assumptions of Theorem 3.4 of Li et al. differ from their setting. The general question I think is one of representability versus learnability - it is possible to represent something but not be able to learn it. \n\n2) It seems that there may be an implicit dependence on the characteristics of the behavior policy that is not stated. Assuming full state support does not seem to be enough - the behavior policy could visit every state but never take a good action. At the limit, suppose the behavior policy always takes the same action - then BC can achieve perfectly accuracy by completely ignoring the state input. Don’t we therefore need an assumption on the minimum return of the behavior policy? Perhaps I missed it? Theorem 5 bounds the error between the estimated and true behavior policy, but this doesn’t address the question of the worst-case error of the policy learned via offline RL with the learned state representation. In the experiments, what happens if you run the algorithm on the “random” data collection setting of D4RL? \n\n3) I would recommend moving Figure 10 from Appendix G.5 to the main paper, as these are the main results - comparisons with other state representation learning methods on image observation inputs. What is the offline RL algorithm that is used for this experiment? What is the result here for the “random” data collection setting?\n\n4) How is co-training with BPR different from TD3+BC? Is the difference that the BC loss is predicted with a separate head instead of being added to the TD3 loss?\n\n5) Did you consider a probabilistic formulation for the state representation? If Z was treated as a latent random variable and the model optimized via variational inference, this would correspond to an information bottleneck on the representation Z, which would compress the information in the state representation.  \n\nNit\n - The theorem numbering starts at 2 instead of 1", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "nUycYdwnHxh", "reviewer": "Reviewer_vYat", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The work focuses on learning representation for offline reinforcement learning. The authors propose a representation learning method based on cloning the behaviour under which the offline dataset was collected. The main motivations of the work is to (1) reduce the difficulty of learning a policy from limited high-dimensional data by projecting the observations into a representative low-dimensional space, and (2) combat the under-parameterization problem of value networks that arises when bootstrapping is used by disentangling the representation learning from policy/value learning. \n\nThe proposed representation learning method is independent of the policy learning RL algorithm and can thus be used with any offline RL algorithm. Authors provide theoretical policy improvement guarantees under some assumption and present empirical results, showing outperformance against baselines. ", "summary_of_the_review": "Interesting, well motivated and well presented paper. However the experiments are is not detailed enough/choices made are not justified clearly enough. No intuition is given for addressing the limitation of the work as it stands. For these reasons my score is 5.", "strength_and_weaknesses": "Strengths:\n\n* The work addresses an important open problem in offline RL. \n* Very well motivated \n* The presentation is clear and coherent making it a pleasant read\n* Some interesting theoretical guarantees are provided\n* The proposed method is evaluated on different criteria (performance against baseline algorithms without pretrained representations, comparison against different representation objectives, change in the effective dimension of the data, and robustness to distractors in pixel-based inputs)\n\nWeaknesses:\n\n* The theoretical analysis provides some intuition on the kind of performance gain is expected when the representation is trained on a data under a policy $\\pi_{\\beta}$. This analysis essentially means that the quality of the $\\pi_{BPR}$ highly depends on the behaviour policy. Authors should discuss address what to do in cases where the behavioural policy is highly adversarial for instance. What does that mean for the learned representation? Is it still beneficial to learn a BPR, or is it harmful then?\n\n* When assessing the effectiveness of BPR against baselines (non-pretrained representation), a fairer comparison would account for the additional 100k training steps BPR used for pretraining. I understand the argument that the representation needed not to be retrained for all of the different offline RL methods, still it feels like an unfair comparison. \n\n* When comparing against other against other representation objectives, a couple things need clarifications:\n  * How were the baseline selected? (the performance criterion used is from Schwarzer et al. 2021, however the SGI representation is not included in the baselines?)\n  * Are the encoders for the different objectives of similar architecture/capacity?\n\n* I would have like to see a discussion or some results comparing BRP with representations that embed the transition dynamics (e.g. self predictive representations) ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SOvvVSSH_A", "reviewer": "Reviewer_jmKm", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new pre-training method called Behavior Prior Representation (BPR) for offline RL that applies Behavioral Cloning with a decoder that learns a compact state representation that is then used by offline RL methods. This compact representation is shown to alleviate the implicit under-parameterization phenomenon that causes over-aliasing of features. The paper presents theoretical performance guarantees and demonstrates significant performance gains with BPR on the D4RL Offline RL benchmarks.", "summary_of_the_review": "This paper is of high quality, has a clear motivation, and a simple and well-founded approach that is evaluated with a substantial amount of experimental and theoretical work.", "strength_and_weaknesses": "Strengths:\n- Clear motivation with plenty of references giving the unfamiliar reader the necessary context\n- A large number of experiments showing significant performance gain\n- Ablation studies to investigate Robustness to Visual Distractions and Effective Dimension\n- Substantial effort to provide theoretical guarantees\n\nWeaknesses:\n- A few parts of the paper seem rushed (in contrast to the remaining part of the paper). E.g. the wordings in the Reproducibility Statement, \"?other representation objectives?\", and it seems to me that state and action are swapped in Figure 1 under BPR.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster