Contrastive Value Learning: Implicit Models for Simple Offline RL
Authors: Bogdan Mazoure
Benjamin Eysenbach
Ofir Nachum
Jonathan Tompson
Venue: Submitted to ICLR 2023 (2023)
Keywords: reinforcement learning
contrastive learning
implicit density models
reward-free learning
offline reinforcement learning
metaworld

Abstract: Model-based reinforcement learning (RL) methods are appealing in the offline setting because they allow an agent to reason about the consequences of actions without interacting with the environment. Prior methods learn a 1-step dynamics model, which predicts the next state given the current state and action. These models do not immediately tell the agent which actions to take, but must be integrated into a larger RL framework. Can we model the environment dynamics in a different way, such that the learned model does directly indicate the value of each action? In this paper, we propose Contrastive Value Learning (CVL), which learns an implicit, multi-step model of the environment dynamics. This model can be learned without access to reward functions, but nonetheless can be used to directly estimate the value of each action, without requiring any TD learning. Because this model represents the multi-step transitions implicitly, it avoids having to predict high-dimensional observations and thus scales to high-dimensional tasks. Our experiments demonstrate that CVL outperforms prior offline RL methods on complex continuous control benchmarks.

Under review as a conference paper at ICLR 2023
CONTRASTIVE VALUE LEARNING: IMPLICIT MODELS FOR SIMPLE OFFLINE RL
Anonymous authors Paper under double-blind review
ABSTRACT
Model-based reinforcement learning (RL) methods are appealing in the offline setting because they allow an agent to reason about the consequences of actions without interacting with the environment. Prior methods learn 1-step dynamics model, which predicts the next state given the current state and action. These models do not immediately tell the agent which actions to take, but must be integrated into larger RL framework. Can we model the environment dynamics in a different way, such that the learned model does directly indicate the value of each action? In this paper, we propose Contrastive Value Learning (CVL), which learns an implicit, multi-step model of the environment dynamics. This model can be learned without access reward functions, but nonetheless can be used to directly estimate the value of each action, without requiring any TD learning. Because this model represents the multi-step transitions implicitly, it avoids having to predict high-dimensional observations and thus scales to nigh-dimensional tasks. Our experiments demonstrate that CVL outperforms prior offline RL methods on complex continuous control benchmarks.
INTRODUCTION
While the offline RL setting is relevant to many real-world applications where the ability for online data collection is limited, it often requires RL algorithms find policies that are not well-supported by the training data. Instead of learning via trilland-emmon offline RL algorithms must leverage logged historical data learn about the outcome of different actions, potentially by capturing environment dynamics as a proxy signal. Many prior approaches for this offline RI. setting have been proposed, whether in model- free (Wu et al., 2019; Fujimoto et al., 2019; Kumar et al., 2020) or model-based (Kidambi et al., 2020; Yu et al., 2021) settings. Our focus will be on those that address this prediction problem head-on: by learning a predictive model of the environment which can be used conjunction with most model-free algorithms.
Prior model-based methods (Yu et al., 2020b; Argenson and Dulac-Arold, 2020; Kidambi et al., 2020; Yu et al., 2021) learn model that predicts the observation at the next time step. This model is then used to generate synthetic data that can be passed to off thel-shehhff RL algorithm. While these approaches can work well on some benchmarks, they can be complex and expensive: the model must predict high-dimensional observations, and determining the value of an action may require unrolling the model for many steps. Learning a model of the environment has not made the RL. problem any simpler. Moreover, as we will show later the paper, the environment dynamics are intertwined with the policy inside the value function; model-based methods aim to decouple these quantities by separately estimating them. On the other hand, we show that one can directly learn long-horizon transition model for a given policy, which is then used to estimate the value function. A natural use case for learning this long-horizon transition model (specifically, a state occupancy measure) from unlabelled data multi-task pretraining, where the implicit dynamics model is trained on trajectory data across a collection of tasks, often exhibiting positive transfer properties. As we demonstrate in our experiments, this multi-task occupancy measure can then be finetuned using rewerd-labelled states on the task of interest, greatly improving performance upon existing pretraining methods as well as tabula rasa approaches.
In this paper, we propose to learn a diffeerntt type of model for offline RL, model which (7) will not require predicting high nigh-dimensional observations and (2) can be directly used to estimate Valuues without requiring either model-based rollouts model-free free temporal difference learning. Precisely, we will lear an implicit model of the discounted state occupancy measure, i.c. a function which takes in a state, action and future state and outputs scalar proportional to the likelihood of visiting the future state under some fixed policy.

---

Under review as a conference paper at ICLR 2023
Q-learning
Contrastive Value Learning (ours)
110
Figure 1: Contrastive Value Learning: A stylized illustration of trajectories (grey) and the rewards future states (e.g., +8, -5). (l.eft) Q-leaming estimates Q-values by "backing up" the rewards at future states. (Right) Our method learns the Q-values by fitting an implicit model estimate the likelihoods future states (bluc), and taking the rewght-weighted average of these likelihoods.
We will learn this implicit model via contrastive learning, treating as a classifier rather than generative model of observations. Once learned, we predict the likelihood of reaching every reward-laheled state. By weighting these predictions by the corresponding rewards, we form an unbiased estimate of the Q-function. Whereas methods like Qllearning estimate the Q-function of a state "backing up" reward values, our approach goes in the opposite direction, "propagating forward" predictions about where the agent will go.
We name our proposed algorithm Contrastive Value Leam CVL is simple algorithm for offline RL which learns the future state occupancy measure using contrastive learning and re-weights it with the future reward samples to construct quantity proportional to the true value function. Because CVL rep- resents multi-step transitions implicitly, it avoids having to predict high-dimensional observations and thus scales high-dimensional tasks. Using the same algorithm, we handle settings where reward-free data is provided, which cannot be directly handled by classical offline RL methods such as FQI (Munos, 2003) or BCQ (Fujimoto al., 2019). We compare our proposed method competitive offline RL baselines, notably CQL (Kumar et al., 2020) and CQL+UDS (Yu et al., 2022) on an offline version of the multi-task Metaworld benchmark (Yu al., 2020a), and find that CVL greatly outperforms the baseline approaches as measured by the rliable library (Agarwal et al., 2021b). Additional experiments on image- based tasks from this same benchmark show that our approach scales high-dimension tasks more seamlessly than the baselines. We also conduct a series of ablation experiments highlighting critical components of our method.
2 RELATED WORKS
Prior work has given rise to multiple offline RL. algorithms, which often rely on behavior regularization in order to be well supposupp by the training data. The key idea of offline RL methods is to balance interpo- lation and extrapolation emrors, while ensuring proper diversity of out-of-dataset actions. Popular offline RL algorithms such as BCQ and CQL rely on a behavior regularization loss (Wu et al., 2019) as a way to control the extrapolation error. This regularization term ensures that the learned policy is well-supported by the data, i.. does not stray too far away from the logging policy. The major issue with current offline RL algorithms is that they fail to fully capture the entire distribution over state-action pairs present in the training data.
To directly learn a value function using policy or value iteration, one needs to have information about the transition model in the form of sequences of state-action pairs, well as the reward emitted by this transition. However, in some real-world scenarios, the reward might only be available for a small subset of data. For instance, in the case of recommending products available in an online catalog the user, the true long-term reward (user buys the product) is only available for users who have browsed the item list for long enough and have purchased a given item. It is possible decompose the value function into reward- dependenn and reward free parts, as was done by (Barreto et al., 2016) through the successor representation framework (Dayan, 1993). More recent approaches (Janner et al., 2020; Eysenbach et al., 2020; 2022) use generative model to learn the occupancy measure over future states for each state-action pair in the dataset; its expectation corresponds to the successor representation. However, learning an explicit multi-step model such as (Janner et al., 2020) can be unstable due to the bootstrapping term in the temporal difference loss. Similarly to model- based approaches, our method will learn reward- -free representation of the world, but will do so without having to predict high- sigh-dinnsiioal observations and without having to do costly autoregressive rollouts. Thus, while our critic is trained without requiring rewards, it is much more similar to value function than standard 1 1-step model.
2

---

Under review as a conference paper at ICLR 2023
Learning conditional probability distribution over highly complex space be a challenging task, which is why it is often easier to instead approximate it using a density ratio specified by an inner product in much lower-dimensional latent space. To learn occupancy measure over future states without passing via the temporal difference route, one can use noise-contrastive estimation (NCE, Gutmann and implicit
Hyvärinen, 2010; Oord et al., 2018) to approximate the corresponding log ratio of densities function. Contrastive learning was originally proposed as an alternative to classical maximum likelihood estimation, but has since then seen successes in static self-supervised learning (He et al., 2020; Chen al., 2020). In reinforcement learning, NCE was shown to improve the robustness of state representations to exogenous noise (Srinivas et al., 2020; Mazoure et al., 2020; Agarwal et al., 2021a) and, more recently, to be an efficient replacement for traditional goal-conditioned methods (Eysenbach et al., 2022).
PRELIMINARIES
learning We assume a Markov decision process M defined by the tuple M (S, T, where S is a state space, So S S is the set of starting states, A is an action space, T= P|-
Reinforcement
is one-step transition function r::S x AAA ('min--mas) is reward function and 7 (0.1) is discount factor. The system starts in one of the initial states Su So- At every timestep / 1,,,,,,,.. the policy S (AA), samples an action G (x<T(-))))) The environment transitions into next state 1 ~ (())),)a))) and emits reward ra a()) The aim is to leam Markovian policy (aa (s) that maximizes the discounted sum of returns over an episode of length H:
H Σ
maxEaT FET O2
Ext(ora))
So
where P f denotes the joint distribution of (84+2 J obtained by executing in the environment for K timesteps starting at timestep t.
We assume that the offline RL algorithm cannot interact with the environment, but instead must learn from an offline dataset of logged trajectories $1,0,, Value-bused RL algorithms maximize cumulative episodic rewards by estimating the state-action value function under policy æ:
H E (r(0un
(T,
for st ES E.. Alternatively, the value function can be written as the expectation of the reward over the discounted occupancy measure:
1
("(s,at)== 1-7 P...
where (s) that the occupancy measure can equivalently be re- re-written in terms of the geometric distribution over the time interval (0,co) for ffinite-horizon rollouts:
- smi as defined in Janner et al. (2020). Note
To (s)s).00)= = -Esy- 110
(4)
This decomposition of the value function has already been used in previous works based on the successor representation (Dayan, 1993; Barreto et al., 2016) and, more recently, "y-mddels (Janner et al., 2020). We will use it to efficiently learn an implicit density ratio proportional to the state occupancy measure using contrastive learning.
Noise-contrastive estimation Noise-contrastive estimation (NCE, Gutmann and Hyvärinen, 2010) spans broad class of learning algorithms, at the core of which is negative sampling (Mikolov et al., 2013), i.e., learning an implicit metric space from positive and negative examples. Given reference samples, samples from a positive distribution (i.e., high similarity with reference points) and samples from a negative distribution (i.e., low similarity with reference points), contrastive learning methods leam an embedding
(AY)) denotes the entire set of distributions over the space X.
3

---

Under review as a conference paper at ICLR 2023
where positive examples are located closer to the reference points than negative examples. One of the most well-known and commonly used NCE objectives is InfoNCE (Oord et al,, 2018), which solves
((()) +(v)
max E 1..0- 0.100
log
Eyyyy ed()) V(rr)
over some hypothesis class $: {0:;A 2} for input space X, latent space 2, (xxxP(()),, y positives (A) and y~P negaivvsi (A)). Contrastive learning has been widely studied in the static unsupervised/ supervised learning settings (Hjelm et al., 2018; Chen et al., 2020; He et al., 2020), as well as in reinforcement leaming (Kim et al., 2018; Mazoure al., 2020) for learning state representations with desirable properties such as alignment and uniformity (Wang and Isola, 2020).
Solving Equation (5) for (6",, 2/*)) yields critic f X y - R which decomposes as (z.z)) = (x) 4/ (1) and, at optimality", captures the log-ratio of positives (2)) and Incgatives(
Plyla] f (x,1) olog Plyl
Implicit dynamics models via NCE. Various prior works (Du and Mordatch, 2019; Mazoure et al., 2020; Nachum and Yang, 2021) have studied the use of NCE to approximate single-step dynamics model, where triplets (11,41,884+1) have higher similarity than (sr,0tt,5t effectively defining positive and negative distributions over trajectory data. More recently, contrastive goal-conditioned RL (Eysenbach et al., 2022) used InfoNCE to condition the critic on goal states sampled from the replay buffer. These methods use asymetric encoders, using ((s;,a)) and ((s++a)), where positive samples of 8t+1r sampled from the discounted state occupancy measure for 120.
The conditional probability distribution of future states given the current state-action pair can be efficiently estimated using an implicit model trained via contrastive learning over positive and negative feature distributions, as shown in Equation (7).
ax)) T ((()-o))
(1)oNCE((N))) = By B,, log ESTESIAA (rrrae) v(s+40))
trajectory data yields critic which, optimality, approximates the future discounted state occupancy measure up to a multiplicatiie term as per Equation (6).
Minimizing ÉITNNCE
(( x log (1,.4
Intuitively, f approximates a H-step dynamics model which has an implicit dependence on policy that collected the training data, but is time-independent since Equation (8) is optimized on average across multiple t,At. Ordinarily, training state-space models is hard when the dimensions are large, e.g. image-based domains. However, by using contrastive learning, we can learn this model without having to require it predict high-dimensional observations, as similarity is evaluated in a lower- -ineessoinii latent space (observe that in Equation (7) the inner product is computed Z, whose dimension we control, instead of X, which specified externally). An apparent limitation of the approach is that the probability of future states 8t+ is recovered only up to a constant. However, it turns out that we can still use this model to get accurate estimates of the Q-values, as is described in the next section.
ESTIMATING AND MAXIMIZING RETURNS VIA CONTRASTIVE LEARNING
In this section, show how NCE be used to learn quantity proportional value function, and how the later can be used in a policy iteration scheme.
4.1 ESTIMATING Q-VALUES USING THE CONTRASTIVE MODEL
As shown in Equation (3), the Q-function at (s) can thought of as evaluating the reward function at states sampled from the discounted occupancy measure Pr (St.04)) That is, to estimate a quantity
Sce Ma and Collins (2018) for exact derivation.
4

---

Under review as a conference paper at ICLR 2023
akin to Q", can first estimate the occupancy measure and take weighted average of rewards over future states using the probabilities from the log-density ratio learned by the contrastive model. Precisely, Equation (3) corresponds to using importance-weighted estimator, where optimal critic that minimizes Equation (7) approximates the density ratio from Equation (8). The positive samples come from the discounted state occupancy measure: we first sample time offset A Geometric(1 -7) (column in the dataset), and then sample state from the distribution of states at this given offset (row in the dataset). As per classical InfoNCE formulation, this forms the joint distribution (s.,at St)),, which is contrasted against the negative distribution of product of marginals p() xp(s:-Ar).
The critic itself can be trained using the occupancy measure formulation specified in Equation (4) over all state-action pairs in given episode. However, Equation (4) needs to be re- re-adjusted to account for finite-horizon truncation of the geometric mass function presented in Definition
Definition 1 (Truncated distribution) Let X be a random variable with distribution function Fx. Y a called the truncated distribution of X with support [m,M] s.t. O<m<M if
PY= Fx Fx(y-m) -m) Fx (x(y-1mm) -y= -y=m.m+1mm
Fx (M) -x
We denote the special case of the truncated geometric distribution TranGGeom(p.m.MM)
The contrastive objective to train the critic to approximate the discounted occupancy measure over a dataset D is then
(oM-nioNNE(Ooe)))= E
log
(10)
EArEAMIA () Maria))
A Truneeommm H)).
AtoThunGoomm1->/: (4-
is possible that multiple optimal critics exist s.t. the multiplicative proportionality constant depends on the action. To avoid this, we adopt a similar approach Eysenbach al. (2022) and introduce regularization term over the partition function, making the critic training objective be
foM-INeNCE Apattition E. Ar,
Σ
(11)
APEAUUM
Now, suppose found an optimal critic f. Combining Equation (4) with Definition obtain the following form of the Q-function for an optimal critic f which minimizes Equation (7):
QNCE(St,at)=
Ar-1
A1=1
E AAn--Tmnaommm--22...
r(s(+ betosaaeee s14
(12)
1
E TAA
Bpp (r(* se) (o.
Here, the offset At is a random variable sampled from TruneGeom(1 where H/ is the horizon of the MDP We can also show that Q(s.a)(((()))) QQs,a') = QNCE(s,a) (NCC(sss)) for sES and a,a' A. which makes the contrastive Q-values suitable for policy evaluation. However, we do not, in general, expect QNCE to recover the optimal Q function, as the recovered Q-values are on-policy with respect
4.2 EFFICIENT ESTIMATION USING RANDOM FOURIER FEATURES
A major issue with using QNCE out-of-the-box is that it is computationally expensive, requiring evaluation of the inner product Φ(s;, ax) (((+)))t with a large number of future states and hence multiple forward passes through sb. The underlying cause of this computational overhead is the RBF kemel term
( ((444)) we instead used linear kemnl,, the constant term d(s;,a,) would be factored out, and we could separately keep track of reward-weighee future expected features. This would (1) reduce
While using the truncated geometric distribution makes Equation (12) proportional the true value function, the relation becomes an equality the infinitehorizon since lim (1 =1
5

---

Under review as a conference paper at ICLR 2023
the computational complexity of N actor updates over D from ((D) to O(D +N) and (2) reduce the variance of the representation if averaging features of future states using exponential moving average. It turns out that the RBF kernel can be approximately linearized by using random Fourier features (Rahimi and Recht, 2007; Nachum and Yang, 2021).
Let E be unit vectors, and let Fw Fw,b(2)= 2r cos (w++b) where W (NNormal(0.1) and b~ Uniform (0,2m) fixed initialization. Then, (() (w.b(y)]ee*
Lemma 1
Lemma 1 is straightforward modification of the result from Rahimi and Recht (2007) and allows us to reduce the RBF kernel to an expectation over d-dimensional random feature vectors:
QNCE(S1,01) =
Ban hww. [B acx)) (e (eo())))) (lertsr) (()++
Fw.old(sr.a)) P Ent x4, WW
(w)
(13)
The advantage of using the RFF approximation is that it allows us to split the exponential term inside the expectation and separately keep track of the policy-dependent, reward-weige weward-weighted future state probability term, while the state-action dependence term is leamed online. Specifically, we keep track of {(T) via an exponential-moving average during the entire duration training".
4.3 LEARNING THE POLICY
Once the policy evaluation phase completes and we have estimate QNCE, we optimize policy to maximize the returns predicted by this Q-value. We can decode the policy by minimizing its Kullback-Leibler divergence to the Boltzmann Q-value distribution (see Haarnoja et al. (2018)), which can be efficiently done by minimizing the following objective:
Ppliicy/ (0) BRIND DKL (T((s))
(14)
Jae (leAeQloaa)/
Note that in discrete action spaces, minimizing Equation (14) leads to a soft version of the greedy policy decoding Tggeddy (s) arggma QNCE @NCe(s,a) for SES. In practice, we approximate the KL term in Equation (14) using N₂ Monte-Cario action samples.
Decoding ı such way can lead sampling out-of-distribution actions in regions where the Q-function might be inaccurate due to poor dataset coverage. To mitigate this issue, we follow prior work (Cobbe al., 2021; Zhao al., 2021; Schwarzer et al., 2021) and add behavior cloning term which prevents the new policy from straying too far away from the data:
(f) (o
(15)
for entropy estimator (()(s))= - En-n(() [logr(a|ss)]. We add this extra loss to Epolicy to learn policy ı which prioritizes high Q-values that are well-supported by the offline dataset D. Thus, the final policy optimization objective becomes
Epolicy (0) = Ppolicy (0) ARChRc(O).
(16)
The policy found by minimizing Falliey has, average, non-decreasing returns, as per Lemma 2.
Lemma 2 (Contrastive policy improvement) Let H be a policy and let QNice
mina Ep (Eeriie (g.4)).
m(s)=argminD) (()) Sac e aie
(17)
*Enl
then (s,aa)) > (s,aa) all (s,a) DH.
"This idea can be adapted to online learning settings well hy clipping policy improvement steps so that & doesn't change fast under newly collected data.

---

Under review as a conference paper at ICLR 2023
Algorithm 1: Contrastive Value Learning (CVL)
Input Dataset "D J1, vo,b networks, temperature parameter т, exponential moving average
parameter /
for epoch j= do
for minibatch B~D do
* Update density ratio estimator using Equation (11)
Update (10++)),,6(++1)))) using Volefcmm (b6));
/* Estimate the contrastive Q-function
(s.aa) < Equation (13) if using RFF, otherwise Equation (12):
/+ Decode policy from Q-function using Equation (16)
Update using V Vo((noicy(0)):
xa Update future state encoder using EMA
(--1) (0b+)
EMA
Update future state features weighted by rewards
*/
c(+1) + EMA Blre+ad;
(++)),
SEMA
The proof of Lemma 2 is located in Section 6.2. Specifically, Lemma 2 tells us that using CVL as a surrogate Q-function corresponds step of conservative policy improvement, where satisfies soft constraints of Equation (14) and small Ep [Dxa (T (x())][((s)) via the BC term.
4.4 PRACTICAL IMPLEMENTATION
We now present our complete method, which can be viewed as actor-critic method for offline RL. We leam the critic via contrastive learning (Equation (11) and learn the policy via Equation (16). We will inter- leave these steps in most of our experiments, but experiments in Section 5.1 show that the critic can be pre- trained e.g. in the presence of unlabeled data from related tasks. We summarize the method in Algorithm 1.
4.5 INTERPRETATIONS AND CONNECTIONS WITH PRIOR WORK
The main distinction between Contrastive Value Learning and prior works consists specifically in representing the Q-values in two-step decomposition: the Q-value is represented as an occupancy measure weighted by the reward signal; the occupancy measure itself is represented using powerful likelihood-based model parameterized using an implicit function. Decoupling the learning of the occupancy measure from reward maximization allows, among others, for efficient pretraining strategies on unlabeled data, i.c. trajectory data without reward information, and can be used to learn provably optimal state representations for any reward function (Touati and Ollivier, 2021). While CVL is similar spirit to the successor representation (Dayan, 1993; Barreto et al., 2016), the occupancy measure learned by CVL. is much richer than that SR, as it captures the entire distribution over future states instead of only the first moment. Another method, y-models (Janner et al., 2020), is closely related CVL, but surrogate single-step TD objective to learn the occupancy measure, similarly to C-learning (Eysenbach et al., 2020).
EXPERIMENTS
Our experiments aim to answer three questions. First, we study how CVL compares with baseline approaches large benchmark of state-based tasks. Our second set of experiments look image-based tasks, testing the hypothesis that CVL scales to these tasks more effectively than the baselines. We conclude with ablation experiments. Our main point of comparison will be a high-perforning offline RL method, CQL (Kumar et al., 2020). While CVL learns an implicit model, that model is structurally much more similar to value-based RL methods than model-based methods, motivating our comparison to balued baseline (CQL). We will also include behavioral cloning as baseline.
Metaworld, We first test approach on the complex MetaWorld benchmark (Yu et al., 2020a), which consists of 50 robotic manipulation tasks such as open a door, pick up an object, reach a certain area of the table, executed by robotic ar (see Figure (left).. This domain is an ideal testbed for CVL,, as it allows
7

---

Under review as a conference paper at ICLR 2023
Median COL
IQM
Mean
CQL+UDS
CVL
2500 3000 3500
2400 3000
3600
4200
2400 2800 3200 3600
Metaworld benchmark, (Left) We evaluate CVL, on 50 tasks from Metaworld, subset of which are shown here. (Right) Compared with three offline RL baselines, CVL achieves Matisticalll-signiffeee improvements in offline performance, Results are reported over random seeds.
Figure 2:
for both full-state and image-based experiments, has dense and informative reward function thus decou- pling the problem of representation learning from exploration, and is challenging for model-free methods which leaves room for improvement. While the original Meta MetaWorld World domain has been used to evaluate online RI. agents, we create an ad hoc dataset suitable for offline learning. To do so, we train Soft Actor- Critic (Haarnoja et al., 2018) from full states each of the 50 tasks separately for 500k frames, and save the resulting replay buffer, which forms the training dataset. As shown in Figure (right), CVL manages to considerably improve upon strong baselines such behavior cloning, CQL and CQL with UDS (Yu et al., 2022y'. We report the results on all tasks the Meta World suite over random seeds, according to the ag- gregation methodology proposed by Agarwal et al. (2021b). Per-environment scores available in Table 4.
Image-based experiments Our working hypothesis that contrastive formulation of the value function acts in itself as a pre-training mechanism via the prism of representation learning. For this reason, we conduct further experiments on 4 image-based tasks from the MetaWorld suite (similarly to full-states, the dataset was obtained from the SAC replay buffer trained rendered images).
Table 1: Offline RL with Images. We compare CVL baselines on four offline, image-based based tasks from Meta World offline image- based tasks. Average + std. dov. are shown for 5 random seeds.
Task
BC
CQL
CVL
door-close door-open
571 + 9.9
4249 269.9
4480 + 305,1
178 1 4.0
2099 1 0.9
3389 76.6
drawer-close
2414 1 1736.5
3964 1 1634.9
2177 1: 1679.5
drawer-open
1030 1 104.2
820 + 56.0
2543 1: 115.0
Results presented in Table show that CVL is also able to learn meaningful Q-values and achieve good empirical performance hard image-based tasks.
5.1 ABLATION EXPERIMENTS.
When is pretraining the model useful? In theory, the model can be pretrained on the data from other tasks, however, we do not always expect this to help (e.g., when the pretraining tasks are very different). We ran an experiment to test this capability. The results, shown in Fig. 3, show that pretraining sometimes speeds up learning. In particular, we observe that pretraining is effective when the pretraining tasks are similar to the target task and contain a diverse set of state-action pairs.
How reliable is the QNCE approximation? Given that contrastive Q-values are proportional the true Q-function, a natural question to ask is how good is QNer at capturing the topology of Q? First, we conduct ablation demonstrating how linearizing the RBF kemne via random Fourier features provides performance gain on the offline MetaWorld tasks Figure 4. Specifically, we hypothesize that this is due the reduced variance of the RFF Q-value estimator which keeps track of future reight-weighted state features using a rolling average.
Next, we qualitatively assess the similarity between contrastive and true Q-values on the continuous Mountain Car enviromment (Moore, 1990) by first pre- treetraining SAC online on the task and then fitting CVL to the data from SAC's replay buffer. Figure 5 (left) shows the contrastive Q-values on a log-scale, evaluated on trajectories from the SAC replay: for comparison, we also show the Q-values learned by online SAC in Figure 5 (right). Note that the value function learned by CVL conserves the same topology as the true value function, up to multiplicative rescaling.
Forr CQL +UDS combine all data from the current task with unlaheled data from related tasks with rewards set O. In the absence of related tasks, we pre-train the critic on the current task with 0 rewards.
8

---

Under review as a conference paper at ICLR 2023
2500
bok-close
dial-um
700
dissssemble
4500
doon-utiock
4500
0 d'awer-open
4000
4000
retur
000
2000
1500
2000
3000
500
400
2530
3000
2300
3000
1500
1500
1000
1000
1200
Dbucet-close
handle
unplugiside
4000
4050
500
600
1300
1000
3000
600
1020
3000
2000
300
750
2000
500
1000
250
C4 0.6
1.0
0.5
0.4
0.8
1.0
266
Gradiant
166
366
Gradient
Figure 3: Offline Learning Curves for Metaworld. Episode return curves function of gradient steps taken during training on 10 random World tasks; curves show mean + standard deviation. Pretraining the reward-free occupancy measure on related tasks allows CVL to outperform baseline approaches and even CVL trained tabula rasa.
t 1.00
CVL (RFF) CVL (ro AFF
0.75
0.50
0.25
0.00
Egicode return (1)
Figure 4: CVL with RFF (orange) performs slightly better than with- out RFF (blue).
Figure 5: Visualizing the estimated Q-values. (Left) Normalized logQNcE learned by CVL offline on the Mountain Car enviromment. (Right) Normal- ized Q learned by online SAC on the same environment.
6 DISCUSSION
This paper presented an RL algorithm that learns a contrastive model of the world, and uses that model obtain values by estimating the likelihood of visiting future states. Our experiments demonstrate that this approach can effectively solve a large number of offline RL tasks, including from image-based observations. Our pretraining results hinted that CVL can be pretrained on datasets from other tasks, and we are excited to pretrain our model on datasets of increasing size.
Limitations. One limitation of our approach is that it corresponds to a single step of policy improvement. This limitation might be lifted by training the contrastive model using a temporal difference update for the contrastive model (Eysenbach et al., 2020; Blier et al., 202 1). A second limitation that the RFF approximation be poor when the feature dimension is small. We tried train the contrastive model using non non-expont ee-exooeentiated features (akin to HaoChen et al. (2021)), ) but failed to achieve satisfactory results. Figuring out how effectively train these spectral models remains an important question.
REPRODUCIBILIIY STATEMENT
We ensure reproducibility of our method via a) releasing the offline Meta World dataset that we used for our main results to allow the community to conduct further research on this domain upon publication, b) releasing the code used obtain results upon publication and c) detailing the hyperparameters and design choices for the implementation of CVL and the computational resources used for our experiments in Section 6.1.

---

Under review as a conference paper at ICLR 2023
REFERENCES
R. Agarwal, M. C. Machado, P. S. Castro, and M. G. Bellemare. Contrastive behavioral similarity emheddings for generalization in reinforcement learning. arXiv preprint arXiv:2 2021a.
R. Agarwal, M. Schwarzer, P. S. Castro, A. C. Courville, and M. Bellemare. Deep reinforcement leamnngg the edge the statistical precipice. Advances in neural information processing systems, 34:29304 2021b.
A. Argenson and G. Dulac-Amold. Model-based offline planning. arXiv preprint arXiv:2008..5556, 2020.
J. I.. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint 16Xiv:160.0064400 2016.
A. Barreto, W. Dabney, R. Munos, J. J. Hunt, T. Schaul, H. Van Hasselt, and D. Silver. Successor features
for transfer in reinforcement learning, arXiv preprint arXiv:1606.05312, 2016.
Blier, C. Tallec, and Y. Ollivier. Learning successor states and goal-dependent values: A mathematical viewpoint. arXiv preprint (tt.xx:2111.012220 2021.
T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
K. W. Cobbe, J. Hilton, O. Klimov, and J. Schulman. Phasic policy gradient. In International Conference on Machine Learning, pages 2020-2027. PMLR, 2021.
P. Dayan. Improving generalization for temporal difference learning: The successor representation. Neural Computation, 5(4))613-6244 1993.
Y. Du and L. Mordatch. Implicit generation and modeling with energy based models. Advances in Neural Information Processing Systems, 32, 2019.
L. Espeholt, H. Soyer, R. Munos, K. Simonyan, Mnih, Ward, Y. Doron, Firoiu, T. Harley, I. Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning pages 1407-1416. PMLR, 2018.
B. Eysenhach, R. Salakhutdinov, and S. Levine. C-learnin:: Learning to achieve goals via recursive classification. arXiv preprint 2rXiv:2011.08909 2020.
B. Eysenbach, T. Zhang, R. Salakhutdinov, and S. Levine. Contrastive learning as goal- conlicondi reinforcement learning. arXiv preprint arXiv:2206. 075668 2022.
Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pages 2052 2062. PMLR, 2019.
M. Gutmann and A. Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 297-304. JMLR Workshop and Conference Proceedings, 2010.
T. Haamoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
J. Z. HaoChen, C. Wei, A. Gaidon, and T. Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. Advances in Neural Information Processing Systems, 34:5000-5011, 2021.
K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9729-9738, 2020.
R. D. Hjelm, A. Fedorov, S. Lavoie-Marchilee K. Grewal. P. Bachman. A. Trischler, and Y. Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint ar08v:/808.065 06670, 2018.
G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
10

---

Under review as a conference paper at ICLR 2023
M. Janner, I. Mordatch, and S. Levine. Generative temporal difference learning for unfinite-horizon prediction. arXiv preprint arXiv:2010. 2020.
R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims. Morel: Model-based offline reinforcement learning. arXiv preprint arXiv:2005.0555,,200 2020.
H. Kim, J. Kim, Y. Jeong, S. Levine, and H. O. Song. Emi: Exploration with mutual information. arXiv preprint arXiv: 1810.0111 2018.
A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint ar arXiv:2006.04779 2020.
Z. Ma and M. Collins. Noise contrastive estimation and negative sampling for conditional models: Consistency and statistical efficiency. arXiv preprint arXiv: 00822 2018
B. Mazoure, R. T. dd Combes, T. Doan, P. Bachman, and R. D. Hjelm. Deep reinforcement and infomax learning. Neural Information Processing Systems, 2020.
T. Mikolov, K. Chen, G, Corrado, and J. Dean. Efficient estimation of word representations in vector space. arXiv preprint 1rXiv:1301.378) 2013.
A. W. Moore. Efficient memorry-asee learning for robot control. 1990.
R. Munos. Error bounds for approximate policy iteration. In ICML,, volume 3, pages 560-567, 2003.
O. Nachum and M. Yang. Provable representation learning for imitation with contrastive fourier features. Advances in Neural Information Processing Systems, 34:30100-300 0001000-0012,, 2021.
A. v. d. Oord, Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03777. 2018.
A. Rahimi and B. Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, 20, 2007.
M. Schwarzer, N. Rajkumar, M. Noukhovitch, Anand, L. Charlin, Hjelm, Bachman, and A. Courville. Pretraining representations for data-efficient reinforcement learning. arXiv preprint arXiv:2106.0499) 2021.
A. Srinivas, M. Laskin, and P. Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning- International Conference on Machine Learning, 2020.
A. Touati and Y. Ollivier. Learning one representation to optimize all rewards. Advances in Neural Information Processing Systems, 34, 2021.
C. X. Wang and W. P. Tay. Practical bounds of kullback-leibler divergence using maximum mean discrepancy. arXiv preprint N0..::2224400031,, 2022.
T. Wang and P. Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. International Conference on Machine Learning, pages 9929-9939. PMLR, 2020.
Y. Wu, G. Tucker, and O. Nachum. Behavior regularized offline reinforcement learning. arXiv preprint 191iv:1911.11388 2019.
Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: benchmark and evaluation for multi-task and meta reinforcement learning. In Conference Robot Learning, pages 1094-1100. PMLR, 2020a.
T. Yu, G. Thomas, L. Yu, S. Ermon, J. Y. Zou. S. Levine, C. Finn, and T. Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 13:14129- 2020b.
T. Yu, A. Kumar, R. Rafailov, A. Rajeswaran, S. Levine. and C. Finn. Combo: Conservative offline model-based policy optimization. Advances neural information processing systems, 34:28954-28966 2021.
11

---

Under review as a conference paper at ICLR 2023
T. Yu, A. Kumar, Y. Chebotar, K. Hausman, C. Finn, and S. Levine. How to leverage unlabeled data in offline reinforcement learning. arXiv preprint arXi::2002.01741, 2022.
Y. Zhao, R. Boney, A. Ili, J. Kannala, and J. Pajarinen. Adaptive behavior cloning regularization for stable offline-to-online reinforcement learning. 2021.
APPENDIX
REPRODUCIBILITY CHECKLIST
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Yes
(b) Did you describe the limitations of your work? See end of Section 4
(c) Did you discuss any potential negative socictal impacts your work? N/A
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Yes
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? Yes, they are outlined in the lemmas.
(b) Did you include complete proofs of all theoretical results? See Appendix Section 6.2
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? Code will be release upon publication, along with the offline Metaworld dataset.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? See Appendix Section 6.1
(c) Did you report error bars (c.g., with respect to the random seed after running experiments multiple times)? Yes, we include multiple various uncertainty estimates and measures of central tendency, e..g as outlined in Agarwal al. (2021b) over random replicates (i.e. seeds)
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? See Appendix Section 6.1
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? N/A
(b) Did you mention the license of the assets? N/A
(c) Did you include any new assets either in the supplemental material or as URL? Due the large size of the offline Metaworld dataset, we will release the code required to re-generate it exactly upon publication.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? N/A
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? N/A
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text instructions given to participants and screenshots, applica- ble? N/A
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? N/A
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? N/A
12

---

Under review as a conference paper at ICLR 2023
6.1 EXPERIMENTAL DETAILS
Model architecture All algorithms (baselines well as CVL) were based on common architecture, where an encoder (IMPALA (Espeholt al., 2018) for image data and two layer DenseNet MLP (Huang et al., 2017) for full-states) generated state features which, combined with actions gave rise to the Q-value and the policy (we used a diagonal Gaussian policy with Tanh bijector, as common for continuous control tasks). The main difference of CVL with the baselines is that the critic is defined implicitly via the dot-prot dot-product current state-action features passed through one encoder, and future state features passed into separate DenseNet. The output of both encoders was optionally normalized using & norm. All methods had a LayerNorm layer (Ba et al., 2016) in between each linear layer to ensure proper feature scaling.
Hyperparameter
Value
Leamingg rate
3 10
Batch size (all but CVL)
512
Discount factor
0.99
Framestack
No
Max gradient norm
100
MIP structure
256 DenseNet
Encoder (full-state)
256 DenseNet MILP
Encoder (pixels)
IMPALA
Add LayerNorm between all layers
Yes
Table 2: Hyperparameters that consistent between methods.
Hyperparameter
Value
CVL.
Batch size
И
Number of future action samples N
10
InfoNCE temperature
Partition function coefficient Aparitioon
0.001
BC coefficient ABC
(Mountain Car), 0.1 (rest)
RFF
Yes
Ey-normalize MLP outputs
Yes
CQL
Regularization coefficient
BC
Entropy regularization coefficient
0.1
Table 3: Hyperparameters that are different between methods.
All experiments were run on the equivalent of 8 V100 GPUs with 64 Gb of RAM and CPUs. For all methods, the corresponding auxiliary loss weights have been selected through best aggregated performance on the drawer and door domains with hyperparameter values of (0.0.01,0.1,1.0).
Dataset composition The offline Meta World dataset was constructed by first pre-training SAC on all 50 tasks from full-states for 500k environment interactions. The replay buffer at the end of the training was then used training dataset for BC, CQL, CQL+UDS and CVL. An identical approach was used to construct the image-based MetaWorld datasets and the Mountain Car dataset.
Pretraining setup When pretraining CVL, we first optimize the critic on unlabeled data from dataset for all the semantically related tasks, i.e. tasks which belong to the same domain, and then finetune both the critic and the policy on reward-labeled data from the target task. Semantically related tasks in Meta World easily identifiable by their domain name, egg. drawer-open and drawer-close belong to
13

---

Under review as a conference paper at ICLR 2023
the drawer domain. We use similar approach when pretraining CQL+UDS, where perform TD updates with all rewards equal to O during the pretraining phase.
6.2 PROOFS
Proof 1 (Random Fourier features approximation, Lemma 1)
For unit vectors z.y Z.E d>0.
EE ( (V2oo((W>t)) (V)os(WW)t)) - - [u11/2]
=exp
-111z111 22x y 11
= =exp(- (222-2222) 2x /))2)
(18)
c" 9-1
e
e
by re-arranging the terms in the result from Rahimi and Recht (2007). Therefore,
e" = LE (os
(19)
Proof (CVL induces single-step of policy improvement, Lemma Since, for the optimal critic f
(' P(s441ls,,ac1)
(20)
P[s
point-wise for every (81,04 EDH then, for 0,
( P|sctauls,,a144
(21)
PP|s
Now, the following relation holds using the previous result
Epp ttar (e Ai)]]
(22)
Tne (1 (,,H,
-QQ (stra))
Using this relation yields
Picelrnais/
(23)
Sac Aiallaaa
Sac (a da Jae (M
follows that
(hinller)!!
argminnDa a(si) nell
aarginnDa (((t)) rell
Sacd Bhellera)/
Socd (
(24)
=T(8))
Now, we invoke Lemma 2 from Haarnjja et al. (2018) by using the equivalence of the policy decoded from contrastive Q-values the policy found by soft policy iteration, which concludes the proof.
14

---

Under review as a conference paper at ICLR 2023
1.00
0.75
Fraction runs
0.50
0.25
0.00
1000
2000
3000
4000
5000
Episode return (T)
Figure 6: Performance profile of BC (red), CQL (green), COL+UDS (orange) and CVL (blue) generated by the rliable library (Agarwal et al., 202 for the offline MetaWorld experiments over random seeds.
Median
IQM
Mean
CVL (BC=0.01)
(BC-0)
3200 3400 3600
3800
3800
4000
4200
3150
3300
3450
Figure 7: Aggregated performance metries for CVL with different behavior cloning weights.
6.3
ADDITIONAL RESULTS
6.3.1 METAWORLD
Ablation the BC coefficient: We ablate the impact of the behavior cloning loss CVL's perfor- mance in Figure 7. We can see that, although adding behavior cloning loss improves the performance by small amount, is not essential to the fundamental functioning of CVL.
6.3.2
MOUNTAIN CAR
Episode return
500
1000
1500
2000
2500
Gradient steps
Figure 8: Evaluation returns on Mountain car during training on data from the SAC replay buffer. The red dotted line indicates highest possible return.
Quantitative evaluation of the contrastive occupancy measure: From Wang and Tay (2022), we know that
MMD(P.Q) KL(F.Q)
(25)
We also know that
=(
(26)
2logNV lITONCE (P QN)
which simplifies the above expression
MMD Hoano
(27)
Figure shows the boper-bound on the MMD between occupancy measures learned with temporal difference and contrastive learning methods.
15

---

Under review as a conference paper at ICLR 2023
Tisk
BC
CQ.
CQLAUDS
CVihlla rasa)
CVI. (Pretmined)
basketball
3188 + 348.9
646 2.2
= 112.6
1138
4171 285.9
bin bin-picking
= 1.5
28 1: 0.7
21 1.9
18 = 1.8
860 : 69.2
box-close
891 + 164.4
311 12.4
296 + 19.8
1496 131.2
4189 + 3524
button- presss-pess
2667 85.3
3445 60.
3420 273.4
3659 519
1906 360.2
button-press--oe pren--es--tooooo
3089 + 256.7
3406
525.7
3505
+ 990.9
3889 + 36.2
548 + 4995
button press trpdown wall
1692 47.0
2095 28.4
2135 66.4
20088 21.7
546 90.8
coffee-button
3490 1435.5
3655
740.6
3431 6898
4259 169.9
149 + 10.8
coffee pull
647 +: 11.7
250 10.3
330 3.22
833 :: 27.2
167 1: 0.6
dial- turn
48.7
276.1
disassemble
215 4.3
215 9.6
217 + 36.0
214 18.6
926 1: 5.6
door-close
3634 + 141.5
4555 200.9
4547 + 215.2
4544 7.6
4313 1940
door-lock
3073 +: 303.7
144.2
271.1
20.6
door-open
828 + 24.7
4526 + 71.7
4531 + 179.0
39K5 + 279.6
613 113.0
door-unlock
1322 + 184.1
4122
4002 80.1
413.7
4618 1: 49.7
drawer close
4619 53.4
4855
4857 :: 2,0
4853 + 6.8
2933 671.8
drawer-open
1727 +: 204.0
2768 + 45.6
2776 25.2
2512 149.3
4664 1: 14.4
faucet-close
4160
1724.2
faueet-open
2052 1: 80.9
4731 401.8
4729 + 561.1
3600 + 221.9
1637 1 64.9
hammer
2158 272.0
898 70.3
126.6
4632 73,6
4630 1:
hand-insert
44 17.8
4.0
2.1
4612 539.8
hundle-press press
4734 36.3
2816
2755 0.8
4861 28
2417 + 169.2
handle-press-side
3820 + 1556.5
4783
478.1
3526
654 - 27.7
hundle-pull
3642 + 968.8
2422
524.1
2436
1286.8
4594 38.6
4636 + 35.8
hundie-pull-side
3418 1002.2
1898 582.6
1757 + 343.2
4660 41.0
2904 + 92.4
lever pull
peg wig-insert-side
11 1.1
17:4.1
= 1.8
15 - 044
12 0.8
peg unplug side
56 1.9
29 +
+ 1.6
4593 24.6
pick-out-of-hole
10 0.2
207 0.4
3.4
1864
5
pick placepacce
1771 416.2
1263 407.6
1306 128.5
2942 +: 454.1
4403 +: 50833
pick blcc-placewwall
19
plate-slide
3979 57.3
2697 475.3
3508 1 747.0
4649 142.6
802 12.4
plate -lide-back
2402 + 333.9
3163 + 1290.3
3014 303.9
4718 + 30688
196 + 5.0
slidk- bick-side
874.6
4669
siid-slidc-
2241 + 536.9
3104 308.1
3015 329.5
2695 - 413.8
1939 + 27.5
push
+: 317.9
494 5.0
463 33
1997 + 196.8
4386 1927
push-back
9.2
08
20,9
push-wall
3327 +: 508.6
689 5.56
628 1:2
4502 = 176.7
4601 +: 2056
reach
3069 + 359,2
920.3
3275 677.8
4819 182.9
4658 204.8
reach-wall
4515 - 93.9
4828 265
4829 1: 49.1
27.0
4825 1: 212
stick-pull
595 - 19.8
297 2.0
441 3.7
4488 52.0
3434 + 1629
stick-push
263 6.1
1.99
1.44
147.5
2814 5513
sweep
817 + 124.3
3086 645.7
3162 1507.1
4127 + 567.2
4461 + 49,5
sweep-into
532 151.3
1974
870.1
2657 1 364.3
506 14.8
window-close
3739 80.4
4478 452.5
4442 13.1
4534 56.2
4519 63.2
window-open
3743 + 147.3
2773 1433.5
2841 11635
4534 109,4
524 1: 320.0
Table 4: Evaluation returns on MetaWorld offline tasks. Average + standard deviation are shown for random seeds.
16

---

Under review as a conference paper at ICLR 2023
6
MMD
4
mmmm
20
40
60
80
100
Gradient steps
Figure Upper-bound on the MMD between occupancy measures estimated via TD and contrastive learning.
17


Reviews:
[{"review_id": "qlB6l0OY2q", "reviewer": "Reviewer_WeYu", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a contrastive learning based method for offline RL where a implicit dynamic model is learned based on contrastive learning. A contrastive Q-function is calculated based on the implicit dynamic model and the Q-function is then used in policy updates. Numerical experiments on robotic manipulation tasks show improved performance compared with prior methods.", "summary_of_the_review": "The proposed new contrastive learning based RL algorithm looks promising, but there are some concerns regarding how the contrastive learning model actually learns.\n\n=== after the rebuttal\nFrom the authors' reply, some of the \"random variables\" like the $\\Delta t$ in the LHS of (8) are actually non-random and are meant to be the averaged values over multiple samples. So those values should be denoted by the corresponding expected values, but which are expected values and which are random variables is very confusing and there are still incorrect steps in the latest version. The paper may contains some good ideas, but a major revision with better clarification seems to be necessary.", "strength_and_weaknesses": "Strength\n- The paper proposes a new RL algorithm which uses contrastive learning to learn an implicit dynamics model. The implicit model is learned to capture the occupancy measure and is then used to estimate a contrastive Q-function which is proportional to the actual Q-function.\n\n- The contrastive Q-function is verified in the numerical experiments to have a similar shape as the SAC Q-function in the continuous Mountain Car environment.\n\n- In the numerical experiments of MetaWorld benchmarks show good performance of the proposed CVL algorithm.\n\nWeaknesses\n- It is not clear from the paper how positive and negative samples are generated for contrastive learning. In (7), positive and negative samples are both denoted by $\\Delta t$ and the distributions for the samples are not specified. In (11) the negative samples are stated to be generated by a truncated geometric distribution, but there is no description on how the parameter $t'$ is selected. Note that this part is different from Eysenbach et al. (2022) where the positive samples are generated by the occupancy measure and negative samples are random state-action pairs.\n\n- Although occupancy measure does not depend on time, what is learned by contrastive learning seems to be the log likelihood ratio of the $\\Delta t$-ahead state given by the RHS of (8). This log likelihood ratio in the RHS of (8) is not the occupancy measure and it should depends on the time difference $\\Delta t$. However, the proposed contrastive learning method attempts to learn the $\\Delta t$-dependent RHS of (8) by the $\\Delta t$-independent LHS of (8). There seems to be something wrong, or I missunderstand some part of the contrastive learning objective.\n\n- The paper claims to focus on offline RL, but the offline aspect is not clearly discussed in the paper. The estimated contrastive Q-function is said to be the Q-function of the policy $\\pi$, but since it's learned using the offline dataset, it actually won't correspond to the Q-function of the current policy. It is very confusing which the policy $\\pi$ refers to in different parts of the paper.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "F563Z8P-S89", "reviewer": "Reviewer_4mmq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel contrastive learning algorithm to estimate the occupancy measure of future states in the offline RL setting. It tackles the well-known issue of extrapolation error in existing algorithms for offline RL, in which rewards outside of the underlying data distribution is not well captured by the model. The contrastive method presented is also able to predict long-term state occupancies without having to rely on an autoregressive model. Additionally, the implicit model captures the full distribution of the occupancy measures rather than just the first moments as seen in prior work.\n", "summary_of_the_review": "Overall, I think this is a nice paper and would of interest to the wider RL community for its theoretical and practical algorithmic contributions. I would lean towards accepting it.\n-------------\nEdit: I thank the authors for their response. Although they addressed my concerns, after discussion, I concur with other reviewers about the issues around the clarity of the technical details. Moreover, I believe there is merit in the work and its improved exposition could make it a strong piece of work. In particular, I encourage the authors to detail how the work differs from and connect back to Eysenbach et al. 2022. \n", "strength_and_weaknesses": "Stengths:\n- The paper is well motivated, with a detailed background on NCE and its drawbacks (mainly computational), leading to the constrastive loss proposed in Eq.11. The use of the approximate linearized kernel seems novel in its application to this setting.\n- The work presents an interesting line of research for value-based offline RL and seems to work well. Rather than improve offline RL with model-based generative models which require expensive rollouts and single-step compounding errors, this work aims to bypass this with a \"classification\" approach that has worked well in other domains. The empirical validation against other offline benchmarks are convincing.\n\nWeaknesses:\n- Figure 5 is a confusing result to showcase what I think is an important detail of this approach. Is there a quantitative measure of the similarities between distributions, e.g. MMD. I agree that this is difficult given one only cares about the topology and not the actual scores, as the former is enough for Q-estimates to be useful for RL, but I am not convinced by this result as is. \n- I have some concerns about the robustness of this approach with generalization to out of unseen examples, i.e. extrapolation error. In particular, it does not seem compatible with model-based offline RL methods, e.g. COMBO, which aim to mitigate this issue with a learned dynamics model. Could the authors provide any insight into how their C-learning method trades off the \"extrapolation vs interpolation\" as seen in offline RL.\n- Along the lines of the above, it would be interesting to see an ablation study of the method's learned policy with and without the behvior cloning term in Eq. 15", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "DvQcfYQLBkd", "reviewer": "Reviewer_qU1j", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper suggests an algorithm for offline reinforcement learning based on first applying a noise contrastive objective to estimate a quantity proportional to $p(s_{t+\\Delta t}|s_t,a_t)/p(s_{t+\\Delta t})$, where $\\Delta t$ belongs to a geometric distribution of future times. That is, the ratio of the discounted future state occupancy conditioned on a state-action pair, to the unconditioned discounted future state occupancy, both under a particular policy. This estimate is then used to construct an estimator for action-values by reweighing a dataset of rewards and states generated under the current policy. Building on this idea, the paper also proposes an alternative action value estimator which decomposes the Q-value as a dot product of two vectors. The first vector depends on the policy and reward function and is tracked as a running average. The other vector depends on the state and action and is learned. This decomposition is motivated by the random features approach of Rahimi and Recht (2007). The proposed approaches are evaluated for offline RL mainly in simulated robotics tasks from MetaWorld and found to compare favourably to alternative approaches to offline RL. It is also demonstrated that when additional offline data without rewards is available, it can be use to pretrain $p(s_{t+\\Delta t}|s_t,a_t)/p(s_{t+\\Delta t})$, which in some cases provides a benefit.", "summary_of_the_review": "The idea is superficially interesting, and the empirical results are superficially promising. However, there appear to be a number of technical and clarity issues, and missing details which lead me to lean strongly towards rejecting the paper in its current state. I am particularly concerned about the correctness of Equation 13 and the lack of detail and clarity in the pseudocode of algorithm 1. If the authors can address these issues, I would consider revising my score.", "strength_and_weaknesses": "The idea is interesting, and the empirical results appear superficially promising. However, there appear to be a number of technical and clarity issues, and missing details which lead me to lean strongly towards rejecting the paper in its current state.\n\nOn the positive side, the derivation of the method for estimating action values using noise contrastive state representations in equation 12 is interesting and appears correct to me. I'm unsure of its practical utility given the contrastive objective only learns a ratio $p(s_{t+\\Delta t}|s_t,a_t)/p(s_{t+\\Delta t})$ (up to a multiplicative constant). Thus it seems like the method still requires sampling from $p(s_{t+\\Delta t})$. Presumably, the variance would be high in many cases if the state space is large and the unconditioned occupancy measure is very different from the discounted occupancy following a particular state-action pair.\n\nThe description of the method in the introduction is a bit convoluted and, in my opinion, makes it harder rather than easier to understand what follows. In particular, phrasing like \"we will learn an implicit model of the discounted state occupancy measure, which answers the question 'where will the agent be in the time-averaged future?'\" gives the impression that the learned model is of a form that it can take a state action pair as input, and predicted the future state occupancy. Based on my understanding, the contrastive objective only learns a function $f^*(s_t,a_t,s_{t+\\Delta t})$ such that $f^*(s_t,a_t,s_{t+\\Delta t})\\approx p(s_{t+\\Delta t}|s_t,a_t)/p(s_{t+\\Delta t})$ and thus can only really be used to correct samples from $p(s_{t+\\Delta t})$ to samples from $p(s_{t+\\Delta t}|s_t,a_t)$. While this is perhaps clarified in the phrase \"treating it as a classifier rather than a generative model of observations\", I feel like this could have been stated much more straightforwardly.\n\nI am concerned with the correctness of equation 13 and the algorithm that is derived from it. On the left-hand side of the equation is a specific value, while the right-hand side is a random variable. This is in itself perhaps only a minor oversight if we assume the intended meaning is that the expectation of the RHS is equal to the LHS. However, $\\xi(\\pi)$, which is never formally defined, should also be a random variable as it depends on the randomly sampled W and b used in the RFF approximation. As far as I can tell, this is not accounted for as $\\xi(\\pi)$ is approximated using a running average, which presumably averages over many different values of W and b. I believe the RFF estimator in Lemma 1 is only correct if the same W and b are used in both factors so I'm not sure from the description how the algorithm actually works in practice or whether it is correct. Perhaps I overlooked something here, and the authors can explain in more detail how the algorithm works and why it is correct. At the very least I feel a more detailed explanation is required in the paper.\n\nIf I'm reading correctly, Figure 5 shows the log of the $Q_{NCE}$ values compared to the normalized SAC Q-values. Why is one on the log scale and the other isn't? I understood that the $Q_{NCE}$ values themselves should be proportional to the regular Q-values, not their log. Or is the intended meaning that both are on the log scale? Moreover, the claim that both figures show the same topology is not at all clear to me from the picture beyond very basic features around the edges.\n\nAlgorithm 1 is not entirely clear in a number of places. $\\psi^{(j+1)}$ is updated both in line 3 by a gradient, and then in line 6 as an exponential moving average, is this really the intended meaning? I think the issue may be that $\\psi^{(j+1)}$ is overloaded to mean both the exponential moving average and the immediate value. On line 7, what is the boldfaced $\\beta$? I can't see that it has been defined anywhere. It is also stated that $\\phi$ and $\\psi$ are updated using the gradient of $l_{Critic}$ but presumably, this is done somehow using samples in the offline dataset as opposed to explicitly computing the gradient of the full expectation so more details here would be helpful. The same is true for the policy update on line 5.\n\nWhile hyperparameters are provided in the appendix, it would help with the interpretation of the results to describe how they were chosen. i.e. tuned for the proposed approach, kept fixed from prior work. ect.\n\nOther Minor Comments Questions and Suggestions\n==============================================\n* I don't understand footnote 3, I thought the reason for the proportionality is that $e^{f(s_t,a_t,s_{t+\\Delta t})}$ is only equal to the suggested ratio up to a constant. Hence, I don't understand this claim.\n\n* In section 4.2: \"It is possible that multiple optimal critics exist s.t. the multiplicative proportionality constant depends on the action.\" Could the authors clarify this? If it can depend on the action, can it also depend on the state $s_t$? And if it can depend on both of these, what makes it a proportionality \"constant\" at all?\n\n* I can't find it stated anywhere how long training proceeded in the main experiments. I see in Figure 3 that for the pretraining experiments a total of 1 million gradient steps are used, is this the same for all experiments and algorithms?\n\n* Why is the batch size for CVL set to H according to table 3? Does this simply mean it trains on all the transitions in an episode at once? Furthermore, how does H compare to the batch size of 512 used by the other methods? Assuming training is done for a fixed number of gradient updates, this might be important in interpreting the results.\n\n* The left-hand side of lemma 1 in the appendix should be in expectation.\n\n* The first appearance of \"occupancy measure\" in the introduction is a bit jarring since it is never before mentioned that the paper will involve learning an occupancy measure", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject