Contrastive Value Learning: Implicit Models for Simple Offline RL
Authors: Bogdan Mazoure
Benjamin Eysenbach
Ofir Nachum
Jonathan Tompson
Venue: CoRL 2023 Poster (2023)
Keywords: reinforcement learning
robotics
metaworld
unsupervised learning
contrastive learning
noise-contrastive estimation
generative model

Abstract: Model-based reinforcement learning (RL) methods are appealing in the offline setting because they allow an agent to reason about the consequences of actions without interacting with the environment. While conventional model-based methods learn a 1-step model, predicting the immediate next state, these methods must be plugged into larger planning or RL systems to yield a policy. Can we model the environment dynamics in a different way, such that the learned model directly indicates the value of each action? In this paper, we propose Contrastive Value Learning (CVL), which learns an implicit, multi-step dynamics model. This model can be learned without access to reward functions, but nonetheless can be used to directly estimate the value of each action, without requiring any TD learning. Because this model represents the multi-step transitions implicitly, it avoids having to predict high-dimensional observations and thus scales to high-dimensional tasks. Our experiments demonstrate that CVL outperforms prior offline RL methods on complex robotics benchmarks.

Contrastive Value Learning: Implicit Models for Simple Offline RL
Bogdan Mazoure
Benjamin Eysenbach +2
Ofrr Nachum Google DeepMind
Jonathan Tompson 3
1 Apple
Prrinceton University
Abstract: Model-based reinforcement learning (RL.) methods appealing in the offline setting because they allow agent to reason about the consequences of actions without interacting with the environment. While conventional model-based methods learn a 1-step model, predicting the immediate next state, these methods must be plugged into larger planning or RL systems yield a policy. Can we model the environment dynamics in a different way, such that the learned model directly indicates the value of each action? In this paper, we propose Contrastive Value Learning (CVL), which learns an implicit, multi-step dynamics model. This model can be learned without access to reward functions, but nonetheless can be used to directly estimate the value of each action, without requiring any TD learning. Because this model represents the multi-step transitions implicitly, it avoids having to predict high-dimensional observations and thus scales to high-dimensional tasks. Our experiments demonstrate that CVL outperforms prior offline RL methods on complex robotics benchmarks.
Introduction
While control from offline demonstrations is relevant to many real-world applications (e.g. sample-efficient pre-training for robots, [1]) case the ability for online data collection is limited, it often requires the algorithms to find policies that not well-supported by the training data. Instead of learning via miil-and-erron offline RL algorithms must leverage logged historical data to learn about the outcome of different actions, potentially by capturing environment dynamics as proxy signal. Many prior approaches for this offline learning setting have been proposed, whether in model-free [2. 3, 4] or model-based [5, 6] settings. Our focus will be on those that address this prediction problem head-on: by learning a predictive model of the environment which can be used in conjunction with most model-free algorithms.
Prior model-based methods [7, 8, 5, 6] learn a model that predicts the observation at the next time step. This model then used to generate synthetic data that can be passed to off-the-shelf RI algorithm. While these approaches can work well on some benchmarks, they can be complex and expensive: the model must predict high-dimensional observations, and determining the value of an action may require unrolling the model for many steps. Learning model of the environment has not made the RL problem any simpler. Moreover, as we will show later in the paper, the environment dynamics are intertwined with the policy inside the value function; model-based methods aim to decouple these quantities by separately estimating them. On the other hand, we show that one can directly learn a long-horizon transition model for given policy, which is then used estimate the value function. A natural use case for learning this long-horizon transition model (specifically, a state occupancy measure) from unlahelled data is multi-task pretraining, where the implicit dynamics model is trained on trajectory data across a collection of tasks, often exhibiting positive transfer properties. As we demonstrate our experiments, this multi-task occupancy measure then be finetuned using Swward-labeledd states on the task of interest, greatly improving performance upon existing pretraining methods as well as tabula rasa approaches.
In this paper, propose to leam different type of model for learning from offline data, model which (7) will not require predicting high-dimensional observations and (2) can be directly used to estimate Q-values without requiring either model-basee rollouts model-free temporal difference learning. Precisely, will learn an implicit model of the discounted state occupancy measure, ie. a function which takes in state, action and future state and outputs scalar proportional to the likelihood of visiting the future state under some fixed policy. We will leamn this implicit model via contrastive learning, treating it as
Work done while at Google.
'hhese authors have contributed equally.
7th Conference on Robot Leaming (CoRI. 2023), Atlanta, USA.

---

Q-learning
Contrastive Value Learning (ours)
+-
+
Contrastive Value Learning: A stylized illustration of trajectories (grey) and the rewards at future states (e.g., +8, -5). (Left) Q-leaming estimates Q-values by "backing up" the rewards at future states. (Right) Our method learns the Q-values by fitting an implicit model estimate the likelihoods future states (blue), and taking the seward-weighted average of these likelihoods.
Figure 1:
classifier rather than a generative model of observations. Once leamed, predict the likelihood of reaching every reward-labeled state. By weighting these predictions by the corresponding rewards, we form unbiased estimate of the Q-function. Whereas methods like Q-learning estimate the Q-function of a state "backing up" reward values, our approach goes in the opposite direction, "propagating forward" predictions about where the robot will go.
We name our proposed algorithm Contrastive Value Learning(CVL), CVL is simple algorithm for model-free control from offline data which learns the future state occupancy measure using contrastive learning and re-weights it with the future reward samples to construct a quantity proportional to the true value function. Because CVL represents multi-step transitions implicitly, it avoids having predict high-dimensional observations and thus scales to high-dimensional tasks. Using the same algorithm, we handle settings where reward- free data is provided, which cannot be directly handled by classical offline RI. methods such as FQI [9] or BCQ [3]. We compare our proposed method to competitive offline offline version of the multi-task Metaworld benchmark [] and find that CVL greatly outperforms the baseline approaches as measured by the rliable library [12]. Additional experiments on image-based tasks from this same benchmark show that our approach scales to high-dimension tasks more seamlessly than the baselines. We also conduct series of ablation experiments highlighting critical components of our method.
RL baselines, notably CQL [4] and CQL+UDS [10]
2 Related works
Prior work has given rise to multiple offline RL algorithms, which often rely on behavior regularization order to be well-supported by the training data. The key idea of offline RL methods to balance interpolation and extrapolation errors, while ensuring proper diversity of out-of-dataset actions. Popular offline RL algorithms such as BCQ and CQL rely on a behavior regularization loss [2] as way to control the extrapolation error. This regularization term ensures that the learned policy is well-supported by the data, i.c does not stray too far away from the logging policy. The major issue with current offline RL algorithms is that they fail to fully capture the entire distribution over state-action pairs present in the training data.
To directly learn a value function using policy or value iteration, one needs to have information about the transition model in the form of sequences of state aactin pairs, well as the reward emitted by this transition. However, in some real-world scenarios, the reward might only be available for a small subset of data. For instance, in the case of recommending products available in online catalog to the user, the true long-term reward (user buys the product) only available for users who have browsed the item list for long enough and have purchased given item. It is possible to decompose the value function into reward- dependent and reward-free parts, as was done by [13] through the successor representation framework [14]. More recent approaches [15, 16, 17] use generative model to leam the occupancy measure over future states for each state-action pair in the dataset; its expectation corresponds to the successor representation. However, learning an explicit multi-step model such as [15] can be unstable due to the bootstrapping term in the temporal difference loss. Similarly to model- based approaches, our method will learn a reward-fres representation of the world, but will do so without having to predict sigh-dinensional observations and without having to do costly autoregressive rollouts. Thus, while our critic is trained without requiring rewards, it is much more similar value function than standard 1-step model.
I Learning conditional probability distribution over a highly complex space can be a challenging task, which is why it is often easier to instead approximate it using a density ratio specified by an inner product

---

in much lower-diment -ineensiienii latent space. To learn an occupancy measure over future states without passing via the temporal difference route, one can use noise-contrastive estimation INCE, 18, 19] to approximate the corresponding log ratio of densities as an implicit function. Contrastive learning was originally proposed as an alternative to classical maximum likelihood estimation, but has since then seen successes in static self-supervised learning [20, 21]. In reinforcement learning, NCE was shown to improve the robustness of state representations to exogenous noise [22, 23, 24] and, more recentl, to be an efficient replacement for traditional goal-conditioned methods [17].
3 Preliminaries
Reinforcement learning, We assume a Markov decision process M defined by the tuple (S, So, A, IP[-|s, ol, r, 7). where S is a state space, So 8 is the set of starting states, A is an action space, P][]s),0.) S S x A - A(S) is a one-step transition function", r x A- [Tiin: mmax] is reward function and Y [0.1) is a discount factor. The system starts one of the initial states So E So- At every timestep t= 1,2,,.., the policy :: 1.S+AA(.4) ((A), samples an action ~ (--o)))). The environment transitions into next state St+1 ~ P[- and emits a reward r; =))) With Markovian policy (a 8), we define the discounted occupancy measure conditionned on (8t,0s) to be
PE (+)+)() -1
Σ Al A PP
A1=1
With this notation in place, the objective is to maximize the discounted sum of returns over H steps:
H
Σ 1 (r(s,a))))))
max E TEll
ousr (8o)
(1)
We will study this problem in the offline setting: rather than learning by trial and error (by interacting with the environment), the algorithm instead must learn from an offline dataset of logged trajectories.
Value-based RL algorithms maximize cumulative episodic rewards by estimating the action value function under a policy æ, which can equivalently be expressed as an expectation under the discounted occupancy measure:
(s) (sur))
Σ
A 1 ((S++S)
E Baa (ra)) 1-y
Note that the occupancy measure equivalently be re-written in terms of the geometric distribution over the time interval (0,00) for infinite-horizon rollouts:
(80,00)) = FE - S++ at [0]]
This decomposition of the value function has already been used previous works based on the successor representation [14, 13] and, more recently, 7-models [15]. We will use it to efficiently learn an implicit density ratio proportional to the state occupancy measure using contrastive learning.
Noise-contrastive estimation Noise- conse-contrastive estimation [NCE, 18] spans a broad class of learning algorithms, at the core of which negative sampling [25]. NCE learns a metric space from positive and negative examples. Given reference samples, samples from a positive distribution (high similarity with reference points) and samples from a negative distribution (low similarity with reference points), contrastive leaming methods learn an embedding where positive examples are located closer to the reference points than negative examples. One of the most well-known and commonly used NCE objectives InfoNCE [19]:
(ar)) (vv)
max 0.0
E BaT
log
(4)
ed(z) vita
positives over some hypothesis class +: (0:X' 2} for input space X' latent space 2, Z.xx))()), y Prosiiiis (A') and P negatives (X'). Contrastive learning has been widely studied in the static unsupervised/ supervised
(xX))) denotes the entire set of distributions over the space X.
3

---

learning settings [26, 21, 20], as well as in reinforcement learning [27, 23] for learning state representations with desirable properties such as alignment and uniformity [28].
Solving Equation (4) for (d)", x/*) yields a ratio estimator f X' y - IR which decompases as (axy)= (x) T 2 (y) and, at optimality', captures the log-ratio of I positives (X)) and neggntives(
8 (x,y) olog P(y|z) Plyl
Implicit dynamics models via NCE. Various prior works [30, 23, 31] have studied the use of NCE to approximate a single-step dynamics model, where triplets (s,, 8t+11 have higher similarity than (5,,01,,1 effectively defining positive and negative distributions over trajectory data. More recently, contrastive goal-conditioned RL [17] used InfoNCE to condition the ratio estimator on goal states sampled from the replay buffer. These methods use asymetric encoders, using (()),a)) and (b(()+)))) where positive samples of St+At are sampled from the discounted state occupancy measure for t20.
The conditional probability distribution of future states given the current state-action pair can be efficiently estimated using an implicit model trained via contrastive learning over positive and negative feature distributions, as shown in Equation (6). Within each batch, the states used in positive examples for one batch element are used negative examples for every other batch element.
eloraa)) (aloaa))((((aaa)) (((1)a)))
-Ea)) log EArEAHUA, (loree) ((c))e))
(6)
Minimizing (INONCE over trajectory data yields a ratio estimator which, at optimality, approximates the future discounted state occupancy measure up multiplicative term per Equation (5),
f (St,,11,))-))) x log PiSt+Ar (T,00;0T]
P(st+AGT]]
Intuitively, f approximates a H-step dynamics model which has an implicit dependence on policy that collected the training data, but is time-independent since Equation (7) is optimized on average across multiple t, Ordinarily, training state-space models is hard when the dimensions are large, e.g. image-based domains. However, by using contrastive learning, we can learn this model without having to require predict high-dimensional observations, as similarity evaluated in a dimeesiiiai latent space (observe that in Equation (6) the inner product is computed in E, whose dimension we control, instead of X, which specified externally). An apparent limitation of the approach is that the probability of future states St+At At is recovered only up to a constant. However, it turns out that we can still use this model to get accurate estimates of the Q-values, as is described in the next section.
4 Estimating and Maximizing Returns via Contrastive Learning
In this section, we show how NCE can be used to learn a quantity proportional to a value function, and how the later can be used in a policy iteration scheme.
4.1 Estimating Q-values using the Contrastive Model
As shown in Equation (2), the Q-function at (st,at) can be thought of evaluating the reward function at states sampled from the discounted occupancy measure PT (St.a.). That is, to estimate a quantity akin to Q",, we can first estimate the occupancy measure and take weighted average of rewards over future states using the probabilities from the log-density ratio leameed by the contrastive model. Precisely, Equation (2) corresponds using an importance importance-eeighted estimator, where an optimal critic that minimizes Equation (6) approximates the density ratio from Equation (7). The positive samples come from the discounted state occupancy measure: we first sample a time offset At ~ Geometric(1 -7) (column in the dataset), and then sample state from the distribution of states at this given offset (row in the dataset). Formally, we can view this as applying infoNCE to a positive distribution S(t+A)) and a negative distribution formed as the product of the marginal distributions, [[I[s,a,]PP(s+++))))
The critic itself can be trained using the occupancy measure formulation specified in Equation (3) over all state-action pairs in given episode. However, Equation (3) needs to be re-adjusted to account for
truncation of the geometric mass function presented in Definition
See [29] for exact derivation.

---

Definition 1 (Truncated distribution) Let X be a random variable with distribution function Fx Y a called the truncated distribution of X with support [m.M] s.t. 0<m M if
P\Y= Fx (o (xly-xxyymmm
(8)
Fx (M) Fx (m)
We denote the special case of the truncated geometric distribution as TrunGeomm(.m..M)
The contrastive objective to train the ratio estimator to approximate the discounted occupancy measure over dataset D is then the dot product of features of current state and action ø with future state e), normalized by the product of negative samples
fInGNNE (ARANCE(O.))) (0.0)=
log
SEARE D.
EArEAuA, Maeea w(stia))
is possible that multiple optimal ratio estimators exist such that the multiplicative proportionality constant depends on the action. To avoid this, we adopt similar approach [17] and introduce a regularization term over the partition function, making the ratio estimator training objective be
Contrastive EInfoNCE + Apatttion Be, Se,
Log Σ
(10)
APEATUA
Now, suppose we found an optimal ratio estimator Combining Equation (3) with Definition 1, obtain the following form of the Q-function for an optimal ratio estimator f which minimizes Equation (6):
(NCR()
Ar-1
r(8 r(8t+ se at (ds
Ju-ai
A1=
1-
H-
BarnTrunGGrenn (1 [/
1 1-7
7 H-t
Bey ai [
Thineeeee (...1r)
(11)
Here, the offset At is a random variable sampled from TruneGeom(1 v,,,, where H the horizon of the MDP. Later on, we show that QNCE(s,a) Q(s,a) for all ES and a,a' EA, which makes the contrastive Q-values suitable for policy evaluation.
4.2 Efficient Estimation using Random Fourier Features
major issue with using QNCE out-of-the-box is that it is computationally expensive, requiring evaluation of the inner product ((5),))) (($++4))) with large number of future states. The underlying cause of this computational overhead is the RBF kernel term (a) (()+ If we instead used a linear kernel, the constant term ((s),a)) would be factored out, and could separately keep track of weighte future expected features. This would (1) reduce the computational complexity of actor updates over 'D from ((D)N) ( + N) and (2) reduce the variance of the representation if averaging features of future states using exponential moving average. It turns out that the RBF kernel can be approximately linearized by using random Fourier features 31].
Lemma 1 (Adapted from [32] Lei a,y Rdd be unit vectors, and let Fw,,(ar) = / 2 cos (Wa+b)
where W NNrmma(O.I) and b~ Uniffor(((.22 Then, E[Fw.o()]] Fw.o())]==
Lemma is straightforward modification of the result from [32] and allows us to reduce the RBF kemel an expectation over d-dimensional random feature vectors:
QxCE(S),0)= 1
EAt- Twwn:eomm---..... E (e (on (ot.laaaaaltttt.a)))aaaaaa.a
/w.o(o(sr,ae)) T BA /ttps///ww//aa///////////a//onae
Fw.o(((sr.ae)) (((())
(12)
5

---

The advantage of using the RFF approximation is that it allows to split the exponential term inside the expectation and separately keep track of the policy-dependent, reward-weighted future state probability term, while the state-action dependence term is learned online. Intuitively, the ((rr) term accumulates the Fourier features of future states re-weighted by the corresponding reward averaged over the geometric mixture of future states. Since does not depend on the current state st. it can be tracked using memory bank (o( (T), which is updated via an pooonentallmovinne average to reduce variance.
4.3 Learning the Policy
Once the policy evaluation phase completes and we have an estimate QNCE. we optimize a policy to maximize the returns predicted by this Q-value. We can decode the policy by minimizing its Kullhack-Leibler divergence to the Boltzmann Q-value distribution (see [33D, which can be efficiently done by minimizing the following objective:
SaEA Alor ]
((srr)/T
Pollicy (0) = EeinD DKL (Tt)
(13)
Note that in discrete action spaces, minimizing Equation (13) leads to soft version of the greedy policy decoding Tgreedy (s) argmax QNCIE( for & S.S. In practice, we approximate the KL. term
in Equation (13) using Na Monte-Carlo action samples T -1)
Decoding in such way lead to sampling out-of-distributtion actions in regions with low dataset coverage, thus making the QNCE estimator less accurate. To mitigate this issue, we follow prior work [34, 35, 36] and add policy behavior cloning term which prevents the new policy from straying too far away from the data:
(BC(() DD (a)s)
(14)
for the entropy estimator H(((s)) = -aaa((() [log=(a| s)], We add this extra loss Policy learn policy æ which prioritizes high Q-values that are well-supported by the offline dataset D. Thus, the final policy optimization objective becomes
P (0) = Policy (0) + ABC(BC(0).
(15)
Lemma tells us that using CVL surrogate Q-function corresponds step of conservative policy improvement, where π satisfies soft constraints of Equation (13) and small EpD (T()))( via the BC term. Its proof is located in Section 6.2.
Lemma 2 (Contrastive policy improvement) Let P be policy and let
Oiee
ming ED CCiie (0.)). ()./ If
(() = argminn (r(s)
chicleril/ ShEA e da
(16)
well
then (=(())) (s,a)) all (8,a) DH.
4.4 Practical Implementation
We now present our complete method, which can be viewed as an actor-critic method for offline RL. We learn the ratio estimator via contrastive learning (Equation (10)) and learn the policy via Equation (15). We will interleave these steps in most of our experiments, but experiments in Section 6.3 show that the ratio estimator can be pretrained e.g. in the presence of unlabeled data from related tasks. We summarize the method in Algorithm 1.
4.5 Interpretations and Connections with Prior work
The main distinction between Contrastive Value Learning and prior works consists specifically in representing the Q-values in a two-step decomposition: the Q-value is represented as an occupancy measure
'This idea can be adapted to online leaming settings well by clipping policy improvement steps so that & doesn't change too fast under newly collected data.

---

Algorithm 1: Contrastive Value Learning (CVL)
Input Dataset D~ J4, 10,c networks, temperature parameter 7, exponential moving average
parameter /
for epoch j=1.2....JJ do
for minibatch B~D do
/* Update density ratio estimator using Equation (10)
Update ((G++,,,(++1 using Vo.w<icce( (6()),66)))
/* Estimate the contrastive Q-function
((s.a) < Equation (12) if using RFF, otherwise Equation (II);
/* Decode policy from Q-function using Equation (15) Update To using (o(fnaicy(0))
/* Update future state encoder using EMA
(-
EMA t Bw00++++++
B)
/* Update future state features weighted by rewards
c(+1) < EEA -B(r++a)):
SEMA
Median
IQM
Mean
BC
COL CQL+UDS CVL
2500 3000
3500
2400
3000
3600
4200
2400 2800 3200 3600
Figure Metaworld benchmark. (Left) evaluate CVL on tasks from Metaworld, a subset which are shown here. (Right) Compared with three offline RL baselines, CVL achieves statistcalllyssinniffii significant improvements in offline performance. Results are reported over random seeds.
weighted by the reward signal; the occupancy measure itself is represented using a powerful likelihood- based model parameterized using an implicit function. Decoupling the learning of the occupancy measure from reward maximization allows, among others, for efficient pretraining strategies on unlabeled data, i.c. trajectory data without reward information, and can used to learn provably optimal state representations for any reward function [37]. While CVL is similar in spirit to the successor representation [14, 13], the occupancy measure learned by CVL much richer than that of SR, as it captures the entire distribution over future states instead of only the first moment. Another method, y-models [155], is closely related to CVL,, but uses a surrogate single-step TD objective to learn the occupancy measure, similarly to C-learnin [16].
5 Experiments
Our experiments aim to answer three questions. First, we study how CVL compares with baseline approaches large benchmark of state-based tasks. Our second set of experiments look image-based tasks, testing the hypothesis that CVL scales to these tasks more effectively than the baselines. We conclude with ablation experiments. Our main point of comparison will be high-performing offline RL method, CQL [4]. While CVL learns an implicit model, that model is structurally more similar to value-based RL methods than model-based methods, motivating our comparison a value-based baseline (CQL).
Metaworld. We first test approach the MetaWorld benchmark [1]], which consists of 50 robotic manipulation tasks such as open a door, pick up an object, reach certain area of the table, executed by robotic arm (see Figure (left)). This domain is an ideal testbed for CVL, as it allows for both full-state and image-based experiments, has a dense and informative reward function thus decoupling the problem of representation learning from exploration, and is challenging for model-free methods which leaves room for improvement. While the original Meta World domain has been used to evaluate online RL agents, we create an ad hoc dataset suitable for offline learning. To do so, we train Soft Actor-Critic [33] from full states on each of the 50 tasks separately for 500k frames, and save the resulting replay buffer, which forms the training dataset. As shown Figure (right), CVL. manages considerably improve upon strong baselines such as behavior cloning, COL and CQL. with UDS [10p.. We report the results on all tasks
FFor CQL+UDS, we combine all data from the current task with unlabeled data from related tasks with rewards set O. In the absence of related tasks, pre-train the ratio estimator the current task with 0 rewards.
7

---

Task
BC
CQL
CVL
OX 2011
door-close
571 +: 9.9
4249 1: 269.9
4480 305.1
1073
door-open
178 + 4.0
2099 0.9
3389 + 76.6
drawer-close
2414 = 1736.5
3964 + 1634,9
2177 = 1679.5
drawer-open
1030 1 104.2
820 56.0
2543 115.0
0.00
Table 1: Offline RL with Images, We compare CVL to baselines on four offiie, maage-based lasks from Meta- World offline image-based tasks on random seeds.
2305
2005
2006 45% 2000
Figure 3: RFF ablation. CVL with RFF (orange) per-
forms slightly better than without RFF (blue).
medium--
medium
mindom
mediumer
medium
random
Trok
mediumer
medium
random
walkerZd
-33+11
+415
walker2d
+11
43=21
-22+22
walker
-4.27
+9 +3
+21 +6
+23 +6
ant
-66+1
-21
-1447
ant
hopper
+59 111
-15 45
+40 1.8
hopper
+21 15
-14 =00
+270
154
happer
-0.32
+133
Table 2: Offine RL. with full states. We compure CVL to CIQL on the mbbriis salee D4KL
Table 3: Comparison successor features. Weoompue successor leatires (129)) CQL on D4RL.
Table 4: Comparison to IQI. We compure IQU (100)
CUL on D4RL.
of the MetaWorld suite over 5 random seeds, according to the aggregation methodology proposed by [12]. Per-environment scores available in Table 7.
D4RL Table 2 shows the relative improvement in normalized scores of CVL over CQL [4], a strong offline RL baseline, the offline RL roboties suite D4RL [38]. Notably, CVL is able outperform CQL on data coming from a random policy. Moreover, Table and Table compare two baselines, successor features (inspired from [39]) and IQL [40]. CQL.
Image-based experiments Our working hypothesis is that contrastive formulation of the value function acts in itself as a pre-training mechanism via the prism representation learning. For this reason, we conduct further experiments on 4 image-based tasks from the MetaWorld suite (similarly to full-states, the dataset was obtained from the SAC replay buffer trained on rendered images). Results presented in Section 5 show that CVL is also able learn meaningful Q-values and achieve good empirical performance on hard image-based tasks.
Ablations In Section 6.3, assess the similarity between contrastive and true Q-values on the contin- wous Mountain Car environment [41] by fitting CVL to the data from SAC's [33] replay buffer. Figure 8 (left) shows the contrastive Q-values on a log-scale, evaluated on trajectories from the SAC replay: for com- parison, we also show the Q-values learned by online SAC in Figure 8 (right). Note that the value function learned by CVL conserves the same topology as the true value function, up to multiplicative rescaling.
Discussion
This paper presented RL algorithm that learns a contrastive model of the world, and uses that model to obtain Q values by estimating the likelihood of visiting future states. Our experiments demonstrate that this approach can effectively solve a large number of offline RL tasks, including from amage-based observations. Our pretraining results hinted that CVL can be pretrained on datasets from other tasks, and we are excited to pretrain our model on datasets of increasing size.
Limitations. One limitation of approach is that it corresponds single step of policy improvement. This limitation might be lifted by training the contrastive model using a temporal difference update for the contrastive model [16, 42]. A second limitation is that the RFF approximation can be poor when the Fourier dimension is small, which has not been a case in our experiments as CVL+RFF performed on par with full kernel CVL. We tried train the contrastive model using non-exponentiated features (akin to [43]), but failed to achieve satisfactory results. Figuring out how to effectively train these spectral models remains an important question.
References
[1] A. Kumar, A. Singh, F. Ebert, Yang, C. Finn, and Levine. Pre-training robots: Offline enables learning new tasks from handful of trials. arXiv preprint 21X1v:2210.05178, 2022.
8

---

[2] Y. Wu, G. Tucker, and O. Nachum. Behavior regularized offline reinforcement learning, arXiv preprint 1,11, 2019.
[3] S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without exploration.
In International Conference Machine Learning, pages 2052-2062. PMLR, 2019.
[4] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.0779, 2020.
[5] R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims. Morel: Model-based offline reinforcement learning. arXiv preprint arXiv:2005,cm 0,950, 2020.
[6] T. Yu, A. Kumar, R. Rafailov, A. Rajeswaran, S. Levine, and C. Finn. Combo: Conservative offline model-basee policy optimization. Advances in neural information processing systems, 34: 28954 28967, 2021.
[7] T. Yu, G. Thomas, L. Yu, S. Ermon, J. Y. Zou, S. Levine, C. Finn, and T. Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129--11119111111 2020.
[8] A. Argenson and G. Dulac-Amold. Model-based offline planning. arXiv preprint 25855:2008.05556 2020.
[9] R. Munos. Error bounds for approximate policy iteration. In JCML, volume 3, pages 560-567, 2003.
[10] T. Yu, A. Kumar, Y. Chebotar, K. Hausman, C. Finn, and S. Levine. How to leverage unlabeled data in offline reinforcement learning. arXiv preprint arXiv:2202.017411 2022.
[11] T. Yu, Quillen, He, R. Julian, Hausman, C. Finn, and Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning. pages 1094 1100. PMLR, 2020.
[12] R. Agarwal, M. Schwarzer, P. S. Castro, C. Courville, and M. Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304-2993 2021.
[13] A. Barreto, Dabney, R. Munos, J. Hunt, Schaul, H. Van Hasselt, and Silver. Successor features for transfer in reinforcement learning. arXiv preprint 1rX1v:1606.05312, 2016.
[14] P. Dayan. Improving generalization for temporal difference learning: The successor representation. Neural Computation, 5(4):613 624, 1993.
[15] M. Janner, Mordatch, and S. Levine. Generative temporal difference learning for infinite-horizon prediction. arXiv preprint 2010::20000.4496 2020.
[16] B. Bysenbach, R. Salakhutdinov, and S. Levine. C-leaming: Learning to achieve goals via recursive classification. arXiv preprint arXiv:2011.0890) 2020.
[17] B. Eysenbach, T. Zhang, R. Salakhutdinov, and S. Levine. Contrastive learning as goal-conditioned reinforcement learning- arXiv preprint arXiv:2206.07568. 2022.
[18] M. Gutmann and A. Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 297-304. JMLR Workshop and Conference Proceedings, 2010.
[19] A. v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 10X7vv/807.03440 2018
[20] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning- In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2020.
[21] T. Chen, S. Komblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607, PMIR, 2020.

---

[22] A. Srinivas, M. Laskin, and P. Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. International Conference on Machine Learning, 2020.
[23] B. Mazoure, R. T. d. Combes, T. Doan, P. Bachman, and R. D. Hjelm. Deep reinforcement and infomax learning. Neural Information Processing Systems, 2020.
[24] R. Agarwal, M. C. Machado, P. S. Castro, and M. G. Bellemare. Contrastive behavioral similarity embeddings for generalization in reinforcement learning. arXiv preprint arXiv:2101.05265, 2021.
[25] T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv: 2013.
[26] R. D. Hjelm, A. Fedorov, S. Lavoie-Marcillonn, Grewal, P. Bachman, A. Trischler, and Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint 1rXiv:1808.06. 06670, 2018.
[27] H. Kim, J. Kim, Y. Jeong, S. Levine, and H. O. Song. Emi: Exploration with mutual information. arXiv preprint arXiv: 181000100, 2018.
[28] T. Wang and P. Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. International Conference on Machine Learning, pages 9929-9939. PMLR, 2020.
[29] Z. Ma and M. Collins. Noise contrastive estimation and negative sampling for conditional models: Consistency and statistical efficiency. arXiv preprint arXiv:1809.01812, 2018.
[30] Y. Du and I. Mordatch. Implicit generation and modeling with energy based models. Advances in Neural Information Processing Systems, 32, 2019.
[31] O. Nachum and M. Yang. Provable representation learning for imitation with contrastive fourier features. Advances in Neural Information Processing Systems, 34:30100-30112, 2021.
[32] A. Rahimi and B. Recht. Random features for large-scale kemnel machines. Advances in neural information processing systems, 20, 2007.
[33] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870 PMLR, 2018.
[34] K. W. Cobbe, Hilton, O. Klimov, and J. Schulman. Phasic policy gradient. In International Conference on Machine Learning, pages 2020-2027. PMLR, 2021.
[35] Y. Zhao, R. Boney, A. Ilin, J. Kannala, and J. Pajarinen. Adaptive behavior cloning regularization for stable offline-to-online reinforcement learning. 2021.
[36] M. Schwarzer, N. Rajkumar, M. Noukhovitch, A. Anand, L. Charlin, D. Hjelm, Bachman, and A. Courville. Pretraining representations for data-efficient reinforcement learning. arXiv preprint wrXii:2106.04 04799, 2021.
[37] A. Touati and Y. Ollivier. Learning one representation to optimize all rewards. Advances in Neural Information Processing Systems, 34, 2021.
[38] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020.
[39] A. Filos, C. Lyle, Y. Gal, S. Levine, N. Jaques, and G. Farquhar. Psiphi-learning: Reinforcement learning with demonstrations using successor features and inverse temporal difference leamnng. In International Conference on Machine Learning, pages 3305-3317. PMLR, 2021.
[40] I. Kostrikov, J. Tompson, R. Fergus, and O. Nachum. Offline reinforcement learning with fisher divergence critic regularization. arXiv preprint arXiv:2103.08050, 2021.
[41] A. W. Moore. Efficient memory- based learning for robot control. 1990.
10

---

[42] L. Blier, C. Tallec, and Y. Ollivier. Learning successor states and goal-dependent values: A mathematical viewpoint. arXiv preprint arXiv:2101.07123, 2021.
[43] J. Z. HaoChen, C. Wei, A. Gaidon, and T. Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. Advances in Neural Information Processing Systems, 34:5000-501, 1, 2021.
[44] L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures, In International Conference on Machine Learning, pages 1407-1416. PMLR, 2018.
[45] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
[46] J. L.. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint 1rXi::1607006450, 2016.
[47] C. X. Wang and W. P. Tay. Practical bounds of kullback-leibler divergence using maximum mean discrepancy, arXiv preprint arX1v:204400003, 2022.


Reviews:
[{"review_id": "pJZ5vBQWPZp", "reviewer": "Reviewer_W8jU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a novel way, named contrastive value learning, to learn the Q function, in which they decompose the Q function by first approximating the occupancy measure using an implicit function, then representing the Q-value as the occupancy measure weighted by the reward signal. The effectiveness of the proposed method is demonstrated on multiple complex robotics benchmarks.", "summary_of_recommendation": "The paper proposes a novel way to learn the Q function, which especially fits the offline RL setting and offline-to-online RL. The proposed method demonstrates good performance on a set of offline RL tasks. I like the presented idea and the paper is generally easy to follow, however, the writing could be improved to minimize ambiguity and several important discussions or experiments are missing in the current version. Based on these concerns, I would suggest Weak Reject.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please check the weakness listed in the Review section.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths: \n- This paper proposes a novel way to learn the Q function. It decomposes the Q function in a two-step way, first, approximating the occupancy measure using an implicit function, then representing the Q-value as the occupancy measure weighted by the reward signal. This decomposition is novel and has benefits in offline settings where the Q value is learned without using TD learning.\n- The proposed method shows obvious improvement over baselines on a set of offline RL tasks. \n\nWeakness:\n- For the writing, I think there are several questions that could be clarified:\n    1. The usage of critic/optimal critic causes unnecessary ambiguity. The critic is often used to represent value function in RL, for example, in SAC, but the \"critic/optimal critic\" in this paper differs. e.g., in Line 152, \"Precisely, Equation (2) corresponds to using an importance-weighted estimator, where an optimal critic that minimizes Equation (6) approximates the density ratio from Equation (7)\".\n    2. Line 260: “In Section 6.3, we qualitatively assess the similarity between contrastive and true Q-values on the continuous Mountain Car environment”. There is no section 6.3 available. \n    3. Might be shown in the appendix, but I haven’t found Table 5 (Line 249) in your paper,\n    4. Line 258: “Results presented in Section 5 show that CVL is also able to learn meaningful Q-values and achieve good empirical performance on hard image-based tasks”. From section 5, I can only find results on offline RL with images, while the analysis of the ability to learn meaningful Q-values is missing. \n    5. In Eq. (12), the equation is shortened by using $\\xi(\\pi)$, it would be better to give an intuitive explanation. Also, the transpose of $F_{w, b}(\\phi(s_t, a_t))$ is missing.\n \n- The paper proposes a way to estimate the Q values, but the analysis of the ability to learn meaningful Q-values is missing. Actually, from the paper, I can not find a thorough discussion of the quality of the learned Q compared to the Q function trained in a regular way.\n- To calculate Eq. (6), the negative samples are very important according to previous results, however, the way to draw negative samples is not very clear in the paper.\n- In Eq.(12), I can understand that the usage of RFF approximation is to achieve efficient estimation, however, the paper doesn't offer enough details on how to implement it and the benefits or drawbacks of using this approximation.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "JKDoegR7d3", "reviewer": "Reviewer_pfTt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces Contrastive Value Learning (CVL) for offline reinforcement learning (RL). Unlike traditional model-based or model-free methods, CVL estimates Q-values through a contrastive learning process. It leverages a contrastive loss function with a geometric distribution over time offsets to enable implicit modeling of the value function. To achieve computational efficiency, the algorithm utilizes random Fourier features. The experiments show that the proposed method obtains promising results on large benchmarks. It is also claimed that the proposed method can benefit from pretraining on unlabeled data.\n\n", "summary_of_recommendation": "The paper is well-written and easy to follow. However, I have questions about the problem statement and experiment settings listed in the above section. I gave my recommendation based on the current version of the paper. However, I will be happy to update it after my questions are resolved.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. The paper mentions the use of random Fourier features for efficient estimation. I was wondering how well the approximation is, and how the computation performance after approximation compares to the baseline model-free algorithms\n\n2. I appreciate the experiments on benchmark tasks such as D4RL. However, only three environments in D4RL are shown in the experiment section, which I believe is not sufficient to obtain a comprehensive understanding of the proposed method's performance. There are a lot of different tasks in different scenarios in D4RL, such as antmaze, mujoco, adroit, etc. I would suggest adding more tasks to the experiments. The baselines are well tested on these benchmarks, so adding these comparisons should not be a burden.\n\n3. How sensitive is the CVL algorithm to the choice of hyperparameters? Were there any specific hyperparameters that significantly influenced its performance? \n\n4. One biggest challenge in offline RL is the problem caused by distributional shift. I would appreciate it if the authors can provide an explicit discussion on how the proposed method deals with out-of-distribution states and actions, compared to other SOTA algorithms.\n\n4. I would suggest moving some of the ablation study to the main body of the paper. \n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Strengths\n\nThe paper is well-structured, provides a clear explanation of the proposed method, and presents good experimental results. I appreciate that there are many discussions on the relationship between the proposed method and prior works. \n\n## Weaknesses\n\nPlease see my questions for rebuttal in the below section.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "2GgIgZZ67j", "reviewer": "Reviewer_X3Ux", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose CVL, an offline RL algorithms based on contrastive learning. An implicit dynamics model is learning via contrastive learning, which is then used to fit a contrastive Q function. This Q function is then used to update a policy. ", "summary_of_recommendation": "The paper has an interesting idea, but the presentation is unclear and confusion in a lot of places. The experiments the authors show are promising, but the authors created their own offline-RL benchmark and the evaluations on the classical benchmarks and baselines are lacking and, therefore, I have some suspicions.\n\n\n=== After Rebuttal ===\n\nI thank the authors for addressing and answering my questions. Based on this I am changing my score to a weak accept.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "See review above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper poses an interesting idea of using contrastive learning in the offline RL setting.\n\nHowever, I lean towards rejection for the following reasons:\n- The paper does not have a clear robotics focus and would, therefore, be better suited for a different conference.\n- I think the paper really lacks in clarity which for me makes it hard to follow and check the claims. \n   - Statements like \"Solving Eq. 4 yields a critic\" are just stated and not well explained. Personally, for me the connection is not obvious. This really is the crucial connection between how contrastive learning and RL can be connected and it should be made more clear.\n   - At least in the preliminaries, the notation seems to slightly change, e.g. sometimes the length-K trajectory uses parenthesis, sometimes it uses curly brackets, sometimes the index k is used and other times it's t or delta t, ... . These inaccuracies make it harder to follow straight away.\n   - The paper ends somewhat abruptly, where the discussion is very short and it feels like the authors just ran out of space and pushed a whole chunk to the appendix.\n- Given that the CVL is presented as an offline-RL algorithm, I think it should be more rigorously benchmarked on standard offline RL tasks with more baselines, e.g. TD3+BC, IQL. Table 2, is an attempt at that, but it only compares against CQL on a small subset of the environments, which makes me somewhat suspicious. Further, there are no uncertainty quantifications in Table 2.", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "0qhrj04r4ZZ", "reviewer": "Reviewer_GZ9B", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Inspired by the large data regime, offline reinforcement learning (RL) changes the way the agent is being learned via trial and error by learning directly from existing data as in supervised learning. One way to find a policy that well-support the training data is by applying model-based RL approaches. Learning a dynamical system that predicts a one-step state prediction is usually challenging, especially with high dimensional inputs. \n\nIn this work, an alternative model is proposed to support implicit multi-step predictions while modeling the value of each action. Hence, the approach is not suffering from the challenge of predicting a high-dimensional state. This approach decouples the learning of value-like quantity into learning the future state occupancy measure using contrastive learning and then re-weighting it with the reward samples.\n\nThis approach outperforms other offline RL baselines on complex robotics benchmarks (MetaWorld).", "summary_of_recommendation": "To summarize, I have only one recommendation for comparing CVL with SR if applicable. I am really excited to see how well CVL can outperform the capabilities of SR. \n\nUPDATE: After discussion, I believe the paper should be accepted.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- As I mentioned, an ablation study that compares both the proposed approach and SR would motivate the use of CVL over SR and verify the claim mentioned in the paper. Is there a point in not including such a study? Is it not feasible for a reason I miss? I believe this study can be similar to the pre-training one where you learn SR for the related tasks, and fine-tune each critic using the reward signals.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "One of the strength points of this work is that learning the future state occupancy measure does not require any TD learning. Second, this approach releases the burden of predicting high dimensional states by learning a quantity proportional to the likelihood of visiting future states. Moreover, the pre-training study shows the capability of CVL for learning some common representations across related tasks. \n\nAs mentioned, this approach is similar to the successor representation in the sense of decoupling the learning of the value function in terms of some fixed features and weighted by the reward signals. However, an ablation study that compares the two approaches does not exist. Also, to verify the claim mentioned in the paper \"While CVL is similar in spirit to the successor representation [14, 13], the occupancy measure learned by CVL is much richer than that of SR, as it captures the entire distribution over future states instead of only the first moment\".", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces Contrastive Value Learning (CVL) for offline reinforcement learning of the Q-function. The ideas are interesting and experimental results are strong. The paper does not have real world robotics experiments. However, while some contributions in reinforcement learning are not necessarily directly applicable to robotics, offline reinforcement learning is one domain with a connection with real-world robots where offline data is often available, and, online data collection is expensive. Reviewers highlighted places where the writing could be clarified. The authors already modified the paper but I suggest to go through the paper again for the final version.
Decision: Accept (Poster)