Zero-Shot Reinforcement Learning from Low Quality Data
Authors: Scott Jeen
Tom Bewley
Jonathan Cullen
Venue: NeurIPS 2024 poster (2024)
Keywords: reinforcement learning
offline reinforcement learning
unsupervised reinforcement learning
zero-shot reinforcement learning
Primary area: reinforcement_learning
TL;DR: We propose methods for improving the performance of zero-shot RL methods when trained on low quality offline datasets.

Abstract: Zero-shot reinforcement learning (RL) promises to provide agents that can perform _any_ task in an environment after an offline, reward-free pre-training phase. Methods leveraging successor measures and successor features have shown strong performance in this setting, but require access to large heterogenous datasets for pre-training which cannot be expected for most real problems. Here, we explore how the performance of zero-shot RL methods degrades when trained on small homogeneous datasets, and propose fixes inspired by _conservatism_, a well-established feature of performant single-task offline RL algorithms. We evaluate our proposals across various datasets, domains and tasks, and show that conservative zero-shot RL algorithms outperform their non-conservative counterparts on low quality datasets, and perform no worse on high quality datasets. Somewhat surprisingly, our proposals also outperform baselines that get to see the task during training. Our code is available via the project page https://enjeeneer.io/projects/zero-shot-rl/.

Zero-Shot Reinforcement Learning from Low Quality Data
Scott Jeen University of Cambridge srj380cam. ac uk
Tom Bewley University of Bristol tomdbezley@gmaill com
Jonathan M. Cullen University of Cambridge jmc990cam ac. uk
Abstract
Zero-shot reinforcement learning (RI.) promises provide agents that can per- form any task in an environment after an offline, reward-free pre-training phase. Methods leveraging successor measures and successor features have shown strong performance in this setting, but require access to large heterogenous datasets for pre-training which cannot be expected for explore how the performance of zero-shot RL methods degrades when trained on small homogeneous datasets, and propose fixes inspired by conservatism, a well cell-established feature of performant single-task offline RL algorithms. We evaluate our proposals across various datasets, domains and tasks, and show that conservative zero-shot RI. algorithms outperform their non-conservative counterparts on low quality datasets, and perform no worse on high quality datasets. Somewhat surprisingly, our proposals also outperform baselines that get to see the task during training. Our code is available via the project page https: httr///njjeeeeeee wtt.ioo/poooo projects/zenee saaro-shot-rr/.
real problems. Here, we
Introduction
Today's large pre-trained models generalise impressively to unscen vision [70] and language [9] tasks, but not to sequential decision-making problems. Zero-shot reinforcement learning (RL) attempts to correct this, asking. informally: we pre-train an agent dataset of reward-free transitions such that it can perform any downstream task in an environment? Recently, methods leveraging successor features [5, 7] and successor measures [6, 82] have emerged as viable zero-shot RL candidates, returning near optimal policies for many unseen tasks [83].
These works have assumed access to a large heterogeneous dataset of transitions for pre-training. In theory, such datasets could be curated by highly-exploratory agents during an upfront data collection phase [32, 10, 18, 62, 63, 35, 48]. However, in practice, deploying such agents in real systems can be time-consuming, costly or dangerous. To avoid these downsides, it would be convenient to skip the data collection phase and pre-train historical datasets. Whilst these common in the real world, they are usually produced by controllers that are not optimising for data heterogeneity [16], making them smaller and less diverse than current zero RL methods expect.
Can we still perform zero-shot RL using these datasets? This the primary question this paper seeks to answer, and one we address in four parts. First, we investigate the performance of existing methods when trained on such datasets, finding their performance suffers because of out-of-distribution state-action value overestimation, a well-observed phenomenon in single-task offline RI. Second, we develop ideas from conservatism single-task offline RL. for use in the zero-shot RI. setting, introducing straightforward regularizer of OOD values or measures that can be used by any zero-shot RL algorithm (Figure 1). Third, we conduct experiments across varied domains, tasks and datasets, showing conservative zero-shot RL proposals outperform their non-conservative counterparts, and surpass the performance methods that get to see the task in advance. Finally, we establish 38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

VC FB Squllaet
Q rcollect
Q Prollect
Q Reval
QFB Faval
VC-PP
Q
10 FL
Foolled
Feoilled
FD
G zevo
B
Scollne
PB
Z
Z
Figure Conservative zero-shot RL.. (Left) Zero-shot RL methods must train on a dataset collected by a behaviour policy optimising against task &collect. yet generalise to new tasks Zeval- Both tasks have associated optimal value functions Q and Q for a given marginal state. (Middle) Existing methods, in this case nrwaardbbckwwac representations (FB), overestimate the value of actions not in the dataset for all tasks, (Right) Value-conservative forward-back for all tasks. Black dots (+) represent state-action samples present the dataset.
representations (VC-FB) suppress the value of actions in the dataset
that our proposals do not hinder performance on large heterogeneous datasets, meaning adopting them presents little downside. We believe the ideas explored in this paper represent step toward real-world deployment of zero-shot RL methods.
2 Preliminaries
Markov decision processes A reward-free Markov Decision Process (MDP) is defined by M A, T, } where S is the state space, A is the action space, T S x A - A(S) is the transition function, where ((X) denotes the set of possible distributions over X, and y E [0, 1) is the discount factor [77]. Given (so. un) E S x A and policy π S - A(A), we denote Pr(-|so, (o, T) and B[|| |8. (o; π] the probabilities and expectations under state-action sequences (8t, de de)120 starting at (80, ) and following policy π, with ~ T( T(- ((-1) and a; x (-1s)). Given a reward function r: S:S R20. the Q function of π for is OF = E.00 [[)(+++1)) so, ag. T).
static offline dataset of reward-free Zero-shot RL For pre-training, the agent has access
transitions D = [(; a,, $1+1) generated by an unknown behaviour policy, and cannot interact with the environment. At test time, reward function Teval specifying task is revealed and the agent must return a policy for the task without any further planning or learning. Ideally, the policy should maximise the expected discounted return on the task 2'
So. an x/. The reward function is specified either via a small dataset of reward -abellled states Diahelled = {(,, Teval |*= with & 10, 000 or Intuitively, the zero-shot RL. problem asks: is it possible to train an agent using a pre-collected dataset of transitions from environment such that, test time, it can return the optimal policy for any task in that environment without any further planning or learning?
explicit function + Teval (s) (like 1 at goal state and 0 elsewhere).
State-of-the-an zero-shot RL methods leverage either successor measures [6] or successor features [5], with the former instantiated by forward backward representations [82] and the latter by universal successor features [7]. The remainder of this section introduces these ideas.
Successor measures The successor measure M (so, a).-) over S is the cumulative discounted time spent in each future state 8t+1 after starting in state so. taking action dg, and following policy m thereafter:
M (Su, ag, X) = En207 Pr(s E S|su, ag, T) A X C.
(1)
The Q function of policy æ for task is the integral of with respect to M
Q. (Su. ag) == Jayes ((s))M (So. ap 84).
(2)
The borward-backward framework representations [82] approximate the successor measures of near-optimal policies for any task. Let p be arbitrary state distribution, and d be representation

---

space, FB representations are composed of forward model F S x A x R - R backward
model B : S RR and polices (Tz)sERs. They are trained such that
00, 2) ((s)p(ds) A so S E A, C SS = (SD. as au. X) ~ F(so, 0u,
(3)
and
( (s) R arg F Frg (,, (5.4)
(4)
Intuitively, Equation says that the approximated successor measure under ns from (50. an) to s is high if their respective forward and backward embeddings are similar i.e. have large dot product. By comparing Equation 2 and Equation 3, we see that an FB representation can be used to approximate the function of x, with respect any reward function as:
Qt (s0: 0o) s Jaes F co, 00,z) (..)))B()))))) (ss)p(ds)
(5)
F(s0, dn, 2) " (e=plr(s)
Training F and B is done with TD learning [71, 78] using transition data sampled from D:
LFB
at, =) B(++) -7 (s++11, (=(+++1), B(x+))²
(F, or. 2) B ((s++))], (6)
where 84 sampled independently of (st, des 8141), F and B are lagging target networks, and 2 task sampling distribution. The policy is trained in an actor-critic formulation [47]. See [82] for full derivation of the TD update, and our Appendix B. for practical implementation details including the specific choice of task sampling distribution Z.
By relating Equations 4 and 5, we find = Ex-plri (r(s)((() for some reward function r. At test time, we can use this property to perform zero-shot RL. Using Dubelled, we estimate the task as Feval As Ex (s) (B) and pass as an argument to u.. Zeval lies within the task sampling distribution 2 used during pre-training, then (s))) Rs arg max, Q (8, a), and hence this policy is approximately optimal for Teval-
(Universal) successor features frattur assume access to basic feature map y : S + Rd that embeds states into a representation space, and defined as the expected discounted sum of future features 3 (So, 40) == EEΣ,>0 7'((s(+1))ss ao, x] [5]. They are made universal by conditioning their predictions on a family of policies
(50; 0n, =) E Σ 7'p(s(41) do, T2
A S,00 E A, Rd
(7)
with
(s) arg max (s,, a, 22, (so, ESExx 2
(8)
Like FB, USFs are trained using TD learning on ((((+1)))))))))))))))))()))))) 2 Lsp )xD. (x)((s, a,, 2) == 2b T
(8t41), ( 2T)))
(9)
where v is a lagging target network, and 2 is the same & sampling distribution used for FB. We refer the reader to [7] for derivation of the TD update and full learning procedure. Test time policy inference is performed similarly to FB. Using Dibellldd the task inferred by performing a linear regression of reval onto the features: Zeval arg min, EgnDannal (()ea)())) - -s(s) =)²] before it is
passed as an argument to the policy.
Zero-Shot RL from Low Quality Data
In this section we introduce methods for improving the performance of zero-shot RL methods on low quality datasets. In Section 3.1, we explore the failure mode of existing methods on such datasets. Then, in Section 3.2, we propose straightforward amendments to these methods that address the failure mode. Finally, in Section 3.3, we illustrate the usefulness of our proposals with a controlled example. We develop our methods within the FB framework because of its superior empirical performance [83], but our proposals are also compatible with USF, We push their derivation Appendix for brevity.
3

---

RND
Random
30
(og(()
20
10
Return (IQM)
1000
1000
500
100000
100
0
100
Learning steps (k)
Learning steps (k)
Figure 2: FB value overestimation with respect to dataset size quality, Log Q values IQM of rollout performance on all Maze tasks for datasets RND and RANDOM. Q values predicted during training increase as both the size and "quality" of the dataset decrease. This contradicts the low return of all resultant policies (note: return of 1000 is the maximum achievable for this task). Informally, say the RND dataset is "high" quality, and RANDOM dataset is "low" quality-see Appendix A.2 for more details.
3.1 Failure Mode of Existing Methods
To investigate the failure mode of existing methods examine the FB loss (Equation 6) more closely. The TD target includes an action produced by the current policy a++1 (St+1))) Equation 4 shows this the policy's 's current best estimate of the Q-maximising action in state s for task z. For finite datasets, this maximisation does not constrain the policy to actions observed in the dataset, and can become biased towards -ittt--disttitttionn (OOD) actions thought to be of high value. In such instances F and B are updated towards targets for which the dataset provides no support. This distribution shift a well-observed phenomenon in single-task offline RL [42, 46, 44], and is exacerbated by small, low-diversity datasets as we explore in Figure 2.
3.2 Mitigating the Distribution Shift
In the single-task setting, the distribution shift is addressed by applying constraints either the policy, value function or model (see Section for a summary of past work). Here we re-purpose single-task value function and model regularisation for use in the zero-shot RL setting. To avoid further complicating zero-shot RL methods, we only consider regularisation techniques that do not introduce new parametric functions. We discuss the implications of this decision in Section 5.
Conservative Q-learning (CQL) [42, 44] regularises the Q function by querying OOD state-action pairs and suppressing their value. This is achieved by adding new term the usual Q loss function
LCoL = a E (awpaaa (Q(s, a)] -E(sa)>=(((s (Q(s, a)]-H(() + Lo.
(10)
where a is a scaling parameter, (*(a(s) is a policy distribution selected to find the maximum value of the current Q function iterate, H(s)) is the entropy of yu used for regularisation, and Lo is the normal TD loss on Q. Equation 10 has the dual effect of minimising the peaks in Q under ja whilst maximising Q for state-action pairs in the dataset.
We can replicate a similar form of regularisation the FB framework, substituting F(s, a, 2) 2 for Q in Equation 10 and adding the normal FB loss (Equation 6)
LVC-FB (Eg (assplaaapp(l))))) =, [F(s, 2) 2==2
-H(p)) + CFB
(11)
The key difference between Equations 10 and 11 is that the former suppresses the value of OOD
actions for one task, whereas the latter does so for all task vectors drawn from Z. We call models learnt with this loss conservavveev hnward-backward
representations (VC-FB).
Because FB derives Q functions from successor measures (Equation 5), and because (by assumption) rewards are non-negative, suppressing the predicted measures for OOD actions provides an alternative route to suppressing their Q values. we did with VC-FB, we can substitute FB's successor measure approximation F(s, a, 2) B(s+) into Equation 10, which yields:

---

(b)
(c)
@
Figure Ignoring -iitt--diitriuution actions. agents are tasked with learning separate policies for reaching @ and @. (a) RND dataset with all "left" actions removed: quivers represent the mean action direction in each state bin. (b) Best FB rollout after I million learning steps. (c) Best VC-FB performance after million learning steps. FB overestimates the value of OOD actions and cannot complete either task; VC-FB synthesises the requisite information from the dataset and completes both tasks.
CMC-FB = Ct aa
B(++]
(l aF, B(s_)] -(-)))-((4)) + Lra. (12)
Equation 12 has the effect of suppressing the expected visitation count goal state s+ when taking an OOD action for all task vectors drawn from Z, which says, informally, if we don't 't know where OOD actions take in the MDP, assume they have low probability of taking us any future states for all tasks. This is analogous to works that regularise model predictions in the single-task offline RL setting [37, 96, 69]. As such, we call this variant representation (MC-FB), Since it is not obvious a priori whether the VC-FB or MC-FB form of conservatism would more effective practice, we evaluate both in Section
consureaineeevvvvive forward-backee backward
Implementing these proposals requires two new model components: 1) a conservative penalty scaling factor and 2) way of obtaining policy distribution μ(als) that maximises the current value or measure iterate. For 1), we observe fixed values of a leading to fragile performance, so dynamically tune it each learning-see Appendix 1.1.4. For 2), the choice of maximum entropy regularisation following [44]'s CQL(H) allows to be approximated conveniently with a log-sum exponential across Q values derived from the current policy distribution and uniform distribution. That this is true is not obvious, so we refer the reader to the detail and derivations in Section 3.2, Appendix A, and Appendix E of [44], as well our adjustments [44]'s theory in Appendix B.1. Code snippets demonstrating the required changes to vanilla FB implementation are provided in Appendix G. We emphasise these additions represent only small increase the number of lines required to implement existing methods.
3.3 A Didactic Example
To understand situations in which conservative zero-shot RL methods may be useful, introduce a modified version of Maze from the EXORL. benchmark [95]. Episodes begin with a point-mass initialised in the upper left of the maze (@), and the agent tasked with selecting æ and y tilt directions such that the mass is moved towards one of two goal locations ( and @). The action space is two-dimensional and bounded in [-1, 1]. We take the RND dataset and remove all "left" actions such that E [0, 1] and a E [], creating a dataset that has the necessary information for solving the tasks, but is inexhaustive (Figure 3 (a)). We train FB and VC-FB on this dataset and plot the highest-reward trajectories- Figure 3 (b) and (c). FB overestimates the value of OOD actions and cannot complete either task. Conversely, VC-FB synthesises the requisite information from the dataset and completes both tasks.
4 Experiments
In this section we perform an empirical study to evaluate our proposals. seek answers to four questions: (Q1) Can our proposals from Section 3 improve FB performance on small and/or low- quality exploratory datasets? (Q2) How does the performance of VC-FB and MC-FB vary with respect to task type and dataset diversity? (Q3) Do sacrifice performance on full datasets for
5

---

SF-LAP
GC-IQL
FB
MC-FB
VC-FB
CQL
Probability of Score 2 Task Score
Zaro-shot Porformance (%)
100
80
0.6
60
0.4
40
0.2
20
0.0
200
400
600
800
1000
Score
Figure 4: Aggregate zero-shot performance on ExORT (Leeft) IQM of task scores across datasets and domains, normalised against the performance of CQL., our baseline. (Right) Performance profiles showing the distribution scores across tasks and domains. Both conservative FB variants stochastically dominate vanilla FB-see [1] for performance profile exposition. The black dashed line represents the IQM CQL performance across all datasets, domains, tasks and seeds.
performance small and/or low-quality datasets? (Q4) If our pre-training dataset only covers behaviour related to one downstream task (i.e. the dataset distribution is narrow and not exploratory), can our proposals from Section 3 improve FB performance that task?
4.1 Setup
Domains We respond to Q1-Q3 using the ExORL benchmark [95]. EXORL provides datasets collected by unsupervised exploratory algorithms on the DeepMMnnd Control Suite [80]. We select three of the same domains as [83]: Walker, Quadruped and Maze, but substitute Jaco for Cheetah. This provides two locomotion domains and two goal-reaching domains. Within each domain, we evaluate on all tasks provided by the DeepMind Control Suite for total of 17 tasks across four domains. Full details are provided Appendix A. We respond to Q4 using the D4RI, benehmark [21]. We select the two MuloCo [81] environments from the Open AI gym [8] that closest resemble those from EXORL: Walker2D and HalfCheetah.
Datasets For Q1-Q3 we pre-train on three datasets of varying quality from EXORL. There is no unambiguous metric for quantifying dataset quality,
the reported performance of offline proxy'. We choose datasets collected via Random Network Distillation (RND) [10], Diversity is All You Need (DIAyN) [18], and RANDOM policies, where agents trained on RND are the most performant, on DIAYN are median performers, and on RANDOM are the least performant. As well as selecting for quality, we also select for size by uniformly sub- sampling 100,000 transitions from each dataset. For Q4 we choose the "medium", "medium-replay" and "medium-expert" datasets from D4RL., each providing different fractions of random, medium and expert task-directed trajectories. More details on the datasets are provided in Appendix A.2.
TD3 on Maze for each dataset
4.2 Baselines
We compare our proposals to baselines from three categories: 1) zero-shot RL methods, 2) goal- conditioned RL (GCRL) methods, and 3) single-task offline RL methods. From category 1), the state-of-the-an successor measure based method, FB, and the state-of-the-an successor feature based method, SF with features from Laplacian eigenfunctions (SF-LAP) [83]. From category 2), use goal-conditioned IQL. (GC-IQL) [60], state-of-thh--a GCRL method that, like our proposals, regularises the value function OOD atttenaciion. We condition GC-IQL the goal state Maze and Jaco, and on the state in Diabelled with highest reward on Walker and Quadruped in lieu of
'e note that [75] propose metrics that describe dataset quality function of the behaviour policy's exploration and exploitation w.r.t. one downstream task. However, since we are interested in generalising any downstream task we cannot use these proposals directly, nor can we easily re-purpose them. We acknowledge that our proxy is imperfect, and that more work is required to better understand what dataset quality means in the context of zero-shot RL.

---

SF-LAP
GC-IQL
MC-FB
VC-FB
CQL
Walker
Maze
Quadruped
Jaco
RND 100k
200
400
200
400
200
0
20
40
DIAYN 100k
200
400
200
400
0
200
0
20
40
Random 100k
200
400
200
400
200
0 20
Zero-shot Performance (abs.)
Figure 5: Performance by dataset/domain on EXORL. IQM scores tasks/seeds with 95% conf. intervals.
well-defined goal state. From category 3),
CQL and offline TD3 trained on the same datasets relabelled with task rewards. CQL approximates what an algorithm with similar mechanistics can achieve when optimising for one task in domain rather than all tasks. Offline TD3 exhibits the best aggregate single-task performance on the ExORL benchmark, so should be indicative of the maximum performance we could expect to extract from dataset. Full implementation details for all algorithms are provided in Appendix B. The full evaluation protocol is described in Appendix A.5. Appendix A.6 provides a breakdown of the computational resources used in this work.
4.3 Results
Q1 We report the aggregate performance of our baselines and proposals on EXORL in Fig- ure 4. Both MC-FB and VC-FB outperform the zero-shot RL and GCRL baselines, achiev- ing 150% and 137% FB's IQM performance respectively. The performance gap between FB and SF-LAP is consistent with the results in [83]. MC-FB and VC-FB outperform our single-task baseline in expectation, reaching 1 111% % and 120% of CQL's IQM performance respectively despite not having access to task- specific reward labels and needing to fit policies for all tasks. This is surprising result, and to the best of our knowledge, the first time a multi- task offline agent has been shown to outperform a single-task analogue. CQL outperforms offline TD3 in aggregate, so we drop offline TD3 from the core analysis, but report its full results in Appendix alongside all other methods. We note FB achieves 80% of single-task offline TD3, which roughly aligns with the 85% performance on the full datasets reported by [83].
VC-FB
MC-FB
FB
Aggriggata IQM Return
300
200
100
10a
10a
107
Q2 We decompose the methods' performance with respect to domain and dataset diversity in Figure 5. The largest gap in performance be- tween the conservative FB variants and FB is on RND. VC-FB and MC-FB reach 2.5 and 1.8x of FB performance respectively, and outperform CQL on three of the four domains. On DIAYN, the conservative variants outperform all methods and reach 1.3x CQL's score. On the RANDOM dataset, all methods perform similarly poorly, except for CQL on Jaco, which outperforms all meth- ods. However, in general, these results suggest the RANDOM dataset is not informative enough to extract valuable policies discussed further response Q3. There appears be little correlation between the type of domain (Appendix 1. and the score achieved by any method. GC-IQL performs particularly well on the goal-reaching domains as expected, but worse than all zero-shot methods on the locomotion tasks, irrespective of whether they are conservative or not. This is presumably because the goal-state used to condition the policy (i.e. the state with highest reward in Drabellea) is poor proxy for the true, dense reward function.
Dataset Size
Figure 6: Performance by dataset size. Aggregate IQM scores across all domains and tasks as RND size is varied. The performance delta between vanilla FB and conservative variants increases as dataset size decreases.
7

---

Table Aggregate performance full ExORL datasets. IQM scores aggregated domains and tasks for all datasels, averaged across three seeds. Both VC-FB MC-FB maintain performance FB; the largest relative performance improvement is on RANDOM.
Dataset
Domain
Task
FB
VC-FB
MC-FB
RND
all
all
389
390
396
DIAYN
all
all
269
280
283
RANDOM
all
all
111
131
133
ALL
all all
256
267
271
Q3 We report the aggregated performance of all FB methods across domains when trained on the full datasets in Table (a full breakdown of results in provided in Appendix C). Both conservative FB variants slightly exceed the performance of vanilla FB in expectation. The largest relative performance improvement on the RANDOM dataset-MC-FB performance 20% higher than FB, compared to 5% higher on DIAYN and 2% higher on RND. This corrob- orates the hypothesis that RANDOM- (0K was not informative enough extract valuable policies.
Table and Figure 4 suggest the performance gap between the conservative FB variants and vanilla FB changes as dataset size varied. We further explore this effect in Figure 6 where we scale the RND dataset size from 10 through 107 and plot aggregate IQM performance of FB, VC- FB and MC-FB across all domains. We find that the performance gap decreases dataset size
SF LAP
MC FB
CQL
VC-FB
Zero-shol Performance (%)
125
100
75
increases. This result to be expected: a larger dataset size for fixed exploration algorithm means dc+1 1 ~ Wa (8)41) in the FB TD update (Equation 6) is more likely to be in the dataset, the policy is less likely to become biased toward OOD actions, and conservatism is less needed.
50
25
Q4 We report the aggregate performance of all zero-shot RL methods and CQL on our D4RL. domains in Figure 7. FB fails all domain-datasee tasks, and reaches only 10% of CQL's aggregate performance. MC-FB and VC-FB improve on FB's considerably (by 5.6 and 6.8 x respectively) but under-perform CQL. SF-LAP outperforms FB, but under-performs VC-FB, MC-FB and CQL.
Figure 7: Aggregate zero-shot performance on D4RL. Aggregate IQM scores across all domains and datasets, normalised against the performance of CQL.
Discussion and Limitations
Performance discrepancy between conservative variants Why does VC-FB outperform MC-FB on both EXORL and D4RL? To understand, we inspect the regularising effect of both models more closely. VC-FB regularises OOD actions on F(s. a, 2)T2, with ~ D, and z ~ 2, whilst MC-FB regularises OOD actions on (s8, a, =) B(++), ), with (8, 8+) ~ and z Z. Note the trailing = in VC-FB is replaced with B(++) MC-FB which ties its updates to DD further. We hypothesised that as (D) reduces, ((++) provides poorer task coverage than = ~ 2, hence the comparable performance on full datasets and divergent performance 100k datasets.
To test this, we evaluate third conservative variant called directed (D)VC-FB which replaces all Z in VC-FB with (s++) such that OOD actions are regularised on F(s, a, B(s+)) (B,+) with (s. 44) ~ D. This ties conservative updates entirely to D, and according to our above hypothesis, DVC-FB should perform worse than VC-FB and MC-FB
the 100k ExORL datasets. See Appendix B. for implementation details. We evaluate this variant on all 100k EXORL datasets, domains and tasks and compare with FB, VC-FB and MC-FB in Table 2. See Appendix C for full breakdown.
We find the aggregate relative performance of each method as expected i.e. DVC-FB < MC-FB VC-FB. As consequence conclude that VC-FB should be preferred for small datascts with

---

Table 2: Aggregated performance of conservative variants employing differing sampling procedures on ExORL. DVC-FB derives all 2s from the hackward model; VC-FB derives all from 2; and MC-FB combines both. Performance correlates with the degree to which : 2.
Dataset
Domain
Task
FB
DVC-FB
MC-FB
VC-FB
ALL (100k)
all
all
99
108
136
148
no prior knowledge of the dataset or test tasks. Of course. for a specific domain-dataset pair, B(s+) with 84 D may happen to cover the tasks well, and MC-FB may outperform VC-FB. We suspect this was the case for all datasets on the Jaco domain for example. Establishing whether this will be true a priori requires either relaxing the restrictions imposed by the zero-shot RL setting, better understanding of the distribution of tasks in 2-space and their relationship to pre-training datasets. The latter is important future work.
Avoiding new parametric functions State-of-the-ane zero-shot RL methods are complex, and we wanted to avoid further complicating them with new parametric functions. This limited our solution- space to CQI--ttle regularisation techniques, but had we relaxed this constraint, other options become available. Methods like AWAC [58]. IQL [40], and X X-QL [25] all require estimate of the state-value function which is not immediately accessible in the FB or USF frameworks. In theory, we could learn antion-independent USF of the form V(s, z) E > (s((s++)]ss) T&J V so
S, 2 E Rd concurrently to F and (or vi for USFs). learnt with expectile regression, this function could be used to implement IQL and X-QL style regularisation; without expectile regression it could be used to compute the advantage weighting required for AWAC. It's possible that implementing these methods could improve downstream performance and reduce computational overhead at the cost of increased training complexity. We leave this worthwhile investigation for future work. We provide detail of negative results related to downstream finetuning of FB models in Appendix E to help inform future research.
D4RL Performance Unlike the EXORL results, VC-FB and MC-FB do not outperform CQL on the D4RL. benchmark. We believe these narrower data distributions require a more careful selection of the conservative penalty scaling factor a. We explore this further in Appendix F, and note this corroborated by findings in the original CQL paper [44]. Methods described above, like IQL, have been shown to be more robust than CQL partly because they bypass a tuning. We expect that exploring the integration of these methods may improve D4RL performance.
6 Related Work
Zero-shot RL Zero-shot RL methods build upon successor representations [15], universal value
function approximators [74], successor features [5] and successor measures [6]. The state-of-the-art methods instantiate these ideas either universal successor features (USFs) [7] or forward-basd backaadd
(FB) representations [82, 83], with recent work showing the latter can be used to perform a range of imitation learning techniques efficiently [65]. A representation learning method is required to learn the features for USFs, with past works using inverse curiosity modules [62], diversity methods
[49, 291, Laplacian eigenfunctions [87], or contrastive learning [13]. No works have yet explored the issues arising when training these methods on low quality offline datasets, and only one has investigated applying these ideas to real- real-world problems [34].
Goal-conditioned RL methods train policies to reach any goal state from any other state, and so can be used to perform zero-shot RL. in goal-reaching environments [60, 54, 93, 19, 85). However, they have no principled mechanism for conditioning policies on "dense" reward functions (as such tasks are not solved by simply reaching particular state), and are not full zero-shot RL methods. concurrent line of work trains policies using sequence models conditioned on reward-labelled histories [12, 33, 45, 68, 99, 11, 24, 76, 91, 90], but, unlike zero-shot RL methods, these works do not have a robust mechanism for generalising to different reward functions as test-time.
Offline RL Offline RL algorithms require regularisation of policies, value functions, models, or combination to manage the offline-to-online distribution shift [46]. Past works regularise policies with explicit constraints [88, 20, 23, 22, 27, 64, 43, 86, 94], via important sampling |66, 79, 50, 57, 26], by leveraging uncertainty in predictions [89, 98, 4, 36), or by minimising OOD action queries [84, 14, 40], a form of imitation learning [72, 73]. Other works constrain value function approximation

---

so OOD action values are not overestimated [44, 42, 52, 53, 51, 92]. Offline model-based RL methods use the model to identify OOD states and penalise predicted rollouts passing through them (97, 37, 96, 2, 55, 67, 69). All of these works have focused on regularising finite number of policies; contrast we extend this line of work to the zero-shot RI. setting which concerned with learning an infinite family of policies.
7 Conclusion
In this paper, explored training agents to perform zero-shot reinforcement learning (RL) from low quality data. We established that the existing methods suffer in this regime because they overestimate the value of diorristriittiinn state-action values, a well-observed pheneomena in single-task offline RL. As a resolution, we proposed a family of conservative zero-shot RL algorithms that regularise value functions or dynamies predictions out-of-distribution state-action pairs. In experiments across various domains, tasks and datasets, we showed our proposals outperform their non-conservative counterparts in aggregate and sometimes surpass our task-specific bascline despite lacking access to reward labels priori. In addition to improving performance when trained on sub-optimal datasets, we showed that performance on large, diverse datasets does not suffer a consequence of our design decisions. Our proposals represent step towards the use of zero-shot RL methods in the real world.
Acknowledgements
We thank Sergey Levine for helpful feedback on the core and finetuning experiments, and Alessandro Abate and Yann Ollivier for reviewing carlier versions of this manuscript. We also thank the anonymous reviewers whose suggestions significantly improved this work. Computational resources were provided by the Cambridge Centre for Data-Driven Discovery (C2D3) and Bristol Advanced Computing Research Centre (ACRC). This work was supported by an EPSRC DTP Studentship (EP/T5 and Emerson Electric.
References
[1] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Belle- mare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304-29320, 2021.
[2] Arthur Argenson and Gabriel Dulac-Aroold. Model-based offline planning. arXiv preprint arXiv:200.005556 2020.
[3] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton, Layer normalization. arXiv preprint 1607.0600..6455 2016.
[4] Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhihong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. arXiv preprint 2rXiv:2202.11566 2022.
[5] André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. Advances in neural information processing systems, 30, 2017.
[6] Léonard Blier, Corentin Tallec, and Yann Ollivier. Learning successor states and goal-dependent values: A mathematical viewpoint. arXiv preprini arXiv:2/01..71133 2021.
[7] Diana Borsa, André Barreto, John Quan, Daniel Mankowitz, Rémi Munos, Hado Van Hasselt, David Silver, and Tom Schaul. Universal successor features approximators. arXiv preprint VrrXiv1882..0760 2018.
[8] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint 11X.iv1606.01500000 2016.
10

---

[9] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[10] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810.1289. 2018
[11] Yevgen Chebotar, Quan Vuong, Alex Irpan, Karol Hausman, Fei Xia, Yao Lu, Aviral Kumar, Tianhe Yu, Alexander Herzog, Karl Pertsch, al. Q-transformer: Scalable offline reinforcement learning via autoregressive q-functions. arXiv preprini arXiv:2309.100 10150, 2023.
[12] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling, Advances in neural information processing systems, 34: 2021.
[13] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1597-1607. PMLR, 2020.
[14] Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith Ross. Bail: Best- action imitation learning for batch deep reinforcement learning. Advances in Neural Information Processing Systems, 33: 13:18333-1836, 2020.
[15] Peter Dayan. Improving generalization for temporal difference learning: The successor repre- sentation. Neural computation, 5(4)6613- 1993.
[16] Gabriel Dulac-Arnold, Daniel Mankowitz, and Todd Hester. Challenges of real-world reinforce- ment learning. arXiv preprini /904. /2901, 2019.
[17] Bradley Efron. Bootstrap methods: another look at the jackknife. In Breakthroughs in statistics: Methodology and distribution, pages 569-593. Springer, 1992.
[18] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070, 2018.
[19] Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Russ R Salakhutdinov. Contrastive learning as goal-conditioned reinforcement learning. Advances in Neural Information Process- ing Systems, 35:35603-333 356200, 2022.
[20] Rasool Fakoor, Jonas W Mueller, Kavosh Asadi, Pratik Chaudhari, and Alexander J Smola. Continuous doubly constrained batch reinforcement learning. Advances in Neural Information Processing Systems, 34: 1 1221-12001222 2021.
[21] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
[22] Scott Fujimoto, Wei-Di Chang, Edward Smith, Shixiang Shane Gu, Doina Precup, and David Meger. For sale: State-action representation learning for deep reinforeement learning. Advances in Neural Information Processing Systems, 36, 2024.
[23] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference machine learning, pages 2052-2062. PMLR, 2019.
[24] Hiroki Furuta, Yutaka Matsuo, and Shixiang Shane Gu. Generalized decision transformer for offline hindsight information matching. arXiv preprint arXiv:2111.10366 2021.
[25] Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent rl without entropy. International Conference on Learning Representations, 2023.
[26] Carles Gelada and Marc G Bellemare. Off-policy deep reinforcement learning by bootstrapping the covariate shift. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3647-3655, 2019.
11

---

[27] Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected-max g-learning operator for simple yet effective offiine and online rl. International Conference Machine Learning, pages 3682-3691. PMLR, 2021.
[28] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870, 2018.
[29] Steven Hansen, Will Dabney, Andre Barreto, Tom Van de Wiele, David Warde-Farley, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. arXiv preprint arXiv:1906.0500 05030, 2019.
[30] Charles R Harris, K Jarrod Millman, Stéfan J Van Der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J Smith, et al. Array programming with numpy. Nature, 585(7825):357-362, 2020.
[31] John D Hunter. Matplotlib: A 2d graphies environment. Computing in science & engineering, 903))90-955 2007.
[32] Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary 2016.
tasks. arXiv preprini
[33] Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34:1273- 1286, 2021.
[34] Scott Jeen, Alessandro Abate, and Jonathan M Cullen. Low emission building control with zero- shot reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 14259-14267, 2023.
[35] Chi Jin, Akshay Krishnamurthy, Max Simchowitz, and Tiancheng Yu. Reward-free exploration for reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, pages 4870-4879, 13-18 Jul 2020.
[36] Ying Jin, Zhuoran Yang, and Zhaoran Wang. pessimism provably efficient for offline rI? In International Conference on Machine Learning, pages 5084-5096. PMLR, 2021.
[37] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. Advances in neural information processing systems, 33:21810 2122218833 2020.
[38] Diederík P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 2014.
[39] Yehuda Koren. On spectral graph drawing. In International Computing and Combinatorics Conference, pages 496-508. Springer, 2003.
[40] Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pages 5774-5783. PMLR, 2021.
[41] Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine. Offline q learning on diverse multi- taski-task data both scales and generalizes. arXiv preprint 4rXiv:2211.15144, 2022.
[42] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[43] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- policy learrnnn via bootstrapping emror reduction. Advances Neural Information Processing Systems, 32, 2019.
12

---

[44] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006. 040079996.0479,, 2020.
[45] Kuang-Huei Lee, Ofir Nachum, Mengjiao Yang. Lisa Lee, Daniel Freeman, Winnie Xu, Ser- gio Guadarrama, Ian Fischer, Eric Jang. Henryk Michalewski, et al. Multi-game decision transformers. Advances in neural information processing systems, 35, 2022.
[46] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
[47] Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR (Poster), 2016.
[48] Hao Liu and Pieter Abbeel. Aps: Active pretraining with successor features. In International Conference Machine Learning, pages 6736-6747. PMLR, 2021.
[49] Hao Liu and Pieter Abbeel. Aps: Active pretraining with successor features. In International Conference on Machine Learning, pages 6747. PMLR, 2021.
[50] Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Off-policy policy gradient with state distribution correction. arXiv preprin! arXiv: 19X4::9904.000 08473, 2019.
[51] Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:1711-1722 2022.
[52] Xiaoteng Ma. Yiqin Yang. Hao Hu, Qihan Liu, Jun Yang, Chongjic Zhang, Qianchuan Zhao, and Bin Liang. Offline reinforcement learning with value-based episodic memory. arXiv preprint 10.0992 2021.
[53] Yecheng Ma, Dinesh Jayaraman, and Osbert Bastani. Conservative offline distributional reinforcement learning. Advances in Neural Information Processing Systems, 34:19235-19247 2021.
[54] Yecheng Jason Ma. Jason Yan, Dinesh Jayaraman, and Osbert Bastani. How far i'll go: Offiine goal-conditioned reinforcement learning via f-advantage regression. arXiv preprint arXiv:2206.0300 03023, 2022.
[55] Tatsuya Matsushima. Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment-efffciiet reinforcement learning via model-based offline optimization. arXiv 2020.
preprint
[56] Wes McKinney et al. pandas: foundational python library for data analysis and statistics. Python for high performance and scientific computing, 14(9):1-9, 2011.
[57] Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior- agnostic estimation of discounted stationary distribution corrections. Advances in neural information processing systems, 32, 2019.
[58] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint 20Xv:22006.0935,, 2020.
[59] Mitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning. arXiv preprint wrXiv22303.05473 2023.
[60] Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. Hiql: Offline goal- conditioned rl with latent states as actions. Advances in Neural Information Processing Systems, 37, 2023.
[61] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vito,o Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
13

---

[62] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity -divven exploration by self-supervised prediction. In International conference on machine learning, pages 2778-
2787. PMLR, 2017.
[63] Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagree- ment. In International conference on machine learning. pages 5062-5071, PMLR, 2019.
[64] Zhiyong Peng, Changlin Han, Yadong Liu, and Zongtan Zhou. Weighted policy constraints for offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 9435-9443, 2023.
[65] Matteo Pirotta, Andrea Tirinzoni, Ahmed Touati, Alessandro Lazaric, and Yann Ollivier. Fast imitation via behavior foundation models. In International Conference on Learning Representa- tions, 2024.
[66] Doina Precup, Richard S Sutton, and Sanjoy Dasgupta. Off-policy with function approximation. In ICML, pages 417-424, 2001.
learning
[67] Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offline reinforcement learning from images with latent space models. In Learning for Dynamics and Control, pages 1168 PMIR, 2021.
[68] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maros Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06117. 2022.
[69] Marc Rigter, Bruno Lacerda, and Nick Hawes. Rambo-rl: Robust adversarial model-based offline reinforcement learning. S. Koyejo, Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 16082-16097. Curran Associates, Inc., 2022.
[70] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and patiern recognition, pages 10684-10695, 2022.
[71] Arthur L Samuel. Some studies in machine learning using the game of checkers. IBM Journal of research and development, 3(3):210-23) 1959.
[72] Stefan Schaal. Learning from demonstration. Advances neural information processing systems, 9, 1996.
[73] Stefan Schaal. Is imitation learning the route to humanoid robots? Trends in cognitive sciences, 3(6):233-242, 1999.
[74] Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function ap- proximators. In International conference machine learning, pages 1312-1320. PMLR, 2015.
[75] Kajetan Schweighofer, Andreas Radler, Marius-Constanin Constantin Dinu, Markus Hofmarcher, Vihang Patil, Angela -Nmmling, Hamid Eghbal-zadee and Sepp Hochreiter. dataset perspective on offline reinforcement learning. arXiv preprint arXiv:211 (tl::2/11.047/// 144414, 2021.
[76] Max Siebenborn, Boris Belousov, Junning Huang, and Jan Peters. How crucial is transformer in decision transformer? arXiv preprint arXiv:2211.14655 2022.
[77] Richard Sutton and Andrew Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018.
[78] Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning. 3:9-44, 1988.
[79] Richard S Sutton, A Rupam Mahmood, and Martha White. An emphatic approach to the problem of off- off-policy lefforal-difference learning. The Journal of Machine Learning Research, -76)) 2016.
14

---

[80] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprini arXiv: 11001800000000 2018.
[81] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012.
[82] Ahmed Touati and Yann Ollivier. Learning one representation to optimize all rewards. Advances in Neural Information Processing Systems, 34:13-23, 2021.
[83] Ahmed Touati, Jérémy Rapin, and Yann Ollivier. Does zero-shot reinforcement learning exist? In The Eleventh International Conference Learning Representations, 2023.
[84] Qing Wang, Jiechao Xiong, Lei Han, Han Liu, Tong Zhang, et al. Exponentially weighted imitation learning for batched historical data. Advances in Neural Information Processing Systems, 31, 2018.
[85] Tongzhou Wang, Antonio Torralba, Phillip Isola, and Amy Zhang. Optimal goal-reaching reinforcement learning via quasimetric learning. In International Conference on Machine Learning, pages 36411 1-343-0 PMLR, 2023.
[86] Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy optimization for offline reinforcement learning, Advances in Neural Information Processing Systems, 35:31278-312911 2022.
[87] Yifan Wu, George Tucker, and Ofir Nachum. The laplacian in rl: Learning representations with efficient approximations. arXiv preprint
2018
[88] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint 19X.v:/911.113600 2019.
[89] Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2105.081440 2021.
[90] Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenenbaum, and Chuang Gan. Prompting decision transformer for few-shot policy generalization. Proceedings of the 39th International Conference on Machine Learning, pages 24631 14631-24645, 17-23 Jul 2022.
[91] Taku Yamagata, Ahmed Khalil, and Raul Santos-Rodrigus Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling offline RL. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pages 38989-39007, 23-29 Jul 2023.
[92] Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. Rorl: Ro- bust offline reinforcement learning via conservative smoothing. Advances in Neural Information Processing Systems, 35:23851-22 -23666, 2022.
[93] Rui Yang, Lin Yong, Xiaoteng Ma, Hao Hu, Chongjie Zhang, and Tong Zhang. What is essential for unseen goal generalization of offline coal-conditioned rl? In International Conference Machine Learning, pages 39543-39571. PMLR, 2023.
[94] Shentao Yang, Zhendong Wang, Huangjic Zheng, Yihao Feng, and Mingyuan Zhou. A behavior regularized implicit policy for offline reinforcement learning. arXiv preprint arXiv:2202.09673 2022.
[95] Denis Yarats, David Brandfonbrener. Hao Liu. Michael Laskin, Pieter Abbeel, Alessandro Lazaric, and Lerrel Pinto. Don't 't change the algorithm, change the data: Exploratory data for offline reinforcement learning. arXiv preprint arXiv:2201. 2022.
[96] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances neural information processing systems, 34:28954-28966 2021.
15

---

[97] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129-14142, 2020.
[98] Andrea Zanette, Martin J Wainwright, and Emma Brunskill. Provable benefits of actor-critic methods for offline reinforcement learning. Advances in neural information processing systems, 34:13626-1364)) 13640, 2021.
[99] Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. In international conference on machine learning, pages 27042-27059. PMLR, 2022.
16

---

Appendices
Experimental Details
18
A.1 ExORL Domains
18
A.2 Datasets
18
A.3
D4RL Domains
19
A.4
D4RL Datasets
19
A.5
Evaluation Protocol
19
A.6
Computational Resources
20
Implementation Details
20
B.1
Forward-Backward Representations
20
B.2
Universal Successor Features
23
B.3
GC-IQL
24
B.4 CQL
24
B.5 TD3
25
B.6 Code References
25
Extended Results
25
Value Conservative Universal Successor Features
28
Negative Results
29
E.1 Downstream Finetuning..
F Learning Curves & Hyperparameter Sensitivity
31
Code Snippets
37
G.I Update Step
37
G.2 Value-Conservative Penalty
38
G.3 Measure-Conservattve Penalty
39
G.4 a Tuning
42
H NeurIPS Paper Checklist
43
17

---

A Experimental Details
A.1 EXORL Domains
We consider two locomotion and two goal-directed domains from the ExORL benchmark [95] which built atop the DeepMind Control Suite [80]. Environments are visualised here: https:
//www www.tyyuuubbee... cwm/c//////////
The domains are summarised in Table 3.
Walker. A two-legged robot required to perform locomotion starting from bent-kneed position. The state and action spaces are 24 and 6-dimensional respectively, consisting of joint torques, velocities and positions. ExORL provides four tasks stand, walk, run and flip. The reward function for stand motivates straightened legs and upright torse; walk and run supersets of stand including reward for small and large degrees of forward velocity; and flip motivates angular velocity of the torso after standing. Rewards are dense.
Quadruped. four-legged robot required to perform locomotion inside a 3D maze. The state and action spaces are 78 and 12-dimensional respectively, consisting of joint torques, velocities and positions. ExORL provides five tasks stand, roll, roll fast, jump and escape. The reward function for stand motivates a minimum torse height and straightened legs; roll and roll fast require the robot to flip from position on back with varying speed; jump adds term motivating vertical displacement to stand; and escape requires the agent to escape from a 3D maze. Rewards are dense.
Maze. A 2D maze with four rooms where the task is to move point-mass to one of the rooms. The state and action spaces are 4 and 2-dimensional respectively; the state space consists of x, y positions and velocities of the mass, the action space is the x, tilt angle. EXORL provides four reaching tasks top left, top right, bottom 1eft and bottom right. The mass always initialised in the top left and the reward is proportional to the distance from the goal, though is sparse i.e. it only registers once the agent reasonably close to the goal.
Jaco. A 3D robotic arm tasked with reaching an object. The state and action spaces are 55 and 6-dimensional respectively and consist of joint torques, velocities and positions. ExORL provides four reaching tasks top left, top right, bottom left and bottom right. The reward is proportional to the distance from the goal object, though is sparse i.c. only registers once the agent is reasonably close to the goal object.
A.2 EXORL Datasets
We train on 100,000 transitions uniformly sampled from three datasets on the EXORL benchmark collected by different unsupervised agents: RANDOM, DIAYN, and RND. The state coverage Maze is depicted in Figure 8. Though harder to visualise, we found that state marginals on higher- dimensional tasks (e.g. Walker) showed similar diversity in state coverage.
RND. An agent whose exploration directed by the predicted error its ensemble of dynamics models. Informally, we say RND datasets exhibit high state diversity.
DIAYN. An agent that attempts sequentially learn of skills, Informally, we say DIAYN datasets exhibit medium state diversity.
Table 3: ExORL domain summary. Dimensionality refers to the relative size of state and action spaces. Type is the task categorisation, either locomotion (satisfy prescribed behaviour until the episode ends) or goal-reaching (achieve specific task terminate the episode). Reward the frequency with which non-zero rewards provided, where dense refers to non-zero rewards every timestep and sparse refers to non-zero rewards only at positions close to the goal. Green and red colours reflect the relative difficulty of these settings.
Domain
Dimensionality
Type
Reward
Walker
Low
Locomotion
Dense
Quadruped
High
Locomotion
Dense
Maze
Low
Goal-reaching
Sparse
Jaco
High
Goal-reaching
Sparse
18

---

RANDOM. A agent that selects actions uniformly at random from the action space. Informally, we say RANDOM datasets exhibit low state diversity.
Random-100k
DIAYN-100K
RND-100k
Figure 8: Maze state coverage by dataset. (left) RANDOM; (middle) DIAYN; (right) RND.
A.3 D4RL Domains
We consider two MuJoCo [81] locomotion tasks from the D4RL benchmark [21], which built atop the v2 Open AI Gym [8]. The below environment descriptions are taken from [8].
Walker2D-v2. A two-dimensional two-legged figure that consist of seven main body parts single torso at the top (with the two legs splitting after the torso), two thighs in the middle below the torso, two legs in the bottom below the thighs, and two feet attached to the legs which the entire body rests. The goal is to walk in the in the forward (right) direction by applying torques on the six hinges connecting the seven body parts.
HalfCheetah-v2. A 2-dimensional robot consisting of 9 body parts and joints connecting them (including two paws). The goal is to apply a torque on the joints to make the cheetah run forward (right) as fast as possible, with positive reward allocated based on the distance moved forward and negative reward allocated for moving backward.
A.4 D4RL Datasets
We consider three goal-directed datasets from D4RL,, each providing different proportion of expert trajectories. The below dataset descriptions are taken from [21].
Medium. Generated by training SAC policy, early-stopping the training, and collecting 1M samples from this partially-trained policy.
Medium- repiua-reppla. Generated by recording all samples in the replay buffer observed during training until the policy reaches the "medium" level of performance.
Medium-expert. Generated by mixing equal amounts expert demonstrations and suboptimal data, either from partially trained policy by unrolling uniform-at at-random policy.
A.5 Evaluation Protocol
We evaluate the cumulative reward (hereafter called score) achieved by VC-FB, MC-FB and our baselines on each task across five seeds. We report task scores as per the best practice recommenda- tions of [1]. Concretely, we run each algorithm for million learning steps, evaluating task scores at checkpoints of 20,000 steps. At each checkpoint, we perform 10 rollouts, record the score of each, and find the interquartile mean (IQM). We average across seeds each checkpoint to create the learning curves reported in Appendix F. From each learning curve, we extract task scores from the learning step for which the all-task IQM maximised across seeds. Results are reported with 95% confidence intervals obtained via stratified bootstrapping [17]. Aggregation across tasks, domains and datasets always performed by evaluating the IQM. Full implementation details are provided in Appendix B.l.
19

---

A.6 Computational Resources
We train our models on NVIDIA A10O GPUs. Training a single-task offline RL method to solve one task on one GPU takes approximately 4 hours. FB and SF solve one domain (for all tasks) on one GPU in approximately hours. Conservative FB variants solve one domain (for all tasks) on one GPU in approximately 12 hours. As result, our core experiments the 100k datasets used approximately 110 GPU days of compute.
B Implementation Details
Here we detail implementations for all methods discussed in this paper. The code required to reproduce our experiments is available via https://github. com/enjeemeee comm/enjennerrrerrooattee
B.1 Berward-Backward Representations
B.1.1
Architecture
The
architecture described below follows the implementation by [83] exactly, other than the batch size which reduce from 1024 512. We did this reduce the computational expense of each run without limiting performance. The hyperparameter study in Appendix J of [83] shows this choice is unlikely to affect FB performance. All other hyperparameters reported in Table 4.
Forward Representation F(s, a, z). The input to the forward representation F is always prepro- cessed. State-action pairs (s, a) and state-task pairs (s, 2) have their own preprocessors which are feedforward MLPs that embed their inputs into 512-dimensional space. These embeddings con- catenated and passed through a third feedforward MLP F which outputs a d-dimensional embedding vector. Note: the forward representation F' is identical to used by USF their implementations are identical (see Table 4).
Backward Representation B(s). The backward representation B is a feedforward MLP that takes state as input and outputs d' didimensional embedding vector.
Actor T(8,2) Like the forward representation, the inputs to the policy network are similarly preprocessed. State-action pairs (s, a) and state-task pairs (s, z) have their own preprocessors which feedforward MLPs that embed their inputs into 5 512-dimensional space. These embeddings are concatenated and passed through third feedforward MLP which outputs a-dimensional vector, where a is the action-space dimensionality. A Tanh activation is used on the last layer to normalise their scale. As per [23]'s recommendations, the policy smoothed by adding Gaussian noise the actions during training. Note the actors used by FB and USFs are identical (see Table 4).
Misc. Layer normalisation [3] and Tanh activations used in the first layer of all MLPs to standardise the inputs.
B.1.2 Task Sampling Distribution
FB representations require method for sampling the task vector z at each learning step. [83] employ mix of two methods, which we replicate:
1. Uniform sampling of z on the hypersphere surface of radius v around the origin of Rdd
2. Biased sampling of passing states D through the backward representation Bss). This also yields vectors on the hypersphere surface due to the I.2 normalisation described above, but the distribution is non-uniform.
We sample 2 50:50 from these methods at each learning step.
B.1.3 Maximum Value Approximator μ
The conservative variants of require access to policy distribution p(lss) that maximises the value of the current Q iterate in expectation. Recall the standard CQL
20

---

Table Hyperparameters for zero-shot RL methods. The additional hyperparameters for Conservative FB representations highlighted in blue
Hyperparameter
Value
Latent dimension d
50 (100 for maze)
dimensions
(1024, 1024) (256, 256, 256) (1024, 1024)
B dimensions
Preprocessor dimensions
Std. deviation for policy smoothing
0,2
Truncation level for policy smoothing
0.3
Learning steps
1,000,000
Batch size
512
Optimiser
Adam [38]
Learning rate
0.0001
Discount y
0.98 (0.99 for maze)
Activations (unless otherwise stated)
ReLU
Target network Polyak smoothing coefficient
0,01
z- z-inference labels
10,000
2 mixing ratio
0.5
Conservative budget r
50 (45 for D4RL)
OOD action samples per policy N
3
LCoL = a (Bo-Daorvtat))1(ss a)] - (s), - R(u)) + Lo,
(13)
where is scaling parameter, s(als) the policy distribution we seek, R regularises and Lo represents the normal TD loss on Q [44]'s most performant CQL variant (CQL(H)) utilises maximum entropy regularisation on μ i.e. R (()) They show that obtaining can be cast as closed-form optimisation problem of the form:
max E ( (x))) [[(x)] H(u) s.t. Em(x) 1, ((x) 2 0 Va,
(14)
and has optimal solution (=)x)= & exp(f(x)), where Z is normalising factor. Plugging Equation 14 into Equation 13 we obtain:
LCor = a Blowllog Σ exp(Q(s. - B(ga)) + Lo-
(15)
In discrete action spaces the logsumexp can be computed exactly; in continuous action spaces [44] approximate it via importance sampling using actions sampled uniformly random, actions from the current policy conditioned on 8, ~ D, and from the current policy conditioned on 8:41 ~ D):
*Conditioning on next states $141 D is not mentioned in the paper, but is present in their official implementation.
21

---

log Σ ((tt())) = log( Σ exp Q(s(. + Σ exp Q(tr +
exp(exp Q++; az)),
3
eep((Q(s,,ai) at) Unif(A)
[exp(Q(s,, (s),
log Bai Lin(t))
+ BaB sx)
T 8t)
(sp(Q((e,)))))
T F(a 18411)
exp(Q(55, anl) Unif(.A)
2N
exp(Q(ss, at))
= log(
Σ
Σ aivat(eilac)
3N
6N
x(a)(8))
ai>-Unif(
exp(Q(s),ai))
Σ
)
3N
<a Sc+1)
(16)
with N hyperparameter defining the number of actions to sample across the action-space, We can substitute u(s 2) for Q(+, a) in the final expression of Equation 17 obtain the equivalent for VC-FB:
exx((F(st, ai, 2), 2)22)
2N Σ
Tcxp(F(st, a2, TT2)
log Σ exp 21, 2== log(
Σ
3N
Unif(A)
6N
T(a
o, Unit(A)(
(arle))))
N
exx((F(sx, ais
Σ
3N
x(a:(81+1)
(17)
In Appendix F, Figure 16 we show how the performance of VC-FB varies with the number of action samples. In general, performance improves with the number of action samples, but limit N to limit computational burden. The formulation for MC-FB is identical other than each value F(s, a, z)"² being replaced with measures F(s, a, B(s4).
B.1.4 Dynamically Tuning
critical hyperparameter' is a which weights the conservative penalty with respect to other losses during each update. We initially trialled constant values of o, but found performance to be fragile to this selection, and lacking robustness across environments. Instead, instantiate their algorithm for dynamically tuning a, which they call Lagrangian dual gradient-descemt on a. We introduce a conservative budget parameterised by r, and a with respect this budget:
follow [44] once again, and
min max (EenD..entalbee T2] (e..))) (F T2] 2] + CFB. (18) a20
Intuitively, this implies that the scale of overestimation I then a set close to 0, and the conservative penalty does not affect the updates. If the scale of overestimation 2 J then a is set proportionally to this gap, and thus the conservative penalty is proportional to the degree of overestimation above T. As above, for the MC-FB variant values a)) T= replaced with measures F(s,a, 2) B(8+).
B.1.5 Algorithm
We summarise the end-to-end implementation of VC-FB as pseudo-code in Algorithm 1. MC-FB representations are trained identically other than line where the conservative penalty is computed for M instead of Q, and in line 12 where Ms are lower bounded via Equation 12.
22

---

Algorithm Pre-training value-conservative forward-backward representations
Require: D: dataset of trajectories
Per, Bøg randomly initialised networks
N, 3, V/, b: learning steps, z-sampling distribution, polyak momentum, batch size
for learning step n 1...N do
2:
{(8), 04 8(+1) ~ Dielo
Sample mini-batch of transitions
3:
{ziheet) Z
4 Sample zs (Appendix B.1.2)
4:
5:
FB Update
6:
{a++1} T($++1, 24)
Sample batch of actions at next states from policy
7:
Update FB given {(s;, Cz 8141, 24)
< Equation 6
8:
9:
// Conservative Update
Quax(s; (8i, a;) Rs log E exp Fs;, ai, 2,) < Compute conservative penalty (Equation 17)
11:
Compute given Quix via Lagrangian dual gradient-descent
< Equation 18
Lower bound Q
Equation 11
13:
14:
Actor Update
15:
{ai} ~ (81; =i)
Sample actions from policy < Standard actor-critic formulation
16:
Update actor to maximise (s; aj 2;)TTa
17:
1/ Update target networks via polyak averaging
19:
OF +- v0 =
Forward target network Backward target network
20:
0 +- v0 (1 0)0
21:
end for
B.1.6 Directed Value-Conservative Forward Backward Representations
VC-FB applies conservative updates using task vectors sampled from Z (which in practice is uniform distribution over the /ddhyperspherre. This will include many vectors corresponding to tasks that are never evaluated in practice downstream applications. Intuitively, it may seem reasonable to direct conservative updates to focus on tasks that are likely to be encountered downstream. One simple way of doing this would be consider the of all goal-reaching tasks for goal states in the training distribution, which corresponds to sampling B(sg) for some 8g 'D. This leads to the following conservative loss function:
LDvc.FB =a. -FeeeDoorullllleaa DD (F B(sg))
- -EIs.a)D. F(s, B(s,)) B(s))) - H(()) + LFB. (19)
We call models learnt via this loss directed (DVC-FB). While were initially open the possibility that DVC-FB updates would be better targeted than those of VC-FB, and would lead to improved downstream task performance, this as discussed in Section 5. We report scores obtained by the DVC-FB method across all 100k datasets, domains and tasks in Appendix C.
be the in experimental settings
B.2
Universal Successor Features
We directly reimplement USFs, with basic features (p() provided by Laplacian eigenfunctions [87]. from [83].
B.2.1 Architecture
USF (5 z). The input to the USF v is always preprocessed. State-action pairs (s, a) and state-task pairs (s, 2) have their own preprocessors which are feedforward MIPs that embed their inputs into 5 312-dimensiona space. These embeddings concatenated and passed through third
23

---

feedforward MLP v which outputs d-dimensional embedding vector. Note this is identical to the implementation of F described in Appendix B.1. All other hyperparameters are reported in Table
4.
Feature Embedding (ss). The feature map (((s is feedforward MLPP that takes state input and outputs d-dimensional embedding vector. The loss function for learning the feature embedding provided in Appendix B.2.2.
Actor m(s, Like the forward representation, the inputs to the policy network are similarly preprocessed. Scate-action pairs (s, a) and state-task pairs (s, 2) have their own preprocessors which are feedforward MLPs that embed their imputs into a 512-dimensional space. These embeddings are concatenated and passed through third feedforward MLP which outputs a-dimensional vector, where a is the action-space dimensionality. their scale. As per [23]'s recommendations, the policy is smoothed by adding Gaussian noise the actions during training. Note this is identical to the implementation of <((8, 2) as described in Appendix B.1.
Tanh activation is used on the last layer to normalise
Misc. Layer normalisation [3] and Tanh activations are used in the first layer all MLPs to standardise the inputs. sampling distribution Z is identical to FB's (Appendix B.1.2).
B.2.2 Laplacian Eigenfunctions Loss
Laplacian cigenfunction features (oss) are learned as [87]. They consider the symmetrized MDP graph Laplacian created by some policy æ, defined as I, = ld (Padiag + d (P?+)=) They learn the eigenfunctions of I. with the following:
min (s [ll((s:) sp( ((s++1)11 X [(p(()) T((+))² 1|((s) l|p((+)
(oo(s))1B >b
(20)
which comes from [39].
B.3 GC-IQL
B.3.1 Architecture
We implement GC-IQL following [60]'s codebase. GC-IQL inherits all functionality from a base soft actor-critic agent [28], adds a soft conservative penalty to goal-conditioned critic's V g) updates. We refer the reader to paper that introduces GC-IQL [60] for details on the loss function used train ((s, g). Hyperparameters are reported in Table
Critic(s). GC-IQL trains double goal-conditioned value functions V(s, g). The critics are feedforward MLPs that take a state-goal pair (s,g) g) as input and output value E R., During training the goals are sampled from the prior G described in Section B.3.2.
Actor The actor is standard feedforward MLP taking the state input and outputting an 2a- dimensional vector, where a is the action-space dimensionality. The actor predicts the mean and standard deviation of Gaussian distribution for each action dimension; during training value is sampled at random, during evaluation the mean is used.
B.3.2 Goal Sampling Distribution
Following [60], goals are sampled from either random states, future states, or the current state with probabilities 0.3, 0.5 and 0.2 respectively. A geometric distribution Geom(1 7) is used for the future state distribution, and the uniform distribution over the offline dataset is used for sampling random states.
B.4 CQL
B.4.1
Architecture
We adopt the same implementation and hyperparameters as is used on the ExORL benchmark. CQL inherits all functionality from base soft actor-critic agent [28], but adds a conservative penalty to the critic updates (Equation 10). Hyperparameters reported in Table 5.
24

---

Critic(s), CQL employs double Q networks, where the target network is updated with Polyak averaging via a momentum coefficient. The critics are feedforward MLPs that take state-action pair (8, a) as input and output a value E RR.
Actor. The actor is standard feedforward MLP taking the state input and outputting 2a- dimensional vector, where a is the action-space dimensionality. The actor predicts the mean and standard deviation of Gaussian distribution for each action dimension; during training value is sampled at random, during evaluation the mean is used.
B.5 TD3
B.5.1
Architecture
We adopt the same implementation and hyperparameters is used the ExORL benchmark. Hyperparameters are reported in Table 5.
Critic(s). TD3 employs double Q networks, where the target network is updated with Polyak averaging via a momentum coefficient. The critics are feedforward MLPs that take a statc-action pair (s, a) imput and output value R.
Actor. The actor is a standard feedforward MLP taking the state as input and outputting an a- dimensional vector, where is the action- space dimensionality. The policy is smoothed by adding Gaussian noise the actions during training.
Misc. As is usual with TD3, layer normalisation [3] is applied to the inputs all networks.
Table 5: Hyperparameters for Non-zero-shot RI.
Hyperparameter
CQL
Offline TD3
GC-IQL
Critic dimensions
(1024, 1024)
(1024, 1024)
(1024, 1024)
Actor dimensions
(1024, 1024)
(1024, 1024)
(1024, 1024)
Learning steps
1,000,000
1,000,000
1,000,000
Batch size
1024
1024
1024
Optimiser
Adam
Adam
Adam
Learning rate
0,0001
0.0001
0.0001
Discount
0.98 (0.99 for maze)
0.98 (0.99 for maze)
0.98 (0.99 for maze)
Activations
ReI.U
ReLU
ReLU
Target network Polyak smoothing coefficient
0.01
0.01
0.01
Sampled Actions Number
CQL
0.01
CQL Lagrange
False
Std. deviation for policy smoothing o Truncation level policy smoothing
0.3
IQL temperature
IQL Expectile
0.7
B.6 Code References
This work was enabled by: NumPy [30], PyTorch [61], Pandas [56] and Matplotlib [31].
C Extended Results
In this section we report full breakdown of our experimental results on the ExORL benchmark by dataset, domain and task. Table reports results for methods trained on the 100k sub-sampled datasets, and Table reports results for methods trained the full datasets.
25

---

Table 6: 100k dataset experimental results on ExORL. For each dataset-domain pair, report the score the step for which the all-task IQM is maximised when averaging across seeds, and the constituent lask scores at that step. Bracketed numbers represent the 95% confidence interval obtained by a stratified bootstrap.
Dataset
Domain
Task
CQL
Offline TD3
GC-IQL
SE-LAP
VC-FB (ours)
DVC-FB (ours)
MC-FB (ours)
Walk
138 (122- 1401
210 cos 2013
115 120
58 cz 100
181 (123 276)
4:6 (425 400
39-1 (oss 512)
217 :181 2061
Walker
Stand
388
362
284
190
558
624
590
480
Run
71 wa 1o
m M]
04 a0
101
-311
134 cT
106 (7> 1m71
Flip
153 (130 37M
162 1345
1713
157 02 17t
70 05 800
163 190
325 1200 2000
250 (215 2001
164 11:3 1901
Stand
167 (23 2091
119 12 336
43 3750
108 (1 1921
131 (s0 1781
331 1200 430
269 300)
171 (T1
Quadruped
Roll Fast
03 - 2019
14 2231
66 144
80 oz 1000
usa - 1277
141 (4) 1989
146 (6 - 207
(19 - 1990
Roll
251 (147 axee
96 x
224 cr
100
130 (r1
141 por
209
1.32 pa0
Jump
128 ia2 223)
i5 2491
2477
91 1000
121
0310 2020
167 230
Escupe
3
3 30-44 90
10 30
7 0i 120
(4 10
(6 18
5 12)
RSD-100L
Reach Top Right
133
1561
457
308
1
D
0
Maze
Reach Top Left
Reach Bottom Right
0a 1
921
302 (1 002)
384 is 7331
1218 1600
0
244 (10
723
Di 15
Reach Botton Left
253
85
25
0
479 ite-
250
384
Reach Top Right
37 00
0a 10
D 30
6
ck
Jaco
Reach Bottom Right
Reach Tap Left
37
21 (22 sn
Oar-m 10
0
10 1
2 ti
O
2a 4)
O
2 20
7
11 (T- 10
9 310
Reach Battom Left
co 200
no
20
3
Walk
126 (132 3301
192 (20 100
24
129 (115- 1331
76 (ss - 1181
122 (se 2400
38 (32 4X
119 :ss 2101
Walker
Stand
246
2915
141
206
237
223
223
209
Run
31 css am
ut+ 631
20 116 700
49 0s 60
css adn
31 mt MMC
Flip
115 (102 12M
72 :47 53)
22 110
100 0z
1130
47 c00 G2)
62 (40 901
47 (43 52
41 (40 50)
Stand
188 (00 201)
261 4720
76 120 2200
285 (145- 4121
278 4401
6150
196 200
172 -
Roll Fast
161 ma 2030
151 (x 3900
98 p1 1080
64 64 189
96 199
43 p2 1373
155 cm maa
78 ost 1991
Quadroped
Roll
322 130
200
166
111
105
130 (T.
183
178
Jump
213 (133- 2001
189 (22 3000
as 300)
128 (14 220
75:00 - 133)
2200
(87 - 121)
1417 (53 -
Escupe
Ba--
di--1 to
2 LD
3 at
6 mm
100K
Reach Top Right
Da--o
0
O
0
0
Maze
Reach Battom Right
Reach Top Left
Du--
O 1
0 20
925 (ms 900
io m
0 -i
D m
0 m
28 (4 1231
0 pi-m mi
(0 104
Dar r5
Reach Botion Left
0o-
37
O
D
0
D
Reach Top Right
53 on
117 X31
ao m
ie -
3mm 10
0a 80
0 p--m mi
Jaco
Reach Bottom Right
Reach Top Left
55
52 c3 so
2 1 to
34
12 51
2
O
Di- 60
0
12
90
26 120 42
23 (11 50
Reach Bottom Left
32.0 am
3 4)
2a 18
a5g
Walk
147 (123 1096
150 u22 1043
23
93 e1 190
251 (1a2 200
262 (1T4 944)
248 (24a 250
260 (175 347)
Walker
Stand
406 4431
262
142
276 9m9
407
455
387
423
Run
38 oss 42)
144
471
19 12 230
58 es to
mm
11.1
82 178
911
87 (50 500
(TL 1071
Flip
149 (LUS 3TM
163 032
179
23 120
141 90 150
193 (135 200)
228 (250
180 (150 2000
182 (153 237)
Stand
299 (142- 4351
848 1722
25L (21a ann
315 (1r
4121
459
1350 4843
447
457
Roll Fast
164 (7s 1989
446 iss aw9
111 us 1000
185 am
a 9
287 PM 3090
250 pa5 190
290 cse
1901
200 1755 2401
Quadruped
Roll
264
709
117
189
460
415
129
456
Jump
196 (L32 26m
(35e BAT:
04
363
418)
(224 207)
391
4113
372
Escupe
6.o. 110
23 :s5 391
aa
16. s9
45 cs om
31 p3
45 ce0 4x
42 oz 1m0
D1AY
Reach Top Right
780 75mh
796
706
O
0
27
Mazs
Reach Batton Right
Reach Top Left
943 pu 200
O mm--n
940 1941 1400
0 m
901 oon
a0
O ti
764 asa
576 4060
Dm- t
910 (:30 1000
Ua m
0 0i-
557 cs0
853 pako 100
Di -t
Reach Botiom Left
799
139
O
0p-m
0
O
Reach Top Right
17 oo 20
0a--m mo
aa m
80-21 2111
2 to
5 mt
0 p--m mi
9ie 100
Jaco
Reach Top Left
Reuch Battom Right
17
10.s-m
0a-
(1
3a
0 11
20- 50
4
70-10
5
0
27 12 t3
D 0
11
Reach Botiom Left
2 17
0
0m
2 31
10 19
in a0e
9ie 140
26

---

Table Full dataset experimental results on ExORL. For each dataset-domain pair, we report the score at the step for which the all-task IQM is maximised when averaging across seeds, and the constituent task scores at that step. Bracketed numbers represent the 95% confidence interval obtained hy a stratified hootstrap.
Dataset
Domain
Task
FB
VC-FB (ours)
MC-FB (ours)
643)
864
792 (226 657)
Stand
928 0223
878 (854 2080
873 012 9340
Walker
Run
1242
320
azs 374
343 0223 300
Flip
325 (682 5981
5-12 (115 371)
598 (st 637)
957 (o0 mil
(177 9500
949 ous 6550
Roll Past
574 (os3
612 (471
505 ras
Quadruped
Roll
(s0s 644)
(21) 9900
600
9000
Jurng
736 mai 1i3)
630 (270 600)
705 (003 707)
Escape
91 y03 125)
59 (20 540
RND
Reach Top Right
(
425 (155
270
Mazo
Reach Bottom Right
Reach Top Left
612 osu 9131
10 00-00
0
151 (138 7800
0 6
173 0013 9340
Reach Bottom Left
268 io- 5380
270 (2- 5001
0 27
Reach Top Right
48
24
Jaco
Reach Top Left
23 6 um
14 (4- a5)
(7 A3)
Reach Rottom Right
60
47
Reach Bottom Leil
27 (12 42)
88 00 14m
( 30)
Walk
148 2201
1455
1821
129 1761
Walker
Stand
318 (201 ass)
255 (252 3000
285 (202 300)
Run
51 -01
17 50
Flip
57 cas 971
83 100 1171
109 os 1at:
Stand
417 100 6031
295 (105 430
210 cas 2877
Rall Fast
110
271 csss 9000
215 (129 9980
Quacruped
Roll
(116 340
c33 255)
303 (300 5300
Jump
287 021 4301
67 (44 om)
161 (13D 194)
Escape
10
7
RANDOM
Reach Top Right
0
0
Maze
Reach Bottom Right
Reach Top Left
00-00
14 618
317 a 6091
0 00
307 o 6141
0.0-0 6g
Reach Bottom Left
10 0i
oo
0 mi 6o
Reach Top Right
5
Jaco
Reach Top Left
30 - 2001
oa - 180
10 313
Reach Bollom Right
15
21
Reach Bottom Left
a a-
18 12--2-0 34
1 (0 3)
Walk
459 6881
536
519
Walker
Stand
478 (451 4941
147 (422 4720
517 4015 6520
Run
87
84
1100
Flip
235 (151 a19
251 (151 3500
301 (013 2550
mss 8141
432
604 /556 AS1)
Roll Fast
497 (190 514)
293 (170 407)
495 (091 4580
Quadruped
Roll
650
761 (716 7040
Jump
628 neen w99
231 ce 3799
608 (914 4220
Escape
65
21 (14 27)
(25
DIAYN
Reach Top Right
0
0 6o
Maze
Reach Tog Left
928 o0r 95m
614 0080
Reach Bottom Right
0
Reach Bottom Left
in- 506
14
y (4)
Reach Top Right
41--7
10
4a 5)
Jaco
Reach Tog Left
5 10)
2 o0 3)
Reach Bottem Right
6a-8)
7
Reach Bottom Left
25 1- 47
12 1i--ue
471
27

---

Table 8: Aggregate zero-shot performance on ExORL for all evaluation statistics recommended by [1]. VC-FB outperforms all methods across all evaluation statisties. " means a higher score is better: means lower score is better. Note that the optimality gap large because s 1000 and for many dataset-domain-taskk the maximum achievable score is far from 1000.
Statistic
SF-LAP
GC-IQL
FB
CQL
MC-FB (ours)
VC-FB (ours)
IQM?
92
95
99
128
136
148
Mean f
87
126
118
138
142
154
Median f
108
104
104
132
133
144
Optimality Gap &
0.92
0.88
0.89
0.87
0.86
0.84
Table D4RL experimental results. For each dataset-domain pair, we report score at the step which IQM is maximised when averaging across seeds. Bracketed numbers represent the 95% confidence interval ohtained by a stratified bootstrap...
Domain
Dataset
CQL
SF-LAP
FB
VC-FB (ours)
MC-FB (ours)
medium
36 (3-36)
(1-7)
3 (-1-8)
39 (35-0)
32 (28-36)
HalfCheetah
medium-expert
43 (7-51)
26 (20-32)
3 (0-6)
27 (24-28)
24 (31-50)
medium-replay
42 (42-4))
29 (29-30)
(2-1
31 (26-36
20 (18-22)
medium
70 (68-73)
7 (1-13)
7 (0:14)
42 (3745)
36 (33-40)
Walker2d
medium-exper expert
102 (97 107)
15 (0:31)
3 (1-5)
82 (72-92)
74 (66-77)
medium-replay
13 (11-14)
11 (8-1)
12 (2-77)
20 (19-11)
22 (19-34)
All
All
48
15
5
34
28
Value Conservative Universal Successor Features
this section, we develop value conservative regularisation for use by Universal Successor Features (USF) [5, 7], the primary alternative to FB for zero-shot RI..
Recall from Section that successor features require state-feature mapping & : IRd which is
usually obtained by some representation learning method [5]. Universal successor features are the expected discounted sum of these features, starting in state so, taking action ag and following the task-dependent policy Te thereafter
(1800 ao, 2) =
Σ rs ((((41)) ao, n
(21)
USFs satisfy a Bellman equation [7] and can be trained using TD-leaming on the Bellman residuals:
LsF B(en, (118), T) (0((+1) (7(65)));7(((4)))))))
(22)
where & is a lagging target network updated via Polyak averaging, and 2 identical to that used for training (Appendix B.1.2). As with FB representations, the policy maximises the Q function defined by tb:
n (s) argmax, (s)
(23)
28

---

and for continuous state and action spaces trained an actor critic formulation. Like FB, USF training requires next action samples 4t+1 ~ T. (3(+1) ) for the TD targets. We therefore expect SFs to suffer the same failure mode discussed in Section 3 (OOD state-action value overestimation) and to benefit from the same featurs
remedial measures (value conservatism). Training
successor features (VC-SF) amounts to substituting the USF Q function definition and loss for FB's in Equation 11:
CvC.SF = a (Bo-Doorldbb) - a, 2) 2
(11 (V5, + CSF.
(24)
Both the maximum value approximator ju(als) (Equation 17, Section B.1.3) and a-tuning (Equation 18, Section B. 1.44 can be extracted identically to the FB case with any occurrence of F(x, a, =) = substituted with (s,s, a, 22" As USFs do not predict successor measures we cannot formulate measure-conservatie USFs.
E Negative Results
In this section we provide detail on experiments we attempted, but which did not provide results significant enough to be included in the main body.
E.1 Downstream Finetuning
If we relax the zero-shot requirement, could pre-trained conservative FB representations be finetuned on new tasks or domains? Base CQL models have been finetuned effectively on unseen tasks using both online and offline data [41], and we had hoped to replicate similar results with VC-FB and MC-FB. We ran offline and online finetuning experiments and provide details on their setups and results below. All experiments were conducted on the Walker domain.
Offline finetuning. We considered a setting where models are trained on low quality dataset initially, before a high quality dataset becomes available downstream. We used models trained on the RANDOM- 100k dataset and finetuned them on both the full RND and RND-100k datasets, with models trained from scratch used as our baseline. Finetuning involved the usual training protocol as described in Algorithm but we limited the number of learning steps to 250k.
We found that though performance improved during finetuning, it improved no quicker than the models trained from scratch. This held for both the full RND and RND-100k datasets. We conclude that the parameter initialisation delivered after training on low quality dataset does not obviously expedite learning when high quality data becomes available.
Online finetuning, We considered the online finetuning setup where a trained representation is deployed in the target environment, required complete specified task, and allowed to collect replay buffer of reward-labelled online experience. We followed a standard online RL protocol where a batch of transitions was sampled from the online replay buffer after each environment step for use in updating the model's s parameters. We experimented with fixing to the target task during in the actor updates (Line 16, Algorithm 1), but found it caused a quick, irrecoverable collapse in actor performance. This suggested uniform samples from Z provide form of regularisation. We granted the agents 500k steps of interaction for online finetuning.
We found that performance never improved beyond the pre-trained (init) performance during finetuning. We speculated that this was similar to the well-documented failure mode of online finetuning of CQL [59], namely taking sub-optimal actions in the real env, observing unexpectedly high reward, and updating their policy toward these sub-optimal actions. But we note that FB representations do not update w.r.t observed rewards, and so conclude this cannot be the failure mode. Instead it seems likely that FB algorithms cannot
29

---

FB
MC-FB
VC-FB (scratch)
VC-FB
FB (scratch)
MC-FB (scratch)
350
300
250
Return (IQM)
150
100
50
0
50
100
150
200
250
Learning steps (k)
Figure 9: Learning curves for methods finetuned on the full RND dataset. Solid lines represent base models trained RANDOM-100K, then finetuned; dashed lines represent models trained from scratch. The finetuned models perform no better than models trained from seratch after 250k learning steps, suggesting model re-training is currently better strategy than offline finetuning.
the narrow, unexploratory experience obtained from attempting perform specific task improve model performance.
We believe resolving issues associated with finetuning conservative FB algorithms once the zero-shot requirement is relaxed is an important future direction and hope that details of our negative attempts to this end help facilitate future research.
30

---

F8
MC-FB
VC-FB (init)
VC-FB
FB (init)
MC-FB (inis)
600
500
Return (IQM)
300
200
0
50
100
150
200
250
Learning steps (k)
Figure 10: Learning curves for online finetuning, The performance at the end of pre- pre-training (init performance) is plotted dashed line for each method. None of the methods consistently outperform their init performance after 250k online transitions.
F Learning Curves & Hyperparameter Sensitivity
EXORL Learning Curves. We report the learning curves for all zero-shot RL methods and CQL in Figures 11. 12, and 13, For all domains except Jaco, the y-axis limit is fixed at 1000 as that is the maximum score achievable in the DeepMind Control Suite. For Jaco-related figures, the y-axis limits fixed at 100 as no method achieves score higher than this.
Hy Hyperparameter Sensitivity. We report the sensitivity of VC-FB and MC-FB to the choice of two new hyperparameters: conservative budget and action samples per policy N on the ExORL benchmark. Figure 14 plots the sensitivity of VC-FB the choice on Walker and Maze domains across RND and RANDOM datasets. Figure 15 plots the sensitivity of MC-FB to the choice of on Walker and Maze domains across RND and RANDOM datasets. Figure 16 plots the sensitivity of MC-FB the choice of N on Walker and Maze domains across RND and RANDOM datasets.
We further explore the sensitivity of VC-FB performance on Walker2D from the D4RL benchmark w.r.t. the choice of conservative budget 7. Figure 17 plots this relationship when trained on the "medium-expert" dataset from D4RL.
31

---

Wi walker-all-lasss-nre
Wa walker-flip-ree
wa walker-run-rrdd
walker-stand-rnd
1000
1000
1000
1000
Return (IQM)
800
800
800
800
600
600
600
600
400
400
400
200
200
200
200
0
walker-walk-re rnd
wa maaleer-llltaskk--adddo
wa walker-flip-andon
wa walker-run-random
1000
1000
1000
1000
Return (IQM)
800
800
800
600
600
600
600
400
400
400
200
200
200
200
0
walker-stand-random
wal walker-wall--anddo
walker-all tikker-alltaskk--daan
walker-flip-diae
1000
1000
1000
1000
Return (IQM)
800
800
800
600
600
600
600
400
400
400
400
200
200
200
200
0
walker-run-diayn
walker-stand-dian
walker-walk-diay
tllss-ll--
1000
1000
1000
1000
Return (IQM)
800
800
800
800
600
600
600
600
400
400
400
400
200
200
200
0
maze-btt-leef--nat
maze-btm-rrghtt-i
maze-top-left-mm
maze-top-right-me
1000
1000
1000
1000
Return (IQM)
800
800
800
800
600
600
600
600
400
400
400
400
200
200
200
200
maze- maaze-alllaasks-raddoo
maze- lom
mazee mrazze-lop-eel--anddo
1000
1000
1000
1000
Return (IQM)
800
800
800
800
600
600
400
400
400
400
200
200
200
200
0.0
0.6
0.0
0.6 0.8 1.0
0.6 0.8
0.0
0.6
Learning steps (x10%)
Learning steps (x10*)
Learning steps (x10%)
Learning steps (x10*)
SF-AP
GC-IQL
FB
VC-FB
MC-FB
CQL
Figure 11: Learning Curves (1/3). Models are evaluated every 20,000 timesteps where we perform 10 rollouts and record the IQM. Curves are the IQM of this value across 5 seeds; shaded areas are one standard deviation.
32

---

wwwze-lop-righl-raaae
mäze-all lalze--llllaskk-diaay
mazz--blmlell-diaa
maaze-bblmrrghh--diaay
1000
1000
1000
1000
Recurn (IQM)
800
800
800
800
600
600
600
600
400
400
400
400
200
200
200
200
0
maze-top-leff-diaye
maze- maze-top-righe tiiiitt--ann
jaco-all-tasks-rnd
mico-btm-left-mmd
1000
1000
100
100
Return (IQM)
800
800
600
600
60
60
400
400
40
200
200
20
20
0
0
nro--btm-right-rnd
lod--top-
jaco-top-right-me
aco-aalltaskk--raddom
100
100
100
100
Return (IQM)
80
60
60
60
60
40
40
40
20
20
0
0
jaco- hico-bbm-lef--random
jaco- boabbbt--
jaco- loo--too-left-randdo
jaco-top- rrc--top-righh-rrnddo
100
100
100
100
Return (IQM)
80
80
60
60
60
60
40
40
40
40
20
0
0
aaco-all-tasks-dee
dico-btm-left-diann
jaco- biagbtt--rg
jaco-top-left-daymn
100
100
100
100
Return (IQM)
80
80
80
80
60
40
40
40
40
20
0
0
jaco-lop-rghhldiayy
quadruped-all-taskkee
quadruped-escape--ee
quadruped-jumae
100
500
500
500
Return (IQM)
80
400
400
400
300
300
300
40
200
200
200
100
100
100
0.2
0.8
0.0
0.6
0.8
0.4
0.8 1.0
0.0
0.6
Learning steps (x10%)
Learning steps (x10%)
Learning steps (x10%)
Learning steps (x10%)
SF-LAP
GC-IQL
FB
VC-FB
MC-FB
CQL
Figure 12: Learning Curves (2/3). Models are evaluated every 20,000 timesteps where we perform 10 rollouts and record the IQM. Curves are the IQM of this value across 5 seeds; shaded areas are one standard deviation.
33

---

quadruped-roll-mnd
quadruped-roll-
rostr--rrdrrol-----rddddd fast-rnd
quadruped-stand-mit quadruped--ll-taskk--.
500
500
Return (IQM)
400
400
300
300
300
300
200
200
200
200
100
100
100
100
quadruped-roll-random quadrrpeddrol--astt--ndoo
500
500
500
Return (IQM)
400
400
300
300
300
300
200
200
200
200
100
100
100
100
diayn quadruppedessaap-dlayn
quadruped-jump-diae
500
1000
1000
1000
Return (IQM)
400
300
600
600
200
400
400
400
100
200
200
200
0.0
0.2
0.4
0.8 1.0
quadruped-roll-dine
quadruped-roll-fast.eoo
quadruped-standd-diayy
Learning steps (x (x109)
1000
1000
1000
Return (IQM)
800
800
800
600
600
400
400
200
200
200
0.0
0.2
0.4
0.6
1.0
0.0
0.4 0.6 1.0
0.0 0.2. 0.6
Learning steps (x10%)
Learning steps (x106)
Learning steps (x10%)
Learning Curves (3/3). Models are evaluated every 20,000 timesteps where we perform 10 rollouts and record the IQM. Curves the 1QM of this value across 5 seeds; shaded areas standard deviation.
Figure 13:
34

---

walker-rnd
Reaur (IQM)
200
100
wolker-random
maze-random
Return (IQM)
100
50
50
25
Conservative budget
Conservative budget
Figure VC-FB sensitivity to conservative budget 7 on Walker Top: RND dataset; bottom: RANDOM dataset. Maximum IQM return across the training run averaged over random seeds
walker-rnd
Return (IQM)
200
200
0
walker random
random
200
Return (IQM)
100
50
50
25
0
0
50
100
50
100
Conservative budget
Conservative budget
Figure MC-FB sensitivity conservative budget Walker and Maze. Top: RND dataset; bottom: RANDOM dataset, Maximum IQM return across training run averaged over random seeds
35

---

walker-md
maze-rnd
400
Return (IQM)
400
200
walker random
maze random
200
100
Return (IQM)
150
75
100
50
25
0
Action samples per policy
Action samples per policy N
Figure 16: MC-FB sensitivity to action samples per policy N on Walker and Maze. Top: RND dataset; bottom: RANDOM dataset. Maximum IQM return across the training run averaged over 3 random seeds.
Walker2D medium expert
100
20
25
35 40 45 50
75
Conservative budget
Figure 17: VC-EB sensitivity to choice of conservative budget 7 on Walker2D from the D4RL benchmark.
36

---

Code Snippets
G.1 Update Step
updato_
self.
observations: torch.Tanzor,
torch.
sest_abserration::
discounte: tercor
7
ze: Tensor
Galculates the loss for the forvard- reprezentation natvork. Forward- representation
opdate
oa the successor aeasure (equation 24, Appendix B)
15
2. Conservative lose: penaliser out distribution actions
observation tenser shape [batch_cize, observation.loaghh actions: action teacor thape Ebatch_eizo, action, nest_observations next obrervation tensor ot [batch_ discousts: discount tensor shape [basch,eize, 25: policy tensor of shape [batch_pige, s_dinension] step: current treizing step
ohservations:
19
23
Returns:
dictionary
for logging
27
update step
xodels
30
core_losa.
31
core_xetrics,
__zext, Rex
35
x2_uext,
39
actor_ate aad__dddddeddd_ddeee
Abservaiins--baaraaae actioas=action next wtt.::/beeaaiiiottttt..oo.oo discouata-diacounta. stap=step.
43
calculate panalcy
50
1f self.ncft.
conservative.peaalty.
conservative.satric..
neasuue
abervntiossebaeevaaat
za-za,
58
F1=F1,
P2=P2.
R_aext-B_nextt
VCFB
66
=lae:
conservative_ponalty..
conserottie.suttrich.
70
self. www.:ullll.cosratiiie....o
37

---

actor_cdd_der=scte::
T1=F1,
F2-F2.
78
79
tune sipha froa consorvarive penslty
sipsa, aepra,netrice self (
consnrvative,1oa alpha coneevativw.penalyy
total_1ose cose_1oss conseevative_1oo
86
step optiniser op..optiiserr wtt.seee..dd......ooooooo
total_lose backward()
90
for paran in self. Fa P...aa.aaraeeerr if paraz. grad not Bone:
91
paraz. Fel..FF.optiniiee opt.iiiiee (t
reture xetrics
G.2 Value-Conservative Penalty
def .value, wwwsorvattio..paaaee.... (
observations: Tensor, next. sext_obscrvations torch. Tensor,
5
ze:
torch. Tensor,
Tenshr
F2: Jurch.Tensor
>>
torch.Tezsor:
torch. Tezsor:
Calculated
penalty
Arga:
13
observations: observation tenser of chape bbstch.cize, obserration_joaggh
observation (batch, abservation_oaaghll
25: tack tensor of shape [batch, z_dineacion] actor_atd_dev standard deviation of the actor foreard exbedding forvard unbadding no.
17
20
Retures:
conservative.pcaalyy. the value conservative ponalty
...
with torch. 20, 20. grad();
observations, wwxt_observations
and Ba
fold action
allaw tanzera dimeasion
dimezsion
paxsed through reshape the output back to saistain the action sasple dimensios ortecttedobbeevattiooo
29
observations.zopon
eatiiooaa... russhhae (self. mollfeedd attin
batchh,izzze -1)
sopeated_ 2o_ood o 2s. re.repeat (ww.ffoood.aeeeeeooooooooo acttion_papples. 1, 1 De.rhppp salf, bizeh
33
owlf.ood..ction_nns
cdd_actions anpty(
36
aiza=(axlf act.ieinnnncctnnnnnon... wwicee-ell...dviiie
aalf, batchh.zzz, self action
37
sepeated_obas oww.rvations.eeeeeee......eee istoo bbservatioss.repee self. altoractr__aste actiommapaata....
ressbape actor_.ctiom__aanil
self belf.betch.size.
rept oow.rttt.oorrttiinn.eeeeede
oost ropeat
self. aelffacttr_accionn actimmmttmmaepee....... 1,
). rushape (sulf (swlf.act actor aetioracctio_saanpl.
self batch -1)
aepeated_ze_acteee
repest acppe 1) reshappe
aetor_current current.actions actor rapeated_obaeree repeated actor,
38

---

stá-sctor, std=sctor_p1d.d07, sanple-
bizch lengtt)
56
actor, actor_ next, aet.otttttiona.................t self actor(
57
copoatee,.oxt..oaaaatttttte oborrvationn, s"ropeated_26.300 atd=actor_atd.der. sauple-Irse,
[ector_,actoo_suaple batch,aize, aetios_length)
get
oed_Fl, aod_F2 = self 28. forvard wer.errdd.eprreenncte rapresentaation ood_ addaccioos, xepeated__9_ood
[ood_action,
acter actor_curreat_ Fl, cutor_cu:rent__2 self .B. forvord, www.ioorrrrpreeeeee.ed representation
69
(epastes.bborrTattiaa..... acter, current.actions, 2ope
disent_dia]
actor setor__ext_Fl, actor_ self. rew.torard
72
actor_nect, actiions 24psttad
[actor_ pesttor_cttionsanpll
batch_aize latent_dix]
xepeated_F1, xepeated_ (e.zepeat(
self.batch, -), repeat acten 1,
76
zeshape (self. acter.actorr.eeeeeeee acter_action,,ne
77
self wwtaaaatttrrioooeeee acticcn,tnnnepee samples,
actor actoo_,cttioo.asppla
bizeh
80
cat_F1 torch.cat(
aod_P1,
actor setor_nexi_Fl, repeatei_F1,
}.
88
din-D,
72 (at)
aod_P2,
actor cotor_current_22,
repeated__2,
96
1, din=0,
100
repanted_ za 28. rapeat toslf tctall 1, 1) rarhapp salf tatal.tt self batch_aize, -1
101
104
convert Qe
105
cal_cat_01 errch.einsun(MM
appeated_zt..reshhee
samplas,
107
108
cal_cat002 torch. oineus ("ed, ad - st, cat_F2, aopested.zt).esthee salf tstal total_lstitnns aaslion, self batch_aize,
109
111
112
0gl lagannaxp
113
torch. ogguxooppeqlleer clt din-0), nean logennerp (ogleerppoll.ctt.
115
116
117
existing
01, (torch.einosn("dd, 8", F, [F1, F2]]
120
Cnaarvatiive_pennlty egl_loggonexp (01 42) mean
121
cassorvarivo.peaally
G.3 Meassure-Conservative Penalty
39

---

observations: torch Iensor, seat_dbaervations torch. Tensor,
torch. Tonsor,
actor_sta,
F1: torch. Tensor,
F2: torch. Tensor,
torch. Tenser,
Ri_next: Torahr
#2_next: torch.Tenser.
12
+> torch. Tezaur:
vHn
Calculstes neasure conservative penaity.
15
Arga:
16
observations: observation tenser of shape [batch_size, observation leagthh) observation tensor abatch_aize, observacion,length))
19
25: task tensor of shape (batch, size, 2, s_dimession] acsor_ctd stazdard devistion of the actor forvard usbadding forvard enbodding no.
20
23
B_nezt: backuard sabedding K1 _i_aex:: successor neasure no.
24
27
penarvativ..eaalyyy the seascre conservative penalty
28
torch. g)_dra
rapeat observationa, nest_ cest_cbaarvations, sa, and Bx
32
fold the action saxple dinession into the batch disezsion passed through zaistain action sample abaervations.
rexhapa output
35
36
oolf.ood, aot_iccioon
repxated_ 2a_ood ra. (aelf action,
()rrephppe(
30
self, basch_
cod_actiona
enpty(
size=(anlf. wow.lf..oddaatiioe act.aatttoot.. self, half.batch_size aelf.
devica=self... devvice
44
Lunifors, (-1,
oberrvttiss.reget::
47
1, self aeng..............a
). ressaape (sell.
swlf.batch..te -1)
repsated_aaztt_
set.
1,
rasbaaa (anlpp((alff.ctaa
+ salf. bitcc_size,, -1)
52
repeated_ 29_actor__ ra.rapatt (self. nall. (ett.
1). 1).rasba
aelf.actor,
batch
55
actor_currest. currest, actrozz,, wwa.tted_oberrreiooss.ccoc repeated_ea__sston iid=sctor_ctdddov, zanple-Trse,
actor,
self actor(
[estor_actios_angpla batch_xiza, lettor_
actor_next_acee self actor(
63
grropested.zt.att
std=actor_.td_der
sanple-Trse,
67
[actor_actioo_.xxppiee + batch, acngot,
ood_Fl, sod_F2 self fB, F8. reprereearrretttiooo shantec_absaraatinns.oom sod_actions. repaated_2s_ood 28_ood [ood_action,canplee
71
lstent_din)
actor cxsro_crrost__Fe Fi, uctor__urrent.1 self .B. www.wrdd representationn
15
curre,currees.ctte
dix]
actor seor__ext_Fl, a2tor_noxt_F2
repeated, wbw.ttttooertttto.....ccoo acter_next, actiona, zoposted.2s,actor
79
bazeh dite
xepeated_Fl, ropeated_F2 (epzepa
40

---

1. resshape (self. aetorracct..... arttoa, sele. bizeh DD F2. ( self setor_action,amplie
r.xeshape
actor, actto,,ptio bazch
87
cat_F1 corch.cat(
ood_F1,
actor, actor_curree currens_ Mestr_ repeated_F1,
}. din-O,
(otch
ood_F2,
99
actor, actor_currene current__2
repeated_
102
J. din-C.
103
106
col_cat_fe torch. tirch. ("adu self samples, self bazeh_oize, -1
+> as", cat_Fi, B_next) B_uoxt) :zeshape
107
col_cat_K2 einsus as", cat_F2, s.uoxt).reshape salf
110
salf, batch_aize, -1
111
lagannexp
114
carch logsuxesp(eal_cc eat_ din-0) mean
115
torch. logguulaouaapptlcccce meaz)).
118
conservative.pennll cog_snggue (Mi_nezt N2_next) mean O
119
cosservative.pealllyy
41

---

G.4 Tuning
def _Luse_elpha{
self, conarratiivn.aeelly:: Barch.Tanaor,
-> torch, Tezsor:
penalty saight (alpha) target pasalty.
Diacussed Appandix B.1.4
8
Arge:
9
conservativa,peaal the current conservative penalty
alpha: updatad Alpha
12
13
alpha
alpha torch. torch.clampaaelf. ang.g nin-B.0, zax-la6) ioss =[ [
16
17
-0.5 alpha (comarvntiva.penlty aelf cog.geg....aarrrttooop......
20
self.critie, aaaiiiie_alppp.....
sore backward (
alpha torch claph log. nin-B.0, desech)
24
42

---

H
NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect on the scope the claims made, e.g., the approach was only tested on few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to to-tet system might not be used reliably to provide closed captions for online lectures because fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems privacy and faireess.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
43

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
This paper does not include theoretical proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimen- tal results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
Code and hyperparameters provided; datasets and environments are open-source.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the case of large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
example, if the contribution
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
model (e.g., large language model), then there should either
44

---

We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source -ource models, may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
See Appendix A and B, and https://githiie http:////ooooeeeeeeeeeeeeraooo-a
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https:
httt:::://iips.ccpubbbbaa /
guides/CodeSuiee www.eeewwwissssooooooooooooooooe ssionPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to repro- duce the results. See the NeurIPS code and data submission guidelines (https: : // //tpp/ipp....//// cc/ waaeeeSubmissionnPe
public/ coolic/guide//
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only a subset experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Appendix A and B. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
45

---

Justification: See Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1 1-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric eтrоr bars that would yield results that are out range (e.g. negative error rates). error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer re- sources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: See Appendix A.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimen- tal runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't itdntt make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:/ /ttp:://neuii.pp.ee wwtpeepppppppppaa///eeaaaaaee.ee cc/publlc//baacceeddddlllll
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
46

---

The authors should make sure preserve anonymity (e.g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
This paper discusses methods for improving the performance of zero-shot RL methods when trained on low quality offline datasets. We do not expect this work directly impact society positively or negatively.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g-. disinformation, generating fake profiles, surveillance). fairness considerations (e.g.. deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to par- ticular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strate- gies (c.g., gated release of models. providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
47

---

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re- packaagd, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification: https://gi httpbbb.tttu..///// wwwphhhww...cceceeeeeeeeeeeeeeee com//enjennser/zero--hooo
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Question:
Answer: [NA]
Justification:
Guidelines:
48

---

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent
Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country institution) were obtained?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For imitial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review.
49


Reviews:
[{"review_id": "aPuWIgjhJe", "reviewer": "Reviewer_qeSX", "summary": "This work proposes modifications to Forward-Backward representations method. The modifications are aimed to improve the method's robustness to dataset quality. The vanilla FB suffers when the dataset does not cover the state space well, and overestimates the quality of actions. The authors show this with a simple example on point-mass. Two modifications are proposed, VC-FB and MC-FB, which take inspiration from CQL and make sure the FB representations are not too optimistic on the transitions not seen in the dataset. \nVC-FB and MC-FB are then thoroughly tested on ExORL and D4RL. The authors show that VC-FB offers the best improvement on ExORL, although none of the proposed modifications beat CQL on D4RL benchmark. Nevertheless, in all settings the modifications greatly improve over vanilla FB.", "questions": "-  How do the proposed modifications affect the performance of the model when the dataset is of good quality? Does it hurt the performance?", "limitations": "The authors have adequately described limitations and potential negative societal impact of their work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is clearly written.\n- The proposed modifications have clear mathematical motivation, similar to CQL\n- Both the modifications improve over the vanilla FB representation, and even outperform CQL on ExORL\n- The experiments are thorough and clear", "weaknesses": "- The novelty is limited: the authors take CQL's method for correcting overly optimal predictions and apply them to FB.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Hi Reviewer qeSX. Thanks very much for engaging with our paper and for the positive comments. See our response to your question below.\n\n**Q1: How do the proposed modifications affect the performance of the model when the dataset is of good quality? Does it hurt the performance?** \n\nWe asked ourselves this question in Q3, Section 4, and responded to it on Lines 221-238, and with Figure 6, Table 1 and Table 8 (Appendix C). We’ll summarise our findings here for convenience.\n\nWe compared the performance of our methods with vanilla FB on the full RND, DIAYN and Random datasets and found our methods showed superior performance on all of them (Table 1). We then ran a second experiment where we trained our methods and FB on different sizes (100k, 500k, 1m, 10m) of RND (the highest quality dataset). Again, we found our method outperformed vanilla FB on each of these, with the discrepancy in performance decreasing as we approached the fully sized dataset (Figure 6). As a result, we’re confident that conservative zero-shot RL methods do not trade better performance on low quality datasets for worse performance on the full, high-quality datasets.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r0Cw0QDBYw", "reviewer": "Reviewer_5D6P", "summary": "This paper identifies an overestimation problem of zero-shot RL algorithms, particularly in low data or low data quality settings. To address this, they propose to use a value conservative method to mitigate the overestimation (from CQL). They showcase that this effectively allows for reducing the overestimation, then empirically show performance improvements on low or bad quality data regimes, without impacting performance on usual regimes.", "questions": "* What is the intuition behind the overestimation reduction of CQL ?", "limitations": "There is a dedicated limitation section that correctly addresses the limitations of the method.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "I am not an expert of zero short RL, but the paper seems sound and is mostly clearly written. The problem seems relevant as zero short RL from low quality data is relevant for robotic applications, as underlined by the authors. I like the first figure, that provides intuition, similar to the ones in DDQN's paper. The method is clearly motivated and explained, the experimental evaluation is clear, the figure and caption allow assessing the defined precise scientific questions. I do not see much flaw in this paper presentation and method, but my qualification is not too high.", "weaknesses": "**Provide a bit more background and intuition about zero-shot RL**. This is not really mandatory, but it would help a broader audience.\n\n**Introduce the RANDOM and RND datasets**. Maybe an introduction of these datasets can help better grasp Figure 2 and 3, and would help the previous point.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Hi Reviewer 5D6P. Thanks very much for engaging with our paper and for the positive comments. See our response to your questions below.\n\n**Q1: Provide a bit more background and intuition about zero-shot RL**.\n\nThanks for the pointer. We’ll add the following text to help intuition after the formal introduction of zero-shot RL in lines 49-55.\n\n*“Intuitively, the zero-shot RL problem asks: is it possible to train an agent using a pre-collected dataset of transitions from an environment such that, at test time, it can return the optimal policy for *any* *task* in that environment without any further planning or learning?”*\n\n**Q2:** **Introduce the RANDOM and RND datasets (to help Figure 2)**. \n\nWe introduce the datasets in lines 177-186 in Section 4.1 (after Figure 2), and provide more detail in Appendix A.2, including a visualisation of the state coverage on Point-mass Maze in Figure 8. We point to Appendix A.2 at the end of Figure 2’s caption in light of not being able to introduce the datasets earlier. We feel that, together, these provide good context on the datasets used in our study.\n\n**Q3: What is the intuition behind the overestimation reduction of CQL?**\n\nWe provide (brief) intuition of CQL’s regularising effect in Lines 120-121, VC-FB’s regularising effect in Lines 124-125, and MC-FB’s regularising effect in Lines 131-134. We’ll summarise these again below for convenience.\n\nIn offline RL $Q$ functions tend to over-value state-action pairs that aren’t in the dataset. Intuitively, CQL amends the $Q$ function’s loss such that the value of these state-action pairs are downweighted, whilst upweighting the value of state-action pairs that are in the dataset (Equation 10).  VC-FB does the same thing, but for all possible downstream tasks (Equation 11), rather than just one task as with CQL. MC-FB is slightly different in that the regulariser operates on expected future visitation counts (measures $M$), rather than expected values ($Q$ functions). The MC-FB loss function downweights the *probability* of reaching some future state $s_+$ from state-action pairs not in the dataset, whilst upweighting the probability of reaching some future state $s_+$ from a state-action pairs that are in the dataset.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ot2QVaEvFN", "reviewer": "Reviewer_ZC95", "summary": "This work addresses zero-shot reinforcement learning (RL), focusing on training agents to perform tasks without explicit rewards during pre-training. The authors investigate the performance degradation that occurs with small, homogeneous datasets and propose conservative zero-shot RL algorithms inspired by single-task offline RL approaches. Experimental results demonstrate that the proposed methods improve performance on sub-optimal datasets without compromising effectiveness on large, diverse datasets.", "questions": "* Could the authors provide a more explicit definition of \"low-quality data\"?\n\n* It is recommended to compare the proposed method with a representative offline GCRL baseline.\n\n* Including related works discussed above would make this study more comprehensive.\n\n* Can this method also be applied to approaches based on successor features?", "limitations": "One limitation is that VC-FB and MC-FB do not outperform CQL on the D4RL benchmark; addressing this issue is left for future work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Zero-shot pretraining for RL is a crucial problem that can enhance generalization in downstream tasks.\n\n* The proposed method is well-motivated, and the writing is clear.\n\n* The experiments are thorough and effectively validate the proposed method.", "weaknesses": "* One weakness of the paper is the lack of a clear definition for \"low-quality data.\" As shown in Figure 8, it might be more accurately described as \"coverage\" rather than quality. Providing a more explicit definition is recommended.\n\n* Additionally, the comparison of baselines is incomplete, as it lacks an offline goal-conditioned baseline, which is highly relevant in the considered tasks. Including a representative offline GCRL baseline, such as GC-IQL, would strengthen the comparison.\n\n* Some related works are missing, particularly in the offline GCRL domain. References [1][2][3][4] are pertinent as they also focus on learning from offline datasets without rewards and deploying for multiple goals. The didactic example in this paper is very similar to Figure 1 in a prior paper [3], which also convey similar information during zero-shot deployment for multiple goals. A discussion with the prior results would be beneficial. Furthermore, a very relevant offline zero-shot RL method [5] is neither cited nor discussed. \n\n\n*References*\n\n[1] Yang R, Lu Y, Li W, et al. Rethinking goal-conditioned supervised learning and its connection to offline rl. ICLR, 2022.\n\n[2] Park S, Ghosh D, Eysenbach B, et al. Hiql: Offline goal-conditioned rl with latent states as actions. Advances in Neural Information Processing Systems, 2024.\n\n[3] Yang R, Yong L, Ma X, et al. What is essential for unseen goal generalization of offline goal-conditioned rl?[C]//International Conference on Machine Learning. PMLR, 2023.\n\n[4] Eysenbach B, Zhang T, Levine S, et al. Contrastive learning as goal-conditioned reinforcement learning[J]. Advances in Neural Information Processing Systems, 2022.\n\n[5] Frans K, Park S, Abbeel P, et al. Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodings[J]. arXiv preprint arXiv:2402.17135, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Hi Reviewer ZC95. Thanks very much for engaging with our paper and for the positive comments. See our response to your questions below.\n\n**Q1: Could the authors provide a more explicit definition of \"low-quality data\"?**\n\nTo the best of our knowledge, there isn’t one metric that formalises the quality and/or coverage of an offline RL dataset. We think of datasets as having two independent characteristics: 1) diversity—how well the state-action space is covered, and 2) size—the absolute number of samples in the dataset. It’s possible to have high diversity and small size (RND-100k) or low diversity and large size (Random-Full). We used “quality” as an umbrella term for both size and diversity, though we agree we could have used “coverage”. If the reviewer feels strongly about this we could replace “low quality” with “low coverage” in the paper.\n\n**Q2-3: It is recommended to compare the proposed method with a representative offline GCRL baseline, and include related GCRL literature.**\n\nZero-shot RL requires methods to generalise to both goal-reaching reward functions *and* “dense” reward functions (i.e. those that define locomotion tasks like Walker-Walk, Walker-Run etc.). We didn’t initially compare our methods to an offline GCRL baseline because such methods are, in principle, only capable of generalising to goal-reaching tasks. Indeed, it is not immediately clear how we would even define a goal for locomotion-type tasks. GCRL methods are therefore limited in a way zero-shot RL methods are not, so we didn’t think they represented a valid baseline.\n\nThat said, we're keen to provide empirical evidence to support this explanation, and have run additional experiments to that end. We evaluated GC-IQL on our 4 ExORL domains when trained on the RND-100k buffer. In lieu of a well-defined goal state for the locomotion tasks, we used the state in $D_{\\text{labelled}}$ with highest reward. We hypothesised that GC-IQL would perform similarly to our proposed methods on the goal-reaching tasks, and less well on the locomotion tasks. We also hypothesised that, because GC-IQL employs conservatism like VC-FB and MC-FB, it would handle the low-quality datasets better than vanilla FB and outperform it in aggregate, despite the naive goal definition for locomotion tasks. We report the aggregated performance of GC-IQL averaged across 5 seeds, compared with zero-shot RL methods based on FB in the table below.\n\n| Domain | Task | Task Type | FB | GC-IQL | VC-FB | MC-FB |\n | --- | --- | --- | --- | --- | --- | --- |\n | Walker | All Tasks | Locomotion | 266 (233–283)  | 218 (164 - 251) |  **396 (381–407)** | 252 (188–288) |\n  | Point-mass Maze | All Tasks | Goal-reaching | 102 (0–181) | **332 (190-436)** | 323 (177–412) | 270 (154–459) |\n | Quadruped | All Tasks | Locomotion | 93 (69–137) | 85 (61-129) | **168 (104–201)** |  104 (38–212) |\n | Jaco | All Tasks | Goal-reaching | 4 (1–6) | 15 (6-25) | 7 (3–12) | **17 (7–26)** |\n | All Domains | All Tasks | - | 97 | 170 | **245** | 178 |\n\nWe find that GC-IQL performs similarly to the VC-FB and MC-FB on the goal-reaching tasks as predicted. On the locomotion tasks it performs worse than all zero-shot methods (irrespective of whether they are conservative or not). This is presumably because the state with highest reward in $D_{\\text{labelled}}$ is a poor proxy for the true, dense reward function.\n\nThank you for this feedback, it has made us realise that we should clarify the difference between GCRL and zero-shot RL in the paper. We will add these results (and associated results for DIAYN and Random datasets), an explanation of the differences between GCRL and zero-shot RL, and the associated literature you cite in the revised manuscript.\n\n**Q4: Can this method also be applied to approaches based on successor features?**\n\nYes, our value-conservative proposals are fully compatible with successor features. We mention this in line 99, and provide a derivation of value conservative successor features in Appendix D.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FL1LfxWaU3", "reviewer": "Reviewer_ZxeR", "summary": "The work investigates methods for zero-shot reinforcement learning (RL) that can be trained on small, homogeneous datasets. This research is driven by the need to make zero-shot RL practical when large, heterogeneous datasets are unavailable. The authors identify the limitations of existing methods that overestimate out-of-distribution (OOD) state-action values when trained on low-quality datasets. They propose incorporating conservatism into zero-shot RL algorithms to mitigate these issues. Their experimental results demonstrate that conservative zero-shot RL methods outperform their non-conservative counterparts on low-quality datasets while maintaining competitive performance on high-quality datasets.", "questions": "See in weakness.", "limitations": "See in weakness.", "rating": 3, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* Addressing Significant Issue: The paper attempts to address a significant gap in zero-shot RL by focusing on the challenges of using small, homogeneous datasets, which are more common in real-world applications.\n\n* Improved Performance: The proposed conservative zero-shot RL methods consistently outperform non-conservative counterparts on \nlow-quality datasets and do not degrade performance on high-quality datasets.", "weaknesses": "1. **The problem definition**: I do not think the settings of this work is so-called Zero-shot RL since the work introduces another dataset. And the code link is invalid.\n2. **Complexity of Implementation**: The introduction of conservatism increases the complexity of the algorithms, which may pose implementation challenges for practitioners.\n3. **Limited Real-World Validation**: While the experiments are thorough, there is limited validation of the proposed methods in real-world settings, which could affect their applicability.\n4. **Comparative Analysis**: The paper could benefit from a more in-depth comparative analysis with other state-of-the-art methods to highlight specific strengths and weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Hi Reviewer ZxeR. See our responses to your comments below. We hope we can work with you to address any misunderstandings. Thank you in advance for your cooperation.\n\n**W1: The problem definition.** **I do not think the settings of this work is so-called Zero-shot RL since the work introduces another dataset.** \n\nWe believe there is a critical misunderstanding in this question which we hope to clarify. We are very much performing zero-shot RL in this work. Zero-shot RL is about using a dataset of transitions collected from an environment to pre-train an agent such that it can return a performant policy for any downstream task in that environment. In our work, we are only changing the quality of this pre-training dataset. We are not introducing new datasets at test-time, and we are not fine-tuning agents on new data. We lay out our problem setting in lines 49-55, and this exactly matches the original zero-shot RL problem as proposed in the canonical work [1].\n\n**W2: Complexity of Implementation. The introduction of conservatism increases the complexity of the algorithms, which may pose implementation challenges for practitioners.**\n\nWe disagree that our proposals would pose implementation challenges for practitioners. In lines 147-148 we stated: *“We emphasise these additions represent only a small increase in the number of lines required to implement existing methods.”* We provide code snippets in Appendix G where the reviewer can see that are proposals constitute a few 10s of extra lines of code on top of the vanilla FB implementation.\n\n**W3: Limited Real-World Validation.** **While the experiments are thorough, there is limited validation of the proposed methods in real-world settings, which could affect their applicability.**\n\nWhilst we agree that evaluating methods on simulated benchmarks raises questions about the real-world applicability of any method, we specifically chose D4RL as one of our testbeds because it is designed to mirror the difficulties/practicalities of real-world deployment. The D4RL paper describes the benchmark as being *“guided by key properties of datasets relevant to real-world applications of offline RL. (cf. abstract)”* And that *“the choice of tasks and datasets [are] motivated by properties reflected in real-world applications, such as narrow data distributions and undirected, logged behavior. (cf. section 7)”*.\n\nBecause of these characteristics, D4RL has become the standard benchmark for evaluating (as closely as is feasible) the real-world applicability of offline RL algorithms, and our work is continuation of that tradition.\n\n**W4: Comparative Analysis.** **The paper could benefit from a more in-depth comparative analysis with other state-of-the-art methods to highlight specific strengths and weaknesses.**\n\nAs discussed in section 4.2, we use both existing state of the art zero-shot RL methods (successor features with laplacian eigenfunctions and FB representations) as baselines. We also compare with the performance of TD3, the best performing single-task RL method on the ExORL benchmark, and CQL, the single-task equivalent of our method. To the best of our knowledge there are no other zero-shot RL methods for us to compare against.\n\nOur comparisons span 5 baseline algorithms, 11 datasets, 6 domains, 19 tasks, and 3/5 seeds for a total of 1086 individual runs. We believe Section 4 (Results) provides evidence of the strengths of our methods w.r.t to the existing state-of-the-art. We explore the weaknesses of our methods thoroughly with additional experiments in Section 5 (Limitations).    \n\n**Code**\n\nFinally, we’re not sure what happened with our code link, it seems to have expired unexpectedly since we submitted this work. We’ve shared a new link with the AC (as per the guidelines) that they should share with you so you can review the code.\n\n**References**\n\n[1]  Ahmed Touati, Jérémy Rapin, and Yann Ollivier. Does zero-shot reinforcement learning exist? ICLR 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper addresses the challenges of zero-shot RL with low-quality data by proposing conservative algorithms that regularize value functions or dynamics predictions for out-of-distribution state-action pairs.

The paper was reviewed by four reviewers with three inclined to suggest for acceptance and one not. The majority of reviewers agree on the strengths, noting that (1) the proposed value conservative method is well motivated, (2) the paper is clearly written, and the approach is sound, and (3) the experiments are thorough and clear. Several weaknesses and questions were discussed during the rebuttal period, and I have not identified any major unresolved issues. 

Considering these strengths, this paper is recommended for acceptance. 
The authors are encouraged to utilize their constructive and detailed discussion with reviewers towards further raising the quality and impact of their work.
Decision: Accept (poster)