Zero-Shot Reinforcement Learning from Low Quality Data

Authors: scott  jeen
tom  bewley
jonathan m cullen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 12:20:18

Abstract: Zero-shot reinforcement learning (RL) promises to provide agents that can perform _any_ task in an environment after an offline, reward-free pre-training phase. Methods leveraging successor measures and successor features have shown strong performance in this setting, but require access to large heterogenous datasets for pre-training which cannot be expected for most real problems. Here, we explore how the performance of zero-shot RL methods degrades when trained on small homogeneous datasets, and propose fixes inspired by _conservatism_, a well-established feature of performant single-task offline RL algorithms. We evaluate our proposals across various datasets, domains and tasks, and show that conservative zero-shot RL algorithms outperform their non-conservative counterparts on low quality datasets, and perform no worse on high quality datasets. Somewhat surprisingly, our proposals also outperform baselines that get to see the task during training. Our code is available via the project page https://enjeeneer.io/projects/zero-shot-rl/.

Main paper: 
 introduction: 
 Today's large pre-trained models generalise impressively to unseen vision [70] and language [9] tasks, but not to sequential decision-making problems. Zero-shot reinforcement learning (RL) attempts to correct this, asking, informally: can we pre-train an agent on a dataset of reward-free transitions such that it can perform any downstream task in an environment? Recently, methods leveraging successor features [5, 7] and successor measures [6, 82] have emerged as viable zero-shot RL candidates, returning near-optimal policies for many unseen tasks [83] .These works have assumed access to a large heterogeneous dataset of transitions for pre-training. In theory, such datasets could be curated by highly-exploratory agents during an upfront data collection phase [32, 10, 18, 62, 63, 35, 48] . However, in practice, deploying such agents in real systems can be time-consuming, costly or dangerous. To avoid these downsides, it would be convenient to skip the data collection phase and pre-train on historical datasets. Whilst these are common in the real world, they are usually produced by controllers that are not optimising for data heterogeneity [16] , making them smaller and less diverse than current zero-shot RL methods expect.Can we still perform zero-shot RL using these datasets? This is the primary question this paper seeks to answer, and one we address in four parts. First, we investigate the performance of existing methods when trained on such datasets, finding their performance suffers because of out-of-distribution state-action value overestimation, a well-observed phenomenon in single-task offline RL. Second, we develop ideas from conservatism in single-task offline RL for use in the zero-shot RL setting, introducing a straightforward regularizer of OOD values or measures that can be used by any zero-shot RL algorithm (Figure 1 ). Third, we conduct experiments across varied domains, tasks and datasets, showing our conservative zero-shot RL proposals outperform their non-conservative counterparts, and surpass the performance of methods that get to see the task in advance. Finally, we establish that our proposals do not hinder performance on large heterogeneous datasets, meaning adopting them presents little downside. We believe the ideas explored in this paper represent a step toward real-world deployment of zero-shot RL methods. 
preliminaries: 
 Markov decision processes A reward-free Markov Decision Process (MDP) is defined by M = {S, A, T , γ} where S is the state space, A is the action space, T : S × A → ∆(S) is the transition function, where ∆(X) denotes the set of possible distributions over X, and γ ∈ [0, 1) is the discount factor [77] . Given (s 0 , a 0 ) ∈ S × A and a policy π : S → ∆(A), we denote Pr(•|s 0 , a 0 , π) and E[•|s 0 , a 0 , π] the probabilities and expectations under state-action sequences (s t , a t ) t≥0 starting at (s 0 , a 0 ) and following policy π, with s t ∼ T (•|s t-1 , a t-1 ) and a t ∼ π(•|s t ). Given a reward function r : S → R ≥0 , the Q function of π for r is Q π r := t≥0 γ t E[r(s t+1 )|s 0 , a 0 , π]. Zero-shot RL For pre-training, the agent has access to a static offline dataset of reward-free transitions D = {(s i , a i , s i+1 )} |D| i=1 generated by an unknown behaviour policy, and cannot interact with the environment. At test time, a reward function r eval specifying a task is revealed and the agent must return a policy for the task without any further planning or learning. Ideally, the policy should maximise the expected discounted return on the task E[ t≥0 γ t r eval (s t+1 )|s 0 , a 0 , π]. The reward function is specified either via a small dataset of reward-labelled states D labelled = {(s i , r eval (s i ))} k i=1 with k ≤ 10, 000 or as an explicit function s → r eval (s) (like 1 at a goal state and 0 elsewhere). Intuitively, the zero-shot RL problem asks: is it possible to train an agent using a pre-collected dataset of transitions from an environment such that, at test time, it can return the optimal policy for any task in that environment without any further planning or learning? State-of-the-art zero-shot RL methods leverage either successor measures [6] or successor features [5] , with the former instantiated by forward backward representations [82] and the latter by universal successor features [7] . The remainder of this section introduces these ideas.The successor measure M π (s 0 , a 0 , •) over S is the cumulative discounted time spent in each future state s t+1 after starting in state s 0 , taking action a 0 , and following policy π thereafter:M π (s 0 , a 0 , X) := t≥0 γ t Pr(s t+1 ∈ X|s 0 , a 0 , π) ∀ X ⊂ S.The Q function of policy π for task r is the integral of r with respect to M π : Q π r (s 0 , a 0 ) := s+∈S r(s + )M π (s 0 , a 0 , s + ).(EQUATIONThe forward-backward framework FB representations [82] approximate the successor measures of near-optimal policies for any task. Let ρ be an arbitrary state distribution, and R d be a representation space. FB representations are composed of a forward model F : S × A × R d → R d , a backward model B : S → R d , and set of polices (π z ) z∈R d . They are trained such that M πz (s 0 , a 0 , X) ≈ X F (s 0 , a 0 , z) ⊤ B(s)ρ(ds) ∀ s 0 ∈ S, a 0 ∈ A, X ⊂ S, z ∈ R d ,andEQUATIONIntuitively, Equation 3 says that the approximated successor measure under π z from (s 0 , a 0 ) to s is high if their respective forward and backward embeddings are similar i.e. have large dot product. By comparing Equation 2 and Equation 3, we see that an FB representation can be used to approximate the Q function of π z with respect to any reward function r as: Q πz r (s 0 , a 0 ) ≈ s∈S r(s)F (s 0 , a 0 , z) ⊤ B(s)ρ(ds) = F (s 0 , a 0 , z)EQUATIONTraining of F and B is done with TD learning [71, 78] using transition data sampled from D:L FB = E (st,at,st+1,s+)∼D,z∼Z [(F (s t , a t , z) ⊤ B(s + ) -γ F (s t+1 , π z (s t+1 ), z) ⊤ B(s + )) 2 -2F (s t , a t , z) ⊤ B(s t+1 )], (6) where s + is sampled independently of (s t , a t , s t+1 ), F and B are lagging target networks, and Z is a task sampling distribution. The policy is trained in an actor-critic formulation [47] . See [82] for a full derivation of the TD update, and our Appendix B.1 for practical implementation details including the specific choice of task sampling distribution Z.By relating Equations 4 and 5, we find z = E s∼ρ [r(s)B(s)] for some reward function r. At test time, we can use this property to perform zero-shot RL. Using D labelled , we estimate the task as z eval ≈ E s∼Dlabelled [r eval (s)B(s)] and pass it as an argument to π z . If z eval lies within the task sampling distribution Z used during pre-training, then π z (s) ≈ arg max a Q πz reval (s, a), and hence this policy is approximately optimal for r eval .(Universal) successor features Successor features assume access to a basic feature map φ : S → R d that embeds states into a representation space, and are defined as the expected discounted sum of future features ψ π (s 0 , a 0 ) := E[ t≥0 γ t φ(s t+1 )|s 0 , a 0 , π] [5] . They are made universal by conditioning their predictions on a family of policies π zEQUATIONwithEQUATIONLike FB, USFs are trained using TD learning onEQUATIONwhere ψ is a lagging target network, and Z is the same z sampling distribution used for FB. We refer the reader to [7] for a derivation of the TD update and full learning procedure. Test time policy inference is performed similarly to FB. Using D labelled , the task is inferred by performing a linear regression of r eval onto the features:z eval := arg min z E s∼Dlabelled [(r eval (s) -φ(s) ⊤ z) 2] before it is passed as an argument to the policy.3 Zero-Shot RL from Low Quality DataIn this section we introduce methods for improving the performance of zero-shot RL methods on low quality datasets. In Section 3.1, we explore the failure mode of existing methods on such datasets. Then, in Section 3.2, we propose straightforward amendments to these methods that address the failure mode. Finally, in Section 3.3, we illustrate the usefulness of our proposals with a controlled example. We develop our methods within the FB framework because of its superior empirical performance [83] , but our proposals are also compatible with USF. We push their derivation to Appendix D for brevity.Figure 2 : FB value overestimation with respect to dataset size n and quality. Log Q values and IQM of rollout performance on all Maze tasks for datasets RND and RANDOM. Q values predicted during training increase as both the size and "quality" of the dataset decrease. This contradicts the low return of all resultant policies (note: a return of 1000 is the maximum achievable for this task). Informally, we say the RND dataset is "high" quality, and the RANDOM dataset is "low" quality-see Appendix A.2 for more details. 
failure mode of existing methods: 
 To investigate the failure mode of existing methods we examine the FB loss (Equation 6) more closely. The TD target includes an action produced by the current policy a t+1 ∼ π z (s t+1 ). Equation 4shows this is the policy's current best estimate of the Q-maximising action in state s for task z.For finite datasets, this maximisation does not constrain the policy to actions observed in the dataset, and so it can become biased towards out-of-distribution (OOD) actions thought to be of high value.In such instances F and B are updated towards targets for which the dataset provides no support. This distribution shift is a well-observed phenomenon in single-task offline RL [42, 46, 44] , and is exacerbated by small, low-diversity datasets as we explore in Figure 2 .In the single-task setting, the distribution shift is addressed by applying constraints to either the policy, value function or model (see Section 6 for a summary of past work). Here we re-purpose single-task value function and model regularisation for use in the zero-shot RL setting. To avoid further complicating zero-shot RL methods, we only consider regularisation techniques that do not introduce new parametric functions. We discuss the implications of this decision in Section 5.Conservative Q-learning (CQL) [42, 44] regularises the Q function by querying OOD state-action pairs and suppressing their value. This is achieved by adding new term to the usual Q loss functionEQUATION)where α is a scaling parameter, µ(a|s) is a policy distribution selected to find the maximum value of the current Q function iterate, H(µ) is the entropy of µ used for regularisation, and L Q is the normal TD loss on Q. Equation 10 has the dual effect of minimising the peaks in Q under µ whilst maximising Q for state-action pairs in the dataset.We can replicate a similar form of regularisation in the FB framework, substituting F (s, a, z) ⊤ z for Q in Equation 10 and adding the normal FB loss (Equation 6)EQUATIONThe key difference between Equations 10 and 11 is that the former suppresses the value of OOD actions for one task, whereas the latter does so for all task vectors drawn from Z. We call models learnt with this loss value-conservative forward-backward representations (VC-FB).Because FB derives Q functions from successor measures (Equation 5), and because (by assumption) rewards are non-negative, suppressing the predicted measures for OOD actions provides an alternative route to suppressing their Q values. As we did with VC-FB, we can substitute FB's successor measure approximation F (s, a, z) ⊤ B(s + ) into Equation 10, which yields: L MC-FB = α • (E s∼D,a∼µ(a|s),z∼Z,s+∼D [F (s, a, z) ⊤ B(s + )] -E (s,a)∼D,z∼Z,s+∼D [F (s, a, z) ⊤ B(s + )] -H(µ)) + L FB . (12)Equation 12 has the effect of suppressing the expected visitation count to goal state s + when taking an OOD action for all task vectors drawn from Z, which says, informally, if we don't know where OOD actions take us in the MDP, we assume they have low probability of taking us to any future states for all tasks. This is analogous to works that regularise model predictions in the single-task offline RL setting [37, 96, 69] . As such, we call this variant a measure-conservative forward-backward representation (MC-FB). Since it is not obvious a priori whether the VC-FB or MC-FB form of conservatism would be more effective in practice, we evaluate both in Section 4.Implementing these proposals requires two new model components: 1) a conservative penalty scaling factor α and 2) a way of obtaining policy distribution µ(a|s) that maximises the current value or measure iterate. For 1), we observe fixed values of α leading to fragile performance, so dynamically tune it at each learning-see Appendix B.1.4. For 2), the choice of maximum entropy regularisation following [44] 's CQL(H) allows µ to be approximated conveniently with a log-sum exponential across Q values derived from the current policy distribution and a uniform distribution. That this is true is not obvious, so we refer the reader to the detail and derivations in Section 3.2, Appendix A, and Appendix E of [44] , as well as our adjustments to [44] 's theory in Appendix B.1.3. Code snippets demonstrating the required changes to a vanilla FB implementation are provided in Appendix G. We emphasise these additions represent only a small increase in the number of lines required to implement existing methods.To understand situations in which a conservative zero-shot RL methods may be useful, we introduce a modified version of Maze from the ExORL benchmark [95] . Episodes begin with a point-mass initialised in the upper left of the maze (⊚), and the agent is tasked with selecting x and y tilt directions such that the mass is moved towards one of two goal locations (⊛ and ⊛). The action space is two-dimensional and bounded in [-1, 1]. We take the RND dataset and remove all "left" actions such that a x ∈ [0, 1] and a y ∈ [-1, 1], creating a dataset that has the necessary information for solving the tasks, but is inexhaustive (Figure 3 (a) ). We train FB and VC-FB on this dataset and plot the highest-reward trajectories-Figure 3 (b) and (c). FB overestimates the value of OOD actions and cannot complete either task. Conversely, VC-FB synthesises the requisite information from the dataset and completes both tasks.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Justification: This paper does not include theoretical proofs. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experiments: 
 In this section we perform an empirical study to evaluate our proposals. We seek answers to four questions: (Q1) Can our proposals from Section 3 improve FB performance on small and/or lowquality exploratory datasets? (Q2) How does the performance of VC-FB and MC-FB vary with respect to task type and dataset diversity? (Q3) Do we sacrifice performance on full datasets for performance on small and/or low-quality datasets? (Q4) If our pre-training dataset only covers behaviour related to one downstream task (i.e. the dataset distribution is narrow and not exploratory), can our proposals from Section 3 improve FB performance on that task?Domains We respond to Q1-Q3 using the ExORL benchmark [95] . ExORL provides datasets collected by unsupervised exploratory algorithms on the DeepMind Control Suite [80] . We select three of the same domains as [83] : Walker, Quadruped and Maze, but substitute Jaco for Cheetah. This provides two locomotion domains and two goal-reaching domains. Within each domain, we evaluate on all tasks provided by the DeepMind Control Suite for a total of 17 tasks across four domains. Full details are provided in Appendix A.1. We respond to Q4 using the D4RL benchmark [21] . We select the two MuJoCo [81] environments from the Open AI gym [8] that closest resemble those from ExORL: Walker2D and HalfCheetah.Datasets For Q1-Q3 we pre-train on three datasets of varying quality from ExORL. There is no unambiguous metric for quantifying dataset quality, so we use the reported performance of offline TD3 on Maze for each dataset as a proxy 1 . We choose datasets collected via Random Network Distillation (RND) [10] , Diversity is All You Need (DIAYN) [18] , and RANDOM policies, where agents trained on RND are the most performant, on DIAYN are median performers, and on RANDOM are the least performant. As well as selecting for quality, we also select for size by uniformly subsampling 100,000 transitions from each dataset. For Q4 we choose the "medium", "medium-replay", and "medium-expert" datasets from D4RL, each providing different fractions of random, medium and expert task-directed trajectories. More details on the datasets are provided in Appendix A.2.We compare our proposals to baselines from three categories: 1) zero-shot RL methods, 2) goalconditioned RL (GCRL) methods, and 3) single-task offline RL methods. From category 1), we use the state-of-the-art successor measure based method, FB, and the state-of-the-art successor feature based method, SF with features from Laplacian eigenfunctions (SF-LAP) [83] . From category 2), we use goal-conditioned IQL (GC-IQL) [60] , a state-of-the-art GCRL method that, like our proposals, regularises the value function at OOD state-actions. We condition GC-IQL on the goal state on Maze and Jaco, and on the state in D labelled with highest reward on Walker and Quadruped in lieu of a well-defined goal state. From category 3), we use CQL and offline TD3 trained on the same datasets relabelled with task rewards. CQL approximates what an algorithm with similar mechanistics can achieve when optimising for one task in a domain rather than all tasks. Offline TD3 exhibits the best aggregate single-task performance on the ExORL benchmark, so it should be indicative of the maximum performance we could expect to extract from a dataset. Full implementation details for all algorithms are provided in Appendix B. The full evaluation protocol is described in Appendix A.5. Appendix A.6 provides a breakdown of the computational resources used in this work.We report the aggregate performance of our baselines and proposals on ExORL in Figure 4 . Both MC-FB and VC-FB outperform the zero-shot RL and GCRL baselines, achieving 150% and 137% of FB's IQM performance respectively. The performance gap between FB and SF-LAP is consistent with the results in [83] . MC-FB and VC-FB outperform our single-task baseline in expectation, reaching 111% and 120% of CQL's IQM performance respectively despite not having access to taskspecific reward labels and needing to fit policies for all tasks. This is a surprising result, and to the best of our knowledge, the first time a multitask offline agent has been shown to outperform a single-task analogue. CQL outperforms offline TD3 in aggregate, so we drop offline TD3 from the core analysis, but report its full results in Appendix C alongside all other methods. We note FB achieves 80% of single-task offline TD3, which roughly aligns with the 85% performance on the full datasets reported by [83] .Q2 We decompose the methods' performance with respect to domain and dataset diversity in Figure 5 . The largest gap in performance between the conservative FB variants and FB is on RND. VC-FB and MC-FB reach 2.5× and 1.8× of FB performance respectively, and outperform CQL on three of the four domains. On DIAYN, the conservative variants outperform all methods and reach 1.3× CQL's score. On the RANDOM dataset, all methods perform similarly poorly, except for CQL on Jaco, which outperforms all methods. However, in general, these results suggest the RANDOM dataset is not informative enough to extract valuable policies-discussed further in response to Q3. There appears to be little correlation between the type of domain (Appendix A.1) and the score achieved by any method. GC-IQL performs particularly well on the goal-reaching domains as expected, but worse than all zero-shot methods on the locomotion tasks, irrespective of whether they are conservative or not. This is presumably because the goal-state used to condition the policy (i.e. the state with highest reward in D labelled ) is a poor proxy for the true, dense reward function. Q3 We report the aggregated performance of all FB methods across domains when trained on the full datasets in Table 1 (a full breakdown of results in provided in Appendix C). Both conservative FB variants slightly exceed the performance of vanilla FB in expectation. The largest relative performance improvement is on the RANDOM dataset-MC-FB performance is 20% higher than FB, compared to 5% higher on DIAYN and 2% higher on RND. This corroborates the hypothesis that RANDOM-100K was not informative enough to extract valuable policies. Aggregate IQM scores across all domains and datasets, normalised against the performance of CQL.Table 1 and Figure 4 suggest the performance gap between the conservative FB variants and vanilla FB changes as dataset size is varied. We further explore this effect in Figure 6 where we scale the RND dataset size from 10 5 through 10 7 and plot aggregate IQM performance of FB, VC-FB and MC-FB across all domains. We find that the performance gap decreases as dataset size increases. This result is to be expected: a larger dataset size for a fixed exploration algorithm means a t+1 ∼ π z (s t+1 ) in the FB TD update (Equation 6) is more likely to be in the dataset, the policy is less likely to become biased toward OOD actions, and conservatism is less needed.Q4 We report the aggregate performance of all zero-shot RL methods and CQL on our D4RL domains in Figure 7 . FB fails all domain-dataset tasks, and reaches only 10% of CQL's aggregate performance. MC-FB and VC-FB improve on FB's considerably (by 5.6 × and 6.8 × respectively) but under-perform CQL. SF-LAP outperforms FB, but under-performs VC-FB, MC-FB and CQL.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: Code and hyperparameters are provided; datasets and environments are open-source. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
discussion and limitations: 
 Performance discrepancy between conservative variants Why does VC-FB outperform MC-FB on both ExORL and D4RL? To understand, we inspect the regularising effect of both models more closely. VC-FB regularises OOD actions on F (s, a, z) ⊤ z, with s ∼ D, and z ∼ Z, whilst MC-FB regularises OOD actions on F (s, a, z) ⊤ B(s + ), with (s, s + ) ∼ D and z ∼ Z. Note the trailing z in VC-FB is replaced with B(s + ) in MC-FB which ties its updates to D further. We hypothesised that as |D| reduces, B(s + ) provides poorer task coverage than z ∼ Z, hence the comparable performance on full datasets and divergent performance on 100k datasets.To test this, we evaluate a third conservative variant called directed (D)VC-FB which replaces all z ∼ Z in VC-FB with B(s + ) such that OOD actions are regularised on F (s, a, B(s + )) ⊤ B(s + ) with (s, s + ) ∼ D. This ties conservative updates entirely to D, and according to our above hypothesis, DVC-FB should perform worse than VC-FB and MC-FB on the 100k ExORL datasets. See Appendix B.1.6 for implementation details. We evaluate this variant on all 100k ExORL datasets, domains and tasks and compare with FB, VC-FB and MC-FB in Table 2 . See Appendix C for a full breakdown.We find the aggregate relative performance of each method is as expected i.e. DVC-FB < MC-FB < VC-FB. As a consequence we conclude that VC-FB should be preferred for small datasets with no prior knowledge of the dataset or test tasks. Of course, for a specific domain-dataset pair, B(s + ) with s + ∼ D may happen to cover the tasks well, and MC-FB may outperform VC-FB. We suspect this was the case for all datasets on the Jaco domain for example. Establishing whether this will be true a priori requires either relaxing the restrictions imposed by the zero-shot RL setting, or better understanding of the distribution of tasks in z-space and their relationship to pre-training datasets. The latter is important future work.Avoiding new parametric functions State-of-the-art zero-shot RL methods are complex, and we wanted to avoid further complicating them with new parametric functions. This limited our solutionspace to CQL-style regularisation techniques, but had we relaxed this constraint, other options become available. Methods like AWAC [58] , IQL [40] , and X-QL [25] all require an estimate of the state-value function which is not immediately accessible in the FB or USF frameworks. In theory, we could learn an action-independent USF of the formV (s, z) = E[ t≥0 γ t φ(s t+1 )|s 0 , π z ] ∀ s 0 ∈ S, z ∈ R d concurrently toF and B (or ψ for USFs). If learnt with expectile regression, this function could be used to implement IQL and X -QL style regularisation; without expectile regression it could be used to compute the advantage weighting required for AWAC. It's possible that implementing these methods could improve downstream performance and reduce computational overhead at the cost of increased training complexity. We leave this worthwhile investigation for future work. We provide detail of negative results related to downstream finetuning of FB models in Appendix E to help inform future research.D4RL Performance Unlike the ExORL results, VC-FB and MC-FB do not outperform CQL on the D4RL benchmark. We believe these narrower data distributions require a more careful selection of the conservative penalty scaling factor α. We explore this further in Appendix F, and note this is corroborated by findings in the original CQL paper [44] . Methods described above, like IQL, have been shown to be more robust than CQL partly because they bypass α tuning. We expect that exploring the integration of these methods may improve D4RL performance. 
related work: 
 Zero-shot RL Zero-shot RL methods build upon successor representations [15] , universal value function approximators [74] , successor features [5] and successor measures [6] . The state-of-the-art methods instantiate these ideas as either universal successor features (USFs) [7] or forward-backward (FB) representations [82, 83] , with recent work showing the latter can be used to perform a range of imitation learning techniques efficiently [65] . A representation learning method is required to learn the features for USFs, with past works using inverse curiosity modules [62] , diversity methods [49, 29] , Laplacian eigenfunctions [87] , or contrastive learning [13] . No works have yet explored the issues arising when training these methods on low quality offline datasets, and only one has investigated applying these ideas to real-world problems [34] .Goal-conditioned RL methods train policies to reach any goal state from any other state, and so can be used to perform zero-shot RL in goal-reaching environments [60, 54, 93, 19, 85] . However, they have no principled mechanism for conditioning policies on "dense" reward functions (as such tasks are not solved by simply reaching a particular state), and so are not full zero-shot RL methods.A concurrent line of work trains policies using sequence models conditioned on reward-labelled histories [12, 33, 45, 68, 99, 11, 24, 76, 91, 90] , but, unlike zero-shot RL methods, these works do not have a robust mechanism for generalising to different reward functions as test-time.Offline RL Offline RL algorithms require regularisation of policies, value functions, models, or a combination to manage the offline-to-online distribution shift [46] . Past works regularise policies with explicit constraints [88, 20, 23, 22, 27, 64, 43, 86, 94] , via important sampling [66, 79, 50, 57, 26] , by leveraging uncertainty in predictions [89, 98, 4, 36] , or by minimising OOD action queries [84, 14, 40] , a form of imitation learning [72, 73] . Other works constrain value function approximation so OOD action values are not overestimated [44, 42, 52, 53, 51, 92] . Offline model-based RL methods use the model to identify OOD states and penalise predicted rollouts passing through them [97, 37, 96, 2, 55, 67, 69] . All of these works have focused on regularising a finite number of policies; in contrast we extend this line of work to the zero-shot RL setting which is concerned with learning an infinite family of policies. 
conclusion: 
 In this paper, we explored training agents to perform zero-shot reinforcement learning (RL) from low quality data. We established that the existing methods suffer in this regime because they overestimate the value of out-of-distribution state-action values, a well-observed pheneomena in single-task offline RL. As a resolution, we proposed a family of conservative zero-shot RL algorithms that regularise value functions or dynamics predictions on out-of-distribution state-action pairs. In experiments across various domains, tasks and datasets, we showed our proposals outperform their non-conservative counterparts in aggregate and sometimes surpass our task-specific baseline despite lacking access to reward labels a priori. In addition to improving performance when trained on sub-optimal datasets, we showed that performance on large, diverse datasets does not suffer as a consequence of our design decisions. Our proposals represent a step towards the use of zero-shot RL methods in the real world. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: This paper discusses methods for improving the performance of zero-shot RL methods when trained on low quality offline datasets. We do not expect this work to directly impact society positively or negatively.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification:Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: https://github.com/enjeeneer/zero-shot-rl.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification:Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A. We consider two locomotion and two goal-directed domains from the ExORL benchmark [95] which is built atop the DeepMind Control Suite [80] . Environments are visualised here: https: //www.youtube.com/watch?v=rAai4QzcYbs. The domains are summarised in Table 3 .Walker. A two-legged robot required to perform locomotion starting from bent-kneed position. The state and action spaces are 24 and 6-dimensional respectively, consisting of joint torques, velocities and positions. ExORL provides four tasks stand, walk, run and flip. The reward function for stand motivates straightened legs and an upright torse; walk and run are supersets of stand including reward for small and large degrees of forward velocity; and flip motivates angular velocity of the torso after standing. Rewards are dense.Quadruped We train on 100,000 transitions uniformly sampled from three datasets on the ExORL benchmark collected by different unsupervised agents: RANDOM, DIAYN, and RND. The state coverage on Maze is depicted in Figure 8 . Though harder to visualise, we found that state marginals on higherdimensional tasks (e.g. Walker) showed a similar diversity in state coverage.RND. An agent whose exploration is directed by the predicted error in its ensemble of dynamics models. Informally, we say RND datasets exhibit high state diversity.DIAYN. An agent that attempts to sequentially learn a set of skills. Informally, we say DIAYN datasets exhibit medium state diversity. We consider two MuJoCo [81] locomotion tasks from the D4RL benchmark [21] , which is built atop the v2 Open AI Gym [8] . The below environment descriptions are taken from [8] .Walker2D-v2. A two-dimensional two-legged figure that consist of seven main body parts -a single torso at the top (with the two legs splitting after the torso), two thighs in the middle below the torso, two legs in the bottom below the thighs, and two feet attached to the legs on which the entire body rests. The goal is to walk in the in the forward (right) direction by applying torques on the six hinges connecting the seven body parts.HalfCheetah-v2. A 2-dimensional robot consisting of 9 body parts and 8 joints connecting them (including two paws). The goal is to apply a torque on the joints to make the cheetah run forward (right) as fast as possible, with a positive reward allocated based on the distance moved forward and a negative reward allocated for moving backward.We consider three goal-directed datasets from D4RL, each providing a different proportion of expert trajectories. The below dataset descriptions are taken from [21] .Medium. Generated by training an SAC policy, early-stopping the training, and collecting 1M samples from this partially-trained policy.Medium-replay. Generated by recording all samples in the replay buffer observed during training until the policy reaches the "medium" level of performance.Medium-expert. Generated by mixing equal amounts of expert demonstrations and suboptimal data, either from a partially trained policy or by unrolling a uniform-at-random policy.We evaluate the cumulative reward (hereafter called score) achieved by VC-FB, MC-FB and our baselines on each task across five seeds. We report task scores as per the best practice recommendations of [1] . Concretely, we run each algorithm for 1 million learning steps, evaluating task scores at checkpoints of 20,000 steps. At each checkpoint, we perform 10 rollouts, record the score of each, and find the interquartile mean (IQM). We average across seeds at each checkpoint to create the learning curves reported in Appendix F. From each learning curve, we extract task scores from the learning step for which the all-task IQM is maximised across seeds. Results are reported with 95% confidence intervals obtained via stratified bootstrapping [17] . Aggregation across tasks, domains and datasets is always performed by evaluating the IQM. Full implementation details are provided in Appendix B.1.We train our models on NVIDIA A100 GPUs. Training a single-task offline RL method to solve one task on one GPU takes approximately 4 hours. FB and SF solve one domain (for all tasks) on one GPU in approximately 4 hours. Conservative FB variants solve one domain (for all tasks) on one GPU in approximately 12 hours. As a result, our core experiments on the 100k datasets used approximately 110 GPU days of compute.Here we detail implementations for all methods discussed in this paper. The code required to reproduce our experiments is available via https://github.com/enjeeneer/zero-shot-rl.B.1 Forward-Backward Representations B.1.1 ArchitectureThe forward-backward architecture described below follows the implementation by [83] exactly, other than the batch size which we reduce from 1024 to 512. We did this to reduce the computational expense of each run without limiting performance. The hyperparameter study in Appendix J of [83] shows this choice is unlikely to affect FB performance. All other hyperparameters are reported in Table 4 .Forward Representation F (s, a, z). The input to the forward representation F is always preprocessed. State-action pairs (s, a) and state-task pairs (s, z) have their own preprocessors which are feedforward MLPs that embed their inputs into a 512-dimensional space. These embeddings are concatenated and passed through a third feedforward MLP F which outputs a d-dimensional embedding vector. Note: the forward representation F is identical to ψ used by USF so their implementations are identical (see Table 4 ).Backward Representation B(s). The backward representation B is a feedforward MLP that takes a state as input and outputs a d-dimensional embedding vector.Actor π(s, z). Like the forward representation, the inputs to the policy network are similarly preprocessed. State-action pairs (s, a) and state-task pairs (s, z) have their own preprocessors which feedforward MLPs that embed their inputs into a 512-dimensional space. These embeddings are concatenated and passed through a third feedforward MLP which outputs a a-dimensional vector, where a is the action-space dimensionality. A Tanh activation is used on the last layer to normalise their scale. As per [23] 's recommendations, the policy is smoothed by adding Gaussian noise σ to the actions during training. Note the actors used by FB and USFs are identical (see Table 4 ).Misc. Layer normalisation [3] and Tanh activations are used in the first layer of all MLPs to standardise the inputs.FB representations require a method for sampling the task vector z at each learning step. [83] employ a mix of two methods, which we replicate: This also yields vectors on the hypersphere surface due to the L2 normalisation described above, but the distribution is non-uniform.We sample z 50:50 from these methods at each learning step.The conservative variants of FB require access to a policy distribution µ(a|s) that maximises the value of the current Q iterate in expectation. Recall the standard CQL loss EQUATION)where α is a scaling parameter, µ(a|s) the policy distribution we seek, R regularises µ and L Q represents the normal TD loss on Q. [44] 's most performant CQL variant (CQL(H)) utilises maximum entropy regularisation on µ i.e. R = H(µ). They show that obtaining µ can be cast as a closed-form optimisation problem of the form:EQUATIONand has optimal solution µ * (x) = 1 Z exp(f (x)), where Z is a normalising factor. Plugging Equation 14into Equation 13 we obtain:EQUATION)In discrete action spaces the logsumexp can be computed exactly; in continuous action spaces [44] approximate it via importance sampling using actions sampled uniformly at random, actions from the current policy conditioned on s t ∼ D, and from the current policy conditioned on s t+1 ∼ D2 :EQUATIONwith N a hyperparameter defining the number of actions to sample across the action-space. We can substitute F (s, a, z) ⊤ z for Q(s, a) in the final expression of Equation 17to obtain the equivalent for VC-FB:EQUATIONIn Appendix F, Figure 16 we show how the performance of VC-FB varies with the number of action samples. In general, performance improves with the number of action samples, but we limit N = 3 to limit computational burden. The formulation for MC-FB is identical other than each value F (s, a, z) T z being replaced with measures F (s, a, z) T B(s + ).A critical hyperparameter is α which weights the conservative penalty with respect to other losses during each update. We initially trialled constant values of α, but found performance to be fragile to this selection, and lacking robustness across environments. Instead, we follow [44] once again, and instantiate their algorithm for dynamically tuning α, which they call Lagrangian dual gradient-descent on α. We introduce a conservative budget parameterised by τ , and set α with respect to this budget:EQUATION)Intuitively, this implies that if the scale of overestimation ≤ τ then α is set close to 0, and the conservative penalty does not affect the updates. If the scale of overestimation ≥ τ then α is set proportionally to this gap, and thus the conservative penalty is proportional to the degree of overestimation above τ . As above, for the MC-FB variant values F (s, a, z) ⊤ z are replaced with measures F (s, a, z) ⊤ B(s + ).We summarise the end-to-end implementation of VC-FB as pseudo-code in Algorithm 1. MC-FB representations are trained identically other than at line 10 where the conservative penalty is computed for M instead of Q, and in line 12 where M s are lower bounded via Equation 12. Update F B given {(s i , a i , s i+1 , a i+1 , z i )} ◁ Equation 68: 9:// Conservative Update 10: 17)Q max (s i , a i ) ≈ log a exp F (s i , a i , z i ) ⊤ z i ◁ Compute conservative penalty (Equation11:Compute α given Q max via Lagrangian dual gradient-descent ◁ Equation 1812:Lower bound Q ◁ Equation 1113: 14:// Actor Update θ - F ← νθ - F + (1 -ν)θ F ◁ Forward target network 20: θ - B ← νθ - B + (1 -ν)θ B ◁ Backward target network 21: end for B.1.6 Directed Value-Conservative Forward Backward RepresentationsVC-FB applies conservative updates using task vectors z sampled from Z (which in practice is a uniform distribution over the √ d-hypersphere). This will include many vectors corresponding to tasks that are never evaluated in practice in downstream applications. Intuitively, it may seem reasonable to direct conservative updates to focus on tasks that are likely to be encountered downstream. One simple way of doing this would be consider the set of all goal-reaching tasks for goal states in the training distribution, which corresponds to sampling z = B(s g ) for some s g ∼ D. This leads to the following conservative loss function:L DVC-FB = α • E s∼D,a∼µ(a|s),sg∼D [F (s, a, B(s g )) ⊤ B(s g )] -E (s,a)∼D,sg∼D [F (s, a, B(s g )) ⊤ B(s g )] -H(µ) + L FB . (19)We call models learnt via this loss directed-VC-FB (DVC-FB). While we were initially open to the possibility that DVC-FB updates would be better targeted than those of VC-FB, and would lead to improved downstream task performance, this turns out not to be the case in our experimental settings as discussed in Section 5. We report scores obtained by the DVC-FB method across all 100k datasets, domains and tasks in Appendix C.We directly reimplement USFs, with basic features φ(s) provided by Laplacian eigenfunctions [87] , from [83] .USF ψ(s, a, z). The input to the USF ψ is always preprocessed. State-action pairs (s, a) and state-task pairs (s, z) have their own preprocessors which are feedforward MLPs that embed their inputs into a 512-dimensional space. These embeddings are concatenated and passed through a third feedforward MLP ψ which outputs a d-dimensional embedding vector. Note this is identical to the implementation of F as described in Appendix B.1. All other hyperparameters are reported in Table 4 .Feature Embedding φ(s). The feature map φ(s) is a feedforward MLP that takes a state as input and outputs a d-dimensional embedding vector. The loss function for learning the feature embedding is provided in Appendix B.2.2.Actor π(s, z). Like the forward representation, the inputs to the policy network are similarly preprocessed. State-action pairs (s, a) and state-task pairs (s, z) have their own preprocessors which are feedforward MLPs that embed their inputs into a 512-dimensional space. These embeddings are concatenated and passed through a third feedforward MLP which outputs a a-dimensional vector, where a is the action-space dimensionality. A Tanh activation is used on the last layer to normalise their scale. As per [23] 's recommendations, the policy is smoothed by adding Gaussian noise σ to the actions during training. Note this is identical to the implementation of π(s, z) as described in Appendix B.1.Misc. Layer normalisation [3] and Tanh activations are used in the first layer of all MLPs to standardise the inputs. z sampling distribution Z is identical to FB's (Appendix B.1.2).Laplacian eigenfunction features φ(s) are learned as per [87] . They consider the symmetrized MDP graph Laplacian created by some policy π, defined as L = Id -1 2 (P π diagρ -1 + diagρ -1(P π ) T ). They learn the eigenfunctions of L with the following:EQUATIONwhich comes from [39] .We implement GC-IQL following [60] 's codebase. GC-IQL inherits all functionality from a base soft actor-critic agent [28] , but adds a soft conservative penalty to the goal-conditioned critic's V (s, g) updates. We refer the reader to paper that introduces GC-IQL [60] for details on the loss function used to train V (s, g). Hyperparameters are reported in Table 5 .Critic(s). GC-IQL trains double goal-conditioned value functions V (s, g). The critics are feedforward MLPs that take a state-goal pair (s, g) as input and output a value ∈ R 1 . During training the goals are sampled from the prior G described in Section B.3.2.Actor. The actor is a standard feedforward MLP taking the state s as input and outputting an 2adimensional vector, where a is the action-space dimensionality. The actor predicts the mean and standard deviation of a Gaussian distribution for each action dimension; during training a value is sampled at random, during evaluation the mean is used.Following [60] , goals are sampled from either random states, future states, or the current state with probabilities 0.3, 0.5 and 0.2 respectively. A geometric distribution Geom(1 -γ) is used for the future state distribution, and the uniform distribution over the offline dataset is used for sampling random states.We adopt the same implementation and hyperparameters as is used on the ExORL benchmark. CQL inherits all functionality from a base soft actor-critic agent [28] , but adds a conservative penalty to the critic updates (Equation 10). Hyperparameters are reported in Table 5 .Critic(s). CQL employs double Q networks, where the target network is updated with Polyak averaging via a momentum coefficient. The critics are feedforward MLPs that take a state-action pair (s, a) as input and output a value ∈ R 1 .Actor. The actor is a standard feedforward MLP taking the state s as input and outputting an 2adimensional vector, where a is the action-space dimensionality. The actor predicts the mean and standard deviation of a Gaussian distribution for each action dimension; during training a value is sampled at random, during evaluation the mean is used.We adopt the same implementation and hyperparameters as is used on the ExORL benchmark. Hyperparameters are reported in Table 5 .Critic(s). TD3 employs double Q networks, where the target network is updated with Polyak averaging via a momentum coefficient. The critics are feedforward MLPs that take a state-action pair (s, a) as input and output a value ∈ R 1 .Actor. The actor is a standard feedforward MLP taking the state s as input and outputting an adimensional vector, where a is the action-space dimensionality. The policy is smoothed by adding Gaussian noise σ to the actions during training.Misc. As is usual with TD3, layer normalisation [3] is applied to the inputs of all networks. This work was enabled by: NumPy [30] , PyTorch [61] , Pandas [56] and Matplotlib [31] .In this section we report a full breakdown of our experimental results on the ExORL benchmark by dataset, domain and task. Table 6 reports results for methods trained on the 100k sub-sampled datasets, and Table 7 reports results for methods trained on the full datasets. and for continuous state and action spaces is trained in an actor critic formulation. Like FB, USF training requires next action samples a t+1 ∼ π z (s t+1 ) for the TD targets. We therefore expect SFs to suffer the same failure mode discussed in Section 3 (OOD state-action value overestimation) and to benefit from the same remedial measures (value conservatism). Training value-conservative successor features (VC-SF) amounts to substituting the USF Q function definition and loss for FB's in Equation 11:EQUATIONBoth the maximum value approximator µ(a|s) (Equation 17, Section B.1.3) and α-tuning (Equation 18, Section B.1.4) can be extracted identically to the FB case with any occurrence of F (s, a, z) ⊤ z substituted with ψ(s, a, z) ⊤ z. As USFs do not predict successor measures we cannot formulate measure-conservative USFs.In this section we provide detail on experiments we attempted, but which did not provide results significant enough to be included in the main body.If we relax the zero-shot requirement, could pre-trained conservative FB representations be finetuned on new tasks or domains? Base CQL models have been finetuned effectively on unseen tasks using both online and offline data [41] , and we had hoped to replicate similar results with VC-FB and MC-FB. We ran offline and online finetuning experiments and provide details on their setups and results below. All experiments were conducted on the Walker domain.Offline finetuning. We considered a setting where models are trained on a low quality dataset initially, before a high quality dataset becomes available downstream. We used models trained on the RANDOM-100k dataset and finetuned them on both the full RND and RND-100k datasets, with models trained from scratch used as our baseline. Finetuning involved the usual training protocol as described in Algorithm 1, but we limited the number of learning steps to 250k.We found that though performance improved during finetuning, it improved no quicker than the models trained from scratch. This held for both the full RND and RND-100k datasets. We conclude that the parameter initialisation delivered after training on a low quality dataset does not obviously expedite learning when high quality data becomes available.Online finetuning. We considered the online finetuning setup where a trained representation is deployed in the target environment, required to complete a specified task, and allowed to collect a replay buffer of reward-labelled online experience. We followed a standard online RL protocol where a batch of transitions was sampled from the online replay buffer after each environment step for use in updating the model's parameters. We experimented with fixing z to the target task during in the actor updates (Line 16, Algorithm 1), but found it caused a quick, irrecoverable collapse in actor performance. This suggested uniform samples from Z provide a form of regularisation. We granted the agents 500k steps of interaction for online finetuning.We found that performance never improved beyond the pre-trained (init) performance during finetuning. We speculated that this was similar to the well-documented failure mode of online finetuning of CQL [59] , namely taking sub-optimal actions in the real env, observing unexpectedly high reward, and updating their policy toward these sub-optimal actions. But we note that FB representations do not update w.r.t observed rewards, and so conclude this cannot be the failure mode. Instead it seems likely that FB algorithms cannot use the narrow, unexploratory experience obtained from attempting to perform a specific task to improve model performance.We believe resolving issues associated with finetuning conservative FB algorithms once the zero-shot requirement is relaxed is an important future direction and hope that details of our negative attempts to this end help facilitate future research. ExORL Learning Curves. We report the learning curves for all zero-shot RL methods and CQL in Figures 11, 12 , and 13. For all domains except Jaco, the y-axis limit is fixed at 1000 as that is the maximum score achievable in the DeepMind Control Suite. For Jaco-related figures, the y-axis limits is fixed at 100 as no method achieves a score higher than this.Hyperparameter Sensitivity. We report the sensitivity of VC-FB and MC-FB to the choice of two new hyperparameters: conservative budget τ and action samples per policy N on the ExORL benchmark. Figure 14 plots the sensitivity of VC-FB to the choice of τ on Walker and Maze domains across RND and RANDOM datasets. Figure 15 plots the sensitivity of MC-FB to the choice of τ on Walker and Maze domains across RND and RANDOM datasets. Figure 16 plots the sensitivity of MC-FB to the choice of N on Walker and Maze domains across RND and RANDOM datasets.We further explore the sensitivity of VC-FB performance on Walker2D from the D4RL benchmark w.r.t. the choice of conservative budget τ . Figure 17 plots this relationship when trained on the "medium-expert" dataset from D4RL. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.We note that[75] propose metrics that describe dataset quality as a function of the behaviour policy's exploration and exploitation w.r.t. one downstream task. However, since we are interested in generalising to any downstream task we cannot use these proposals directly, nor can we easily re-purpose them. We acknowledge that our proxy is imperfect, and that more work is required to better understand what dataset quality means in the context of zero-shot RL.Conditioning on next states st+1 ∼ D is not mentioned in the paper, but is present in their official implementation. 


Decision: This paper addresses the challenges of zero-shot RL with low-quality data by proposing conservative algorithms that regularize value functions or dynamics predictions for out-of-distribution state-action pairs.

The paper was reviewed by four reviewers with three inclined to suggest for acceptance and one not. The majority of reviewers agree on the strengths, noting that (1) the proposed value conservative method is well motivated, (2) the paper is clearly written, and the approach is sound, and (3) the experiments are thorough and clear. Several weaknesses and questions were discussed during the rebuttal period, and I have not identified any major unresolved issues. 

Considering these strengths, this paper is recommended for acceptance. 
The authors are encouraged to utilize their constructive and detailed discussion with reviewers towards further raising the quality and impact of their work.

Reviews: review: paper_summary: paper_summary: This work proposes modifications to Forward-Backward representations method. The modifications are aimed to improve the method's robustness to dataset quality. The vanilla FB suffers when the dataset does not cover the state space well, and overestimates the quality of actions. The authors show this with a simple example on point-mass. Two modifications are proposed, VC-FB and MC-FB, which take inspiration from CQL and make sure the FB representations are not too optimistic on the transitions not seen in the dataset. 
VC-FB and MC-FB are then thoroughly tested on ExORL and D4RL. The authors show that VC-FB offers the best improvement on ExORL, although none of the proposed modifications beat CQL on D4RL benchmark. Nevertheless, in all settings the modifications greatly improve over vanilla FB.
strength_weakness: strength_weakness: - The novelty is limited: the authors take CQL's method for correcting overly optimal predictions and apply them to FB.
questions: questions: -  How do the proposed modifications affect the performance of the model when the dataset is of good quality? Does it hurt the performance?
limitations: limitations: The authors have adequately described limitations and potential negative societal impact of their work.
paper_summary: paper_summary: This paper identifies an overestimation problem of zero-shot RL algorithms, particularly in low data or low data quality settings. To address this, they propose to use a value conservative method to mitigate the overestimation (from CQL). They showcase that this effectively allows for reducing the overestimation, then empirically show performance improvements on low or bad quality data regimes, without impacting performance on usual regimes.
strength_weakness: strength_weakness: **Provide a bit more background and intuition about zero-shot RL**. This is not really mandatory, but it would help a broader audience.

**Introduce the RANDOM and RND datasets**. Maybe an introduction of these datasets can help better grasp Figure 2 and 3, and would help the previous point.
questions: questions: * What is the intuition behind the overestimation reduction of CQL ?
limitations: limitations: There is a dedicated limitation section that correctly addresses the limitations of the method.
paper_summary: paper_summary: This work addresses zero-shot reinforcement learning (RL), focusing on training agents to perform tasks without explicit rewards during pre-training. The authors investigate the performance degradation that occurs with small, homogeneous datasets and propose conservative zero-shot RL algorithms inspired by single-task offline RL approaches. Experimental results demonstrate that the proposed methods improve performance on sub-optimal datasets without compromising effectiveness on large, diverse datasets.
strength_weakness: strength_weakness: * One weakness of the paper is the lack of a clear definition for "low-quality data." As shown in Figure 8, it might be more accurately described as "coverage" rather than quality. Providing a more explicit definition is recommended.

* Additionally, the comparison of baselines is incomplete, as it lacks an offline goal-conditioned baseline, which is highly relevant in the considered tasks. Including a representative offline GCRL baseline, such as GC-IQL, would strengthen the comparison.

* Some related works are missing, particularly in the offline GCRL domain. References [1][2][3][4] are pertinent as they also focus on learning from offline datasets without rewards and deploying for multiple goals. The didactic example in this paper is very similar to Figure 1 in a prior paper [3], which also convey similar information during zero-shot deployment for multiple goals. A discussion with the prior results would be beneficial. Furthermore, a very relevant offline zero-shot RL method [5] is neither cited nor discussed. 


*References*

[1] Yang R, Lu Y, Li W, et al. Rethinking goal-conditioned supervised learning and its connection to offline rl. ICLR, 2022.

[2] Park S, Ghosh D, Eysenbach B, et al. Hiql: Offline goal-conditioned rl with latent states as actions. Advances in Neural Information Processing Systems, 2024.

[3] Yang R, Yong L, Ma X, et al. What is essential for unseen goal generalization of offline goal-conditioned rl?[C]//International Conference on Machine Learning. PMLR, 2023.

[4] Eysenbach B, Zhang T, Levine S, et al. Contrastive learning as goal-conditioned reinforcement learning[J]. Advances in Neural Information Processing Systems, 2022.

[5] Frans K, Park S, Abbeel P, et al. Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodings[J]. arXiv preprint arXiv:2402.17135, 2024.
questions: questions: * Could the authors provide a more explicit definition of "low-quality data"?

* It is recommended to compare the proposed method with a representative offline GCRL baseline.

* Including related works discussed above would make this study more comprehensive.

* Can this method also be applied to approaches based on successor features?
limitations: limitations: One limitation is that VC-FB and MC-FB do not outperform CQL on the D4RL benchmark; addressing this issue is left for future work.
paper_summary: paper_summary: The work investigates methods for zero-shot reinforcement learning (RL) that can be trained on small, homogeneous datasets. This research is driven by the need to make zero-shot RL practical when large, heterogeneous datasets are unavailable. The authors identify the limitations of existing methods that overestimate out-of-distribution (OOD) state-action values when trained on low-quality datasets. They propose incorporating conservatism into zero-shot RL algorithms to mitigate these issues. Their experimental results demonstrate that conservative zero-shot RL methods outperform their non-conservative counterparts on low-quality datasets while maintaining competitive performance on high-quality datasets.
strength_weakness: strength_weakness: 1. **The problem definition**: I do not think the settings of this work is so-called Zero-shot RL since the work introduces another dataset. And the code link is invalid.
2. **Complexity of Implementation**: The introduction of conservatism increases the complexity of the algorithms, which may pose implementation challenges for practitioners.
3. **Limited Real-World Validation**: While the experiments are thorough, there is limited validation of the proposed methods in real-world settings, which could affect their applicability.
4. **Comparative Analysis**: The paper could benefit from a more in-depth comparative analysis with other state-of-the-art methods to highlight specific strengths and weaknesses.
questions: questions: See in weakness.
limitations: limitations: See in weakness.

Comments: title: Response by authors
Response from authors
Thank you very much for your clarifications
comment: Thanks for pointing us toward this work. We currently cite [2], and use their proxy for dataset quality (downstream performance on certain tasks) to rank our datasets. However, we were unaware of [1], and the metrics they propose are helpful, although not directly applicable to our setting as we’ll explain, but we’re keen to try to use these ideas.

In [1] they propose metrics for measuring the *exploitation (TQ)* and *exploration (SACo)* of the behaviour policy that creates the dataset. *TQ* is estimated as the mean reward of trajectories in the dataset w.r.t. a single downstream task of interest. Since we’re interested in generalising to any downstream task, in principle we’d need to calculate TQ for an infinite set of reward functions as specified by our task sampling distribution $\mathcal{Z}$ which is clearly intractable. As an alternative, we could calculate TQ w.r.t our test reward functions only, but our datasets are made up of state-action-next-state transition samples, not of full trajectories, so we can’t do this either. Consequently, TQ doesn’t quite fit our problem setting. However, we can measure *SACo;* the ratio of unique state-action pairs contained in a dataset w.r.t. some reference dataset. We used their codebase to do so w.r.t. an idealised dataset of 100k unique state-action transitions. See the metrics below, note we had to vary the number of discretisation bins to to get meaningful results across environments with different sizes of state and action spaces.

|  | Walker (5 bins) | Point-mass Maze (25 bins) | Quadruped (3 bins) | Jaco (4 bins) |
| --- | --- | --- | --- | --- |
| RND-100k | 0.993 | 0.557 | 0.657 | 0.981 |
| DIAYN-100k | 0.432 | 0.395 | 0.587 | 0.983 |
| Random-100k | 0.477 | 0.036 | 0.526 | 0.801 |
|  |  |  |  |  |

Though this is a step in the right direction it is still not a complete picture. For that, we would need a TQ-like metric that is applicable to our setting. This has raised an interesting discussion which we’ll add to the paper to highlight the limitations of these existing approaches. We will add this table to Appendix B (Datasets), cite this work, update text in the main body to mention this method and to discuss its limitations.
Thanks for clarifying your question. We hint at the setting you describe when we train on the medium-expert dataset in our D4RL experiments (a dataset containing a mix of ~optimal and suboptimal trajectories for the downstream task). We report those results in Table 10 Appendix C. We found VC-FB and MC-FB were performant in this setting, unlike FB which fails catastrophically, but didn’t match the performance of CQL. Our intuition is that the success of our methods in this setting depends closely on the choice of $\tau$ which reflects how “conservative” our methods are w.r.t. OOD state-actions. If $\tau$ is too high, the methods are not conservative w.r.t. OOD state-actions and we get catastrophic failures akin to FB. If $\tau$ is too low the methods are too conservative about OOD state-actions and the policy struggles if it deviates from the ~optimal trajectories described by the dataset. Similar findings are discussed in the original CQL paper in Appendix F. Of course, this is far from perfect, and we’d like to look at more robust methods in future work.
Thank you very much for your clarifications. I hope that they will help improve your manuscript, and make it more accessible to a broader audience.
Thank you for your response. My question regarding "good quality" dataset was rather about what would happen if the dataset you're pre-training on actually contains trajectories that solve the downstream task. In section 4.1 you say that you use offline TD3 performance on a given dataset as a proxy for dataset quality. My question is basically what will happen if you push the quality (as measured by TD3) even higher than you have so far. For example, you could take online TD3 replay buffer. Since there's not much time for you to collect the data and train on it, I'm just asking for you intuition about this.
Thank you for the responses. While most issues have been resolved, I think the first question remains unaddressed. The authors claim that "quality" is composed of "diversity" and "data size," which I disagree with. Does this imply that data with low diversity but large size is considered high quality? Additionally, "coverage" or "diversity" has been studied in previous works such as [1][2], whereas "quality" is often referred to as the average return of the trajectories. I suggest the authors reconsider their definition or terminology to ensure rigorousness of a good work.

[1] Schweighofer K, Radler A, Dinu M C, et al. A Dataset Perspective on Offline Reinforcement Learning[J]. arXiv preprint arXiv:2111.04714, 2021.

[2] Yarats D, Brandfonbrener D, Liu H, et al. Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning[J]. arXiv preprint arXiv:2201.13425, 2022.