Survival Instinct in Offline Reinforcement Learning
Authors: Anqi Li
Dipendra Misra
Andrey Kolobov
Ching-An Cheng
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Offline RL
safe RL
TL;DR: We find that offline RL can produce surprisingly good policies even when trained on wrong reward labels. We provide explanations and discuss practical implications.

Abstract: We present a novel observation about the behavior of offline reinforcement learning (RL) algorithms: on many benchmark datasets, offline RL can produce well-performing and safe policies even when trained with "wrong" reward labels, such as those that are zero everywhere or are negatives of the true rewards. This phenomenon cannot be easily explained by offline RL's return maximization objective. Moreover, it gives offline RL a degree of robustness that is uncharacteristic of its online RL counterparts, which are known to be sensitive to reward design. We demonstrate that this surprising robustness property is attributable to an interplay between the notion of *pessimism* in offline RL algorithms and certain implicit biases in common data collection practices. As we prove in this work, pessimism endows the agent with a *survival instinct*, i.e., an incentive to stay within the data support in the long term, while the limited and biased data coverage further constrains the set of survival policies. Formally, given a reward class -- which may not even contain the true reward -- we identify conditions on the training data distribution that enable offline RL to learn a near-optimal and safe policy from any reward within the class. We argue that the survival instinct should be taken into account when interpreting results from existing offline RL benchmarks and when creating future ones. Our empirical and theoretical results suggest a new paradigm for offline RL, whereby an agent is "nudged" to learn a desirable behavior with imperfect reward but purposely biased data coverage. Please visit our website [https://survival-instinct.github.io](https://survival-instinct.github.io) for accompanied code and videos.

Survival Instinct in Offline Reinforcement Learning
Anqi Li University of Washington
Dipendra Misra Microsoft Research
Andrey Kolobov Microsoft Research
Ching-An Cheng Microsoft Research
Abstract
We present novel observation about the behavior of offline reinforcement learning (RL) algorithms: on many benchmark datasets, offline RL can produce well- performing and safe policies even when trained with "wrong" reward labels, such those that are zero everywhere or are negatives of the true rewards. This phe- nomenon cannot be easily explained by offline RL's return maximization objective. Moreover, it gives offline RL degree of robustness that is uncharacteristic of its online RL counterparts, which are known to be sensitive to reward design. We demonstrate that this surprising robustness property is attributable an interplay between the notion of pessimism in offline RI algorithms and certain implicit biases in common data collection practices. As prove in this work, pessimism endows the agent with a survival instinct, i.c., an incentive to stay within the data support in the long term, while the limited and biased data coverage further constrains the set of survival policies. Formally, given a reward class - which may not even contain the true reward we identify conditions on the training data distribution that enable offline RL to learn a opar-optimal and safe policy from any reward within the class. We argue that the survival instinct should be taken into account when interpreting results from existing offline RL benchmarks and when creating future ones. Our empirical and theoretical results suggest new paradigm for offline RL,, whereby an agent is "nudged" to learn desirable behavior with imperfect reward but purposely biased data coverage. Please visit our website https://survival-iistnnee iitthuub..ttttboooooooooo highru for accompanied code and videos.
Introduction
In offline reinforcement learning (RL), an agent optimizes its performance given an offline dataset. Despite being its main objective, we find that return maximization is not sufficient for explaining some of its empirical behaviors. In particular, in many existing benchmark datasets, we observe that offline RL can produce surprisingly good policies even when trained utterly wrong reward labels.
In Fig. [ we present such results on the hopper task from D4RL 1. popular offline RI. benchmark, using a state-of-tthe-ar offline RI algorithm ATAC (2) The goal of an RI. agent in the hopper task to move forward as fast as possible while avoiding falling down. We trained ATAC agents on the original datasets and on three modified versions of each dataset, with "wrong" rewards: /) zero: assigning a zero reward to all transitions, 2) random: labeling each transition with a reward sampled uniformly from [0, I]. and 3) negative: using the negation of the true reward. Although these wrong rewards contain no information about the underlying task or are even misleading, the policies learned from them in Fig- (left) often perform significantly better than the behavior (data collection) policy and the behavior cloning (BC) policy. They even outperform policies trained with the true reward (denoted as original in Fig. ID in some cases. This is puzzling, since RL is notorious for being sensitive to reward mis-specification [3:6]: in general, maximizing the wrong rewards with RI. leads to sub-optimal performance that can be worse than simply performing BC. In addition, these wrong- wrong-reward reward policies demonstrate "safe" behavior, which keeps the hopper from falling down for longer period than other comparators in Fig. (right). This is yet another peculiarity hard to link to
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

BC
original
zero
random
negative
1000
100
75-
750
500
250-
medluatt--eplay meddemeeeperr
inteuum repley meduum-espprt
(a) Normalized scores
(b) Episode lengths
Figure 1: On the hopper task from D4RL M. ATAC 22, an offline RL algorithm, can produce high-perfornance and safe policies even when trained wrong rewards.
return maximization, none of the wrong rewards encourage the agent to stay alive. As will show empirically in Section these effects are not unique to ATAC or the hopper task. They occur with multiple offline RL algorithms,, including ATAC 22, PSPI (7). IQL 8 CQL [9] and the Decision benchmarks.
Transformer (DT) [10]. on dozens datasets from D4RL 1 and Meta- World
This robustness of offline RL is not only counter-intuitive but also cannot be explained fully by the provides performance guarantees only when the data reward is the true reward. Although offline imitation learning (IL) [14 makes assumptions about reward, it only shows that the learned policy can achieve performance comparable to that of the behavior policy, not beyond. Robust offline RL [17H21] shows that specialized algorithms can perform well when the size of data perturbation is small. In contrast, we demonstrate that standard offline RL algorithms can produce good policies even when completely change the reward. Constrained offline RL algorithms [22)24] can learn safe behaviors when constraint violations are both explicitly labeled and optimized. However, in the phenomena observe, no safety signal given to off-the-shelf, unconstrained offline RL. algorithms. Recently, [25] observes a similar robustness phenomena of offline RL, but does not provide a complete explanation In Appendix B we discuss the gap between the related work and our findings in more detail.
literature. Offline RL. theory
In this work, we provide explanation for this seemingly surprising observation. In theory, we prove that this robustness property attributed to the interplay between the use of pessimism offline RL algorithms and the implicit bias in typical data collection processes. Offline RL algorithmm often use pessimism to avoid taking actions that lead to unknown future events. We show that this risk-averse tendency bakes a "survival instinct" into the agent, an incentive stay within the data coverage in the long term. On the other hand, the limited coverage offline data further constrains the set of survival policies (policies that remain in the data support in the long term). When this set of survival policies correlates with policies that achieve high returns w.r.t. the true reward (as the example in Fig. robust behavior emerges.
Our theoretical and empirical results have two important implications. First and foremost, offline RL has survival instinct that leads to inherent robustness and safety properties that online RL does not have. Unlike online RL, offline RL is doubly robust: as long as the data reward is correct or the data has positive implicit bias, pessimistic offline RL agent can perform well. Moreover, offline RL is safe as long as the data only contains safe states; thus safe RL in the offline setup can be achieved without specialized algorithms. Second, because of the existence of the survival instinct, the data coverage has profound impact on offline RL. While a large data coverage improves the best policy that can be learned by offline RL with the true reward, it also make offline RL more sensitive to imperfect rewards. In other words, collecting a large set of diverse data might not be necessary or helpful (see Section 4. This goes against the common wisdom in the RL community that data should be as exploratory as possible (26228
We emphasize that survival instinct's implications should be taken into account when interpreting results on existing offline RL benchmarks as well as when designing future ones. We should treat the evaluation of offline RL algorithms differently from online RL algorithms. We suggest evaluating the performance of an offline RL algorithm by training it with wrong rewards in addition to the true reward so that we can isolate the performance due to return maximization from the compounded effects of survival instinct and implicit data bias. We propose to use this performance as a score to quantify the data bias (see Eq. (3) in Section 4.1 in practice.
We learned about this paper after the submission. Our theoretical results not only validate their conjecture the expert dataset, but also predict how this robustness happen in data collected by suboptimal policies.

---

We believe that our findings shed new light on RL applicability and research. To practitioners, we demonstrate that offline RL does not always require the correct reward to succeed. This opens up the possibility of using offline RI. in domains where obtain high-quality rewards is challenging. Research- wise, the existence of the survival instinct raises the question of how to design data collection or data filtering procedures that would help offline RL to leverage this instinct in order to improve RL's performance with incorrect or missing reward labels. While in this paper we focus on positive data biases, we caution that in practice the data bias might be negatively correlated with a user's intention. In that case, running offline RL even with the right reward would not lead to the right behavior.
2 Background
The goal of offiine RI is to solve an unknown Markov decision process (MDP) from offline data. Typ- ically, an offline dataset is a collection of tuples, D == {(s,, a, (')(s, a) ~ ()(;)), r- r(s,, a), P(-(s,a) a)). where r is the reward, P captures the MDP's transition dynamics, and μ denotes the state-action data distribution induced by some data collection process. Modern offline RL algorithms adopt pessimism address the issue of policy learning when μ does not have the full state-action space as its support. The basic idea is to optimize for a performance lower bound that penalizes actions leading to -ustooot future states. Such a penalty can take the form of behavior regular- ization [8] [29] (30, negative bonuses to discourage visiting less frequent state-action pairs [12 [3 [3),
pruning less frequent actions [33 [3]. adversarial training
331 or value penalties in
modified dynamic programming 9 36
We are interested in settings where the offline RL agent learns not from D itself but from its corrupted version D with wrong reward F, i.e., D = {(s, a, F, F, s') a's) D, F f(s, a) E []], We assume is from reward class R which may not necessarily contain the true reward r r- and we wish to explain why offline RL can learn good behaviors from the corrupted dataset D (e.g., in Fig. D. To this end, we introduce notations and assumptions we will use in the paper.
Notation We focus on the setting of infinite-horizon discounted MDPs. We denote the task MDP that the agent aims to solve as M = A. P,7), where S is the state space, A is the action space. and y E [0, is the discount. Without loss of generality, we assume r S x AA + [0, 1]. Let A(U) denote the set of probability distributions over a set U. We denote a policy as
S + A(A). For a reward function r, we define a policy I's state value function as (s) = B,,P E 2 (rr( (so = s] and the aate-actioon value function as Q, a) = r(s.a) a) + -BBss-(()s..)) VV (s').. Solving MDP A requires learning a policy that maximizes the return at initial state distribution do E A(S), that is, max, Ey [V* (+)]. We denote the optimal policy as T² and the optimal value functions as V and Q². Given dg. we define the average state- action visitation of policy π as d' ds((s)) = (1 - >),,,, .IEE= 7 dī (s, a)], where dī (s, a) denotes the probability of visiting (s, a) at time i when running starting at an initial state sampled from do- Note that (1 Te [V7 [s)] = E(e.a)aa (r(s, a)). For a function f S A + TR, we use the shorthand f(ss, T) Ea + (s(s, a)}; similarly for f S + R, we write (p) E.- e.g..
v, Eg [V, (s)]. For distribution p. we use supp(p) to denote its support.
Assumption We make the typical assumption in offline RL that the data distribution μ assigns positive probabilities to these state-actions visited by running the optimal policy 'r" starting from dg-
Assumption 1 (Single-policy Concentrability). We assume SuP
d (s.a) p(s.a]
< 0o.
This is a standard assumption in the offline RL literature, which in the worst case is a necessary condition to have no regret w.r.t. π² B7). There are generalized notions 7 34 of this kind, which are weaker but requires other smoothness assumptions. We note that Assumption does not assume that μ is the average state-action visitation frequency of single behavior policy, nor does assume
has a full coverage of all states and actions or all policy distributions 27 38
3 Why Offline RL can Learn Right Behaviors from Wrong Rewards
In this section, we provide conditions under which offline RL's aforementioned robustness w.r.t. misspecified rewards emerges. Our main finding is summarized in the theorem below.
Theorem 1. (Informal) Under Assumption 7 and certain regularity assumptions, if an offline
RL algorithm Algo is set be sufficiently pessimistic and the data distribution P. has a pos- itive bias, for any data reward F E R. the policy à learned by Algo from the dataset D
3

---

has performance guarantee V (do) - (V) < O(1) as well as safety guarantee (1 7) Di=gg 7 Prro (3r [0. s, S, & supp(p)]a) S O(1) for small u that decreases to zero as the degree of pessimism and dataset size increase.
In other words, the robustness originates from an interplay between pessimism and an implicit positive bias in data. Here is the insight on why Theorem is true. Because of pessimism, offline implicitly solves a constrained MDP (CMDP) problem [39 that enforces the policy to stay within the data support. When combined with a training data distribution that has a positive bias (e.g., all policies staying within data support are near-optimal) such a survival instinct results in robustness to reward misspecification.
RL endows agents with "survival instinct"
Overall, Theorem 1 has two important implications:
1. Offline RL is doubly robust: it can learn near optimal policies so long as either the reward label is correct or the data distribution is positively biased;
2. Offline RL is intrinsically safe when data are safe, regardless of reward labeling, without the need of explicitly modeling safety constraints.
In the remaining of this section, we provide details and discussion of the statements above. The complete theoretical statements and proofs can be found in Appendix D
3.1 Intuitions for Survival Instinct and Positive Data Bias
We first use a grid world example to build some intuitions. Fig. 2 shows goal-directed problem, where the true reward is +1 and 1 upon touching the key and the lava, respectively. The offline data is suboptimal and does not have full support. All data trajectories that touched the lava were stopped early, while others were allowed to continue until the end of the episode. The goal state (key) is absorbing state, where the agent can stay forever beyond the episode length. We use the wrong rewards in Fig. [ to train PEVI [12,- finite-horizee tabular offline RL method. PEVI performs dynamic programming similar to value iteration, but with pessimistic value initialization and an instantaneous pessimism-inducing penalty of O ( //(s.))) where ns, a) is the empirical count in data. The penalties ensure that the learned value lower bounds the true one.
Figure 2: A grid world. BC (red); offline RL with wrong rewards (blue). The opacity indicates the frequency of a state in the data (more opaque means more frequent). Offline RL with the three wrong rewards produces the same policy.
We see the PEVI agent learned with any wrong reward in Fig. iis able to solve the problem despite data imperfection, while the BC agent that mimics the data directly fails. The main reasons are: 1) There is a data bias whereby longer trajectories end closer to the goal (especially, the longest trajectories are the ones that reach the goal, since the goal is an absorbing state). We call this length bias (see Section (.3). This positive data bias is due to bad trajectories (touching the lava or not reaching the goal) being cut short or timing out. 2) Pessimism in PEVI gives the agent algorithmic bias (i.e., survival instinct) that favors longer data trajectories. Because of the pessimistic value initialization, PEVI treats trajectories shorter than the episode length as having the lowest return. As result, by maximizing the pessimistically estimated values, PEVI learns good behaviors despite wrong rewards by leveraging the survival instinct and positive data bias together. We now make this claim more general.
3.2 Survival Instinct
Survival instinct is pessimism induced behavior that offline RL algorithms tend to favor policies leading to iupsup trajectories. We formally characterize this risk aversion behavior by the concept of constrained MDP (CMDP)
39 which we define below.
Definition 1. Let f.g: : S (-1, 1). A CMDP C(S, A, P, 7) is a constrained optimization problem: max, VJ (do) s.t. V (do) S 0. Let म denote its optimal policy. For 8 2 0. we define the set of d-approximately optimal policies 11 (6) == {- VF' (do) - (do) S V,, (do) S 6).
We prove that when trained on D. offline RL, because of its pessimism, implicitly solves the CMDP below even when the algorithm does not explicitly model any constraints:
Cf( C(S,A, cu, P,?),
(I)
where ()) = a) = 0) indicates whether (s, a) is out of the support of A i.c., the

---

constraint in Eq. (T) enforces the agent's trajectories to stay within the data support. Note the constraint in this CMDP is feasible because of Assumption[[]
Proposition 1 (Survival Instinet). (Informal) Under certain regularity conditions CT),), the policy learned by an offline RI. algorithm Algo with the offline dataset π is t-approximately optimal with respect to (P), ( for some small which decreases as the algorithm becomes more pessimistic.
Proposition 1 says if an offline RL algorithm sufficiently pessimistic, then the learned policy is approximately optimal to (F). The learned policy has not only small regret with respect to the data reward F, but also small chances of escaping the support of the data distribution μ.
We highlight that the survival instinct in Proposition [ is a long-tern behavior. Such a survival behavior cannot be achieved by myopically taking actions in the data support in general (such as BI For instance, when some trajectories generating the data are truncated (e.g., due to early- stopping intervention for safety reasons), taking in-support actions may still lead to out-of-support states in the future, as the BC agent in Section 3.1
BC)
Proof Sketch The key to prove Proposition is to show that an offline RL algorithm by pessimism has small regret for not just F but a set of reward functions consistent with i' on data but different outside of data, including the Lagrange reward of (I) (i.e., f λcμ. with large enough À 2 0). We call this property admissibility and we prove in Appendix F that many existing offiin RI algorithms
admissible, including model-free algorithms ATAC 2 VI-LCB [31 PPI/PQI [I3] PSPI (7).
as well as model-algorie aogoel-aggoitthm, ARMOR [34, 40). MOPO 32), MOReL 13 and CPPO [14]. By
Proposition these algorithms have survival instinct. Please see Appendix D for details.
3.3 Positive Data Bias
In addition to survival instinet, another key factor is an implicit positive bias in common offline datasets. Typically these data manifest meaningful behaviors. For example, in collecting data for goal-oriented problems, data recording is normally stopped if the agent fails to reach the goal within certain time limit. Another example is problems (like robotics or healthcare) where wrong decisions can have detrimental consequences, i.e., problems that safe RL studies. In these domains, the data are collected by qualified policies only or under an intervention mechanism to prevent catastrophic failures. Such one-sided bias can creates an effect that staying within data support would lead to meaningful behaviors. Below we formally define the positive data bias; Later in Section 4 (Fig. 5) we provide empirical estimates of the degree of positive data bias.
Definition 2 (Positive Data Bias). A distribution μs is ! --oositively biased w.r.t. reward class R if
max max V. (do) (do) Sc+ FER TElL .,, (6)
(2)
for all 8 2 0, where 11; (d) denotes the set of 6-approximately optimal policy of (F).
We measure the degree of positive data bias based on how bad policy can perform in terms of the true reward r when approximately solving the CMDP (F) in (1) defined with the wrong reward i E R. If the data distribution is oc-positively biased, then any approximately optimal policy to (F) (which includes the policies learned by offline RL, shown earlier) can achieve high return in the true reward r. We also can view the degree of positiveness as reflecting whether 7 provides a similar ranking as r, among policies within the support of μ. When there is positive bias, offline RL can learn with to perform well under even when i is not aligned with globally. This is in contrast to online RL, which requires global alignment due to the exploratory nature of online RI.
Examples
We provide few examples of positive data bias. (See Appendix D.4 for proofs.)
The distribution d= induced by any policy is co-poiitively biased for any rewards resulting from potential- reward shaping 41 since it provides the same ranking for all policies.
For an II. setup, the data distribution μ is oo-positively biased for any rewards f : S x A -> R if the data is generated by the optimal policy (i.e., μ = d=' ). This is because following the optimal
'The use of an indicator function not crucial here. We can extend the current analysis here to other costs that are zero the support and strictly positive out of the support.
BC requires stronger condition, e.g., the data are all complete trajectories without intervention or timeouts.
5

---

policy is the only way to stay within the data support. In Section 4 we empirically show that offline RL algorithms can learn good policies with wrong rewards on D4RL expert datasets, which are collected by near-optimal policies.
A positive bias happens when longer trajectories in the data have smaller optimality gap. This is a generalized formal definition of the length bias mentioned in Section [3.] This condition is typically satisfied when intervention is taken in the data collection process (despite the data collection policy being suboptimal), as in the motivating example in Fig. Later in Section 4 we
will investigate deeper into this kind of length bias empirically.
Remark We highlight that the positive data bias assumption in Definition 2 is different from assuming that the data is collected by expert policies, which is typical in the IL literature. Positive data bias assumption can hold in cases when data is generated by highly suboptimal policies, which we observe in the hopper example from Section I On the other hand, there are also cases where IL can learn well, while positive data bias does not exist (e.g., when learning from data collected by a stochastic expert policy that covers highly suboptimal actions with small probabilities).
3.4 Summary: Offfine RL is Doubly Robust and Intrinsically Safe
We have discussed the survival instinct from pessimism and the potential positive data bias in common datasets. In short, survival instinct enables offline RL algorithms to learn policies that benefit from a favorable inductive bias of staying within the support of a positive data distribution. As a result, offline RL becomes robust to reward mis- siiisppecfiittiin (namely, Theorem I.
Below we discuss some direct implications of Theorem First, we can view this phenomenon as a doubly robust property of offline RL. We borrow the name "doubly robust" from the offline policy evaluation literature [42] to highlight the robustness of offline RL to reward mis-specification as an offline policy optimization approach.
Corollary 1. Under Assumption ! offline RL can learn near optimal policy, long the reward is correct, or the data has a positive implicit bias.
Theorem aalso implies that offline RL is intrinsically safe learning algorithm, unlike its counterpart online RL. where additional penalties or constraints need to be explicitly modelled 473
Corollary 2. If μs only covers safe states and there exists a safe policy staying within the support of Hs, then the policy of offline RL only visits safe states with high probability (see Theorem !
We should note that covering safe states is a very mild assumption the safe RL literature e.g., compared with all (data) states have safe action. It does not require the data collection policies that generate μ are safe. This condition can be easily satisfied by filtering out unsafe states in post processing. The existence of a safe policy is also mild and common assumption. In Section 4.2 validate this inherent safety property on offline SafetyGymnasium
49 an offline safe RL benchmark.
4 Experiments
We conduct two group of experiments. Section 4.1 ing that multiple offline RL. algorithms can be robust to reward mis-specification on a variety of datasets. In Section 4.2, we experimentally validate the inherent safety of offline RL algorithms stated in Corollary 2 We show that admissible offline RL algorithms, without modifications, can achieve ofate-oftthe-art performance on an offline safe RL benchmark 49
we conduct large scale experiment, show-
4.1 Robustness to Mis-specified Rewards
We empirically study the performance of offline RL algorithms under wrong rewards in a variety
of tasks. We use the same set of wrong rewards as in Fig. [1] 1) zero: the zero reward, 2) random: labeling each transition with reward value randomly sampled from Unif 1]. and 3) negative:
the negation of true reward. We consider five offline RL algorithms, ATAC 22. PSPI 7. IQL. (8)
CQL [9] and decision transformer (DT [10] We deliberately choose offline RL algorithms cover those that are provably pessimistic 22 7 and those that are popular among practitioners g (9) as
well as an unconventional offline RL algorithm [IO]. We consider a variety of tasks from D4RL and Meta-
Meta-World World [1I] ranging from safety-critical tasks (i.e., the agent dies when reaching bad states),
"We condition the decision transformer on the return of a trajectory sampled randomly from trajectories that achieve top 10% of returns in terms of data reward.

---

ATAC
50
PSPI
50
hopper
walker2d
halfcheetah
behavior
original
random
negative
IQL
50
100
CQL
100 DT
1900
mell rostay ITAT eop
7903
med
mend roclay most eop
Wend vockes mos eup
hopper
walker2d
halfcheetah
Figure 3: Normalized scores for locomotion tasks from D4RL [.] The mean and standard error for normalized scores are computed across 10 random seeds. For each random seed, we evaluate the final policy of each algorithm over 50 episodes.
goal-oriented tasks, and tasks that belong to neither. We train a total of around 16k offline RL agents (see Appendix C.8 Please see Appendix for details.
Messages We would like to convey three main messages through our experiments. First, implicit data bias can exist naturally in wide range of datasets, and offline RL algorithms that sufficiently pessimistic can leverage such bias to succeed when given wrong rewards. Second, offline RL algorithms, regardless how pessimistic they are, become sensitive reward when the data does not possess a positive bias. Third, offline RL algorithms without explicit pessimism, e.g., IQL (8]. can sometimes still be pessimistic enough achieve good performance under wrong rewards.
Remark negative results We consider "negative" results, i.e., when offline fails under wrong rewards, as important as the positive ones. Since they tell us how positive data bias can be broken avoided. We hope our study provide insights researchers who are interested in actively incorporating positive bias data collection, as well as who hope to design offline RI. benchmarks specifically with without positive data bias.
4.1.1 Locomotion Tasks from D4RL
We evaluate offline RL algorithms three locomotion tasks, hopper, walker2d and half chlethhe..h from D4RL [ widely used offline RL benchmark. For each task, we consider five datasets with different qualites: random, medium, medium-replay. medium-expert, and expert. We refer readers to D for the construction of these datasets. We measure policy performance in D4RL normalized scores [I]. We provide three baselines 1) behavior: normalized score directly computed from the dataset, 2) BC: the behavior cloning policy, and 3) original: the policy produced by the offline RL algorithm using the original dataset (with true reward). The normalized scores for baselines and offline RL algorithms with wrong rewards are presented in Fig. 3 The exact numbers can be found in tables in Appendix C
Positive bias exists in some D4RL datasets. We visualize the length bias of D4RL datasets Fig. Here are our main observations. 7) Most datasets for hopper and walker2d, with the
'We remove terminal transitions from hopper and valker2d datasets when learning with wrong rewards. In Appendix include ablation study on the effect of removing terminals when using reward.
7

---

medium
medium-replay
medium-expert
expert
100
100
100-
100-
160
hopper
1000
0
1000
0
1000
0
1000
0
1000
100
100
100
100
100
walker2d 50
1000
1000
0
1000
1000
1000
100
100
100
100
100
halfcheetah 50
50
50
1000
1000
0
1000
Figure 4: A visualization of length bias in datasets from D4RL. [1. Each plot corresponds to dataset for a task (row) with a dataset (column). Each trajectory in a dataset represented as a data point with the z-coordinate being its episode length and y-coordinate being its normalized score.
exception of the medium-replay datasets, have strong length bias, where longer trajectories have higher returns. This length bias due to the safety-critictt nature of hopper and walker2d tasks, as the trajectories get terminated when reaching bad states. The length bias is especially salient in pepper-medium, where the normalized score is almost proportional to episode length. 2) The medium-replay datasets for hopper and walker2d have more diverse behavior, the bias is smaller. 3) All half cheetah datasets do not have length bias, as they all have the same length of 1000. 4) hopper-expert dataset has length bias, while walker2d and hal cheetah-expeeree datasets do not have an obvious length bias. However, it is worth noting that all expert datasets have an IL-type positive bias as discussed in Section 3.3
Offline RL can learn good policies with wrong rewards on datasets with strong length bias. On datasets with strong length bias (aopper- hopper-medioee and 2elkerr2dmmediun we observe that ATAC and PSPI with wrong rewards generally produce well-performing policies, in a few cases even out-performing the policies learned from the true reward (original in Fig. 4 DT is mostly insensitive to reward quality. IQL and CQL with wrong rewards can sometimes achieve good performance; among the two, we find IQL to be more robust to wrong rewards and CQL with wrong rewards almost fails completely on all walker2d datasets.
Offline RL can learn good policies with wrong rewards on expert datasets In Section 3 we point out that the data has a positive bias when it is generated by the optimal policy. In Fig. 3 observe that all offline RL algorithms, when trained with wrong rewards, can achieve expert-level performance on walker2d and half thetthh eeaeh-expettt datasets. ATAC, PSPI and DT perform well on the hopper-expers dataset while IQL and CQL receive lower scores when using wrong rewards.
Offline RL needs stronger reward signals on datasets with diverse behavior policy. The medium-replay datasets of hopper and walker2d are generated from multiple behavior policies. Due to their diverse nature, they are a multiple ways to stay within data support in a long term. As result, the survival instinct of offline RL by itself is not sufficient to guarantee good performance. Here algorithms with wrong rewards generally under-perform the policies trained with true reward. As datasets have a more diverse coverage, they can be less positively biased and offline RI. requires a stronger reward signal to differentiate good survival policies and the bad ones. Practically speaking, when high-quality reward is not available, a diverse dataset can even hurt offline RL performance. This is contrary to the common belief that larger data support is more preferable (26-28).
Offline RL requires good reward to perform well datasets without length bias. In all four halfcheetah datasets, the trajectories all have the same length, as is demonstrated in Fig. 4 This means that there no data bias. We observe that all algorithms with wrong rewards at best perform similarly as the behavior and BC policies in most cases; this is an imitation-like effect due to the survival instinct. In the half cheetah -eahhmmedium iumiimm--ee medpum-expeet dataset, since there are variety of surviving trajectories, with score from 0 to near 100, we observe that the performance of the resulting policy degrades as data reward becomes more different from the reward.
Offfine RL algorithms with provable pessimism produce safer policies. For hopper and 2alker2d, we observe that policies learned by ATAC and PSPI, regardless of the reward, can keep the agent from falling for longer. The episode lengths of IQL and CQL policies are often
8

---

FAnd
mid
mad-aally nap
hopper
PCD
rend
med
walker2d
madreplay pidd
FAnd
halfcheetah
mas mplay meding
PP
Figure 5: Estimated positive data bias of all 15 D4RL datasets given by ATAC. Datasets marked by i... hopper- expert and walker2d expert, have infinite positive bias.
comparable to that of the behavior policies. We provide statistics of episode lengths in Appendix C In Section 4.2 we provide further results validating the safety properties of ATAC and PSPI.
On empirically estimating positive data bias Inspired by Definition 2 we propose an empirical estimate of positive data bias. Let Jzao- Faand and Jneg be the return (with respect to the true reward) of an offline RL algorithm learned with zero, random, and negative rewards, respectively. Then, given an estimated optimal return J+, we propose to empirically estimate the positive data bias by
estimated positive bias = J /m/max J" -min( Jug
(3)
In Fig. 5 we visualize the estimated positive data bias of all 15 D4RL datasets given by ATAC. Estimated positive data bias of all D4RL and Meta-World datasets is given in Fig. 9
Remark on benchmarking offline RL Our observations on the popular D4RL datasets raise an interesting question for benchmarking offline RL algorithms. An implicit positive data bias can give certain algorithms a hidden advantage, as they can already achieve good performance without using the reward. Without controlling data bias, it is hard differentiate whether the performance of those algorithms are due to their ability to maximize returns or simply due to their survival instinct.
4.1.2 Goal-oriented Manipulation Tasks from Meta-World
We study goal-oriented manipulation tasks from the Meta-World benehmark goal-oriented tasks from Meta-Worlds the 10 tasks from the MT-10 set and the 5 testing tasks from the MI.-45 set. For each task, we generate a dataset of 110 trajectories using the scripted policies provided by MT: 10 are produced by the scripted policy, and 100 are generated by perturbing the scripted policy with a zero-mean Gaussian noise of standard deviation 0.5. In the datasets, unsuccessful trajectories receive time-out signal after 500 steps (maximum episode length for Meta-World),
We consider 15
For each task, we measure the success rate of policies for 50 new goals unseen during training. Similar to D4RL experiments, consider BC policies and policies trained with the true reward (original) as baselines. Since the training dataset is generated for a different set of goals, we do not compare the success rate of learned policies with the success rate in data.
Goal-oriented problems have data bias. Goal-oriented problems are fundamentally different from safety-critical tasks (such as hopper and walker2d). good goal-oriented policy should reach the goal as fast as possible rather than wandering around until the end of an episode. But another form of length bias still exists here, as successful trajectories are labeled with a termination signal that indicates an unbounded length and failed trajectories have bounded length (see Section 3.1)
Offline RL can learn with wrong rewards on goal-oriented tasks. The success rate of learned policies with different rewards are shown in Fig. 6 We observe that ATAC and PSPI with wrong rewards generally achieve comparable or better success rate than BC policies. The exceptions are often due that ATAC or PSPI, even with the true reward, does not work well as BC in those tasks, e.g. drawer-open, bin- tinpiikkig and box-close. This could be caused by overfitting or optimization errors (due to challenges of dynamics programming over long horizon problems). In a number of tasks, such as peg-insert-side side, push and reach, ATAC and PSPI with wrong rewards can out-perform BC by margin. This shows that data collection for goal-oriented problems have positive bias that offline RI. algorithms can succeed without true reward. This is remarkable as when learning with random and negative rewards, unsuccessful trajectories are long and generally have significantly higher return, as reward for each step is non-negative. The offline RI. algorithms need to be sufficiently pessimistic and be able to plan over a long horizon to propagate pessimism to the unsuccessful data trajectories. For IQL, there is gentle performance decrease the reward becomes more different than the true reward, even though policies learned with wrong rewards are not much worse than BC in many cases. CQL shows robustness reward in few tasks such as button-press, press, drawer-close and door-un unlock. Interestingly, DT performs almost uniformly across all rewards, potentially because DT does not explicitly maximize returns.

---

0.5
1.0
PSPI
button press
doar open
dhawer cl0ce
drower Rpon
Insert
pikk place
reach
window clsse
Window open
Din picking
cour lock
door unlack
side
dlsse
visert
BC
original
zero
random
negative
IQL
0.5
1.0
CQL
DT
0.0
button
3bor
drower
drawer
pog Insert
pick
Push
reach
window
window
Bin
Dox
COOr
dbor unlock
hand
open
open
open
plcking
Figure 6: Success rate for gral-oriented tasks from Meta-War World (II] The average success rate and
confidence interval are computed across 10 random seeds. For each seed, we evaluate final policies for 50 episodes, each with new goal unseen in the dataset.
Remark on BC performance It is noticeable that BC policies in general achieve high performance, in many tasks often out-performing offline RL policies learned with true reward. This effect has also been observed in existing work on similar tasks [50]. We would like to clarify that the goal of experiments is to study the behavior of offline RL algorithms when trained with wrong rewards, rather than showing that offline RL performs better than BC. We refer interested readers to existing when using offline RL algorithms is is not preferable over BC.
studies [50 51
4.2 Inherent Safety Properties of Offline RL Algorithms
We show offline RI. algorithms (without any modifications) can behave safe RL algorithm. We conduct experiments on offline SafetyGymnasie
(49) using ATAC (2) and PSPI (7]. We first remove all transitions with non-zero cost and then run offline RL algorithms the filtered datasets.
The results summarized in Table in Appendix C.6 We observe that offline RL with this naive data filtering strategy (using less data) can achieve comparable performance as the per-task best performing state-of-the-or offline safe RL algorithm, which uses the full dataset and has knowledge of the cost target. Our agents in general incur low cost except for the circle tasks. We hypothesize that learning safe policy from the circle datasets hard, as other baselines also struggle to produce safe policies. Note that these are preliminary results, and better performance might be achievable through using a more sophisticated data filtering strategy. Please see Appendix C.6 for experiment details.
5 Concluding Remarks
We present unexpected results on the robustness of offline RL to reward mis mis-specification. We show that this property originates from the interaction between the survival instinet of offline RL and hidden positive biases in common data collection processes. Our findings suggest extra considerations should taken when interpreting offline RL results and designing future benchmarks. In addition, our findings open new space for offline RL research: the possibility of designing algorithms that proactively leverage the survival instinct learn policies for domains where rewards are nontrivial to specify or unavailable. Please Appendix A for discussion limitations and broader impacts.
10

---

References
[1] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine, "D4rl: Datasets for deep data-driven reinforcement learning," arXiv preprint arXiv:2004.07219, 2020.
[2] C.-A. Cheng, T. Xic, N. Jiang, and A. Agarwal, "Adversarially trained actor critic for offline reinforcement learning," in International Conference Machine Learning, PMLR, 2022.
[3] J. Leike, M. Martic, V. Krakovna, P. A. Ortega, T. Everitt, A. Lefrancq. L. Orseau, and S. Legg, "Ai safety gridworlds," arXiv preprint arXiv:1711.0988. 2017.
[4] D. Hadfield-Menell, Milli, P. Abbeel, S. J. Russell, and A. Dragan, "Inverse reward design." Advances in neural information processing systems, vol. 30, 2017.
[5] B. Ibarz, J. Leike, T. Pohlen, G. Irving. S. Legg, and D. Amodei, "Reward learning from human preferences and demonstrations in atari," Advances in neural information processing systems, vol. 31, 2018.
[6] A. Pan, K. Bhatia, and J. Steinhardt, "The effects of reward misspecification: Mapping and mitigating misaligned models," in International Conference on Learning Representations, 2022.
[7] T. Xie, C.-A. Cheng, N. Jiang, P. Mineiro, and A. Agarwal, "Bellman-consistent pessimism for offline reinforcement learning," Advances in neural information processing systems, vol. 34, pp. 6683-6694, 2021.
[8] I. Kostrikov, A. Nair, and S. Levine, "Offline reinforcement learning with implicit q-learning," in International Conference on Learning Representations, 2021.
[9] A. Kumar, A. Zhou, G. Tucker, and S. Levine. "Conservative q-aarnnig for offline reinforcement learning," " Advances in Neural Information Processing Systems, vol. 33, pp. 1179-1191, 2020.
[10] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and Mordatch, "Decision transformer: Reinforcement learning via sequence modeling," Advances in neural information processing systems, vol. 34, pp. 15084-15097, 2021.
[11] Yu, D. Quillen, Z. He, R. Julian, Hausman, C. Finn, and Levine, "Meta-world: benchmark and evaluation for multi-task and meta reinforcement learning," in Conference on robot learning, pp. 1094-1100, PMLR, 2020.
[12] Y. Jin, Z. Yang, and Z. Wang. "Is pessimism provably efficient for offline r!?," in International Conference on Machine Learning, pp. 5084-5096 PMIR, 2021.
[13] Y. Liu, A. Swaminathan. A. Agarwal, and E. Brunskill, "Provably good batch off-policy reinforcement learning without great exploration," Advances in neural information processing systems, vol. 33, pp. 1264-1274, 2020.
[14] M. Uehara and W. Sun, "Pessimistic model-based offline reinforcement learning under partial coverage," in International Conference on Learning Representations, 2022.
[15] J. Chang, M. Uchara, D. Sreenivas, R. Kidambi, and W. Sun, "Mitigating covariate shift in imitation learning via offline data with partial coverage," Advances in Neural Information Processing Systems, vol. 34, pp. 965-979, 2021.
[16] Y. Ma, A. Shen, D. Jayaraman, and O. Bastani, "Versatile offline imitation from observations and examples via regularized occupancyy matching." in International Conference on Machine Learning, pp. 14639-14666 PMLR, 2022.
[17] X. Zhang, Y. Chen, X. Zhu, and W. Sun, "Comuption-robus offline reinforcement learning," in International Conference on Artificial Intelligence and Statistics, pp. 5757-5773, PMLR, 2022.
[18] F. Wu, L. Li, H. Zhang, B. Kailkhura, K. Kenthapadi, D. Zhao, and B. Li, "Copa: Certifying robust policies for offline reinforcement learning against poisoning attacks," " in International Conference Learning Representations, 2023.

---

[19] Z. Zhou, Z. Zhou, Q. Bai, L. Qiu, J. Blanchet, and P. Glynn, "Finite- sample regret bound for distributionally robust offline tabular reinforcement learning," International Conference on Artificial Intelligence and Statistics, pp. 3331-3339, PMLR, 2021.
[20] X. Ma, Z. Liang, L. Xia, J. Zhang. J. Blanchet, M. Liu, Q. Zhao, and Z. Zhou, "Distribution- ally robust offline reinforcement learning with linear function approximation." arXiv preprins 2209. 06620, 2022.
[21] K. Panaganti, Z. Xu, D. Kalathil, and M. Ghavamzadeh, "Robust reinforcement learning using offline data," in Advances Neural Information Processing Systems, 2022.
[22] H. Le, C. Voloshin, and Yue, "Batch policy learning under constraints," in International Conference on Machine Learning, pp. 3703-3712, PMLR, 2019.
[23] H. Xu, X. Zhan, and X. Zhu, "Constraints penalized q-learning for safe offline reinforcement learning," " in AAAI, 2022.
[24] Lee, C. Paduraru, J. Mankowitz, N. Heess, D. Precup. K.-E. Kim, and A. Guez, "Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation," in International Conference on Learning Representations, 2022.
[25] D. Shin, Dragan, and D. S. Brown, "Benchmarks and algorithms for offline preference- based reward learning." Transactions on Machine Learning Research, 2023.
[26] R. Munos, "Error bounds for approximate policy iteration," in International Conference on Machine Learning, vol. 3. pp. 560-567, Citeseer, 2003.
[27] Chen and N. Jiang, "Information- -hhoorrtti considerations in batch reinforcement learning," in International Conference on Machine Learning, pp. 1042-1051, PMLR, 2019.
(28) D. Yarats, D. Brandfoobrener, H. Liu, M. Laskin, P. Abbeel, A. Lazaric, and L. Pinto, "Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning." arXiv preprint arXiv: 13425, 2022.
[29] S. Fujimoto and S. S. Gu, "A minimalist approach to offline reinforcement learning," Advances neural information processing systems, vol. 34, pp. 20132-20145, 2021.
[30] Y. Wu, G. Tucker, and O. Nachum, "Behavior regularized offline reinforcement learning," arXiv preprint arXiv: 191 1361, 2019.
[31] P. Rashidinejad, B. Zhu, C. Ma, J. Jiao, and S. Russell, "Bridging offline reinforcement learning and imitation learning: A tale of pessimism," Advances Neural Information Processing Systems, vol. 34, pp. 11702-11716, 2021.
[32] T. Yu, G. Thomas, L. Yu, S. Ermon, J. Y. Zou, S. Levine, C. Finn, and T. Ma, "Mopo: Model- based offline policy optimization," Advances in Neural Information Processing Systems, vol. 33, pp. 14129-14144 2020.
[33] R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims, "Morel: Model-based offline reinforcement learning," Advances in neural information processing systems, vol. 33, pp. 21810- 21 21823, 2020.
[34] T. Xie, M. Bhardwaj, N. Jiang, and C.-A. Cheng, "Armor: A model-based framework for improving arbitrary baseline policies with offline data," arXiv preprint arXiv:2211.0455x 2022.
[35] M. Rigter, Lacerda, and N. Hawes, "Rambo-RL: Robust adversarial model-based offline reinforcement learning," Advances in neural information processing systems, 2022.
[36] T. Yu, A. Kumar, R. Rafailov, A. Rajeswaran, S. Levine, and C. Finn, "Combo: Conservative offline model-based policy optimization," Advances in neural information processing systems, vol. 34, pp. 28954-28967, 2021.
[37] W. Zhan, B. Huang, A. Huang, N. Jiang, and J. Lee, "Offline reinforcement learning with realizability and single-policy concentrability." Conference on Learning Theory, pp. 2730- 2775, PMLR, 2022.
12

---

[38] D. J. Foster, A. Krishnamurthy, D. Simchi-Levi, and Y. Xu, "Offline reinforcement learning: Fundamental barriers for value function approximation, in Conference on Learning Theory, pp. -4899-3489, PMLR, 2022.
[39] E. Altman, Constrained Markov decision processes, vol. 7. CRC press, 1999.
[40] M. Bhardwaj, T. Xie, B. Boots, N. Jiang, and C.-A. Cheng, "Adversarial model for offline reinforcement learning," arXiv preprint arXiv:2302.11048, 2023.
[41] A. Y. Ng. D. Harada, and S. Russell, "Policy invariance under reward transformations: Theory and application to reward shaping," in International Conference Machine Learning, vol. 99, pp. 278-287, Citeseer, 1999.
[42] M. Dudík, J. Langford, and L. Li, "Doubly robust policy evaluation and learning," in Proceed- ings of the 28th International Conference on International Conference on Muchine Learning, pp. 1097-1104, 2011.
[43] Achiam, D. Held, A. Tamar, and P. Abbeel, "Constrained policy optimization," in International conference on machine learning, pp. 22-3122311 PMLR, 2017.
[44] A. Wachi and Y. Sui, "Safe reinforcement learning in constrained markov decision processes," in International Conference on Machine Learning, pp. 9797-9806, PMLR, 2020.
[45] N. C. Wagener, B. Boots, and C.-A. Cheng, "Safe reinforcement learning using advantage- bbase intervention," International Conference on Machine Learning, pp. 10630-10640, PMLR, 2021.
[46] S. Paternain, M. Calvo-Fullana, F. F. Chamon, and A. Ribeiro, "Safe policies for reinforcement learning via primal-dual methods," IEEE Transactions on Automatic Control, 2022.
[47] H.-A Nguyen and C.-A. Cheng, "Provable reset-free reinforcement learning by no-regret reduction," arXiv preprint arXiv:2301.02388, 2023.
[48] A. Hans, D. SchneegaB, A. M. Schäfer, and S. Udluft, "Safe exploration for reinforcement learning.." " in ESANN, pp. 143-148, Citeseer, 2008
[49] Z. Liu, 7. Guo, H. Lin, Y. Yao, Zhu, Z. Cen, H. Hu, W. Yu, T. Zhang, J. Tan, al., "Datasets and benchmarks for offline safe reinforcement learning," arXiv preprint arXiv:2306.09300 2023.
[50] G. Zhou, L. Ke, S. Srinivasa, A. Gupta, A. Rajeswaran, and V. Kumar, "Real world offline reinforcement learning with realistic data source," arXiv preprint arXiv:22 00.779 2022.
[51] A. Kumar, J. Hong, Singh, and S. Levine, "Should run offline reinforcement learning or behavioral cloning?," in International Conference on Learning Representations, 2022.
[52] Gebru, J. Morgenstem, B. Vecchione, J. W. Vaughan, H. Wallach, H. D. Iii, and Crawford, "Datasheets for datasets," Communications of the ACM, vol. 64, no. 12, pp. 86-92, 2021.
[53] A. Singh, A. Yu, J. Yang, J. Zhang, A. Kumar, and S. Levine, "Cog: Connecting new skills to past experience with offline reinforcement learning," in Conference on Robot Learning, PMLR, 2020.
[54] H. Hu, Y. Yang, Q. Zhao, and C. Zhang, "The provable benefits of unsupervised data sharing for offline reinforcement learning." " in International Conference on Learning Representations, 2023.
[55] A. Li, B. Boots, and C.-A. Cheng. "Mahalo: Unifying offline reinforcement learning and imitation learning from observations." arXiv preprint arXiv:2303.17156, 2023.
[56] T. Yu, A. Kumar, Y. Chebotar, K. Hausman, C. Finn, and S. Levine, "How to leverage unlabeled data in offline reinforcement learning," in International Conference on Machine Learning, pp. 25611-25635 PMLR, 2022.
13

---

[57] Li, Hu, H. Xu, Liu, X. Zhan, Q.-S Jia, and Y.-Q. Zhang, "Mind the gap: Offline policy optimization for imperfect rewards," " in International Conference on Learning Representations. 2023.
[58] X. B. Peng, A. Kumar, G. Zhang, and S. Levine, "Advantage-weighhed regression: Simple and scalable off-policy reinforcement learning," arXiv preprint arxiv:1910.00177, 2019.
[59] K. Zoina, A. Novikov, K. Konyushkova, C. Gulcehre, Z. Wang. Y. Aytar, M. Denil, N. de Freitas, and S. Reed, "Offline learning from demonstrations and unlabeled experience," arXiv preprint arXiv:2011. 13885, 2020.
[60] H. Xu, X. Zhan, H. Yin, and H. Qin, "Diseriminntor-wciihhdd offline imitation learning from suboptimal demonstrations," in International Conference on Machine Learning, pp. 24725- 24742, PMLR, 2022.
[61] G.-H. Kim, S. Sco, J. Lee, W. Jeon, H. Hwang, H. Yang, and K.-E. Kim, "Demodice: Offline imitation learning with supplementary imperfect demonstrations," in International Conference on Learning Representations, 2021.
[62] 7. Zhu, K. Lin, B. Dai, and J. Zhou, "Off-policy imitation learning from observations," Advances in Neural Information Processing Systems, vol. 33, pp. 12402-12413, 2020.
[63] B. Dai, O. Nachum, Y. Chow, L. Li, C. Szepesvári, and D. Schuurmans, "Coindice: Off-policy confidence interval estimation,"" " Advances neural information processing systems, vol. 33, pp. 9398-9411 1, 2020.
[64] R. Kidambi, J. Chang, and W. Sun, "Mobile: Model-based imitation learning from observation alone," Advances in Neural Information Processing Systems, vol. 34, pp. 28598-2866 1, 2021.
[65] S. Yue, G. Wang, W. Shao, Z. Zhang, S. Lin, J. Ren, and J. Zhang, "CLARE: Conservative model- based reward learning for offline inverse reinforcement learning," in International Conference on Learning Representations, 2023.
[66] B. D. Ziebart, A. L. Maas, J. A. Bagnell, A. K. Dey, et al., "Maximum entropy inverse reinforcement learning.." in Proceedings of the AAAI Conference on Artificial Intelligence. vol. 8, pp. 1433-1438, Chicago, IL, USA, 2008.
[67] M. Smith, L. Maystre, Z. Dai, and K. Ciosek, "A strong baseline for batch imitation learning." arXiv preprint arXiv: 022802788 2023.
[68] Y. Chen, X. Zhang, K. Zhang, M. Wang, and X. Zhu, "Byzantine-robee online and offline distributed reinforeement learning." in International Conference on Artificial Intelligence and Statistics, pp. 3230-3269, PMLR, 2023.
[69] R. Yang, C. Bai, X. Ma, Z. Wang, C. ZZhang, and L. Han, "Rorl: Robust offline reinforcement learning via conservative smoothing," in Advances in Neural Information Processing Systems, 2022.
[70] L. Shi and Y. Chi, "Distributionally robust model-based offline reinforcement learning with optr-oltimal sample complexity." arXiv preprint arXiv:2208.088 057677 2022.
[71] Z. Liu, Z. Guo, Y. Yao, Z. Cen, W. Yu, T. Zhang, and D. Zhao, "Constrained decision transformer for offline safe reinforcement learning," arXiv preprint arXiv:2302.07351,200 2023.
[72] N. Polosky, B. C. Da Silva, M. Fiterau, and J. Jagannath, "Constrained offline policy optimiza- tion," in International Conference on Machine Learning, pp. 17801-17810, PMLR, 2022.
[73] W. Xu, Y. Ma, K. Xu, H. Bastani, and O. Bastani, "Uniformly conservative exploration in reinforcement learning," in International Conference on Artificial Intelligncee and Statistics, pp. 10856-10870, PMLR, 2023.
[74] S. Lee, Y. Seo, K. Lee, P. Abbeel, and J. Shin, "Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble," Conference on Robot Learning, pp. 1702-1712, PMLR, 2022.
14

---

[75] W. Yu, C. K. Liu, and G. Turk, "Policy transfer with strategy optimization," in International Conference on Learning Representations, 2019.
[76] H. Van Hasselt, Y. Doron, F. Strub, M. Hessel, N. Sonnerat, and J. Modayil, reinforeeme learning and the deadly triad," arXiv preprint
2018.
[77] R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction. MIT press, 2018.
[78] M. Riedmiller, "Neural fitted q iteration-firs, experiences with data efficient neural reinforce- ment learning method," in Machine Learning: ECML 2005: 16th European Conference on Machine Learning, Porto, Portugal, October 3-7, 2005, Proceedings 16, pp. 317-328, Springer, 2005.
15

---

Contents
Introduction
Background
3
Why Offline RL can Learn Right Behaviors from Wrong Rewards
Intuitions for Survival Instinet and Positive Data Bias
Survival Instinet
3,3
Positive Data Bias
3.4
Summary: Offline RL Doubly Robust and Intrinsically Safe
Experiments
Robustness Mis-specilied Rewards
Locomotion Tasks from D4RL
4.1.2
Goal-oriented Manipulation Tasks from Meta- World
4.2
Inherent Safety Properties Offline RL Algorithms
10
5
Concluding Remarks
10
Limitations and Broader Impacts
17
Related Work
18
Experiment Details
19
Benchmark Datasets
20
Implementation Details
Hyperparameters
21
Episode Lengths for hopper and walker2d Tasks
22
C.5
Effect of Removing Terminal Transitions
22
C.6
Results Safety Gymnasium Datasets
23
Ablation Study COL Architecture
24
C.8
Compute Usage
24
Technical Details of Section 3
34
D.I
Definitions
34
D.2
Formal Statement of the Main Theorem
35
D.3 Survival Instinct
36
D.4
Implicit Data Bias
37
D.4.1
Proof of RL setup in Proposition|
38
D.4.2
Proof of IL setup in Proposition
39
D.4.3 Proof of Length Bias in Proposition
39
D.5
Proof of Theorem 2
40
16

---

D.6
Technical Details of Constrained MDP
41
Tinite-Horizon Version
42
Main Theorem for Finite-horizon Problems
43
Length Bias
44
Goal-oriented Problems
44
Algorithm Specific Results
45
F.1
ATAC
46
E.1
Notations
46
F.1.2
Analysis
47
F2
PSPI
48
E3
ARMOR
49
CPPO
50
VI-LCB
51
F.6
PPI and POI
52
MOReL and MOPO
54
E771
MORel
54
17.2
MOPO
56
E7.3
Proof of Technical Lemma
57
Grid World
57
A Limitations and Broader Impacts
Limitations We would like point out that results do not suggest that offline RL can always learn with wrong rewards (since the data may not have the needed positive bais). This is shown in our experimental results. In addition, our theoretical analysis makes certain simplifications (e.g., ignoring the optimization difficulty of finding a survival policy), which may not hold in practice. We experimented with limited number of offline RL algorithms and benchmarks. It would be interesting to see if similar experimental results can be obtained by, e.g., model- offline RL algorithms and other types of dataset such as the ones which use image observations. Future work may also explore approaches to quantifying positive data bias without having to run offline RL algorithms.
Broader Impacts Our discovery might shed light on potential positive impact of offline RL. By survival instinct, offline RL may be able to train sequential-decision policies that have good behaviors without needing to collect negative data. This is different from the common belief that RI can learn to behave well only if has seen both positive (success) and negative (failure) data. This ability to learn from one-sided data is important especially for applications where collecting negative data is unethical, e.g., learning not to produce hateful speech by first collecting hateful speech, learning not to harm patients in medical applications by first harming some, etc.
On the flip side, also by survival instinct, offline RL can be prone to existing societal (e.g., gender, racial) biases in data, and, moreover, such bias cannot be easily corrected by relabeling data with different reward. As result, when using an offline RL algorithm, more strategic thinking on data collection might be needed. We encourage researchers and practitioners to collect datasets using methodology such as those proposed in 52 and provide details such as how data was collected and cleaned that users can assess whether it is appropriate to train offline RL with these datasets.
17

---

B Related Work
Offline RL Offline RL studies the problem of return maximization given an offline dataset. Offline
RL algorithms can be broadly classified into model- based approaches, e.g., [22 33 14
35]
34]. and
model-free approaches, e.g.. [12 29 7 g Since the agent needs to learn without online data
collection, offline RI. becomes more challenging when the offline dataset does not provide good coverage over the state-action distribution of all feasible policies. There have been two common strategies to handle insufficient coverage, behavior regularization approaches, e.g., [2 30 R. which restricts the learned policy to be close to the behavior policy, and value regularization approaches, e.g., 12 13 7 2 34] which provide a pessimistic value estimates for the learned policy. Both approaches can be viewed as optimizing for performance lower bound of the agent's return.
Offline RL commonly assumes compatibility between offline dataset and MDP, ie., the offline dataset should be generated from the task MDP of interest. Our work is distinctive in the offline RL literature as we study the behavior of existing offline RL algorithms when data reward differs from the true MDP reward. Existing tools for analyzing offline RL are not directly applicable to our setting: policies with good performance under data reward, as guaranteed by such tools, do not necessarily attain high return under the true reward. Our novel analysis is made possible by studying the properties of offline data distribution, an aspect mostly neglected by existing work in the literature.
Offline RL with imperfect reward In the offline RL literature, there is line of work studying scenarios where the reward is missing in subset of the dataset (53/56) when the data reward is misspecified [57]. These approaches propose to construct new reward function and apply offline RL algorithms on the relabeled dataset. For example, [53 56) label the missing reward with a constant
minimum reward. [55] [54] learn pessimistic reward function.
57 optimize for reward function
which minimizes the reward difference among expert data.
Different from this line of work, we are interested in running offline RL algorithms using the misspecified reward it is. We show both theoretically and empirically that offline RL algorithms can produce well-performing policies with rewards different from the true reward as long as the underlying data distribution satisfies certain conditions. It can be an interesting future work to combine these techniques with our analysis to explore the robustness of offline RL algorithms under the class of learned reward functions.
After paper submission, we learned that (25] observed a similar robustness phenomena of offline RL. However, the authors did not provide complete explanation. They found that offline RL algorithms such as AWR (58) empirically can learn well with random or zero rewards on D4RL locomotion datasets. For the expert datasets,, they conjectured that this robustness may be due the fact that offline RI. algorithms hehave like imitation learning on expert data, but they did not provide details of why offline RL algorithms have this robustness behavior on other datasets. Here we provide formal theoretical proof to show that this robustness is due to the interaction between the survival instinct of offline RL and an implicit positive data bias. Our results support 251's conjecture on the expert dataset; we show that expert datasets generated by an optimal policy can theoretically be proved to be infinitely positively biased, and we also empirically estimate the positive bias on the D4RL datasets in Fig. []] which agrees with the theoretical prediction. Our results further provide conditions for positive data bias beyond the expert data scenario, such as the length bias in Section 3 These new insights show why offline RI. algorithms can learn good policies with wrong rewards, on datasets collected by non-expert suboptimal policies. Finally, we present strong empirical evidence to show this robustness phenomenon happens in large number of existing benchmarks with multiple offline RL algorithms.
Offline IL Offline imitation learning (IL) can be considered as a special case of offline RL when reward missing from the entire dataset, or when the agent has the largest uncertainty about the reward [2]. In offline II. the learner is instead given the information on whether a transition is generated by an expert. The goal of offline IL is to produce policy that has comparable performance
with the expert. [59] learns a discriminator reward to classify expert and non-expert data, and apply offline RL algorithms with the learned reward. [60] uses the discriminator as the weight for weighted
proposes DICE 63 -style algorithms to minimize the divergence between the state-action or snete-(next state) occupancy measure of the learner and the expert. [15 64
behavior cloning. [6 16 62
minimize state-action or state-(next state) distribution divergence under a pessimistic model.
65
18

---

extends maximum entropy inverse RL 666 to an offline setting. [67 proposes to run offline RL under an indicator reward function on whether the data is expert data. [55] learns a pessimistic reward
function where the expert data labeled as the maximum reward.
Our experiments with zero reward function is similar policy being the expert. However, we show that offline RL, algorithms can sometimes achieve significant performance gain over behavior and behavior cloning policies, which is a phenomenon that cannot be explained by existing analysis in offline IL. We provide an explanation to this puzzle: in Appendix D.4 we show that the data distribution is oc-positively biased (according our definition) for offline RL, when the data is generated by the optimal policy of the true MDP. This means that an admissible offline RL algorithm can achieve near-optimal performance with any reward function from its corresponding admissible reward class in this case. We empirically show that in Section 4
offline II. setting with the behavior
Robust offfine RL Robust offline RL T7 68 18
20 aims to develop new offline
RL algorithms for cases when the compatibility assumption does not hold, i.e., the offline dataset
may not be generated from the task MDP. Among this line of work, [21 19 20 70 study the
robust MDP setting, where the dataset can be generated from any MDP within 8-ball of nominal
MDP. An adversary can choose any MDP from the 5-ball when evaluating the learned policy. (T7] considers when the adversary can modify fraction of transition tuples in the offline dataset, while 18 considers when up to K trajectories can be added in or removed from the original dataset. (68] uses the formulation of Byzantine-robust: when multiple offline datasets are presented to the learner, among them an a fraction are corrupted. [69] assumes the dataset is generated from the true MDP, but the state presented to the policy during test time can be corrupted up distance.
Our work is fundamentally different from this line of work that we study the inherent and somewhat unintended robustness of offline RL algorithms. We show that, despite originally designed under the compatibility assumption, offline RL. algorithms show strong robustness against perturbation of reward. Another salient difference is that robust offline RL typically requires an upper bound on the size of perturbation to the true or nominal MDP, and most of proposed algorithms assume knowledge of this upper bound. However, our work does not make assumptions moreover, the algorithms we study are not even aware of the existence of such perturbation. Indeed, empirically show that offline RL algorithms, without any modifications, can succeed even when data reward is a constant or the negation of the true reward. Additionally, in this work, we consider the reward perturbation while most work in robust offline RL focuses more dynamics perturbation.
the size of perturbation and,
Constrained and safe offline RL Our work is related to the literature of constrained and safe offline RL 222
7 24 (72). In constrained offline RL, the agent solves for CMDP given an offline dataset {(s, a, r, C, (ssss) s')}- The goal of constrained offline RL is to produce a policy which maximizes the expected return while ensuring that constraint violation is below a given upper bound. [2 uses an offline policy evaluation oracle to solve for the Lagrangian relaxation of the CMDP. 23 constructs a pessimistic estimation of value and constraint violation. [24] 772 proposes DICE 66
23
style algorithms to solve for the optimal state-action occupancy measure. 771 proposes a variant of decision transformer [10] capable of producing policies for different constraint violation upper bounds during test time.
Our work, instead, considers different scenario where the constraint is only given implicitly through the support of the offline dataset. We assume that any transition in the dataset satisties the constraint, which can be achieved through intervention during data collection. We show that offline RL can inherently produce approximately safe policies due to its survival instinct. Although our assumption seems more restrictive as in that we require a constraint violation-free dataset, we argue that our assumption is more practical in many safety critical scenarios: compared to executing unsafe actions, it is perhaps more reasonable intervene before constraint violation happens. We believe that our findings open up new possibilities for applying offline RI. as a safe RI. oracle, which can be especially promising in offline-online RL setup 73 74.
C Experiment Details
In the experiments, we train a total of around 16k offline RI. agents (see Appendix CarendixCC.eee for details). In this appendix, we provide details of experiment setup, implementation of algorithms, hyperparameter
19

---

(a) D4RL !
(b) Meta- World |
Figure 7: We study offline RL with wrong rewards for a variety of tasks: (a) three locomotion tasks from D4RL M: hopper (top left), walker2d (top right) and halfcheetah (bottom). the figures are from (75) (b) 15 goal-oriented tasks from Meta-Worts World M. The 15 tasks are composed of the 10 tasks from MT-10 (top 2 rows) and the 5 testing tasks from ML-45 (bottom row).
tuning. as well as additional experimental results and ablation studies. At the end of the appendix, we include Table 13 and Table 14 which contain the actual numbers used in generating Fig. and Fig. o
C.1 Benchmark Datasets
In the experiments, we consider three tasks (15 datasets) from D4RL [1. 15 datasets collected from the World [II] benchmark, and 16 datasets from SaferyGymnasium [49]. The D4RL and Meta-World tasks are visualized in Fig- Below provide more details these tasks.
D4RL We consider three locomotion tasks from D4RL [1, hopper, walker2d and halfcheetal (see Fig. 7. For each task, we use five datasets of different behavior policy quality: random, medium, medium cediummreppay: medium expert, and expert.
hopper:
The goal of hopper is move forward as fast as possible without falling down. An episode terminates when either of the three happens: 1) the height of the hopper is below threshold, 2) the body of the hopper is too tilted, or 3) the velocity of any joint is too large. The original true reward function (which has been used to train online RL agents) is
1 1 Ax - 0.001|/7||3
Thopper 1 [hopper is alive]
A
where Az is the displacement in the forward direction, and is the applied torque. The true reward function is designed such that the hopper can receive bonus when staying alive and moving forward. We show that, for a few hopper datasets, offline RL agent can succeed without such a carefully designed reward, and even with the negation of the true reward.
walker2d: The walker2d task is similar to hopper, except that the walker2d agent has higher degree of freedom. We observe similarly that offline RL can learn good behaviors without reward that is specifically designed to encourage staying alive and moving forward.
halfcheetah:
The halfcheetas task differs from the two tasks above since episodes here always have the same length and never terminates early. The true reward of half cheetah is
Ar 0.1|/7|12. At
I Thalfcheetah
Since the half datasets (with the exception of the expert dataset) do not have positive bias, we find that offline RL agent is sensitive to reward quality.
Meta-Worke We consider 15 goal-oriented manipulation tasks from [1 see Fig. We use the 10 tasks from the MT-10 set and the 5 testing tasks from the ML-45 set. Each Meet--orrh task comes with a hand-crafted reward function that encourage progress toward task completion, e.g.. when the end-effector grasps the target object, when the object is placed at the goal, etc. We refer readers to [1] for the specific reward functions used for each tasks. We make few modifications to the Meta- World environment: /) we shift and scale the true reward such that the agent receives
We use the v2 version of all three tasks.
20

---

reward of range 1-1, OJ rather than [0, 10] of the original environment, 2) we terminate an episode and mark the last state with terminal flag when the goal is achieved. We show that, data collected by goal-oriented tasks can have a positive bias, and an offline RI. agent can make use of such bias to learn the right behavior even with wrong rewards. We would like clarify that the data generated by the original Meta-World environment (with no terminal signals) does not have positive bias in general, since all trajectories, success or failure, would have a same length of 500.
We experiment with 16 datasets from offline SafetyGymnasium (49 The 16 datasets are given by two embodiments, point and car, with 4 tasks, button, circle, goal, and push, each with 2 difficulty levels.
It is worth noting that [49] focuses on learning from dataset which contains both safe and unsafe behaviors, while our results (Corollary 2 focus on offline RL's ability to produce safe policies given dataset that only contains safe states. Therefore, we use naïve filtering strategy of removing all transitions with non-zero cost and then run offline RL algorithms on the filtered datasets.
Many algorithms benchmarked in [49] use a cost target, which sets the maximum total cost allowed by safety. [49] uses three different cost targets, {20, 40, 80), and reports the normalized total reward and cost for each algorithm averaged over the three targets. By contrast, standard offline RL algorithms do not use such a cost target. We report results of offline RL algorithms according to an "effective cost target" of 34.29, which is similar to how BC-All results are presented in [49
C.2 Implementation Details
We use the implementation of ATAC and PSPI from make a small modification to the ATAC and PSPI policy class. Instead a tanh-Gaussian policy, we use a scaled lanhiGGassian policy, i.e., c- tanh(N (m, Σ)) with e 2 1. We find scaled tanh-Gaussian policies to be more numerically stable when the data has a lot of on-boundary actions, i.e., actions that takes the value of either -1 or 1 on any dimension. For PSPI, we let the critic f to minimize for fs, π) for all states from the dataset (rather than all initial states). We find it to provide better results when the effective horizon given by the discount factor is smaller than the actual episode length. We use the implementation of IQL and BC from httss//ggtuu.oo/ggwwoo//////// httpg/gtom////L-PPy//.. Plrta//oh// and the implementation of CQL from https::/ggthub..oomoo coonyyuu---no geng/CQL For decision transformer, we use the implementation from https::/gittubbbc com/ wttpauu..oooo//l/ccciiiiioooo traiionnttnsffomee We feed the decision transformer with wrong rewards (consistent with data reward) during testing.
We
C.3 Hyperparameters
We use the default values (ones provided by the original paper by the implementation use) for most of the hyperparameters for all algorithms. For each algorithm, we tune one or two hy- perparameters (with 4 combinations most) which affect the degree of pessimisn The values of hyperparamters of all algorithms are given in Table 4 where the choices of hyperparameters used in tuning are highlighted in blue. The tuned hyperparameter values for all experiments and all algorithms are provided in Table 1
For D4RL and Meta- World experiments, we tune hyperparameter per each dataset and per reward label. During tuning, we evaluate each combination of hyperparameter(s) for 2 random seeds for D4RL and 3 random seeds for Meta-World. We choose the best-performing values and report the results of these hyperparameters over 10 new random seeds (not including the tuning seeds). We find the D4RL scores for the true reward to be close what is reported in the original papers.
For SafetyGyynasium. we follow procedures from 49 and report results of the best hyperparameters over 3 random seeds. We take the safe agent with the highest cumulative reward if there is at least one safe agent, and take the agent with the lowest cumulative cost if there are no safe agents.
For decision transformer, do not tune hyperparameters since it has notion of pessimism.
21

---

Hyperparameter
Value
Bellman consistency coefficient 3
{0.1, 100
Number of hidden layers
Number of hidden units
256
Nonlinearity
ReLU
Policy distribution
scaled tanh-Gaussian
Action scale
1.0 for D4RL and SafetyGymnasium
1.2 for Meta World
Fast learning rate Tau
5 X 10
Slow learning rate 7ysoww
10-7
Minibatch size [mmin]]
256
Gradient steps
10
Warmstart (BC) steps
10a
Temporal difference loss weight w
0.5
Target smoothing coefficient
0.005
Discount factor
0.99 for D4RL and SafetyGymnasium
0.998 for Meta-World
Minimum target value Visin Maximum target value Viuu
2 min(-1, (,,n--1,Fmmi
I-7 ma Fmx))
Table 1: Hyperparameters for ATAC and PSPI
Hyperparameter
Value
Inverse temperature 3
(0.3,3)
Expectile т
(0.7,0.9)
Number of hidden layers
3
Number of hidden units
256
Nonlincarity
ReLU
Policy distribution
Gaussian
Learning rate
3 x 10
Minibatch size [Dmini
256
Gradient steps
10
Target smoothing cocfficient a
0.005
Discount factor Y
0.99 for D4RL
0.998 for Meta-Workd
Table 2: Hyperparameters for IQL
C.4 Episode Lengths for hopper and walker2d Tasks
We show the episode lengths for hopper and walker2d tasks Fig. g cies learned by ATAC and PSPI can consistently keep the agent from falling down for a sig- nificant longer period than the behavior policies. The only exception is when running PSPI on hopper medirm--emm----me expert dataset with random reward. This is potentially due to the fact that only 2 seeds for hyperparameter tuning. The average normalized scores and episode lengths during tuning 101.1 and 925.0, respectively, higher than the final values of 58.8 and 546.0. We observe that DT policies also tend to keep the agent alive for longer number of steps than behavior policies.
We observe that poli-
For IQL and CQL, it is relatively rare for them to achieve long episode lengths when trained with wrong rewards. IQL and CQL, when using negative reward, can keep the agent from falling for longer in hopper-medine and hooper-medium-eee eeppert datasets. We would like to note that our hyperparameters are tuned based on normalized scores rather than episode lengths. The statistics of episode lengths are included in Table 5
C.5 Effect of Removing Terminal Transitions
For hopper and walker2d tasks, we remove the terminal transitions, i.e., the transitions where the hopper or walker falls, from the datasets with modified rewards. This is to satisfy the safety
22

---

Hyperparameter
Value
Critic pessimism coefficient
10 100)
Number of hidden layers
2"
Number of hidden units
256
Nonlinearity
ReLU
Policy distribution
tanh-Gaussian"
Learning rate v)
10
Minibatch size [Pamn]
256
Gradient steps
10
Target smoothing coefficient
0.005
Discount factor 7
0.99 D4RL
0.998 for Meta-World
Table 3: Hyperparameters for CQL. "We include an ablation study of CQL with 3 hidden layers and scaled tanh-Gaussian policy, i.c. the same architecture as ATAC and PSPI, Appendix C.7
Hyperparameter
Value
Number of hidden layers
3
Number of hidden units
128
Nonlinearity
ReLU
Context length
20
Dropout
0.1
Warmup steps
2 10
Learning rate 1)
5 x 10-4
Minihatch size (Dminil
256
Gradient steps
7 10 for DARL
5 x 10 for Meta-Woon World
Gradient norm clip
0.25
Weight decay ceturn-to-goal conditioning
10-4
Return randomly sampled from the top 10%
of trajectory returns in the data, w.r.t. the data reward
Table 4: Hyperparameters for the decision transformer (DT)
condition in Corollary A terminal signal implicitly adds an absorbing but in this case, unsafe, state into the data distribution μ.. This means that staying within data support in long term would not provide safety, as the agent can stay within support by falling. For D4RL datasets particular, need to remove terminal transitions (rather than just the terminal flag) since terminal transitions in the original datasets do not contain valid next states !
This, however, means that there are two underlying variables: /) the data reward, and 2) whether terminal transitions are removed, when comparing offline RL with wrong rewards and offline RL on the original dataset. To separate the effeec of two variables, study the performance of offline RL algorithms with the original true reward but the terminal transitions removed. The results are listed in the No-terminal column in Table 13 We observe that ATAC and PSPI with the original true reward show similar results whether the terminal transitions are removed. IQL and CQL with true reward, interestingly, does significantly worse when the terminal transitions removed. This is potentially due to the increased instability of target network 76 77
when there are no terminal signals.
C.6 Results on Safety Gymnasium Datasets
In T we present results on point datasets and car from Safety Gymnasium [49 For baselines. we select the best none-BC Algorithms from [49]: the best-performing offline safe RL agent selected
'This is also why we mark goal completion as terminal Meta- Meta-World experiments. For goal- tasks, terminal signal upon task completion implicitly creates an absorbing goal state in the data distribution.
"The authors D4RL datasets set next state to a place-holdes value when transition marked as terminal.
We do not study decision transformer (DT) in this ablation since DT does terminal signals.
23

---

3030
ATAC
530
hopper
walker2d
1000
PSPI
hopper
walker2d
behavior
BC
original
zero
random
negative
1000
IQL
mid-4ap
hopper
walker2d
CQL
rand
med
mad Htr
rand
med-resty
mideep
400
hopper
walker2d
3000
500
12nd
FCP
CAP
010
reapy
CND
hopper
walker2d
Figure 8: Episode lengths for hopper and walker2d datasets from D4RL. The mean and standard error of episode lengths are computed across 10 random seeds. For each random seed, we evaluate the final policy of each algorithm over 50 episodes. Note that the hyperparameters are tuned based on normalized scores rather than episode lengths.
using the same criterion (safe agent with highest reward if there is at least one safe agent, and agent with lowest cost otherwise). We observe that ATAC and PSPI with naive filtering strategy can achieve comparable performance with state-of-the-au offline safe RL algorithms benchmarked [49]
C.7 Ablation Study CQL Architecture
Our results are generated by CQL with the architecture in Table 3 which is given by the original implementation. In this study, examine the performance of CQL when using the same architecture as ATAC and PSPI, i.e., with 3 hidden layers and scaled tanh-Gaussian. The normalized scores for D4RL datasets and success We find the performance of CQL with 3 hidden layers and scaled lanh-Gaussian policy, denoted as CQL3, to be similar with the performance of CQL with the original architecture.
for Meta-Was World tasks are included in Table Table respectively.
C.8 Compute Usage
For ATAC, PSPI, IQL, CQL and BC, each run takes around 5.5 hours on an NC4as T4_v3 Azure virtual machine for D4RL, Meta-World, and SafetyGymnasium experiments. For D4RL experiments (including ablation studies), we use total (algorithms) x (D4RL. datasets) x 5 (reward types) x (hyperparameters) x 2 (seeds) 3000 runs for hyperparameter tuning and

---

Dataset
Behavior
BC
Algo
Original
Zero
Random
Negaiiee
Notterminal
ATAC
19.3=130
714.22 120.0
552.7
567
hopper random
PSPL
99.2=622
354.--113000
113.9-13100
749.14:2245
517.3+142.9
22.1
93.2:1..0
IQL
108.744.
137.1=18.0
12.8:0.11
12.1=2.7
14.5
48.4+12.
41.8+2099
28.4+3.8
6.0:::000
16.7=
DT
245.248.
2.9.72..66
103.2-2.0
295.1 32..8
ATAC
808.4=15.3
099.==0.3
998.9=10
997.6=199
840.2=199
hopper medium
PSPI
906.8 16.9
897.7=16.2
993.0=6.4
907.8::6600
833.9=29.7
457.2
$40.5 1.2
IQL
571.2 8.5
538.7
594.8 20.3
631.4121.7
446.4165.88
351.1=35.9
594.7-1-23.9
447.2-97.4
962.2-17.00
503.7+123.0
DT
630.4.16.7
641.1
638.8 6.1
633.4.19.1
AIAC
1000.0+0000
904.3:2:4.3
309.4:35.3
909.6
10000000.0
hopper medium replay
PSPI
1000.0:10.00
1000:0::0.0
995.9=3.9
1000.0::0.0
1000.0:10.0
197.0
74.
IQL
598.5 19.8
222-5
22.3.1..2
65.55
500.9153..6
CQL
801
38.2
24.9::2.6
217.5:: 123.7
153.8=91
116.4293.2
DT
934.5
13.6
342.31.33.4
836.2.1.20.6
825.6.1.1.2
ATAC
93..1=1311
10.5
73..7
19
885.1 336.1
hopper medium expert
PSPI
974.516.1
848.01144..
546.01884.4
968.914.2
969.8.1..3
622.2
541.84:1.
IQL
980.445.11
535.413.4 3.4
581 7.77
883.2-28.5
735.3-162.8
COL
800,8=314
516.6472.77
633.1+87.9
901.9=55.7
489. 103.0
DT
817.4=22.5
813.8:222.0
8178 17.0
852.1=1744 17.4
AIAC
998.810.5
997.3=1111
9.5
998.811.2
999.910011
hopper expert
PSPI
9.
998.8.100
998.4-0..8
995.8.117
3.000
972.8
998.410.66
TQL
993.3:11.7
607 107.7:1:122
658 91
439 130.
102.0
238.3
906.7=25.9
1.9
1..9
603.2++10.00
1.27
DT
963.1
966.0=2499
972.92299
979.4:12.99
227.6=10.2
23L0=110
14:46=10:4
214.8::922
189.02
walker2d
PSPI
221.1 13.7
143.81100.5 10.5
165.7=19.8
135.8:11.2
135.9:10.5
81.211..1
240.5.13.8
83.71:4.9
79.91121.4
182.341.66
CQL
7829
56.0::23.11
188.7=50.9
34.7=5.1
37.2+ 10.2
228.447.6
210.9=:10.2
75.2+11.2
203.3=11.11
ATAC
17699=17.6
999.710.3
997.8=1.5
999.8:1.22
990.215599
walker2d
PSPI
989.111..2
998.2.1 1.7
1000.0110.0
973.0.14.9
991.7.14.7
895.3422.4
930.1
331.1-37.33
976.5-2.44
739.
CQL
991.0+3.00
105.3+344.7
86,9+:32.7
130.4=35.0
968.7+3.3
955.6=7.5
951.5=477
9.4.9-1699
ATAC
923.1=16.4
967.8 21.
968.41199 19.8
817.4=3.66
985.8
walker2d medium replay
PSPL
314.4=30.4
997.1
995.3
931.:=33.2
997,341122
276.3
312.715.8
1:2.5
211.11:3..1
119.5
41.81:.4
CQL
14.91=14.9
71.35:27.5
35.1=6.9
72.8++23.3
87.5+26.77
776.1
10.7
14.
74.88=17.8
929
179.421539
9994=00.6
1000:0:0::0
960.
walker2d mediumm expert
PSPL
1000.01:0000
918. 918.9:121.3
1000.00:0.00
960.2:1..3
987.9199.X
976.911..5
999.6.0.44
989.0129
CQI.
9.8.411.5
55.811.0.44
44.34 17.4
$14.5::11..3
12.6
1000.0110.0
9.9
1000.010000
99..511.44
1000.0+0000
100000:0000
993.6=3.6
1000:0::::0
1000:0:0000
walker2d expert
PSPL
1000.0:10.0
1000.0:10.0
951.7-22.1
997.4:11.7
1000.0:10.0
1000.0.1000
1000.010.00
29.0
CQL
1000.0+10.00
1000.0-10.00
1000.0+:0.00
1000.0:10.0
998.3+1.6
1000.0+:..0
1000.0=00.0
1000.0+0..0
1000.0+:0.0
Table Episode lengths for hopper and walker2d datasets. The mean and standard error episode lengths are computed across 10 random seeds. For each random seed, we evaluate the final policy of each algorithm over 50 episodes.
(algorithms) X 15 (D4RL datasets) x 5 (reward types) 10 (seeds) = 4500 runs for generating the final results. For Meta-World experiments, use 5 (algorithms) 15 (Meta-World datasets) x 4 (reward types) x (hyperparameters) x 3 (seeds) 3600 runs for hyperparameter tuning and 6 (algorithms) x 15 Adeta-World datasets) x 4 (reward types) 10 (seeds) 3600 runs for gen- erating the final results. SameetyGymnsiumm experiments require a total of 2 (algorithms) x 16 (SafetyGyymnasium datasets) (hyperparameters) x (seeds) 384 runs. This amount to 15084 runs or 82962 hours of training on NC4as_T4, T4__3 v3 virtual machines.
For DT, we do not tune any parameters at large scale. Each run takes around 2.5 hours NC6s_vd 1166s_222oo ND6s) Azure virtual machine. train total 30 (D4RL and WerldWorld datasets) x 4 (reward types) X 10 (seeds) = 1200 DT agents.
25

---

ATAC
PSPI
Best None-BC Algorithm from
Task
reward
cost J
reward t
cost
reward
cost
algorithm
PointButton
0.08
0.82
0.10
1.04
0.13
1.35
COpiDICE 24
PointButton2
0.14
0.89
0.19
1.29
0.15
1.51
COpiDICE 24
PointCirclel
0.40
2.79
0.34
2.39
0.59
0.69
CDT 71
PointCirele2
0.61
4,45
0.55
2.84
0.64
1.05
CDT 71
PointGoall
0.58
0.94
0.59
0.90
0.71
0.98
BCQ-Lag
PointGoal2
0.55
2.30
0.45
2.55
1.31
CPQ 2
1
0.18
0.51
0.20
0.85
0.33
0.86
BCQ-Lag 22
PointPush2
0.10
0.68
0.10
0.81
0.23
0.99
BCQ-Lag 23
CarButton
-0.06
1.25
-0.08
1.11
0.21
1.60
CDT
CarButton2
-0.11
1.22
-0.08
1.04
0.13
1,58
CDT 71
CarCirelel
0.66
5.51
0.64
5.92
0.60
1.73
CDT
CarCircle2
0.64
5.96
0.64
5.99
0.66
2.53
CDT [71
CarGoall
0.50
0.99
0.41
0.78
0.47
0.78
BCQ-Lag [2
CarGoal2 0.24
1.00
0.24
1.05
0.25
0.91
COptiDICE 24
CarPushl 0.33
0,96
0.32
0.75
0.31
0.40
CDT 711
CarPush2 0.10
0.95
0.11
0.81
0.09
1.07
COptiDICE 24
Table 6: Results on point and car datasets from offline SafetyGymnasium
49 over random seeds.
An agent is considered safe the cumulative cost is no larger than 1. Unsafe agent with cumulative cost more than 1 (total cost 34.29)
Cumulative reward and cost are normalized as described in 449
shown in gray.
Datase
Algo
Original
Zero
Random
Negaiive
Ne-iemmnnal
hopper-eandom
COL
3.0:0.77
20:11.
182002
0,7=0.0
1.240.3
CQLs
11.24:2.
2.0:0.
7.74:3.5
3.9=29
10.14:3.2
hopper medium
COL
85.0.13.77
55.0.111.7
36.9.19.8
89.0.11.5
46.3
CQL
48.0:1-2.2
61.7411.2
75.8:16.9
hopper medium replay
COL
79.71:3.7
1.910.1
7.6.1..7
5.8
4.4.1..8
CQL.s
1.0
1.8:1000
188:0.00
1.620.22
1.8:10.0
mupper-medium-eee expert
CQL
49.718.1
60.71994
68.6:16.1
47.4111..D
CQLs
104.7:1144
40.6+
17.3:99.0
48.2488.7
30.0::-9.5
walker22
CQL
4.0.1.18
09100.
4.3.1.44
0.4.1..0
CQL.s
4.8.1-022
1.4.0.88
3.3:1122
0.9
.0240.11
walker2d mediumm
COL
81.4:10.3
2.0.1.99
1.2.0.7
14.5:1..4
2.1:0088
CQLs
81.0:044
5.0:4.77
-0200000
13.347.7
0.1::0.3
walker2d-medium--ee
CQL
74.1::122
0.54003
0.240.11
0.2=033
0.410.4
CQLs
7..8:11.33
-0220.11
0.140.1
0.1-033
0.34-0.3
walker/d-meddum-expee
CQI.
109.3110.3
0.8:00.5
0.120.2
28.019.11
40210.0
CQLs
108.0+0.6
-0.2+0.1
-0.3.0.1
26.8=110
-0200.0
halfcheetab-random
CQL
3071:0.6
10000.8
-1400.3
-5.410.3
CQLs
31.3::0.6
0.54:0.
1.940.1
-7.0::04
halfchestal-mediing
CQL.
57.211.5
43.5:0.11
42.740.1
COLs
65.1+
43.1++0.1
43.3+0..1
42.2+0.1
salicheettl-mmccun--epllla
CQL
48.2::5.4
$4.2:077
33.11:099
32.41100
COL.
47.8:1.5.2
38.31110
37.7111
32.6:11.2
hallchectal-mmccuu--qpperr
COL
75.24:1.0
21242.11
69.6-288
42.42002
COL,
88.0::1.7
75.54:1.7
80.0:12.55
42.6:10.1
Table D4RL normalized scores for CQL with the original architecture Table 3 (CQL) and with the same architecture ATAC and PSPI (CQLs), i.c., with 3 hidden layers and scaled Banh-Gaussian find that CQL performs similarly under the two choices of architecture.
26

---

Dataset
Algo
Original
Zero
Random
Negarive
button press
CQL
0.88
0.8310.10
0.84
0.94
CQL
0.78+0.10
0.90+10.66
0.73-00.11
0.68+0.11
door open
CQL
0.82 10.99
0.07
0.12.1..00 0.08
0.001
CQL
0.47.10.15
0.29-10.14
0.00
0.38-40.15
drawor-close
CQL
0.91
0.99:0.01
0.96==0.22
0.97+0.02
CQLs
0.99++0.01
0.93
0.97=0.02
0.98=+0.01
drawer-open
CQL
0.78
0.66:10.07
0.65
0.84
CQLa
0.86=
0.79=00.9
0.78=0006
0.80=+0.05
peg-insert-sie
CQL
0.08
0.00
0.02--0.01
0.00+10.00
CQLa
0.10+
0.05+00.0
0.08=00.02
0.03+0.02
pick-place
CQL
0.00
0.06:00:2
0.05
0.07
CQLa
0.00-10.00
0.00
0.00:10.00
0.00:10.00
push
CQL
0.00
0.01 =0.00
0.01=00000
0.03=0002
CQLa
0.03:10000
0.00::0.00
0.00=-0.00
0.01
reach
CQL
0.17
0.29
0.25==0.05
0.23220.03
CQL.
0.35
0.35+10.03
0.37--0.55
0.22440.06
window-close
CQL
1.0010.00
1.0010.000
0.59
0.55
CQL.
0.97-10.03
0.75
0.77.10.12
0.98-10.02
window-open
CQL
0.00
0.82::0.5
CQL
1..07
0.94
0.93
0.77
bin-picking
CQL.
0.00110.00
0.02
0.00
0.00+10.00
CQLa
0.00:1:0.0
0.00=0000
0.00==0.00
0.00++0.00
hox-close
CQL
0.04:10.01
0.07=10.02
0.01 0.01
0.05
CQLa
0.03+:0.02
0.00+0.00
0.01=00.00
0.01
door-lock
CQI.
0.75.10.04
0.61 10.10
0.6110.11
0.60
CQLa
0.63:1:006
0.64::0009
0.57=00.10
0.62+:0.09
door-unlock
CQL
0.87:00033
0.77=0.09
0.80==0.04
0.93+0.03
CQLa
0.92-:0.02
0
0.71=0.11
0.76:10.11
hand-insert
CQL
0.1340004
022
0.3420.04
0.32+0.03
CQLa
0.04::0.03
0.08
0.08
Table Meta-World success rate for CQL with the original architecture in Table 3 (CQL) and with the same architecture as ATAC and PSPI (CQLs), i.c., with 3 hidden layers and scaled tanh-Gaussian. We find that CQL performs similarly under the two choices of architecture.
27

---

D4RL Datasets
ATAC
104
med-eslay
rend
mekrapley,
mud-rally meker halfcheetah
hopper
walker2d
107
X
PSPI
Fee
madi-nelay meseep hopper
BCG
rend
mascoalay maslea
fand
med
maS-rapay medess
eap
IQL
167
med
rend
med-rssly, redieg walker2d
med med raplay
hopper
halfcheetah
10a
mask-nelay medeep hopper
rend
Lonplay masleeg
fand
matirapler medecs
wep
DT
10a
hopper
walker2d
halfcheetah
Meta- World Datasets
ATAC
104
S00r
PFO
pice
merh
WindOw
RSx
cDor
hard
SIde
PSPI
1.0
1ution
091x
DAR
DeSh
6041
Tard
Brition
2008
[PF IMNT
Ns
hOw
0037
Luiion
Iurd
X
DT
1Rech
hard
Figure Estimated positive data bias of D4RI. and Meta- World datasets given 6 offline RI. algorithms, ATAC, PSPI, IQI,, CQL, and DT. Datasets marked by "x" have infinite positive bias.
28

---

Dataset
Original
7ero
Random
Negative
No-terminal
= 100.0
1.0
10.0
hupper medium
10.0
100.0
8
100.0
8 1.0
100.0
hooper-medium--ee repllay
8 = 100.0
3 1.0
8
= 1.0
B 0.1
8
100.0
hopper-msdiun-xxpetr
B 10.0
10.0
0 1.0
0
1.0
hopper- expert
0.1
1.0
8 0.1
8 0.1
walker2d random
8 10.0
= 0.1
1.0
8 0.1
100.0
walker20-medium
10.0
100.0
mallkerdd-medium-ee
= 100.0
0 10.0
10.0
= 10.0
100.0
D 10.0
0 10.0
0 = 1.0
0 0.1
10.0
wxlker2d-expert
halfcheetab-randoon
8 = 0.1
= 0.1
= 0.1
0 0.1
ballcheelah lis medium
B 1.0
= 100.0
3 1.0
3 0.1
balfcheetallmee mecuuuuc-am
1.0
halfcheetah wechhh--diduudeeeeeeee expert
8 0.
0.1
0.1
3 0.1
hallcheetah expert
B 0.1
100.0
0.1
0.1
bulloo-press
B 10.0
8 10.0
1.
3 100.0
door-open
= 100.0
0 = 10.0
= 10.0
== 1.0
drawer-close
8
8
8 100.0
drawer open
B 10.0
==10
8 10.0
3 10.0
insert side
8 = 1.0
== 1.0
$= 1.0
3= 1.0
pick-place
B
== 10.0
3= 1.0
== 1.0
1.0
100
0=10
1.0
reach
8 10.0
100.0
$ = 10.0
8 1.0
window-elose
8 = 1.0
8 1.0
3= 1.0
3 3= 1.0
window open
=110
8
100.0
bin- prckpickng
3 L.
8
3=10
3 1.0
box-close
8 = 1.0
8 1.0
3= 1.0
0 == 1.0
door-lock
3 10.0
3=
door-unlock
100.0
B 10.0
10.0
1000
hand insert
3 10.0
= 1.0
= 1.0
3 == 1.0
point- point-butioni
polnt-bution2
3 1.0
point-circlel
8 m 0.1
point puint-circle?
goal
8
goall
3 = 1.0
point-push1
0 10.0
point-passh pushh
8 1.0
car-button
8 0.1
car hutton2
8 1.0
car-oircle2
= 100.0
car-goall
B 10.0
car-goal2
push
100.0
car-push2
3= 0.1
Table 9: Tuned hyperparameter values for ATAC
29

---

Dataset
Original
7ero
Random
Negative
No-terminal
10.0
B
100.0
100.0
hupper medium
100.0
10.0
8
100.0
8
1.0
100.0
hooper-mediin--ee replay
B 10.0
B 10.0
8
10.0
3 1.0
8 1.0
B 10.0
100.0
3 1.0
0
1.0
hopper- expert
8 1.0
1.0
8 0.1
8 1.0
walker2d random
8 10.0
8 10.0
8 10.0
8 1.0
8 1.0
walker20-medium
10.0
0.1
10.0
mallkerdd-medium-ee
D= 10.0
0
10.0
10.0
0 1.0
100.0
3 = 1.0
= 1.0
0
10.0
D 0.1
10.0
wxlker2d-expert
8
8
halfcheetab-randoan
# = 100.0
= 100.0
= 0.1
0 = 1.0
ballcheelah las medium
B = 1.0
3 == 10.0
3 1.0
3 0.1
balfcheetallmee mecduti--mnddu-epplll
10.0
=
10.0
halfcheetah wechhh--ddiiiideeeeeee expert
8 0.1
1.0
0.1
3 0.1
hallcheetah expert
B
=
10.0
3 0.1
bullon-press
3 = 0.1
B 10.0
1.0
3 100.0
door-open
D 10.0
0 1.0
0 = 10.0
0= 0.1
drawer-close
drawer open
100.0
B 10.0
8 10.0
3 10.0
insert side
8 = 1.0
= 1.0
3= 1.0
8 = 10.0
pick-place
3 10.0
3
3 10.0
== 1.0
100.0
D
3=10
0 100
reach
100.0
8 100.0
$ = 10.0
= 10.0
window-elose
3 0.1
0.1
3= 1.0
3 == 0.1
window open
8
8
bin- prckpickng
3 1.
8 1.
3 10
3 1.0
box-close
8 = 1.0
0 = 100
3 1.0
== 1.0
door-lock
3
= 10.0
10.0
door-unlock
100.0
10.0
10.0
10.0
hand insert
3 10.0
100.0
= 1.0
3 1.0
point- point-butioni
polnt-bution2
g 10.0
point-circlel
8 1.0
point puint-circle2
goal
8
goall
3 = 1.0
point-push1
point-passh pushh
10.0
car-button
8 0.1
car hutton2
8 1.0
car-oircle2
= 100.0
car-goall
3 = 1.0
car-goal2
push
100.0
car-push2
3 1.0
Table 10: Tuned hyperparameter values for PSPI
30

---

Datnset
Original
7em
Random
Negative
No-temiinal
hopper-rundon
B 0.3,
3.0,T 0.9
0.3,
0.3.T 0.9
3.0, r 0.7
hopper medium
$ 0.3, = 0.7
3.0,
0.7
0.3,
0.7
8 0.3.
0.7
0.3, 0.7
hopper-medium- medium-replay
$ == 3.0, = 0.7
8 = 330,7
0.7
3
= 3.0,
-.7
8
=
0.3.7
= 0.9
= 3,0, r = 0.9
hogpet-mediun-exper
B = 0.3, 0.7
= 0.3,7
0.7
0 0.3,
3.07
0.7
= 0.3,T 0.7
hopper-expert
B 0.3,
0.7
3.,
0.7
8 3.0,
0.7
8
0.3.7
0.9
0.3, 0.7
walker2d random
B 3.0, = 0.9
0.3.7
= 0.7
8
0.3, r = 0.7
8 0.3 = 0.7
3 0.3, r = 0.7
walker24-mediom
B 0.3, 0.7
0.3,
0.7
3.0, =.7
0.3.7
0.7
0.3, 0.7
walkerZd malkerzd-mediume medium--ppllyy
D = 3.0,7 = 0.7
8
3,0,
0.7
3.0,7 0.7
0.3,7
0.7
8 3,0, 0.7
walkerzd-msdiun--raa
P D= 3.0,7 = 0.7
0.3.7
= 2.77
D
=
=0331
-.7
8
0.377
0.7
D 0.3,T
= 0.7
walker2d-expert
B 3.0,
0.3,7
0.7
0.3,
03,77
0.7
3.0.T 0.7
hallcheetab-random
3=0.0 = 0.9
8 =
0.3,7
= 0.7
3.0, = 0.9
8= 0337
0.7
hullcheetah- medium
3= 3.0, = 0.9
8
0.3.7
0.9
3 3= 0.3, r U.7
8 0.3.T
0.7
D 330.0 0.9
0.3,7
-077
0 0.3, 0.7
0.9
weffchettah-mediing expert
8 3.0,T 0.7
0.3,7
0.9
8 0.3,
0.7
8
3.0,7
0.7
hallcheetah expert
3 == 3.0, = 0.7
= 3.0
= 0.7
3 = 3.0,
0.7
8 =
0.3,T
= 0.7
bulton-press
3 == 0.3. = U.7
8 0.3,
0.7
8 0.3, U.7
8 0.3.7 0.7
door-open
D= 3.0, = 0.7
8 3.0
= 0.7
0 0=
0.3,T
0.7
0.3,7
0.7
drawer-close
B 0.3, 0.7
0.3,7
0.9
8 0.3,
0.7
0.3.7
0.7
drawer-open
B 0.3, 0.7
8 0.3,
0.7
0.3,
8
3.0
0.7
peg insert side
$ 0.3. = 0.7
8 = 3.0,
= 0.7
8 = 0.3,
0.7
8
0.3.
= 0.7
pick-place
3= 0.3,7 0.9
= 3.0,7
0.7
3 0.3,T
0.7
0.3,T
0.7
push
D= 0.3, 0.9
3,0
0.7
0.3,
8
0.3.7
0.9
reach
g 0.3, 0.7
8
0.3,
0.7
0 0.3.
0.7
8
0.3.7
0.7
window-close
B 0.3,T = 0.9
0.3.7
0.9
0 0.3, 0.7
8
0.3,7
0.7
window open
8= 0.3, 0.9
3.0,
0.9
0.3,
0.3.
0.9
bitt-picking
3.0,0 r U.7
0=0,37 == 0.7
B 0.3, 0.7
8
U.3.7
0.7
box close
3 U.3.T = 0.9
8=
1=3.3,7
0.7
3 3= 0.33
= 0.7
8
0.3.7
0.9
door-lock
D= 0.3,7 0.7
8 0.3.r
= 0.7
0 0= 003, 0.7
0.3,7
0.7
door-unlock
8 0.3,T 0.7
3.0,
0.7
0 0.3, 0.9
8 0.3.7
0.7
hand insert
3 == 0.3.
= 0.9
8 3.0,
0.7
8= 0.3, = 0.7
0.3.7
= 0.7
Table 11: Tuned hyperparameter values for IQL
Dataset hopper-random
Original
Zero
Random
Negative
terminal
100.0
0.1
1.0
hopper medium
1.0
10.0
10.0
10.0
10.0
hopper-madiun-cppaay
a 10.0
Q 10.0
0
100.0
0 10.0
10.0
hopper-madiume exppe
a 10.0
100.0
100.0
10.0
10.0
hopper
10.0
1.0
0.1
10.0
100.0
walker2d-randoe
a = 0.1
a = 100.0
== 100.0
a 100.0
10.0
walker2d-medium
α 10.0
10.0
10.0
10.0
walker244-ediim--eee replay
10.0
10.0
1.0
a 100.0
10.0
a 10.0
a = 0.1
= 1.0
10.0
0
100.0
walker2d expert
a 10.0
10.0
100.0
a 100.0
10.0
halfcheetah-random
a = 0.1
a = 0.1
0 = 1.0
a 100.0
balfcheetah-mediun
a 0.1
a 1.0
0 = 1.0
a 100.0
srchhetah-meddum-
0.1
100.0
100.0
100.0
hal icheethhh medium experi
= 100.0
= 1.0
= 1.0
a 100.0
hallchectall-xppet
a = 100.0
a 10.0
a 100.0
a 100.0
button-press
100.0
100.0
10.0
100.0
door-open
0 1.0
0 10.0
0 100.0
10.0
drawer cluse
100.0
10.0
drawer-open
100.0
100.0
100.0
10.0
peg peg-insert-side side
1.0
100.0
100.0
10.0
pick-place
a = 100.0
a
100.0
0
100.0
α
100.0
push
100.0
100.0
100.0
100.0
Reach
100.0
100.0
100.0
10.0
window-cled
a 1.0
a 0.1
a 10.0
a 10.0
wilddw-openn
100.0
100.0
100.0
100.0
bin-picking
100,0
100.0
100,0
100.0
box-close
= 1.0
a = 100.0
100.0
α 100.0
door-lock
0 1.0
0 10.0
a 10.0
a 100.0
door unlock
100.0
10.0
10.0
10.0
hand- -ind--insert
= 160.0
= 100.0
a 100.0
a 100.0
Table 12: Tuned hyperparameter values for CQL
31

---

Dataset
Behavior
BC
Algo
Original
Zero
Random
Negauve
No-terinal
ATAC
5.620.88
22.6+38
18,4
$0.51111
1888+4,5
hopper random
PSPI
7.24:0.3
12.343.5
17.3+339
23.5+3.7
17.1=4,4
1.2
3.6:0.22
IQL
7.7400
5.0:0.7
1.0:0.00
0940.11
1.0:0.00
CQL
3.0:0.77
2.0:11
18:00.
0.74:00
1.200.5
DT
11.5::0.3
11.04:04
6.5:10011
12.3208
ATAC
84.311.
94.0+0.66
02.3+0.3
915=005
88.1=200
hopper medium
PSP
94.54:1.7
18,44:1.3
92.7::0.5
89.4:0.6
3.02+:300
44.3
53.31...1
IQL
57.4.0099
53.1100.4
59.0.1..1
63.012.1
43.01...
85.0:13.7
55.0+11..7
36.9:99.8
89.0
46.3+1155
DT
62.41:0.6
63.51:0.8
6..31:0.66
62.61088
AIAC
1013=0.2
63.743.9
77.24299
48.1=2211
9.400.6
hopper medium replay
PSPI
10144103
32.41:133
46.917722
31.0:0.0
9..1
15.0
26.71.0.6
IQL
61.01:19
16.5
1.5.1..3
3.5
6.1.3.11
CQL
79.74:3.7
199:011
7.6::3.7
5.422.7
4.42.88
DT
84.0.10.9
60.71.288
67.8.11.1
68.812.6
AIAC
104
106
92.248.0
904
98,924,1
hupper medium expert
PSPI
1...910.6
93.315.0
58.8.1944
95.3100.3
108.3.1066
64.8
53.640.11
IQL
109.
52.840.3
60.5.8..7
94.413.7
77.9
CQL
88 88.6633.6
49.748.11
60.7494
88 88.6
17.+110
DT
89.9+22.7
89.8+-2.6
89.74:2.2
93.842233
AIAC
11111:0.1
110.9=0.1
110.7
110.920.1
111.400.
hopper expert
PSPI
110.910.2
111.10011
111.11.11
111.010.2
110,9.10.3
108.5
110.6:0.0
IQL
110.3:0.2
66.7
73.5
48.5
24.3
CQL
101.2=3.0
0.7+0.00
0.740..0
66.0+12..
0.6:0.00
DT
107.2:20.7
107.7=0.6
108.5:0.3
109.0:0.3
6.4:50.7
3.24:099
5.640.66
4.E11
walker2d 1.4:10.0
PSPI
6.3:1.0.4
2.1:10.88
16:10.5
10100.
5.5:10.0
7.2.1..1
1.
CQL
4.0
0,9+0.9
4.3-144
-040000
0.2:0.4
6.5+:0.22
5.340.5
1.7400.0
6.4402
ATAC
86.8:11.6
76.6.00.3
76.8:0.77
7272002
86.420.6
walker2d medium
PSPI
87.7..0.7
7122,0,5
72.2.10.6
77.5100.5
05
68.1500.5
1.141:111
6284330
CQL
414
2.0+00.9
1.2+00.7
14.5#74
2.1+088
76.6:t00.3
75.74-05
7..8+:0.2
75.5+00.3
ATAC
85.7211.3
67.242.66
66.5.T.66
42.3267
80.22
walker2d medium replay
PSPI
84.9:33.2
69.9+:0.7
68.1
$6.244555
81.3
19.310.4
57.71:3.6
15.512.22
3.241:.2
CQL
1..1
0,5++0.3
-0.2=0.11
0.24:033
0.40044
56.3::1.3
13.94:100
62.0
13.541.5
1089H53
96.444.4
93.62488
72.1=0.3
108.9445
walker2d medium expert
PSPI
110.0=0.1
60.6-66.3
109.2=00.1
75.5220.7
111.7113
103.0.11.0
11101102
106.2:1044
70.71:299
CQL
00.
0.8.00.5
0.1::0.2
28.0-19.1
0.2.10.00
10.
1084=0.1
108.5:0022
108.41022
110.20011
00%.0=0.1
007.1
107.020.0
110.3000
walker2d expert
PSPI
100.3=0..
107
102.1
107 002.2
109.5:0000
108.3110.0
110.11.11
0.0
111.0.10.1
CQL
108.3+0.22
108 0.2
108.0=00
108.1 1.2.2
108.34033
109.0
108.9=0.1
109.0=0.1
109.0=0.11
ATAC
2.2:0.00
2.200.0
22::0.00
22200.00
halfehoctah random
PSPI
2.6:0.44
2.3:00.0
2.2::0.00
2.3+:000
-0.1
1.91000
18.6:10.3
18110.1
200:0.11
2.2::000
CQL
30.7.10.6
1.0.0.88
-14.0.3
5..10.3
2.3.0000
2.3000
2.3:0.00
2.300.00
ATAC
21.220.1
43.1=0.11
43.0=0.11
414+0.1
halfcheetah medium
PSPI
49.4::0.1
43.0
42.9-0.1
42.810011
40.7
42.21:0.0
IQL
50.0.1..1
39.3128
42.3.00.1
34.7110.3
CQI
57.211.55
43.240.11
43.5
42.710.11
43.0=10.0
43.0-1:.0
43.0
43.0=0.0
ATAC 47.410.1
31.712.2
31.1
40.3-10.2
halfcheetah medium replay
PSPI
48.5::0.2
35.6:10.6
36.1
35.5
27.2
35.9.1..1
IQL
43.710.2
36.1
35.
18.1
CQL
48.245544
342200.7
33.1
32.4+ 1.0
38 6.6.1
372200.2
37.1
37.0=20.2
ATAC
91.5.0066
46.42.0
4..91
41.010.11
halfehectah medium expert
86.74119
52.6+117
49.8+
42.240.1
64.4
67.9.10.4
IQL
94.810.1
56.5133.2
70.8.10.88
37.9-0022
CQL
75.24
81.242.1
69.6+22.8
42.400.2
78.81088
81.1:077
1.
78.32:99
ATAC
94.0:044
91.4004
92.412
93.120.1
halfehoctah expert
PSPI
94.6::0.11
92.9+0033
88.0+1..3
93.5-00.3
88.1
92.9::0.0
96.8-10.1
92.910011
92:10000
89.310.1
CQL
92.3::099
93.940.44
93.740066
94.0
92.610011
92.110.1
92.610.1
Table 13: D4RL normalized scores. The mean and standard emoo of normalized scores computed across 10 random seeds. For each random seed, we evaluate the final policy over 50 episodes.
32

---

Dutaset
BC
Algo
Original
Zero
Random
Negative
ATAC
1.0000.00
0.90=00.9
1.0000.00
0.90
PSPI
0.86=0.10
0.90=0.09
0.84
0.90=0.09
button-press
0.90::0000
IQL
0.9810.01
0.97=
0.94
0.88 0.8820002
CQL
0.94
DT
1.0+00000
1.00=0.00
1.00=0.00
1.00
ATAC
100000000
1.00=0.00
1.0000.00
0.95 0.95 00.04
0.0=0000
0.95=0.33
0.96=00.2
0.009
door-open
0.81 02000
IQL
0.94
0.06
0.92:10.03
0.92 0.02
CQL
0.82-10.09
DT
1.00=0.00
1.00=0.00
1.00=0.00
1.00
ATAC
0.89 0.09
0.07
1.00
1.00 1.000
0.77
0.96=0.04
1.00:00000
1.0000.00
drawer close
0.99.10.000
IQL
0.98 0.01
0.99
0.99
0.97.10.01
CQL
0.99=0.01
0.97
DT
0.99=0.01
1.00=0.00
1.00=0.00
1.00
ATAC
0.72
0.09
0.41 0.41
0.69
PSPI
00
0.42=0.09
0.57=0.07
0. 0.74=0.06
drawer-open
0.88=0001
IQL
0.93=#0.2
0.04=0.02
0.73
0.52
CQL
0.78-00.4
0.66=0.07
0.65=00.07
0.84+0.03
DT
1.00=0.00
1.00=0.00
1.00=0.00
1.00=0000
ATAC
0.02
0.86=0.06
0.83
0.92
PSPI
0.75
0.73=0.07
0.92=0.03
0.90
rig-ineet--sie
0.58::0.01
IQL
0.60
0.61
0.34
00.01
DT
0.07
0.00=0.00
0.01
0.00=
ATAC
0.38
0.35=0.05
0.35
0.24
0.47
0.16=0.05
0.31=0.04
0.14=00.4
pick-place
0.15:10.11
IQL
0.11
0.16=0002
0.1710.01
0.10
CQL
10.03
DT
0.06=0001
0.06=0.01
0.06=0001
0.06=0001
ATAC
0.
0.48=0.08
0.34
0.27
0.45
0.42=0.44
0.35
0.37
push
0.19.10.01
IQL
0.101..00
0.18 0.02
0.14
0.13:10.01
CQL
0.00
0.00
0.03
DT
0.04
0.06=0.02
0.04
0.07
ATAC
0.86-10003
0.01
0.8311003
0.81 0.003
0.92:00.22
0.77=0.04
0.84
0.84
reach
0.59-10.01
IQL.
0.51 1.002
0.04
0.57
0.58 1.002
CQL
17.00.03
0.29=0.04
0.23--0.33
DT
0.67
0.79=0.02
0.81
0.83
ATAC
0.90
0.00=0.00
0.90
0.86:10.1 10
PSPI
0.90=
0.99=0.01
0.89
0.92=00.06
window-close
1.00=0.00
IQL
0.99
1.00=0.00
0.99
0.99:
CQL
1.00=0.00
1.00=0.00
0.59=0.13
0.55
DT
1.00
0.98=0.01
0.99=00.0
1.00
ATAC
0.51 0..120.13
0.740.11 10
1.0000000
1.00
0.00=0.00
0.98=0.01
1.00=0.00
0. 0.70
w window-open
0.98:20.00
IQL
0
0.98
0.98:0000
0.002
COL
DT
0.91 0.01
0.91=0.11
0.89::002
0.90=10.1
ATAC
0.41
0.49=0.09
0.31
0.37
0.29=0010
0.41=0.08
0.33-10.99
0.30-10.1
bin picking
0.72:10.01
IQL
0.391.0.2
0.59-0.07
0.35
1.311.0.04
CQL
DT
0.00=0.00
0.00=0.00
0.00=0000
0.00
ATAC
0.20=00.0
0.50=0.13
0.32=0.11
0.33=0.12
0.34
0.28:10.09
0.24-10.08
box-clood close
0.57:10.01
IQL
0.58 0.002
0.54
0.40
0.23 0.001
CQL
0.04
0.07
0.00
DT
0.02=0.00
0.03=0.01
0.04=0.02
00.03
ATAC 1.00
0.98
0.70
0.62.10.12
0.82-0.110
door-lock
0.96:10.00
IQL.
0.99-10.00
0.96-
0.91 0.91
0.69 0.60410.04
CQL
0.75-00.04
0.61
0.61
0.60
DT
0.002
0.83=0.02
0.80=0.02
0.78=
ATAC
0.90
0.03
0.98
0.94
PSPI
0.95
0.98=0.01
0.88=0009
0.98
door-unlock
0.86=20.00
IQL.
0.87-10.02
0.82
0.86
0.81-10.02
CQL
0.87-0003
0.77=0.09
0.89
0.93-00.3
DT
0
0.97=0.01
0.98=00.11
0.98-20.11
ATAC
0.80
0.36=0.08
0.80
0.61 20.04
0.00
0.62=0.08
0.75
0.81
hand iinert
0.39::0.01
IQL
0.35
0.34=0.0Z
0.18:10.03
0.14:10.02
CQL
DT
0.24-00.33
0.21
0.28
0.26::0.02
Table 14: Meta- Meta-World success rate. The mean and standard error of success rate computed across 10 random seeds. For each random seed, we evaluate the final policy over 50 episodes.
33

---

D Technical Details of Section 3
Here we provide the details of our theory on the robustness of offline RL in Section
Theorem 1. (Informal) Under Assumption L and certain regularity assumptions. if an offline RI. algorithm Algo is set to be sufficiently pessimistic and the data distribution p has a pos- itive bias, for any data reward F E R. the policy &t learned by Algo from the dataset D has performance guaraniee " 7) - 7 (3T & [0, 11,1,,8 87 & supp(p)(a) S O(1) for small u that decreases to zero as the degree of pessimism and dataset size increase.
V²((n) (dn) I O(1) as well as safety guarantee (1
Below we first provide in Appendix D.1 some background for stating the formal version Theorem in
Appendix D.2 Then we present the formal claims on survival instinct and conditions on positive data bias in Appendix D.3 and Appendix [D44 respectively. Finally, we prove Theorem iin Appendix [D5 Our discussion here will be focusing the discounted infinite-horizon step. We discuss the finite horizon variant in Appendix
D.1 Definitions
First, we introduce some definitions so that we can state the formal version of Theorem constrained MDPs.
1
We define
Definition 3 (Constrained MDP), Let f.g 8 8 A [-1, A CMDP problem C(S, f.g. P.y) defined as
max
VF(d),,
V (do) S 0. s.t.
Let n denote its optimal policy. For 8 2 0, we define the set of 6-approximately optimal policies,
II (,, fa VF (do) VT (do) - 6, V (do) s 8)
We will drop the subscript S.g from IT,,, when it is clear from the context. In addition, we define a sensitivity function to approximately optimal policies,
x(6) = max *EIT((8)
(d( V (do)
The sensitivity function measures how much the approximately optimal policies (which can slightly violate the constraint of the CMDP) would perform better than the optimal policy of the CMDP (which satisfies the constraint). By definition, x=(6) E j0, 2 and its value can be smaller than 2
1-7
By duality, a CMDP problem is related to saddle-point problem induced by its constraint. We review the definition of saddle points and this fact below.
Definition 4. Given a bifunction L(x, v) X' ) + R. we say (xt, n!) E X x y is saddle point of C. if for all & E X' and E y it holds that C(x,y") S L(x .v) S L(xt.y).
Lemma (Duality). (39] Let II denote the space of policies. Consider a CMDP C = f. C(S,A,
r). Define the Lagrangian L(T, A) = =V7 ( for II and X 2 0. Let at and X denote the optimal policy and Lagrange multiplier, i.e..
tt arg max min L(T, and Xt arg min max C(T, A)
X20
X20
Then together form a saddle point (Tt,At At) to the Lagrangian L.
We also introduce a definition to characterize the degree of pessimism of offline RL algorithms. We call it an admissibility condition defined below, which measures the size of the data-consistent reward functions that the offline RL algorithm can have small regret.
Definition 5. [Admissibility] For R. > 0, we say an offline RL algorithm Algo is R-admissible with respect to R. if for any 7 & R, given D, Algo learns policy & satisfying the following with high probability: for any policy æ II,
max V (do) V (do) s E( (F)
34

---

where we define a data- conssitent reward class
(F((F) = {F F(s,a) a) = a(, E supp(n) and F(s, a) E [- 1, E E A},
( ji) is some regret upper bound such that [(,, (1) = o(1) if supses denotes a term that vanishes as the dataset size becomes infinite.
d [xxa] oo, and (1))
pls,a)
In Appendix F provide proofs of the degrees of admissibility for various existing algorithms, including model-free algorithms ATAC 2 VI-LCB [3 PPI/PQI [13], PSPI [7]. as well as model-
algorithms. ARMOR 34 40), MOPO 32 MOReL 33 and CPPO [14] We show that their degree
of admissibility increases as the algorithm becomes more pessimistic (by setting the hyperparameters).
Finally, recall the definition of positive data bias in Section 3 for completeness, where the implicit CMDP (F) is defined in (T.
Definition 2 (Positive Data Bias). A distribution μ is ! --osittivlyy biased w.r.t. reward class R if
max max VAT (do) - V," (do) c FER xEll (6)
(2)
for all 5 2 0, where mI (o) denotes the set of 5-approximately optimal policy of C,)
Note that the CMDP (F) is feasible because of the assumption on concentrability (Assumption 1
That is, we know '²" is feasible policy, as (1 - V. (do) E.- Ex--[1[(s, a) & supp(μ)]] =
implied by sups,a p(3,0)
d' (e.a)
D.2 Formal Statement of the Main Theorem
Now state the formal version of Theorem Assumption 2 (Data Assumption).
1. For all i E R. f(s, a) E [-1,1] for all E S and E A.
2. The data distribution f -pooitivvey biased with respect to R.
3. There is a policy ı satisfying Ve, S
4. For any E R, the CMDP (F) C(S,A, c,, P.,,)) has sensitivity function
x(0) K for some K do and lims k()) 0. In addition, its solution mt satisfies d= (s.a) sups.a p(s,a)
< x.
Theorem (Main Result). Under the data assumption Assumption consider an offline RL algorithm Algo that is sufficiently pessimistic to be (K + -adammiiseeeee with respect to R. Then for any E R, with high probability, the policy learned by Algo from the dataset D = ((,,,, p, F(s,a), a), S' -P(ls,a)) a}} has both performance guarantee
V. (do) (doo s sc+
and safety guarantee
(1 7) Σ ? Prob (3r [0,t]s, & supp(p) #) S
where 8 k(8) + (1) and o(1) denotes a term that vanishes as the dataset size becomes infinite.
In Assumption [ the first assumption supposes the data reward (which may be the wrong reward) is bounded; the second assumption is that the data distribution is positively biased in view of the reward class R. Finally, we make a regularity assumption on the convergence of x()) to zero
d= (s.a) < 0o to rule out some unrealistic R and P combinations. We note that the and sup..0 8.0. ((3,0)
35

---

condition on the bounded density ratio is always true for tabular problem, and a degenerate case can happen when the state-action space is, e.g., continuous. On the other hand, a degenerate case that K. is discontinuous at 8 = 0 can happen due to the infinite problem horizon in the worst case; roughly speaking, this happens when it is impossible to determine the constraint violation using finite-horizon estimator. We provide the following lemma to quantify the rate of s(0), which is proved in Appendix [D6 In Appendix horizon setting.
E
we show this regularity condition is always true in the finite
Lemma 2. Suppose there is X S L with some L < 3o such that Xt E. arg minx max, VF AVE, (do). Then r<() I Lô.
Under the data assumption Assumption 2. Theorem shows that if the offline RL algorithm Algo is set to be sufficiently pessimistic (relative to the upper bound of the sensitivity function k, which we know most _ I-y by the first assumption in Assumption 2) then Algo have both performance guarantee to the true reward r and the a safety guarantee that the learned policy would have a small probability of leaving the data support.
We recall from Section that Theorem 2 is the effects of the survival instinct of offline RL and the positive bias in the data distribution. As we will show, the admissibility condition in Theorem (2 would ensure that the survival instinct, which is what we stated informally as Proposition]] [ In addition, we will provide concrete sufficient conditions for the data distribution being positively bias, as we alluded
in Section
3.3
We discuss these details below and then we prove Theorem in Appendix D.5
D.3 Survival Instinct
We show the formal version of Proposition [ which says that if the offline algorithm is admissible
then it has a survival instinet. That is, implicitly solves the CMDP in (1 even though the constraint
is never explicitly modeled in the algorithm. In Appendix [1] we prove the admissibility for various
existing algorithms, including model-free algorithms ATAC [2]. VI-LCB
31 PPI/PQI [13,, PSPI (7).
as well as model-ale pooel-algorithms, ARMOR (34 40). MOPO 32) MOReL 33 and CPPO (4)
Proposition 2. For F E R. assume the CMDP := C(S, A,,,A,F,G, P.r)) has sensitivity function
such that lims-0+ <(8)- O and k²()) S K for some K S T Consider an offline RI. algorithm Algo that is ( 5 + addadisssb with respect to R. Then, with high probability, the policy ât learned by Algo with the dataset D = [(s, a, (s), ~ u, = F(ss a), s' P(-ss,a)3 satisfies
Vir (do)
V (do) S ((1)
V cu (dn) 5 + x + o()
where o11) denotes a term that vanishes as the dataset size becomes infinite.
Proof of Proposition To begin, we introduce some extra notations. Given (T), we define
relaxed CMDP problem: max, V (do), V. c, (do) S å and its Lagrangian L5(T, A) == (do) -
X(V (de) - 6), where 2 O denotes the Lagrange multiplier. Let 73. and X denote the optimal policy and Lagrange multiplier of this relaxed CMDP, and let - denote the optimal policy to the CMDP (7). Finally we define shorthand r| (s. a) = F(s, a) (A} + (s,
Now we bound the regret and the constraint violation of Ā in the CMDP (F). The following holds for any 8 2 0. For the regret, can derive
VF¹ (do) - V (do) = VF' (do) - (2$ + 1)V3 (do) - VF
(V (do) = 0)
< VF' (do) - (2f + +1V (do) V (do) + (2) + 1)VV (do)
(V (do) 2 0)
V.T' (do) - v,5 (do).
36

---

Here we use the additivity of value function over reward, namely, for any reward functions and policy n, we have V + Vi3 tr2 Similarly for the constraint violation, we can derive
ep (do) = (Vc.. (do) - 6) +
= (A} + 1)( (do) - 6) 23 (do) - 8) + 8
(do) + (1/ + 1) 6, (dn) 5) (do) 2/(V² co (do) 5) +
Cs(t, (t) Cf A/ + +1)+0
S Co(T)) 2/ +1 - Ls( (f,,1/1 +1 +8
(2l (do) (do)) +
S VT (do) V (do) 5 + ki()
where the first inequality is due to that (TTJ. 2f) a saddle point to Ls (see Definition 4 and the second inequality follows from
(dn) V (do) (Ag 1)Ve " (do);
< V. (do) - (A) + 1)Va' (do)
(using Ve S Varh) (Definition of k.)
S VI (do) + x(6) - (A) + 1)v=' cp (do)
S VI (do) + x(0)
Next we bound on V.5. (dg) V (do). By Lemma 5 and Assumption 2 we know 2/ S K so pt (s, a) E - - 2, 1]. This and the definition of Cu together imply that r E RR ). Since the offline RL algorithm Algo is 2 -admissible, we have V (do) V(((d)) s E(rl, .p). Finally, we ds,a) < cc, so E(Tl, 14) o(1). Thus, we have use the fact that 1 by definition satisfies sups,a ((s,a)) proved
Vi (do) (do) S (1)) v (do) S (1) + +
That is, â E (F)((F) with 8 +
D.4 Implicit Data Bias
Below we provide example conditions for data distributions to have a positive implicit bias. Proposition Under Assumption 7 the followings are true.
(RI setup) Suppose for any in R, there is h - R. such that
(1,
s.a E supp(p)
[r(s,a) + TEg [M++)] - h()) F(x, a)) S
E2, otherwise
for some 61 < co. Then μ is 1-y 2e1 -positiivv biased R.
2. (IL setup) If μ = d= then μ is cc-positively biased with respect R - ( A [-1, yf d= such that V." (do) - V. (do) S i and n" is deterministic, then μ e' ppsiitively biased to the previous R.
is 1-2
3. (Length bias) Suppose V (do) S 8 implies V (do) V (do) S 2'016)) Then p is I-7
O ( biased with respect to R (i: F: S x A (-1, Below we list some sufficient intervention conditions on data collection that leads to the length bias:
37

---

(a) For all (s.a) E supp(u), Rmax Fmax-r(s,a) é, where Rmax = a S 1. (b) For all (8,a) E supp(p), (,(,, (8) S
The first example is the typical RL assumption, which says μ always has positive bias to R if all rewards in R. provide the same ranking of policies in the support [4T]. The second assumes the data collectd by a single behavior policy, which includes the typical IL assumption that the data is collected by an optimal policy. Finally, the third example describes a length bias. Note that V (s, a) measures the probability of going out of data support after taking at which is disproportional to expected length the agent can survive. This condition assumes longer trajectories in the data to have smaller optimality gap.
This length bias condition is typically satisfied when intervention is taken the data collection process (despite the data collection policy being suboptimal), as we saw in the motivating example in Fig. T and Section 3.1 and empirically in Section 4 We give two sufficient conditions on such interventions. The first one is that the data collection is stopped when the instantaneous rewards is far from the maximum rewards, and the second is that the data collection agent does not take an action that no policy (even the optimal one) can perform well in the future. The consequence of these types of interventions that longer trajectories perform better generally. These conditions are satisfied in hopper and walker2d dataset (except for -medium-replas in Section 4 well as in the grid world example in Section 3.1
The proof of Proposition is based on the performance difference lemma, which can be proved by telescoping sum decomposition.
Lemma 3 (Performance Difference Lemma). Consider a discounted infinite- MDP M P.7), For any policy and any h S > R. it holds that
1 Esard a) + PB [h(s')] h())].
(do) - h(do)
In particular, this implies for any policies n,
V, (do) (do) 1
E, Br 10 (s,a) (s)).
D.4.1 Proof of RL setup in Proposition
Proof. Given E R, consider some h: S x A + R satisfying the assumption. Define
f(s,a) = r(s, a) + Ble (k())) h(s).
Notice
(1 (V (do) (do))
(1 - (V, (do) h(db)) - (do) h((o))
= Exandr [r(s, a) +
(h(s')) h[s)]] - E. (r(s,a) + +.B (la(ss())) - -(ss)]
= Eg [f(s, a)] Egaand [f(s, a)]
=(1 (V, (do) (7(())
By Lemma for any E (8), it holds
- (V= (do) - V (do))
(1 - [VV (do) VF
Egaande [i(s, a)] Esaand [f(s, a)]
Bgande (8(x, a)] EEgand (8 a)] + Egangr (f(s, a) F(x, a)] Es (F(s
- f(s, a))
(1 (V, (do) (do)) + Eeand (f(s - -((s, Exa-er (f(, i(s, a))
38

---

Since nt is assumed to be in the support of in Assumption
! we have
Eg s [[ss a) f(s, a [A(s, a) F(s,a))
E s.and [[s, a) ((s,a))(s,a)) E supp(μ)) ax (f(s, F(s,a))(s; E supp(u))
Es (((s a) F(s,,))((s.)) supp(u)]
$ 261 - sup [[s, a) - f(s, a)l Beand [A[s, a & supp(u)]]
$ 261 + -2 (db)
Putting these inequalities together, we have shown
(1 (1-7)(V,') (do) V," (1 (VF (do) - Vy" + 2e + V2, (do)
Since (8), implies
(V= (do) V (do)) $ (1 7)6 2e1 (1 7)e26 2c1 0(6)
Thus, the distribution is 1-2 -osiiioiyy biased.
2e,
D.4.2 Proof of IL setup in Proposition 3
Proof. Let μ d= We can write
(1 (1= (do) (do))
=Ev (s) O (s,a)]
Ean [V, (s) OF (s.a))(s,a) E supp(r)) Ea (V." (s) Q7 (s) & supp(u)]
56
So the distribution is cc-positively biased. For the case with the suboptimal expert n², we can derive similarly
(1 - -V, (do) - V (do))
(1 - (V, (do) V," (do) + (1 (V, (do) - V. (do))
< é' + Em [V.T" (s) O, [s)
c' + Ea² [ (s) OF (s,a))(s, supp(ui] Est [V."' (s) OF (s, (s,a)](s, a) & supp(n)]
< 8
Therefore, is - -ositively biased.
D.4.3 Proof of Length Bias in Proposition 3
Item 3a
Proof. Suppose Rmax - S for (s, supp(u).
(1 - (1=)((="' (do) - V,7 (do))
Egg (r(s,a)) a)] Es (r(x, a)]
= E (r(s - E ()(s,a)((ss E supp(r)) - E. (s., & supp(u))
I Rmax E.g [ (Ruax - ())((s,)) supp(n)) -
Egr [e'][e E supp(u)) + [Rmax (s, a) & supp(#))
< + (1 - R7)
39

---

Item 3b
Proof. Suppose a, supp(u) only if (s) Q7(s, a) s é'. Then
(1-7)(V," (do) - (do()))
-BarlV (s) O. (s.a)
=Ea-lV," (s) Q (s.a))(s,a) supp(u)) EV (s) - Q, (s, & supp(u)]
Sé'+o
(s) Note that more broadly, we can relax the above condition to that Qü, (s. -V2, (s) (s) - (s, a) S 3 + O(a), for potentially smaller 3, where we recall that V Q: are the optimal value functions to (minimizing) the cost Cu and V,"" Q are the optimal value functions to (maximizing) the reawrd r. We can use the lemma below to prove the degree of positive bias is 1 B
a implies
in this more general case. The above is a special case for a = 1, since (s, a) = when s, is not in the support of μ.
Lemma 4. Define A {a E A Q2 a) V2, (s) a.. Suppose for all E 8, A
satisfies V (s) Q (8, S B. For such that V (da) S 6, it holds for any x¹
+ 8/a
VF (do) (do) v 1-7
Proof.
(1 (1-7) (do)
V.7 (du))
Ex- V (s) O7' (s,a)l
Ex V (s) - OF (s,a)la E AT + E EV (s) OF (s, $ A
1
S 3 +
Eng [1 AgIl
1
On the other hand, we have
(1 7)6 2 (1 - V) (do) = (1 (V? (do) - V (do))
Ex- 12,, (s, a) V2, (s)la A.] E. (Q; - V., (s)la & A.)
Ba=10 (s, a)
V. (a)la A.)
al [1 & A.]]
Therefore,
(1 - 7V, (do) - (do)) 5
D.5
Proof of Theorem 2
Proof. By Proposition 2 we have proved
Varl (do) (do) s o(1)
cn (da) S o(1) + 5 + k(0)
That is, A E (C,(())) with = 8 x(6) + o(1)
Then by I --positively assumption on μe, we have the performance bound
V. (do) - V (do) S O())
40

---

To bound of out of support probability, we derive
20
V Gp
(do)
y aa [[)
20
= -
Σ (u((,a)) =0]]
(1 Prob (3r E [0, $ supp(u) (t)
where second equality follows 45 [Lemma 2].
D.6 Technical Details of Constrained MDP
Lemma 5. For g: S A |0, oo), let C(S, A, 9, P. Assume (IS,A,f.s P.Y) is feasible. Consider a relaxed CMDP, max, Vj (do), S.S. (do) S
be CMDP with sensitivity function kc(8).
x(6) and let A] denote its optimal Lagrange multiplier. Then X] I 8
Proof. Denote the optimal policy of the relaxed CMDP as mo! and the optimal policy of the orig-
inal CMDP t!. By construction, =/} (o). Therefore, we have V (do) VJ (da) max (8) (do) -VF (do) k()).
To bound A1 we use that the fact that (TT X1) is a saddle point to the Lagrangian of the relaxed problem La A) = (do) XVV (do) ö). As a result, we can derive
VT' (dn) > (do) 8) CTT A/) = Ca(T)) Ai) s Cs( 0) V (do).
Since 1/ 9 (do) 0, the above inequality implies
2/ < V (do) V7 (do)
k(d) 8
8
Lemma 2. Suppose there is X L with some L so such that At e arg minAzo max, V7 AV (do). Then k(() & T.S.
11 [4 Theorem 4.1] we know that (I1, A1) is saddle-point to the CMDP Proof. First note by
( C(S A. F, C.. 7 in (I. Let f = and g = Cu. Define the Lagrangian Cs(T,)) 0) =
(((d)) X(V (dn) 8) for the relaxed CMDP problem (do) V (dg). Consider
saddle-point (T). A/) to this relaxed problem.
By duality, we can first derive
max V7 VF' (do) = max VJ - VF' (do) = Lo(T)) VV (do) x:V; (d)5s (do)s8
=ETI
Then we upper bound Ls (T( 2\):
La(T5, xf) Ca(Tth111)
(db - (V (do) 6)
vd) - , o! (do) + 5X
[(T)) X1) + +At
5 L(T), ) +8
L(m 0) + +
We use this theorem because the CMDP here does satisfy the Slater's condition.
41

---

Thus we have
max (7(d)) -0-- VF (do) [(=), +ax Vp (do) V7 (do) + 821 VF (do) 621
rem (6)
E Finite-Horizon Version
We discuss how to interpret Theorem 2 in the finite-horizon setup. Assumption (Data Assumption).
For all F E R. F(s, a) E [-1, 1] for & and a A.
2. The data distribution
-positivvly biased with respect R.
3. There is a policy т satisfying Ve,- S
4. For any F R, the CMDP (F) = C(S,A,i, P.n) has sensitivity function
k(8) S K for some K < oo and limg-so x(6) = 0. In addition, its solution .! satisfies
(s.a)
suPs,0 p[s,a]
X.0.
Theorem 2 (Main Result). Under the data assumption in Assumption consider an offline RL algorithm Algo that is sufficiently pessimistic to be ( - 2)-admissiblee with respect to R. Then for any F R, with high probubility, the policy & learned by Algo from the dataset D a,, p. i(s, a), P(-|s, (ss,aa)})))))) has both performance guarantee
V (do) (do) 1 5c+0())
and safety guarantee
(1 7)
? Prob (3r [0, & supp(n) #) S
where 1= 8 8 o(1) and o(1) denotes a term that vanishes as the dataset size becomes infinite.
To translate the previous notation to the finite-horizon setting, suppose the state contains time information and the state space is layered. That is, S U#= S,, where H the problem horizon, and S, denotes the set of states at time t. For example, for a trajectory so: 82. ................................ 8 starting from O, have s, S,, for t E [0, H 1]. Therefore, previous notation to model time-varying functions naturally (needed in the finite horizon), without explicitly listing the time dependency. In this section, with abuse of notation, define the value ViT
Notation
the
of policy ı to reward r at 8 E S, as
(s) = E.,
F(8e: 0r) 87 =
MDP and CMDP Problems
The task MDP becomes My = A,, and solving it means
max B
Σ
r(s az) 30 ~ do
and the CMDP problem in I becomes
H -1
H
max Ex, 4
Σ
F(sr, at) 80 do
s.l. ExP
Σ
cn (st, az)
80 ~ do 50
42

---

E.1 Main Theorem for Finite-horizon Problems
Assumption (Data Assumption (Finite Horizon)).
1. For all i E R. f(s, a) E [-1,1] for all 8 E S and a E A.
2. The data distribution j -positively biased with respect R.
d: 3. The solution to the CDMP æ² satisfies supa,a p(s,a) < co, where we note that d= is defined without discounts but as the average over the episode length.
Theorem (Main Result (Finite Horizon)). Under the data assumption in Assumption 2 consider an offline algorithm Algo that is sufficiently pessimistic be & + adddmissibbbee with respect
10 R. Then for any i E RR with high probability, the policy & learned by Algo from the dataset D = s.) p. a), P(-ss, a}} has both performance guarantee
V (do) - (do) sc+
and safety guarantee
Prob (3r E [0, H 17 & supp(+))#)
where: = 5 k(6) + o(1) and o(1) denotes a term that vanishes as the dataset size becomes infinite.
The main difference between the infinite-horizon setup and the finite-horizon setup is that the finite- horizon setup always satisfies the regularity assumption (the third point) in Assumption 2 In addition, the safety guarantee more explicit compared the discounted horizon counterpart. This is because Prob (3r [0, H 1], & supp(u)((t I V (do) for the finite horizon, whereas we need an additional conversion in Appendix D.5
6p
We prove lim6- s(
is always true for the finite horizon version.
Proposition 4. For the H -horizzon constrained MDP of (I have optimal dual variable A 2H + 1 and k(() S min(2H, (2H 1)6}-
Proof. Define the Lagrange reward rf (s, a) = F(s.a) X Nee (s, a). Let denote the optimal policy to the Lagrange reward. Notice that V (do) 2 1 for any such that V (do) O. Therefore, with Xt 2H.
(d - XtV (do)
max V X
max m:Ve (do)>0
v:V= (do)20
-
max V2 Xt (do)
V: (do)=0
Therefore, I| satisfies V= (do) 0. This implies G,
max V (do)
Xtv= (do) 2 min max V (do) - AV (do) V= (do)
6x
X
On the other hand, it is always true that
max V (do) Atv (do) > min max V.5 (do) AV (du)
X20
Therefore,
A min (do) - AV= (do)
cu
Consider X = 2H + By Lemma 2 we have x()) S (2H + 1)8. addition k(6) < 2H by definition.
43

---

E.2 Length Bias
For finite-horizon problems, we provide a simple sufficient condition for length bias, which assumes long trajectories obtained in data collections are near optimal.
Proposition 5. Suppose the data are generated by rolling out policies starting from t = 0 and the initials state distribution dg- Let { = (sg, ag. .1.....877 11c-1, aTg-1) denote a trajectory of lenghh Tg. If with probability one (over randomness of {) that Te = H implies
H-1 Σ
r(st, 2 V (do) He
then the data distribution is 1 He positively biased.
Proof. Consider some π E II (8). We can derive
V," V17 (do) (do) - Ex
r(se, az)
H
P
V. (do)
Σ
r(5 0z)
3r [0, H 11, & supp(n)
1=0
H-1
E
(dox(d))
Σ
r r(s,, az) VT E [0, H - 1, E supp(w)
1=0
S H x Prob (3r E [0, H 1] $ supp(u))xx) + HC
< x V (do) + He
$ Hō + Hc
Remark on the existence of full-length trajectories. Note that we do not explicitly make an assumption on the likeliness of full-length trajectories in Proposition 5 This is because the probability of having full-length trajectory in the data distribution is guaranteed to be strictly greater than zero given the concentrability assumption in Assumption Notice that time information is part of the state in finite horizon problem, so Assumption Assumptionßimppies Timplies that there is non-zero probability of having full- horizon data trajectories (otherwise, the concentrability coefficient would be oo). When Assumption holds, statistical errors due to finite dataset size are included in the term Theorem
E.3 Goal-oriented Problems
Finally we make remark on positive data bias in the goal-oriented finite-horized setting. In the hefiite-horizon setting that a goal is marked an absorbing state, which means that the agent once entering will stay there forever. The exact instantancous (wrong) reward obtained at this absorbing state not relevant (it can be anything in [-1, 1)) since the admissibility condition has already accounted for the range of the associated Lagrange reward. Namely, the agent is set pessimistic such that it views partial transitionsstraaectores obtaining a worse return than 1-7" bound of the return the absorbing goal state.
which lower
To make sure the finite-horizon setting has the same kind of positive data bias, one way is to virtually extend the problem's original horizon (say H) by (as least) one and let the goal state be the only state where the agent can stay until the last time step H (i.c., all the other trajectories continue maximally up to time step H and therefore have length at most H). Then we apply the offline
We use zero-based numbering here where 0 is the initial step and H is the last step for a problem with horizon H 1.
44

---

RL algorithm to this extended problem (e.g., of horizon H + 1). We would need to set the horizon in the previous theoretical results accordingly for this longer horizon.
The reason for this extension is to ensure that the effect of absorbing state in the infinite-horioon setting (which ensures trajectories going into the absorbing state is by definition the longest) can carry over to the finite horizon case. If we apply the agent directly to solve the H step problem without such an extension, there may be other trajectories which can be long as goal-reaching one. As result, there is no positive data bias.
An alternate way to create the positive data bias in the finite horizon setting is to truncate trajectories that do not reach goal at time step И - 1 to be no longer than - 1. That is, they now time out at time step H 2, whereas only goal-reaching trajectories continue up to time step H 1.
F Algorithm Specific Results
In this appendix, provide few examples of admissible offline RL algorithms. We choose algorithms to cover both model-free 22731 and model-hased [34]
40
14
32 33 approaches.
The pessimism in these algorithms constructed through different ways, including adversarial
training (2 7 34 40 [4], value bonus 31 32
and action truncation
33
These algorithms are
listed in Table 15
Adversarial Trinnnng
Value Ronus
Action Truncation
Model
ATAC 2 (Apperdix
VLLCB 31
(Appendix ES
PPU/QQ (Appendix F6
PSPI (Appendix
Model-based
ARMOR
CPPO [4] (Appendix
40 (Appendx
F4
MOPO
R Appendii
MOReL
R3
(Appendix
F7
Table 15:
We show that all of the offline RL algorithms above are admissible. Note that this is not complete list of all admissible offline RL algorithms.
We recall the definition of admissibility below.
Definition 5. [Admissibility] For R 2 0, we say an offline RI algorithm Algo is R-admissible with
respect to R. if for any i E RR given D, Algo learns policy a satisfying the following with high probability: for any policy æ E II,
(doo V22 (do) S ((T,
max (F)
where we define a data-consistent reward class
R (F = (F F(s, a) = F(s,a), suрр(и) and E (-
E A},
d (saa E(T, 11) is some regret upper bound such that E(τ, p)) o(1) if supses p(s.a) denotes term that vanishes as the dataset size becomes infinite.
co, and o(1)
For the sake of clarity, we consider the tabular setting in our analysis. That is, we assume the state space and the action space AA are countable and finite. (We use |S] and [.A] to denote their cardinalities). To establish admissibility, we sometimes make minor changes over the function classes these algorithms, because some algorithms assume that the reward is known or has value bounded by [0. 1j. We highlight these changes in blue. We would like to clarify that the goal of our analysis is to prove that these algorithms are admissible rather than provide a tight bound for their performance. We recommend readers who are interested in performance bound read the original papers, their bound can be tighter than ours.
One remarkable technical details in our analysis is that there is no need of using an union bound over all rewards (f) the admissibility definition when proving these high probability statements. Instead we found that we can reuse the bound proved for single reward directly. The main reason that all rewards in admissibility definition agree with each other on the support of the data distribution, and the statistical analysis of concentration only happens within this support. Outside of the support, a uniform bound based on the reward range can used to bound the error. This will be made more clearly later in the derivations.
45

---

F1 ATAC
We first show that ATAC 2 an admissible offline RL algorithm. We first introduce notations that we will use for analyzing ATAC, which will also be useful for studying PSPI in Appendix F.2
F.111 Notations
For any U.v A(S x A), we denote (n (s, a) max(u(s, a)) 0). For any μ E ASS x A) and any f : S x A + R. we define (H, f) = Lisa)esxa ((), For an MDP with transition probability P: 8 x A A(S), define (PT ()(s) = -= any f S x A and S A(A).
m) for
Given function class F S (S A R), policy π S ((A) and reward : S x A R. we introduce Bellman eπror transfer coefficient 7 to measure the distribution shift between two probability measure 14 and v.
Definition 6 (Bellman error transfer coefficient). The Bellman error transfer coefficient between and ju under function class F, policy ı and reward r is defined as,
I F P" f 12, JEF
max
I F- P fl2,,1
We note that the Bellman transfer coefficient is weaker notion than the density ratio, which is established in the lemma below.
Lemma For any distributions v. E A(S x A), function class F S (8 x A + R), policy
:::: S - A(A) and reward F S A + E. we have
v(s,a)
F(
sup (s.))SXAA u(s,a)
(5)
Proof.
|lf F 20--pffil
fll,
Σ
vfs.a) ( f(s,a))
(K,0)SSA
u(s,a) ((.0)
Σ
(f(s,a) - F((, Pd as),
m(s,a)
(a,0)ESxA
Σ u(s,a) sup v(s,a)
(sa)ESxA a(),a) (f(s,a)- P(s.a)
(s.0)E8xA
v(s,a)
Σ
((s,a)(f(s;a) - a(s, - f(s
= sup (alDEExAM(ss,.
(s.a)ESXA
v(s,a) (ca))SxAAMs,,0))
sup
||ʃ f- pd 112
Therefore,
1f - 7 Px f12
8 ((1,,,,,,,,7 max
FEF [] - pTl f133,
v(s,a)
|lf - P f 112,,
sup
max
(aA)ESXA ((s,a)
fef f P f 11
sup v(s,a)
(s.4) (u(s,a)
46

---

F.1.2 Analysis
ATAC considers a critic function class F and policy class II. Since we consider the tabular setting, we choose F: (S x [- Vmax. V mmax|
13 with 1-7 I Vinxx < oo and II (S - A(A)). ATAC
formulates offline RI as the following Stackelberge game,
f E arg max (") Tell
(6)
s.t. f E arg min (,(",, (E,
JEF
with 3 2 0 being hyperparameter, and
(D(=, = - -
(7)
{p(T) : [B)]))))_((()))--Fr--)
- -/-f(s ,)) min Epl ((() P'Ey
For any policy n. the critic f provides relative pessimistic (with respect to the behavior policy p) value estimate of the policy æ. The hyperparameter / balances pessimism, given by Cz( and Bellman consistency, given by (T). The learned policy maximizes the relative pessimistic value estimate given by the critic.
We state the performance guarantee of ATAC with respect to any comparator policy π under the set of data-consistent reward functions {f F(s, a) = F(s, a), V(s, a) E supp(e) and |F(s, a)l < (1-7) 7 V(ss, a) S A} in the following proposition. We make additional assumption on the data distribution for ATAC, which is needed in its original proof.
Assumption For ATAC, we assume 4 the (mixture) average of aateeactioon visitation distribution for dg-
Proposition 6. Fix some F S x A 11, Consider F (S A) - --maxx Vmax] with T-y S V < 0o and 11 (S + A(A)). Let àt be the solution to (6) and let 11 be any comparator policy. Let E A(s,a) be an arbitrary distribution. Under Assumption 4 for any
5 E (0, 1]. choosing θ
C
with probability 1- - 8, it holds that
Vinc
(iS)|A) log( (1))
Vinax (8 |)|
||||||||| log(1/6)) 1/3
(d 1,F+PP55 -
(do) (do) sō
(1 (1-7) (1-7)|Dp1.3
1 1-7
(8)
for all F such that F(s, a) = F(8,()) Vs, a) E supp(μ) and [F(8, a)l S (1 (1-7) ((, E A
Proof. We first show that F: (S A + |-V mox: Vimx.) is both realizable and Bellman complete
with repect to F. For any 7 such that [F(s, a) (1-1))
(1-7) - we have Q7 (s, a) E (-mamx Vmmx). This means that QF E F for any E II, 8 E S, A. Moreover, for any E F. a(ssa)++P [(, <
[F(s a)) + 1PT f(s, a)l < (1 - (11-))Vm Vmux + VV = Vmax. In other words, (F(s, a) + p* f(s F
for any f fEF. F.
D. That is, solving the Stackelberg game (6) given by D is equivalent to solving the game given by D. The rest of the proof follows from Theorem C. 12 in (55] by choosing the reward class to only contain the reward r, ie., g = (F) (which implies dg = 0) and using dy.n = ((S)IAA) (og()/s))
By construction of F, we have D = a," D,F= a)}
Note that the above derivation does not need additional union bound to cover all rewards in {f : (Fs, a) = F(x, a), V(s, a) E supp(u) and iFss, a)l S (1- V(8, a) S x A). This is because the concentration analysis is only taken on the support of the data distribution, where all rewards in this reward class agree, and a uniform bound based on Vmax is used for out of support places, which again applies to all the rewards in this reward class.
The original theoretical statement of ATAC assumes F to contain only non- negative functions. However, the analysis can be extended to any F with bounded value, such [-Vmax Vnsa). We note that the practical implementation of ATAC using function approximators does not make assumption on J being non negative.
47

---

We then show that ATAC 2 is admissible based on Proposition 6
R2 Corollary 3 (ATAC is admissible). For any Vmax I- ATAC is R-admissible with respect to R.
for any R C (8 x x (-,,,,
Proof. Given any ÿ E R, we want to show that, with high probability, VF (do) (do) (do) - o(1)
for all F Rx(() and all policy such that (UP(s,)) p(s,a) 0C. For any such x, define
(s.a)
d'(s.a) C. == a) p(s,a)) we have (v; p1, F. F,,, I Coo for any F. By taking v = dm in Proposition [6] we have, with probability 1 - 6,
Vinax (S log( 1og((/))) 1/3
(d d F 1T)
(d) (do)
(do) sō
(1 2 2/3
1-7
V Vmux (Cx|S]||A| log( 1/6)) 1/3
= 0
= =0(1)
(1 (1-7)
for all such that F(8,a) ax,, ((s.a) a) E supp(u) and (F(s a) - (1-7) V V(s, ESX..
Since Vina R I-y; have that V (do) V (do) o11) for all (
F.2 PSPI
We show that PSPI (7) is also admissible. PSPI 7 is similar to ATAC. Given critic function class
F (S x A + |- Vmx Vima]] with 1- S Vmax < do and policy class 11 (S + A(A)), PSPI
solves the following Stackelburg game,
A arg max (1 - 1f (do, Well
(9)
s.t. f arg min (1 f) T) + (T JEF
with 3 2 0 being hyperparameter, and
Ep(T) (((s,, m))
-
min PeF
- -F-76s,, 9.
In PSPI, the critic fr provides an absolute pessimistic value estimate of policy π. The hyperparameter trades off pessimism and Bellman-cone consistency. The learned policy á maximizes such a pessimistic value. We state the performance guarantee of the learned policy & with respect to any compara- tor policy under the set of data-consistent reward functions {F F(s, a) i(s, a), V(s
Sμрр(μ4 and [F(s, a) < (1 7 V(s, a) E S x A} in the following proposition.
Proposition
77 55 Let be the solution to (6) and æ E II be any comparator policy. Let
A(s, be arbitrary distribution. For any 8 E (0, 11. choosing 3 è
CID
Vas V
(1811A1)og(((1))
with probability 6,
V7 - (do) cō (
(d p= - -
+
(1 (1-7)) (1/))/D///3
1-
(10)
for all T such that F(s,a) a) = F(s, V(s a) E supp(u) and [F(8, S (1 max. V(s
E S A
Proof. The proof is similar to that of Proposition 6 First, observe that F (S A + [-Vaax Vmnx is both realizable and Bellman complete with respect to F. The rest of the proof follows from Theorem D.I in (55) (taking G = (f)) and Lemma 6
1*Similar to ATAC, we extend the critic function class to contain functions that can take negative values. The theoretical analysis can be generalized to this case. The practical implementation of PSPI can directly handle critics which take negative values.
48

---

The proposition above implies that PSPI 7 is admissible. We omit the proof of the corollary below as it the same as the proof of Corollary 3
Corollary (PSPI is admissible). For any Vmax 2 R I-7' any R S (S A [-,, 11).
PSPI is R-admissible with respect to R. for
E3 ARMOR
We show that ARMOR 34 40] is admissible. ARMOR is a model-based offline RL algorithm. We denote model as M = (S. A. PM. -V..) Y), where Par S x A A(S)) is the model dynamics, and FM S x A + | Rmux Rmax 15 is the reward function with S Raux < 00. Since we consider the tabular setting,
model class that contains all possible dynamics and all reward
functions bounded within [- Rmax, R.aa]. i.e., Mmoodel = {M : PM E (S x A + A(S) , TM e (8 x A [- Rmax: Rmax))) For any reference policy कeती S - (AA), ARMOR solves the
following two-player game,
f arg max min
MEM: (do) - VM (do)
(11)
EETT
where (i)( = EK PM Ei= I'M (st: 0s) ~ dol. and
MB = -IM E Mmdelt (n(M)
min
ED(M') S a),
(12)
M Mmodel
with
{D(M)= = Σ - log Pa (s' + (rM(s, F)?
(13)
Intuitively, ARMOR constructs a relative pessimistic model (with respect Tret) M which is also approximately consistent with data. The learned policy म maximizes the value estimate given by the pessimistic model. We define the generalized single policy concentrability characterize the distribution shift from μ to the state-action visitation d" of any policy æ.
Definition 7 (Generalized Single-policy Concentrability [40]). We define the generalized single- policy concentration for policy æ, model class Mmodel, reward and data distribution μ
Ex FF]
Cmodet (T;; Minoeet, F) == sup
TF (((;)
MEMmodee with 8(M;F) Drv (a P()ss (FM ((s.))))
The single-policy concentrability Cmodet ( (x) can be considered as a model-based version of the
Bellman error transfer coefficient in Definition 6 Following the same steps as the proof of Lemma
it can be shown that Emodet (T;; Mmmodet r) S SUP(s,a)ESxA p(ssa)"
(s.a) We provide the high-probability
performance guarantee for the learned policy Ā. This implies that ARMOR is an admissible algorithm with a sufficiently large Rmax.
Proposition 8. For any 5 10, there exists an absolute constant such that when choosing a = c |A log()/a). for any comparator policy a E II, with probability - 5, the policy î learned by ARMOR [T] satisfies
V (do) - (do) 5 ( (Vemontal (TT Mmodel, r)
Rmar 7)2
|| (+/)
(14)
[D]
for all F such that F(s, a) = i(s, a), V(s E supp(H) and (F(s, a)l $ Rmari V(s S A.
Proof. By construction, we have M = (S,A,,,F,)) E Mmodell Since F(s,a) F(s, a) for any (s, a) E supp(μ), we have M = Mg. The rest of the proof follows mostly from Theorem 2 in [40 by taking Net = μr. By similar argument as in the proof of Proposition 6 we do not need an addi- tional union bound to cover all rewards in (F F(s, a) = i(s, a), V(s, a) E supp(μ) and [F(s a) <
Rmaxx ((s, a) S A}.
15The theoretical statement of ARMOR assumes the value of TM is bounded by [0, 1]. The original analysis can be extended as long as the value of r' is bounded by finite value Rmax < 20. The practical implementation of ARMOR does not assume the reward only takes value in [0, 1).
49

---

Corollary (ARMOR is admissible), For any Rmax 2 R. ARMOR is R-admissiible with respect to for any R C (8 x A [],
Proof. Given any F E R, we want to show that, with high probability, V7 (do) d (4.a)
o(1 for all F E RR(() and all policy such that (a.(x, p(s,a)
< 00. For any such π,,
"oodet (T;; Mmodel, F) S sup (s.a))) pla,a)
d (s.a]
Coo < co. By Proposition 8 with probability
1 8, we have
VZ(d) (do) (do) 10000ovo
Rmax
|| logllo/(/// o(1).
(15)
71-
[D]
for all F such that F(s,a) = f(s, a), V(s,a) E supp(u) and [F(s,a) < R, ((s E ES A. There- fore, (do) (do) (1) for all E (a( with high probability.
F.4 CPPO
[4] is another admissible model-based offline algorithm. Again, we denote model as M- = (S, A, PM, rM; 77) where PM x A + A(S) is the model dynamics, and rw x A - [-Rmax, Rmax 16 is the reward function with S Rmax < co. CPPO solves the two-player game,
t = arg max min MEM
(do)
(16)
sell
with (d)) = Ey, Ei= 7² Ta (st, 4,)s30 ~ dol and My D defined in 12) CPPO constructs a pessimistic model M which is also approximately consistent with data. The learned policy क maximizes the value estimate given by the pessimistic model. We provide the high-probability
performance guarantee for the learned policy f. This implies that CPPO is an admissible algorithm with sufficiently large Raaax
Proposition 9. For any d (0, 1]. there exists an absolute constant e such that when choosing S| A (og) any comparator policy π E 11, with probability 1 - 8, the policy î learned
by CPPO [6 satisfies
(do) - V (do) 5 ( (VErmodeel TT; Mmodel, F)
Rmax
S| log(1/))
(17)
(1 7)2
[D]
for all i such that F(s, a) = i(s,, a), (() E supp(n) and (F(s. S Rmary V(s, E x A.
Proof. By construction, we have M = (S,A,, E Mmodee- Since F(s, a) = F(s, a) for any (s, a) supp(μ), we have Mg Mg. The rest of the proof follows similarly from the proof of Theorem in 40 By Lemma 5 from [0]. with high probability, M E MM. Therefore,
(do) - V (do) = V (do) (do) S V (do)
min MEMS (do))
S (do)
min (Vi) (do))
MEMS
< max MEMO
V (do) Vi (do)
where the second inequality follows from the optimality of म. By (20) from 40 for any M E MG.
Following a similar argument to the proof of Proposition 6 there is no need to take the union bound with respect to which concludes the proof.
IV (do) - (i (do) S O (v Cmoddtt (T; Mmodel, r)
Ruc (1-a)
[ A Alog((/))
We omit the proof for the following corollary as it is the same of the proof of Corollary 5
Corollary 6 (CPPO is admissible). For any Rmax 2 R. CPPO is R-admissible with respect to Roror
any R C (S x A + 1)).
50

---

Algorithm VI-LCB 31
1: Input: Batch dataset D, discount factor 7, confidence level 6, and value range Vmax-
2: Set == log I-7
3: Randomly split D into J +1 sets D. {(5, Oz Ti, 5() for E {0,1,... , with m J+1
4: Set mu(s, a) == En= 1( a.) = (s, a)} based on dataset Do.
5: For all E A and s E S, initialize Qu = - Vo(s) = - Vmax and set To(s) = arg max, (n(s a).
6: for 1 J do
7: Initialize ry (s. a)
and set P3 to be a random probability vector.
8: Set mx(8, == >i =i= [ss, a.) } based on dataset D.
9: Compute penalty (8 a) for L 2000 2000log + + S| A /8)
(e(3, = 2/max (s)))
10:
for (s,a) E AS do
ms, a) 2 then
12:
Set a,a to be empirical transitions and (2 a) be empirical average of rewards. end if
14:
Set (s) + re(s, a) - by(s, + + V1
end for
16: Compute V.mid(s) (s) <- max. as) and (s) E arg max, Qr a).
for s do
18:
if V.midds (s) < (s) then
19:
Vi(s) + (s) and ग. (s) + Tj-1(s)) (s).
20:
else
21:
(1(8) + (s) and (s) + m. miidd (8).
22: end if
23:
end for
24:
end for
25:
Return ==
F5 VI-LCB
We show VI-LCB 31 is admissible. VI-LCB is pessimistic version of value iteration. It adds negative bonuses to the Bellman backup step in order to underestimate the value when there are missing data. By being pessimistic in value estimation, it can overcome the issue of μu being non- exploratory. We recap the VI-LCB algorithm Algorithm T We highlight the changes we make in blue. These changes are due to that originally the authors in 31 assume the rewards are non-negative, but the rewards in the admissibility definition Definition can take negative values. Therefore, we make these changes accordingly. We state and prove the guarantee below
Proposition 10. For Viaax 2 [])
R VI-LCB is R-admissible with respect to any R. C (S A
Proof. First, we introduce factor of 2 in their proof of Lemma 1 31 to accomodate that the rewards here can be negative. This is reflected in the updated bonus definition in Algorithm [ This updated Lemma of (31] now captures the good event that the bonus can upper bound the error of the empirical estimate of Bellman backup. It shows this good event is true with high probability.
We remark that while originally Lemma 1 of 331 is proved for single reward function, it actually holds for simultancously for all the reward functions in RR( without the need of introducing
16e original algorithm assumes that the reward function known. Here we consider the generalization of the algorithm with unknown reward. We referenced to lemmas and equations based on the version arXiv:2103.12021.
51

---

Algorithm 2 Pessimistic Policy Iteration (PPI) 13
Input: Batch dataset D, function class F, probability estimator ji, hyperparameter b.
2: for i E {0, (0,..., - do
3: for e {O..... J do
4: fi.t11 arg me (f
5: end for
6: ft+1 < arg max, Ep [ fi.J-1]
7: end for
8: Return ã
Algorithm 3 Pessimistic Q Iteration (PQI) 13
1: Input: Batch dataset D, function class F, probability estimator áú, hyperparameter b.
2: for i E {0,............ (0,....II 1} do
3: fi11 6 arg min (f.
4: (t41(s) &- arg max, < (s+1(s,a)
5: end for
6: Return #
additionally union bound. The reason is that in the proof of Lemma in [31] the concentration is used only for the estimating the Bellman on the data support. This bound would apply to all rewards in Rx( since they agree exactly the data. For state-action pairs out of the support, the proof takes uniform bound based on the size of Vaa which also applies to all rewards (F)
Therefore, we can apply the updated Lemma 1 of (3T] to prove the desired high probability bound needed in the admissibility condition. Under this good event, we can use the upper bound (54b)
in 31 to bound the regret. Consider some such that C' == sups,a [[s,,a < co. Suppose D has N transitions. Take some Vmax r-y. Then running VI-LCB with J iterations ensures for any
F E Ra(F),
64V max 1-
LIS|AACC(] +1
V (do) V (do) 2 2V max
where we mark the changes due to using rewards which can takes negative values blue. Setting log as in 31), with probability - 5, it holds that
1-7
Vmax
L|S|IACC
(do) (do)
(do) 5 n VVVax
N
|SILA C log
=
Viax
N(1 7)3
Thus, VI-LCB is R-admissible.
F.6 PPI and PQI
We show PPI and PQI proposed in 13 are admissible, We present their algorithms in Algorithm and Algorithm 3 with (Ç f) a) = Ç(s. fs, a) for any Ç. S x A R. These algorithms use probability estimator of the data distribution (denoted as fi), which in the tabular case is the empirical distribution. Based on A. they define a filter function
((s, a; ft, b) [[A(s, a) 6]
This filter function classifies whether state-action pair in the support, and it is used to modify the Bellman operators in dynamics programming as shown in line 6 of Algorithm 2 and line 4 of
We add back an extra dependency on [A| as their original proof assumes is deterinistic. which is not
necessarily the case here.
52

---

Algorithm 3 As result, the Bellman backup and the policy optimization only consider in-support actions, which mitigates the issue of learning with non-exploratory μ.. The loss functions (i.e., L in Algorithm 2 and Algorithm 3 denote the squared Bellman error (with a target network) as used in We omit the details here.
fitted Q iteration 78
In summary, PPI and PQI follow the typical policy iteration and value iteration schemes, except that the backup and the argmax are only taken within the observed actions actions with sufficiently evidence to be in the support when using function approximators). This idea is similar to the spirit of the later IQL algorithm [ except IQL doesn not construct the filter function explicitly but relies instead on expectile maximization.
Proposition 11. Suppose F (S A +
[- Vmaxx Viaa
19 For Vmax 2
PPI/PQI is I-7'
R
R-admissible with respect any R C (S X A 1I)
Proof. We prove for PPI; the proof for PQI follows similarly. Consider some т such that = ( sup,,a p[s,a] < co. Suppose 1) has N transitions. Take some Vmax 2 I and some F from R
We the Corollary in [I3]: With I large enough, it holds with probability 1 1-
Vmaxx (1 (1-7)3
S ||
|||(//1))1100) ln(1/))
sups,a,t (s a)
(i(o - VF 50
cp + (1 - >>) -.E.-.P(Me pl s 26]]
N
b
where = (r, = 1 (NN) ). Further,
-1-71 (l-7))mrrPPlluie a) s 26]] Σ
d= (s, 26]
8,0
Σ a(s dt (8,a) 6,0
1im(s, a) s 26]
m(s,a)
$ C, S 26]].
We can then upper bound the performance difference above as
(1 Vimxx
sup sups,, (s)
VN + CEp(1[u(s, s 201))
VF (du) S 5ō
(1-
N +
b
We bound the latter two terms by tuning Notice [T,(1[((8, S 26]] S |||b at most. This implies
inf sups.att
s.a.t
di () 1
+ CE,[Llu(s,a) s 26]]
VN
sup a (F a) + C1S|||A/22
inf
b
VN
VCISTA N1/1
0
Thus.
VIAX
(1SILA))((1))) N
VCSSTAA
V (do) V (do) 5
(1-
N1/4
We can apply this similar argument used in the previous proofs to show that this bound simultaneously applies to all F E (f) since the high-probability concentration analysis is only taken on the data distribution where all rewards in RF) are equal. Thus, when Vmax I PQI) R-admissibile.
R PPQ (and similarly
The original algorithms PPI PQI assumes (S A [0, Vimx)). note that our extension Viux, Vinx do change the performance bound absorbed the Ô notation.
53

---

F.7 MOReL and MOPO
MOReL 33 and MOPO 32 are two popular model- based offline RL algorithms. The two algorithms operate in a similar manner, except that MOReL [33] uses the truncation approach (similar to PPI and PQI [3] which classifies stteeacttions into known and unknown sets, whereas MOPO 32] uses negative bonuses (similar to VI-LCB [1]. In this section, we show both algorithms are admissible.
We denote model M (S, A, Parraa,, 7), where PM S A ((S) is the model dynamics, and T' x A + (-1, with I Rmax < oo being the reward function. The model class is denoted as Mmodel.
F.7.1 MOReL
We analyze a variant of MOReL. which also learns the reward function, whereas MOReL in the originally paper [3 assumes the reward function is given. We suppose model reward r' and model dynamics PM are learned by
M'
PM E arg min (p(M)
MEMmodel
where
{M(M)
Σ
- log
-))
We define the set known state-actions Kg as
(s.
{Irm (s, a) (s, Vaxx (ww (Ma s st).
Kg
sup M2EM
(18)
with
My = {M Mmodel (n(M
min
Ep(M') S at,
MMmde
We note that a can be chosen as [ log 1/8) so that Ma contains model (S,A, with probability - for any 8 (0, 1] (see 40 for details).
Remark Note that we change the definition for the set of known state-actions. In 33 the set is given by
- ((s,a) (rar(s a) F(s, a)] VVv
a), P(+(s,a) 5
(19)
However, is not impossible to use such a set in practice since the reward function F and true model P(-[s, a) are not given the learner. Our known state-action set Kçç, comparison, is easier to construct. It simply measures the maximum disagreement between two data-consistent models. With considerate choice of (see 40)), Kc is always subset of KG Our definition also matches better with the practical implementation in 3 which uses the disagreement between any two models in an ensemble.
Then MOReL learns policy # by solving the MDP (S,A,P,F, where
rir(s,)), -X,
8, E Kr otherwise
a(
a() = (s')ss 1 = st),
s, E Kg otherwise
Note MORel. introduces absorbing state st.
Below we prove performance guarantee of MOReL. The proof here is different from that in the original paper, because here we need to consider reward learning and the original proof does not provide an exact rate that converges zero the data size increases. (In the original paper there is non-zero bias term that depends on the comparator policy.)
Both MOPO and MOReL assume that the reward function is known. Here study the variants of the two algorithms which also learn the reward function.
54

---

Proposition 12 (MOReL Performance Guarantee). Given data D = {(s, a, P.)) size N of an unknown MDP P. Suppose r. E [-1, 1] and (f. P) My
D' Choose
min(1, m(V (154)))))))))))))) probability - 6, it holds that
Let 2 R. For x such that sups,a u(8.0) C < oc, with
(s.a))
(IS ( :)) N
R ) with Vinx ==
1/4
(1-7) (V-7)(V7 - V; 0o Vinux
1-7 ilr E RR()) = (r: F(s a) = =((s V(s,a) E supp(p) and F(s, E ((0,,,),
Proof. We extend i as F(st, ,a) = - for all a A. Let M = (P,
First, by the optimality of â. write
(do) V2 (do)
= (do) Vi (do) (7 V
M (do) + V (dd)) (do
SVØ (do) - (do) + V (do) V (do)
where MM denotes the value of policy т with respect to the reward and dynamics P. The last inequality follows from the optimality of क on M.
Then we notice the fact that Kgg S {upp(()) with a small &. The proof of the lemma below is given in Appendix F.7.3.
Lemma 7. Ifc 1, S supp(n).
Let d= denote the average state-action visitation of a with respect to P. (do) (do))
- (do) (1- 7) (V
EF sa [((, ( a) + - P/s (5)] F(s, a) - 7B, Pla
E and [f(s, a) + y1 P| 1/2² (5') F(s, a) 7B V (s)) $ Kgl
$5 + aand ((s + (.an Kel
- E Baand [f((s)) - - P1000
- F
Kg]
<5 + R st Vmax F(8, a) + V (s, a) E Kc]
<
where in the second inequality we use Kg S supp(u) (which implies F = F in KG²).
Similarly we can show
(1 1--1 7)-) (V (do) - V2 (do))
F(s,a) 2- { + Es [[( a) + 7T. Mls -(((a)) -B (V & Kc]
2 - { 2V 2V and an [1[(s, a) & Ke]]
Combining the three inequalities above gives
(1 7) (V. (do) v (do)) < 2V [1[(s, a) & Kell + 2
By the concentratability assumption, we can further upper bound
Ex aL)(s,a) & Kell I [1 a) Kg]]
Therefore we have
(1 - 7) (V77 (do) V (do) < 2V Va an aor [A[[(, a) & Keel + 2
E, Ee (esppl (s, a)) + 2E
Vax
E sup Psr B L (E(s, PM]]
S Vimax
+
55

---

where the last step is Markov inequality,
(s
sup Mr..s
{IM (s, ( + max DTV (1s, (-18, PMr (1),,)))
and
E(s, PM) (rM(s, - F(ss ( (
Now upper bound the expectation of E over μ.. With probability greater than 1 - 6,
E (s, a,T, PM)) S Bull(ra(ss,)) (o
2 - (7Vmax (/00://60)))000////////////////r
- (1 + 7 Vmnx) /2 R. ( a))
P(.ss, a))
1 max v2
Ep (TM f(s,
DDv (1s P(11s, a))
|S| A log(
S max v2N
N
where the last step is based on Lemma
Lemma 8. (4 With probability
8, for any MDP model M,
(ED(M) min ED(M) + |S| |A| log(
EpllrM(s, a) r(s,,))) DrV Dr)(P
Thus, we have
V2
v 1/1/ log(
max
(1-7) (VF V/ (do)) so
+ +
EVN
(1 log(()
)
1/4
0
Viaax
can apply the previous analysis technique to MOReL. show is admissible without addition union bounds, which leads to the corollary below.
By Proposition 12
Corollary 7. For X 2 R, MOReL is R-admissible with respect any R C (S A A [-1,1)
F.7.2 MOPO
MОРO 332 is very similar to MOReL except that it uses negative bonuses (like VI-LCB) instead of truncation. In the original paper of MOPO, the authors assume the reward given. Here we consider variant that also learns the reward. Specifically, given Pis and ris above it solves the MDP (S, A, PM, f, 7), where
f(s.a) = (s)
While the original MOPO paper does not give a specific design b(s, a) that provably correct, in principle making MOPO provably correct is possible. Essentially, we need to choose large enough bonus to cover the error of the model-based Bellman operator (defined by the estimated reward and dynamics). This would lead to a bonus of order Vmax (see the analysis of VI-I.CB in Appendix [F5). Then we can proceed with an analysis that combines that MOReL and VI-LCB to prove MOPO's performance guarantee. This can then be used to show (like the previous proofs) that MOPO is admissible. We omit the proof here.
Corollary 8. Suppose b(s,a) a) O(Vnax) for (8,a) a) supp(u)). For Vina 2
Jt. 1-gx
MOPO is
R-admissible with respect any R C (S A [I).
56

---

F.7.3 Proof of Technical Lemma
Proof of Lemma Consider any (3,a) & supp(μ). Given any M = we construct M = (8, A. Par, (M,, with TM = rM for all (8,a) E (S x A) {(5,a)), and TMr (s. a) = -i a)). By construction, since M and M agrees on all state-actions except (s, a). which is not in data support, we have M' E EM However, we have that
E M..
{|M (5, TM (3, (3, + yV DrV (PM (-1,,), (-1,
sup
M
(3, a) (3, a) 7V Drv DannDDrrPPalliee (M
=|ra a) (3. 2126.
This means that (8,0) & Kg.. Therefore Kgg C supp(u)..
G
Grid World
In this section, we describe the full details of the grid world study that was discussed in Section 3.1.
Environment. We consider goal-directed navigation in the environment shown in Fig. I0 We use a fixed grid world layout shown in the figure. The agent's state is given by a tuple (xr, y, d) where (ar, y) [5] represents the 2-d coordinate and d {N, W, E, S) encodes the North, West, East, and South direction respectively that the agent is facing. The agent's action space is A = [/,1,) where denotes the action of moving to grid square in front of the agent, / denotes left turn of 90 degrees, and r denotes a right turn of 90 degrees. The agent can through the lava (orange wavy square) but cannot go through the wall (grey square). The goal (key) an absorbing state. The agent gets reward of +1 when first visiting the goal followed by a reward of O for then staying in the goal. The agent gets a reward of -1 for reaching lava and reward of -0.01 for all other actions to incentivize the agent to reach the goal along the shortest safe path. We use a horizon of H 20. The agent deterministically starts in the top-right corner shown in Fig. 10
Dataset. We collect dataset of 500 episodes by taking 100 identical optimal episodes along with 400 identical episodes that all touch the topmost lava field. We introduce a length bias by terminating an episode if the agent touches the lava or fails reach the goal H steps (i.e., the agent doesn't I survive). In addition to the original setting with the observed reward, we consider three additional settings where the rewards in the dataset are replaced by: (i) constant zero reward, (ii) their negative, and (iii) random number sampled uniformly in [0, 1].
a
S R
Figure 10: Left: A goal-directed gridworld navigation task where the agent (red triangle) has reach the goal (key) and avoid lava (orange waves). Center: Shows the length bias that helps offline RL succeed even when rewards are entirely replaced with random or adversarial values. Right: Results on the grid world task. Behavior cloning fails to solve the task (red path) while using an offline RL method leads to success (blue path) even with wrong rewards. The opacity of the grid square overlay indicates how frequent that state is in the dataset (more opaque means more frequent).
57

---

Algorithm 4 PEVI(S, A, H, 3, Vmaa) 12]. We are given access to set D of episodes (r). Additionally, we are given a hyperparameter 3 E R20 that controls pessimism. The hyperparameters Vmin Vmax E RR indicate bounds on policy value with Vmin.h and V /max,h denoting the minimum and maximum policy values respectively for time step h E [H]. Unlike the original PEVI algorithm of [12. we treat Viiin Vaxx as hyperparameters.
Set VH+1 for h H..... do for E S do
for E A do
nn(s, = ΣEED
1(8 = = ag = =a}
n, a) > 0
()) an(s,a] ETED 1,0 = art + (i+1(5%+11)
Qn(s,a) = Qn(s,a) (ns(s.a)
@n(s,a) a) = max (Vnin,k min (Q, a), Vmmxxt
Else
Qn(8,a) a) = VininA
end for
The (8) ) = arg max ())
(s(s))
end for end for Return π (TI:H)
Methods. We evaluate two methods: behavior cloning which simply takes the action with the highest empirical count in given state and random action if the state was unseen, and PEVI [12 offline RI. method with guarantees for tabular MDP. We provide a pseudocode of PEVI in Algorithm Unlike the original PEVI [I2), we provide minimum and maximum bounds for the value function for each time step as an additional hyperparameter. These bounds should be dictated by our CMDP framework and should penalize out-of-distribution actions. In principle, we need set these bounds such that the algorithm is admissible. Please see Appendix D and Appendix for details.
standard value iteration with two key changes. Firstly, it uses pessimistic value initialization where the learned value Vx((s) of a state at time step / is set to the lowest possible return Vmin,h if the state is unseen time step h. Secondly, when performing value iteration it adds a pessimism penalty of -8/ /ns.a) to the dataset reward where nsss, a)) is the empirical state-action count in D. This pessimism penalty ensures that the learned value function lower bounds the true value function (hence a pessimistic estimate), In our experiments, we vary 3 and set Vmin, = Vmin. - 3( - h + 1) where Vmin. is the minimum possible return for the given reward type that we are working with. The value of Vmin. -H h + 1) for the original reward, 1 for the negative reward, and for the zero reward and random reward cases). We define Viaax, = Vmax, where Vmax.h is the maximum possible return for the given reward type. The value of V Vmaa.JA is 1 of the original reward, (H - h + 1) for the random reward and the negative reward, and for the zero reward case.
Intuitively, PEVI performs dynamic programming similar
We show numerical results Table 16 We visualize results in Fig. 10 The behavior cloning simply imitates the most common trajectory in the dataset which results in going to the lava (shown in red in the figure). In contrast, the PEVI is able to reach the goal in every reward situation.
Results.
Explanation. The reason why PEVI works can be understood simply by pessimistic initialization and length bias. The length bias in this setting visualized in the center of Fig. TO PEVI assigns the final state sy of a failed trajectory will get pessimistic value of Vi(s() = V Viii... This value isn' 't updated as we never take any action in s due to timeout. our case, of will be the state where the agent first visits the lava. In contrast, the final state of a successful trajectory 8 gets assigned value of If V Viin is sufficiently negative, then PEVI will only consider policies that stay all H steps within the data support. Due to length bias, this is not true for the non-surviving trajectories that
58

---

Reward Type
Behavior Cloning
PEVI
Original Reward
-1.19 = 0.00
0.92 = 0.00
Zero Reward
-1.19 + 0.00
0.92 0.00
Random Reward
-1.19 + 0.00
0.92 + 0.00
Negative Reward
-1.19 + 0.00
0.92 0.00
Table 16: Gridworld results: Mean return on 1000 test episodes. PEVI achieves the optimal return
of V* 0.92.
reach the lava. Further, the only trajectories our case that complete end up reaching and staying in the goal. Therefore, under the assumption that Vnin,h all sufficiently small, PEVI will learn a policy reaches the goal irrespective the correctness the reward in the data. The sufficiency of the negativity of V is implied by our CMDP framework. Lastly, note that the world is deterministic, we didn't t need to use the pessimism arising from the -8 penalty term. n(s,a)
59


Reviews:
[{"review_id": "PJ22eGfLvI", "reviewer": "Reviewer_NS2u", "summary": "In this paper, the authors show that offline RL algorithms have an implicit survival instinct that often allows them to learn good policies with incorrect rewards. The authors argue that this is due to the data being positively biased and the pessimism that constrains offline RL algorithms to the data. This argument is supported by theory saying that when these conditions are met, the learned policy is close to the optimal policy and meets certain safety guarantees. They show several ways in which data bias can be introduced that benefit offline RL algorithms, such as length bias. Lastly, this phenomenon is shown empirically across several offline RL algorithms and benchmarks.", "questions": "* Line 217 you say that any policy is infinitely positively biased for any rewards resulting from potential-based reward shaping. Isn’t positive bias a property of the offline dataset and not the reward? This didn’t make sense to me. \n\n* Do you have any hypotheses for why some algorithms have different levels of survival instincts? Is it because of the amount of pessimism in the algorithms?\n\n* I still have several questions about length bias:\n    * I understand that often in benchmarks longer trajectories correspond to higher returns, but it is not clear to me why this always helps. For instance, suppose we have a dataset with trajectories where half the time the robot fell and half the time it did not. If we label all rewards with -1, wouldn’t the agent be incentivized to fall as fast as possible to incur less negative rewards.\n    * Perhaps I am misunderstanding the assumptions in Proposition 5, but it seems like there is no assumption on how likely long trajectories are to appear in the dataset, only on the returns of long trajectories. I would think there has to be some assumption on the former.", "limitations": "Although I do not think this work necessarily needs to address these limitations, it would be beneficial to have some discussion about them in my opinion:\n\n* While the paper does cover many relevant offline RL benchmarks, it does not explore all types of datasets. In particular, it does not appear that the authors explored cases where there are only sub trajectories of good behavior that need to be stitched together.\n\n* The assumption of positively biased data in Theorem 1 seems strong. Since the definition of positively biased data is about the value of the delta-optimal policy set of the CMDP, there seems to be no intuition for when Theorem 1 will hold when looking at properties of the offline dataset only. In other words, it does not appear that there are any methods that practitioners could use to estimate the role of survival instincts for any arbitrary dataset.\n\n* While there are theoretical results for offline model-based RL works, the paper’s experiments focus on model-free offline RL algorithms only.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "* Overall, I really enjoyed this paper. While pessimism is known to play an important role in offline RL, I think that this paper does a good job of illuminating how important pessimism is in these offline benchmarks. I think this will be an important paper for the offline RL community, especially when considering these benchmark tasks.\n\n* The paper is very well written.\n\n* The paper is very comprehensive with a strong Appendix section detailing experiment details and additional results.\n\n* The paper has a lot of theory explaining the survival instinct phenomenon and showing how algorithms in the literature have this survival instinct.", "weaknesses": "* I still do not fully understand the argument around length bias (see Questions), and I am not convinced that this is a main cause of the phenomenon shown in Figure 3. In particular, how do we know specifically the length of the trajectories is the important property? I would assume that the medium and medium-expert trajectories datasets just have more support over actions that prevent the agent from falling down. The length of the trajectory just seems like a byproduct of these good actions. More discussion or evidence here would be appreciated.\n\n* The main paper makes it unclear how data was generated for the grid world. While the appendix does have some more information about that, a little more should be explained here to make the example more understandable.\n     * The offline agent is used on the “wrong reward” but what is the wrong reward? It seems like in the appendix you consider three wrong rewards, but which one is used? It seems like different rewards here could change the outcome drastically.\n\n* (Line 219) The authors claim that a dataset generated by the optimal policy is infinitely positively biased regardless of the reward. This seems like a bold claim to make for all MDPs and all reward functions without a proof. \n\n* What happens with expert D4RL datasets? It would be interesting to see what performance is on this, especially since the claim is that datasets from the optimal policy are infinitely positively biased regardless of the reward.\n\n* Potential typo on Line 1415: The agent starts deterministically in the upper left it seems, not right.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Cause of Phenomenon in Fig 3**\n\nWe would like to first clarify that length bias means the length of an in-support trajectory *positively correlates* with its return (line 220), not that the dataset has many long trajectories. In fact, a dataset would have no length bias if it has lots of long bad trajectories, e.g., halfcheetah datasets. \n\nWe argue that length bias is one of the main causes of Fig. 3’s phenomenon. First, we visualize the presence of length bias in most datasets by showing the positive correlation in Fig. 4. Second, when trained on datasets without length bias, e.g., medium-replay and halfcheetah datasets, we observe that offline RL algorithms are more susceptible to wrong rewards.\n\nIn fact, the medium-replay datasets and halfcheetah datasets are more diverse and cover more full-length trajectories than some other datasets with length bias. For example, hopper-medium-replay has 44 full-length trajectories and yet hopper-medium has only 1. This suggests that *\"have more support over actions ... from falling down\"* is not the main reason for the robustness to wrong rewards. \n \n**Grid World Example**\n\nPEVI produces the same policy and has the same performance under all three wrong rewards (zero, random, and negative) in the grid world example. We will clarify this and add more details on the data generation process in the revision. \n \n**Positive Bias from Optimal Behavioral Policy**\n\nIn Appendix C, we provide a proof (Prop. 3 and Appendix C.4.2) of positive bias due to optimal, as well as deterministic near-optimal, behavioral policies. Intuitively, if data is from an optimal policy, all the actions in the support are optimal and therefore by survival instinct the learned policy will be optimal. The point below also empirically validates our statement.\n \n**D4RL Expert Datasets**\n\nIn Appendix B.6, we report the results on the expert datasets of hopper, walker, and halfcheetah from D4RL. In Fig. 9, we show that in most scenarios (13 out of 15), offline RL algorithms can consistently achieve expert-level performance using all three wrong rewards, which verifies our theoretical statement. \n \n**Positive Bias from Potential-based Reward Shaping**\n\nWe first want to note that there is a typo in line 217: any *policy* $\\to$ any *data distribution*. \n\nTo answer the reviewer’s question, positive data bias is a property of a data distribution $\\mu$ *with respect to a reward class* $\\tilde{\\mathcal{R}}$, per Definition 2. In line 217, we intended to convey that, if we consider a reward class $\\tilde{\\mathcal{R}}$ consists of potential-based reward shaping rewards, then any data distribution is $\\infty$-positively biased with respect to $\\tilde{\\mathcal{R}}$. This follows from the fact that the ordering of policies under a potential-based shaped reward is the same as the ordering under the original reward. We will revise the wordings in line 271 to clarify this.\n\n**Levels of Survival Instinct**\n\nThere are a few causes. First, different offline RL algorithms are designed based on different notions of pessimism, e.g. behavior regularization, pessimistic values, etc. These different notions may or may not be translatable to the R-admissibility condition (Definition 4 in Appendix C) that controls the survival instinct. Second, offline RL algorithms often have hyperparameters that adjust the level of pessimism, which, as the reviewer suggested, can affect the level of survival instinct. In Appendix E, we theoretically show how to set hyperparameters of several algorithms to be sufficiently pessimistic. In the experiments, we conducted a limited hyperparameter search, which might not always have yielded the same level of survival instinct as one another. Finally, as noted in the limitations, practical implementations may have slightly different properties than the theoretical algorithms due to numerical issues and implementation details.\n\n**Dataset with -1 Reward**\n\nThe behavior of an offline RL agent is subject to how \"falling\" is interpreted. Consider the robot example in the review. Suppose that each trajectory ends immediately when the robot falls. If falling is interpreted as \"entering an absorbing state of value 0\", the offline RL agent *would* fall, as the reviewer suggested.\n\nBut since we stop the trajectory upon falling, we find it more appropriate to consider falling as “entering a state that is unknown” (Footnote 4 & Appendix B.5). In this case, as the agent never sees what happens after falling, an offline RL algorithm with sufficient pessimism would imagine these unknown states have lower values than getting -1 rewards and hence avoid falling.\n\nWe note that using -1 reward is actually similar to our experiments with the *negative* reward for hopper and walker. Fig. 7 shows offline RL algorithms are able to learn policies that avoid falling with the negative reward. We also observed that offline RL algorithms can learn good policies with -1 reward in our earlier preliminary experiments.\n \n\n**Assumption on Prop. 5**\n\nThe reason why we do not explicitly make an assumption on the likeliness of long trajectories in Prop. 5 is that it is covered already by the third point in Assumption 3. Notice that time information is part of the state in a finite horizon problem, so Assumption 3 implies that there is a non-zero probability of having full-horizon data trajectories (otherwise, the concentrability coefficient would be $\\infty$). When Assumption 3 holds, statistical errors due to finite dataset size are included in the $\\iota$ term in Theorem 1 (line 132). We will clarify this in the revision. \n\n**Limitations**\n\nWe agree with the reviewer on these limitations and will discuss them in the final version of the manuscript. In particular, we find it to be interesting future work to quantify the amount of positive bias of a data distribution, including the size of reward class $\\tilde{\\mathcal{R}}$ it has positive bias with respect to, without having to actually run offline RL algorithms.\n"}, {"review_id": "rcNzugiJxw", "reviewer": "Reviewer_Sf8f", "summary": "This  paper reports a very interesting new phenomenon that would be interesting to the offline RL community. It demonstrates that even when trajectories have the wrong reward labels, offline RL can learn good policies. The paper's experiments attempt to dissect the reasons for why this surprising phenomenon emerges, and argues that special to offline RL algorithms, pessimism endows the agent with survival instinct to stay within the data support, so that safe policies can be learned within these constraints.  \n", "questions": "See questions present in the above sections. \nAnd one more question, about the suggestion at the final part of the abstract (and conclusion): \n-  if the implicit bias for long trajectories in offline data is the sole feature that is responsible for this wrong labels phenomenon, then is it still reasonable to suggest that \"whereby an agent is “nudged” to learn a desirable behavior with 20 imperfect reward but purposely biased data coverage\"? \n\nIn other words, does the \"survival instinct\" as discussed in the paper really have the sensitivity to make such a (next step) hypothesis reasonable? ", "limitations": "The authors have a limitations section, and are fair. They also nicely clarify that of course, offline RL does not always learn from wrong rewards. ", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The phenomenon that has been identified is surprising; the experiments done to dissect this phenomenon are clear and solid. Overall a solid paper.  ", "weaknesses": "I have only one concern, and welcome further clarification by the authors. The authors claim that their algorithm demonstrates this robustness phenomenon arises due to the survival instinct of offline RL algorithm. But it seems from the paper that the circumstance of this only arises in the situation that the data collected for offline RL has long timescale trajectories. Is this the case? For instance, it does not arise in other circumstance investigated, like when there are multiple sources of data, or other circumstances that deviate from this central feature. \n\nIf this is the case, then the authors should state this clearly. This is still very much an interesting phenomenon, but is slightly more limited than the first impression one gets when reading the abstract and introduction that this is a potentially quite a universal phenomenon in offline RL. \n\nIf long trajectories is not the only circumstance in which this phenomenon, the authors can make the paper even more clearer by listing all the circumstnaces in which this phenomenon arise, which would be very helpful for the reader. \n\nRegardless, as long as the authors match their conclusions to their demonstrations, this paper demonstrates indeed a very interesting phenomenon arising in offline RL that is worthy of publication. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**When positive bias arises & clarification on \"long timescale trajectories\"**\n\n\nWe thank reviewer Sf8f for providing feedback to our manuscript. The robustness to reward is attributable to an interplay between survival instinct in ofﬂine RL algorithms (due to their pessimistic nature) and positive data bias. There can be *multiple* possible factors that cause positive data bias. Length bias is one such factor which requires the length of the trajectory to correlate with higher returns (see line 220-221). Note that length bias is not the same as the dataset having more long trajectories. Further, there can be other factors of positive data bias (Prop. 3 in Appendix C), e.g., potential-based reward shaping or near-optimal behavioral policies (see Appendix B.5 for experiments) which are not related to problem horizon or data trajectory length. \n\n**Positive bias is not a universal phenomenon but it can be common**\n\nWe want to highlight that we do *not* intend to mean all existing offline RL datasets have positive data bias. In fact, we explicitly highlighted this in Section 4 (line 258) and showed examples like halfcheetah datasets, where there is no positive data bias. Nonetheless, we also point out that positive data bias often arises (due to factors listed in Prop. 3) in common data collection practices, such as intervention of unsafe or bad rollouts, or using expert demonstrations. \n\n\n**On nudging the agent to learn desirable behavior**\n\nFinally, to answer the reviewer’s question about how to purposely increase positive bias in data. Let us consider length bias as an example. During data collection, we can early stop a data collection episode if the agent is not performing well. This creates a positive correlation between data trajectory length and performance, and thereby increases the length bias in the dataset. Recall that survival instinct gives offline RL agents the incentive to produce trajectories that stay within support in a long term. Therefore, offline RL algorithms, due to survival instinct, can learn better with such a dataset. We are happy to answer any follow-up questions.\n"}, {"review_id": "FbMrIGFlfm", "reviewer": "Reviewer_k1TU", "summary": "This work displays a new observation that offline reinforcement learning (RL) algorithms can develop efficient policies, even when trained with incorrect reward labels. The authors attribute this resilience to the pessimistic nature of offline RL algorithms and the inherent biases in data collection processes. The findings introduce a ‘survival instinct’ in the systems that inform how we understand offline RL benchmarks and create future ones. The study, therefore, recommends a new strategy for offline RL that promotes learning of desired behavior through flawed rewards but intentional bias in data coverage.", "questions": "3. Regarding the intuition about the positive data bias, I am pretty interested in the experiments in the Antmaze dataset from D4RL, where it is a multi-goal task and there does not exist positive data bias claimed by the authors.\n4. A detailed analysis of the influence of the three types of “wrong” reward labels can make this work more complete and robust.\n5. More experiments about safe RL are highly appreciated [1].\n6. In line 247 about the safe RL. \"This condition can be easily satisfied by filtering out unsafe states in post processing.\" From the previous work [2], one biggest reasons why we prefer offline RL over BC is that offline RL can learn that those states are dangerous. What does the author think about this?\n\n[1] Liu Z, Guo Z, Lin H, et al. Datasets and Benchmarks for Offline Safe Reinforcement Learning[J]. arXiv preprint arXiv:2306.09303, 2023. \n\n[2] Kumar A, Hong J, Singh A, et al. When should we prefer offline reinforcement learning over behavioral cloning?[J]. arXiv preprint arXiv:2204.05618, 2022.\n", "limitations": "see the above", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. A good warm-up experiment to help readers understand the motivation.\n2. Sound theoretical analysis to show the property of ‘survival instinct’ in current offline RL algorithms. \n3. Comprehensive experiments on current D4RL and meta-world benchmarks with different offline RL algorithms. ", "weaknesses": "1. There exists a gap between the intuition (longer trajectories in the data have a smaller optimality gap) and the definition of the Positive Data Bias. On the other hand, Positive Data Bias does not have a quantitative value for each dataset so that the reader can clearly see the difference between the different datasets.\n2. Authors claim that offline RL is intrinsically safe. However, missing relevant experiments to validate their claims. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Quantifying Positive Bias**\n\nWe would like to clarify that positive bias is a general concept of when approximate optimality to the CMDP implies approximate optimality w.r.t the true reward, while *\"longer trajectories in the data have a smaller optimality gap\"* refers to a special form of positive data bias called *length bias*. In Prop. 3 (Appendix C.4), we provide a few examples of positive data bias, including length bias.  \n \nWe argue that the performance gap between running an offline RL algorithm with true reward and other data rewards can be used to measure positive data bias (like our experiments). An interesting future direction is to quantify the positive bias in a dataset without having to run offline RL algorithms. We will discuss this in the limitations.\n \n**Inherent Safety**\n\nOur results on episode lengths for D4RL datasets in Fig. 1 (b) and Appendix B.4 are evident that offline RL is inherently safe. For hopper and walker, the cost can be defined by whether the agent falls, and an episode stops when the agent incurs a non-zero cost, i.e., an agent is the safest if it does not fall. In Fig. 7, we show that offline RL algorithms, e.g., ATAC and PSPI, can achieve long episode lengths even when the behavior policy is unsafe. We conduct new experiments on offline safety gymnasium [1] below, as the reviewer suggested.\n\n**Antmaze**\n\nAs discussed in Section 4 (line 260), we acknowledge not all datasets have positive bias. With that said, AntMaze datasets technically have a positive bias, albeit a weak one. Although AntMaze datasets are collected by multi-goal policies, the terminal flags are *re-labeled based on a single test-time goal*. In other words, trajectories reaching the test-time goal are marked as terminal (which mathematically has an infinite length) while others have a bounded length of at most 1001. We refer the reviewer to Appendices D.2 and D.3 where we formally discuss how this creates a positive data bias. \n \nHowever, such a length bias is hard to be picked up by a deep offline RL implementation, because failed trajectories, which can be very long, would have similar *discounted* returns as the successful ones. In fact, it has been proved that, for AntMaze medium and large datasets, good performance can only be achieved when using *an algorithm-specific transformed reward*: IQL requires one to be subtracted from the rewards and CQL requires the reward to be shifted and scaled to [-5, 5] [2].  During our earlier experiments, we found the results on the AntMaze inconclusive, as it is hard to separate optimization difficulty from the effect of different data rewards.\n \n**Analysis on Wrong Rewards**\n\nEmpirically, we observe that, for ATAC and PSPI, the learned policies from zero and random rewards are similar, as the random reward is in expectation a constant. In more diverse datasets, such as the medium-replay datasets, we often find policies learned with the negative reward worse than those learned using zero and random reward. \n \nOn the theoretical side, Definition 2 allows a dataset to have positive bias w.r.t. one of the wrong rewards, but not with the others. In Prop. 3 (Appendix C), we provide several factors (reward shaping reward, expert demonstrations, and length bias) that can lead to positive data bias. \n\n**Extra Experiments on Offline Safety RL**\n\nWe conduct experiments on offline safety gymnasium [1] and include the results in the attached PDF. We make a few remarks which are important to interpreting the results.\n1. Our notion of inherent safety is different from [1]. Ours (Corollary 2) focus on offline RL’s ability to produce safe policies given a dataset that *only* contains safe states, while [1] focuses on learning the concept of safety from a dataset which contains *both* safe and unsafe behaviors. Our notion of inherent safety can be more favorable when it is inadmissible/unethical to collect unsafe data. \n2. For this reason, the datasets in [1] contain unsafe states. We use a naïve filtering strategy, which removes all transitions with non-zero costs. This ensures our data only covers safe states, but it also means that we use less data than the comparators, algorithms in [1]. Better results may be possible with a more sophisticated filtering strategy.\n3. Many algorithms from [1] use a cost target, which sets the maximum total cost allowed by safety. [1] uses three different cost targets, {20, 40, 80}, and reports the normalized total reward and cost for each algorithm averaged over the three targets. By contrast, standard offline RL algorithms do not use such a cost target. We report results of offline RL algorithms according to an “effective cost target” of 34.29, which is similar to how BC_All results are presented in [1]. \n\nWe observe that *offline RL with a naïve data filtering strategy achieves comparable performance as the per-task best performing offline safe RL algorithm*. Our agents in general incur low cost except for in the circle tasks. We hope these new results can give the reviewer more confidence on the safety property of offline RL.\n \n**Filtering Unsafe States**\n\nFiltering often naturally happens during data collection, e.g., a robot is likely to be stopped when it’s about to crash. In this case, we have a dataset with only safe states, and some of the trajectories are incomplete due to intervention. By Corollary 2, offline RL can learn a safe policy on such a dataset, *provided that a safe in-support policy exists*. This is because due to survival instinct, offline RL algorithms would 1) assign a low value to such an unsafe state and 2) propagate this low value to states and actions encountered earlier. BC, however, may follow an unsafe trajectory and enter an unsafe state. This is demonstrated in the D4RL episode length results in Appendix B.4. \n \n[1] Liu et al. Datasets and Benchmarks for Offline Safe Reinforcement Learning. 2023\n\n[2] Tarasov et al. CORL: Research-oriented deep offline reinforcement learning library. 2022"}, {"review_id": "nC6KrgqhJG", "reviewer": "Reviewer_kbuj", "summary": "The authors make an interesting and important observation about offline Reinforcement Learning. They note that an agent can learn from offline data even when the reward signal is not the same one as that used to train the online agent. Moreover it can be vastly different, or even with no reward signal at all, within certain types of task and with certain datasets. In particular, when a dataset has limited and biased data coverage, the aim of an offline agent to stay within the data distribution means that it has an equivalent of a survival instinct. This point is noted first empirically and then proven formally as well as being shown in an extensive set off environments and settings.\n\nThe conditions necessary of the training data for an agent to learn a near-optimal policy are identified. Finally, a perspective on offline RL is given which means that the reward signal can be essentially ignored, so long as the appropriately biased dataset is provided.", "questions": "I think that a discussion under which circumstances an adversarial reward signal could be designed to attack an offline agent may be interesting. While it is clear that with the appropriately biased data, a random reward signal has little effect, there may be circumstances where a strong enough adversarial reward signal and an inappropriately biased dataset may not be sufficient to lead to near optimal performance.\n\nIs it possible to make claims in the multi-agent offline setting similar to those in the single agent setting?", "limitations": "Limitations of the work are given, but none of the potential societal impact of this work has been discussed. Given that Offline RL may well play a large role in ML systems going forward, this should be mentioned.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The originality and quality of the paper are both very strong. The combination of formal mathematical proofs of the insensitivity to the rewards and the experiments which show this to be true are married well together.  The paper as a whole is very well written, and the appendix is in particular very thorough and shows a good understanding of the need for transparency and clarity in experimental settings. The amount of formality within the bulk of the paper is also appropriate, giving enough to understand the results, but not to swamp them.\n\nThe nature of the findings are themselves original and are likely to have impact in the field of offline RL.", "weaknesses": "I see few weaknesses overall, although there are a few typos throughout:\n\nIn most of the figures, the number of seeds is provided, though in figure 1 it is not. This should be added.\nLine 120: \"worst case a necessary\"\nLine 143: \"In following\"\nLine 178: \"s, a\" may be clearer as (s,a).\nLine 214: \"This is contrast\"\nLine 227: \"Positive data bias assumption\"\nFootnote on page 5 has an additional bracket at the end.\nLine 232: \"benefits\"->\"benefit\"\nLine 245: \"states have a safe action\"\nLine 247: \"is also mild\"\nFootnote on page 6: \"we include a ablation\"\nLine 290: \"with wrong can\"\nLine 295 needs to be rewritten.\nLine 310: \"with provably pessimism\"\nLine 315: \"A implicit\"\nLine 318: \"due the\"\nLine 332: \"the end of episode\"\nLine 334: \"there is no timeouts\"\nLine 915 in the appendix: \"the followings are true\"\nLine 1025: \"first we the\"\nLine 1089: \"and let the goal state to be\"\n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are happy that the reviewer overall feels positive about our paper. We want to thank the reviewer for reviewing our paper and pointing out typos in detail. We will fix these typos in the final version. \n \n**Adversarial Reward**\n\nRegarding the question on adversarial reward, we agree with the reviewer that when a dataset does not have positive bias, offline RL may suffer from bad data reward. Indeed, in our experiments on the halfcheetah datasets, we did observe that the wrong data reward can make offline RL algorithms produce bad policies (see Fig. 3). From Definition 2, *theoretically*, an admissible (see Definition 4) offline RL learning a positively biased data can guard against any adversarial reward within the reward class $\\tilde{\\mathcal{R}}$ . However, we currently don’t have way to *empirically* verify whether a deep offline RL implementation is admissible, or quantify the amount of positive bias of a data distribution, like the size of reward class $\\tilde{\\mathcal{R}}$. Currently, our suggestion is to actually run the offline RL algorithm with wrong data rewards and see whether the learned behavior differs from that given by learning with the true reward. One interesting future direction is to investigate whether/how offline RL algorithms can be robust to adversarial reward that changes during training (Definition 2 only concerns a fixed reward).\n\n**Multi-agent Offline RL**\n\nAs the reviewer also pointed out, exploring whether multi-agent offline RL algorithms have a similar survival instinct is another interesting future direction. We will discuss these future research directions in the limitations in the final version of the manuscript. \n\n**Societal Impact**\n\nLastly, we agree with the reviewer on the growing societal impact of offline RL going forward. We think the discovery here might shed light on a potential positive impact of offline RL. By survival instinct, offline RL may be able to train sequential-decision policies that have good behaviors without needing to collect negative data. This is different from the common belief that RL can learn to behave well only if it has seen both positive (success) and negative (failure) data. This ability to learn from one-sided data is important especially for applications where collecting negative data is unethical, e.g., learning not to produce hateful speech by first collecting hateful speech, learning not to harm patients in medical applications by first harming some, etc. \n\nOn the flip side, also by survival instinct, offline RL can be prone to existing societal (e.g., gender, racial) biases in data, and, moreover, such a bias cannot be easily corrected by relabeling data with a different reward. As a result, when using an offline RL algorithm, more strategic thinking on data collection might be needed. We encourage researchers and practitioners to collect datasets using methodology such as those proposed in [1] and provide details such as how data was collected and cleaned so that users can assess whether it is appropriate to train offline RL algorithms with these datasets.\n\n[1]  Gebru, Timnit, et al. “Datasheets for datasets.” Communications of the ACM 2021."}]

Author rebuttal:
We thank all reviewers for providing feedback to our manuscript. It is encouraging to see that reviewers generally find our empirical findings and theoretical analysis relevant to the offline RL community. We are excited to see that the reviewers propose a few interesting future research directions, and we are happy to provide a discussion on these future directions in the revision. We also would like to thank reviewers for pointing out typos in our manuscript, which will be fixed in the final version. We will address specific questions and concerns from each reviewer in the individual responses.


In the attached PDF, we include a table which has additional experimental results on offline safety gymnasium [1] as per suggestions by reviewer k1TU. We observe that *offline RL with a naïve data filtering strategy can achieve comparable performance as the per-task best performing state-of-the-art offline safe RL algorithm*. We hope our new results can give reviewers more confidence on the safety property of offline RL. 

[1] Liu et al. Datasets and Benchmarks for Offline Safe Reinforcement Learning. 2023.

Meta-review: This paper explores a phenomenon in offline reinforcement learning (RL) where agents learn effective policies even with incorrect reward signals. The authors argue that this resilience arises due to the inherent pessimism in offline RL algorithms and biases in data collection, introducing a 'survival instinct'. Empirical demonstrations and theory, demonstrate this observation across various environments. The paper has clear exposition, comprehensive experiments, and sound theoretical analysis. Some reviewers raised concerns about the clarity of some arguments, especially regarding data biases and the claim of intrinsic safety. Some also highlighted the need for specific clarifications around situations where this phenomenon manifests and several typographical errors scattered throughout the paper. The authors addressed most of the reviewers' concerns, and the reviewers found the work to be potentially impactful for future work.
Decision: Accept (spotlight)