Survival Instinct in Offline Reinforcement Learning
and Implicit Human Bias in Data

Anqi Li 1 Dipendra Misra 2 Andrey Kolobov 2 Ching-An Cheng 2

Abstract

when trained on utterly wrong reward labels.

We present a novel observation about the behavior
of offline reinforcement learning (RL) algorithms:
on many benchmark datasets, offline RL can produce well-performing and safe policies even whentrained with “wrong” reward labels, such as thosethat are zero everywhere or are negatives of thetrue rewards. This phenomenon cannot be easily explained by offline RL’s return maximization
objective. Moreover, it gives offline RL a degreeof robustness that is uncharacteristic of its online
RL counterparts, which are known to be sensitive
to reward design. We demonstrate that this surprising robustness property is attributable to aninterplay between the notion of pessimism in offline RL algorithms and certain human biases implicit in common data collection practices. As weprove in this work, pessimism endows the agent
with a survival instinct, i.e., an incentive to staywithin the data support in the long term, while
the limited and biased data coverage further constrains the set of survival policies. We argue thatthe survival instinct should be taken into accountwhen interpreting results from existing offline RL
benchmarks and when creating future ones. Our
empirical and theoretical results suggest a newparadigm for RL, whereby an agent is “nudged”
to learn a desirable behavior with imperfect reward but purposely biased data coverage.

In Figure 1, we present such results on the hopper
task from D4RL (Fu et al., 2020), a popular offline RL
benchmark, using a state-of-the-art offline RL algorithm
ATAC (Cheng et al., 2022). The goal of an RL agent in
the hopper task is to move forward as fast as possible
while avoiding falling down. We trained ATAC agents on
the original datasets and on three modified versions of eachdataset, with “wrong” rewards: 1) zero: assigning a zeroreward to all transitions, 2) random: labeling each transition
with a reward sampled uniformly from [0, 1], and 3) negative: using the negation of the true reward. Although thesewrong rewards contain no information about the underlying
task or are even misleading, the policies learned from themin Figure 1 (left) often perform significantly better than the
behavior (data collection) policy and the behavior cloning(BC) policy. They even outperform policies trained with thetrue reward (denoted as original in Figure 1) in some cases.
This is puzzling, since RL is notorious for being sensitive
to reward mis-specification (Leike et al., 2017; HadfieldMenell et al., 2017; Ibarz et al., 2018; Pan et al., 2022):
in general, maximizing the wrong rewards with RL leads
to sub-optimal performance that can be worse than simply
performing BC. In addition, these wrong-reward policies
demonstrate a “safe” behavior, which keeps the hopperfrom falling down for a longer period than other comparators in Figure 1 (right). This is yet another peculiarity hardto link to return maximization, as none of the wrong rewards
encourage the agent to stay alive. As we will show empirically in Section 4, these effects are not unique to ATAC
or the hopper task. They occur with multiple offline RL
algorithms, including ATAC (Cheng et al., 2022), PSPI (Xieet al., 2021), IQL (Kostrikov et al., 2021), CQL (Kumar
et al., 2020) and the Decision Transformer (DT) (Chen et al.,2021), on dozens of datasets from D4RL (Fu et al., 2020)
and Meta-World (Yu et al., 2020a) benchmarks.

1. Introduction
In offline reinforcement learning (RL), an agent optimizes
its performance given an offline dataset. Despite being
its main objective, we find that return maximization is not
sufficient for explaining some of its empirical behaviors. In
particular, in many existing benchmark datasets, we observethat offline RL can produce surprisingly good policies evenThis robustness of offline RL is not only counter-intuitive
but also cannot be explained fully by the literature. Offline
RL theory (Cheng et al., 2022; Xie et al., 2021; Jin et al.,2021; Liu et al., 2020) provides performance guarantees
only when the data reward is the true reward. Although offline imitation learning (IL) (Uehara and Sun, 2022; Chang

1

University of Washington 2 Microsoft Research. Correspondence to: Anqi Li <anqil4@cs.washington.edu>.
Interactive Learning with Implicit Human Feedback Workshop at
ICML 2023.

1

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

implications. First and foremost, offline RL has a survival
instinct that leads to inherent robustness and safety properties that online RL does not have. Unlike online RL, offline
RL is doubly robust: as long as the data reward is corrector the data has a positive implicit bias, a pessimistic offline
RL agent can perform well. Moreover, offline RL is safe aslong as the data only contains safe states; thus safe RL in theoffline setup can be achieved without specialized algorithms.
Second, because of the existence of the survival instinct, thedata coverage has a profound impact on offline RL. Whilea large data coverage improves the best policy that can belearned by offline RL with the true reward, it can also makeoffline RL more sensitive to imperfect rewards. In otherwords, collecting a large set of diverse data might not benecessary or helpful (see Section 4). This goes against thecommon wisdom in the RL community that data should be
as exploratory as possible (Munos, 2003; Chen and Jiang,
2019; Yarats et al., 2022).

(a) Normalized scores(b) Episode lengths

Figure 1: On the hopper task from D4RL (Fu et al., 2020),
ATAC (Cheng et al., 2022), an offline RL algorithm, can produce high-performance and safe policies even when trainedon wrong rewards.

We emphasize that survival instinct’s implications should
be taken into account when interpreting results on existing offline RL benchmarks and designing future ones. We
should treat the evaluation of offline RL algorithms differently from online RL algorithms. We suggest evaluating
the performance of an offline RL algorithm by training itwith wrong rewards in addition to the true reward in order
to ablate the performance due to return maximization fromthe effects of survival instinct and implicit data bias.

et al., 2021; Ma et al., 2022b) makes no assumptions about
reward, it only shows that the learned policy can achieve
performance comparable to that of the behavior policy, notbeyond. Robust offline RL (Zhang et al., 2022; Wu et al.,2023; Zhou et al., 2021; Ma et al., 2022a; Panaganti et al.,2022) shows that specialized algorithms can perform wellwhen the size of data perturbation is small. In contrast, wedemonstrate that standard offline RL algorithms can produce good policies even when we completely change thereward. Constrained offline RL algorithms (Le et al., 2019;
Xu et al., 2022b; Lee et al., 2022a) can learn safe behaviors when constraint violations are both explicitly labeled
and optimized. However, in the phenomena we observe, nosafety signal is given to off-the-shelf, unconstrained offline
RL algorithms. In Appendix A, we discuss the gap between
the related work and our findings in more detail.

We believe that our findings shed new light on RL applicability and research. To practitioners, we demonstrate thatoffline RL does not always require the correct reward tosucceed. This opens up the possibility of using offline RL in
domains where obtain high-quality rewards is challenging.
Research-wise, the existence of the survival instinct raises
the question of how to design data collection or data filtering procedures that would help offline RL to leverage thisinstinct in order to improve RL’s performance with incorrector missing reward labels.

In this work, we provide an explanation for this seemingly
surprising observation. We prove that this robustness property is attributed to the interplay between the use of pessimism in offline RL algorithms and the implicit bias intypical data collection processes. Offline RL algorithms
often use pessimism to avoid taking actions that lead to unknown future events. We show that this risk-averse tendency
bakes a “survival instinct” into the agent, an incentive tostay within the data coverage in the long term. On the other
hand, the limited coverage of offline data further constrains
the set of survival policies (policies that remain in the data
support in the long term). When this set of survival policies
correlates with policies that achieve high returns w.r.t. thetrue reward (as in the example in Figure 1), robust behavior
emerges.

2. BackgroundThe goal of offline RL is to solve an unknown Markov decision process (MDP) from offline data. Typically, an offline
dataset is a collection of tuples, D := {(s, a, r, s′ )|(s, a) ∼µ(·, ·), r = r(s, a), s′ ∼ P (·|s, a)}, where r is the reward,
P captures the MDP’s transition dynamics, and µ denotes
the state-action data distribution induced by some data collection process. Modern offline RL algorithms adopt pessimism to address the issue of policy learning when µ does
not have the full state-action space as its support. The basic idea is to optimize for a performance lower bound thatpenalizes actions leading to out-of-support future states.
Such a penalty can take the form of behavior regularization (Kostrikov et al., 2021; Fujimoto and Gu, 2021; Wu
et al., 2019), negative bonuses to discourage visiting lessOur theoretical and empirical results have two important2

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Datafrequent state-action pairs (Jin et al., 2021; Rashidinejad
et al., 2021; Yu et al., 2020b), pruning less frequent actions(Liu et al., 2020; Kidambi et al., 2020), adversarial training (Cheng et al., 2022; Xie et al., 2021; Uehara and Sun,
2022; Xie et al., 2022; Rigter et al., 2022) or value penalties
in modified dynamic programming (Kumar et al., 2020; Yu
et al., 2021).

frequency of a single behavior policy, nor does it assumeµ has a full coverage of all states and actions or all policy
distributions (Chen and Jiang, 2019; Foster et al., 2022).

3. Why Offline RL can Learn Right Behaviors
from Wrong Rewards
We provide conditions under which offline RL’s aforementioned robustness w.r.t. misspecified rewards emerges. Our
main finding is summarized in the theorem below.

We are interested in settings where the offline RL agent
learns not from D itself but from its corrupted version D̃with a wrong reward r̃, i.e., D̃ := {(s, a, r̃, s′ )|(s, a, s′ ) ∈D, r̃ = r̃(s, a) ∈ [−1, 1]}. We assume r̃ is from a reward
class R̃ — which may not necessarily contain the true reward r — and we wish to explain why offline RL can learn
good behaviors from the corrupted dataset D̃ (e.g., as in Figure 1). To this end, we introduce notations and assumptions
we will use in the paper.

Theorem 3.1. (Informal) Under Assumption 2.1 and certain
regularity assumptions, if an offline RL algorithm Algo is
set to be sufficiently pessimistic and the data distribution µ
has a positive bias, for any data reward r̃ ∈ R̃, the policyπ̂ learned by Algo from the dataset D̃ has performance
guarantee∗

Vrπ (d0 ) − Vrπ̂ (d0 ) ≤ O(ι)

Notation We focus on the setting of infinite-horizon discounted MDPs. We denote the task MDP that the agent aims
to solve as M = (S, A, r, P, γ), where S is the state space,
A is the action space, and γ ∈ [0, 1) is the discount. Without
loss of generality, we assume r : S × A → [0, 1]. Let ∆(U)
denote the set of probability distributions over a set U. We
denote a policy as π : S → ∆(A). For a reward functionr, we P
define a policy π’s state value function as Vrπ (s) :=∞Eπ,P [ t=0 γ t r(st , at )|s0 = s] and the state-action valuefunction as Qπr (s, a) := r(s, a) + γEs′ ∼P (·|s,a) [Vrπ (s′ )].
Solving MDP M requires learning a policy that maximizes
the return at an initial state distribution d0 ∈ ∆(S), thatis, maxπ Es∼d0 [Vrπ (s)]. We denote the optimal policy as
π ∗ and the optimal value functions as Vr∗ and Q∗r . Given
d0 , we define the average state-action
P∞ visitation of a policy π as dπ (s, a) := (1 − γ)Eπ,P [ t=0 γ t dπt (s, a)], wheredπt (s, a) denotes the probability of visiting (s, a) at time
t when running π starting at an initial state sampled fromd0 . Note that (1 − γ)Es∼d0 [Vrπ (s)] = E(s,a)∼dπ [r(s, a)].
For a function f : S × A → R, we use the shorthandf (s, π) = Ea∼π|s [f (s, a)]; similarly for f : S → R, wewrite f (p) = Es∼p [f (s)], e.g., Vrπ (d0 ) = Es∼d0 [Vrπ (s)].
For a distribution p, we use supp(p) to denote its support.

as well as safety guaranteeP∞(1 − γ) t=0 γ t Prob (∃τ ∈ [0, t], sτ ∈/ supp(µ)|π̂) ≤ O(ι)
for a small ι that decreases to zero as the degree of pessimism and dataset size increase.
Here is our key insight: because of pessimism, offline RL endows agents with a “survival instinct” — it implicitly solves
a constrained MDP (CMDP) problem that enforces the policy to stay within the data support. When combined witha training data distribution that has a positive bias (e.g., allpolicies staying within data support are near-optimal) such asurvival instinct results in robustness to reward misspecification. Overall, Theorem 3.1 has two important implications:
1. Offline RL is doubly robust: it can learn near optimal
policies so long as either the reward label is correct or
the data distribution is positively biased;
2. Offline RL is intrinsically safe when data are safe,
regardless of reward labeling, without the need of explicitly modeling safety constraints.
In the following, we provide details and discussion of thestatements above. The complete theoretical statements andproofs can be found in Appendix C.

Assumption We make the typical assumption in offline
RL that the data distribution µ assigns positive probabilities
to these state-actions visited by running the optimal policyπ ∗ starting from d0 .

3.1. Warm-up: Intuitions for Survival Instinct andPositive Data Bias
We first use a grid world example to build some intuitions.
Figure 2 shows a goal-directed problem, where the true
reward is +1 and −1 upon touching the key and the lava,
respectively. The offline data is suboptimal and does nothave full support. All data trajectories that touched thelava were stopped early, while others were allowed to continue until the end of the episode. The goal state (key) isan absorbing state, where the agent can stay beyond theepisode length. We use the wrong rewards in Figure 1 toAssumption 2.1 (Single-policy Concentrability). We asπ∗(s,a)
sume sups∈S,a∈A dµ(s,a)< ∞.

This is a standard assumption in the offline RL literature,
which in the worst case is a necessary condition to have noregret w.r.t. π ∗ (Zhan et al., 2022). There are generalized notions (Xie et al., 2021; 2022) which are weaker but requires
other smoothness assumptions. We note that Assumption 2.1
does not assume that µ is the average state-action visitation
3

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

We prove that offline RL trained on D̃, because of its pessimism, implicitly solves the CMDP:Cµ (r̃) := C(S, A, r̃, cµ , P, γ),(1)where cµ (s, a) := 1[µ(s, a) = 0] indicates whether s, a isout of the support of µ1 .
Proposition 3.3 (Survival Instinct). (Informal) Under certain regularity conditions on Cµ (r̃), the policy learned by
an offline RL algorithm Algo with the offline dataset D̃ isι-approximately optimal to Cµ (r̃), for some small ι which
decreases as the algorithm becomes more pessimistic.

Figure 2: A grid world. BC (red); offline RL with wrong
rewards (blue). The opacity indicates the frequency of astate in the data (more opaque means more frequent).

Proposition 3.3 says if an offline RL algorithm is sufficiently
pessimistic, then the learned policy is approximately optimal
to Cµ (r̃). The learned policy has not only small regretwith respect to the data reward r̃, but also small chances of
escaping the support of the data distribution µ.

train PEVI (Jin et al., 2021), a finite-horizon, tabular offline
RL method. PEVI performs dynamic programming similar
to value iteration, but with a pessimistic value initialization and an instantaneous pessimism-inducing penalty of√O(−1/ n(s,a)), where n(s, a) is the empirical count in data.
The penalties ensure that the learned value lower bounds thetrue one.

We highlight that the survival instinct in Proposition 3.3
is a long-term behavior. Such a survival behavior cannot
be achieved by myopically staying within the action support of the data distribution in general (such as BC).2 For
instance, when some trajectories generating the data aretruncated (e.g., due to early-stopping or intervention forsafety reasons), taking in-support actions may still lead toout-of-support states in the future, as the BC agent in Section 3.1.

We see the offline RL agent learned with any wrong reward
in Figure 1 is able to solve the problem despite data imperfection, while the BC agent that mimics the data directly
fails. The main reasons are: 1) There is a data bias whereby
longer trajectories are closer to the goal, which we call alength bias (see Section 3.3). This positive data bias isdue to bad trajectories (touching the lava or not reaching
the goal) being cut short or timing out. 2) Pessimism in
PEVI gives the agent an algorithmic bias (survival instinct)
that favors longer trajectories. Because of the pessimistic
value initialization, PEVI treats trajectories shorter than theepisode length as having the lowest return. As a result,by maximizing the pessimistically estimated values, PEVI
learns good behaviors despite wrong rewards by leveraging
the survival instinct and positive data bias together. We now
make this claim more general.

The key to prove Proposition 3.3 is to show that an offline
RL algorithm by pessimism has small regret for not just r̃
but a set of reward functions consistent with r̃ on data but
different outside of data, including the Lagrange reward of(1) (i.e., r̃ − λcµ , with λ ≥ 0). We call this property admissibility and we show in Appendix E that many existingoffline RL algorithms are admissible, including model-freealgorithms ATAC (Cheng et al., 2022), VI-LCB (Rashidinejad et al., 2021) PPI/PQI (Liu et al., 2020), PSPI (Xie et al.,2021), as well as model-algorithms, ARMOR (Xie et al.,2022; Bhardwaj et al., 2023), MOPO (Yu et al., 2020b),
MOReL (Kidambi et al., 2020), and CPPO (Uehara andSun, 2022). By Proposition 3.3 these algorithms have survival instinct. Please see Appendix C for details.

3.2. Survival Instinct
The first key factor to the robustness of offline RL is asurvival instinct originating from its pessimistic reasoning: offline RL algorithms tend to favor policies leading
to in-support trajectories. Equivalently, offline RL implicitly solves a CMDP where the constraint enforces the agent’s
trajectories to stay within the data support, even when thealgorithm does not explicitly model any constraints. We introduce some definitions to characterize this behavior more
concretely.

3.3. Positive Data Bias
In addition to survival instinct, another key factor is an
implicit positive human bias in common offline datasets.
Typically these data manifest meaningful behaviors: For
example, in collecting data for goal-oriented problems, data
recording is normally stopped if the agent fails to reach theDefinition 3.2. Let f, g : S × A → [−1, 1]. A CMDP
C(S, A, f, g, P, γ) is a constrained optimization problem:
maxπ Vfπ (d0 ) s.t. Vgπ (d0 ) ≤ 0. Let π † denote its optimal
policy. For δ ≥ 0, we define the set of δ-approximately†
optimal policies Π† (δ) := {π : Vfπ (d0 ) − Vfπ (d0 ) ≤πδ, Vg (d0 ) ≤ δ}.

1
The use of an indicator function is not crucial here. We can
extend the current analysis here to other costs that are zero in the
support and strictly positive out of the support.
2
BC requires a stronger condition to have survival instinct,
such as when the data distribution µ is a (mixture of) averagestate-action visitation(s).

4Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

goal within certain time limit. Another example is problems
where wrong decisions can have detrimental consequences,such as in robotics and healthcare scenarios (i.e. problems
where safe RL focuses). In these domains, the data arecollected by qualified policies only or under an intervention
mechanism to prevent catastrophic failures.

we observe in the hopper example from Section 1. On
the other hand, there are also cases where IL can learn well,while positive data bias does not exist.
3.4. Summary: Offline RL is Doubly Robust andIntrinsically Safe
We have discussed the survival instinct from pessimism andthe potential positive data bias in common datasets. Since
survival instinct makes the policy of offline RL benefits froma favorable inductive bias of staying within the support of apositive data distribution, together they imply the robustnessof offline RL (namely, Theorem 3.1).

Such a one-sided bias can creates an effect that staying
within data support would lead to meaningful behaviors.
Below we formally define the positive data bias.
Definition 3.4 (Positive Data Bias). A distribution µ is1
ϵ -positively biased w.r.t. a reward class R̃ if∗

max max Vrπ (d0 ) − Vrπ (d0 ) ≤ ϵ + O(δ)
r̃∈R̃ π∈Π† (δ)

We can view this phenomenon as a doubly robust property
of offline RL. We borrow the name “doubly robust” fromthe offline policy evaluation literature (Dudı́k et al., 2011)
to highlight the robustness of offline RL to reward misspecification as an offline policy optimization approach.

(2)

for all δ ≥ 0, where Π† (δ) denotes the set of δapproximately optimal policy of Cµ (r̃).

Corollary 3.5. Under Assumption 2.1, offline RL can learna near optimal policy, as long as the reward is correct, or
the data has a positive implicit bias.

We measure the degree of positive data bias based on how
bad a policy can perform in terms of the true reward r whenapproximately solving the CMDP Cµ (r̃) in (1) defined withthe wrong reward r̃ ∈ R̃. If the data distribution µ is ∞positively biased, then any approximately optimal policy toCµ (r̃) (which includes the policies learned by offline RL,
as shown earlier) can achieve high return in the true rewardr. We also can view the degree of positiveness as reflecting
whether r̃ provides a similar ranking as r, among policies
within the support of µ. When there is a positive bias, offline
RL can learn with r̃ to perform well under r, even whenr̃ is not aligned with r globally. This is contrast to online
RL, which requires global alignment due to the exploratory
nature of online RL.

Theorem 3.1 also implies that offline RL is an intrinsicallysafe learning algorithm, unlike its counterpart online RL
where additional penalties or constraints need to be explicitly modelled (Achiam et al., 2017; Wachi and Sui, 2020;
Wagener et al., 2021; Paternain et al., 2022; Nguyen andCheng, 2023).
Corollary 3.6. If µ only covers safe states and there exists
a safe policy staying within the support of µ, then the policy
of offline RL only visits safe states with high probability(see Theorem 3.1).
We should note that covering safe states is a very mild
assumption in the safe RL literature (Wagener et al., 2021;
Hans et al., 2008), e.g., compared with all (data) states
have a safe action. It does not require the data collection
policies that generate µ are safe. This condition can be easily
satisfied by filtering out unsafe states in post processing.
The existence of a safe policy is also mild and common
assumption.

Here we provide a few example data distributions that havea positive data bias. First, any policy is ∞-positively biased for any rewards resulting from potential-based reward
shaping (Ng et al., 1999), since it provides the same ranking for all policies. Second, for a setup similar to IL, thedata distribution µ is ∞-positively biased for any rewardsr̃ : S × A → R if it is generated by the optimal policy,⋆i.e., µ = dπ . Finally, a data distribution can have a positive bias when longer trajectories in the data have smaller
optimality gap. This condition is typically satisfied whenintervention is taken in the data collection process (despite
the data collection policy being suboptimal), as in the motivating example in Figure 1. Later in Section 4, we will
investigate deeper into this kind of length bias empirically.
We formalize these conditions and provide more technicaldiscussions in Appendix C.

4. Experiments
We empirically study the performance of offline RL algorithms under wrong rewards in a variety of tasks. We use thesame set of wrong rewards as in Figure 1. 1) zero: the zeroreward, 2) random: labeling each transition with a reward
value randomly sampled from Unif[0, 1], and 3) negative:
the negation of true reward. We consider five offline RL algorithms, ATAC (Cheng et al., 2022), PSPI (Xie et al., 2021),
IQL (Kostrikov et al., 2021), CQL (Kumar et al., 2020) anddecision transformer (DT)3 (Chen et al., 2021). We deliber-

Remark We highlight that the positive data bias assumption in Definition 3.4 is different from assuming that thedata is collected by expert policies, which is typical in the
IL literature. Positive data bias assumption can hold in cases
when data is generated by highly suboptimal policies, which3

We condition the decision transformer on the return of a trajectory sampled randomly from trajectories that achieve top 10%of returns in terms of data reward.

5

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

Figure 3: Normalized scores for locomotion tasks from D4RL. The mean and standard error for normalized scores arecomputed across 10 random seeds. For each random seed, we evaluate the final policy of each algorithm over 50 episodes.
ately choose offline RL algorithms to cover those that areprovably pessimistic (Cheng et al., 2022; Xie et al., 2021)
and those that are popular among practitioners (Kostrikov
et al., 2021; Kumar et al., 2020), as well as an unconventional offline RL algorithm (Chen et al., 2021). We consider
a variety of tasks from D4RL (Fu et al., 2020) and MetaWorld (Yu et al., 2020a) ranging from safety-critical tasks(i.e., the agent dies when reaching bad states), goal-oriented
tasks, and tasks that belong to neither. We train a total ofaround 16k offline RL agents (see Appendix B.8). Please
see Appendix B for details.

with or without positive data bias.
4.1. Locomotion Tasks from D4RL
We evaluate offline RL algorithms on three locomotion
tasks, hopper, walker2d4 and halfcheetah, from
D4RL (Fu et al., 2020), a widely used offline RL benchmark. For each task, we consider four datasets with different
qualites: random, medium, medium-replay, and mediumexpert. We refer readers to (Fu et al., 2020) for the construction of these datasets. We measure policy performance in
D4RL normalized scores (Fu et al., 2020). We provide threebaselines 1) behavior: normalized score directly computedfrom the dataset, 2) BC: the behavior cloning policy, and 3)
original: the policy produced by the offline RL algorithm using the original dataset (with true reward). The normalized
scores for baselines and offline RL algorithms with wrong
rewards are presented in Figure 3. The exact numbers canbe found in tables in Appendix B.

Messages We would like to convey three main messages
through our experiments. First, implicit data bias can exist
naturally in a wide range of datasets, and offline RL algorithms that are sufficiently pessimistic can leverage such abias to succeed when given wrong rewards. Second, offline
RL algorithms, regardless of how pessimistic they are, become sensitive to reward when the data does not possess apositive bias. Third, offline RL algorithms without explicit
pessimism, e.g., IQL (Kostrikov et al., 2021), can sometimes
still be pessimistic enough to achieve good performance under wrong rewards.

Data bias exists in some D4RL datasets. We visualize the length bias of D4RL datasets in Figure 4. Hereare our main observations. 1) Most datasets for hopper
and walker2d, with the exception of the medium-replay
datasets, have a strong length bias, where longer trajectories have higher returns. This length bias is due to thesafety-critical nature of hopper and walker2d tasks, asRemark on negative results We consider “negative” results, i.e., when offline RL fails under wrong rewards, as
important as the positive ones. Since they tell us how positive data bias can be broken or avoided. We hope our study
can provide insights to researchers who are interested in actively incorporating positive bias in data collection, as wellas who hope to design offline RL benchmarks specifically4
We remove terminal transitions from hopper andwalker2d datasets when learning with wrong rewards. In Appendix B, we include a ablation study on the effect of removing
terminals when using true reward.

6

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

Offline RL requires good reward to perform well on
datasets without length bias. In all four halfcheetah
datasets, the trajectories all have the same length, as isdemonstrated in Figure 4. This means that there is no data
bias. We observe that all algorithms with wrong rewards
at best perform similarly as the behavior and BC policies
in most cases; this is an imitation-like effect due to the survival instinct. In the halfcheetah-medium-expert
dataset, since there are a variety of surviving trajectories,
with score from 0 to near 100, we observe that the performance of the resulting policy degrades as data reward
becomes more different from the true reward.

Figure 4: A visualization of length bias in datasets from
D4RL (Fu et al., 2020). Each plot corresponds to a dataset
for a task (row) with a dataset (column). Each trajectory ina dataset is represented as a data point with the x-coordinate
being its episode length and y-coordinate being its normalized score.

Offline RL algorithms with provably pessimism produce
safer policies. For hopper and walker2d, we observethat policies learned by ATAC and PSPI, regardless of the
reward, can keep the agent from falling for longer. The
episode lengths of IQL and CQL policies are often comparable to that of the behavior policies. We provide statistics
of episode lengths in Appendix B.

the trajectories get terminated when reaching bad states.
The length bias is especially salient in hopper-medium,
where the normalized score is almost proportional to episodelength. 2) The medium-replay datasets for hopper andwalker2d have more diverse behavior, so the bias issmaller. 3) All halfcheetah datasets do not have lengthbias, as they all have the same length of 1000.

Remark on benchmarking offline RL Our observations
on the popular D4RL datasets raise an interesting question
for benchmarking offline RL algorithms. A implicit positive
data bias can give certain algorithms a hidden advantage, as
they can already achieve good performance without using
the reward. Without controlling data bias, it is hard todifferentiate whether the performance of those algorithms
are due to their ability to maximize returns or simply due
the their survival instinct.

Offline RL can learn good policies with wrong rewards
on datasets with strong length bias. On datasets withstrong length bias (hopper-random, hopper-medium
and walker2d-medium), we observe that ATAC
and PSPI with wrong rewards generally produce wellperforming policies, in a few cases even out-performing
the policies learned from the true reward (original in Figure 4). DT is mostly insensitive to reward quality. IQL andCQL with wrong can sometimes achieve good performance;
among the two, we find IQL to be more robust to wrong rewards and CQL with wrong rewards almost fails completelyon all walker2d datasets.

4.2. Goal-oriented Manipulation Tasks from
Meta-World
We study goal-oriented manipulation tasks from the MetaWorld benchmark (Yu et al., 2020a). We consider 15 goaloriented tasks from Meta-World: the 10 tasks from theMT-10 set and the 5 testing tasks from the ML-45 set. For
each task, we generate a dataset of 110 trajectories using
the scripted policies provided by (Yu et al., 2020a); 10 are
produced by the scripted policy, and 100 are generated byperturbing the scripted policy with a zero-mean Gaussian
noise of standard deviation 0.5. In the datasets, unsuccessful
trajectories receive a time-out signal after 500 steps (maximum episode length for Meta-World).

Offline RL needs stronger reward signals on datasetswith diverse behavior policy. The medium-replay
datasets of hopper and walker2d are generated frommultiple behavior policies. Due to their diverse nature, they
are a multiple ways to stay within data support in a long termwith various performance. As a result, the survival instinctof offline RL by itself is not sufficient to guarantee good
performance. Generally, algorithms with wrong rewards
under-perform the policies trained with true reward in theexperiments. As datasets have a more diverse coverage, they
can become less positively biased and offline RL requires astronger reward signal to differentiate good survival policies
and the bad ones. Practically speaking, when high-quality
reward labeling is not available, a diverse dataset can evenhurt offline RL performance. This is contrary to the common
belief that larger data support is more preferable (Munos,
2003; Chen and Jiang, 2019; Yarats et al., 2022).

We measure the success rate of policies for 50 new goals
unseen during training. Similar to D4RL experiments, weconsider BC policies and policies trained with the true reward (original) as baselines. Since the training dataset isgenerated for a different set of goals, we do not compare thesuccess rate of learned policies with the success rate in data.
Goal-oriented problems have data bias. Goal-oriented
problems are fundamentally different from safety-criticaltasks (such as hopper and walker2d). A good goaloriented policy should reach the goal as fast as possible
7

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

Figure 5: Success rate for goal-oriented tasks from Meta-World. The average success rate and confidence interval arecomputed across 10 random seeds. For each seed, we evaluate final policies for 50 episodes, each with a new goal unseen in
the dataset.
rather than wandering around until the end of episode. Nevertheless, another form of length bias still exists here, because successful trajectories are labeled with a termination
signal that indicates there is no timeouts and failed trajectories have a bounded length (see Section 3.1).

robustness to reward in a few tasks such as button-press,
drawer-close and door-unlock. Interestingly, DT performs
almost uniformly across all rewards, potentially because DT
does not explicitly maximize returns.
Remark on BC performance It is noticeable that BC
policies in general achieve high performance, in many tasks
often out-performing offline RL policies learned with true
reward. This effect has also been observed in existing workon similar tasks (Zhou et al., 2022). We would like to clarify
that the goal of our experiments is to study the behavior
of offline RL algorithms when trained with wrong rewards,
rather than showing that offline RL performs better thanBC. We refer interested readers to existing studies (Zhouet al., 2022; Kumar et al., 2022) on when using offline RL
algorithms is or is not preferable over BC.

Offline RL can learn with wrong rewards on goaloriented tasks. The success rate of learned policies withdifferent rewards are shown in Figure 5. We observe that
ATAC and PSPI with wrong rewards generally achieve comparable or better success rate than BC. The exceptions areoften due to that ATAC or PSPI, even with the true reward,
does not work as well as BC in those tasks, e.g. drawer-open,
bin-picking and box-close. This could be caused by overfitting or optimization errors (due to challenges of dynamics
programming over long horizon problems). In a number oftasks, such as peg-insert-side, push and reach, ATAC andPSPI with wrong rewards can out-perform BC by a margin.
This shows that data collection for goal-oriented problems
have a positive bias. This is remarkable as when learningwith random and negative rewards, unsuccessful trajectories
are long and generally have significantly higher return, as
reward for each step is non-negative. The offline RL algorithms need to be sufficiently pessimistic and be able to planover a long horizon to propagate pessimism to the unsuccessful data trajectories. For IQL, there is a gentle performancedecrease as the reward becomes more different than the true
reward, even though policies learned with wrong rewards
are not much worse than BC in many cases. CQL shows

5. Concluding Remarks
We present unexpected results on the robustness of offline
RL to reward mis-specification. We show that this property
originates from the interaction between the survival instinctof offline RL and hidden positive biases in common datacollection processes. Our findings suggest extra considerations should be taken when interpreting offline RL results
and designing future benchmarks. In addition, our findings
open a new space for offline RL research: the possibility ofdesigning algorithms that proactively leverage the survival
instinct to learn policies for domains where rewards are
nontrivial to specify or unavailable.
8

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

References

Fujimoto, S. and Gu, S. S. (2021). A minimalist approach
to offline reinforcement learning. Advances in neural
information processing systems, 34:20132–20145.

Achiam, J., Held, D., Tamar, A., and Abbeel, P. (2017). Constrained policy optimization. In International conferenceon machine learning, pages 22–31. PMLR.

Hadfield-Menell, D., Milli, S., Abbeel, P., Russell, S. J., andDragan, A. (2017). Inverse reward design. Advances inneural information processing systems, 30.

Altman, E. (1999). Constrained Markov decision processes,
volume 7. CRC press.

Hans, A., Schneegaß, D., Schäfer, A. M., and Udluft, S.
(2008). Safe exploration for reinforcement learning. In
ESANN, pages 143–148. Citeseer.

Bhardwaj, M., Xie, T., Boots, B., Jiang, N., and Cheng, C.A. (2023). Adversarial model for offline reinforcement
learning. arXiv preprint arXiv:2302.11048.

Hu, H., Yang, Y., Zhao, Q., and Zhang, C. (2023). The
provable benefits of unsupervised data sharing for offline
reinforcement learning. In International Conference on
Learning Representations.

Chang, J., Uehara, M., Sreenivas, D., Kidambi, R., and Sun,W. (2021). Mitigating covariate shift in imitation learning
via offline data with partial coverage. Advances in Neural
Information Processing Systems, 34:965–979.

Ibarz, B., Leike, J., Pohlen, T., Irving, G., Legg, S., andAmodei, D. (2018). Reward learning from human preferences and demonstrations in atari. Advances in neural
information processing systems, 31.

Chen, J. and Jiang, N. (2019). Information-theoretic considerations in batch reinforcement learning. In International Conference on Machine Learning, pages 1042–
1051. PMLR.

Jin, Y., Yang, Z., and Wang, Z. (2021). Is pessimism provably efficient for offline rl? In International Conference
on Machine Learning, pages 5084–5096. PMLR.

Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A.,
Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I.
(2021). Decision transformer: Reinforcement learning
via sequence modeling. Advances in neural information
processing systems, 34:15084–15097.

Kidambi, R., Chang, J., and Sun, W. (2021). Mobile:
Model-based imitation learning from observation alone.
Advances in Neural Information Processing Systems,
34:28598–28611.

Chen, Y., Zhang, X., Zhang, K., Wang, M., and Zhu, X.
(2023). Byzantine-robust online and offline distributed
reinforcement learning. In International Conference on
Artificial Intelligence and Statistics, pages 3230–3269.
PMLR.

Kidambi, R., Rajeswaran, A., Netrapalli, P., and Joachims,
T. (2020). Morel: Model-based offline reinforcement
learning. Advances in neural information processing
systems, 33:21810–21823.

Cheng, C.-A., Xie, T., Jiang, N., and Agarwal, A. (2022).
Adversarially trained actor critic for offline reinforcement
learning. In International Conference on Machine Learning. PMLR.

Kim, G.-H., Seo, S., Lee, J., Jeon, W., Hwang, H., Yang,
H., and Kim, K.-E. (2021). Demodice: Offline imitation learning with supplementary imperfect demonstrations. In International Conference on Learning Representations.

Dai, B., Nachum, O., Chow, Y., Li, L., Szepesvári, C., andSchuurmans, D. (2020). Coindice: Off-policy confidence
interval estimation. Advances in neural information processing systems, 33:9398–9411.

Kostrikov, I., Nair, A., and Levine, S. (2021). Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations.

Dudı́k, M., Langford, J., and Li, L. (2011). Doubly robust
policy evaluation and learning. In Proceedings of the 28th
International Conference on International Conference on
Machine Learning, pages 1097–1104.

Kumar, A., Hong, J., Singh, A., and Levine, S. (2022).
Should i run offline reinforcement learning or behavioral
cloning? In International Conference on Learning Representations.

Foster, D. J., Krishnamurthy, A., Simchi-Levi, D., and Xu,
Y. (2022). Offline reinforcement learning: Fundamental
barriers for value function approximation. In Conference
on Learning Theory, pages 3489–3489. PMLR.

Kumar, A., Zhou, A., Tucker, G., and Levine, S. (2020).
Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems,
33:1179–1191.

Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S.
(2020). D4rl: Datasets for deep data-driven reinforcement
learning. arXiv preprint arXiv:2004.07219.

Le, H., Voloshin, C., and Yue, Y. (2019). Batch policy
learning under constraints. In International Conference
on Machine Learning, pages 3703–3712. PMLR.
9

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

Lee, J., Paduraru, C., Mankowitz, D. J., Heess, N., Precup,
D., Kim, K.-E., and Guez, A. (2022a). Coptidice: Offline
constrained reinforcement learning via stationary distribution correction estimation. In International Conference
on Learning Representations.

Pan, A., Bhatia, K., and Steinhardt, J. (2022). The effects
of reward misspecification: Mapping and mitigating misaligned models. In International Conference on Learning
Representations.
Panaganti, K., Xu, Z., Kalathil, D., and Ghavamzadeh, M.
(2022). Robust reinforcement learning using offline data.
In Advances in Neural Information Processing Systems.

Lee, S., Seo, Y., Lee, K., Abbeel, P., and Shin, J. (2022b).
Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot
Learning, pages 1702–1712. PMLR.

Paternain, S., Calvo-Fullana, M., Chamon, L. F., andRibeiro, A. (2022). Safe policies for reinforcement learning via primal-dual methods. IEEE Transactions on Automatic Control.

Leike, J., Martic, M., Krakovna, V., Ortega, P. A., Everitt,T., Lefrancq, A., Orseau, L., and Legg, S. (2017). Ai
safety gridworlds. arXiv preprint arXiv:1711.09883.

Polosky, N., Da Silva, B. C., Fiterau, M., and Jagannath,
J. (2022). Constrained offline policy optimization. In
International Conference on Machine Learning, pages
17801–17810. PMLR.

Li, A., Boots, B., and Cheng, C.-A. (2023a). Mahalo: Unifying offline reinforcement learning and imitation learningfrom observations. arXiv preprint arXiv:2303.17156.

Rashidinejad, P., Zhu, B., Ma, C., Jiao, J., and Russell, S.
(2021). Bridging offline reinforcement learning and imitation learning: A tale of pessimism. Advances in Neural
Information Processing Systems, 34:11702–11716.

Li, J., Hu, X., Xu, H., Liu, J., Zhan, X., Jia, Q.-S., andZhang, Y.-Q. (2023b). Mind the gap: Offline policy
optimization for imperfect rewards. In International Conference on Learning Representations.

Riedmiller, M. (2005). Neural fitted q iteration–first experiences with a data efficient neural reinforcement learning
method. In Machine Learning: ECML 2005: 16th European Conference on Machine Learning, Porto, Portugal, October 3-7, 2005. Proceedings 16, pages 317–328.
Springer.

Liu, Y., Swaminathan, A., Agarwal, A., and Brunskill, E.
(2020). Provably good batch off-policy reinforcement
learning without great exploration. Advances in neural
information processing systems, 33:1264–1274.
Liu, Z., Guo, Z., Yao, Y., Cen, Z., Yu, W., Zhang, T.,
and Zhao, D. (2023). Constrained decision transformer
for offline safe reinforcement learning. arXiv preprint
arXiv:2302.07351.

Rigter, M., Lacerda, B., and Hawes, N. (2022). Rambo-RL:
Robust adversarial model-based offline reinforcement
learning. Advances in neural information processing
systems.

Ma, X., Liang, Z., Xia, L., Zhang, J., Blanchet, J., Liu,
M., Zhao, Q., and Zhou, Z. (2022a). Distributionallyrobust offline reinforcement learning with linear function
approximation. arXiv preprint arXiv:2209.06620.

Shi, L. and Chi, Y. (2022). Distributionally robust modelbased offline reinforcement learning with near-optimal
sample complexity. arXiv preprint arXiv:2208.05767.
Singh, A., Yu, A., Yang, J., Zhang, J., Kumar, A., andLevine, S. (2020). Cog: Connecting new skills to pastexperience with offline reinforcement learning. In Conference on Robot Learning. PMLR.

Ma, Y., Shen, A., Jayaraman, D., and Bastani, O. (2022b).
Versatile offline imitation from observations and examples via regularized state-occupancy matching. In International Conference on Machine Learning, pages 14639–
14663. PMLR.

Smith, M., Maystre, L., Dai, Z., and Ciosek, K. (2023).
A strong baseline for batch imitation learning. arXiv
preprint arXiv:2302.02788.

Munos, R. (2003). Error bounds for approximate policy iteration. In International Conference on Machine Learning,volume 3, pages 560–567. Citeseer.

Sutton, R. S. and Barto, A. G. (2018). Reinforcement learning: An introduction. MIT press.

Ng, A. Y., Harada, D., and Russell, S. (1999). Policy invariance under reward transformations: Theory and application to reward shaping. In International Conference on
Machine Learning, volume 99, pages 278–287. Citeseer.

Uehara, M. and Sun, W. (2022). Pessimistic model-basedoffline reinforcement learning under partial coverage. In
International Conference on Learning Representations.

Nguyen, H.-A. and Cheng, C.-A. (2023). Provable resetfree reinforcement learning by no-regret reduction. arXiv
preprint arXiv:2301.02389.

Van Hasselt, H., Doron, Y., Strub, F., Hessel, M., Sonnerat,N., and Modayil, J. (2018). Deep reinforcement learningand the deadly triad. arXiv preprint arXiv:1812.02648.
10

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

Wachi, A. and Sui, Y. (2020). Safe reinforcement learningin constrained markov decision processes. In International Conference on Machine Learning, pages 9797–
9806. PMLR.

Yu, T., Kumar, A., Rafailov, R., Rajeswaran, A., Levine,S., and Finn, C. (2021). Combo: Conservative offlinemodel-based policy optimization. Advances in neural
information processing systems, 34:28954–28967.

Wagener, N. C., Boots, B., and Cheng, C.-A. (2021). Safe
reinforcement learning using advantage-based intervention. In International Conference on Machine Learning,
pages 10630–10640. PMLR.

Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn,C., and Levine, S. (2020a). Meta-world: A benchmark
and evaluation for multi-task and meta reinforcement
learning. In Conference on robot learning, pages 1094–
1100. PMLR.

Wu, F., Li, L., Zhang, H., Kailkhura, B., Kenthapadi, K.,
Zhao, D., and Li, B. (2023). Copa: Certifying robust
policies for offline reinforcement learning against poisoning attacks. In International Conference on Learning
Representations.

Yu, T., Thomas, G., Yu, L., Ermon, S., Zou, J. Y., Levine, S.,
Finn, C., and Ma, T. (2020b). Mopo: Model-based offline
policy optimization. Advances in Neural Information
Processing Systems, 33:14129–14142.

Wu, Y., Tucker, G., and Nachum, O. (2019). Behavior
regularized offline reinforcement learning. arXiv preprint
arXiv:1911.11361.

Yu, W., Liu, C. K., and Turk, G. (2019). Policy transfer
with strategy optimization. In International Conference
on Learning Representations.

Xie, T., Bhardwaj, M., Jiang, N., and Cheng, C.-A. (2022).
Armor: A model-based framework for improving arbitrary baseline policies with offline data. arXiv preprint
arXiv:2211.04538.

Yue, S., Wang, G., Shao, W., Zhang, Z., Lin, S., Ren, J., andZhang, J. (2023). CLARE: Conservative model-based
reward learning for offline inverse reinforcement learning. In International Conference on Learning Representations.

Xie, T., Cheng, C.-A., Jiang, N., Mineiro, P., and Agarwal,
A. (2021). Bellman-consistent pessimism for offline reinforcement learning. Advances in neural information
processing systems, 34:6683–6694.

Zhan, W., Huang, B., Huang, A., Jiang, N., and Lee, J.
(2022). Offline reinforcement learning with realizability and single-policy concentrability. In Conference on
Learning Theory, pages 2730–2775. PMLR.

Xu, H., Zhan, X., Yin, H., and Qin, H. (2022a).
Discriminator-weighted offline imitation learning from
suboptimal demonstrations. In International Conference
on Machine Learning, pages 24725–24742. PMLR.

Zhang, X., Chen, Y., Zhu, X., and Sun, W. (2022).
Corruption-robust offline reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pages 5757–5773. PMLR.

Xu, H., Zhan, X., and Zhu, X. (2022b). Constraints penalized q-learning for safe offline reinforcement learning. In
AAAI.

Zhou, G., Ke, L., Srinivasa, S., Gupta, A., Rajeswaran,A., and Kumar, V. (2022). Real world offline reinforcement learning with realistic data source. arXiv preprint
arXiv:2210.06479.

Xu, W., Ma, Y., Xu, K., Bastani, H., and Bastani, O.
(2023). Uniformly conservative exploration in reinforcement learning. In International Conference on Artificial
Intelligence and Statistics, pages 10856–10870. PMLR.

Zhou, Z., Zhou, Z., Bai, Q., Qiu, L., Blanchet, J., and Glynn,
P. (2021). Finite-sample regret bound for distributionallyrobust offline tabular reinforcement learning. In International Conference on Artificial Intelligence and Statistics,
pages 3331–3339. PMLR.

Yang, R., Bai, C., Ma, X., Wang, Z., Zhang, C., and Han, L.
(2022). Rorl: Robust offline reinforcement learning via
conservative smoothing. In Advances in Neural Information Processing Systems.

Zhu, Z., Lin, K., Dai, B., and Zhou, J. (2020). Off-policy
imitation learning from observations. Advances in Neural
Information Processing Systems, 33:12402–12413.

Yarats, D., Brandfonbrener, D., Liu, H., Laskin, M., Abbeel,P., Lazaric, A., and Pinto, L. (2022). Don’t change thealgorithm, change the data: Exploratory data for offline reinforcement learning. arXiv preprint arXiv:2201.13425.

Ziebart, B. D., Maas, A. L., Bagnell, J. A., Dey, A. K., et al.
(2008). Maximum entropy inverse reinforcement learning. In Proceedings of the AAAI Conference on Artificial
Intelligence, volume 8, pages 1433–1438. Chicago, IL,USA.

Yu, T., Kumar, A., Chebotar, Y., Hausman, K., Finn, C., andLevine, S. (2022). How to leverage unlabeled data in offline reinforcement learning. In International Conference
on Machine Learning, pages 25611–25635. PMLR.
11

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

Zolna, K., Novikov, A., Konyushkova, K., Gulcehre, C.,
Wang, Z., Aytar, Y., Denil, M., de Freitas, N., and Reed,
S. (2020). Offline learning from demonstrations andunlabeled experience. arXiv preprint arXiv:2011.13885.

12

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

A. Related Work
Offline RL Offline RL studies the problem of return maximization given an offline dataset. Offline RL algorithms
can be broadly classified into model-based approaches, e.g., (Yu et al., 2020b; Kidambi et al., 2020; Uehara and Sun,
2022; Rigter et al., 2022; Xie et al., 2022), and model-free approaches, e.g., (Jin et al., 2021; Fujimoto and Gu, 2021; Xieet al., 2021; Cheng et al., 2022; Kumar et al., 2020; Kostrikov et al., 2021). Since the agent needs to learn without online
data collection, offline RL becomes more challenging when the offline dataset does not provide good coverage over thestate-action distribution of all feasible policies. There have been two common strategies to handle insufficient coverage,
behavior regularization approaches, e.g., (Fujimoto and Gu, 2021; Wu et al., 2019; Kostrikov et al., 2021), which restricts
the learned policy to be close to the behavior policy, and value regularization approaches, e.g., (Jin et al., 2021; Liu et al.,2020; Xie et al., 2021; Cheng et al., 2022; Kumar et al., 2020; Xie et al., 2022) which provide a pessimistic value estimates
for the learned policy. Both approaches can be viewed as optimizing for a performance lower bound of the agent’s return.
Offline RL commonly assumes compatibility between offline dataset and MDP, i.e., the offline dataset should be generated
from the task MDP of interest. Our work is distinctive in the offline RL literature as we study the behavior of existing offline
RL algorithms when data reward differs from the true MDP reward. Existing tools for analyzing offline RL are not directly
applicable to our setting: policies with good performance under data reward, as guaranteed by such tools, do not necessarily
attain high return under the true reward. Our novel analysis is made possible by studying the properties of offline data
distribution, an aspect mostly neglected by existing work in the literature.
Offline RL with imperfect reward In the offline RL literature, there is a line of work studying scenarios where the reward
is missing in a subset of the dataset (Singh et al., 2020; Hu et al., 2023; Li et al., 2023a; Yu et al., 2022) or when the data
reward is misspecified (Li et al., 2023b). These approaches propose to construct a new reward function and apply offline
RL algorithms on the relabeled dataset. For example, (Singh et al., 2020; Yu et al., 2022) label the missing reward with aconstant minimum reward. (Li et al., 2023a; Hu et al., 2023) learn a pessimistic reward function. (Li et al., 2023b) optimize
for a reward function which minimizes the reward difference among expert data.
Different from this line of work, we are interested in running offline RL algorithms using the misspecified reward as it is.
We show both theoretically and empirically that offline RL algorithms can produce well-performing policies with rewards
different from the true reward as long as the underlying data distribution satisfies certain conditions. It can be an interesting
future work to combine these techniques with our analysis to explore the robustness of offline RL algorithms under the classof learned reward functions.
Offline IL Offline imitation learning (IL) can be considered as a special case of offline RL when reward is missing fromthe entire dataset, or when the agent has the largest uncertainty about the reward (Cheng et al., 2022). In offline IL, thelearner is instead given the information on whether a transition is generated by an expert. The goal of offline IL is to produce
a policy that has comparable performance with the expert. (Zolna et al., 2020) learns a discriminator reward to classify expert
and non-expert data, and apply offline RL algorithms with the learned reward. (Xu et al., 2022a) uses the discriminator as
the weight for weighted behavior cloning. (Kim et al., 2021; Ma et al., 2022b; Zhu et al., 2020) proposes DICE (Dai et al.,2020)-style algorithms to minimize the divergence between the state-action or state-(next state) occupancy measure of thelearner and the expert. (Chang et al., 2021; Kidambi et al., 2021) minimize state-action or state-(next state) distribution
divergence under a pessimistic model. (Yue et al., 2023) extends maximum entropy inverse RL (Ziebart et al., 2008) to an
offline setting. (Smith et al., 2023) proposes to run offline RL under an indicator reward function on whether the data is
expert data. (Li et al., 2023a) learns a pessimistic reward function where the expert data labeled as the maximum reward.
Our experiments with zero reward function is similar to an offline IL setting with the behavior policy being the expert.
However, we show that offline RL algorithms can sometimes achieve significant performance gain over behavior andbehavior cloning policies, which is a phenomenon that cannot be explained by existing analysis in offline IL. We provide
an explanation to this puzzle: in Appendix C.4, we show that the data distribution is ∞-positively biased (according ourdefinition) for offline RL, when the data is generated by the optimal policy of the true MDP. This means that an admissible
offline RL algorithm can achieve near-optimal performance with any reward function from its corresponding admissible
reward class in this case. We empirically show that in Appendix B.6.
Robust offline RL Robust offline RL (Zhang et al., 2022; Chen et al., 2023; Wu et al., 2023; Panaganti et al., 2022; Yanget al., 2022; Zhou et al., 2021; Ma et al., 2022a; Shi and Chi, 2022) aims to develop new offline RL algorithms for cases
when the compatibility assumption does not hold, i.e., the offline dataset may not be generated from the task MDP. Among13

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

this line of work, (Panaganti et al., 2022; Zhou et al., 2021; Ma et al., 2022a; Shi and Chi, 2022) study the robust MDP
setting, where the dataset can be generated from any MDP within a δ-ball of a nominal MDP. An adversary can choose any
MDP from the δ-ball when evaluating the learned policy. (Zhang et al., 2022) considers when the adversary can modify ϵfraction of transition tuples in the offline dataset, while (Wu et al., 2023) considers when up to K trajectories can be addedin or removed from the original dataset. (Chen et al., 2023) uses the formulation of Byzantine-robust: when multiple offline
datasets are presented to the learner, among them an α fraction are corrupted. (Yang et al., 2022) assumes the dataset isgenerated from the true MDP, but the state presented to the policy during test time can be a corrupted up to ϵ distance.
Our work is fundamentally different from this line of work in that we study the inherent and somewhat unintendedrobustness of offline RL algorithms. We show that, despite originally designed under the compatibility assumption, offline
RL algorithms show strong robustness against perturbation of reward. Another salient difference is that robust offline RL
typically requires an upper bound on the size of perturbation to the true or nominal MDP, and most of proposed algorithms
assume knowledge of this upper bound. However, our work does not make assumptions on the size of perturbation and,moreover, the algorithms we study are not even aware of the existence of such perturbation. Indeed, we empirically showthat offline RL algorithms, without any modifications, can succeed even when data reward is a constant or the negationof the true reward. Additionally, in this work, we consider the reward perturbation while most work in robust offline RL
focuses more on dynamics perturbation.
Constrained and safe offline RL Our work is related to the literature of constrained and safe offline RL (Le et al., 2019;
Xu et al., 2022b; Liu et al., 2023; Lee et al., 2022a; Polosky et al., 2022). In constrained offline RL, the agent solves
for a CMDP given an offline dataset {(s, a, r, c, s′ )}. The goal of constrained offline RL is to produce a policy which
maximizes the expected return while ensuring that constraint violation is below a given upper bound. (Le et al., 2019) uses
an offline policy evaluation oracle to solve for the Lagrangian relaxation of the CMDP. (Xu et al., 2022b) constructs apessimistic estimation of value and constraint violation. (Lee et al., 2022a; Polosky et al., 2022) proposes DICE (Dai et al.,2020)-style algorithms to solve for the optimal state-action occupancy measure. (Liu et al., 2023) proposes a variant ofdecision transformer (Chen et al., 2021) capable of producing policies for different constraint violation upper bounds during
test time.
Our work, instead, considers a different scenario where the constraint is only given implicitly — through the support ofthe offline dataset. We assume that any transition in the dataset satisfies the constraint, which can be achieved throughintervention during data collection. We show that offline RL can inherently produce approximately safe policies due to its
survival instinct. Although our assumption seems more restrictive as in that we require a constraint violation-free dataset,
we argue that our assumption is more practical in many safety critical scenarios: compared to executing unsafe actions,
it is perhaps more reasonable to intervene before constraint violation happens. We believe that our findings open up new
possibilities for applying offline RL as a safe RL oracle, which can be especially promising in an offline-online RL setup (Xuet al., 2023; Lee et al., 2022b).

B. Experiment Details
In the experiments, we train a total of around 16k offline RL agents (see Appendix B.8 for details). In this appendix, we
provide details of experiment setup, implementation of algorithms, hyperparameter tuning, as well as additional experimental
results and ablation studies. At the end of the appendix, we include Table 13 and Table 14 which contain the actual numbers
used in generating Figure 3 and Figure 5.
B.1. Benchmark Datasets
In the experiments, we consider three tasks (15 datasets5 ) from D4RL (Fu et al., 2020) and 15 datasets collected from theMeta-World (Yu et al., 2020a) benchmark. The tasks are visualized in Figure 6. Below we provide more details on thesetasks.
D4RL We consider three locomotion tasks from D4RL (Fu et al., 2020), hopper, walker2d and halfcheetah6(see Figure 6). In the main paper, for each task, we use four datasets of different behavior policy quality: random, medium,medium-replay, and medium expert. We use the expert datasets in Appendix B.6 to study the robustness of5
6

We use 12 datasets (without expert datasets) in the main experiments.
We use the v2 version of all three tasks.

14

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data(a) D4RL (Fu et al., 2020)(b) Meta-World (Yu et al., 2020a)

Figure 6: We study offline RL with wrong rewards for a variety of tasks: (a) three locomotion tasks from D4RL (Fuet al., 2020); hopper (top left), walker2d (top right) and halfcheetah (bottom). the figures are from (Yu et al., 2019) (b) 15
goal-oriented tasks from Meta-World (Yu et al., 2020a). The 15 tasks are composed of the 10 tasks from MT-10 (top 2 rows)
and the 5 testing tasks from ML-45 (bottom row).
offline RL on data generated by near-optimal policies.
hopper: The goal of hopper is to move forward as fast as possible without falling down. An episode terminates wheneither of the three happens: 1) the height of the hopper is below a threshold, 2) the body of the hopper is too tilted,or 3) the velocity of any joint is too large. The original true reward function (which has been used to train online RL
agents) is1
rhopper := 1[hopper is alive] +∆x − 0.001∥τ ∥2 ,∆twhere ∆x is the displacement in the forward direction, and τ is the applied torque. The true reward function is designed
such that the hopper can receive bonus when staying alive and moving forward. We show that, for a few hopper
datasets, offline RL agent can succeed without such a carefully designed reward, and even with the negation of the true
reward.
walker2d: The walker2d task is similar to hopper, except that the walker2d agent has a higher degree of freedom.
We observe similarly that offline RL can learn good behaviors without a reward that is specifically designed to encouragestaying alive and moving forward.
halfcheetah: The halfcheetah task differs from the two tasks above since episodes here always have the same lengthand never terminates early. The true reward of halfcheetah isrhalfcheetah :=1∆x − 0.1∥τ ∥2 .
∆t

Since the halfcheetah datasets (except for the expert dataset, see Appendix B.6) do not have a positive bias, wefind that offline RL agent is sensitive to reward quality.
Meta-World We consider 15 goal-oriented manipulation tasks from (Yu et al., 2020a), see Figure 6. We use the 10 tasks
from the MT-10 set and the 5 testing tasks from the ML-45 set. Each Meta-World task comes with a hand-crafted reward
function that encourage progress toward task completion, e.g., when the end-effector grasps the target object, when theobject is placed at the goal, etc. We refer readers to (Yu et al., 2020a) for the specific reward functions used for each tasks.
We make a few modifications to the Meta-World environment: 1) we shift and scale the true reward such that the agent
receives a reward of range [−1, 0] rather than [0, 10] of the original environment, 2) we terminate an episode and mark thelast state with a terminal flag when the goal is achieved. We show that, data collected by goal-oriented tasks can have apositive bias, and an offline RL agent can make use of such bias to learn the right behavior even with wrong rewards. We
would like to clarify that the data generated by the original Meta-World environment (with no terminal signals) does nothave a positive bias in general, since all trajectories, success or failure, would have a same length of 500.
15

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

B.2. Implementation Details
We use the implementation of ATAC and PSPI from https://github.com/chinganc/lightATAC. We make a small modification
to the ATAC and PSPI policy class. Instead of a tanh-Gaussian policy, we use a scaled tanh-Gaussian policy, i.e.,c · tanh(N (µ, Σ)) with c ≥ 1. We find scaled tanh-Gaussian policies to be more numerically stable when the data has a lotof on-boundary actions, i.e., actions that takes the value of either −1 or 1 on any dimension. For PSPI, we let the critic f tominimize for f (s, π) for all states from the dataset (rather than all initial states). We find it to provide better results when theeffective horizon given by the discount factor is smaller than the actual episode length. We use the implementation of IQL
and BC from https://github.com/gwthomas/IQL-PyTorch/, and the implementation of CQL from https://github.com/younggeng/CQL. For decision transformer, we use the implementation from https://github.com/kzl/decision-transformer. We feed
the decision transformer with wrong rewards (consistent with data reward) during testing.
B.3. Hyperparameters
We use the default values (ones provided by the original paper or by the implementation we use) for most of the hyperparameters for all algorithms. For each algorithm, we tune one or two hyperparameters (with 4 combinations at most) which
affect the degree of pessimism7 . The values of hyperparamters of all algorithms are given in Table 1–4, where the choices ofhyperparameters used in tuning are highlighted in blue.
We tune hyperparameter per each dataset and per reward label. The tuned values are provided in Table 9–12. During tuning,
we evaluate each combination of hyperparameter(s) for 2 random seeds for D4RL and 3 random seeds for Meta-World. We
choose the best-performing values and report the results of these hyperparameters over 10 new random seeds (not including
the tuning seeds). We find the D4RL scores for the true reward to be close to what is reported in the original papers.
Hyperparameter

Value

Bellman consistency coefficient β
Number of hidden layers
Number of hidden units
Nonlinearity
Policy distribution
Action scale c{0.1, 1, 10, 100}
3
256
ReLU
scaled tanh-Gaussian
1.0 for D4RL
1.2 for Meta-World
5 × 10−4
5 × 10−7
256
106
105
0.5
0.005
0.99 for D4RL
0.998 for Meta-World2min(−1, r̃min )1−γ
2
max(1, r̃max )1−γ

Fast learning rate ηfast
Slow learning rate ηslow
Minibatch size |Dmini |Gradient steps
Warmstart (BC) steps
Temporal difference loss weight w
Target smoothing coefficient τ
Discount factor γ
Minimum target value Vmin
Maximum target value Vmax

Table 1: Hyperparameters for ATAC and PSPI
B.4. Episode Lengths for hopper and walker2d Tasks
We show the episode lengths for hopper and walker2d tasks in Figure 7. We observe that policies learned by ATAC andPSPI can consistently keep the agent from falling down for a significant longer period than the behavior policies. The only
exception is when running PSPI on hopper-medium-expert dataset with random reward. This is potentially due tothe fact that we use only 2 seeds for hyperparameter tuning. The average normalized scores and episode lengths during
tuning are 101.1 and 925.0, respectively, higher than the final values of 58.8 and 546.0. We observe that DT policies also
tend to keep the agent alive for a longer number of steps than behavior policies.
For IQL and CQL, it is relatively rare for them to achieve long episode lengths when trained with wrong rewards.
7

For decision transformer, we do not tune hyperparameters since it has no notion of pessimism.

16

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data
Hyperparameter

Value

Inverse temperature β
Expectile τ
Number of hidden layers
Number of hidden units
Nonlinearity
Policy distribution
Learning rate η
Minibatch size |Dmini |Gradient steps
Target smoothing coefficient α
Discount factor γ{0.3, 3}{0.7, 0.9}
3
256
ReLU
Gaussian
3 × 10−4
256
106
0.005
0.99 for D4RL
0.998 for Meta-World

Table 2: Hyperparameters for IQL
Hyperparameter

Value

Critic pessimism coefficient α
Number of hidden layers
Number of hidden units
Nonlinearity
Policy distribution
Learning rate η
Minibatch size |Dmini |Gradient steps
Target smoothing coefficient τ
Discount factor γ{0.1, 1, 10, 100}
2∗
256
ReLU
tanh-Gaussian∗
3 × 10−4
256
106
0.005
0.99 for D4RL
0.998 for Meta-World

Table 3: Hyperparameters for CQL. ∗ We include an ablation study of CQL with 3 hidden layers and scaled tanh-Gaussianpolicy, i.e., the same architecture as ATAC and PSPI, in Appendix B.7.

IQL and CQL, when using negative reward, can keep the agent from falling for longer in hopper-medium andhopper-medium-expert datasets. We would like to note that our hyperparameters are tuned based on normalized
scores rather than episode lengths. The statistics of episode lengths are included in Table 5.
B.5. Effect of Removing Terminal Transitions
For hopper and walker2d tasks, we remove the terminal transitions, i.e., the transitions where the hopper or walker
falls, from the datasets with modified rewards. This is to satisfy the safety condition in Section 3.4. A terminal signal
implicitly adds an absorbing8 , but in this case, unsafe, state into the data distribution µ. This means that staying within
data support in a long term would not provide safety, as the agent can stay within support by falling. For D4RL datasets inparticular, we need to remove terminal transitions (rather than just the terminal flag) since terminal transitions in the original
datasets do not contain valid next states.9
This, however, means that there are two underlying variables: 1) the data reward, and 2) whether terminal transitions are
removed, when comparing offline RL with wrong rewards and offline RL on the original dataset. To separate the effect of
two variables, we study the performance of offline RL algorithms with the original true reward but the terminal transitions
removed. The results10 are listed in the No-terminal column in Table 13. We observe that ATAC and PSPI with the originaltrue reward show similar results whether the terminal transitions are removed. IQL and CQL with true reward, interestingly,
does significantly worse when the terminal transitions are removed. This is potentially due to the increased instability of
8This is also why we mark goal completion as terminal in Meta-World experiments. For goal-oriented tasks, terminal signal upon taskcompletion implicitly creates an absorbing goal state in the data distribution.
9
The authors of D4RL datasets set the next state to be a place-holder value when a transition is marked as terminal.
10
We do not study decision transformer (DT) in this ablation since DT does not use terminal signals.

17

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data
Hyperparameter

Value

Number of hidden layers
Number of hidden units
Nonlinearity
Context length K
Dropout
Warmup steps
Learning rate η
Minibatch size |Dmini |Gradient steps

3
128
ReLU
20
0.1
2 × 102
5 × 10−4
256
7 × 104 for D4RL
5 × 103 for Meta-World
0.25
10−4
Return randomly sampled from the top 10%of trajectory returns in the data, w.r.t. the data reward

Gradient norm clip
Weight decay
Return-to-goal conditioning

Table 4: Hyperparameters for the decision transformer (DT)target network (Van Hasselt et al., 2018; Sutton and Barto, 2018) when there are no terminal signals.
B.6. Offline RL with Expert Datasets
In Section 3, we point out that data has a positive bias when it is generated by the expert policy, which is different fromthe length bias. We now experimentally11 validate this claim. We consider the expert datasets from the D4RL locomotion
tasks. As is visualized in Figure 8, walker2d-expert and halfcheetah-expert datasets do not have a length bias:
almost all trajectories are of lengh 1000. We observe that all offline algorithms except decision transformer, when trainedwith wrong rewards, can achieve expert-level performance in these two datasets (see Figure 9). ATAC, PSPI and DT also
perform well on the hopper-expert dataset while IQL and CQL receive lower scores when using wrong rewards. The
actual normalized scores are listed in Table 6.
B.7. Ablation Study on CQL Architecture
Our results are generated by CQL with the architecture in Table 3, which is given by the original implementation. In thisstudy, we examine the performance of CQL when using the same architecture as ATAC and PSPI, i.e., with 3 hidden layers
and scaled tanh-Gaussian. The normalized scores for D4RL datasets and success rate for Meta-World tasks are includedin Table 7 and Table 8, respectively. We find the performance of CQL with 3 hidden layers and scaled tanh-Gaussian policy,
denoted as CQL3 , to be similar with the performance of CQL with the original architecture.
B.8. Compute UsageFor ATAC, PSPI, IQL, CQL and BC, each run takes around 5.5 hours on an NC4as T4 v3 Azure virtual machine for both
D4RL and Meta-World experiments. For D4RL experiments (including ablation studies), we use a total of 5 (algorithms) ×
15 (D4RL datasets) × 5 (reward types) × 4 (hyperparameters) × 2 (seeds) = 3000 runs for hyperparameter tuning and
6 (algorithms) × 15 (D4RL datasets) × 5 (reward types) × 10 (seeds) = 4500 runs for generating the final results. For
Meta-World experiments, we use 5 (algorithms) × 15 (Meta-World datasets) × 4 (reward types) × 4 (hyperparameters) ×
3 (seeds) = 3600 runs for hyperparameter tuning and 6 (algorithms) × 15 (Meta-World datasets) × 4 (reward types) ×
10 (seeds) = 3600 runs for generating the final results. This amount to 14700 runs or 66165 hours of training on
NC4as T4 v3 virtual machines.
For DT, we do not tune any parameters at large scale. Each run takes around 2.5 hours on an NC6s v2 or ND6s Azure virtual
machine. We train a total of 30 (D4RL and Meta-World datasets) × 4 (reward types) × 10 (seeds) = 1200 DT agents.

11

We will formally prove it in Appendix C.

18

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

DatasetBehaviorBC

hopperrandom

22.1

93.2±5.0

hoppermedium457.2

540.5±1.2hoppermediumreplay197.0345.4±7.0hoppermediumexpert622.2

541.8±1.1

walker2d
random20.4

81.2±2.1

walker2d
medium839.6

895.3±2.4

walker2d
mediumreplay276.3

312.7±5.8

walker2d
mediumexpert912.8

976.9±3.5

Algo
ATAC
PSPI
IQL
CQL
DT
ATAC
PSPI
IQL
CQL
DT
ATAC
PSPI
IQL
CQL
DT
ATAC
PSPI
IQL
CQL
DT

Original79.3±13.0
99.2±6.2
108.7±4.0
48.4±12.5
245.2±8.2
808.4±15.3
906.8±16.9
571.2±8.5851.1±35.9
630.4±6.7
1000.0±0.0
1000.0±0.0
598.5±19.8
801.9±38.2
934.5±13.6
930.1±13.1
974.5±6.1
980.4±5.1
800.8±31.4
817.4±22.5

Zero714.2±120.0
354.5±113.0
137.1±18.0
41.8±20.9
239.7±8.6
999.7±0.3
897.7±16.2
538.7±4.0594.7±123.9
641.1±8.2
904.5±44.3
1000.0±0.0
222.5±33.2
24.9±2.6
742.3±33.4
965.3±10.5
848.0±44.6535.4±3.4516.6±72.7
813.8±22.0

Random552.7±123.4
513.9±131.0
12.8±0.1
28.4±3.8
103.2±2.0998.9±1.0
993.0±6.4594.8±20.3
447.2±97.4
638.8±6.1
909.4±35.3
995.9±3.9
22.3±4.2
217.5±123.7
836.2±20.6
836.5±71.7
546.0±84.4
581.2±74.7
633.1±87.9
817.8±17.0

Negative
958.8±39.1
749.1±124.5
12.1±2.7
6.0±0.0295.1±32.8
997.6±1.9907.8±6.0
631.4±21.7
962.2±17.0
633.4±9.1
909.6±29.5
1000.0±0.0
88.1±65.5
153.8±91.8
825.6±31.2
998.3±1.6988.9±4.2
883.2±28.5
901.9±55.7
852.1±17.4

No-terminal
567.2±144.9
517.3±142.9
14.5±1.0
16.7±10.1–
840.2±19.8
833.9±29.7
446.4±65.8
503.7±123.0–
1000.0±0.0
1000.0±0.0
100.9±53.6
116.4±93.2–
885.1±36.3
969.8±5.3
735.3±62.8489.1±103.0
–

ATAC
PSPI
IQL
CQL
DT
ATAC
PSPI
IQL
CQL
DT
ATAC
PSPI
IQL
CQL
DT
ATAC
PSPI
IQL
CQL
DT

227.6±10.2
221.1±13.7240.5±3.8
182.9±79.5
228.4±7.6976.9±17.6
989.1±7.2
930.1±12.3
991.0±3.0
968.7±3.3
923.1±16.4
914.4±30.4659.9±37.5
945.1±14.9776.1±21.3
929.8±44.4
1000.0±0.0
999.6±0.4
998.4±1.5
1000.0±0.0

231.0±11.0
143.8±10.5
83.7±4.956.0±23.1
210.9±10.2
999.7±0.3
998.2±1.7
831.1±37.3
105.3±34.7
955.6±7.5967.8±21.5
997.1±2.7313.5±27.5
71.3±27.5
755.5±10.7
979.4±15.9
918.9±21.3
989.0±2.955.8±20.4
999.1±0.9

149.6±10.4165.7±9.8
79.9±21.4
188.7±50.9
75.2±1.2
997.8±1.5
1000.0±0.0
976.5±7.4
86.9±32.7
951.5±4.7
968.4±19.8
995.3±3.2
211.1±37.1
35.1±6.9
841.9±14.8
999.4±0.6
1000.0±0.0
808.8±60.9
44.3±17.4
1000.0±0.0

214.8±9.2135.8±1.2
182.3±1.6
34.7±5.1
203.3±11.1
999.8±0.2
973.0±4.9739.1±30.7321.5±107.8
954.9±6.9817.4±83.6
931.8±53.2
119.5±25.1
72.8±23.3
740.8±17.8
1000.0±0.0
960.2±8.3856.7±21.7
514.5±112.3
999.5±0.4

189.0±19.8
135.9±0.5
62.8±23.0
37.2±10.2–
990.2±5.9991.7±4.7517.4±11.2
130.4±35.0–
985.8±10.0
997.3±1.2
41.8±3.487.5±26.7–
960.3±36.0
987.9±9.8334.1±12.5
12.6±2.2–

Table 5: Episode lengths for hopper and walker2d datasets. The mean and standard error of episode lengths are computed
across 10 random seeds. For each random seed, we evaluate the final policy of each algorithm over 50 episodes.

DatasetBehaviorBC

hopperexpert108.5

110.6±0.0

walker2d
experthalfcheetah
expert

107.1

88.1

Algo
ATAC
PSPI
IQL
CQL
DT

Original
111.1±0.1
110.9±0.2
110.3±0.2
101.2±3.0
107.2±0.7

Zero110.9±0.1
111.1±0.1
66.7±12.8
0.7±0.0
107.7±0.6

Random110.7±0.2
111.1±0.1
73.5±11.2
0.7±0.0
108.5±0.3

Negative110.9±0.1
111.0±0.2
48.5±11.8
66.0±12.5
109.0±0.3

No-terminal
111.4±0.0110.9±0.3
24.3±8.9
0.6±0.0–

108.3±0.0

ATAC
PSPI
IQL
CQL
DT

110.2±0.1
109.3±0.0110.9±0.1
108.3±0.2
109.0±0.1

108.0±0.1
107.9±0.1
108.4±0.0108.2±0.2
108.9±0.1

107.1±0.5102.1±2.8103.0±3.5108.0±0.1
109.0±0.1

107.8±0.0
107.8±0.2
108.3±0.1
108.1±0.2
109.0±0.1

110.3±0.0109.5±0.0
111.0±0.1
108.3±0.3–

92.9±0.0

ATAC
PSPI
IQL
CQL
DT

94.0±0.4
94.6±0.1
96.8±0.1
92.3±0.992.6±0.1

92.4±0.492.9±0.3
92.9±0.1
93.9±0.492.8±0.1

92.4±1.2
88.0±1.3
92.9±0.093.7±0.692.6±0.1

93.1±0.1
93.5±0.3
89.3±0.1
94.0±0.292.8±0.1

–
–
–
–
–

Table 6: D4RL expert dataset normalized scores. The mean and standard error of normalized scores are computed across 10
random seeds. For each random seed, we evaluate the final policy of each algorithm over 50 episodes.

19Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

Figure 7: Episode lengths for hopper and walker2d datasets from D4RL. The mean and standard error of episodelengths are computed across 10 random seeds. For each random seed, we evaluate the final policy of each algorithm over 50
episodes. Note that the hyperparameters are tuned based on normalized scores rather than episode lengths.

Figure 8: A visualization of hopper-expert, walker2d-expert and halfcheetah-expert datasets from D4RL. Each plotcorresponds to a dataset for a task (row) with a dataset (column). Each trajectory in a dataset is represented as a data pointwith the x-coordinate being its episode length and y-coordinate being its normalized score.
20

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

Figure 9: Normalized scores for expert datasets of hopper, walker2d and halfcheetah from D4RL (Fu et al., 2020). The
mean and standard error of normalized scores are computed across 10 random seeds. For each random seed, we evaluate thefinal policy of each algorithm over 50 episodes.

Datasethopper-random
hopper-medium
hopper-medium-replay
hopper-medium-expert

walker2d-random
walker2d-medium
walker2d-medium-replay
walker2d-medium-expert

halfcheetah-random
halfcheetah-medium
halfcheetah-medium-replay
halfcheetah-medium-expert

Algo
CQL
CQL3
CQL
CQL3
CQL
CQL3
CQL
CQL3

Original
3.0±0.7
11.2±2.285.0±3.7
48.0±12.2
79.7±3.7
100.3±1.6
88.8±3.6
104.7±1.4

Zero
2.0±1.1
2.0±0.455.0±11.7
61.7±12.2
1.9±0.1
1.8±0.049.7±8.1
40.6±10.7

Random
1.8±0.2
7.7±3.536.9±9.817.3±9.8
7.6±3.7
1.8±0.060.7±9.4
17.3±9.0

Negative
0.7±0.0
3.9±2.989.0±1.575.8±6.9
5.8±2.7
1.6±0.2
88.6±6.1
48.2±8.7

No-terminal
1.2±0.5
10.1±3.2
46.3±11.5
32.8±10.3
4.4±2.8
1.8±0.047.4±11.0
30.0±9.5

CQL
CQL3
CQL
CQL3
CQL
CQL3
CQL
CQL3

4.0±1.8
4.8±0.281.4±0.3
81.0±0.474.1±1.276.8±1.3
109.3±0.3
108.0±0.6

0.9±0.9
1.4±0.8
2.0±0.9
5.0±4.7
0.5±0.3-0.2±0.1
0.8±0.5-0.2±0.1

4.3±1.4
3.3±1.2
1.2±0.7-0.2±0.0-0.2±0.1
0.1±0.1
0.1±0.2-0.3±0.1-0.4±0.0
0.9±0.6
14.5±7.4
13.3±7.7
0.2±0.3
0.1±0.3
28.0±9.1
26.8±11.0

0.2±0.4-0.2±0.1
2.1±0.8
0.1±0.3
0.4±0.4
0.3±0.3-0.2±0.0-0.2±0.0

CQL
CQL3
CQL
CQL3
CQL
CQL3
CQL
CQL3

30.7±0.631.3±0.657.2±1.565.1±0.648.2±5.447.8±5.275.2±1.0
88.0±1.7

1.0±0.8
0.5±0.443.2±0.1
43.1±0.1
34.2±0.738.3±1.0
81.2±2.175.5±1.7-1.4±0.3
1.9±0.143.5±0.1
43.3±0.1
33.1±0.937.7±1.1
69.6±2.880.0±2.5-5.4±0.3-7.0±0.442.7±0.142.2±0.1
32.4±1.0
32.6±1.242.4±0.242.6±0.1

–
–
–
–
–
–
–
–

Table 7: D4RL normalized scores for CQL with the original architecture in Table 3 (CQL) and with the same architecture as
ATAC and PSPI (CQL3 ), i.e., with 3 hidden layers and scaled tanh-Gaussian. We find that CQL performs similarly under
the two choices of architecture.

21

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data
Datasetbutton-press
door-open
drawer-close
drawer-open
peg-insert-side
pick-place
push
reach
window-close
window-open
bin-picking
box-close
door-lockdoor-unlock
hand-insert

Algo
CQL
CQL3
CQL
CQL3
CQL
CQL3
CQL
CQL3
CQL
CQL3
CQL
CQL3
CQL
CQL3
CQL
CQL3
CQL
CQL3
CQL
CQL3
CQL
CQL3
CQL
CQL3
CQL
CQL3
CQL
CQL3
CQL
CQL3

Original
0.88±0.07
0.78±0.10
0.82±0.09
0.47±0.15
0.91±0.04
0.99±0.01
0.78±0.04
0.86±0.03
0.08±0.02
0.10±0.02
0.01±0.00
0.00±0.00
0.00±0.00
0.03±0.01
0.17±0.03
0.35±0.05
1.00±0.00
0.97±0.03
0.83±0.03
0.84±0.07
0.00±0.00
0.00±0.00
0.04±0.01
0.03±0.02
0.75±0.04
0.63±0.06
0.87±0.03
0.92±0.02
0.13±0.04
0.04±0.03

Zero
0.83±0.10
0.90±0.06
0.07±0.07
0.29±0.14
0.99±0.01
0.93±0.03
0.66±0.07
0.79±0.09
0.00±0.00
0.05±0.03
0.06±0.02
0.00±0.00
0.01±0.00
0.00±0.00
0.29±0.04
0.35±0.03
1.00±0.00
0.75±0.12
0.82±0.05
0.94±0.02
0.02±0.02
0.00±0.00
0.07±0.02
0.00±0.00
0.61±0.10
0.64±0.09
0.77±0.09
0.73±0.11
0.22±0.06
0.08±0.03

Random
0.84±0.09
0.73±0.11
0.12±0.08
0.00±0.00
0.96±0.02
0.97±0.02
0.65±0.07
0.78±0.06
0.02±0.01
0.08±0.02
0.05±0.02
0.00±0.00
0.01±0.00
0.00±0.00
0.25±0.05
0.37±0.05
0.59±0.13
0.77±0.12
0.92±0.02
0.93±0.02
0.00±0.00
0.00±0.00
0.01±0.00
0.01±0.00
0.61±0.11
0.57±0.10
0.89±0.04
0.71±0.11
0.34±0.04
0.18±0.03

Negative
0.94±0.03
0.68±0.11
0.00±0.00
0.38±0.15
0.97±0.02
0.98±0.01
0.84±0.03
0.80±0.05
0.00±0.00
0.03±0.02
0.07±0.03
0.00±0.00
0.03±0.02
0.01±0.01
0.23±0.03
0.22±0.06
0.55±0.13
0.98±0.02
0.83±0.07
0.77±0.09
0.00±0.00
0.00±0.00
0.05±0.01
0.01±0.00
0.60±0.08
0.62±0.09
0.93±0.03
0.76±0.11
0.32±0.03
0.08±0.02

Table 8: Meta-World success rate for CQL with the original architecture in Table 3 (CQL) and with the same architecture as
ATAC and PSPI (CQL3 ), i.e., with 3 hidden layers and scaled tanh-Gaussian. We find that CQL performs similarly under
the two choices of architecture.
Datasethopper-random
hopper-medium
hopper-medium-replay
hopper-medium-expert
hopper-expert

Original
β = 100.0
β = 10.0
β = 100.0
β = 1.0
β = 0.1

Zero
β = 100.0
β = 100.0
β = 1.0
β = 10.0
β = 1.0

Randomβ = 10.0
β = 100.0
β = 1.0
β = 10.0
β = 1.0

Negative
β = 1.0
β = 1.0
β = 0.1
β = 1.0
β = 0.1

No-terminal
β = 10.0
β = 100.0
β = 100.0
β = 1.0
β = 0.1

walker2d-random
walker2d-medium
walker2d-medium-replay
walker2d-medium-expert
walker2d-expert

β = 10.0
β = 10.0
β = 100.0
β = 10.0
β = 1.0

β = 0.1
β = 1.0
β = 10.0
β = 10.0
β = 100.0

β = 1.0
β = 1.0
β = 10.0
β = 1.0
β = 0.1

β = 0.1
β = 0.1
β = 10.0
β = 0.1
β = 0.1

β = 100.0
β = 100.0
β = 100.0
β = 10.0
β = 1.0

halfcheetah-random
halfcheetah-medium
halfcheetah-medium-replay
halfcheetah-medium-expert
halfcheetah-expert

β = 0.1
β = 1.0
β = 1.0
β = 0.1
β = 0.1

β = 0.1
β = 100.0
β = 1.0
β = 0.1
β = 100.0

β = 0.1
β = 1.0
β = 1.0
β = 0.1
β = 0.1

β = 0.1
β = 0.1
β = 0.1
β = 0.1
β = 0.1–––––

button-press
door-open
drawer-close
drawer-open
peg-insert-side
pick-place
push
reach
window-close
window-open
bin-picking
box-close
door-lockdoor-unlock
hand-insert

β = 10.0
β = 100.0
β = 1.0
β = 10.0
β = 1.0
β = 1.0
β = 1.0
β = 10.0
β = 1.0
β = 0.1
β = 1.0
β = 1.0
β = 10.0
β = 100.0
β = 10.0

β = 10.0
β = 10.0
β = 10.0
β = 1.0
β = 1.0
β = 10.0
β = 1.0
β = 100.0
β = 1.0
β = 1.0
β = 1.0
β = 1.0
β = 1.0
β = 10.0
β = 1.0

β = 1.0
β = 10.0
β = 10.0
β = 10.0
β = 1.0
β = 1.0
β = 1.0
β = 10.0
β = 1.0
β = 10.0
β = 1.0
β = 1.0
β = 1.0
β = 10.0
β = 1.0

β = 100.0
β = 1.0
β = 100.0
β = 10.0
β = 1.0
β = 1.0
β = 1.0
β = 1.0
β = 1.0
β = 100.0
β = 1.0
β = 1.0
β = 0.1
β = 10.0
β = 1.0

–
–
–
–
–
–
–
–
–
–
–
–
–
–
–

Table 9: Tuned hyperparameter values for ATAC

22

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data
Datasethopper-random
hopper-medium
hopper-medium-replay
hopper-medium-expert
hopper-expert

Original
β = 10.0
β = 100.0
β = 10.0
β = 1.0
β = 1.0

Zero
β = 10.0
β = 10.0
β = 10.0
β = 10.0
β = 1.0

Randomβ = 100.0
β = 100.0
β = 10.0
β = 100.0
β = 1.0

Negative
β = 100.0
β = 1.0
β = 1.0
β = 1.0
β = 0.1

No-terminal
β = 100.0
β = 100.0
β = 1.0
β = 1.0
β = 1.0

walker2d-random
walker2d-medium
walker2d-medium-replay
walker2d-medium-expert
walker2d-expert

β = 10.0
β = 10.0
β = 10.0
β = 1.0
β = 1.0

β = 10.0
β = 10.0
β = 10.0
β = 1.0
β = 1.0

β = 10.0
β = 10.0
β = 10.0
β = 10.0
β = 10.0

β = 1.0
β = 0.1
β = 1.0
β = 0.1
β = 0.1

β = 1.0
β = 10.0
β = 100.0
β = 10.0
β = 1.0

halfcheetah-random
halfcheetah-medium
halfcheetah-medium-replay
halfcheetah-medium-expert
halfcheetah-expert

β = 100.0
β = 1.0
β = 10.0
β = 0.1
β = 0.1

β = 100.0
β = 10.0
β = 1.0
β = 1.0
β = 1.0

β = 0.1
β = 1.0
β = 10.0
β = 0.1
β = 10.0

β = 1.0
β = 0.1
β = 0.1
β = 0.1
β = 0.1–––––

button-press
door-open
drawer-close
drawer-open
peg-insert-side
pick-place
push
reach
window-close
window-open
bin-picking
box-close
door-lockdoor-unlock
hand-insert

β = 0.1
β = 10.0
β = 1.0
β = 100.0
β = 1.0
β = 10.0
β = 100.0
β = 100.0
β = 0.1
β = 10.0
β = 1.0
β = 1.0
β = 1.0
β = 100.0
β = 10.0

β = 10.0
β = 1.0
β = 1.0
β = 10.0
β = 1.0
β = 1.0
β = 10.0
β = 100.0
β = 0.1
β = 10.0
β = 1.0
β = 1.0
β = 1.0
β = 10.0
β = 100.0

β = 1.0
β = 10.0
β = 10.0
β = 10.0
β = 1.0
β = 10.0
β = 1.0
β = 10.0
β = 1.0
β = 10.0
β = 1.0
β = 1.0
β = 10.0
β = 10.0
β = 1.0

β = 100.0
β = 0.1
β = 10.0
β = 10.0
β = 10.0
β = 1.0
β = 1.0
β = 10.0
β = 0.1
β = 0.1
β = 1.0
β = 1.0
β = 10.0
β = 10.0
β = 1.0

–
–
–
–
–
–
–
–
–
–
–
–
–
–
–

Table 10: Tuned hyperparameter values for PSPI
Datasethopper-random
hopper-medium
hopper-medium-replay
hopper-medium-expert
hopper-expert

Original
β = 0.3, τ = 0.7
β = 0.3, τ = 0.7
β = 3.0, τ = 0.7
β = 0.3, τ = 0.7
β = 0.3, τ = 0.7

Zero
β = 3.0, τ = 0.9
β = 3.0, τ = 0.7
β = 3.0, τ = 0.7
β = 0.3, τ = 0.7
β = 3.0, τ = 0.7

Randomβ = 0.3, τ = 0.7
β = 0.3, τ = 0.7
β = 3.0, τ = 0.7
β = 0.3, τ = 0.7
β = 3.0, τ = 0.7

Negativeβ = 0.3, τ = 0.9
β = 0.3, τ = 0.7
β = 0.3, τ = 0.9
β = 3.0, τ = 0.7
β = 0.3, τ = 0.9

No-terminal
β = 3.0, τ = 0.7
β = 0.3, τ = 0.7
β = 3.0, τ = 0.9
β = 0.3, τ = 0.7
β = 0.3, τ = 0.7

walker2d-random
walker2d-medium
walker2d-medium-replay
walker2d-medium-expert
walker2d-expert

β = 3.0, τ = 0.9
β = 0.3, τ = 0.7
β = 3.0, τ = 0.7
β = 3.0, τ = 0.7
β = 3.0, τ = 0.7

β = 0.3, τ = 0.7
β = 0.3, τ = 0.7
β = 3.0, τ = 0.7
β = 0.3, τ = 0.7
β = 0.3, τ = 0.7

β = 0.3, τ = 0.7
β = 3.0, τ = 0.7
β = 3.0, τ = 0.7
β = 0.3, τ = 0.7
β = 0.3, τ = 0.9

β = 0.3, τ = 0.7
β = 0.3, τ = 0.7
β = 0.3, τ = 0.7
β = 0.3, τ = 0.7
β = 0.3, τ = 0.7

β = 0.3, τ = 0.7
β = 0.3, τ = 0.7
β = 3.0, τ = 0.7
β = 0.3, τ = 0.7
β = 3.0, τ = 0.7

halfcheetah-random
halfcheetah-medium
halfcheetah-medium-replay
halfcheetah-medium-expert
halfcheetah-expert

β = 3.0, τ = 0.9
β = 3.0, τ = 0.9
β = 3.0, τ = 0.9
β = 3.0, τ = 0.7
β = 3.0, τ = 0.7

β = 0.3, τ = 0.7
β = 0.3, τ = 0.9
β = 0.3, τ = 0.7
β = 0.3, τ = 0.9
β = 3.0, τ = 0.7

β = 3.0, τ = 0.9
β = 0.3, τ = 0.7
β = 0.3, τ = 0.7
β = 0.3, τ = 0.7
β = 3.0, τ = 0.7

β = 0.3, τ = 0.7
β = 0.3, τ = 0.7
β = 3.0, τ = 0.9
β = 3.0, τ = 0.7
β = 0.3, τ = 0.7

–
–
–
–
–

button-press
door-open
drawer-close
drawer-open
peg-insert-side
pick-place
push
reach
window-close
window-open
bin-picking
box-close
door-lockdoor-unlock
hand-insert

β = 0.3, τ = 0.7
β = 3.0, τ = 0.7
β = 0.3, τ = 0.7
β = 0.3, τ = 0.7
β = 0.3, τ = 0.7
β = 0.3, τ = 0.9
β = 0.3, τ = 0.9
β = 0.3, τ = 0.7
β = 0.3, τ = 0.9
β = 0.3, τ = 0.9
β = 3.0, τ = 0.7
β = 0.3, τ = 0.9
β = 0.3, τ = 0.7
β = 0.3, τ = 0.7
β = 0.3, τ = 0.9

β = 0.3, τ = 0.7
β = 3.0, τ = 0.7
β = 0.3, τ = 0.9
β = 0.3, τ = 0.7
β = 3.0, τ = 0.7
β = 3.0, τ = 0.7
β = 3.0, τ = 0.7
β = 0.3, τ = 0.7
β = 0.3, τ = 0.9
β = 3.0, τ = 0.9
β = 0.3, τ = 0.7
β = 0.3, τ = 0.7
β = 0.3, τ = 0.7
β = 3.0, τ = 0.7
β = 3.0, τ = 0.7

β = 0.3, τ = 0.7
β = 0.3, τ = 0.7
β = 0.3, τ = 0.7
β = 0.3, τ = 0.7
β = 0.3, τ = 0.7
β = 0.3, τ = 0.7
β = 0.3, τ = 0.9
β = 0.3, τ = 0.7
β = 0.3, τ = 0.7
β = 0.3, τ = 0.7
β = 0.3, τ = 0.7
β = 0.3, τ = 0.7
β = 0.3, τ = 0.7
β = 0.3, τ = 0.9
β = 0.3, τ = 0.7

β = 0.3, τ = 0.7
β = 0.3, τ = 0.7
β = 0.3, τ = 0.7
β = 3.0, τ = 0.7
β = 0.3, τ = 0.7
β = 0.3, τ = 0.7
β = 0.3, τ = 0.9
β = 0.3, τ = 0.7
β = 0.3, τ = 0.7
β = 0.3, τ = 0.9
β = 0.3, τ = 0.7
β = 0.3, τ = 0.9
β = 0.3, τ = 0.7
β = 0.3, τ = 0.7
β = 0.3, τ = 0.7

–
–
–
–
–
–
–
–
–
–
–
–
–
–
–

Table 11: Tuned hyperparameter values for IQL
23

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

Datasethopper-random
hopper-medium
hopper-medium-replay
hopper-medium-expert
hopper-expert

Original
α = 0.1
α = 1.0
α = 10.0
α = 10.0
α = 10.0

Zeroα = 0.1
α = 10.0
α = 10.0
α = 100.0
α = 1.0

Randomα = 100.0
α = 10.0
α = 100.0
α = 100.0
α = 0.1

Negative
α = 0.1
α = 10.0
α = 10.0
α = 10.0
α = 10.0

No-terminal
α = 1.0
α = 10.0
α = 10.0
α = 10.0
α = 100.0

walker2d-random
walker2d-medium
walker2d-medium-replay
walker2d-medium-expert
walker2d-expert

α = 0.1
α = 10.0
α = 10.0
α = 10.0
α = 10.0

α = 100.0
α = 10.0
α = 10.0
α = 0.1
α = 10.0

α = 100.0
α = 1.0
α = 1.0
α = 1.0
α = 100.0

α = 100.0
α = 10.0
α = 100.0
α = 10.0
α = 100.0

α = 10.0
α = 10.0
α = 10.0
α = 100.0
α = 10.0

halfcheetah-random
halfcheetah-medium
halfcheetah-medium-replay
halfcheetah-medium-expert
halfcheetah-expert

α = 0.1
α = 0.1
α = 0.1
α = 100.0
α = 100.0

α = 0.1
α = 1.0
α = 100.0
α = 1.0
α = 10.0

α = 1.0
α = 1.0
α = 100.0
α = 1.0
α = 100.0

α = 100.0
α = 100.0
α = 100.0
α = 100.0
α = 100.0

–
–
–
–
–

button-press
door-open
drawer-close
drawer-open
peg-insert-side
pick-place
push
reach
window-close
window-open
bin-picking
box-close
door-lockdoor-unlock
hand-insert

α = 100.0
α = 1.0
α = 100.0
α = 100.0
α = 1.0
α = 100.0
α = 100.0
α = 100.0
α = 1.0
α = 100.0
α = 100.0
α = 1.0
α = 1.0
α = 100.0
α = 100.0

α = 100.0
α = 10.0
α = 10.0
α = 100.0
α = 100.0
α = 100.0
α = 100.0
α = 100.0
α = 0.1
α = 100.0
α = 100.0
α = 100.0
α = 10.0
α = 10.0
α = 100.0

α = 10.0
α = 100.0
α = 10.0
α = 100.0
α = 100.0
α = 100.0
α = 100.0
α = 100.0
α = 10.0
α = 100.0
α = 100.0
α = 100.0
α = 10.0
α = 10.0
α = 100.0

α = 100.0
α = 10.0
α = 10.0
α = 10.0
α = 10.0
α = 100.0
α = 100.0
α = 10.0
α = 10.0
α = 100.0
α = 100.0
α = 100.0
α = 100.0
α = 10.0
α = 100.0

–
–
–
–
–
–
–
–
–
–
–
–
–
–
–

Table 12: Tuned hyperparameter values for CQL

24

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

DatasetBehaviorBC

hopperrandom

1.2

3.6±0.2

hoppermedium44.3

53.3±0.1

hoppermediumreplay15.0

26.7±0.6

hoppermediumexpert64.853.6±0.1

walker2d
random

0.0

1.4±0.0

walker2d
medium62.0

68.1±0.5

walker2d
mediumreplay14.8

19.3±0.4

walker2d
mediumexpert82.7

103.0±1.0

halfcheetah
random

-0.1

1.9±0.0

halfcheetah
medium

40.7

42.2±0.0

halfcheetah
mediumreplay27.2

35.9±0.1

halfcheetah
mediumexpert64.467.9±0.4

Algo
ATAC
PSPI
IQL
CQL
DT
ATAC
PSPI
IQL
CQL
DT
ATAC
PSPI
IQL
CQL
DT
ATAC
PSPI
IQL
CQL
DT

Original5.6±0.8
7.2±0.3
7.7±0.2
3.0±0.7
11.5±0.3
84.8±1.5
94.5±1.7
57.4±0.9
85.0±3.762.4±0.6
101.5±0.2
101.4±0.3
61.0±1.979.7±3.784.0±0.9
104.2±1.5108.9±0.6109.2±0.5
88.8±3.6
89.9±2.7

Zero
22.6±3.8
12.3±3.5
5.0±0.7
2.0±1.1
11.0±0.4
94.0±0.688.4±1.3
53.1±0.4
55.0±11.7
63.5±0.8
65.7±3.932.4±1.3
16.5±2.5
1.9±0.1
60.7±2.8
106.4±1.1
93.3±5.052.8±0.3
49.7±8.1
89.8±2.6

Random
18.4±3.817.3±3.9
1.0±0.0
1.8±0.2
6.5±0.1
92.3±0.3
92.7±0.5
59.0±2.1
36.9±9.863.3±0.677.2±2.946.9±7.2
1.5±0.3
7.6±3.767.8±1.1
92.2±8.0
58.8±9.460.5±8.7
60.7±9.4
89.7±2.2

Negative
30.5±1.1
23.5±3.7
0.9±0.1
0.7±0.0
12.3±0.8
91.5±0.589.4±0.6
63.0±2.1
89.0±1.562.6±0.848.1±2.1
31.0±0.0
3.5±2.2
5.8±2.768.8±2.690.4±0.4
95.3±0.3
94.5±3.788.6±6.1
93.8±2.3

No-terminal
18.8±4.517.1±4.4
1.0±0.0
1.2±0.5–88.1±2.087.2±3.0
43.0±7.3
46.3±11.5–
99.4±0.6
99.1±0.4
6.1±3.1
4.4±2.8–98.9±4.1
108.3±0.6
77.9±7.647.4±11.0
–

ATAC
PSPI
IQL
CQL
DT
ATAC
PSPI
IQL
CQL
DT
ATAC
PSPI
IQL
CQL
DT
ATAC
PSPI
IQL
CQL
DT

6.4±0.7
6.3±0.4
7.2±0.1
4.0±1.8
6.5±0.2
86.8±1.687.7±0.778.4±1.1
81.4±0.376.6±0.385.7±1.584.9±3.2
57.7±3.674.1±1.256.3±1.3
105.9±5.4110.0±0.1
111.0±0.2
109.3±0.3
108.5±0.1

5.8±0.5
2.1±0.8
1.3±0.1
0.9±0.9
5.3±0.576.6±0.371.2±0.562.8±3.0
2.0±0.975.7±0.567.2±2.6
69.9±0.7
15.5±2.2
0.5±0.353.9±1.0
96.8±4.480.6±6.3
106.2±0.4
0.8±0.5
108.4±0.1

3.2±0.9
1.6±0.5
1.4±0.7
4.3±1.4
1.7±0.076.8±0.772.2±0.6
72.0±2.2
1.2±0.775.8±0.2
66.5±1.6
68.1±2.2
3.2±1.2-0.2±0.1
62.0±1.1
93.6±4.8
109.2±0.1
71.5±7.4
0.1±0.2
108.5±0.2

5.6±0.6
1.0±0.5
6.7±0.0-0.4±0.0
6.4±0.272.7±0.277.5±0.553.3±3.3
14.5±7.475.5±0.342.3±6.756.2±4.5
0.2±0.7
0.2±0.353.5±1.572.7±0.375.5±0.770.7±2.928.0±9.1
108.4±0.2

4.9±1.1
5.5±0.0
1.8±1.0
0.2±0.4–
86.4±0.685.2±0.539.3±0.9
2.1±0.8–80.2±1.0
81.3±0.8
0.7±0.1
0.4±0.4–
108.9±4.5111.7±1.3
28.0±0.7-0.2±0.0
–

ATAC
PSPI
IQL
CQL
DT
ATAC
PSPI
IQL
CQL
DT
ATAC
PSPI
IQL
CQL
DT
ATAC
PSPI
IQL
CQL
DT

2.2±0.0
2.6±0.4
18.6±0.330.7±0.6
2.3±0.051.2±0.1
49.4±0.1
50.0±0.1
57.2±1.543.0±0.0
47.4±0.1
48.5±0.2
43.7±0.2
48.2±5.4
38.6±0.1
91.5±0.6
86.7±1.994.8±0.175.2±1.078.8±0.8

2.2±0.0
2.3±0.0
1.8±0.1
1.0±0.8
2.3±0.043.1±0.1
43.0±0.1
39.3±2.843.2±0.1
43.0±0.033.7±2.2
35.6±0.636.1±0.534.2±0.737.2±0.2
46.4±2.052.6±1.756.5±3.2
81.2±2.1
81.1±0.7

2.2±0.0
2.2±0.0
2.0±0.1-1.4±0.3
2.3±0.0
43.0±0.1
42.9±0.1
42.3±0.1
43.5±0.1
43.0±0.0
31.1±2.536.1±0.735.3±0.3
33.1±0.937.1±0.244.9±1.249.8±1.970.8±0.8
69.6±2.879.6±1.4

2.2±0.0
2.3±0.0
2.2±0.0-5.4±0.3
2.3±0.0
41.4±0.142.8±0.1
34.7±0.342.7±0.1
43.0±0.0
40.3±0.2
35.5±0.518.1±0.4
32.4±1.0
37.0±0.2
41.0±0.142.2±0.1
37.9±0.242.4±0.278.3±0.9

–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–

Table 13: D4RL normalized scores. The mean and standard error of normalized scores are computed across 10 random
seeds. For each random seed, we evaluate the final policy over 50 episodes.

25

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data
DatasetBCbutton-press

0.98±0.00

door-open

0.81±0.02

drawer-close

0.99±0.00

drawer-open

0.88±0.01

peg-insert-side

0.58±0.01

pick-place

0.15±0.01

push

0.19±0.01

reach

0.59±0.01

window-close

1.00±0.00

window-open

0.98±0.00

bin-picking

0.72±0.01

box-close

0.57±0.01

door-lock

0.96±0.00

door-unlock

0.86±0.00

hand-insert

0.39±0.01

Algo
ATAC
PSPI
IQL
CQL
DT
ATAC
PSPI
IQL
CQL
DT
ATAC
PSPI
IQL
CQL
DT
ATAC
PSPI
IQL
CQL
DT
ATAC
PSPI
IQL
CQL
DT
ATAC
PSPI
IQL
CQL
DT
ATAC
PSPI
IQL
CQL
DT
ATAC
PSPI
IQL
CQL
DT
ATAC
PSPI
IQL
CQL
DT
ATAC
PSPI
IQL
CQL
DT
ATAC
PSPI
IQL
CQL
DT
ATAC
PSPI
IQL
CQL
DT
ATAC
PSPI
IQL
CQL
DT
ATAC
PSPI
IQL
CQL
DT
ATAC
PSPI
IQL
CQL
DT

Original
1.00±0.00
0.86±0.10
0.98±0.01
0.88±0.07
1.00±0.00
1.00±0.00
1.00±0.00
0.94±0.02
0.82±0.09
1.00±0.00
0.89±0.09
0.77±0.11
0.98±0.01
0.91±0.04
0.99±0.01
0.72±0.07
0.61±0.10
0.93±0.02
0.78±0.04
1.00±0.00
0.82±0.04
0.75±0.09
0.60±0.03
0.08±0.02
0.07±0.01
0.38±0.06
0.47±0.04
0.11±0.02
0.01±0.00
0.06±0.01
0.48±0.09
0.47±0.05
0.10±0.02
0.00±0.00
0.04±0.01
0.86±0.03
0.92±0.02
0.51±0.02
0.17±0.03
0.67±0.03
0.98±0.01
0.90±0.08
0.99±0.01
1.00±0.00
1.00±0.00
0.51±0.13
1.00±0.00
0.97±0.01
0.83±0.03
0.91±0.01
0.41±0.12
0.29±0.10
0.39±0.02
0.00±0.00
0.00±0.00
0.20±0.09
0.34±0.10
0.58±0.02
0.04±0.01
0.02±0.00
1.00±0.00
0.78±0.12
0.99±0.00
0.75±0.04
0.81±0.02
0.97±0.01
0.95±0.04
0.87±0.02
0.87±0.03
0.97±0.01
0.80±0.03
0.81±0.02
0.35±0.03
0.13±0.04
0.24±0.03

Zero
0.90±0.09
0.90±0.09
0.97±0.01
0.83±0.10
1.00±0.00
1.00±0.00
0.95±0.03
0.86±0.03
0.07±0.07
1.00±0.00
0.97±0.03
0.96±0.04
0.99±0.01
0.99±0.01
1.00±0.00
0.35±0.09
0.42±0.09
0.84±0.02
0.66±0.07
1.00±0.00
0.88±0.06
0.73±0.07
0.61±0.03
0.00±0.00
0.00±0.00
0.35±0.05
0.16±0.05
0.16±0.02
0.06±0.02
0.06±0.01
0.48±0.08
0.42±0.04
0.18±0.02
0.01±0.00
0.06±0.02
0.81±0.05
0.77±0.04
0.59±0.04
0.29±0.04
0.79±0.02
1.00±0.00
0.99±0.01
1.00±0.00
1.00±0.00
0.98±0.01
0.74±0.10
0.98±0.01
0.98±0.01
0.82±0.05
0.91±0.01
0.49±0.09
0.41±0.08
0.59±0.07
0.02±0.02
0.00±0.00
0.50±0.13
0.44±0.10
0.54±0.02
0.07±0.02
0.03±0.01
0.98±0.01
0.80±0.12
0.96±0.01
0.61±0.10
0.83±0.02
0.93±0.05
0.98±0.01
0.88±0.02
0.77±0.09
0.97±0.01
0.36±0.08
0.62±0.08
0.34±0.02
0.22±0.06
0.21±0.02

Random
1.00±0.00
0.84±0.10
0.94±0.02
0.84±0.09
1.00±0.00
1.00±0.00
0.96±0.02
0.92±0.03
0.12±0.08
1.00±0.00
1.00±0.00
1.00±0.00
0.99±0.00
0.96±0.02
1.00±0.00
0.41±0.08
0.57±0.07
0.73±0.04
0.65±0.07
1.00±0.00
0.83±0.05
0.92±0.03
0.34±0.02
0.02±0.01
0.01±0.00
0.35±0.05
0.31±0.04
0.17±0.01
0.05±0.02
0.06±0.01
0.34±0.04
0.35±0.08
0.14±0.01
0.01±0.00
0.04±0.01
0.83±0.03
0.84±0.03
0.57±0.02
0.25±0.05
0.81±0.02
0.90±0.09
0.89±0.07
0.99±0.01
0.59±0.13
0.99±0.00
1.00±0.00
1.00±0.00
0.98±0.00
0.92±0.02
0.89±0.02
0.51±0.12
0.33±0.09
0.35±0.04
0.00±0.00
0.00±0.00
0.32±0.11
0.28±0.09
0.40±0.04
0.01±0.00
0.04±0.02
0.70±0.14
0.99±0.01
0.91±0.01
0.61±0.11
0.80±0.02
0.98±0.01
0.88±0.09
0.86±0.02
0.89±0.04
0.98±0.01
0.80±0.03
0.75±0.04
0.18±0.03
0.34±0.04
0.28±0.02

Negative
0.90±0.09
0.90±0.09
0.88±0.02
0.94±0.03
1.00±0.00
0.95±0.04
0.81±0.09
0.92±0.02
0.00±0.00
1.00±0.00
1.00±0.00
1.00±0.00
0.97±0.01
0.97±0.02
1.00±0.00
0.69±0.09
0.74±0.06
0.52±0.03
0.84±0.03
1.00±0.00
0.92±0.03
0.91±0.03
0.11±0.01
0.00±0.00
0.00±0.00
0.24±0.05
0.14±0.04
0.10±0.01
0.07±0.03
0.06±0.01
0.27±0.06
0.37±0.08
0.13±0.01
0.03±0.02
0.07±0.01
0.81±0.03
0.84±0.04
0.58±0.02
0.23±0.03
0.83±0.02
0.86±0.10
0.92±0.06
0.99±0.00
0.55±0.13
1.00±0.00
1.00±0.00
0.70±0.12
0.96±0.02
0.83±0.07
0.90±0.01
0.37±0.12
0.30±0.11
0.31±0.04
0.00±0.00
0.00±0.00
0.53±0.12
0.24±0.08
0.23±0.01
0.05±0.01
0.05±0.03
0.62±0.12
0.82±0.10
0.69±0.04
0.60±0.08
0.78±0.02
0.94±0.03
0.98±0.01
0.83±0.02
0.93±0.03
0.98±0.01
0.61±0.04
0.81±0.03
0.14±0.02
0.32±0.03
0.26±0.02

Table 14: Meta-World success rate. The mean and standard error of success rate are computed across 10 random seeds. For
each random seed, we evaluate the final policy over 50 episodes.
26

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

C. Technical Details of Section 3
Here we provide the details of our theory on the robustness of offline RL in Section 3.
Theorem 3.1. (Informal) Under Assumption 2.1 and certain regularity assumptions, if an offline RL algorithm Algo is set
to be sufficiently pessimistic and the data distribution µ has a positive bias, for any data reward r̃ ∈ R̃, the policy π̂ learned
by Algo from the dataset D̃ has performance guarantee∗

Vrπ (d0 ) − Vrπ̂ (d0 ) ≤ O(ι)
as well as safety guarantee(1 − γ)P∞t/ supp(µ)|π̂) ≤ O(ι)
t=0 γ Prob (∃τ ∈ [0, t], sτ ∈for a small ι that decreases to zero as the degree of pessimism and dataset size increase.
Below we first provide in Appendix C.1 some background for stating the formal version Theorem 3.1 in Appendix C.2. Then
we present the formal claims on survival instinct and conditions on positive data bias in Appendix C.3 and Appendix C.4,
respectively. Finally, we prove Theorem 3.1 in Appendix C.5. Our discussion here will be focusing the discounted
infinite-horizon step. We discuss the finite horizon variant in Appendix D.
C.1. DefinitionsFirst, we introduce some definitions so that we can state the formal version of Theorem 3.1. We define constrained MDPs.
Definition C.1 (Constrained MDP). Let f, g : S × A → [−1, 1]. A CMDP problem C(S, A, f, g, P, γ) is defined asmaxπVfπ (d0 ),s.t.

Vgπ (d0 ) ≤ 0.

Let π † denote its optimal policy. For δ ≥ 0, we define the set of δ-approximately optimal policies,†Π†f,g (δ) = {π : Vfπ (d0 ) − Vfπ (d0 ) ≤ δ, Vgπ (d0 ) ≤ δ}
We will drop the subscript f, g from Π†f,g when it is clear from the context. In addition, we define a sensitivity function toapproximately optimal policies,†κ(δ) := max Vfπ (d0 ) − Vfπ (d0 )π∈Π† (δ)

The sensitivity function measures how much the approximately optimal policies (which can slightly violate the constraintof the CMDP) would perform better than the optimal policy of the CMDP (which satisfies the constraint). By definition,22] and its value can be smaller than 1−γ.
κ(δ) ∈ [0, 1−γ
By duality, a CMDP problem is related to a saddle-point problem induced by its constraint. We review the definition of
saddle points and this fact below.
Definition C.2. Given a bifunction L(x, y) : X × Y → R, we say (x† , y † ) ∈ X × Y is a saddle point of L, if for all x ∈ X
and y ∈ Y it holds that L(x, y † ) ≤ L(x† , y † ) ≤ L(x† , y).

Lemma C.3 (Duality). (Altman, 1999) Let Π denote the space of policies. Consider a CMDP C := C(S, A, f, g, P, γ).
Define the Lagrangian L(π, λ) := Vfπ (d0 ) − λVgπ (d0 ) for π ∈ Π and λ ≥ 0. Let π † and λ† denote the optimal policy and
Lagrange multiplier, i.e.,π † ∈ arg max min L(π, λ) and λ† ∈ arg min max L(π, λ)
π

λ≥0

λ≥0

π

Then together they form a saddle point (π † , λ† ) to the Lagrangian L.
We also introduce a definition to characterize the degree of pessimism of offline RL algorithms. We call it an admissibility
condition defined below, which measures the size of the data-consistent reward functions that the offline RL algorithm canhave a small regret.
27

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

Definition C.4. [Admissibility] For R ≥ 0, we say an offline RL algorithm Algo is R-admissible with respect to R̃, if for
any r̃ ∈ R̃, given D̃, Algo learns a policy π̂ satisfying the following with high probability: for any policy π ∈ Π,
max Vr̄π (d0 ) − Vr̄π̂ (d0 ) ≤ E(π, µ),r̄∈RR (r̃)

where we define a data-consistent reward class
RR (r̃) := {r̄ : r̄(s, a) = r̃(s, a), ∀(s, a) ∈ supp(µ) and r̄(s, a) ∈ [−R, 1] , ∀(s, a) ∈ S × A} ,π(s,a)< ∞, and o(1) denotes a term thatE(π, µ) is some regret upper bound such that E(π, µ) = o(1) if sups∈S,a∈A dµ(s,a)
vanishes as the dataset size becomes infinite.

In Appendix E, we provide proofs of the degrees of admissibility for various existing algorithms, including model-freealgorithms ATAC (Cheng et al., 2022), VI-LCB (Rashidinejad et al., 2021) PPI/PQI (Liu et al., 2020), PSPI (Xie et al., 2021),
as well as model-algorithms, ARMOR (Xie et al., 2022; Bhardwaj et al., 2023), MOPO (Yu et al., 2020b), MOReL (Kidambi
et al., 2020), and CPPO (Uehara and Sun, 2022). We show that their degree of admissibility increases as the algorithm
becomes more pessimistic (by setting the hyperparameters).
Finally, we recall the definition of positive data bias in Section 3 for completeness, where the implicit CMDP Cµ (r̃) is
defined in (1).
Definition C.5 (Positive Data Bias). A distribution µ is 1ϵ -positively biased w.r.t. a reward class R̃ if∗

max max Vrπ (d0 ) − Vrπ (d0 ) ≤ ϵ + O(δ)(2)

r̃∈R̃ π∈Π† (δ)

for all δ ≥ 0, where Π† (δ) denotes the set of δ-approximately optimal policy of Cµ (r̃).
Note that the CMDP Cµ (r̃) is feasible because of the assumption on concentrability (Assumption 2.1). That is, we know π ∗
π∗

(s,a)
is feasible policy, as (1 − γ)Vcπµ (d0 ) = Edπ∗ [1[(s, a) ∈/ supp(µ)]] = 0 is implied by sups,a dµ(s,a)< ∞.
∗C.2. Formal Statement of the Main Theorem
Now we state the formal version of Theorem 3.1.
Assumption C.6 (Data Assumption).
1. For all r̃ ∈ R̃, r̃(s, a) ∈ [−1, 1] for all s ∈ S and a ∈ A.
2. The data distribution µ is 1ϵ -positively biased with respect to R̃.
3. There is a policy π satisfying Vcµ ≤ 0.
4. For any r̃ ∈ R̃, the CMDP Cµ (r̃) := C(S, A, r̃, cµ , P, γ) in (1) has a sensitivity function κ(δ) ≤ K for some K < ∞
π†

(s,a)
and limδ→0+ κ(δ) = 0. In addition, its solution π † satisfies sups,a dµ(s,a)< ∞.

Theorem C.7 (Main Result). Under the data assumption in Assumption C.6, consider an offline RL algorithm Algo thatis sufficiently pessimistic to be ( Kδ + 2)-admissible with respect to R̃. Then for any r̃ ∈ R̃, with high probability, thepolicy π̂ learned by Algo from the dataset D̃ := {(s, a, r̃, s′ )|(s, a) ∼ µ, r̃ = r̃(s, a), s′ ∼ P (·|s, a)} has both performance
guarantee∗

Vrπ (d0 ) − Vrπ̂ (d0 ) ≤ ϵ + O(ι)
and safety guarantee(1 − γ)∞X
t=0

γ t Prob (∃τ ∈ [0, t]sτ ∈/ supp(µ)|π̂) ≤ ι

where ι := δ + κ(δ) + o(1) and o(1) denotes a term that vanishes as the dataset size becomes infinite.
28

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

In Assumption C.6, the first assumption supposes the data reward (which may be the wrong reward) is bounded; the second
assumption is that the data distribution is positively biased in view of the reward class R̃. Finally, we make a regularity
π†

(s,a)
< ∞ to rule out some unrealistic R̃ and P combinations.
assumption on the convergence of κ(δ) to zero and sups,a dµ(s,a)
We note that the condition on the bounded density ratio is always true for tabular problem, and a degenerate case can happen
when the state-action space is, e.g., continuous. On the other hand, a degenerate case that κ is discontinuous at δ = 0
can happen due to the infinite problem horizon in the worst case; roughly speaking, this happens when it is impossible todetermine the constraint violation using a finite-horizon estimator. We provide the following lemma to quantify the rate of
κ(δ), which is proved in Appendix C.6. In Appendix D, we show this regularity condition is always true in the finite horizon
setting.

Lemma C.8. Suppose there is λ† ≤ L with some L < ∞ such that λ† ∈ arg minλ≥0 maxπ Vr̃π (d0 ) − λVcπµ (d0 ). Thenκ(δ) ≤ Lδ.
Under the data assumption Assumption C.6, Theorem C.7 shows that if the offline RL algorithm Algo is set to be sufficiently
1
pessimistic (relative to the upper bound of the sensitivity function κ, which we know is at most 1−γby the first assumptionin Assumption C.6), then Algo have both a performance guarantee to the true reward r and the a safety guarantee that thelearned policy would have a small probability of leaving the data support.
We recall from Section 3 that Theorem C.7 is the effects of the survival instinct of offline RL and the positive bias in thedata distribution. As we will show, the admissibility condition in Theorem C.7 would ensure that the survival instinct,
which is what we stated informally as Proposition 3.3. In addition, we will provide concrete sufficient conditions for thedata distribution being positively bias, as we alluded in Section 3.3. We discuss these details below and then we prove
Theorem C.7 in Appendix C.5.
C.3. Survival Instinct
We show the formal version of Proposition 3.3 which says that if the offline algorithm is admissible then it has a survival
instinct. That is, it implicitly solves the CMDP in (1) even though the constraint is never explicitly modeled in the algorithm.
In Appendix E, we prove the admissibility for various existing algorithms, including model-free algorithms ATAC (Chenget al., 2022), VI-LCB (Rashidinejad et al., 2021) PPI/PQI (Liu et al., 2020), PSPI (Xie et al., 2021), as well as modelalgorithms, ARMOR (Xie et al., 2022; Bhardwaj et al., 2023), MOPO (Yu et al., 2020b), MOReL (Kidambi et al., 2020),
and CPPO (Uehara and Sun, 2022).
Proposition C.9. For r̃ ∈ R̃, assume the CMDP Cµ (r̃) := C(S, A, r̃, cµ , P, γ) has a sensitivity function such that1limδ→0+ κ(δ) = 0 and κ(δ) ≤ K for some K ≤ 1−γ. Consider an offline RL algorithm Algo that is ( Kδ + 2)-admissible
with respect to R̃. Then, with high probability, the policy π̂ learned by Algo with the dataset D̃ := {(s, a, r̃, s′ )|(s, a) ∼µ, r̃ = r̃(s, a), s′ ∼ P (·|s, a)} satisfies†

Vr̃π (d0 ) − Vr̃π̂ (d0 ) ≤ o(1)Vcπ̂µ (d0 ) ≤ δ + κ(δ) + o(1)where o(1) denotes a term that vanishes as the dataset size becomes infinite.
Proof of Proposition C.9. To begin, we introduce some extra notations. Given Cµ (r̃), we define a relaxed CMDP problem:
maxπ Vr̃π (d0 ), s.t.Vcπµ (d0 ) ≤ δ and its Lagrangian Lδ (π, λ) := Vr̃π (d0 ) − λ(Vcπµ (d0 ) − δ), where λ ≥ 0 denotes the
Lagrange multiplier. Let πδ† and λ†δ denote the optimal policy and Lagrange multiplier of this relaxed CMDP, and let π †denote the optimal policy to the CMDP Cµ (r̃). Finally we define a shorthand r† (s, a) := r̃(s, a) − (λ†δ + 1)cµ (s, a).

Now we bound the regret and the constraint violation of π̂ in the CMDP Cµ (r̃). The following holds for any δ ≥ 0. For the
regret, we can derive†††(Vcπµ (d0 ) = 0)††

(Vcπ̂µ (d0 ) ≥ 0)

Vr̃π (d0 ) − Vr̃π̂ (d0 ) = Vr̃π (d0 ) − (λ†δ + 1)Vcπµ (d0 ) − Vr̃π̂ (d0 )≤ Vr̃π (d0 ) − (λ†δ + 1)Vcπµ (d0 ) − Vr̃π̂ (d0 ) + (λ†δ + 1)Vcπ̂µ (d0 )†= Vrπ† (d0 ) − Vrπ̂† (d0 ).
29†

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

Here we use the additivity of value function over reward, namely, for any reward functions r1 , r2 and policy π, we haveVrπ1 + Vrπ2 = Vrπ1 +r2 . Similarly for the constraint violation, we can deriveVcπ̂µ (d0 ) = (Vcπ̂µ (d0 ) − δ) + δ= (λ†δ + 1)(Vcπ̂µ (d0 ) − δ) − λ†δ (Vcπ̂µ (d0 ) − δ) + δ= −Vr̃π̂ (d0 ) + (λ†δ + 1)(Vcπ̂µ (d0 ) − δ) + Vr̃π̂ (d0 ) − λ†δ (Vcπ̂µ (d0 ) − δ) + δ= Lδ (π̂, λ†δ ) − Lδ (π̂, λ†δ + 1) + δ≤ Lδ (πδ† , λ†δ + 1) − Lδ (π̂, λ†δ + 1) + δπ†= Vr†δ (d0 ) − Vrπ̂† (d0 ) + δ†≤ Vrπ† (d0 ) − Vrπ̂† (d0 ) + δ + κ(δ)

where the first inequality is due to that (πδ† , λ†δ ) is a saddle point to Lδ (see Definition C.2), and the second inequality follows
from
π†

π†

π†

π†

†

Vr†δ (d0 ) = Vr̃ δ (d0 ) − (λ†δ + 1)Vcµδ (d0 ),≤ Vr̃ δ (d0 ) − (λ†δ + 1)Vcπµ (d0 )
†

π†

†

(using Vcπµ ≤ Vcµδ )†≤ Vr̃π (d0 ) + κ(δ) − (λ†δ + 1)Vcπµ (d0 )(Definition of κ.)†≤ Vrπ† (d0 ) + κ(δ)†

K†
Next we bound on Vrπ† (d0 ) − Vrπ̂† (d0 ). By Lemma C.13 and Assumption C.6, we know λ†δ ≤ K
δ , so r (s, a) ∈ [− δ − 2, 1].
K
This and the definition of cµ together imply that r† ∈ Rµ (r̃, K
δ ). Since the offline RL algorithm Algo is δ -admissible,
π†

†

(s,a)
we have Vrπ† (d0 ) − Vrπ̂† (d0 ) ≤ E(π † , µ). Finally, we use the fact that π † by definition satisfies sups,a dµ(s,a)< ∞, so†E(π , µ) = o(1). Thus, we have proved†

Vr̃π (d0 ) − Vr̃π̂ (d0 ) ≤ o(1)Vcπ̂µ (d0 ) ≤ o(1) + δ + κ(δ)

That is, π̂ ∈ Πι (Cµ (r̃)) with ι = δ + κ(δ) + o(1).

C.4. Implicit Data Bias
Below we provide example conditions for data distributions to have a positive implicit bias.
Proposition C.10. Under Assumption 2.1, the followings are true.
1. (RL setup) Suppose for any r̃ in R̃, there is h : S → R such that(
ϵ1 , s, a ∈ supp(µ)|r(s, a) + γEs′ ∼P|s,a [h(s)] − h(s) − r̃(s, a)| ≤ϵ2 , otherwise
for some ϵ1 , ϵ2 < ∞. Then µ is 1−γ2ϵ1 -positively biased to R̃.
∗e

2. (IL setup) If µ = dπ , then µ is ∞-positively biased with respect to R̃ = {r̃ : r̃ : S × A → [−1, 1]}. If µ = dπ such∗eϵ′that Vrπ (d0 ) − Vrπ (d0 ) ≤ 1−γ
and π e is deterministic, then µ is 1−γϵ′ -positively biased to the previous R̃.
′

3. (Length bias) Suppose Vcπµ (d0 ) ≤ δ implies Vr∗ (d0 ) − Vrπ (d0 ) ≤ ϵ +O(δ)
1−γ . Then µ is O

1−γ
ϵ′



-positively biased withrespect to R̃ = {r̃ : r̃ : S × A → [−1, 1]}. Below we list some sufficient intervention conditions on data collectionthat leads to the length bias:
30

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data(a) For all (s, a) ∈ supp(µ), Rmax − r(s, a) ≤ ϵ′ , where Rmax = sups∈S,a∈A r(s, a) ≤ 1.
(b) For all (s, a) ∈ supp(µ), Q∗r (s, a) − Vr∗ (s) ≤ ϵ′ .
The first example is the typical RL assumption, which says µ always has a positive bias to R̃ if all rewards in R̃ provide
the same ranking of policies in the support (Ng et al., 1999). The second assumes the data is collectd by a single behavior
policy, which includes the typical IL assumption that the data is collected by an optimal policy. Finally, the third example
describes a length bias. Note that Vcπµ (s, a) measures the probability of going out of data support after taking a at s, which
is disproportional to expected length the agent can survive. This condition assumes longer trajectories in the data to havesmaller optimality gap.
This length bias condition is typically satisfied when intervention is taken in the data collection process (despite the datacollection policy being suboptimal), as we saw in the motivating example in Figure 1 and Section 3.1, and empiricallyin Section 4. We give two sufficient conditions on such interventions. The first one is that the data collection is stopped
when the instantaneous rewards is far from the maximum rewards, and the second is that the data collection agent does
not take an action that no policy (even the optimal one) can perform well in the future. The consequence of these types ofinterventions is that longer trajectories perform better generally. These conditions are satisfied in hopper and walker2d
dataset (except for -medium-replay) in Section 4, as well as in the grid world example in Section 3.1.
The proof of Proposition C.10 is based on the performance difference lemma, which can be proved by a telescoping sum
decomposition.
Lemma C.11 (Performance Difference Lemma). Consider a discounted infinite-horizon MDP M = (S, A, r, P, γ). For
any policy π and any h : S → R, it holds that
Vrπ (d0 ) − h(d0 ) =1
Es,a∼dπ [r(s, a) + γEs′ ∼P |s,a [h(s′ )] − h(s)].
1−γ

In particular, this implies for any policies π, π ′ ,′

Vrπ (d0 ) − Vrπ (d0 ) =′′1
Es,a∼dπ [Qπr (s, a) − Vrπ (s)].
1−γC.4.1. P ROOF OF RL SETUP IN P ROPOSITION C.10
Proof. Given r̃ ∈ R̃, consider some h : S × A → R satisfying the assumption. Definer̂(s, a) = r(s, a) + γEs′ ∼P|s,a [h(s)] − h(s).
Notice∗(1 − γ)(Vrπ (d0 ) − Vrπ (d0 ))∗= (1 − γ)(Vrπ (d0 ) − h(d0 ) − Vrπ (d0 ) + h(d0 ))= Es,a∼dπ∗ [r(s, a) + γEs′ ∼P |s,a [h(s′ )] − h(s)] − Es,a∼dπ [r(s, a) + γEs′ ∼P |s,a [h(s′ )] − h(s)]= Es,a∼dπ∗ [r̂(s, a)] − Es,a∼dπ [r̂(s, a)]∗= (1 − γ)(Vr̂π (d0 ) − Vr̂π (d0 ))By Lemma C.11, for any π ∈ Π† (δ), it holds∗(1 − γ)(Vrπ (d0 ) − Vrπ (d0 ))∗= (1 − γ)(Vr̂π (d0 ) − Vr̂π (d0 ))= Es,a∼dπ∗ [r̂(s, a)] − Es,a∼dπ [r̂(s, a)]= Es,a∼dπ∗ [r̃(s, a)] − Es,a∼dπ [r̃(s, a)] + Es,a∼dπ∗ [r̂(s, a) − r̃(s, a)] − Es,a∼dπ [r̂(s, a) − r̃(s, a)]∗= (1 − γ)(Vr̃π (d0 ) − Vr̃π (d0 )) + Es,a∼dπ∗ [r̂(s, a) − r̃(s, a)] − Es,a∼dπ [r̂(s, a) − r̃(s, a)]31

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

Since π ∗ is assumed to be in the support of µ in Assumption 2.1, we haveEs,a∼dπ∗ [r̂(s, a) − r̃(s, a)] − Es,a∼dπ [r̂(s, a) − r̃(s, a)]= Es,a∼dπ∗ [r̂(s, a) − r̃(s, a)|(s, a) ∈ supp(µ)] − Es,a∼dπ [r̂(s, a) − r̃(s, a)|(s, a) ∈ supp(µ)]− Es,a∼dπ [r̂(s, a) − r̃(s, a)|(s, a) ∈/ supp(µ)]≤ 2ϵ1 − sup |r̂(s, a) − r̃(s, a)| × Es,a∼dπ [1[s, a ∈/ supp(µ)]]
s,a≤ 2ϵ1 + (1 − γ)ϵ2 Vcπµ (d0 )Putting these two inequalities together, we have shown∗∗(1 − γ)(Vrπ (d0 ) − Vrπ (d0 )) ≤ (1 − γ)(Vr̃π (d0 ) − Vr̃π (d0 )) + 2ϵ + ϵ2 (1 − γ)Vcπµ (d0 )Since π ∈ Π† (δ), it implies∗(1 − γ)(Vrπ (d0 ) − Vrπ (d0 )) ≤ (1 − γ)δ + 2ϵ1 + (1 − γ)ϵ2 δ = 2ϵ1 + O(δ)
Thus, the distribution is 1−γ2ϵ1 -positively biased.
C.4.2. P ROOF OF IL SETUP IN P ROPOSITION C.10∗

Proof. Let µ = dπ . We can write∗(1 − γ)(Vrπ (d0 ) − Vrπ (d0 ))
∗

∗

∗

∗

= Edπ [Vrπ (s) − Qπr (s, a)]∗∗= Edπ [Vrπ (s) − Qπr (s, a)|(s, a) ∈ supp(µ)] + Edπ [Vrπ (s) − Qπr (s, a)|(s, a) ∈/ supp(µ)]≤δ

So the distribution is ∞-positively biased. For the case with the suboptimal expert π e , we can derive similarly∗(1 − γ)(Vrπ (d0 ) − Vrπ (d0 ))
∗

e

e

e

e

e

e

= (1 − γ)(Vrπ (d0 ) − Vrπ (d0 )) + (1 − γ)(Vrπ (d0 ) − Vrπ (d0 ))≤ ϵ′ + Edπ [Vrπ (s) − Qπr (s, a)]ee= ϵ′ + Edπ [Vrπ (s) − Qπr (s, a)|(s, a) ∈ supp(µ)] + Edπ [Vrπ (s) − Qπr (s, a)|(s, a) ∈/ supp(µ)]≤ ϵ′ + δ

Therefore, µ is 1−γϵ′ -positively biased.
C.4.3. P ROOF OF L ENGTH B IAS IN P ROPOSITION C.10
Item 3aProof. Suppose Rmax − r(s, a) ≤ ϵ′ for (s, a) ∈ supp(µ).
∗(1 − γ)(Vrπ (d0 ) − Vrπ (d0 ))= Edπ∗ [r(s, a)] − Edπ [r(s, a)]= Edπ∗ [r(s, a)] − Edπ [r(s, a)|(s, a) ∈ supp(µ)] − Edπ [r(s, a)|(s, a) ∈/ supp(µ)]≤ Rmax − Edπ [(Rmax − ϵ′ )|(s, a) ∈ supp(µ)] − 0= Edπ [ϵ′ |(s, a) ∈ supp(µ)] + [Rmax |(s, a) ∈/ supp(µ)]≤ ϵ′ + (1 − γ)Rmax δ

32

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

Item 3b
Proof. Suppose s, a ∈ supp(µ) only if Vr∗ (s) − Q∗r (s, a) ≤ ϵ′ . Then∗

(1 − γ)(Vrπ (d0 ) − Vrπ (d0 ))
∗

∗

∗

∗

= Edπ [Vrπ (s) − Qπr (s, a)]∗∗= Edπ [Vrπ (s) − Qπr (s, a)|(s, a) ∈ supp(µ)] + Edπ [Vrπ (s) − Qπr (s, a)|(s, a) ∈/ supp(µ)]≤ ϵ′ + δNote that more broadly, we can relax the above condition to that Q∗cµ (s, a) − Vc∗µ (s) < α implies Vr∗ (s) − Q∗r (s, a) ≤β + O(α), for a potentially smaller β, where we recall that Vc∗µ , Q∗cµ are the optimal value functions to (minimizing) the
cost cµ and Vr∗ , Q∗r are the optimal value functions to (maximizing) the reawrd r. We can use the lemma below to prove thedegree of positive bias is 1−γβ in this more general case. The above is a special case for α = 1, since cµ (s, a) = 1 when s, a
is not in the support of µ.
′

∗
∗
α
π
Lemma C.12. Define Aα
s = {a ∈ A : Qcµ (s, a) − Vcµ (s) < α}. Suppose for all s ∈ S, a ∈ As satisfies Vr (s) −π′π′Qr (s, a) ≤ β. For π such that Vcµ (d0 ) ≤ δ, it holds for any π′

Vrπ (d0 ) − Vrπ (d0 ) ≤β + δ/α1−γProof.
′

′

′

′

′

(1 − γ)(Vrπ (d0 ) − Vrπ (d0 )) = Edπ [Vrπ (s) − Qπr (s, a)]

′

′

π
π
= Edπ [Vrπ (s) − Qπr (s, a)|a ∈ Aα
/ Aαs ] + Edπ [Vr (s) − Qr (s, a)|a ∈s]1≤β+Edπ [1[a ∈
/ Aα
s ]]
1−γ

On the other hand, we have
(1 − γ)δ ≥ (1 − γ)Vcπµ (d0 ) = (1 − γ)(Vcπµ (d0 ) − Vc∗µ (d0 ))= Edπ [Q∗cµ (s, a) − Vc∗µ (s)|a ∈ As ] + Edπ [Q∗cµ (s, a) − Vc∗µ (s)|a ∈/ As ]≥ Edπ [Q∗cµ (s, a) − Vc∗µ (s)|a ∈/ As ]≥ αEdπ [1[a ∈/ As ]]
Therefore,′(1 − γ)(Vrπ (d0 ) − Vrπ (d0 )) ≤ β +δ
αC.5. Proof of Theorem C.7
Proof. By Proposition C.9, we have proved†

Vr̃π (d0 ) − Vr̃π̂ (d0 ) ≤ o(1)Vcπ̂µ (d0 ) ≤ o(1) + δ + κ(δ)

That is, π̂ ∈ Πι (Cµ (r̃)) with ι = δ + κ(δ) + o(1).
33Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

Then by 1ϵ -positively assumption on µ, we have the performance bound†Vrπ (d0 ) − Vrπ̂ (d0 ) ≤ O(ι)
To bound of out of support probability, we deriveVcπ̂µ (d0 ) =∞X

γ t Edπ̂t [1[µ(s, a) = 0]]t=0= (1 − γ)≥ (1 − γ)

∞
X
t=0
∞
X
t=0

γt

t
X

!

Edπ̂τ [1[µ(s, a) = 0]]τ =0γ t Prob (∃τ ∈ [0, t]sτ ∈/ supp(µ)|π̂)

where second equality follows (Wagener et al., 2021)[Lemma 2].
C.6. Technical Details of Constrained MDP
Lemma C.13. For g : S × A → [0, ∞), let C(S, A, f, g, P, γ) be a CMDP with sensitivity function κ(δ). Assume
C(S, A, f, g, P, γ) is feasible. Consider a relaxed CMDP, maxπ Vfπ (d0 ), s.t. Vgπ (d0 ) ≤ δ and let λ†δ ≥ 0 denote its optimal

Lagrange multiplier. Then λ†δ ≤ κ(δ)δ .

Proof. Denote the optimal policy of the relaxed CMDP as πδ† and the optimal policy of the original CMDP as π † . By
π†

†

†

construction, πδ† ∈ Π† (δ). Therefore, we have Vf δ (d0 ) − Vfπ (d0 ) = maxπ∈Π† (δ) Vfπ (d0 ) − Vfπ (d0 ) = κ(δ).
To bound λ†δ , we use that the fact that (πδ† , λ†δ ) is a saddle point to the Lagrangian of the relaxed problem Lδ (π, λ) :=Vfπ (d0 ) − λ(Vgπ (d0 ) − δ). As a result, we can derive
†

π†

†

Vfπ (d0 ) − λ†δ (Vgπ (d0 ) − δ) = Lδ (π † , λ†δ ) ≤ Lδ (πδ† , λ†δ ) ≤ Lδ (πδ† , 0) = Vf δ (d0 ).
†Since Vgπ (d0 ) = 0, the above inequality implies†π†
Vf δ (d0 ) − Vfπ (d0 )
κ(δ)
†
λδ ≤
=

δ

δ

Lemma C.14. Suppose there is λ† ≤ L with some L < ∞ such that λ† ∈ arg minλ≥0 maxπ Vr̃π (d0 ) − λVcπµ (d0 ). Thenκ(δ) ≤ Lδ.
Proof. First we note by12 (Nguyen and Cheng, 2023, Theorem 4.1) we know that (π † , λ† ) is a saddle-point to the CMDPCµ (r̃) = C(S, A, r̃, cµ , P, γ) in (1). Let f = r̃ and g = cµ . Define the Lagrangian Lδ (π, δ) = Vfπ (d0 ) − λ(Vgπ (d0 ) − δ)for the relaxed CMDP problem maxπ:Vgπ (d0 )≤δ Vfπ (d0 ). Consider a saddle-point (πδ† , λ†δ ) to this relaxed problem .
By duality, we can first derive†max Vfπ (d0 ) − Vfπ (d0 ) =π∈Π† (δ)
12max

π:Vgπ (d0 )≤δ††Vfπ (d0 ) − Vfπ (d0 ) = Lδ (πδ† , λ†δ ) − Vfπ (d0 )We use this theorem because the CMDP here does not satisfy the Slater’s condition.

34

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

Then we upper bound Lδ (πδ† , λ†δ ):Lδ (πδ† , λ†δ ) ≤ Lδ (πδ† , λ† )
π†

π†

π†

π†

= Vf δ (d0 ) − λ† (Vf δ (d0 ) − δ)= Vf δ (d0 ) − λ† Vf δ (d0 ) + δλ†= L(πδ† , λ† ) + δλ†≤ L(π † , λ† ) + δλ†≤ L(π † , 0) + δλ† .
Thus we have
†

†

†

†

max Vfπ (d0 ) − Vfπ (d0 ) ≤ L(π † , 0) + δλ† − Vfπ (d0 ) = Vfπ (d0 ) + δλ† − Vfπ (d0 ) = δλ†π∈Π† (δ)

D. Finite-Horizon Version
We discuss how to interpret Theorem C.7 in the finite-horizon setup.
Assumption D.1 (Data Assumption).
1. For all r̃ ∈ R̃, r̃(s, a) ∈ [−1, 1] for all s ∈ S and a ∈ A.
2. The data distribution µ is 1ϵ -positively biased with respect to R̃.
3. There is a policy π satisfying Vcµ ≤ 0.
4. For any r̃ ∈ R̃, the CMDP Cµ (r̃) := C(S, A, r̃, cµ , P, γ) in (1) has a sensitivity function κ(δ) ≤ K for some K < ∞
π†

(s,a)
and limδ→0+ κ(δ) = 0. In addition, its solution π † satisfies sups,a dµ(s,a)< ∞.

Theorem C.7 (Main Result). Under the data assumption in Assumption C.6, consider an offline RL algorithm Algo thatis sufficiently pessimistic to be ( Kδ + 2)-admissible with respect to R̃. Then for any r̃ ∈ R̃, with high probability, thepolicy π̂ learned by Algo from the dataset D̃ := {(s, a, r̃, s′ )|(s, a) ∼ µ, r̃ = r̃(s, a), s′ ∼ P (·|s, a)} has both performance
guarantee∗

Vrπ (d0 ) − Vrπ̂ (d0 ) ≤ ϵ + O(ι)
and safety guarantee(1 − γ)∞X
t=0

γ t Prob (∃τ ∈ [0, t]sτ ∈/ supp(µ)|π̂) ≤ ι

where ι := δ + κ(δ) + o(1) and o(1) denotes a term that vanishes as the dataset size becomes infinite.
Notation First we the finite-horizon setup. To translate the previous notation
to the finite-horizon setting, we suppose theSH−1state s contains time information and the state space is layered. That is, S = 0=1 St , where H is the problem horizon, andSt denotes the set of states at time t. For example, for a trajectory s0 , s2 , . . . , sH−1 starting from t = 0, we have st ∈ St ,for t ∈ [0, H − 1]. Therefore, we can use the previous notation to model time-varying functions naturally (needed in thefinite horizon), without explicitly listing the time dependency. In this section, with abuse of notation, we define the value
Vrπ of a policy π to reward r at s ∈ Sτ as
"H−1
#
X
π
Vr (s) := Eπ,Pr̃(st , at ) | sτ = s
t=τ35

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

MDP and CMDP Problems

The task MDP becomes MH = (S, A, r, P, H) and solving it means"H−1#
X
max Eπ,Pr(st , at ) | s0 ∼ d0
πt=0

and the CMDP problem in (1) becomes"H−1#
X
max Eπ,Pr̃(st , at ) | s0 ∼ d0
π

s.t.

Eπ,P

"H−1
X

t=0

t=0

#
cµ (st , at ) | s0 ∼ d0 ≤ 0D.1. Main Theorem for Finite-horizon Problems
Assumption D.2 (Data Assumption (Finite Horizon)).
1. For all r̃ ∈ R̃, r̃(s, a) ∈ [−1, 1] for all s ∈ S and a ∈ A.
2. The data distribution µ is 1ϵ -positively biased with respect to R̃.
π † (s,a)†3. The solution to the CDMP π † satisfies sups,a dµ(s,a) < ∞, where we note that dπ is defined without discounts but as
the average over the episode length.
Theorem D.3 (Main Result (Finite Horizon)). Under the data assumption in Assumption C.6, consider an offline RL
algorithm Algo that is sufficiently pessimistic to be ( Kδ + 2)-admissible with respect to R̃. Then for any r̃ ∈ R̃, with highprobability, the policy π̂ learned by Algo from the dataset D̃ := {(s, a, r̃, s′ )|(s, a) ∼ µ, r̃ = r̃(s, a), s′ ∼ P (·|s, a)} hasboth performance guarantee∗

Vrπ (d0 ) − Vrπ̂ (d0 ) ≤ ϵ + O(ι)
and safety guarantee
Prob (∃τ ∈ [0, H − 1], sτ ∈/ supp(µ)|π̂) ≤ ι
where ι := δ + κ(δ) + o(1) and o(1) denotes a term that vanishes as the dataset size becomes infinite.
The main difference between the infinite-horizon setup and the finite-horizon setup is that the finite-horizon setup always
satisfies the regularity assumption (the third point) in Assumption C.6. In addition, the safety guarantee is more explicit
compared the discounted infinite-horizon counterpart. This is because Prob (∃τ ∈ [0, H − 1], sτ ∈/ supp(µ)|π̂) ≤ Vcπ̂µ (d0 )for the finite horizon, whereas we need an additional conversion in Appendix C.5.
We prove limδ→0+ κ(δ) = 0 is always true for the finite horizon version.
Proposition D.4. For the H-horizon constrained MDP of (1), we have an optimal dual variable λ† = 2H + 1 andκ(δ) ≤ min{2H, (2H + 1)δ}.
Proof. Define the Lagrange reward r† (s, a) = r̃(s, a) − λ† cµ (s, a). Let π̂ † denote the optimal policy to the Lagrange
reward. Notice that Vcπµ (d0 ) ≥ 1 for any π such that Vcπµ (d0 ) > 0. Therefore, with λ† > 2H,
max

π:Vcπµ (d0 )>0Vr̃π (d0 ) − λ† Vcπµ (d0 ) ≤max

π:Vcπµ (d0 )>0

< −H
≤

max

π:Vcπµ (d0 )=0Vr̃π (d0 ) − λ†Vr̃π (d0 ) − λ† Vcπµ (d0 )†Therefore, π † satisfies Vcπµ (d0 ) = 0. This implies†

max Vr̃π (d0 ) − λ† Vcπµ (d0 ) ≥ min max Vr̃π (d0 ) − λVcπµ (d0 ) = Vr̃π (d0 )
π

λ≥0

π

36

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

On the other hand, it is always true thatmax Vr̃π (d0 ) − λ† Vcπµ (d0 ) ≥ min max Vr̃π (d0 ) − λVcπµ (d0 )
π

λ≥0

π

Therefore,λ† ∈ min max Vr̃π (d0 ) − λVcπµ (d0 )λ≥0πConsider λ† = 2H + 1. By Lemma C.8, we have κ(δ) ≤ (2H + 1)δ. In addition κ(δ) ≤ 2H by definition.
D.2. Length Bias
For finite-horizon problems, we provide a simple sufficient condition for length bias, which assumes long trajectories
obtained in data collections are near optimal.
Proposition D.5. Suppose the data are generated by rolling out policies starting from t = 0 and the initials state distribution
d0 . Let ξ = (s0 , a0 , s1 , . . . , sTξ −1 , aTξ −1 ) denote a trajectory of length Tξ . If with probability one (over randomness of ξ)that Tξ = H implies
H−1
X
t=0

∗

r(st , at ) ≥ V π (d0 ) − Hϵ′1then the data distribution is Hϵ′ -positively biased.

Proof. Consider some π ∈ Π† (δ). We can derive
Vrπ∗ (d0 ) − Vrπ (d0 ) = Vrπ∗ (d0 ) − Eπ,P

"H−1
X

"
= Eπ,P

Vrπ∗ (d0 ) −r(st , at )

t=0
H−1
X
t=0

"
+ Eπ,P#

Vrπ∗ (d0 ) −#r(st , at ) | ∃τ ∈ [0, H − 1], sτ ∈/ supp(µ)

H−1
X
t=0

#
r(st , at ) | ∀τ ∈ [0, H − 1], sτ ∈ supp(µ)≤ H × Prob (∃τ ∈ [0, H − 1], sτ ∈/ supp(µ)|π) + Hϵ′≤ H × Vcπµ (d0 ) + Hϵ′≤ Hδ + Hϵ′

D.3. Goal-oriented Problems
Finally we make a remark on positive data bias in the goal-oriented finite-horizon setting. In the infinite-horizon settingthat a goal is marked as an absorbing state, which means that the agent once entering will stay there forever. The exact
instantaneous (wrong) reward obtained at this absorbing state is not relevant (it can be anything in [−1, 1]) since theadmissibility condition has already accounted for the range of the associated Lagrange reward. Namely, the agent is set
1
pessimistic such that it views partial transitions/trajectories obtaining a worse return than − 1−γ, which is a lower bound of
the return the absorbing goal state.
To make sure the finite-horizon setting has the same kind of positive data bias, one way is to virtually extend the problem’s
original horizon (say H) by (as least) one and let the goal state to be the only state where the agent can stay until the lasttime step13 H (i.e., all the other trajectories continue maximally up to time step H − 1 and therefore have a length at most13

We use zero-based numbering here where 0 is the initial step and H is the last step for a problem with horizon H + 1.

37

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

H). Then we apply the offline RL algorithm to this extended problem (e.g., of horizon H + 1). We would need to set thehorizon in the previous theoretical results accordingly for this longer horizon.
The reason for this extension is to ensure that the effect of absorbing state in the infinite-horizon setting (which ensures
trajectories going into the absorbing state is by definition the longest) can carry over to the finite horizon case. If we apply
the agent directly to solve the H step problem without such an extension, there may be other trajectories which can be as
long as a goal-reaching one. As a result, there is no positive data bias.
An alternate way to create the positive data bias in the finite horizon setting is to truncate trajectories that do not reach goalat time step H − 1 to be no longer than H − 1. That is, they now time out at time step H − 2, whereas only goal-reaching
trajectories can continue up to time step H − 1.

E. Algorithm Specific Results
In this appendix, we provide a few examples of admissible offline RL algorithms. We choose algorithms to cover bothmodel-free (Cheng et al., 2022; Xie et al., 2021; Rashidinejad et al., 2021; Liu et al., 2020) and model-based (Xie et al.,2022; Bhardwaj et al., 2023; Uehara and Sun, 2022; Yu et al., 2020b; Kidambi et al., 2020) approaches. The pessimism inthese algorithms are constructed through different ways, including adversarial training (Cheng et al., 2022; Xie et al., 2021;
2022; Bhardwaj et al., 2023; Uehara and Sun, 2022), value bonus (Rashidinejad et al., 2021; Yu et al., 2020b) and action
truncation (Liu et al., 2020; Kidambi et al., 2020). These algorithms are listed in Table 15.

Model-free
Model-based

Adversarial Training
ATAC (Cheng et al., 2022) (Appendix E.1)
PSPI (Xie et al., 2021) (Appendix E.2)
ARMOR (Xie et al., 2022; Bhardwaj et al., 2023) (Appendix E.3)
CPPO (Uehara and Sun, 2022) (Appendix E.4)

Value Bonus

Action Truncation

VI-LCB (Rashidinejad et al., 2021) (Appendix E.5)

PPI/PQI (Liu et al., 2020) (Appendix E.6)

MOPO (Yu et al., 2020b) (Appendix E.7)

MOReL (Kidambi et al., 2020) (Appendix E.7)

Table 15: We show that all of the offline RL algorithms above are admissible. Note that this is not a complete list of alladmissible offline RL algorithms.
We recall the definition of admissibility below.
Definition E.1. [Admissibility] For R ≥ 0, we say an offline RL algorithm Algo is R-admissible with respect to R̃, if for
any r̃ ∈ R̃, given D̃, Algo learns a policy π̂ satisfying the following with high probability: for any policy π ∈ Π,
max Vr̄π (d0 ) − Vr̄π̂ (d0 ) ≤ E(π, µ),r̄∈RR (r̃)

where we define a data-consistent reward class
RR (r̃) := {r̄ : r̄(s, a) = r̃(s, a), ∀(s, a) ∈ supp(µ) and r̄(s, a) ∈ [−R, 1] , ∀(s, a) ∈ S × A} ,π(s,a)
E(π, µ) is some regret upper bound such that E(π, µ) = o(1) if sups∈S,a∈A dµ(s,a)< ∞, and o(1) denotes a term thatvanishes as the dataset size becomes infinite.

For the sake of clarity, we consider the tabular setting in our analysis. That is, we assume the state space S and the action
space A are countable and finite. (We use |S| and |A| to denote their cardinalities). To establish admissibility, we sometimes
make minor changes over the function classes in these algorithms, because some algorithms assume that the reward is knownor has value bounded by [0, 1]. We highlight these changes in blue. We would like to clarify that the goal of our analysis
is to prove that these algorithms are admissible rather than provide a tight bound for their performance. We recommend
readers who are interested in performance bound to read the original papers, as their bound can be tighter than ours.
One remarkable technical details in our analysis is that there is no need of using an union bound over all rewards in RR (r̃)
in the admissibility definition when proving these high probability statements. Instead we found that we can reuse the boundproved for a single reward directly. The main reason is that all rewards in the admissibility definition agree with each otheron the support of the data distribution, and the statistical analysis of concentration only happens within this support. Outsideof the support, a uniform bound based on the reward range can be used to bound the error. This will be made more clearlylater in the derivations.
38Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

E.1. ATAC
We first show that ATAC (Cheng et al., 2022) is an admissible offline RL algorithm. We first introduce notations that we will
use for analyzing ATAC, which will also be useful for studying PSPI in Appendix E.2.
E.1.1. N OTATIONS
For any µ, ν ∈ ∆(S × A), we denote
P (µ \ ν)(s, a) := max(µ(s, a) − ν(s, a), 0). For any µ ∈ ∆(S × A) and any f :S × A → R, we define ⟨µ, f ⟩ := (s,a)∈S×A µ(s, a)f (s, a). For an MDP with transition probability P : S × A → ∆(S),
we define (P π f )(s, a) := γEs′ ∼P (·|s,a) f (s′ , π) for any f : S × A → R and π : S × ∆(A).
Given function class F ⊆ (S × A → R), policy π : S → ∆(A) and reward r̄ : S × A → R, we introduce Bellman error
transfer coefficient (Xie et al., 2021; Cheng et al., 2022) to measure the distribution shift between two probability measure µand ν.
Definition E.2 (Bellman error transfer coefficient). The Bellman error transfer coefficient between ν and µ under function
class F, policy π and reward r̄ is defined as,∥f − r̄ − P π f ∥22,ν
.
f ∈F ∥f − r̄ − P π f ∥22,µC (ν; µ, F, π, r̄) := max(3)

We note that the Bellman error transfer coefficient is a weaker notion than the density ratio, which is established in thelemma below.
Lemma E.3. For any distributions ν, µ ∈ ∆(S × A), function class F ⊆ (S × A → R), policy π : S → ∆(A) and rewardr̄ : S × A → R, we have!
ν(s, a).
(4)
C (ν; µ, F, π, r̄) ≤sup(s,a)∈S×A µ(s, a)Proof.
∥f − r̄ − P π f ∥22,ν =X(s,a)∈S×A

ν(s, a) f (s, a) − r̄(s, a) − P π f (s, a)22ν(s, a)f (s, a) − r̄(s, a) − P π f (s, a)µ(s, a)(s,a)∈S×A!
X
2
ν(s, a)f (s, a) − r̄(s, a) − P π f (s, a)≤µ(s, a)sup(s,a)∈S×A µ(s, a)(s,a)∈S×A!
X
2
ν(s, a)=supµ(s, a) f (s, a) − r̄(s, a) − P π f (s, a)(s,a)∈S×A µ(s, a)(s,a)∈S×A!
ν(s, a)=sup∥f − r̄ − P π f ∥22,µ(s,a)∈S×A µ(s, a)=Xµ(s, a)Therefore,∥f − r̄ − P π f ∥22,ν
f ∈F ∥f − r̄ − P π f ∥22,µ!
∥f − r̄ − P π f ∥22,µν(s, a)≤supmax
f ∈F ∥f − r̄ − P π f ∥2(s,a)∈S×A µ(s, a)2,µC (ν; µ, F, π, r̄) = max=

supν(s, a)(s,a)∈S×A µ(s, a)39

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

E.1.2. A NALYSIS
ATAC considers a critic function class F and a policy class Π. Since we consider the tabular setting, we choose F :1(S × A → [−Vmax , Vmax ])14 with 1−γ≤ Vmax < ∞ and Π : (S → ∆(A)). ATAC formulates offline RL as the following
Stackelberge game,π̂ ∈ arg max LD̃ (π, f π )

(5)

π∈Π

s.t.

f π ∈ arg min LD̃ (π, f ) + βED̃ (π, f ),f ∈Fwith β ≥ 0 being hyperparameter, andLD̃ (π, f ) := ED̃ f (s, π) − f (s, a) ,

2 

2 
ED̃ (π) := ED̃ f (s, a) − r̃ − γf (s′ , π)
− minED̃ f ′ (s, a) − r̃ − γf (s′ , π) .
′

(6)

f ∈F

For any policy π, the critic f π provides a relative pessimistic (with respect to the behavior policy µ) value estimate of thepolicy π. The hyperparameter β balances pessimism, given by LD̃ (π, f ), and Bellman consistency, given by ED̃ (π). The
learned policy π̂ maximizes the relative pessimistic value estimate given by the critic.
We state the performance guarantee of ATAC with respect to any comparator policy π under the set of data-consistent
reward functions {r̄ : r̄(s, a) = r̃(s, a), ∀(s, a) ∈ supp(µ) and |r̄(s, a)| ≤ (1 − γ)Vmax , ∀(s, a) ∈ S × A} in the following
proposition. We make an additional assumption on the data distribution for ATAC, which is needed in its original proof.
Assumption E.4. For ATAC, we assume µ is the (mixture) average of state-action visitation distribution for d0 .
1
≤ Vmax < ∞ and
Proposition E.5. Fix some r̃ : S × A → [−1, 1]. Consider F : (S × A) → [−Vmax , Vmax ] with 1−γ
Π : (S → ∆(A)). Let π̂ be the solution to (5) and let π ∈ Π be any comparator policy. Let ν ∈!∆(s, a) be an arbitraryr
C|D̃|21 3
2 , with probability 1 − δ,distribution. Under Assumption E.4, for any δ ∈ (0, 1], choosing β = Θ̃ Vmax
|S||A| log(1/δ)

it holds thatVr̄π (d0 ) − Vr̄π̂ (d0 ) ≤Õ

Vmax C (ν; µ, F, π, r̄)|S||A| log(1/δ)(1 − γ)|D̃|1/3

1/3 !
+⟨dπ \ ν, r̄ + P π f π − f π ⟩
1−γ

(7)

for all r̄ such that r̄(s, a) = r̃(s, a), ∀(s, a) ∈ supp(µ) and |r̄(s, a)| ≤ (1 − γ)Vmax , ∀(s, a) ∈ S × A.
Proof. We first show that F : (S × A → [−Vmax , Vmax ]) is both realizable and Bellman complete with repect to r̄. For any
r̄ such that |r̄(s, a)| ≤ (1 − γ)Vmax , we have Qπr̄ (s, a) ∈ [−Vmax , Vmax ]. This means that Qπr̄ ∈ F for any π ∈ Π, s ∈ S,a ∈ A. Moreover, for any f ∈ F, |r̄(s, a) + P π f (s, a)| ≤ |r̄(s, a)| + |P π f (s, a)| ≤ (1 − γ)Vmax + γVmax = Vmax . In otherwords, (r̄(s, a) + P π f (s, a)) ∈ F for any f ∈ F.
By construction of r̄, we have D̄ := {(s, a, r̄, s′ )|(s, a, s′ ) ∈ D, r̄ = r̄(s, a)} ≡ D̃. That is, solving the Stackelberg
game (5) given by D̃ is equivalent to solving the game given by D̄. The rest of the proof follows from Theorem C.12 in (Liet al., 2023a) by choosing the reward class to only contain the reward r̄, i.e., G = {r̄} (which implies dG = 0) and using
dF ,Π = Õ(|S||A| log(1/δ)).

Note that the above derivation does not need an additional union bound to cover all rewards in {r̄ : r̄(s, a) =r̃(s, a), ∀(s, a) ∈ supp(µ) and |r̄(s, a)| ≤ (1 − γ)Vmax , ∀(s, a) ∈ S × A}. This is because the concentration analysis
is only taken on the support of the data distribution, where all rewards in this reward class agree, and a uniform bound basedon Vmax is used for out of support places, which again applies to all the rewards in this reward class.

We then show that ATAC (Cheng et al., 2022) is admissible based on Proposition E.5.
14

The original theoretical statement of ATAC assumes F to contain only non-negative functions. However, the analysis can be extended
to any F with bounded value, such as [−Vmax , Vmax ]. We note that the practical implementation of ATAC using function approximators
does not make assumption on F being non-negative.

40

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in DataR
Corollary E.6 (ATAC is admissible). For any Vmax ≥ 1−γ, ATAC is R-admissible with respect to R̃ for any R̃ ⊆ (S × A →[−1, 1]).

Proof. Given any r̃ ∈ R̃, we want to show that, with high probability, Vr̄π (d0 ) − Vr̄π̂ (d0 ) = o(1) for all r̄ ∈ RR (r̃)
π
π
(s,a)
(s,a)
< ∞. For any such π, define C∞ := sup(s,a)∈S×A dµ(s,a)
and all policy π such that sup(s,a)∈S×A dµ(s,a), we haveπC (ν; µ, F, π, r̄) ≤ C∞ for any r̄. By taking ν = d in Proposition E.5, we have, with probability 1 − δ,Vr̄π (d0 ) − Vr̄π̂ (d0 ) ≤ Õ= ÕVmax C∞ |S||A| log(1/δ)

1/3 !

Vmax C∞ |S||A| log(1/δ)

1/3 !

(1 − γ)|D̃|1/3+(1 − γ)|D̃|1/3⟨dπ \ dπ , r̄ + P π f π − f π ⟩1−γ= o(1)

R
for all r̄ such that r̄(s, a) = r̃(s, a), ∀(s, a) ∈ supp(µ) and |r̄(s, a)| ≤ (1 − γ)Vmax , ∀(s, a) ∈ S × A. Since Vmax ≥ 1−γ,
we have that Vr̄π (d0 ) − Vr̄π̂ (d0 ) = o(1) for all r̄ ∈ RR (r̃).

E.2. PSPI
We show that PSPI (Xie et al., 2021) is also admissible. PSPI (Xie et al., 2021) is similar to ATAC. Given critic function
1≤ Vmax < ∞ and policy class Π : (S → ∆(A)), PSPI solves the followingclass F : (S × A → [−Vmax , Vmax ])15 with 1−γ
Stackelburg game,π̂ ∈ arg max (1 − γ)f π (d0 , π)

(8)

π∈Π

s.t.

f π ∈ arg min (1 − γ)f (d0 , π) + βED̃ (π, f ),f ∈Fwith β ≥ 0 being hyperparameter, and

2 
 ′
2 
′
ED̃ (π) := ED̃ f (s, a) − r̃ − γf (s′ , π)
− min
E
f
(s,
a)
−
r̃
−
γf
(s
,
π)
.
D̃
′
f ∈F

In PSPI, the critic f π provides an absolute pessimistic value estimate of policy π. The hyperparameter β trades off
pessimism and Bellman-consistency. The learned policy π̂ maximizes such a pessimistic value. We state the performance
guarantee of the learned policy π̂ with respect to any comparator policy π under the set of data-consistent reward functions{r̄ : r̄(s, a) = r̃(s, a), ∀(s, a) ∈ supp(µ) and |r̄(s, a)| ≤ (1 − γ)Vmax , ∀(s, a) ∈ S × A} in the following proposition.
Proposition E.7 ((Xie et al., 2021; Li et al., 2023a)). Let π̂ be the solution to (5) and let π ∈ Π be any comparator!
r
C|D̃|21 3
2 , with
policy. Let ν ∈ ∆(s, a) be an arbitrary distribution. For any δ ∈ (0, 1], choosing β = Θ̃ Vmax
|S||A| log(1/δ)

probability 1 − δ,Vr̄π (d0 ) − Vr̄π̂ (d0 ) ≤Õ

Vmax C (ν; µ, F, π, r̄)|S||A| log(1/δ)(1 − γ)|D̃|1/3

1/3 !
+⟨dπ \ ν, r̄ + P π f π − f π ⟩
1−γ

(9)

for all r̄ such that r̄(s, a) = r̃(s, a), ∀(s, a) ∈ supp(µ) and |r̄(s, a)| ≤ (1 − γ)Vmax , ∀(s, a) ∈ S × A.
Proof. The proof is similar to that of Proposition E.5. First, observe that F : (S × A → [−Vmax , Vmax ]) is both realizable
and Bellman complete with respect to r̄. The rest of the proof follows from Theorem D.1 in (Li et al., 2023a) (taking
G = {r̄}) and Lemma E.3.
15
Similar to ATAC, we extend the critic function class to contain functions that can take negative values. The theoretical analysis can begeneralized to this case. The practical implementation of PSPI can directly handle critics which take negative values.

41Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

The proposition above implies that PSPI (Xie et al., 2021) is admissible. We omit the proof of the corollary below as it is thesame as the proof of Corollary E.6.
R
Corollary E.8 (PSPI is admissible). For any Vmax ≥ 1−γ, PSPI is R-admissible with respect to R̃ for any R̃ ⊆ (S × A →[−1, 1]).

E.3. ARMOR
We show that ARMOR (Xie et al., 2022; Bhardwaj et al., 2023) is admissible. ARMOR is a model-based offline RL
algorithm. We denote a model as M = (S, A, PM , rM , γ), where PM : S × A → ∆(S) is the model dynamics, andrM : S × A → [−Rmax , Rmax ]16 is the reward function with 1 ≤ Rmax < ∞. Since we consider the tabular setting,
we use a model class that contains all possible dynamics and all reward functions bounded within [−Rmax , Rmax ], i.e.,
Mmodel = {M : PM ∈ (S × A → ∆(S)), rM ∈ (S × A → [−Rmax , Rmax ])}. For any reference policy πref : S → ∆(A),
ARMOR solves the following two-player game,πref
ππ̂ = arg max minα VM(d0 ) − VM
(d0 )
π∈Π

M ∈M

(10)

D̃

P∞
π
where VM
(d0 ) := Eπ,PM [ t=0 γ t rM (st , at )|s0 ∼ d0 ], andED̃ (M ′ ) ≤ α},(11)− log PM (s′ |s, a) + (rM (s, a) − r̃)2 .

(12):= {M ∈ Mmodel : ED̃ (M ) −
Mα
D̃
with
ED̃ (M ) :=X
(s,a,r̃,s′ )∈D̃min

M ′ ∈Mmodel

Intuitively, ARMOR constructs a relative pessimistic model (with respect to πref ) M which is also approximately consistent
with data. The learned policy π̂ maximizes the value estimate given by the pessimistic model. We define the generalized
single policy concentrability to characterize the distribution shift from µ to the state-action visitation dπ of any policy π.
Definition E.9 (Generalized Single-policy Concentrability (Bhardwaj et al., 2023)). We define the generalized single-policy
concentration for policy π, model class Mmodel , reward r̄ and data distribution µ as
Cmodel (π; Mmodel , r̄) :=Edπ [E(M ; r̄)]M ∈Mmodel Eµ [E(M ; r̄)]supwith E(M ; r̄) = DT V (PM (·|s, a), P (·|s, a))2 + (rM (s, a) − r̄(s, a))2 .
The single-policy concentrability Cmodel (π) can be considered as a model-based version of the Bellman error transfer coefficient in Definition E.2. Following the same steps as the proof of Lemma E.3, it can be shown that Cmodel (π; Mmodel , r̄) ≤π(s,a)
sup(s,a)∈S×A dµ(s,a). We provide the high-probability performance guarantee for the learned policy π̂. This implies that
ARMOR is an admissible algorithm with a sufficiently large Rmax .
Proposition E.10. For any δ ∈ (0, 1], there exists an absolute constant c such that when choosing α = c|S|2 |A| log(1/δ),for any comparator policy π ∈ Π, with probability 1 − δ, the policy π̂ learned by ARMOR (10) satisfies
s
!
p
Rmax
|S|2 |A| log(1/δ)ππ̂
Vr̄ (d0 ) − Vr̄ (d0 ) ≤ O
Cmodel (π; Mmodel , r̄),(13)(1 − γ)2|D̃|
for all r̄ such that r̄(s, a) = r̃(s, a), ∀(s, a) ∈ supp(µ) and |r̄(s, a)| ≤ Rmax , ∀(s, a) ∈ S × A.
Proof. By construction, we have M̄ = (S, A, P, r̄, γ) ∈ Mmodel . Since r̄(s, a) = r̃(s, a) for any (s, a) ∈ supp(µ), we
have Mα≡ Mα. The rest of the proof follows mostly from Theorem 2 in (Bhardwaj et al., 2023) by taking πref = µ. ByD̄D̃
a similar argument as in the proof of Proposition E.5, we do not need an additional union bound to cover all rewards in{r̄ : r̄(s, a) = r̃(s, a), ∀(s, a) ∈ supp(µ) and |r̄(s, a)| ≤ Rmax , ∀(s, a) ∈ S × A}.
16

The theoretical statement of ARMOR assumes the value of rM is bounded by [0, 1]. The original analysis can be extended as long as
the value of rM is bounded by a finite value Rmax < ∞. The practical implementation of ARMOR does not assume the reward only takes
value in [0, 1].

42

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

Corollary E.11 (ARMOR is admissible). For any Rmax ≥ R, ARMOR is R-admissible with respect to R̃ for any
R̃ ⊆ (S × A → [−1, 1]).
Proof. Given any r̃ ∈ R̃, we want to show that, with high probability, Vr̄π (d0 ) − Vr̄π̂ (d0 ) = o(1) for all r̄ ∈ RR (r̃) and all
π
π
(s,a)
(s,a)
:= C∞ <
policy π such that sup(s,a)∈S×A dµ(s,a)< ∞. For any such π, Cmodel (π; Mmodel , r̄) ≤ sup(s,a)∈S×A dµ(s,a)∞. By Proposition E.10, with probability 1 − δ, we haveVr̄π (d0 ) − Vr̄π̂ (d0 ) ≤ O

p

Rmax
C∞
(1 − γ)2s|S|2 |A| log(1/δ)|D̃|!
= o(1).

(14)for all r̄ such that r̄(s, a) = r̃(s, a), ∀(s, a) ∈ supp(µ) and |r̄(s, a)| ≤ R, ∀(s, a) ∈ S × A. Therefore, Vr̄π (d0 ) − Vr̄π̂ (d0 ) =o(1) for all r̄ ∈ RR (r̃) with high probability.
E.4. CPPOCPPO(Uehara and Sun, 2022) is another admissible model-based offline algorithm. Again, we denote a model as M =(S, A, PM , rM , γ), where PM : S × A → ∆(S) is the model dynamics, and rM : S × A → [−Rmax , Rmax ]17 is the reward
function with 1 ≤ Rmax < ∞. CPPO solves the two-player game,ππ̂ = arg max minα VM
(d0 )
π∈Π

(15)

M ∈M

D̃

P∞
π
with VM(d0 ) := Eπ,PM [ t=0 γ t rM (st , at )|s0 ∼ d0 ] and Mα
defined in (11). CPPO constructs a pessimistic model M
D̃
which is also approximately consistent with data. The learned policy π̂ maximizes the value estimate given by the pessimistic
model. We provide the high-probability performance guarantee for the learned policy π̂. This implies that CPPO is anadmissible algorithm with sufficiently large Rmax .
Proposition E.12. For any δ ∈ (0, 1], there exists an absolute constant c such that when choosing α = c|S|2 |A| log(1/δ),for any comparator policy π ∈ Π, with probability 1 − δ, the policy π̂ learned by CPPO (15) satisfies
Vr̄π (d0 ) − Vr̄π̂ (d0 ) ≤ O

pRmax
Cmodel (π; Mmodel , r̄)(1 − γ)2s|S|2 |A| log(1/δ)
|D̃|

!
,

(16)

for all r̄ such that r̄(s, a) = r̃(s, a), ∀(s, a) ∈ supp(µ) and |r̄(s, a)| ≤ Rmax , ∀(s, a) ∈ S × A.
Proof. By construction, we have M̄ = (S, A, P, r̄, γ) ∈ Mmodel . Since r̄(s, a) = r̃(s, a) for any (s, a) ∈ supp(µ), we
have Mα≡ Mα. The rest of the proof follows similarly from the proof of Theorem 2 in (Bhardwaj et al., 2023). By
D̄D̃
Lemma 5 from (Bhardwaj et al., 2023), with high probability, M̄ ∈ Mα
. Therefore,
D̄

π
π̂
π
π̂
Vr̄π (d0 ) − Vr̄π̂ (d0 ) = VM̄
(d0 ) − VM̄(d0 ) ≤ VM̄(d0 ) − minα VM(d0 )M ∈MD̄

ππ≤ VM̄(d0 ) − minα (VM(d0 ))M ∈MD̄ππ≤ maxα |VM̄(d0 ) − VM(d0 )|M ∈MD̄

where the second inequality follows from the optimality of π̂. By (20) from (Bhardwaj et al., 2023), for any M ∈ Mα
,
D̄
p

q 2
|S| |A| log(1/δ)
Rmaxππ|VM̄ (d0 ) − VM (d0 )| ≤ O
Cmodel (π; Mmodel , r̄) (1−γ)2. Following a similar argument to the proof|D̃|of Proposition E.5, there is no need to take the union bound with respect to r̄, which concludes the proof.
We omit the proof for the following corollary as it is the same of the proof of Corollary E.11.
Corollary E.13 (CPPO is admissible). For any Rmax ≥ R, CPPO is R-admissible with respect to R̃ for any R̃ ⊆ (S × A →[−1, 1]).
43

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

Algorithm 1 VI-LCB (Rashidinejad et al., 2021)
1: Input: Batch dataset D̃, discount factor γ, confidence level δ, and value range Vmax .
N
2: Set J := log
1−γ .
N
3: Randomly split D̃ into J + 1 sets D̃t = {(si , ai , ri , s′i )}m
i=1 for j ∈ {0, 1, · · · , J} with m = J+1 .

4: Set m0 (s, a) := i=1 1{(si , ai ) = (s, a)} based on dataset D̃0 .
5: For all a ∈ A and s ∈ S, initialize Q0 (s, a) = −Vmax , V0 (s) = −Vmax and set π0 (s) = arg maxa m0 (s, a).
6: for j = 1, · · · , J doj
7:
Initialize rt (s, a) = 0 and set Ps,ato be a random probability vector.

Pm

8:
9:

Pm
Set mt (s, a) := i=1 1{(si , ai ) = (s, a)} based on dataset D̃t .
Compute penalty bt (s, a) for L = 2000 log(2(J + 1) |S| |A| /δ)sbt (s, a) := 2VmaxL.
max(mt (s, a), 1)

10:for (s, a) ∈ (S, A) do
11:
if mt (s, a) ≥ 1 thenj
12:
Set Ps,ato be empirical transitions and rt (s, a) be empirical average of rewards.
13:
end ifjVj−1 .
14:
Set Qj (s, a) ← rt (s, a) − bt (s, a) + γPs,a
15:
end for16:
Compute Vtmid (s) ← maxa Qt (s, a) and πtmid (s) ∈ arg maxa Qt (s, a).
17:
for s ∈ S do
18:
if Vtmid (s) ≤ Vj−1 (s) then19:
Vt (s) ← Vj−1 (s) and πt (s) ← πj−1 (s).
20:
else
21:
Vt (s) ← Vtmid (s) and πt (s) ← πtmid (s).
22:
end if23:
end for
24: end for
25: Return π̂ = πJ

E.5. VI-LCB
We show VI-LCB (Rashidinejad et al., 2021) is admissible. VI-LCB is a pessimistic version of value iteration. It adds
negative bonuses to the Bellman backup step in order to underestimate the value when there are missing data. By being
pessimistic in value estimation, it can overcome the issue of µ being non-exploratory. We recap the VI-LCB algorithm in
Algorithm 1. We highlight the changes we make in blue. These changes are due to that originally the authors in (Rashidinejad
et al., 2021) assume the rewards are non-negative, but the rewards in the admissibility definition Definition C.4 can takenegative values. Therefore, we make these changes accordingly. We state and prove the guarantee below.18
R
Proposition E.14. For Vmax ≥ 1−γ, VI-LCB is R-admissible with respect to any R̃ ⊆ (S × A → [−1, 1]).

Proof. First, we introduce a factor of 2 in their proof of Lemma 1 (Rashidinejad et al., 2021) to accomodate that the
rewards here can be negative. This is reflected in the updated bonus definition in Algorithm 1. This updated Lemma 1of (Rashidinejad et al., 2021) now captures the good event that the bonus can upper bound the error of the empirical estimateof Bellman backup. It shows this good event is true with high probability.
We remark that while originally Lemma 1 of (Rashidinejad et al., 2021) is proved for a single reward function, it actuallyholds for simultaneously for all the reward functions in RR (r̃), without the need of introducing additionally a union bound.
The reason is that in the proof of Lemma 1 in (Rashidinejad et al., 2021), the concentration is used only for the estimating
17

The original algorithm assumes that the reward function is known. Here we consider the generalization of the algorithm with unknown
reward.
18
We referenced to lemmas and equations based on the version arXiv:2103.12021.

44

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

Algorithm 2 Pessimistic Policy Iteration (PPI) (Liu et al., 2020)
1: Input: Batch dataset D̃, function class F, probability estimator µ̂, hyperparameter b.
2: for i ∈ {0, . . . , I − 1} do
3:for j ∈ {0, . . . , J} do
4:fi,j+1 ← arg minf ∈F LD̃ (f, fi,j , π̂i )5:
end for6:π̂i+1 ← arg maxπ ED̃ [Eπ [ζ ◦ fi,J+1 ]7: end for
8: Return π̂ = π̂I

Algorithm 3 Pessimistic Q Iteration (PQI) (Liu et al., 2020)
1: Input: Batch dataset D̃, function class F, probability estimator µ̂, hyperparameter b.
2: for i ∈ {0, . . . , I − 1} do
3:fi+1 ← arg minf ∈F LD̃ (f, fi )4:π̂i+1 (s) ← arg maxa ζ ◦ fi+1 (s, a)
5: end for
6: Return π̂ = π̂I

the Bellman on the data support. This bound would apply to all rewards in RR (r̃) since they agree exactly on the data. For
state-action pairs out of the support, the proof takes a uniform bound based on the size of Vmax , which also applies to all
rewards in RR (r̃).
Therefore, we can apply the updated Lemma 1 of (Rashidinejad et al., 2021) to prove the desired high probability boundneeded in the admissibility condition. Under this good event, we can use the upper bound in (54b) in (Rashidinejad et al.,π(s,a)2021) to bound the regret. Consider some π such that C := sups,a dµ(s,a) < ∞. Suppose D̃ has N transitions. Take someR
Vmax ≥ 1−γ. Then running VI-LCB with J iterations ensures19 for any r̄ ∈ RR (r̃),
64VmaxVr̄π (d0 ) − Vr̄π̂ (d0 ) ≤ γ J 2Vmax +1−γr

L|S||A|C(J + 1)
NN
where we mark the changes due to using rewards which can takes negative values in blue. Setting J = log
1−γ as in(Rashidinejad et al., 2021), with probability 1 − δ, it holds that!
r
V
L|S||A|CJ
max
Vr̄π (d0 ) − Vr̄π̂ (d0 ) ≤ Õ γ J Vmax +
1−γ
N


s
|S||A|C log 1δ= Õ Vmax
N (1 − γ)3

Thus, VI-LCB is R-admissible.
E.6. PPI and PQI
We show PPI and PQI proposed in (Liu et al., 2020) are admissible. We present their algorithms in Algorithm 2 andAlgorithm 3, with (ζ ◦ f )(s, a) := ζ(s, a)f (s, a) for any ζ, f : S × A → R. These algorithms use a probability estimatorof the data distribution (denoted as µ̂), which in the tabular case is the empirical distribution. Based on µ̂, they define a filter
functionζ(s, a; µ̂, b) = 1[µ̂(s, a) ≥ b]
This filter function classifies whether a state-action pair is in the support, and it is used to modify the Bellman operators indynamics programming as shown in line 6 of Algorithm 2 and line 4 of Algorithm 3. As a result, the Bellman backup and
19

We add back an extra dependency on |A| as their original proof assumes π is deterinistic, which is not necessarily the case here.

45

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Datathe policy optimization only consider in-support actions, which mitigates the issue of learning with non-exploratory µ. The
loss functions (i.e., LD̃ ) in Algorithm 2 and Algorithm 3 denote the squared Bellman error (with a target network) as used in
fitted Q iteration (Riedmiller, 2005). We omit the details here.
In summary, PPI and PQI follow the typical policy iteration and value iteration schemes, except that the backup and theargmax are only taken within the observed actions (or actions with sufficiently evidence to be in the support when using
function approximators). This idea is similar to the spirit of the later IQL algorithm (Kostrikov et al., 2021), except IQL
doesn not construct the filter function explicitly but relies instead on expectile maximization.
R
Proposition E.15. Suppose F = (S × A → [−Vmax , Vmax ])20 . For Vmax ≥ 1−γ, PPI/PQI is R-admissible with respect

any R̃ ⊆ (S × A → [−1, 1]).

π(s,a)

Proof. We prove for PPI; the proof for PQI follows similarly. Consider some π such that C := sups,a dµ(s,a) < ∞. SupposeRD̃ has N transitions. Take some Vmax ≥ 1−γ
and some r̄ from RR (r̃). We use the Corollary 1 in (Liu et al., 2020): With I
large enough, it holds with probability 1 − δ,
!!
r
π
sup
d
(s,
a)
V
|S||A|ln(1/δ)max
s,a,tt
Vr̄π (d0 ) − Vr̄π̂ (d0 ) ≤ Õ+ϵµ + (1 − γ)Eπ∗ ,P [1[µ(s, a) ≤ 2b]]
(1 − γ)3Nbwhere ϵµ = DT V (µ̂, µ) = O( √1N ). Further,(1 − γ)Eπ∗ ,P [1[µ(s, a) ≤ 2b]] =

X

=

X

s,a

dπ (s, a)1[µ(s, a) ≤ 2b]∗∗µ(s, a)s,adπ (s, a)1[µ(s, a) ≤ 2b]µ(s, a)≤ CEµ [1[µ(s, a) ≤ 2b]].
We can then upper bound the performance difference above as
!!
r
Vmax
|S||A| ln(1/δ) sups,a,t dπt (s, a) 1ππ̂
√ + CEµ [1[µ(s, a) ≤ 2b]]Vr̄ (d0 ) − Vr̄ (d0 ) ≤ Õ+(1 − γ)3NbN
We bound the latter two terms by tuning b. Notice Eµ [1[µ(s, a) ≤ 2b]] ≤ |S||A|2b at most. This implies
sups,a,t dπt (s, a) 1
√ + CEµ [1[µ(s, a) ≤ 2b]]bbNsups,a,t dπt (s, a) 1
√ + C|S||A|2b
≤ inf
b
b
N
!
p
C|S||A|
=O
N 1/4

inf

Thus,
Vr̄π (d0 ) − Vr̄π̂ (d0 ) ≤ ÕVmax
(1 − γ)3r|S||A| ln(1/δ)
+
N

!!
p
C|S||A|
N 1/4

We can apply this similar argument used in the previous proofs to show that this bound simultaneously applies to allr̄ ∈ RR (r̃) since the high-probability concentration analysis is only taken on the data distribution where all rewards inR
RR (r̃) are equal. Thus, when Vmax ≥ 1−γ, PPQ (and similarly PQI) is R-admissibile.
20

The original algorithms of PPI and PQI assumes F = (S × A → [0, Vmax ]). We note that our extension to [−Vmax , Vmax ] do notchange the performance bound as it absorbed by the Õ notation.

46

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

E.7. MOReL and MOPO
MOReL (Kidambi et al., 2020) and MOPO (Yu et al., 2020b) are two popular model-based offline RL algorithms. The twoalgorithms operate in a similar manner, except that MOReL (Kidambi et al., 2020) uses the truncation approach (similar
to PPI and PQI (Liu et al., 2020)) which classifies state-actions into known and unknown sets, whereas MOPO (Yu et al.,2020b) uses negative bonuses (similar to VI-LCB (Rashidinejad et al., 2021)). In this section, we show both algorithms areadmissible.
We denote a model as M = (S, A, PM , rM , γ), where PM : S × A → ∆(S) is the model dynamics, and rM : S × A →[−1, 1]21 with 1 ≤ Rmax < ∞ being the reward function. The model class is denoted as Mmodel .
E.7.1. MOR E L
We analyze a variant of MOReL which also learns the reward function, whereas MOReL in the originally paper (Kidambi
et al., 2020) assumes the reward function is given. We suppose model reward rM and model dynamics PM are learned by
⋆
⋆
rM
, PM
∈ arg min ED̃ (M )M ∈Mmodel

whereED̃ (M ) :=X
(s,a,r̃,s′ )∈D̃

− log PM (s′ |s, a) + (rM (s, a) − r̃)2 .

We define the set of known state-actions Kξ asonKξ = (s, a) :sup|rM1 (s, a) − rM2 (s, a)| + γVmax DT V (PM1 (·|s, a), PM2 (·|s, a)) ≤ ξ ,(17)

M1 ,M2 ∈Mα

D̃

with
:= {M ∈ Mmodel : ED̃ (M ) −
Mα
D̃

min

M ′ ∈MmodelED̃ (M ′ ) ≤ α},

We note that α can be chosen as Θ(|S|2 |A| log 1/δ) so that Mα contains model (S, A, P, r̃, γ) with probability 1 − δ forany δ ∈ (0, 1] (see (Bhardwaj et al., 2023) for details).
RemarkNote that we change the definition for the set of known state-actions. In (Kidambi et al., 2020), the set is given byKξ0 = {(s, a) : |rM (s, a) − r̃(s, a)| + γVmax DT V (PM (·|s, a), P (·|s, a)) ≤ ξ}.

(18)

However, it is not impossible to use such a set in practice since the reward function r̃ and true model P (·|s, a) are notgiven to the learner. Our known state-action set Kξ , in comparison, is easier to construct. It simply measures the maximum
disagreement between two data-consistent models. With a considerate choice of α (see (Bhardwaj et al., 2023)), Kξ isalways a subset of Kξ0 . Our definition also matches better with the practical implementation in (Kidambi et al., 2020) whichuses the disagreement between any two models in an ensemble.
Then MOReL learns policy π̂ by solving the MDP (S, A, P̂ , r̂, γ), where(⋆rM(s, a),
s, a ∈ Kξ
r̂(s, a) =−X,
otherwise(⋆PM(s′ |s, a),s, a ∈ Kξ
P̂ (s′ |s, a) =′†1(s = s ),otherwise
Note MOReL introduces an absorbing state s† .
Below we prove a performance guarantee of MOReL. The proof here is different from that in the original paper, because
here we need to consider reward learning and the original proof does not provide an exact rate that converges to zero as thedata size increases. (In the original paper there is a non-zero bias term that depends on the comparator policy.)
21
Both MOPO and MOReL assume that the reward function is known. Here we study the variants of the two algorithms which also
learn the reward function.

47

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

Proposition E.16 (MOReL Performance Guarantee). Given data D̃ = {(s, a, r̃, s′ )} of size N of an unknown MDP⋆(S, A, r̃, P, γ). Suppose r̃, rM∈ [−1, 1] and (r̃, P ) ∈ Mα
. Choose ξ = min(1, O(Vmax (|S|2 |A| log(1/δ)/N )1/4 )). Let
D̃
π

(s,a)
= C < ∞, with probability 1 − δ, it holds thatX ≥ R. For π such that sups,a dµ(s,a)

(1 − γ) Vr̄π (d0 ) − Vr̄π̂ (d0 ) ≤ O Vmax



|S|2 |A| log( 1δ )N

1/4 !
,

with Vmax :=R
1−γfor all r̄ ∈ RR (r̃) := {r̄ : r̄(s, a) = r̃(s, a), ∀(s, a) ∈ supp(µ) and r̄(s, a) ∈ [−R, 1] , ∀(s, a) ∈ S × A}.
Proof. We extend r̃ as r̃(s† , a) = −X for all a ∈ A. Let M̂ = (r̂, Pˆ).
First, by the optimality of π̂, we writeVr̄π (d0 ) − Vr̄π̂ (d0 )

π
π
π̂
π̂
=Vr̄π (d0 ) − VM̂(d0 ) + VM̂(d0 ) − VM̂(d0 ) + VM̂(d0 ) − Vr̄π̂ (d0 )ππ̂(d0 ) + VM̂(d0 ) − Vr̄π̂ (d0 )≤Vr̄π (d0 ) − VM̂πwhere VM̂denotes the value of policy π with respect to the reward r̂ and dynamics P̂ . The last inequality follows from theoptimality of π̂ on M̂ .
Then we notice the fact that Kξ ⊆ supp(µ) with a small ξ. The proof of the lemma below is given in Appendix E.7.3.
Lemma E.17. If ξ ≤ 1, Kξ ⊆ supp(µ).
Let dˆπ denote the average state-action visitation of π with respect to P̂ .
π̂
(1 − γ) VM̂
(d0 ) − Vr̄π̂ (d0 )=Es,a∼dˆπ̂ [r̂(s, a) + γEs′ ∼P̂ |s,a [Vr̄π̂ (s′ )] − r̄(s, a) − γEs′ ∼P |s,a [Vr̄π̂ (s′ )]]≤Es,a∼dˆπ̂ [r̂(s, a) + γEs′ ∼P̂ |s,a [Vr̄π̂ (s′ )] − r̄(s, a) − γEs′ ∼P |s,a [Vr̄π̂ (s′ )]|(s, a) ∈/ Kξ ]+ Es,a∼dˆπ̂ [|r̂(s, a) − r̄(s, a)| + γVmax DT V (P̂ (·|s, a), P (·|s, a))|(s, a) ∈ Kξ ]≤ξ + Es,a∼dˆπ̂ [r̂(s, a) + γEs′ ∼P̂ |s,a [Vr̄π̂ (s′ )] − r̄(s, a) − γEs′ ∼P |s,a [Vr̄π̂ (s′ )]|(s, a) ∈/ Kξ ]≤ξ + Es,a∼dˆπ̂ [−Vmax − r̄(s, a) + γVmax |(s, a) ∈/ Kξ ]≤ξ

where in the second inequality we use Kξ ⊆ supp(µ) (which implies r̃ = r̄ in Kξ ).
Similarly we can showπ
(1 − γ) VM̂
(d0 ) − Vr̄π (d0 )ππ≥ − ξ + Es,a∼dπ [r̂(s, a) + γEs′ ∼P̂ |s,a [VM̂(s′ )] − r̄(s, a) − γEs′ ∼P |s,a [VM̂(s′ )]|(s, a) ∈/ Kξ ]≥ − ξ − 2Vmax Es,a∼dπ [1[(s, a) ∈/ Kξ ]]
Combining the three inequalities above gives(1 − γ) Vr̄π (d0 ) − Vr̄π̂ (d0 ) ≤ 2Vmax Es,a∼dπ [1[(s, a) ∈/ Kξ ]] + 2ξBy the concentratability assumption, we can further upper boundEs,a∼dπ [1[(s, a) ∈/ Kξ ]] ≤ CEs,a∼µ [1[(s, a) ∈/ Kξ ]]
48

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

Therefore we have
(1 − γ) Vr̄π (d0 ) − Vr̄π̂ (d0 ) ≤ 2Vmax Es,a∼µ [1[(s, a) ∈/ Kξ ]] + 2ξEs,a∼µ [Esup (s, a)]+ 2ξξsuprM ,PM Es,a∼µ [E(s, a; rM , PM )]≤ 4Vmax+ 2ξξ≤ 2Vmax

where the last step is Markov inequality,
Esup (s, a) =supM1 ,M2 ∈MαD̃

|rM1 (s, a) − rM2 (s, a)| + γVmax DT V (PM1 (·|s, a), PM2 (·|s, a)) ,and
E(s, a; rM , PM ) = |rM (s, a) − r̃(s, a)| + γVmax DT V (PM (·|s, a), P (·|s, a)).
Now we upper bound the expectation of E over µ. With probability greater than 1 − δ,qEµ [(|rM (s, a) − r̃(s, a)| + γVmax DT V (PM (·|s, a), P (·|s, a)))2 ]√ q≤ 2 Eµ [(rM (s, a) − r̃(s, a))2 + (γVmax )2 DT V (PM (·|s, a), P (·|s, a))2 ]√ q≤ (1 + γVmax ) 2 Eµ [(rM (s, a) − r̃(s, a))2 + DT V (PM (·|s, a), P (·|s, a))2 ]√ q≤ Vmax 2 Eµ [(rM (s, a) − r̃(s, a))2 + DT V (PM (·|s, a), P (·|s, a))2 ]s√|S|2 |A| log( 1δ )≤ Vmax 2
N

Eµ [E(s, a; rM , PM )] ≤where the last step is based on Lemma E.18.
Lemma E.18. (Xie et al., 2022) With probability 1 − δ, for any MDP model M ,22

Eµ [|rM (s, a) − r(s, a)| + DT V (PM (·|s, a), P (·|s, a)) ] ≤ O

ED (M ) − minM ′ ED (M ′ ) + |S|2 |A| log( 1δ )N.

Thus, we haveq


|S|2 |A| log( 1δ )
 + 2ξ
√(1 − γ) Vr̄π (d0 ) − Vr̄π̂ (d0 ) ≤ O 
ξ N 2
1/4 !
|S| |A| log( 1δ )≤ O Vmax
.
N


2
Vmax

By Proposition E.16, we can apply the previous analysis technique to MOReL to show it is admissible without addition
union bounds, which leads to the corollary below.
Corollary E.19. For X ≥ R, MOReL is R-admissible with respect any R̃ ⊆ (S × A → [−1, 1]).
49

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

E.7.2. MOPOMOPO (Yu et al., 2020b) is very similar to MOReL except that it uses negative bonuses (like VI-LCB) instead of truncation.
In the original paper of MOPO, the authors assume the reward is given. Here we consider a variant that also learns the⋆⋆⋆
reward. Specifically, given PM
and rM
above it solves the MDP (S, A, PM, r̂, γ), where⋆r̂(s, a) = rM(s, a) − b(s, a)

While the original MOPO paper does not give a specific design of b(s, a) that is provably correct, in principle making MOPO
provably correct is possible. Essentially, we need to choose a large enough bonus to cover the error of the model-based
Bellman operator (defined by the estimated reward and dynamics). This would lead to a bonus of order Vmax (see theanalysis of VI-LCB in Appendix E.5). Then we can proceed with an analysis that combines that of MOReL and VI-LCB toprove MOPO’s performance guarantee. This can then be used to show (like the previous proofs) that MOPO is admissible.
We omit the proof here.
R
Corollary E.20. Suppose b(s, a) = Θ(Vmax ) for (s, a) ∈/ supp(µ). For Vmax ≥ 1−γ, MOPO is R-admissible with respect

any R̃ ⊆ (S × A → [−1, 1]).

E.7.3. P ROOF OF T ECHNICAL L EMMA
Proof of Lemma E.17. Consider any (s̄, ā) ∈/ supp(µ). Given any M = (S, A, PM , rM , γ) ∈ Mα
, we construct M ′ =D̃(S, A, PM , rM ′ , γ) with rM ′ = rM for all (s, a) ∈ (S × A) \ {(s̄, ā)}, and rM ′ (s̄, ā) = −sign(rM (s̄, ā)). By construction,
since M and M ′ agrees on all state-actions except (s̄, ā), which is not in data support, we have M ′ ∈ Mα
. However, weD̃
have that
sup|rM1 (s̄, ā) − rM2 (s̄, ā)| + γVmax DT V (PM1 (·|s̄, ā), PM2 (·|s̄, ā))M1 ,M2 ∈MαD̃

≥|rM (s̄, ā) − rM ′ (s̄, ā)| + γVmax DT V (PM (·|s̄, ā), PM (·|s̄, ā))=|rM (s̄, ā) − rM ′ (s̄, ā)| ≥ 1 ≥ ξ.

This means that (s̄, ā) ∈/ Kξ . Therefore Kξ ⊆ supp(µ).

F. Grid WorldIn this section, we describe the full details of the grid world study that was discussed in Section 3.1.
Environment. We consider goal-directed navigation in the environment shown in Figure 10. We use a fixed grid world
layout shown in the figure. The agent’s state is given by a tuple (x, y, d) where (x, y) ∈ [5]2 represents the 2-d coordinate

Length Bias

Vmin,ts0t<latexit sha1_base64="S9y1Ww888+FmkLj5dzQ9g2YzUfg=">AAAB8HicdVDLSgMxFM3UV62vqks3wSK4kGFmWtu6K7pxWcE+pB1KJs20oUlmSDJCGfoVblwo4tbPceffmGkrqOiBC4dz7uXee4KYUaUd58PKrayurW/kNwtb2zu7e8X9g7aKEolJC0cskt0AKcKoIC1NNSPdWBLEA0Y6weQq8zv3RCoaiVs9jYnP0UjQkGKkjXTXHqScijM9GxRLjn1Rr3rnHnRsx6l55WpGvFrFK0PXKBlKYInmoPjeH0Y44URozJBSPdeJtZ8iqSlmZFboJ4rECE/QiPQMFYgT5afzg2fwxChDGEbSlNBwrn6fSBFXasoD08mRHqvfXib+5fUSHdb9lIo40UTgxaIwYVBHMPseDqkkWLOpIQhLam6FeIwkwtpkVDAhfH0K/ydtz3bLtndTKTUul3HkwRE4BqfABTXQANegCVoAAw4ewBN4tqT1aL1Yr4vWnLWcOQQ/YL19AjIbkK0=</latexit><latexit sha1_base64="sKd5ox1s7JN0w0Wwc6DPZJ/6dUY=">AAAB63icdVDLSgMxFM3UV62vqks3wSK6GjJjpXVXdOOygn1AO5RMmmlDk8yQZIRS+gtuXCji1h9y59+YaUdQ0QMXDufcy733hAln2iD04RRWVtfWN4qbpa3tnd298v5BW8epIrRFYh6rbog15UzSlmGG026iKBYhp51wcp35nXuqNIvlnZkmNBB4JFnECDaZpE8HZlCuILeGLnzfh8hFC1hSRajuX0IvVyogR3NQfu8PY5IKKg3hWOuehxITzLAyjHA6L/VTTRNMJnhEe5ZKLKgOZotb5/DEKkMYxcqWNHChfp+YYaH1VIS2U2Az1r+9TPzL66UmqgczJpPUUEmWi6KUQxPD7HE4ZIoSw6eWYKKYvRWSMVaYGBtPyYbw9Sn8n7R91zt3/dtqpXGVx1EER+AYnAEP1EAD3IAmaAECxuABPIFnRziPzovzumwtOPnMIfgB5+0TBuGOPA==</latexit>tHaTimeout(Failure)<latexit sha1_base64="CUm81vWGY9iupv2gUQ1O9EsZVFQ=">AAAB7nicdVDLSgNBEOyNrxhfUY9eBoPgaZldI4m3oJccI5gHJEuYncwmQ2YfzMwKYclHePGgiFe/x5t/4yRZQUULGoqqbrq7/ERwpTH+sApr6xubW8Xt0s7u3v5B+fCoo+JUUtamsYhlzyeKCR6xtuZasF4iGQl9wbr+9Gbhd++ZVDyO7vQsYV5IxhEPOCXaSF2NBoKh5rBcwXYNX7qui7CNlzCkinHdvUJOrlQgR2tYfh+MYpqGLNJUEKX6Dk60lxGpORVsXhqkiiWETsmY9Q2NSMiUly3PnaMzo4xQEEtTkUZL9ftERkKlZqFvOkOiJ+q3txD/8vqpDupexqMk1Syiq0VBKpCO0eJ3NOKSUS1mhhAqubkV0QmRhGqTUMmE8PUp+p90XNu5sN3baqVxncdRhBM4hXNwoAYNaEIL2kBhCg/wBM9WYj1aL9brqrVg5TPH8APW2yeXlI8W</latexit><latexit sha1_base64="BSBMOshdpJGx9RcX7WQMQZ1DdtI=">AAAB6HicdVDLSgNBEJz1GeMr6tHLYBA8LbNrJPEW9OIxAfOAZAmzk95kzOyDmVkhLPkCLx4U8eonefNvnCQrqGhBQ1HVTXeXnwiuNCEf1srq2vrGZmGruL2zu7dfOjhsqziVDFosFrHs+lSB4BG0NNcCuokEGvoCOv7keu537kEqHke3epqAF9JRxAPOqDZSkw5KZWJXyYXrupjYZAFDKoTU3Evs5EoZ5WgMSu/9YczSECLNBFWq55BEexmVmjMBs2I/VZBQNqEj6Bka0RCUly0OneFTowxxEEtTkcYL9ftERkOlpqFvOkOqx+q3Nxf/8nqpDmpexqMk1RCx5aIgFVjHeP41HnIJTIupIZRJbm7FbEwlZdpkUzQhfH2K/ydt13bObbdZKdev8jgK6BidoDPkoCqqoxvUQC3EEKAH9ISerTvr0XqxXpetK1Y+c4R+wHr7BP/MjRI=</latexit>

0<latexit sha1_base64="VnW+u71je3AqmI1QMUE8Ge09bzA=">AAAB6HicdVDLSgMxFM3UV62vqks3wSK4GjLT2tZd0Y3LFuwD2qFk0kwbm8kMSUYoQ7/AjQtF3PpJ7vwbM20FFT1w4XDOvdx7jx9zpjRCH1ZubX1jcyu/XdjZ3ds/KB4edVSUSELbJOKR7PlYUc4EbWumOe3FkuLQ57TrT68zv3tPpWKRuNWzmHohHgsWMIK1kVpoWCwh+7JedS9ciGyEam65mhG3VnHL0DFKhhJYoTksvg9GEUlCKjThWKm+g2LtpVhqRjidFwaJojEmUzymfUMFDqny0sWhc3hmlBEMImlKaLhQv0+kOFRqFvqmM8R6on57mfiX1090UPdSJuJEU0GWi4KEQx3B7Gs4YpISzWeGYCKZuRWSCZaYaJNNwYTw9Sn8n3Rc2ynbbqtSalyt4siDE3AKzoEDaqABbkATtAEBFDyAJ/Bs3VmP1ov1umzNWauZY/AD1tsn3VCM/A==</latexit>

s1<latexit sha1_base64="SrOcaRcqD6kTkp9wxRGOrqamJXU=">AAAB6nicdVDLSgNBEOyNrxhfUY9eBoPgaZldI4m3oBePEc0DkiXMTmaTIbMPZmaFsOQTvHhQxKtf5M2/cZKsoKIFDUVVN91dfiK40hh/WIWV1bX1jeJmaWt7Z3evvH/QVnEqKWvRWMSy6xPFBI9YS3MtWDeRjIS+YB1/cjX3O/dMKh5Hd3qaMC8ko4gHnBJtpFs1cAblCrZr+Nx1XYRtvIAhVYzr7gVycqUCOZqD8nt/GNM0ZJGmgijVc3CivYxIzalgs1I/VSwhdEJGrGdoREKmvGxx6gydGGWIgliaijRaqN8nMhIqNQ190xkSPVa/vbn4l9dLdVD3Mh4lqWYRXS4KUoF0jOZ/oyGXjGoxNYRQyc2tiI6JJFSbdEomhK9P0f+k7drOme3eVCuNyzyOIhzBMZyCAzVowDU0oQUURvAAT/BsCevRerFel60FK585hB+w3j4BQIGNyA==</latexit>sh

Terminal
sH+1 (Success)<latexit sha1_base64="fEV+3LGZJGOaa94XCeR2gmM9WCk=">AAAB7nicdVDLSgNBEOz1GeMr6tHLYBAEYZldI4m3oJccI5gHJEuYncwmQ2YfzMwKYclHePGgiFe/x5t/4yRZQUULGoqqbrq7/ERwpTH+sFZW19Y3Ngtbxe2d3b390sFhW8WppKxFYxHLrk8UEzxiLc21YN1EMhL6gnX8yc3c79wzqXgc3elpwryQjCIecEq0kTpqkDXOndmgVMZ2FV+6rouwjRcwpIJxzb1CTq6UIUdzUHrvD2OahizSVBCleg5OtJcRqTkVbFbsp4olhE7IiPUMjUjIlJctzp2hU6MMURBLU5FGC/X7REZCpaahbzpDosfqtzcX//J6qQ5qXsajJNUsostFQSqQjtH8dzTkklEtpoYQKrm5FdExkYRqk1DRhPD1KfqftF3bubDd20q5fp3HUYBjOIEzcKAKdWhAE1pAYQIP8ATPVmI9Wi/W67J1xcpnjuAHrLdPARmPWw==</latexit><latexit sha1_base64="kBmI5+Tw4YXcdcEE87iZtxD+Jgs=">AAAB6nicdVDLSgNBEOyNrxhfUY9eBoPgaZldI4m3oBePEc0DkiXMTmaTIbMPZmaFsOQTvHhQxKtf5M2/cZKsoKIFDUVVN91dfiK40hh/WIWV1bX1jeJmaWt7Z3evvH/QVnEqKWvRWMSy6xPFBI9YS3MtWDeRjIS+YB1/cjX3O/dMKh5Hd3qaMC8ko4gHnBJtpFs1GA/KFWzX8LnrugjbeAFDqhjX3Qvk5EoFcjQH5ff+MKZpyCJNBVGq5+BEexmRmlPBZqV+qlhC6ISMWM/QiIRMedni1Bk6McoQBbE0FWm0UL9PZCRUahr6pjMkeqx+e3PxL6+X6qDuZTxKUs0iulwUpALpGM3/RkMuGdViagihkptbER0TSag26ZRMCF+fov9J27WdM9u9qVYal3kcRTiCYzgFB2rQgGtoQgsojOABnuDZEtaj9WK9LlsLVj5zCD9gvX0Ck92N/w==</latexit>

a?h = ⇡h? (sh )<latexit sha1_base64="zER7H1iFHVV8fYcw2dWL519non4=">AAACBXicdVDLSgMxFM34rPU16lIXwSLUTcmMldaFUHTjsoJ9QGccMmmmDc08SDJCGbpx46+4caGIW//BnX9j+gIVPXDh5Jx7yb3HTziTCqFPY2FxaXllNbeWX9/Y3No2d3abMk4FoQ0S81i0fSwpZxFtKKY4bSeC4tDntOUPLsd+644KyeLoRg0T6oa4F7GAEay05JkH+NaRCguvD8+hk7D5qyi9/rFnFlCpgk5t24aohCbQpIxQ1T6D1kwpgBnqnvnhdGOShjRShGMpOxZKlJthoRjhdJR3UkkTTAa4RzuaRjik0s0mV4zgkVa6MIiFrkjBifp9IsOhlMPQ150hVn352xuLf3mdVAVVN2NRkioakelHQcqhiuE4EthlghLFh5pgIpjeFZI+FpgoHVxehzC/FP5PmnbJOinZ1+VC7WIWRw7sg0NQBBaogBq4AnXQAATcg0fwDF6MB+PJeDXepq0LxmxmD/yA8f4FnryYBA==</latexit>

Figure 10: Left: A goal-directed gridworld navigation task where the agent (red triangle) has to reach the goal (key) andavoid lava (orange waves). Center: Shows the length bias that helps offline RL succeed even when rewards are entirely
replaced with random or adversarial values. Right: Results on the grid world task. Behavior cloning fails to solve the task
(red path) while using an offline RL method leads to success (blue path) even with wrong rewards. The opacity of the gridsquare overlay indicates how frequent that state is in the dataset (more opaque means more frequent).
50

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Dataand d ∈ {N, W, E, S} encodes the North, West, East, and South direction respectively that the agent is facing. The agent’s
action space is A = {f, l, r} where f denotes the action of moving to grid square in front of the agent, l denotes a left turnof 90 degrees, and r denotes a right turn of 90 degrees. The agent can go through the lava (orange wavy square) but cannot
go through the wall (grey square). The goal (key) is an absorbing state. The agent gets a reward of +1 when first visiting thegoal followed by a reward of 0 for then staying in the goal. The agent gets a reward of -1 for reaching lava and a reward of-0.01 for all other actions to incentivize the agent to reach the goal along the shortest safe path. We use a horizon of H = 20.
The agent deterministically starts in the top-right corner shown in Figure 10.
Dataset. We collect a dataset D of 500 episodes by taking 100 identical optimal episodes along with 400 identical episodes
that all touch the topmost lava field. We introduce a length bias by terminating an episode if the agent touches the lava or
fails to reach the goal in H − 1 steps (i.e., the agent doesn’t survive). In addition to the original setting with the observed
reward, we consider three additional settings where the rewards in the dataset are replaced by: (i) a constant zero reward, (ii)their negative, and (iii) a random number sampled uniformly in [0, 1].
Algorithm 4 PEVI(S, A, H, D, β, Vmin , Vmax ) (Jin et al., 2021). We are given access to a set D of episodes (τ ). Additionally,
we are given a hyperparameter β ∈ R≥0 that controls pessimism. The hyperparameters Vmin , Vmax ∈ RH indicate bounds
on policy value with Vmin,h and Vmax,h denoting the minimum and maximum policy values respectively for time steph ∈ [H]. Unlike the original PEVI algorithm of (Jin et al., 2021), we treat Vmin , Vmax as hyperparameters.
Set VH+1 = 0
for h = H, · · · , 1 dofor s ∈ S dofor a ∈ A do P
nh (s, a) = τ ∈D 1{sτh = s ∧ aτh = a}If nh (s, a) > 0
P
1
τ
τ
τ
τ
Q̃h (s, a) = nh (s,a)τ ∈D 1{sh = s ∧ ah = a}{rh + Vh+1 (sh+1 )}Q̄h (s, a) = Q̃h (s, a) − √ βnh (s,a)b h (s, a) = max{Vmin,h , min{Q̄h (s, a), Vmax,h }}QElseb h (s, a) = Vmin,hQ
end forb h (s, a)πh (s) = arg maxa∈A Q
b
Vh (s) = Qh (s, πh (s))
end forend for
Return π = (π1:H )Methods. We evaluate two methods: behavior cloning which simply takes the action with the highest empirical count in agiven state and a random action if the state was unseen, and PEVI (Jin et al., 2021) an offline RL method with guarantees fortabular MDP. We provide a pseudocode of PEVI in Algorithm 4. Unlike the original PEVI in (Jin et al., 2021), we provide
minimum and maximum bounds for the value function for each time step as an additional hyperparameter. These bounds
should be dictated by our CMDP framework and should penalize out-of-distribution actions. In principle, we need to set
these bounds such that the algorithm is admissible. Please see Appendix C and Appendix D for details.
Intuitively, PEVI performs dynamic programming similar to a standard value iteration with two key changes. Firstly, it
uses pessimistic value initialization where the learned value Vh (s) of a state s at time step h is set to the lowest possible
return Vmin,h if the state is unseen at time step h. Secondly, when performing value iteration it adds a pessimism penalty of√−β/ n(s,a) to the dataset reward where n(s, a) is the empirical state-action count in D. This pessimism penalty ensures thatthe learned value function lower bounds the true value function (hence a pessimistic estimate). In our experiments, we varyβ and set Vmin,h = Ṽmin,h − β(H − h + 1) − 1 where Ṽmin,h is the minimum possible return for the given reward type that
we are working with. The value of Ṽmin,h is −(H − h + 1) for the original reward, −1 for the negative reward, and 0 forthe zero reward and random reward cases). We define Vmax,h = Ṽmax,h where Ṽmax,h is the maximum possible return for
the given reward type. The value of Ṽmax,h is 1 of the original reward, (H − h + 1) for the random reward and the negative51

Survival Instinct in Offline Reinforcement Learning and Implicit Human Bias in Data

Reward Type
Original Reward
Zero Reward
Random Reward
Negative Reward

Behavior Cloning−1.19 ± 0.00
−1.19 ± 0.00
−1.19 ± 0.00
−1.19 ± 0.00

PEVI
0.92 ± 0.00
0.92 ± 0.00
0.92 ± 0.00
0.92 ± 0.00

Table 16: Gridworld results: Mean return on 1000 test episodes. PEVI achieves the optimal return of V ⋆ = 0.92.
reward, and 0 for the zero reward case.
Results. We show numerical results in Table 16. We visualize results in Figure 10. The behavior cloning simply imitates
the most common trajectory in the dataset which results in going to the lava (shown in red in the figure). In contrast, the
PEVI is able to reach the goal in every reward situation.
Explanation. The reason why PEVI works can be understood simply by pessimistic initialization and length bias. The
length bias in this setting is visualized in the center of Figure 10. PEVI assigns the final state s′t of a failed trajectory will geta pessimistic value of Vt (s′t ) = Vmin,t . This value isn’t updated as we never take any action in s′t due to timeout. In ourcase, s′t will be the state where the agent first visits the lava. In contrast, the final state of a successful trajectory sH+1 gets
assigned a value of 0. If Vmin is sufficiently negative, then PEVI will only consider policies that stay all H steps within
the data support. Due to length bias, this is not true for the non-surviving trajectories that reach the lava. Further, theonly trajectories in our case that complete end up reaching and staying in the goal. Therefore, under the assumption that
Vmin,h are all sufficiently small, PEVI will learn a policy reaches the goal irrespective of the correctness of the reward in
the data. The sufficiency of the negativity of Vmin,h is implied by our CMDP framework. Lastly, note that as the world isdeterministic, we didn’t need to use the pessimism arising from the √ −β penalty term.
n(s,a)

52