Title: Survival Instinct in Offline Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Offline RL

safe RL
Abstract: We present a novel observation about the behavior of offline reinforcement learning (RL) algorithms: on many benchmark datasets, offline RL can produce well-performing and safe policies even when trained with "wrong" reward labels, such as those that are zero everywhere or are negatives of the true rewards. This phenomenon cannot be easily explained by offline RL's return maximization objective. Moreover, it gives offline RL a degree of robustness that is uncharacteristic of its online RL counterparts, which are known to be sensitive to reward design. We demonstrate that this surprising robustness property is attributable to an interplay between the notion of *pessimism* in offline RL algorithms and certain implicit biases in common data collection practices. As we prove in this work, pessimism endows the agent with a *survival instinct*, i.e., an incentive to stay within the data support in the long term, while the limited and biased data coverage further constrains the set of survival policies. Formally, given a reward class -- which may not even contain the true reward -- we identify conditions on the training data distribution that enable offline RL to learn a near-optimal and safe policy from any reward within the class. We argue that the survival instinct should be taken into account when interpreting results from existing offline RL benchmarks and when creating future ones. Our empirical and theoretical results suggest a new paradigm for offline RL, whereby an agent is "nudged" to learn a desirable behavior with imperfect reward but purposely biased data coverage. Please visit our website [https://survival-instinct.github.io](https://survival-instinct.github.io) for accompanied code and videos.

Survival Instinct in Offline Reinforcement Learning

Abstract

We present a novel observation about the behavior of offline reinforcement learning (RL) algorithms: on many benchmark datasets, offline RL can produce wellperforming and safe policies even when trained with 'wrong' reward labels, such as those that are zero everywhere or are negatives of the true rewards. This phenomenon cannot be easily explained by offline RL's return maximization objective. Moreover, it gives offline RL a degree of robustness that is uncharacteristic of its online RL counterparts, which are known to be sensitive to reward design. We demonstrate that this surprising robustness property is attributable to an interplay between the notion of pessimism in offline RL algorithms and certain implicit biases in common data collection practices. As we prove in this work, pessimism endows the agent with a survival instinct , i.e., an incentive to stay within the data support in the long term, while the limited and biased data coverage further constrains the set of survival policies. Formally, given a reward class - which may not even contain the true reward - we identify conditions on the training data distribution that enable offline RL to learn a near-optimal and safe policy from any reward within the class. We argue that the survival instinct should be taken into account when interpreting results from existing offline RL benchmarks and when creating future ones. Our empirical and theoretical results suggest a new paradigm for offline RL, whereby an agent is 'nudged' to learn a desirable behavior with imperfect reward but purposely biased data coverage. Please visit our website https://survival-instinct.github.io for accompanied code and videos.

1 Introduction

In offline reinforcement learning (RL), an agent optimizes its performance given an offline dataset. Despite being its main objective, we find that return maximization is not sufficient for explaining some of its empirical behaviors. In particular, in many existing benchmark datasets, we observe that offline RL can produce surprisingly good policies even when trained on utterly wrong reward labels.

In Fig. 1, we present such results on the hopper task from D4RL [1], a popular offline RL benchmark, using a state-of-the-art offline RL algorithm ATAC [2]. The goal of an RL agent in the hopper task is to move forward as fast as possible while avoiding falling down. We trained ATAC agents on the original datasets and on three modified versions of each dataset, with 'wrong' rewards: 1) zero : assigning a zero reward to all transitions, 2) random : labeling each transition with a reward sampled uniformly from [0 , 1] , and 3) negative : using the negation of the true reward. Although these wrong rewards contain no information about the underlying task or are even misleading, the policies learned from them in Fig. 1 (left) often perform significantly better than the behavior (data collection) policy and the behavior cloning (BC) policy. They even outperform policies trained with the true reward (denoted as original in Fig. 1) in some cases. This is puzzling, since RL is notorious for being sensitive to reward mis-specification [3-6]: in general, maximizing the wrong rewards with RL leads to sub-optimal performance that can be worse than simply performing BC. In addition, these wrong-reward policies demonstrate a 'safe' behavior, which keeps the hopper from falling down for a longer period than other comparators in Fig. 1 (right). This is yet another peculiarity hard to link to

(a) Normalized scores

(b) Episode lengths

behavior

BC

original

zero

random

negative

1000

100

75

750

50

500

25

250

random

medium

medium-replaymedium-expert

0

random

medium

medium-replaymedium-expert

return maximization, as none of the wrong rewards encourage the agent to stay alive. As we will show empirically in Section 4, these effects are not unique to ATAC or the hopper task. They occur with multiple offline RL algorithms, including ATAC [2], PSPI [7], IQL [8], CQL [9] and the Decision Transformer (DT) [10], on dozens of datasets from D4RL [1] and Meta-World [11] benchmarks.

This robustness of offline RL is not only counter-intuitive but also cannot be explained fully by the literature. Offline RL theory [2, 7, 12, 13] provides performance guarantees only when the data reward is the true reward. Although offline imitation learning (IL) [14-16] makes no assumptions about reward, it only shows that the learned policy can achieve performance comparable to that of the behavior policy, not beyond. Robust offline RL [17-21] shows that specialized algorithms can perform well when the size of data perturbation is small. In contrast, we demonstrate that standard offline RL algorithms can produce good policies even when we completely change the reward. Constrained offline RL algorithms [22-24] can learn safe behaviors when constraint violations are both explicitly labeled and optimized. However, in the phenomena we observe, no safety signal is given to off-the-shelf, unconstrained offline RL algorithms. Recently, [25] observes a similar robustness phenomena of offline RL, but does not provide a complete explanation. 1 In Appendix B, we discuss the gap between the related work and our findings in more detail.

In this work, we provide an explanation for this seemingly surprising observation. In theory, we prove that this robustness property is attributed to the interplay between the use of pessimism in offline RL algorithms and the implicit bias in typical data collection processes. Offline RL algorithms often use pessimism to avoid taking actions that lead to unknown future events. We show that this risk-averse tendency bakes a 'survival instinct' into the agent, an incentive to stay within the data coverage in the long term. On the other hand, the limited coverage of offline data further constrains the set of survival policies (policies that remain in the data support in the long term). When this set of survival policies correlates with policies that achieve high returns w.r.t. the true reward (as in the example in Fig. 1), robust behavior emerges.

Our theoretical and empirical results have two important implications. First and foremost, offline RL has a survival instinct that leads to inherent robustness and safety properties that online RL does not have. Unlike online RL, offline RL is doubly robust : as long as the data reward is correct or the data has a positive implicit bias, a pessimistic offline RL agent can perform well. Moreover, offline RL is safe as long as the data only contains safe states; thus safe RL in the offline setup can be achieved without specialized algorithms. Second, because of the existence of the survival instinct, the data coverage has a profound impact on offline RL. While a large data coverage improves the best policy that can be learned by offline RL with the true reward, it can also make offline RL more sensitive to imperfect rewards. In other words, collecting a large set of diverse data might not be necessary or helpful (see Section 4). This goes against the common wisdom in the RL community that data should be as exploratory as possible [26-28].

We emphasize that survival instinct's implications should be taken into account when interpreting results on existing offline RL benchmarks as well as when designing future ones. We should treat the evaluation of offline RL algorithms differently from online RL algorithms. We suggest evaluating the performance of an offline RL algorithm by training it with wrong rewards in addition to the true reward so that we can isolate the performance due to return maximization from the compounded effects of survival instinct and implicit data bias. We propose to use this performance as a score to quantify the data bias (see Eq. (3) in Section 4.1) in practice.

We believe that our findings shed new light on RL applicability and research. To practitioners, we demonstrate that offline RL does not always require the correct reward to succeed. This opens up the possibility of using offline RL in domains where obtain high-quality rewards is challenging. Researchwise, the existence of the survival instinct raises the question of how to design data collection or data filtering procedures that would help offline RL to leverage this instinct in order to improve RL's performance with incorrect or missing reward labels. While in this paper we focus on positive data biases, we caution that in practice the data bias might be negatively correlated with a user's intention. In that case, running offline RL even with the right reward would not lead to the right behavior.

2 Background

The goal of offline RL is to solve an unknown Markov decision process (MDP) from offline data. Typically, an offline dataset is a collection of tuples, D := { ( s, a, r, s ′ ) | ( s, a ) ∼ µ ( · , · ) , r = r ( s, a ) , s ′ ∼ P ( ·| s, a ) } , where r is the reward, P captures the MDP's transition dynamics, and µ denotes the state-action data distribution induced by some data collection process. Modern offline RL algorithms adopt pessimism to address the issue of policy learning when µ does not have the full state-action space as its support. The basic idea is to optimize for a performance lower bound that penalizes actions leading to out-of-support future states. Such a penalty can take the form of behavior regularization [8, 29, 30], negative bonuses to discourage visiting less frequent state-action pairs [12, 31, 32], pruning less frequent actions [13, 33], adversarial training [2, 7, 14, 34, 35] or value penalties in modified dynamic programming [9, 36].

We are interested in settings where the offline RL agent learns not from D itself but from its corrupted version ˜ D with a wrong reward ˜ r , i.e., ˜ D := { ( s, a, ˜ r, s ′ ) | ( s, a, s ′ ) ∈ D , ˜ r = ˜ r ( s, a ) ∈ [ -1 , 1] } . We assume ˜ r is from a reward class ˜ R -which may not necessarily contain the true reward r -and we wish to explain why offline RL can learn good behaviors from the corrupted dataset ˜ D (e.g., as in Fig. 1). To this end, we introduce notations and assumptions we will use in the paper.

Notation We focus on the setting of infinite-horizon discounted MDPs. We denote the task MDP that the agent aims to solve as M = ( S , A , r, P, γ ) , where S is the state space, A is the action space, and γ ∈ [0 , 1) is the discount. Without loss of generality, we assume r : S × A → [0 , 1] . Let ∆( U ) denote the set of probability distributions over a set U . We denote a policy as π : S → ∆( A ) . For a reward function r , we define a policy π 's state value function as V π r ( s ) := E π,P [ ∑ ∞ t =0 γ t r ( s t , a t ) | s 0 = s ] and the state-action value function as Q π r ( s, a ) := r ( s, a ) + γ E s ′ ∼ P ( ·| s,a ) [ V π r ( s ′ )] . Solving MDP M requires learning a policy that maximizes the return at an initial state distribution d 0 ∈ ∆( S ) , that is, max π E s ∼ d 0 [ V π r ( s )] . We denote the optimal policy as π ∗ and the optimal value functions as V ∗ r and Q ∗ r . Given d 0 , we define the average stateaction visitation of a policy π as d π ( s, a ) := (1 -γ ) E π,P [ ∑ ∞ t =0 γ t d π t ( s, a )] , where d π t ( s, a ) denotes the probability of visiting ( s, a ) at time t when running π starting at an initial state sampled from d 0 . Note that (1 -γ ) E s ∼ d 0 [ V π r ( s )] = E ( s,a ) ∼ d π [ r ( s, a )] . For a function f : S × A → R , we use the shorthand f ( s, π ) = E a ∼ π | s [ f ( s, a )] ; similarly for f : S → R , we write f ( p ) = E s ∼ p [ f ( s )] , e.g., V π r ( d 0 ) = E s ∼ d 0 [ V π r ( s )] . For a distribution p , we use supp ( p ) to denote its support.

Assumption We make the typical assumption in offline RL that the data distribution µ assigns positive probabilities to these state-actions visited by running the optimal policy π ∗ starting from d 0 .

This is a standard assumption in the offline RL literature, which in the worst case is a necessary condition to have no regret w.r.t. π ∗ [37]. There are generalized notions [7, 34] of this kind, which are weaker but requires other smoothness assumptions. We note that Assumption 1 does not assume that µ is the average state-action visitation frequency of a single behavior policy, nor does it assume µ has a full coverage of all states and actions or all policy distributions [27, 38].

3 Why Offline RL can Learn Right Behaviors from Wrong Rewards

In this section, we provide conditions under which offline RL's aforementioned robustness w.r.t. misspecified rewards emerges. Our main finding is summarized in the theorem below.

Theorem 1. (Informal) Under Assumption 1 and certain regularity assumptions, if an offline RL algorithm Algo is set to be sufficiently pessimistic and the data distribution µ has a positive bias, for any data reward ˜ r ∈ ˜ R , the policy ˆ π learned by Algo from the dataset ˜ D

has performance guarantee V π ∗ r ( d 0 ) -V ˆ π r ( d 0 ) ≤ O ( ι ) as well as safety guarantee (1 -γ ) ∑ ∞ t =0 γ t Prob( ∃ τ ∈ [0 , t ] , s τ / ∈ supp ( µ ) | ˆ π ) ≤ O ( ι ) for a small ι that decreases to zero as the degree of pessimism and dataset size increase.

In other words, the robustness originates from an interplay between pessimism and an implicit positive bias in data. Here is the insight on why Theorem 1 is true. Because of pessimism, offline RL endows agents with a 'survival instinct' - it implicitly solves a constrained MDP (CMDP) problem [39] that enforces the policy to stay within the data support. When combined with a training data distribution that has a positive bias (e.g., all policies staying within data support are near-optimal) such a survival instinct results in robustness to reward misspecification.

Overall, Theorem 1 has two important implications:

Offline RL is doubly robust : it can learn near optimal policies so long as either the reward label is correct or the data distribution is positively biased;

Offline RL is intrinsically safe when data are safe, regardless of reward labeling, without the need of explicitly modeling safety constraints.

In the remaining of this section, we provide details and discussion of the statements above. The complete theoretical statements and proofs can be found in Appendix D.

3.1 Intuitions for Survival Instinct and Positive Data Bias

We first use a grid world example to build some intuitions. Fig. 2 shows a goal-directed problem, where the true reward is +1 and -1 upon touching the key and the lava, respectively. The offline data is suboptimal and does not have full support. All data trajectories that touched the lava were stopped early, while others were allowed to continue until the end of the episode. The goal state (key) is an absorbing state, where the agent can stay forever beyond the episode length. We use the wrong rewards in Fig. 1 to train PEVI [12], a finite-horizon, tabular offline RL method. PEVI performs dynamic programming similar to value iteration, but with a pessimistic value initialization and an instantaneous pessimism-inducing penalty of O ( -1 / √ n ( s,a ) ) , where n ( s, a ) is the empirical count in data. The penalties ensure that the learned value lower bounds the true one.

We see the PEVI agent learned with any wrong reward in Fig. 1 is able to solve the problem despite data imperfection, while the BC agent that mimics the data directly fails. The main reasons are: 1) There is a data bias whereby longer trajectories end closer to the goal (especially, the longest trajectories are the ones that reach the goal, since the goal is an absorbing state). We call this a length bias (see Section 3.3). This positive

data bias is due to bad trajectories (touching the lava or not reaching the goal) being cut short or timing out. 2) Pessimism in PEVI gives the agent an algorithmic bias (i.e., survival instinct) that favors longer data trajectories. Because of the pessimistic value initialization, PEVI treats trajectories shorter than the episode length as having the lowest return. As a result, by maximizing the pessimistically estimated values, PEVI learns good behaviors despite wrong rewards by leveraging the survival instinct and positive data bias together . We now make this claim more general.

3.2 Survival Instinct

Survival instinct is a pessimism induced behavior that offline RL algorithms tend to favor policies leading to in-support trajectories . We formally characterize this risk aversion behavior by the concept of constrained MDP (CMDP) [39], which we define below.

Definition 1. Let f, g : S ×A → [ -1 , 1] . A CMDP C ( S , A , f, g, P, γ ) is a constrained optimization problem: max π V π f ( d 0 ) s.t. V π g ( d 0 ) ≤ 0 . Let π † denote its optimal policy. For δ ≥ 0 , we define the set of δ -approximately optimal policies Π † f,g ( δ ) := { π : V π † f ( d 0 ) -V π f ( d 0 ) ≤ δ, V π g ( d 0 ) ≤ δ } .

We prove that when trained on ˜ D , offline RL, because of its pessimism, implicitly solves the CMDP below even when the algorithm does not explicitly model any constraints:

where c µ ( s, a ) := ✶ [ µ ( s, a ) = 0] indicates whether ( s, a ) is out of the support of µ 2 i.e., the

constraint in Eq. (1) enforces the agent's trajectories to stay within the data support. Note the constraint in this CMDP is feasible because of Assumption 1.

Proposition 1 (Survival Instinct) . (Informal) Under certain regularity conditions on C µ (˜ r ) , the policy learned by an offline RL algorithm Algo with the offline dataset ˜ D is ι -approximately optimal with respect to C µ (˜ r ) , for some small ι which decreases as the algorithm becomes more pessimistic.

Proposition 1 says if an offline RL algorithm is sufficiently pessimistic, then the learned policy is approximately optimal to C µ (˜ r ) . The learned policy has not only small regret with respect to the data reward ˜ r , but also small chances of escaping the support of the data distribution µ .

We highlight that the survival instinct in Proposition 1 is a long-term behavior. Such a survival behavior cannot be achieved by myopically taking actions in the data support in general (such as BC). 3 For instance, when some trajectories generating the data are truncated (e.g., due to earlystopping or intervention for safety reasons), taking in-support actions may still lead to out-of-support states in the future, as the BC agent in Section 3.1.

Proof Sketch The key to prove Proposition 1 is to show that an offline RL algorithm by pessimism has small regret for not just ˜ r but a set of reward functions consistent with ˜ r on data but different outside of data, including the Lagrange reward of (1) (i.e., ˜ r -λc µ , with a large enough λ ≥ 0 ). We call this property admissibility and we prove in Appendix F that many existing offline RL algorithms are admissible, including model-free algorithms ATAC [2], VI-LCB [31] PPI/PQI [13], PSPI [7], as well as model-algorithms, ARMOR [34, 40], MOPO [32], MOReL [33], and CPPO [14]. By Proposition 1 these algorithms have survival instinct. Please see Appendix D for details.

3.3 Positive Data Bias

In addition to survival instinct, another key factor is an implicit positive bias in common offline datasets. Typically these data manifest meaningful behaviors. For example, in collecting data for goal-oriented problems, data recording is normally stopped if the agent fails to reach the goal within certain time limit. Another example is problems (like robotics or healthcare) where wrong decisions can have detrimental consequences, i.e., problems that safe RL studies. In these domains, the data are collected by qualified policies only or under an intervention mechanism to prevent catastrophic failures. Such a one-sided bias can creates an effect that staying within data support would lead to meaningful behaviors. Below we formally define the positive data bias; Later in Section 4 (Fig. 5), we provide empirical estimates of the degree of positive data bias.

Definition 2 (Positive Data Bias) . A distribution µ is 1 ϵ -positively biased w.r.t. a reward class ˜ R if

for all δ ≥ 0 , where Π † ˜ r,c µ ( δ ) denotes the set of δ -approximately optimal policy of C µ (˜ r ) .

We measure the degree of positive data bias based on how bad a policy can perform in terms of the true reward r when approximately solving the CMDP C µ (˜ r ) in (1) defined with the wrong reward ˜ r ∈ ˜ R . If the data distribution µ is ∞ -positively biased, then any approximately optimal policy to C µ (˜ r ) (which includes the policies learned by offline RL, as shown earlier) can achieve high return in the true reward r . We also can view the degree of positiveness as reflecting whether ˜ r provides a similar ranking as r , among policies within the support of µ . When there is a positive bias, offline RL can learn with ˜ r to perform well under r , even when ˜ r is not aligned with r globally. This is in contrast to online RL, which requires global alignment due to the exploratory nature of online RL.

Examples We provide a few examples of positive data bias. (See Appendix D.4 for proofs.)

The distribution d π induced by any policy π is ∞ -positively biased for any rewards resulting from potential-based reward shaping [41] since it provides the same ranking for all policies.

For an IL setup, the data distribution µ is ∞ -positively biased for any rewards ˜ r : S × A → R if the data is generated by the optimal policy (i.e., µ = d π ⋆ ). This is because following the optimal

policy is the only way to stay within the data support. In Section 4, we empirically show that offline RL algorithms can learn good policies with wrong rewards on D4RL expert datasets, which are collected by near-optimal policies.

A positive bias happens when longer trajectories in the data have smaller optimality gap. This is a generalized formal definition of the length bias mentioned in Section 3.1. This condition is typically satisfied when intervention is taken in the data collection process (despite the data collection policy being suboptimal), as in the motivating example in Fig. 1. Later in Section 4, we will investigate deeper into this kind of length bias empirically.

Remark We highlight that the positive data bias assumption in Definition 2 is different from assuming that the data is collected by expert policies, which is typical in the IL literature. Positive data bias assumption can hold in cases when data is generated by highly suboptimal policies, which we observe in the hopper example from Section 1. On the other hand, there are also cases where IL can learn well, while positive data bias does not exist (e.g., when learning from data collected by a stochastic expert policy that covers highly suboptimal actions with small probabilities).

3.4 Summary: Offline RL is Doubly Robust and Intrinsically Safe

We have discussed the survival instinct from pessimism and the potential positive data bias in common datasets. In short, survival instinct enables offline RL algorithms to learn policies that benefit from a favorable inductive bias of staying within the support of a positive data distribution. As a result, offline RL becomes robust to reward mis-specification (namely, Theorem 1).

Below we discuss some direct implications of Theorem 1. First, we can view this phenomenon as a doubly robust property of offline RL. We borrow the name 'doubly robust' from the offline policy evaluation literature [42] to highlight the robustness of offline RL to reward mis-specification as an offline policy optimization approach.

Corollary 1. Under Assumption 1, offline RL can learn a near optimal policy, as long as the reward is correct, or the data has a positive implicit bias.

Theorem 1 also implies that offline RL is an intrinsically safe learning algorithm, unlike its counterpart online RL where additional penalties or constraints need to be explicitly modelled [43-47].

Corollary 2. If µ only covers safe states and there exists a safe policy staying within the support of µ , then the policy of offline RL only visits safe states with high probability (see Theorem 1).

We should note that covering safe states is a very mild assumption in the safe RL literature [45, 48], e.g., compared with all (data) states have a safe action. It does not require the data collection policies that generate µ are safe. This condition can be easily satisfied by filtering out unsafe states in post processing. The existence of a safe policy is also mild and common assumption. In Section 4.2, we validate this inherent safety property on offline SafetyGymnasium [49], an offline safe RL benchmark.

4 Experiments

We conduct two group of experiments. In Section 4.1, we conduct large scale experiment, showing that multiple offline RL algorithms can be robust to reward mis-specification on a variety of datasets. In Section 4.2, we experimentally validate the inherent safety of offline RL algorithms stated in Corollary 2. We show that admissible offline RL algorithms, without modifications, can achieve state-of-the-art performance on an offline safe RL benchmark [49].

4.1 Robustness to Mis-specified Rewards

We empirically study the performance of offline RL algorithms under wrong rewards in a variety of tasks. We use the same set of wrong rewards as in Fig. 1. 1) zero: the zero reward, 2) random: labeling each transition with a reward value randomly sampled from Unif [0 , 1] , and 3) negative: the negation of true reward. We consider five offline RL algorithms, ATAC [2], PSPI [7], IQL [8], CQL [9] and decision transformer (DT) 4 [10]. We deliberately choose offline RL algorithms to cover those that are provably pessimistic [2, 7] and those that are popular among practitioners [8, 9], as well as an unconventional offline RL algorithm [10]. We consider a variety of tasks from D4RL [1] and Meta-World [11] ranging from safety-critical tasks (i.e., the agent dies when reaching bad states),

100

ATAC

50

100

PSPI

50

0

rand

med

med-replay med-exp

exp

rand

med

med-replay med-exp

exp

rand

med

med-replay med-exp

exp

hopper

walker2d

halfcheetah

behavior

BC

original

zero

random

negative

100

IQL

50

100

CQL

50

100

DT

50

0

rand

med

med-replay med-exp

exp

rand

med

med-replay med-exp

exp

rand

med

med-replay med-exp

exp

walker2d

halfcheetah

hopper

goal-oriented tasks, and tasks that belong to neither. We train a total of around 16 k offline RL agents (see Appendix C.8). Please see Appendix C for details.

Messages We would like to convey three main messages through our experiments. First, implicit data bias can exist naturally in a wide range of datasets, and offline RL algorithms that are sufficiently pessimistic can leverage such a bias to succeed when given wrong rewards. Second, offline RL algorithms, regardless of how pessimistic they are, become sensitive to reward when the data does not possess a positive bias. Third, offline RL algorithms without explicit pessimism, e.g., IQL [8], can sometimes still be pessimistic enough to achieve good performance under wrong rewards.

Remark on negative results We consider 'negative' results, i.e., when offline RL fails under wrong rewards, as important as the positive ones. Since they tell us how positive data bias can be broken or avoided. We hope our study can provide insights to researchers who are interested in actively incorporating positive bias in data collection, as well as who hope to design offline RL benchmarks specifically with or without positive data bias.

4.1.1 Locomotion Tasks from D4RL

Weevaluate offline RL algorithms on three locomotion tasks, hopper , walker2d 5 and halfcheetah , from D4RL [1], a widely used offline RL benchmark. For each task, we consider five datasets with different qualites: random , medium , medium-replay , medium-expert , and expert . We refer readers to [1] for the construction of these datasets. We measure policy performance in D4RL normalized scores [1]. We provide three baselines 1) behavior: normalized score directly computed from the dataset, 2) BC: the behavior cloning policy, and 3) original: the policy produced by the offline RL algorithm using the original dataset (with true reward). The normalized scores for baselines and offline RL algorithms with wrong rewards are presented in Fig. 3. The exact numbers can be found in tables in Appendix C.

Positive bias exists in some D4RL datasets. We visualize the length bias of D4RL datasets in Fig. 4. Here are our main observations. 1) Most datasets for hopper and walker2d , with the

random

medium

medium-replay

medium-expert

expert

100

100

100

100

100

hopper

50

50

50

50

50

0

0

1000

0

0

1000

0

0

1000

0

0

1000

0

0

1000

100-

100

100

100

100

walker2d

50

50

50

50

50

0

0

1000

0

0

1000

0

0

1000

0

0

1000

0

0

1000

100

100

100

100

100

halfcheetah50

50

50

50

50

0

0

1000

0

0

1000

0

0

1000

0

0

1000

0

0

1000

exception of the medium-replay datasets, have a strong length bias, where longer trajectories have higher returns. This length bias is due to the safety-critical nature of hopper and walker2d tasks, as the trajectories get terminated when reaching bad states. The length bias is especially salient in hopper-medium , where the normalized score is almost proportional to episode length. 2) The medium-replay datasets for hopper and walker2d have more diverse behavior, so the bias is smaller. 3) All halfcheetah datasets do not have length bias, as they all have the same length of 1000 . 4) hopper-expert dataset has a length bias, while walker2d and halfcheetah-expert datasets do not have an obvious length bias. However, it is worth noting that all expert datasets have an IL-type positive bias as discussed in Section 3.3.

Offline RL can learn good policies with wrong rewards on datasets with strong length bias. On datasets with strong length bias ( hopper-random , hopper-medium and walker2d-medium ), we observe that ATAC and PSPI with wrong rewards generally produce well-performing policies, in a few cases even out-performing the policies learned from the true reward (original in Fig. 4). DT is mostly insensitive to reward quality. IQL and CQL with wrong rewards can sometimes achieve good performance; among the two, we find IQL to be more robust to wrong rewards and CQL with wrong rewards almost fails completely on all walker2d datasets.

Offline RL can learn good policies with wrong rewards on expert datasets In Section 3, we point out that the data has a positive bias when it is generated by the optimal policy. In Fig. 3, we observe that all offline RL algorithms, when trained with wrong rewards, can achieve expert-level performance on walker2d and halfcheetah-expert datasets. ATAC, PSPI and DT perform well on the hopper-expert dataset while IQL and CQL receive lower scores when using wrong rewards.

Offline RL needs stronger reward signals on datasets with diverse behavior policy. The medium-replay datasets of hopper and walker2d are generated from multiple behavior policies. Due to their diverse nature, they are a multiple ways to stay within data support in a long term. As a result, the survival instinct of offline RL by itself is not sufficient to guarantee good performance. Here algorithms with wrong rewards generally under-perform the policies trained with true reward. As datasets have a more diverse coverage, they can be less positively biased and offline RL requires a stronger reward signal to differentiate good survival policies and the bad ones. Practically speaking, when high-quality reward is not available, a diverse dataset can even hurt offline RL performance. This is contrary to the common belief that larger data support is more preferable [26-28].

Offline RL requires good reward to perform well on datasets without length bias. In all four halfcheetah datasets, the trajectories all have the same length, as is demonstrated in Fig. 4. This means that there is no data bias. We observe that all algorithms with wrong rewards at best perform similarly as the behavior and BC policies in most cases; this is an imitation-like effect due to the survival instinct. In the halfcheetah-medium-expert dataset, since there are a variety of surviving trajectories, with score from 0 to near 100 , we observe that the performance of the resulting policy degrades as data reward becomes more different from the true reward.

Offline RL algorithms with provable pessimism produce safer policies. For hopper and walker2d , we observe that policies learned by ATAC and PSPI, regardless of the reward, can keep the agent from falling for longer. The episode lengths of IQL and CQL policies are often

102

101

100

rand

med

med-replay med-exp

exp

rand

med

med-replay med-exp

exp

rand

med

med-replay med-exp

exp

hopper

walker2d

halfcheetah

comparable to that of the behavior policies. We provide statistics of episode lengths in Appendix C. In Section 4.2, we provide further results validating the safety properties of ATAC and PSPI.

On empirically estimating positive data bias Inspired by Definition 2, we propose an empirical estimate of positive data bias. Let J zero , J rand and J neg be the return (with respect to the true reward) of an offline RL algorithm learned with zero, random, and negative rewards, respectively. Then, given an estimated optimal return ˆ J ⋆ , we propose to empirically estimate the positive data bias by

In Fig. 5, we visualize the estimated positive data bias of all 15 D4RL datasets given by ATAC. Estimated positive data bias of all D4RL and Meta-World datasets is given in Fig. 9.

Remark on benchmarking offline RL Our observations on the popular D4RL datasets raise an interesting question for benchmarking offline RL algorithms. An implicit positive data bias can give certain algorithms a hidden advantage, as they can already achieve good performance without using the reward. Without controlling data bias, it is hard to differentiate whether the performance of those algorithms are due to their ability to maximize returns or simply due to their survival instinct.

4.1.2 Goal-oriented Manipulation Tasks from Meta-World

We study goal-oriented manipulation tasks from the Meta-World benchmark [11]. We consider 15 goal-oriented tasks from Meta-World: the 10 tasks from the MT-10 set and the 5 testing tasks from the ML-45 set. For each task, we generate a dataset of 110 trajectories using the scripted policies provided by [11]; 10 are produced by the scripted policy, and 100 are generated by perturbing the scripted policy with a zero-mean Gaussian noise of standard deviation 0 . 5 . In the datasets, unsuccessful trajectories receive a time-out signal after 500 steps (maximum episode length for Meta-World).

For each task, we measure the success rate of policies for 50 new goals unseen during training. Similar to D4RL experiments, we consider BC policies and policies trained with the true reward (original) as baselines. Since the training dataset is generated for a different set of goals, we do not compare the success rate of learned policies with the success rate in data.

Goal-oriented problems have data bias. Goal-oriented problems are fundamentally different from safety-critical tasks (such as hopper and walker2d ). A good goal-oriented policy should reach the goal as fast as possible rather than wandering around until the end of an episode. But another form of length bias still exists here, as successful trajectories are labeled with a termination signal that indicates an unbounded length and failed trajectories have a bounded length (see Section 3.1).

Offline RL can learn with wrong rewards on goal-oriented tasks. The success rate of learned policies with different rewards are shown in Fig. 6. We observe that ATAC and PSPI with wrong rewards generally achieve comparable or better success rate than BC policies. The exceptions are often due to that ATAC or PSPI, even with the true reward, does not work as well as BC in those tasks, e.g. drawer-open, bin-picking and box-close. This could be caused by overfitting or optimization errors (due to challenges of dynamics programming over long horizon problems). In a number of tasks, such as peg-insert-side, push and reach, ATAC and PSPI with wrong rewards can out-perform BC by a margin. This shows that data collection for goal-oriented problems have a positive bias that offline RL algorithms can succeed without true reward. This is remarkable as when learning with random and negative rewards, unsuccessful trajectories are long and generally have significantly higher return, as reward for each step is non-negative. The offline RL algorithms need to be sufficiently pessimistic and be able to plan over a long horizon to propagate pessimism to the unsuccessful data trajectories. For IQL, there is a gentle performance decrease as the reward becomes more different than the true reward, even though policies learned with wrong rewards are not much worse than BC in many cases. CQL shows robustness to reward in a few tasks such as button-press, drawer-close and door-unlock. Interestingly, DT performs almost uniformly across all rewards, potentially because DT does not explicitly maximize returns.

1.0

ATAC

0.5

0.0

1.0

PSPI

0.5

0.0

button

Joop

drawer

drawer peg insert

pick

push

reach

window

window

box

oop

door

hand

press

open

close

open

side

place

close

open

picking

close

lock

unlock

insert

BC

original

zero

random

negative

1.0

IQL

0.5

0.0

1.0

CQL

0.5

0.0

1.0

DT

0.5

0.0

button

door

drawer

drawer peginsert

pick

push

reach

window

window

box

door

door

hand

press

open

close

open

side

place

open

picking

lock

unlock

insert

Remark on BC performance It is noticeable that BC policies in general achieve high performance, in many tasks often out-performing offline RL policies learned with true reward. This effect has also been observed in existing work on similar tasks [50]. We would like to clarify that the goal of our experiments is to study the behavior of offline RL algorithms when trained with wrong rewards, rather than showing that offline RL performs better than BC. We refer interested readers to existing studies [50, 51] on when using offline RL algorithms is or is not preferable over BC.

4.2 Inherent Safety Properties of Offline RL Algorithms

We show offline RL algorithms (without any modifications) can behave as a safe RL algorithm. We conduct experiments on offline SafetyGymnasium [49] using ATAC [2] and PSPI [7]. We first remove all transitions with non-zero cost and then run offline RL algorithms on the filtered datasets.

The results are summarized in Table 6 in Appendix C.6. We observe that offline RL with this naïve data filtering strategy (using less data) can achieve comparable performance as the per-task best performing state-of-the-art offline safe RL algorithm , which uses the full dataset and has knowledge of the cost target. Our agents in general incur low cost except for the circle tasks. We hypothesize that learning a safe policy from the circle datasets is hard, as other baselines also struggle to produce safe policies. Note that these are preliminary results, and better performance might be achievable through using a more sophisticated data filtering strategy. Please see Appendix C.6 for experiment details.

5 Concluding Remarks

We present unexpected results on the robustness of offline RL to reward mis-specification. We show that this property originates from the interaction between the survival instinct of offline RL and hidden positive biases in common data collection processes. Our findings suggest extra considerations should be taken when interpreting offline RL results and designing future benchmarks. In addition, our findings open a new space for offline RL research: the possibility of designing algorithms that proactively leverage the survival instinct to learn policies for domains where rewards are nontrivial to specify or unavailable. Please see Appendix A for a discussion on limitations and broader impacts.

References

J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine, 'D4rl: Datasets for deep data-driven reinforcement learning,' arXiv preprint arXiv:2004.07219 , 2020.

C.-A. Cheng, T. Xie, N. Jiang, and A. Agarwal, 'Adversarially trained actor critic for offline reinforcement learning,' in International Conference on Machine Learning , PMLR, 2022.

J. Leike, M. Martic, V. Krakovna, P. A. Ortega, T. Everitt, A. Lefrancq, L. Orseau, and S. Legg, 'Ai safety gridworlds,' arXiv preprint arXiv:1711.09883 , 2017.

D. Hadfield-Menell, S. Milli, P. Abbeel, S. J. Russell, and A. Dragan, 'Inverse reward design,' Advances in neural information processing systems , vol. 30, 2017.

B. Ibarz, J. Leike, T. Pohlen, G. Irving, S. Legg, and D. Amodei, 'Reward learning from human preferences and demonstrations in atari,' Advances in neural information processing systems , vol. 31, 2018.

A. Pan, K. Bhatia, and J. Steinhardt, 'The effects of reward misspecification: Mapping and mitigating misaligned models,' in International Conference on Learning Representations , 2022.

T. Xie, C.-A. Cheng, N. Jiang, P. Mineiro, and A. Agarwal, 'Bellman-consistent pessimism for offline reinforcement learning,' Advances in neural information processing systems , vol. 34, pp. 6683-6694, 2021.

I. Kostrikov, A. Nair, and S. Levine, 'Offline reinforcement learning with implicit q-learning,' in International Conference on Learning Representations , 2021.

A. Kumar, A. Zhou, G. Tucker, and S. Levine, 'Conservative q-learning for offline reinforcement learning,' Advances in Neural Information Processing Systems , vol. 33, pp. 1179-1191, 2020.

L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch, 'Decision transformer: Reinforcement learning via sequence modeling,' Advances in neural information processing systems , vol. 34, pp. 15084-15097, 2021.

T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine, 'Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning,' in Conference on robot learning , pp. 1094-1100, PMLR, 2020.

Y. Jin, Z. Yang, and Z. Wang, 'Is pessimism provably efficient for offline rl?,' in International Conference on Machine Learning , pp. 5084-5096, PMLR, 2021.

Y. Liu, A. Swaminathan, A. Agarwal, and E. Brunskill, 'Provably good batch off-policy reinforcement learning without great exploration,' Advances in neural information processing systems , vol. 33, pp. 1264-1274, 2020.

M. Uehara and W. Sun, 'Pessimistic model-based offline reinforcement learning under partial coverage,' in International Conference on Learning Representations , 2022.

J. Chang, M. Uehara, D. Sreenivas, R. Kidambi, and W. Sun, 'Mitigating covariate shift in imitation learning via offline data with partial coverage,' Advances in Neural Information Processing Systems , vol. 34, pp. 965-979, 2021.

Y. Ma, A. Shen, D. Jayaraman, and O. Bastani, 'Versatile offline imitation from observations and examples via regularized state-occupancy matching,' in International Conference on Machine Learning , pp. 14639-14663, PMLR, 2022.

X. Zhang, Y. Chen, X. Zhu, and W. Sun, 'Corruption-robust offline reinforcement learning,' in International Conference on Artificial Intelligence and Statistics , pp. 5757-5773, PMLR, 2022.

F. Wu, L. Li, H. Zhang, B. Kailkhura, K. Kenthapadi, D. Zhao, and B. Li, 'Copa: Certifying robust policies for offline reinforcement learning against poisoning attacks,' in International Conference on Learning Representations , 2023.

Z. Zhou, Z. Zhou, Q. Bai, L. Qiu, J. Blanchet, and P. Glynn, 'Finite-sample regret bound for distributionally robust offline tabular reinforcement learning,' in International Conference on Artificial Intelligence and Statistics , pp. 3331-3339, PMLR, 2021.

X. Ma, Z. Liang, L. Xia, J. Zhang, J. Blanchet, M. Liu, Q. Zhao, and Z. Zhou, 'Distributionally robust offline reinforcement learning with linear function approximation,' arXiv preprint arXiv:2209.06620 , 2022.

K. Panaganti, Z. Xu, D. Kalathil, and M. Ghavamzadeh, 'Robust reinforcement learning using offline data,' in Advances in Neural Information Processing Systems , 2022.

H. Le, C. Voloshin, and Y. Yue, 'Batch policy learning under constraints,' in International Conference on Machine Learning , pp. 3703-3712, PMLR, 2019.

H. Xu, X. Zhan, and X. Zhu, 'Constraints penalized q-learning for safe offline reinforcement learning,' in AAAI , 2022.

J. Lee, C. Paduraru, D. J. Mankowitz, N. Heess, D. Precup, K.-E. Kim, and A. Guez, 'Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation,' in International Conference on Learning Representations , 2022.

D. Shin, A. D. Dragan, and D. S. Brown, 'Benchmarks and algorithms for offline preferencebased reward learning,' Transactions on Machine Learning Research , 2023.

R. Munos, 'Error bounds for approximate policy iteration,' in International Conference on Machine Learning , vol. 3, pp. 560-567, Citeseer, 2003.

J. Chen and N. Jiang, 'Information-theoretic considerations in batch reinforcement learning,' in International Conference on Machine Learning , pp. 1042-1051, PMLR, 2019.

D. Yarats, D. Brandfonbrener, H. Liu, M. Laskin, P. Abbeel, A. Lazaric, and L. Pinto, 'Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning,' arXiv preprint arXiv:2201.13425 , 2022.

S. Fujimoto and S. S. Gu, 'A minimalist approach to offline reinforcement learning,' Advances in neural information processing systems , vol. 34, pp. 20132-20145, 2021.

Y. Wu, G. Tucker, and O. Nachum, 'Behavior regularized offline reinforcement learning,' arXiv preprint arXiv:1911.11361 , 2019.

P. Rashidinejad, B. Zhu, C. Ma, J. Jiao, and S. Russell, 'Bridging offline reinforcement learning and imitation learning: A tale of pessimism,' Advances in Neural Information Processing Systems , vol. 34, pp. 11702-11716, 2021.

T. Yu, G. Thomas, L. Yu, S. Ermon, J. Y. Zou, S. Levine, C. Finn, and T. Ma, 'Mopo: Modelbased offline policy optimization,' Advances in Neural Information Processing Systems , vol. 33, pp. 14129-14142, 2020.

R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims, 'Morel: Model-based offline reinforcement learning,' Advances in neural information processing systems , vol. 33, pp. 2181021823, 2020.

T. Xie, M. Bhardwaj, N. Jiang, and C.-A. Cheng, 'Armor: A model-based framework for improving arbitrary baseline policies with offline data,' arXiv preprint arXiv:2211.04538 , 2022.

M. Rigter, B. Lacerda, and N. Hawes, 'Rambo-RL: Robust adversarial model-based offline reinforcement learning,' Advances in neural information processing systems , 2022.

T. Yu, A. Kumar, R. Rafailov, A. Rajeswaran, S. Levine, and C. Finn, 'Combo: Conservative offline model-based policy optimization,' Advances in neural information processing systems , vol. 34, pp. 28954-28967, 2021.

W. Zhan, B. Huang, A. Huang, N. Jiang, and J. Lee, 'Offline reinforcement learning with realizability and single-policy concentrability,' in Conference on Learning Theory , pp. 27302775, PMLR, 2022.

D. J. Foster, A. Krishnamurthy, D. Simchi-Levi, and Y. Xu, 'Offline reinforcement learning: Fundamental barriers for value function approximation,' in Conference on Learning Theory , pp. 3489-3489, PMLR, 2022.

E. Altman, Constrained Markov decision processes , vol. 7. CRC press, 1999.

M. Bhardwaj, T. Xie, B. Boots, N. Jiang, and C.-A. Cheng, 'Adversarial model for offline reinforcement learning,' arXiv preprint arXiv:2302.11048 , 2023.

A. Y. Ng, D. Harada, and S. Russell, 'Policy invariance under reward transformations: Theory and application to reward shaping,' in International Conference on Machine Learning , vol. 99, pp. 278-287, Citeseer, 1999.

M. Dudík, J. Langford, and L. Li, 'Doubly robust policy evaluation and learning,' in Proceedings of the 28th International Conference on International Conference on Machine Learning , pp. 1097-1104, 2011.

J. Achiam, D. Held, A. Tamar, and P. Abbeel, 'Constrained policy optimization,' in International conference on machine learning , pp. 22-31, PMLR, 2017.

A. Wachi and Y. Sui, 'Safe reinforcement learning in constrained markov decision processes,' in International Conference on Machine Learning , pp. 9797-9806, PMLR, 2020.

N. C. Wagener, B. Boots, and C.-A. Cheng, 'Safe reinforcement learning using advantage-based intervention,' in International Conference on Machine Learning , pp. 10630-10640, PMLR, 2021.

S. Paternain, M. Calvo-Fullana, L. F. Chamon, and A. Ribeiro, 'Safe policies for reinforcement learning via primal-dual methods,' IEEE Transactions on Automatic Control , 2022.

H.-A. Nguyen and C.-A. Cheng, 'Provable reset-free reinforcement learning by no-regret reduction,' arXiv preprint arXiv:2301.02389 , 2023.

A. Hans, D. Schneegaß, A. M. Schäfer, and S. Udluft, 'Safe exploration for reinforcement learning.,' in ESANN , pp. 143-148, Citeseer, 2008.

Z. Liu, Z. Guo, H. Lin, Y. Yao, J. Zhu, Z. Cen, H. Hu, W. Yu, T. Zhang, J. Tan, et al. , 'Datasets and benchmarks for offline safe reinforcement learning,' arXiv preprint arXiv:2306.09303 , 2023.

G. Zhou, L. Ke, S. Srinivasa, A. Gupta, A. Rajeswaran, and V. Kumar, 'Real world offline reinforcement learning with realistic data source,' arXiv preprint arXiv:2210.06479 , 2022.

A. Kumar, J. Hong, A. Singh, and S. Levine, 'Should i run offline reinforcement learning or behavioral cloning?,' in International Conference on Learning Representations , 2022.

T. Gebru, J. Morgenstern, B. Vecchione, J. W. Vaughan, H. Wallach, H. D. Iii, and K. Crawford, 'Datasheets for datasets,' Communications of the ACM , vol. 64, no. 12, pp. 86-92, 2021.

A. Singh, A. Yu, J. Yang, J. Zhang, A. Kumar, and S. Levine, 'Cog: Connecting new skills to past experience with offline reinforcement learning,' in Conference on Robot Learning , PMLR, 2020.

H. Hu, Y. Yang, Q. Zhao, and C. Zhang, 'The provable benefits of unsupervised data sharing for offline reinforcement learning,' in International Conference on Learning Representations , 2023.

A. Li, B. Boots, and C.-A. Cheng, 'Mahalo: Unifying offline reinforcement learning and imitation learning from observations,' arXiv preprint arXiv:2303.17156 , 2023.

T. Yu, A. Kumar, Y. Chebotar, K. Hausman, C. Finn, and S. Levine, 'How to leverage unlabeled data in offline reinforcement learning,' in International Conference on Machine Learning , pp. 25611-25635, PMLR, 2022.

J. Li, X. Hu, H. Xu, J. Liu, X. Zhan, Q.-S. Jia, and Y .-Q. Zhang, 'Mind the gap: Offline policy optimization for imperfect rewards,' in International Conference on Learning Representations , 2023.

X. B. Peng, A. Kumar, G. Zhang, and S. Levine, 'Advantage-weighted regression: Simple and scalable off-policy reinforcement learning,' arXiv preprint arXiv:1910.00177 , 2019.

K. Zolna, A. Novikov, K. Konyushkova, C. Gulcehre, Z. Wang, Y. Aytar, M. Denil, N. de Freitas, and S. Reed, 'Offline learning from demonstrations and unlabeled experience,' arXiv preprint arXiv:2011.13885 , 2020.

H. Xu, X. Zhan, H. Yin, and H. Qin, 'Discriminator-weighted offline imitation learning from suboptimal demonstrations,' in International Conference on Machine Learning , pp. 2472524742, PMLR, 2022.

G.-H. Kim, S. Seo, J. Lee, W. Jeon, H. Hwang, H. Yang, and K.-E. Kim, 'Demodice: Offline imitation learning with supplementary imperfect demonstrations,' in International Conference on Learning Representations , 2021.

Z. Zhu, K. Lin, B. Dai, and J. Zhou, 'Off-policy imitation learning from observations,' Advances in Neural Information Processing Systems , vol. 33, pp. 12402-12413, 2020.

B. Dai, O. Nachum, Y. Chow, L. Li, C. Szepesvári, and D. Schuurmans, 'Coindice: Off-policy confidence interval estimation,' Advances in neural information processing systems , vol. 33, pp. 9398-9411, 2020.

R. Kidambi, J. Chang, and W. Sun, 'Mobile: Model-based imitation learning from observation alone,' Advances in Neural Information Processing Systems , vol. 34, pp. 28598-28611, 2021.

S. Yue, G. Wang, W. Shao, Z. Zhang, S. Lin, J. Ren, and J. Zhang, 'CLARE: Conservative modelbased reward learning for offline inverse reinforcement learning,' in International Conference on Learning Representations , 2023.

B. D. Ziebart, A. L. Maas, J. A. Bagnell, A. K. Dey, et al. , 'Maximum entropy inverse reinforcement learning.,' in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 8, pp. 1433-1438, Chicago, IL, USA, 2008.

M. Smith, L. Maystre, Z. Dai, and K. Ciosek, 'A strong baseline for batch imitation learning,' arXiv preprint arXiv:2302.02788 , 2023.

Y. Chen, X. Zhang, K. Zhang, M. Wang, and X. Zhu, 'Byzantine-robust online and offline distributed reinforcement learning,' in International Conference on Artificial Intelligence and Statistics , pp. 3230-3269, PMLR, 2023.

R. Yang, C. Bai, X. Ma, Z. Wang, C. Zhang, and L. Han, 'Rorl: Robust offline reinforcement learning via conservative smoothing,' in Advances in Neural Information Processing Systems , 2022.

L. Shi and Y. Chi, 'Distributionally robust model-based offline reinforcement learning with near-optimal sample complexity,' arXiv preprint arXiv:2208.05767 , 2022.

Z. Liu, Z. Guo, Y. Yao, Z. Cen, W. Yu, T. Zhang, and D. Zhao, 'Constrained decision transformer for offline safe reinforcement learning,' arXiv preprint arXiv:2302.07351 , 2023.

N. Polosky, B. C. Da Silva, M. Fiterau, and J. Jagannath, 'Constrained offline policy optimization,' in International Conference on Machine Learning , pp. 17801-17810, PMLR, 2022.

W. Xu, Y. Ma, K. Xu, H. Bastani, and O. Bastani, 'Uniformly conservative exploration in reinforcement learning,' in International Conference on Artificial Intelligence and Statistics , pp. 10856-10870, PMLR, 2023.

S. Lee, Y. Seo, K. Lee, P. Abbeel, and J. Shin, 'Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble,' in Conference on Robot Learning , pp. 1702-1712, PMLR, 2022.

W. Yu, C. K. Liu, and G. Turk, 'Policy transfer with strategy optimization,' in International Conference on Learning Representations , 2019.

H. Van Hasselt, Y. Doron, F. Strub, M. Hessel, N. Sonnerat, and J. Modayil, 'Deep reinforcement learning and the deadly triad,' arXiv preprint arXiv:1812.02648 , 2018.

R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction . MIT press, 2018.

M. Riedmiller, 'Neural fitted q iteration-first experiences with a data efficient neural reinforcement learning method,' in Machine Learning: ECML 2005: 16th European Conference on Machine Learning, Porto, Portugal, October 3-7, 2005. Proceedings 16 , pp. 317-328, Springer, 2005.

Contents

G Grid World

A Limitations and Broader Impacts

Limitations We would like to point out that our results do not suggest that offline RL can always learn with wrong rewards (since the data may not have the needed positive bais). This is shown in our experimental results. In addition, our theoretical analysis makes certain simplifications (e.g., ignoring the optimization difficulty of finding a survival policy), which may not hold in practice. We experimented with a limited number of offline RL algorithms and benchmarks. It would be interesting to see if similar experimental results can be obtained by, e.g., model-based offline RL algorithms and on other types of dataset such as the ones which use image observations. Future work may also explore approaches to quantifying positive data bias without having to run offline RL algorithms.

Broader Impacts Our discovery might shed light on a potential positive impact of offline RL. By survival instinct, offline RL may be able to train sequential-decision policies that have good behaviors without needing to collect negative data. This is different from the common belief that RL can learn to behave well only if it has seen both positive (success) and negative (failure) data. This ability to learn from one-sided data is important especially for applications where collecting negative data is unethical, e.g., learning not to produce hateful speech by first collecting hateful speech, learning not to harm patients in medical applications by first harming some, etc.

On the flip side, also by survival instinct, offline RL can be prone to existing societal (e.g., gender, racial) biases in data, and, moreover, such a bias cannot be easily corrected by relabeling data with a different reward. As a result, when using an offline RL algorithm, more strategic thinking on data collection might be needed. We encourage researchers and practitioners to collect datasets using methodology such as those proposed in [52] and provide details such as how data was collected and cleaned so that users can assess whether it is appropriate to train offline RL with these datasets.

57

B Related Work

Offline RL Offline RL studies the problem of return maximization given an offline dataset. Offline RL algorithms can be broadly classified into model-based approaches, e.g., [32, 33, 14, 35, 34], and model-free approaches, e.g., [12, 29, 7, 2, 9, 8]. Since the agent needs to learn without online data collection, offline RL becomes more challenging when the offline dataset does not provide good coverage over the state-action distribution of all feasible policies. There have been two common strategies to handle insufficient coverage, behavior regularization approaches, e.g., [29, 30, 8], which restricts the learned policy to be close to the behavior policy, and value regularization approaches, e.g., [12, 13, 7, 2, 9, 34] which provide a pessimistic value estimates for the learned policy. Both approaches can be viewed as optimizing for a performance lower bound of the agent's return.

Offline RL commonly assumes compatibility between offline dataset and MDP, i.e., the offline dataset should be generated from the task MDP of interest. Our work is distinctive in the offline RL literature as we study the behavior of existing offline RL algorithms when data reward differs from the true MDPreward. Existing tools for analyzing offline RL are not directly applicable to our setting: policies with good performance under data reward, as guaranteed by such tools, do not necessarily attain high return under the true reward. Our novel analysis is made possible by studying the properties of offline data distribution, an aspect mostly neglected by existing work in the literature.

Offline RL with imperfect reward In the offline RL literature, there is a line of work studying scenarios where the reward is missing in a subset of the dataset [53-56] or when the data reward is misspecified [57]. These approaches propose to construct a new reward function and apply offline RL algorithms on the relabeled dataset. For example, [53, 56] label the missing reward with a constant minimum reward. [55, 54] learn a pessimistic reward function. [57] optimize for a reward function which minimizes the reward difference among expert data.

Different from this line of work, we are interested in running offline RL algorithms using the misspecified reward as it is . We show both theoretically and empirically that offline RL algorithms can produce well-performing policies with rewards different from the true reward as long as the underlying data distribution satisfies certain conditions. It can be an interesting future work to combine these techniques with our analysis to explore the robustness of offline RL algorithms under the class of learned reward functions.

After paper submission, we learned that [25] observed a similar robustness phenomena of offline RL. However, the authors did not provide a complete explanation. They found that offline RL algorithms such as AWR [58] empirically can learn well with random or zero rewards on D4RL locomotion datasets. For the expert datasets, they conjectured that this robustness may be due to the fact that offline RL algorithms behave like imitation learning on expert data, but they did not provide details of why offline RL algorithms have this robustness behavior on other datasets. Here we provide a formal theoretical proof to show that this robustness is due to the interaction between the survival instinct of offline RL and an implicit positive data bias. Our results support [25]'s conjecture on the expert dataset; we show that expert datasets generated by an optimal policy can theoretically be proved to be infinitely positively biased, and we also empirically estimate the positive bias on the D4RL datasets in Fig. 5, which agrees with the theoretical prediction. Our results further provide conditions for positive data bias beyond the expert data scenario, such as the length bias in Section 3. These new insights show why offline RL algorithms can learn good policies with wrong rewards, on datasets collected by non-expert suboptimal policies. Finally, we present strong empirical evidence to show this robustness phenomenon happens in a large number of existing benchmarks with multiple offline RL algorithms.

Offline IL Offline imitation learning (IL) can be considered as a special case of offline RL when reward is missing from the entire dataset, or when the agent has the largest uncertainty about the reward [2]. In offline IL, the learner is instead given the information on whether a transition is generated by an expert. The goal of offline IL is to produce a policy that has comparable performance with the expert. [59] learns a discriminator reward to classify expert and non-expert data, and apply offline RL algorithms with the learned reward. [60] uses the discriminator as the weight for weighted behavior cloning. [61, 16, 62] proposes DICE [63]-style algorithms to minimize the divergence between the state-action or state-(next state) occupancy measure of the learner and the expert. [15, 64] minimize state-action or state-(next state) distribution divergence under a pessimistic model. [65]

extends maximum entropy inverse RL [66] to an offline setting. [67] proposes to run offline RL under an indicator reward function on whether the data is expert data. [55] learns a pessimistic reward function where the expert data labeled as the maximum reward.

Our experiments with zero reward function is similar to an offline IL setting with the behavior policy being the expert. However, we show that offline RL algorithms can sometimes achieve significant performance gain over behavior and behavior cloning policies, which is a phenomenon that cannot be explained by existing analysis in offline IL. We provide an explanation to this puzzle: in Appendix D.4, we show that the data distribution is ∞ -positively biased (according our definition) for offline RL, when the data is generated by the optimal policy of the true MDP. This means that an admissible offline RL algorithm can achieve near-optimal performance with any reward function from its corresponding admissible reward class in this case. We empirically show that in Section 4.

Robust offline RL Robust offline RL [17, 68, 18, 21, 69, 19, 20, 70] aims to develop new offline RL algorithms for cases when the compatibility assumption does not hold, i.e., the offline dataset may not be generated from the task MDP. Among this line of work, [21, 19, 20, 70] study the robust MDP setting, where the dataset can be generated from any MDP within a δ -ball of a nominal MDP. An adversary can choose any MDP from the δ -ball when evaluating the learned policy. [17] considers when the adversary can modify ϵ fraction of transition tuples in the offline dataset, while [18] considers when up to K trajectories can be added in or removed from the original dataset. [68] uses the formulation of Byzantine-robust: when multiple offline datasets are presented to the learner, among them an α fraction are corrupted. [69] assumes the dataset is generated from the true MDP, but the state presented to the policy during test time can be a corrupted up to ϵ distance.

Our work is fundamentally different from this line of work in that we study the inherent and somewhat unintended robustness of offline RL algorithms. We show that, despite originally designed under the compatibility assumption, offline RL algorithms show strong robustness against perturbation of reward. Another salient difference is that robust offline RL typically requires an upper bound on the size of perturbation to the true or nominal MDP, and most of proposed algorithms assume knowledge of this upper bound. However, our work does not make assumptions on the size of perturbation and, moreover, the algorithms we study are not even aware of the existence of such perturbation. Indeed, we empirically show that offline RL algorithms, without any modifications, can succeed even when data reward is a constant or the negation of the true reward. Additionally, in this work, we consider the reward perturbation while most work in robust offline RL focuses more on dynamics perturbation.

Constrained and safe offline RL Our work is related to the literature of constrained and safe offline RL [22, 23, 71, 24, 72]. In constrained offline RL, the agent solves for a CMDP given an offline dataset { ( s, a, r, c, s ′ ) } . The goal of constrained offline RL is to produce a policy which maximizes the expected return while ensuring that constraint violation is below a given upper bound. [22] uses an offline policy evaluation oracle to solve for the Lagrangian relaxation of the CMDP. [23] constructs a pessimistic estimation of value and constraint violation. [24, 72] proposes DICE [63]style algorithms to solve for the optimal state-action occupancy measure. [71] proposes a variant of decision transformer [10] capable of producing policies for different constraint violation upper bounds during test time.

Our work, instead, considers a different scenario where the constraint is only given implicitly -through the support of the offline dataset. We assume that any transition in the dataset satisfies the constraint, which can be achieved through intervention during data collection. We show that offline RL can inherently produce approximately safe policies due to its survival instinct. Although our assumption seems more restrictive as in that we require a constraint violation-free dataset, we argue that our assumption is more practical in many safety critical scenarios: compared to executing unsafe actions, it is perhaps more reasonable to intervene before constraint violation happens. We believe that our findings open up new possibilities for applying offline RL as a safe RL oracle, which can be especially promising in an offline-online RL setup [73, 74].

C Experiment Details

In the experiments, we train a total of around 16 k offline RL agents (see Appendix C.8 for details). In this appendix, we provide details of experiment setup, implementation of algorithms, hyperparameter

(a) D4RL [1]

tuning, as well as additional experimental results and ablation studies. At the end of the appendix, we include Table 13 and Table 14 which contain the actual numbers used in generating Fig. 3 and Fig. 6.

C.1 Benchmark Datasets

In the experiments, we consider three tasks (15 datasets) from D4RL [1], 15 datasets collected from the Meta-World [11] benchmark, and 16 datasets from SafetyGymnasium [49]. The D4RL and Meta-World tasks are visualized in Fig. 7. Below we provide more details on these tasks.

D4RL Weconsider three locomotion tasks from D4RL [1], hopper , walker2d and halfcheetah 6 (see Fig. 7). For each task, we use five datasets of different behavior policy quality: random , medium , medium-replay , medium expert , and expert .

hopper: The goal of hopper is to move forward as fast as possible without falling down. An episode terminates when either of the three happens: 1) the height of the hopper is below a threshold, 2) the body of the hopper is too tilted, or 3) the velocity of any joint is too large. The original true reward function (which has been used to train online RL agents) is

✶ where ∆ x is the displacement in the forward direction, and τ is the applied torque. The true reward function is designed such that the hopper can receive bonus when staying alive and moving forward. We show that, for a few hopper datasets, offline RL agent can succeed without such a carefully designed reward, and even with the negation of the true reward.

walker2d: The walker2d task is similar to hopper , except that the walker2d agent has a higher degree of freedom. We observe similarly that offline RL can learn good behaviors without a reward that is specifically designed to encourage staying alive and moving forward.

halfcheetah: The halfcheetah task differs from the two tasks above since episodes here always have the same length and never terminates early. The true reward of halfcheetah is

Since the halfcheetah datasets (with the exception of the expert dataset) do not have a positive bias, we find that offline RL agent is sensitive to reward quality.

Meta-World We consider 15 goal-oriented manipulation tasks from [11], see Fig. 7. We use the 10 tasks from the MT-10 set and the 5 testing tasks from the ML-45 set. Each Meta-World task comes with a hand-crafted reward function that encourage progress toward task completion, e.g., when the end-effector grasps the target object, when the object is placed at the goal, etc. We refer readers to [11] for the specific reward functions used for each tasks. We make a few modifications to the Meta-World environment: 1) we shift and scale the true reward such that the agent receives a

reward of range [ -1 , 0] rather than [0 , 10] of the original environment, 2) we terminate an episode and mark the last state with a terminal flag when the goal is achieved. We show that, data collected by goal-oriented tasks can have a positive bias, and an offline RL agent can make use of such bias to learn the right behavior even with wrong rewards. We would like to clarify that the data generated by the original Meta-World environment (with no terminal signals) does not have a positive bias in general, since all trajectories, success or failure, would have a same length of 500 .

SafetyGymnasium We experiment with 16 datasets from offline SafetyGymnasium [49]. The 16 datasets are given by two embodiments, point and car , with 4 tasks, button , circle , goal , and push , each with 2 difficulty levels.

It is worth noting that [49] focuses on learning from a dataset which contains both safe and unsafe behaviors, while our results (Corollary 2) focus on offline RL's ability to produce safe policies given a dataset that only contains safe states. Therefore, we use a naïve filtering strategy of removing all transitions with non-zero cost and then run offline RL algorithms on the filtered datasets.

Many algorithms benchmarked in [49] use a cost target, which sets the maximum total cost allowed by safety. [49] uses three different cost targets, { 20 , 40 , 80 } , and reports the normalized total reward and cost for each algorithm averaged over the three targets. By contrast, standard offline RL algorithms do not use such a cost target. We report results of offline RL algorithms according to an 'effective cost target' of 34 . 29 , which is similar to how BC-All results are presented in [49].

C.2 Implementation Details

We use the implementation of ATAC and PSPI from https://github.com/chinganc/lightATAC. We make a small modification to the ATAC and PSPI policy class. Instead of a tanh -Gaussian policy, we use a scaled tanh -Gaussian policy, i.e., c · tanh( N ( µ, Σ)) with c ≥ 1 . We find scaled tanh -Gaussian policies to be more numerically stable when the data has a lot of on-boundary actions, i.e., actions that takes the value of either -1 or 1 on any dimension. For PSPI, we let the critic f to minimize for f ( s, π ) for all states from the dataset (rather than all initial states). We find it to provide better results when the effective horizon given by the discount factor is smaller than the actual episode length. We use the implementation of IQL and BC from https://github.com/gwthomas/IQL-PyTorch/, and the implementation of CQL from https://github.com/young-geng/CQL. For decision transformer, we use the implementation from https://github.com/kzl/decision-transformer. We feed the decision transformer with wrong rewards (consistent with data reward) during testing.

C.3 Hyperparameters

We use the default values (ones provided by the original paper or by the implementation we use) for most of the hyperparameters for all algorithms. For each algorithm, we tune one or two hyperparameters (with 4 combinations at most) which affect the degree of pessimism 7 . The values of hyperparamters of all algorithms are given in Table 1-4, where the choices of hyperparameters used in tuning are highlighted in blue. The tuned hyperparameter values for all experiments and all algorithms are provided in Table 9-12.

For D4RL and Meta-World experiments, we tune hyperparameter per each dataset and per reward label. During tuning, we evaluate each combination of hyperparameter(s) for 2 random seeds for D4RL and 3 random seeds for Meta-World. We choose the best-performing values and report the results of these hyperparameters over 10 new random seeds (not including the tuning seeds). We find the D4RL scores for the true reward to be close to what is reported in the original papers.

For SafetyGymnasium, we follow procedures from [49] and report results of the best hyperparameters over 3 random seeds. We take the safe agent with the highest cumulative reward if there is at least one safe agent, and take the agent with the lowest cumulative cost if there are no safe agents.

C.4 Episode Lengths for hopper and walker2d Tasks

We show the episode lengths for hopper and walker2d tasks in Fig. 8. We observe that policies learned by ATAC and PSPI can consistently keep the agent from falling down for a significant longer period than the behavior policies. The only exception is when running PSPI on hopper-medium-expert dataset with random reward. This is potentially due to the fact that we use only 2 seeds for hyperparameter tuning. The average normalized scores and episode lengths during tuning are 101 . 1 and 925 . 0 , respectively, higher than the final values of 58 . 8 and 546 . 0 . We observe that DT policies also tend to keep the agent alive for a longer number of steps than behavior policies.

For IQL and CQL, it is relatively rare for them to achieve long episode lengths when trained with wrong rewards. IQL and CQL, when using negative reward, can keep the agent from falling for longer in hopper-medium and hopper-medium-expert datasets. We would like to note that our hyperparameters are tuned based on normalized scores rather than episode lengths. The statistics of episode lengths are included in Table 5.

C.5 Effect of Removing Terminal Transitions

For hopper and walker2d tasks, we remove the terminal transitions, i.e., the transitions where the hopper or walker falls, from the datasets with modified rewards. This is to satisfy the safety

condition in Corollary 2. A terminal signal implicitly adds an absorbing 8 , but in this case, unsafe, state into the data distribution µ . This means that staying within data support in a long term would not provide safety, as the agent can stay within support by falling. For D4RL datasets in particular, we need to remove terminal transitions (rather than just the terminal flag) since terminal transitions in the original datasets do not contain valid next states. 9

This, however, means that there are two underlying variables: 1) the data reward, and 2) whether terminal transitions are removed, when comparing offline RL with wrong rewards and offline RL on the original dataset. To separate the effect of two variables, we study the performance of offline RL algorithms with the original true reward but the terminal transitions removed. The results 10 are listed in the No-terminal column in Table 13. We observe that ATAC and PSPI with the original true reward show similar results whether the terminal transitions are removed. IQL and CQL with true reward, interestingly, does significantly worse when the terminal transitions are removed. This is potentially due to the increased instability of target network [76, 77] when there are no terminal signals.

C.6 Results on Safety Gymnasium Datasets

In Table 6, we present results on point datasets and car from Safety Gymnasium [49]. For baselines, we select the best none-BC Algorithms from [49]: the best-performing offline safe RL agent selected

ATAC

1000

500

0

rand

med

med-replay

med-exp

exp

rand

med

med-replay

med-exp

exp

hopper

walker2d

PSPI

1000

500

0

rand

med

med-replay

med-exp

exp

rand

med

med-replay

med-exp

exp

hopper

walker2d

behavior

BC

original

zero

random

negative

IQL

1000

500

0

rand

med

med-replay

med-exp

exp

rand

med

med-replay

med-exp

exp

hopper

walker2d

CQL

1000

500

0

rand

med

med-replay

med-exp

exp

rand

med

med-replay

med-exp

exp

hopper

walker2d

1000

DT

500

0

rand

med

med-replay

med-exp

exp

rand

med

med-replay

med-exp

exp

walker2d

hopper

using the same criterion (safe agent with highest reward if there is at least one safe agent, and agent with lowest cost otherwise). We observe that ATAC and PSPI with a naïve filtering strategy can achieve comparable performance with state-of-the-art offline safe RL algorithms benchmarked in [49].

C.7 Ablation Study on CQL Architecture

Our results are generated by CQL with the architecture in Table 3, which is given by the original implementation. In this study, we examine the performance of CQL when using the same architecture as ATAC and PSPI, i.e., with 3 hidden layers and scaled tanh -Gaussian. The normalized scores for D4RL datasets and success rate for Meta-World tasks are included in Table 7 and Table 8, respectively. We find the performance of CQL with 3 hidden layers and scaled tanh -Gaussian policy, denoted as CQL 3 , to be similar with the performance of CQL with the original architecture.

C.8 Compute Usage

For ATAC, PSPI, IQL, CQL and BC, each run takes around 5 . 5 hours on an NC4as_T4_v3 Azure virtual machine for D4RL, Meta-World, and SafetyGymnasium experiments. For D4RL experiments (including ablation studies), we use a total of 5 (algorithms) × 15 (D4RL datasets) × 5 (reward types) × 4 (hyperparameters) × 2 (seeds) = 3000 runs for hyperparameter tuning and

±

±

±

±

6 (algorithms) × 15 (D4RL datasets) × 5 (reward types) × 10 (seeds) = 4500 runs for generating the final results. For Meta-World experiments, we use 5 (algorithms) × 15 (Meta-World datasets) × 4 (reward types) × 4 (hyperparameters) × 3 (seeds) = 3600 runs for hyperparameter tuning and 6 (algorithms) × 15 (Meta-World datasets) × 4 (reward types) × 10 (seeds) = 3600 runs for generating the final results. SafetyGymnasium experiments require a total of 2 (algorithms) × 16 (SafetyGymnasium datasets) × 4 (hyperparameters) × 3 (seeds) = 384 runs. This amount to 15084 runs or 82962 hours of training on NC4as_T4_v3 virtual machines.

For DT, we do not tune any parameters at large scale. Each run takes around 2 . 5 hours on an NC6s_v2 or ND6s Azure virtual machine. We train a total of 30 (D4RL and Meta-World datasets) × 4 (reward types) × 10 (seeds) = 1200 DT agents.

±

±

±

±

±

±

±

±

10²

10

10°

10²

101

100

10²

10

10°

10²

102

10°

10²

101

100

rand rand

rand rand

rand

D4RL Datasets med

walker2d med-replay med-exp

med med

med med

walker2d med-replaymed-exp

med-replay med-exp walker2d

med-replay med-exp walker2d

X

med-replaymed-exp walker2d

hopper med-replay med-exp

med-replay med-exp hopper

med-replay med-exp hopper

med-replay med-exp hopper

med-replay med-exp hopper

halfcheetah med

med-replay med-exp medmed-replay med-exp

halfcheetah med

halfcheetah med-replay med-exp

med-replay med-exp med

×

exp exp

exp exp

exp exp

exp

X

exp

X

exp

ATAC

exp

PSPI

exp

IQL

exp

CQL

exp

DT

exp rand

rand rand

rand rand

rand rand

rand rand

rand med

pau med

med med

med-replay med-exp

Meta-World Datasets

102

ATAC

101

100

button

drawer

drawer

peg insert

pick

ysnd

reach

window

box

door

hand

press

open

e

open

side

place

e

open

picking

lock

unlock

insert

102

PSPI

101

100

button

press

open

door

drawer

drawer

open

peginsert

side

place

pick

push

reach

window

window

open

picking

box

lock

door

insert

hand

102

IQL

101

100

button

drawer

drawer

peg insert

pick

ysnd

reach

window

window

box

oop

hand

press

open

open

side

place

open

picking

lock

insert

102

CQL

101

100

button

Joop

drawer

drawer

peginsert

pick

push

reach

window

window

box

door

hand

press

open

open

side

place

open

picking

lock

insert

10²

X

X

DT

10

100

button

press

open

oop

drawer

close

drawer

open

peginsert

side

place

pick

push

reach

window

close

window

open

picking

box

oop

lock

unlock

oop

insert

hand

Dataset

Original

Zero

Random

Negative

No-terminal

hopper-random

β

= 100

.

0

β

= 100

.

0

β

= 10

.

0

β

= 1

.

0

β

= 10

.

0

hopper-medium

β

= 10

.

0

β

= 100

.

0

β

= 100

.

0

β

= 1

.

0

β

= 100

.

0

hopper-medium-replay

β

= 100

.

0

β

= 1

.

0

β

= 1

.

0

β

= 0

.

1

β

= 100

.

0

hopper-medium-expert

β

= 1

.

0

β

= 10

.

0

β

= 10

.

0

β

= 1

.

0

β

= 1

.

0

hopper-expert

β

= 0

.

1

β

= 1

.

0

β

= 1

.

0

β

= 0

.

1

β

= 0

.

1

walker2d-random

β

= 10

.

0

β

= 0

.

1

β

= 1

.

0

β

= 0

.

1

β

= 100

.

0

walker2d-medium

β

= 10

.

0

β

= 1

.

0

β

= 1

.

0

β

= 0

.

1

β

= 100

.

0

walker2d-medium-replay

β

= 100

.

0

β

= 10

.

0

β

= 10

.

0

β

= 10

.

0

β

= 100

.

0

walker2d-medium-expert

β

= 10

.

0

β

= 10

.

0

β

= 1

.

0

β

= 0

.

1

β

= 10

.

0

walker2d-expert

β

= 1

.

0

β

= 100

.

0

β

= 0

.

1

β

= 0

.

1

β

= 1

.

0

halfcheetah-random

β

= 0

.

1

β

= 0

.

1

β

= 0

.

1

β

= 0

.

1

-

β

= 1

.

0

β

= 100

.

0

β

= 1

.

0

β

= 0

.

1

-

β

= 1

.

0

β

= 1

.

0

β

= 1

.

0

β

= 0

.

1

-

β

= 0

.

1

β

= 0

.

1

β

= 0

.

1

β

= 0

.

1

-

halfcheetah-expert

β

= 0

.

1

β

= 100

.

0

β

= 0

.

1

β

= 0

.

1

-

button-press

β

= 10

.

0

β

= 10

.

0

β

= 1

.

0

β

= 100

.

0

-

door-open

β

= 100

.

0

β

= 10

.

0

β

= 10

.

0

β

= 1

.

0

-

drawer-close

β

= 1

.

0

β

= 10

.

0

β

= 10

.

0

β

= 100

.

0

-

drawer-open

β

= 10

.

0

β

= 1

.

0

β

= 10

.

0

β

= 10

.

0

-

peg-insert-side

β

= 1

.

0

β

= 1

.

0

β

= 1

.

0

β

= 1

.

0

-

pick-place

β

= 1

.

0

β

= 10

.

0

β

= 1

.

0

β

= 1

.

0

-

push

β

= 1

.

0

β

= 1

.

0

β

= 1

.

0

β

= 1

.

0

-

reach

β

= 10

.

0

β

= 100

.

0

β

= 10

.

0

β

= 1

.

0

-

window-close

β

= 1

.

0

β

= 1

.

0

β

= 1

.

0

β

= 1

.

0

-

window-open

β

= 0

.

1

β

= 1

.

0

β

= 10

.

0

β

= 100

.

0

-

bin-picking

β

= 1

.

0

β

= 1

.

0

β

= 1

.

0

β

= 1

.

0

-

box-close

β

= 1

.

0

β

= 1

.

0

β

= 1

.

0

β

= 1

.

0

-

door-lock

β

= 10

.

0

β

= 1

.

0

β

= 1

.

0

β

= 0

.

1

-

door-unlock

β

= 100

.

0

β

= 10

.

0

β

= 10

.

0

β

= 10

.

0

-

hand-insert

β

= 10

.

0

β

= 1

.

0

β

= 1

.

0

β

= 1

.

0

-

point-button1

β

= 1

.

0

-

-

-

-

point-button2

β

= 1

.

0

-

-

-

-

point-circle1

β

= 0

.

1

-

-

-

-

point-circle2

β

= 0

.

1

-

-

-

-

point-goal1

β

= 0

.

1

-

-

-

-

point-goal2

β

= 1

.

0

-

-

-

-

point-push1

β

= 10

.

0

-

-

-

-

point-push2

β

= 1

.

0

-

-

-

-

car-button1

β

= 0

.

1

-

-

-

-

car-button2

β

= 1

.

0

-

-

-

-

car-circle1

β

= 0

.

1

-

-

-

-

car-circle2

β

= 100

.

0

-

-

-

-

car-goal1

β

= 10

.

0

-

-

-

-

β

= 1

.

0

-

-

-

-

car-push1

β

= 100

.

0

-

-

-

-

car-push2

β

= 0

.

1

-

-

-

-

±

±

±

±

±

±

±

±

D Technical Details of Section 3

Here we provide the details of our theory on the robustness of offline RL in Section 3.

Theorem 1. (Informal) Under Assumption 1 and certain regularity assumptions, if an offline RL algorithm Algo is set to be sufficiently pessimistic and the data distribution µ has a positive bias, for any data reward ˜ r ∈ ˜ R , the policy ˆ π learned by Algo from the dataset ˜ D has performance guarantee V π ∗ r ( d 0 ) -V ˆ π r ( d 0 ) ≤ O ( ι ) as well as safety guarantee (1 -γ ) ∑ ∞ t =0 γ t Prob( ∃ τ ∈ [0 , t ] , s τ / ∈ supp ( µ ) | ˆ π ) ≤ O ( ι ) for a small ι that decreases to zero as the degree of pessimism and dataset size increase.

Below we first provide in Appendix D.1 some background for stating the formal version Theorem 1 in Appendix D.2. Then we present the formal claims on survival instinct and conditions on positive data bias in Appendix D.3 and Appendix D.4, respectively. Finally, we prove Theorem 1 in Appendix D.5. Our discussion here will be focusing the discounted infinite-horizon step. We discuss the finite horizon variant in Appendix E.

D.1 Definitions

First, we introduce some definitions so that we can state the formal version of Theorem 1. We define constrained MDPs.

Definition 3 (Constrained MDP) . Let f, g : S × A → [ -1 , 1] . A CMDP problem C ( S , A , f, g, P, γ ) is defined as

Let π † denote its optimal policy. For δ ≥ 0 , we define the set of δ -approximately optimal policies ,

We will drop the subscript f, g from Π † f,g when it is clear from the context. In addition, we define a sensitivity function to approximately optimal policies,

The sensitivity function measures how much the approximately optimal policies (which can slightly violate the constraint of the CMDP) would perform better than the optimal policy of the CMDP (which satisfies the constraint). By definition, κ ( δ ) ∈ [0 , 2 1 -γ ] and its value can be smaller than 2 1 -γ .

By duality, a CMDP problem is related to a saddle-point problem induced by its constraint. We review the definition of saddle points and this fact below.

Definition 4. Given a bifunction L ( x, y ) : X × Y → R , we say ( x † , y † ) ∈ X × Y is a saddle point of L , if for all x ∈ X and y ∈ Y it holds that L ( x, y † ) ≤ L ( x † , y † ) ≤ L ( x † , y ) .

Lemma 1 (Duality) . [39] Let Π denote the space of policies. Consider a CMDP C := C ( S , A , f, g, P, γ ) . Define the Lagrangian L ( π, λ ) := V π f ( d 0 ) -λV π g ( d 0 ) for π ∈ Π and λ ≥ 0 . Let π † and λ † denote the optimal policy and Lagrange multiplier, i.e.,

Then together they form a saddle point ( π † , λ † ) to the Lagrangian L .

We also introduce a definition to characterize the degree of pessimism of offline RL algorithms. We call it an admissibility condition defined below, which measures the size of the data-consistent reward functions that the offline RL algorithm can have a small regret.

Definition 5. [Admissibility] For R ≥ 0 , we say an offline RL algorithm Algo is R -admissible with respect to ˜ R , if for any ˜ r ∈ ˜ R , given ˜ D , Algo learns a policy ˆ π satisfying the following with high probability: for any policy π ∈ Π ,

where we define a data-consistent reward class

E ( π, µ ) is some regret upper bound such that E ( π, µ ) = o (1) if sup s ∈S ,a ∈A d π ( s,a ) µ ( s,a ) < ∞ , and o (1) denotes a term that vanishes as the dataset size becomes infinite.

In Appendix F, we provide proofs of the degrees of admissibility for various existing algorithms, including model-free algorithms ATAC [2], VI-LCB [31] PPI/PQI [13], PSPI [7], as well as modelalgorithms, ARMOR [34, 40], MOPO [32], MOReL [33], and CPPO [14]. We show that their degree of admissibility increases as the algorithm becomes more pessimistic (by setting the hyperparameters).

Finally, we recall the definition of positive data bias in Section 3 for completeness, where the implicit CMDP C µ (˜ r ) is defined in (1).

Definition 2 (Positive Data Bias) . A distribution µ is 1 ϵ -positively biased w.r.t. a reward class ˜ R if

for all δ ≥ 0 , where Π † ˜ r,c µ ( δ ) denotes the set of δ -approximately optimal policy of C µ (˜ r ) .

Note that the CMDP C µ (˜ r ) is feasible because of the assumption on concentrability (Assumption 1). That is, we know π ∗ is feasible policy, as (1 -γ ) V π ∗ c µ ( d 0 ) = E d π ∗ [ ✶ [( s, a ) / ∈ supp ( µ )]] = 0 is implied by sup s,a d π ∗ ( s,a ) µ ( s,a ) < ∞ .

D.2 Formal Statement of the Main Theorem

Now we state the formal version of Theorem 1.

Assumption 2 (Data Assumption) .

For all ˜ r ∈ ˜ R , ˜ r ( s, a ) ∈ [ -1 , 1] for all s ∈ S and a ∈ A .

The data distribution µ is 1 ϵ -positively biased with respect to ˜ R .

There is a policy π satisfying V c µ ≤ 0 .

For any ˜ r ∈ ˜ R , the CMDP C µ (˜ r ) := C ( S , A , ˜ r, c µ , P, γ ) in (1) has a sensitivity function κ ( δ ) ≤ K for some K < ∞ and lim δ → 0 + κ ( δ ) = 0 . In addition, its solution π † satisfies sup s,a d π † ( s,a ) µ ( s,a ) < ∞ .

Theorem 2 (Main Result) . Under the data assumption in Assumption 2, consider an offline RL algorithm Algo that is sufficiently pessimistic to be ( K δ + 2) -admissible with respect to ˜ R . Then for any ˜ r ∈ ˜ R , with high probability, the policy ˆ π learned by Algo from the dataset ˜ D := { ( s, a, ˜ r, s ′ ) | ( s, a ) ∼ µ, ˜ r = ˜ r ( s, a ) , s ′ ∼ P ( ·| s, a ) } has both performance guarantee and safety guarantee

where ι := δ + κ ( δ ) + o (1) and o (1) denotes a term that vanishes as the dataset size becomes infinite.

In Assumption 2, the first assumption supposes the data reward (which may be the wrong reward) is bounded; the second assumption is that the data distribution is positively biased in view of the reward class ˜ R . Finally, we make a regularity assumption on the convergence of κ ( δ ) to zero and sup s,a d π † ( s,a ) µ ( s,a ) < ∞ to rule out some unrealistic ˜ R and P combinations. We note that the

condition on the bounded density ratio is always true for tabular problem, and a degenerate case can happen when the state-action space is, e.g., continuous. On the other hand, a degenerate case that κ is discontinuous at δ = 0 can happen due to the infinite problem horizon in the worst case; roughly speaking, this happens when it is impossible to determine the constraint violation using a finite-horizon estimator. We provide the following lemma to quantify the rate of κ ( δ ) , which is proved in Appendix D.6. In Appendix E, we show this regularity condition is always true in the finite horizon setting.

Lemma 2. Suppose there is λ † ≤ L with some L < ∞ such that λ † ∈ arg min λ ≥ 0 max π V π ˜ r ( d 0 ) -λV π c µ ( d 0 ) . Then κ ( δ ) ≤ Lδ .

Under the data assumption Assumption 2, Theorem 2 shows that if the offline RL algorithm Algo is set to be sufficiently pessimistic (relative to the upper bound of the sensitivity function κ , which we know is at most 1 1 -γ by the first assumption in Assumption 2), then Algo have both a performance guarantee to the true reward r and the a safety guarantee that the learned policy would have a small probability of leaving the data support.

We recall from Section 3 that Theorem 2 is the effects of the survival instinct of offline RL and the positive bias in the data distribution. As we will show, the admissibility condition in Theorem 2 would ensure that the survival instinct, which is what we stated informally as Proposition 1. In addition, we will provide concrete sufficient conditions for the data distribution being positively bias, as we alluded in Section 3.3. We discuss these details below and then we prove Theorem 2 in Appendix D.5.

D.3 Survival Instinct

We show the formal version of Proposition 1 which says that if the offline algorithm is admissible then it has a survival instinct. That is, it implicitly solves the CMDP in (1) even though the constraint is never explicitly modeled in the algorithm. In Appendix F, we prove the admissibility for various existing algorithms, including model-free algorithms ATAC [2], VI-LCB [31] PPI/PQI [13], PSPI [7], as well as model-algorithms, ARMOR [34, 40], MOPO [32], MOReL [33], and CPPO [14].

Proposition 2. For ˜ r ∈ ˜ R , assume the CMDP C µ (˜ r ) := C ( S , A , ˜ r, c µ , P, γ ) has a sensitivity function such that lim δ → 0 + κ ( δ ) = 0 and κ ( δ ) ≤ K for some K ≤ 1 1 -γ . Consider an offline RL algorithm Algo that is ( K δ +2) -admissible with respect to ˜ R . Then, with high probability, the policy ˆ π learned by Algo with the dataset ˜ D := { ( s, a, ˜ r, s ′ ) | ( s, a ) ∼ µ, ˜ r = ˜ r ( s, a ) , s ′ ∼ P ( ·| s, a ) } satisfies

where o (1) denotes a term that vanishes as the dataset size becomes infinite.

Proof of Proposition 2. To begin, we introduce some extra notations. Given C µ (˜ r ) , we define a relaxed CMDP problem: max π V π ˜ r ( d 0 ) , s.t. V π c µ ( d 0 ) ≤ δ and its Lagrangian L δ ( π, λ ) := V π ˜ r ( d 0 ) -λ ( V π c µ ( d 0 ) -δ ) , where λ ≥ 0 denotes the Lagrange multiplier. Let π † δ and λ † δ denote the optimal policy and Lagrange multiplier of this relaxed CMDP, and let π † denote the optimal policy to the CMDP C µ (˜ r ) . Finally we define a shorthand r † ( s, a ) := ˜ r ( s, a ) -( λ † δ +1) c µ ( s, a ) .

Now we bound the regret and the constraint violation of ˆ π in the CMDP C µ (˜ r ) . The following holds for any δ ≥ 0 . For the regret, we can derive

Here we use the additivity of value function over reward, namely, for any reward functions r 1 , r 2 and policy π , we have V π r 1 + V π r 2 = V π r 1 + r 2 . Similarly for the constraint violation, we can derive

where the first inequality is due to that ( π † δ , λ † δ ) is a saddle point to L δ (see Definition 4), and the second inequality follows from

Next we bound on V π † r † ( d 0 ) -V ˆ π r † ( d 0 ) . By Lemma 5 and Assumption 2, we know λ † δ ≤ K δ , so r † ( s, a ) ∈ [ -K δ -2 , 1] . This and the definition of c µ together imply that r † ∈ R µ (˜ r, K δ ) . Since the offline RL algorithm Algo is K δ -admissible, we have V π † r † ( d 0 ) -V ˆ π r † ( d 0 ) ≤ E ( π † , µ ) . Finally, we use the fact that π † by definition satisfies sup s,a d π † ( s,a ) µ ( s,a ) < ∞ , so E ( π † , µ ) = o (1) . Thus, we have proved

That is, ˆ π ∈ Π ι ( C µ (˜ r )) with ι = δ + κ ( δ ) + o (1) .

D.4 Implicit Data Bias

Below we provide example conditions for data distributions to have a positive implicit bias. Proposition 3. Under Assumption 1, the followings are true.

(RL setup) Suppose for any ˜ r in ˜ R , there is h : S → R such that

for some ϵ 1 , ϵ 2 < ∞ . Then µ is 1 -γ 2 ϵ 1 -positively biased to ˜ R .

(IL setup) If µ = d π ∗ , then µ is ∞ -positively biased with respect to ˜ R = { ˜ r : ˜ r : S × A → [ -1 , 1] } . If µ = d π e such that V π ∗ r ( d 0 ) -V π e r ( d 0 ) ≤ ϵ ′ 1 -γ and π e is deterministic, then µ is 1 -γ ϵ ′ -positively biased to the previous ˜ R .

(Length bias) Suppose V π c µ ( d 0 ) ≤ δ implies V ∗ r ( d 0 ) -V π r ( d 0 ) ≤ ϵ ′ + O ( δ ) 1 -γ . Then µ is O ( 1 -γ ϵ ′ ) -positively biased with respect to ˜ R = { ˜ r : ˜ r : S × A → [ -1 , 1] } . Below we list some sufficient intervention conditions on data collection that leads to the length bias:

Notice

By Lemma 3, for any π ∈ Π † ( δ ) , it holds

(a) For all ( s, a ) ∈ supp ( µ ) , R max -r ( s, a ) ≤ ϵ ′ , where R max = sup s ∈S ,a ∈A r ( s, a ) ≤ 1 .

(b) For all ( s, a ) ∈ supp ( µ ) , Q ∗ r ( s, a ) -V ∗ r ( s ) ≤ ϵ ′ .

The first example is the typical RL assumption, which says µ always has a positive bias to ˜ R if all rewards in ˜ R provide the same ranking of policies in the support [41]. The second assumes the data is collectd by a single behavior policy, which includes the typical IL assumption that the data is collected by an optimal policy. Finally, the third example describes a length bias. Note that V π c µ ( s, a ) measures the probability of going out of data support after taking a at s , which is disproportional to expected length the agent can survive. This condition assumes longer trajectories in the data to have smaller optimality gap.

This length bias condition is typically satisfied when intervention is taken in the data collection process (despite the data collection policy being suboptimal), as we saw in the motivating example in Fig. 1 and Section 3.1, and empirically in Section 4. We give two sufficient conditions on such interventions. The first one is that the data collection is stopped when the instantaneous rewards is far from the maximum rewards, and the second is that the data collection agent does not take an action that no policy (even the optimal one) can perform well in the future. The consequence of these types of interventions is that longer trajectories perform better generally. These conditions are satisfied in hopper and walker2d dataset (except for -medium-replay) in Section 4, as well as in the grid world example in Section 3.1.

The proof of Proposition 3 is based on the performance difference lemma, which can be proved by a telescoping sum decomposition.

Lemma 3 (Performance Difference Lemma) . Consider a discounted infinite-horizon MDP M = ( S , A , r, P, γ ) . For any policy π and any h : S → R , it holds that

In particular, this implies for any policies π, π ′ ,

D.4.1 Proof of RL setup in Proposition 3

Proof. Given ˜ r ∈ ˜ R , consider some h : S × A → R satisfying the assumption. Define

Since π ∗ is assumed to be in the support of µ in Assumption 1, we have

Putting these two inequalities together, we have shown

Since π ∈ Π † ( δ ) , it implies

Thus, the distribution is 1 -γ 2 ϵ 1 -positively biased.

D.4.2 Proof of IL setup in Proposition 3

Proof. Let µ = d π ∗ . We can write

So the distribution is ∞ -positively biased. For the case with the suboptimal expert π e , we can derive similarly

Therefore, µ is 1 -γ ϵ ′ -positively biased.

D.4.3 Proof of Length Bias in Proposition 3 Item 3a

Item 3b

Proof. Suppose s, a ∈ supp ( µ ) only if V ∗ r ( s ) -Q ∗ r ( s, a ) ≤ ϵ ′ . Then

Note that more broadly, we can relax the above condition to that Q ∗ c µ ( s, a ) -V ∗ c µ ( s ) < α implies V ∗ r ( s ) -Q ∗ r ( s, a ) ≤ β + O ( α ) , for a potentially smaller β , where we recall that V ∗ c µ , Q ∗ c µ are the optimal value functions to (minimizing) the cost c µ and V ∗ r , Q ∗ r are the optimal value functions to (maximizing) the reawrd r . We can use the lemma below to prove the degree of positive bias is 1 -γ β in this more general case. The above is a special case for α = 1 , since c µ ( s, a ) = 1 when s, a is not in the support of µ .

Lemma 4. Define A α s = { a ∈ A : Q ∗ c µ ( s, a ) -V ∗ c µ ( s ) < α } . Suppose for all s ∈ S , a ∈ A α s satisfies V π ′ r ( s ) -Q π ′ r ( s, a ) ≤ β . For π such that V π c µ ( d 0 ) ≤ δ , it holds for any π ′

Proof.

D.5 Proof of Theorem 2

Proof. By Proposition 2, we have proved

That is, ˆ π ∈ Π ι ( C µ (˜ r )) with ι = δ + κ ( δ ) + o (1) .

Then by 1 ϵ -positively assumption on µ , we have the performance bound

To bound of out of support probability, we derive

where second equality follows [45][Lemma 2].

D.6 Technical Details of Constrained MDP

Lemma 5. For g : S×A → [0 , ∞ ) , let C ( S , A , f, g, P, γ ) be a CMDP with sensitivity function κ ( δ ) . Assume C ( S , A , f, g, P, γ ) is feasible. Consider a relaxed CMDP, max π V π f ( d 0 ) , s.t. V π g ( d 0 ) ≤ δ and let λ † δ ≥ 0 denote its optimal Lagrange multiplier. Then λ † δ ≤ κ ( δ ) δ .

Proof. Denote the optimal policy of the relaxed CMDP as π † δ and the optimal policy of the original CMDP as π † . By construction, π † δ ∈ Π † ( δ ) . Therefore, we have V π † δ f ( d 0 ) -V π † f ( d 0 ) = max π ∈ Π † ( δ ) V π f ( d 0 ) -V π † f ( d 0 ) = κ ( δ ) .

To bound λ † δ , we use that the fact that ( π † δ , λ † δ ) is a saddle point to the Lagrangian of the relaxed problem L δ ( π, λ ) := V π f ( d 0 ) -λ ( V π g ( d 0 ) -δ ) . As a result, we can derive

Since V π † g ( d 0 ) = 0 , the above inequality implies

Lemma 2. Suppose there is λ † ≤ L with some L < ∞ such that λ † ∈ arg min λ ≥ 0 max π V π ˜ r ( d 0 ) -λV π c µ ( d 0 ) . Then κ ( δ ) ≤ Lδ .

Proof. First we note by 11 [47, Theorem 4.1] we know that ( π † , λ † ) is a saddle-point to the CMDP C µ (˜ r ) = C ( S , A , ˜ r, c µ , P, γ ) in (1). Let f = ˜ r and g = c µ . Define the Lagrangian L δ ( π, δ ) = V π f ( d 0 ) -λ ( V π g ( d 0 ) -δ ) for the relaxed CMDP problem max π : V π g ( d 0 ) ≤ δ V π f ( d 0 ) . Consider a saddle-point ( π † δ , λ † δ ) to this relaxed problem .

By duality, we can first derive

Then we upper bound L δ ( π † δ , λ † δ ) :

Thus we have

E Finite-Horizon Version

We discuss how to interpret Theorem 2 in the finite-horizon setup. Assumption 2 (Data Assumption) .

For all ˜ r ∈ ˜ R , ˜ r ( s, a ) ∈ [ -1 , 1] for all s ∈ S and a ∈ A .

The data distribution µ is 1 ϵ -positively biased with respect to ˜ R .

There is a policy π satisfying V c µ ≤ 0 .

For any ˜ r ∈ ˜ R , the CMDP C µ (˜ r ) := C ( S , A , ˜ r, c µ , P, γ ) in (1) has a sensitivity function κ ( δ ) ≤ K for some K < ∞ and lim δ → 0 + κ ( δ ) = 0 . In addition, its solution π † satisfies sup s,a d π † ( s,a ) µ ( s,a ) < ∞ .

Theorem 2 (Main Result) . Under the data assumption in Assumption 2, consider an offline RL algorithm Algo that is sufficiently pessimistic to be ( K δ + 2) -admissible with respect to ˜ R . Then for any ˜ r ∈ ˜ R , with high probability, the policy ˆ π learned by Algo from the dataset ˜ D := { ( s, a, ˜ r, s ′ ) | ( s, a ) ∼ µ, ˜ r = ˜ r ( s, a ) , s ′ ∼ P ( ·| s, a ) } has both performance guarantee

where ι := δ + κ ( δ ) + o (1) and o (1) denotes a term that vanishes as the dataset size becomes infinite.

Notation To translate the previous notation to the finite-horizon setting, we suppose the state s contains time information and the state space is layered. That is, S = ⋃ H -1 0=1 S t , where H is the problem horizon, and S t denotes the set of states at time t . For example, for a trajectory s 0 , s 2 , . . . , s H -1 starting from t = 0 , we have s t ∈ S t , for t ∈ [0 , H -1] . Therefore, we can use the previous notation to model time-varying functions naturally (needed in the finite horizon), without explicitly listing the time dependency. In this section, with abuse of notation, we define the value V π r of a policy π to reward r at s ∈ S τ as

MDP and CMDP Problems The task MDP becomes M H = ( S , A , r, P, H ) and solving it means

and the CMDP problem in (1) becomes

and safety guarantee

E.1 Main Theorem for Finite-horizon Problems

Assumption 3 (Data Assumption (Finite Horizon)) .

For all ˜ r ∈ ˜ R , ˜ r ( s, a ) ∈ [ -1 , 1] for all s ∈ S and a ∈ A .

The data distribution µ is 1 ϵ -positively biased with respect to ˜ R .

The solution to the CDMP π † satisfies sup s,a d π † ( s,a ) µ ( s,a ) < ∞ , where we note that d π † is defined without discounts but as the average over the episode length.

Theorem 3 (Main Result (Finite Horizon)) . Under the data assumption in Assumption 2, consider an offline RL algorithm Algo that is sufficiently pessimistic to be ( K δ +2) -admissible with respect to ˜ R . Then for any ˜ r ∈ ˜ R , with high probability, the policy ˆ π learned by Algo from the dataset ˜ D := { ( s, a, ˜ r, s ′ ) | ( s, a ) ∼ µ, ˜ r = ˜ r ( s, a ) , s ′ ∼ P ( ·| s, a ) } has both performance guarantee

where ι := δ + κ ( δ ) + o (1) and o (1) denotes a term that vanishes as the dataset size becomes infinite.

The main difference between the infinite-horizon setup and the finite-horizon setup is that the finitehorizon setup always satisfies the regularity assumption (the third point) in Assumption 2. In addition, the safety guarantee is more explicit compared the discounted infinite-horizon counterpart. This is because Prob( ∃ τ ∈ [0 , H -1] , s τ / ∈ supp ( µ ) | ˆ π ) ≤ V ˆ π c µ ( d 0 ) for the finite horizon, whereas we need an additional conversion in Appendix D.5.

We prove lim δ → 0 + κ ( δ ) = 0 is always true for the finite horizon version.

Proposition 4. For the H -horizon constrained MDP of (1) , we have an optimal dual variable λ † = 2 H +1 and κ ( δ ) ≤ min { 2 H, (2 H +1) δ } .

Proof. Define the Lagrange reward r † ( s, a ) = ˜ r ( s, a ) -λ † c µ ( s, a ) . Let ˆ π † denote the optimal policy to the Lagrange reward. Notice that V π c µ ( d 0 ) ≥ 1 for any π such that V π c µ ( d 0 ) > 0 . Therefore, with λ † > 2 H ,

Therefore, π † satisfies V π † c µ ( d 0 ) = 0 . This implies

On the other hand, it is always true that

Therefore,

Consider λ † = 2 H + 1 . By Lemma 2, we have κ ( δ ) ≤ (2 H + 1) δ . In addition κ ( δ ) ≤ 2 H by definition.

and safety guarantee

E.2 Length Bias

For finite-horizon problems, we provide a simple sufficient condition for length bias, which assumes long trajectories obtained in data collections are near optimal.

Proposition 5. Suppose the data are generated by rolling out policies starting from t = 0 and the initials state distribution d 0 . Let ξ = ( s 0 , a 0 , s 1 , . . . , s T ξ -1 , a T ξ -1 ) denote a trajectory of length T ξ . If with probability one (over randomness of ξ ) that T ξ = H implies

then the data distribution is 1 Hϵ ′ -positively biased.

Proof. Consider some π ∈ Π † ( δ ) . We can derive

Remark on the existence of full-length trajectories. Note that we do not explicitly make an assumption on the likeliness of full-length trajectories in Proposition 5. This is because the probability of having a full-length trajectory in the data distribution is guaranteed to be strictly greater than zero given the concentrability assumption in Assumption 3. Notice that time information is part of the state in a finite horizon problem, so Assumption 3 implies that there is a non-zero probability of having fullhorizon data trajectories (otherwise, the concentrability coefficient would be ∞ ). When Assumption 3 holds, statistical errors due to finite dataset size are included in the ι term Theorem 2.

E.3 Goal-oriented Problems

Finally we make a remark on positive data bias in the goal-oriented finite-horizon setting. In the infinite-horizon setting that a goal is marked as an absorbing state, which means that the agent once entering will stay there forever. The exact instantaneous (wrong) reward obtained at this absorbing state is not relevant (it can be anything in [ -1 , 1] ) since the admissibility condition has already accounted for the range of the associated Lagrange reward. Namely, the agent is set pessimistic such that it views partial transitions/trajectories obtaining a worse return than -1 1 -γ , which is a lower bound of the return the absorbing goal state.

To make sure the finite-horizon setting has the same kind of positive data bias, one way is to virtually extend the problem's original horizon (say H ) by (as least) one and let the goal state be the only state where the agent can stay until the last time step 12 H (i.e., all the other trajectories continue maximally up to time step H -1 and therefore have a length at most H ). Then we apply the offline

RL algorithm to this extended problem (e.g., of horizon H +1 ). We would need to set the horizon in the previous theoretical results accordingly for this longer horizon.

The reason for this extension is to ensure that the effect of absorbing state in the infinite-horizon setting (which ensures trajectories going into the absorbing state is by definition the longest) can carry over to the finite horizon case. If we apply the agent directly to solve the H step problem without such an extension, there may be other trajectories which can be as long as a goal-reaching one. As a result, there is no positive data bias.

An alternate way to create the positive data bias in the finite horizon setting is to truncate trajectories that do not reach goal at time step H -1 to be no longer than H -1 . That is, they now time out at time step H -2 , whereas only goal-reaching trajectories can continue up to time step H -1 .

F Algorithm Specific Results

In this appendix, we provide a few examples of admissible offline RL algorithms. We choose algorithms to cover both model-free [2, 7, 31, 13] and model-based [34, 40, 14, 32, 33] approaches. The pessimism in these algorithms are constructed through different ways, including adversarial training [2, 7, 34, 40, 14], value bonus [31, 32] and action truncation [13, 33]. These algorithms are listed in Table 15.

We recall the definition of admissibility below.

Definition 5. [Admissibility] For R ≥ 0 , we say an offline RL algorithm Algo is R -admissible with respect to ˜ R , if for any ˜ r ∈ ˜ R , given ˜ D , Algo learns a policy ˆ π satisfying the following with high probability: for any policy π ∈ Π ,

where we define a data-consistent reward class

E ( π, µ ) is some regret upper bound such that E ( π, µ ) = o (1) if sup s ∈S ,a ∈A d π ( s,a ) µ ( s,a ) < ∞ , and o (1) denotes a term that vanishes as the dataset size becomes infinite.

For the sake of clarity, we consider the tabular setting in our analysis. That is, we assume the state space S and the action space A are countable and finite. (We use |S| and |A| to denote their cardinalities). To establish admissibility, we sometimes make minor changes over the function classes in these algorithms, because some algorithms assume that the reward is known or has value bounded by [0 , 1] . We highlight these changes in blue. We would like to clarify that the goal of our analysis is to prove that these algorithms are admissible rather than provide a tight bound for their performance. We recommend readers who are interested in performance bound to read the original papers, as their bound can be tighter than ours.

One remarkable technical details in our analysis is that there is no need of using an union bound over all rewards in R R (˜ r ) in the admissibility definition when proving these high probability statements. Instead we found that we can reuse the bound proved for a single reward directly. The main reason is that all rewards in the admissibility definition agree with each other on the support of the data distribution, and the statistical analysis of concentration only happens within this support. Outside of the support, a uniform bound based on the reward range can be used to bound the error. This will be made more clearly later in the derivations.

F.1 ATAC

We first show that ATAC [2] is an admissible offline RL algorithm. We first introduce notations that we will use for analyzing ATAC, which will also be useful for studying PSPI in Appendix F.2.

F.1.1 Notations

For any µ, ν ∈ ∆( S × A ) , we denote ( µ \ ν )( s, a ) := max( µ ( s, a ) -ν ( s, a ) , 0) . For any µ ∈ ∆( S × A ) and any f : S × A → R , we define ⟨ µ, f ⟩ := ∑ ( s,a ) ∈S×A µ ( s, a ) f ( s, a ) . For an MDP with transition probability P : S × A → ∆( S ) , we define ( P π f )( s, a ) := γ E s ′ ∼ P ( ·| s,a ) f ( s ′ , π ) for any f : S × A → R and π : S × ∆( A ) .

Given function class F ⊆ ( S × A → R ) , policy π : S → ∆( A ) and reward ¯ r : S × A → R , we introduce Bellman error transfer coefficient [7, 2] to measure the distribution shift between two probability measure µ and ν .

Definition 6 (Bellman error transfer coefficient) . The Bellman error transfer coefficient between ν and µ under function class F , policy π and reward ¯ r is defined as,

We note that the Bellman error transfer coefficient is a weaker notion than the density ratio, which is established in the lemma below.

Lemma 6. For any distributions ν, µ ∈ ∆( S × A ) , function class F ⊆ ( S × A → R ) , policy π : S → ∆( A ) and reward ¯ r : S × A → R , we have

Proof.

Therefore,

F.1.2 Analysis

ATAC considers a critic function class F and a policy class Π . Since we consider the tabular setting, we choose F : ( S × A → [ -V max , V max ]) 13 with 1 1 -γ ≤ V max < ∞ and Π : ( S → ∆( A )) . ATAC formulates offline RL as the following Stackelberge game,

with β ≥ 0 being hyperparameter, and

For any policy π , the critic f π provides a relative pessimistic (with respect to the behavior policy µ ) value estimate of the policy π . The hyperparameter β balances pessimism, given by L ˜ D ( π, f ) , and Bellman consistency, given by E ˜ D ( π ) . The learned policy ˆ π maximizes the relative pessimistic value estimate given by the critic.

We state the performance guarantee of ATAC with respect to any comparator policy π under the set of data-consistent reward functions { ¯ r : ¯ r ( s, a ) = ˜ r ( s, a ) , ∀ ( s, a ) ∈ supp ( µ ) and | ¯ r ( s, a ) | ≤ (1 -γ ) V max , ∀ ( s, a ) ∈ S × A} in the following proposition. We make an additional assumption on the data distribution for ATAC, which is needed in its original proof.

Assumption 4. For ATAC, we assume µ is the (mixture) average of state-action visitation distribution for d 0 .

Proposition 6. Fix some ˜ r : S × A → [ -1 , 1] . Consider F : ( S × A ) → [ -V max , V max ] with 1 1 -γ ≤ V max < ∞ and Π : ( S → ∆( A )) . Let ˆ π be the solution to (6) and let π ∈ Π be any comparator policy. Let ν ∈ ∆( s, a ) be an arbitrary distribution. Under Assumption 4, for any

for all ¯ r such that ¯ r ( s, a ) = ˜ r ( s, a ) , ∀ ( s, a ) ∈ supp ( µ ) and | ¯ r ( s, a ) | ≤ (1 -γ ) V max , ∀ ( s, a ) ∈ S ×A .

Proof. We first show that F : ( S × A → [ -V max , V max ]) is both realizable and Bellman complete with repect to ¯ r . For any ¯ r such that | ¯ r ( s, a ) | ≤ (1 -γ ) V max, we have Q π ¯ r ( s, a ) ∈ [ -V max , V max ] . This means that Q π ¯ r ∈ F for any π ∈ Π , s ∈ S , a ∈ A . Moreover, for any f ∈ F , | ¯ r ( s, a )+ P π f ( s, a ) | ≤ | ¯ r ( s, a ) | + |P π f ( s, a ) | ≤ (1 -γ ) V max + γV max = V max. In other words, (¯ r ( s, a ) + P π f ( s, a )) ∈ F for any f ∈ F .

By construction of ¯ r , we have ¯ D := { ( s, a, ¯ r, s ′ ) | ( s, a, s ′ ) ∈ D , ¯ r = ¯ r ( s, a ) } ≡ ˜ D . That is, solving the Stackelberg game (6) given by ˜ D is equivalent to solving the game given by ¯ D . The rest of the proof follows from Theorem C.12 in [55] by choosing the reward class to only contain the reward ¯ r , i.e., G = { ¯ r } (which implies d G = 0 ) and using d F , Π = ˜ O ( |S||A| log( 1 / δ )) .

Note that the above derivation does not need an additional union bound to cover all rewards in { ¯ r : ¯ r ( s, a ) = ˜ r ( s, a ) , ∀ ( s, a ) ∈ supp ( µ ) and | ¯ r ( s, a ) | ≤ (1 -γ ) V max , ∀ ( s, a ) ∈ S × A} . This is because the concentration analysis is only taken on the support of the data distribution, where all rewards in this reward class agree, and a uniform bound based on V max is used for out of support places, which again applies to all the rewards in this reward class.

We then show that ATAC [2] is admissible based on Proposition 6.

Corollary 3 (ATAC is admissible) . For any V max ≥ R 1 -γ , ATAC is R -admissible with respect to ˜ R for any ˜ R⊆ ( S × A → [ -1 , 1]) .

Proof. Given any ˜ r ∈ ˜ R , we want to show that, with high probability, V π ¯ r ( d 0 ) -V ˆ π ¯ r ( d 0 ) = o (1) for all ¯ r ∈ R R (˜ r ) and all policy π such that sup ( s,a ) ∈S×A d π ( s,a ) µ ( s,a ) < ∞ . For any such π , define C ∞ := sup ( s,a ) ∈S×A d π ( s,a ) µ ( s,a ) , we have C ( ν ; µ, F , π, ¯ r ) ≤ C ∞ for any ¯ r . By taking ν = d π in Proposition 6, we have, with probability 1 -δ ,

for all ¯ r such that ¯ r ( s, a ) = ˜ r ( s, a ) , ∀ ( s, a ) ∈ supp ( µ ) and | ¯ r ( s, a ) | ≤ (1 -γ ) V max , ∀ ( s, a ) ∈ S ×A . Since V max ≥ R 1 -γ , we have that V π ¯ r ( d 0 ) -V ˆ π ¯ r ( d 0 ) = o (1) for all ¯ r ∈ R R (˜ r ) .

F.2 PSPI

We show that PSPI [7] is also admissible. PSPI [7] is similar to ATAC. Given critic function class F : ( S × A → [ -V max , V max ]) 14 with 1 1 -γ ≤ V max < ∞ and policy class Π : ( S → ∆( A )) , PSPI solves the following Stackelburg game,

with β ≥ 0 being hyperparameter, and

In PSPI, the critic f π provides an absolute pessimistic value estimate of policy π . The hyperparameter β trades off pessimism and Bellman-consistency. The learned policy ˆ π maximizes such a pessimistic value. We state the performance guarantee of the learned policy ˆ π with respect to any comparator policy π under the set of data-consistent reward functions { ¯ r : ¯ r ( s, a ) = ˜ r ( s, a ) , ∀ ( s, a ) ∈ supp ( µ ) and | ¯ r ( s, a ) | ≤ (1 -γ ) V max , ∀ ( s, a ) ∈ S × A} in the following proposition.

Proposition 7 ([7, 55]) . Let ˆ π be the solution to (6) and let π ∈ Π be any comparator policy. Let ν ∈ ∆( s, a ) be an arbitrary distribution. For any δ ∈ (0 , 1] , choosing β = ˜ Θ ( 1 V max 3 √ C | ˜ D| 2 ( |S||A| log( 1 / δ ) ) 2 ) , with probability 1 -δ , for all ¯ r such that ¯ r ( s, a ) = ˜ r ( s, a ) , ∀ ( s, a ) ∈ supp ( µ ) and | ¯ r ( s, a ) | ≤ (1 -γ ) V max , ∀ ( s, a ) ∈ S ×A .

Proof. The proof is similar to that of Proposition 6. First, observe that F : ( S × A → [ -V max , V max ]) is both realizable and Bellman complete with respect to ¯ r . The rest of the proof follows from Theorem D.1 in [55] (taking G = { ¯ r } ) and Lemma 6.

The proposition above implies that PSPI [7] is admissible. We omit the proof of the corollary below as it is the same as the proof of Corollary 3.

Corollary 4 (PSPI is admissible) . For any V max ≥ R 1 -γ , PSPI is R -admissible with respect to ˜ R for any ˜ R⊆ ( S × A → [ -1 , 1]) .

F.3 ARMOR

We show that ARMOR [34, 40] is admissible. ARMOR is a model-based offline RL algorithm. We denote a model as M = ( S , A , P M , r M , γ ) , where P M : S × A → ∆( S ) is the model dynamics, and r M : S × A → [ -R max , R max ] 15 is the reward function with 1 ≤ R max < ∞ . Since we consider the tabular setting, we use a model class that contains all possible dynamics and all reward functions bounded within [ -R max , R max ] , i.e., M model = { M : P M ∈ ( S × A → ∆( S )) , r M ∈ ( S × A → [ -R max , R max ]) } . For any reference policy π ref : S → ∆( A ) , ARMOR solves the following two-player game,

where V π M ( d 0 ) := E π,P M [ ∑ ∞ t =0 γ t r M ( s t , a t ) | s 0 ∼ d 0 ] , and with

Intuitively, ARMOR constructs a relative pessimistic model (with respect to π ref ) M which is also approximately consistent with data. The learned policy ˆ π maximizes the value estimate given by the pessimistic model. We define the generalized single policy concentrability to characterize the distribution shift from µ to the state-action visitation d π of any policy π .

Definition 7 (Generalized Single-policy Concentrability [40]) . We define the generalized singlepolicy concentration for policy π , model class M model , reward ¯ r and data distribution µ as

The single-policy concentrability C model ( π ) can be considered as a model-based version of the Bellman error transfer coefficient in Definition 6. Following the same steps as the proof of Lemma 6, it can be shown that C model ( π ; M model , ¯ r ) ≤ sup ( s,a ) ∈S×A d π ( s,a ) µ ( s,a ) . Weprovide the high-probability performance guarantee for the learned policy ˆ π . This implies that ARMOR is an admissible algorithm with a sufficiently large R max.

Proposition 8. For any δ ∈ (0 , 1] , there exists an absolute constant c such that when choosing α = c |S| 2 |A| log( 1 / δ ) , for any comparator policy π ∈ Π , with probability 1 -δ , the policy ˆ π learned by ARMOR (11) satisfies

for all ¯ r such that ¯ r ( s, a ) = ˜ r ( s, a ) , ∀ ( s, a ) ∈ supp ( µ ) and | ¯ r ( s, a ) | ≤ R max , ∀ ( s, a ) ∈ S × A .

Proof. By construction, we have ¯ M = ( S , A , P, ¯ r, γ ) ∈ M model . Since ¯ r ( s, a ) = ˜ r ( s, a ) for any ( s, a ) ∈ supp ( µ ) , we have M α ˜ D ≡ M α ¯ D . The rest of the proof follows mostly from Theorem 2 in [40] by taking π ref = µ . By a similar argument as in the proof of Proposition 6, we do not need an additional union bound to cover all rewards in { ¯ r : ¯ r ( s, a ) = ˜ r ( s, a ) , ∀ ( s, a ) ∈ supp ( µ ) and | ¯ r ( s, a ) | ≤ R max , ∀ ( s, a ) ∈ S × A} .

Corollary 5 (ARMOR is admissible) . For any R max ≥ R , ARMOR is R -admissible with respect to ˜ R for any ˜ R⊆ ( S × A → [ -1 , 1]) .

Proof. Given any ˜ r ∈ ˜ R , we want to show that, with high probability, V π ¯ r ( d 0 ) -V ˆ π ¯ r ( d 0 ) = o (1) for all ¯ r ∈ R R (˜ r ) and all policy π such that sup ( s,a ) ∈S×A d π ( s,a ) µ ( s,a ) < ∞ . For any such π , C model ( π ; M model , ¯ r ) ≤ sup ( s,a ) ∈S×A d π ( s,a ) µ ( s,a ) := C ∞ < ∞ . By Proposition 8, with probability 1 -δ , we have

for all ¯ r such that ¯ r ( s, a ) = ˜ r ( s, a ) , ∀ ( s, a ) ∈ supp ( µ ) and | ¯ r ( s, a ) | ≤ R, ∀ ( s, a ) ∈ S × A . Therefore, V π ¯ r ( d 0 ) -V ˆ π ¯ r ( d 0 ) = o (1) for all ¯ r ∈ R R (˜ r ) with high probability.

F.4 CPPO

CPPO[14] is another admissible model-based offline algorithm. Again, we denote a model as M = ( S , A , P M , r M , γ ) , where P M : S × A → ∆( S ) is the model dynamics, and r M : S × A → [ -R max , R max ] 16 is the reward function with 1 ≤ R max < ∞ . CPPO solves the two-player game,

with V π M ( d 0 ) := E π,P M [ ∑ ∞ t =0 γ t r M ( s t , a t ) | s 0 ∼ d 0 ] and M α ˜ D defined in (12). CPPO constructs a pessimistic model M which is also approximately consistent with data. The learned policy ˆ π maximizes the value estimate given by the pessimistic model. We provide the high-probability performance guarantee for the learned policy ˆ π . This implies that CPPO is an admissible algorithm with sufficiently large R max.

Proposition 9. For any δ ∈ (0 , 1] , there exists an absolute constant c such that when choosing α = c |S| 2 |A| log( 1 / δ ) , for any comparator policy π ∈ Π , with probability 1 -δ , the policy ˆ π learned by CPPO (16) satisfies

for all ¯ r such that ¯ r ( s, a ) = ˜ r ( s, a ) , ∀ ( s, a ) ∈ supp ( µ ) and | ¯ r ( s, a ) | ≤ R max , ∀ ( s, a ) ∈ S × A .

Proof. By construction, we have ¯ M = ( S , A , P, ¯ r, γ ) ∈ M model . Since ¯ r ( s, a ) = ˜ r ( s, a ) for any ( s, a ) ∈ supp ( µ ) , we have M α ˜ D ≡ M α ¯ D . The rest of the proof follows similarly from the proof of Theorem 2 in [40]. By Lemma 5 from [40], with high probability, ¯ M ∈ M α ¯ D . Therefore,

where the second inequality follows from the optimality of ˆ π . By (20) from [40], for any M ∈ M α ¯ D , | V π ¯ M ( d 0 ) -V π M ( d 0 ) | ≤ O ( √ C model ( π ; M model , ¯ r ) R max (1 -γ ) 2 √ |S| 2 |A| log( 1 / δ ) | ˜ D| ) . Following a similar argument to the proof of Proposition 6, there is no need to take the union bound with respect to ¯ r , which concludes the proof.

We omit the proof for the following corollary as it is the same of the proof of Corollary 5.

Corollary 6 (CPPO is admissible) . For any R max ≥ R , CPPO is R -admissible with respect to ˜ R for any ˜ R⊆ ( S × A → [ -1 , 1]) .

Algorithm 1 VI-LCB [31]

1: Input: Batch dataset ˜ D , discount factor γ , confidence level δ , and value range V max .

3: Randomly split ˜ D into J +1 sets ˜ D t = { ( s i , a i , r i , s ′ i ) } i m =1 for j ∈ { 0 , 1 , · · · , J } with m = N J +1 .

2: Set J := log N 1 -γ .

4: Set m 0 ( s, a ) := ∑ m i =1 ✶ { ( s i , a i ) = ( s, a ) } based on dataset ˜ D 0 . 5: For all a ∈ A and s ∈ S , initialize Q 0 ( s, a ) = -V max , V 0 ( s ) = -V max and set π 0 ( s ) = arg max a m 0 ( s, a ) .

7: Initialize r t ( s, a ) = 0 and set P j s,a to be a random probability vector.

6: for j = 1 , · · · , J do

8: Set m t ( s, a ) := ∑ m i =1 ✶ { ( s i , a i ) = ( s, a ) } based on dataset ˜ D t . 9: Compute penalty b t ( s, a ) for L = 2000log(2( J +1) |S| |A| /δ )

be empirical average of rewards.

F.5 VI-LCB

We show VI-LCB [31] is admissible. VI-LCB is a pessimistic version of value iteration. It adds negative bonuses to the Bellman backup step in order to underestimate the value when there are missing data. By being pessimistic in value estimation, it can overcome the issue of µ being nonexploratory. We recap the VI-LCB algorithm in Algorithm 1. We highlight the changes we make in blue. These changes are due to that originally the authors in [31] assume the rewards are non-negative, but the rewards in the admissibility definition Definition 5 can take negative values. Therefore, we make these changes accordingly. We state and prove the guarantee below. 17

Proposition 10. For V max ≥ R 1 -γ , VI-LCB is R -admissible with respect to any ˜ R ⊆ ( S × A → [ -1 , 1]) .

Proof. First, we introduce a factor of 2 in their proof of Lemma 1 [31] to accomodate that the rewards here can be negative. This is reflected in the updated bonus definition in Algorithm 1. This updated Lemma 1 of [31] now captures the good event that the bonus can upper bound the error of the empirical estimate of Bellman backup. It shows this good event is true with high probability.

We remark that while originally Lemma 1 of [31] is proved for a single reward function, it actually holds for simultaneously for all the reward functions in R R (˜ r ) , without the need of introducing

Algorithm 2 Pessimistic Policy Iteration (PPI) [13]

1: Input: Batch dataset ˜ D , function class F , probability estimator ˆ µ , hyperparameter b .

3: for j ∈ { 0 , . . . , J } do

2: for i ∈ { 0 , . . . , I -1 } do

4: f i,j +1 ← arg min f ∈F L ˜ D ( f, f i,j , ˆ π i )

5: end for

6: ˆ π i +1 ← arg max π E ˜ D [ E π [ ζ ◦ f i,J +1 ]

7: end for

8: Return ˆ π = ˆ π I

Algorithm 3 Pessimistic Q Iteration (PQI) [13]

1: Input: Batch dataset ˜ D , function class F , probability estimator ˆ µ , hyperparameter b .

3: f i +1 ← arg min f ∈F L ˜ D ( f, f i )

2: for i ∈ { 0 , . . . , I -1 } do

4: ˆ π i +1 ( s ) ← arg max a ζ ◦ f i +1 ( s, a )

5: end for

6: Return ˆ π = ˆ π I

additionally a union bound. The reason is that in the proof of Lemma 1 in [31], the concentration is used only for the estimating the Bellman on the data support. This bound would apply to all rewards in R R (˜ r ) since they agree exactly on the data. For state-action pairs out of the support, the proof takes a uniform bound based on the size of V max , which also applies to all rewards in R R (˜ r ) .

Therefore, we can apply the updated Lemma 1 of [31] to prove the desired high probability bound needed in the admissibility condition. Under this good event, we can use the upper bound in (54b) in [31] to bound the regret. Consider some π such that C := sup s,a d π ( s,a ) µ ( s,a ) < ∞ . Suppose ˜ D has N transitions. Take some V max ≥ R 1 -γ . Then running VI-LCB with J iterations ensures 18 for any ¯ r ∈ R R (˜ r ) ,

where we mark the changes due to using rewards which can takes negative values in blue. Setting J = log N 1 -γ as in [31], with probability 1 -δ , it holds that

Thus, VI-LCB is R -admissible.

F.6 PPI and PQI

We show PPI and PQI proposed in [13] are admissible. We present their algorithms in Algorithm 2 and Algorithm 3, with ( ζ ◦ f )( s, a ) := ζ ( s, a ) f ( s, a ) for any ζ, f : S × A → R . These algorithms use a probability estimator of the data distribution (denoted as ˆ µ ), which in the tabular case is the empirical distribution. Based on ˆ µ , they define a filter function

ζ ( s, a ; ˆ µ, b ) = ✶ [ˆ µ ( s, a ) ≥ b ] This filter function classifies whether a state-action pair is in the support, and it is used to modify the Bellman operators in dynamics programming as shown in line 6 of Algorithm 2 and line 4 of

Algorithm 3. As a result, the Bellman backup and the policy optimization only consider in-support actions, which mitigates the issue of learning with non-exploratory µ . The loss functions (i.e., L ˜ D ) in Algorithm 2 and Algorithm 3 denote the squared Bellman error (with a target network) as used in fitted Q iteration [78]. We omit the details here.

In summary, PPI and PQI follow the typical policy iteration and value iteration schemes, except that the backup and the argmax are only taken within the observed actions (or actions with sufficiently evidence to be in the support when using function approximators). This idea is similar to the spirit of the later IQL algorithm [8], except IQL doesn not construct the filter function explicitly but relies instead on expectile maximization.

Proposition 11. Suppose F = ( S × A → [ -V max , V max ]) 19 . For V max ≥ R 1 -γ , PPI/PQI is R -admissible with respect any ˜ R⊆ ( S × A → [ -1 , 1]) .

Proof. We prove for PPI; the proof for PQI follows similarly. Consider some π such that C := sup s,a d π ( s,a ) µ ( s,a ) < ∞ . Suppose ˜ D has N transitions. Take some V max ≥ R 1 -γ and some ¯ r from R R (˜ r ) . We use the Corollary 1 in [13]: With I large enough, it holds with probability 1 -δ ,

where ϵ µ = D TV (ˆ µ, µ ) = O ( √ N ) . Further,

We can then upper bound the performance difference above as

We bound the latter two terms by tuning b . Notice E µ [ ✶ [ µ ( s, a ) ≤ 2 b ]] ≤ |S||A| 2 b at most. This implies

Thus,

We can apply this similar argument used in the previous proofs to show that this bound simultaneously applies to all ¯ r ∈ R R (˜ r ) since the high-probability concentration analysis is only taken on the data distribution where all rewards in R R (˜ r ) are equal. Thus, when V max ≥ R 1 -γ , PPQ (and similarly PQI) is R -admissibile.

F.7 MOReL and MOPO

MOReL[33] and MOPO [32] are two popular model-based offline RL algorithms. The two algorithms operate in a similar manner, except that MOReL [33] uses the truncation approach (similar to PPI and PQI [13]) which classifies state-actions into known and unknown sets, whereas MOPO [32] uses negative bonuses (similar to VI-LCB [31]). In this section, we show both algorithms are admissible.

We denote a model as M = ( S , A , P M , r M , γ ) , where P M : S × A → ∆( S ) is the model dynamics, and r M : S × A → [ -1 , 1] 20 with 1 ≤ R max < ∞ being the reward function. The model class is denoted as M model .

F.7.1 MOReL

We analyze a variant of MOReL which also learns the reward function, whereas MOReL in the originally paper [33] assumes the reward function is given. We suppose model reward r M and model dynamics P M are learned by

where

We define the set of known state-actions K ξ as

with

We note that α can be chosen as Θ( |S| 2 |A| log 1 / δ ) so that M α contains model ( S , A , P, ˜ r, γ ) with probability 1 -δ for any δ ∈ (0 , 1] (see [40] for details).

Remark Note that we change the definition for the set of known state-actions. In [33], the set is given by

However, it is not impossible to use such a set in practice since the reward function ˜ r and true model P ( ·| s, a ) are not given to the learner. Our known state-action set K ξ , in comparison, is easier to construct. It simply measures the maximum disagreement between two data-consistent models. With a considerate choice of α (see [40]), K ξ is always a subset of K 0 ξ . Our definition also matches better with the practical implementation in [33] which uses the disagreement between any two models in an ensemble.

Then MOReL learns policy ˆ π by solving the MDP ( S , A , ˆ P, ˆ r, γ ) , where

✶ Note MOReL introduces an absorbing state s † .

Below we prove a performance guarantee of MOReL. The proof here is different from that in the original paper, because here we need to consider reward learning and the original proof does not provide an exact rate that converges to zero as the data size increases. (In the original paper there is a non-zero bias term that depends on the comparator policy.)

Proposition 12 (MOReL Performance Guarantee) . Given data ˜ D = { ( s, a, ˜ r, s ′ ) } of size N of an unknown MDP ( S , A , ˜ r, P, γ ) . Suppose ˜ r, r ⋆ M ∈ [ -1 , 1] and (˜ r, P ) ∈ M α ˜ D . Choose ξ = min(1 , O ( V max ( |S| 2 |A| log( 1 / δ ) / N ) 1 / 4 )) . Let X ≥ R . For π such that sup s,a d π ( s,a ) µ ( s,a ) = C < ∞ , with probability 1 -δ , it holds that

for all ¯ r ∈ R R (˜ r ) := { ¯ r : ¯ r ( s, a ) = ˜ r ( s, a ) , ∀ ( s, a ) ∈ supp ( µ ) and ¯ r ( s, a ) ∈ [ -R, 1] , ∀ ( s, a ) ∈ S × A} .

Proof. We extend ˜ r as ˜ r ( s † , a ) = -X for all a ∈ A . Let ˆ M = (ˆ r, ˆ P ) . First, by the optimality of ˆ π , we write

where V π ˆ M denotes the value of policy π with respect to the reward ˆ r and dynamics ˆ P . The last inequality follows from the optimality of ˆ π on ˆ M .

Then we notice the fact that K ξ ⊆ supp ( µ ) with a small ξ . The proof of the lemma below is given in Appendix F.7.3.

Lemma 7. If ξ ≤ 1 , K ξ ⊆ supp ( µ ) .

Let ˆ d π denote the average state-action visitation of π with respect to ˆ P .

where in the second inequality we use K ξ ⊆ supp ( µ ) (which implies ˜ r = ¯ r in K ξ ). Similarly we can show

(1 -γ ) ( V π ˆ M ( d 0 ) -V π ¯ r ( d 0 ) ) ≥-ξ + E s,a ∼ d π [ˆ r ( s, a ) + γ E s ′ ∼ ˆ P | s,a [ V π ˆ M ( s ′ )] -¯ r ( s, a ) -γ E s ′ ∼ P | s,a [ V π ˆ M ( s ′ )] | ( s, a ) / ∈ K ξ ] ≥-ξ -2 V max E s,a ∼ d π [ ✶ [( s, a ) / ∈ K ξ ]] Combining the three inequalities above gives

By the concentratability assumption, we can further upper bound

Therefore we have

where the last step is Markov inequality,

and

Now we upper bound the expectation of E over µ . With probability greater than 1 -δ ,

where the last step is based on Lemma 8.

Lemma 8. [34] With probability 1 -δ , for any MDP model M ,

Thus, we have

By Proposition 12, we can apply the previous analysis technique to MOReL to show it is admissible without addition union bounds, which leads to the corollary below.

Corollary 7. For X ≥ R , MOReL is R -admissible with respect any ˜ R⊆ ( S × A → [ -1 , 1]) .

F.7.2 MOPO

MOPO [32] is very similar to MOReL except that it uses negative bonuses (like VI-LCB) instead of truncation. In the original paper of MOPO, the authors assume the reward is given. Here we consider a variant that also learns the reward. Specifically, given P ⋆ M and r ⋆ M above it solves the MDP ( S , A , P ⋆ M , ˆ r, γ ) , where

While the original MOPO paper does not give a specific design of b ( s, a ) that is provably correct, in principle making MOPO provably correct is possible. Essentially, we need to choose a large enough bonus to cover the error of the model-based Bellman operator (defined by the estimated reward and dynamics). This would lead to a bonus of order V max (see the analysis of VI-LCB in Appendix F.5). Then we can proceed with an analysis that combines that of MOReL and VI-LCB to prove MOPO's performance guarantee. This can then be used to show (like the previous proofs) that MOPO is admissible. We omit the proof here.

Corollary 8. Suppose b ( s, a ) = Θ( V max ) for ( s, a ) / ∈ supp ( µ ) . For V max ≥ R 1 -γ , MOPO is R -admissible with respect any ˜ R⊆ ( S × A → [ -1 , 1]) .

F.7.3 Proof of Technical Lemma

Proof of Lemma 7. Consider any (¯ s, ¯ a ) / ∈ supp ( µ ) . Given any M = ( S , A , P M , r M , γ ) ∈ M α ˜ D , we construct M ′ = ( S , A , P M , r M ′ , γ ) with r M ′ = r M for all ( s, a ) ∈ ( S × A ) \ { (¯ s, ¯ a ) } , and r M ′ (¯ s, ¯ a ) = -sign ( r M (¯ s, ¯ a )) . By construction, since M and M ′ agrees on all state-actions except (¯ s, ¯ a ) , which is not in data support, we have M ′ ∈ M α ˜ D . However, we have that

This means that (¯ s, ¯ a ) / ∈ K ξ . Therefore K ξ ⊆ supp ( µ ) .

G Grid World

In this section, we describe the full details of the grid world study that was discussed in Section 3.1.

Environment. We consider goal-directed navigation in the environment shown in Fig. 10. We use a fixed grid world layout shown in the figure. The agent's state is given by a tuple ( x, y, d ) where ( x, y ) ∈ [5] 2 represents the 2-d coordinate and d ∈ { N,W,E,S } encodes the North, West, East, and South direction respectively that the agent is facing. The agent's action space is A = { f, l, r } where f denotes the action of moving to grid square in front of the agent, l denotes a left turn of 90 degrees, and r denotes a right turn of 90 degrees. The agent can go through the lava (orange wavy square) but cannot go through the wall (grey square). The goal (key) is an absorbing state. The agent gets a reward of +1 when first visiting the goal followed by a reward of 0 for then staying in the goal. The agent gets a reward of -1 for reaching lava and a reward of -0.01 for all other actions to incentivize the agent to reach the goal along the shortest safe path. We use a horizon of H = 20 . The agent deterministically starts in the top-right corner shown in Fig. 10.

Dataset. We collect a dataset D of 500 episodes by taking 100 identical optimal episodes along with 400 identical episodes that all touch the topmost lava field. We introduce a length bias by terminating an episode if the agent touches the lava or fails to reach the goal in H -1 steps (i.e., the agent doesn't survive). In addition to the original setting with the observed reward, we consider three additional settings where the rewards in the dataset are replaced by: (i) a constant zero reward, (ii) their negative, and (iii) a random number sampled uniformly in [0 , 1] .

Fo

a

/star

h

=

π

/star

h

(

s

h

)

s

h

a

s

1

s

′

t

Timeout

(Failure)

Terminal

(Success)

Length Bias

V

min,t

0

s

H

+1

t

H

E

<latexi sh

1_b

64="SrOc

R

qD

kT

p9w

G

mJXU

>A

B

n

dV

L

gN

E

y

f

Y

oP

Z

I

3

0

M

F

Q

vH

K

5

2/

W

j

7

+

C8

z

u

<latexi sh

1_b

64="kBmI5+Tw

YXcd

E

87

Z

D

Jg

>A

n

V

LS

N

Oy

r

fU

9

oP

3

0

M

F

Q

vH

K

2/

W

j

q

R

C

p

G

z

u

<latexi sh

1_b

64="BS

MO

dpJG

9RcX7WQ

Z

D

I

>A

H

V

L

gN

E

z

r

Y

8

P

f

m

k

5

y

C

U

on

v

q

T

w

u

2

j

F

+

0

3

K

/

<latexi sh

1_b

64="zER7H

F

V

8fYcw2dWL5

9no

>A

CBX

D

SgM

3

rPU

I

T

m

j

J

QG

0

p

+

/

y

q

N

k

Z

K

O

uv

<latexi sh

1_b

64="

Kd5o

7JN0w

W

c

DPZ

/

UY

>A

B

3

V

LSgM

F

2vqk

Gj pX

O

y

n

R

m

8

Q

I

+

u

C

9

f

r

z

E

H

T

<latexi sh

1_b

64="CUm8

vWGY9

up

2g

Q

O

E

ZVF

>A

B7n cd

DLS

N

y

r

f

oP

I

3

J

5

H

w

zM

K

/

R

q

T

+

X

0

k

j

<latexi sh

1_b

64="fEV+3LGZJ

O

9

XC

W

k

>A

B7n cd

D

S

N

z

r

H

Y

I

o

5

u

w

Q

K

P

F

/

y

U

q

pT

0

8

j

v

<latexi sh

1_b

64="S9y

Ww8

+FmkLj5dzQ

g2Y

>A

B

H

c

VD

M

3

vq

K

G

u

7p

E

J

0o

C

P

r

O

X

/

N

I

R

n

T

Z

<latexi sh

1_b

64="VnW+u7

j

3AqmI

QMUE8G

09

z

>

B

H

cd

DLSg

F

2v

k

w

K

T

Z

Y

o

/

PpJ

X

O

RC

y

5

r

N

f

Algorithm 4 PEVI ( S , A , H, D , β, V min , V max ) [12]. We are given access to a set D of episodes ( τ ). Additionally, we are given a hyperparameter β ∈ R ≥ 0 that controls pessimism. The hyperparameters V min , V max ∈ R H indicate bounds on policy value with V min ,h and V max ,h denoting the minimum and maximum policy values respectively for time step h ∈ [ H ] . Unlike the original PEVI algorithm of [12], we treat V min , V max as hyperparameters.

Methods. We evaluate two methods: behavior cloning which simply takes the action with the highest empirical count in a given state and a random action if the state was unseen, and PEVI [12] an offline RL method with guarantees for tabular MDP. We provide a pseudocode of PEVI in Algorithm 4. Unlike the original PEVI in [12], we provide minimum and maximum bounds for the value function for each time step as an additional hyperparameter. These bounds should be dictated by our CMDP framework and should penalize out-of-distribution actions. In principle, we need to set these bounds such that the algorithm is admissible. Please see Appendix D and Appendix E for details.

Intuitively, PEVI performs dynamic programming similar to a standard value iteration with two key changes. Firstly, it uses pessimistic value initialization where the learned value V h ( s ) of a state s at time step h is set to the lowest possible return V min,h if the state is unseen at time step h . Secondly, when performing value iteration it adds a pessimism penalty of -β / √ n ( s,a ) to the dataset reward where n ( s, a ) is the empirical state-action count in D . This pessimism penalty ensures that the learned value function lower bounds the true value function (hence a pessimistic estimate). In our experiments, we vary β and set V min ,h = ˜ V min ,h -β ( H -h + 1) -1 where ˜ V min ,h is the minimum possible return for the given reward type that we are working with. The value of ˜ V min ,h is -( H -h +1) for the original reward, -1 for the negative reward, and 0 for the zero reward and random reward cases). We define V max ,h = ˜ V max ,h where ˜ V max ,h is the maximum possible return for the given reward type. The value of ˜ V max ,h is 1 of the original reward, ( H -h +1) for the random reward and the negative reward, and 0 for the zero reward case.

Results. We show numerical results in Table 16. We visualize results in Fig. 10. The behavior cloning simply imitates the most common trajectory in the dataset which results in going to the lava (shown in red in the figure). In contrast, the PEVI is able to reach the goal in every reward situation.

Explanation. The reason why PEVI works can be understood simply by pessimistic initialization and length bias. The length bias in this setting is visualized in the center of Fig. 10. PEVI assigns the final state s ′ t of a failed trajectory will get a pessimistic value of V t ( s ′ t ) = V min , t . This value isn't updated as we never take any action in s ′ t due to timeout. In our case, s ′ t will be the state where the agent first visits the lava. In contrast, the final state of a successful trajectory s H +1 gets assigned a value of 0. If V min is sufficiently negative, then PEVI will only consider policies that stay all H steps within the data support. Due to length bias, this is not true for the non-surviving trajectories that

-

±

±

reach the lava. Further, the only trajectories in our case that complete end up reaching and staying in the goal. Therefore, under the assumption that V min ,h are all sufficiently small, PEVI will learn a policy reaches the goal irrespective of the correctness of the reward in the data. The sufficiency of the negativity of V min ,h is implied by our CMDP framework. Lastly, note that as the world is deterministic, we didn't need to use the pessimism arising from the -β √ n ( s,a ) penalty term.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors make an interesting and important observation about offline Reinforcement Learning. They note that an agent can learn from offline data even when the reward signal is not the same one as that used to train the online agent. Moreover it can be vastly different, or even with no reward signal at all, within certain types of task and with certain datasets. In particular, when a dataset has limited and biased data coverage, the aim of an offline agent to stay within the data distribution means that it has an equivalent of a survival instinct. This point is noted first empirically and then proven formally as well as being shown in an extensive set off environments and settings.\n\nThe conditions necessary of the training data for an agent to learn a near-optimal policy are identified. Finally, a perspective on offline RL is given which means that the reward signal can be essentially ignored, so long as the appropriately biased dataset is provided.\n\nSTRENGTHS:\nThe originality and quality of the paper are both very strong. The combination of formal mathematical proofs of the insensitivity to the rewards and the experiments which show this to be true are married well together.  The paper as a whole is very well written, and the appendix is in particular very thorough and shows a good understanding of the need for transparency and clarity in experimental settings. The amount of formality within the bulk of the paper is also appropriate, giving enough to understand the results, but not to swamp them.\n\nThe nature of the findings are themselves original and are likely to have impact in the field of offline RL.\n\nWEAKNESSES:\nI see few weaknesses overall, although there are a few typos throughout:\n\nIn most of the figures, the number of seeds is provided, though in figure 1 it is not. This should be added.\nLine 120: \"worst case a necessary\"\nLine 143: \"In following\"\nLine 178: \"s, a\" may be clearer as (s,a).\nLine 214: \"This is contrast\"\nLine 227: \"Positive data bias assumption\"\nFootnote on page 5 has an additional bracket at the end.\nLine 232: \"benefits\"->\"benefit\"\nLine 245: \"states have a safe action\"\nLine 247: \"is also mild\"\nFootnote on page 6: \"we include a ablation\"\nLine 290: \"with wrong can\"\nLine 295 needs to be rewritten.\nLine 310: \"with provably pessimism\"\nLine 315: \"A implicit\"\nLine 318: \"due the\"\nLine 332: \"the end of episode\"\nLine 334: \"there is no timeouts\"\nLine 915 in the appendix: \"the followings are true\"\nLine 1025: \"first we the\"\nLine 1089: \"and let the goal state to be\"\n\nQUESTIONS:\nI think that a discussion under which circumstances an adversarial reward signal could be designed to attack an offline agent may be interesting. While it is clear that with the appropriately biased data, a random reward signal has little effect, there may be circumstances where a strong enough adversarial reward signal and an inappropriately biased dataset may not be sufficient to lead to near optimal performance.\n\nIs it possible to make claims in the multi-agent offline setting similar to those in the single agent setting?\n\nLIMITATIONS:\nLimitations of the work are given, but none of the potential societal impact of this work has been discussed. Given that Offline RL may well play a large role in ML systems going forward, this should be mentioned.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work displays a new observation that offline reinforcement learning (RL) algorithms can develop efficient policies, even when trained with incorrect reward labels. The authors attribute this resilience to the pessimistic nature of offline RL algorithms and the inherent biases in data collection processes. The findings introduce a ‘survival instinct’ in the systems that inform how we understand offline RL benchmarks and create future ones. The study, therefore, recommends a new strategy for offline RL that promotes learning of desired behavior through flawed rewards but intentional bias in data coverage.\n\nSTRENGTHS:\n1. A good warm-up experiment to help readers understand the motivation.\n2. Sound theoretical analysis to show the property of ‘survival instinct’ in current offline RL algorithms. \n3. Comprehensive experiments on current D4RL and meta-world benchmarks with different offline RL algorithms.\n\nWEAKNESSES:\n1. There exists a gap between the intuition (longer trajectories in the data have a smaller optimality gap) and the definition of the Positive Data Bias. On the other hand, Positive Data Bias does not have a quantitative value for each dataset so that the reader can clearly see the difference between the different datasets.\n2. Authors claim that offline RL is intrinsically safe. However, missing relevant experiments to validate their claims.\n\nQUESTIONS:\n3. Regarding the intuition about the positive data bias, I am pretty interested in the experiments in the Antmaze dataset from D4RL, where it is a multi-goal task and there does not exist positive data bias claimed by the authors.\n4. A detailed analysis of the influence of the three types of “wrong” reward labels can make this work more complete and robust.\n5. More experiments about safe RL are highly appreciated [1].\n6. In line 247 about the safe RL. \"This condition can be easily satisfied by filtering out unsafe states in post processing.\" From the previous work [2], one biggest reasons why we prefer offline RL over BC is that offline RL can learn that those states are dangerous. What does the author think about this?\n\n[1] Liu Z, Guo Z, Lin H, et al. Datasets and Benchmarks for Offline Safe Reinforcement Learning[J]. arXiv preprint arXiv:2306.09303, 2023. \n\n[2] Kumar A, Hong J, Singh A, et al. When should we prefer offline reinforcement learning over behavioral cloning?[J]. arXiv preprint arXiv:2204.05618, 2022.\n\nLIMITATIONS:\nsee the above", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis  paper reports a very interesting new phenomenon that would be interesting to the offline RL community. It demonstrates that even when trajectories have the wrong reward labels, offline RL can learn good policies. The paper's experiments attempt to dissect the reasons for why this surprising phenomenon emerges, and argues that special to offline RL algorithms, pessimism endows the agent with survival instinct to stay within the data support, so that safe policies can be learned within these constraints.\n\nSTRENGTHS:\nThe phenomenon that has been identified is surprising; the experiments done to dissect this phenomenon are clear and solid. Overall a solid paper.\n\nWEAKNESSES:\nI have only one concern, and welcome further clarification by the authors. The authors claim that their algorithm demonstrates this robustness phenomenon arises due to the survival instinct of offline RL algorithm. But it seems from the paper that the circumstance of this only arises in the situation that the data collected for offline RL has long timescale trajectories. Is this the case? For instance, it does not arise in other circumstance investigated, like when there are multiple sources of data, or other circumstances that deviate from this central feature. \n\nIf this is the case, then the authors should state this clearly. This is still very much an interesting phenomenon, but is slightly more limited than the first impression one gets when reading the abstract and introduction that this is a potentially quite a universal phenomenon in offline RL. \n\nIf long trajectories is not the only circumstance in which this phenomenon, the authors can make the paper even more clearer by listing all the circumstnaces in which this phenomenon arise, which would be very helpful for the reader. \n\nRegardless, as long as the authors match their conclusions to their demonstrations, this paper demonstrates indeed a very interesting phenomenon arising in offline RL that is worthy of publication.\n\nQUESTIONS:\nSee questions present in the above sections. \nAnd one more question, about the suggestion at the final part of the abstract (and conclusion): \n-  if the implicit bias for long trajectories in offline data is the sole feature that is responsible for this wrong labels phenomenon, then is it still reasonable to suggest that \"whereby an agent is “nudged” to learn a desirable behavior with 20 imperfect reward but purposely biased data coverage\"? \n\nIn other words, does the \"survival instinct\" as discussed in the paper really have the sensitivity to make such a (next step) hypothesis reasonable?\n\nLIMITATIONS:\nThe authors have a limitations section, and are fair. They also nicely clarify that of course, offline RL does not always learn from wrong rewards.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors show that offline RL algorithms have an implicit survival instinct that often allows them to learn good policies with incorrect rewards. The authors argue that this is due to the data being positively biased and the pessimism that constrains offline RL algorithms to the data. This argument is supported by theory saying that when these conditions are met, the learned policy is close to the optimal policy and meets certain safety guarantees. They show several ways in which data bias can be introduced that benefit offline RL algorithms, such as length bias. Lastly, this phenomenon is shown empirically across several offline RL algorithms and benchmarks.\n\nSTRENGTHS:\n* Overall, I really enjoyed this paper. While pessimism is known to play an important role in offline RL, I think that this paper does a good job of illuminating how important pessimism is in these offline benchmarks. I think this will be an important paper for the offline RL community, especially when considering these benchmark tasks.\n\n* The paper is very well written.\n\n* The paper is very comprehensive with a strong Appendix section detailing experiment details and additional results.\n\n* The paper has a lot of theory explaining the survival instinct phenomenon and showing how algorithms in the literature have this survival instinct.\n\nWEAKNESSES:\n* I still do not fully understand the argument around length bias (see Questions), and I am not convinced that this is a main cause of the phenomenon shown in Figure 3. In particular, how do we know specifically the length of the trajectories is the important property? I would assume that the medium and medium-expert trajectories datasets just have more support over actions that prevent the agent from falling down. The length of the trajectory just seems like a byproduct of these good actions. More discussion or evidence here would be appreciated.\n\n* The main paper makes it unclear how data was generated for the grid world. While the appendix does have some more information about that, a little more should be explained here to make the example more understandable.\n     * The offline agent is used on the “wrong reward” but what is the wrong reward? It seems like in the appendix you consider three wrong rewards, but which one is used? It seems like different rewards here could change the outcome drastically.\n\n* (Line 219) The authors claim that a dataset generated by the optimal policy is infinitely positively biased regardless of the reward. This seems like a bold claim to make for all MDPs and all reward functions without a proof. \n\n* What happens with expert D4RL datasets? It would be interesting to see what performance is on this, especially since the claim is that datasets from the optimal policy are infinitely positively biased regardless of the reward.\n\n* Potential typo on Line 1415: The agent starts deterministically in the upper left it seems, not right.\n\nQUESTIONS:\n* Line 217 you say that any policy is infinitely positively biased for any rewards resulting from potential-based reward shaping. Isn’t positive bias a property of the offline dataset and not the reward? This didn’t make sense to me. \n\n* Do you have any hypotheses for why some algorithms have different levels of survival instincts? Is it because of the amount of pessimism in the algorithms?\n\n* I still have several questions about length bias:\n    * I understand that often in benchmarks longer trajectories correspond to higher returns, but it is not clear to me why this always helps. For instance, suppose we have a dataset with trajectories where half the time the robot fell and half the time it did not. If we label all rewards with -1, wouldn’t the agent be incentivized to fall as fast as possible to incur less negative rewards.\n    * Perhaps I am misunderstanding the assumptions in Proposition 5, but it seems like there is no assumption on how likely long trajectories are to appear in the dataset, only on the returns of long trajectories. I would think there has to be some assumption on the former.\n\nLIMITATIONS:\nAlthough I do not think this work necessarily needs to address these limitations, it would be beneficial to have some discussion about them in my opinion:\n\n* While the paper does cover many relevant offline RL benchmarks, it does not explore all types of datasets. In particular, it does not appear that the authors explored cases where there are only sub trajectories of good behavior that need to be stitched together.\n\n* The assumption of positively biased data in Theorem 1 seems strong. Since the definition of positively biased data is about the value of the delta-optimal policy set of the CMDP, there seems to be no intuition for when Theorem 1 will hold when looking at properties of the offline dataset only. In other words, it does not appear that there are any methods that practitioners could use to estimate the role of survival instincts for any arbitrary dataset.\n\n* While there are theoretical results for offline model-based RL works, the paper’s experiments focus on model-free offline RL algorithms only.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}