Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism

√ N in offline RL , where N is the batch dataset sample size . In contextual bandits , we prove that LCB is adaptively optimal for the entire data composition range , achieving a smooth transition from imitation learning to offline RL . We further show that LCB is almost adaptively optimal in tabular MDPs . 1 Introduction . Reinforcement learning ( RL ) algorithms have recently achieved tremendous empirical success including beating Go champions [ 45 , 46 ] and surpassing professionals in Atari games [ 30 , 31 ] , to name a few . Most success stories , however , are in the realm of online RL in which active data collection is necessary . This online paradigm falls short of leveraging previously-collected datasets and dealing with scenarios where online exploration is not possible [ 10 ] . To tackle these issues , offline 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . ( or batch ) reinforcement learning [ 24 , 26 ] arises in which the agent aims at achieving competence by exploiting a batch dataset without access to online exploration . This paradigm is useful in a diverse array of application domains such as healthcare [ 51 , 15 , 35 ] , autonomous driving [ 57 , 4 , 36 ] , and recommendation systems [ 47 , 12 , 49 ] . The key component of offline RL is a pre-collected dataset from an unknown stochastic environment . Broadly speaking , there exist two types of data composition for which offline RL algorithms have shown promising empirical and theoretical success ; see Figure 1 for an illustration . Expert data . One end of the spectrum includes datasets collected by following an expert policy . For such datasets , imitation learning algorithms ( e.g. , behavior cloning [ 40 ] ) are shown to be effective in achieving a small sub-optimality w.r.t . the expert policy . Recently , Rajaraman et al . [ 39 ] showed that the behavior cloning algorithm achieves the minimal sub-optimality of 1/N in episodic Markov decision processes ( MDPs ) , where N is the sample size in the expert dataset . Uniform coverage data . On the other end of the spectrum lies the datasets with uniform coverage , which aim to cover all states and actions , even the states never visited or actions never taken by satisfactory policies . Most vanilla offline RL algorithms are only suited in this region and are shown to diverge—both empirically [ 11 , 22 ] and theoretically [ 2 , 7 ] —for narrower datasets [ 10 , 20 ] , such as those collected via human demonstrations or hand-crafted policies . In this regime , a widely-adopted requirement is the bounded uniform concentrability coefficient which assumes that the ratio of the state-action occupancy density of any policy and the data distribution is bounded uniformly over all states and actions [ 32 , 9 , 6 , 52 ] . Another common assumption is uniformly lower bounded data distribution on all states and actions [ 43 , 1 ] , which ensures all states and actions are visited with sufficient probabilities . Algorithms developed for this regime are demonstrated to achieve a 1/ √ N sub-optimality competing with the optimal policy [ 53 , 16 , 50 ] . 1.1 Motivating questions . Both of these two ends impose strong assumptions on the dataset : at one extreme , we hope for a solely expert-driven dataset ; at the other extreme , we require the dataset to cover every , even sub-optimal , actions . In practice , there are numerous scenarios where the dataset deviates from these two extremes , which has motivated new offline RL benchmark datasets with different data compositions [ 10 , 20 ] . With this need in mind , the first and foremost question is regarding offline RL formulations : Question 1 ( Formulation ) Can we propose an offline RL framework that accommodates the entire data composition range ? We answer this question affirmatively by proposing a new formulation for offline RL that smoothly interpolates between two regimes : expert data and data with uniform coverage . More specifically , we characterize the data composition in terms of the ratio between the state-action occupancy density of an optimal policy d ? ( s , a ) 1 and that of the data distribution µ ( s , a ) , i.e. , we define C ? to be the smallest constant that satisfies d ? ( s , a ) /µ ( s , a ) ≤ C ? for all s ∈ S and a ∈ A ; see Definition 1 for a precise characterization . In words , C ? can be viewed as a measure of the deviation between the data distribution and the distribution induced by the optimal policy . C ? = 1 describes the expert datasets as by definition , the behavior policy is identical to the optimal policy . In contrast , when C ? > 1 , the dataset is no longer purely expert-driven : it could contain “ spurious ” samples—states and actions that are not visited by the optimal policy . As another example , when the data distribution is lower bounded by µmin over all states and actions , C ? is upper bounded by µ−1min . 1Our developments can accommodate arbitrary competing policies , however , we restrict ourselves to the optimal policy for ease of presentation . Assuming a finite C ? is the weakest concentrability requirement [ 42 , 13 , 52 ] that is currently enjoyed only by some online algorithms such as CPI [ 18 ] . C ? imposes a much weaker assumption in contrast to other concentrability requirements which involve taking a maximum over all policies ; see [ 42 ] for a hierarchy of different concentrability definitions . We would like to immediately point out that existing works on offline RL either do not specify the dependency of sub-optimality on data coverage [ 17 , 55 ] , or do not have a batch data coverage assumption that accommodates the entire data spectrum [ 54 , 19 ] . For instance , Yin et al . [ 54 ] requires a uniformly lower bounded data distribution that traces an optimal policy , which implies that optimal actions should be included in states not visited by the optimal policy . Furthermore , this characterization of data coverage does not recover imitation learning : even if the behavior policy is exactly equal to the optimal policy , data distribution lower bound can be arbitrarily small . Further discussion of related work is presented in Appendix A . With this formulation in mind , a natural next step is designing offline RL algorithms that can handle various data compositions , i.e. , for all C ? ≥ 1 . Recently , efforts have been made toward reducing the offline dataset requirements based on a shared intuition : the agent should act conservatively and avoid states and actions less covered in the offline dataset . Based on this intuition , a variety of model-based [ 55 , 19 , 56 ] and model-free [ 22 , 33 , 11 , 34 , 25 , 37 , 44 , 14 , 28 , 23 , 3 ] offline RL algorithms are proposed that achieve promising empirical results . However , it is observed empirically that existing model-free methods perform better when the dataset is nearly expert-driven whereas existing model-based methods perform better when the dataset is randomly-collected [ 55 , 5 , 56 ] . It remains unclear whether a single algorithm exists that performs well regardless of data composition— an important challenge from a practical perspective [ 21 , 10 , 20 ] . More importantly , the knowledge of the dataset composition may not be available a priori to assist in selecting the right algorithm . In practice , imitation learning often succeeds with very few samples in contrast to offline RL [ 41 ] . Unifying offline RL and imitation learning via a single algorithm is thus beneficial , as it can result in tremendous sample savings , in case the dataset has good coverage on an expert policy . This motivates the second question on the algorithm design : Question 2 ( Adaptive algorithm design ) Can we design algorithms that can achieve minimal suboptimality when facing different dataset compositions ( i.e. , different C ? ) ? Furthermore , can this be achieved in an adaptive manner , i.e. , without knowing C ? beforehand ? To answer the second question , we analyze a pessimistic variant of a value-based method in which we first form a lower confidence bound ( LCB ) for the value function of a policy using the batch data and then seek to find a policy that maximizes the LCB . The idea of pessimism has appeared in the literature of risk minimization [ 48 , 8 ] . A similar algorithm design has recently appeared in [ 17 ] . It turns out that such a simple algorithm—fully agnostic to the data composition—achieves almost optimal performance in multi-armed bandits and MDPs , and optimally solves the offline learning problem in contextual bandits . Results summary . Figure 2 summarizes our theoretical findings . For multi-armed bandits , we prove that LCB achieves a √ C ? /N sub-optimality for any C ? ≥ 1 . Yet , we prove lower bounds showing that LCB can not be adaptively optimal for any C ? ≥ 1 if the knowledge of C ? is not available . For contextual bandits with at least two contexts , we prove that LCB enjoys a rate of√ ( C ? − 1 ) /N + 1/N , which translates to a fast rate of 1/N when C ? ≈ 1 akin to the performance of behavioral cloning and smoothly transitions from 1/N to 1/ √ N as C ? increases . This rate matches the information theoretic limit , showing adaptive optimality of LCB in contextual bandits . Establishing the C ? −1 dependency requires a novel analysis based on a careful policy sub-optimality decomposition and directly analyzing the probability of taking wrong actions . For MDPs , we similarly show that LCB achieves a fast rate of 1/N for C ? ≈ 1 and a rate of √ C ? /N for larger values of C ? . We conjecture that LCB upper bound also has the form √ ( C ? − 1 ) /N . We verify this conjecture in a simple example and show that establishing the C ? − 1 dependency in MDPs requires a delicate analysis that accounts for the value gap between optimal and sub-optimal actions . 2 Background and problem formulation . Notation . The probability simplex over set X is denoted by ∆ ( X ) . We write x . y when there exists c > 0 such that x ≤ cy and write x y if c1 , c2 > 0 exist such that c1|x| ≤ |y| ≤ c2|x| . We write x ∨ y to denote the supremum of x and y . We write f ( x ) = O ( g ( x ) ) if M > 0 , x0 exist such that |f ( x ) | ≤ Mg ( x ) for all x ≥ x0 . We use Õ ( · ) to be the big-O notation ignoring logarithmic factors . We write f ( x ) = Ω ( g ( x ) ) if M > 0 , x0 exist such that |f ( x ) | ≥Mg ( x ) for all x ≥ x0 . 2.1 Markov decision processes . An infinite-horizon discounted MDP is described by a tuple M = ( S , A , P , R , ρ , γ ) , where S is a finite state space with S = |S| , A is a finite action space , P : S ×A 7→ ∆ ( S ) is a transition matrix , R : S × A 7→ ∆ ( [ 0 , 1 ] ) encodes a family of reward distributions with r : S × A 7→ [ 0 , 1 ] as the expected reward function , ρ : S 7→ ∆ ( S ) is the initial distribution , and γ ∈ [ 0 , 1 ) is a discount factor . A stationary deterministic policy π : S 7→ A is a function that maps a state to an action . Correspondingly , the ( normalized ) state-action discounted occupancy measure dπ : S ×A 7→ [ 0 , 1 ] is defined as dπ ( s , a ) : = ( 1−γ ) ∑∞ t=0 γ t Pt ( st = s , at = a ; π ) for all s ∈ S , a ∈ A , where Pt ( st = s , at = a ; π ) denotes the probability of st = s , at = a after executing policy π and starting from s0 ∼ ρ ( · ) . The value function V π : S 7→ R of the policy π is defined as V π ( s ) : = E [ ∑∞ t=0 γ trt | s0 = s , at = π ( st ) for all t ≥ 0 ] for s ∈ S. The quality function ( or Q-function ) Qπ : S × A → R of policy π is defined analogously Qπ ( s , a ) : = E [ ∑∞ t=0 γ trt | s0 = s , a0 = a , at = π ( st ) for all t ≥ 1 ] . It is convenient to define a scalar summary of the performance of a policy π as J ( π ) : = Es∼ρ [ V π ( s ) ] . It is well known [ 38 ] that a stationary deterministic policy π ? exists that simultaneously maximizes V π ( s ) for all s ∈ S , and hence maximizing the expected value J ( π ) . We use shorthands V ? : = V π ? and Q ? : = Qπ ? to denote the optimal value function and Q-function .

Summary: This work presents a simple algorithm for offline reinforcement learning based on optimizing the sum of lower confidence bounds of rewards for each transition (where the confidence bound is constricted using the offline dataset of transitions).  Theoretical bounds are given on the suboptimality of the resulting policy, based on the quantity $C^*$, which bounds the proportion of non-optimal transitions in the dataset.  Conditional on there being a finite ratio of non-optimal to optimal actions in each state, the presented algorithm has good performance in both the "expert data" case, in which the behavioral policy that generated the data is close to optimal, and the "uniform coverage" case, in which the behavioral policy covers all possible transitions.