Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism

√ N in offline RL , where N is the batch dataset sample size . In contextual bandits , we prove that LCB is adaptively optimal for the entire data composition range , achieving a smooth transition from imitation learning to offline RL . We further show that LCB is almost adaptively optimal in tabular MDPs . 1 Introduction . Reinforcement learning ( RL ) algorithms have recently achieved tremendous empirical success including beating Go champions [ 45 , 46 ] and surpassing professionals in Atari games [ 30 , 31 ] , to name a few . Most success stories , however , are in the realm of online RL in which active data collection is necessary . This online paradigm falls short of leveraging previously-collected datasets and dealing with scenarios where online exploration is not possible [ 10 ] . To tackle these issues , offline 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . ( or batch ) reinforcement learning [ 24 , 26 ] arises in which the agent aims at achieving competence by exploiting a batch dataset without access to online exploration . This paradigm is useful in a diverse array of application domains such as healthcare [ 51 , 15 , 35 ] , autonomous driving [ 57 , 4 , 36 ] , and recommendation systems [ 47 , 12 , 49 ] . The key component of offline RL is a pre-collected dataset from an unknown stochastic environment . Broadly speaking , there exist two types of data composition for which offline RL algorithms have shown promising empirical and theoretical success ; see Figure 1 for an illustration . Expert data . One end of the spectrum includes datasets collected by following an expert policy . For such datasets , imitation learning algorithms ( e.g. , behavior cloning [ 40 ] ) are shown to be effective in achieving a small sub-optimality w.r.t . the expert policy . Recently , Rajaraman et al . [ 39 ] showed that the behavior cloning algorithm achieves the minimal sub-optimality of 1/N in episodic Markov decision processes ( MDPs ) , where N is the sample size in the expert dataset . Uniform coverage data . On the other end of the spectrum lies the datasets with uniform coverage , which aim to cover all states and actions , even the states never visited or actions never taken by satisfactory policies . Most vanilla offline RL algorithms are only suited in this region and are shown to diverge—both empirically [ 11 , 22 ] and theoretically [ 2 , 7 ] —for narrower datasets [ 10 , 20 ] , such as those collected via human demonstrations or hand-crafted policies . In this regime , a widely-adopted requirement is the bounded uniform concentrability coefficient which assumes that the ratio of the state-action occupancy density of any policy and the data distribution is bounded uniformly over all states and actions [ 32 , 9 , 6 , 52 ] . Another common assumption is uniformly lower bounded data distribution on all states and actions [ 43 , 1 ] , which ensures all states and actions are visited with sufficient probabilities . Algorithms developed for this regime are demonstrated to achieve a 1/ √ N sub-optimality competing with the optimal policy [ 53 , 16 , 50 ] . 1.1 Motivating questions . Both of these two ends impose strong assumptions on the dataset : at one extreme , we hope for a solely expert-driven dataset ; at the other extreme , we require the dataset to cover every , even sub-optimal , actions . In practice , there are numerous scenarios where the dataset deviates from these two extremes , which has motivated new offline RL benchmark datasets with different data compositions [ 10 , 20 ] . With this need in mind , the first and foremost question is regarding offline RL formulations : Question 1 ( Formulation ) Can we propose an offline RL framework that accommodates the entire data composition range ? We answer this question affirmatively by proposing a new formulation for offline RL that smoothly interpolates between two regimes : expert data and data with uniform coverage . More specifically , we characterize the data composition in terms of the ratio between the state-action occupancy density of an optimal policy d ? ( s , a ) 1 and that of the data distribution µ ( s , a ) , i.e. , we define C ? to be the smallest constant that satisfies d ? ( s , a ) /µ ( s , a ) ≤ C ? for all s ∈ S and a ∈ A ; see Definition 1 for a precise characterization . In words , C ? can be viewed as a measure of the deviation between the data distribution and the distribution induced by the optimal policy . C ? = 1 describes the expert datasets as by definition , the behavior policy is identical to the optimal policy . In contrast , when C ? > 1 , the dataset is no longer purely expert-driven : it could contain “ spurious ” samples—states and actions that are not visited by the optimal policy . As another example , when the data distribution is lower bounded by µmin over all states and actions , C ? is upper bounded by µ−1min . 1Our developments can accommodate arbitrary competing policies , however , we restrict ourselves to the optimal policy for ease of presentation . Assuming a finite C ? is the weakest concentrability requirement [ 42 , 13 , 52 ] that is currently enjoyed only by some online algorithms such as CPI [ 18 ] . C ? imposes a much weaker assumption in contrast to other concentrability requirements which involve taking a maximum over all policies ; see [ 42 ] for a hierarchy of different concentrability definitions . We would like to immediately point out that existing works on offline RL either do not specify the dependency of sub-optimality on data coverage [ 17 , 55 ] , or do not have a batch data coverage assumption that accommodates the entire data spectrum [ 54 , 19 ] . For instance , Yin et al . [ 54 ] requires a uniformly lower bounded data distribution that traces an optimal policy , which implies that optimal actions should be included in states not visited by the optimal policy . Furthermore , this characterization of data coverage does not recover imitation learning : even if the behavior policy is exactly equal to the optimal policy , data distribution lower bound can be arbitrarily small . Further discussion of related work is presented in Appendix A . With this formulation in mind , a natural next step is designing offline RL algorithms that can handle various data compositions , i.e. , for all C ? ≥ 1 . Recently , efforts have been made toward reducing the offline dataset requirements based on a shared intuition : the agent should act conservatively and avoid states and actions less covered in the offline dataset . Based on this intuition , a variety of model-based [ 55 , 19 , 56 ] and model-free [ 22 , 33 , 11 , 34 , 25 , 37 , 44 , 14 , 28 , 23 , 3 ] offline RL algorithms are proposed that achieve promising empirical results . However , it is observed empirically that existing model-free methods perform better when the dataset is nearly expert-driven whereas existing model-based methods perform better when the dataset is randomly-collected [ 55 , 5 , 56 ] . It remains unclear whether a single algorithm exists that performs well regardless of data composition— an important challenge from a practical perspective [ 21 , 10 , 20 ] . More importantly , the knowledge of the dataset composition may not be available a priori to assist in selecting the right algorithm . In practice , imitation learning often succeeds with very few samples in contrast to offline RL [ 41 ] . Unifying offline RL and imitation learning via a single algorithm is thus beneficial , as it can result in tremendous sample savings , in case the dataset has good coverage on an expert policy . This motivates the second question on the algorithm design : Question 2 ( Adaptive algorithm design ) Can we design algorithms that can achieve minimal suboptimality when facing different dataset compositions ( i.e. , different C ? ) ? Furthermore , can this be achieved in an adaptive manner , i.e. , without knowing C ? beforehand ? To answer the second question , we analyze a pessimistic variant of a value-based method in which we first form a lower confidence bound ( LCB ) for the value function of a policy using the batch data and then seek to find a policy that maximizes the LCB . The idea of pessimism has appeared in the literature of risk minimization [ 48 , 8 ] . A similar algorithm design has recently appeared in [ 17 ] . It turns out that such a simple algorithm—fully agnostic to the data composition—achieves almost optimal performance in multi-armed bandits and MDPs , and optimally solves the offline learning problem in contextual bandits . Results summary . Figure 2 summarizes our theoretical findings . For multi-armed bandits , we prove that LCB achieves a √ C ? /N sub-optimality for any C ? ≥ 1 . Yet , we prove lower bounds showing that LCB can not be adaptively optimal for any C ? ≥ 1 if the knowledge of C ? is not available . For contextual bandits with at least two contexts , we prove that LCB enjoys a rate of√ ( C ? − 1 ) /N + 1/N , which translates to a fast rate of 1/N when C ? ≈ 1 akin to the performance of behavioral cloning and smoothly transitions from 1/N to 1/ √ N as C ? increases . This rate matches the information theoretic limit , showing adaptive optimality of LCB in contextual bandits . Establishing the C ? −1 dependency requires a novel analysis based on a careful policy sub-optimality decomposition and directly analyzing the probability of taking wrong actions . For MDPs , we similarly show that LCB achieves a fast rate of 1/N for C ? ≈ 1 and a rate of √ C ? /N for larger values of C ? . We conjecture that LCB upper bound also has the form √ ( C ? − 1 ) /N . We verify this conjecture in a simple example and show that establishing the C ? − 1 dependency in MDPs requires a delicate analysis that accounts for the value gap between optimal and sub-optimal actions . 2 Background and problem formulation . Notation . The probability simplex over set X is denoted by ∆ ( X ) . We write x . y when there exists c > 0 such that x ≤ cy and write x y if c1 , c2 > 0 exist such that c1|x| ≤ |y| ≤ c2|x| . We write x ∨ y to denote the supremum of x and y . We write f ( x ) = O ( g ( x ) ) if M > 0 , x0 exist such that |f ( x ) | ≤ Mg ( x ) for all x ≥ x0 . We use Õ ( · ) to be the big-O notation ignoring logarithmic factors . We write f ( x ) = Ω ( g ( x ) ) if M > 0 , x0 exist such that |f ( x ) | ≥Mg ( x ) for all x ≥ x0 . 2.1 Markov decision processes . An infinite-horizon discounted MDP is described by a tuple M = ( S , A , P , R , ρ , γ ) , where S is a finite state space with S = |S| , A is a finite action space , P : S ×A 7→ ∆ ( S ) is a transition matrix , R : S × A 7→ ∆ ( [ 0 , 1 ] ) encodes a family of reward distributions with r : S × A 7→ [ 0 , 1 ] as the expected reward function , ρ : S 7→ ∆ ( S ) is the initial distribution , and γ ∈ [ 0 , 1 ) is a discount factor . A stationary deterministic policy π : S 7→ A is a function that maps a state to an action . Correspondingly , the ( normalized ) state-action discounted occupancy measure dπ : S ×A 7→ [ 0 , 1 ] is defined as dπ ( s , a ) : = ( 1−γ ) ∑∞ t=0 γ t Pt ( st = s , at = a ; π ) for all s ∈ S , a ∈ A , where Pt ( st = s , at = a ; π ) denotes the probability of st = s , at = a after executing policy π and starting from s0 ∼ ρ ( · ) . The value function V π : S 7→ R of the policy π is defined as V π ( s ) : = E [ ∑∞ t=0 γ trt | s0 = s , at = π ( st ) for all t ≥ 0 ] for s ∈ S. The quality function ( or Q-function ) Qπ : S × A → R of policy π is defined analogously Qπ ( s , a ) : = E [ ∑∞ t=0 γ trt | s0 = s , a0 = a , at = π ( st ) for all t ≥ 1 ] . It is convenient to define a scalar summary of the performance of a policy π as J ( π ) : = Es∼ρ [ V π ( s ) ] . It is well known [ 38 ] that a stationary deterministic policy π ? exists that simultaneously maximizes V π ( s ) for all s ∈ S , and hence maximizing the expected value J ( π ) . We use shorthands V ? : = V π ? and Q ? : = Qπ ? to denote the optimal value function and Q-function .

Summary: This is a strong paper that is well-motivated and well-written. I enjoy reading the paper, thinking about the questions and learning the new results.  Contribution on problem formulation: The paper provides a new offline RL framework that accommodates the entire data composition range, smoothly interpolating between two regimes: expert data and data with uniform coverage.   Contribution on algorithm design and the upper bounds: The paper provides a lower-confidence-bound algorithm that achieves almost optimal performance in multi-armed bandits and MDPs, and optimally solves the offline learning problem in contextual bandits. It is interesting to see that in all the three settings, LCB achieves a faster rate of 1/N for nearly-expert datasets compared to the usual rate of 1/\sqrt{N} in offline RL.  Contribution on the lower bounds: The paper provides information-theoretical lower bounds to capture the fundamental hardness for this new offline RL framework. The paper also has interesting discussion on whether the knowledge of C* is necessary to design optimal algorithm.