Towards Instance-Optimal Offline Reinforcement Learning with Pessimism

O H∑ h=1 ∑ sh , ah dπ ? h ( sh , ah ) √ VarPsh , ah ( V ? h+1 + rh ) dµh ( sh , ah ) √ 1 n . ( 1 ) In complementary , we also prove a per-instance information-theoretical lower bound under the weak assumption that dµh ( sh , ah ) > 0 if d π ? h ( sh , ah ) > 0 . Different from the previous minimax lower bounds , the per-instance lower bound ( via local minimaxity ) is a much stronger criterion as it applies to individual instances separately . Here π ? is a optimal policy , µ is the behavior policy and dµh is the marginal state-action probability . We call ( 1 ) the intrinsic offline reinforcement learning bound since it directly implies all the existing optimal results : minimax rate under uniform data-coverage assumption , horizon-free setting , single policy concentrability , and the tight problem-dependent results . Later , we extend the result to the assumption-free regime ( where we make no assumption on µ ) and obtain the assumption-free intrinsic bound . Due to its generic form , we believe the intrinsic bound could help illuminate what makes a specific problem hard and reveal the fundamental challenges in offline RL . 1 Introduction . In offline reinforcement learning ( offline RL Levine et al . [ 2020 ] , Lange et al . [ 2012 ] ) , the goal is to learn a reward-maximizing policy in an unknown environment ( Markov Decision Process or MDP ) using the historical data coming from a ( fixed ) behavior policy µ . Unlike online RL , where the agent can keep interacting with the environment and gain new feedback by exploring unvisited state-action space , offline RL usually populates when such online interplays are expensive or even unethical . Due to its nature of without the access to interact with the MDP model ( which causes the distributional mismatches ) , most of the literature that study the sample complexity / provable efficiency of offline RL ( e.g . Le et al . [ 2019 ] , Chen and Jiang [ 2019 ] , Xie and Jiang [ 2020 , 2021 ] , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Yin et al . [ 2021a , b ] , Ren et al . [ 2021 ] , Rashidinejad et al . [ 2021 ] , Xie et al . [ 2021b ] ) rely on making different data-coverage assumptions for making the problem learnable and provide the near-optimal worst-case performance bounds that depend on their data-coverage coefficients . Those results are valuable in general as they do not depend on the structure of the particular problem , therefore , remain valid even for pathological MDPs . But is this good enough ? In practice , the empirical performances of offline reinforcement learning ( e.g . Gulcehre et al . [ 2020 ] , Fu et al . [ 2020 , 2021 ] , Janner et al . [ 2021 ] ) are often far better than what those non-adaptive / problem-independent bounds would indicate . Although empirical evidence can help explain why we may observe better or worse performances on different MDPs , a systematic understanding of what types of decision processes and what kinds of behavior policies are inherently easier or more challenging for offline RL is lacking . Besides , despite the fact that a non-adaptive bound can learn even the pathological examples within the assumption family , there is no guarantee for the instances outside the family . However , practical offline reinforcement learning problems are usually beyond the scope of certain data-coverage assumptions , which limits the applicability of those results . Can we make as few assumptions as possible ? Or even more , what can we guarantee when no assumption is made about offline learning ? Those motivate us to derive the provably efficient bounds that are adaptive to the individual instances but only require minimal assumptions so they can be widely applied in most cases . Ideally , such bounds should characterize the system structures of the specific problems , hold even for peculiar instances that do not satisfy the standard data-coverage assumptions , and recover the worst-case guarantees when the assumptions are satisfied . As mentioned in Zanette and Brunskill [ 2019 ] , a fully adaptive characterization in RL is important as it might bring considerable saving in the time spent designing domain-specific RL solutions and in training a human expert to judge and recognize the complexity of different problems . 1.1 Our contribution . In this work , we provide the analysis for the adaptive pessimistic value iteration ( APVI ) ( Algorithm 1 ) with finite horizon time-inhomogeneous ( non-stationary ) MDPs and derive a strong adaptive bound that is near-optimal under the weak assumption dµh ( sh , ah ) > 0 if d π ? h ( sh , ah ) > 0 ( Theorem 4.1 ) . Specifically , our bound ( quantity ( 1 ) ) explicitly depends on the marginal importance ratios ( between the optimal policy π ? and the behavior policy µ ) and the per-step conditional variances . In addition , we provide an instance-dependent ( local minimax ) lower bound ( Theorem 4.3 ) to certify ( 1 ) is nearly optimal at the instance level for offline learning and call it the intrinsic offline learning bound . The intrinsic bound has the following consequences . • In the non-adaptive / worst-case regime ( 4.1-4.3 ) , the intrinsic bound implies Õ ( H3/dm 2 ) complexity under the uniform data-coverage 2.1 , Õ ( H3SC ? / 2 ) complexity under the single policy concentrability assumption 2.3 and Õ ( H/dm 2 ) complexity when the sum of rewards is bounded by 1 . All of those are optimal in their respectively regimes [ Yin et al. , 2021a , Rashidinejad et al. , 2021 , Xie et al. , 2021b , Ren et al. , 2021 ] ; • In the adaptive domain ( 4.4 ) , the intrinsic bound implies the tight problem-dependent counterpart of Zanette and Brunskill [ 2019 ] , yields Õ ( H3/ndm ) fast convergence in the deterministic systems , has improved complexity in the partially deterministic systems and a family of highly mixing problems , and remains optimal when reducing to the tabular contextual bandits . Beyond the above , due to the generic form of the intrinsic bound , we could come up with as many problem instances ( that are of our interests ) as possible and study their properties . In this sense , the intrinsic bound helps illuminate the fundamental nature of offline RL . Furthermore , as a step towards assumption-free offline reinforcement learning , we build a modified AVPI and obtain an adaptive bound that could characterize the suboptimality gap in the state-action space that is agnostic to the behavior policy ( Theorem 5.1 ) . To the best of our knowledge , all of these results are the first of its kinds . 1.2 Related work . Finite sample analysis for offline reinforcement learning can be traced back to Szepesvári and Munos [ 2005 ] , Antos et al . [ 2008a , b ] for the infinite horizon discounted setting via Fitted Q-Iteration ( FQI ) type function approximation algorithms . [ Chen and Jiang , 2019 , Le et al. , 2019 , Xie and Jiang , 2021 , 2020 ] follow this line of research and derive the information-theoretical bounds . Recently , Xie and Jiang [ 2021 ] considers the offline RL with only the realizability assumption , Liu et al . [ 2020 ] , Chang et al . [ 2021 ] considers the offline RL without sufficient coverage and Kidambi et al . [ 2020 ] , Uehara and Sun [ 2021 ] uses the model-based approach for addressing offline RL . Under those weak coverage assumption , their finite sample analysis are suboptimal ( e.g . in terms of the effective horizon ( 1− γ ) −1 ) . Recently , Yin et al . [ 2021a , b ] , Ren et al . [ 2021 ] study the finite horizon case . In the linear MDP case , Jin et al . [ 2020 ] studies the pessimistic algorithm for offline policy learning under only the compliance assumption , and , concurrently , Xie et al . [ 2021a ] proposes the general pessimistic function approximation framework with instantiation in linear MDP and Zanette et al . [ 2021 ] shows actor-critic style algorithm is near-optimal for linear Bellman complete model . In addition , Wang et al . [ 2021 ] , Zanette [ 2021 ] prove some exponential lower bounds under their linear function approximation assumptions . Among them , there are a few works that achieve the sample optimality under their respective assumptions . Under the uniform data coverage ( minimal state-action probability dm > 0 ) , Yin et al . [ 2021a ] first proves the optimal Õ ( H3/dm 2 ) complexity in the time-inhomogeneous MDP . Recently , Yin et al . [ 2021b ] designs the offline variance reduction algorithm to achieve the optimal Õ ( H2/dm 2 ) rate for the time-homogeneous case . Under the setting where the total cumulative reward is bounded by 1 , Ren et al . [ 2021 ] obtains the horizon-free result with Õ ( 1/dm ) . More recently , Rashidinejad et al . [ 2021 ] considers the single concentrability coefficient C ? : = maxs , a dπ ? ( s , a ) /dµ ( s , a ) and derives the upper bound Õ [ ( 1 − γ ) −5SC ? / 2 ] in the infinite horizon setting which is recently improved by the concurrent work Xie et al . [ 2021b ] . While those worst-case guarantees are desirable , none of them can explain the hardness of the individual problems.1 2 Preliminaries . Episodic non-stationary ( time-varying ) reinforcement learning . A finite-horizon Markov Decision Process ( MDP ) is denoted by a tuple M = ( S , A , P , r , H , d1 ) [ Sutton and Barto , 2018 ] , where S is the finite state space and A is the finite action space with S : = |S| < ∞ , A : = |A| < ∞ . A nonstationary transition kernel Ph : S ×A× S 7→ [ 0 , 1 ] maps each state action ( sh , ah ) to a probability distribution Ph ( ·|sh , ah ) and Ph can be different across the time . Besides , r : S ×A 7→ R is the expected instantaneous reward function satisfying 0 ≤ r ≤ 1. d1 is the initial state distribution . H is the horizon . A policy π = ( π1 , . . . , πH ) assigns each state sh ∈ S a probability distribution over actions according to the map sh 7→ πh ( ·|sh ) ∀h ∈ [ H ] . An MDP together with a policy π induce a random trajectory s1 , a1 , r1 , . . . , sH , aH , rH , sH+1 with s1 ∼ d1 , ah ∼ π ( ·|sh ) , sh+1 ∼ Ph ( ·|sh , a ) , ∀h ∈ [ H ] and rh is a random realization given the observed sh , ah . Q-values , Bellman ( optimality ) equations . The value function V πh ( · ) ∈ RS and Q-value function Qπh ( · , · ) ∈ RS×A for any policy π is defined as : V πh ( s ) = Eπ [ ∑H t=h rt|sh = s ] , Qπh ( s , a ) = Eπ [ ∑H t=h rt|sh , ah = s , a ] , ∀s , a ∈ S , A , h ∈ [ H ] . The performance is defined as vπ : = Ed1 [ V π1 ] = Eπ , d1 [ ∑H t=1 rt ] , where we denote V πh , Q π h as column vectors and Ph ∈ RSA×S the transition matrix , then the vector form Bellman ( optimality ) equations follow ∀h ∈ [ H ] : Qπh = rh + PhV π h+1 , V π h = Ea∼πh [ Qπh ] , Q ? h = rh + PhV ? h+1 , V ? h = maxaQ ? h ( · , a ) .In addition , we denote the per-step marginal state-action occupancy dπh ( s , a ) as : d π h ( s , a ) : = P [ sh = s|s1 ∼ d1 , π ] · πh ( a|s ) , which is the marginal state-action probability at time h. Offline setting and the goal . The offline RL requires the agent to find a policy π such that the performance vπ is maximized , given only the episodic data D = { ( sτh , aτh , rτh , sτh+1 ) } h∈ [ H ] τ∈ [ n ] rolled out from some behavior policy µ . The offline nature requires we can not change µ and in particular we 1We do mention Zanette et al . [ 2021 ] is near-optimal in their setting , but it is unclear whether it remains optimal in the standard setting where Qπ ∈ [ 0 , H ] , since there is an additional H factor by rescaling . do not assume the functional knowledge of µ . That is to say , given the batch data D and a targeted accuracy > 0 , the offline RL seeks to find a policy πalg such that v ? − vπalg ≤ .

Summary: This paper studies the capability and intrinsic hardness of pessimistic value iteration algorithm in the finite-horizon tabular MDP case when data is i.i.d. collected from a behavior policy. By a fine-grained analysis, the PEVI (reduced to the i.i.d. tabular case) is modified to incorporate empirical variance into the pessimistic penalty term. These empirical quantities converge to their population counterparts, which is also shown by a lower bound as the intrinsic hardness of learning the MDP with data collected by the policy. Extensive examples & special cases are analyzed to provide a comprehensive analysis of pessimistic offline RL in this setting.