Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimism

Offline reinforcement learning , which seeks to utilize offline/historical data to optimize sequential decision-making strategies , has gained surging prominence in recent studies . Due to the advantage that appropriate function approximators can help mitigate the sample complexity burden in modern reinforcement learning problems , existing endeavors usually enforce powerful function representation models ( e.g . neural networks ) to learn the optimal policies . However , a precise understanding of the statistical limits with function representations , remains elusive , even when such a representation is linear . Towards this goal , we study the statistical limits of offline reinforcement learning with linear model representations . To derive the tight offline learning bound , we design the variance-aware pessimistic value iteration ( VAPVI ) , which adopts the conditional variance information of the value function for time-inhomogeneous episodic linear Markov decision processes ( MDPs ) . VAPVI uses the estimated variances of the value functions to reweight the Bellman residuals in the least-square pessimistic value iteration and provides improved offline learning bounds over the existing best-known results ( whereas the Bellman residuals are equally weighted ) . More importantly , our learning bounds are expressed in terms of system quantities , which provide natural instance-dependent characterizations that previous results are short of . We hope our results draw a clearer picture of what offline learning should look like when linear representations are provided . 1 INTRODUCTION . Offline reinforcement learning ( offline RL or batch RL Lange et al . ( 2012 ) ; Levine et al . ( 2020 ) ) is the framework for learning a reward-maximizing policy in an unknown environment ( Markov Decision Process or MDP ) 1 using the logged data coming from some behavior policy µ . Function approximations , on the other hand , are well-known for generalization in standard supervised learning . Offline RL with function representation/approximation , as a result , provides generalization across large state-action spaces for the challenging sequential decision-making problems when no iteration is allowed ( as opposed to online learning ) . This paradigm is crucial to the success of modern RL problems as many deep RL algorithms find their prototypes in the literature of offline RL . For example , Xie and Jiang ( 2020 ) provides a view that Fitted Q-Iteration Gordon ( 1999 ) ; Ernst et al . ( 2005 ) can be considered as the theoretical prototype of the deep Q-networks algorithm ( DQN ) Mnih et al . ( 2015 ) with neural networks being the function representors . On the empirical side , there are a huge body of deep RL-based algorithms Mnih et al . ( 2015 ) ; Silver et al . ( 2017 ) ; Fujimoto et al . ( 2019 ) ; Kumar et al . ( 2019 ) ; Wu et al . ( 2019 ) ; Kidambi et al . ( 2020 ) ; Yu et al . ( 2020 ) ; Kumar et al . ( 2020 ) ; Janner et al . ( 2021 ) ; Chen et al . ( 2021a ) that utilize function approximations to achieve respective successes in the offline regime . However , it is also realized that practical function approximation schemes can be quite sample inefficient ( e.g . millions of samples are needed for deep Q-network to solve certain Atari games Mnih et al . ( 2015 ) ) . 1The environment could have other forms as well , e.g . partially-observed MDP ( POMDP ) or non-markovian decision process ( NMDP ) . To understand this phenomenon , there are numerous studies consider how to achieve sample efficiency with function approximation from the theoretical side , as researchers find sample efficient algorithms are possible with particular model representations , in either online RL ( e.g . Yang and Wang ( 2019 ; 2020 ) ; Modi et al . ( 2020 ) ; Jin et al . ( 2020 ) ; Ayoub et al . ( 2020 ) ; Jiang et al . ( 2017 ) ; Du et al . ( 2019 ) ; Sun et al . ( 2019 ) ; Zanette et al . ( 2020 ) ; Zhou et al . ( 2021 ) ; Jin et al . ( 2021a ) ; Du et al . ( 2021 ) ) or offline RL ( e.g . Munos ( 2003 ) ; Chen and Jiang ( 2019 ) ; Xie and Jiang ( 2020 ) ; Jin et al . ( 2021b ) ; Xie et al . ( 2021a ) ; Duan et al . ( 2021 ) ; Nguyen-Tang et al . ( 2021 ) ; Zanette et al . ( 2021 ) ) . Among them , the linear MDP model ( Yang and Wang , 2020 ; Jin et al. , 2020 ) , where the transition is represented as a linear combinations of the given d-dimensional feature , is ( arguably ) the most studied setting in function approximation and there are plenty of extensions based upon it ( e.g . generalized linear model ( Wang et al. , 2021b ) , reward-free RL ( Wang et al. , 2020 ) , gap-dependent analysis ( He et al. , 2021 ) or generative adversarial learning ( Liu et al. , 2021 ) ) . Given its prosperity , however , there are still unknowns for understanding function representations in RL , especially in the offline case . • While there are surging researches in showing provable sample efficiency ( polynomial sample complexity is possible ) under a variety of function approximation schemes , how to improve the sample efficiency for a given class of function representations remains understudied . For instance , given a neural network approximation class , an algorithm that learns the optimal policy with complexity O ( H10 ) is far worse than the one that can learn in O ( H3 ) sample complexity , despite that both algorithms are considered sample efficient . Therefore , how to achieve the “ optimal ” /tight sample complexity when function approximation is provided is a valuable question to consider . On the other hand , it is known that tight sample complexity , due to the limit of the existing statistical analysis tools , can be very tough to obtain when function representation has a very complicated form . However , does this mean tight analysis is not hopeful even when the representation is linear ? • Second , in the existing analysis of offline RL ( with function approximation or simply the tabular MDPs ) , the learning bounds depend either explicitly on the data-coverage quantities ( e.g . uniform concentrability coefficients Chen and Jiang ( 2019 ) ; Xie and Jiang ( 2020 ) , uniform visitation measure Yin et al . ( 2021 ) ; Yin and Wang ( 2021a ) and single concentrability Rashidinejad et al . ( 2021 ) ; Xie et al . ( 2021b ) ) or the horizon length H ( Jin et al. , 2021b ; Uehara and Sun , 2021 ) . While those results are valuable as they do not depend on the structure of the particular problem ( therefore , remain valid even for pathological MDPs ) , in practice , the empirical performances of offline reinforcement learning are often far better than those non-adaptive bounds would indicate . Can the learning bounds reflect the nature of individual MDP instances when the MDP model has a certain function representation ? In this work , we think about offline RL from the above two aspects . In particular , we consider the fundamental linear model representations and ask the following question of interest : Can we achieve the statistical limits for offline RL when models have linear representations ? 1.1 RELATED THEORETICAL WORK . Offline RL with general function representations . The finite sample analysis of offline RL with function approximation is initially conducted by Fitted Q-Iteration ( FQI ) type algorithms and can be dated back to ( Munos , 2003 ; Szepesvári and Munos , 2005 ; Antos et al. , 2008a ; b ) . Later , Chen and Jiang ( 2019 ) ; Le et al . ( 2019 ) ; Xie and Jiang ( 2020 ) follow this line of research and derive the improved learning results . However , owing to the aim for tackling general function approximation , those learning bounds are expressed in terms of the stringent concentrability coefficients ( therefore , are less adaptive to individual instances ) and are usually only information-theoretical , due to the computational intractability of the optimization procedure over the general function classes . Other works impose weaker assumptions ( e.g . partial coverage ( Liu et al. , 2020 ; Kidambi et al. , 2020 ; Uehara and Sun , 2021 ) ) , and their finite sample analysis are generally suboptimal in terms of H or the effective horizon ( 1− γ ) −1 . Offline RL with tabular models . For tabular MDPs , tight learning bounds can be achieved under several data-coverage assumptions . For the class of problems with uniform data-visitation measure dm , the near-optimal sample complexity bound has the rate O ( H3/dm 2 ) for time-inhomogeneous MDPs ( Yin et al. , 2021 ) and O ( H2/dm ) for time-homogeneous MDPs ( Yin and Wang , 2021a ; Ren et al. , 2021 ) . Under the single concentrability assumption , the tight rate O ( H3SC ? / 2 ) is obtained by Xie et al . ( 2021b ) . In particular , the recent study Yin and Wang ( 2021b ) introduces the intrinsic offline learning bound that is not only instance-dependent but also subsumes previous optimal results . Offline RL with linear model representations . Recently , there is more focus on studying the provable efficient offline RL under the linear model representations . Jin et al . ( 2021b ) first shows offline RL with linear MDP is provably efficient by the pessimistic value iteration . Their analysis deviates from their lower bound by a factor of d ·H ( check their Theorem 4.4 and 4.6 ) . Later , Xie et al . ( 2021a ) considers function approximation under the Bellman-consistent assumptions , and , when realized to linear MDP setting , improve the sample complexity guarantee of Jin et al . ( 2021b ) by an order O ( d ) ( Theorem 3.2 ) .2 However , their improvement only holds for finite action space ( due to the dependence log |A| ) and by the direct reduction ( from Theorem 3.1 ) their result does not imply a computationally tractable algorithm . Concurrently , Zanette et al . ( 2021 ) considers the Linear Bellman Complete model and designs the actor-critic style algorithm that achieves tight result under the assumption that the value function is bounded by 1 . While their algorithm is efficient ( which is based on solving a sequence of second-order cone programs ) , the resulting learning bound requires the action space to be finite due to the mirror descent updates in the Actor procedure ( Agarwal et al. , 2021 ) . Besides , assuming the value function to be less than 1 simplifies the challenges in dealing with horizon H since when rescaling their result to [ 0 , H ] , there is a H factor blow-up , which makes no horizon improvement comparing to Jin et al . ( 2021b ) . On the other hand , Wang et al . ( 2021a ) ; Zanette ( 2021 ) study the statistical hardness of offline RL with linear representations by proofing the exponential lower bounds . As a result , none of the existing algorithms can achieve the statistical limit for the well-structured linear MDP model with the general ( infinite or continuous ) state-action spaces . Variance-aware studies . Talebi and Maillard ( 2018 ) first incorporate the variance structure in online tabular MDPs and Zanette and Brunskill ( 2019 ) tighten the result . For linear MDPs , Zhou et al . ( 2021 ) first uses variance structure to achieve the near-optimal result and the Weighted OFUL incorporates the variance structure explicitly in the regret bound . Recently , Variance-awareness is also considered in Zhang et al . ( 2021 ) for horizon-free setting and for OPE problem ( Min et al. , 2021 ) .

Summary: This paper proposes a re-weighting of the Bellman update for offline RL, using the variance term, that leads to an improved bound for offline RL. The paper is well written and easy to follow, with the key improvement (theoretically) easy to see. The core idea of the algorithm is a simple trick to re-weight the update and the analysis follows as in other offline RL works examining the statistical significance for offline value estimation.