Breaking the Sample Complexity Barrier to Regret-Optimal Model-Free Reinforcement Learning

√ H2SAT ( modulo log factors ) with T the total number of samples . While several competing solution paradigms have been proposed to minimize regret , they are either memoryinefficient , or fall short of optimality unless the sample size exceeds an enormous threshold ( e.g. , S6A4 poly ( H ) for existing model-free methods ) . To overcome such a large sample size barrier to efficient RL , we design a novel model-free algorithm , with space complexity O ( SAH ) , that achieves near-optimal regret as soon as the sample size exceeds the order of SApoly ( H ) . In terms of this sample size requirement ( also referred to the initial burn-in cost ) , our method improves — by at least a factor of S5A3 — upon any prior memory-efficient algorithm that is asymptotically regret-optimal . Leveraging the recently introduced variance reduction strategy ( also called reference-advantage decomposition ) , the proposed algorithm employs an early-settled reference update rule , with the aid of two Q-learning sequences with upper and lower confidence bounds . The design principle of our early-settled variance reduction method might be of independent interest to other RL settings that involve intricate exploration-exploitation tradeoffs . 1 Introduction . Contemporary reinforcement learning ( RL ) has to deal with unknown environments with unprecedentedly large dimensionality . How to make the best use of samples in the face of high-dimensional state/action space lies at the core of modern RL practice . An ideal RL algorithm would learn to act favorably even when the number of available data samples scales sub-linearly in the ambient dimension of the model , i.e. , the number of parameters needed to describe the transition dynamics of the environment . The challenge is further compounded when this task needs to be accomplished with limited memory . Simultaneously achieving the desired sample and memory efficiency is particularly challenging when it comes to online episodic RL scenarios . In contrast to the simulator setting that permits sampling of any state-action pair , an agent in online episodic RL is only allowed to draw sample trajectories by executing a policy in the unknown Markov decision process ( MDP ) , where the initial states are pre-assigned and might even be chosen by an adversary . Careful deliberation needs to be undertaken when deciding what policies to use to allow for effective interaction with the unknown environment , ∗Department of Electrical and Computer Engineering , Princeton University , Princeton , NJ 08544 , USA . †Department of Electrical and Computer Engineering , Carnegie Mellon University , Pittsburgh , PA 15213 , USA . ‡Department of Electronic Engineering , Tsinghua University , Beijing 100084 , China . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . how to optimally balance exploitation and exploration , and how to process and store the collected information intelligently without causing redundancy . 1.1 Regret-optimal model-free RL ? A sample size barrier . In order to evaluate and compare the effectiveness of RL algorithms in high dimension , a recent body of works sought to develop a finite-sample theoretical framework to analyze the algorithms of interest , with the aim of delineating the dependency of algorithm performance on all salient problem parameters in a non-asymptotic fashion ( Dann et al. , 2017 ; Kakade , 2003 ) . Such finite-sample guarantees are brought to bear towards understanding and tackling the challenges in the samplestarved regime commonly encountered in practice . To facilitate discussion , let us take a moment to summarize the state-of-the-art theory for episodic finite-horizon MDPs with non-stationary transition kernels , focusing on minimizing cumulative regret — a metric that quantifies the performance difference between the learned policy and the true optimal policy — with the fewest number of samples . Here and throughout , we denote by S , A , and H the size of the state space , the size of the action space , and the horizon length of the MDP , respectively , and let T represent the sample size . In addition , the immediate reward gained at each time step is assumed to lie between 0 and 1 . Fundamental regret lower bound . Following the arguments in Jaksch et al . ( 2010 ) ; Auer et al . ( 2002 ) , the recent works Jin et al . ( 2018 ) ; Domingues et al . ( 2021 ) developed a fundamental lower bound on the expected total regret for this setting . Specifically , this lower bound claims that : no matter what algorithm to use , one can find an MDP such that the accumulated regret incurred by the algorithm necessarily exceeds the order of ( lower bound ) √ H2SAT , ( 1 ) as long as T ≥ H2SA.4 This sublinear regret lower bound in turn imposes a sampling limit if one wants to achieve ε average regret . Model-based RL . Moving beyond the lower bound , let us examine the effectiveness of model-based RL — an approach that can be decoupled into a model estimation stage ( i.e. , estimating the transition kernel using available data ) and a subsequent stage of planning using the learned model ( Jaksch et al. , 2010 ; Azar et al. , 2017 ; Efroni et al. , 2019 ; Agrawal and Jia , 2017 ; Pacchiano et al. , 2020 ) . In order to ensure a sufficient degree of exploration , Zhang et al . ( 2020b ) came up with an algorithm called MVP that blends model-based learning and the optimism principle , which achieves a regret bound5 Õ ( √ H2SAT ) that nearly attains the lower bound ( 1 ) as T tends to infinity . Caution needs to be exercised , however , that existing theory does not guarantee the near optimality of this algorithm unless the sample size T surpasses T ≥ S3AH4 , a threshold that is significantly larger than the dimension of the underlying model . This threshold can also be understood as the initial burn-in cost of the algorithm , namely , a sampling burden needed for the algorithm to exhibit the desired performance . In addition , model-based algorithms typically require storing the estimated probability transition kernel , resulting in a space complexity that could be as high as O ( S2AH ) ( Azar et al. , 2017 ; Zhang et al. , 2020b ) . Model-free RL . Another competing solution paradigm is the model-free approach , which circumvents the model estimation stage and attempts to learn the optimal values directly ( Strehl et al. , 2006 ; Jin et al. , 2018 ; Bai et al. , 2019 ; Yang et al. , 2021 ) . In comparison to the model-based counterpart , the model-free approach holds the promise of low space complexity , as it eliminates the need of storing a full description of the model . In fact , in a number of previous works ( e.g. , Strehl et al . ( 2006 ) ; Jin et al . ( 2018 ) ) , an algorithm is declared to be model-free only if its space complexity is o ( S2AH ) regardless of the sample size T . • Memory-efficient model-free methods . Jin et al . ( 2018 ) proposed the first memory-efficient modelfree algorithm — which is an optimistic variant of classical Q-learning — that achieves a regret bound proportional to √ T with a space complexity O ( SAH ) . Compared to the lower bound ( 1 ) , however , the regret bound in Jin et al . ( 2018 ) is off by a factor of √ H and hence suboptimal for 4Given that a trivial upper bound on the regret is T , one needs to impose a lower bound T ≥ H2SA in order for ( 1 ) to be meaningful . 5Here and throughout , we use the standard notation f ( n ) = O ( g ( n ) ) to indicate that f ( n ) /g ( n ) is bounded above by a constant as n grows . The notation Õ ( · ) resembles O ( · ) except that it hides any logarithmic scaling . The notation f ( n ) = o ( g ( n ) ) means that limn→∞ f ( n ) /g ( n ) = 0. the optimal regret Õ ( √ H2SAT ) , and the memory complexity , with all logarithmic factors omitted for simplicity of presentation . The red text highlights the suboptimal part of the respective algorithms . problems with long horizon . This drawback has recently been overcome in Zhang et al . ( 2020c ) by leveraging the idea of variance reduction ( or the so-called “ reference-advantage decomposition ” ) for large enough T . While the resulting regret matches the information-theoretic limit asymptotically , its optimality in the non-asymptotic regime is not guaranteed unless the sample size T exceeds ( see Zhang et al . ( 2020c , Lemma 7 ) ) T ≥ S6A4H28 , a requirement that is even far more stringent than the burn-in cost imposed by Azar et al . ( 2017 ) . • A memory-inefficient “ model-free ” variant . The recent work Menard et al . ( 2021 ) put forward a novel sample-efficient variant of Q-learning called UCB-M-Q , which relies on a carefully chosen momentum term for bias reduction . This algorithm is guaranteed to yield near-optimal regret Õ ( √ H2SAT ) as soon as the sample size exceeds T ≥ SApoly ( H ) , which is a remarkable improvement vis-à-vis previous regret-optimal methods ( Azar et al. , 2017 ; Zhang et al. , 2020c ) . Nevertheless , akin to the model-based approach , it comes at a price in terms of the space and computation complexities , as the space required to store all bias-value function is O ( S2AH ) and the computation required is O ( ST ) , both of which are larger by a factor of S than other model-free algorithms like Zhang et al . ( 2020c ) . In view of this memory inefficiency , UCB-M-Q falls short of fulfilling the definition of model-free algorithms in Strehl et al . ( 2006 ) ; Jin et al . ( 2018 ) . See Menard et al . ( 2021 , Section 3.3 ) for more detailed discussions . A more complete summary of prior results can be found in Table 1 . 1.2 A glimpse of our contributions . In brief , while it is encouraging to see that both model-based and model-free approaches allow for near-minimal regret as T tends to infinity , they are either memory-inefficient , or require the sample size to exceed a threshold substantially larger than the model dimensionality . In fact , no prior algorithms have been shown to be simultaneously regret-optimal and memory-efficient unless T ≥ S6A4 poly ( H ) , which constitutes a stringent sample size barrier constraining their utility in the sample-starved and memory-limited regime . The presence of this sample complexity barrier motivates one to pose a natural question : Is it possible to design an algorithm that accommodates a significantly broader . sample size range without compromising regret optimality and memory efficiency ? In this paper , we answer this question affirmatively , by designing a new model-free algorithm , dubbed as Q-EarlySettled-Advantage , that enjoys the following performance guarantee . Theorem 1 ( informal ) . The proposed Q-EarlySettled-Advantage algorithm , which has a space complexityO ( SAH ) , achieves near-optimal regret Õ ( √ H2SAT ) as soon as the sample size exceeds T ≥ SApoly ( H ) . The proof of this theorem can be found in the full version Li et al . ( 2021c ) . As can be seen from Table 1 , the space complexity of the proposed algorithm is O ( SAH ) , which is far more memoryefficient than both the model-based approach in Azar et al . ( 2017 ) and the UCB-M-Q algorithm in Menard et al . ( 2021 ) ( both of these prior algorithms require S2AH units of space ) . In addition , the sample size requirement T ≥ SApoly ( H ) of our algorithm improves — by a factor of at least S5A3 — upon that of any prior algorithm that is simultaneously regret-optimal and memory-efficient . In fact , this requirement is nearly sharp in terms of the dependency on both S and A , and was previously achieved only by the UCB-M-Q algorithm at a price of a much higher storage burden . Let us also briefly highlight the key ideas of our algorithm . As an optimistic variant of variancereduced Q-learning , Q-EarlySettled-Advantage leverages the recently-introduced referenceadvantage decompositions for variance reduction ( Zhang et al. , 2020c ) . As a distinguishing feature from prior algorithms , we employ an early-stopped reference update rule , with the assistance of two Q-learning sequences that incorporate upper and lower confidence bounds , respectively . The design of our early-stopped variance reduction scheme , as well as its analysis framework , might be of independent interest to other settings that involve managing intricate exploration-exploitation trade-offs .

Summary: This paper addresses the problem of designing a memory-efficient and sample-efficient algorithm for episodic MDPs with smaller initial warm-up phase. The paper proposes a new model-free algorithm "UCB-Q-Double-Advantage" that extends the optimistic Q-learning and the reference-advantage decomposition for variance reduction using two reference-advantage functions, i.e. the updated reference-advantage function and the old reference-advantage function. Use of these two functions with a well-curated update rule leads to near-optimal regret with SAH^2 warm-up actions, O(SAH) memory complexity, \tilde{O}(optimal x H) sample complexity, and O(T) time complexity. The final performance is O(H) worse than optimal in terms of sample and warm-up complexities.