Uniform-PAC Bounds for Reinforcement Learning with Linear Function Approximation

1 Introduction . Designing efficient reinforcement learning ( RL ) algorithms for environments with large state and action spaces is one of the main tasks in the RL community . To achieve this goal , function approximation , which uses a class of predefined functions to approximate either the value function or transition dynamic , has been widely studied in recent years . Specifically , a series of recent works [ 11 , 13 , 18 , 24 , 3 , 27 ] have studied RL with linear function approximation with provable guarantees . They show that with linear function approximation , one can either obtain a sublinear regret bound against the optimal value function [ 13 , 24 , 3 , 27 ] or a polynomial sample complexity bound [ 14 ] ( Probably Approximately Correct ( PAC ) bound for short ) in finding a near-optimal policy [ 11 , 18 ] . However , neither the regret bound or PAC bound is a perfect performance measure . As discussed in detail by [ 7 ] , these two measures fail to guarantee the convergence to the optimal policy . Therefore , an algorithm with high probability regret and/or PAC bound guarantees do not necessarily learn the optimal policy , and can perform badly in practice . In detail , one can face the following two situations : • An algorithm with a sublinear regret suggests that the summation of the suboptimality gaps ∆t ( the gap between the values of the current adapted policy and optimal policy , see Definition 3.3 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . for details . ) in the first T rounds is bounded by o ( T ) . However , this algorithm may be arbitrarily suboptimal infinitely times 1 , thus it fails to converge to the optimal policy . • An algorithm is ( , δ ) -PAC suggests that with probability at least 1−δ , the number of suboptimality gaps ∆t that are greater than will be at most polynomial in and log ( 1/δ ) . The formal definition of ( , δ ) -PAC can be found in Definition 3.4 . However , this algorithm may still have gaps satisfying /2 < ∆t < infinitely often , thus fails to converge to the optimal policy . To overcome the limitations of regret and PAC guarantees , Dann et al . [ 7 ] proposed a new performance measure called uniform-PAC , which is a strengthened notion of the PAC framework . Specifically , an algorithm is uniform-PAC if there exists a function of the target accuracy and the confidence parameter δ that upper bounds the number of suboptimality gaps satisfying ∆t > simultaneously for all > 0 with probability at least 1− δ . The formal definition of uniform-PAC can be found in Definition 3.6 . Algorithms that are uniform-PAC converge to an optimal policy with high probability , and yield both PAC and high probability regret bounds . In addition , they proposed a UBEV algorithm for learning tabular MDPs , which is uniform-PAC . Nevertheless , UBEV is designed for tabular MDPs , and it is not clear how to incorporate function approximation into UBEV to scale it up for large ( or even infinite ) state and action space . Therefore , a natural question arises : Can we design a provable efficient uniform-PAC RL algorithm with linear function approximation ? In this work , we answer this question affirmatively . In detail , we propose new algorithms for both contextual linear bandits and linear Markov decision processes ( MDPs ) [ 22 , 13 ] . Both of them are uniform-PAC , and their sample complexity is comparable to that of the state-of-the-art algorithms which are not uniform-PAC . Our key contributions are highlighted as follows . • We begin with contextual linear bandits problem as a “ warm-up ” example of the RL with linear function approximation ( with horizon length equals 1 ) . We propose a new algorithm called uniformPAC OFUL ( UPAC-OFUL ) , and show that our algorithm is uniform-PAC with Õ ( d2/ 2 ) sample complexity , where d is the dimension of contexts and is the accuracy parameter . In addition , this result also implies an Õ ( d √ T ) regret in the first T round and matches the result of OFUL algorithm [ 1 ] up to a logarithmic factor . The key idea of our algorithm is a novel minimax linear predictor and a multi-level partition scheme to select the training samples from past observations . To the best of our knowledge , this is the first algorithm with a uniform PAC-bound for contextual bandits problems . • We also consider RL with linear function approximation in episodic linear MDPs , where the transition kernel admits a low-rank factorization . We propose an algorithm dubbed uniForm-PAC Least-sqUare value iTEration ( FLUTE ) , which adapts the novel techniques we developed in the contextual linear bandits setting , and show that our algorithm is uniform-PAC with Õ ( d3H5/ 2 ) sample complexity , where d is the dimension of the feature mapping , H is the length of episodes and is the accuracy parameter . This result further implies an Õ ( √ d3H4T ) regret in the first T steps and matches the result of LSVI-UCB algorithm [ 13 ] up to a √ H-factor , while LSVI-UCB is not uniform-PAC . Again , FLUTE is the first uniform-PAC RL algorithm with linear function approximation . Notation We use lower case letters to denote scalars , and use lower and upper case bold face letters to denote vectors and matrices respectively . For any positive integer n , we denote by [ n ] the set { 1 , . . . , n } . For a vector x ∈ Rd , we denote by ‖x‖1 the Manhattan norm and denote by ‖x‖2 the Euclidean norm . For a vector x ∈ Rd and matrix Σ ∈ Rd×d , we define ‖x‖Σ = √ x > Σx . For two sequences { an } and { bn } , we write an = O ( bn ) if there exists an absolute constant C such that an ≤ Cbn . We use Õ ( · ) to further hide the logarithmic factors . For logarithmic regret , we use Õ ( · ) to hide all logarithmic terms except log T . 1Suppose the suboptimality gaps satisfy ∆t = 1 { t = i2 , i = 1 , . . . } , then the regret in the first T rounds is upper bounded by O ( √ T ) , and the constant 1-gap will appear infinitely often 2 Related Work . 2.1 Linear bandits . There is a series of works focusing on the stochastic linear bandits problem . These works can be categorized into two groups : the works aim at providing sublinear regret guarantee and the works providing PAC bound for linear best-arm identification problem . In detail , for finite action set with K arms , Auer [ 2 ] proposed a SupLinRel algorithm which achieves anO ( √ dT log3 ( TK ) ) regret , where K is the number of arms . Chu et al . [ 5 ] proposed a SupLinUCB algorithm which has the same regret bound as SupLinRel , but is easier to implement . Li et al . [ 17 ] proposed a Variable-Confidence-Level ( VCL ) SupLinUCB algorithm and improved the regret bound to O ( √ dT log T logK ) . For infinite action set , Dani et al . [ 6 ] proposed a Confidence Ball algorithm with an O ( d √ T log3 T ) regret and proved an Ω ( d √ T ) lower bound . Abbasi-Yadkori et al . [ 1 ] proposed OFUL algorithm and improved the regret bound to O ( d √ T log2 T ) . When the reward has a bounded variance , Zhou et al . [ 27 ] , Zhang et al . [ 25 ] proposed algorithms with variance-aware confident sets , and obtained tight variance-dependent regret bounds . For the best-arm identification problem , to find an arm which is -suboptimal , Soare et al . [ 19 ] proposed a G-allocation strategy with an Õ ( d/ 2 ) sample complexity . Karnin [ 15 ] proposed an Explore-Verify framework which improves the sample complexity by some logarithmic factors . Xu et al . [ 21 ] proposed a LinGapE algorithm whose sample complexity matches the lower bound up to some K factors . Tao et al . [ 20 ] proposed an ALBA algorithm which improves the sample complexity to have a linear dimension dependence . Fiez et al . [ 8 ] studied transductive linear bandits and proposed an algorithm with sample complexity similar to linear bandits . Compared with best-arm identification , the contextual bandits setting we focus on is more challenging since the action set will change at each round . 2.2 RL with linear function approximation . Recently , a line of work focuses on analyzing RL with linear function approximation . To mention a few , Jiang et al . [ 11 ] studied MDPs with low Bellman rank and proposed an OLIVE algorithm , which has the PAC guarantee . Yang and Wang [ 22 ] studied the linear transition model and proposed a sample-optimal Q-learning method with a generative model . Jin et al . [ 13 ] studied the linear MDP model and proposed an LSVI-UCB algorithm under the online RL setting ( without a generative model ) with Õ ( √ d3H3T ) regret . Later , Zanette and Brunskill [ 23 ] studied the low inherent Bellman error model and proposed an ELEANOR algorithm with a better regret Õ ( dH √ T ) using a global planning oracle . Modi et al . [ 18 ] studied the linearly combined model ensemble and proposed a provable sample-efficient algorithm . Jia et al . [ 10 ] , Ayoub et al . [ 3 ] studied the linear mixture MDPs and proposed a UCRL-VTR algorithm with an Õ ( d √ H3T ) regret . Recently Zhou et al . [ 27 ] improved the regret bound to Õ ( dH √ T ) with a new algorithm design and a new Bernstein inequality . However , all of these works aim at deriving PAC sample complexity guarantee or regret bound , and none of them has the uniform PAC guarantee for learning MDPs with linear function approximation . Our work will fill this gap in the linear MDP setting [ 22 , 13 ] . 3 Preliminaries . We consider episodic Markov Decision Processes ( MDPs ) in this work . Each episodic MDP is denoted by a tuple M ( S , A , H , { rh } Hh=1 , { Ph } Hh=1 ) . Here , S is the state space , A is the finite action space , H is the length of each episode , rh : S × A → [ 0 , 1 ] is the reward function at stage h and Ph ( s′|s , a ) is the transition probability function at stage h which denotes the probability for state s to transfer to state s′ with action a at stage h. A policy π : S × [ H ] → A is a function which maps a state s and the stage number h to an action a . For any policy π and stage h ∈ [ H ] , we define the action-value function Qπh ( s , a ) and value function V π h ( s ) as follows Qπh ( s , a ) = rh ( s , a ) + E [ H∑ h′=h+1 rh′ ( sh′ , π ( sh′ , h ′ ) ) ∣∣sh = s , ah = a ] , V πh ( s ) = Qπh ( s , π ( s , h ) ) , where sh′+1 ∼ Ph ( ·|sh′ , ah′ ) . We define the optimal value function V ∗h and the optimal actionvalue function Q∗h as V ∗ h ( s ) = maxπ V π h ( s ) and Q ∗ h ( s , a ) = maxπ Q π h ( s , a ) . By definition , the value function V πh ( s ) and action-value function Q π h ( s , a ) are bounded in [ 0 , H ] . For any function V : S → R , we denote [ PhV ] ( s , a ) = Es′∼Ph ( ·|s , a ) V ( s′ ) . Therefore , for each stage h ∈ [ H ] and policy π , we have the following Bellman equation , as well as the Bellman optimality equation : Qπh ( s , a ) = rh ( s , a ) + [ PhV πh+1 ] ( s , a ) , Q∗h ( s , a ) = rh ( s , a ) + [ PhV ∗h+1 ] ( s , a ) , ( 3.1 ) where V πH+1 = V ∗ H+1 = 0 . At the beginning of the episode k , the agent determines a policy πk to be followed in this episode . At each stage h ∈ [ H ] , the agent observes the state skh , chooses an action following the policy πk and observes the next state with skh+1 ∼ Ph ( ·|skh , akh ) . We consider linear function approximation in this work . Therefore , we make the following linear MDP assumption , which is firstly proposed in [ 22 , 13 ] . Assumption 3.1 . MDPM ( S , A , H , { rh } Hh=1 , { Ph } Hh=1 ) is a linear MDP such that for any stage h ∈ [ H ] , there exists an unknown vector µh , an unknown measure θh ( · ) : S → Rd and a known feature mapping φ : S ×A → Rd , such that for each ( s , a ) ∈ S ×A and s′ ∈ S , Ph ( s′|s , a ) = 〈 φ ( s , a ) , θh ( s ′ ) 〉 , rh ( s , a ) = 〈 φ ( s , a ) , µh 〉 . For simplicity , we assume that µh , θn ( · ) and φ ( · , · ) satisfy ‖φ ( s , a ) ‖2 ≤ 1 for all s , a , ‖µh‖2 ≤ √ d and ∥∥θh ( S ) ∥∥2 ≤ √d . The linear MDP assumption automatically suggests that for any policy π , the action-value function Qπh is always a linear function of the given feature mapping φ , which is summarized in the following proposition . Proposition 3.2 ( Proposition 2.3 , [ 13 ] ) . For any policy π , there exist weights { wπh } Hh=1 such that for any s , a , h ∈ S ×A× [ H ] , Qπh ( s , a ) = 〈φ ( s , a ) , wπh〉 . Next we define the regret and ( , δ ) -PAC formally . Definition 3.3 . For an RL algorithm Alg , we define its regret on learning an MDP M ( S , A , H , r , P ) in the first K episodes as the sum of the suboptimality for episode k = 1 , . . . , K , Regret ( K ) = K∑ k=1 V ∗1 ( s k 1 ) − V πk 1 ( s k 1 ) , where πk is the policy in the k-th episode . Definition 3.4 . For an RL algorithm Alg and a fixed , let π1 , π2 , . . . be the policies generated by Alg . Let N = ∑∞ k=1 1 { V ∗1 ( sk1 ) − V πk 1 ( s k 1 ) > } be the number of episodes whose suboptimality gap is greater than . Then we say Alg is ( , δ ) -PAC with sample complexity f ( , δ ) if P ( N > f ( , δ ) ) ≤ δ . Remark 3.5 . Dann et al . [ 7 ] suggested that an algorithm with a sublinear regret is not necessarily to be an ( , δ ) -PAC algorithm . However , with some modification , Jin et al . [ 12 ] showed that any algorithm with a sublinear regret can be converted to a new algorithm which is ( , δ ) -PAC , which does not contradict with the claim by [ 7 ] . For example , Ghavamzadeh et al . [ 9 ] and Zhang et al . [ 26 ] proposed algorithms with Õ ( √ SAHT ) regret , and both algorithms can be converted into new algorithms which are ( , δ ) -PAC with sample complexity Õ ( SAH2/ 2 ) . Both regret and PAC guarantees are not perfect . As Dann et al . [ 7 ] showed , an algorithm with sub-linear regret or ( δ , ) -PAC bound may fail to converge to the optimal policy . For an ( δ , ) -PAC algorithm with ∆t = /2 ( t ∈ N ) , it still has linear regret O ( T ) and will never converge to the optimal policy . For an algorithm with a sub-linear regret bound , a constant sub-optimality gap may still occur infinite times . Therefore , Dann et al . [ 7 ] proposed uniform-PAC algorithms , which are defined formally as follows . Definition 3.6 . For an RL algorithm Alg , let π1 , π2 , . . . be the policies generated by Alg . Let N = ∑∞ k=1 1 { V ∗1 ( sk1 ) − V πk 1 ( s k 1 ) > } be the number of episodes whose suboptimality gap is greater than . We say Alg is uniform-PAC for some δ ∈ ( 0 , 1 ) with sample complexity f ( , δ ) if P ( ∃ > 0 , N > f ( , δ ) ) ≤ δ . The following theorem suggests that a uniform-PAC algorithm is automatically a PAC algorithm and an algorithm with sublinear regret . Algorithm 1 Uniform-PAC OFUL ( UPAC-OFUL ) Require : Regularization parameter λ , confidence radius βl ( l ∈ N ) 1 : Set Cl ← ∅ , l ∈ N and the total level S1 = 1 2 : for round k = 1 , 2 , .. do 3 : for all level l ∈ [ Sk ] do 4 : Set Σlk = λI + ∑ i∈Cl xix > i 5 : Set blk = ∑ i∈Cl xiri and w l k = ( Σ l k ) −1blk 6 : end for 7 : Receive the action set Dk 8 : Choose action xk ← argmaxx∈Dk min1≤l≤Sk ( w l k ) > x + βl √ x > ( Σlk ) −1x 9 : Set level lk = 1 10 : while √ x > k ( Σ lk k ) −1xk ≤ 2−lk and lk ≤ Sk do 11 : lk ← lk + 1 12 : end while 13 : Add the new element k to the set Clk and receive the reward rk 14 : Set the total level Sk+1 as Sk+1 = maxl : |Cl| > 0 l 15 : end for Theorem 3.7 ( Theorem 3 , [ 7 ] ) . If an algorithm Alg is uniform-PAC for some δ ≥ 0 , with sample complexity Õ ( C1/ + C2/ 2 ) , where C1 , C2 are constant and depend only on S , A , H , log ( 1/δ ) . Then , we have the following results : • 1 : Alg will converge to optimal policies with high probability at least 1−δ : P ( limk→+∞ V ∗ 1 ( s k 1 ) − V πk1 ( s k 1 ) = 0 ) ≥ 1− δ • 2 : With probability at least 1 − δ , for each K ∈ N , the regret for Alg in the first K episodes is upper bounded by Õ ( √ C2K + C1 + C2 ) . • 3 : For each ≥ 0 , Alg is also ( , δ ) -PAC with the same sample complexity Õ ( C1/ + C2/ 2 ) . Theorem 3.7 suggests that uniform-PAC is stronger than both the PAC and regret guarantees . In the remainder of this paper , we aim at developing uniform-PAC RL algorithms with linear function approximation .

Summary: The paper a proposes optimistic algorithm for reinforcement learning in linear MDPs and proves the first uniform-PAC bound for this setting. Unlike existing result, this stronger notion of a PAC bound ensures high-probability regret as well as convergence to the optimal policy. Along the way, the paper also provides an algorithm with uniform-PAC guarantees for the contextual linear bandit setting. The algorithms are similar to existing UCB algorithms but employ a sample-splitting technique. This technique essentially ensures the following key property for uniform-PAC results: the confidence bound for a certain state-action pair cannot increase arbitrarily with the number of episodes (as is the case with existing UCBs that typically have log(T) dependence in this setting and therefore increase with T unless an informative observation was gathered).