Provably Efficient Reinforcement Learning with Linear Function Approximation under Adaptivity Constraints

√ d3H3T +dHT/B ) regret , where d is the dimension of the feature mapping , H is the episode length , T is the number of interactions and B is the number of batches . Our result suggests that it suffices to use only √ T/dH batches to obtain Õ ( √ d3H3T ) regret . For the rare policy switch model , our proposed LSVI-UCB-RareSwitch algorithm enjoys an Õ ( √ d3H3T [ 1 + T/ ( dH ) ] dH/B ) regret , which implies that dH log T policy switches suffice to obtain the Õ ( √ d3H3T ) regret . Our algorithms achieve the same regret as the LSVI-UCB algorithm ( Jin et al. , 2020 ) , yet with a substantially smaller amount of adaptivity . We also establish a lower bound for the batch learning model , which suggests that the dependency on B in our regret bound is tight . 1 Introduction . Real-world reinforcement learning ( RL ) applications often come with possibly infinite state and action space , and in such a situation classical RL algorithms developed in the tabular setting are not applicable anymore . A popular approach to overcoming this issue is by applying function approximation techniques to the underlying structures of the Markov decision processes ( MDPs ) . For example , one can assume that the transition probability and the reward are linear functions of a known feature mapping φ : S ×A → Rd , where S and A are the state space and action space , and d is the dimension of the embedding . This gives rise to the so-called linear MDP model ( Yang and Wang , 2019 ; Jin et al. , 2020 ) . Assuming access to a generative model , efficient algorithms under ∗Equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . this setting have been proposed by Yang and Wang ( 2019 ) and Lattimore et al . ( 2020 ) . For online finite-horizon episodic linear MDPs , Jin et al . ( 2020 ) proposed an LSVI-UCB algorithm that achieves Õ ( √ d3H3T ) regret , where H is the planning horizon ( i.e. , length of each episode ) and T is the number of interactions . However , all the aforementioned algorithms require the agent to update the policy in every episode . In practice , it is often unrealistic to frequently switch the policy in the face of big data , limited computing resources as well as inevitable switching costs . Thus one may want to batch the data stream and update the policy at the end of each period . For example , in clinical trials , each phase ( batch ) of the trial amounts to applying a medical treatment to a batch of patients in parallel . The outcomes of the treatment are not observed until the end of the phase and will be subsequently used to design experiments for the next phase . Choosing the appropriate number and sizes of the batches is crucial to achieving nearly optimal efficiency for the clinical trial . This gives rise to the limited adaptivity setting , which has been extensively studied in many online learning problems including predictionfrom-experts ( PFE ) ( Kalai and Vempala , 2005 ; Cesa-Bianchi et al. , 2013 ) , multi-armed bandits ( MAB ) ( Arora et al. , 2012 ; Cesa-Bianchi et al. , 2013 ) and online convex optimization ( Jaghargh et al. , 2019 ; Chen et al. , 2020 ) , to mention a few . Nevertheless , in the RL setting , learning with limited adaptivity is relatively less studied . Bai et al . ( 2019 ) introduced two notions of adaptivity in RL , local switching cost and global switching cost , that are defined as follows Nlocal = K−1∑ k=1 H∑ h=1 ∑ s∈S 1 { πkh ( s ) 6= πk+1h ( s ) } and Nglobal = K−1∑ k=1 1 { πk 6= πk+1 } , ( 1.1 ) where πk = { πkh : S → A } h∈ [ H ] is the policy for the k-th episode of the MDP , πk 6= πk+1 means that there exists some ( h , s ) ∈ [ H ] ×S such that πkh ( s ) 6= π k+1 h ( s ) , and K is the number of episodes . Then they proposed a Q-learning method with UCB2H exploration that achieves Õ ( √ H3SAT ) regret with O ( H3SA log ( T/ ( AH ) ) local switching cost for tabular MDPs , but they did not provide tight bounds on the global switching cost . In this paper , based on the above motivation , we aim to develop online RL algorithms with linear function approximation under adaptivity constraints . In detail , we consider time-inhomogeneous2 episodic linear MDPs ( Jin et al. , 2020 ) where both the transition probability and the reward function are unknown to the agent . In terms of the limited adaptivity imposed on the agent , we consider two scenarios that have been previously studied in the online learning literature ( Perchet et al. , 2016 ; Abbasi-Yadkori et al. , 2011 ) : the batch learning model and the rare policy switch model . More specifically , in the batch learning model ( Perchet et al. , 2016 ) , the agent is forced to pre-determine the number of batches ( or equivalently batch size ) . Within each batch , the same policy is used to select actions , and the policy is updated only at the end of this batch . The amount of adaptivity in the batch learning model is measured by the number of batches , which is expected to be as small as possible . In contrast , in the rare policy switch model ( Abbasi-Yadkori et al. , 2011 ) , the agent can adaptively choose when to switch the policy and therefore start a new batch in the learning process as long as the total number of policy updates does not exceed the given budget on the number of policy switches . The amount of adaptivity in the rare policy switch model can be measured by the number of policy switches , which turns out to be the same as the global switching cost introduced in Bai et al . ( 2019 ) . It is worth noting that for the same amount of adaptivity3 , the rare policy switch model can be seen as a relaxation of the batch learning model since the agent in the batch learning model can only change the policy at pre-defined time steps . In our work , for each of these limited adaptivity models , we propose a variant of the LSVI-UCB algorithm ( Jin et al. , 2020 ) , which can be viewed as an RL algorithm with full adaptivity in the sense that it switches the policy at a per-episode scale . Our algorithms can attain the same regret as LSVI-UCB , yet with a substantially smaller number of batches/policy switches . This enables parallel learning and improves the large-scale deployment of RL algorithms with linear function approximation . The main contributions of this paper are summarized as follows : 2We say an episodic MDP is time-inhomogeneous if its reward and transition probability are different at different stages within each episode . See Definition 3.2 for details . 3The number of batches in the batch learning model is comparable to the number of policy switches in the rare policy switch model . • For the batch learning model , we propose an LSVI-UCB-Batch algorithm for linear MDPs and show that it enjoys an Õ ( √ d3H3T + dHT/B ) regret , where d is the dimension of the feature mapping , H is the episode length , T is the number of interactions and B is the number of batches . Our result suggests that it suffices to use only √ T/dH batches , rather than T batches , to obtain the same regret Õ ( √ d3H3T ) achieved by LSVI-UCB ( Jin et al. , 2020 ) in the fully sequential decision model . We also prove a lower bound of the regret for this model , which suggests that the required number of batches Õ ( √ T ) is sharp . • For the rare policy switch model , we propose an LSVI-UCB-RareSwitch algorithm for linear MDPs and show that it enjoys an Õ ( √ d3H3T [ 1 + T/ ( dH ) ] dH/B ) regret , where B is the number of policy switches . Our result implies that dH log T policy switches are sufficient to obtain the same regret Õ ( √ d3H3T ) achieved by LSVI-UCB . The number of policy switches is much smaller than that4 of the batch learning model when T is large . Concurrent to our work , Gao et al . ( 2021 ) proposed an algorithm achieving Õ ( √ d3H3T ) regret with a O ( dH logK ) global switching cost in the rare policy switch model . They also proved a Ω ( dH/ log d ) lower bound on the global switching cost . The focus of our paper is different from theirs : our goal is to design efficient RL algorithms under a switching cost budget B , while their goal is to achieve the optimal rate in terms of T with as little switching cost as possible . On the other hand , for the rare policy switch model , our proposed algorithm ( LSVI-UCB-RareSwitch ) along its regret bound can imply their results by optimizing our regret bound concerning the switching cost budget B . The rest of the paper is organized as follows . In Section 2 we discuss previous works related to this paper , with a focus on RL with linear function approximation and online learning with limited adaptivity . In Section 3 we introduce necessary preliminaries for MDPs and adaptivity constraints . Sections 4 and 5 present our proposed algorithms and the corresponding theoretical results for the batch learning model and the rare policy switch model respectively . In Section 6 we present the numerical experiment which supports our theory . Finally , we conclude our paper and point out a future direction in Section 7 . Notation We use lower case letters to denote scalars and use lower and upper case boldface letters to denote vectors and matrices respectively . For any real number a , we write [ a ] + = max ( a , 0 ) . For a vector x ∈ Rd and matrix Σ ∈ Rd×d , we denote by ‖x‖2 the Euclidean norm and define ‖x‖Σ = √ x > Σx . For any positive integer n , we denote by [ n ] the set { 1 , . . . , n } . For any finite set A , we denote by |A| the cardinality of A . For two sequences { an } and { bn } , we write an = O ( bn ) if there exists an absolute constant C such that an ≤ Cbn , and we write an = Ω ( bn ) if there exists an absolute constant C such that an ≥ Cbn . We use Õ ( · ) to further hide the logarithmic factors . 2 Related Works . Reinforcement Learning with Linear Function Approximation Recently , there have been many advances in RL with function approximation , especially the linear case . Jin et al . ( 2020 ) proposed an efficient algorithm for the first time for linear MDPs of which the transition probability and the rewards are both linear functions with respect to a feature mapping φ : S ×A → Rd . Under similar assumptions , different settings ( e.g. , discounted MDPs ) have also been studied in Yang and Wang ( 2019 ) ; Du et al . ( 2020 ) ; Zanette et al . ( 2020 ) ; Neu and Pike-Burke ( 2020 ) and He et al . ( 2021 ) . A parallel line of work studies linear mixture MDPs ( a.k.a . linear kernel MDPs ) based on a ternary feature mapping ψ : S ×A× S → Rd ( see Jia et al . ( 2020 ) ; Zhou et al . ( 2021b ) ; Cai et al . ( 2020 ) ; Zhou et al . ( 2021a ) ) . For other function approximation settings , we refer readers to generalized linear model ( Wang et al. , 2021 ) , general function approximation with Eluder dimension ( Wang et al. , 2020 ; Ayoub et al. , 2020 ) , kernel approximation ( Yang et al. , 2020 ) , function approximation with disagreement coefficients ( Foster et al. , 2021 ) and bilinear classes ( Du et al. , 2021 ) . Online Learning with Limited Adaptivity As we mentioned before , online learning with limited adaptivity has been studied in two popular models of adaptivity constraints : the batch learning model and the rare policy switch model . 4The number of policy switches is identical to the number of batches in the batch learning model . For the batch learning model , Altschuler and Talwar ( 2018 ) proved that the optimal regret bound for prediction-from-experts ( PFE ) is Õ ( √ T log n ) when the number of batches B = Ω ( √ T log n ) , and min ( Õ ( T log n/B ) , T ) when B = O ( √ T log n ) , exhibiting a phase-transition phenomenon5 . Here T is the number of rounds and n is the number of actions . For general online convex optimization , Chen et al . ( 2020 ) showed that the minimax regret bound is Õ ( T/ √ B ) . Perchet et al . ( 2016 ) studied batched 2-arm bandits , and Gao et al . ( 2019 ) studied the batched multi-armed bandits ( MAB ) . Dekel et al . ( 2014 ) proved a Ω ( T/ √ B ) lower bound for batched MAB , and Altschuler and Talwar ( 2018 ) further characterized the dependence on the number of actions n and showed that the corresponding minimax regret bound is min ( Õ ( T √ n/ √ B ) , T ) . For batched linear bandits with adversarial contexts , Han et al . ( 2020 ) showed that the minimax regret bound is Õ ( √ dT+dT/B ) where d is the dimension of the context vectors . Better rates can be achieved for batched linear bandits with stochastic contexts as shown in Esfandiari et al . ( 2021 ) ; Han et al . ( 2020 ) ; Ruan et al . ( 2020 ) . For the rare policy switch model , the minimax optimal regret bound for PFE is O ( √ T log n ) in terms of both the expected regret ( Kalai and Vempala , 2005 ; Geulen et al. , 2010 ; Cesa-Bianchi et al. , 2013 ; Devroye et al. , 2015 ) and high-probability guarantees ( Altschuler and Talwar , 2018 ) , where T is the number of rounds , and n is the number of possible actions . For MAB , the minimax regret bound has been shown to be Õ ( T 2/3n1/3 ) by Arora et al . ( 2012 ) ; Dekel et al . ( 2014 ) . For stochastic linear bandits , Abbasi-Yadkori et al . ( 2011 ) proposed a rarely switching OFUL algorithm achieving Õ ( d √ T ) regret with log ( T ) batches . Ruan et al . ( 2020 ) proposed an algorithm achieving Õ ( √ dT ) regret with less than O ( d log d log T ) batches for stochastic linear bandits with adversarial contexts . For episodic RL with finite state and action space , Bai et al . ( 2019 ) proposed an algorithm achieving Õ ( √ H3SAT ) regret with O ( H3SA log ( T/ ( AH ) ) ) local switching cost where S and A are the number of states and actions respectively . They also provided a Ω ( HSA ) lower bound on the local switching cost that is necessary for sublinear regret . For the global switching cost , Zhang et al . ( 2021 ) proposed an MVP algorithm with at most O ( SA log ( KH ) ) global switching cost for time-homogeneous tabular MDPs .

Summary: This paper examines two variants of the LSVI-UCB algorithm in limited adaptiveness, which is the case where policies cannot be updated continuously. To deal with the limited adaptivity, the authors consider two models and proposes two algorithms, respectively. The LSVI-UCB-Batch algorithm corresponds to the batch learning model where we update the policy every $K/B$ epochs. The authors prove a $\tilde{O}(\sqrt{d^3 H^3 T} + d H T/B)$ regret and a $\Omega(dH \sqrt{T} + d H T/B)$ regret lower bound under this scenario. The LSVI-UCB-RareSwitch algorithm corresponds to the rare policy switch model where we update the policy when $det(\Lambda_h^k) > \eta det (\Lambda_h)$. The authors prove a $\tilde{O}(\sqrt{d^3 H^3 T[1 + T/(dH)]^{dH/B}})$ regret for this case. Experimental results in this paper show that the performance of both algorithms is comparable to that of the LSVI-UCB algorithm, with a substantially reduced number of policy switches.