Provably Efficient Reinforcement Learning with Linear Function Approximation under Adaptivity Constraints

√ d3H3T +dHT/B ) regret , where d is the dimension of the feature mapping , H is the episode length , T is the number of interactions and B is the number of batches . Our result suggests that it suffices to use only √ T/dH batches to obtain Õ ( √ d3H3T ) regret . For the rare policy switch model , our proposed LSVI-UCB-RareSwitch algorithm enjoys an Õ ( √ d3H3T [ 1 + T/ ( dH ) ] dH/B ) regret , which implies that dH log T policy switches suffice to obtain the Õ ( √ d3H3T ) regret . Our algorithms achieve the same regret as the LSVI-UCB algorithm ( Jin et al. , 2020 ) , yet with a substantially smaller amount of adaptivity . We also establish a lower bound for the batch learning model , which suggests that the dependency on B in our regret bound is tight . 1 Introduction . Real-world reinforcement learning ( RL ) applications often come with possibly infinite state and action space , and in such a situation classical RL algorithms developed in the tabular setting are not applicable anymore . A popular approach to overcoming this issue is by applying function approximation techniques to the underlying structures of the Markov decision processes ( MDPs ) . For example , one can assume that the transition probability and the reward are linear functions of a known feature mapping φ : S ×A → Rd , where S and A are the state space and action space , and d is the dimension of the embedding . This gives rise to the so-called linear MDP model ( Yang and Wang , 2019 ; Jin et al. , 2020 ) . Assuming access to a generative model , efficient algorithms under ∗Equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . this setting have been proposed by Yang and Wang ( 2019 ) and Lattimore et al . ( 2020 ) . For online finite-horizon episodic linear MDPs , Jin et al . ( 2020 ) proposed an LSVI-UCB algorithm that achieves Õ ( √ d3H3T ) regret , where H is the planning horizon ( i.e. , length of each episode ) and T is the number of interactions . However , all the aforementioned algorithms require the agent to update the policy in every episode . In practice , it is often unrealistic to frequently switch the policy in the face of big data , limited computing resources as well as inevitable switching costs . Thus one may want to batch the data stream and update the policy at the end of each period . For example , in clinical trials , each phase ( batch ) of the trial amounts to applying a medical treatment to a batch of patients in parallel . The outcomes of the treatment are not observed until the end of the phase and will be subsequently used to design experiments for the next phase . Choosing the appropriate number and sizes of the batches is crucial to achieving nearly optimal efficiency for the clinical trial . This gives rise to the limited adaptivity setting , which has been extensively studied in many online learning problems including predictionfrom-experts ( PFE ) ( Kalai and Vempala , 2005 ; Cesa-Bianchi et al. , 2013 ) , multi-armed bandits ( MAB ) ( Arora et al. , 2012 ; Cesa-Bianchi et al. , 2013 ) and online convex optimization ( Jaghargh et al. , 2019 ; Chen et al. , 2020 ) , to mention a few . Nevertheless , in the RL setting , learning with limited adaptivity is relatively less studied . Bai et al . ( 2019 ) introduced two notions of adaptivity in RL , local switching cost and global switching cost , that are defined as follows Nlocal = K−1∑ k=1 H∑ h=1 ∑ s∈S 1 { πkh ( s ) 6= πk+1h ( s ) } and Nglobal = K−1∑ k=1 1 { πk 6= πk+1 } , ( 1.1 ) where πk = { πkh : S → A } h∈ [ H ] is the policy for the k-th episode of the MDP , πk 6= πk+1 means that there exists some ( h , s ) ∈ [ H ] ×S such that πkh ( s ) 6= π k+1 h ( s ) , and K is the number of episodes . Then they proposed a Q-learning method with UCB2H exploration that achieves Õ ( √ H3SAT ) regret with O ( H3SA log ( T/ ( AH ) ) local switching cost for tabular MDPs , but they did not provide tight bounds on the global switching cost . In this paper , based on the above motivation , we aim to develop online RL algorithms with linear function approximation under adaptivity constraints . In detail , we consider time-inhomogeneous2 episodic linear MDPs ( Jin et al. , 2020 ) where both the transition probability and the reward function are unknown to the agent . In terms of the limited adaptivity imposed on the agent , we consider two scenarios that have been previously studied in the online learning literature ( Perchet et al. , 2016 ; Abbasi-Yadkori et al. , 2011 ) : the batch learning model and the rare policy switch model . More specifically , in the batch learning model ( Perchet et al. , 2016 ) , the agent is forced to pre-determine the number of batches ( or equivalently batch size ) . Within each batch , the same policy is used to select actions , and the policy is updated only at the end of this batch . The amount of adaptivity in the batch learning model is measured by the number of batches , which is expected to be as small as possible . In contrast , in the rare policy switch model ( Abbasi-Yadkori et al. , 2011 ) , the agent can adaptively choose when to switch the policy and therefore start a new batch in the learning process as long as the total number of policy updates does not exceed the given budget on the number of policy switches . The amount of adaptivity in the rare policy switch model can be measured by the number of policy switches , which turns out to be the same as the global switching cost introduced in Bai et al . ( 2019 ) . It is worth noting that for the same amount of adaptivity3 , the rare policy switch model can be seen as a relaxation of the batch learning model since the agent in the batch learning model can only change the policy at pre-defined time steps . In our work , for each of these limited adaptivity models , we propose a variant of the LSVI-UCB algorithm ( Jin et al. , 2020 ) , which can be viewed as an RL algorithm with full adaptivity in the sense that it switches the policy at a per-episode scale . Our algorithms can attain the same regret as LSVI-UCB , yet with a substantially smaller number of batches/policy switches . This enables parallel learning and improves the large-scale deployment of RL algorithms with linear function approximation . The main contributions of this paper are summarized as follows : 2We say an episodic MDP is time-inhomogeneous if its reward and transition probability are different at different stages within each episode . See Definition 3.2 for details . 3The number of batches in the batch learning model is comparable to the number of policy switches in the rare policy switch model . • For the batch learning model , we propose an LSVI-UCB-Batch algorithm for linear MDPs and show that it enjoys an Õ ( √ d3H3T + dHT/B ) regret , where d is the dimension of the feature mapping , H is the episode length , T is the number of interactions and B is the number of batches . Our result suggests that it suffices to use only √ T/dH batches , rather than T batches , to obtain the same regret Õ ( √ d3H3T ) achieved by LSVI-UCB ( Jin et al. , 2020 ) in the fully sequential decision model . We also prove a lower bound of the regret for this model , which suggests that the required number of batches Õ ( √ T ) is sharp . • For the rare policy switch model , we propose an LSVI-UCB-RareSwitch algorithm for linear MDPs and show that it enjoys an Õ ( √ d3H3T [ 1 + T/ ( dH ) ] dH/B ) regret , where B is the number of policy switches . Our result implies that dH log T policy switches are sufficient to obtain the same regret Õ ( √ d3H3T ) achieved by LSVI-UCB . The number of policy switches is much smaller than that4 of the batch learning model when T is large . Concurrent to our work , Gao et al . ( 2021 ) proposed an algorithm achieving Õ ( √ d3H3T ) regret with a O ( dH logK ) global switching cost in the rare policy switch model . They also proved a Ω ( dH/ log d ) lower bound on the global switching cost . The focus of our paper is different from theirs : our goal is to design efficient RL algorithms under a switching cost budget B , while their goal is to achieve the optimal rate in terms of T with as little switching cost as possible . On the other hand , for the rare policy switch model , our proposed algorithm ( LSVI-UCB-RareSwitch ) along its regret bound can imply their results by optimizing our regret bound concerning the switching cost budget B . The rest of the paper is organized as follows . In Section 2 we discuss previous works related to this paper , with a focus on RL with linear function approximation and online learning with limited adaptivity . In Section 3 we introduce necessary preliminaries for MDPs and adaptivity constraints . Sections 4 and 5 present our proposed algorithms and the corresponding theoretical results for the batch learning model and the rare policy switch model respectively . In Section 6 we present the numerical experiment which supports our theory . Finally , we conclude our paper and point out a future direction in Section 7 . Notation We use lower case letters to denote scalars and use lower and upper case boldface letters to denote vectors and matrices respectively . For any real number a , we write [ a ] + = max ( a , 0 ) . For a vector x ∈ Rd and matrix Σ ∈ Rd×d , we denote by ‖x‖2 the Euclidean norm and define ‖x‖Σ = √ x > Σx . For any positive integer n , we denote by [ n ] the set { 1 , . . . , n } . For any finite set A , we denote by |A| the cardinality of A . For two sequences { an } and { bn } , we write an = O ( bn ) if there exists an absolute constant C such that an ≤ Cbn , and we write an = Ω ( bn ) if there exists an absolute constant C such that an ≥ Cbn . We use Õ ( · ) to further hide the logarithmic factors . 2 Related Works . Reinforcement Learning with Linear Function Approximation Recently , there have been many advances in RL with function approximation , especially the linear case . Jin et al . ( 2020 ) proposed an efficient algorithm for the first time for linear MDPs of which the transition probability and the rewards are both linear functions with respect to a feature mapping φ : S ×A → Rd . Under similar assumptions , different settings ( e.g. , discounted MDPs ) have also been studied in Yang and Wang ( 2019 ) ; Du et al . ( 2020 ) ; Zanette et al . ( 2020 ) ; Neu and Pike-Burke ( 2020 ) and He et al . ( 2021 ) . A parallel line of work studies linear mixture MDPs ( a.k.a . linear kernel MDPs ) based on a ternary feature mapping ψ : S ×A× S → Rd ( see Jia et al . ( 2020 ) ; Zhou et al . ( 2021b ) ; Cai et al . ( 2020 ) ; Zhou et al . ( 2021a ) ) . For other function approximation settings , we refer readers to generalized linear model ( Wang et al. , 2021 ) , general function approximation with Eluder dimension ( Wang et al. , 2020 ; Ayoub et al. , 2020 ) , kernel approximation ( Yang et al. , 2020 ) , function approximation with disagreement coefficients ( Foster et al. , 2021 ) and bilinear classes ( Du et al. , 2021 ) . Online Learning with Limited Adaptivity As we mentioned before , online learning with limited adaptivity has been studied in two popular models of adaptivity constraints : the batch learning model and the rare policy switch model . 4The number of policy switches is identical to the number of batches in the batch learning model . For the batch learning model , Altschuler and Talwar ( 2018 ) proved that the optimal regret bound for prediction-from-experts ( PFE ) is Õ ( √ T log n ) when the number of batches B = Ω ( √ T log n ) , and min ( Õ ( T log n/B ) , T ) when B = O ( √ T log n ) , exhibiting a phase-transition phenomenon5 . Here T is the number of rounds and n is the number of actions . For general online convex optimization , Chen et al . ( 2020 ) showed that the minimax regret bound is Õ ( T/ √ B ) . Perchet et al . ( 2016 ) studied batched 2-arm bandits , and Gao et al . ( 2019 ) studied the batched multi-armed bandits ( MAB ) . Dekel et al . ( 2014 ) proved a Ω ( T/ √ B ) lower bound for batched MAB , and Altschuler and Talwar ( 2018 ) further characterized the dependence on the number of actions n and showed that the corresponding minimax regret bound is min ( Õ ( T √ n/ √ B ) , T ) . For batched linear bandits with adversarial contexts , Han et al . ( 2020 ) showed that the minimax regret bound is Õ ( √ dT+dT/B ) where d is the dimension of the context vectors . Better rates can be achieved for batched linear bandits with stochastic contexts as shown in Esfandiari et al . ( 2021 ) ; Han et al . ( 2020 ) ; Ruan et al . ( 2020 ) . For the rare policy switch model , the minimax optimal regret bound for PFE is O ( √ T log n ) in terms of both the expected regret ( Kalai and Vempala , 2005 ; Geulen et al. , 2010 ; Cesa-Bianchi et al. , 2013 ; Devroye et al. , 2015 ) and high-probability guarantees ( Altschuler and Talwar , 2018 ) , where T is the number of rounds , and n is the number of possible actions . For MAB , the minimax regret bound has been shown to be Õ ( T 2/3n1/3 ) by Arora et al . ( 2012 ) ; Dekel et al . ( 2014 ) . For stochastic linear bandits , Abbasi-Yadkori et al . ( 2011 ) proposed a rarely switching OFUL algorithm achieving Õ ( d √ T ) regret with log ( T ) batches . Ruan et al . ( 2020 ) proposed an algorithm achieving Õ ( √ dT ) regret with less than O ( d log d log T ) batches for stochastic linear bandits with adversarial contexts . For episodic RL with finite state and action space , Bai et al . ( 2019 ) proposed an algorithm achieving Õ ( √ H3SAT ) regret with O ( H3SA log ( T/ ( AH ) ) ) local switching cost where S and A are the number of states and actions respectively . They also provided a Ω ( HSA ) lower bound on the local switching cost that is necessary for sublinear regret . For the global switching cost , Zhang et al . ( 2021 ) proposed an MVP algorithm with at most O ( SA log ( KH ) ) global switching cost for time-homogeneous tabular MDPs .

Summary: This paper considers online episodic RL with linear function approximation under adaptivity constraints. In particular, two types of limited adaptivity models are considered: - Batch learning model, where the agent is allowed to update the policy at $B$ pre-specifed batch grids. Using a batched version of LSVI-UCB we can achieve $\tilde{O}(\sqrt{d^3 H^3 T} + dHT/B)$ regret with $B$ uniform batch grids, so $\sqrt{T/dH}$ batches are sufficient. A lower bound is provided which shows the tight dependency on $B$. - Rare policy switch model, where the agent is allowed to switch policy ($i.e., \pi^k \ne \pi^{k-1}$) for at most $B$ times. Using a modified version of the rare switching OFUL we can achieve $\tilde{O}(\sqrt{d^3 H^3 T[1+ T/(dH)]^{dH/B}} )$ regret with $B$ switches, so $dH \log T$ switches are sufficient. No lower bound is provided in this setting.  The authors test out their algorithm with the hard-to-learn linear MDP (Zhou et al. 2021b) and the numerical results match theoretical gurantees.