Optimism in Reinforcement Learning with Generalized Linear Function Approximation

( H √ d3T ) whereH is the horizon , d is the dimensionality of the state-action features and T is the number of episodes . This is the first statistically and computationally efficient algorithm for reinforcement learning with generalized linear functions . 1 INTRODUCTION . We study episodic reinforcement learning problems with infinitely large state spaces , where the agent must use function approximation to generalize across states while simultaneously engaging in strategic exploration . Such problems form the core of modern empirical/deep-RL , but relatively little work focuses on exploration , and even fewer algorithms enjoy strong sample efficiency guarantees . On the theoretical side , classical sample efficiency results from the early 00s focus on “ tabular ” environments with small finite state spaces ( Kearns & Singh , 2002 ; Brafman & Tennenholtz , 2002 ; Strehl et al. , 2006 ) , but as these methods scale with the number of states , they do not address problems with infinite or large state spaces . While this classical work has inspired practically effective approaches for large state spaces ( Bellemare et al. , 2016 ; Osband et al. , 2016 ; Tang et al. , 2017 ) , these methods do not enjoy sample efficiency guarantees . More recent theoretical progress has produced provably sample efficient algorithms for complex environments where function approximation is required , but these algorithms are relatively impractical ( Krishnamurthy et al. , 2016 ; Jiang et al. , 2017 ) . In particular , these methods are computationally inefficient or rely crucially on strong dynamics assumptions ( Du et al. , 2019b ) . In this paper , with an eye toward practicality , we study a simple variation of Q-learning , where we approximate the optimal Q-function with a generalized linear model . The algorithm is appealingly simple : collect a trajectory by following the greedy policy corresponding to the current model , perform a dynamic programming back-up to update the model , and repeat . The key difference over traditional Q-learning-like algorithms is in the dynamic programming step . Here we ensure that the updated model is optimistic in the sense that it always overestimates the optimal Q-function . This optimism is essential for our guarantees . Optimism in the face of uncertainty is a well-understood and powerful algorithmic principle in shorthorizon ( e.g , . bandit ) problems , as well as in tabular reinforcement learning ( Azar et al. , 2017 ; Dann et al. , 2017 ; Jin et al. , 2018 ) . With linear function approximation , Yang & Wang ( 2019 ) and Jin et al . ( 2019 ) show that the optimism principle can also yield provably sample-efficient algorithms , when the environment dynamics satisfy certain linearity properties . Their assumptions are always satisfied in tabular problems , but are somewhat unnatural in settings where function approximation is required . Moreover as these assumptions are directly on the dynamics , it is unclear how their analysis can accommodate other forms of function approximation , including generalized linear models . In the present paper , we replace explicit dynamics assumptions with expressivity assumptions on the function approximator , and , by analyzing a similar algorithm to Jin et al . ( 2019 ) , we show that the optimism principle succeeds under these strictly weaker assumptions.1 More importantly , the relaxed assumption facilitates moving beyond linear models , and we demonstrate this by providing the first practical and provably efficient RL algorithm with generalized linear function approximation . The paper is organized as follows : In Section 2 we formalize our setting , introduce the optimistic closure assumption , and discuss related assumptions in the literature . In Section 3 we study optimistic closure in detail and verify that it is strictly weaker than the recently proposed Linear MDP assumption . Our main algorithm and results are presented in Section 4 , with the main proof in Section A . We close with some final remarks and future directions in Section 5 . 2 PRELIMINARIES . We consider episodic reinforcement learning in a finite-horizon markov decision process ( MDP ) with possibly infinitely large state space S , finite action space A , initial distribution µ ∈ ∆ ( S ) , transition operator P : S × A → ∆ ( S ) , reward function R : S × A → ∆ ( [ 0 , 1 ] ) and horizon H . The agent interacts with the MDP in episodes and , in each episode , a trajectory ( s1 , a1 , r1 , s2 , a2 , r2 , . . . , sH , aH , rH ) is generated where s1 ∼ µ , for h > 1 we have sh ∼ P ( · | sh−1 , ah−1 ) , rh ∼ R ( sh , ah ) , and actions a1 : H are chosen by the agent . For normalization , we assume that ∑H h=1 rh ∈ [ 0 , 1 ] almost surely . A ( deterministic , nonstationary ) policy π = ( π1 , · · · , πH ) consists of H mappings πh : S → A , where πh ( sh ) denotes the action to be taken at time point h if at state sh ∈ S The value function for a policy π is a collection of functions ( V π1 , . . . , V π H ) where V π h : S → R is the expected future reward the policy collects if it starts in a particular state at time point h. Formally , V πh ( s ) , E [ H∑ h′=h rh′ | sh = s , ah : H ∼ π ] . The value for a policy π is simply V π , Es1∼µ [ V π1 ( s1 ) ] , and the optimal value is V ? , maxπ V π , where the maximization is over all nonstationary policies . The typical goal is to find an approximately optimal policy , and in this paper , we measure performance by the regret accumulated over T episodes , Reg ( T ) , TV ? − E [ T∑ t=1 H∑ h=1 rh , t ] . Here rh , t is the reward collected by the agent at time point h in the tth episode . We seek algorithms with regret that is sublinear in T , which demonstrates the agent ’ s ability to act near-optimally over the long run . 2.1 Q-VALUES AND FUNCTION APPROXIMATION . For any policy π , the state-action value function , or the Q-function is a sequence of mappings Qπ = ( Qπ1 , . . . , Q π H ) where Q π h : S ×A → R is defined as Qπh ( s , a ) , E [ H∑ h′=h rh′ | sh = s , ah = a , ah+1 : H ∼ π ] . The optimal Q-function is Q ? h , Q π ? h where π ? , argmaxπ V π is the optimal policy . In the value-based function approximation setting , we use a function class G to model Q ? . In this paper , we always take G to be a class of generalized linear models ( GLMs ) , defined as follows : Let d ∈ N be a dimensionality parameter and let Bd , { x ∈ Rd : ‖x‖2 ≤ 1 } be the ` 2 ball in Rd . 1This is also mentioned as a remark in Jin et al . ( 2019 ) . Definition 1 . For a known feature map φ : S × A → Bd and a known link function f : [ −1 , 1 ] 7→ [ −1 , 1 ] the class of generalized linear models is G , { ( s , a ) 7→ f ( 〈φ ( s , a ) , θ〉 ) : θ ∈ Bd } . As is standard in the literature ( Filippi et al. , 2010 ; Li et al. , 2017 ) , we assume the link function satisfies certain regularity conditions . Assumption 1. f ( · ) is either monotonically increasing or decreasing . Furthermore , there exist absolute constants 0 < κ < K < ∞ and M < ∞ such that κ ≤ |f ′ ( z ) | ≤ K and |f ′′ ( z ) | ≤M for all |z| ≤ 1 . For intuition , two example link functions are the identity map f ( z ) = z and the logistic map f ( z ) = 1/ ( 1 + e−z ) with bounded z . It is easy to verify that both of these maps satisfy Assumption 1 . 2.2 EXPRESSIVITY ASSUMPTIONS : REALIZABILITY AND OPTIMISTIC CLOSURE . To obtain sample complexity guarantees that scale polynomially with problem parameters in the function approximation setting , it is necessary to posit expressivity assumptions on the function class G ( Krishnamurthy et al. , 2016 ; Du et al. , 2019a ) . The weakest such condition is realizability , which posits that the optimal Q function is in G , or at least well-approximated by G. Realizability alone suffices for provably efficient algorithms in the “ contextual bandits ” setting where H = 1 ( Li et al. , 2017 ; Filippi et al. , 2010 ; Abbasi-Yadkori et al. , 2011 ) , but it does not seem to be sufficient whenH > 1 . Indeed in these settings it is common to make stronger expressivity assumptions ( Chen & Jiang , 2019 ; Yang & Wang , 2019 ; Jin et al. , 2019 ) . Following these works , our main assumption is a closure property of the Bellman update operator Th . This operator has type Th : ( S ×A → R ) → ( S ×A → R ) and is defined for all s ∈ S , a ∈ A as Th ( Q ) ( s , a ) , E [ rh + VQ ( sh+1 ) | sh = s , ah = a ] , VQ ( s ) , max a∈A Q ( s , a ) . The Bellman update operator for time point H is simply TH ( Q ) ( s , a ) , E [ rH | sH = s , aH = a ] , which is degenerate . To state the assumption , we must first define the enlarged function class Gup . For a d × d matrix A , A 0 denotes that A is positive semi-definite . For a positive semi-definite matrix A , ‖A‖op is the matrix operator norm , which is just the largest eigenvalue , and ‖x‖A , √ x > Ax is the matrix Mahalanobis seminorm . For a fixed constant Γ ∈ R+ that we will set to be polynomial in d and log ( T ) , define Gup , { ( s , a ) 7→ 1 ∧ f ( 〈φ ( s , a ) , θ〉 ) + γ ‖φ ( s , a ) ‖A : θ ∈ Bd , A 0 , ‖A‖op ≤ 1 } , Here we use a∧b , min { a , b } . The class Gup contains G in addition to all possible upper confidence bounds that arise from solving least squares regression problems using the class G. We now state our main expressivity assumption , which we call optimistic closure . Assumption 2 ( Optimistic closure ) . For any 1 ≤ h < H and g ∈ Gup , we have Th ( g ) ∈ G. In words , when we perform a Bellman backup on any upper confidence bound function for time point h + 1 , we obtain a generalized linear function at time h. While this property seems quite strong , we note that a similar notion is mentioned informally in Jin et al . ( 2019 ) and that related closure-type assumptions are common in the literature ( see Section 2.3 for detailed discussion ) . More importantly , we will prove in Section 3 that optimistic closure is actually strictly weaker than previous assumptions used in our RL setting where exploration is required . Before turning to these discussions , we mention two basic properties of optimistic closure . Fact 1 ( Optimistic closure and realizability ) . Optimistic closure implies that Q ? ∈ G ( realizability ) . Proof . We will solve for Q ? via dynamic programming , starting from time point H . In this case , the Bellman update operator is degenerate , and we start by observing that TH ( g ) ≡ Q ? H for all g. Consequently we have Q ? H ∈ G. Next , inductively we assume that we have Q ? h+1 ∈ G , which implies that Q ? h+1 ∈ Gup as we may take the same parameter θ and set A ≡ 0 . Then , by the standard Bellman fixed-point characterization , we know that Q ? h = Th ( Q ? h+1 ) , at which point Assumption 2 yields that Q ? h ∈ G. Fact 2 ( Optimistic closure in tabular settings ) . If S is finite and φ ( s , a ) = es , a is the standard-basis feature map , then under Assumption 1 we have optimistic closure . Proof . We simply verify that G contains all mappings from ( s , a ) 7→ [ 0 , 1 ] , at which point the result is immediate . To see why , observe that via Assumption 1 we know that f is invertible ( it is monotonic with derivative bounded from above and below ) . Then , note that any function ( s , a ) 7→ [ 0 , 1 ] can be written as a vector v ∈ [ 0 , 1 ] |S|×|A| . For such a vector v , if we define θs , a , f−1 ( vs , a ) we have that f ( 〈es , a , θ〉 ) = vs , a . Hence G contains all functions , so we trivially have optimistic closure .

Summary: The authors studies an episodic MDP learning problem, where they propose to study an Optimistic Closure assumption which allows the Q function to be expressed as a generalized linear function plus a positive semi-definite quadratic form. They motivate the assumption by showing that the assumption allows the tabular MDP case to be modeled, and that the Optimistic Closure is in fact a strictly weaker assumption than the linear MDP assumption made in previous related works. The authors then proceed to the design and analysis of the LSVI-UCB algorithm, which involves estimating the the parameter of the GLM model by a ridge estimator and adding an optimistic exploration bonus to the Q function. The authors propose a regret bound for the algorithm.