Optimistic Policy Optimization with General Function Approximations

√ T -regret that scales polynomially in the eluder dimension of the general model class . Here T is the number of steps taken by the agent . In particular , we specialize such a regret to handle two nonparametric model classes ; one based on reproducing kernel Hilbert spaces and another based on overparameterized neural networks . 1 INTRODUCTION . Reinforcement learning with neural networks achieved impressive empirical breakthroughs ( Mnih et al. , 2015 ; Silver et al. , 2016 ; 2017 ; Berner et al. , 2019 ; Vinyals et al. , 2019 ) . These algorithms are often based on policy optimization ( Williams , 1992 ; Baxter & Bartlett , 2000 ; Sutton et al. , 2000 ; Kakade , 2002 ; Schulman et al. , 2015 ; 2017 ) . Compared with value-based approaches , which iteratively estimate the optimal value function , policy-based approaches directly optimize the expected total reward , which leads to more steady policy improvement . In particular , as shown in this paper , policy optimization generates steadily improving stochastic policies and consequently allow adversarial environments . On the other hand , policy optimization often suffers from a lack of computational and statistical efficiency in practice , which calls for the principled design of efficient algorithms . Specifically , in terms of computational efficiency , the recent progress ( Abbasi-Yadkori et al. , 2019a ; b ; Bhandari & Russo , 2019 ; Liu et al. , 2019 ; Agarwal et al. , 2019 ; Wang et al. , 2019 ) establishes the convergence of policy optimization to a globally optimal policy given sufficiently many data points , even in the presence of neural networks . However , in terms of sample efficiency , it remains less understood how to sequentially acquire the data points used in policy optimization while balancing exploration and exploitation , especially in the presence of neural networks , despite the recent progress ( Cai et al. , 2019 ; Agarwal et al. , 2020 ) . In particular , such a lack of sample efficiency prohibits the principled applications of policy optimization in critical domains , e.g. , autonomous driving and dynamic treatment , where data acquisition is expensive . In this paper , we aim to provably achieve sample efficiency in model-based policy optimization , which is quantified via the lens of regret . In particular , we focus on the episodic setting with general function approximations on the transition kernel . Such a setting is studied by Russo & Van Roy ( 2013 ; 2014 ) ; Osband & Van Roy ( 2014 ) ; Ayoub et al . ( 2020 ) ; Wang et al . ( 2020 ) , which however focus on value iteration . In contrast , policy optimization remains less understood , despite its critical role in practice . To this end , we propose an optimistic policy optimization algorithm , which achieves exploration by incorporating optimism into policy evaluation and propagating it through policy improvement . In particular , we establish a κ ( P ) · √ H3T -regret of the proposed algorithm , which matches that of existing value iteration algorithms but additionally allow the reward function to adversarially vary across each episode . Here T is the number of steps , H is the length of each episode , and κ ( P ) is the model capacity , which is defined based on the eluder dimension . Moreover , we instantiate the proposed algorithm for the special cases of reproducing kernel Hilbert spaces and overparameterized neural networks , both of which are infinite-dimensional model classes . Our work is related to the study on computational efficiency of policy optimization ( Fazel et al. , 2018 ; Yang et al. , 2019 ; Abbasi-Yadkori et al. , 2019a ; b ; Bhandari & Russo , 2019 ; Liu et al. , 2019 ; Agarwal et al. , 2019 ; Wang et al. , 2019 ) . These works assume either the transition model is known or there exists a well-explored behavior policy such that the policy update direction can be estimated accurately . With such assumptions , the tradeoff between exploration and exploitation is absent and their focus is solely on the computational aspect . In addition , our work is related to the works on adversarial MDP ( Even-Dar et al. , 2009 ; Yu et al. , 2009 ; Neu et al. , 2010a ; b ; Zimin & Neu , 2013 ; Neu et al. , 2012 ; Rosenberg & Mansour , 2019b ; a ) . The algorithm in these work directly estimates the visitation measure and their algorithm utilize mirror descent to handle adversarial reward functions . Furthermore , our work is closely related the recent work on the sample complexity of policy optimization methods Cai et al . ( 2019 ) , which only focus on the tabular and linear settings . In contrast , our work consider general function approximation setting , which is significantly more general . Moreover , the construction of optimistic policy evaluation is related to Ayoub et al . ( 2020 ) , where the similar approach is incorporated in estimating the optimal value function . The theoretical foundation of such a type of optimistic estimation is innovated by Russo & Van Roy ( 2014 ) in the bandit problem . In particular , to characterizing the optimism and accuracy of the optimistic evaluation , we rely on the notion of the eluder dimension proposed by Russo & Van Roy ( 2014 ) , which is further instantiated by this paper to the cases of kernel and neural function approximations . 1.1 NOTATIONS . We denote by ‖ · ‖p the ` p-norm of a vector when p ∈ N or the spectral norm of a matrix when p = 2 . For any two distributions p1 , p2 over a discrete set A , we denote by DKL ( p1 ‖ p2 ) the KL-divergence DKL ( p1 ‖ p2 ) = ∑ a∈A p1 ( a ) log p1 ( a ) p2 ( a ) . For any a , b , x ∈ R , we define the clamp function clamp ( x , a , b ) = b , if x > b , x , if a ≤ x ≤ b , a , if x < a . ( 1.1 ) 2 PRELIMINARIES . 2.1 ONLINE REINFORCEMENT LEARNING WITH ADVERSARIAL REWARDS . We consider an episodic MDP ( S , A , H , { Ph } Hh=1 , { rh } Hh=1 ) , where S is a continuous state space , A is a discrete action space , H is the number of steps in each episode , { Ph } Hh=1 represent the unknown transition model , and { rh } Hh=1 represent the reward function . In particular , for any h ∈ [ H ] , Ph represents the transition kernel from a state-action pair ( sh , ah ) at the h-th step to the next state sh+1 , while rh represents the reward function at the h-th step that maps a state-action pair to a deterministic reward . Moreover , we allow the reward function to vary across episodes and denote by rkh the reward function at the h-th step of the k-th episode . In particular , rkh depends on the trajectories before the k-th episode begins , possibly in an adversarial manner , and remains unobservable until the k-th episode ends . Without loss of generality , we assume each episode starts from a fixed state s1 and all rewards fall in the interval [ 0 , 1 ] . For any h ∈ [ H ] , a policy πh represents the conditional distribution of the action given the state at the h-th step . We drop the subscript h to represent the collection of policies at all steps and still refer to such a collection as a policy when it is clear from the context . For any ( k , h ) ∈ N× [ H ] , given a policy π and reward functions { rkh } Hh=1 , the value function and Q-function at the h-th step of the k-th episode are defined by V π , kh ( s ) = Eπ [ H∑ j=h rkj ( sj , aj ) ∣∣∣ sh = s ] , Qπ , kh ( s , a ) = Eπ [ H∑ j=h rkj ( sj , aj ) ∣∣∣ sh = s , ah = a ] for any ( s , a ) ∈ S × A . Here the subscript π in the expectation Eπ [ · ] denotes all actions are taken according to the policy π except for the one given in the condition . An online algorithm aims to construct and execute a sequence of policies { πk } k≥1 and minimize the regret Regret ( T ) = max π K∑ k=1 ( V π , k1 ( s1 ) − V πk , k 1 ( s1 ) ) , ( 2.1 ) where K is the number of episodes and T = KH is the number of steps taken by the algorithm . 2.2 REPRODUCING KERNEL HILBERT SPACE . We say H is a reproducing kernel Hilbert space ( RKHS ) on a set Y with the reproducing kernel K : Y × Y → R if there exists an inner product 〈· , ·〉H such that , for any f ∈ H and x ∈ Y , we have f ( x ) = 〈f , Kx〉H . Here Kx represents the function K ( x , · ) , which is the Riesz representation of the evaluation functional at x ( Schölkopf et al. , 2002 ) . When the reproducing kernel K is continuous , symmetric , and positive definite , Mercer ’ s theorem ( Steinwart & Christmann , 2008 ) says K has the representation K ( x , y ) = ∞∑ j=1 λjφj ( x ) φj ( y ) , for any x , y ∈ Y , ( 2.2 ) where { φj } ∞j=1 is an orthonormal basis of L2 ( Y ) and λ1 ≥ λ2 ≥ · · · ≥ 0 . See more details on RKHS in Section A . 3 ALGORITHM . Framework : Before the k-th episode begins , we construct the policy πk based on πk−1 and { Qk−1h } Hh=1 , which are the policy in the ( k − 1 ) -th episode and estimators of { Q πk−1 , k−1 h } Hh=1 , respectively . Then , we execute the policy πk in the k-th episode and correspondingly update the Q-function estimators { Qkh } Hh=1 using the reward function { rkh } Hh=1 , which is observed after the k-th episode ends . Policy Improvement : For any ( k , h ) ∈ [ K ] × [ H ] , we parametrize πkh by πkh ( a | s ) = exp { Ekh ( s , a ) } ∑ a′∈A exp { Ekh ( s , a′ ) } , for any ( s , a ) ∈ S ×A . ( 3.1 ) Here Ekh is the potential function , which is initialized as the zero function and updated by Ekh ( s , a ) = E k−1 h ( s , a ) + α ·Q k−1 h ( s , a ) . ( 3.2 ) Here α > 0 is the stepsize of policy improvement . Equivalently , we have πkh ( · | s ) ∝ πk−1h ( · | s ) · exp ( α ·Qk−1h ( s , · ) ) for any s ∈ S. To see ( 3.2 ) is a policy improvement step , note that πkh is the maximizer of Lkh ( πh ) = Eπk−1 [ 〈Qk−1h ( sh , · ) , πh ( · | sh ) 〉A + α −1 ·DKL ( πh ( · | sh ) ∥∥πk−1h ( · | sh ) ) ] . This is the same as the update in Politex ( Abbasi-Yadkori et al. , 2019a ) , which originates at MDP-E ( Even-Dar et al. , 2009 ) . It is also very close to a slightly changed variant of the one-step objective in the proximal policy optimization ( PPO ) algorithm ( Schulman et al. , 2017 ; 2015 ) . Policy Evaluation : Let P be a known class of transition models such that Ph ∈ P for any h ∈ [ H ] , which is specified in Section 4 . Also , for any P ∈ P , s ∈ S , a ∈ A , and V : S → [ 0 , H ] , we define zP ( s , a , V ) = ∫ S V ( s′ ) · P ( s′ | s , a ) ds′ . ( 3.3 ) For any ( k , h ) ∈ [ K ] × [ H ] , we construct a confidence set of the transition model Ph and correspondingly the optimistic Q-function estimator Qkh using the data collected before the k-th episode begins . Note that we do not use the data collected from the k-th episode although they are available , which , however , is only used to simplify the analysis . Let V kH+1 be the zero function . Inspired by Ayoub et al . ( 2020 ) , given the optimistic value function estimators { V τh+1 } k−1 τ=1 from the first ( k − 1 ) episodes , we construct the confidence set Pkh of Ph by Pkh = { P ∈ P ∣∣∣ k−1∑ τ=1 ( zP ( s τ h , a τ h , V τ h+1 ) − zPkh ( s τ h , a τ h , V τ h+1 ) ) 2 ≤ β } , ( 3.4 ) where P kh = argmin P∈P k−1∑ τ=1 ( V τh+1 ( s τ h+1 ) − zP ( sτh , aτh , V τh+1 ) ) 2 for a threshold β > 0 , which represents the degree of optimism . Then , for any ( s , a ) ∈ S ×A , given the optimistic value function estimator V kh+1 , we define the optimistic Q-function estimator Q k h by Qkh ( s , a ) = r k h ( s , a ) + max P∈Pkh zP ( s , a , V k h+1 ) ( 3.5 ) and correspondingly update the optimistic value function estimator by V kh ( s ) = 〈Qkh ( s , · ) , πkh ( · | s ) 〉A for any s ∈ S . We apply the clamp function defined in ( 1.1 ) to the second term on the right-hand side of ( 3.5 ) to ensure it falls in the range [ 0 , H − h ] , which is due to the assumption that all rewards fall in the range [ 0 , 1 ] . Implementation : The full algorithm is presented in Algorithm 1 . Given a parametrization of the model class P , we can apply the projected stochastic gradient descent ( PSGD ) algorithm to solve the constrained minimization problem in Line 11 of Algorithm 1 . In particular , for kernel function approximations in Section 4.1 , it reduces to a convex optimization problem , which allows the PSGD algorithm to converge to a global minimizer . Meanwhile , for neural function approximations , it reduces to an approximately convex problem in the overparametrized regime ( Arora et al. , 2019 ) , which leads to the same global convergence guarantee . Also , to implement Lines 12 and 13 of Algorithm 1 , it suffices to solve a constrained maximization problem ( Feng et al. , 2020 ) , where the constraint is defined in Line 12 . The Lagrangian relaxation of such a constrained maximization problem can be solved by the PSGD algorithm in the same manner of Line 11 . In addition , to instantiate the update of Qkh in Line 13 , it suffices to solve a least-squares regression problem . In summary , we can instantiate the aforementioned steps through supervised learning oracles , which can be implemented in a computationally efficient manner .

Summary: The paper proposes an optimistic policy optimization algorithm. It is theoretically shown that the algorithm has sublinear regret for multiple model classes such as kernel function and NTK. The policy update rule also allows the algorithm to have sublinear regret for  adversarial reward function. The technique for analyzing nonparametric model classes can potentially be extended to other reinforcement learning algorithms based on optimism.