Provable Model-based Nonlinear Bandit and Reinforcement Learning: Shelve Optimism, Embrace Virtual Curvature

1 Introduction . Recent progresses demonstrate many successful applications of deep reinforcement learning ( RL ) in robotics [ 49 ] , games [ 7 , 66 ] , computational biology [ 56 ] , etc . However , theoretical understanding of deep RL algorithms is limited . Last few years witnessed a plethora of results on linear function approximations in RL [ 80 , 65 , 40 , 73 , 71 , 19 , 3 , 32 ] , but the analysis techniques appear to strongly rely on ( approximate ) linearity and hard to generalize to neural networks . The goal of this paper is to theoretically analyze model-based nonlinear bandit and RL with neural net approximation , which achieves amazing sample-efficiency in practice ( see e.g. , [ 37 , 10 , 30 , 31 , 12 ] ) . We focus on the setting where the state and action spaces are continuous . Past theoretical work on model-based RL studies families of dynamics with restricted complexity measures such as Eluder dimension [ 60 ] , witness rank [ 68 ] , the linear dimensionality [ 77 ] , and others [ 58 , 42 , 17 ] . Implications of these complexity measures have been studied , e.g. , finite mixture of dynamics [ 6 ] and linear models [ 64 ] have bounded Eluder dimensions . However , it turns out that none of the complexity measures apply to the family of MDPs with even barely nonlinear dynamics , e.g. , MDPs with dynamics parameterized by all one-layer neural network with a single activation unit ( and with bounded weight norms ) . For example , in Theorem 5.2 , we will prove that one-layer neural nets do not have polynomially-bounded Eluder dimension.1 ( See more evidence below . ) The limited progress on neural net approximation is to some extent not surprising . Given a deterministic dynamics with known neural net parameters , finding the best parameterized policy still involves optimizing a complex non-concave function , which is in general computationally intractable . More 1This result is also proved by the concurrent Li et al . [ 50 , Theorem 8 ] independently . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . fundamentally , we find that it is also statistically intractable for solving the one-hidden-layer neural net bandit problem ( which is a strict sub-case of deep RL ) . In other words , it requires exponential ( in the input dimension ) samples to find the global maximum ( see Theorems 5.1 ) . This also shows that conditions in past work that guarantee global convergence can not apply to neural nets . Given these strong impossibility results , we propose to reformulate the problem to finding an approximate local maximum policy with guarantees . This is in the same vein as the recent fruitful paradigm in non-convex optimization where researchers disentangle the problem into showing that all local minima are good and fast convergence to local minima ( e.g. , see [ 28 , 26 , 27 , 29 , 48 ] ) . In RL , local maxima can often be global as well for many cases [ 4 ] .2 Zero-order optimization or policy gradient algorithms can converge to local maxima and become natural potential competitors . They are widely believed to be less sample-efficient than the modelbased approach because the latter can leverage the extrapolation power of the parameterized models . Theoretically , our formulation aims to characterize this phenomenon with results showing that the model-based approach ’ s sample complexity mostly depends ( polynomially ) on the complexity of the model class , whereas policy gradient algorithms ’ sample complexity polynomially depend on the dimensionality of policy parameters ( in RL ) or actions ( in bandit ) . Our technical goal is to answer the following question : Can we design algorithms that converge to approximate local maxima with sample complexities that depend only and polynomially on the complexity measure of the dynamics/reward class ? We note that this question is open even if the dynamics hypothesis class is finite , and the complexity measure is the logarithm of its size . The question is also open even for nonlinear bandit problems ( where dynamics class is replaced by reward function class ) , with which we start our research . We consider first nonlinear bandit with deterministic reward where the reward function is given by η ( θ , a ) for action a ∈ A under instance θ ∈ Θ . We use sequential Rademacher complexity [ 62 , 63 ] to capture the complexity of the reward function η . Our main result for nonlinear bandit is stated as follows . Theorem 1.1 ( Informal version of Theorem 3.1 ) . Suppose the sequential Rademacher complexity of a loss function class ( defined later ) induced by the reward function class { η ( θ , · ) : θ ∈ Θ } is bounded by √ R ( Θ ) Tpolylog ( T ) . Then , there exists an algorithm ( ViOlin , Alg . 1 ) that finds an -approximate local maximum with Õ ( R ( Θ ) −8 ) samples . In contrast to zero-order optimization , which does not use the parameterization of η and has a sample complexity depending on the action dimension , our bound only depends on the complexity of the reward function class . This suggests that our algorithm exploits the extrapolation power of the reward function class . To the best of our knowledge , this is the first action-dimension-free result for both linear and nonlinear bandit problems . More concretely , we instantiate our theorem to the following settings and get new results that leverage the model complexity ( more in Section 3.1 ) . 1 . Linear bandit with finite parameter space Θ . Because η is concave in action a , our result leads to a sample complexity O ( poly ( log|Θ| , 1/ ) ) for finding an -approximate optimal action . In this case both zero-order optimization and the SquareCB algorithm in Foster , Rakhlin [ 24 ] have sample complexity/regret that depend on the dimension of action space dA . 2 . Linear bandit with s-sparse or structured instance parameters . Our algorithm ViOlin achieves an O ( poly ( s , 1/ ) ) sample complexity when the instance/model parameter is s-sparse and the reward is deterministic . The sample complexity of zero-order optimization depends polynomially on dA . Carpentier , Munos [ 9 ] achieve a stronger Õ ( s √ T ) regret bound for s-sparse linear bandits with actions set A = Sd−1 . In contrast , our ViOlin algorithm applies more generally to any structured instance parameter set . Other related results either leverage the rather strong anti-concentration assumption on the action set [ 72 ] , or have implicit dimension dependency [ 33 , Remark 4.3 ] . 3 . Two-layer neural nets bandit . Our algorithm finds an -approximate local maximum Õ ( poly ( 1/ ) ) . Zero-order optimization can also find a local maximum but with Ω ( dA ) samples . 2The all-local-maxima-are-global condition only needs to hold to the ground-truth total expected reward function . This potentially can allow disentangled assumptions on the ground-truth instance and the hypothesis class . Optimistic algorithms in this case have an exponential sample complexity ( see Theorem 5.3 ) . Moreover , when the second layer of the ground-truth network contains all negative weights and the activation is convex and monotone , all local maxima are global because the reward is concave in the input ( action ) [ 5 ] . The results for bandit can be extended to model-based RL with deterministic nonlinear dynamics and deterministic reward . Theorem 1.2 ( Informal version of Theorem 4.4 ) . Consider RL problems with deterministic dynamics class and stochastic policy class that satisfy some Lipschitz properties . Suppose the sequential Rademacher complexity of the ` 2 losses for learning the dynamics is bounded by√ R ( Θ ) Tpolylog ( T ) . Then , ViOlin for RL ( Alg . 2 ) finds an -approximate local maximum policy with Õ ( R ( Θ ) −8 ) samples . To the best of our knowledge , this is the first model-based RL algorithms with provable finite sample complexity guarantees ( for local convergence ) for general nonlinear dynamics . The work of [ 55 ] is the closest prior work which also shows local convergence , but its conditions likely can not be satisfied by any parameterized models ( including linear models ) . We also present a concrete example of RL problems with nonlinear models satisfying our Lipschitz assumptions in Example 4.3 of Section 4 , which may also serve as a testbed for future model-based deep RL analysis . Other prior works on model-based RL do not apply to one-hidden-layer neural nets because they conclude global convergence which is not possible for one-hidden-layer neural nets in the worst case . Technical novelty : exploring by model-based curvature estimate . The main challenge is that the optimism-in-face-of-uncertainty exploration principle seems to be too aggressive—for a nonlinear model class , even if the ground-truth model is linear , optimistic exploration leads to exponentially large model cumulative prediction errors and exponential sample complexity ( see Theorem 5.3 ) . We use an online learning oracle to learn the dynamics , which can dramatically reduce the model prediction errors by hedging the risks . However , now , choosing actions or policies by maximizing virtual return may lack exploration . The main novelty of our algorithm is that we augment the online learning loss with gradient and Hessian estimation errors . The key insight is that , in order to ensure sufficient exploration for converging to local maxima , it suffices for the model to predict the virtual return , its gradient and Hessian reasonably accurately . We refer to the approach as “ model-based curvature estimate ” . Our algorithm alternates between maximizing virtual return ( over action or policy ) and learning the model parameters by an online learner . Because the algorithm leverages model extrapolation , the sample complexity of model-based curvature prediction depends on the model complexity instead of action dimension in the zerooptimization approach . Foster , Rakhlin [ 24 ] also propose algorithms that do not rely on UCB— their exploration strategy either relies on the finite discrete action space , or leverages the linear structure in the action space and has action-dimension dependency . In contrast , our algorithms ’ exploration relies more on the learning of the model . Consequently , our sample complexity can be action-dimension-free . 2 Problem Setup and Preliminaries . In this section , we first introduce our problem setup for nonlinear bandit and reinforcement learning , and then the preliminary for online learning and sequential Rademacher complexity . 2.1 Nonlinear Bandit Problem with Deterministic Reward . We consider deterministic nonlinear bandits with continuous actions . Let θ ∈ Θ be the parameter that specifies the bandit instance , a ∈ RdA the action , and η ( θ , a ) ∈ [ 0 , 1 ] the reward function . Let θ ? denote the unknown ground-truth parameter . Throughout the paper , we work under the realizability assumption that θ ? ∈ Θ . A bandit algorithm aims to maximize η ( θ ? , a ) . Let a ? = argmaxa η ( θ ? , a ) be the optimal action ( breaking tie arbitrarily ) . Let ‖H‖sp be the spectral norm of a matrix H . We assume that the reward function , its gradient and Hessian matrix are Lipschitz , which are standard in the optimization literature ( e.g. , Johnson , Zhang [ 41 ] , Ge et al . [ 26 ] ) . Assumption 2.1 . We assume that for all θ ∈ Θ , supa ‖∇aη ( θ , a ) ‖2 ≤ ζg and supa ∥∥∇2aη ( θ , a ) ∥∥sp ≤ ζh . For every θ ∈ Θ and a1 , a2 ∈ RdA , ∥∥∇2aη ( θ , a1 ) −∇2aη ( θ , a2 ) ∥∥sp ≤ ζ3rd ‖a1 − a2‖2 . As a motivation to consider deterministic rewards , we prove in Theorem B.1 for a special case that no algorithm can find a local maximum in less than √ dA steps . The result implies that an actiondimension-free sample complexity bound is impossible with standard sub-Gaussian noise.3 Approximate local maxima . In this paper , we aim to find a local maximum of the real reward function η ( θ ? , · ) . A point x is an ( g , h ) -approximate local maximum of a twice-differentiable function f ( x ) if ‖∇f ( x ) ‖2 ≤ g , and λmax ( ∇2f ( x ) ) ≤ h. As argued in Sec . 1 and proved in Sec . 5 , because reaching a global maximum is computational and statistically intractable for nonlinear problems , we only aim to reach a local maximum . Sample complexity and local regret for converging to local maxima . Let at be the action that the algorithm takes at time step t. The sample complexity for converging to approximate local maxima is defined to be the minimal number of steps T such that there exists t ∈ [ T ] where at is an ( g , h ) approximate local maximum with probability at least 1−δ . We also define the “ local regret ” by comparing with an approximate local maximum . We defer the discussion to Appendix C.5 .

Summary: This paper studies model-based nonlinear bandit and reinforcement learning and proposes the ViOlin algorithm (Virtual Ascent with Online Model Learner). The main contributions are as follows.  1) For bandit problems, with deterministic reward functions satisfying Assumption 2.1 (Lipschitz over continuous actions, smooth over actions, and the third order derivative over action is bounded), and with sublinear sequential Rademacher complexity, they show that the ViOlin algorithm achieves an $\epsilon$ approximate local maximum using $\tilde{O}(\epsilon^{-8})$ iterations, which does not depend on the dimension of action space.  Some instantiations are shown, including linear bandit with finite model class, with sparse or structured model vectors, and two-layer neural nets, which is the first result of non-linear bandit with neural network parameterization.  The main idea is to fit the estimation of gradient and Hessian, and show that the current greedy action can be improved until the parameter is close enough to an approximate local maximum.  2) For RL, with deterministic transition, and Assumptions 4.1 and 4.2 (regularity of value function and policy), they show that the similar ViOlin (Algorithm 2) achieves an $\epsilon$ approximate local maximum using also sublinear iterations.  The difference with bandit design is that the loss function is using one-step model prediction errors rather than gradient and Hessian.  3) The authors show some lower bound results. In particular, (i) using ReLU activations in one-layer neural network bandits, finding global optimal action takes exponentially large iterations. (ii) the failure of Eluder dimension: one-layer neural networks has its Eluder dimensions exponentially large. (iii) the failure of UCB: with linear reward function, and two-layer neural network as the hypothesis function class, UCB would take exponential sample complexity to find the optimal action.