Title: On the Convergence and Sample Complexity Analysis of Deep Q-Networks with $\epsilon$-Greedy Exploration
Venue: NeurIPS 2023
Accepted: yes
Keywords: Reinforcement learning

Deep Q Network

Convergence analysis

Sample complexity

Generalization analysis
Abstract: This paper provides a theoretical understanding of deep Q-Network (DQN) with the $\varepsilon$-greedy exploration in deep reinforcement learning.
Despite the tremendous empirical achievement of the DQN, its theoretical characterization remains underexplored.
First, the exploration strategy is either impractical or ignored in the existing analysis.  
Second, in contrast to conventional Q-learning algorithms, the DQN employs the target network and experience replay to acquire an unbiased estimation of the mean-square Bellman error (MSBE) utilized in training  the Q-network. However,
the existing theoretical analysis of DQNs lacks convergence analysis or bypasses the technical challenges by deploying a significantly overparameterized neural network, which is not computationally efficient. 
This paper provides the first theoretical convergence and sample complexity analysis of the
  practical setting of DQNs with $\epsilon$-greedy policy. We prove an iterative procedure with decaying $\epsilon$ converges to the optimal Q-value function geometrically. Moreover, a higher level of $\epsilon$ values enlarges the region of convergence but slows down the convergence, while the opposite holds for a lower level of $\epsilon$ values. Experiments justify our established theoretical insights on DQNs.

On the Convergence and Sample Complexity Analysis of Deep Q-Networks with ε -Greedy Exploration

Abstract

This paper provides a theoretical understanding of Deep Q-Network (DQN) with the ε -greedy exploration in deep reinforcement learning. Despite the tremendous empirical achievement of the DQN, its theoretical characterization remains underexplored. First, the exploration strategy is either impractical or ignored in the existing analysis. Second, in contrast to conventional Q-learning algorithms, the DQN employs the target network and experience replay to acquire an unbiased estimation of the mean-square Bellman error (MSBE) utilized in training the Qnetwork. However, the existing theoretical analysis of DQNs lacks convergence analysis or bypasses the technical challenges by deploying a significantly overparameterized neural network, which is not computationally efficient. This paper provides the first theoretical convergence and sample complexity analysis of the practical setting of DQNs with ε -greedy policy. We prove an iterative procedure with decaying ε converges to the optimal Q-value function geometrically. Moreover, a higher level of ε values enlarges the region of convergence but slows down the convergence, while the opposite holds for a lower level of ε values. Experiments justify our established theoretical insights on DQNs.

1 Introduction

Reinforcement learning (RL) is a sequential decision-making process for a learning agent taking actions in the environment. RL has found important applications in autonomous control [37, 36], healthcare [16], Internet of Things [40, 33], and natural language processing [70]. The environment of an RL problem is modeled as a Markov decision process (MDP) with an underlying state transition probability matrix. The goal of the problem is to find an optimal policy to select the best actions to maximize the immediate and future rewards. Q-learning [77] has been recognized as one of the most promising and efficient learning algorithms for seeking the optimal policy because it does not require the knowledge of the model of the environment (namely, 'model free') and can learn from data generated from a non-optimal policy (namely, 'off-policy'). Traditional Q-learning approaches are centered on tabular methods [82, 41] or linear function approximations [65, 13] to estimate the optimal action-value (Q-value) function. However, tabular methods require sample complexity scaling in the order of state space, which is impractical for modern RL problems involving large or even infinite state space [82]. Q-learning with linear function approximation is limited to applications only when the transition matrix admits a linear feature representation [23].

Due to the remarkable advancements in deep neural networks (DNNs), the Deep Q-network (DQN) framework has emerged as a powerful approach that leverages the expressive power of non-linear functions and the ability to generalize to unknown states. DQN has proven to be a pioneering solution that addresses challenges encountered by traditional approaches. In the DQN framework [51], the Q-value function is approximated using a DNN, and the algorithm iteratively updates the Qvalue function by collecting data following the ε -greedy policy. The ε -greedy policy is the simplest approach to balance exploration and exploitation. Namely, with a probability of ε , we select a random action ( exploration ), and with a probability of 1 -ε , we choose the best action according to the current estimated Q-value function ( exploitation ). ε -greedy is myopic compared with other strategic explorations, e.g., Thompson sampling-based [71, 55, 25] and optimism in the face of the uncertainty (OFU)-based [30, 31] ones. However, implementing these strategic explorations is not computationally efficient in DQNs [17], while DQNs equipped with ε -greedy policy have shown empirical success in diversified applications, e.g., the game of Go [63], Atari games [51], robotics [34], and autonomous vehicles [61, 12, 60].

Despite the numerical success, the theoretical understanding of DQN remains elusive, which could prevent its applications in domains requiring reliable and safe decision-making. First, updating the Q-value function involves minimizing a mean-squared Bellman error (MSBE) function. However, existing convergence analysis and statistical properties are predominantly limited to linear models, failing to capture the complexities present in non-linear neural networks like DQN. Second, the sample complexity required for the convergence of MSBE is still not well comprehended. Achieving the desired accuracy in this context often demands a sample complexity that grows exponentially with the input dimension [46], rendering it impractical and inefficient in real-world scenarios. Third, the optimal selection of the ε value in DQN remains a gap in existing research. The hyperparameter tuning in algorithms involved with neural networks can be arduous and time-consuming. For instance, without a well-designed hyperparameter configuration, only a small fraction (e.g., 1%) of the possible combinations yield satisfactory results in neural network training [76].

Contributions. To the best of our knowledge, this paper presents the first theoretical study with convergence analysis for Deep Q-Networks (DQNs) utilizing the ε -greedy policy. A comparison with existing works can be found in Table 1. The paper focuses on the Q-value function approximated by a DNN. It offers a comprehensive analysis of the convergence of DQNs and provides insights into the estimation error of the learned Q-value function, accompanied by an analysis of the sample complexity. The key contributions of this study are as follows:

1. The convergence analysis of DQN with bounded estimation errors. Assuming the existence of a DNN with unknown weights W ⋆ that matches the optimal Q-value function, this paper proves that the learned model via DQNs equipped with ε -greedy policy through the (accelerated) gradient descent algorithm converges linearly to W ⋆ up to some characterizable estimation error.

2. The theoretical characterization for a wide selection range of ε over iterations. This paper provides lower and upper bounds of ε at each iteration for the convergence of DQNs and, in particular, characterizes the sample complexity, estimation error, and convergence rate of the DQN equipped with ε -greedy with decreasing ε . Moreover, this paper proves that a higher level of ε values leads to an enlarged region of convergence, which relaxes the requirement on the initialization, while a lower level of ε values leads to faster convergence.

3. The sample complexity analysis for learning a desired Q-value function. We quantify the required sample complexity, depending on the neural network parameters and distribution shift of the collected data, for the learned model to converge to the desired accuracy. Typically, the estimation error of the converged model scales in the order of 1 / √ N s , where N s is the number of samples.

2 Related Works.

Q-learning with linear function approximation. In the setting of linear function approximation, the Q-function is assumed to be a linear function of either the feature mapping [83, 32, 92] or a mixture of some basis kernels [91, 52]. Early works mainly focus on the algorithm design [6, 48, 2] and convergence analysis [38, 47, 67, 14, 69] but lacks theoretical guarantees with polynomial sample complexity. Assuming the underlying Q-function can be exactly represented as a linear function of the feature mapping with some unknown parameters, several sample-efficient algorithms are proposed to find the ground-truth mapping with finite-sample guarantee [11, 80], and the sample complexity depends linearly on the feature dimension [80].

Q-learning with non-linear function approximation. Recent approaches with non-linear function approximation mainly fall into the frameworks of the Bellman Eluder (BE) dimension [30, 18, 58, 27], Neural Tangent Kernel (NTK) [81, 10, 79, 10, 20, 53], and Besov regularity [68, 29, 54, 46, 24]. The Eluder dimension is at least in an exponential order even for a two-layer neural network [19], leading to uncharacterizable sample complexity. The NTK framework linearizes deep neural networks to tackle convergence in non-linear models. However, it requires using computationally inefficient extremely wide neural networks [81]. Moreover, the sample complexity can exponentially increase with the input feature dimension, necessitating a substantial number of samples for accurate estimation. Furthermore, the NTK approach fails to explain the advantages of non-linear neural networks over linear function approximation [46, 24]. Besov space requires the neural networks to be sparse and makes an unpractical assumption that the algorithm can find the global optimal of the non-convex objective function [29, 54, 46, 24]. To the best of our knowledge, only [46] considers ε -greedy policy with theoretical analysis applicable to DQNs. However, the model is limited to sparse neural networks, and it cannot characterize the case of varying ε .

Supervised learning with neural networks. Compared with Q-learning in RL, where the label of the sampled data depends on the currently estimated Q function, analyzing supervised learning is less challenging, where sampled data label is known and fixed across the training. Existing theoretical results for supervised learning are largely built upon NTK [28, 21, 39, 15, 43, 45], model recovery [90, 26, 3, 64, 88, 85, 87], and structured data [44, 62, 8, 1, 35, 78, 84, 42]. Due to the high non-convexity of neural networks [59], the one-hidden-layer neural network is still a state-of-theart practice for convergence analysis and generalization guarantees. Additional assumptions, e.g., Gaussian distribution [89, 7], linear separable data [75, 9] on the data distribution, are needed for finite-sample analysis.

3 Problem Formulation: Notation, Background, and Algorithm

The Markov Decision Process and Q-learning. A discounted Markov decision process (MDP) is defined as ( S , A , P , r, γ ) , where S is the state space, A is the action set. P : S×A -→ ∆( S ) is the transition operator, and p a s , s ′ := P ( s ′ | s , a ) denotes the transition probability from current state s and action a to the next state s ′ . In addition, r : S × A -→ [ -R max , R max ] is the reward function, and γ ∈ (0 , 1) is the discount factor.

At a state s ∈ S , the agent takes action a ∈ A according to some behavior policy π , denoted as a ∼ π ( s ) (or a = π ( s ) for deterministic policy). Then, the system moves to the next state s ′ following the transition probability p a s , s ′ . Meanwhile, the agent receives an immediate reward r ( s , a ) from the environment. Let { s i , a i } ∞ i =0 be the generated sequential data given the behavior policy π and transition probability P . We define the state-value function V π at state s as

which is the expected total discounted reward starting from the state s . For any state-action ( s , a ) , the corresponding Q-value (or action-value) function Q π of a policy π is defined as

Then, the goal of the agent is to find an optimal policy π ⋆ that maximizes the state-value function in (1) for all states, which is equivalent to

where (3) is known as the Bellman equation. With the optimal Q-value function Q ⋆ , the optimal policy can be derived via π ⋆ ( s ) = argmax a Q ⋆ ( s , a ) [77, 66].

The Deep Neural Network Model. The DQN utilizes a deep neural network (DNN) H : R d -→ R to approximate the optimal Q-value function Q ⋆ in (3). Specifically, given input x ∈ R d , the output of the L -hidden-layer DNN with K neurons in each hidden layer is defined as

where W 1 ∈ R d × K , W l ∈ R K × K with l = 2 , · · · , L , and W = [ vec ( W 1 ) ⊤ , · · · , vec ( W L ) ⊤ ] ⊤ is the concatenation of the vectorization of all parameter matrices. ϕ ( · ) is the nonlinear activation function, and we consider the ReLU activation function, i.e., ϕ ( z ) = max { 0 , z } . Then, the Q-value function Q ( s , a ) is parameterized using the DNN as

where x : S × A -→ R d is the feature mapping of the state-action pair. Without loss of generality, we assume | x ( s , a ) | ≤ 1 . Then, the goal of DQN-based Q-learning is to minimize the mean squared Bellman error (MSBE) as

where µ is the distribution of ( s , a ) following the optimal policy π ⋆ .

3.1 The Deep Q-Network Algorithm

Algorithm 1 Deep Q-Network

1: Input : Number of iterations T × M , and experience replay buffer size N , exploration probability { ε t } T t =1 , step size η , and momentum parameter β .

2: Initialize the Q-network with weights W (0 , 0) .

3: for t = 0 , 1 , 2 , · · · , T -1 do

4: Select the initial weights W ( t, 0) .

5: for m = 0 , 1 , 2 , · · · , M -1 do

6: Sample data and store in the experience replay buffer D t following ε t -greedy policy, namely, at state s n , with probability ε t , select a random action a n , otherwise select a n = argmax a Q ( W ( t, 0) ; s n , a ) .

7: Sample random mini-batch of transition D ( m ) t from the replay buffer D t .

8: Set y n = r n + γ max a Q ( W ( t, 0) ; s ′ n , a ) for n ∈ 1 , 2 , · · · , |D ( m ) t | .

9: Perform a gradient descent step

10: end for

11: Set W ( t +1 , 0) = W ( t,M ) .

12: end for

The learning problem (6) is solved via the DQN equipped with ε -greedy exploration, as summarized in Algorithm 1. In t -th outer loop, we initialize the Q-value function using currently estimated weights W ( t, 0) as Q ( W ( t, 0) ) (line 4). Then, for each inner loop, the agent selects and executes actions according to the ε -greedy policy (line 6), namely, with probability ε , we select a random action, and with probability 1 -ε , we select the action based on the greedy policy with respect to Q ( W ( t, 0) ) . The data are stored in a replay buffer with size N (line 7). Then, we sample a minibatch of independent samples from D t , denoted as D ( m ) t for the m -th inner loop (line 8). Next, we update the current weights using a mini-batch (accelerated) gradient descent algorithm (line 9). The gradient descent (GD) direction in this step at point W is represented as

where

Note that (7) can be viewed as the gradient of

which is the approximation to (6) via fixing the max a Q ( W ) as max a Q ( W ( t, 0) ) . After moving along the GD direction, accelerated gradient descent (AGD) adds a momentum term, denoted by β ( W ( t,m ) -W ( t,m -1) ) to accelerate the convergence rate [57]. Vanilla SGD can be viewed as a special case of AGD by letting β = 0 . After updating neuron weights in the inner loop, we set the network as the currently estimated Q-value function Q ( W ( t, 0) ) (line 11) and repeat the steps above.

4 Theoretical Results

4.1 Takeaways of the Theoretical Findings

We consider the general setup of DQNs with ε -greedy under some commonly used assumptions. To the best of our knowledge, we provide the first theoretical characterization of both the convergence and sample complexity analysis for DQNs with ε -greedy. The major notations are summarized in Table 2. We first briefly introduce the key takeaways of our results, and the formal theoretical results are introduced in Section 4.3.

̸

(T1) Theoretical characterizations of { ε t } T t =1 for convergence. Weprove that for a wide selection of { ε t } values that decrease over time, Algorithm 1 converges to Q ⋆ linearly up to some estimation error. Let c ε measure the value level of ε t 's. A higher level of ε values (i.e., a larger c ε ) leads to an enlarged region of convergence (in the order of c ε ), measured by the distance from the initialization W (0 , 0) to W ∗ . Thus, larger ε values relax the requirements on W (0 , 0) . A lower level of ε values (i.e., a smaller c ε ) leads to faster convergence with a rate in the order of c ε . Our findings explain the intuition that the agent tends to explore more at the beginning and exploit more after gaining enough knowledge during the exploration.

(T2) Convergence to the optimal Q-value function Q ⋆ with geometric decay. The learned models converge to the ground truth model Q ⋆ with a geometric decay up to some bounded estimation error. The convergence rate is upper bounded by γ + c ε · (1 -γ ) . When γ is close to one, the problem emphasizes long-term rewards. While the immediate reward can be observed directly, the future reward needs to be estimated by the learned Q function as shown in (7). Therefore, with a large γ , the learned Q function needs more iterations to converge, which leads to a slow convergence rate.

(T3) Sample complexity for achieving a desired estimation error of the optimal Q-value function. With the proper selection of { ε t } T t =1 , the estimation error of the learned model scales in the order of C T (1 -γ ) 2 √ N s . C T is the fraction of actions following the current greedy policy that differ from the ones following the optimal policy. N s is the number of samples. With a smaller discounted factor γ , the problem focuses more on the immediate reward, which can be observed directly, making Q ⋆ easier to learn. The learned model achieves a small estimation error given a small distribution shift, a large sample size, or a small discounted factor.

4.2 Assumptions

We propose assumptions commonly used in existing RL and neural network learning theories and notations to simplify the presentation. Assumption 1 assumes the existence of a good approximation of DNN to Q ⋆ , which guarantees (6) is solvable. The assumption is commonly used in both deep learning theories [90, 86] and reinforcement learning theories [46, 24]. Assumption 2 assumes the samples from experience replay are independent and identically distributed (i.i.d.), which follows the assumptions in existing theoretical analysis of DQN [24, 53] and matches the intuition of using experience replay to break temporal dependence among the samples. Specifically, we have

Assumption 1. There exists a DNN with weights W ⋆ such that minimizes (6) as f ( W ⋆ ) = 0 .

Assumption 1 assumes that the Q-value function with some unknown ground truth W ⋆ 1 can represent the optimal Q-value function.

Assumption 2. Suppose the mini-batch data are i.i.d. samples from the replay buffer following the distribution µ t , which is the stationary distribution of the behavior policy at t -th outer loop.

Assumption 2 assumes the mini-batch at the t -th outer loop are i.i.d. samples following µ t , where µ t is the stationary distribution of ( s , a ) generated by ε t -greedy policy at t -th outer loop 2 . The minibatch samples are close to being independent given the experience size in practice is sufficiently large ( ∼ millions [51]).

In what follows, we define two quantities C t and ρ to simplify the presentation of theoretical results.

Definition 1. C t ∈ [0 , 1] is the fraction of non-optimal state-action pair ( s , a ) in the greedy policy with respect to Q ( W ( t, 0) ) , i.e., the fraction of ( s , a ) pairs that satisfy

̸

among all ( s , a ) pairs following the greedy policy at t -th outer loop as a = argmax a ′ Q ( W ( t, 0) ; s , a ′ ) .

C t can be viewed as the difference between behavior policy and optimal policy. Theorem 1 shows the general results for any value of C t . Nevertheless, the greedy policy is improved over time, e.g., updating the weights every few steps (line 11 in Algorithm 1). Hence, C t depends on W ( t, 0) and is expected to decrease as W ( t, 0) approaching W ⋆ [56, 92], which will be discussed in Corollary 3.

Definition 2. Let ρ be the value of

ρ suggests the radius of the local convex region of the objective function. We provide the lower bound for ρ in Lemma 7 (see the proof in Appendix E.2), suggesting a sufficiently large local convex region near W ⋆ .

4.3 Major Theoretical Results

Lemma 1 characterizes the convergent point when minimizing (9) using the mini-batch gradient descent in the t -th outer loop under certain conditions. Specifically, given that the initial weights at the t -th outer loop are sufficiently close to the ground truth as shown in (12) and the replay buffer is large enough as shown in (13), the distance between the learned model weights W ( t +1 , 0) and W ⋆ are bounded from above as shown in (14).

Lemma 1 (Estimation error of W ( t +1 , 0) ) . Suppose Assumptions 1 & 2 hold and the initial neuron weights at the t -th outer loop satisfy

The step size η is 1 /T , and the size of the replay buffer is

Then, with the high probability of at least 1 -q -d , the neuron weights W ( t +1 , 0) generated from Algorithm 1 satisfy

Remark 1 (Large replay buffer reduces the estimation error and the requirement for W ( t, 0) ) . From (14), a larger N leads to a reduced distance between W ( t +1 , 0) and W ⋆ . Moreover, (12) implies that if we increase the size of replay buffer N , the upper bound of e t increases, indicating the algorithm can tolerant a large range of W ( t, 0) .

In the following corollary, we show the upper and lower bound of ε t at the t -th outer loop. The lower bound guarantees that the RHS of (12) is larger than 0 , so we have a sufficiently large radius for convergence. The upper bound ensures (14) be less than e t , indicating an improved estimation of Q ⋆ across the iterations.

Corollary 1 (Range of ε ) . Given the assumptions and conditions in Theorem 1 hold. To ensure the existence of a good initialization at iteration t , ε t needs to satisfy

To ensure the estimated learned model is improving over iterations, ε t needs to satisfy

Remark 2 (Reduce ε t as t increases) . The lower bound can always be smaller than the upper bound given a sufficiently large N as shown in (13). From both (15) and (16), we know that ε t needs to decrease as e t decreases. Specifically, the lower bound of ε t is a linear function of e t , and the upper bound of ε t is a linear function of e t . Namely, we need a relatively large ε 0 at the beginning since the distance between initial point W (0 , 0) and W ⋆ is large. As t increases, e t , which is the distance of learned neuron weights W ( t, 0) to the ground truth W ⋆ , becomes smaller, and we should decrease ε t to guarantee an improved Q ( t +1 , 0) over Q ( t, 0) .

Theorem 1 shows that the learned model from Algorithm 1 converges to the optimal Q-value function Q ⋆ with geometric decay up to an estimation error shown in (20).

Theorem 1 (Convergence to Q ⋆ ) . Suppose Assumptions 1 and 2 hold, the buffer size N satisfies (13) . Let us define C max be a constant that is larger than C t for 1 ≤ t ≤ T , when ε t satisfy

for a fixed constant c ε ∈ (0 , (1 -γ ) 2 ] , and the initialization satisfies

Then, with the high probability of at least 1 -T · q -d , we have

the learned weights decay geometrically with

the returned model Q ( W ( T, 0) ) exhibits an estimation error in the order of 1 / √ N with

Remark 3 (Selection of ε t ) . The value of ε t is influenced by three key factors: c ε , C max , and the current estimation error bound e t . The constant c ε is fixed and controls the magnitude of the values in the sequence ε t T t =1 , providing a way to regulate the level of ε t . Regarding C max , C t tends to decrease as the iteration index t increases, indicating a progressive improvement in the policy and resulting in a smaller data distribution shift. Hence, to estimate C max , we can leverage C 0 , which is obtained by collecting data based on the policy max a Q ( W (0 , 0) ; s , a ) . Moreover, with c ε fixed, the estimation error e t follows a geometric decay pattern, as depicted in (19). This behavior allows us to use the expression ( γ + c ε · (1 -γ ) ) t · e 0 as an estimate for e t .

Remark 4 (Geometric decay to Q ⋆ ) . From (19) and (20), we know that the learned model from the proposed algorithm converges to Q ⋆ with a geometric decay up to some estimation error. The convergence rate is in the order of γ + c ε · (1 -γ ) , and the estimation error is in the order of (1 -γ ) -2 · C max / √ N . As we mentioned in the takeaways in Section 4.1, a small discounted factor γ leads to a fast convergence rate. We have a reduced estimation with a large buffer with size N , a small distribution shift C T , and a small γ .

Remark 5 (Larger ε values for enlarged region of convergence and smaller ε values for faster convergence) . From (17), we know that c ε controls the value level of ε t . From (18), a larger c ε (i.e., a higher level of ε values) increases the upper bound of ∥ W (0 , 0) -W ⋆ ∥ 2 and, thus, enlarges the proper region of W (0 , 0) . (19) indicates that the convergence rate is in the order of c ε . A smaller c ε (i.e., lower level of ε values) leads to faster convergence.

In the following corollary, we provide the sample complexity for achieving δ estimation error as shown in (21), where ˜ Ω( · ) omits some log factors. The corollary can be obtained by letting (20) to be less than a desired accuracy δ .

Corollary 2 (Sample complexity) . To achieve an estimation error of δ , the required number of samples, referred to as the sample complexity, needs to satisfy

Remark 6 (Sample complexity) . (21) shows that the sample complexity is a linear function of d and L , where d is the feature mapping of the state-action pair ( s , a ) and L is the number of layers. Given the freedom of degree of W is a linear function of d and L , respectively, the sample complexity is almost order-wise optimal with respect to d and L .

The following corollary presents a tighter bound for the model estimation error compared with (20). This improvement arises from a stronger assumption on C t , which becomes a function of ∥ W ( t, 0) -W ⋆ ∥ 2 . Compared with (20), the estimation error bound in (24) and (25) considers the cases that the behavior policy is improved as W ( t, 0) becomes closer to W ⋆ . As a result, we achieve a more precise and tighter estimation of the error in the model.

Corollary 3 (Distribution shift and estimation error) . Assume that C t is H¨ older continuous with a factor α as

3 for 0 < α ≤ 1 and all t ∈ [ T ] . When ε t satisfy

the estimation error of the Q-function satisfies

In the special case that C t = O ( ∥ W ( t, 0) -W ⋆ ∥ 2 ) , then

Remark 7 (Reduced or zero estimation error when behavior policy is improved over iterations) . Recall the definition of C t in Definition 1. If e t is zero, i.e., W t = W ⋆ , the action selected following the greedy policy is always the optimal action, which means that C t = 0 . Therefore, it is reasonable to assume that C t is H¨ older continuous as shown in (22). When α > 0 , we can see that (24) is less than (20), indicating a reduced estimation error and sample complexity. Typically, if C t has an order of growth less than e t near W ⋆ , a zero estimation error is achievable.

4.4 The Roadmap of Proofs, Comparison with Existing Works, and Limitations

The proof of Theorem 1 draws inspiration from the model estimation framework in the supervised learning setting [90, 88]. The key idea is to use a population risk function (PRF) to characterize the objective function in (9). By satisfying certain conditions such as having sufficient training samples and a bounded data distribution shift, the approximation error between the PRF and objective function can be bounded. This allows for the characterization of the optimization problem in (9) by analyzing the landscape and convergence properties of the PRF.

In comparison to existing proofs based on the model estimation frameworks, this paper addresses two additional challenges. Firstly, it extends the proof from one-hidden-layer neural networks to multi-layer neural networks. This extension is achieved by providing new tools for characterizing the Hessian matrix (refer to Lemma 3) and concentration bound (refer to Lemmas 3 and 6). Additionally, this paper characterizes the differences between the two functions caused by the interaction of neuron weights across layers in the gradient and Hessian matrix. Secondly, the paper extends the proof from supervised learning settings to Q-learning settings. This requires characterizing the additional error term caused by the data distribution shift and 'noisy' labels (refer to Lemma 3) because the empirical risk function is no longer an expectation of the defined population risk function.

Existing state-of-the-art theoretical results on Q-learning with neural network approximations primarily revolve around the NTK and Besov regularity frameworks. In the NTK framework, the networks are assumed to be extremely over-parameterized, requiring an impractical projection step and resulting in error bounds that cannot be characterized for networks with finite width. In the Besov regularity framework, the neural network needs to be sparse, which does not align with the DQN algorithm. Furthermore, the analysis in the Besov regularity framework relies on the achievability of the global minimizer of the non-convex problem in (9), which cannot be guaranteed using GD algorithms. This paper takes a significant step towards bridging the gap between theoretical understanding and practical applications of DQN by addressing these challenges. However, there still remains a gap between the theoretical results and numerical findings. Future research directions include devising efficient exploration strategies for DQNs to further enhance their performance and extending the theoretical analysis to variants of DQNs and policy gradient-based methods.

5 Numerical Experiments

In this section, we provide numerical justification that our theoretical findings are aligned with practical DQNs through the Atari Pong game, which is commonly used for DQNs in [50, 51, 73]. We take the Double DQN (DDQN) [73], one of the most popular variants of DQN, as the backbone in the setup. DDQN differs from DQN only in (8) via changing y = r + γ · Q ( W ( t, 0) ; s ′ , a ⋆ ) to y = r + γ · Q ( W ( t,m ) ; s ′ , a ⋆ ) , where a ⋆ = argmax a Q ( W ( t, 0) ; s ′ , a ) . DDQNoutperforms DQN in the relief of overoptimism and the improvement of stability. Our numerical experiments on DDQN also indicates that our analysis applies to the variants of DQN.

The input to the network is 84 × 84 × 4 images, where the last dimension represents the number of frames in history. The network is a convolutional neural network consisting of three convolutional layers and one fully-connected layer. The algorithm terminates if the average score over the recent 20 episodes does not improve or the algorithm reaches the maximum episode set as 200 , which is around 4 × 10 5 training steps. The testing score is calculated based on a similar setup as the training process by fixing the maximum memory size N as 2000 and greedy policy, i.e., ε = 0 . Each point in the plot is averaged over 10 experiments with an error bar representing the standard deviation.

Estimation errors with respect to the sample complexity N . As the Q-value function is the estimate of the expected cumulative reward, we use the difference between the reward obtained from the estimated Q-value function and the maximum reward as the estimation error of the learned model to the optimal Q-value function, which is also consistent with the experiments in [50, 51, 73]. Given that the full test score in the Pong game is 21 , we set the test error as the value of ( 21 - test score ) in each experiment. The ε t in ε -greedy policy decreases geometrically from 1 to 0 . 01 . We vary the number of samples in the replay buffer from 400 to 2500 . Figure 1 shows that the test error is almost linear in 1 / √ N , which is consistent with our characterization in (20). In addition, experiments with a large N have a shorter error bar indicating a more stable learning performance with a large sample complexity as shown in (12).

Convergence with different selections of ε . Figure 2 illustrates the convergence rate when ε t in the ε -greedy policy changes. For each point, ε 0 is selected as the value in the x-axis, and we decrease ε t geometrically as the iteration t increases. Each point is averaged over 5 independent trials. We can see that the convergence rate is a linear function of c ε , matching our findings in (19).

Performance with different selections of ε . We investigate the effect of ε -greedy policy during the training. In Figure 3, we show the test scores using ε -greedy policy with (1) geometrically decreasing ε from 1 to 0 . 01 , (2) fixed ε as 0 . 1 , and (3) fixed ε as 0 . Each test score in the curve is averaged over the past 10 episodes to smooth the trend. One can observe that a gradually decreasing

0.02

0.03

0.04

0.05

0

10

20

Test Error in the Score

0.01

0.02

0.03

0.04

0.05

0.986

0.988

0.99

0.992

0.994

0.996

Convergence rate

100

200

Episode

-20

-10

0

10

20

Test Score

decreases to 0.01

Fixed  =0.1

Fixed  =0

ε leads to a better score than fixing ε to be the final value. The test score of ε = 0 . 1 shares a similar trend to the ε -greedy policy but with a slower speed, matching our findings in (19) that a small ε leads to a slow convergence rate. The test score of ε = 0 has the fastest convergence rate at the early stage, but the convergent point is the worst and the most unstable, matching our findings in (18) that a small ε leads to a reduced radius of convergence.

6 Conclusions and Discussions

This paper provides the first convergence and sample complexity analysis of the DQN algorithm equipped with ε -greedy exploration. We establish the theoretical guarantee for the convergence of the learned model to the optimal Q-value function Q ⋆ , which can be used to derive the optimal policy. We provide a nearly optimal sample complexity for achieving an arbitrarily small estimation error. We also prove that ε -greedy with decreasing ε achieves both an enlarged radius of convergence and an improved convergence rate. Future directions include the generalization of the theoretical analysis to the variants of DQNs and the design of efficient exploration strategies for DQNs.

One of the anonymous reviewers raised concerns about (12) regarding its demanding requirements on the initial policy. We would like to clarify that (12) primarily concerns the optimization analysis of the objective function rather than the initial policy. Instead, we impose only a minor assumption on the initial policy, with no specific environmental constraints. In our work, C 0 quantifies the initial policy's difference from the optimal policy and is independent of (12) in our primary theoretical results. With a sufficiently large replay buffer, C 0 can approach one, except when it equals 1, indicating an extreme divergence from the optimal policy in all states. Thus, our initial policy assumption is minimal. Considering the highly non-convex nature of deep neural network objective functions with countless local minima, (12) represents the state-of-the-art assumption for optimizing deep neural networks.

As mentioned by the anonymous reviewers, the selections of ε t in (17) depends on e t = ∥ W ( t ) -W ⋆ ∥ 2 , which is unknown to the agent. Here, we would like to clarify that e t can be replaced by its upper bound in (19) and lower bound in (20). By plugging (19) and (20) into (17), we have

As mentioned by one of the anonymous reviewers, Corollary 3 relies on an assumption that depends on the algorithm's trajectory, which lacks mathematical rigor. Here, we would like to clarify that although this equation depends on the algorithm's trajectory, it can be easily derived from a timeindependent equation

Additionally, it is worth mentioning the difference between (22) in this paper and (2) in [92]. Specifically, we want to highlight that the equation above, (27), leads to (22). In contrast, (2) in [92] requires the following condition to hold for for all W 1 and W 2 :

As a comparison, (27) only requires W 2 to be the ground truth and W 1 to be some weights near the ground truth. In other words, (27) is a sufficient condition for (22). While equation (2) in [92] does not hold with epsilon-greedy, (27) can hold with Q-learning using epsilon-greedy, thus ensuring the mathematical rigor of (22).

Acknowledgment

This work was done when Shuai was a postdoc at Rensselaer Polytechnic Institute (RPI). We thank Dr. Tianyi Chen at RPI for providing remarkable and inspiring insight into the analysis of temporal difference learning. This work was supported by AFOSR FA9550-20-1-0122, ARO W911NF-21-10255, NSF 1932196, and the Rensselaer-IBM AI Research Collaboration (http://airc.rpi.edu), part of the IBM AI Horizons Network (http://ibm.biz/AIHorizons). We thank all anonymous reviewers for their constructive comments.

References

Zeyuan Allen-Zhu and Yuanzhi Li. Feature purification: How adversarial training performs robust deep learning. In 2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS) , pages 977-988. IEEE, 2022.

Kamyar Azizzadenesheli, Emma Brunskill, and Animashree Anandkumar. Efficient exploration through bayesian deep q-networks. In 2018 Information Theory and Applications Workshop (ITA) , pages 1-9. IEEE, 2018.

Ainesh Bakshi, Rajesh Jayaram, and David P Woodruff. Learning two layer rectified neural networks in polynomial time. In Conference on Learning Theory , pages 195-268. PMLR, 2019.

Jalaj Bhandari, Daniel Russo, and Raghav Singal. A finite time analysis of temporal difference learning with linear function approximation. In Conference on learning theory , pages 16911692. PMLR, 2018.

Rajendra Bhatia. Matrix analysis , volume 169. Springer Science & Business Media, 2013.

Steven J Bradtke and Andrew G Barto. Linear least-squares algorithms for temporal difference learning. Machine learning , 22(1):33-57, 1996.

Alon Brutzkus and Amir Globerson. Globally optimal gradient descent for a convnet with gaussian inputs. In Proceedings of the 34th International Conference on Machine LearningVolume 70 , pages 605-614. JMLR. org, 2017.

Alon Brutzkus and Amir Globerson. An optimization and generalization analysis for maxpooling networks. In Uncertainty in Artificial Intelligence , pages 1650-1660. PMLR, 2021.

Alon Brutzkus, Amir Globerson, Eran Malach, and Shai Shalev-Shwartz. Sgd learns overparameterized networks that provably generalize on linearly separable data. In International Conference on Learning Representations , 2018.

Qi Cai, Zhuoran Yang, Jason D Lee, and Zhaoran Wang. Neural temporal-difference learning converges to global optima. Advances in Neural Information Processing Systems , 32, 2019.

Yichen Chen, Lihong Li, and Mengdi Wang. Scalable bilinear pi learning using state and action features. In International Conference on Machine Learning , pages 834-843. PMLR, 2018.

Yu Fan Chen, Michael Everett, Miao Liu, and Jonathan P How. Socially aware motion planning with deep reinforcement learning. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 1343-1350. IEEE, 2017.

Zaiwei Chen, Sheng Zhang, Thinh T Doan, Siva Theja Maguluri, and John-Paul Clarke. Performance of q-learning with linear function approximation: Stability and finite-time analysis. arXiv preprint arXiv:1905.11425 , page 4, 2019.

Girish Chowdhary, Miao Liu, Robert Grande, Thomas Walsh, Jonathan How, and Lawrence Carin. Off-policy reinforcement learning with gaussian processes. IEEE/CAA Journal of Automatica Sinica , 1(3):227-238, 2014.

Mohammed Nowaz Rabbani Chowdhury, Shuai Zhang, Meng Wang, Sijia Liu, and Pin-Yu Chen. Patch-level routing in mixture-of-experts is provably sample-efficient for convolutional neural networks. arXiv preprint arXiv:2306.04073 , 2023.

Antonio Coronato, Muddasar Naeem, Giuseppe De Pietro, and Giovanni Paragliola. Reinforcement learning for intelligent healthcare applications: A survey. Artificial Intelligence in Medicine , 109:101964, 2020.

Chris Dann, Yishay Mansour, Mehryar Mohri, Ayush Sekhari, and Karthik Sridharan. Guarantees for epsilon-greedy reinforcement learning with function approximation. In International Conference on Machine Learning , pages 4666-4689. PMLR, 2022.

Kefan Dong, Jian Peng, Yining Wang, and Yuan Zhou. Root-n-regret for learning in markov decision processes with function approximation and low bellman rank. In Conference on Learning Theory , pages 1554-1557. PMLR, 2020.

Kefan Dong, Jiaqi Yang, and Tengyu Ma. Provable model-based nonlinear bandit and reinforcement learning: Shelve optimism, embrace virtual curvature. Advances in Neural Information Processing Systems , 34:26168-26182, 2021.

Simon S Du, Jason D Lee, Gaurav Mahajan, and Ruosong Wang. Agnostic q -learning with function approximation in deterministic systems: Near-optimal bounds on approximation error and sample complexity. Advances in Neural Information Processing Systems , 33:2232722337, 2020.

Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. In International Conference on Learning Representations , 2018.

Simon S. Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. In International Conference on Learning Representations , 2019.

Yaqi Duan, Zeyu Jia, and Mengdi Wang. Minimax-optimal off-policy evaluation with linear function approximation. In International Conference on Machine Learning , pages 2701-2709. PMLR, 2020.

Jianqing Fan, Zhaoran Wang, Yuchen Xie, and Zhuoran Yang. A theoretical analysis of deep q-learning. In Learning for Dynamics and Control , pages 486-489. PMLR, 2020.

Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, et al. Noisy networks for exploration. arXiv preprint arXiv:1706.10295 , 2017.

Rong Ge, Jason D. Lee, and Tengyu Ma. Learning one-hidden-layer neural networks with landscape design. In International Conference on Learning Representations , 2018.

Haque Ishfaq, Qiwen Cui, Viet Nguyen, Alex Ayoub, Zhuoran Yang, Zhaoran Wang, Doina Precup, and Lin Yang. Randomized exploration in reinforcement learning with general value function approximation. In International Conference on Machine Learning , pages 4607-4616. PMLR, 2021.

Arthur Jacot, Franck Gabriel, and Cl´ ement Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems , 2018.

Xiang Ji, Minshuo Chen, Mengdi Wang, and Tuo Zhao. Sample complexity of nonparametric off-policy evaluation on low-dimensional manifolds using deep networks. arXiv preprint arXiv:2206.02887 , 2022.

Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, John Langford, and Robert E Schapire. Contextual decision processes with low bellman rank are pac-learnable. In International Conference on Machine Learning , pages 1704-1713. PMLR, 2017.

Chi Jin, Qinghua Liu, and Sobhan Miryoosefi. Bellman eluder dimension: New rich classes of rl problems, and sample-efficient algorithms. Advances in neural information processing systems , 34:13406-13418, 2021.

Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory , pages 21372143. PMLR, 2020.

Shusen Jing, Songyang Zhang, and Zhi Ding. Reinforcement learning for robust header compression under model uncertainty. arXiv preprint arXiv:2309.13291 , 2023.

Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning , pages 651-673. PMLR, 2018.

Stefani Karp, Ezra Winston, Yuanzhi Li, and Aarti Singh. Local signal adaptivity: Provable feature learning in neural networks beyond kernels. Advances in Neural Information Processing Systems , 34:24883-24897, 2021.

Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research , 32(11):1238-1274, 2013.

Petar Kormushev, Sylvain Calinon, and Darwin G Caldwell. Reinforcement learning in robotics: Applications and real-world challenges. Robotics , 2(3):122-148, 2013.

Michail G Lagoudakis and Ronald Parr. Least-squares policy iteration. The Journal of Machine Learning Research , 4:1107-1149, 2003.

Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. In International Conference on Learning Representations , 2018.

Lei Lei, Yue Tan, Kan Zheng, Shiwen Liu, Kuan Zhang, and Xuemin Shen. Deep reinforcement learning for autonomous internet of things: Model, applications and challenges. IEEE Communications Surveys & Tutorials , 22(3):1722-1760, 2020.

Gen Li, Yuting Wei, Yuejie Chi, Yuantao Gu, and Yuxin Chen. Sample complexity of asynchronous q-learning: Sharper analysis and variance reduction. Advances in neural information processing systems , 33:7031-7043, 2020.

Hongkang Li, Meng Wang, Sijia Liu, and Pin-Yu Chen. A theoretical understanding of shallow vision transformers: Learning, generalization, and sample complexity. In The Eleventh International Conference on Learning Representations , 2023.

Hongkang Li, Meng Wang, Sijia Liu, Pin-Yu Chen, and Jinjun Xiong. Generalization guarantee of training graph convolutional networks with graph topology sampling. In International Conference on Machine Learning , pages 13014-13051. PMLR, 2022.

Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient descent on structured data. In Advances in Neural Information Processing Systems , pages 8157-8166, 2018.

Yuanzhi Li, Tengyu Ma, and Hongyang R Zhang. Learning over-parametrized two-layer neural networks beyond ntk. In Conference on learning theory , pages 2613-2682. PMLR, 2020.

Fanghui Liu, Luca Viano, and Volkan Cevher. Understanding deep neural function approximation in reinforcement learning via ε -greedy exploration. arXiv preprint arXiv:2209.07376 , 2022.

Hamid R Maei, Csaba Szepesv´ ari, Shalabh Bhatnagar, and Richard S Sutton. Toward offpolicy learning control with function approximation. In Proceedings of the 27th International Conference on Machine Learning (ICML-10) , pages 719-726, 2010.

Francisco S Melo and M Isabel Ribeiro. Q-learning with linear function approximation. In International Conference on Computational Learning Theory , pages 308-322. Springer, 2007.

A Yu Mitrophanov. Sensitivity and convergence of uniformly ergodic markov chains. Journal of Applied Probability , 42(4):1003-1014, 2005.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 , 2013.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature , 518(7540):529-533, 2015.

Aditya Modi, Nan Jiang, Ambuj Tewari, and Satinder Singh. Sample complexity of reinforcement learning using linearly combined model ensembles. In International Conference on Artificial Intelligence and Statistics , pages 2010-2020. PMLR, 2020.

Thanh Nguyen-Tang, Sunil Gupta, A Tuan Nguyen, and Svetha Venkatesh. Offline neural contextual bandits: Pessimism, optimization and generalization. In International Conference on Learning Representations , 2022.

Thanh Nguyen-Tang, Sunil Gupta, Hung Tran-The, and Svetha Venkatesh. On sample complexity of offline reinforcement learning with deep reLU networks in besov spaces. Transactions on Machine Learning Research , 2022.

Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems , 29, 2016.

Theodore Perkins and Doina Precup. A convergent form of approximate policy iteration. Advances in neural information processing systems , 15, 2002.

Boris T Polyak. Introduction to optimization. New York: Optimization Software, Inc , 1987.

Daniel Russo and Benjamin Van Roy. Eluder dimension and the sample complexity of optimistic exploration. Advances in Neural Information Processing Systems , 26, 2013.

Itay Safran and Ohad Shamir. Spurious local minima are common in two-layer relu neural networks. In International Conference on Machine Learning , pages 4430-4438, 2018.

Wilko Schwarting, Javier Alonso-Mora, and Daniela Rus. Planning and decision-making for autonomous vehicles. Annual Review of Control, Robotics, and Autonomous Systems , 1:187210, 2018.

Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint arXiv:1610.03295 , 2016.

Zhenmei Shi, Junyi Wei, and Yingyu Liang. A theoretical analysis on feature learning in neural networks: Emergence from inputs and advantage over fixed features. In International Conference on Learning Representations , 2022.

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature , 529(7587):484-489, 2016.

Mahdi Soltanolkotabi, Adel Javanmard, and Jason D Lee. Theoretical insights into the optimization landscape of over-parameterized shallow neural networks. IEEE Transactions on Information Theory , 65(2):742-769, 2018.

Zhao Song, Ronald E Parr, Xuejun Liao, and Lawrence Carin. Linear feature encoding for reinforcement learning. Advances in neural information processing systems , 29, 2016.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Richard S Sutton, Hamid Maei, and Csaba Szepesv´ ari. A convergent o ( n ) temporal-difference algorithm for off-policy learning with linear function approximation. Advances in neural information processing systems , 21, 2008.

Taiji Suzuki. Adaptivity of deep reLU network for learning in besov and mixed smooth besov spaces: optimal rate and curse of dimensionality. In International Conference on Learning Representations , 2019.

Manel Tagorti and Bruno Scherrer. On the rate of convergence and error bounds for lstd ( λ ). In International Conference on Machine Learning , pages 1521-1529. PMLR, 2015.

Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference on Learning Representations , 2018.

William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika , 25(3-4):285-294, 1933.

Joel A Tropp. User-friendly tail bounds for sums of random matrices. Foundations of computational mathematics , 12(4):389-434, 2012.

Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence , 2016.

Roman Vershynin. Introduction to the non-asymptotic analysis of random matrices. arXiv preprint arXiv:1011.3027 , 2010.

Gang Wang, Georgios B Giannakis, and Jie Chen. Learning relu networks on linearly separable data: Algorithm, optimality, and generalization. IEEE Transactions on Signal Processing , 67(9):2357-2370, 2019.

Yulong Wang, Haoxin Zhang, and Guangwei Zhang. cpso-cnn: An efficient pso-based algorithm for fine-tuning hyper-parameters of convolutional neural networks. Swarm and Evolutionary Computation , 49:114-123, 2019.

Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning , 8(3):279-292, 1992.

Zixin Wen and Yuanzhi Li. Toward understanding the feature learning process of selfsupervised contrastive learning. In International Conference on Machine Learning , pages 11112-11122. PMLR, 2021.

Pan Xu and Quanquan Gu. A finite-time analysis of q-learning with neural network function approximation. In International Conference on Machine Learning , pages 10555-10565. PMLR, 2020.

Lin Yang and Mengdi Wang. Sample-optimal parametric q-learning using linearly additive features. In International Conference on Machine Learning , pages 6995-7004. PMLR, 2019.

Zhuoran Yang, Chi Jin, Zhaoran Wang, Mengdi Wang, and Michael I Jordan. On function approximation in reinforcement learning: optimism in the face of large state spaces. In Proceedings of the 34th International Conference on Neural Information Processing Systems , pages 13903-13916, 2020.

Ming Yin, Yu Bai, and Yu-Xiang Wang. Near-optimal provable uniform convergence in offline policy evaluation for reinforcement learning. In International Conference on Artificial Intelligence and Statistics , pages 1567-1575. PMLR, 2021.

Andrea Zanette, Alessandro Lazaric, Mykel Kochenderfer, and Emma Brunskill. Learning near optimal policies with low inherent bellman error. In International Conference on Machine Learning , pages 10978-10989. PMLR, 2020.

Shuai Zhang, Meng Wang, Pin-Yu Chen, Sijia Liu, Songtao Lu, and Miao Liu. Joint edgemodel sparse learning is provably efficient for graph neural networks. In The Eleventh International Conference on Learning Representations , 2023.

Shuai Zhang, Meng Wang, Sijia Liu, Pin-Yu Chen, and Jinjun Xiong. Fast learning of graph neural networks with guaranteed generalizability:one-hidden-layer case. In 2020 International Conference on Machine Learning (ICML) , 2020.

Shuai Zhang, Meng Wang, Sijia Liu, Pin-Yu Chen, and Jinjun Xiong. Why lottery ticket wins? a theoretical perspective of sample complexity on pruned neural networks. In Thirtyfifth Conference on Neural Information Processing Systems (NeurIPS) , 2021.

Shuai Zhang, Meng Wang, Sijia Liu, Pin-Yu Chen, and Jinjun Xiong. How unlabeled data improve generalization in self-training? a one-hidden-layer theoretical analysis. In International Conference on Learning Representations , 2022.

Shuai Zhang, Meng Wang, Jinjun Xiong, Sijia Liu, and Pin-Yu Chen. Improved linear convergence of training cnns with generalizability guarantees: A one-hidden-layer case. IEEE Transactions on Neural Networks and Learning Systems , 2020.

Yuchen Zhang, Jason D. Lee, and Michael I. Jordan. L1-regularized neural networks are improperly learnable in polynomial time. In Proceedings of The 33rd International Conference on Machine Learning , volume 48, pages 993-1001, 2016.

Kai Zhong, Zhao Song, Prateek Jain, Peter L Bartlett, and Inderjit S Dhillon. Recovery guarantees for one-hidden-layer neural networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pages 4140-4149. JMLR. org, https://arxiv.org/abs/1706.03175, 2017.

Dongruo Zhou, Jiafan He, and Quanquan Gu. Provably efficient reinforcement learning for discounted mdps with feature mapping. In International Conference on Machine Learning , pages 12793-12802. PMLR, 2021.

Shaofeng Zou, Tengyu Xu, and Yingbin Liang. Finite-sample analysis for sarsa with linear function approximation. Advances in neural information processing systems , 32, 2019.

Supplementary Materials for:

On the Convergence and Sample Complexity Analysis of Deep Q-Network with Epsilon-Greedy Exploration

The structure of the appendix mainly follows the roadmap of the proof described in Section 4.4.

In Appendix A, we define the characterizable population risk function in (31) to approximate the objective function. Also, some notations to simplify the analysis are introduced in Appendix A, and we recommend the readers to refer to Table 3 for the major notations used in the proofs.

In Appendix B, we provide the proof for Lemma 1 and Theorem 1 following the steps as (1) Characterization of the local convex region of population risk function (Lemma 2), (2) Characterization of the distance between the population risk function and the objective function (Lemma 3), (3) Characterization of the convergence of two consecutive iterations W ( t,m +1) and W ( t,m ) , and (4) Mathematical induction over the t and m to obtain the error bound between the convergent point W ( T, 0) and the desired point W ⋆ .

In Appendix C, we provide the preliminary lemmas and the whole proof for Lemma 2, which characterizes the local convex region of the non-convex population risk function.

In Appendix D, we provide the preliminary lemmas and the whole proof for Lemma 3, which characterizes the difference of g t and the gradient descent of defined population risk function in (31).

In Appendix E, we provide the proofs for the preliminary lemmas in proving Lemmas 2 and 3.

Before moving to the details, we provide an overview of the techniques in the proofs.

(P1.) The local convex region near W ⋆ . To characterize the local convex region, we first bound the Hessian matrix of the defined population risk function in (31) at W ∗ . Then, we derive the changes in the Hessian matrix when the neuron weights move around the W ∗ . Specifically, we prove that when neuron weights W are not far away W ⋆ , then the Hessian matrix in this region is always positive-definite, indicating that a local convex region near W ∗ . [90] considers the onehidden-layer neural network, and the lower bound of the Hessian matrix only holds for Gaussian input. Instead, in this paper, we consider multi-layer cases and need to derive a lower bound for the Hessian matrix for all the layers. Instead, the input of the intermediate layer cannot be proved to be Gaussian but belong to sub-Gaussian distribution. Therefore, we built the proof for the lower bound of the Hessian matrix when the input belongs to the sub-Gaussian distribution. Compared with Gaussian input, Sub-Gaussian does not have a closed form of the probability density function. Instead of directly calculating the lower bound, we convert the problem into proving a series of functions are linearly independent over a Hilbert space (see Lemma 7 and the proof in Appendix E). Instead of directly calculating the distance of the population risk function in different points, we characterize a Gaussian variable such that the distance over the sub-Gaussian distribution can be upper bounded by the one over the Gaussian variable (see Lemma 6 and the proof in Appendix E).

(P2.) The difference between the gradient g t and the population risk function. With the local convex region of the population risk function, we can characterize the convergence of the population risk function. With Lemma 3, we can prove that the distance between the population risk function and g t is small enough, the behaviors of the iterations via g t can be described by the ones in the population risk function with some additional error terms. Compared with the proof in [90], We need to address the extension from supervised learning settings to Q learning settings and the extension from the one-hidden-layer neural networks to the multi-layer neural networks. First, similar to challenges in (P.1), we provide a new concentration bound to characterize the distance between the two functions for the intermediate layers (see I 1 in the proof of Lemma 3). Second, the distance between the two functions has an additional error term due to the inconsistency of the label defined in (31) and (8) (see I 2 in the proof of Lemma 3). Third, we need to develop a new concentration bound to characterize the error term caused by the distribution shift when training samples are collected by ε -greedy policy (see I 3 in the proof of Lemma 3).

(P3.) The convergence analysis of Algorithm 1. When the initialization is not far away from W ⋆ , the initialization lies in the local convex region of W ⋆ for the population risk function. When we have enough samples N and a large enough ε t , we can guarantee that the distance between the g t and the gradient of the population risk function is small enough such that the iterations following g t converges to a point nearby W ⋆ as well. However, if ε t is too large, the convergent point nearby W ⋆ can be even worse than the initial point. To avoid this issue, we have an upper bound for selecting ε t , and the upper bound decreases as ∥ W ( t, 0) -W ⋆ ∥ decreases over t . Therefore, we build the convergence analysis of Algorithm 1.

A Definitions and Notations

In this section, we implement the details of algorithms described in Algorithm 1, and some important notations are defined to simplify the presentation of the proof.

A.1 Definition of the Empirical Risk Function and Its Corresponding Notations

Recall that the goal of Q -learning is to find the Q ⋆ -function to minimize (6). Therefore, we have

Since W ⋆ is the global minimal to (6), we have

Therefore, the population risk function is defined as

where µ ∗ is the distribution of the sampled data following the optimal policy π ⋆ .

The gradient of the (31) is

As W ⋆ is one of the ground truths to f ( W ) , i.e., f ( W ⋆ ) achieves the minimum value as f ( W ⋆ ) = 0 ≤ f ( W ) for any other W . Given f is a smooth function, we have the gradient of f with respect to any W ℓ at the ground truth W ⋆ equals to zero, namely,

In addition, without special descriptions, α = [ α ⊤ 1 , α ⊤ 2 , · · · , α ⊤ K ] ⊤ stands for any unit vector that in R K ℓ K ℓ -1 with α j ∈ R K ℓ -1 ( K 0 = d ). Therefore, we have

A.2 Notations in Algorithm 1

Recall that the gradient in the t -th loop is

Then, we define g ( m ) t ( W ℓ ; W ) as the components of g ( m ) t ( W ) with respect to W ℓ . Recall that in (4) we have

Then, with the definition of g ( m ) t ( W ℓ ; W ) , we have

To simplify the analysis, the update of W ( t,m ) is analyzed in the form of

One can see that (38) returns the same W ( t,m +1) as the gradient step at line 9 in Algorithm 1.

A.3 Notations for the Deep Neural Networks.

Let n denote the dimension of W defined in (4). We denote n l as the dimension of the vectorized neuron weights in the ℓ -th layer, namely, n ℓ = dim(vec( W ℓ )).

Then, let h ( ℓ ) ( W ) denote the input in the ℓ -th layer (or the output in the ( ℓ -1) -th layer) with respect the neuron weights as W , and h (1) = ( s , a ) , where

h ( ℓ ) ( W ) may be shortened as h ( ℓ ) when the neuron weights are clear from the contexts. Then, we denote the dimension of h ( ℓ ) as K ℓ , where

Then, Q ( W ; s , a ) can be written as

where w ℓ,k denotes the k -th neuron weights in the ℓ -th layer. Then, we define a group of functions J ℓ ( W ) ∈ R n -→ R K such that

Then, the gradient of Q can be represented as

where J ℓ,k stands for the k -th component of J ℓ .

A.4 Notations for Order-wise Analysis

Without loss of generality, we consider the case that d ≫ K . If K ≫ d , we can always switch the order of K and d in the proof. Let σ i ( L ) denote the i -th largest singular value of W ⋆ L . In this paper, we consider the case that W ⋆ L is will-conditioned and bounded, i.e., σ 1 ( L ) and σ 1 ( L ) /σ K ( L ) can be viewed as the constant and will be ignored in the analysis. In addition, some constant numbers will be ignored in most steps. In particular, we use h 1 ( z ) ≳ ( or ≲ , ≂ ) h 2 ( z ) to denote there exists some positive constant C such that h 1 ( z ) ≥ ( or ≤ , =) C · h 2 ( z ) when z ∈ R is sufficiently large.

B Proof of Lemma 1 and Theorem 1

The main idea in proving Theorem 1 is to characterize the gradient descent term by the Mean Value Theorem (MVT) in Lemma 4 as shown in (47) and (48). The MVT is not directly applied in g t because it is not smooth. However, the population risk functions defined in (31), which are the expectations over random variables, are smooth. Lemma 2 characterizes the bounds of the Hessian matrix defined in (49). Lemma 3 characterizes the bounds of gradient differences between the population risk function defined in (31) and g t in (7) as shown in (60). Furthermore, according to Lemma 3, we know that the distance ∥∇ ℓ f ( W ) - ∇ ℓ f ( W ∗ ) ∥ 2 is upper bounded in the order of ∥ W -W ∗ ∥ 2 as shown in (60). Then, we can establish the connection between ∥ W ( t,m +1) -W ∗ ∥ 2 and ∥ W ( t,m ) -W ∗ ∥ 2 as shown in (59). Then, by mathematical induction over m , one can characterize the iteration of {∥ W ( t, 0) -W ∗ ∥ 2 } T t =1 as shown in (65), which completes the proof of Lemma 1. Finally, selecting ε t based on (68) for all t ∈ [ T ] , we derive the error bound of ∥ W ( T, 0) -W ⋆ ∥ 2 by mathematical induction over t , which completes the proof of Theorem 1.

Lemma 2. Given any W ∈ R n , let W satisfy

for some constant c I ∈ (0 , 1) . Then, for the f defined in (31) , we have

Lemma 3. Let f be the function defined in (31) . Let g t be the function defined in (7) . Then, we have

with probability at least 1 -q -K ℓ .

Lemma 4 (Mean Value Theorem) . Let U ⊂ R d 1 be open and f : U -→ R d 2 be continuously differentiable, and x ∈ U , h ∈ R d 1 vectors such that the line segment x + t h , 0 ≤ t ≤ 1 remains in U . Then we have:

where ∇ f denotes the Jacobian matrix of f .

Proof of Theorem 1. Let W ℓ denote the neuron weights in the ℓ -th layer. From Algorithm 1 and (38), in the s -th iteration and t -th episode, we have

From (31), we can see that W ⋆ is the global optimal to f because f ( W ⋆ ) achieves the minimum value as 0 . Therefore, we have ∇ ℓ f t ( W ⋆ ) = 0 . Since ∇ ℓ f is a smooth function W ⋆ , from the Mean Value Theorem in Lemma 4, we have

For notational convenience, we use H to denote the integration as

Then, we have

Let H = S Λ S T be the eigen-decomposition of H . Then, we define

eigenvalues. Let λ ( Λ ) i be the i -th eigenvalue of H ( ℓ ) t , then the corresponding i -th eigenvalue of (51), denoted by λ ( A ) i , satisfies

By simple calculation, we have

Specifically, we have

and λ ( A ) i achieves the minimum λ ( A ) ⋆ i = ∣ ∣ ∣ 1 -√ ηλ ( Λ ) i ∣ ∣ ∣ when β ⋆ = ( 1 -√ ηλ ( Λ ) i ) 2 . From Lemma 2, for any a ∈ R d with ∥ a ∥ 2 = 1 , we have

where λ max ≂ 1 K , and λ min ≂ ρ K 2 . Therefore, we have

Thus, when η ≤ 1 2 λ ( Λ ) max ≲ K , ∥ A ( β ⋆ ) ∥ 2 can be bounded by

Therefore, we have

Take the sum of (58) from ℓ = 1 to ℓ = L , we have

For some small constant c N ≥ 0 , let

which requires

Then, the sample complexity

Therefore, we have

By mathematical induction, when M = log γ -1 and η = 1 /T = 1 / Θ( N ) , we have

From Algorithm 1, we know that W ( t +1 , 0) = W ( t,M ) . To guarantee that iteration converge to the ground truth W ⋆ , namely, ∥ W ( t +1 , 0) -W ⋆ ∥ 2 < ∥ W ( t, 0) -W ⋆ ∥ , we need

To guarantee that ε T ≥ 0 , then we have

we have

which completes the proof.

C Proof of Lemma 2

Lemma 2 provides the lower and upper bounds for the eigenvalues of the Hessian matrix of population risk function in (31). According to Weyl's inequality in Lemma 5, the eigenvalues of ∇ 2 ℓ f ( · ) at any fixed point W can be bounded in the form of (75). Therefore, we first provide the lower and upper bounds for ∇ 2 ℓ f at the desired ground truth W ⋆ . Then, the bounds for ∇ 2 ℓ f at any other point W is bounded through (31) by utilizing the conclusion in Lemma 6. Lemma 6 illustrates the distance between the Hessian matrix of f at W and W ∗ . Lemma 7 provides the lower bound of E x ( ∑ K j =1 α ⊤ j ∂Q ∂ w ℓ,k ( W ⋆ ) ) 2 when x belongs to sub-Gaussian distribution, which is used in proving the lower bound of the Hessian matrix in (76).

Specifically, let

Lemma 5 (Weyl's inequality, [5]) . Let B = A + E be a matrix with dimension m × m . Let λ i ( B ) and λ i ( A ) be the i -th largest eigenvalues of B and A , respectively. Then, we have

Lemma 6. Let f ( W ) be the population risk function defined in (31) . If W is close to W ⋆ such that we have

Lemma 7. Suppose the following assumptions hold:

1. { w j } K j =1 ∈ R K ℓ are linear independent,

2. p H ( h ) : R K ℓ -→ [0 1] be the probability density for h such that E h ∥ h ∥ 2 2 ≤ + ∞ .

Let α ∈ R K 1 K 2 be the unit vector defined in (34) , we have

where R⊂ R K ℓ with ∫ R f H ( h ) > 0 . Moreover, if further assuming P is Gaussian distribution and R = R K ℓ , we have ρ > 0 . 091 .

Lemma 8. Let h ( ℓ ) ( W ) be the function defined in (39) . When W is sufficiently close to W ⋆ , i.e., ∥ W -W ⋆ ∥ 2 is smaller than some positive constant c < 1 , we have

Proof of Lemma 2. Let λ max ( W ) and λ min ( W ) denote the largest and smallest eigenvalues of ∇ 2 ℓ f ( W ) at a point W , respectively. Then, from Lemma 5, we have

Then, we provide the lower bound of the Hessian matrix of the population function at W ⋆ . Let P be the distribution for h ( ℓ ) ( W ) when x ∼ µ t with probability density function denoted as p H . For any α ∈ R K ℓ K defined in (34) with ∥ α ∥ 2 = 1 , we have

̸

where the last inequality comes from Lemma 7, and Lemma 7 holds since h ( ℓ ) belongs to subGaussian distribution and W ℓ is full rank.

Next, the upper bound of ∇ 2 ℓ f can be bounded as

(77)

Then, given (71), we have

Therefore, we have

Combining (79) and Lemma 6, we have

Therefore, from (80) and (75), we have

which completes the proof.

D Proof of Lemma 3

Before illustrating the whole proof, we first introduce some preliminary lemmas and definitions. Lemma 9 is the concentration theorem for independent random matrices. The definitions of the sub-Gaussian and sub-exponential variables are summarized in Definitions 3 and 4, and it is easy to verify that any bounded variables belong to sub-Gaussian distribution. Lemmas 10 and 11 serve as the technical tools in bounding matrix norms under the framework of the confidence interval.

The error bound between ∥∇ ℓ f -g t ∥ 2 is divided into bounding I 1 , I 2 , and I 3 as shown in (91). I 1 in (92) represent the deviation of the mean of several random variables to their expectation, which can be bounded through concentration inequality, i.e, Chernoff bound. I 2 in (93) come from the inconsistency of 'noisy' label in (8) and the 'ground truth' label in the population risk function (31). I 3 in (94) come from the data distribution shift defined in Definition 1.

Lemma 9 ([72], Theorem 1.6) . Consider a finite sequence { Z k } of independent, random matrices with dimensions d 1 × d 2 . Assume that such a random matrix satisfies

Define

E ( Z k ) = 0 and ∥ Z k ∥ ≤ R almost surely .

Then for all t ≥ 0 , we have

Definition 3 (Definition 5.7, [74]) . A random variable X is called a sub-Gaussian random variable if it satisfies

for all p ≥ 1 and some constant c 1 > 0 . In addition, we have

for all s ∈ R and some constant c 2 > 0 , where ∥ X ∥ ϕ 2 is the sub-Gaussian norm of X defined as ∥ X ∥ ψ 2 = sup p ≥ 1 p -1 / 2 ( E | X | p ) 1 /p .

Moreover, a random vector X ∈ R d belongs to the sub-Gaussian distribution if one-dimensional marginal α ⊤ X is sub-Gaussian for any α ∈ R d , and the sub-Gaussian norm of X is defined as ∥ X ∥ ψ 2 = sup ∥ α ∥ 2 =1 ∥ α ⊤ X ∥ ψ 2 .

Definition 4 (Definition 5.13, [74]) . A random variable X is called a sub-exponential random variable if it satisfies

for all p ≥ 1 and some constant c 3 > 0 . In addition, we have

for s ≤ 1 / ∥ X ∥ ψ 1 and some constant c 4 > 0 , where ∥ X ∥ ψ 1 is the sub-exponential norm of X defined as ∥ X ∥ ψ 1 = sup p ≥ 1 p -1 ( E | X | p ) 1 /p .

Lemma 10 (Lemma 5.2, [74]) . Let B (0 , 1) ∈ { α ∣ ∣ ∥ α ∥ 2 = 1 , α ∈ R d } denote a unit ball in R d . Then, a subset S ξ is called a ξ -net of B (0 , 1) if every point z ∈ B (0 , 1) can be approximated to within ξ by some point α ∈ B (0 , 1) , i.e., ∥ z -α ∥ 2 ≤ ξ . Then the minimal cardinality of a ξ -net S ξ satisfies

Lemma 11 (Lemma 5.3, [74]) . Let A be an d 1 × d 2 matrix, and let S ξ ( d ) be a ξ -net of B (0 , 1) in R d for some ξ ∈ (0 , 1) . Then

Proof of Lemma 3. From (7), we know that

From (31), we know that

Then, from (88) and (89), we have

where D t and µ t are equivalent because of Assumption 2. Then, we have

For convenience, we define I 1 , I 2 , and I 3 in the following ways with x n := ( s n , a n ) be the feature mapping of state-action pair ( s n , a n ) .

Then, I 1 is defined as

I 2 is defined as

and I 3 is defined as

where

from (43). Therefore, we have

Next, we will provide the bound for ∥ I 1 ∥ 2 , ∥ I 2 ∥ 2 , and ∥ I 3 ∥ 2 .

Bound of I 1 . We first divide the data in D t into two parts, namely, D t, 1 and D t, 2 . D t, 1 includes the state-action pair ( s , a ) such that a n is randomly selected from action space A , and D t, 2 includes the state-action pair ( s , a ) such that a n is selected based on the greedy policy with respect to Q ( W ( t, 0) ) .

Then, we define a random variable Z ( ℓ, 1) = ( Q ( x ; W ) -Q ( x ; W ⋆ ) ) · J ℓ,k · α T h ( ℓ ) ( W ) with x ∼ D t, 1 and Z ( ℓ, 1) n = ( Q ( x n ; W ) -Q ( x n ; W ⋆ ) ) · J ℓ,k · α T h ( ℓ ) n ( W ) as the realization of Z (1) ℓ for n = 1 , 2 · · · , N , where α ∈ R d is any fixed unit vector with ∥ α ∥ 2 ≤ 1 . We know that s and a are independent for x ∼ D t, 1 . Let Σ 1 denote the covariance matrix of x ∼ D t, 1 . Moreover, x ( s , a ) is bounded by 1 , then we have ∥ Σ 1 ∥ 2 ≤ 1 .

Similar to Z ( ℓ, 1) , we define a random variable Z ( ℓ, 2) = ( Q ( x ; W ) -Q ( x ; W ⋆ ) ) ·J ℓ,k · α T h ( ℓ ) ( W ) with x ∼ D t, 2 and Z ( ℓ, 2) n = ( Q ( x n ; W ) -Q ( x n ; W ⋆ ) ) · J ℓ,k · α T h ( ℓ ) n ( W ) as the realization of Z ( ℓ, 2) for n = 1 , 2 · · · , N . Differ from Z ( ℓ, 1) , s and a are dependent for x ∼ D t, 2 . Let Σ 2 denote the covariance matrix of x ∼ D t, 1 . Then, we have ∥ Σ 2 ∥ 2 ≤ 1 + max j ρ x j ,a ≤ 2 , where ρ x j ,a denotes the correlation between a and x j .

According to the definition of (92), we can rewrite I 1 as

Then, for any p ∈ N + , we have

where C 1 is a positive constant.

From Definition 4, we know that Z ( ℓ, 1) belongs to sub-exponential distribution with ∥ Z ( ℓ, 1) ∥ ψ 1 ≤ C 1 ∥ W -W ⋆ ∥ 2 . Therefore, by Chernoff inequality, we have

for some positive constant C and any s ∈ R .

Let t = C 1 ∥ W -W ⋆ ∥ 2 √ d log q N and s = 2 C ∥ W -W ⋆ ∥ 2 · t for some large constant q > 0 . Then, we have

with probability at least 1 -q .

Similar to (98), we have

where C 2 = 2 · C 1 . Then, we have

with probability at least 1 -q -d .

From Lemma 11 and (97), we have with probability at least

From Lemma 10, we know that |S 1 2 ( d ) | ≤ 5 d . Therefore, the probability for (103) holds is at least 1 -( q 5 ) -d . Because q ≫ 5 , we denote the probability as 1 -q -d for convenience.

Then, we have

Similarly to (105), let us define ˜ a ⋆ n = arg max a Q ( W ; s n , a ) . Then, we have

Combining (105) and (106), we have

Following the definition of Z in (98), we define

Therefore, from (105) and (106), we know

Following the steps in (98) to (100), we have

with probability at least 1 -q -d , where the last inequality holds when N ≳ d · log q · log |A| . Bound of I 3 . We have

Then, we have

Then, we have

Therefore, the bound of I 3 can be found as

where C d = |A| .

In conclusion, let α ∈ R Kd and α j ∈ R d with α = [ α T 1 , α T 2 , · · · , α T K ] T , we have

with probability at least 1 -q -d .

E Additional proof of the lemmas in Appendix C

E.1 Proof of Lemma 6

The distance of the second order derivatives of the population risk function f ( · ) at point W and W ⋆ can be converted into bounding P 1 , P 2 , which are defined in (116). The major idea in proving P 1 is to connect the error bound to the angle between W and W ⋆ given h ( ℓ ) belongs to the sub-Gaussian distribution.

Proof of Lemma 6. From the definition of f in (31), we have

where h = h ( ℓ ) ( W ) and h ⋆ = h ( ℓ ) ( W ⋆ ) .

Then, we have

For any a ∈ R K ℓ with ∥ a ∥ 2 = 1 , we have

Then, we have

Utilizing the Gram-Schmidt process, we can demonstrate the existence of a set of normalized orthonormal vectors denoted as B = { a , b , c , a ⊥ 4 , · · · , a ⊥ d } ∈ R d . This set forms an orthogonal and normalized basis for R d , wherein the subspace spanned by a , b , c includes a , w ℓ,j 2 , and w ∗ ℓ,j 2 . Then, for any x ∈ R d , we have a unique z = [ z 1 , z 2 , · · · , z d ] ⊤ such that

Because (i) a , w ℓ,j 2 , and w ∗ ℓ,j 2 belongs to the subspace spanned by vectors { a , b , c } and (ii) a ⊥ 4 , · · · , a ⊥ d , · · · are orthogonal to a , b , and c . Then, we know that

where ˜ h = z 1 a + z 2 b + z 3 c . Similar to (119), we have w ⊤ ℓ,j 2 h = w ⊤ ℓ,j 2 ˜ h and a ⊤ h = a ⊤ ˜ h .

Then, we define I 4 as

where | J h ( z ) | is the determinant of the Jacobian matrix ∂ h ∂ z . Since z is a representation of h based on an orthogonal and normalized basis, we have | J h ( z ) | = 1 . According to (119), I 4 can be rewritten as

where in the last equality we abuse f Z ( z 1 , z 2 , z 3 ) to represent the probability density function of ( z 1 , z 2 , z 3 ) defined in region R z .

Next, we show that z is rotational invariant over R z . Let R = [ a b c · · · a ⊥ d ] , we have h = Rz . For any z (1) and z (2) with ∥ z (1) ∥ 2 = ∥ z (2) ∥ 2 . We define h (1) = R z (1) and h (2) = R z (2) . Since x is rotational invariant and ∥ h (1) ∥ 2 = ∥ h (2) ∥ 2 = ∥ z (1) ∥ 2 = ∥ z (2) ∥ 2 , then we know h (1) and h (2) has the same distribution density. Then, z (1) and z (2) has the same distribution density as well. Therefore, z is rotational invariant over R z .

Then, we consider spherical coordinates with z 1 = Rcosϕ 1 , z 2 = Rsinϕ 1 sinϕ 2 , z 3 = Rsinϕ 1 cosϕ 2 . Hence, we have

Since z is rotational invariant, we have that

Then, we have

where the first equality holds because ϕ ′ ( w ⊤ i,,j 2 h ) only depends on the direction of h , and ¯ x := h /R = (cos ϕ 1 , sin ϕ 1 sin ϕ 2 , sin ϕ 1 cos ϕ 2 ) in the last inequality.

Because z belongs to the sub-Gaussian distribution, we have F z ( R ) ≥ 1 -2 e -R 2 σ 2 for some constant σ > 0 . Then, the integration of R can be represented as

where the last inequality comes from the calculation that

Then, we define ˜ x ∈ R K ℓ belongs to Gaussian distribution as ˜ x ∼ N ( 0 , σ 2 I ) . Therefore, we have

where ˜ x belongs to Gaussian distribution.

Therefore, the inequality bound over a sub-Gaussian distribution is bounded by the one over a Gaussian distribution. In the following contexts, we provide the upper bound of E ˜ x ∣ ∣ ϕ ′ ( w ⊤ ℓ,j 2 ˜ x ) -ϕ ′ ( w ⋆T ℓ,j 2 ˜ x ) | .

Define a set A 1 = { x | ( w ⋆ ⊤ ℓ,j 2 ˜ x )( w ⊤ ℓ,j 2 ˜ x ) < 0 } . If ˜ x ∈ A 1 , then w ⋆ ⊤ ℓ,j 2 ˜ x and w ⊤ ℓ,j 2 ˜ x have different signs, which means the value of ϕ ′ ( w ⊤ ℓ,j 2 ˜ x ) and ϕ ′ ( w ⋆ ⊤ ℓ,j 2 ˜ x ) are different. This is equivalent to say that

Moreover, if ˜ x ∈ A 1 , then we have

Let us define a set A 2 such that

Hence, we have that

Since ˜ x ∼ N ( 0 , ∥ a ∥ 2 2 I ) , θ ˜ x , w ⋆ ℓ,j 2 belongs to the uniform distribution on [ -π, π ] , we have

Hence, (124) and (132) suggest that

The same bound that is shown in (133) holds for P 2 as well.

Therefore, we have

where α ∈ R Kd and α j ∈ R K ℓ with α = [ α ⊤ 1 , α ⊤ 2 , · · · , α ⊤ K ] ⊤ .

E.2 Proof of Lemma 7

̸

We aim to prove that ∫ R ( ∑ K j =1 α ⊤ h ϕ ′ ( w ⊤ ℓ,j h ) ) 2 p H ( h ) · d h is strictly greater than zero for any α . Therefore, the ρ in (2) is strictly greater than zero. The proof is inspired by Theorem 3.1 in [22]. It is obviously that ( ∑ K j =1 α ⊤ h ϕ ′ ( w ⊤ ℓ,j h ) ) 2 is greater or equal to zero. Given ( ∑ K j =1 α ⊤ h ϕ ′ ( w ⊤ ℓ,j h ) ) 2 is continuous, we only need to show that α such that ∑ K j =1 α ⊤ h ϕ ′ ( w ⊤ ℓ,j h ) = 0 for any α , namely, { h ϕ ′ ( w ⊤ ℓ,j h ) } K j =1 are linear independent. Compared with Theorem 3.1 in [22], we need to address two challenges: (1) the neuron weights w is the random variable in [22] while the input h is the random variable in this paper and (2) the random variable belongs to Gaussian distribution in [22] while the random variable belongs to sub-Gaussian distribution in this paper.

Proof of Lemma 7. Let H be a Hilbert space on R K ℓ , and the inner product of H is defined as

where the Lebesgue measure of R over R K ℓ is non-zero. Instead of directly proving ∫ R ( ∑ K k =1 α ⊤ h ϕ ′ ( w ⊤ k h ) ) 2 f H ( h ) · d h > 0 for any α , we note that it is sufficient to prove that { h ϕ ′ ( w ⊤ k h ) } k ∈ [ K ] are linear independent over the Hilbert space H . Namely, if { h ϕ ′ ( w ⊤ k h ) } k ∈ [ K ] are linear independent, we have

̸

Therefore, we can know that ∫ R ( ∑ K j =1 α ⊤ h ϕ ′ ( w ⊤ ℓ,j h ) ) 2 p H ( h ) · d h is strictly greater than zero.

Next, we provide the whole proof for that { xϕ ′ ( w ⊤ k h ) } k ∈ [ K ] are linear independent over the Hilbert space H .

We define a group of functions { ψ j ( h ) } K j =1 , where ψ j ( h ) = h ϕ ′ ( w ⊤ j h ) . From the assumption in Lemma 7, we can justify that E h ∼D | ψ j ( h ) | 2 ≤ E h ∼D | h | 2 < ∞ .

̸

̸

Let X i = { h | w ⊤ i h = 0 } for any i ∈ [ K ] . For any fixed k , we can justify that X k cannot be covered by other sets {X k } j = k as long as w k does not parallel to any other weights w j with j = k . Namely, X k ̸⊂ ∪ j = k X j . The idea of proving the claim above is that the intersection of X j and X k is only a hyperplane in X k . The union of finite many hyperplanes is not even a measurable space and thus cannot cover the original space. Formally, we provide the formal proof for this claim as follows.

̸

̸

Let λ be the Lebesgue measure on X k , then λ ( X k ) > 0 . When w j does not parallel to w k , X k ∩X j is only a hyperplane in X k for j = k . Hence, we have λ ( X j ∩ X k ) = 0 . Next, we have

̸

̸

Therefore, we have

̸

̸

Therefore, we have X k / ( ∪ j = k X j ) is not empty, which means that X k ̸⊂ ∪ j = k X j .

̸

̸

̸

Next, Since X k / ( ∪ j = k X j ) is not an empty set, there exists a point z k ∈ X k / ( ∪ j = k X j ) and r 0 > 0 such that

̸

where B ( z k , r ) stands for a ball centered at z k with a radius of r . Then, we divide B ( z k , r ) into two disjoint subsets such that

Because z k is a boundary point of { h | w ⊤ k h = 0 } , both B + r and B -r are non-empty.

̸

Note that ψ j ( h ) is continuous at any point except for the ones in X j . Then, for any j = k , we know that ϕ j ( w ⊤ k h ) is continuous at point z k since z k ̸∈ X j . Hence, it is easy to verify that

While for ψ k , we know that ψ k ( h ) ≡ 0 for h ∈ B -r , (ii) ψ k ( h ) = h for h ∈ B + r . Hence, it is easy to verify that

Now let us proof that { ψ j } K j =1 are linear independent by contradiction. Suppose { ψ j } K j =1 are linear dependent, we have

Then, we have

Then, we have

where the last equality comes from (141) and (142).

Note that z k cannot be 0 because z k ̸∈ X j . Therefore, we have α k = 0 . Similarly to (145), we can obtain that α j = 0 by define z j following the definition of z k for any j ∈ [ K ] . Then, we know that (143) holds if and only if α = 0 , which contradicts the assumption that { ψ j } K j =1 are linear dependent.

In conclusion, we know that { ψ j } K j =1 are linear independent, and ∫ R ( ∑ K j =1 α ⊤ h ϕ ′ ( w ⊤ ℓ,j h ) ) 2 p H ( h ) · d h is strictly greater than zero.

̸

E.3 Proof of Lemma 8

Proof of Lemma 8. From the definition of (39), we have

With the assumption in the Lemma 8 such that W is close enough to W ⋆ , we have

Therefore, we have

Then, we have

which completes the proof.

F Additional experiments

In this section, we provide numerical justification that our theoretical findings are aligned with DDQN through the Atari Breakout game The neural network follows the same architecture as the one used in Section 5. The algorithm terminates if the average score over the recent 100 episodes does not improve or the algorithm reaches the maximum episode set as 200 , which is around 4 × 10 5 training steps. The testing score is calculated based on a similar setup as the training process by fixing the maximum memory size N as 2000 and greedy policy, i.e., ε = 0 . Each point in the plot is averaged over 10 experiments with an error bar representing the standard deviation.

Estimation errors with respect to the sample complexity N . We follow the setup in Section 5 to use the expected cumulative reward as the estimation error of the learned model to the optimal Q-value function. The ε t in ε -greedy policy decreases geometrically from 1 to 0 . 01 . We vary the number of samples in the replay buffer from 3000 to 10000 . Figure 4 shows that the test error is almost linear in 1 / √ N , which is consistent with our characterization in (20). In addition, experiments with a large N have a shorter error bar indicating a more stable learning performance with a large sample complexity as shown in (12).

Convergence with different selections of ε . Figure 5 illustrates the convergence rate when ε t in the ε -greedy policy changes. For each point, ε 0 is selected as the value in the x-axis, and we decrease ε t geometrically as the iteration t increases. Each point is averaged over 10 independent trials. We can see that the convergence rate is a linear function of c ε , matching our findings in (19).

0.01

0.012

0.014

0.016

0.018

0.02

13.5

14

14.5

15

15.5

16

Score

G Extension to non i.i.d. samples

Assumption 3. At any fixed outer iteration t , the behavior policy π t and transition kernel P t satisfy

for some constant λ > 0 and ν ∈ (0 , 1) , where d TV denotes the total-variation distance between the probability measures.

Assumption 3 assumes the Markov chain { s t } induced by the behavior policy, i.e., ε t -greedy policy at t -th outer loop, is uniformly ergodic with the corresponding invariant measure P t . Compared with i.i.d. cases, we need to handle an additional error term when bounding the distance between the g t and ∇ f as shown in (91). Therefore, the upper bound in Lemma 3 changes, which suggests an additional term in the final bound.

We present the major theoretical findings for non-i.i.d. samples in Theorem 2. The major proofs in this context follow similar steps to the proof of Theorem 1, with slight changes in the error bound between the sequences g t and ∇ f . In this section, we omit the details of the proof for Theorem 2 but provide the proof for Lemma 3 under the assumptions outlined in Assumption 2 to simplify the presentation.

Theorem 2 (Convergence for non-i.i.d. case) . Suppose Assumption 1 and (143) hold, the buffer size N satisfies (13). Let us define C max be a constant that is larger than C t for 1 ≤ t ≤ T and C d = |A| · (1 + log ν λ -1 + 1 1 -ν ) , when ε t satisfy

for a fixed constant c ε ∈ (0 , (1 -γ ) 2 ] , and the initialization satisfies

Then, with the high probability of at least 1 -T · q -d , we have

( C1 ) The learned weights decay geometrically with

( C2 ) the returned model Q ( W ( T, 0) ) exhibits an estimation error as

where τ ⋆ = min { t | λν t ≤ 1 / ( N · T ) } .

0.01

0.02

0.03

0.04

0.05

0.976

0.978

0.98

0.982

0.984

0.986

Convergence rate

Proof of Lemma 3 under Assumption 2. Recall that in (91), we have

Bound of I 1 and I 2 . Compared with (91), the upper bound for I 1 and I 2 is the same as those shown in (103) and (109), respectively.

Bound of I 3 . Following (111), the upper bound of I 3 can be characterized as

and

From Theorem 3.1 in [49], we know that

Therefore, the bound of I 3 can be found as

where C d = |A| · (1 + log ν λ -1 + 1 1 -ν ) .

Bound of I 4 . I 4 is the bias of the data because the data ( s , a ) at iteration t depends on the neural network parameters W . Let us define ¯ g t as

and

It is easy to verify that

Then, we have

Therefore, we have

Then, we need to bound δ t ( W ( t -τ, 0) ) .

Let us define the observed tuple O t ( s , a, s ′ ) as the collection of the state, action, and the next state at the t -th outer loop. Note that

forms a Markov chain introduced by the policy π t -τ .

Let ˜ W ( t -τ, 0) and ˜ O t be independently drawn from the marginal distributions of ˜ W ( t -τ, 0) and O t , respectively.

With Lemma 9 in [4], we have

By definition, we have E ∆ t ( ˜ W ( t -τ, 0) , ˜ O t ) = 0 and

Therefore, we have

where the last inequality comes from the fact that the step size η t is non-increasing.

Choose τ ⋆ = min { t = 0 , 1 , 2 , · · · | λν τ ≤ η T } . When t ≤ τ ⋆ , we choose τ = t and have

When t > τ ⋆ , we can choose τ = τ ⋆ and obtain

Combining (169) and (170), we have

where τ ⋆ = min { t | λν t ≤ η T } .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies DQN (one of the most popular RL algorithms in the literature) with $\\epsilon$-greedy behavior policy. Specifically, the authors provide the convergence and sample complexity guarantees and numerically verify the algorithm's performance.\n\nSTRENGTHS:\nThis paper is well-written, and the appendix seems of high quality.\n\nWEAKNESSES:\nMajor Comments:\n\n(1) Corollary 1: It is unclear to me how to choose $\\epsilon_t$ based on this result as $e_t$ is precisely the error of the algorithm, which is unknown to the agent.\n\n(2) Theorem 1: The authors refer to $W^{T,0}$ as the convergent point of the algorithm, which is confusing because $W^{T,0}$ depends on the iteration number. In addition, it is unclear to me why Theorem 1 implies the almost sure convergence (or any other kind of asymptotic convergence) of the weights.\n\nIn Eq. (19), I saw $t$ on the LHS. and $T$ on the RHS. Is this a typo? The claim of geometric decay based on Eq. (19) is not entirely convincing because $\\epsilon_t$ is not a constant but an implicit function of $t$. This also impacts the claim in Corollary 2. From my understanding, Corollary 2 is derived solely based on Eq. (20) because the transient term in Eq. (19) decays at a geometric rate, thereby does not contribute to any polynomial sample complexity. \n\n(3) Corollary 3: The assumption in Eq. (22) is not completely rigorous. In general, one should not make any assumptions about the trajectory of the very algorithm that is being studied.\n\nQUESTIONS:\nWhen using linear function approximation, Q-learning can be divergent [1,2] even if the target network is used as in this paper. In addition, there are several impossibility results in the literature [3,4] stating that unless certain assumptions are made (such as Bellman completeness, or both $V^*$ and $Q^*$ belong to the function class), the sample complexity can be exponential. While this paper uses neural network approximation, it is similar to linear function approximation when there is only one layer (the difference is only about the activation function). Then, (i) why is the algorithm naturally stable? and (ii) why does this algorithm achieve polynomial sample complexity? It is important to investigate the fundamental reason behind these questions to provide a better understanding of the behavior of Q-learning with function approximation.\n\n[1] Baird, L. (1995). Residual algorithms: Reinforcement learning with function approximation. In Machine Learning Proceedings 1995 (pp. 30-37). Morgan Kaufmann.\n\n[2] Chen, Z., Clarke, J. P., & Maguluri, S. T. (2022). Target Network and Truncation Overcome The Deadly Triad in $ Q $-Learning. arXiv preprint arXiv:2203.02628.\n\n[3] Zanette, A. (2021, July). Exponential lower bounds for batch reinforcement learning: Batch rl can be exponentially harder than online rl. In International Conference on Machine Learning (pp. 12287-12297). PMLR.\n\n[4] Wang, R., Foster, D. P., & Kakade, S. M. (2020). What are the statistical limits of offline RL with linear function approximation?. arXiv preprint arXiv:2010.11895.\n\nLIMITATIONS:\nThe comments in the weakness section (especially comment (3)) should be the theoretical limitations of this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies non-asymptotic behavior of deep Q-network where the inner loop uses momentum and the behavior policy is time-varying epsilon greedy policy. With nice choice of initialization and exploring strategy $\\epsilon_t$, the authors prove that the DQN algorithm enjoys $1/\\sqrt{N}$ sample complexity.\n\nThe analysis of inner loop iteration follows the spirit of bounding the error between the gradient of population risk function and its estimation, with a novel lower bound on the Hessian of population risk function.\n\nSTRENGTHS:\n1. The paper manages to derive sample complexity deep Q-network under epsilon-greedy policy, which has been difficult due to time-varying policy. The authors tackle this problem by defining $C_t$, a quantity that measures the difference of behavior policy and optimal policy. The theoretical contribution seems to be meaningful. \n\n2. Moreover, the technique to lower bound the Hessian matrix, levereging on a series of functions that are linearly independent over a Hilbert space,  seems to be novel.\n\nWEAKNESSES:\n1. The exploration strategy $\\epsilon_t$ requires prior knowledge of optimal policy, which is not possible. Moreover, the upper and lower bound for $\\epsilon_t$ is not really intuitive.\n\n2. The work assumes that mini-batch is sampled from time-varying stationary distribution. The transition matrix induced by behavior policy at every time step is assumed to be have stationary distribution.\n\n3. The step-size depends on the number of outer loop iteartions, $T$.\n\nQUESTIONS:\n1. Please give reference or more explanation for the definition of local convex region in equation (11). It looks like norm, but what does it have to do with convexity? Why do we need to care about its lower bound?\n\n2. To prove local convexity of $f(W)$ around $W^*$, the lower bound on the hessian of $f(W)$, Lemma 6 plays a key role, but I think the details are quite omitted. The coordinate transform to bound $I_4$ from line 856~861 is not really intuitive. Is this a standard techinque or have been used in other literature? Please give more details and explanation.\n\n3. Regarding proof of Lemma 7, what is the intuition behind leveraging on a series of functions that are linearly independent over a Hilbert space?\n\n\n- Miscellaneous\n\nRegarding the footnote in page 6, which comments on the extension to non i.i.d. case, I think there is typo. It should be Appendix G not F. Moreover, what is the result on the non i.i.d. case? Mentioning about only the Lemma seems to be incomplete. Please add a full theorem or proposition about the non-i.i.d. case if to argue as in the footnote. \n\nIn Algorithm 1, how do we initialize the buffer and does it work like First-in-First-out queue or do we have to collect $N$ new samples whenever we start the inner loop?\n\n In line 713,  $M=\\log \\gamma$ does not make sense since $M$ should be positive while $\\log \\gamma$ is negative.\n\nLIMITATIONS:\nEven though, there are strict assumptions such as prior knowledge on optimal policy to select the exploration strategy $\\epsilon_t$, stationarity of Markov chain for every time step, and strict conditions on the initialization, I think the paper paved a way for analysis of DQN algorithm, and theoretical contribution is substantial. Hence, I am leaning towards acceptance.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper provides a theoretical analysis of the converge of DQN in deep reinforcement learning. Its contributions are mostly theoretical and concern the important setting of epsilon-greedy exploration DQN with general function approximation. The paper includes convergence and sample complexity analysis for DQN as well as bounds for varying the exploration epsilon. Its results extend prior work with more restrictive assumptions (e.g. no epsilon greedy exploration in Xu and Gu, 2020).

STRENGTHS:
The paper is well written and clear. The main strength of the paper is that as far as we know, this is the first theoretical work to tackle convergence of the Bellman error under both neural approximation and epsilon greedy exploration, without overly restrictive assumptions (two-layer networks etc). We appreciated for instance the fact that even as the epsilon exploration bounds are somewhat loose (as shown in empirical sections), they qualitatively explain crucial phenomena known to RL practitioners such as the necessity to anneal the exploration epsilon. The clarity of exposition also contrasts with the technicity of proofs (e.g. using tools from sub-Gaussian concentration of measure). Overall we find the theoretical contribution to be valuable and significant.

WEAKNESSES:
As far as weaknesses go, there are minor orthographic typos that remain in the main text, and should be addressed in second reading.

QUESTIONS:
Question to authors: It is not immediately obvious to me which strong intuitions we should have about the Holder exponent of quantity C_t that intervenes in Corollary 3, or if we could attempt to measure a pathwise estimate empirically. Could we detail this point further, perhaps in supplementary ?
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07

comments: SUMMARY:
This paper analyzes the sample complexity of using $\epsilon$-greedy for exploration in the setting of general function approximation.

STRENGTHS:
This paper provides an convergence analysis of using $\epsilon$-greedy in Q-learning.

WEAKNESSES:
In my opinion, this paper is studying optimization rather than sample complexity. Equation (18) is saying that the initial Q value is accurate. However, in the research of sample complexity, we are concerning that how to explore the environment even when the initial policy is nothing and we know nothing about the environment. Here we have a good initial point, which weaken the need of exploration.

QUESTIONS:
1. The definition of Theta in Equation 12 is not found in the main text.

2. Why is the requirement of the initial point in Equation 18 become higher as \rho grows bigger? What is the intuition behind this phenomenon?

LIMITATIONS:
See 'Weakness' Session
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-08