SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$C_t$  can be viewed as the difference between behavior policy and optimal policy. Theorem 1 shows the general results for any value of  $C_t$ . Nevertheless, the greedy policy is improved over time, e.g., updating the weights every few steps (line 11 in Algorithm 1). Hence,  $C_t$  depends on  $\pmb{W}^{(t,0)}$  and is expected to decrease as  $\pmb{W}^{(t,0)}$  approaching  $\pmb{W}^{\star}$  [56, 92], which will be discussed in Corollary 3.
Definition 2. Let  $\rho$  be the value of
$$
\rho := \min  _ {\| \boldsymbol {\alpha} \| _ {2} \geq 1} \mathbb {E} _ {(\boldsymbol {s}, a) \sim \pi^ {\star}} \left(\boldsymbol {\alpha} ^ {\top} \nabla_ {\boldsymbol {W}} Q (\boldsymbol {W} ^ {\star}; \boldsymbol {s}, a)\right) ^ {2}. \tag {11}
$$
$\rho$  suggests the radius of the local convex region of the objective function. We provide the lower bound for  $\rho$  in Lemma 7 (see the proof in Appendix E.2), suggesting a sufficiently large local convex region near  $W^{\star}$ .
4.3 Major Theoretical Results
Lemma 1 characterizes the convergent point when minimizing (9) using the mini-batch gradient descent in the  $t$ -th outer loop under certain conditions. Specifically, given that the initial weights at the  $t$ -th outer loop are sufficiently close to the ground truth as shown in (12) and the replay buffer is large enough as shown in (13), the distance between the learned model weights  $\pmb{W}^{(t + 1,0)}$  and  $\pmb{W}^{\star}$  are bounded from above as shown in (14).
Lemma 1 (Estimation error of  $\mathbf{W}^{(t + 1,0)}$ ). Suppose Assumptions 1 & 2 hold and the initial neuron weights at the t-th outer loop satisfy
$$
e _ {t} := \| \boldsymbol {W} ^ {(t, 0)} - \boldsymbol {W} ^ {\star} \| _ {F} \leq \mathcal {O} \left(1 - \frac {1 - \Theta (\varepsilon_ {t})}{\Theta (\sqrt {N})}\right) \cdot \frac {\rho \cdot \| \boldsymbol {W} ^ {\star} \| _ {F}}{K}, \tag {12}
$$
The step size  $\eta$  is  $1 / T$ , and the size of the replay buffer is
$$
N = \Omega \left(\rho^ {- 2} \cdot K ^ {3} \cdot L \cdot d \cdot \log q \cdot T\right). \tag {13}
$$
Then, with the high probability of at least  $1 - q^{-d}$ , the neuron weights  $\mathbf{W}^{(t + 1,0)}$  generated from Algorithm 1 satisfy
$$
\left\| \boldsymbol {W} ^ {(t + 1, 0)} - \boldsymbol {W} ^ {\star} \right\| _ {F} \leq (1 - \Theta (\varepsilon_ {t})) \cdot \gamma \cdot \left\| \boldsymbol {W} ^ {(t, 0)} - \boldsymbol {W} ^ {\star} \right\| _ {F} + \frac {C _ {t} + (1 - C _ {t}) \varepsilon_ {t}}{\Theta (\sqrt {N})} \cdot \frac {| \mathcal {A} | \cdot R _ {\max }}{1 - \gamma}. \tag {14}
$$
Remark 1 (Large replay buffer reduces the estimation error and the requirement for  $\mathbf{W}^{(t,0)}$ ). From (14), a larger  $N$  leads to a reduced distance between  $\mathbf{W}^{(t+1,0)}$  and  $\mathbf{W}^{\star}$ . Moreover, (12) implies that if we increase the size of replay buffer  $N$ , the upper bound of  $e_t$  increases, indicating the algorithm can tolerate a large range of  $\mathbf{W}^{(t,0)}$ .
<sup>1</sup>Note that  $\mathbf{W}^*$  does not need to be unique. We abuse the notation  $\| \mathbf{W} - \mathbf{W}^* \|_2$  to denote the minimum distance of  $\mathbf{W}$  to any  $\mathbf{W}^*$  satisfying Assumption 1.
2The framework can be extended into non i.i.d. samples, see Appendix G.
In the following corollary, we show the upper and lower bound of  $\varepsilon_{t}$  at the  $t$ -th outer loop. The lower bound guarantees that the RHS of (12) is larger than 0, so we have a sufficiently large radius for convergence. The upper bound ensures (14) be less than  $e_t$ , indicating an improved estimation of  $Q^{\star}$  across the iterations.
Corollary 1 (Range of  $\varepsilon$ ). Given the assumptions and conditions in Theorem 1 hold. To ensure the existence of a good initialization at iteration  $t$ ,  $\varepsilon_t$  needs to satisfy
$$
\varepsilon_ {t} \geq 1 - \Theta (\sqrt {N}) \cdot (1 - e _ {t}), \tag {15}
$$
To ensure the estimated learned model is improving over iterations,  $\varepsilon_{t}$  needs to satisfy
$$
\varepsilon_ {t} \leq \frac {(1 - \gamma) ^ {2} \cdot \Theta (\sqrt {N}) \cdot e _ {t}}{(1 - C _ {t}) \cdot | \mathcal {A} | \cdot R _ {\max }} - \frac {C _ {t}}{1 - C _ {t}}. \tag {16}
$$
Remark 2 (Reduce  $\varepsilon_{t}$  as  $t$  increases). The lower bound can always be smaller than the upper bound given a sufficiently large  $N$  as shown in (13). From both (15) and (16), we know that  $\varepsilon_{t}$  needs to decrease as  $e_t$  decreases. Specifically, the lower bound of  $\varepsilon_{t}$  is a linear function of  $e_t$ , and the upper bound of  $\varepsilon_{t}$  is a linear function of  $e_t$ . Namely, we need a relatively large  $\varepsilon_0$  at the beginning since the distance between initial point  $\mathbf{W}^{(0,0)}$  and  $\mathbf{W}^{\star}$  is large. As  $t$  increases,  $e_t$ , which is the distance of learned neuron weights  $\mathbf{W}^{(t,0)}$  to the ground truth  $\mathbf{W}^{\star}$ , becomes smaller, and we should decrease  $\varepsilon_{t}$  to guarantee an improved  $Q^{(t + 1,0)}$  over  $Q^{(t,0)}$ .
Theorem 1 shows that the learned model from Algorithm 1 converges to the optimal Q-value function  $Q^{\star}$  with geometric decay up to an estimation error shown in (20).
Theorem 1 (Convergence to  $Q^{\star}$ ). Suppose Assumptions 1 and 2 hold, the buffer size  $N$  satisfies (13). Let us define  $C_{\mathrm{max}}$  be a constant that is larger than  $C_t$  for  $1 \leq t \leq T$ , when  $\varepsilon_t$  satisfy
$$
\varepsilon_ {t} = \frac {c _ {\varepsilon} \cdot \Theta (\sqrt {N}) \cdot e _ {t}}{\left(1 - C _ {\max }\right) \cdot | \mathcal {A} | \cdot R _ {\max }} - \frac {C _ {\max }}{1 - C _ {\max }} \tag {17}
$$
for a fixed constant  $c_{\varepsilon} \in (0, (1 - \gamma)^2]$ , and the initialization satisfies
$$
\left\| \boldsymbol {W} ^ {(0, 0)} - \boldsymbol {W} ^ {\star} \right\| _ {F} \leq \mathcal {O} \left(1 - \frac {1 - c _ {\varepsilon}}{\Theta (\sqrt {N})}\right) \cdot \frac {\rho \cdot \left\| \boldsymbol {W} ^ {\star} \right\| _ {F}}{K}. \tag {18}
$$
Then, with the high probability of at least  $1 - T \cdot q^{-d}$ , we have
1. the learned weights decay geometrically with
$$
\left\| \boldsymbol {W} ^ {(t + 1, 0)} - \boldsymbol {W} ^ {\star} \right\| _ {F} \leq \left(\gamma + c _ {\varepsilon} \cdot (1 - \gamma)\right) \cdot \left\| \boldsymbol {W} ^ {(t, 0)} - \boldsymbol {W} ^ {\star} \right\| _ {F}, \forall t \leq \log_ {\gamma} (1 / N). \tag {19}
$$
2. the returned model  $Q(\mathbf{W}^{(T,0)})$  exhibits an estimation error in the order of  $1 / \sqrt{N}$  with
$$
\sup  _ {(\mathbf {s}, a)} \left| Q \left(\boldsymbol {W} ^ {(T, 0)}\right) - Q ^ {\star} \right| = \Theta \left(\left\| \boldsymbol {W} ^ {(T, 0)} - \boldsymbol {W} ^ {\star} \right\| _ {F}\right) \leq \frac {C _ {\max } \cdot | \mathcal {A} | \cdot R _ {\max }}{(1 - \gamma) ^ {2} \cdot \Theta (\sqrt {N})}, \tag {20}
$$
where  $T\geq \log_{\gamma}(1 / N)$
Remark 3 (Selection of  $\varepsilon_{t}$ ). The value of  $\varepsilon_{t}$  is influenced by three key factors:  $c_{\varepsilon}$ ,  $C_{\mathrm{max}}$ , and the current estimation error bound  $e_t$ . The constant  $c_{\varepsilon}$  is fixed and controls the magnitude of the values in the sequence  $\varepsilon_{t_{t = 1}}^T$ , providing a way to regulate the level of  $\varepsilon_{t}$ . Regarding  $C_{\mathrm{max}}$ ,  $C_t$  tends to decrease as the iteration index  $t$  increases, indicating a progressive improvement in the policy and resulting in a smaller data distribution shift. Hence, to estimate  $C_{\mathrm{max}}$ , we can leverage  $C_0$ , which is obtained by collecting data based on the policy  $\max_a Q(\pmb {W}^{(0,0)};\pmb {s},a)$ . Moreover, with  $c_{\varepsilon}$  fixed, the estimation error  $e_t$  follows a geometric decay pattern, as depicted in (19). This behavior allows us to use the expression  $(\gamma +c_{\varepsilon}\cdot (1 - \gamma))^t\cdot e_0$  as an estimate for  $e_t$ .
Remark 4 (Geometric decay to  $Q^{\star}$ ). From (19) and (20), we know that the learned model from the proposed algorithm converges to  $Q^{\star}$  with a geometric decay up to some estimation error. The convergence rate is in the order of  $\gamma + c_{\varepsilon} \cdot (1 - \gamma)$ , and the estimation error is in the order of  $(1 - \gamma)^{-2} \cdot C_{\max} / \sqrt{N}$ . As we mentioned in the takeaways in Section 4.1, a small discounted factor  $\gamma$  leads to a fast convergence rate. We have a reduced estimation with a large buffer with size  $N$ , a small distribution shift  $C_T$ , and a small  $\gamma$ .
Remark 5 (Larger  $\varepsilon$  values for enlarged region of convergence and smaller  $\varepsilon$  values for faster convergence). From (17), we know that  $c_{\varepsilon}$  controls the value level of  $\varepsilon_{t}$ . From (18), a larger  $c_{\varepsilon}$  (i.e., a higher level of  $\varepsilon$  values) increases the upper bound of  $\| W^{(0,0)} - W^{\star} \|_2$  and, thus, enlarges the proper region of  $W^{(0,0)}$ . (19) indicates that the convergence rate is in the order of  $c_{\varepsilon}$ . A smaller  $c_{\varepsilon}$  (i.e., lower level of  $\varepsilon$  values) leads to faster convergence.
In the following corollary, we provide the sample complexity for achieving  $\delta$  estimation error as shown in (21), where  $\widetilde{\Omega} (\cdot)$  omits some log factors. The corollary can be obtained by letting (20) to be less than a desired accuracy  $\delta$ .
Corollary 2 (Sample complexity). To achieve an estimation error of  $\delta$ , the required number of samples, referred to as the sample complexity, needs to satisfy
$$
N _ {s} = N \cdot \log \gamma = \widetilde {\Omega} \left((1 - \gamma) ^ {4} \cdot C _ {\max } \cdot | \mathcal {A} | ^ {2} R _ {\max } ^ {2} \cdot K ^ {3} \cdot L \cdot d \cdot T / \delta^ {2}\right). \tag {21}
$$
Remark 6 (Sample complexity). (21) shows that the sample complexity is a linear function of  $d$  and  $L$ , where  $d$  is the feature mapping of the state-action pair  $(s, a)$  and  $L$  is the number of layers. Given the freedom of degree of  $W$  is a linear function of  $d$  and  $L$ , respectively, the sample complexity is almost order-wise optimal with respect to  $d$  and  $L$ .
The following corollary presents a tighter bound for the model estimation error compared with (20). This improvement arises from a stronger assumption on  $C_t$ , which becomes a function of  $\| \pmb{W}^{(t,0)} - \pmb{W}^{\star} \|_2$ . Compared with (20), the estimation error bound in (24) and (25) considers the cases that the behavior policy is improved as  $\pmb{W}^{(t,0)}$  becomes closer to  $\pmb{W}^{\star}$ . As a result, we achieve a more precise and tighter estimation of the error in the model.
Corollary 3 (Distribution shift and estimation error). Assume that  $C_t$  is Hölder continuous with a factor  $\alpha$  as
$$
C _ {t} = \mathcal {O} \left(\left\| \boldsymbol {W} ^ {(t, 0)} - \boldsymbol {W} ^ {\star} \right\| _ {2} ^ {\alpha}\right), \tag {22}
$$
${}^{3}$  for  $0 < \alpha  \leq  1$  and all  $t \in  \left\lbrack  T\right\rbrack$  . When  ${\varepsilon }_{t}$  satisfy
$$
\varepsilon_ {t} = \frac {c _ {\varepsilon} \cdot \Theta (\sqrt {N}) \cdot e _ {t}}{\left(1 - C _ {t}\right) \cdot | \mathcal {A} | \cdot R _ {\max }} - \frac {C _ {t}}{1 - C _ {t}}, \tag {23}
$$
the estimation error of the  $Q$ -function satisfies
$$
\sup  _ {(\boldsymbol {s}, a)} \left| Q \left(\boldsymbol {W} ^ {(T, 0)}\right) - Q ^ {\star} \right| = \frac {\left(| \mathcal {A} | \cdot R _ {\max }\right) ^ {\frac {1}{1 - \alpha}}}{(1 - \gamma) ^ {\frac {2}{1 - \alpha}} \cdot \Theta \left(N ^ {\frac {1}{2 (1 - \alpha)}}\right)}. \tag {24}
$$
In the special case that  $C_t = \mathcal{O}(\| \pmb{W}^{(t,0)} - \pmb{W}^\star \|_2)$ , then
$$
\sup  _ {(\boldsymbol {s}, a)} \left| Q \left(\boldsymbol {W} ^ {(T, 0)}\right) - Q ^ {\star} \right| = 0. \tag {25}
$$
Remark 7 (Reduced or zero estimation error when behavior policy is improved over iterations). Recall the definition of  $C_t$  in Definition 1. If  $e_t$  is zero, i.e.,  $W_t = W^\star$ , the action selected following the greedy policy is always the optimal action, which means that  $C_t = 0$ . Therefore, it is reasonable to assume that  $C_t$  is Hölder continuous as shown in (22). When  $\alpha > 0$ , we can see that (24) is less than (20), indicating a reduced estimation error and sample complexity. Typically, if  $C_t$  has an order of growth less than  $e_t$  near  $W^\star$ , a zero estimation error is achievable.
4.4 The Roadmap of Proofs, Comparison with Existing Works, and Limitations
The proof of Theorem 1 draws inspiration from the model estimation framework in the supervised learning setting [90, 88]. The key idea is to use a population risk function (PRF) to characterize the objective function in (9). By satisfying certain conditions such as having sufficient training samples and a bounded data distribution shift, the approximation error between the PRF and objective function can be bounded. This allows for the characterization of the optimization problem in (9) by analyzing the landscape and convergence properties of the PRF.
<sup>3</sup>Although this equation depends on the algorithm's trajectory, it can be easily derived from a time-independent equation  $|\pi_W(\pmb{s}|a) - \pi^{\star}(\pmb{s}|a) \leq C||\pmb{W} - \pmb{W}^{\star}||_2$ . Also, this equation is a weaker condition than (2) in [92], which holds universally across the entire space and model parameter space.
In comparison to existing proofs based on the model estimation frameworks, this paper addresses two additional challenges. Firstly, it extends the proof from one-hidden-layer neural networks to multi-layer neural networks. This extension is achieved by providing new tools for characterizing the Hessian matrix (refer to Lemma 3) and concentration bound (refer to Lemmas 3 and 6). Additionally, this paper characterizes the differences between the two functions caused by the interaction of neuron weights across layers in the gradient and Hessian matrix. Secondly, the paper extends the proof from supervised learning settings to Q-learning settings. This requires characterizing the additional error term caused by the data distribution shift and "noisy" labels (refer to Lemma 3) because the empirical risk function is no longer an expectation of the defined population risk function.
Existing state-of-the-art theoretical results on Q-learning with neural network approximations primarily revolve around the NTK and Besov regularity frameworks. In the NTK framework, the networks are assumed to be extremely over-parameterized, requiring an impractical projection step and resulting in error bounds that cannot be characterized for networks with finite width. In the Besov regularity framework, the neural network needs to be sparse, which does not align with the DQN algorithm. Furthermore, the analysis in the Besov regularity framework relies on the achievability of the global minimizer of the non-convex problem in (9), which cannot be guaranteed using GD algorithms. This paper takes a significant step towards bridging the gap between theoretical understanding and practical applications of DQN by addressing these challenges. However, there still remains a gap between the theoretical results and numerical findings. Future research directions include devising efficient exploration strategies for DQNs to further enhance their performance and extending the theoretical analysis to variants of DQNs and policy gradient-based methods.
In this section, we provide numerical justification that our theoretical findings are aligned with practical DQNs through the Atari Pong game, which is commonly used for DQNs in [50, 51, 73]. We take the Double DQN (DDQN) [73], one of the most popular variants of DQN, as the backbone in the setup. DDQN differs from DQN only in (8) via changing  $y = r + \gamma \cdot Q(\pmb{W}^{(t,0)}; s', a^{\star})$  to  $y = r + \gamma \cdot Q(\pmb{W}^{(t,m)}; s', a^{\star})$ , where  $a^{\star} = \operatorname{argmax}_{a} Q(\pmb{W}^{(t,0)}; s', a)$ . DDQN outperforms DQN in the relief of overoptimism and the improvement of stability. Our numerical experiments on DDQN also indicate that our analysis applies to the variants of DQN.
The input to the network is  $84 \times 84 \times 4$  images, where the last dimension represents the number of frames in history. The network is a convolutional neural network consisting of three convolutional layers and one fully-connected layer. The algorithm terminates if the average score over the recent 20 episodes does not improve or the algorithm reaches the maximum episode set as 200, which is around  $4 \times 10^5$  training steps. The testing score is calculated based on a similar setup as the training process by fixing the maximum memory size  $N$  as 2000 and greedy policy, i.e.,  $\varepsilon = 0$ . Each point in the plot is averaged over 10 experiments with an error bar representing the standard deviation.
Estimation errors with respect to the sample complexity  $N$ . As the Q-value function is the estimate of the expected cumulative reward, we use the difference between the reward obtained from the estimated Q-value function and the maximum reward as the estimation error of the learned model to the optimal Q-value function, which is also consistent with the experiments in [50, 51, 73]. Given that the full test score in the Pong game is 21, we set the test error as the value of (21 - test score) in each experiment. The  $\varepsilon_{t}$  in  $\varepsilon$ -greedy policy decreases geometrically from 1 to 0.01. We vary the number of samples in the replay buffer from 400 to 2500. Figure 1 shows that the test error is almost linear in  $1 / \sqrt{N}$ , which is consistent with our characterization in (20). In addition, experiments with a large  $N$  have a shorter error bar indicating a more stable learning performance with a large sample complexity as shown in (12).
Convergence with different selections of  $\varepsilon$ . Figure 2 illustrates the convergence rate when  $\varepsilon_{t}$  in the  $\varepsilon$ -greedy policy changes. For each point,  $\varepsilon_0$  is selected as the value in the x-axis, and we decrease  $\varepsilon_{t}$  geometrically as the iteration  $t$  increases. Each point is averaged over 5 independent trials. We can see that the convergence rate is a linear function of  $c_{\varepsilon}$ , matching our findings in (19).
Performance with different selections of  $\varepsilon$ . We investigate the effect of  $\varepsilon$ -greedy policy during the training. In Figure 3, we show the test scores using  $\varepsilon$ -greedy policy with (1) geometrically decreasing  $\varepsilon$  from 1 to 0.01, (2) fixed  $\varepsilon$  as 0.1, and (3) fixed  $\varepsilon$  as 0. Each test score in the curve is averaged over the past 10 episodes to smooth the trend. One can observe that a gradually decreasing
Figure 1: Test error in scores against the number of samples.
Figure 2: The convergence rate against the value of  $c_{\varepsilon}$ .
Figure 3: Test scores of the cases of  $\varepsilon$ -greedy down to different values.
$\varepsilon$  leads to a better score than fixing  $\varepsilon$  to be the final value. The test score of  $\varepsilon = 0.1$  shares a similar trend to the  $\varepsilon$ -greedy policy but with a slower speed, matching our findings in (19) that a small  $\varepsilon$  leads to a slow convergence rate. The test score of  $\varepsilon = 0$  has the fastest convergence rate at the early stage, but the convergent point is the worst and the most unstable, matching our findings in (18) that a small  $\varepsilon$  leads to a reduced radius of convergence.
This paper provides the first convergence and sample complexity analysis of the DQN algorithm equipped with  $\varepsilon$ -greedy exploration. We establish the theoretical guarantee for the convergence of the learned model to the optimal Q-value function  $Q^{\star}$ , which can be used to derive the optimal policy. We provide a nearly optimal sample complexity for achieving an arbitrarily small estimation error. We also prove that  $\varepsilon$ -greedy with decreasing  $\varepsilon$  achieves both an enlarged radius of convergence and an improved convergence rate. Future directions include the generalization of the theoretical analysis to the variants of DQNs and the design of efficient exploration strategies for DQNs.
One of the anonymous reviewers raised concerns about (12) regarding its demanding requirements on the initial policy. We would like to clarify that (12) primarily concerns the optimization analysis of the objective function rather than the initial policy. Instead, we impose only a minor assumption on the initial policy, with no specific environmental constraints. In our work,  $C_0$  quantifies the initial policy's difference from the optimal policy and is independent of (12) in our primary theoretical results. With a sufficiently large replay buffer,  $C_0$  can approach one, except when it equals 1, indicating an extreme divergence from the optimal policy in all states. Thus, our initial policy assumption is minimal. Considering the highly non-convex nature of deep neural network objective functions with countless local minima, (12) represents the state-of-the-art assumption for optimizing deep neural networks.
As mentioned by the anonymous reviewers, the selections of  $\varepsilon_{t}$  in (17) depends on  $e_t = \| \pmb{W}^{(t)} - \pmb{W}^\star \|_2$ , which is unknown to the agent. Here, we would like to clarify that  $e_t$  can be replaced by its upper bound in (19) and lower bound in (20). By plugging (19) and (20) into (17), we have
$$
\varepsilon_ {t} = \max  \left\{\frac {c _ {\varepsilon} \cdot \Theta (\sqrt {N}) \cdot \left(\gamma + c _ {\varepsilon} \cdot (1 - \gamma)\right) ^ {t} e _ {0}}{(1 - C _ {\operatorname* {m a x}}) \cdot | \mathcal {A} | \cdot R _ {\operatorname* {m a x}}} - \frac {C _ {\operatorname* {m a x}}}{1 - C _ {\operatorname* {m a x}}}, \quad c _ {\varepsilon} \cdot \frac {C _ {\operatorname* {m a x}}}{1 - C _ {\operatorname* {m a x}}} \right\}. \tag {26}
$$
As mentioned by one of the anonymous reviewers, Corollary 3 relies on an assumption that depends on the algorithm's trajectory, which lacks mathematical rigor. Here, we would like to clarify that although this equation depends on the algorithm's trajectory, it can be easily derived from a time-independent equation
$$
\left| \pi_ {\boldsymbol {W}} (\boldsymbol {s} | a) - \pi^ {\star} (\boldsymbol {s} | a) \right| \leq C \| \boldsymbol {W} - \boldsymbol {W} ^ {\star} \| _ {2}. \tag {27}
$$
Additionally, it is worth mentioning the difference between (22) in this paper and (2) in [92]. Specifically, we want to highlight that the equation above, (27), leads to (22). In contrast, (2) in [92] requires the following condition to hold for all  $\mathbf{W}_1$  and  $\mathbf{W}_2$ :
$$
\left| \pi_ {\boldsymbol {W} _ {1}} (\boldsymbol {s} | a) - \pi_ {\boldsymbol {W} _ {2}} (\boldsymbol {s} | a) \leq C \right| \left| \left| \boldsymbol {W} _ {1} - \boldsymbol {W} _ {2} \right| \right| _ {2} \tag {28}
$$
As a comparison, (27) only requires  $W_{2}$  to be the ground truth and  $W_{1}$  to be some weights near the ground truth. In other words, (27) is a sufficient condition for (22). While equation (2) in [92] does not hold with epsilon-greedy, (27) can hold with Q-learning using epsilon-greedy, thus ensuring the mathematical rigor of (22).
This work was done when Shuai was a postdoc at Rensselaer Polytechnic Institute (RPI). We thank Dr. Tianyi Chen at RPI for providing remarkable and inspiring insight into the analysis of temporal difference learning. This work was supported by AFOSR FA9550-20-1-0122, ARO W911NF-21-1-0255, NSF 1932196, and the Rensselaer-IBM AI Research Collaboration (http://airc.rpi.edu), part of the IBM AI Horizons Network (http://ibm.biz/AIHorizons). We thank all anonymous reviewers for their constructive comments.