SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where the Lebesgue measure of  $\mathcal{R}$  over  $\mathbb{R}^{K_{\ell}}$  is non-zero. Instead of directly proving  $\int_{\mathcal{R}}\left(\sum_{k = 1}^{K}\pmb{\alpha}^{\top}\pmb{h}\phi^{\prime}(\pmb{w}_{k}^{\top}\pmb {h})\right)^{2}f_{H}(\pmb {h})\cdot d\pmb {h} > 0$  for any  $\pmb{\alpha}$ , we note that it is sufficient to prove that  $\{h\phi^{\prime}(w_k^\top h)\}_{k\in [K]}$  are linear independent over the Hilbert space  $\mathcal{H}$ . Namely, if  $\{h\phi^{\prime}(\pmb{w}_k^\top \pmb {h})\}_{k\in [K]}$  are linear independent, we have
$$
\boldsymbol {\alpha} ^ {\top} \boldsymbol {h} \phi^ {\prime} \left(\boldsymbol {w} _ {k} ^ {\top} \boldsymbol {h}\right) \neq 0 \quad \text {a l m o s t e v e r w h e r e}. \tag {136}
$$
Therefore, we can know that  $\int_{\mathcal{R}}\left(\sum_{j = 1}^{K}\pmb{\alpha}^{\top}\pmb{h}\phi^{\prime}(\pmb{w}_{\ell ,j}^{\top}\pmb {h})\right)^{2}p_{H}(\pmb {h})\cdot d\pmb{h}$  is strictly greater than zero.
Next, we provide the whole proof for that  $\{x\phi^{\prime}(\pmb{w}_k^\top \pmb {h})\}_{k\in [K]}$  are linear independent over the Hilbert space  $\mathcal{H}$ .
We define a group of functions  $\{\psi_j(\pmb{h})\}_{j=1}^K$ , where  $\psi_j(\pmb{h}) = \pmb{h}\phi'(\pmb{w}_j^\top \pmb{h})$ . From the assumption in Lemma 7, we can justify that  $\mathbb{E}_{\pmb{h} \sim \mathcal{D}}|\psi_j(\pmb{h})|^2 \leq \mathbb{E}_{\pmb{h} \sim \mathcal{D}}|\pmb{h}|^2 < \infty$ .
Let  $\mathcal{X}_i = \{\pmb{h} \mid \pmb{w}_i^\top \pmb{h} = 0\}$  for any  $i \in [K]$ . For any fixed  $k$ , we can justify that  $\mathcal{X}_k$  cannot be covered by other sets  $\{\mathcal{X}_k\}_{j \neq k}$  as long as  $\pmb{w}_k$  does not parallel to any other weights  $\pmb{w}_j$  with  $j \neq k$ . Namely,  $\mathcal{X}_k \not\subset \cup_{j \neq k} \mathcal{X}_j$ . The idea of proving the claim above is that the intersection of  $\mathcal{X}_j$  and  $\mathcal{X}_k$  is only a hyperplane in  $\mathcal{X}_k$ . The union of finite many hyperplanes is not even a measurable space and thus cannot cover the original space. Formally, we provide the formal proof for this claim as follows.
Let  $\lambda$  be the Lebesgue measure on  $\mathcal{X}_k$ , then  $\lambda(\mathcal{X}_k) > 0$ . When  $\boldsymbol{w}_j$  does not parallel to  $\boldsymbol{w}_k$ ,  $\mathcal{X}_k \cap \mathcal{X}_j$  is only a hyperplane in  $\mathcal{X}_k$  for  $j \neq k$ . Hence, we have  $\lambda(\mathcal{X}_j \cap \mathcal{X}_k) = 0$ . Next, we have
$$
\lambda \left(\mathcal {X} _ {k} \cap \left(\cup_ {j \neq k} \mathcal {X} _ {k}\right)\right) \leq \sum_ {j \neq k} \lambda \left(\mathcal {X} _ {k} \cap \mathcal {X} _ {j}\right) = 0. \tag {137}
$$
$$
\lambda \left(\mathcal {X} _ {k} / \left(\cup_ {j \neq k} \mathcal {X} _ {k}\right)\right) = \lambda \left(\mathcal {X} _ {k}\right) - \lambda \left(\mathcal {X} _ {k} \cap \left(\cup_ {j \neq k} \mathcal {X} _ {k}\right)\right) = \lambda \left(\mathcal {X} _ {k}\right) > 0. \tag {138}
$$
Therefore, we have  $\mathcal{X}_k / (\cup_{j\neq k}\mathcal{X}_j)$  is not empty, which means that  $\mathcal{X}_k\notin \cup_{j\neq k}\mathcal{X}_j$
Next, Since  $\mathcal{X}_k / (\cup_{j\neq k}\mathcal{X}_j)$  is not an empty set, there exists a point  $z_{k}\in \mathcal{X}_{k} / (\cup_{j\neq k}\mathcal{X}_{j})$  and  $r_0 > 0$  such that
$$
\mathcal {B} \left(\boldsymbol {z} _ {k}, r\right) \cap \mathcal {D} _ {j} = \emptyset \quad \text {w i t h} \quad \forall r \leq r _ {0} \text {a n d} j \neq k, \tag {139}
$$
where  $\mathcal{B}(\pmb{z}_k, r)$  stands for a ball centered at  $\pmb{z}_k$  with a radius of  $r$ . Then, we divide  $\mathcal{B}(\pmb{z}_k, r)$  into two disjoint subsets such that
$$
\mathcal {B} _ {r} ^ {+} = \mathcal {B} \left(\boldsymbol {z} _ {k}, r\right) \cap \left\{\boldsymbol {h} \mid \boldsymbol {w} _ {k} ^ {\top} \boldsymbol {h} > 0 \right\}, \tag {140}
$$
$$
\mathcal {B} _ {r} ^ {-} = \mathcal {B} \left(\boldsymbol {z} _ {k}, r\right) \cap \left\{\boldsymbol {h} \mid \boldsymbol {w} _ {k} ^ {\top} \boldsymbol {h} <   0 \right\}.
$$
Because  $z_{k}$  is a boundary point of  $\{h|\pmb{w}_{k}^{\top}\pmb{h} = 0\}$ , both  $\mathcal{B}_r^+$  and  $\mathcal{B}_r^-$  are non-empty.
Note that  $\psi_j(h)$  is continuous at any point except for the ones in  $\mathcal{X}_j$ . Then, for any  $j \neq k$ , we know that  $\phi_j(\boldsymbol{w}_k^\top \boldsymbol{h})$  is continuous at point  $\boldsymbol{z}_k$  since  $\boldsymbol{z}_k \notin \mathcal{X}_j$ . Hence, it is easy to verify that
$$
\lim  _ {r \rightarrow 0 _ {+}} \frac {1}{\lambda \left(\mathcal {B} _ {r} ^ {+}\right)} \int_ {\mathcal {B} _ {r} ^ {+}} \psi_ {k} (\boldsymbol {h}) d \boldsymbol {h} = \lim  _ {r \rightarrow 0 _ {-}} \frac {1}{\lambda \left(\mathcal {B} _ {r} ^ {-}\right)} \int_ {\mathcal {B} _ {r} ^ {+}} \psi_ {k} (\boldsymbol {h}) d \boldsymbol {h} = \psi_ {k} \left(\boldsymbol {z} _ {k}\right). \tag {141}
$$
While for  $\psi_{k}$ , we know that  $\psi_{k}(\pmb{h})\equiv 0$  for  $\pmb {h}\in \mathcal{B}_r^-$  (ii)  $\psi_{k}(\pmb {h}) = \pmb{h}$  for  $\pmb {h}\in \mathcal{B}_r^+$ . Hence, it is easy to verify that
$$
\lim  _ {r \rightarrow 0 _ {+}} \frac {1}{\lambda \left(\mathcal {B} _ {r} ^ {+}\right)} \int_ {\mathcal {B} _ {r} ^ {+}} \psi_ {k} (\boldsymbol {h}) d \boldsymbol {h} = \boldsymbol {z} _ {k} \tag {142}
$$
$$
\lim  _ {r \to 0 -} \frac {1}{\lambda \left(\mathcal {B} _ {r} ^ {-}\right)} \int_ {\mathcal {B} _ {r} ^ {+}} \psi_ {k} (\boldsymbol {h}) d \boldsymbol {h} = 0.
$$
Now let us proof that  $\{\psi_j\}_{j=1}^K$  are linear independent by contradiction. Suppose  $\{\psi_j\}_{j=1}^K$  are linear dependent, we have
$$
\sum_ {j = 1} ^ {K} \alpha_ {j} \psi_ {j} (\boldsymbol {h}) \equiv 0, \quad \forall \boldsymbol {h}. \tag {143}
$$
$$
\lim  _ {r \rightarrow 0 _ {+}} \frac {1}{\lambda \left(\mathcal {B} _ {r} ^ {+}\right)} \int_ {\mathcal {B} _ {r} ^ {+}} \sum_ {j = 1} ^ {K} \alpha_ {j} \psi_ {j} (\boldsymbol {h}) d \boldsymbol {h} = 0 \tag {144}
$$
$$
\lim  _ {r \rightarrow 0 +} \frac {1}{\lambda \left(\mathcal {B} _ {r} ^ {-}\right)} \int_ {\mathcal {B} _ {r} ^ {+}} \sum_ {j = 1} ^ {K} \alpha_ {j} \psi_ {j} (\boldsymbol {h}) d \boldsymbol {h} = 0
$$
$$
\begin{array}{l} 0 = \lim  _ {r \rightarrow 0 _ {+}} \frac {1}{\lambda \left(\mathcal {B} _ {r} ^ {+}\right)} \int_ {\mathcal {B} _ {r} ^ {+}} \sum_ {j = 1} ^ {K} \alpha_ {j} \psi_ {j} (\boldsymbol {h}) d \boldsymbol {h} - \lim  _ {r \rightarrow 0 _ {+}} \frac {1}{\lambda \left(\mathcal {B} _ {r} ^ {-}\right)} \int_ {\mathcal {B} _ {r} ^ {+}} \sum_ {j = 1} ^ {K} \alpha_ {j} \psi_ {j} (\boldsymbol {h}) d \boldsymbol {h} \tag {145} \\ = \alpha_ {k} z _ {k} \\ \end{array}
$$
where the last equality comes from (141) and (142).
Note that  $\mathbf{z}_k$  cannot be  $\mathbf{0}$  because  $\mathbf{z}_k \notin \mathcal{X}_j$ . Therefore, we have  $\alpha_k = 0$ . Similarly to (145), we can obtain that  $\alpha_j = 0$  by define  $\mathbf{z}_j$  following the definition of  $\mathbf{z}_k$  for any  $j \in [K]$ . Then, we know that (143) holds if and only if  $\alpha = \mathbf{0}$ , which contradicts the assumption that  $\{\psi_j\}_{j=1}^K$  are linear dependent.
In conclusion, we know that  $\{\psi_j\}_{j=1}^K$  are linear independent, and  $\int_{\mathcal{R}} \left( \sum_{j=1}^{K} \boldsymbol{\alpha}^\top \boldsymbol{h} \phi'(\boldsymbol{w}_{\ell,j}^\top \boldsymbol{h}) \right)^2 p_H(\boldsymbol{h}) \cdot d\boldsymbol{h}$  is strictly greater than zero.
E.3 Proof of Lemma 8
Proof of Lemma 8. From the definition of (39), we have
$$
\begin{array}{l} \left\| \boldsymbol {h} ^ {(\ell)} (\boldsymbol {W}) - \boldsymbol {h} ^ {(\ell)} \left(\boldsymbol {W} ^ {\star}\right) \right\| _ {2} \\ = \| \phi \left(\boldsymbol {W} _ {\ell - 1} ^ {\top} \boldsymbol {h} ^ {(\ell - 1)} (\boldsymbol {W})\right) - \phi \left(\boldsymbol {W} _ {\ell - 1} ^ {\star \top} \boldsymbol {h} ^ {(\ell - 1)} (\boldsymbol {W} ^ {\star})\right) \| _ {2} \\ = \| \phi \left(\boldsymbol {W} _ {\ell - 1} ^ {\top} \boldsymbol {h} ^ {(\ell - 1)} (\boldsymbol {W})\right) - \phi \left(\boldsymbol {W} _ {\ell - 1} ^ {\star \top} \boldsymbol {h} ^ {(\ell - 1)} (\boldsymbol {W})\right) \\ + \phi \left(\boldsymbol {W} _ {\ell - 1} ^ {\star \top} \boldsymbol {h} ^ {(\ell - 1)} (\boldsymbol {W})\right) - \phi \left(\boldsymbol {W} _ {\ell - 1} ^ {\star \top} \boldsymbol {h} ^ {(\ell - 1)} \left(\boldsymbol {W} ^ {\star}\right)\right) \| _ {2} \tag {146} \\ \leq \left\| \phi \left(\boldsymbol {W} _ {\ell - 1} ^ {\top} \boldsymbol {h} ^ {(\ell - 1)} (\boldsymbol {W})\right) - \phi \left(\boldsymbol {W} _ {\ell - 1} ^ {\star \top} \boldsymbol {h} ^ {(\ell - 1)} (\boldsymbol {W})\right) \right\| _ {2} \\ + \left\| \phi \left(\boldsymbol {W} _ {\ell - 1} ^ {\star \top} \boldsymbol {h} ^ {(\ell - 1)} (\boldsymbol {W})\right) - \phi \left(\boldsymbol {W} _ {\ell - 1} ^ {\star \top} \boldsymbol {h} ^ {(\ell - 1)} (\boldsymbol {W} ^ {\star})\right) \right\| _ {2} \\ \leq \left\| \boldsymbol {W} _ {\ell - 1} - \boldsymbol {W} _ {\ell - 1} ^ {\star} \right\| _ {2} \cdot \left\| \boldsymbol {h} ^ {(\ell - 1)} (\boldsymbol {W}) \right\| _ {2} + \left\| \boldsymbol {h} ^ {(\ell - 1)} (\boldsymbol {W}) - \boldsymbol {h} ^ {(\ell - 1)} (\boldsymbol {W} ^ {\star}) \right\| _ {2}. \\ \end{array}
$$
With the assumption in the Lemma 8 such that  $W$  is close enough to  $\pmb{W}^{\star}$ , we have
$$
\left\| \boldsymbol {W} _ {i} \right\| _ {2} \leq \left\| \boldsymbol {W} _ {i} ^ {\star} \right\| _ {2} + \left\| \boldsymbol {W} _ {i} - \boldsymbol {W} _ {i} ^ {\star} \right\| _ {2} \lesssim 1. \tag {147}
$$
$$
\left\| \boldsymbol {h} ^ {(i)} (\boldsymbol {W}) \right\| _ {2} \leq \left\| \boldsymbol {W} _ {i} \right\| _ {2} \dots \left\| \boldsymbol {W} _ {1} \right\| _ {2} \cdot \left\| \boldsymbol {x} \right\| _ {2} \lesssim \| \boldsymbol {x} \| _ {2}. \tag {148}
$$
$$
\begin{array}{l} \| \pmb {h} ^ {(\ell)} (\pmb {W}) - \pmb {h} ^ {(\ell)} (\pmb {W} ^ {\star}) \| _ {2} \\ \leq \left\| \boldsymbol {W} _ {\ell - 1} - \boldsymbol {W} _ {\ell - 1} ^ {\star} \right\| _ {2} \cdot \left\| \boldsymbol {x} \right\| _ {2} + \left\| \boldsymbol {h} ^ {(\ell - 1)} (\boldsymbol {W}) - \boldsymbol {h} ^ {(\ell - 1)} (\boldsymbol {W} ^ {\star}) \right\| _ {2} \\ \leq \sum_ {i = 1} ^ {\ell - 1} \| \boldsymbol {W} _ {i} - \boldsymbol {W} _ {i} ^ {\star} \| _ {2} \cdot \| \boldsymbol {x} \| _ {2} + \| \boldsymbol {h} ^ {(1)} (\boldsymbol {W}) - \boldsymbol {h} ^ {(1)} (\boldsymbol {W} ^ {\star}) \| _ {2} \\ = \sum_ {i = 1} ^ {\ell - 1} \| \boldsymbol {W} _ {i} - \boldsymbol {W} _ {i} ^ {\star} \| _ {2} \cdot \| \boldsymbol {x} \| _ {2} + \| \boldsymbol {x} - \boldsymbol {x} \| _ {2} \tag {149} \\ = \sum_ {i = 1} ^ {\ell - 1} \| \boldsymbol {W} _ {i} - \boldsymbol {W} _ {i} ^ {\star} \| _ {2} \cdot \| \boldsymbol {h} ^ {(i - 1)} (\boldsymbol {W}) \| _ {2} \\ \leq \| \boldsymbol {W} - \boldsymbol {W} ^ {\star} \| _ {2} \cdot \| \boldsymbol {x} \| _ {2}, \\ \end{array}
$$
In this section, we provide numerical justification that our theoretical findings are aligned with DDQN through the Atari Breakout game. The neural network follows the same architecture as the one used in Section 5. The algorithm terminates if the average score over the recent 100 episodes does not improve or the algorithm reaches the maximum episode set as 200, which is around  $4 \times 10^{5}$  training steps. The testing score is calculated based on a similar setup as the training process by fixing the maximum memory size  $N$  as 2000 and greedy policy, i.e.,  $\varepsilon = 0$ . Each point in the plot is averaged over 10 experiments with an error bar representing the standard deviation.
Estimation errors with respect to the sample complexity  $N$ . We follow the setup in Section 5 to use the expected cumulative reward as the estimation error of the learned model to the optimal Q-value function. The  $\varepsilon_{t}$  in  $\varepsilon$ -greedy policy decreases geometrically from 1 to 0.01. We vary the number of samples in the replay buffer from 3000 to 10000. Figure 4 shows that the test error is almost linear in  $1 / \sqrt{N}$ , which is consistent with our characterization in (20). In addition, experiments with a large  $N$  have a shorter error bar indicating a more stable learning performance with a large sample complexity as shown in (12).
Convergence with different selections of  $\varepsilon$ . Figure 5 illustrates the convergence rate when  $\varepsilon_{t}$  in the  $\varepsilon$ -greedy policy changes. For each point,  $\varepsilon_0$  is selected as the value in the x-axis, and we decrease  $\varepsilon_{t}$  geometrically as the iteration  $t$  increases. Each point is averaged over 10 independent trials. We can see that the convergence rate is a linear function of  $c_{\varepsilon}$ , matching our findings in (19).
Figure 4: Test error in scores against the number of samples.
Figure 5: The convergence rate against the value of  $c_{\varepsilon}$ .
G Extension to non i.i.d. samples
Assumption 3. At any fixed outer iteration  $t$ , the behavior policy  $\pi_t$  and transition kernel  $\mathcal{P}_t$  satisfy
$$
\sup  _ {\boldsymbol {s} \in \mathcal {S}} d _ {T V} \left(\mathbb {P} \left(\boldsymbol {s} _ {\tau} \in \cdot\right) \mid \boldsymbol {s} _ {0} = \boldsymbol {s}\right), \mathcal {P} _ {t}) \leq \lambda \nu^ {\tau}, \quad \forall \tau \geq 0 \tag {150}
$$
for some constant  $\lambda > 0$  and  $\nu \in (0,1)$ , where  $d_{TV}$  denotes the total-variation distance between the probability measures.
Assumption 3 assumes the Markov chain  $\{s_t\}$  induced by the behavior policy, i.e.,  $\varepsilon_{t}$ -greedy policy at  $t$ -th outer loop, is uniformly ergodic with the corresponding invariant measure  $\mathcal{P}_t$ . Compared with i.i.d. cases, we need to handle an additional error term when bounding the distance between the  $g_{t}$  and  $\nabla f$  as shown in (91). Therefore, the upper bound in Lemma 3 changes, which suggests an additional term in the final bound.
We present the major theoretical findings for non-i.i.d. samples in Theorem 2. The major proofs in this context follow similar steps to the proof of Theorem 1, with slight changes in the error bound between the sequences  $g_{t}$  and  $\nabla f$ . In this section, we omit the details of the proof for Theorem 2 but provide the proof for Lemma 3 under the assumptions outlined in Assumption 2 to simplify the presentation.
Theorem 2 (Convergence for non-i.i.d. case). Suppose Assumption 1 and (143) hold, the buffer size  $N$  satisfies (13). Let us define  $C_{\mathrm{max}}$  be a constant that is larger than  $C_t$  for  $1 \leq t \leq T$  and  $C_d = |\mathcal{A}| \cdot \left(1 + \log_\nu \lambda^{-1} + \frac{1}{1 - \nu}\right)$ , when  $\varepsilon_t$  satisfy
$$
\varepsilon_ {t} = \frac {c _ {\varepsilon} \cdot \Theta (\sqrt {N}) \cdot e _ {t}}{\left(1 - C _ {\max }\right) \cdot C _ {d} \cdot R _ {\max }} - \frac {C _ {\max }}{1 - C _ {\max }} \tag {151}
$$
for a fixed constant  $c_{\varepsilon}\in (0,(1 - \gamma)^{2}]$ , and the initialization satisfies
$$
\left\| \boldsymbol {W} ^ {(0, 0)} - \boldsymbol {W} ^ {\star} \right\| _ {F} \leq \mathcal {O} \left(1 - \frac {1 - c _ {\varepsilon}}{\Theta (\sqrt {N})}\right) \cdot \frac {\rho \cdot \| \boldsymbol {W} ^ {\star} \| _ {F}}{K}. \tag {152}
$$
(C1) The learned weights decay geometrically with
$$
\left| \left| \boldsymbol {W} ^ {(t + 1, 0)} - \boldsymbol {W} ^ {\star} \right| \right| _ {F} \leq \left(\gamma + c _ {\varepsilon} \cdot (1 - \gamma)\right) \cdot \left| \left| \boldsymbol {W} ^ {(t, 0)} - \boldsymbol {W} ^ {\star} \right| \right| _ {F} + \frac {(2 + \gamma) R _ {\max } \tau^ {\star}}{(1 - \gamma) \Theta (N)}, \tag {153}
$$
(C2) the returned model  $Q(\mathbf{W}^{(T,0)})$  exhibits an estimation error as
$$
\sup  _ {(s, a)} \left| Q \left(\boldsymbol {W} ^ {(T, 0)}\right) - Q ^ {\star} \right| \leq \frac {C _ {\max } \cdot C _ {d} \cdot R _ {\max }}{(1 - \gamma) ^ {2} \cdot \Theta (\sqrt {N \cdot T})} + \frac {(2 + \gamma) R _ {\max } \tau^ {\star}}{(1 - \gamma) \Theta (N \cdot T)}, \tag {154}
$$
where  $\tau^{\star} = \min \{t\mid \lambda \nu^{t}\leq 1 / (N\cdot T)\}$
Proof of Lemma 3 under Assumption 2. Recall that in (91), we have
$$
\begin{array}{l} g _ {t} (\boldsymbol {w} _ {\ell , k}; \boldsymbol {W}) - \frac {\partial f}{\partial \boldsymbol {w} _ {\ell , k}} (\boldsymbol {W}) \\ = \left[ \frac {1}{N} \sum_ {n = 1} ^ {N} \left(Q (\boldsymbol {W}; \boldsymbol {s} _ {n}, a _ {n}) - Q (\boldsymbol {W} ^ {\star}; \boldsymbol {s} _ {n}, a _ {n})\right) \cdot \frac {\partial Q (\boldsymbol {W} ; \boldsymbol {s} _ {n} , a _ {n})}{\partial \boldsymbol {w} _ {\ell , k}} \right. \\ \left. - \mathbb {E} _ {\left(\boldsymbol {s}, a\right) \sim \mathcal {D} _ {t}} \left(Q (\boldsymbol {W}; \boldsymbol {s}, a) - Q \left(\boldsymbol {W} ^ {\star}; \boldsymbol {s}, a\right)\right) \cdot \frac {\partial Q (\boldsymbol {W} ; \boldsymbol {s} , a)}{\partial \boldsymbol {w} _ {\ell , k}} \right] \tag {155} \\ + \left[ \frac {1}{N} \sum_ {n = 1} ^ {N} \gamma \cdot \left(\max  _ {a} Q (\boldsymbol {s} _ {n}, a; \boldsymbol {W} ^ {\star}) - \max  _ {a} Q (\boldsymbol {s} _ {n}, a; \boldsymbol {W} ^ {(t, 0)})\right) \cdot \frac {\partial Q (\boldsymbol {W} ; \boldsymbol {s} _ {n} , a _ {n})}{\partial \boldsymbol {w} _ {\ell , k}} \right] \\ + \mathbb {E} _ {(\boldsymbol {s}, a) \sim \mu_ {t}} g _ {t} (\boldsymbol {w} _ {\ell , k}; \boldsymbol {W}) - \frac {\partial f}{\partial \boldsymbol {w} _ {\ell , k}} (\boldsymbol {W}) \\ + \mathbb {E} _ {(\boldsymbol {s}, a) \sim \mathcal {D} _ {t}, \mathcal {P}} \left[ g _ {t} \left(\boldsymbol {w} _ {\ell , k}; \boldsymbol {W}\right) - \mathbb {E} _ {(\boldsymbol {s}, a) \sim \mu_ {t}, \mathcal {P}} g _ {t} \left(\boldsymbol {w} _ {\ell , k}; \boldsymbol {W}\right) \right] \\ := I _ {1} + I _ {2} + I _ {3} + I _ {4}. \\ \end{array}
$$
Bound of  $I_{1}$  and  $I_{2}$ . Compared with (91), the upper bound for  $I_{1}$  and  $I_{2}$  is the same as those shown in (103) and (109), respectively.
Bound of  $I_{3}$ . Following (111), the upper bound of  $I_{3}$  can be characterized as
$$
\begin{array}{l} \| I _ {3} \| _ {2} \leq \frac {R _ {\max }}{1 - \gamma} \cdot \left[ (1 - \varepsilon) \cdot \left| \int_ {(s, a)} \int_ {s ^ {\prime}} \left(\mu^ {\star} (d s, d a) \mathcal {P} (d s ^ {\prime} | s, a) - \mu_ {t, 1} (d s, d a) \mathcal {P} (d s ^ {\prime} | s, a)\right) \right| \right. \tag {156} \\ + \varepsilon \cdot \left| \int_ {(\boldsymbol {s}, a)} \int_ {\boldsymbol {s} ^ {\prime}} \left(\mu^ {\star} (d \boldsymbol {s}, d a) \mathcal {P} (d \boldsymbol {s} ^ {\prime} | \boldsymbol {s}, a) - \mu_ {t, 2} (d \boldsymbol {s}, d a) \mathcal {P} (d \boldsymbol {s} ^ {\prime} | \boldsymbol {s}, a)\right) \right]. \\ \end{array}
$$
$$
\begin{array}{l} \left| \int_ {(s, a)} \int_ {s ^ {\prime}} \left(\mu^ {\star} (d s, d a) \mathcal {P} (d s ^ {\prime} | s, a) - \mu_ {t, 1} (d s, d a) \mathcal {P} (d s ^ {\prime} | s, a)\right) \right| \\ = \left| \int_ {(s, a)} \int_ {s ^ {\prime}} \left(\mathcal {P} ^ {\star} (d s) \pi^ {\star} (d a | s) \mathcal {P} (d s ^ {\prime} | s, a) - \mathcal {P} _ {t, 1} (d s) \pi_ {t, 1} (d a | d s) \mathcal {P} (d s ^ {\prime} | s, a)\right) \right| \tag {157} \\ \leq \left| \int_ {(\boldsymbol {s}, a)} \int_ {\boldsymbol {s} ^ {\prime}} \left(\mathcal {P} ^ {\star} (d \boldsymbol {s}) - \mathcal {P} _ {t, 1} (d \boldsymbol {s})\right) \pi^ {\star} (d a | \boldsymbol {s}) \mathcal {P} (d \boldsymbol {s} ^ {\prime} | \boldsymbol {s}, a) \right| \\ + \left| \int_ {(\boldsymbol {s}, a)} \int_ {\boldsymbol {s} ^ {\prime}} \mathcal {P} _ {t, 1} (d \boldsymbol {s}) \left(\pi_ {t, 1} (d a | d \boldsymbol {s}) - \pi^ {\star} (d a | d \boldsymbol {s})\right) \mathcal {P} (d \boldsymbol {s} ^ {\prime} | \boldsymbol {s}, a) \right|. \\ \end{array}
$$
From Theorem 3.1 in [49], we know that
$$
\left| \int_ {(s, a)} \left(\mathcal {P} ^ {\star} (d s) - \mathcal {P} _ {t, 1} (d s)\right) \right| \leq | \mathcal {A} | \left(\log_ {\nu} \lambda^ {- 1} + \frac {1}{1 - \nu}\right) C _ {t} \tag {158}
$$
$$
\text {a n d} \quad \| \pi_ {t, 1} (d a | d s) - \pi^ {\star} (d a | d s) \| \leq C _ {t}.
$$
$$
\begin{array}{l} \left\| I _ {3} \right\| _ {2} \leq \frac {R _ {\max }}{1 - \gamma} \cdot | \mathcal {A} | \cdot \left((1 - \varepsilon) C _ {t} + \varepsilon \cdot C _ {t}\right) \cdot \left(1 + \log_ {\nu} \lambda^ {- 1} + \frac {1}{1 - \nu}\right) \tag {159} \\ = C _ {d} \cdot \left(C _ {t} + (1 - C _ {t}) \varepsilon\right) \cdot \frac {R _ {\max}}{1 - \gamma}, \\ \end{array}
$$
where  $C_d = |\mathcal{A}|\cdot \left(1 + \log_\nu \lambda^{-1} + \frac{1}{1 - \nu}\right)$
Bound of  $I_4$ .  $I_4$  is the bias of the data because the data  $(s, a)$  at iteration  $t$  depends on the neural network parameters  $W$ . Let us define  $\bar{g}_t$  as
$$
\bar {g} _ {t} \left(\boldsymbol {w} _ {\ell , k}; \boldsymbol {W}\right) = \mathbb {E} _ {\mu_ {t}, \mathcal {P}} g _ {t} \left(\boldsymbol {w} _ {\ell , k}; \boldsymbol {W}\right) \tag {160}
$$
$$
\Delta_ {t} = g _ {t} \left(\boldsymbol {w} _ {\ell , k}; \boldsymbol {W}\right) - \bar {g} _ {t} \left(\boldsymbol {w} _ {\ell , k}; \boldsymbol {W}\right). \tag {161}
$$
It is easy to verify that
$$
\left\| g _ {t} \left(\boldsymbol {w} _ {\ell , k}; \boldsymbol {W}\right) - g _ {t} \left(\widetilde {\boldsymbol {w}} _ {\ell , k}; \widetilde {\boldsymbol {W}}\right) \right\| \leq (1 + \gamma) \cdot \left\| \boldsymbol {W} - \widetilde {\boldsymbol {W}} \right\|,
$$
$$
\left\| \bar {g} _ {t} \left(\boldsymbol {w} _ {\ell , k}; \boldsymbol {W}\right) - \bar {g} _ {t} \left(\tilde {\boldsymbol {w}} _ {\ell , k}; \widetilde {\boldsymbol {W}}\right) \right\| \leq (1 + \gamma) \cdot \left\| \boldsymbol {W} - \widetilde {\boldsymbol {W}} \right\|, \tag {162}
$$
$$
a n d \quad \| g _ {t} \| \lesssim \frac {R _ {\max }}{1 - \gamma}.
$$
$$
\Delta_ {t} (\boldsymbol {W}) - \Delta_ {t} (\widetilde {\boldsymbol {W}}) \lesssim (1 + \gamma) \cdot \| \boldsymbol {W} - \widetilde {\boldsymbol {W}} \| _ {2}. \tag {163}
$$
$$
\Delta_ {t} \left(\boldsymbol {W} ^ {(t, 0)}\right) \leq \Delta_ {t} \left(\boldsymbol {W} ^ {(t - \tau , 0)}\right) + \frac {1 + \gamma}{1 - \gamma} \cdot R _ {\max } \cdot \sum_ {i = t - \tau} ^ {t - 1} \eta_ {i}. \tag {164}
$$
Then, we need to bound  $\delta_t(\pmb{W}^{(t - \tau, 0)})$ .
Let us define the observed tuple  $O_{t}(s, a, s')$  as the collection of the state, action, and the next state at the  $t$ -th outer loop. Note that
$$
\boldsymbol {W} ^ {(t - \tau , 0)} \longrightarrow \boldsymbol {s} _ {t - \tau} \longrightarrow \boldsymbol {s} _ {t} \longrightarrow O _ {t} \tag {165}
$$
forms a Markov chain introduced by the policy  $\pi_{t - \tau}$
Let  $\widetilde{\pmb{W}}^{(t - \tau ,0)}$  and  $\widetilde{O}_t$  be independently drawn from the marginal distributions of  $\widetilde{\pmb{W}}^{(t - \tau ,0)}$  and  $O_{t}$  respectively.
With Lemma 9 in [4], we have
$$
\mathbb {E} \Delta_ {t} \left(\boldsymbol {W} ^ {(t - \tau , 0)}, O _ {t}\right) - \mathbb {E} \Delta_ {t} \left(\widetilde {\boldsymbol {W}} ^ {(t - \tau , 0)}, \widetilde {O} _ {t}\right) \lesssim 2 \sup  _ {\boldsymbol {w}, O} | \Delta_ {t} (\boldsymbol {W}, O) | \cdot \lambda \cdot \nu^ {\tau}. \tag {166}
$$
By definition, we have  $\mathbb{E}\Delta_t(\widetilde{\boldsymbol{W}}^{(t - \tau ,0)},\widetilde{\boldsymbol{O}}_t) = 0$  and
$$
\left| \Delta_ {t} (\boldsymbol {W}, O) \right| \leq \frac {2 R _ {\max }}{1 - \gamma}. \tag {167}
$$
$$
\begin{array}{l} \mathbb {E} \Delta_ {t} \left(\boldsymbol {W} ^ {(t, 0)}\right) \leq \mathbb {E} \Delta_ {t} \left(\boldsymbol {W} ^ {(t - \tau , 0)}\right) + \frac {1 + \gamma}{1 - \gamma} \cdot R _ {\max } \cdot \sum_ {i = t - \tau} ^ {t - 1} \eta_ {i} \tag {168} \\ \leq \frac {R _ {\max }}{1 - \gamma} \left(\lambda \cdot \nu^ {\tau} + (1 + \gamma) \cdot \tau \cdot \eta_ {t - \tau}\right), \\ \end{array}
$$
where the last inequality comes from the fact that the step size  $\eta_t$  is non-increasing.
Choose  $\tau^{\star} = \min \left\{t = 0,1,2,\dots \mid \lambda \nu^{\tau}\leq \eta_{T}\right\}$ . When  $t\leq \tau^{\star}$ , we choose  $\tau = t$  and have
$$
\mathbb {E} \Delta_ {t} \left(\boldsymbol {W} ^ {(t, 0)}\right) \leq \frac {R _ {\max }}{1 - \gamma} \cdot \tau^ {\star} \cdot \eta_ {0}. \tag {169}
$$
When  $t > \tau^{\star}$ , we can choose  $\tau = \tau^{\star}$  and obtain