SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma 10 (Lemma 5.2, [74]). Let  $\mathcal{B}(0,1) \in \{\alpha \mid \| \alpha \|_2 = 1, \alpha \in \mathbb{R}^d\}$  denote a unit ball in  $\mathbb{R}^d$ . Then, a subset  $S_{\xi}$  is called a  $\xi$ -net of  $\mathcal{B}(0,1)$  if every point  $z \in \mathcal{B}(0,1)$  can be approximated to within  $\xi$  by some point  $\alpha \in \mathcal{B}(0,1)$ , i.e.,  $\| z - \alpha \|_2 \leq \xi$ . Then the minimal cardinality of a  $\xi$ -net  $S_{\xi}$  satisfies
$$
\left| \mathcal {S} _ {\xi} \right| \leq (1 + 2 / \xi) ^ {d}. \tag {86}
$$
Lemma 11 (Lemma 5.3, [74]). Let  $\mathbf{A}$  be an  $d_1 \times d_2$  matrix, and let  $S_{\xi}(d)$  be a  $\xi$ -net of  $\mathcal{B}(0,1)$  in  $\mathbb{R}^d$  for some  $\xi \in (0,1)$ . Then
$$
\left\| \boldsymbol {A} \right\| _ {2} \leq (1 - \xi) ^ {- 1} \max  _ {\boldsymbol {\alpha} _ {1} \in \mathcal {S} _ {\xi} (d _ {1}), \boldsymbol {\alpha} _ {2} \in \mathcal {S} _ {\xi} (d _ {2})} | \boldsymbol {\alpha} _ {1} ^ {\top} \boldsymbol {A} \boldsymbol {\alpha} _ {2} |. \tag {87}
$$
Proof of Lemma 3. From (7), we know that
$$
\begin{array}{l} g _ {t} (\boldsymbol {w} _ {\ell , k}; \boldsymbol {W}) \\ = \frac {1}{N} \sum_ {n = 1} ^ {N} \left(Q (\boldsymbol {W}; \boldsymbol {s} _ {n}, a _ {n}) - y _ {n} ^ {(t)}\right) \cdot \frac {\partial Q (\boldsymbol {W} ; \boldsymbol {s} _ {n} , a _ {n})}{\partial \boldsymbol {w} _ {\ell , k}} \\ = \frac {1}{N} \sum_ {n = 1} ^ {N} \left(Q (\boldsymbol {W}; \boldsymbol {s} _ {n}, a _ {n}) - Q (\boldsymbol {W} ^ {\star}; \boldsymbol {s} _ {n}, a _ {n}) + \gamma \cdot \max  _ {a} Q (\boldsymbol {s} _ {n}, a; \boldsymbol {W} ^ {\star}) \right. \\ \left. - \gamma \cdot \max  _ {a} Q \left(\boldsymbol {s} _ {n}, a; \boldsymbol {W} ^ {(t, 0)}\right)\right) \cdot \frac {\partial Q (\boldsymbol {W} ; \boldsymbol {s} _ {n} , a _ {n})}{\partial \boldsymbol {w} _ {\ell , k}} \tag {88} \\ = \frac {1}{N} \sum_ {n = 1} ^ {N} \left(Q (\boldsymbol {W}; \boldsymbol {s} _ {n}, a _ {n}) - Q (\boldsymbol {W} ^ {\star}; \boldsymbol {s} _ {n}, a _ {n})\right) \cdot \frac {\partial Q (\boldsymbol {W} ; \boldsymbol {s} _ {n} , a _ {n})}{\partial \boldsymbol {w} _ {\ell , k}} \\ + \frac {1}{N} \sum_ {n = 1} ^ {N} \gamma \cdot \left(\max  _ {a} Q (\boldsymbol {s} _ {n}, a; \boldsymbol {W} ^ {\star}) - \max  _ {a} Q (\boldsymbol {s} _ {n}, a; \boldsymbol {W} ^ {(t, 0)})\right) \cdot \frac {\partial Q (\boldsymbol {W} ; \boldsymbol {s} _ {n} , a _ {n})}{\partial \boldsymbol {w} _ {\ell , k}}. \\ \end{array}
$$
From (31), we know that
$$
\frac {\partial f}{\partial \boldsymbol {w} _ {\ell , k}} (\boldsymbol {W}) = \mathbb {E} _ {(\boldsymbol {s}, a) \sim \mu^ {\star}} \left(Q (\boldsymbol {W}; \boldsymbol {s}, a) - Q (\boldsymbol {W} ^ {\star}; \boldsymbol {s}, a)\right) \cdot \frac {\partial Q (\boldsymbol {W} ; \boldsymbol {s} , a)}{\partial \boldsymbol {w} _ {\ell , k}}. \tag {89}
$$
Then, from (88) and (89), we have
$$
\begin{array}{l} g _ {t} (\pmb {w} _ {\ell , k}; \pmb {W}) - \frac {\partial f}{\partial \pmb {w} _ {\ell , k}} (\pmb {W}) = g _ {t} (\pmb {w} _ {\ell , k}; \pmb {W}) - \mathbb {E} _ {(\pmb {s}, a) \sim \mathcal {D} _ {t}} g _ {t} (\pmb {w} _ {\ell , k}; \pmb {W}) \\ + \mathbb {E} _ {(\boldsymbol {s}, a) \sim \mu_ {t}} g _ {t} (\boldsymbol {w} _ {\ell , k}; \boldsymbol {W}) - \frac {\partial f}{\partial \boldsymbol {w} _ {\ell , k}} (\boldsymbol {W}), \tag {90} \\ \end{array}
$$
where  $\mathcal{D}_t$  and  $\mu_t$  are equivalent because of Assumption 2. Then, we have
$$
\begin{array}{l} g _ {t} (\boldsymbol {w} _ {\ell , k}; \boldsymbol {W}) - \frac {\partial f}{\partial \boldsymbol {w} _ {\ell , k}} (\boldsymbol {W}) \\ = \left[ \frac {1}{N} \sum_ {n = 1} ^ {N} \left(Q (\boldsymbol {W}; \boldsymbol {s} _ {n}, a _ {n}) - Q (\boldsymbol {W} ^ {\star}; \boldsymbol {s} _ {n}, a _ {n})\right) \cdot \frac {\partial Q (\boldsymbol {W} ; \boldsymbol {s} _ {n} , a _ {n})}{\partial \boldsymbol {w} _ {\ell , k}} \right. \\ \left. - \mathbb {E} _ {(\boldsymbol {s}, a) \sim \mu_ {t}} \left(Q (\boldsymbol {W}; \boldsymbol {s}, a) - Q (\boldsymbol {W} ^ {\star}; \boldsymbol {s}, a)\right) \cdot \frac {\partial Q (\boldsymbol {W} ; \boldsymbol {s} , a)}{\partial \boldsymbol {w} _ {\ell , k}} \right] \tag {91} \\ + \left[ \frac {1}{N} \sum_ {n = 1} ^ {N} \gamma \cdot \left(\max  _ {a} Q (\boldsymbol {s} _ {n}, a; \boldsymbol {W} ^ {\star}) - \max  _ {a} Q (\boldsymbol {s} _ {n}, a; \boldsymbol {W} ^ {(t, 0)})\right) \cdot \frac {\partial Q (\boldsymbol {W} ; \boldsymbol {s} _ {n} , a _ {n})}{\partial \boldsymbol {w} _ {\ell , k}} \right] \\ + \mathbb {E} _ {(\boldsymbol {s}, a) \sim \mu_ {t}} g _ {t} (\boldsymbol {w} _ {\ell , k}; \boldsymbol {W}) - \frac {\partial f}{\partial \boldsymbol {w} _ {\ell , k}} (\boldsymbol {W}). \\ \end{array}
$$
For convenience, we define  $I_1, I_2$ , and  $I_3$  in the following ways with  $\boldsymbol{x}_n \coloneqq (s_n, a_n)$  be the feature mapping of state-action pair  $(s_n, a_n)$ .
Then,  $I_{1}$  is defined as
$$
\begin{array}{l} \boldsymbol {I} _ {1} := \frac {1}{N} \sum_ {n = 1} ^ {N} \left(Q (\boldsymbol {W}; \boldsymbol {s} _ {n}, a _ {n}) - Q \left(\boldsymbol {W} ^ {\star}; \boldsymbol {s} _ {n}, a _ {n}\right)\right) \cdot \frac {\partial Q (\boldsymbol {W} ; \boldsymbol {s} _ {n} , a _ {n})}{\partial \boldsymbol {w} _ {\ell , k}} \tag {92} \\ - \mathbb {E} _ {(\boldsymbol {s}, a) \sim \mathcal {D} _ {t}} \left(Q (\boldsymbol {W}; \boldsymbol {s}, a) - Q (\boldsymbol {W} ^ {\star}; \boldsymbol {s}, a)\right) \cdot \frac {\partial Q (\boldsymbol {W} ; \boldsymbol {s} , a)}{\partial \boldsymbol {w} _ {\ell , k}}, \\ \end{array}
$$
$I_{2}$  is defined as
$$
\boldsymbol {I} _ {2} := \frac {1}{N} \sum_ {n = 1} ^ {N} \gamma \cdot \left(\max  _ {a} Q \left(\boldsymbol {s} _ {n} ^ {\prime}, a; \boldsymbol {W} ^ {\star}\right) - \max  _ {a} Q \left(\boldsymbol {s} _ {n} ^ {\prime}, a; \boldsymbol {W} ^ {(t, 0)}\right)\right) \cdot \frac {\partial Q (\boldsymbol {W} ; \boldsymbol {s} _ {n} , a _ {n})}{\partial \boldsymbol {w} _ {\ell , k}}, \tag {93}
$$
and  $I_{3}$  is defined as
$$
\boldsymbol {I} _ {3} := \mathbb {E} _ {(\boldsymbol {s}, a) \sim \mu_ {t}} g _ {t} (\boldsymbol {w} _ {\ell , k}; \boldsymbol {W}) - \frac {\partial f}{\partial \boldsymbol {w} _ {\ell , k}} (\boldsymbol {W}), \tag {94}
$$
$$
\frac {\partial Q (\boldsymbol {W} ; \boldsymbol {s} _ {n} , \boldsymbol {a} _ {n})}{\partial \boldsymbol {w} _ {\ell , k}} = \frac {1}{K} \mathcal {J} _ {\ell , k} \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , k} ^ {\top} \boldsymbol {h} ^ {\ell}\right) \boldsymbol {h} ^ {\ell} (\boldsymbol {W}) \tag {95}
$$
from (43). Therefore, we have
$$
\left\| g _ {t} \left(\boldsymbol {w} _ {\ell , k}; \boldsymbol {W}\right) - \frac {\partial f _ {t}}{\partial \boldsymbol {w} _ {\ell , k}} (\boldsymbol {W}) \right\| _ {2} \leq \| \boldsymbol {I} _ {1} \| _ {2} + \| \boldsymbol {I} _ {2} \| _ {2} + \| \boldsymbol {I} _ {3} \| _ {2}. \tag {96}
$$
Next, we will provide the bound for  $\| I_1\| _2,\| I_2\| _2$  , and  $\| I_3\| _2$
Bound of  $I_{1}$ . We first divide the data in  $\mathcal{D}_t$  into two parts, namely,  $\mathcal{D}_{t,1}$  and  $\mathcal{D}_{t,2}$ .  $\mathcal{D}_{t,1}$  includes the state-action pair  $(s,a)$  such that  $a_{n}$  is randomly selected from action space  $\mathcal{A}$ , and  $\mathcal{D}_{t,2}$  includes the state-action pair  $(s,a)$  such that  $a_{n}$  is selected based on the greedy policy with respect to  $Q(\pmb{W}^{(t,0)})$ .
Then, we define a random variable  $Z^{(\ell,1)} = \left(Q(\boldsymbol{x};\boldsymbol{W}) - Q(\boldsymbol{x};\boldsymbol{W}^{\star})\right) \cdot \mathcal{J}_{\ell,k} \cdot \boldsymbol{\alpha}^{T}\boldsymbol{h}^{(\ell)}(\boldsymbol{W})$  with  $\boldsymbol{x} \sim \mathcal{D}_{t,1}$  and  $Z_{n}^{(\ell,1)} = \left(Q(\boldsymbol{x}_{n};\boldsymbol{W}) - Q(\boldsymbol{x}_{n};\boldsymbol{W}^{\star})\right) \cdot \mathcal{J}_{\ell,k} \cdot \boldsymbol{\alpha}^{T}\boldsymbol{h}_{n}^{(\ell)}(\boldsymbol{W})$  as the realization of  $Z_{\ell}^{(1)}$  for  $n = 1,2,\dots,N$ , where  $\boldsymbol{\alpha} \in \mathbb{R}^{d}$  is any fixed unit vector with  $\| \boldsymbol{\alpha}\|_{2} \leq 1$ . We know that  $s$  and  $a$  are independent for  $\boldsymbol{x} \sim \mathcal{D}_{t,1}$ . Let  $\Sigma_{1}$  denote the covariance matrix of  $\boldsymbol{x} \sim \mathcal{D}_{t,1}$ . Moreover,  $\boldsymbol{x}(s,a)$  is bounded by 1, then we have  $\| \Sigma_{1}\|_{2} \leq 1$ .
Similar to  $Z^{(\ell,1)}$ , we define a random variable  $Z^{(\ell,2)} = (Q(\boldsymbol{x};\boldsymbol{W}) - Q(\boldsymbol{x};\boldsymbol{W}^{\star})) \cdot \mathcal{J}_{\ell,k} \cdot \boldsymbol{\alpha}^T \boldsymbol{h}^{(\ell)}(\boldsymbol{W})$  with  $\boldsymbol{x} \sim \mathcal{D}_{t,2}$  and  $Z_{n}^{(\ell,2)} = (Q(\boldsymbol{x}_{n};\boldsymbol{W}) - Q(\boldsymbol{x}_{n};\boldsymbol{W}^{\star})) \cdot \mathcal{J}_{\ell,k} \cdot \boldsymbol{\alpha}^T \boldsymbol{h}_{n}^{(\ell)}(\boldsymbol{W})$  as the realization of  $Z^{(\ell,2)}$  for  $n = 1,2,\dots,N$ . Differ from  $Z^{(\ell,1)}$ ,  $s$  and  $a$  are dependent for  $\boldsymbol{x} \sim \mathcal{D}_{t,2}$ . Let  $\Sigma_{2}$  denote the covariance matrix of  $\boldsymbol{x} \sim \mathcal{D}_{t,1}$ . Then, we have  $\| \Sigma_{2}\|_{2} \leq 1 + \max_{j} \rho_{x_{j},a} \leq 2$ , where  $\rho_{x_{j},a}$  denotes the correlation between  $a$  and  $x_{j}$ .
According to the definition of (92), we can rewrite  $I_{1}$  as
Then, for any  $p\in \mathbb{N}^+$  , we have
$$
\begin{array}{l} \left(\mathbb {E} \left| Z ^ {(1)} \right| ^ {p}\right) ^ {1 / p} = \left(\mathbb {E} _ {\boldsymbol {x} \sim \mathcal {D} _ {t, 1}} \left| Q (\boldsymbol {W}; \boldsymbol {x}) - Q (\boldsymbol {W} ^ {\star}; \boldsymbol {x}) \right| ^ {p} \cdot \left| \mathcal {J} _ {\ell , k} \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , k} ^ {\top} \boldsymbol {x}\right) \right| \cdot \left| \boldsymbol {\alpha} ^ {T} \boldsymbol {h} ^ {\ell} \right| ^ {p}\right) ^ {1 / p} \\ \leq \left(\mathbb {E} _ {\boldsymbol {x} \sim \mathcal {D} _ {t, 1}} | Q (\boldsymbol {W}; \boldsymbol {x}) - Q (\boldsymbol {W} ^ {\star}; \boldsymbol {x}) | ^ {p} \cdot | \boldsymbol {\alpha} ^ {T} \boldsymbol {h} ^ {\ell} | ^ {p}\right) ^ {1 / p} \tag {98} \\ \leq \left(\mathbb {E} _ {\boldsymbol {x} \sim \mathcal {D} _ {t, 1}} \left| \| \boldsymbol {W} - \boldsymbol {W} ^ {\star} \| _ {2} \cdot \| \boldsymbol {x} \| _ {2} \right| ^ {p} \cdot | \boldsymbol {\alpha} ^ {T} \boldsymbol {x} | ^ {p}\right) ^ {1 / p} \\ \leq C _ {1} \cdot \| \boldsymbol {W} - \boldsymbol {W} ^ {\star} \| _ {2} \cdot p \\ \end{array}
$$
where  $C_1$  is a positive constant.
From Definition 4, we know that  $Z^{(\ell,1)}$  belongs to sub-exponential distribution with  $\| Z^{(\ell,1)} \|_{\psi_1} \leq C_1 \| W - W^{\star} \|_2$ . Therefore, by Chernoff inequality, we have
$$
\mathbb {P} \left\{\left| \frac {1}{N} \sum_ {n = 1} ^ {N} Z _ {n} ^ {(\ell , 1)} (j) - \mathbb {E} Z ^ {(\ell , 1)} (j) \right| <   t \right\} \leq 1 - \frac {e ^ {- C \left(C _ {1} \| \boldsymbol {W} - \boldsymbol {W} ^ {\star} \| _ {2}\right) ^ {2} \cdot N s ^ {2}}}{e ^ {N s t}} \tag {99}
$$
for some positive constant  $C$  and any  $s \in \mathbb{R}$ .
Let  $t = C_1\| \pmb {W} - \pmb{W}^\star \| _2\sqrt{\frac{d\log q}{N}}$  and  $s = \frac{2}{C\|\pmb{W} - \pmb{W}^\star\|_2}\cdot t$  for some large constant  $q > 0$ . Then, we have
$$
\left| \frac {1}{N} \sum_ {n = 1} ^ {N} Z _ {n} ^ {(\ell , 1)} (j) - \mathbb {E} Z ^ {(\ell , 1)} (j) \right| \lesssim C _ {1} \| \boldsymbol {W} - \boldsymbol {W} ^ {\star} \| _ {2} \cdot \sqrt {\frac {d \log q}{N}} \tag {100}
$$
with probability at least  $1 - q^{-d}$
Similar to (98), we have
$$
\left(\mathbb {E} \left| Z ^ {(\ell , 2)} \right| ^ {p}\right) ^ {1 / p} \leq C _ {2} \cdot \left\| \boldsymbol {W} - \boldsymbol {W} ^ {\star} \right\| _ {2} \cdot p, \tag {101}
$$
where  $C_2 = 2\cdot C_1$  . Then, we have
$$
\left| \frac {1}{N} \sum_ {n = 1} ^ {N} Z _ {n} ^ {(\ell , 2)} (j) - \mathbb {E} Z ^ {(\ell , 2)} (j) \right| \lesssim 2 C _ {1} \| \boldsymbol {W} - \boldsymbol {W} ^ {\star} \| _ {2} \cdot \sqrt {\frac {d \log q}{N}} \tag {102}
$$
with probability at least  $1 - q^{-d}$ .
From Lemma 11 and (97), we have
$$
\begin{array}{l} \| I _ {1} \| _ {2} \leq 2 \cdot \frac {1}{K ^ {2}} \left[ \varepsilon \cdot \left| \frac {1}{\varepsilon N} \sum_ {n \in \mathcal {D} _ {t, 1}} Z _ {n} ^ {(\ell , 1)} (j) - \mathbb {E} Z ^ {(\ell , 1)} (j) \right| \right. \\ + (1 - \varepsilon) \cdot \left| \frac {1}{(1 - \varepsilon) N} \sum_ {n \in \mathcal {D} _ {t, 2}} Z _ {n} ^ {(\ell , 2)} (j) - \mathbb {E} Z ^ {(\ell , 2)} (j) \right| \tag {103} \\ \lesssim \frac {2 - \varepsilon}{K ^ {2}} \| \boldsymbol {W} - \boldsymbol {W} ^ {\star} \| _ {2} \cdot \sqrt {\frac {d \log q}{N}} \\ \end{array}
$$
with probability at least  $1 - |\mathcal{S}_{\frac{1}{2}}(d)| \cdot q^{-d}$ .
From Lemma 10, we know that  $|S_{\frac{1}{2}}(d)| \leq 5^d$ . Therefore, the probability for (103) holds is at least  $1 - \left(\frac{q}{5}\right)^{-d}$ . Because  $q \gg 5$ , we denote the probability as  $1 - q^{-d}$  for convenience.
Bound of  $I_{2}$ . Let  $a_{n}^{\star} = \arg \max_{a\in \mathcal{A}}Q(\pmb{W}^{\star};\pmb{s}_{n}^{\prime},\pmb {a})$ . While for  $Q(W)$ , we have
$$
\max  _ {a} Q \left(\boldsymbol {W}; \boldsymbol {s} _ {n} ^ {\prime}, \boldsymbol {a}\right) \geq Q \left(\boldsymbol {W}; \boldsymbol {s} _ {n} ^ {\prime}, \boldsymbol {a} ^ {\star}\right). \tag {104}
$$
$$
\begin{array}{l} \max  _ {a} Q \left(\boldsymbol {W} ^ {\star}; \boldsymbol {s} _ {n} ^ {\prime}, \boldsymbol {a}\right) - \max  _ {a} Q \left(\boldsymbol {W}; \boldsymbol {s} _ {n} ^ {\prime}, \boldsymbol {a}\right) = Q \left(\boldsymbol {W} ^ {\star}; \boldsymbol {s} _ {n} ^ {\prime}, \boldsymbol {a} _ {n} ^ {\star}\right) - \max  _ {a} Q \left(\boldsymbol {W}; \boldsymbol {s} _ {n} ^ {\prime}, \boldsymbol {a}\right) \tag {105} \\ \leq Q \left(\boldsymbol {W} ^ {\star}; \boldsymbol {s} _ {n} ^ {\prime}, \boldsymbol {a} _ {n} ^ {\star}\right) - Q \left(\boldsymbol {W}; \boldsymbol {s} _ {n} ^ {\prime}, \boldsymbol {a} _ {n} ^ {\star}\right). \\ \end{array}
$$
Similarly to (105), let us define  $\tilde{a}_n^\star = \arg \max_a Q(\pmb {W};\pmb {s}_n,\pmb {a})$  . Then, we have
$$
\max  _ {a} Q \left(\boldsymbol {W} ^ {\star}; \boldsymbol {s} _ {n} ^ {\prime}, \boldsymbol {a}\right) - \max  _ {a} Q \left(\boldsymbol {W}; \boldsymbol {s} _ {n} ^ {\prime}, \boldsymbol {a}\right) \geq Q \left(\boldsymbol {W} ^ {\star}; \boldsymbol {s} _ {n} ^ {\prime}, \tilde {\boldsymbol {a}} _ {n} ^ {\star}\right) - Q \left(\boldsymbol {W}; \boldsymbol {s} _ {n} ^ {\prime}, \tilde {\boldsymbol {a}} _ {n} ^ {\star}\right). \tag {106}
$$
Combining (105) and (106), we have
$$
\left| \max  _ {a} Q \left(\boldsymbol {W} ^ {\star}; \boldsymbol {s} _ {n} ^ {\prime}, \boldsymbol {a}\right) - \max  _ {a} Q \left(\boldsymbol {W}; \boldsymbol {s} _ {n} ^ {\prime}, \boldsymbol {a}\right) \right| \leq \max  _ {a} \left| Q \left(\boldsymbol {W} ^ {\star}; \boldsymbol {s} _ {n} ^ {\prime}, \boldsymbol {a}\right) - Q \left(\boldsymbol {W}; \boldsymbol {s} _ {n} ^ {\prime}, \boldsymbol {a}\right) \right|. \tag {107}
$$
Following the definition of  $Z^{(\ell,1)}$  in (98), we define
$$
Z ^ {(\ell , 3)} (j) = \left(\max  _ {a} Q (\boldsymbol {W} ^ {\star}; \boldsymbol {s} _ {n} ^ {\prime}, \boldsymbol {a}) - \max  _ {a} Q (\boldsymbol {W}; \boldsymbol {s} _ {n} ^ {\prime}, \boldsymbol {a})\right) \cdot \mathcal {J} _ {\ell , k} \phi^ {\prime} (\boldsymbol {w} _ {\ell , k} ^ {\top} \boldsymbol {h} ^ {(\ell)}) \cdot \boldsymbol {\alpha} ^ {\top} \boldsymbol {h} ^ {(\ell)}.
$$
Therefore, from (105) and (106), we know
$$
\begin{array}{l} (\mathbb {E} | Z ^ {(3)} | ^ {p}) ^ {1 / p} \leq \left(\mathbb {E} _ {\boldsymbol {x} \sim \mathcal {D} _ {t}} \left| \max  _ {a} Q (\boldsymbol {W} ^ {\star}; \boldsymbol {s} _ {n} ^ {\prime}, \boldsymbol {a}) - \max  _ {a} Q (\boldsymbol {W}; \boldsymbol {s} _ {n} ^ {\prime}, \boldsymbol {a}) \right| ^ {p} \right. \\ \left. \cdot \left| \mathcal {J} _ {\ell , k} \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , k} ^ {\top} \boldsymbol {h} ^ {(\ell)}\right) \right| ^ {p} \cdot \left| \boldsymbol {\alpha} ^ {\top} \boldsymbol {h} _ {n} ^ {(\ell)} \right| ^ {p}\right) ^ {1 / p} \tag {108} \\ \leq \left(\mathbb {E} _ {\boldsymbol {x} \sim \mathcal {D} _ {t}} \max  _ {a} \left| Q (\boldsymbol {W} ^ {\star}; \boldsymbol {s} _ {n} ^ {\prime}, \boldsymbol {a}) - Q (\boldsymbol {W}; \boldsymbol {s} _ {n} ^ {\prime}, \boldsymbol {a}) \right| ^ {p} \cdot | \boldsymbol {\alpha} ^ {\top} \boldsymbol {h} _ {n} ^ {(\ell)} | ^ {p}\right) ^ {1 / p} \\ \lesssim (2 - \varepsilon) \cdot \left\| \boldsymbol {W} - \boldsymbol {W} ^ {\star} \right\| _ {2} \cdot \log | \mathcal {A} | \cdot p. \\ \end{array}
$$
Following the steps in (98) to (100), we have
$$
\begin{array}{l} \left\| I _ {2} \right\| _ {2} \lesssim \frac {(1 - \varepsilon / 2) \gamma}{K} \cdot \left(\left\| W - W ^ {\star} \right\| _ {2} \cdot \sqrt {\frac {d \cdot \log q \cdot \log | \mathcal {A} |}{N}} + \mathbb {E} Z ^ {(\ell , 3)}\right) \\ \lesssim \frac {(1 - \varepsilon / 2) \gamma}{K} \cdot \left(\| \boldsymbol {W} - \boldsymbol {W} ^ {\star} \| _ {2} \cdot \left(\sqrt {\frac {d \cdot \log q \cdot \log | \mathcal {A} |}{N}} + C\right) \right. \tag {109} \\ \lesssim \frac {(1 - \varepsilon / 2) \gamma}{K} \cdot \| \boldsymbol {W} - \boldsymbol {W} ^ {\star} \| _ {2} \\ \end{array}
$$
with probability at least  $1 - q^{-d}$ , where the last inequality holds when  $N \gtrsim d \cdot \log q \cdot \log |\mathcal{A}|$ .
Bound of  $I_{3}$ . We have
$I_{3}$
$$
\begin{array}{l} = \mathbb {E} _ {(\boldsymbol {s}, a) \sim \mu_ {t}} g _ {t} (\boldsymbol {w} _ {\ell , k}; \boldsymbol {W}) - \frac {\partial f}{\partial \boldsymbol {w} _ {\ell , k}} (\boldsymbol {W}) \\ = \mathbb {E} _ {\left(\boldsymbol {s}, a\right) \sim \mu_ {t}} \left(Q (\boldsymbol {W}; \boldsymbol {s}, a) - Q \left(\boldsymbol {W} ^ {\star}; \boldsymbol {s}, a\right)\right) \cdot \frac {\partial Q (\boldsymbol {W} ; \boldsymbol {s} , a)}{\partial \boldsymbol {w} _ {\ell , k}} \\ - \mathbb {E} _ {(\boldsymbol {s}, a) \sim \mu^ {\star}} \left(Q (\boldsymbol {W}; \boldsymbol {s}, a) - Q (\boldsymbol {W} ^ {\star}; \boldsymbol {s}, a)\right) \cdot \frac {\partial Q (\boldsymbol {W} ; \boldsymbol {s} , a)}{\partial \boldsymbol {w} _ {\ell , k}} \\ = \mathbb {E} _ {\left(\boldsymbol {s}, a\right) \sim \mu_ {t}} \left(Q (\boldsymbol {W}; \boldsymbol {s}, a) - r (\boldsymbol {s}, a) - \gamma \cdot \mathbb {E} _ {\boldsymbol {s} ^ {\prime} \sim p _ {\boldsymbol {s}, \boldsymbol {s} ^ {\prime}} ^ {a}} \max  _ {a ^ {\prime}} Q \left(\boldsymbol {W} ^ {\star}; \boldsymbol {s} ^ {\prime}, a ^ {\prime}\right)\right) \cdot \frac {\partial Q (\boldsymbol {W} ; \boldsymbol {s} , a)}{\partial \boldsymbol {w} _ {\ell , k}} \tag {110} \\ - \mathbb {E} _ {(\boldsymbol {s}, a) \sim \mu^ {\star}} \Big (Q (\boldsymbol {W}; \boldsymbol {s}, a) - r (\boldsymbol {s}, a) - \gamma \cdot \mathbb {E} _ {\boldsymbol {s} ^ {\prime} \sim p _ {\boldsymbol {s}, \boldsymbol {s} ^ {\prime}} ^ {a}} \max  _ {a ^ {\prime}} Q (\boldsymbol {W} ^ {\star}; \boldsymbol {s} ^ {\prime}, a ^ {\prime}) \Big) \cdot \frac {\partial Q (\boldsymbol {W} ; \boldsymbol {s} , a)}{\partial \boldsymbol {w} _ {\ell , k}} \\ = \mathbb {E} _ {(\boldsymbol {s}, a) \sim \mu_ {t}, \boldsymbol {s} ^ {\prime} \sim p _ {s, s ^ {\prime}} ^ {a}} \left(Q (\boldsymbol {W}; \boldsymbol {s}, a) - r (\boldsymbol {s}, a) - \gamma \cdot \max  _ {a ^ {\prime}} Q (\boldsymbol {W} ^ {\star}; \boldsymbol {s} ^ {\prime}, a ^ {\prime})\right) \cdot \frac {\partial Q (\boldsymbol {W} ; \boldsymbol {s} , a)}{\partial \boldsymbol {w} _ {\ell , k}} \\ - \mathbb {E} _ {(\boldsymbol {s}, a) \sim \mu^ {\star}, \boldsymbol {s} ^ {\prime} \sim p _ {\boldsymbol {s}, \boldsymbol {s} ^ {\prime}} ^ {a}} \Big (Q (\boldsymbol {W}; \boldsymbol {s}, a) - r (\boldsymbol {s}, a) - \gamma \cdot \max  _ {a ^ {\prime}} Q (\boldsymbol {W} ^ {\star}; \boldsymbol {s} ^ {\prime}, a ^ {\prime}) \Big) \cdot \frac {\partial Q (\boldsymbol {W} ; \boldsymbol {s} , a)}{\partial \boldsymbol {w} _ {\ell , k}} \\ \end{array}
$$
$$
\begin{array}{l} \left| I _ {3} \right| = \left| \int_ {(s, a)} \int_ {s ^ {\prime}} \left(Q (\boldsymbol {W}; s, a) - r (s, a) - \gamma \cdot \max  _ {a ^ {\prime}} Q (\boldsymbol {W} ^ {\star}; s ^ {\prime}, a ^ {\prime})\right) \cdot \frac {\partial Q (\boldsymbol {W} ; s , a)}{\partial \boldsymbol {w} _ {\ell , k}} \right. \\ \cdot \left(\mu^ {\star} (d s, d a) \mathcal {P} (d s ^ {\prime} | s, a) - \mu_ {t} (d s, d a) \mathcal {P} (d s ^ {\prime} | s, a)\right) \\ \leq \left| Q (\boldsymbol {W}; \boldsymbol {s}, a) - r (\boldsymbol {s}, a) - \gamma \cdot \max  _ {a ^ {\prime}} Q (\boldsymbol {W} ^ {\star}; \boldsymbol {s} ^ {\prime}, a ^ {\prime}) \right| \cdot \left| \frac {\partial Q (\boldsymbol {W} ; \boldsymbol {s} , a)}{\partial \boldsymbol {w} _ {\ell , k}} \right| \\ \cdot \left| \int_ {(\boldsymbol {s}, a)} \int_ {\boldsymbol {s} ^ {\prime}} \left(\mu^ {\star} (d \boldsymbol {s}, d a) \mathcal {P} (d \boldsymbol {s} ^ {\prime} | \boldsymbol {s}, a) - \mu_ {t} (d \boldsymbol {s}, d a) \mathcal {P} (d \boldsymbol {s} ^ {\prime} | \boldsymbol {s}, a)\right) \right| \\ = \left| Q (\boldsymbol {W}; \boldsymbol {s}, a) - r (\boldsymbol {s}, a) - \gamma \cdot \max  _ {a ^ {\prime}} Q \left(\boldsymbol {W} ^ {\star}; \boldsymbol {s} ^ {\prime}, a ^ {\prime}\right) \right| \cdot \left| \frac {\partial Q (\boldsymbol {W} ; \boldsymbol {s} , a)}{\partial \boldsymbol {w} _ {\ell , k}} \right| \tag {111} \\ \cdot \left[ (1 - \varepsilon) \cdot \left| \int_ {(\boldsymbol {s}, a)} \int_ {\boldsymbol {s} ^ {\prime}} \left(\mu^ {\star} (d \boldsymbol {s}, d a) \mathcal {P} (d \boldsymbol {s} ^ {\prime} | \boldsymbol {s}, a) - \mu_ {t, 1} (d \boldsymbol {s}, d a) \mathcal {P} (d \boldsymbol {s} ^ {\prime} | \boldsymbol {s}, a)\right) \right| \right. \\ + \varepsilon \cdot \left| \int_ {(\boldsymbol {s}, a)} \int_ {\boldsymbol {s} ^ {\prime}} \left(\mu^ {\star} (d \boldsymbol {s}, d a) \mathcal {P} (d \boldsymbol {s} ^ {\prime} | \boldsymbol {s}, a) - \mu_ {t, 2} (d \boldsymbol {s}, d a) \mathcal {P} (d \boldsymbol {s} ^ {\prime} | \boldsymbol {s}, a)\right) \right| \\ \leq \frac {R _ {\max }}{1 - \gamma} \cdot \left[ (1 - \varepsilon) \cdot \left| \int_ {(s, a)} \int_ {s ^ {\prime}} \left(\mu^ {\star} (d s, d a) \mathcal {P} (d s ^ {\prime} | s, a) - \mu_ {t, 1} (d s, d a) \mathcal {P} (d s ^ {\prime} | s, a)\right) \right| \right. \\ \left. + \varepsilon \cdot \left| \int_ {(\boldsymbol {s}, a)} \int_ {\boldsymbol {s} ^ {\prime}} \left(\mu^ {\star} (d \boldsymbol {s}, d a) \mathcal {P} (d \boldsymbol {s} ^ {\prime} | \boldsymbol {s}, a) - \mu_ {t, 2} (d \boldsymbol {s}, d a) \mathcal {P} (d \boldsymbol {s} ^ {\prime} | \boldsymbol {s}, a)\right) \right] \right]. \\ \end{array}
$$
$$
\begin{array}{l} \left| \int_ {(s, a)} \int_ {\mathbf {s} ^ {\prime}} \left(\mu^ {*} (d \mathbf {s}, d a) \mathcal {P} (d \mathbf {s} ^ {\prime} | \mathbf {s}, a) - \mu_ {t, 1} (d \mathbf {s}, d a) \mathcal {P} (d \mathbf {s} ^ {\prime} | \mathbf {s}, a)\right) \right| \\ = \left| \int_ {(\boldsymbol {s}, a)} \int_ {\boldsymbol {s} ^ {\prime}} \left(\mathcal {P} ^ {\star} (d \boldsymbol {s}) \pi^ {\star} (d a | \boldsymbol {s}) \mathcal {P} (d \boldsymbol {s} ^ {\prime} | \boldsymbol {s}, a) - \mathcal {P} _ {t, 1} (d \boldsymbol {s}) \pi_ {t, 1} (d a | d \boldsymbol {s}) \mathcal {P} (d \boldsymbol {s} ^ {\prime} | \boldsymbol {s}, a)\right) \right| \\ \leq \left| \int_ {(\boldsymbol {s}, a)} \int_ {\boldsymbol {s} ^ {\prime}} \left(\mathcal {P} ^ {\star} (d \boldsymbol {s}) - \mathcal {P} _ {t, 1} (d \boldsymbol {s})\right) \pi^ {\star} (d a | \boldsymbol {s}) \mathcal {P} (d \boldsymbol {s} ^ {\prime} | \boldsymbol {s}, a) \right| \tag {112} \\ + \left| \int_ {(\boldsymbol {s}, a)} \int_ {\boldsymbol {s} ^ {\prime}} \mathcal {P} _ {t, 1} (d \boldsymbol {s}) \left(\pi_ {t, 1} (d a | d \boldsymbol {s}) - \pi^ {\star} (d a | d \boldsymbol {s})\right) \mathcal {P} (d \boldsymbol {s} ^ {\prime} | \boldsymbol {s}, a) \right| \\ \leq \left| \mathcal {A} \right| \cdot C _ {t}. \\ \end{array}
$$
Therefore, the bound of  $I_{3}$  can be found as