SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left| f _ {2} \left(\phi \left(\frac {\left[ \nabla_ {\boldsymbol {\theta}} f _ {1} \left(\boldsymbol {\chi} _ {k , \widehat {i}} ^ {s}\right) ; \nabla_ {\boldsymbol {\theta}} f _ {1} \left(\boldsymbol {\chi} _ {k , \widehat {i}} ^ {q}\right) \right]}{c _ {g} ^ {\prime} L}, \boldsymbol {\chi} _ {k, \widehat {i}} ^ {q}\right); \boldsymbol {\theta} _ {2} ^ {(k - 1)}\right) - \left(r _ {k} - f _ {1} \left(\boldsymbol {\chi} _ {k, \widehat {i}} ^ {q}; \boldsymbol {\theta} _ {1} ^ {(k - 1)}\right)\right) \right| \\ \leq \left| f _ {2} \left(\phi \left(\frac {\left[ \nabla_ {\boldsymbol {\theta}} f _ {1} \left(\chi_ {k , \widehat {i}} ^ {s}\right) ; \nabla_ {\boldsymbol {\theta}} f _ {1} \left(\chi_ {k, \widehat {i}} ^ {q}\right) \right]}{c _ {g} ^ {\prime} L}, \chi_ {k, \widehat {i}} ^ {q}\right); \boldsymbol {\theta} _ {2} ^ {(k - 1)}\right) \right| + \left| f _ {1} \left(\chi_ {k, \widehat {i}} ^ {q}; \boldsymbol {\theta} _ {1} ^ {(k - 1)}\right) \right| + 1 \\ \leq 1 + 2 \gamma_ {1} \\ \end{array}
$$
by triangle inequality and applying the generalization result of FC networks (Lemma D.5) on  $f_{1}(\cdot ;\pmb{\theta}_{1}),f_{2}(\cdot ;\pmb{\theta}_{2})$ .
For the brevity of notation, we use  $\nabla f_{1}(\mathcal{T}_{k,\widehat{i}})$  to denote  $\phi\left(\frac{[\nabla_{\theta}f_{1}(\boldsymbol{\chi}_{k,\widehat{i}}^{s});\nabla_{\theta}f_{1}(\boldsymbol{\chi}_{k,\widehat{i}}^{q})]}{c_{g}^{\prime}L},\boldsymbol{\chi}_{k,\widehat{i}}^{q}\right)$  and apply  $(\boldsymbol{\chi}_k,r_k)$  as  $(\boldsymbol{\chi}_{k,\widehat{i}}^{q},r_{k,\widehat{i}})$  for the following proof. Define the difference sequence as
$$
\begin{array}{l} V _ {\tau} ^ {(1)} = \mathbb {E} \left[ \left| f _ {2} \left(\nabla f _ {1} \left(\mathcal {T} _ {\tau , \hat {i}}\right); \boldsymbol {\theta} _ {2} ^ {(\tau - 1)}\right) - \left(r _ {\tau} - f _ {1} \left(\boldsymbol {\chi} _ {\tau}; \boldsymbol {\theta} _ {1} ^ {(\tau - 1)}\right)\right) \right| \right] \\ - \left| f _ {2} \left(\nabla f _ {1} (\mathcal {T} _ {\widehat {\tau}, i}); \pmb {\theta} _ {2} ^ {(\tau - 1)}\right) - \left(r _ {\tau} - f _ {1} (\pmb {\chi} _ {\tau}; \pmb {\theta} _ {1} ^ {(\tau - 1)})\right) \right|. \\ \end{array}
$$
Since the past rewards and the received arm-reward pairs  $(\boldsymbol{\chi}_{\tau}, r_{\tau})$  are generated by the same reward mapping function, we have the expectation
$$
\begin{array}{l} \mathbb {E} \left[ V _ {\tau} ^ {(1)} \mid F _ {\tau} \right] = \mathbb {E} \left[ \left| f _ {2} \left(\nabla f _ {1} \left(\mathcal {T} _ {\tau , \widehat {i}}\right); \boldsymbol {\theta} _ {2} ^ {(\tau - 1)}\right) - \left(r _ {\tau} - f _ {1} \left(\boldsymbol {\chi} _ {\tau}; \boldsymbol {\theta} _ {1} ^ {(\tau - 1)}\right)\right) \right| \right] \\ - \mathbb {E} \left[ \left| f _ {2} \left(\nabla f _ {1} \left(\mathcal {T} _ {\tau , \widehat {i}}\right); \boldsymbol {\theta} _ {2} ^ {(\tau - 1)}\right) - \left(r _ {\tau} - f _ {1} \left(\boldsymbol {\chi} _ {\tau}; \boldsymbol {\theta} _ {1} ^ {(\tau - 1)}\right)\right) \right| | F _ {\tau} \right] = 0. \\ \end{array}
$$
where  $F_{\tau}$  denotes the filtration given the past records  $\mathcal{P}_{\tau}$ , up to round  $\tau \in [k]$ . This also gives the fact that  $V_{\tau}^{(1)}$  is a martingale difference sequence. Then, after applying the martingale difference sequence over  $[k]$ , we have
$$
\begin{array}{l} \frac {1}{k} \sum_ {\tau \in [ k ]} V _ {\tau} ^ {(1)} = \frac {1}{k} \sum_ {\tau \in [ k ]} \mathbb {E} \left[ \left| f _ {2} \left(\nabla f _ {1} \left(\mathcal {T} _ {\tau , \widehat {i}}\right); \boldsymbol {\theta} _ {2} ^ {(\tau - 1)}\right) - \left(r _ {\tau} - f _ {1} \left(\boldsymbol {\chi} _ {\tau}; \boldsymbol {\theta} _ {1} ^ {(\tau - 1)}\right)\right) \right| \right] \\ \left. \right. - \frac {1}{k} \sum_ {\tau \in [ k ]} \left| f _ {2} \left(\nabla f _ {1} \left(\mathcal {T} _ {\widehat {\tau}, i}\right); \boldsymbol {\theta} _ {2} ^ {(\tau - 1)}\right) - \left(r _ {\tau} - f _ {1} \left(\boldsymbol {\chi} _ {\tau}; \boldsymbol {\theta} _ {1} ^ {(\tau - 1)}\right)\right)\right|. \\ \end{array}
$$
Then, by applying the Azuma-Hoeffding inequality, it leads to
$$
\mathbb {P} \left[ \frac {1}{k} \sum_ {\tau \in [ k ]} V _ {\tau} ^ {(1)} - \frac {1}{k} \sum_ {\tau \in [ k ]} \mathbb {E} [ V _ {\tau} ^ {(1)} ] \geq (1 + 2 \gamma_ {1}) \sqrt {\frac {2 \log (1 / \delta)}{k}} \right] \leq \delta
$$
Since the expectation of  $V_{\tau}^{(1)}$  is zero, with the probability at least  $1 - \delta$  and an existing set of parameters  $\pmb{\theta}_{2}$  s.t.  $\| \pmb{\theta}_{2} - \pmb{\theta}_{2}^{(0)} \| \leq \mathcal{O}\left(\frac{k^{3}}{\rho\sqrt{m}}\log m\right)$ , the above inequality implies
$$
\begin{array}{l} \frac {1}{k} \sum_ {\tau \in [ k ]} V _ {\tau} ^ {(1)} \leq (1 + 2 \gamma_ {1}) \sqrt {\frac {2 \log (1 / \delta)}{k}} \Rightarrow \\ \mathbb {E} _ {\mathcal {T} _ {k, i} \sim \mathcal {P} (\mathcal {T})} \mathbb {E} _ {\{\boldsymbol {\theta} _ {1} ^ {(k - 1)}, \boldsymbol {\theta} _ {2} ^ {(k - 1)} \}} \left[ \left| f _ {2} \left(\nabla f _ {1} \left(\mathcal {T} _ {\tau , \hat {i}}\right); \boldsymbol {\theta} _ {2} ^ {(k - 1)}\right) - \left(r _ {\tau} - f _ {1} \left(\boldsymbol {\chi} _ {\tau}; \boldsymbol {\theta} _ {1} ^ {(k - 1)}\right)\right) \right| \right] \\ = \frac {1}{k} \sum_ {\tau \in [ k ]} \mathbb {E} \left[ \left| f _ {2} \left(\nabla f _ {1} \left(\mathcal {T} _ {\tau , \hat {i}}\right); \boldsymbol {\theta} _ {2} ^ {(\tau - 1)}\right) - \left(r _ {k} - f _ {1} \left(\boldsymbol {\chi} _ {\tau}; \boldsymbol {\theta} _ {1} ^ {(\tau - 1)}\right)\right) \right| \right] \\ \leq \frac {1}{k} \sum_ {\tau \in [ k ]} \left| f _ {2} \left(\nabla f _ {1} \left(\mathcal {T} _ {\widehat {\tau}, i}\right); \boldsymbol {\theta} _ {2} ^ {(\tau - 1)}\right) - \left(r _ {\tau} - f _ {1} \left(\boldsymbol {\chi} _ {\tau}; \boldsymbol {\theta} _ {1} ^ {(\tau - 1)}\right)\right) \right| + (1 + 2 \gamma_ {1}) \sqrt {\frac {2 \log (1 / \delta)}{k}} \\ \leq \frac {1}{k} \sum_ {\tau \in [ k ]} \left| f _ {2} \left(\nabla f _ {1} \left(\mathcal {T} _ {\widehat {\tau}, i}\right); \boldsymbol {\theta} _ {2}\right) - \left(r _ {\tau} - f _ {1} \left(\boldsymbol {\chi} _ {\tau}; \boldsymbol {\theta} _ {1} ^ {(\tau - 1)}\right)\right) \right| + \frac {3 L}{\sqrt {2 k}} + (1 + 2 \gamma_ {1}) \sqrt {\frac {2 \log (1 / \delta)}{k}} \\ \leq \frac {1}{\sqrt {k}} \sqrt {\sum_ {\tau \in [ k ]} \left| f _ {2} \left(\nabla f _ {1} \left(\mathcal {T} _ {\widehat {\tau} , i}\right) ; \boldsymbol {\theta} _ {2}\right) - \left(r _ {\tau} - f _ {1} \left(\chi_ {\tau} ; \boldsymbol {\theta} _ {1} ^ {(\tau - 1)}\right)\right) \right| ^ {2}} + \frac {3 L}{\sqrt {2 k}} + (1 + 2 \gamma_ {1}) \sqrt {\frac {2 \log (1 / \delta)}{k}} \\ \leq \sqrt {\frac {2 \xi_ {1}}{k}} + \frac {3 L}{\sqrt {2 k}} + (1 + 2 \gamma_ {1}) \sqrt {\frac {2 \log (1 / \delta)}{k}}. \\ \end{array}
$$
where the first equality is due to the sampling of candidate tasks and the model parameters. Here, the upper bound (i) is derived by applying the conclusions of Lemma D.6 and Lemma D.10, and the inequality (ii) is derived by adopting Lemma D.6 while defining the empirical loss to be  $\frac{1}{2}\sum_{\tau \in [k]}\left|f_2\left(\nabla f_1(\mathcal{T}_{\tau,\widehat{i}});\pmb{\theta}_2\right) - \left(r_\tau - f_1(\pmb{\chi}_\tau;\pmb{\theta}_1^{(\tau-1)})\right)\right|^2 \leq \xi_1$ . Finally, applying the union bound would give the aforementioned results.
Here, analogous to the trained parameters, we consider the shadow parameters as  $\{\pmb{\theta}_1^{(k),*},\pmb{\theta}_2^{(k),*}\} \sim \{\widetilde{\pmb{\theta}}_1^{(\tau),*},\widetilde{\pmb{\theta}}_2^{(\tau),*}\}_{\tau \in [k]}$ . Similarly, each pair  $\{\widetilde{\pmb{\theta}}_1^{(\tau),*},\widetilde{\pmb{\theta}}_2^{(\tau),*}\}$  is separately trained on past received rewards of the optimal arm(s)  $\{r_{\tau ',i^*}\}_{\tau '\in [\tau ]},\mathcal{T}_{\tau ',i^*\in \Omega_k^*}$  and past exploration scores of the optimal arm(s)  $\{e_{\tau ',i^*}\}_{\tau '\in [\tau ]},\mathcal{T}_{\tau ',i^*\in \Omega_k^*}$  with  $J_{\pmb{\theta}}$ -iteration GD, starting from the random initialization  $\{\pmb{\theta}_1^{(0)},\pmb{\theta}_2^{(0)}\}$ .
Corollary D.2. For the constants  $0 < \rho \leq \mathcal{O}(1 / L)$  and  $\xi_1\in (0,1)$ , given the past records  $\mathcal{P}_{k - 1}$  we suppose  $m,\eta_{1},J$  satisfy the conditions in Theorem 5.2, and randomly draw the parameters  $\{\pmb {\theta}_1^{(k),*},\pmb {\theta}_2^{(k),*}\} \sim \{\widetilde{\pmb{\theta}}_1^{(\tau),*},\widetilde{\pmb{\theta}}_2^{(\tau),*}\}_{\tau \in [k]}$ . For the optimal arm  $\mathcal{T}_{k,i^*}\in \Omega_{task}^k$ , consider its union set with the collection of past optimal arms  $\mathcal{P}_{k - 1}^{*}\cup \{\mathcal{T}_{k,i^{*}},r_{k,i^{*}}\}$  are generated by a fixed policy when witness the candidate arms  $\{\Omega_{task}^{(\tau)}\}_{\tau \in [k]}$ , with  $\mathcal{P}_{k - 1}^{*}$  being the collection chosen by this policy. Then, with probability at least  $1 - \delta$ , we have
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {T} _ {k, i} \sim \mathcal {P} (\mathcal {T})} \left[ | f _ {2} \left(\phi (\frac {[ \nabla_ {\pmb {\theta}} f _ {1} (\pmb {\chi} _ {k} ^ {s , *}) ; \nabla_ {\pmb {\theta}} f _ {1} (\pmb {\chi} _ {k} ^ {q , *}) ]}{c _ {g} ^ {\prime} L}, \pmb {\chi} _ {k} ^ {q, *}\); \pmb {\theta} _ {2} ^ {(k - 1), *}\right) - \left(r _ {\tau} - f _ {1} (\pmb {\chi} _ {k} ^ {q, *}; \pmb {\theta} _ {1} ^ {(k - 1), *})\right) | \left| \Omega_ {t a s k} ^ {(k)}, \mathcal {P} _ {k - 1} ^ {*} \right. \right] \\ \leq \frac {1}{\sqrt {k}} \cdot \left(\sqrt {2 \xi_ {1}} + \frac {3 L}{\sqrt {2}} + (1 + \gamma_ {1}) \sqrt {2 \log (\frac {k}{\delta})}\right) + \Gamma_ {k} \\ \end{array}
$$
where  $r_{\tau ,i^*}$  is the corresponding reward generated by the mapping function given an arm  $\chi_{\tau ,i^{*}}$  , and
$$
\Gamma_ {k} = \left(1 + \mathcal {O} (\frac {k L ^ {3} \log^ {5 / 6} (m)}{\rho^ {1 / 3} m ^ {1 / 6}})\right) \cdot \mathcal {O} (\frac {k ^ {4} L}{\rho \sqrt {m}} \log (m)) + \mathcal {O} \left(\frac {k ^ {5} L ^ {2} \log^ {1 1 / 6} (m)}{\rho^ {4 / 3} m ^ {1 / 6}}\right).
$$
Proof. This corollary is the direct application of Lemma D.1 by following a similar proof procedure. First, suppose the shadow models  $f_{1}(\cdot ;\pmb{\theta}_{2}),f_{2}(\cdot ;\pmb{\theta}_{2})$  are trained on the alternative trajectory  $\mathcal{P}_{k - 1}^{*}$ . Analogous to the proof of Lemma D.1, we can define the following martingale difference sequence with regard to the previous records  $\mathcal{P}_{k - 1}^{*}$  up to round  $\tau \in [t]$  as
$$
\begin{array}{l} V _ {\tau} ^ {(1), *} = \mathbb {E} \bigg [ \bigg | f _ {2} \bigg (\nabla f _ {1} (\mathcal {T} _ {\tau , i ^ {*}}); \pmb {\theta} _ {2} ^ {(\tau - 1), *}) - \left(r _ {\tau} ^ {*} - f _ {1} (\pmb {\chi} _ {\tau} ^ {*}; \pmb {\theta} _ {1} ^ {(\tau - 1), *})\right) \bigg | \bigg ] \\ \left. \right. - \left| f _ {2} \left(\nabla f _ {1} \left(\mathcal {T} _ {\tau , i ^ {*}}\right); \boldsymbol {\theta} _ {2} ^ {(\tau - 1), *}\right) - \left(r _ {\tau} ^ {*} - f _ {1} \left(\boldsymbol {\chi} _ {\tau} ^ {*}; \boldsymbol {\theta} _ {1} ^ {(\tau - 1), *}\right)\right)\right|. \\ \end{array}
$$
Since the records in set  $\mathcal{P}_{k - 1}^{*}$  are sharing the same reward mapping function, we have the expectation
$$
\begin{array}{l} \mathbb {E} \left[ V _ {\tau} ^ {(1), *} \mid F _ {\tau} ^ {*} \right] = \mathbb {E} \left[ \left| f _ {2} \left(\nabla f _ {1} \left(\mathcal {T} _ {\tau , i ^ {*}}\right); \boldsymbol {\theta} _ {2} ^ {(\tau - 1), *}\right) - \left(r _ {\tau} ^ {*} - f _ {1} \left(\boldsymbol {\chi} _ {\tau} ^ {*}; \boldsymbol {\theta} _ {1} ^ {(\tau - 1), *}\right)\right) \right| \right] \\ - \mathbb {E} \left[ \left| f _ {2} \left(\nabla f _ {1} \left(\mathcal {T} _ {\tau , i ^ {*}}\right); \boldsymbol {\theta} _ {2} ^ {(\tau - 1), *}\right) - \left(r _ {\tau} ^ {*} - f _ {1} \left(\boldsymbol {\chi} _ {\tau} ^ {*}; \boldsymbol {\theta} _ {1} ^ {(\tau - 1), *}\right)\right) \right| F _ {\tau} ^ {*} \right] = 0. \\ \end{array}
$$
where  $F_{\tau}^{*}$  denotes the filtration given the past records  $\mathcal{P}_{k - 1}^{*}$ . The mean value of  $V_{\tau}^{(1),*}$  across different time steps will be
$$
\begin{array}{l} \frac {1}{k} \sum_ {\tau \in [ k ]} V _ {\tau} ^ {(1), *} = \frac {1}{k} \sum_ {\tau \in [ k ]} \mathbb {E} \left[ \left| f _ {2} \left(\nabla f _ {1} \left(\mathcal {T} _ {\tau , i ^ {*}}\right); \boldsymbol {\theta} _ {2} ^ {(\tau - 1), *}\right) - \left(r _ {\tau} ^ {*} - f _ {1} \left(\boldsymbol {\chi} _ {\tau} ^ {*}; \boldsymbol {\theta} _ {1} ^ {(\tau - 1), *}\right)\right) \right| \right] \\ - \frac {1}{k} \sum_ {\tau \in [ k ]} \left| f _ {2} \left(\nabla f _ {1} (\mathcal {T} _ {\tau , i ^ {*}}); \pmb {\theta} _ {2} ^ {(\tau - 1), *}\right) - \left(r _ {\tau} ^ {*} - f _ {1} (\pmb {\chi} _ {\tau} ^ {*}; \pmb {\theta} _ {1} ^ {(\tau - 1), *})\right) \right|. \\ \end{array}
$$
with the expectation of zero. Afterwards, applying the Azuma-Hoeffding inequality, with a constant  $\delta \in (0,1)$ , it leads to
$$
\mathbb {P} \left[ \frac {1}{k} \sum_ {\tau \in [ k ]} V _ {\tau} ^ {(1), *} - \frac {1}{k} \sum_ {\tau \in [ k ]} \mathbb {E} [ V _ {\tau} ^ {(1), *} ] \geq (1 + 2 \gamma_ {1}) \sqrt {\frac {2 \log (1 / \delta)}{k}} \right] \leq \delta
$$
To bound the output difference between the shadow model  $f_{1}(\cdot;\pmb{\theta}_{1}^{(k-1),*}), f_{2}(\cdot;\pmb{\theta}_{2}^{(k-1),*})$  and the model we trained based on received records  $f_{1}(\cdot;\pmb{\theta}_{1}^{(k-1)}), f_{2}(\cdot;\pmb{\theta}_{2}^{(k-1)})$ , we apply the conclusion from Lemma D.11, which leads to that given arbitrary input vectors  $\pmb{x}, \pmb{x}'$ , we have
$$
\begin{array}{l} | f _ {1} (\pmb {x}; \pmb {\theta} _ {1} ^ {(k - 1), *}) - f _ {1} (\pmb {x}; \pmb {\theta} _ {1} ^ {(k - 1)}) |, | f _ {2} (\pmb {x} ^ {\prime}; \pmb {\theta} _ {2} ^ {(k - 1), *}) - f _ {2} (\pmb {x} ^ {\prime}; \pmb {\theta} _ {2} ^ {(k - 1)}) | \leq \\ \left(1 + \mathcal {O} (\frac {k L ^ {3} \log^ {5 / 6} (m)}{\rho^ {1 / 3} m ^ {1 / 6}})\right) \cdot \mathcal {O} (\frac {k ^ {3} L}{\rho \sqrt {m}} \log (m)) + \mathcal {O} \left(\frac {k ^ {4} L ^ {2} \log^ {1 1 / 6} (m)}{\rho^ {4 / 3} m ^ {1 / 6}}\right). \\ \end{array}
$$
Finally, combining all the results will finish the proof.
We will also be able to have the performance guarantee under the batch settings. Recall that given a batch of chosen tasks  $\Omega_k \subset \Omega_{\mathrm{task}}^{(k)}, k \in [K]$ , we have the meta-parameters adapted to this batch of tasks being  $\Theta^{(K - 1)}[\Omega_K]$ , which we consider as the input for the  $f_1(\cdot; \theta_1)$  model, where the tasks within each collection are sampled from the task distribution. Thus, chosen task batches from different iterations are also independent from each other. Intuitively, we can also define the corresponding reward for arm batch  $\Omega_k$  as  $r_k = h(\Theta^{(k - 1)}[\Omega_k])$ . Then, we bound the batch settings with the following lemma and corollary.
Lemma D.3. For the constants  $c_{g}^{\prime} > 0$ ,  $\rho \in (0, \mathcal{O}\left(\frac{1}{L}\right))$  and  $\xi_{1} \in (0,1)$ , given the past records  $\mathcal{P}_{k-1}$ , we suppose  $m, \eta_{1}, \eta_{2}$  satisfy the conditions in Theorem 5.2, and randomly draw the parameter  $\{\pmb{\theta}_{1}^{(k)}, \pmb{\theta}_{2}^{(k)}\} \sim \{\widetilde{\pmb{\theta}}_{1}^{(\tau)}, \widetilde{\pmb{\theta}}_{2}^{(\tau)}\}_{\tau \in [k]}$ . Consider the past records  $\mathcal{P}_{k-1}$  up to round  $k$  are generated by a fixed policy when witness the candidate arms  $\{\Omega_{task}^{(\tau)}\}_{\tau \in [k]}$ . Then, with probability at least  $1 - \delta$  given the pair of chosen arm batch and the reward  $(\Omega_{k}, r_{k})$  in round  $k$ , we have
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {T} _ {k, i} \sim \mathcal {P} (\mathcal {T})} \left[ | f _ {2} \Bigg (\phi \big (\frac {[ \nabla_ {\boldsymbol {\theta}} f _ {1} (\boldsymbol {\chi} _ {k} ^ {s}) ; \nabla_ {\boldsymbol {\theta}} f _ {1} (\boldsymbol {\chi} _ {k} ^ {q}) ]}{c _ {g} ^ {\prime} L}, \boldsymbol {\chi} _ {k} ^ {q}); \boldsymbol {\theta} _ {2} ^ {(k - 1)} \big) - \left(r _ {\tau} - f _ {1} (\boldsymbol {\chi} _ {k} ^ {q}; \boldsymbol {\theta} _ {1} ^ {(k - 1)})\right) | \mid \Omega_ {t a s k} ^ {(k)}, \mathcal {P} _ {k - 1} \right] \\ \leq \frac {1}{\sqrt {k}} \cdot \left(\sqrt {2 \xi_ {1}} + \frac {3 L}{\sqrt {2}} + (1 + 2 \gamma_ {1}) \sqrt {2 \log (\frac {k}{\delta})}\right) \\ \end{array}
$$
Proof. The proof of this lemma is analogous to the proof of Lemma D.1. First, we can derive the output upper bound
$$
\begin{array}{l} \left| f _ {2} \left(\phi \left(\frac {\left[ \nabla_ {\boldsymbol {\theta}} f _ {1} \left(\chi_ {k} ^ {s}\right) ; \nabla_ {\boldsymbol {\theta}} f _ {1} \left(\chi_ {k} ^ {q}\right) \right]}{c _ {g} ^ {\prime} L}, \chi_ {k} ^ {q}\right); \boldsymbol {\theta} _ {2} ^ {(k - 1)}\right) - \left(r _ {k} - f _ {1} \left(\chi_ {k} ^ {q}; \boldsymbol {\theta} _ {1} ^ {(k - 1)}\right)\right) \right| \\ \leq \left| f _ {2} \left(\phi \left(\frac {\left[ \nabla_ {\boldsymbol {\theta}} f _ {1} \left(\boldsymbol {\chi} _ {k} ^ {s}\right) ; \nabla_ {\boldsymbol {\theta}} f _ {1} \left(\boldsymbol {\chi} _ {k} ^ {q}\right) \right]}{c _ {g} ^ {\prime} L}, \boldsymbol {\chi} _ {k} ^ {q}\right); \boldsymbol {\theta} _ {2} ^ {(k - 1)}\right) \right| + \left| f _ {1} \left(\boldsymbol {\chi} _ {k} ^ {q}; \boldsymbol {\theta} _ {1} ^ {(k - 1)}\right) \right| + 1 \\ \leq 1 + 2 \gamma_ {1} \\ \end{array}
$$
by triangle inequality and applying the generalization result of FC networks (Lemma D.5) on  $f_{1}(\cdot ;\pmb{\theta}_{1}),f_{2}(\cdot ;\pmb{\theta}_{2})$ , where  $c_{g}^{\prime} > 0$  is a positive number to scale the concatenated gradient vector.
For the brevity of notation, we use  $\nabla f_{1}(\Omega_{k})$  to denote  $\phi (\frac{[\nabla_{\theta}f_{1}(\boldsymbol{\chi}_{k}^{s});\nabla_{\theta}f_{1}(\boldsymbol{\chi}_{k}^{q})]}{c_{g}^{\prime}L},\boldsymbol{\chi}_{k}^{q})$  and apply  $(\boldsymbol {\chi}_k,r_k)$  as  $(\boldsymbol{\chi}_k^q,r_k)$  for the following proof. Define the difference sequence as
$$
\begin{array}{l} V _ {\tau} ^ {(2)} = \mathbb {E} \left[ \left| f _ {2} \left(\nabla f _ {1} \left(\Omega_ {\tau}\right); \boldsymbol {\theta} _ {2} ^ {(\tau - 1)}\right) - \left(r _ {\tau} - f _ {1} \left(\boldsymbol {\chi} _ {\tau}; \boldsymbol {\theta} _ {1} ^ {(\tau - 1)}\right)\right) \right| \right] \\ \left. \right. - \left| f _ {2} \left(\nabla f _ {1} (\Omega_ {\tau}); \boldsymbol {\theta} _ {2} ^ {(\tau - 1)}\right) - \left(r _ {\tau} - f _ {1} (\boldsymbol {\chi} _ {\tau}; \boldsymbol {\theta} _ {1} ^ {(\tau - 1)})\right)\right|. \\ \end{array}
$$
Since the past rewards and the received arm batch-reward pairs  $(\chi_{\tau}, r_{\tau})$  are generated by the same reward mapping function, we have the expectation
$$
\begin{array}{l} \mathbb {E} \left[ V _ {\tau} ^ {(2)} \mid F _ {\tau} \right] = \mathbb {E} \left[ \left| f _ {2} \left(\nabla f _ {1} \left(\Omega_ {\tau}\right); \boldsymbol {\theta} _ {2} ^ {(\tau - 1)}\right) - \left(r _ {\tau} - f _ {1} \left(\chi_ {\tau}; \boldsymbol {\theta} _ {1} ^ {(\tau - 1)}\right)\right) \right| \right] \\ - \mathbb {E} \bigg [ \bigg | f _ {2} \Big (\nabla f _ {1} (\Omega_ {\tau}); \pmb {\theta} _ {2} ^ {(\tau - 1)} \Big) - \Big (r _ {\tau} - f _ {1} (\pmb {\chi} _ {\tau}; \pmb {\theta} _ {1} ^ {(\tau - 1)}) \Big) \bigg | \big | F _ {\tau} \bigg ] = 0. \\ \end{array}
$$
where  $F_{\tau}$  denotes the filtration given the past records  $\mathcal{P}_{\tau}$ , up to round  $\tau \in [k]$ . This also gives the fact that  $V_{\tau}^{(2)}$  is a martingale difference sequence. Then, after applying the martingale difference sequence over  $[k]$ , we have
$$
\begin{array}{l} \frac {1}{k} \sum_ {\tau \in [ k ]} V _ {\tau} ^ {(2)} = \frac {1}{k} \sum_ {\tau \in [ k ]} \mathbb {E} \left[ \left| f _ {2} \left(\nabla f _ {1} \left(\Omega_ {\tau}\right); \boldsymbol {\theta} _ {2} ^ {(\tau - 1)}\right) - \left(r _ {\tau} - f _ {1} \left(\boldsymbol {\chi} _ {\tau}; \boldsymbol {\theta} _ {1} ^ {(\tau - 1)}\right)\right) \right| \right] \\ - \frac {1}{k} \sum_ {\tau \in [ k ]} \left| f _ {2} \left(\nabla f _ {1} (\Omega_ {\tau}); \pmb {\theta} _ {2} ^ {(\tau - 1)}\right) - \left(r _ {\tau} - f _ {1} (\pmb {\chi} _ {\tau}; \pmb {\theta} _ {1} ^ {(\tau - 1)})\right) \right|. \\ \end{array}
$$
By the Azuma-Hoeffding inequality, it leads to  $\mathbb{P}\Big[\frac{1}{k}\sum_{\tau \in [k]}V_{\tau}^{(2)} - \frac{1}{k}\sum_{\tau \in [k]}\mathbb{E}[V_{\tau}^{(2)}]\geq (1 + 2\gamma_1)\sqrt{\frac{2\log(1 / \delta)}{k}}\Big]\leq \delta$ . As we have discussed, the tasks within each collection are sampled from the task distribution, which makes chosen task batches from different iterations  $\Omega_k, k\in [K]$  are also independent from each other. Since the expectation of  $V_{\tau}^{(2)}$  is zero, with the probability at least  $1 - \delta$  and an existing set of parameters  $\pmb{\theta}_{2}$  s.t.  $\| \pmb{\theta}_2 - \pmb{\theta}_2^{(0)}\| \leq \mathcal{O}\left(\frac{k^3}{\rho\sqrt{m}}\log m\right)$ , the above inequality implies
$$
\begin{array}{l} \frac {1}{k} \sum_ {\tau \in [ k ]} V _ {\tau} ^ {(1)} \leq (1 + 2 \gamma_ {1}) \sqrt {\frac {2 \log (1 / \delta)}{k}} \Rightarrow \\ \mathbb {E} _ {\mathcal {T} _ {k, i} \sim \mathcal {P} (\mathcal {T})} \mathbb {E} _ {\left\{\boldsymbol {\theta} _ {1} ^ {(k - 1)}, \boldsymbol {\theta} _ {2} ^ {(k - 1)} \right\}} \left[ \left| f _ {2} \left(\nabla f _ {1} (\Omega); \boldsymbol {\theta} _ {2} ^ {(k - 1)}\right) - \left(r - f _ {1} (\boldsymbol {\chi}; \boldsymbol {\theta} _ {1} ^ {(k - 1)})\right) \right| \right] \\ = \frac {1}{k} \sum_ {\tau \in [ k ]} \mathbb {E} \left[ \left| f _ {2} \left(\nabla f _ {1} \left(\Omega_ {\tau}\right); \boldsymbol {\theta} _ {2} ^ {(\tau - 1)}\right) - \left(r _ {k} - f _ {1} \left(\boldsymbol {\chi} _ {\tau}; \boldsymbol {\theta} _ {1} ^ {(\tau - 1)}\right)\right) \right| \right] \\ \leq \frac {1}{k} \sum_ {\tau \in [ k ]} \left| f _ {2} \left(\nabla f _ {1} \left(\Omega_ {\tau}\right); \boldsymbol {\theta} _ {2} ^ {(\tau - 1)}\right) - \left(r _ {\tau} - f _ {1} \left(\boldsymbol {\chi} _ {\tau}; \boldsymbol {\theta} _ {1} ^ {(\tau - 1)}\right)\right) \right| + (1 + 2 \gamma_ {1}) \sqrt {\frac {2 \log (1 / \delta)}{k}} \\ \leq \frac {1}{k} \sum_ {\tau \in [ k ]} \left| f _ {2} \left(\nabla f _ {1} \left(\Omega_ {\tau}\right); \boldsymbol {\theta} _ {2}\right) - \left(r _ {\tau} - f _ {1} \left(\boldsymbol {\chi} _ {\tau}; \boldsymbol {\theta} _ {1} ^ {(\tau - 1)}\right)\right) \right| + \frac {3 L}{\sqrt {2 k}} + (1 + 2 \gamma_ {1}) \sqrt {\frac {2 \log (1 / \delta)}{k}} \\ \leq \frac {1}{\sqrt {k}} \sqrt {\sum_ {\tau \in [ k ]} \left| f _ {2} \left(\nabla f _ {1} \left(\Omega_ {\tau}\right); \boldsymbol {\theta} _ {2}\right) - \left(r _ {\tau} - f _ {1} \left(\boldsymbol {\chi} _ {\tau}; \boldsymbol {\theta} _ {1} ^ {(\tau - 1)}\right)\right) \right| ^ {2}} + \frac {3 L}{\sqrt {2 k}} + (1 + 2 \gamma_ {1}) \sqrt {\frac {2 \log (1 / \delta)}{k}} \\ \leq_ {\mathrm {(i i)}} \sqrt {\frac {2 \xi_ {2}}{k}} + \frac {3 L}{\sqrt {2 k}} + (1 + 2 \gamma_ {1}) \sqrt {\frac {2 \log (1 / \delta)}{k}}. \\ \end{array}
$$
where the first equality is due to the sampling of candidate tasks and the model parameters. Here, the upper bound (i) is derived by applying the conclusions of Lemma D.6 and Lemma D.10, and the inequality (ii) is derived by adopting Lemma D.6 while defining the empirical loss to be  $\frac{1}{2}\sum_{\tau \in [k]}\left|f_2\left(\nabla f_1(\Omega_\tau);\pmb{\theta}_2\right) - \left(r_\tau -f_1(\chi_\tau;\pmb{\theta}_1^{(\tau -1)})\right)\right|^2\leq \xi_2$ . Finally, applying the union bound would give the aforementioned results.