SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \frac {\nu (s , x , a)}{\bar {\mu} _ {\beta} (s , x , a)} \leq \frac {\sum_ {\hat {x} \in \mathcal {X}} \nu (s , \hat {x} , a)}{\bar {\mu} _ {\beta} (s , x , a)} = \frac {\sum_ {\hat {x} \in \mathcal {X}} \nu (s , \hat {x} , a)}{\sum_ {\hat {x} \in \mathcal {X}} \mu (s , \hat {x} , a)} \times \frac {\sum_ {\hat {x} \in \mathcal {X}} \mu (s , \hat {x} , a)}{\bar {\mu} _ {\beta} (s , x , a)} \\ = \frac {\nu (s , a)}{\mu (s , a)} \times \frac {\sum_ {\hat {x} \in \mathcal {X}} \mu (s , \hat {x} , a)}{\bar {\mu} _ {\beta} (s , x , a)} \\ \leq_ {(1)} C \times \frac {\mu (s , a)}{\bar {\mu} _ {\beta} (s , x , a)} \\ = _ {(2)} C \left(\frac {\mu (s , a)}{\frac {\mu (s , x , a)}{m + 1} + \frac {m}{m + 1} \cdot \sum_ {\hat {x} \in \mathcal {X}} \mu (s , \hat {x} , a) \beta (x)}\right) \\ \leq C \left(\frac {(m + 1) \mu (s , a)}{m \sum_ {\hat {x} \in \mathcal {X}} \mu (s , \hat {x} , a) \beta (x)}\right) \\ \leq C \left(\frac {(m + 1) \mu (s , a)}{m \sum_ {\hat {x} \in \mathcal {B}} \mu (s , \hat {x} , a) \beta (x)}\right) \\ \leq C \cdot \frac {m + 1}{m} \cdot \frac {1}{\sigma_ {1}}, \\ \end{array}
$$
where the inequality (1) holds because of Assumption A.3, the equality (2) holds by substituting equation (8) and the last inequality is true because the fact that  $\beta(x) \geq \sigma_1, \forall x \in \mathcal{B}$ .
The next lemma transforms the norm in terms of distribution  $\nu$  to distribution  $\bar{\mu}_{\beta}$  (Eq. (8)).
Lemma A.2. Let  $\nu$  be any admissible distribution,  $\bar{\mu}_{\beta}$  denote the new data distribution defined in Eq. (8) after generating virtual samples with  $\beta(x)$ . If  $\beta(x) \geq \sigma_1, \forall x \in \mathcal{B}$ , then under Assumption A.3, for any function  $f: \mathcal{S} \times \mathcal{X} \times \mathcal{A} \to \mathbb{R}$ , we have  $\|f\|_{\underline{2},\nu} \leq \sqrt{\frac{m+1}{m}\frac{C}{\sigma_1}} \|f\|_{\underline{2},\bar{\mu}_{\beta}}$ , where
$$
\| f \| _ {\underline {{2}}, \nu} = \left(\sum_ {(s, x, a) \in \mathcal {S} \times \mathcal {X} \times \mathcal {A}, x \in \mathcal {B}} | f (s, x, a) | ^ {2} \nu (s, x, a)\right) ^ {1 / 2}.
$$
Proof. For any function  $f$ , we have
$$
\| f \| _ {\underline {{2}}, \nu} = \left(\sum_ {(s, x, a) \in \mathcal {S} \times \mathcal {X} \times \mathcal {A}, x \in \mathcal {B}} | f (s, x, a) | ^ {2} \nu (s, x, a)\right) ^ {1 / 2}
$$
$$
\begin{array}{l} \leq \left(\sum_ {(s, x, a) \in \mathcal {S} \times \mathcal {X} \times \mathcal {A}, x \in \mathcal {B}} | f (x, x, a) | ^ {2} \bar {\mu} _ {\beta} (s, x, a) \frac {(m + 1) C}{m \sigma_ {1}}\right) ^ {1 / 2} \\ \leq \sqrt {\frac {(m + 1) C}{m \sigma_ {1}}} \| f \| _ {\underline {{2}}, \bar {\mu} _ {\beta}}, \\ \end{array}
$$
where the first inequality is a result of Lemma A.1.
Lemma A.3. Consider two functions  $f, f': \mathcal{S} \times \mathcal{X} \times \mathcal{A} \to \mathbb{R}$  and define a policy  $\pi_{f,f'}(s,x) := \arg \min_{a \in \mathcal{A}} \min \{f(s,x,a), f'(s,x,a)\}$ . Then we have  $\forall \nu \in \Delta(\mathcal{S} \times \mathcal{X} \times \mathcal{A})$ ,
$$
\left\| V _ {f} - V _ {f ^ {\prime}} \right\| _ {2, P (\nu)} = \left\| f - f ^ {\prime} \right\| _ {2, P (\nu) \times \pi_ {f, f ^ {\prime}}}. \tag {9}
$$
$$
\begin{array}{l} \left\| V _ {f} - V _ {f ^ {\prime}} \right\| _ {2, P (\nu)} ^ {2} \\ = \sum_ {(s, x, a) \in \mathcal {S} \times \mathcal {X} \times \mathcal {A}} \sum_ {s ^ {\prime} \in \mathcal {S}} P (s ^ {\prime} | s, a) \left(\min  _ {a ^ {\prime} \in \mathcal {A}} f (s ^ {\prime}, g (s, x, a, s ^ {\prime}), a ^ {\prime}) - \min  _ {a ^ {\prime} \in \mathcal {A}} f ^ {\prime} (s ^ {\prime}, g (s, x, a, s ^ {\prime}), a ^ {\prime})\right) ^ {2} \\ \leq \sum_ {(s, x, a) \in \mathcal {S} \times \mathcal {X} \times \mathcal {A}} \sum_ {s ^ {\prime} \in \mathcal {S}} P (s ^ {\prime} | s, a) \left(f (s ^ {\prime}, g (s, x, a, s ^ {\prime}), \pi_ {f, f ^ {\prime}} (s ^ {\prime}, g (s, x, a, s ^ {\prime}))\right) \\ \left. - f ^ {\prime} \left(s ^ {\prime}, g (s, x, a, s ^ {\prime}), \pi_ {f, f ^ {\prime}} \left(s ^ {\prime}, g (s, x, a, s ^ {\prime})\right)\right)\right) ^ {2} \\ = \| f - f ^ {\prime} \| _ {2, P (v) \times \pi_ {f, f ^ {\prime}}} ^ {2}. \\ \end{array}
$$
Lemma A.4. Under Assumptions A.3 and A.4, for any admissible distribution  $\nu \in \Delta (\mathcal{S}\times \mathcal{X}\times \mathcal{A})$  and a data distribution  $\bar{\mu}_{\beta}$  associated with a virtual sample distribution  $\beta (x)$ , define  $P(\nu)$  as a distribution generated as  $s_i^\prime \sim P(\nu)$ , then for any policy  $\pi$ , and  $f,f^{\prime}:\mathcal{S}\times \mathcal{X}\times \mathcal{A}\to \mathbb{R}$ , we have
$$
\left\| f - Q ^ {*} \right\| _ {\underline {{2}}, \nu} \leq \sqrt {\frac {(m + 1) C}{m \sigma}} \left\| f - \mathcal {T} f ^ {\prime} \right\| _ {\underline {{2}}, \bar {\mu} _ {\beta}} + \gamma \left\| f ^ {\prime} - Q ^ {*} \right\| _ {2, P (\nu) \times \pi_ {f ^ {\prime}, Q ^ {*}}} \tag {10}
$$
$$
\begin{array}{l} \| f - Q ^ {*} \| _ {2, \nu} = \| f - \mathcal {T} f ^ {\prime} + \mathcal {T} f ^ {\prime} - Q ^ {*} \| _ {2, \nu} \\ \leq_ {(1)} \| f - \mathcal {T} f ^ {\prime} \| _ {2, \nu} + \| \mathcal {T} f ^ {\prime} - Q ^ {*} \| _ {2, \nu} \\ \leq_ {(2)} \sqrt {\frac {(m + 1) C}{m \sigma}} \| f - \mathcal {T} f ^ {\prime} \| _ {\underline {{2}}, \bar {\mu} _ {\beta}} + \| \mathcal {T} f ^ {\prime} - Q ^ {*} \| _ {\underline {{2}}, \nu} \\ \leq_ {(3)} \sqrt {\frac {(m + 1) C}{m \sigma}} \| f - \mathcal {T} f ^ {\prime} \| _ {\underline {{2}}, \bar {\mu} _ {\beta}} + \gamma \| V _ {f ^ {\prime}} - V ^ {*} \| _ {2, P (\nu)} \\ \leq \sqrt {\frac {(m + 1) C}{m \sigma}} \| f - \mathcal {T} f ^ {\prime} \| _ {\underline {{2}}, \bar {\mu} _ {\beta}} + \gamma \| f ^ {\prime} - Q ^ {*} \| _ {2, P (\nu) \times \pi_ {f ^ {\prime}, Q ^ {*}}}, \\ \end{array}
$$
where inequality (1) holds because of triangle inequality, inequality (2) comes from lemma A.2, inequality (3) holds because
$$
\begin{array}{l} \left\| \mathcal {T} f ^ {\prime} - Q ^ {*} \right\| _ {2, \nu} ^ {2} = \left\| \mathcal {T} ^ {*} f ^ {\prime} - \mathcal {T} Q ^ {*} \right\| _ {2, \nu} ^ {2} = \mathbb {E} _ {(s, x, a) \sim \nu} \left[ \left(\left(\mathcal {T} f ^ {\prime}\right) (s, x, a) - \left(\mathcal {T} Q ^ {*}\right) (s, x, a)\right) ^ {2} \right] \\ = \mathbb {E} _ {(s, x, a) \sim \nu} \left[ \left(\gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ V _ {f ^ {\prime}} \left(s ^ {\prime}, g (s, x, a, s ^ {\prime})\right) - V ^ {*} \left(s ^ {\prime}, g (s, x, a, s ^ {\prime})\right) \right]\right) ^ {2} \right] \\ \leq \gamma^ {2} \mathbb {E} _ {(s, x, a) \sim \nu , s ^ {\prime} \sim P (\cdot | s, a)} \left[ \left(V _ {f ^ {\prime}} \left(s ^ {\prime}, g (s, x, a, s ^ {\prime})\right) - V ^ {*} \left(s ^ {\prime}, g (s, x, a, s ^ {\prime})\right)\right) ^ {2} \right] \\ = \gamma^ {2} \mathbb {E} _ {s ^ {\prime} \sim P (\nu)} \left[ \right.\left( \right.V _ {f ^ {\prime}} \left(s ^ {\prime}, g (s, x, a, s ^ {\prime})\right) - V ^ {*} \left(s ^ {\prime}, g (s, x, a, s ^ {\prime}))\right) ^ {2} \left. \right] \\ = \gamma^ {2} \| V _ {f ^ {\prime}} - V ^ {*} \| _ {2, P (\nu)} ^ {2}, \\ \end{array}
$$
and the last inequality holds due to Lemma A.3.
Lemma A.5. For a given data sample  $(s,x,a,r,s',a')$  generated from a data distribution  $\mu$ , such that  $(s,x,a) \sim \mu, s' \sim P(\cdot | s, a)$ ,  $x' = g(s,x,a,s')$ , for any  $f, f' \in \mathcal{F}$ , define  $V_f(s,x) = \min_{a'} f(s,x,a')$ , then
$$
\begin{array}{l} \mathbb {E} \left[ \left(f (s, x, a) - r - \gamma V _ {f ^ {\prime}} \left(s ^ {\prime}, x ^ {\prime}\right)\right) ^ {2} \right] \\ = \| f - \mathcal {T} f ^ {\prime} \| _ {2, \mu} ^ {2} + \gamma^ {2} \mathbb {E} _ {(s, x, a) \sim \mu} [ \operatorname {V a r} \left(V _ {f ^ {\prime}} \left(s ^ {\prime}, x ^ {\prime}\right) \mid s, x, a\right) ] \tag {11} \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} \left[ \left(f (s, x, a) - r - \gamma V _ {f ^ {\prime}} \left(s ^ {\prime}, x ^ {\prime}\right)\right) ^ {2} \right] \\ = \mathbb {E} \left[ \left(f (s, x, a) - (\mathcal {T} f ^ {\prime}) (s, x, a) + (\mathcal {T} f ^ {\prime}) (s, x, a) - \left(r + \gamma V _ {f ^ {\prime}} \left(s ^ {\prime}, x ^ {\prime}\right)\right)\right) ^ {2} \right] \\ = \mathbb {E} \left[ \left(f (s, x, a) - \left(\mathcal {T} f ^ {\prime}\right) (s, x, a)\right) ^ {2} \right] + \mathbb {E} \left[ \left(\left(\mathcal {T} f ^ {\prime}\right) (s, x, a) - \left(r + \gamma V _ {f ^ {\prime}} \left(s ^ {\prime}, x ^ {\prime}\right)\right)\right) ^ {2} \right] \\ + \underbrace {2 \mathbb {E} \left[ \left(f (s , x , a) - \left(\mathcal {T} f ^ {\prime}\right) (s , x , a)\right) \left(\left(\mathcal {T} f ^ {\prime}\right) (s , x , a) - \left(r + \gamma V _ {f ^ {\prime}} \left(s ^ {\prime} , x ^ {\prime}\right)\right)\right) \right]} _ {(1) = 0} \\ = \mathbb {E} \left[ (f (s, x, a) - (\mathcal {T} f ^ {\prime}) (s, x, a)) ^ {2} \right] + \gamma^ {2} \mathbb {E} _ {(s, x, a) \sim \mu} [ \operatorname {V a r} \left(V _ {f ^ {\prime}} \left(s ^ {\prime}, x ^ {\prime}\right) | s, x, a\right) ] \\ = \| f - \mathcal {T} f ^ {\prime} \| _ {2, \mu} ^ {2} + \gamma^ {2} \mathbb {E} _ {(s, x, a) \sim \mu} [ \operatorname {V a r} (V _ {f ^ {\prime}} (s ^ {\prime}, x ^ {\prime}) | s, x, a) ], \\ \end{array}
$$
where the equation  $(1) = 0$  because that condition on  $(s,x,a)$ , we have  $f$  and  $V_{f'}$  are independent.
Lemma A.6. Under Algorithm 2, at iteration  $k$ , we have
$$
\mathcal {L} _ {\hat {\mu} _ {\beta_ {k}}} \left(f; f ^ {\prime}\right) - \mathcal {L} _ {\hat {\mu} _ {\beta_ {k}}} \left(\mathcal {T} f; f ^ {\prime}\right) = \| f - \mathcal {T} f ^ {\prime} \| _ {2, \bar {\mu} _ {\beta_ {k}}} ^ {2}, \tag {12}
$$
where  $\mathcal{L}_{\hat{\mu}_{\beta_k}}(f;f') = \mathbb{E}[\mathcal{L}_{\hat{D}_k}(f;f')]$
Proof. Recall that  $\hat{D}_k = D \cup D_k$  and  $|\hat{D}_k| = nm + n$ . The expectation is w.r.t. the random draw of the dataset  $D$  and the random generation of dataset  $D_k$  with virtual sample distribution  $\beta_k$ . We know that
$$
\begin{array}{l} \mathcal {L} _ {\hat {D} _ {k}} \left(f; f ^ {\prime}\right) = \frac {1}{\left| \hat {D} _ {k} \right|} \sum_ {(s, x, a, r, s ^ {\prime}, x ^ {\prime}) \in D} \left(f (s, x, a) - r - \gamma V _ {f ^ {\prime}} \left(s ^ {\prime}, x ^ {\prime}\right)\right) ^ {2} \\ + \frac {1}{\left| \hat {D} _ {k} \right|} \sum_ {(s, x, a, r, s ^ {\prime}, x ^ {\prime}) \in D _ {k}} \left(f (s, x, a) - r - \gamma V _ {f ^ {\prime}} \left(s ^ {\prime}, x ^ {\prime}\right)\right) ^ {2} \\ \end{array}
$$
Let  $\mathcal{M}_k^{(s,x,a,s',x')}$  denote the set of virtual samples that are associated with the real sample  $(s,x,a,s',x')$  at iteration  $k$ . Then
$$
\begin{array}{l} \mathcal {L} _ {\hat {\mu} _ {\beta_ {k}}} (f; f ^ {\prime}) := \mathbb {E} [ \mathcal {L} _ {\hat {D} _ {k}} (f; f ^ {\prime}) ] \\ = \frac {n}{n m + n} \left(\| f - \mathcal {T} f ^ {\prime} \| _ {2, \mu} ^ {2} + \gamma^ {2} \mathbb {E} _ {(s, x, a) \sim \mu} [ \operatorname {V a r} \left(V _ {f ^ {\prime}} \left(s ^ {\prime}, x ^ {\prime}\right) | s, x, a\right) ]\right) \quad (\text {b y}) \\ + \frac {1}{n m + n} \mathbb {E} \left[ \sum_ {(s, x, a, r, s ^ {\prime}, x ^ {\prime}) \in D} \sum_ {(s, \bar {x}, a, \bar {r}, \bar {s} ^ {\prime}, \bar {x} ^ {\prime}) \in \mathcal {M} _ {k} ^ {(s, x, a, r, s ^ {\prime}, x ^ {\prime})}} \left(f (s, \bar {x}, a) - \bar {r} - \gamma V _ {f ^ {\prime}} (\bar {s} ^ {\prime}, \bar {x} ^ {\prime})\right) ^ {2} \right] \\ = \frac {n}{n m + n} \left(\| f - \mathcal {T} f ^ {\prime} \| _ {2, \mu} ^ {2} + \gamma^ {2} \mathbb {E} _ {(s, x, a) \sim \mu} [ \operatorname {V a r} \left(V _ {f ^ {\prime}} \left(s ^ {\prime}, x ^ {\prime}\right) | s, x, a\right) ]\right) \\ + \frac {1}{n m + n} \mathbb {E} \left[ \sum_ {(s, x, a, r, s ^ {\prime}, x ^ {\prime}) \in D} \mathbb {E} \left[ \sum_ {(s, \bar {x}, a, \bar {r}, \bar {s} ^ {\prime}, \bar {x} ^ {\prime}) \in \mathcal {M} _ {k} ^ {(s, x, a, r, s ^ {\prime}, x ^ {\prime})}} (f (s, \bar {x}, a) - \bar {r} - \gamma V _ {f ^ {\prime}} (\bar {s} ^ {\prime}, \bar {x} ^ {\prime})) ^ {2} \mid s, x, a, r, s ^ {\prime}, x ^ {\prime} \right] \right] \\ = \frac {n}{n m + n} \left(\| f - \mathcal {T} f ^ {\prime} \| _ {2, \mu} ^ {2} + \gamma^ {2} \mathbb {E} _ {(s, x, a) \sim \mu} [ \operatorname {V a r} \left(V _ {f ^ {\prime}} \left(s ^ {\prime}, x ^ {\prime}\right) | s, x, a\right) ]\right) \\ \end{array}
$$
$$
\begin{array}{l} + \frac {m}{n m + n} \mathbb {E} \left[ \sum_ {(s, x, a, r, s ^ {\prime}, x ^ {\prime}) \in D} \sum_ {\bar {x} \in \mathcal {X}} \beta_ {k} (\bar {x}) \left(f (s, \bar {x}, a) - \bar {r} - \gamma V _ {f ^ {\prime}} (\bar {s} ^ {\prime}, \bar {x} ^ {\prime})\right) ^ {2} \right] \\ = \frac {n}{n m + n} \left(\| f - \mathcal {T} f ^ {\prime} \| _ {2, \mu} ^ {2} + \gamma^ {2} \mathbb {E} _ {(s, x, a) \sim \mu} [ \operatorname {V a r} \left(V _ {f ^ {\prime}} \left(s ^ {\prime}, x ^ {\prime}\right) | s, x, a\right) ]\right) \\ + \frac {m n}{n m + n} \sum_ {(s, x, a) \in \mathcal {S} \times \mathcal {X} \times \mathcal {A}} \mu (s, x, a) \sum_ {\bar {x} \in \mathcal {X}} \beta_ {k} (\bar {x}) \left(f (s, \bar {x}, a) - \bar {r} - \gamma V _ {f ^ {\prime}} (\bar {s} ^ {\prime}, \bar {x} ^ {\prime})\right) ^ {2} \\ = \frac {n}{n m + n} \left(\| f - \mathcal {T} f ^ {\prime} \| _ {2, \mu} ^ {2} + \gamma^ {2} \mathbb {E} _ {(s, x, a) \sim \mu} [ \operatorname {V a r} (V _ {f ^ {\prime}} (s ^ {\prime}, x ^ {\prime}) | s, x, a) ]\right) \\ + \frac {m n}{n m + n} \sum_ {(s, \bar {x}, a) \in \mathcal {S} \times \mathcal {X} \times \mathcal {A}} \mu (s, a) \beta_ {k} (\bar {x}) \left(f (s, \bar {x}, a) - \bar {r} - \gamma V _ {f ^ {\prime}} (\bar {s} ^ {\prime}, \bar {x} ^ {\prime})\right) ^ {2} \\ = \frac {n}{n m + n} \left(\| f - \mathcal {T} f ^ {\prime} \| _ {2, \mu} ^ {2} + \gamma^ {2} \mathbb {E} _ {(s, x, a) \sim \mu} [ \operatorname {V a r} \left(V _ {f ^ {\prime}} \left(s ^ {\prime}, x ^ {\prime}\right) | s, x, a\right) ]\right) \\ + \frac {n m}{n m + n} \left(\| f - \mathcal {T} f ^ {\prime} \| _ {2, \mu_ {k}} ^ {2} + \gamma^ {2} \mathbb {E} _ {(s, x, a) \sim \mu_ {k}} [ \operatorname {V a r} \left(V _ {f ^ {\prime}} \left(s ^ {\prime}, x ^ {\prime}\right) | s, x, a\right) ]\right), \quad (\text {b y}) \\ \end{array}
$$
where  $\bar{r} = R(s,\bar{x},a),\mu_k(s,x,a) = \sum_{x'\in \mathcal{X}}\mu (s,x',a)\beta_k(x) = \mu (s,a)\beta_k(x)$ . Since we have  $\bar{\mu}_{\beta_k}(s,x,a) = \frac{1}{m + 1}\mu (s,x,a) + \frac{m}{m + 1}\mu (s,a)\beta_k(x)$ .
$$
\mathcal {L} _ {\hat {\mu} _ {\beta_ {k}}} (f; f ^ {\prime}) - \mathcal {L} _ {\hat {\mu} _ {\beta_ {k}}} (\mathcal {T} f ^ {\prime}; f ^ {\prime}) = \| f - \mathcal {T} f ^ {\prime} \| _ {2, \bar {\mu} _ {\beta_ {k}}} ^ {2}.
$$
The next lemma shows an upper bound on  $\| f_{k + 1} - \mathcal{T}f_k\|_{2,\bar{\mu}_{\beta_k}}^2$
Lemma A.7. Given the MDP  $M = (\mathcal{S},\mathcal{X},P,R,\gamma)$ , we assume that the  $Q$ -function classes  $\mathcal{F}$  satisfies  $\forall f\in \mathcal{F},\mathcal{T}f\in \mathcal{F}$ . The dataset  $D$  is generated as:  $(s,x,a)\sim \mu ,r = R(s,x,a),s^{\prime}\sim P(\cdot |s,a),x^{\prime} = g(s,x,a,s^{\prime})$ , and the new dataset  $\hat{D}_k = D\cup D_k$  is generated by following Alg. 1 with virtual sample generating distribution  $\beta_{k}(x)$  at kth iteration. Then with probability at least  $1 - \delta, \forall f\in \mathcal{F}$ , and  $k = 0,\ldots ,K$  we hvae
$$
\left\| f _ {k + 1} - \mathcal {T} f _ {k} \right\| _ {2, \bar {\mu} _ {\beta_ {k}}} ^ {2} \leq 5 \left(\frac {1}{n} + \frac {1}{m}\right) V _ {\max } ^ {2} \log (n K | \mathcal {F} | ^ {2} / \delta) + \frac {3 \delta V _ {\max } ^ {2}}{n} \tag {13}
$$
Proof. Using Lemma A.6 we know that
$$
\| f - \mathcal {T} f ^ {\prime} \| _ {2, \bar {\mu} _ {\beta_ {k}}} ^ {2} = \mathcal {L} _ {\hat {\mu} _ {\beta_ {k}}} (f; f ^ {\prime}) - \mathcal {L} _ {\hat {\mu} _ {\beta_ {k}}} (\mathcal {T} f; f ^ {\prime}).
$$
Then it is sufficient to bound  $\| f - \mathcal{T}f^{\prime}\|_{2,\bar{\mu}_{\beta_k}}^2$  by bounding
$$
\mathcal {L} _ {\hat {\mu} _ {\beta_ {k}}} (f; f ^ {\prime}) - \mathcal {L} _ {\hat {\mu} _ {\beta_ {k}}} (\mathcal {T} f; f ^ {\prime}) = \mathbb {E} [ \mathcal {L} _ {\hat {D} _ {k}} (f; f ^ {\prime}) - \mathcal {L} _ {\hat {D} _ {k}} (\mathcal {T} f; f ^ {\prime}) ].
$$
For any  $f, f'$ , recall that
$$
\begin{array}{l} \mathcal {L} _ {\hat {D} _ {k}} \left(f; f ^ {\prime}\right) = \underbrace {\frac {1}{| \hat {D} _ {k} |} \sum_ {(s , x , a , r , s ^ {\prime} , x ^ {\prime}) \in D} \left(f (s , x , a) - r - \gamma V _ {f ^ {\prime}} \left(s ^ {\prime} , x ^ {\prime}\right)\right) ^ {2}} _ {\mathcal {L} _ {D} \left(f ; f ^ {\prime}\right)} \\ + \underbrace {\frac {1}{| \hat {D} _ {k} |} \sum_ {(s , x , a , r , s ^ {\prime} , x ^ {\prime}) \in D _ {k}} \left(f (s , x , a) - r - \gamma V _ {f ^ {\prime}} (s ^ {\prime} , x ^ {\prime})\right) ^ {2}} _ {\mathcal {L} _ {D _ {k}} (f ; f ^ {\prime})}. \\ \end{array}
$$
For any  $f, f'$  define
$$
Y (f; f ^ {\prime}) := \left(f (s, x, a) - r - \gamma V _ {f ^ {\prime}} \left(s ^ {\prime}, x ^ {\prime}\right)\right) ^ {2} - \left(\mathcal {T} f ^ {\prime} (s, x, a) - r - \gamma V _ {f ^ {\prime}} \left(s ^ {\prime}, x ^ {\prime}\right)\right) ^ {2}
$$
Then for each  $(s,x,a,s^{\prime},x^{\prime})\in D$  we get i.i.d. variables  $Y_{1}(f;f^{\prime}),\ldots ,Y_{n}(f;f^{\prime})$
$$
X _ {i} (f; f ^ {\prime}) := \left(f \left(s _ {i}, \hat {x} _ {i}, a _ {i}\right) - \hat {r} _ {i} - \gamma V _ {f ^ {\prime}} \left(s _ {i} ^ {\prime}, \hat {x} _ {i} ^ {\prime}\right)\right) ^ {2} - \left(\mathcal {T} f ^ {\prime} \left(s _ {i}, \hat {x} _ {i}, a _ {i}\right) - \hat {r} _ {i} - \gamma V _ {f ^ {\prime}} \left(s _ {i} ^ {\prime}, \hat {x} _ {i} ^ {\prime}\right)\right) ^ {2},
$$
where  $(s_i, \hat{x}_i, a_i, \hat{r}_i, s_i', \hat{x}_i')$  is an augmented sample based on the  $i$ th real sample  $(s_i, x_i, a_i, r_i, s_i')$ . Denote the  $m$  i.i.d virtual samples by  $X_{i_1}(f; f'), \ldots, X_{i_m}(f; f')$ . Therefore
$$
\mathcal {L} _ {\hat {D} _ {k}} \left(f; f ^ {\prime}\right) - \mathcal {L} _ {\hat {D} _ {k}} \left(\mathcal {T} f ^ {\prime}; f ^ {\prime}\right) = \frac {n}{n m + n} \times \frac {1}{n} \sum_ {i = 1} ^ {n} Y _ {i} \left(f; f ^ {\prime}\right) + \frac {n m}{n m + n} \times \frac {1}{n m} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {m} X _ {i j} \left(f; f ^ {\prime}\right). \tag {14}
$$
Taking the expectations on both sides, we obtain for any  $f,f^{\prime}\in \mathcal{F}$
$$
\mathcal {L} _ {\hat {\mu} _ {\beta_ {k}}} (f; f ^ {\prime}) - \mathcal {L} _ {\hat {\mu} _ {\beta_ {k}}} (\mathcal {T} f ^ {\prime}; f ^ {\prime}) = \frac {n}{n m + n} \mathbb {E} [ Y (f; f ^ {\prime}) ] + \frac {n m}{n m + n} \times \frac {1}{n} \mathbb {E} \left[ \sum_ {i = 1} ^ {n} X _ {i} (f; f ^ {\prime}) \right]
$$
We need to introduce  $\frac{1}{n}\sum_{i = 1}^{n}Y_{i}(f;f^{\prime})$  and  $\frac{1}{m}\sum_{i = 1}^{n}\sum_{j = 1}^{m}X_{ij}(f;f^{\prime})$  to bound the above terms. For the first term, we know that the variance of  $Y$  can be bounded by:
$\operatorname{Var}(Y(f; f')) \leq \mathbb{E}[Y(f; f')^2]$
$$
\begin{array}{l} = \mathbb {E} \left[ \left(\left(f (s, x, a) - r - \gamma V _ {f ^ {\prime}} \left(s ^ {\prime}, x ^ {\prime}\right)\right) ^ {2} - \left(\mathcal {T} f ^ {\prime} (s, x, a) - r - \gamma V _ {f} \left(s ^ {\prime}, x ^ {\prime}\right)\right) ^ {2}\right) ^ {2} \right] \\ = \mathbb {E} \left[ \left(f (s, x, a) - \mathcal {T} f ^ {\prime} (s, x, a)\right) ^ {2} \left(f (s, x, a) + \mathcal {T} f ^ {\prime} (s, x, a) - 2 r - 2 \gamma V _ {f ^ {\prime}} \left(s ^ {\prime}, x ^ {\prime}\right)\right) ^ {2} \right] \\ \leq 4 V _ {\max} ^ {2} \mathbb {E} \left[ \left(f (s, x, a) - \mathcal {T} f ^ {\prime} (s, x, a)\right) ^ {2} \right] \\ = 4 V _ {\max } ^ {2} \| f - \mathcal {T} f ^ {\prime} \| _ {2, \mu} ^ {2} \\ = 4 V _ {\max } ^ {2} \mathbb {E} \left[ Y \left(f; f ^ {\prime}\right) \right], \tag {15} \\ \end{array}
$$
where the last equality is true because
$$
\begin{array}{l} \mathbb {E} [ Y (f; f ^ {\prime}) ] = \mathbb {E} [ \mathcal {L} _ {D} (f; f ^ {\prime}) ] - \mathbb {E} [ \mathcal {L} _ {D} (\mathcal {T} f ^ {\prime}; f ^ {\prime}) ] \\ = \| f - \mathcal {T} f ^ {\prime} \| _ {2, \mu} ^ {2} + \gamma^ {2} \mathbb {E} _ {(s, x, a) \sim \mu} [ \operatorname {V a r} (V _ {f ^ {\prime}} (s ^ {\prime}, x ^ {\prime}) | s, x, a ] \\ - \| \mathcal {T} f ^ {\prime} - \mathcal {T} f ^ {\prime} \| _ {2, \mu} ^ {2} - \gamma^ {2} \mathbb {E} _ {(s, x, a) \sim \mu} [ \operatorname {V a r} (V _ {f ^ {\prime}} (s ^ {\prime}, x ^ {\prime}) | s, x, a ] \quad (\text {u s i n g L e m m a A . 5}) \\ = \| f - \mathcal {T} f ^ {\prime} \| _ {2, \mu} ^ {2}. \\ \end{array}
$$
Then by applying Bernstein's inequality, together with a union bound over all  $f, f' \in \mathcal{F}$ , we obtain with probability  $1 - \delta$  we have
$$
\begin{array}{l} \mathbb {E} \left[ Y (f; f ^ {\prime}) \right] - \frac {1}{n} \sum_ {i = 1} ^ {n} Y _ {i} (f; f ^ {\prime}) \leq \sqrt {\frac {2 \operatorname {V a r} (Y (f ; f ^ {\prime})) \log (| \mathcal {F} | ^ {2} / \delta)}{n}} + \frac {4 V _ {\max} ^ {2} \log (| \mathcal {F} | ^ {2} / \delta)}{3 n} \\ \leq \sqrt {\frac {8 V _ {\operatorname* {m a x}} ^ {2} \mathbb {E} [ Y (f ; f ^ {\prime}) ] \log (| \mathcal {F} | ^ {2} / \delta)}{n}} + \frac {4 V _ {\operatorname* {m a x}} ^ {2} \log (| \mathcal {F} | ^ {2} / \delta)}{3 n} \tag {16} \\ \end{array}
$$
For the second term, note that for any given  $i$ th sample  $(s_i, x_i, a_i, s_i', x_i')$  all the variables  $\{X_{i_j}\}$  are i.i.d. Then following a similar argument, then for all  $f, f' \in \mathcal{F}$ , we have with probability at least  $1 - \delta / n$ ,
$$
\begin{array}{l} \mathbb {E} \left[ X _ {i} \left(f; f ^ {\prime}\right) \mid s _ {i}, x _ {i}, a _ {i} \right] - \frac {1}{m} \sum_ {j = 1} ^ {m} X _ {i _ {j}} \left(f; f ^ {\prime}\right) \\ \leq \sqrt {\frac {8 V _ {\operatorname* {m a x}} ^ {2} \mathbb {E} [ X _ {i} (f ; f ^ {\prime}) | s _ {i} , x _ {i} , a _ {i} ] \log (n | \mathcal {F} | / \delta)}{m}} + \frac {4 V _ {\operatorname* {m a x}} ^ {2} \log (n | \mathcal {F} | ^ {2} / \delta)}{3 m} \\ \end{array}
$$
Then it is easy to obtain that we have for all  $f, f' \in \mathcal{F}$ ,
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} \left[ X _ {i} (f; f ^ {\prime}) - \frac {1}{m} \sum_ {j = 1} ^ {m} X _ {i _ {j}} (f; f ^ {\prime}) \right]
$$
$$
\leq \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\sqrt {\frac {8 V _ {\operatorname* {m a x}} ^ {2} \mathbb {E} \left[ X _ {i} \left(f ; f ^ {\prime}\right) \right] \log \left(n \mid \mathcal {F} \right| / \delta}{m}} + \frac {4 V _ {\operatorname* {m a x}} ^ {2} \log \left(n \mid \mathcal {F} \right| ^ {2} / \delta)}{3 m}\right) + \frac {\delta V _ {\operatorname* {m a x}} ^ {2}}{n} \tag {17}
$$
Combining Eq.(17) and Eq.(16) we can obtain with probability at least  $1 - \delta$ , for all  $f, f' \in \mathcal{F}$ ,
$$
\begin{array}{l} \frac {n}{n + n m} \times \mathbb {E} \left[ Y (f; f ^ {\prime}) \right] - \frac {n}{n + n m} \times \frac {1}{n} \sum_ {i = 1} ^ {n} Y _ {i} (f; f ^ {\prime}) + \frac {n m}{n m + n} \times \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} \left[ X _ {i} (f; f ^ {\prime}) - \frac {1}{m} \sum_ {j = 1} ^ {m} X _ {i _ {j}} (f; f ^ {\prime}) \right] \\ \leq \frac {1}{1 + m} \times \left(\sqrt {\frac {8 V _ {\operatorname* {m a x}} ^ {2} \mathbb {E} [ Y (f ; f ^ {\prime}) ] \log (| \mathcal {F} | ^ {2} / \delta)}{n}} + \frac {4 V _ {\operatorname* {m a x}} ^ {2} \log (| \mathcal {F} | ^ {2} / \delta)}{3 n}\right) \\ + \frac {n m}{n m + n} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \left(\sqrt {\frac {8 V _ {\operatorname* {m a x}} ^ {2} \mathbb {E} \left[ X _ {i} \left(f ; f ^ {\prime}\right) \right] \log (n | \mathcal {F} | / \delta)}{m}} + \frac {4 V _ {\operatorname* {m a x}} ^ {2} \log (n | \mathcal {F} | ^ {2} / \delta)}{3 m}\right) + \frac {\delta V _ {\operatorname* {m a x}} ^ {2}}{n}\right) \tag {18} \\ \end{array}
$$
Let  $f = f_{k + 1}, f' = f_k$ , then according to Algorithm 2 we know that
$$
f_{k + 1} = \mathcal{\tilde{T}}_{k,\mathcal{F}}f_{k}:= \operatorname *{arg  min}_{f\in \mathcal{F}}\mathcal{L}_{\mathcal{D}_{k}}(f;f_{k}).
$$
According to Eq. (14), we have