SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Part (ii) follows immediately from the definition of  $K$ . Now consider (iii). Write  $\mathbb{F} = \mathbb{F}_1 + \mathbb{F}_2 = \mathbb{G}_1 \oplus \mathbb{G}_2$ , where  $\mathbb{G}_1 = \mathbb{F}_1$  and  $\mathbb{G}_2 = \mathbb{F}_2 \cap \mathbb{F}_1^\perp$  satisfies  $\mathbb{G}_2 \perp \mathbb{G}_1$ . Let  $f \in \mathbb{F}$ ,  $f \neq 0$ , be arbitrary and write  $f = g_1 + g_2$  for  $g_i \in \mathbb{G}_i$ ,  $i = 1, 2$ . Then
$$
\begin{array}{l} \frac {\| L (\theta) (f) \| _ {\mathbb {Y}} ^ {2}}{\| f \| _ {\mathbb {X}} ^ {2}} \leq 2 \frac {\| L (\theta) (g _ {1}) \| _ {\mathbb {Y}} ^ {2} + \| L (\theta) (g _ {2}) \| _ {\mathbb {Y}} ^ {2}}{\| g _ {1} \| _ {\mathbb {X}} ^ {2} + \| g _ {2} \| _ {\mathbb {X}} ^ {2}} \\ \leq 2 \left(\frac {\| L (\theta) (g _ {1}) \| _ {\mathbb {Y}} ^ {2}}{\| g _ {1} \| _ {\mathbb {X}} ^ {2}} + \frac {\| L (\theta) (g _ {2}) \| _ {\mathbb {Y}} ^ {2}}{\| g _ {2} \| _ {\mathbb {X}} ^ {2}}\right) \\ \leq 2 \left(K (\mathbb {G} _ {1}) (\theta) + K (\mathbb {G} _ {2}) (\theta)\right) \\ \leq 2 \left(K \left(\mathbb {F} _ {1}\right) (\theta) + K \left(\mathbb {F} _ {2}\right) (\theta)\right), \\ \end{array}
$$
where in the last step we used the fact that  $\mathbb{G}_2\subseteq \mathbb{F}_2$  . Since  $f$  was arbitrary, the result now follows.
Finally, consider (iv). By the definition of  $K$ ,  $\mathcal{B}(\mathbb{F})$  and linearity,
$$
\begin{array}{l} K (\mathcal {B} (\mathbb {F})) (\theta) = \sup  \left\{\| L (\theta) (f / \| f \| _ {\mathbb {X}}) \| _ {\mathbb {Y}} ^ {2}: f \in \mathbb {F} \backslash \{0 \} \right\} \\ = \sup \left\{\frac {\| L (\theta) (f) \| _ {\mathbb {Y}} ^ {2}}{\| f \| _ {\mathbb {X}} ^ {2}}: f \in \mathbb {F} \backslash \{0 \} \right\} = K (\mathbb {F}) (\theta), \\ \end{array}
$$
For the remaining results it is useful to introduce some additional notation. Let
$$
\overline {{\mathbb {Y}}} = \oplus_ {c = 1} ^ {C} \oplus_ {i = 1} ^ {m _ {c}} \mathbb {Y} _ {c}
$$
be the overall measurement space. Note that this is a Hilbert space, when equipped with suitable norm. If  $y = (y_{ic})$  is an arbitrary element of  $\mathbb{Y}$ , we take this norm to be
$$
\| y \| _ {\overline {{\mathbb {Y}}}} = \sqrt {\sum_ {c = 1} ^ {C} \sum_ {i = 1} ^ {m _ {c}} \| y _ {i c} \| _ {\mathbb {Y} _ {c}} ^ {2}}.
$$
Now let  $m = m_{1} + \ldots +m_{C}$ $D = \otimes_{c = 1}^{C}\otimes_{i = 1}^{m_c}D_c$  be the tensor-product space and write  $(\theta_{ic})$  for an arbitrary element of  $D$  , where  $\theta_{ic}\in D_c$  for  $i = 1,\dots ,m_c$  and  $c = 1,\dots ,C$  . Then we define the overall sampling operator
$$
M: D \rightarrow \mathcal {B} \left(\mathbb {X} _ {0}, \overline {{\mathbb {Y}}}\right), \theta := \left(\theta_ {i c}\right) \mapsto \left(\frac {1}{\sqrt {\nu_ {c} \left(\theta_ {i c}\right) m _ {c}}} L _ {c} \left(\theta_ {i c}\right)\right). \tag {E.2}
$$
Note that the operator on the right-hand side is given by
$$
f \in \mathbb {X} _ {0} \mapsto \left(\frac {1}{\sqrt {\nu_ {c} (\theta_ {i c}) m _ {c}}} L _ {c} (\theta_ {i c}) (f)\right) \in \overline {{\mathbb {Y}}}.
$$
Further, we note that the nondegeneracy condition is equivalent to
$$
\alpha \| f \| _ {\mathbb {X}} ^ {2} \leq \mathbb {E} _ {\overline {{\Theta}} \sim \tau} \| M (\overline {{\Theta}}) (f) \| _ {\mathbb {Y}} ^ {2} \leq \beta \| f \| _ {\mathbb {X}} ^ {2}, \quad \forall f \in \mathbb {X} _ {0}. \tag {E.3}
$$
Here  $\tau = \otimes_{c=1}^{C} \otimes_{i=1}^{m_c} \mu_c$  is the probability measure of the full sample  $(\Theta_{ic}) := \overline{\Theta}$ .
Next, given measurements as in (2.1) or (4.4), we define the scaled measurements  $\bar{y} = (\bar{y}_{ic})\in \mathbb{Y}$  by
$$
\bar {y} _ {i c} = \frac {1}{\sqrt {\nu_ {c} \left(\theta_ {i c}\right) m _ {c}}} y _ {i c}, \quad i = 1, \dots , m _ {c}, c = 1, \dots , C. \tag {E.4}
$$
Then, with  $M$  as defined in E.2, the regression problem (2.2) is equivalent to
$$
\hat {f} \in \underset {f \in \mathbb {F}} {\operatorname {a r g m i n}} \| \bar {y} - M (\Theta) (f) \| _ {\overline {{\mathbb {Y}}}} ^ {2}. \tag {E.5}
$$
With this in hand, we now prove Theorem 4.2. For this, it is first useful to prove a special case of Theorem 4.2 when  $\mathbb{F}$  is itself a subspace.
Lemma E.2 (Special case of Theorem 4.2 for subspaces). Consider the setup of Theorem 4.2, where  $\mathbb{F}$  is a subspace of dimension at most  $n$ . Suppose that
$$
m _ {c} \geq c _ {\gamma} \cdot \alpha^ {- 1} \cdot \operatorname * {e s s} _ {\theta \sim \rho_ {c}} \left\{K _ {c} (\mathbb {F}) (\theta) / \nu_ {c} (\theta) \right\} \cdot \log (2 n / \delta), \quad \forall c = 1, \dots , C,
$$
for some  $0 < \gamma < 1$ , where
$$
c _ {\gamma} = \left((1 + \gamma) \log (1 + \gamma) - \gamma\right) ^ {- 1}. \tag {E.6}
$$
Then (2.5) holds with probability at least  $1 - \delta$ .
Proof. The proof of the result in this case is a straightforward application of the matrix Chernoff bound [115]. Let  $\{f_i\}_{i=1}^n$  be an orthonormal basis for  $\mathbb{F}$ . Write an arbitrary  $f \in \mathbb{F}$  as  $f = \sum_{i=1}^n c_i f_i$ , so that  $\|f\|_{\mathbb{X}} = \|c\|_{\ell^2}$ , where  $c = (c_i)_{i=1}^n \in \mathbb{C}^n$ . Then
$$
\begin{array}{l} \| M (\Theta) (f) \| _ {\mathbb {Y}} ^ {2} = \sum_ {c = 1} ^ {C} \frac {1}{m _ {c}} \sum_ {i = 1} ^ {m _ {c}} \frac {1}{\nu_ {c} (\Theta_ {i c})} \| L _ {c} (\Theta_ {i c}) (f) \| _ {\mathbb {Y} _ {c}} ^ {2} \\ = \sum_ {c = 1} ^ {C} \frac {1}{m _ {c}} \sum_ {i = 1} ^ {m _ {c}} \frac {1}{\nu_ {c} (\Theta_ {i c})} \sum_ {j, k = 1} ^ {n} \overline {{c _ {j}}} c _ {k} \langle L _ {c} (\Theta_ {i c}) (f _ {j}), L _ {c} (\Theta_ {i c}) (f _ {k}) \rangle_ {\mathbb {Y} _ {c}} \\ = c ^ {*} \left(\sum_ {c = 1} ^ {C} \sum_ {i = 1} ^ {m _ {c}} X _ {i c}\right) c, \\ \end{array}
$$
where  $X_{ic}$  are the independent, self-adjoint,  $n\times n$  random matrices with
$$
X _ {i c} = \left(\frac {1}{m _ {c} \nu_ {c} (\Theta_ {i c})} \langle L _ {c} (\Theta_ {i c}) (f _ {j}), L _ {c} (\Theta_ {i c}) (f _ {k}) \rangle_ {\mathbb {Y} _ {c}}\right) _ {j, k = 1} ^ {d}.
$$
Thus, the result is equivalent to the condition
$$
(1 - \epsilon) \alpha \leq \lambda_ {\min } \left(\sum_ {c = 1} ^ {C} \sum_ {i = 1} ^ {m _ {c}} X _ {i c}\right) \leq \lambda_ {\max } \left(\sum_ {c = 1} ^ {C} \sum_ {i = 1} ^ {m _ {c}} X _ {i c}\right) \leq (1 + \epsilon) \beta .
$$
Notice that  $X_{ic}\succeq 0$  and, due to (E.3),
$$
\alpha \leq \lambda_ {\min } \left(\sum_ {c = 1} ^ {C} \sum_ {i = 1} ^ {m _ {c}} \mathbb {E} X _ {i c}\right) \leq \lambda_ {\max } \left(\sum_ {c = 1} ^ {C} \sum_ {i = 1} ^ {m _ {c}} \mathbb {E} X _ {i c}\right) \leq \beta .
$$
Now, with  $f = \sum_{i=1}^{n} c_i f_i \in \mathbb{F}$ , we have
$$
c ^ {*} X _ {i c} c = \frac {1}{m _ {c} \nu_ {c} (\Theta_ {i c})} \| L _ {c} (\Theta_ {i c}) (f) \| _ {\mathbb {Y} _ {c}} ^ {2} \leq \frac {1}{m _ {c} \nu_ {c} (\Theta_ {i c})} K _ {c} (\mathbb {F}) (\Theta_ {i c}) \| f \| _ {\mathbb {X}} ^ {2},
$$
$$
\lambda_{\max}(X_{ic})\leq m_{c}^{-1}\operatorname *{ess}\sup_{\theta \sim \rho_{c}}\{K_{c}(\mathbb{F})(\theta) / \nu_{c}(\theta)\} ,
$$
almost surely. The result now follows from the matrix Chernoff bound.
Proof of Theorem 4.2. Let  $f \in \mathcal{B}(\mathbb{F} - \mathbb{F})$ . Then there is  $j \in \{1, \dots, d\}$  and  $f_j \in \mathbb{F}_j \cap \mathcal{B}(\mathbb{F} - \mathbb{F})$  such that  $\| f - f_j \| \leq t$ , where  $t = \sqrt{\alpha \epsilon / 2}$ . Observe that
$$
\| M (\overline {{\Theta}}) (f _ {j}) \| _ {\overline {{\mathbb {Y}}}} + \| M (\overline {{\Theta}}) (f - f _ {j}) \| _ {\overline {{\mathbb {Y}}}} \geq \| M (\overline {{\Theta}}) (f) \| _ {\overline {{\mathbb {Y}}}} \geq \| M (\overline {{\Theta}}) (f _ {j}) \| _ {\overline {{\mathbb {Y}}}} - \| M (\overline {{\Theta}}) (f - f _ {j}) \| _ {\overline {{\mathbb {Y}}}}.
$$
Now, with probability one,
$$
\| M (\overline {{\Theta}}) (f - f _ {j}) \| _ {\mathbb {Y}} ^ {2} = \sum_ {c = 1} ^ {C} \frac {1}{m _ {c}} \sum_ {i = 1} ^ {m _ {c}} \frac {1}{\nu_ {c} (\Theta_ {i c})} \| L _ {c} (\Theta_ {i c}) (f - f _ {j}) \| _ {\mathbb {Y} _ {c}} ^ {2} \leq \| f - f _ {j} \| ^ {2} \leq \alpha \epsilon / 2.
$$
Using this and the fact that  $\beta \geq \alpha$ , we deduce that
$$
\| M (\overline {{\Theta}}) (f _ {j}) \| _ {\overline {{\mathbb {Y}}}} + \beta \epsilon / 2 \geq \| M (\overline {{\Theta}}) (f) \| _ {\overline {{\mathbb {Y}}}} \geq \| M (\overline {{\Theta}}) (f _ {j}) \| _ {\overline {{\mathbb {Y}}}} - \alpha \epsilon / 2.
$$
Now let  $E$  be the event
$$
(1 - \epsilon) \alpha \| f \| _ {\mathbb {X}} ^ {2} \leq \| M (\overline {{\Theta}}) (f) \| _ {\mathbb {Y}} ^ {2} \leq (1 + \epsilon) \beta \| f \| _ {\mathbb {X}} ^ {2}, \quad \forall f \in \mathbb {F} - \mathbb {F},
$$
and note that our goal is to show that  $\mathbb{P}(E) \geq 1 - \delta$ . By linearity,  $E$  is equivalent to
$$
(1 - \epsilon) \alpha \leq \| M (\overline {{\Theta}}) (f) \| _ {\overline {{\mathbb {Y}}}} ^ {2} \leq (1 + \epsilon) \beta , \quad \forall f \in \mathcal {B} (\mathbb {F} - \mathbb {F}).
$$
Hence, by the above argument, we have
$$
\mathbb {P} (E) \geq \mathbb {P} \left(\bigcap_ {j = 1} ^ {d} E _ {j}\right),
$$
where  $E_{j}$  is the event
$$
(1 - \epsilon / 2) \alpha \| f _ {j} \| _ {\mathbb {X}} ^ {2} \leq \| M (\overline {{\Theta}}) (f _ {j}) \| _ {\overline {{\mathbb {Y}}}} ^ {2} \leq (1 + \epsilon / 2) \beta \| f _ {j} \| _ {\mathbb {X}} ^ {2}, \quad \forall f _ {j} \in \mathbb {F} _ {j}.
$$
By the union bound
$$
\mathbb {P} (E ^ {c}) \leq \sum_ {j = 1} ^ {d} \mathbb {P} (E _ {j} ^ {c}).
$$
Suppose first that  $n \geq 1$ . Then  $K(\mathbb{F}_j)(\theta) \leq K(\overline{\mathbb{F}})(\theta)$  and the condition (4.1) on the  $m_c$  and Lemma E.2 give that  $\mathbb{P}(E_j^c) \leq \delta / d$ .
The gives the result for  $n \geq 1$ .
For  $n = 0$ , we have that  $\mathbb{F}_j = \{f_i\}$  is a singleton. Notice that  $\mathbb{F}_j \subseteq \operatorname{span}(\mathbb{F}_j)$  and  $K_c(\mathbb{F}_j)(\theta) = K_c(\operatorname{span}(\mathbb{F}_j))(\theta)$ . Hence we may apply Lemma E.2 with  $n = 1$ . We deduce once more that  $\mathbb{P}(E_j^c) \leq \delta / d$ . This gives the result for  $n = 0$  as well.
Remark E.3 (The norm in Theorem 4.2) Write  $\mathbb{F}' = \mathcal{B}(\mathbb{F} - \mathbb{F})$ . Then it follows from the definition of the  $K_{c}$  that
$$
\| f _ {1} - f _ {2} \| \leq \sqrt {\sum_ {c = 1} ^ {C} \operatorname {e s s s u p} _ {\theta \sim \rho_ {c}} K _ {c} (\mathbb {F} ^ {\prime} - \mathbb {F} ^ {\prime}) / \nu_ {c} (\theta_ {c})} \| f _ {1} - f _ {2} \| _ {\mathbb {X}}, \quad \forall f _ {1}, f _ {2} \in \mathbb {F} ^ {\prime}.
$$
Therefore, the cover in Theorem 4.2 could be constructed with respect to the  $\mathbb{X}$ -norm, up to a change in the constant  $t$ . Notice that the quantity that bounds  $\| f_1 - f_2\|$  in terms of  $\| f_1 - f_2\|_{\mathbb{X}}$  involves the generalized Christoffel functions  $K_{c}(\mathbb{F}^{\prime} - \mathbb{F}^{\prime})$  and weight functions  $\nu_{c}$ . Therefore, sampling according to the generalized Christoffel function (on a slightly modified set) also minimizes this constant, and therefore the size  $d$  of the cover in Theorem 4.2.
Proof of Corollary 4.3. If  $\mathbb{F} - \mathbb{F} = \cup_{i=1}^{d}\mathbb{F}_{i}$  is a union of  $d$  subspaces of dimension at most  $n$ , then clearly
$$
\mathcal {B} (\mathbb {F} - \mathbb {F}) = \cup_ {i = 1} ^ {d} \mathcal {B} (\mathbb {F} _ {i}) \subset \cup_ {i = 1} ^ {d} \mathbb {F} _ {i}.
$$
It follows that  $\{\mathbb{F}_i\}_{i = 1}^d$  provides a  $(\| \cdot \| ,n,t)$ -subspace covering of  $\mathcal{B}(\mathbb{F} - \mathbb{F})$  for any  $t\geq 0$ . Hence we may apply Theorem 4.2 with this cover. Since  $\widetilde{\mathbb{F}} = \cup_{i = 1}^{d}\mathbb{F}_{i} = \mathbb{F} - \mathbb{F}$  in this case, the result now follows immediately.
Proof of Corollary 4.4. When  $n = 0$ , the sets  $\mathbb{F}_j = \{f_j\}$  are singletons and we must have  $\mathbb{F}_j \subseteq \mathcal{B}(\mathbb{F} - \mathbb{F})$  by definition of the covering. Hence  $\widetilde{\mathbb{F}} \subseteq \mathcal{B}(\mathbb{F} - \mathbb{F})$ , which gives that  $K_{c}(\widetilde{\mathbb{F}})(\theta) \leq K_{c}(\mathcal{B}(\mathbb{F} - \mathbb{F}))(\theta)$ . Using Lemma E.1, we deduce that  $K_{c}(\widetilde{\mathbb{F}})(\theta) \leq K_{c}(\mathbb{F} - \mathbb{F})(\theta)$ . This implies the result.
Proof of Lemma 4.6. Since  $K(\mathbb{F})$  is integrable, the quantity  $\kappa(\mathbb{F})$  is well defined. Also,  $K(\mathbb{F})(\theta) > 0$  almost everywhere, due to the assumption on  $L$  and  $\mathbb{F}$ . In particular,  $\kappa(\mathbb{F}) > 0$ . It follows that  $\nu^{\star}$  is measurable, positive almost everywhere and integrable with  $\int_{D} \nu^{\star}(\theta) \, \mathrm{d}\rho(\theta) = 1$ . Clearly,
$$
\operatorname *{ess  sup}_{\theta \sim \rho}\{K(\theta) / \nu^{\star}(\theta)\} = \kappa (\mathbb{F}).
$$
We now show that this is the optimal value. Let  $\nu$  be any other such function. Then
$$
\operatorname * {e s s} _ {\theta \sim \rho} \{K (\mathbb {F}) (\theta) / \nu (\theta) \} \geq \int_ {D} \frac {K (\mathbb {F}) (\theta)}{\nu (\theta)} \nu (\theta)   \mathrm {d} \rho (\theta) = \int_ {D} K (\mathbb {F}) (\theta)   \mathrm {d} \rho (\theta) = \kappa (\mathbb {F}).
$$
This gives the result.
Proof of Corollary 4.7. We first prove the result when  $d = 1$ . Let  $\{f_i\}_{i=1}^n$  be an orthonormal basis for  $\mathbb{F}_1 = \mathbb{F} - \mathbb{F}$ . Then Lemma E.1 gives that
$$
K _ {c} (\mathbb {F} - \mathbb {F}) (\theta) \leq \sum_ {i = 1} ^ {n} \| L _ {c} (\theta) (f _ {i}) \| _ {\mathbb {Y} _ {c}} ^ {2}.
$$
$$
\sum_ {c = 1} ^ {C} \kappa_ {c} (\mathbb {F} - \mathbb {F}) = \sum_ {c = 1} ^ {C} \int_ {D _ {c}} K _ {c} (\mathbb {F} - \mathbb {F}) (\theta) \mathrm {d} \rho_ {c} (\theta) \leq \sum_ {i = 1} ^ {n} \sum_ {c = 1} ^ {C} \int_ {D _ {c}} \| L _ {c} (\theta) (f _ {i}) \| _ {\mathbb {Y} _ {c}} ^ {2} \mathrm {d} \rho_ {c} (\theta).
$$
Assumption 2.4 and orthonormality now imply that
$$
\sum_ {c = 1} ^ {C} \kappa_ {c} (\mathbb {F} - \mathbb {F}) \leq \beta \sum_ {i = 1} ^ {n} \| f _ {i} \| _ {\mathbb {X}} ^ {2} \leq \beta n.
$$
Now suppose that  $d > 1$ . Lemma E.1 gives that
$$
K _ {c} (\mathbb {F} - \mathbb {F}) (\theta) = \max  _ {j = 1, \dots , d} K _ {c} (\mathbb {F} _ {j}) (\theta)
$$
$$
K _ {c} (\mathbb {F} - \mathbb {F}) (\theta) \leq \sum_ {j = 1} ^ {d} K _ {c} (\mathbb {F} _ {j}) (\theta).
$$
Using the result for  $d = 1$ , we get
$$
\sum_ {c = 1} ^ {C} \kappa_ {c} (\mathbb {F} - \mathbb {F}) \leq \beta n d,
$$
Remark E.4 (Lower bound on  $\sum_{c=1}^{C} \kappa_c(\mathbb{F} - \mathbb{F})$ ) Suppose that  $\dim(\mathbb{Y}_c) \leq p \in \mathbb{N}, \forall c$ . Then Lemma E.1 gives, for a subspace  $\mathbb{G}$  with an orthonormal basis  $\{g_i\}_{i=1}^n$ ,
$$
K _ {c} (\mathbb {G}) (\theta) \geq \frac {1}{p} \sum_ {i = 1} ^ {n} \| L _ {c} (\theta) (g _ {i}) \| _ {\mathbb {Y} _ {c}} ^ {2}.
$$
Using this and following the steps of the above proof, we deduce the lower bound
$$
\sum_ {c = 1} ^ {C} \kappa_ {c} (\mathbb {F} - \mathbb {F}) \geq \alpha n / p.
$$
Finally, we prove Theorem 4.8. For this, we first require the following lemma.
Lemma E.5. Suppose that a realization  $\theta = (\theta_{ic})\in D$  of the  $\Theta_{ic}$  gives that
$$
\alpha^ {\prime} \| f \| _ {\mathbb {X}} ^ {2} \leq \| M (\theta) (f) \| _ {\mathbb {Y}} ^ {2}, \quad \forall f \in \mathbb {F} - \mathbb {F}, \tag {E.7}
$$
for some  $\alpha' > 0$ . Then any  $\gamma$ -minimizer  $\hat{f}$  of (2.2) satisfies
$$
\| \hat {f} - f ^ {*} \| _ {\mathbb {X}} \leq \inf  _ {f \in \mathbb {F}} \left\{\| f ^ {*} - f \| _ {\mathbb {X}} + \frac {2}{\sqrt {\alpha^ {\prime}}} \| M (\theta) (f ^ {*} - f) \| _ {\overline {{\mathbb {Y}}}} \right\} + \frac {2}{\sqrt {\alpha^ {\prime}}} \| \bar {e} \| _ {\overline {{\mathbb {Y}}}} + \frac {\gamma}{\sqrt {\alpha^ {\prime}}},
$$
where  $\bar{e} = \left(\frac{1}{\sqrt{\nu_c(\theta_{ic})m_c}} e_{iC}\right)$ .
Proof. Let  $f \in \mathbb{F}$ . We write
$$
\begin{array}{l} \| \hat {f} - f ^ {*} \| _ {\mathbb {X}} \leq \| \hat {f} - f \| _ {\mathbb {X}} + \| f ^ {*} - f \| _ {\mathbb {X}} \\ \leq 1 / \sqrt {\alpha^ {\prime}} \| M (\theta) (\hat {f} - f) \| _ {\overline {{\mathbb {Y}}}} + \| f ^ {*} - f \| _ {\mathbb {X}} \\ \leq 1 / \sqrt {\alpha^ {\prime}} \| M (\theta) (\hat {f}) - \bar {y} \| _ {\overline {{\mathbb {Y}}}} + 1 / \sqrt {\alpha^ {\prime}} \| \bar {y} - M (\theta) (f) \| _ {\overline {{\mathbb {Y}}}} + \| f ^ {*} - f \| _ {\mathbb {X}}. \\ \end{array}
$$
We now use the fact that  $\hat{f}$  is a  $\gamma$ -minimizer and the fact that  $\bar{y} = M(\theta)(f^{*}) + \bar{e}$  to get
$$
\begin{array}{l} \left\| \hat {f} - f ^ {*} \right\| _ {\mathbb {X}} \leq 2 / \sqrt {\alpha^ {\prime}} \| M (\theta) (f) - \bar {y} \| _ {\overline {{\mathbb {Y}}}} + \| f ^ {*} - f \| _ {\mathbb {X}} + \gamma / \sqrt {\alpha^ {\prime}} \\ \leq 2 / \sqrt {\alpha^ {\prime}} \| M (\theta) (f ^ {*} - f) \| _ {\overline {{\mathbb {Y}}}} + 2 / \sqrt {\alpha^ {\prime}} \| \bar {e} \| _ {\overline {{\mathbb {Y}}}} + \| f ^ {*} - f \| _ {\mathbb {X}} + \gamma / \sqrt {\alpha^ {\prime}}, \\ \end{array}
$$
Proof of Theorem 4.8. Let  $E$  be the event
$$
(1 - \epsilon) \alpha \| f \| _ {\mathbb {X}} ^ {2} \leq \| M (\overline {{\Theta}}) (f) \| _ {\mathbb {Y}} ^ {2} \leq (1 + \epsilon) \beta \| f \| _ {\mathbb {X}} ^ {2}, \quad \forall f \in \mathbb {F} - \mathbb {F}.
$$
Due Theorem 4.2 and the given assumptions, we have that  $\mathbb{P}(E^c) \leq \delta$ . Now recall that  $\tau$  is the probability distribution of the full draw  $\overline{\Theta}$ . Then
$$
\mathbb {E} \| f ^ {*} - \check {f} \| _ {\mathbb {X}} ^ {2} = \int_ {E} \| f ^ {*} - \check {f} \| _ {\mathbb {X}} ^ {2} d \tau + \int_ {E ^ {c}} \| f ^ {*} - \check {f} \| _ {\mathbb {X}} ^ {2} d \tau .
$$
In the first integral, we use the fact that  $\mathcal{T}_{\sigma}$  is a contraction to get
$$
\| f ^ {*} - \check {f} \| _ {\mathbb {X}} = \| \mathcal {T} _ {\sigma} (f) - \mathcal {T} _ {\sigma} (\hat {f}) \| _ {\mathbb {X}} \leq \| f ^ {*} - \hat {f} \| _ {\mathbb {X}}.
$$
In the second integral, we use the fact that  $\| \mathcal{T}_{\sigma}(g)\|_{\mathbb{X}}\leq \sigma$  to get
$$
\left\| f ^ {*} - \tilde {f} \right\| _ {\mathbb {X}} \leq 2 \sigma .
$$
$$
\mathbb {E} \| f ^ {*} - \tilde {f} \| _ {\mathbb {X}} ^ {2} = \int_ {E} \| f ^ {*} - \hat {f} \| _ {\mathbb {X}} ^ {2} \mathrm {d} \tau + 4 \sigma^ {2} \delta . \tag {E.8}
$$
Now, for the first term, we fix  $f \in \mathbb{F}$  and apply Lemma E.5. Since  $\alpha' \geq (1 - \epsilon)\alpha$  whenever  $E$  holds, we get
$$
\begin{array}{l} \int_ {E} \| f ^ {*} - \hat {f} \| _ {\mathbb {X}} ^ {2} \mathrm {d} \tau \\ \leq 4 \int_ {E} \left(\| f ^ {*} - f \| _ {\mathbb {X}} ^ {2} + \frac {4}{(1 - \epsilon) \alpha} \| M (\overline {{\Theta}}) (f ^ {*} - f) \| _ {\mathbb {Y}} ^ {2} + \frac {4}{(1 - \epsilon) \alpha} \| \bar {e} \| _ {\mathbb {Y}} ^ {2} + \frac {\gamma^ {2}}{(1 - \epsilon) \alpha}\right) d \tau . \\ \end{array}
$$
For the second term, we use Assumption 2.4 and (E.3) to get
$$
\begin{array}{l} \int_ {E} \| f ^ {*} - \hat {f} \| _ {\mathbb {X}} ^ {2} \mathrm {d} \tau \leq 4 \left(1 + \frac {4 \beta}{(1 - \epsilon) \alpha}\right) \| f ^ {*} - f \| _ {\mathbb {X}} ^ {2} \\ + \frac {1 6}{(1 - \epsilon) \alpha} \sum_ {c = 1} ^ {C} \frac {1}{m _ {c}} \sum_ {i = 1} ^ {m _ {c}} \| e _ {i c} \| _ {\mathbb {Y} _ {c}} ^ {2} \int_ {D _ {c}} \frac {1}{\nu_ {c} (\theta_ {c})} \mathrm {d} \mu_ {c} (\theta_ {c}) + \frac {4 \gamma^ {2}}{(1 - \epsilon) \alpha} \\ \end{array}
$$
$$
\begin{array}{l} \int_ {E} \| f ^ {*} - \hat {f} \| _ {\mathbb {X}} ^ {2} \mathrm {d} \tau \leq 4 \left(1 + \frac {4 \beta}{(1 - \epsilon) \alpha}\right) \| f ^ {*} - f \| _ {\mathbb {X}} ^ {2} \\ + \frac {1 6}{(1 - \epsilon) \alpha} \sum_ {c = 1} ^ {C} \frac {1}{m _ {c}} \sum_ {i = 1} ^ {m _ {c}} \| e _ {i c} \| _ {\mathbb {Y} _ {c}} ^ {2} \rho_ {c} (D _ {c}) + \frac {4 \gamma^ {2}}{(1 - \epsilon) \alpha}. \\ \end{array}
$$
Combining this with (E.8) and recalling that  $f$  was arbitrary now gives the result.
Remark E.6 As noted, the noise bound in Theorem 4.8 assumes  $\rho_c(D_c) < \infty$ . If this fails, one may also bound the noise term by assuming the noise takes the form
$$
e _ {i c} = L _ {c} \left(\Theta_ {i c}\right) (g), \quad i = 1, \dots , m _ {c}, c = 1, \dots , C,
$$
for some  $g\in \mathbb{X}$  with  $\| g\|_{\mathbb{X}}\ll 1$  . In this case, one has
$$
\mathbb {E} \| \bar {e} \| _ {\mathbb {Y}} ^ {2} = \sum_ {c = 1} ^ {C} \frac {1}{m _ {c}} \sum_ {i = 1} ^ {m _ {c}} \mathbb {E} \left(\frac {1}{\nu_ {c} (\Theta_ {i c})} \| L _ {c} (\Theta_ {i c}) (g) \| _ {\mathbb {Y} _ {c}} ^ {2}\right) = \sum_ {c = 1} ^ {C} \int_ {D _ {c}} \| L (\theta) (g) \| _ {\mathbb {Y} _ {c}} ^ {2} \mathrm {d} \rho_ {c} (\theta) \leq \beta \| g \| _ {\mathbb {X}} ^ {2}.
$$
Hence Theorem 4.8 holds with  $N = \beta \| g\|_{\mathbb{X}}^2$ . For other ways to estimate the noise term under different noise models, see, e.g., [93].
Proof of Theorem 2.7. Part (i) follows from Theorem 4.2, Corollary 4.3 and Lemma 4.6 with the (arbitrary) value  $\epsilon = 1/2$ . Part (ii) follows from part (i) and Theorem 4.8. Finally, part (iii) follows from Corollary 4.7.