SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\ell (\mu , y, \lambda t _ {1} + (1 - \lambda) t _ {2}) \leq \lambda \ell (\mu , y, t _ {1}) + (1 - \lambda) \ell (\mu , y, t _ {2}),
$$
i.e., convexity of  $\ell$
Proof. of Proposition 4.3 From Lemma 4.2 we get that  $\ell$  is nonnegative and convex. The existence, uniqueness and the representation formulas follow then from the standard representer theorem, cf. e.g., [20, Theorem 5.5].
Furthermore, for all  $M\in \mathbb{N}_{+}$  we have
$$
\begin{array}{l} \lambda \| f _ {M, \lambda} ^ {*} \| _ {M} ^ {2} \leq \frac {1}{N} \sum_ {n = 1} ^ {N} \ell_ {M} (\vec {x} _ {n} ^ {[ M ]}, y _ {n} ^ {[ M ]}, f _ {M, \lambda} ^ {*} (\vec {x} _ {n} ^ {[ M ]})) + \lambda \| f _ {M, \lambda} ^ {*} \| _ {M} ^ {2} \\ \leq \mathcal {R} _ {\ell_ {M}, D _ {N} ^ {[ M ]}, \lambda} (0) \\ \leq N C _ {\ell}, \\ \end{array}
$$
hence  $\| f_{M,\lambda}^{*}\|_{M}\leq \sqrt{\frac{NC_{\ell}}{\lambda}}$
Let  $f \in H_{k}$  and  $(f_{M})_{M}$ ,  $f_{M} \in H_{M}$ , such that  $f_{M} \xrightarrow{\mathcal{P}_{1}} f$ . From  $D_{N}^{[M]} \xrightarrow{\mathcal{P}_{1}} D_{N}$  and the continuity of  $\ell_{M}$ , together with  $\ell_{M} \xrightarrow{\mathcal{P}_{1}} \ell$  and the boundedness of  $\{y_{n}^{[M]} \mid M \in \mathbb{N}_{+}, n = 1, \ldots, N\} \subseteq Y$  and  $\{f_{M}(\vec{x}_{n}^{[M]}) \mid M \in \mathbb{N}_{+}, N = 1, \ldots, N\}$  we find that
$$
\lim _ {M} \frac {1}{N} \sum_ {n = 1} ^ {N} \ell_ {M} (\vec {x} _ {n} ^ {[ M ]}, y _ {n} ^ {[ M ]}, f _ {M} (\vec {x} _ {n} ^ {[ M ]})) = \frac {1}{N} \sum_ {n = 1} ^ {N} \ell (\mu_ {n}, y _ {n}, f (\mu_ {n})).
$$
Combining this with Lemma 2.4 and Lemma 2.5 then establishes that  $\mathcal{R}_{\ell_M, D_N^{[M]}, \lambda} \xrightarrow{\Gamma} \mathcal{R}_{\ell, D_N, \lambda}$  and the remaining claims follow from Proposition B.3 and the uniqueness of the minimizers.
Proof. of Lemma 4.4 Let  $\epsilon >0$  be arbitrary. Recall from the proof of Proposition 4.3 that for all  $M\in \mathbb{N}_{+}$  we have  $\| f_{M,\lambda}^{*}\|_{M}\leq \sqrt{\frac{NC_{\ell}}{\lambda}}$ , and hence for all  $\vec{x}\in X^M$  we have
$$
\begin{array}{l} \left| f _ {M, \lambda} ^ {*} (\vec {x}) \right| \leq \left\| f _ {M, \lambda} ^ {*} \right\| _ {k} \left\| k _ {M} (\cdot , \vec {x}) \right\| _ {k} \\ \leq \sqrt {\frac {N C _ {\ell}}{\lambda}} \sqrt {C _ {k}}. \\ \end{array}
$$
A similar argument applies to  $f_{\lambda}^{*} \in H_{k}$ , so we can find a compact set  $K \subseteq \mathbb{R}$  with
$$
\left\{f _ {M, \lambda} ^ {*} \left(\vec {x} _ {n} ^ {[ M ]}\right) \mid M \in \mathbb {N} _ {+}, n = 1, \dots , N \right\} \cup \left\{f _ {\lambda} ^ {*} \left(\mu_ {n}\right) \mid n = 1, \dots , N \right\} \subseteq K.
$$
Choose now  $m_{\epsilon} \in \mathbb{N}_{+}$  such that for all  $m \geq m_{\epsilon}$  we have
$$
\sup_{\substack{\vec{x}\in X^{M_{m}}\\ y\in Y}}|\ell_{M_{m}}(\vec{x},y,f^{*}_{M_{m},\lambda}(\vec{x})) - \ell_{M_{m}}(\vec{x},y,f^{*}_{\lambda}(\hat{\mu} [\vec{x}]))| \leq \frac{\epsilon}{3}
$$
$$
\sup_{\substack{\vec{x}\in X^{M_{m}}\\ y\in Y,t\in K}}|\ell_{M_{m}}(\vec{x},y,t) - \ell (\hat{\mu} [\vec{x} ],y,t)|\leq \frac{\epsilon}{3}
$$
$$
\left| \int_ {X ^ {M _ {m}} \times Y} \ell (\hat {\mu} [ \vec {x} ], y, f _ {\lambda} ^ {*} (\hat {\mu} [ \vec {x} ])) \mathrm {d} P ^ {[ M _ {m} ]} (\vec {x}, y) - \int_ {\mathcal {P} (X) \times Y} \ell (\mu , y, f _ {\lambda} ^ {*} (\mu)) \mathrm {d} (\mu , y) \right| \leq \frac {\epsilon}{3}.
$$
Such a  $m_{\epsilon}$  exists since  $f_{M_m,\lambda}^* \xrightarrow{\mathcal{P}_1} f_\lambda^*$  and all  $\ell_{M_m}$  are uniformly Lipschitz continuous (first inequality),  $\ell_{M_m} \xrightarrow{\mathcal{P}_1} \ell$  and  $Y$  and  $K$  are compact (second inequality), and  $P^{[M]} \xrightarrow{\mathcal{P}_1} P$  as well as that  $(\mu, y) \mapsto \ell(\mu, y, f_\lambda^*(\mu))$  is continuous and bounded (third inequality). We now have
$$
\begin{array}{l} \left| \mathcal {R} _ {\ell_ {M _ {m}}, P ^ {[ M _ {m} ]}} \left(f _ {M _ {m}, \lambda} ^ {*}\right) - \mathcal {R} _ {\ell , P} \left(f _ {\lambda} ^ {*}\right) \right| \\ \leq \left| \int_ {X ^ {M _ {m} \times Y}} \ell_ {M _ {m}} (\vec {x}, y, f _ {M _ {m}, \lambda} ^ {*} (\vec {x})) - \ell_ {M _ {m}} (\vec {x}, y, f _ {\lambda} ^ {*} (\hat {\mu} [ \vec {x} ])) \mathrm {d} P ^ {[ M _ {m} ]} (\vec {x}, y) \right| \\ + \left| \int_ {X ^ {M _ {m} \times Y}} \ell_ {M _ {m}} (\vec {x}, y, f _ {\lambda} ^ {*} (\hat {\mu} [ \vec {x} ])) - \ell (\hat {\mu} [ \vec {x} ], y, f _ {\lambda} ^ {*} (\hat {\mu} [ \vec {x} ])) \mathrm {d} P ^ {[ M _ {m} ]} (\vec {x}, y) \right| \\ + \left| \int_ {X ^ {M _ {m}} \times Y} \ell (\hat {\mu} [ \vec {x} ], y, f _ {\lambda} ^ {*} (\hat {\mu} [ \vec {x} ])) \mathrm {d} P ^ {[ M _ {m} ]} (\vec {x}, y) - \int_ {\mathcal {P} (X) \times Y} \ell (\mu , y, f _ {\lambda} ^ {*} (\mu)) \mathrm {d} (\mu , y) \right| \\ \leq \int_ {X ^ {M _ {m} \times Y}} | \ell_ {M _ {m}} (\vec {x}, y, f _ {M _ {m}, \lambda} ^ {*} (\vec {x})) - \ell_ {M _ {m}} (\vec {x}, y, f _ {\lambda} ^ {*} (\hat {\mu} [ \vec {x} ])) | d P ^ {[ M _ {m} ]} (\vec {x}, y) \\ + \int_ {X ^ {M _ {m} \times Y}} | \ell_ {M _ {m}} (\vec {x}, y, f _ {\lambda} ^ {*} (\hat {\mu} [ \vec {x} ])) - \ell (\hat {\mu} [ \vec {x} ], y, f _ {\lambda} ^ {*} (\hat {\mu} [ \vec {x} ])) | d P ^ {[ M _ {m} ]} (\vec {x}, y) \\ + \frac {\epsilon}{3} \\ \leq \epsilon , \\ \end{array}
$$
Proof. of Proposition 4.5 Observe that all  $k_{M}$  are bounded measurable kernels,  $\mathcal{R}_{\ell_M,P^{[M]}}(f_M) < \infty$  for all  $f\in H_M$ ,  $\ell_{M}$  is a convex,  $P^{[M]}$ -integrable Nemitskii loss (cf. Remark 4.1) and hence [20, Lemma 5.1, Theorem 5.2] guarantee the existence and uniqueness of  $f_{M,\lambda}^{*}$ . A completely analogous argument shows the existence and uniqueness of  $f_{\lambda}^{*}$ .
We now show that  $\mathcal{R}_{\ell_M,P^{[M]},\lambda}\xrightarrow{\Gamma}\mathcal{R}_{\ell,P,\lambda}$ . For the  $\Gamma$ -lim inf-inequality, let  $f_{M}\in H_{M}$ ,  $f\in H_{k}$  be arbitrary with  $f_{M}\xrightarrow{\mathcal{P}_{1}}f$ , and let  $\epsilon >0$ . Choose  $M_{\epsilon}\in \mathbb{N}_{+}$  so large that for all  $M\geq M_{\epsilon}$
$$
\left| \int \ell (\hat {\mu} [ \vec {x} ], y, f (\hat {\mu} [ \vec {x} ]) \mathrm {d} P ^ {[ M ]} (\vec {x}, y)) - \int \ell (\mu , y, f (\mu)) \mathrm {d} P (\mu , y) \right| \leq \frac {\epsilon}{2}
$$
(this is possible since  $(\mu, y) \mapsto \ell(\mu, y, f(\mu))$  is bounded and continuous and  $P^{[M]} \xrightarrow{\mathcal{P}_1} P$ ) and
$$
\left| \ell_ {M} (\vec {x}, y, f _ {M} (\vec {x})) - \ell (\hat {\mu} [ \vec {x} ], y, f (\hat {\mu} [ \vec {x} ])) \right| \leq \frac {\epsilon}{2}
$$
for all  $\vec{x} \in X^M$ ,  $y \in Y$  (this is possible due to the same argument used in the proof of Lemma 4.4). For  $M \geq M_{\epsilon}$  we then find
$$
\begin{array}{l} \mathcal {R} _ {\ell , P, \lambda} (f) = \int \ell (\mu , y, f (\mu)) \mathrm {d} P (\mu , y) + \lambda \| f \| _ {k} ^ {2} \\ \leq \int \ell_ {M} (\vec {x}, y, f _ {M} (\vec {x})) \mathrm {d} P ^ {[ M ]} (\vec {x}, y) \\ + \left| \int \ell (\hat {\mu} [ \vec {x} ], y, f (\hat {\mu} [ \vec {x} ]) \mathrm {d} P ^ {[ M ]} (\vec {x}, y)) - \int \ell (\mu , y, f (\mu)) \mathrm {d} P (\mu , y) \right| \\ + \left| \int \ell_ {M} (\vec {x}, y, f _ {M} (\vec {x})) - \ell (\hat {\mu} [ \vec {x} ], y, f (\hat {\mu} [ \vec {x} ])) \mathrm {d} P ^ {[ M ]} (\vec {x}, y) \right| + \lambda \| f \| _ {k} ^ {2} \\ \leq \int \ell_ {M} (\vec {x}, y, f _ {M} (\vec {x})) \mathrm {d} P ^ {[ M ]} (\vec {x}, y) + \lambda \lim _ {M} \inf \| f _ {M} \| _ {M} ^ {2} + \epsilon , \\ \end{array}
$$
where we used Lemma 2.4 in the last inequality.
For the  $\Gamma$ -lim sup-inequality, let  $f \in H_k$  be arbitrary and let  $(f_M)_M$  be the recovery sequence from Lemma 2.5. The desired inequality then follows by repeating the arguments from above.
Finally, using exactly the same argument as in the proof of Proposition 4.3 shows that  $\| f_{M,\lambda}^{*}\|_{M}\leq \sqrt{\frac{NC_{\ell}}{\lambda}}$ , so we can apply Proposition B.3 and the result follows.
Proof. of Proposition 4.7 Let  $(\epsilon_{n})_{n} \subseteq \mathbb{R}_{>0}$  with  $\epsilon_{m} \searrow 0$ . We construct a strictly increasing sequence  $(M_{n})_{n}$  such that
$$
\left| \mathcal {R} _ {\ell_ {M _ {n}}, P ^ {[ M _ {n} ]}} ^ {H _ {M _ {n}} *} - \mathcal {R} _ {\ell , P} ^ {H _ {k} *} \right| \leq \epsilon_ {n}
$$
for all  $n\in \mathbb{N}_{+}$
We start with  $n = 1$ : Since  $A_{2}(0) = 0$  and  $A_{2}$  is continuous in  $0$ , cf. [20, Lemma 5.15], there exists  $\lambda_1' \in \mathbb{R}_{>0}$  such that  $A_{2}(\lambda) \leq \frac{\epsilon_{1}}{3}$  for all  $0 < \lambda \leq \lambda_1'$ . From Assumption 4.6 we get  $\lambda_1'' \in \mathbb{R}_{>0}$  such that for all  $M \in \mathbb{N}_+$  we have  $A_{2}^{[M]}(\lambda) \leq \frac{\epsilon_{1}}{3}$  for all  $0 < \lambda \leq \lambda_1''$ . Define now  $\lambda_{1} = \min\{\lambda_{1}', \lambda_{1}''\}$ , and observe that  $\lambda_{1} > 0$ . Proposition 4.5 ensures the existence of a strictly increasing sequence  $(M_m^{(1)})_m \subseteq \mathbb{N}_+$  with
$$
\mathcal{R}^{H_{M_{m}^{(1)}}{}^{*}}_{\ell_{M_{m}^{(1)}},P^{[M_{m}^{(1)}]},\lambda_{1}}\to \mathcal{R}^{H_{k}{}^{*}}_{\ell ,P,\lambda_{1}}
$$
for  $m\to \infty$  . Choose  $m_{1}\in \mathbb{N}_{+}$  such that for all  $m\geq m_1$  we have
$$
\left|\mathcal{R}^{H_{M_{m}^{(1)}}*}_{\ell_{M_{m}^{(1)}},P[M_{m}^{(1)}] , \lambda_{1}} - \mathcal{R}^{H_{k}*}_{\ell ,P,\lambda_{1}}\right|\leq \frac{\epsilon_{1}}{3}.
$$
We now set  $M_1 = M_{m_1}^{(1)}$  and get that
$$
\begin{array}{l} \left|\mathcal{R}^{H_{M_{1}}*}_{\ell_{M_{1}},P^{[M_{1}]}} - \mathcal{R}^{H_{k}*}_{\ell ,P}\right|\leq \left|\mathcal{R}^{H_{M_{m_{1}}^{(1)}}*}_{\ell_{M_{m_{1}}^{(1)}},P^{[M_{m_{1}}^{(1)]}}} - \mathcal{R}^{H_{M_{m_{1}}^{(1)}}*}_{\ell_{M_{m_{1}}^{(1)}},P^{[M_{m_{1}}^{(1)}]},\lambda_{1}}\right| + \left|\mathcal{R}^{H_{M_{m_{1}}^{(1)}}*}_{\ell_{M_{m_{1}}^{(1)}},P^{[M_{m_{1}}^{(1)}]},\lambda_{1}} - \mathcal{R}^{H_{k}*}_{\ell ,P,\lambda_{1}}\right| \\ + \left| \mathcal {R} _ {\ell , P, \lambda_ {1}} ^ {H _ {k} *} - \mathcal {R} _ {\ell , P} ^ {H _ {k} *} \right| \\ \leq A _ {2} ^ {[ M _ {m} ^ {(1)} ]} (\lambda_ {1}) + \frac {\epsilon_ {1}}{3} + A _ {2} (\lambda_ {1}) \\ \leq \epsilon_ {1}. \\ \end{array}
$$
We can now repeat the argument from above inductively: Suppose we have constructed our subsequence up to  $n \in \mathbb{N}_+$ , i.e.,  $M_1, \ldots, M_n$ . Choose  $\lambda' \in \mathbb{R}_{>0}$  such that  $A_2(\lambda) \leq \frac{c_{n+1}}{3}$  for
all  $0 < \lambda \leq \lambda'$  (exists due to continuity), and  $\lambda'' \in \mathbb{R}_{>0}$  such that for all  $M \in \mathbb{N}_+$  we have  $A_2^{[M]}(\lambda) \leq \frac{\epsilon_{n+1}}{3}$  for all  $0 < \lambda \leq \lambda''$  (using Assumption 4.6). Define now  $\lambda_{n+1} = \min\{\lambda', \lambda''\}$ , and observe that  $\lambda_{n+1} > 0$ . Proposition 4.5 ensures the existence of a strictly increasing sequence  $\left(M_m^{(n+1)}\right)_m$  such that
$$
\mathcal{R}^{H_{M_{m}^{(n + 1)}}*}_{\ell_{M_{m}^{(n + 1)}},P^{[M_{m}^{(n + 1)}]},\lambda_{n + 1}}\to \mathcal{R}^{H_{k}*}_{\ell ,P,\lambda_{n + 1}}
$$
for  $m\to \infty$  . Choose  $m_{n + 1}$  such that for all  $m\geq m_{n + 1}$  we have
$$
\left|\mathcal{R}^{H_{M_{m}^{(n + 1)}}*}_{\ell_{M_{m}^{(n + 1)}},P^{[M_{m}^{(n + 1)}]},\lambda_{n + 1}} - \mathcal{R}^{H_{k}*}_{\ell ,P,\lambda_{n + 1}}\right|\\ \leq \frac{\epsilon_{n + 1}}{3}.
$$
Define now  $M_{n + 1} = \max \{M_n + 1, M_{m_{n + 1}}^{(n + 1)}\}$ , then we get
$$
\begin{array}{l} \left|\mathcal{R}^{H_{M_{n + 1}}*}_{\ell_{M_{n + 1}},P^{[M_{n + 1}]}} - \mathcal{R}^{H_{k}*}_{\ell ,P}\right|\leq \left|\mathcal{R}^{H_{M_{m_{n + 1}}^{(n + 1)}}*}_{\ell_{M_{m_{n + 1}}^{(n + 1)}},P^{[M_{m_{n + 1}}^{(n + 1)}]}} - \mathcal{R}^{H_{M_{m_{n + 1}}^{(n + 1)}}*}_{\ell_{M_{m_{n + 1}}^{(n + 1)}},P^{[M_{m_{n + 1}}^{(n + 1)}]},\lambda_{n + 1}}\right| \\ + \left| \begin{array}{c} \mathcal {R} ^ {H _ {M _ {m _ {n + 1}} ^ {(n + 1)}} *} \\ \ell_ {M _ {m _ {n + 1}} ^ {(n + 1)}}, P ^ {[ M _ {m _ {n + 1}} ^ {(n + 1)} ], \lambda_ {n + 1}} \end{array} - \mathcal {R} _ {\ell , P, \lambda_ {n + 1}} ^ {H _ {k} *} \right| \\ + \left| \mathcal {R} _ {\ell , P, \lambda_ {n + 1}} ^ {H _ {k} *} - \mathcal {R} _ {\ell , P} ^ {H _ {k} *} \right| \\ \leq A _ {2} ^ {M _ {m _ {n + 1}} ^ {(n + 1)}} \left(\lambda_ {n + 1}\right) + \frac {\epsilon_ {n + 1}}{3} + A _ {2} \left(\lambda_ {n + 1}\right) \\ \leq \epsilon_ {n + 1}. \\ \end{array}
$$
The resulting sequence  $(M_n)_n$  fulfills then
$$
\mathcal {R} _ {\ell_ {M _ {n}}, P ^ {[ M _ {n} ]}} ^ {H _ {M _ {n}} *} \to \mathcal {R} _ {\ell , P} ^ {H _ {k} *}
$$
for  $n\to \infty$
B Additional technical results
In this section we state and prove two technical results that play an important role in the proofs of the main results.
B.1 A characterization of RKHS functions
Here we recall the following characterization of RKHS functions from [31, Section I.4]. Let  $\mathcal{X} \neq \emptyset$  be arbitrary. For  $k: \mathcal{X} \times \mathcal{X} \to \mathbb{R}$  symmetric and positive semidefinite and some  $f \in \mathbb{R}^{\mathcal{X}}$  as well as  $N \in \mathbb{N}_{+}$ ,  $\vec{x} \in \mathcal{X}^{N}$ ,  $\vec{\alpha} \in \mathbb{R}^{N}$  define
$$
\mathcal {E} (\vec {x}, \vec {\alpha}, f) = \sum_ {n = 1} ^ {N} \alpha_ {n} f (x _ {n})
$$
$$
\mathcal {W} (\vec {x}, \vec {\alpha}, k) = \sqrt {\sum_ {i , j = 1} ^ {N} \alpha_ {i} \alpha_ {j} k (x _ {j} , x _ {i})},
$$
where we might omit some arguments if they are clear. Furthermore, define
$$
\mathcal {D} (\vec {x}, \vec {\alpha}, f, k) = \left\{ \begin{array}{l l} \frac {\mathcal {E} (\vec {x} , \vec {\alpha} , f)}{\mathcal {W} (\vec {x} , \vec {\alpha} , k)} & \text {i f} \mathcal {E} (\vec {x}, \vec {\alpha}, f) \neq 0, \mathcal {W} (\vec {x}, \vec {\alpha}, k) \neq 0 \\ 0 & \text {i f} \mathcal {E} (\vec {x}, \vec {\alpha}, f) = \mathcal {W} (\vec {x}, \vec {\alpha}, k) = 0 \\ \infty & \text {i f} \mathcal {E} (\vec {x}, \vec {\alpha}, f) \neq 0, \mathcal {W} (\vec {x}, \vec {\alpha}, k) = 0 \end{array} \right.
$$
$$
\mathcal{N}(f,k) = \sup_{\substack{(\vec{x},\vec{\alpha})\in \mathcal{X}^{N}\times \mathbb{R}^{N}\\ N\in \mathbb{N}_{+}}}\mathcal{D}(\vec{x},\vec{\alpha},f,k).
$$
We collect now some simple facts that will be used repeatedly.
Let  $\vec{x} \in \mathcal{X}^N$ ,  $\vec{\alpha} \in \mathbb{R}^N$ ,  $N \in \mathbb{N}_+$ , be arbitrary, and define
$$
f = \sum_ {n = 1} ^ {N} \alpha_ {n} k (\cdot , x _ {n}) \in H _ {k} ^ {\text {p r e}}.
$$
1. By construction,  $\mathcal{W}(\vec{x},\vec{\alpha},k)\in \mathbb{R}_{\geq 0}$  (recall that  $k$  is positive semidefinite).
2. Since  $f \in H_k^{\mathrm{pre}}$ , its RKHS norm has an explicit form and we find
$$
\| f \| _ {k} = \sqrt {\sum_ {i , j = 1} ^ {N} \alpha_ {i} \alpha_ {j} k \left(x _ {j} , x _ {i}\right)} = \mathcal {W} (\vec {x}, \vec {\alpha}, k).
$$
This also implies that  $f \equiv 0$  if and only if  $\mathcal{W}(\vec{x},\vec{\alpha},k) = 0$ .
3. If  $\mathcal{W}(\vec{x},\vec{\alpha},k) > 0$  , then
$$
\begin{array}{l} \mathcal {D} (\vec {x}, \vec {\alpha}, f, k) = \frac {\mathcal {E} (\vec {x} , \vec {\alpha} , f)}{\mathcal {W} (\vec {x} , \vec {\alpha} , k)} \\ = \frac {\sum_ {i = 1} ^ {N} \alpha_ {i} f (x _ {i})}{\sqrt {\sum_ {i , j = 1} ^ {N} \alpha_ {i} \alpha_ {j} k (x _ {j} , x _ {i})}} \\ = \frac {\sum_ {i , j = 1} ^ {N} \alpha_ {i} \alpha_ {j} k (x _ {j} , x _ {i})}{\sqrt {\sum_ {i , j = 1} ^ {N} \alpha_ {i} \alpha_ {j} k (x _ {j} , x _ {i})}} \\ = \frac {\mathcal {W} (\vec {x} , \vec {\alpha} , k) ^ {2}}{\mathcal {W} (\vec {x} , \vec {\alpha} , k)} = \mathcal {W} (\vec {x}, \vec {\alpha}, k). \\ \end{array}
$$
We can now state the characterization result.
Theorem B.1. Let  $k: \mathcal{X} \times \mathcal{X} \to \mathbb{R}$  be a kernel and  $f \in \mathbb{R}^{\mathcal{X}}$ . Then  $f \in H_k$  if and only if  $\mathcal{N}(f, k) < \infty$ . If  $f \in H_k$ , then  $\|f\|_k = \mathcal{N}(f, k)$ .
For convenience, we provide a full self-contained proof of this result.
Proof. Step 1 First, we show that for  $f \in H_k$ , we have  $\| f \|_k = \mathcal{N}(f, k)$ .
$\mathcal{N}(f,k)\leq \| f\| _k$  : Let  $N\in \mathbb{N}_{+}$  and  $(\vec{x},\vec{\alpha})\in \mathcal{X}^N\times \mathbb{R}^N$  be arbitrary. Observe that
$$
\begin{array}{l} \mathcal {E} (\vec {x}, \vec {\alpha}, f) = \sum_ {n = 1} ^ {N} \alpha_ {n} f (x _ {n}) \\ = \sum_ {n = 1} ^ {N} \alpha_ {n} \langle f, k (\cdot , x _ {n}) \rangle_ {k} \\ = \langle f, \sum_ {n = 1} ^ {N} \alpha_ {n} k (\cdot , x _ {n}) \rangle_ {k} \\ \leq \| f \| _ {k} \left\| \sum_ {n = 1} ^ {N} \alpha_ {n} k (\cdot , x _ {n}) \right\| _ {k} \\ = \| f \| _ {k} \mathcal {W} (\vec {x}, \vec {\alpha}, k). \\ \end{array}
$$
If  $\mathcal{W}(\vec{x},\vec{\alpha},k) = \|\sum_{n=1}^{N}\alpha_{n}k(\cdot,x_{n})\|_{k} = 0$ , then  $\sum_{n=1}^{N}\alpha_{n}k(\cdot,x_{n}) = 0_{H_{k}}$ , hence  $\mathcal{E}(\vec{x},\vec{\alpha},f) = \langle f,0_{H_k}\rangle_k = 0$  and by definition  $\mathcal{D}(\vec{x},\vec{\alpha},f,k) = 0 \leq \|f\|_k$ .
If  $\mathcal{W}(\vec{x},\vec{\alpha},k) > 0$  we can rearrange to get
$$
\frac {\mathcal {E} (\vec {x} , \vec {\alpha} , f)}{\mathcal {W} (\vec {x} , \vec {\alpha} , k)} = \mathcal {D} (\vec {x}, \vec {\alpha}, f, k) \leq \| f \| _ {k}.
$$
Since  $(\vec{x},\vec{\alpha})$  was arbitrary, we find that  $\mathcal{N}(\vec{x},\vec{\alpha},f,k)\leq \| f\| _k$
$\mathcal{N}(f,k)\geq \| f\| _k$  : Let  $\epsilon >0$  and choose  $f_{\epsilon} = \sum_{n = 1}^{N}\alpha_{n}k(\cdot ,x_{n})\in H_{k}^{\mathrm{pre}}$  such that  $\| f - f_{\epsilon}\| _k <   \epsilon$  If  $\mathcal{W}(\vec{x},\vec{\alpha},k) = \| f_{\epsilon}\| _k = 0$  , then  $f_{\epsilon} = 0_{H_k}$  and hence  $\mathcal{E}(\vec{x},\vec{\alpha},f) = \langle f,f_{\epsilon}\rangle_{k} = \langle f,0_{H_{k}}\rangle_{k} = 0.$  By definition, this then shows
$$
\mathcal {D} (\vec {x}, \vec {\alpha}, f) = 0 = \| f _ {\epsilon} \| _ {k} \geq \| f \| _ {k} - \epsilon .
$$
Before we continue, note that for all  $f_{1}, f_{2} \in H_{k}$  we have
$$
\begin{array}{l} \left| \mathcal {E} (\vec {x}, \vec {\alpha}, f _ {1}) - \mathcal {E} (\vec {x}, \vec {\alpha}, f _ {2}) \right| = \left| \sum_ {n = 1} ^ {N} \alpha_ {n} \left(f _ {1} \left(x _ {n}\right) - f _ {2} \left(x _ {n}\right)\right) \right| \\ = \left| \sum_ {n = 1} ^ {N} \alpha_ {n} \langle f _ {1} - f _ {2}, k (\cdot , x _ {n}) \rangle_ {k} \right| \\ = \left| \langle f _ {1} - f _ {2}, \sum_ {n = 1} ^ {N} \alpha_ {n} k (\cdot , x _ {n}) \rangle_ {k} \right| \\ \leq \left\| f _ {1} - f _ {2} \right\| _ {k} \left\| f _ {\epsilon} \right\| _ {k}. \\ \end{array}
$$
Assume now that  $\mathcal{W}(\vec{x},\vec{\alpha},k) > 0$  , then we get
$$
\begin{array}{l} \mathcal {D} (\vec {x}, \vec {\alpha}, f, k) = \frac {\mathcal {E} (\vec {x} , \vec {\alpha} , f)}{\mathcal {W} (\vec {x} , \vec {\alpha} , k)} \\ \geq \frac {\mathcal {E} (\vec {x} , \vec {\alpha} , f _ {\epsilon})}{\mathcal {W} (\vec {x} , \vec {\alpha} , k)} - \frac {\| f - f _ {\epsilon} \| _ {k} \| f _ {\epsilon} \| _ {k}}{\mathcal {W} (\vec {x} , \vec {\alpha} , k)} \\ \geq \frac {\mathcal {E} (\vec {x} , \vec {\alpha} , f _ {\epsilon})}{\mathcal {W} (\vec {x} , \vec {\alpha} , k)} - \frac {\epsilon \| f _ {\epsilon} \| _ {k}}{\mathcal {W} (\vec {x} , \vec {\alpha} , k)} \\ = \mathcal {W} (\vec {x}, \vec {\alpha}, k) - \epsilon \\ = \| f _ {\epsilon} \| _ {k} - \epsilon \\ \geq \| f \| _ {k} - 2 \epsilon \\ \end{array}
$$
Altogether, by definition of  $\mathcal{N}(f,k)$ , we get that
$$
\mathcal {N} (f, k) \geq \mathcal {D} (\vec {x}, \vec {\alpha}, f, k) \geq \| f \| _ {k} - 2 \epsilon .
$$
Since  $\epsilon > 0$  was arbitrary, we find that  $\mathcal{N}(f,k) \geq \| f\|_k$ .
Step 2 Let  $f \in \mathbb{R}^{\mathcal{X}}$  be arbitrary. We show that if  $\mathcal{N}(f, k) < \infty$ , then
$$
\ell_ {f}: H _ {k} ^ {\mathrm {p r e}} \to \mathbb {R}
$$
$$
\sum_ {n = 1} ^ {N} \alpha_ {n} k (\cdot , x _ {n}) \mapsto \sum_ {n = 1} ^ {N} \alpha_ {n} f (x _ {n})
$$
is a well-defined, linear and continuous (w.r.t.  $\| \cdot \| _k$ ) map.
To establish the well-posedness, let  $(\vec{x},\vec{\alpha})\in \mathcal{X}^N\times \mathbb{R}^N$  and  $(\vec{y},\vec{\beta})\in \mathcal{X}^{M}\times \mathbb{R}^{M}$  such that
$$
\sum_ {n = 1} ^ {N} \alpha_ {n} k (\cdot , x _ {n}) = \sum_ {m = 1} ^ {M} \beta_ {m} k (\cdot , y _ {m}) \in H _ {k} ^ {\text {p r e}}.
$$
$$
\sum_ {n = 1} ^ {N} \alpha_ {n} k (\cdot , x _ {n}) + \sum_ {m = 1} ^ {M} (- \beta_ {m}) k (\cdot , y _ {m}) = 0 _ {H _ {k}}
$$
and hence  $\mathcal{W}((\vec{x},\vec{y}),(\vec{\alpha}, - \vec{\beta}),k) = \| \sum_{n = 1}^{N}\alpha_{n}k(\cdot ,x_{n}) + \sum_{m = 1}^{M}(-\beta_{m})k(\cdot ,y_{m})\|_{k} = 0.$  Assume now that
$$
\sum_ {n = 1} ^ {N} \alpha_ {n} f (x _ {n}) \neq \sum_ {m = 1} ^ {m} \beta_ {m} f (x _ {m}),
$$
then we get that
$$
\sum_ {n = 1} ^ {N} \alpha_ {n} f (x _ {n}) + \sum_ {m = 1} ^ {m} (- \beta_ {m}) f (x _ {m}) = \mathcal {E} ((\vec {x}, \vec {y}), (\vec {\alpha}, - \vec {\beta}), f) \neq 0
$$
which by definition implies that  $\mathcal{D}((\vec{x},\vec{y}),(\vec{\alpha}, - \vec{\beta}),f,k) = \infty$  and therefore  $\mathcal{N}(f,k) = \infty$ , a contradiction.
The linearity is then clear. Finally, to show the continuity, let  $H_{k}^{\mathrm{pre}} \ni f_{0} = \sum_{n=1}^{N} \alpha_{n} k(\cdot, x_{n})$  be arbitrary and set  $\vec{x} = (x_{1} \cdots x_{N}), \vec{\alpha} = (\alpha_{1} \cdots \alpha_{N})$ , then
$$
\begin{array}{l} | \ell_ {f} (f _ {0}) | = \left| \sum_ {n = 1} ^ {N} \alpha_ {n} f (x _ {n}) \right| \\ = | \mathcal {E} (\vec {x}, \vec {\alpha}, f) | \\ \leq \mathcal {N} (f, k) \mathcal {W} (\vec {x}, \vec {\alpha}, k) \\ = \mathcal {N} (f, k) \| f _ {0} \| _ {k}. \\ \end{array}
$$
Since  $\mathcal{N}(f,k)$  is finite and independent of  $f_0$ , and  $\ell_f$  is a linear map, this shows the continuity of  $\ell_f$ .
Step 3 Let  $f \in \mathbb{R}^{\mathcal{X}}$  such that  $\mathcal{N}(f, k) < \infty$ . Since according to Step 2  $\ell_f$  is a linear and continuous map on  $H_k^{\mathrm{pre}}$  and the latter is dense in  $H_k$ , there exists a unique linear and continuous extension  $\bar{\ell}_f: H_k \to \mathbb{R}$  of  $\ell_f$ . Furthermore, from the Riesz Representation Theorem there exists a unique  $\hat{f} \in H_k$  with  $\bar{\ell}_f = \langle \cdot, \hat{f} \rangle_k$ . For all  $x \in \mathcal{X}$  we then get
$$
\begin{array}{l} \hat {f} (x) = \langle \hat {f}, k (\cdot , x) \rangle_ {k} \\ = \langle k (\cdot , x), \hat {f} \rangle_ {k} \\ = \bar {\ell} _ {f} (k (\cdot , x)) \\ = \ell_ {f} (k (\cdot , x)) \\ = f (x), \\ \end{array}
$$
hence  $f = \hat{f}\in H_{k}$
B.2 A  $\Gamma$ -convergence argument
We use repeatedly the concept of  $\Gamma$ -convergence, see for example [32]. For convenience, in this section we summarize the well-known and standard main argument, roughly following [33, Chapter 5].
Definition B.2. Let  $F_{M}: H_{M} \to \mathbb{R} \cup \{\infty\}$  and  $F: H_{k} \to \mathbb{R} \cup \{\infty\}$ . We say that  $F_{M}$  is converges to  $F$  and write  $F_{M} \xrightarrow{\Gamma} F$ , if
1. For all sequences  $(f_M)_M, f_M \in H_M$ , with  $f_M \xrightarrow{\mathcal{P}_1} f$  for some  $f \in H_k$ , we have
$$
F (f) \leq \lim  _ {M} \inf  F _ {M} (f _ {M}).
$$
2. For all  $f \in H_k$  there exists a sequence  $(f_M)_M$  with  $f_M \in H_M$  such that  $f_M \xrightarrow{\mathcal{P}_1} f$  and
$$
F(f)\geq \limsup_{M}F_{M}(f_{M}).
$$
The sequence in the second item is commonly called a recovery sequence (for  $f$ ).
Proposition B.3. Let  $F_{M} \xrightarrow{\Gamma} F$  and  $f_{M}^{*} \in \operatorname{argmin}_{f \in H_{M}} F_{M}(f)$  for all  $M \in \mathbb{N}$  (in particular, all the minima are attained). If there exists  $B \in \mathbb{R}_{\geq 0}$  such that  $\| f_{M}^{*} \|_{M} \leq B$  for all  $M \in \mathbb{N}$ , then there exists a subsequence  $(f_{M_{\ell}}^{*})_{\ell}$  and  $f^{*} \in H_{k}$  such that  $f_{M_{\ell}}^{*} \xrightarrow{\mathcal{P}_{1}} f^{*}$ . Furthermore,  $F_{M_{\ell}}(f_{M_{\ell}}^{*}) \to F(f^{*})$ .
Proof. From Theorem 2.3 we get the existence of  $(f_{M_\ell}^*)_\ell$  and  $f^* \in H_k$ , and that  $f_{M_\ell}^* \xrightarrow{\mathcal{P}_1} f^*$ . Let  $f \in H_k$  be arbitrary and let  $(f_M)_M$  be a recovery sequence for  $f$ . We then have