SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\lambda_ {i} = \max  _ {\substack {\boldsymbol {u} \in \mathbb {R} ^ {N} \backslash \{\boldsymbol {0} \} \\ \boldsymbol {u} ^ {T} \boldsymbol {u} _ {j} = 0, \forall j <   i}} \frac {\boldsymbol {u} ^ {T} \boldsymbol {A} \boldsymbol {u}}{\boldsymbol {u} ^ {T} \boldsymbol {u}} \tag{101}
$$
where the eigenvectors  $\mathbf{u}_i$  are the respective maximizers.
Proof. Horn and Johnson [26], Theorem 4.2.2, modified appropriately to handle eigenvalues in decreasing order, and where we choose  $i_1, \ldots, i_k = 1, \ldots, N - i + 1$ .
We will prove the main claim below in two stages: first, on representer weight space  $R_{k,\pmb{x}}$ , then on the full RKHS  $H_{k}$ . We begin with the former.
$$
u ^ {(i)} (\cdot) = \underset {u \in R _ {k, \pi}} {\arg \max } \left\{\sum_ {i = 1} ^ {N} u \left(x _ {i}\right) ^ {2}: \| u \| _ {H _ {k}} = 1, \langle u, u ^ {(j)} \rangle_ {H _ {k}} = 0, \forall j <   i \right\}. \tag {102}
$$
Proof. Define  $\pmb{w}_j = \mathbf{U}\pmb{\Lambda}^{-1/2}\pmb{u}_j$ , where we recall that  $\pmb{u}_j$  are the respective eigenvectors of the kernel matrix. From the Result 12 form of the Courant-Fischer Min-Max Theorem, the reproducing property, and the explicit form of the RKHS inner product on  $R_{k,\pmb{x}}$  in the basis defined by the canonical basis functions, we can conclude that
$$
\lambda_ {i} = \max  _ {\substack {\boldsymbol {u} \in \mathbb {R} ^ {N} \backslash \{\boldsymbol {0} \} \\ \boldsymbol {u} ^ {T} \boldsymbol {u} _ {j} = 0, \forall j <   i}} \frac {\boldsymbol {u} ^ {T} \mathbf {K} _ {\boldsymbol {x x}} \boldsymbol {u}}{\boldsymbol {u} ^ {T} \boldsymbol {u}} = \max  _ {\substack {\boldsymbol {w} \in \mathbb {R} ^ {N} \backslash \{\boldsymbol {0} \} \\ \boldsymbol {w} ^ {T} \mathbf {K} _ {\boldsymbol {x x}} \boldsymbol {w} _ {j} = 0, \forall j <   i}} \frac {\boldsymbol {w} ^ {T} \mathbf {K} _ {\boldsymbol {x x}} ^ {2} \boldsymbol {w}}{\boldsymbol {w} ^ {T} \mathbf {K} _ {\boldsymbol {x x}} \boldsymbol {w}} \tag{103}
$$
$$
\begin{array}{l} = \max  _ {\substack {\boldsymbol {w} \in \mathbb {R} ^ {N} \backslash \{\boldsymbol {0} \} \\ \boldsymbol {w} ^ {T} \boldsymbol {K} _ {\boldsymbol {x x}} \boldsymbol {w} _ {j} = 0, \forall j <   i}} \frac {\left\| \mathbf {K} _ {\boldsymbol {x x}} \boldsymbol {w} \right\| _ {2} ^ {2}}{\boldsymbol {w} ^ {T} \boldsymbol {K} _ {\boldsymbol {x x}} \boldsymbol {w}} = \max  _ {\substack {\boldsymbol {w} \in \mathbb {R} ^ {N} \backslash \{\boldsymbol {0} \} \\ \boldsymbol {w} ^ {T} \boldsymbol {K} _ {\boldsymbol {x x}} \boldsymbol {w} _ {j} = 0, \forall j <   i}} \frac {\left\| \sum_ {i = 1} ^ {N} w _ {i} k (x _ {i} , \boldsymbol {x}) \right\| _ {2} ^ {2}}{\left\| \sum_ {i = 1} ^ {N} w _ {i} k (x _ {i} , \cdot) \right\| _ {H _ {k}}} (104) \\ = \max  _ { \begin{array}{c} u \in R _ {k, x} \backslash \{0 \} \\ \langle u, u ^ {(j)} \rangle_ {H _ {k}} = 0, \forall j <   i \end{array} } \frac {\| u (\boldsymbol {x}) \| _ {2} ^ {2}}{\| u \| _ {H _ {k}} ^ {2}} = \max  _ { \begin{array}{c} u \in R _ {k, x} \\ \| u \| _ {H _ {k}} = 1 \\ \langle u, u ^ {(j)} \rangle_ {H _ {k}} = 0, \forall j <   i \end{array} } \sum_ {i = 1} ^ {N} u \left(x _ {i}\right) ^ {2} (105) \\ \end{array}
$$
Next, we use a Representative-Theorem-type orthogonality argument to extend the optimization problem to all of  $H_{k}$ . The argument will rely heavily on the Projection Theorem for Hilbert spaces: for an overview of the setting, see Lang [28], Chapter 5.
Proposition 14. We have
$$
u ^ {(i)} (\cdot) = \underset {u \in H _ {k}} {\arg \max } \left\{\sum_ {i = 1} ^ {N} u \left(x _ {i}\right) ^ {2}: \| u \| _ {H _ {k}} = 1, \langle u, u ^ {(j)} \rangle_ {H _ {k}} = 0, \forall j <   i \right\}. \tag {106}
$$
Proof. Let  $H_{k}^{(i)}$  be the orthogonal complement of the finite-dimensional subspace  $\operatorname{span}\{u^{(j)}, j = 1,..,i - 1\}$  in  $H_{k}$  and note by feasibility that  $u^{(i)} \in H_{k}^{(i)}$ . Consider the decomposition of  $u^{(i)}$  into its projection onto an orthogonal complement with respect to the finite-dimensional subspace  $H_{k}^{(i)} \cap R_{k,\mathbf{x}}$ , namely
$$
u ^ {(i)} (\cdot) = u ^ {\parallel} (\cdot) + u ^ {\perp} (\cdot) \tag {107}
$$
where this notation is defined as  $u^{\parallel} = \mathrm{proj}_{H_k^{(i)}\cap R_{k,\varpi}}u^{(i)}$  and  $u^{\perp} = u^{(i)} - u^{\parallel}$ . Observe that
$$
u ^ {\perp} \left(x _ {i}\right) = \left\langle u ^ {\perp}, k \left(x _ {i}, \cdot\right) \right\rangle_ {H _ {k}} = \left\langle \operatorname {p r o j} _ {H _ {k} ^ {(i)}} u ^ {\perp}, k \left(x _ {i}, \cdot\right) \right\rangle_ {H _ {k}} = \left\langle u ^ {\perp}, \operatorname {p r o j} _ {H _ {k} ^ {(i)}} k \left(x _ {i}, \cdot\right) \right\rangle_ {H _ {k}} = 0 \tag {108}
$$
where the first equality follows from the reproducing property, the second equality follows because  $u^{\perp} \in H_{k}^{(i)}$ , the third equality follows because the projection is orthogonal and therefore self-adjoint, and last equality follows since  $u^{\perp}$  by definition lives in the orthogonal complement of  $H_{k}^{(i)} \cap R_{k,\mathbf{x}}$  within  $H_{k}^{(i)}$ , with the subspace inner product. Moreover, by the Projection Theorem and the fact that  $H_{k}^{(i)}$  inherits its norm from  $H_{k}$ , note that
$$
\left\| u ^ {(i)} \right\| _ {H _ {k}} ^ {2} = \left\| u ^ {\parallel} \right\| _ {H _ {k}} ^ {2} + \left\| u ^ {\perp} \right\| _ {H _ {k}} ^ {2}. \tag {109}
$$
We now argue by contradiction: suppose that  $u^{\perp} \neq 0$ . This means that  $\| u^{\perp} \|_{H_k} > 0$ . From the above, we have
$$
\sum_ {i = 1} ^ {N} u ^ {(i)} \left(x _ {i}\right) ^ {2} = \sum_ {i = 1} ^ {N} u ^ {\parallel} \left(x _ {i}\right) ^ {2} \quad \| u ^ {\parallel} \| _ {H _ {k}} <   \| u ^ {(i)} \| _ {H _ {k}}. \tag {110}
$$
Define the function
$$
u ^ {\prime} (\cdot) = \frac {\| u ^ {(i)} \| _ {H _ {k}}}{\| u ^ {\parallel} \| _ {H _ {k}}} u ^ {\parallel} (\cdot) \tag {111}
$$
and note that  $\| u^{\prime}\| = 1$ , and  $\langle u^{\prime},u^{(j)}\rangle = 0$  for all  $j < i$  since  $u^{\parallel}\in H_k^{(i)}$ , which makes  $u^{\prime}$  a feasible solution to the optimization problem considered. At the same time, it satisfies
$$
\sum_ {i = 1} ^ {N} u ^ {\prime} \left(x _ {i}\right) ^ {2} = \frac {\left\| u ^ {(i)} \right\| _ {H _ {k}} ^ {2}}{\left\| u ^ {\parallel} \right\| _ {H _ {k}} ^ {2}} \sum_ {i = 1} ^ {N} u ^ {\parallel} \left(x _ {i}\right) ^ {2} > \sum_ {i = 1} ^ {N} u ^ {\parallel} \left(x _ {i}\right) ^ {2} = \sum_ {i = 1} ^ {N} u ^ {(i)} \left(x _ {i}\right) ^ {2} \tag {112}
$$
which shows that  $u^{(i)}$ , contrary to its definition, is not actually optimal—contradiction. We conclude that  $u^{\perp} = 0$ , which means that  $u^{(i)} \in H_k^{(i)} \cap R_{k,\mathbf{x}}$ , and the claim follows from Lemma 13.
This means that the top eigenvalues can be understood as the largest possible squared sums of canonical basis functions evaluated at the data, subject to RKHS-norm constraints. In situations where  $k(x,\cdot)$  is continuous, non-negative, and decays as one moves away from the data, it is intuitively clear that such sums will be maximized by placing weight on canonical basis functions which cover as much of the data as possible. This mirrors the RKHS view of kernel principal component analysis.