SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. First, let  $\beta \in \mathbb{R}^{s_{\phi}}$  be the ridge regression solution involving  $\widetilde{\mathbf{X}}$  and  $y$ , i.e.,
$$
\beta \in \operatorname *{arg  min}_{x\in \mathbb{R}^{s_{\phi}}}\frac{1}{n}\sum_{i = 1}^{n}\left|y_{i} - \widetilde{\mathbf{X}}_{i*}x\right|^{2} + \lambda s\| x\|_{2}^{2}.
$$
Note that, by construction, for every  $i\in [n]$ ,  $f_{[\widetilde{\mathbf{X}},y,\phi ]}^{\lambda}\left(\widetilde{\mathbf{X}}_{i*}\right)$  is equal to  $\widetilde{\mathbf{X}}_{i*}\cdot \beta$
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \left| y _ {i} - f _ {[ \widetilde {\mathbf {X}}, y, \phi ]} ^ {\lambda} \left(\widetilde {\mathbf {X}} _ {i *}\right) \right| ^ {2} = \frac {1}{n} \sum_ {i = 1} ^ {n} \left| y _ {i} - \widetilde {\mathbf {X}} _ {i *} \beta \right| ^ {2} = \inf  _ {\| \widetilde {f} \| _ {\mathcal {H}} \leq \sqrt {2}} \frac {1}{n} \sum_ {i = 1} ^ {n} \left| y _ {i} - \widetilde {f} \left(\mathbf {X} _ {i *}\right) \right| ^ {2} \tag {10}
$$
where the first equality holds by definition of  $f_{[\widetilde{\mathbf{X}},y,\phi]}^{\lambda}\left(\widetilde{\mathbf{X}}_{i*}\right)$  for every  $i\in [n]$ , and the second equality follows from the proof of Lemma 6 of [LTOS21] which indicates that  $s\| \beta \| _2^2\leq 2$ .
$$
\begin{array}{l} \inf  _ {\left\| \widetilde {f} \right\| _ {\mathcal {H}} \leq \sqrt {2}} \frac {1}{n} \sum_ {i = 1} ^ {n} \left| y _ {i} - \widetilde {f} \left(\mathbf {X} _ {i *}\right) \right| ^ {2} \\ = \inf  _ {\| \widetilde {f} \| _ {\mathcal {H}} \leq \sqrt {2}} \frac {1}{n} \sum_ {i = 1} ^ {n} \left| y _ {i} - f _ {[ \mathbf {X}, y, k ]} ^ {\lambda} \left(\mathbf {X} _ {i *}\right) + f _ {[ \mathbf {X}, y, k ]} ^ {\lambda} \left(\mathbf {X} _ {i *}\right) - \widetilde {f} \left(\mathbf {X} _ {i *}\right) \right| ^ {2} \tag {11} \\ = \inf  _ {\| \widetilde {f} \| _ {\mathcal {H}} \leq \sqrt {2}} \frac {1}{n} \sum_ {i = 1} ^ {n} \left| y _ {i} - f _ {[ \mathbf {X}, y, k ]} ^ {\lambda} (\mathbf {X} _ {i *}) \right| ^ {2} + \frac {1}{n} \sum_ {i = 1} ^ {n} \left| \widetilde {f} (\mathbf {X} _ {i *}) - f _ {[ \mathbf {X}, y, k ]} ^ {\lambda} (\mathbf {X} _ {i} *) \right| ^ {2} \\ + \frac {2}{n} \sum_ {i = 1} ^ {n} \left(y _ {i} - f _ {[ \mathbf {X}, y, k ]} ^ {\lambda} \left(\mathbf {X} _ {i *}\right)\right) \left(f _ {[ \mathbf {X}, y, k ]} ^ {\lambda} \left(\mathbf {X} _ {i *}\right) - \widetilde {f} \left(\mathbf {X} _ {i *}\right)\right) \\ \end{array}
$$
where the first equality holds by adding and subtracting the same amount, while the last equality holds by simple expansion.
To properly bound the terms above, we rely on Lemma 7 of [LTOS21].
Lemma 8 (Restatement of Lemma 7 [LTOS21]). Under Assumption 1 and the definitions in Theorem 2, and for
$$
\widetilde {f} ^ {\lambda} \in \underset {\widetilde {f} \in \widetilde {\mathcal {H}}} {\arg \min} \frac {1}{n} \sum_ {i = 1} ^ {n} \left| f _ {[ \mathbf {X}, y, k ]} ^ {\lambda} (\mathbf {X} _ {i *}) - \widetilde {f} (\mathbf {X} _ {i *}) \right| ^ {2} + \lambda \left\| \widetilde {f} \right\| _ {\widetilde {\mathcal {H}}},
$$
where  $\widetilde{\mathcal{H}}$  is the corresponding RKHS space of  $\widetilde{\mathbf{X}}$ , it holds that
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \left(y _ {i} - f _ {[ \mathbf {X}, y, k ]} ^ {\lambda} \left(\mathbf {X} _ {i *}\right)\right) \left(f _ {[ \mathbf {X}, y, k ]} ^ {\lambda} \left(\mathbf {X} _ {i *}\right) - \widetilde {f} ^ {\lambda} \left(\mathbf {X} _ {i *}\right)\right) \leq \lambda .
$$
Hence, combining Lemma 8 with (11) yields
$$
\begin{array}{l} \inf_{\| \widetilde{f}\|_{\mathcal{H}}\leq \sqrt{2}}\frac{1}{n}\sum_{i = 1}^{n}\left|y_{i} - \widetilde{f}\left(\mathbf{X}_{i*}\right)\right|^{2} \\ \leq \frac {1}{n} \sum_ {i = 1} ^ {n} \left| y _ {i} - f _ {[ \mathbf {X}, y, k ]} ^ {\lambda} \left(\mathbf {X} _ {i *}\right) \right| ^ {2} + 2 \lambda + \inf  _ {\left\| \widetilde {f} \right\| _ {\mathcal {H}} \leq \sqrt {2}} \frac {1}{n} \sum_ {i = 1} ^ {n} \left| \widetilde {f} \left(\mathbf {X} _ {i *}\right) - f _ {[ \mathbf {X}, y, k ]} ^ {\lambda} \left(\mathbf {X} _ {i *}\right) \right| ^ {2} \tag {12} \\ \leq \frac {1}{n} \sum_ {i = 1} ^ {n} \left| y _ {i} - f _ {[ \mathbf {X}, y, k ]} ^ {\lambda} \left(\mathbf {X} _ {i *}\right) \right| ^ {2} + 4 \lambda \\ \end{array}
$$
where the last inequality holds by Lemma 4 (or equivalently Lemma 6 of [LTOS21]), and thus concluding Theorem 2.
C Regarding  $s_{\phi}$
In what follows, we discuss cases for determining the quantity  $s_{\phi}$ . To quantify  $s_{\phi}$  which in turn determines the size of the distilled set, we need to measure  $d_{K}$  which is the trace of  $\mathbf{K}\left(\mathbf{K} + n\lambda I_n\right)^{-1}$ :
Table 1: Table 1 from [LTOS19]. The trade-off in the worst case for the squared error loss.
SAMPLING SCHEME	SPECTRUM	NUMBER OF FEATURES
WEIGHTED RFF	finite rank	\(s_{\phi}\in \Omega (1)\)
\(\lambda_{i}\propto A^{i}\)	\(s_{\phi}\in \Omega (\log n\cdot \log \log n)\)
\(\lambda_{i}\propto i^{-2t}(t\geq 1)\)	\(s_{\phi}\in \Omega (n^{1/2t}\cdot \log n)\)
\(\lambda_{i}\propto i^{-1}\)	\(s_{\phi}\in \Omega (\sqrt{n}\cdot \log n)\)
PLAIN RFF	finite rank	\(s_{\phi}\in \Omega (\sqrt{n})\)
\(\lambda_{i}\propto A^{i}\)	\(s_{\phi}\in \Omega (\sqrt{n}\cdot \log \log n)\)
\(\lambda_{i}\propto i^{-2t}(t\geq 1)\)	\(s_{\phi}\in \Omega (\sqrt{n}\cdot \log n)\)
\(\lambda_{i}\propto i^{-1}\)	\(s_{\phi}\in \Omega (\sqrt{n}\cdot \log n)\)
- For the case where  $K$  has a finite rank, i.e., the number of positive eigenvalues is lower than  $n$ , then  $s_{\phi} \in \Omega(1)$ .
- As for the exponential decay, it occurs when the kernel is Gaussian and the marginal distribution of the input data (e.g., images) is sub-Gaussian. In such a case, it was shown in [Bac17] that  $d_{K} \in O\left(\log \frac{1}{\lambda}\right)$ . Thus  $s_{\phi}$  is poly-logarithmic in  $n$  when  $\lambda := O\left(\frac{1}{\sqrt{n}}\right)$ .
- For the case where the Hilbert space  $\mathcal{H}$  is also a Sobolov space of order  $\gamma$  larger or equal to 1, then  $d_{K} \in O\left(\frac{1}{\lambda^{2\gamma}}\right)$  which in the case of  $\lambda := O\left(\frac{1}{\sqrt{n}}\right)$ , we have  $s_{\phi} \in \Omega\left(n^{\frac{1}{4\gamma}}\right)$ .
- In the general case, where the decay of the eigenvalues admits  $\lambda_{i} \propto O(i^{-1})$ , then  $d_{k}$  in the worst case is bounded by  $O\left(\frac{1}{\text{lambda}}\right)$ , and thus, via Theorem 2, we can deduce that  $s_{\phi} \in O(\sqrt{n} \log n)$  for the case of  $\lambda \in O\left(\frac{1}{\sqrt{n}}\right)$ .
Note that the choice of  $\lambda \in O\left(\frac{1}{\sqrt{n}}\right)$  is used in the literature of KRR to ensure that the learning rate is  $\frac{1}{\sqrt{n}}$  as shown in [RR17].
D Multi-class Experiments
In fig. 7 and fig. 8 we test our bound for the multi-class case on MNIST 0,1,2 digits with similar settings as in the binary classification case. The results justify our bounds.
Figure 7: Distillation results for MNIST multi class 0, 1, 2 classification  $(n = 3)$
Figure 8: Distillation results for MNIST multi class 0, 1, 2 classification  $(n = 3)$ ; testing the  $8\lambda$  bound
E On the rank of  $\widetilde{\mathbf{X}}$  and its connection to the distilled set  $S$
First, recall that  $\widetilde{\mathbf{X}}$  is the RFF image of  $X$ . There are mainly two cases connecting the rank of  $\widetilde{\mathbf{X}}$  to  $S$ :
(I) The rank of  $\widetilde{\mathbf{X}}$  is  $s_{\phi}$ , i.e.,  $\widetilde{\mathbf{X}}$  is of full rank.
(II) Otherwise.
Regarding Case (I). For such a case, Theorem 3 is exactly what we need to prove the existence of a distilled set  $S$ .
Handling Case (II). For this case as well, one can also choose to ensure that the rank of  $\mathbf{S}$  (matrix form of  $S$ ) is  $s_{\phi}$  (full rank) and apply the same derivations done in the proof of Theorem 3 without any violation from the perspective of the validity of our proof. However, due to the nature of this case, one can also ensure that the size of  $S$  (number of instances) is much smaller than  $s_{\phi}$ , specifically,  $r$  (the rank of  $\widetilde{\mathbf{X}}$ ). To that end, must ensure that
(A) the following equation
$$
\left(\widetilde {\mathbf {S}} ^ {T} \widetilde {\mathbf {S}} + \lambda n s _ {\phi} \lambda \mathbf {I} _ {s _ {\phi}}\right) \left(\widetilde {\mathbf {X}} ^ {T} \widetilde {\mathbf {X}} + \lambda n s _ {\phi} \lambda \mathbf {I} _ {s _ {\phi}}\right) ^ {- 1} \widetilde {\mathbf {X}} ^ {T} y = \widetilde {\mathbf {S}} ^ {T} y \mathbf {s},
$$
has a solution, i.e.,  $y_{\mathbf{S}}$  exists, and also
(B) we need to ensure that (6) also has a solution, i.e., there exists  $\alpha \in \mathbb{R}^r$  such that
$$
\frac {1}{s _ {\phi}} \widetilde {\mathbf {X}} ^ {T} \left(\frac {1}{s _ {\phi}} \widetilde {\mathbf {X}} \widetilde {\mathbf {X}} ^ {T} + n \lambda \mathbf {I} _ {n}\right) ^ {- 1} \mathbf {f} _ {\mathbf {S}} [ \mathbf {X} ] = \mathbf {A} \alpha .
$$
We observe that since  $\widetilde{\mathbf{X}}$  is rank deficient, it adds some obstacles to our proofs; when attempting to ensure both (A) and (B). Specifically speaking, the rank of  $\left(\widetilde{\mathbf{S}}^T\widetilde{\mathbf{S}} + \lambda ns_\phi \lambda \mathbf{I}_{s_\phi}\right)\left(\widetilde{\mathbf{X}}^T\widetilde{\mathbf{X}} + \lambda ns_\phi \lambda \mathbf{I}_{s_\phi}\right)^{-1}\widetilde{\mathbf{X}}^T$  is equal to the rank of  $\widetilde{\mathbf{X}}$  (due to properties of rank function) and since we aim to maintain that  $S$  has exactly the number of rows while also its RFF image  $\widetilde{\mathbf{S}}$  has a rank equal to that of  $\widetilde{\mathbf{X}}$ , we obtain that the linear system of equations in (A) might have no solution at all. This is unless the row span of  $\widetilde{\mathbf{S}}$  coincides with that of  $\widetilde{\mathbf{X}}$ .
Having an  $S$  such that the row span of its RFF image coincides with the row span of  $\tilde{\mathbf{X}}$ , also ensures the existence of a solution concerning the linear system of equations present at (B).
F One of many ways to construct  $S$
Following the previous section, to properly construct  $S$  that would satisfy the proof of Theorem 3, we discuss two possible constructions:
- If  $\tilde{X}$  is of full rank (the rank is  $s_{\phi}$ ), then there exists a subset  $S$  of the rows of  $X$  such that its image in the RFF space is a linearly independent subset  $\tilde{S}$  of the rows of  $\tilde{X}$ . Such a subset is of full rank as it is linearly independent. Such an algorithm can be accessed via [J23].
- If  $\tilde{X}$  is rank deficient, i.e.,  $r < s_{\phi}$  where  $r$  denotes the rank of  $\tilde{X}$ , then the problem by itself requires  $r$  independent vectors, i.e., one can represent  $\left(\tilde{X}^T\tilde{X} + \lambda ns_{\phi}\lambda I_{s_{\phi}}\right)^{-1}\tilde{X}^Ty$  by  $Az$  where  $A \in \mathbb{R}^{s_{\phi} \times r}$  and  $z \in \mathbb{R}^r$ . This is since the rank of  $\left(\tilde{X}^T\tilde{X} + \lambda ns_{\phi}\lambda I_{s_{\phi}}\right)^{-1}\tilde{X}^T$  is at max the minimum between the rank of  $\left(\tilde{X}^T\tilde{X} + \lambda ns_{\phi}\lambda I_{s_{\phi}}\right)^{-1}$  and the rank of  $\tilde{X}^T$ . Such rank is bounded from above by  $r$  since the rank of  $\left(\tilde{X}^T\tilde{X} + \lambda ns_{\phi}\lambda I_{s_{\phi}}\right)^{-1}$  is  $s_{\phi}$ .
To this end, we can invoke the first case with respect to  $(B,z)$ . Thus, by choosing  $S$  to contain the  $r$  linearly independent rows in  $\tilde{X}$  and an additional row from  $\tilde{X}$  (not already in  $S$ ), there exists  $y_{S}$  such that,
$$
\left(\tilde {S} ^ {T} \tilde {S} + \lambda n s _ {\phi} \lambda I _ {s _ {\phi}}\right) B z = \tilde {S} ^ {T} y _ {\mathbf {S}}.
$$
Connection to subset selection and label solve. Observe that according to this section, we showed that our bounds in Theorem 3 hold for a subset of the data but with different (potentially learned) labels (not necessarily from the set of input labels) which can be thought of as a hybrid between distillation and subset selection. Intuitively, this justifies the label-solve approach of [NCL20], where here we showed that a subset of the data can achieve similar bounds but with learning the labels.