SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Alaa Maalouf*  
MIT CSAIL
Murad Tukan* DataHeroes
Noel Loo  
MIT CSAIL
Ramin Hasani  
MIT CSAIL
Mathias Lechner  
MIT CSAIL
Daniela Rus  
MIT CSAIL
Dataset Distillation is the task of synthesizing small datasets from large ones while still retaining comparable predictive accuracy to the original uncompressed dataset. Despite significant empirical progress in recent years, there is little understanding of the theoretical limitations/guarantees of dataset distillation, specifically, what excess risk is achieved by distillation compared to the original dataset, and how large are distilled datasets? In this work, we take a theoretical view on kernel ridge regression (KRR) based methods of dataset distillation such as Kernel Inducing Points. By transforming ridge regression in random Fourier features (RFF) space, we provide the first proof of the existence of small (size) distilled datasets and their corresponding excess risk for shift-invariant kernels. We prove that a small set of instances exists in the original input space such that its solution in the RFF space coincides with the solution of the original data. We further show that a KRR solution can be generated using this distilled set of instances which gives an approximation towards the KRR solution optimized on the full input data. The size of this set is linear in the dimension of the RFF space of the input set or alternatively near linear in the number of effective degrees of freedom, which is a function of the kernel, number of datapoints, and the regularization parameter  $\lambda$ . The error bound of this distilled set is also a function of  $\lambda$ . We verify our bounds analytically and empirically.
Motivated by the growing data demands of modern deep learning, dataset distillation [ZB21, NNXL21, ZNB22, WZTE18] aims to summarize large datasets into significantly smaller synthetic distilled datasets, which when trained on retain high predictive accuracy, comparable to the original dataset. These distilled datasets have applications in continual learning [ZNB22, SCCB22], architecture search  $\left[\mathrm{SRL}^{+}19\right]$ , and privacy preservation [CKF22]. Recent years have seen the development of numerous distillation algorithms, but despite this progress, the field has remained largely empirical. Specifically, there is little understanding of what makes one dataset "easier to distill" than another, or whether such small synthetic datasets even exist.
This work aims to fill this gap by providing the first bounds on the sufficient size and relative error associated with distilled datasets. Noting prior work relating neural network training to kernel ridge regression (KRR), we consider dataset distillation in the kernel ridge regression settings with shift-invariant kernels. By casting the problem into the Random Fourier Feature (RFF) space, we show that: The size and relative error of distilled datasets is governed by the kernel's "number of effective degrees of freedom",  $d_k^\lambda$ . Specifically, in Section 4, we show that distilled sets of size  $\Omega(d_k^\lambda \log d_k^\lambda)$ , exist, with  $12\lambda + 2\mathcal{L}_\lambda$  predictive error on the training dataset, and only  $8\lambda$  error with
*Equal contribution. Correspondence E-mail: alaam@mit.edu.
respect to the optimal solution computed on the full dataset, where  $\lambda$  is the kernel ridge regression regularization parameter and  $\mathcal{L}_{\lambda}$  the KRR training error on the original dataset; see Theorem 3 and Remark 7 for full details.
These bounds hold in practice for both real and synthetic datasets. In section 5, we validate our theorem by distilling synthetic and real datasets with varying sizes and values of  $d_k^\lambda$ , showing that in all scenarios our bounds accurately predict the error associated with distillation.
**Coresets.** Coresets are weighted selections from a larger training dataset, which, when used for training, yield similar outcomes as if the whole dataset was used [MSSW18, MBL20, PDM22,  $\mathrm{MEM}^{+}22$ ]. The key benefit of using coresets is that they significantly speed up the training process, unlike when the full data set is used. Current coresets methods incorporate clustering techniques [FL11, LBK16, BLHK16], bilevel optimization [BMK20], sensitivity analysis [MSSW18, HCB16, TMF20, MSF20], and surrogate models for approximation  $\left[\mathrm{TZM}^{+}23\right]$ . Newer strategies are specifically designed for neural networks, where before each training epoch, coresets are chosen such that their gradients align with the gradients of the entire dataset [MBL20, PDM22,  $\mathrm{TZM}^{+}23$ ], followed by training the model on the chosen coreset. Although coresets are usually theoretically supported, these methods fall short when the aim is to compute a coreset once for a full training procedure.
Dataset Distillation. To this end, dataset distillation algorithms construct synthetic datasets (not necessarily a subset from the original input) such that gradient descent training on the synthetic datapoints results in high predictive accuracy on the real dataset. Cast as a bilevel optimization problem, early methods involve unrolling training computation graph [WZTE18] for a few gradient descent steps and randomly sampled weight initializations. More sophisticated methods aim to approximate the unrolled computation using kernel methods [NCL21, NNXL21, ZNB22, LHR22a, LHLR23a], surrogate objectives such gradient matching [ZMB21, ZB21], trajectory matching  $\mathrm{[CWT^{+}22]}$  or implicit gradients [LHLR23b]. The kernel-induced points (KIP) algorithm [NCL21, NNXL21] is a technique that employs Neural Tangent Kernel (NTK) theory[JGH18, LHR22b] to formulate the ensuing loss:  $\mathcal{L}_{KIP} = \frac{1}{2}\| y_t - K_{TS}K_{SS}^{-1}y_S\|_2^2$ . This loss signifies the predictive loss of training infinitely wide networks on distilled datapoints  $X_{S}$  with corresponding labels  $y_{S}$ , on the original training set and labels  $X_{T},y_{T}$ , with  $K_{\cdot ,\cdot}$  being the NTK. Dataset distillation is closely related to the use of inducing points to accelerate Gaussian Processes [SG05, TRB16], for which convergence rates exist, but the existence of such inducing points is not unknown [BRVDW19].
From dataset distillation to kernel ridge regression. Kernel ridge regression (KRR) extends the linear machine learning ridge regression model by using a kernel function to map input data into higher-dimensional feature spaces, allowing for more complex non-linear relationships between variables to be captured [Mur12]. Various methods have been proposed to improve and accelerate the training process of kernel ridge regression. Most notably, Random Fourier Features [RR07] approximates shift-invariant kernel functions by mapping the input data into a lower-dimensional feature space using a randomized cosine transformation. This has been shown to work effectively in practice due to regularizing effects [JSS+20], as well as providing approximation bounds to the full kernel ridge regression [SS15, AKM+17, LTOS19]. Training infinite-width neural networks can be cast as kernel ridge regression with the Neural Tangent Kernel (NTK) [JGH18], which allows a closed-form solution of the infinite-width neural network's predictions, enabling kernel-based dataset distillation algorithms such as [NCL21, NNXL21, LHAR22a].
Goal. We provide the first provable guarantees on the existence and approximation error of a small distilled dataset in the KRR settings. We first provide notations that will be used throughout the paper.
Notations. Let  $\mathcal{H}$  be a Hilbert space with  $\| \cdot \|_{\mathcal{H}}$  as its norm. For a vector  $a\in \mathbb{R}^n$ , we use  $\| a\| _2$  to denote its Euclidean norm, and  $a_{i}$  to denote its  $i$ th entry for every  $i\in [n]$ . For any positive integer  $n$ , we use the convention  $[n] = \{1,2,\dots ,n\}$ . Let  $A\in \mathbb{R}^{n\times m}$  be a matrix, then, for every  $i\in [n]$  and  $j\in [m]$ ,  $A_{i*}$  denotes the  $i$ th row of  $A$ ,  $A_{*j}$  denotes the  $j$ th column of  $A$ , and  $A_{i,j}$  is the  $j$ th entry of the  $i$ th row of  $A$ . Let  $B\in \mathbb{R}^{n\times n}$ , then we denote the trace of  $B$  by  $Tr(B)$ . We use  $\mathbf{I}_m\in \mathbb{R}^{m\times m}$  to denote the identity matrix. Finally, vectors are addressed as column vectors unless stated otherwise.
3.1 Kernel ridge regression
Let  $\mathbf{X} \in \mathbb{R}^{n \times d}$  be a matrix and let  $y \in \mathbb{R}^n$  be a vector. Let  $k: \mathbb{R}^d \times \mathbb{R}^d \to [0, \infty)$  be a kernel function, and let  $\mathbf{K} \in \mathbb{R}^{n \times n}$  be its corresponding kernel matrix with respect to the rows of  $\mathbf{X}$ ; i.e.,  $\mathbf{K}_{i,j} = k(\mathbf{X}_{i*}, \mathbf{X}_{j*})$  for every  $i, j \in [n]$ . Let  $\lambda > 0$  be a regularization parameter. The goal of kernel ridge regression (KRR) involving  $\mathbf{X}, y, k$ , and  $\lambda$  is to find
$$
\alpha_ {[ \mathbf {X}, y, k ]} ^ {\lambda} \in \operatorname * {a r g   m i n} _ {\alpha \in \mathbb {R} ^ {n}} \frac {1}{n} \| y - \mathbf {K} \alpha \| _ {2} ^ {2} + \lambda \alpha^ {T} \mathbf {K} \alpha . \tag {1}
$$
We use the notation  $f_{[\mathbf{X},y,k]}^{\lambda} \colon \mathbb{R}^{d} \to \mathbb{R}$  to denote the in-sample prediction by applying the KRR solution obtained on  $\mathbf{X}, y$  and  $\lambda$  using the kernel  $k$ , i.e., for every  $x \in \mathbb{R}^{d}$ ,
$$
f _ {[ \mathbf {X}, y, k ]} ^ {\lambda} (x) = \sum_ {i = 1} ^ {n} \alpha_ {[ \mathbf {X}, y, k ] _ {i}} ^ {\lambda} k \left(\mathbf {X} _ {i *}, x\right). \tag {2}
$$
To provide our theoretical guarantees on the size and approximation error for the distilled datasets, the following assumption will be used in our theorem and proofs.
Assumption 1. We inherit the same theoretical assumptions used at [LTOS21] for handling the KRR problem: (I) Let  $\mathcal{F}$  be the set of all functions mapping  $\mathbb{R}^d$  to  $\mathbb{R}$ . Let  $f^{*} \in \mathcal{F}$  be the minimizer of  $\sum_{i=1}^{n} |y_i - f(\mathbf{X}_{i*})|^2$ , subject to the constraint that for every  $x \in \mathbb{R}^d$  and  $y \in \mathbb{R}$ ,  $y = f^{*}(x) + \epsilon$ , where  $\mathbb{E}(\epsilon) = 0$  and  $\text{Var}(\epsilon) = \sigma^2$ . Furthermore, we assume that  $y$  is bounded, i.e.,  $|y| \leq y_0$ . (II) We assume that  $\left\| f_{[\mathbf{X},y,k]}^{\lambda} \right\|_{\mathcal{H}} \leq 1$ . (III) For a kernel  $k$ , denote with  $\lambda_1 \geq \dots \geq \lambda_n$  the eigenvalues of the kernel matrix  $\mathbf{K}$ . We assume that the regularization parameter satisfies  $0 \leq n\lambda \leq \lambda_1$ .
The logic behind our assumptions. First, the idea behind Assumption (I) is that the pair  $(\mathbf{X},y)$  can be linked through some function that can be from either the same family of kernels that we support (i.e., shift-invariant) or any other kernel function. In the context of neural networks, the intuition behind Assumption (I) is that there exists a network from the desired architectures that gives a good approximation for the data. Assumption (II) aims to simplify the bounds used throughout the paper as it is a pretty standard assumption, characteristic to the analysis of random Fourier features [LTOS19, RR17]. Finally, Assumption (III) is to prevent underfitting. Specifically speaking, the largest eigenvalue of  $\mathbf{K}\left(\mathbf{K} + n\lambda \mathbf{I}_n\right)^{-1}$  is  $\frac{\lambda_1}{(\lambda_1 + n\lambda)}$ . Thus, in the case of  $n\lambda >\lambda_{1}$ , the in-sample prediction is dominated by the term  $n\lambda$ . Throughout the following analysis, we will use the above assumptions. Hence, for the sake of clarity, we will not repeat them, unless problem-specific clarifications are required.
Connection to Dataset distillation of neural networks. Since the neural network kernel in the case of infinite width networks describes a Gaussian distribution [JGH18], we aim at proving the existence of small sketches (distilled sets) for the input data with respect to the KRR problem with Gaussian kernel. However, the problem with this approach is that the feature space (in the Gaussian kernel corresponding mapping) is rather intangible or hard to map to, and sketch (distilled set) construction techniques require the representation of these points in the feature space.
To resolve this problem, we use a randomized approximated feature map, e.g., random Fourier features (RFF), and weighted random Fourier features (Weighted RFF). The dot product between every two mapped vectors in this approximated feature map aims to approximate their Gaussian kernel function [RR07]. We now restate a result connecting ridge regression in the RFF space (or alternatively weighted RFF), and KRR in the input space.
Theorem 2 (A result of the proof of Theorem 1 and Corollary 2 of [LTOS21]). Let  $\mathbf{X} \in \mathbb{R}^{n \times d}$  be an input matrix,  $y \in \mathbb{R}^n$  be an input label vector;  $k: \mathbb{R}^d \times \mathbb{R}^d \to [0, \infty)$  be a shift-invariant kernel function, and  $\mathbf{K} \in \mathbb{R}^{n \times n}$ , where  $\forall i, j \in [n]: \mathbf{K}_{i,j} = k(\mathbf{X}_{i*}, \mathbf{X}_{j*})$ . Let  $\lambda > 0$ , and let  $d_{\mathbf{K}}^{\lambda} = Tr\left(\mathbf{K}(\mathbf{K} + n\lambda \mathbf{I}_n)^{-1}\right)$ . Let  $s_\phi \in \Omega\left(d_{\mathbf{K}}^\lambda \log\left(d_{\mathbf{K}}^\lambda\right)\right)$  be a positive integer. Then, there exists a pair  $(\phi, \widetilde{\mathbf{X}})$  such that (i)  $\phi$  is a mapping  $\phi: \mathbb{R}^d \to \mathbb{R}^{s_\phi}$  (which is based on either the weighted RFF function or the RFF function [LTOS21]), (ii)  $\widetilde{\mathbf{X}}$  is a matrix  $\widetilde{\mathbf{X}} \in \mathbb{R}^{n \times s_\phi}$  where for every  $i \in [n]$ ,
$\widetilde{\mathbf{X}}_{i*} \coloneqq \phi(\mathbf{X}_{i*})$ , and (iii)  $(\phi, \widetilde{\mathbf{X}})$  satisfies
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \left| y _ {i} - f _ {[ \widetilde {\mathbf {X}}, y, \phi ]} ^ {\lambda} (\widetilde {\mathbf {X}} _ {i *}) \right| ^ {2} \leq \frac {1}{n} \sum_ {i = 1} ^ {n} \left| y _ {i} - f _ {[ \mathbf {X}, y, k ]} ^ {\lambda} (\mathbf {X} _ {i *}) \right| ^ {2} + 4 \lambda ,
$$
where  $f_{[\widetilde{\mathbf{X}},y,\phi]}^{\lambda}:\mathbb{R}^{s_{\phi}}\to \mathbb{R}$  such that for every row vector  $z\in \mathbb{R}^{s_{\phi}}$ ,  $f_{[\widetilde{\mathbf{X}},y,\phi]}^{\lambda}(z) = z\left(\widetilde{\mathbf{X}}^{T}\widetilde{\mathbf{X}} +\lambda ns_{\phi}\lambda \mathbf{I}_{s_{\phi}}\right)^{-1}\widetilde{\mathbf{X}}^{T}y$ . Note that, Table 1 gives bounds on  $s_{\phi}$  when  $\lambda \propto \frac{1}{\sqrt{n}}$ .
Intuition behind Theorem 2. Theorem 2 bounds the difference (additive approximation error) between (i) the MSE loss between the ground truth labels and the predictions obtained by applying Kernel Ridge regression (KRR) on the raw (original) data, and (ii) the MSE between the ground truth labels and the predictions obtained when applying Ridge regression on the mapped (full) training data via random Fourier features  $(RFF)$ . Theorem 2 will be utilized to set the minimal dimension of the  $RFF$  which yields the desired additive approximation, i.e.,  $4\lambda$ . Thus the intuition behind using this theorem is to link the dimension of the  $RFF$  with the size of the distilled set. In other words, we use this error bound and sufficient size (of the minimal dimension of the  $RFF$ ) to provide proof of the sufficient small size of the distilled set.
4 Main result: on the existence of small distilled sets
In what follows, we show that for any given matrix  $\mathbf{X} \in \mathbb{R}^{n \times d}$  and a label vector  $y \in \mathbb{R}^n$ , there exists a matrix  $\mathbf{S} \in \mathbb{R}^{s_\phi \times d}$  and a label vector  $y_{\mathbf{S}} \in \mathbb{R}^r$  such that the fitting solution in the RFF space mapping of  $\mathbf{S}$  is identical to that of the fitted solution on the RFF space mapping of  $\mathbf{X}$ . With such  $\mathbf{S}$  and  $y_{\mathbf{S}}$ , we proceed to provide our main result showing that one can construct a solution for KRR in the original space of  $\mathbf{S}$  which provably approximates the quality of the optimal KRR solution involving  $\mathbf{X}$  and  $y$ . Thus, we obtain bounds on the minimal distilled set size required for computing a robust approximation, as well as bounds on the error for such a distilled set.
We now provide Theorem 3 followed by its proof of the existence of a small distilled set. Then we provide extensive details and intuitive explanations about the steps of the proof.
Theorem 3 (On the existence of some distilled data). Let  $\mathbf{X} \in \mathbb{R}^{n \times d}$  be a matrix,  $y \in \mathbb{R}^n$  be a label vector,  $k: \mathbb{R}^d \times \mathbb{R}^d \to [0, \infty)$  be a kernel function,  $\Upsilon = (0,1) \cup \{2\}$ , and let  $s_\phi$  and  $\widetilde{\mathbf{X}}$  be defined as in Theorem 2. Assume that the rank of  $\widetilde{\mathbf{X}}$  is  $s_\phi$ , then, there exists a matrix  $\mathbf{S} \in \mathbb{R}^{s_\phi \times d}$  and a label vector  $y_{\mathbf{S}}$  such that
(i) the weighted RFF mapping  $\widetilde{\mathbf{S}}\in \mathbb{R}^{s_{\phi}\times s_{\phi}}$  of  $\mathbf{S}$ , satisfies that  $\left(\widetilde{\mathbf{X}}^T\widetilde{\mathbf{X}} +\lambda ns_\phi \lambda \mathbf{I}_{s_\phi}\right)^{-1}\widetilde{\mathbf{X}}^T y = \left(\widetilde{\mathbf{S}}^T\widetilde{\mathbf{S}} +\lambda ns_\phi \lambda \mathbf{I}_{s_\phi}\right)^{-1}\widetilde{\mathbf{S}}^T y_{\mathbf{S}}$ , and
(ii) there exists an in-sample prediction  $f_{[\mathbf{S},y_{\mathbf{S}},k]}^{\lambda ,\mathbf{X},y}$  (not necessarily the optimal on  $\mathbf{S}$  and  $y_{\mathbf{s}}$ ) satisfying
$$
\begin{array}{l} \frac {1}{n} \sum_ {i = 1} ^ {n} \left| f _ {[ \mathbf {X}, y, k ]} ^ {\lambda} \left(\mathbf {X} _ {i *}\right) - f _ {[ \mathbf {S}, y _ {\mathbf {S}}, k ]} ^ {\lambda , \mathbf {X}, y} \left(\mathbf {X} _ {i *}\right) \right| ^ {2} \leq \min  _ {\tau \in \Upsilon} \left(2 \max  \left\{\tau , \frac {4}{\tau^ {2}} \right\} + \right. \tag {3} \\ \left. 2 \min \left\{1 + \tau , \frac {4 (1 + \tau)}{3 \tau} \right\}\right) \lambda , \\ \end{array}
$$
$$
\begin{array}{l} \frac {1}{n} \sum_ {i = 1} ^ {n} \left| y _ {i} - f _ {[ \mathbf {S}, y \mathbf {S}, k ]} ^ {\lambda , \mathbf {X}, y} \left(\mathbf {X} _ {i *}\right) \right| ^ {2} \leq \min  _ {\tau \in \Upsilon} \frac {\min  \left\{1 + \tau , \frac {4 (1 + \tau)}{3 \tau} \right\}}{n} \sum_ {i = 1} ^ {n} \left| y _ {i} - f _ {[ \mathbf {X}, y, k ]} ^ {\lambda} \left(\mathbf {X} _ {i *}\right) \right| ^ {2} \\ + \left(4 \min  \left\{1 + \tau , \frac {4 (1 + \tau)}{3 \tau} \right\} + 2 \max  \left\{\tau , \frac {4}{\tau^ {2}} \right\}\right) \lambda . \tag {4} \\ \end{array}
$$
Proof. Let  $\mathbf{S}$  be any matrix in  $\mathbb{R}^{s_{\phi} \times d}$  such its weighted RFF mapping  $\widetilde{\mathbf{S}}$  is of rank equal to that of  $\widetilde{\mathbf{X}}$  and for every  $i \in [s_{\phi}]$ ,  $\sum_{j=1}^{n} k(\mathbf{S}_{i*}, \mathbf{X}_{i*}) \neq 0$ .
Proof of (i). To ensure (i), we need to find a corresponding proper  $y_{\mathbf{S}}$ . We observe that
$$
\left(\widetilde {\mathbf {S}} ^ {T} \widetilde {\mathbf {S}} + \lambda n s _ {\phi} \lambda \mathbf {I} _ {s _ {\phi}}\right) \left(\widetilde {\mathbf {X}} ^ {T} \widetilde {\mathbf {X}} + \lambda n s _ {\phi} \lambda \mathbf {I} _ {s _ {\phi}}\right) ^ {- 1} \widetilde {\mathbf {X}} ^ {T} y = \widetilde {\mathbf {S}} ^ {T} y \mathbf {S}
$$
Let  $b = \left(\widetilde{\mathbf{S}}^T\widetilde{\mathbf{S}} + \lambda ns_{\phi}\lambda \mathbf{I}_{s_{\phi}}\right)\left(\widetilde{\mathbf{X}}^T\widetilde{\mathbf{X}} + \lambda ns_{\phi}\lambda \mathbf{I}_{s_{\phi}}\right)^{-1}\widetilde{\mathbf{X}}^T y$ , be the left-hand side term above.  $b$  is a vector of dimension  $s_{\phi}$ . Hence we need to solve  $b = \widetilde{\mathbf{S}}^T y\mathbf{S}$  for  $y_{\mathbf{S}}$ . Since  $\mathbf{S}$  has full rank then we have a linear system involving  $s_{\phi}$  variables and  $s_{\phi}$  equations. Thus, the solution to such a system exists and is  $y_{\mathbf{S}} = \left(\widetilde{\mathbf{S}}^T\right)^{\dagger}b$ , where  $(\cdot)^{\dagger}$  denotes the pseudo-inverse of the given matrix.
Proof of (ii). Inspired by [LHAR22a] and [NCL20], the goal is to find a set of instances that their in-sample prediction with respect to the input data  $(\mathbf{X}$  in our context) would lead to an approximation towards the solution that one would achieve if the KRR was used only with the input data. To that end, we introduce the following Lemm.
Lemma 4 (Restatement of Lemma 6 [LTOS21]). Under Assumption 1 and the definitions in Theorem 2, for every  $f \in \mathcal{H}$  with  $\| f\|_{\mathcal{H}} \leq 1$ , with constant probability, it holds that
$$
\inf_{\substack{\sqrt{s_{\phi}}\| \beta \|_{2}\leq \sqrt{2}\\ \beta \in \mathbb{R}^{s_{\phi}}}}\sum_{i = 1}^{n}\frac{1}{n}\left|f\left(\mathbf{X}_{i*}\right) - \widetilde{\mathbf{X}}_{i*}\beta \right|^{2}\leq 2\lambda .
$$
Note that Lemma 4 shows that for every in-sample prediction function with respect to  $\mathbf{X}$ , there exists a query  $\beta \in \mathbb{R}^{s_{\phi}}$  in the RFF space of that input data such that the distance between the in-prediction sample function in the input space and the in-sample prediction in the RFF space is at  $2\lambda$ .
Furthermore, at [LTOS21] it was shown that  $\beta$  is defined as  $\beta = \frac{1}{s_{\phi}}\widetilde{\mathbf{X}}^T\left(\widetilde{\mathbf{X}}\widetilde{\mathbf{X}}^T + n\lambda \mathbf{I}_{s_{\phi}}\right)^{-1}\mathbf{f}[\mathbf{X}]$ , where  $\mathbf{f}[\mathbf{X}]_i = f(\mathbf{X}_{i*})$  for every  $i \in [n]$ .
We thus set out to find an in-sample prediction function that is defined over  $\mathbf{S}$  such that by its infimum by Lemma 4 would be the same solution  $\beta$  that the ridge regression on  $\widetilde{\mathbf{X}}$  attains with respect to the  $y$ . Specifically speaking, we want to find an in-sample prediction  $f_{[\mathbf{S}, y \mathrm{~s}, k]}^{(\lambda, \mathbf{X}, y)}(\cdot)$  such that
$$
\beta = \frac {1}{s _ {\phi}} \widetilde {\mathbf {X}} ^ {T} \left(\frac {1}{s _ {\phi}} \widetilde {\mathbf {X}} \widetilde {\mathbf {X}} ^ {T} + n \lambda \mathbf {I} _ {s _ {\phi}}\right) ^ {- 1} \mathbf {f} _ {\mathbf {S}} [ \mathbf {X} ], \tag {5}
$$
where (i)  $\mathbf{f}_{\mathbf{S}}[\mathbf{X}] \in \mathbb{R}^n$  such that for every  $i \in [n]$ ,  $\mathbf{f}_{\mathbf{S}}[\mathbf{X}]_i = f_{[\mathbf{S},y_{\mathbf{S}},k]}^{(\lambda,\mathbf{X},y)}(\mathbf{X}_{i*})$ , and (ii)  $f_{[\mathbf{S},y_{\mathbf{S}},k]}^{\lambda,\mathbf{X},y}(\cdot) = \sum_{i=1}^{s_\phi+1} \alpha_i k(\mathbf{S}_{i*},\cdot)$  such that  $\alpha \in \mathbb{R}^{s_\phi}$ .
Hence we need to find an in-sample prediction function  $f_{[\mathbf{S},y\mathbf{s},k]}^{\lambda ,\mathbf{X},y}$  satisfying 5. Now, notice that  $\beta \in \mathbb{R}^{s_{\phi}}$ ,  $\mathbf{f}_{\mathbf{S}}[\mathbf{X}] \in \mathbb{R}^n$  and  $\widetilde{\mathbf{X}}^T\left(\frac{1}{s_\phi}\widetilde{\mathbf{X}}\widetilde{\mathbf{X}}^T + n\lambda \mathbf{I}_n\right)^{-1} \in \mathbb{R}^{s_\phi \times n}$ . Due to the fact that we aim to find  $f_{[\mathbf{S},y\mathbf{s},k]}^{\lambda ,\mathbf{X},y}$ , such a task boils down to finding  $\alpha \in \mathbb{R}^{s_{\phi}}$  which defines  $f_{[\mathbf{S},y\mathbf{s},k]}^{\lambda ,\mathbf{X},y}$  as in (ii). The above problem can be reduced to a system of linear equations where the number of equalities is  $s_{\phi}$ , while the number of variables is  $s_{\phi}$ .
To do so, we denote  $\frac{1}{s_{\phi}}\widetilde{\mathbf{X}}^T\left(\frac{1}{s_{\phi}}\widetilde{\mathbf{X}}\widetilde{\mathbf{X}}^T + n\lambda \mathbf{I}_n\right)^{-1}$  by  $\hat{\mathbf{A}}$ , and observe that we aim to solve
$$
\beta = \hat {\mathbf {A}} f _ {\mathbf {S}} ^ {\lambda} [ \mathbf {X} ] = \hat {\mathbf {A}} \left[ \begin{array}{c} \sum_ {i = 1} ^ {s _ {\phi}} \alpha_ {i} k (\mathbf {S} _ {i *}, \mathbf {X} _ {1 *}) \\ \sum_ {i = 1} ^ {s _ {\phi}} \alpha_ {i} k (\mathbf {S} _ {i *}, \mathbf {X} _ {2 *}) \\ \vdots \\ \sum_ {i = 1} ^ {s _ {\phi}} \alpha_ {i} k (\mathbf {S} _ {i *}, \mathbf {X} _ {n *}) \end{array} \right].
$$