SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We now show that every entry  $b_{j}$  ( $j \in [s_{\phi}]$ ) in  $\beta$  can be rewritten as inner products between another pair of vectors in  $\mathbb{R}^{s_{\phi}}$  instead of the inner product between two vectors in  $\mathbb{R}^n$ . Formally, for every  $j \in [s_{\phi}]$ , it holds that
$$
\beta_ {j} = \hat {\mathbf {A}} _ {j *} \left[ \begin{array}{c} \sum_ {i = 1} ^ {s _ {\phi}} \alpha_ {i} k (\mathbf {S} _ {i *}, \mathbf {X} _ {1 *}) \\ \sum_ {i = 1} ^ {s _ {\phi}} \alpha_ {i} k (\mathbf {S} _ {i *}, \mathbf {X} _ {2 *}) \\ \vdots \\ \sum_ {i = 1} ^ {s _ {\phi}} \alpha_ {i} k (\mathbf {S} _ {i *}, \mathbf {X} _ {n *}) \end{array} \right] = \left[ \sum_ {t = 1} ^ {n} \hat {\mathbf {A}} _ {j, t} k (\mathbf {S} _ {1 *}, \mathbf {X} _ {t *}), \dots , \sum_ {t = 1} ^ {n} \hat {\mathbf {A}} _ {j, t} k (\mathbf {S} _ {(s _ {\phi}) *}, \mathbf {X} _ {t *}) \right] \left[ \begin{array}{c} \alpha_ {1} \\ \vdots \\ \alpha_ {s _ {\phi}} \end{array} \right].
$$
Thus, for every  $j \in [s_{\phi}]$ , define  $\mathbf{A}_{j*} = \left[\sum_{t=1}^{n} \hat{\mathbf{A}}_{j,t} k\left(\mathbf{S}_{1*}, \mathbf{X}_{t*}\right), \dots, \sum_{t=1}^{n} \hat{\mathbf{A}}_{j,t} k\left(\mathbf{S}_{s_{\phi}*}, \mathbf{X}_{t*}\right)\right] \in \mathbb{R}^{s_{\phi}}$ . In other words,  $A$  is a result of a Hadamard multiplication of  $\hat{A}$  and a 1-rank matrix  $G$  such that each of its rows is equal to  $\left[\sum_{t=1}^{n} k\left(\mathbf{S}_{1*}, \mathbf{X}_{t*}\right), \dots, \sum_{t=1}^{n} k\left(\mathbf{S}_{s_{\phi}*}, \mathbf{X}_{t*}\right)\right]$ .
Since the rank of  $\hat{A}$  is full, i.e.,  $\mathrm{rank}(A) = \mathrm{rank}\left(\widetilde{\mathbf{X}}^T\left(\frac{1}{s_\phi}\widetilde{\mathbf{X}}\widetilde{\mathbf{X}}^T + n\lambda \mathbf{I}_n\right)^{-1}\right) = \mathrm{rank}\left(\widetilde{\mathbf{X}}\right) = s_\phi$  by assumption, then it holds that
$$
\operatorname {r a n k} (A) = \operatorname {r a n k} \left(\hat {A} \circ G\right) = \operatorname {r a n k} \left(D _ {v} \hat {A} D _ {u}\right) = \operatorname {r a n k} \left(\hat {A}\right),
$$
where the first equality holds by definition of  $A$  and  $\circ$  denoting the Hadamard multiplication product, the second inequality holds since by construction of  $\mathbf{S}$  and  $G = uv^{T}$  such that  $u, v \in \mathbb{R}^{s_{\phi}}$  are vector with non-zero entries, and  $D_{u}, D_{v} \in \mathbb{R}^{s_{\phi} \times s_{\phi}}$  are diagonal matrices where their diagonal are  $u$  and  $v$  respectively. The last inequality holds by property of rank function, i.e., the rank of any product of pair of square matrices  $C$  and  $D$  such that  $D$  is of full rank is equal to the rank of  $C$ .
The right-hand side of (5) can reformulated as
$$
\frac {1}{s _ {\phi}} \widetilde {\mathbf {X}} ^ {T} \left(\frac {1}{s _ {\phi}} \widetilde {\mathbf {X}} \widetilde {\mathbf {X}} ^ {T} + n \lambda \mathbf {I} _ {n}\right) ^ {- 1} \mathbf {f} _ {\mathbf {S}} [ \mathbf {X} ] = \mathbf {A} \alpha , \tag {6}
$$
where now we only need to solve  $\beta = \mathbf{A}\alpha$ . Such a linear system of equations has a solution since we have  $s_{\phi}$  variables (the length of  $\alpha$ ) and  $s_{\phi}$  equations and the rank  $A$  is equal to  $s_{\phi}$ . For simplicity, a solution to the above equality would be  $\alpha := (\mathbf{A})^\dagger \beta$ . To proceed in proving (ii) with all of the above ingredients, we utilize the following tool.
Lemma 5 (Special case of Definition 6.1 from  $[\mathrm{BFL}^{+}16])$ . Let  $X$  be a set, and let  $\left(X,\| \cdot \| _2^2\right)$  be a 2-metric space i.e., for every  $x,y,z\in X$ ,  $\| x - y\| _2^2\leq 2\left(\| x - z\| _2^2 +\| y - z\| _2^2\right)$ . Then, for every  $\varepsilon \in (0,1)$ , and  $x,y,z\in X$ ,
$$
(1 - \varepsilon) \| y - z \| _ {2} ^ {2} - \frac {4}{\varepsilon^ {2}} \| x - z \| _ {2} ^ {2} \leq \| x - y \| _ {2} ^ {2} \leq \frac {4}{\varepsilon^ {2}} \| x - z \| _ {2} ^ {2} + (1 + \varepsilon) \| y - z \| _ {2} ^ {2}. \tag {7}
$$
We note that Lemma 5 implies that  $x,y,z\in \mathbb{R}^d$
$$
\left\| x - y \right\| _ {2} ^ {2} \leq \min  _ {\tau \in \Upsilon} \max  \left\{\tau , \frac {4}{\tau^ {2}} \right\} \| x - z \| _ {2} ^ {2} + \min  \left\{1 + \tau , \frac {4 (1 + \tau)}{3 \tau} \right\} \| y - z \| _ {2} ^ {2}. \tag {8}
$$
where for  $\tau = 2$  we get the inequality associated with the property of 2-metric, and for any  $\tau \in (0,1)$  we obtain the inequality (7).
We thus observe that
$$
\begin{array}{l} \frac {1}{n} \sum_ {i = 1} ^ {n} \left| f _ {[ \mathbf {X}, y, k ]} ^ {\lambda} (\mathbf {X} _ {i *}) - f _ {[ \mathbf {S}, y _ {\mathbf {S}}, k ]} ^ {\lambda , \mathbf {X}, y} (\mathbf {X} _ {i *}) \right| ^ {2} \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \left| f _ {[ \mathbf {X}, y, k ]} ^ {\lambda} (\mathbf {X} _ {i *}) - f _ {[ \widetilde {\mathbf {X}}, y, \phi ]} ^ {\lambda} (\widetilde {\mathbf {X}} _ {i *}) + f _ {[ \widetilde {\mathbf {X}}, y, \phi ]} ^ {\lambda} (\widetilde {\mathbf {X}} _ {i *}) - f _ {[ \mathbf {S}, y _ {\mathbf {S}}, k ]} ^ {\lambda , \mathbf {X}, y} (\mathbf {X} _ {i *}) \right| ^ {2} \\ \leq \min _ {\tau \in \Upsilon} \frac {\max  \left\{\tau , \frac {4}{\tau^ {2}} \right\}}{n} \sum_ {i = 1} ^ {n} \left| f _ {[ \mathbf {X}, y, k ]} ^ {\lambda} \left(\mathbf {X} _ {i *}\right) - f _ {[ \widetilde {\mathbf {X}}, y, \phi ]} ^ {\lambda} \left(\widetilde {\mathbf {X}} _ {i *}\right) \right| ^ {2} + \\ \frac {\operatorname * {m i n} \left\{1 + \tau , \frac {4 (1 + \tau)}{3 \tau} \right\}}{n} \sum_ {i = 1} ^ {n} \left| f _ {[ \widetilde {\mathbf {X}}, y, \phi ]} ^ {\lambda} (\widetilde {\mathbf {X}} _ {i *}) - f _ {[ \mathbf {S}, y _ {\mathbf {S}}, k ]} ^ {\lambda , \mathbf {X}, y} (\mathbf {X} _ {i *}) \right| ^ {2} \\ \leq \min  _ {\tau \in \Upsilon} 2 \max  \left\{\tau , \frac {4}{\tau^ {2}} \right\} \lambda + 2 \min  \left\{1 + \tau , \frac {4 (1 + \tau)}{3 \tau} \right\} \lambda \\ = \min _ {\tau \in \Upsilon} \left(2 \max \left\{\tau , \frac {4}{\tau^ {2}} \right\} + 2 \min \left\{1 + \tau , \frac {4 (1 + \tau)}{3 \tau} \right\}\right) \lambda , \\ \end{array}
$$
where the first equality holds by adding and subtracting the same term, the first inequality holds by Lemma 5, and the second inequality holds by combining the way  $f_{[\mathbf{S},y\mathbf{s},k]}^{\lambda ,\mathbf{X},y}$  was defined and Theorem 2. Finally, to conclude the proof of Theorem 3, we derive 4
$$
\begin{array}{l} \frac {1}{n} \sum_ {i = 1} ^ {n} \left| y _ {i} - f _ {[ \mathbf {S}, y _ {\mathbf {S}}, k ]} ^ {\lambda , \mathbf {X}, y} \left(\mathbf {X} _ {i *}\right) \right| ^ {2} = \frac {1}{n} \sum_ {i = 1} ^ {n} \left| y _ {i} - f _ {[ \widetilde {\mathbf {X}}, y, \phi ]} ^ {\lambda} \left(\widetilde {\mathbf {X}} _ {i *}\right) + f _ {[ \widetilde {\mathbf {X}}, y, \phi ]} ^ {\lambda} \left(\widetilde {\mathbf {X}} _ {i *}\right) - f _ {[ \mathbf {S}, y _ {\mathbf {S}}, k ]} ^ {\lambda , \mathbf {X}, y} \left(\mathbf {X} _ {i *}\right) \right| ^ {2} \\ \leq \min  _ {\tau \in \Upsilon} \frac {\min  \left\{1 + \tau , \frac {4 (1 + \tau)}{3 \tau} \right\}}{n} \sum_ {i = 1} ^ {n} \left| y _ {i} - f _ {[ \widetilde {\mathbf {X}}, y, \phi ]} ^ {\lambda} \left(\widetilde {\mathbf {X}} _ {i *}\right) \right| ^ {2} \\ + \frac {\operatorname* {m a x} \left\{\tau , \frac {4}{\tau^ {2}} \right\}}{n} \sum_ {i = 1} ^ {n} \left| f _ {[ \widetilde {\mathbf {X}}, y, \phi ]} ^ {\lambda} \left(\widetilde {\mathbf {X}} _ {i *}\right) - f _ {[ \mathbf {S}, y \mathbf {S}, k ]} ^ {\lambda , \mathbf {X}, y} \left(\mathbf {X} _ {i *}\right) \right| ^ {2} \\ \leq \min  _ {\tau \in \Upsilon} \frac {\min  \left\{1 + \tau , \frac {4 (1 + \tau)}{3 \tau} \right\}}{n} \sum_ {i = 1} ^ {n} \left| y _ {i} - f _ {[ \widetilde {\mathbf {X}}, y, \phi ]} ^ {\lambda} (\widetilde {\mathbf {X}} _ {i *}) \right| ^ {2} + 2 \max  \left\{\tau , \frac {4}{\tau^ {2}} \right\} \lambda \\ \leq \min  _ {\tau \in \Upsilon} \frac {\min  \left\{1 + \tau , \frac {4 (1 + \tau)}{3 \tau} \right\}}{n} \sum_ {i = 1} ^ {n} \left| y _ {i} - f _ {[ \mathbf {X}, y, k ]} ^ {\lambda} \left(\mathbf {X} _ {i *}\right) \right| ^ {2} \\ + \left(4 \min  \left\{1 + \tau , \frac {4 (1 + \tau)}{3 \tau} \right\} + 2 \max  \left\{\tau , \frac {4}{\tau^ {2}} \right\}\right) \lambda , \tag {9} \\ \end{array}
$$
where the equality holds by adding and subtracting the same term, the first inequality holds by (8), and the second inequality follows as a result of the way  $f_{\mathbf{S}}^{\lambda}$  was constructed and the fact that  $\beta$  is its infimum based on Lemma 4, and the last inequality holds by Theorem 2.
Intuition behind Theorem 3. The goal of Theorem 3 is to prove the existence of a small distilled set  $\mathbf{S}$  (its size is a function of the minimal dimension of the RFF mapping required to ensure the provable additive approximation stated in Theorem 2) satisfying that: (i) The Ridge regression model trained on the mapped training data via RFF is identical to that of the Ridge regression model trained on the mapped small distilled set via RFF, (ii) more importantly there exists a KRR solution formulated for  $\mathbf{S}$  with respect to the loss of the whole big data  $\mathbf{X}$ , which approximates the KRR solution on the whole data  $\mathbf{X}$  (which is the goal of KRR-based dataset distillation techniques). Thus, (iii) we derive bounds on the difference (approximation error) between (1) The MSE between the ground truth labels of the full data and their corresponding predictions obtained by the specific KRR model (we previously described) on our distilled set and (2) The MSE between the ground truth labels and the predictions obtained when applying KRR on the whole data  $\mathbf{X}$ .
The heart of our approach lies in connecting the minimal dimension of the RFF required for provable additive approximation and the size of the distilled set. This is first done by showing that the distilled set can be any set  $S$  of instances from the input space (e.g., images) and their corresponding labels, as long as the corresponding labels must maintain a certain property. Specifically speaking, the labels of the distilled set need to be in correlation with the normal of the best hyperplane found to fit the mapped training data via RFF  $\widetilde{\mathbf{X}}$  via the Ridge regression model trained on  $\left(\widetilde{\mathbf{X}}, y\right)$ ,
i.e.,  $\left(\widetilde{\mathbf{S}}^T\widetilde{\mathbf{S}} +\lambda ns_\phi \lambda I_{s_\phi}\right)\left(\widetilde{\mathbf{X}}^T\widetilde{\mathbf{X}} +\lambda ns_\phi \lambda I_{s_\phi}\right)^{-1}\widetilde{\mathbf{X}}^T y = \widetilde{\mathbf{S}}^T y\mathbf{s}$ . From here, the idea hinges upon showing the existence of a KRR model (represented by a prediction function) that would be dependent on the prediction function that can be obtained from applying the Ridge regression problem to the mapped full training data via RFF. With such a model, the idea is to retrieve the predictions obtained when using the Ridge regression problem from the mapped training data via RFF via the use of some KRR model used on the distilled set. We thus show that through careful mathematical derivations, equation reformulation (involving), and solving a system of equations, one is able to show the existence of a KRR solution that would allow us to use Theorem 2. Finally, to obtain our bounds, we also rely on the use of the weak triangle inequality. To that end, we now utilize the described KRR model on the distilled data together with Theorem 2 to achieve (iii).
Remark 6. Note that the assumption that  $\widetilde{\mathbf{X}}$  is of full rank (i.e.,  $s_{\phi}$ ) can be dropped easily from Theorem 3, and as a result, we obtain that  $S$  can contain  $r$  (rank of  $\widetilde{\mathbf{X}}$ ) instances (rows of  $\mathbf{S}$ ). For additional details, please refer to Section  $E$  in the Appendix.
To simplify the bounds stated at Theorem 3, we provide the following remark.
Remark 7. By fixing  $\tau \coloneqq 2$ , the bounds in Theorem 3 become
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \left| f _ {[ \mathbf {X}, y, k ]} ^ {\lambda} \left(\mathbf {X} _ {i *}\right) - f _ {[ \mathbf {S}, y _ {\mathbf {S}}, k ]} ^ {\lambda , \mathbf {X}, y} \left(\mathbf {X} _ {i *}\right) \right| ^ {2} \leq 8 \lambda ,
$$
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \left| y _ {i} - f _ {[ \mathbf {S}, y _ {\mathbf {S}}, k ]} ^ {\lambda , \mathbf {X}, y} \left(\mathbf {X} _ {i *}\right) \right| ^ {2} \leq \frac {2}{n} \sum_ {i = 1} ^ {n} \left| y _ {i} - f _ {[ \mathbf {X}, y, k ]} ^ {\lambda} \left(\mathbf {X} _ {i *}\right) \right| ^ {2} + 1 2 \lambda .
$$
As for fixing  $\tau \coloneqq \varepsilon \in (0,1)$ , we obtain that
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \left| y _ {i} - f _ {[ \mathbf {S}, y _ {\mathbf {S}}, k ]} ^ {\lambda , \mathbf {X}, y} (\mathbf {X} _ {i *}) \right| ^ {2} \leq \frac {1 + \varepsilon}{n} \sum_ {i = 1} ^ {n} \left| y _ {i} - f _ {[ \mathbf {X}, y, k ]} ^ {\lambda} (\mathbf {X} _ {i *}) \right| ^ {2} + \left(4 (1 + \varepsilon) + \frac {8}{\varepsilon^ {2}}\right) \lambda .
$$
5 Experimental Study
To validate our theoretical bounds, we performed distillation on three datasets: two synthetic datasets consisting of data generated from a Gaussian Random Field, and classification of two clusters, and one real dataset of MNIST binary and multi-class classification. Full experimental details for all experiments are available in the appendix.
2d Gaussian Random Fields. We first test our bounds by distilling data generated from the Gaussian Process prior induced by a kernel,  $k$  on 2d data. We use a squared exponential kernel with lengthscale parameter  $l = 1.5$ :  $k(x,x^{\prime}) = e^{-\frac{||x - x^{\prime}||_2^2}{2l^2}}$ . For  $\mathbf{X}$ , we sample  $n = 10^{5}$  datapoints from  $\mathcal{N}(0,\sigma_x^2)$  with  $\sigma_{x}\in [0.25,5.0]$ . We then sample  $y\sim \mathcal{N}(0,K_{XX} + \sigma_y^2 I_n)$ ,  $\sigma_{y} = 0.01$ . We fix  $\lambda = 10^{-5}$  and distill down to  $s = d_k^\lambda \log d_k^\lambda$ . The resulting values of  $d_k^\lambda$ ,  $s$ , and compression ratios are plotted in fig. 2. We additionally plot the predicted upper bound given by Remark 7 and the actual distillation loss. Our predicted upper bound accurately bounds the actual distillation loss. To better visualize how distillation affects the resulting KRR prediction, we show the KRR predictive function  $f_{\mathbf{X}}^{\lambda}$  and the distilled predictive  $f_{\mathbf{S}}^{\lambda}$  for  $\sigma_{x} = 5.0$  in fig. 1b and fig. 1c.
Two Gaussian Clusters Classification. Our second synthetic dataset is one consisting of two Gaussian clusters centered at  $(-2,0)$  and  $(2,0)$ , with labels  $-1$  and  $+1$ , respectively. Each cluster contains 5000 datapoints so that  $n = 10^5$ . Each cluster as standard deviation  $\sigma_x \in [0.25, 5.0]$ . Additionally, two allow the dataset to be easily classified, we clip the  $x$  coordinates of clusters 1 and
(c)
Figure 1: (a) visualizes the two clusters dataset with  $n = 300$  and  $\sigma_{x} = 1.5$ . (b) and (c) visualizing the KRR predictive functions generated by the original dataset (b) and the distilled dataset (c) for the Gaussian Random Field experiment for  $\sigma_{x} = 5.0$ . The distilled dataset is able to capture all the nuances of the original dataset with a fraction of the datapoints.
Figure 2: Distillation results for synthetic data generated by a Gaussian Random Field  $(n = 3)$
Gaussian Random Field
clusters 2 to not exceed/drop below  $-0.4$  and  $0.4$ , for the two clusters, respectively. This results in a margin between the two classes. We visualize the dataset for  $n = 300$  and  $\sigma = 1.5$  in fig. 1a. We use the same squared exponential kernel as before with  $l = 1.5$ , fix  $\lambda = 10^{-5}$ , and distill with the same protocol as before. We likewise plot  $d_k^\lambda$ ,  $s$ , and compression ratios and distillation losses in fig. 3, again with our bound accurately containing the true distillation loss.
Figure 3: Distillation results for synthetic data of two Gaussian clusters  $(n = 3)$
Two Clusters
Real Word Datasets Classification. For our next test, we first consider binary classification on (i) MNIST 0 and 1 digits, (ii) SVHN 0 and 1 digit, and (iii) CIFAR-10 ship vs deer; all with labels  $-1$  and  $+1$ , respectively. We use the same squared-exponential kernel with  $l = 13.9$ ,  $l = 3.0$ , and  $l = 8.0$ , for MNIST, SVHN and CIFAR-10, respectively, which was chosen to maximize the marginal-log-likelihood, treating the problem as Gaussian Process regression. We vary  $n \in [500, 10000]$ , with an equal class split, and perform the same distillation protocol. Here, we additionally scale  $\lambda \propto \frac{1}{\sqrt{n}}$  such that  $\lambda = 10^{-4}$  when  $n = 5000$ . Distilling yields fig. 4, fig. 5, and fig. 6, showing that our bounds can accurately predict distillation losses for real-world datasets. In fig. 7 and fig. 8 in the appendix, we test our bound for the multi-class case on MNIST 0,1,2 digits with similar settings as in the binary classification case. The results justify our bounds.
Figure 4: Distillation results for MNIST binary 0 vs. 1 classification  $(n = 3)$
MNIST 0 vs. 1 Classification
Figure 5: Distillation results for SVHN binary 0 vs. 1 classification  $(n = 3)$
SVHN 0/1 Classification Optimizing  $8\lambda$
Figure 6: Distillation results for CIFAR-10 binary deer vs. ship classification  $(n = 3)$
CIFAR-10 Deer vs. Ship Classification
In this study, we adopt a theoretical perspective to provide bounds on the (sufficient) size and approximation error of distilled datasets. By leveraging the concept of random Fourier features (RFF), we prove the existence of small distilled datasets and we bound their corresponding excess risk when using shift-invariant kernels. Our findings indicate that the size of the guaranteed distilled data is a function of the "number of effective degrees of freedom," which relies on factors like the kernel, the number of points, and the chosen regularization parameter,  $\lambda$ , which also controls the excess risk. In particular, we demonstrate the existence of a small subset of instances within the original input space, where the solution in the RFF space coincides with the solution found using the input data in the RFF space. Subsequently, we show that this distilled subset of instances can be utilized to generate a KRR solution that approximates the KRR solution obtained from the complete input data. To validate these findings, we conducted empirical examinations on both synthetic and real-world datasets supporting our claim. While this study provides a vital first step in understanding the theoretical limitations of dataset distillation, the proposed bounds are not tight, as seen by the gap between the theoretical upper bound and the empirical distillation loss in section 5. Future work could look at closing this gap, as well as better understanding the tradeoff between distillation size and relative error.
This research has been funded in part by the Office of Naval Research Grant Number Grant N00014-18-1-2830, DSTA Singapore, and the J. P. Morgan AI Research program.
$\left[\mathrm{AKM}^{+}17\right]$  Haim Avron, Michael Kapralov, Cameron Musco, Christopher Musco, Ameya Velingker, and Amir Zandieh. Random fourier features for kernel ridge regression:
[SRL+19] Felipe Petroski Such, Aditya Rawal, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Generative teaching networks: Accelerating neural architecture search by learning to generate synthetic training data. CoRR, abs/1912.07768, 2019.
[SS15] Danica J Sutherland and Jeff Schneider. On the error of random fourier features. arXiv preprint arXiv:1506.02785, 2015.
[TMF20] Murad Tukan, Alaa Maalouf, and Dan Feldman. Coresets for near-convex functions. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2020.
[TRB16] Dustin Tran, Rajesh Ranganath, and David M. Blei. The variational gaussian process, 2016.
[TZM+23] Murad Tukan, Samson Zhou, Alaa Maalouf, Daniela Rus, Vladimir Braverman, and Dan Feldman. Provable data subset selection for efficient neural network training. arXiv preprint arXiv:2303.05151, 2023.
[WZTE18] Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811.10959, 2018.
[ZB21] Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning, 2021.
[ZMB21] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In International Conference on Learning Representations, 2021.
[ZNB22] Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regression. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2022.
All experiments unless otherwise stated present the average/standard deviations of  $n = 3$  runs. Each run consists of a random subset of MNIST 0/1 digits for MNIST binary classification, or random positions of sampled datapoints for synthetic data, and different samples from the GP for the Gaussian Random Field experiment. Distilled datasets are initialized as subsets of the original training data. We distill for 20000 iterations with Adam optimizer with a learning rate of 0.002 optimizing both images/data positions and labels. We use full batch gradient descent for the synthetic datasets and a maximum batch size of 2000 for the MNIST experiment. For the MNIST experiment we found that particularly for larger values of  $n$ , with minibatch training, we could obtain lower distillation losses by optimizing for longer, so the closing of the gap between the upper bound and experiment values in fig. 4 may be misleading: longer optimization could bring the actual distillation loss lower.
To ensure that assumption (II) is fulfilled, we scale the labels such that  $\left\| f_{[\mathbf{X},y,k]}^{\lambda}\right\|_{\mathcal{H}} = 1$ . For example, if we are working with MNIST binary classification, with labels  $\{+1, -1\}$ , we first compute  $\left\| f_{[\mathbf{X},y,k]}^{\lambda}\right\|_{\mathcal{H}} = r$  using  $\{+1, -1\}$  labels, then rescale the labels by  $1/r$  so that the labels are  $\{+\frac{1}{r}, -\frac{1}{r}\}$ . Suppose this results in some upper bound  $\mathcal{L}_U$  and some real distillation loss  $\mathcal{L}_R$ . For the corresponding plots in figs. 2 to 4, we plot  $r^2 \mathcal{L}_U$  and  $r^2 \mathcal{L}_R$ . We do this because the  $r$  values for different parameters (such as  $n$  or  $\sigma_x$ ) could be different, and scaling for the plots allows the values to be comparable.
In the figures for the upper bounds on the distillation loss we plot the smallest value of the upper bounds in remark 7.
A.1 Code
Code is available in the supplementary material.
B Proof of Technical Results
B.1 Proof of Theorem 2
Note that the proof of Theorem 2 follows from the proofs of Theorem 1 and Corollary 1 of [LTOS21]. For completeness, we provide the following proof.
Theorem 2 (A result of the proof of Theorem 1 and Corollary 2 of [LTOS21]). Let  $\mathbf{X} \in \mathbb{R}^{n \times d}$  be an input matrix,  $y \in \mathbb{R}^n$  be an input label vector,  $k: \mathbb{R}^d \times \mathbb{R}^d \to [0, \infty)$  be a shift-invariant kernel function, and  $\mathbf{K} \in \mathbb{R}^{n \times n}$ , where  $\forall i, j \in [n]: \mathbf{K}_{i,j} = k(\mathbf{X}_{i*}, \mathbf{X}_{j*})$ . Let  $\lambda > 0$ , and let  $d_{\mathbf{K}}^{\lambda} = Tr\left(\mathbf{K}(\mathbf{K} + n\lambda \mathbf{I}_n)^{-1}\right)$ . Let  $s_\phi \in \Omega\left(d_{\mathbf{K}}^{\lambda}\log\left(d_{\mathbf{K}}^{\lambda}\right)\right)$  be a positive integer. Then, there exists a pair  $(\phi, \widetilde{\mathbf{X}})$  such that (i)  $\phi$  is a mapping  $\phi: \mathbb{R}^d \to \mathbb{R}^{s_\phi}$  (which is based on either the weighted RFF function or the RFF function [LTOS21]), (ii)  $\widetilde{\mathbf{X}}$  is a matrix  $\widetilde{\mathbf{X}} \in \mathbb{R}^{n \times s_\phi}$  where for every  $i \in [n]$ ,  $\widetilde{\mathbf{X}}_{i*} := \phi(\mathbf{X}_{i*})$ , and (iii)  $(\phi, \widetilde{\mathbf{X}})$  satisfies