Title: On the Size and Approximation Error of Distilled Datasets
Venue: NeurIPS 2023
Accepted: yes
Keywords: Dataset Distillation

Size and Approximation Error
Abstract: Dataset Distillation is the task of synthesizing small datasets from large ones while still retaining comparable predictive accuracy to the original uncompressed dataset. Despite significant empirical progress in recent years, there is little understanding of the theoretical limitations/guarantees of dataset distillation, specifically, what excess risk is achieved by distillation compared to the original dataset, and how large are distilled datasets? In this work, we take a theoretical view on kernel ridge regression (KRR) based methods of dataset distillation such as Kernel Inducing Points. By transforming ridge regression in random Fourier features (RFF) space, we provide the first proof of the existence of small (size) distilled datasets and their corresponding excess risk for shift-invariant kernels. We prove that a small set of instances exists in the original input space such that its solution in the RFF space coincides with the solution of the original data. We further show that a KRR solution can be generated using this distilled set of instances which gives an approximation towards the KRR solution optimized on the full input data. The size of this set is linear in the dimension of the RFF space of the input set or alternatively near linear in the number of effective degrees of freedom, which is a function of the kernel, number of data points, and the regularization parameter $\lambda$. The error bound of this distilled set is also a function of $\lambda$.  We verify our bounds analytically and empirically.

On the Size and Approximation Error of Distilled Sets

Abstract

Dataset Distillation is the task of synthesizing small datasets from large ones while still retaining comparable predictive accuracy to the original uncompressed dataset. Despite significant empirical progress in recent years, there is little understanding of the theoretical limitations/guarantees of dataset distillation, specifically, what excess risk is achieved by distillation compared to the original dataset, and how large are distilled datasets? In this work, we take a theoretical view on kernel ridge regression (KRR) based methods of dataset distillation such as Kernel Inducing Points. By transforming ridge regression in random Fourier features (RFF) space, we provide the first proof of the existence of small (size) distilled datasets and their corresponding excess risk for shift-invariant kernels. We prove that a small set of instances exists in the original input space such that its solution in the RFF space coincides with the solution of the original data. We further show that a KRR solution can be generated using this distilled set of instances which gives an approximation towards the KRR solution optimized on the full input data. The size of this set is linear in the dimension of the RFF space of the input set or alternatively near linear in the number of effective degrees of freedom, which is a function of the kernel, number of datapoints, and the regularization parameter λ . The error bound of this distilled set is also a function of λ . We verify our bounds analytically and empirically.

1 Introduction

Motivated by the growing data demands of modern deep learning, dataset distillation [ZB21, NNXL21, ZNB22, WZTE18] aims to summarize large datasets into significantly smaller synthetic distilled datasets, which when trained on retain high predictive accuracy, comparable to the original dataset. These distilled datasets have applications in continual learning [ZNB22, SCCB22], architecture search [SRL + 19], and privacy preservation [CKF22]. Recent years have seen the development of numerous distillation algorithms, but despite this progress, the field has remained largely empirical. Specifically, there is little understanding of what makes one dataset 'easier to distill" than another, or whether such small synthetic datasets even exist.

This work aims to fill this gap by providing the first bounds on the sufficient size and relative error associated with distilled datasets. Noting prior work relating neural network training to kernel ridge regression (KRR), we consider dataset distillation in the kernel ridge regression settings with shift-invariant kernels. By casting the problem into the Random Fourier Feature (RFF) space, we show that: The size and relative error of distilled datasets is governed by the kernel's 'number of effective degrees of freedom", d λ k . Specifically, in Section 4, we show that distilled sets of size Ω( d λ k log d λ k ) , exist, with 12 λ +2 L λ predictive error on the training dataset, and only 8 λ error with

respect to the optimal solution computed on the full dataset, where λ is the kernel ridge regression regularization parameter and L λ the KRR training error on the original dataset; see Theorem 3 and Remark 7 for full details.

These bounds hold in practice for both real and synthetic datasets . In section 5, we validate our theorem by distilling synthetic and real datasets with varying sizes and values of d λ k , showing that in all scenarios our bounds accurately predict the error associated with distillation.

2 Related work

Coresets. Coresets are weighted selections from a larger training dataset, which, when used for training, yield similar outcomes as if the whole dataset was used [MSSW18, MBL20, PDM22, MEM + 22]. The key benefit of using coresets is that they significantly speed up the training process, unlike when the full data set is used. Current coresets methods incorporate clustering techniques [FL11, LBK16, BLHK16], bilevel optimization [BMK20], sensitivity analysis [MSSW18, HCB16, TMF20, MSF20], and surrogate models for approximation [TZM + 23]. Newer strategies are specifically designed for neural networks, where before each training epoch, coresets are chosen such that their gradients align with the gradients of the entire dataset [MBL20, PDM22, TZM + 23], followed by training the model on the chosen coreset. Although coresets are usually theoretically supported, these methods fall short when the aim is to compute a coreset once for a full training procedure.

Dataset Distillation. To this end, dataset distillation algorithms construct synthetic datasets (not necessarily a subset from the original input) such that gradient descent training on the synthetic datapoints results in high predictive accuracy on the real dataset. Cast as a bilevel optimization problem, early methods involve unrolling training computation graph [WZTE18] for a few gradient descent steps and randomly sampled weight initializations. More sophisticated methods aim to approximate the unrolled computation using kernel methods [NCL21, NNXL21, ZNB22, LHAR22a, LHLR23a], surrogate objectives such gradient matching [ZMB21, ZB21], trajectory matching [CWT + 22] or implicit gradients [LHLR23b]. The kernel-induced points (KIP) algorithm [NCL21, NNXL21] is a technique that employs Neural Tangent Kernel (NTK) theory[JGH18, LHAR22b] to formulate the ensuing loss: L KIP = 1 2 ∥ y t -K TS K -1 SS y S ∥ 2 2 . This loss signifies the predictive loss of training infinitely wide networks on distilled datapoints X S with corresponding labels y S , on the original training set and labels X T , y T , with K · , · being the NTK. Dataset distillation is closely related to the use of inducing points to accelerate Gaussian Processes [SG05, TRB16], for which convergence rates exist, but the existence of such inducing points is not unknown [BRVDW19].

From dataset distillation to kernel ridge regression. Kernel ridge regression (KRR) extends the linear machine learning ridge regression model by using a kernel function to map input data into higher-dimensional feature spaces, allowing for more complex non-linear relationships between variables to be captured [Mur12]. Various methods have been proposed to improve and accelerate the training process of kernel ridge regression. Most notably, Random Fourier Features [RR07] approximates shift-invariant kernel functions by mapping the input data into a lower-dimensional feature space using a randomized cosine transformation. This has been shown to work effectively in practice due to regularizing effects [JSS + 20], as well as providing approximation bounds to the full kernel ridge regression [SS15, AKM + 17, LTOS19]. Training infinite-width neural networks can be cast as kernel ridge regression with the Neural Tangent Kernel (NTK) [JGH18], which allows a closed-form solution of the infinite-width neural network's predictions, enabling kernel-based dataset distillation algorithms such as [NCL21, NNXL21, LHAR22a].

3 Background

Goal. We provide the first provable guarantees on the existence and approximation error of a small distilled dataset in the KRR settings. We first provide notations that will be used throughout the paper.

Notations. Let H be a Hilbert space with ∥·∥ H as its norm. For a vector a ∈ R n , we use ∥ a ∥ 2 to denote its Euclidean norm, and a i to denote its i th entry for every i ∈ [ n ] . For any positive integer n , we use the convention [ n ] = { 1 , 2 , · · · , n } . Let A ∈ R n × m be a matrix, then, for every i ∈ [ n ] and j ∈ [ m ] , A i ∗ denotes the i th row of A , A ∗ j denotes the j th column of A , and A i,j is the j th entry of the i th row of A . Let B ∈ R n × n , then we denote the trace of B by Tr ( B ) . We use I m ∈ R m × m to denote the identity matrix. Finally, vectors are addressed as column vectors unless stated otherwise.

3.1 Kernel ridge regression

Let X ∈ R n × d be a matrix and let y ∈ R n be a vector. Let k : R d × R d → [0 , ∞ ) be a kernel function, and let K ∈ R n × n be its corresponding kernel matrix with respect to the rows of X ; i.e., K i,j = k ( X i ∗ , X j ∗ ) for every i, j ∈ [ n ] . Let λ > 0 be a regularization parameter. The goal of kernel ridge regression (KRR) involving X , y, k, and λ is to find

We use the notation f λ [ X ,y,k ] : R d → R to denote the in-sample prediction by applying the KRR solution obtained on X , y and λ using the kernel k , i.e., for every x ∈ R d ,

To provide our theoretical guarantees on the size and approximation error for the distilled datasets, the following assumption will be used in our theorem and proofs.

Assumption 1. We inherit the same theoretical assumptions used at [LTOS21] for handling the KRR problem: (I) Let F be the set of all functions mapping R d to R . Let f ∗ ∈ F be the minimizer of n ∑ i =1 | y i -f ( X i ∗ ) | 2 , subject to the constraint that for every x ∈ R d and y ∈ R , y = f ∗ ( x ) + ϵ , where E ( ϵ ) = 0 and Var ( ϵ ) = σ 2 . Furthermore, we assume that y is bounded, i.e., | y | ≤ y 0 . (II) We assume that ∥ ∥ ∥ f λ [ X ,y,k ] ∥ ∥ ∥ H ≤ 1 . (III) For a kernel k , denote with λ 1 ≥ · · · ≥ λ n the eigenvalues of the kernel matrix K . We assume that the regularization parameter satisfies 0 ≤ nλ ≤ λ 1 .

The logic behind our assumptions. First, the idea behind Assumption (I) is that the pair ( X , y ) can be linked through some function that can be from either the same family of kernels that we support (i.e., shift-invariant) or any other kernel function. In the context of neural networks, the intuition behind Assumption (I) is that there exists a network from the desired architectures that gives a good approximation for the data. Assumption (II) aims to simplify the bounds used throughout the paper as it is a pretty standard assumption, characteristic to the analysis of random Fourier features [LTOS19, RR17]. Finally, Assumption (III) is to prevent underfitting. Specifically speaking, the largest eigenvalue of K ( K + nλ I n ) -1 is λ 1 ( λ 1 + nλ ) . Thus, in the case of nλ > λ 1 , the in-sample prediction is dominated by the term nλ . Throughout the following analysis, we will use the above assumptions. Hence, for the sake of clarity, we will not repeat them, unless problem-specific clarifications are required.

Connection to Dataset distillation of neural networks. Since the neural network kernel in the case of infinite width networks describes a Gaussian distribution [JGH18], we aim at proving the existence of small sketches (distilled sets) for the input data with respect to the KRR problem with Gaussian kernel. However, the problem with this approach is that the feature space (in the Gaussian kernel corresponding mapping) is rather intangible or hard to map to, and sketch (distilled set) construction techniques require the representation of these points in the feature space.

To resolve this problem, we use a randomized approximated feature map, e.g., random Fourier features (RFF), and weighted random Fourier features (Weighted RFF). The dot product between every two mapped vectors in this approximated feature map aims to approximate their Gaussian kernel function [RR07]. We now restate a result connecting ridge regression in the RFF space (or alternatively weighted RFF), and KRR in the input space.

Theorem 2 (A result of the proof of Theorem 1 and Corollary 2 of [LTOS21]) . Let X ∈ R n × d be an input matrix, y ∈ R n be an input label vector, k : R d × R d → [0 , ∞ ) be a shift-invariant kernel function, and K ∈ R n × n , where ∀ i, j ∈ [ n ] : K i,j = k ( X i ∗ , X j ∗ ) . Let λ > 0 , and let d λ K = Tr ( K ( K + nλ I n ) -1 ) . Let s ϕ ∈ Ω ( d λ K log ( d λ K )) be a positive integer. Then, there exists a pair ( ϕ, ˜ X ) such that (i) ϕ is a mapping ϕ : R d → R s ϕ (which is based on either the weighted RFF function or the RFF function [LTOS21]), (ii) ˜ X is a matrix ˜ X ∈ R n × s ϕ where for every i ∈ [ n ] ,

˜ X i ∗ := ϕ ( X i ∗ ) , and (iii) ( ϕ, ˜ X ) satisfies

where f λ [ ˜ X ,y,ϕ ] : R s ϕ → R such that for every row vector z ∈ R s ϕ , f λ [ ˜ X ,y,ϕ ] ( z ) = z ( ˜ X T ˜ X + λns ϕ λ I s ϕ ) -1 ˜ X T y . Note that, Table 1 gives bounds on s ϕ when λ ∝ 1 √ n .

Intution behind Theorem 2. Theorem 2 bounds the difference (additive approximation error) between (i) the MSE loss between the ground truth labels and the predictions obtained by applying Kernel Ridge regression (KRR) on the raw (original) data, and (ii) the MSE between the ground truth labels and the predictions obtained when applying Ridge regression on the mapped (full) training data via random Fourier features ( RFF ). Theorem 2 will be utilized to set the minimal dimension of the RFF which yields the desired additive approximation, i.e., 4 λ . Thus the intuition behind using this theorem is to link the dimension of the RFF with the size of the distilled set. In other words, we use this error bound and sufficient size (of the minimal dimension of the RFF ) to provide proof of the sufficient small size of the distilled set.

4 Main result: on the existence of small distilled sets

In what follows, we show that for any given matrix X ∈ R n × d and a label vector y ∈ R n , there exists a matrix S ∈ R s ϕ × d and a label vector y S ∈ R r such that the fitting solution in the RFF space mapping of S is identical to that of the fitted solution on the RFF space mapping of X . With such S and y S , we proceed to provide our main result showing that one can construct a solution for KRR in the original space of S which provably approximates the quality of the optimal KRR solution involving X and y . Thus, we obtain bounds on the minimal distilled set size required for computing a robust approximation, as well as bounds on the error for such a distilled set.

We now provide Theorem 3 followed by its proof of the existence of a small distilled set. Then we provide extensive details and intuitive explanations about the steps of the proof.

Theorem 3 (On the existence of some distilled data) . Let X ∈ R n × d be a matrix, y ∈ R n be a label vector, k : R d × R d → [0 , ∞ ) be a kernel function, Υ = (0 , 1) ∪ { 2 } , and let s ϕ and ˜ X be defined as in Theorem 2. Assume that the rank of ˜ X is s ϕ , then, there exists a matrix S ∈ R s ϕ × d and a label vector y S such that

(ii) there exists an in-sample prediction f λ, X ,y [ S ,y S ,k ] (not necessarily the optimal on S and y s ) satisfying

and

Proof. Let S be any matrix in R s ϕ × d such its weighted RFF mapping ˜ S is of rank equal to that of ˜ X and for every i ∈ [ s ϕ ] , n ∑ j =1 k ( S i ∗ , X i ∗ ) = 0 .

̸

Proof of (i). To ensure (i), we need to find a corresponding proper y S . We observe that

Let b = ( ˜ S T ˜ S + λns ϕ λ I s ϕ )( ˜ X T ˜ X + λns ϕ λ I s ϕ ) -1 ˜ X T y , be the left-hand side term above. b is a vector of dimension s ϕ . Hence we need to solve b = ˜ S T y S for y S . Since S has full rank then we have a linear system involving s ϕ variables and s ϕ equations. Thus, the solution to such a system exists and is y S = ( ˜ S T ) † b , where ( · ) † denotes the pseudo-inverse of the given matrix.

Proof of (ii). Inspired by [LHAR22a] and [NCL20], the goal is to find a set of instances that their in-sample prediction with respect to the input data ( X in our context) would lead to an approximation towards the solution that one would achieve if the KRR was used only with the input data. To that end, we introduce the following Lemm.

Lemma 4 (Restatement of Lemma 6 [LTOS21]) . Under Assumption 1 and the definitions in Theorem 2, for every f ∈ H with ∥ f ∥ H ≤ 1 , with constant probability, it holds that

Note that Lemma 4 shows that for every in-sample prediction function with respect to X , there exists a query β ∈ R s ϕ in the RFF space of that input data such that the distance between the inprediction sample function in the input space and the in-sample prediction in the RFF space is at 2 λ . Furthermore, at [LTOS21] it was shown that β is defined as β = 1 s ϕ ˜ X T ( ˜ X ˜ X T + nλ I s ϕ ) -1 f [ X ] , where f [ X ] i = f ( X i ∗ ) for every i ∈ [ n ] .

We thus set out to find an in-sample prediction function that is defined over S such that by its infimum by Lemma 4 would be the same solution β that the ridge regression on ˜ X attains with respect to the y . Specifically speaking, we want to find an in-sample prediction f λ, X ,y [ S ,y S ,k ] ( · ) such that

where (i) f S [ X ] ∈ R n such that for every i ∈ [ n ] , f S [ X ] i = f λ, X ,y [ S ,y S ,k ] ( X i ∗ ) , and (ii) f λ, X ,y [ S ,y S ,k ] ( · ) = s ϕ +1 ∑ α i k ( S i ∗ , · ) such that α ∈ R s ϕ .

Hence we need to find an in-sample prediction function f λ, X ,y [ S ,y S ,k ] satisfying 5. Now, notice that β ∈ R s ϕ , f S [ X ] ∈ R n and ˜ X T ( 1 s ϕ ˜ X ˜ X T + nλ I n ) -1 ∈ R s ϕ × n . Due to the fact that we aim to find f λ, X ,y [ S ,y S ,k ] , such a task boils down to finding α ∈ R s ϕ which defines f λ, X ,y [ S ,y S ,k ] as in (ii). The above problem can be reduced to a system of linear equations where the number of equalities is s ϕ , while the number of variables is s ϕ .

To do so, we denote 1 s ϕ ˜ X T ( 1 s ϕ ˜ X ˜ X T + nλ I n ) -1 by ˆ A , and observe that we aim to solve

We now show that every entry b j ( j ∈ [ s ϕ ] ) in β can be rewritten as inner products between another pair of vectors in R s ϕ instead of the inner product between two vectors in R n . Formally, for every j ∈ [ s ϕ ] , it holds that

Thus, for every j ∈ [ s ϕ ] , define A j ∗ = [ n ∑ t =1 ˆ A j,t k ( S 1 ∗ , X t ∗ ) , · · · , n ∑ t =1 ˆ A j,t k ( S s ϕ ∗ , X t ∗ ) ] ∈ R s ϕ ˆ

.

In other words, A is a the result of a Hadamard multiplication of A and a 1 -rank matrix G such that each of its rows is equal to [ n ∑ t =1 k ( S 1 ∗ , X t ∗ ) , · · · , n ∑ t =1 k ( S s ϕ ∗ , X t ∗ ) ] .

Since the rank of ˆ A is full, i.e., rank ( A ) = rank ( ˜ X T ( 1 s ϕ ˜ X ˜ X T + nλ I n ) -1 ) = rank ( ˜ X ) = s ϕ by assumption, then it holds that

where the first equality holds by definition of A and ◦ denoting the Hadamard multiplication product, the second inequality holds since by construction of S and G = uv T such that u, v ∈ R s ϕ are vector with non-zero entries, and D u , D v ∈ R s ϕ × s ϕ are diagonal matrices where their diagonal are u and v respectively. The last inequality holds by property of rank function, i.e., the rank of any product of pair of square matrices C and D such that D is of full rank is equal to the rank of C .

The right-hand side of (5) can reformulated as

where now we only need to solve β = A α. Such a linear system of equations has a solution since we have s ϕ variables (the length of α ) and s ϕ equations and the rank A is equal to s ϕ . For simplicity, a solution to the above equality would be α := ( A ) † β. To proceed in proving (ii) with all of the above ingredients, we utilize the following tool.

Lemma 5 (Special case of Definition 6.1 from [BFL + 16]) . Let X be a set, and let ( X, ∥·∥ 2 2 ) be a 2 -metric space i.e., for every x, y, z ∈ X , ∥ x -y ∥ 2 2 ≤ 2 ( ∥ x -z ∥ 2 2 + ∥ y -z ∥ 2 2 ) . Then, for every ε ∈ (0 , 1) , and x, y, z ∈ X ,

We note that Lemma 5 implies that x, y, z ∈ R d

where for τ = 2 we get the inequality associated with the property of 2 -metric, and for any τ ∈ (0 , 1) , we obtain the inequality (7).

We thus observe that

where the first equality holds by adding and subtracting the same term, the first inequality holds by Lemma 5, and the second inequality holds by combining the way f λ, X ,y [ S ,y S ,k ] was defined and Theorem 2. Finally, to conclude the proof of Theorem 3, we derive 4

where the equality holds by adding and subtracting the same term, the first inequality holds by (8), and the second inequality follows as a result of the way f λ S was constructed and the fact that β is its infimum based on Lemma 4, and the last inequality holds by Theorem 2.

Intuition behind Theorem 3. The goal of Theorem 3 is to prove the existence of a small distilled set S (its size is a function of the minimal dimension of the RFF mapping required to ensure the provable additive approximation stated in Theorem 2) satisfying that: (i) The Ridge regression model trained on the mapped training data via RFF is identical to that of the Ridge regression model trained on the mapped small distilled set via RFF, (ii) more importantly there exists a KRR solution formulated for S with respect to the loss of the whole big data X , which approximates the KRR solution on the whole data X (which is the goal of KRR-based dataset distillation techniques). Thus, (iii) we derive bounds on the difference (approximation error) between (1) The MSE between the ground truth labels of the full data and their corresponding predictions obtained by the specific KRR model (we previously described) on our distilled set and (2) The MSE between the ground truth labels and the predictions obtained when applying KRR on the whole data X .

The heart of our approach lies in connecting the minimal dimension of the RFF required for provable additive approximation and the size of the distilled set. This is first done by showing that the distilled set can be any set S of instances from the input space (e.g., images) and their corresponding labels, as long as the corresponding labels must maintain a certain property. Specifically speaking, the labels of the distilled set need to be in correlation with the normal of the best hyperplane found to fit the mapped training data via RFF

˜

X

via the Ridge regression model trained on

˜

X

, y

,

i.e., ( ˜ S T ˜ S + λns ϕ λI s ϕ )( ˜ X T ˜ X + λns ϕ λI s ϕ ) ˜ X T y = ˜ S T y S . From here, the idea hinges upon showing the existence of a KRR model (represented by a prediction function) that would be dependent on the prediction function that can be obtained from applying the Ridge regression problem to the mapped full training data via RFF. With such a model, the idea is to retrieve the predictions obtained when using the Ridge regression problem from the mapped training data via RFF via the use of some KRR model used on the distilled set. We thus show that through careful mathematical derivations, equation reformulation (involving ), and solving a system of equations, one is able to show the existence of a KRR solution that would allow us to use Theorem 2. Finally, to obtain our bounds, we also rely on the use of the weak triangle inequality. To that end, we now utilize the described KRR model on the distilled data together with Theorem 2 to achieve (iii).

Remark 6. Note that the assumption that ˜ X is of full rank (i.e., s ϕ ) can be dropped easily from Theorem 3, and as a result, we obtain that S can contain r (rank of ˜ X ) instances (rows of S ). For additional details, please refer to Section E in the Appendix.

To simplify the bounds stated at Theorem 3, we provide the following remark.

Remark 7. By fixing τ := 2 , the bounds in Theorem 3 become

and

As for fixing τ := ε ∈ (0 , 1) , we obtain that

5 Experimental Study

To validate our theoretical bounds, we performed distillation on three datasets: two synthetic datasets consisting of data generated from a Gaussian Random Field, and classification of two clusters, and one real dataset of MNIST binary and multi-class classification. Full experimental details for all experiments are available in the appendix.

2d Gaussian Random Fields. We first test our bounds by distilling data generated from the Gaussian Process prior induced by a kernel, k on 2d data. We use a squared exponential kernel with lengthscale parameter l = 1 . 5 : k ( x, x ′ ) = e -|| x -x ′ || 2 2 2 l 2 . For X , we sample n = 10 5 datapoints from N (0 , σ 2 x ) , with σ x ∈ [0 . 25 , 5 . 0] . We then sample y ∼ N (0 , K XX + σ 2 y I n ) , σ y = 0 . 01 . We fix λ = 10 -5 and distill down to s = d λ k log d λ k . The resulting values of d λ k , s , and compression ratios are plotted in fig. 2. We additionally plot the predicted upper bound given by Remark 7 and the actual distillation loss. Our predicted upper bound accurately bounds the actual distillation loss. To better visualize how distillation affects the resulting KRR prediction, we show the KRR predictive function f λ X and the distilled predictive f λ S for σ x = 5 . 0 in fig. 1b and fig. 1c.

Two Gaussian Clusters Classification. Our second synthetic dataset is one consisting of two Gaussian clusters centered at ( -2 , 0) and (2 , 0) , with labels -1 and +1 , respectively. Each cluster contains 5000 datapoints so that n = 10 5 . Each cluster as standard deviation σ x ∈ [0 . 25 , 5 . 0] . Additionally, two allow the dataset to be easily classified, we clip the x coordinates of clusters 1 and

(

)

4

2

0

2

4

4

2

0

2

4

Two Clusters,

n

=300,

x

=1.5

(a)

20

15

10

5

0

5

10

15

20

20

15

10

5

0

5

10

15

20

Training Data Predictive,

f

X

(b)

20

15

10

5

0

5

10

15

20

20

15

10

5

0

5

10

15

20

Distilled Data Predictive,

f

S

(c)

Gaussian Random Field

0

1

2

3

4

5

x

0

100

200

300

400

dk

0

1

2

3

4

5

x

0

500

1000

1500

2000

2500

Distilled Set Size, s

0

1

2

3

4

5

x

10 1

10 2

Compression Ratio

0

1

2

3

4

5

x

10

3

10

2

Predicted

Distillation Loss

clusters 2 to not exceed/drop below -0 . 4 and 0 . 4 , for the two clusters, respectively. This results in a margin between the two classes. We visualize the dataset for n = 300 and σ = 1 . 5 in fig. 1a. We use the same squared exponential kernel as before with l = 1 . 5 , fix λ = 10 -5 , and distill with the same protocol as before. We likewise plot d λ k , s , and compression ratios and distillation losses in fig. 3, again with our bound accurately containing the true distillation loss.

0

1

2

3

4

5

x

0

100

200

300

400

dk

0

1

2

3

4

5

x

0

500

1000

1500

2000

2500

Distilled Set Size, s

0

1

2

3

4

5

x

10 1

10 2

10 3

Compression Ratio

0

1

2

3

4

5

x

10

5

10

4

10

3

10

2

10

1

Predicted

Distillation Loss

Two Clusters

Real Word Datasets Classification. For our next test, we first consider binary classification on (i) MNIST 0 and 1 digits, (ii) SVHN 0 and 1 digit, and (iii) CIFAR-10 ship vs deer; all with labels -1 and +1 , respectively. We use the same squared-exponential kernel with l = 13 . 9 , l = 3 . 0 , and l = 8 . 0 , for MNIST, SVHN and CIFAR-10, respectively, which was chosen to maximize the marginallog-likelihood, treating the problem as Gaussian Process regression. We vary n ∈ [500 , 10000] , with an equal class split, and perform the same distillation protocol. Here, we additionally scale λ ∝ 1 √ n such that λ = 10 -4 when n = 5000 . Distilling yields fig. 4, fig. 5, and fig. 6, showing that our bounds can accurately predict distillation losses for real-world datasets. In fig. 7 and fig. 8 in the appendix, we test our bound for the multi-class case on MNIST 0,1,2 digits with similar settings as in the binary classification case. The results justify our bounds.

2000

4000

6000

8000

10000

40

60

80

100

120

dk

2000

4000

6000

8000

10000

200

300

400

500

Distilled Set Size, s

2000

4000

6000

8000

10000

10 1

3×10 0

4×10 0

6×10 0

Compression Ratio

2000

4000

6000

8000

10000

10

5

6×10 6

2×10 5

2000

4000

6000

8000

10000

10

1

Predicted

Distillation Loss

MNIST 0 vs. 1 Classification

SVHN0/1ClassificationOptimizing8A

DistilledSetSize,s

CompressionRatio

入

UpperBoundvs.ActualLoss

120

600

3×10-5

100

110

100

500

101

2 ×10-5

10-1

Predicted

90

400

6 ×100

102

Distillation Loss

80

4 ×100

70

300

3×100

105

10-3

60

2×100

200040006000800010000

200040006000800010000

200040006000800010000

200040006000800010000

200040006000800010000

TrainingSetSize,n

Training SetSize,n

TrainingSetSize,n

CIFAR-10 Deer vs. Ship Classification

2000

4000

6000

8000

10000

40

60

80

100

120

dk

2000

4000

6000

8000

10000

200

300

400

500

Distilled Set Size, s

2000

4000

6000

8000

10000

10 1

3×10 0

4×10 0

6×10 0

Compression Ratio

2000

4000

6000

8000

10000

10

5

6×10 6

2×10 5

2000

4000

6000

8000

10000

10

1

Predicted

Distillation Loss

6 Conclusion

In this study, we adopt a theoretical perspective to provide bounds on the (sufficient) size and approximation error of distilled datasets. By leveraging the concept of random Fourier features (RFF), we prove the existence of small distilled datasets and we bound their corresponding excess risk when using shift-invariant kernels. Our findings indicate that the size of the guaranteed distilled data is a function of the "number of effective degrees of freedom," which relies on factors like the kernel, the number of points, and the chosen regularization parameter, λ , which also controls the excess risk. In particular, we demonstrate the existence of a small subset of instances within the original input space, where the solution in the RFF space coincides with the solution found using the input data in the RFF space. Subsequently, we show that this distilled subset of instances can be utilized to generate a KRR solution that approximates the KRR solution obtained from the complete input data. To validate these findings, we conducted empirical examinations on both synthetic and real-world datasets supporting our claim. While this study provides a vital first step in understanding the theoretical limitations of dataset distillation, the proposed bounds are not tight, as seen by the gap between the theoretical upper bound and the empirical distillation loss in section 5. Future work could look at closing this gap, as well as better understanding the tradeoff between distillation size and relative error.

References

[AKM + 17] Haim Avron, Michael Kapralov, Cameron Musco, Christopher Musco, Ameya Velingker, and Amir Zandieh. Random fourier features for kernel ridge regression:

Approximation bounds and statistical guarantees. In International conference on machine learning , pages 253-262. PMLR, 2017.

[Bac17] Francis Bach. On the equivalence between kernel quadrature rules and random feature expansions. The Journal of Machine Learning Research , 18(1):714-751, 2017.

[BFL + 16] Vladimir Braverman, Dan Feldman, Harry Lang, Adiel Statman, and Samson Zhou. New frameworks for offline and streaming coreset constructions. arXiv preprint arXiv:1612.00889 , 2016.

[BLHK16] Olivier Bachem, Mario Lucic, S. Hamed Hassani, and Andreas Krause. Approximate k-means++ in sublinear time. In Dale Schuurmans and Michael P. Wellman, editors, Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA , pages 1459-1467. AAAI Press, 2016.

[BMK20] Zalán Borsos, Mojmir Mutny, and Andreas Krause. Coresets via bilevel optimization for continual learning and streaming. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) , volume 33, pages 14879-14890, 2020.

[BRVDW19] David Burt, Carl Edward Rasmussen, and Mark Van Der Wilk. Rates of convergence for sparse variational Gaussian process regression. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 862-871. PMLR, 09-15 Jun 2019.

[CKF22] Dingfan Chen, Raouf Kerkouche, and Mario Fritz. Private set generation with discriminative information. In Neural Information Processing Systems (NeurIPS) , 2022.

[CWT + 22] George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022.

[FL11] Dan Feldman and Michael Langberg. A unified framework for approximating and clustering data. In Proceedings of the forty-third annual ACM symposium on Theory of computing , pages 569-578, 2011.

[HCB16] Jonathan H. Huggins, Trevor Campbell, and Tamara Broderick. Coresets for scalable bayesian logistic regression. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) , pages 4080-4088, 2016.

[J23] Matt J. Extract linearly independent subset of matrix columns, 2023.

[JGH18] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems , 31, 2018.

[JSS + 20] Arthur Jacot, Berfin Simsek, Francesco Spadaro, Clément Hongler, and Franck Gabriel. Implicit regularization of random feature models. In International Conference on Machine Learning , pages 4631-4640. PMLR, 2020.

[LBK16] Mario Lucic, Olivier Bachem, and Andreas Krause. Strong coresets for hard and soft bregman clustering with applications to exponential family mixtures. In Artificial intelligence and statistics , pages 1-9. PMLR, 2016.

[LHAR22a] Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. Efficient dataset distillation using random feature approximation. arXiv preprint arXiv:2210.12067 , 2022.

[LHAR22b] Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. Evolution of neural tangent kernels under benign and adversarial training. arXiv preprint arXiv:2210.12030 , 2022.

[LHLR23a] Noel Loo, Ramin Hasani, Mathias Lechner, and Daniela Rus. Dataset distillation fixes dataset reconstruction attacks. arXiv preprint arXiv:2302.01428 , 2023.

[LHLR23b] Noel Loo, Ramin Hasani, Mathias Lechner, and Daniela Rus. Dataset distillation with convexified implicit gradients, 2023.

[LTOS19] Zhu Li, Jean-Francois Ton, Dino Oglic, and Dino Sejdinovic. Towards a unified analysis of random fourier features. In International conference on machine learning , pages 3905-3914. PMLR, 2019.

[LTOS21] Zhu Li, Jean-Francois Ton, Dino Oglic, and Dino Sejdinovic. Towards a unified analysis of random fourier features. The Journal of Machine Learning Research , 22(1):4887-4937, 2021.

[MBL20] Baharan Mirzasoleiman, Jeff A. Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 6950-6960. PMLR, 2020.

[MEM + 22] Alaa Maalouf, Gilad Eini, Ben Mussay, Dan Feldman, and Margarita Osadchy. A unified approach to coreset learning. IEEE Transactions on Neural Networks and Learning Systems , 2022.

[MSF20] Alaa Maalouf, Adiel Statman, and Dan Feldman. Tight sensitivity bounds for smaller coresets. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining , pages 2051-2061, 2020.

[MSSW18] Alexander Munteanu, Chris Schwiegelshohn, Christian Sohler, and David Woodruff. On coresets for logistic regression. Advances in Neural Information Processing Systems , 31, 2018.

[Mur12] Kevin P Murphy. Machine learning: a probabilistic perspective . MIT press, 2012.

[NCL20] Timothy Nguyen, Zhourong Chen, and Jaehoon Lee. Dataset meta-learning from kernel ridge-regression. arXiv preprint arXiv:2011.00050 , 2020.

[NCL21] Timothy Nguyen, Zhourong Chen, and Jaehoon Lee. Dataset meta-learning from kernel ridge-regression. In International Conference on Learning Representations , 2021.

[NNXL21] Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks. In Thirty-Fifth Conference on Neural Information Processing Systems , 2021.

[PDM22] Omead Pooladzandi, David Davini, and Baharan Mirzasoleiman. Adaptive second order coresets for data-efficient machine learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 17848-17869. PMLR, 2022.

[RR07] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems , 20, 2007.

[RR17] Alessandro Rudi and Lorenzo Rosasco. Generalization properties of learning with random features. Advances in neural information processing systems , 30, 2017.

[SCCB22] Mattia Sangermano, Antonio Carta, Andrea Cossu, and Davide Bacciu. Sample condensation in online continual learning, 2022.

[SG05] Edward Snelson and Zoubin Ghahramani. Sparse gaussian processes using pseudoinputs. In Y. Weiss, B. Schölkopf, and J. Platt, editors, Advances in Neural Information Processing Systems , volume 18. MIT Press, 2005.

[SRL + 19] Felipe Petroski Such, Aditya Rawal, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Generative teaching networks: Accelerating neural architecture search by learning to generate synthetic training data. CoRR , abs/1912.07768, 2019.

[SS15] Danica J Sutherland and Jeff Schneider. On the error of random fourier features. arXiv preprint arXiv:1506.02785 , 2015.

[TMF20] Murad Tukan, Alaa Maalouf, and Dan Feldman. Coresets for near-convex functions. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) , 2020.

[TRB16] Dustin Tran, Rajesh Ranganath, and David M. Blei. The variational gaussian process, 2016.

[TZM + 23] Murad Tukan, Samson Zhou, Alaa Maalouf, Daniela Rus, Vladimir Braverman, and Dan Feldman. Provable data subset selection for efficient neural network training. arXiv preprint arXiv:2303.05151 , 2023.

[WZTE18] Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811.10959 , 2018.

[ZB21] Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning , 2021.

[ZMB21] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In International Conference on Learning Representations , 2021.

[ZNB22] Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regression. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) , 2022.

A Experiment Details

All experiments unless otherwise stated present the average/standard deviations of n = 3 runs. Each run consists of a random subset of MNIST 0/1 digits for MNIST binary classification, or random positions of sampled datapoints for synthetic data, and different samples from the GP for the Gaussian Random Field experiment. Distilled datasets are initialized as subsets of the original training data. We distill for 20000 iterations with Adam optimizer with a learning rate of 0 . 002 optimizing both images/data positions and labels. We use full batch gradient descent for the synthetic datasets and a maximum batch size of 2000 for the MNIST experiment. For the MNIST experiment we found that particularly for larger values of n , with minibatch training, we could obtain lower distillation losses by optimizing for longer, so the closing of the gap between the upper bound and experiment values in fig. 4 may be misleading: longer optimization could bring the actual distillation loss lower.

To ensure that assumption (II) is fulfilled, we scale the labels such that ∥ ∥ ∥ f λ [ X ,y,k ] ∥ ∥ ∥ H = 1 . For example, if we are working with MNIST binary classification, with labels { +1 , -1 } , we first compute ∥ ∥ ∥ f λ [ X ,y,k ] ∥ ∥ ∥ H = r using { +1 , -1 } labels, then rescale the labels by 1 /r so that the labels are { + 1 r , -1 r } . Suppose this results in some upper bound L U and some real distillation loss L R . For the corresponding plots in figs. 2 to 4, we plot r 2 L U and r 2 L R . We do this because the r values for different parameters (such as n or σ x ) could be different, and scaling for the plots allows the values to be comparable.

In the figures for the upper bounds on the distillation loss we plot the smallest value of the upper bounds in remark 7.

A.1 Code

Code is available in the supplementary material.

B Proof of Technical Results

B.1 Proof of Theorem 2

Note that the proof of Theorem 2 follows from the proofs of Theorem 1 and Corollary 1 of [LTOS21]. For completeness, we provide the following proof.

Theorem 2 (A result of the proof of Theorem 1 and Corollary 2 of [LTOS21]) . Let X ∈ R n × d be an input matrix, y ∈ R n be an input label vector, k : R d × R d → [0 , ∞ ) be a shift-invariant kernel function, and K ∈ R n × n , where ∀ i, j ∈ [ n ] : K i,j = k ( X i ∗ , X j ∗ ) . Let λ > 0 , and let d λ K = Tr ( K ( K + nλ I n ) -1 ) . Let s ϕ ∈ Ω ( d λ K log ( d λ K )) be a positive integer. Then, there exists a pair ( ϕ, ˜ X ) such that (i) ϕ is a mapping ϕ : R d → R s ϕ (which is based on either the weighted RFF function or the RFF function [LTOS21]), (ii) ˜ X is a matrix ˜ X ∈ R n × s ϕ where for every i ∈ [ n ] , ˜ X i ∗ := ϕ ( X i ∗ ) , and (iii) ( ϕ, ˜ X ) satisfies

where f λ [ ˜ X ,y,ϕ ] : R s ϕ → R such that for every row vector z ∈ R s ϕ , f λ [ ˜ X ,y,ϕ ] ( z ) = z ( ˜ X T ˜ X + λns ϕ λ I s ϕ ) -1 ˜ X T y . Note that, Table 1 gives bounds on s ϕ when λ ∝ 1 √ n .

Proof. First, let β ∈ R s ϕ be the ridge regression solution involving ˜ X and y , i.e.,

Note that, by construction, for every i ∈ [ n ] , f λ [ ˜ X ,y,ϕ ] ( ˜ X i ∗ ) is equal to ˜ X i ∗ · β .

Thus,

where the first equality holds by definition of f λ [ ˜ X ,y,ϕ ] ( ˜ X i ∗ ) for every i ∈ [ n ] , and the second equality follows from the proof of Lemma 6 of [LTOS21] which indicates that s ∥ β ∥ 2 2 ≤ 2 . Note that,

where the first equality holds by adding and subtracting the same amount, while the last equality holds by simple expansion.

To properly bound the terms above, we rely on Lemma 7 of [LTOS21].

Lemma 8 (Restatement of Lemma 7 [LTOS21]) . Under Assumption 1 and the definitions in Theorem 2, and for

where ˜ H is the corresponding RKHS space of ˜ X , it holds that

Hence, combining Lemma 8 with (11) yields

where the last inequality holds by Lemma 4 (or equivalently Lemma 6 of [LTOS21]), and thus concluding Theorem 2.

C Regarding s ϕ

In what follows, we discuss cases for determining the quantity s ϕ . To quantify s ϕ which in turn determines the size of the distilled set, we need to measure d K which is the trace of K ( K + nλI n ) -1 :

For the case where K has a finite rank, i.e., the number of positive eigenvalues is lower than n , then s ϕ ∈ Ω(1) .

As for the exponential decay, it occurs when the kernel is Gaussian and the marginal distribution of the input data (e.g., images) is sub-Gaussian. In such a case, it was shown in [Bac17] that d K ∈ O ( log 1 λ ) . Thus s ϕ is poly-logarithmic in n when λ := O ( 1 √ n ) .

For the case where the Hilbert space H is also a Sobolov space of order γ larger or equal to 1, then d K ∈ O ( 1 λ 2 γ ) which in the case of λ := O ( 1 √ n ) , we have s ϕ ∈ Ω ( n 1 4 γ ) .

In the general case, where the decay of the eigenvalues admits λ i ∝ O ( i -1 ) , then d k in the worst case is bounded by O ( 1 lambda , and thus, via Theorem 2, we can deduce that s ϕ ∈ O ( √ n log n ) for the case of λ ∈ O ( 1 √ n ) .

Note that the choice of λ ∈ O ( 1 √ n ) is used in the literature of KRR to ensure that the learning rate is 1 √ n as shown in [RR17].

D Multi-class Experiments

In fig. 7 and fig. 8 we test our bound for the multi-class case on MNIST 0,1,2 digits with similar settings as in the binary classification case. The results justify our bounds.

MNIST0/1/2ClassificationOptimizing12入

d

DistilledSetSize，s

CompressionRatio

入

UpperBoundvs.ActualLoss

1600

6×10-4

10-1

250

1400

6×100

4 × 10-4

10-2

1200

200

4 ×100

3 ×10-4

103

1000

3 ×100

10~4

150

800

2 ×10-4

600

2×100

10-5

Predicted

100

10-6

Distillation Loss

400

2500500075001000012500

2500500075001000012500

2500500075001000012500

250050007500 1000012500

25005000

7500 1000012500

MNiST0/1/2ClassificationOptimizing8A

d

DistilledSetSize,s

CompressionRatio

入

UpperBoundvs.ActualLoss

16001

6×10-4

10-1

250

1400

6×100

102

4 ×10-4

200

1200

4 ×100

3 ×10-4

10-3

1000

3 ×100

10-4

150

800

2 ×10-4

10-5

600

2 ×100

10-6

Predicted

100

400

107

Distillation Loss

2500500075001000012500

2500500075001000012500

2500500075001000012500

250050007500 10000 12500

25005000

7500 1000012500

E On the rank of ˜ X and its connection to the distilled set S

First, recall that ˜ X is the RFF image of X . There are mainly two cases connecting the rank of ˜ X to S :

(I) The rank of ˜ X is s ϕ , i.e., ˜ X is of full rank.

(II) Otherwise.

Regarding Case (I). For such a case, Theorem 3 is exactly what we need to prove the existence of a distilled set S .

Handling Case (II). For this case as well, one can also choose to ensure that the rank of S (matrix form of S ) is s ϕ (full rank) and apply the same derivations done in the proof of Theorem 3 without any violation from the perspective of the validity of our proof. However, due to the nature of this case, one can also ensure that the size of S (number of instances) is much smaller than s ϕ , specifically, r (the rank of ˜ X ). To that end, must ensure that

(A) the following equation

has a solution, i.e., y S exists, and also

(B) we need to ensure that (6) also has a solution, i.e., there exists α ∈ R r such that

We observe that since ˜ X is rank deficient adds some obstacles to our proofs; when attempting to ensure both (A) and (B). Specifically speaking, the rank of ( ˜ S T ˜ S + λns ϕ λ I s ϕ )( ˜ X T ˜ X + λns ϕ λ I s ϕ ) -1 ˜ X T is equal to the rank of ˜ X (due to properties of rank function) and since we aim to maintain that S has exactly that number of rows while also its RFF image ˜ S has a rank equal to that of ˜ X , we obtain that the linear system of equations in (A) might have no solution at all. This is unless the row span of ˜ S coincides with that of ˜ X .

Having an S such that the row span of its RFF image coincides with the row span of ˜ X , also ensures the existence of a solution concerning the linear system of equations present at (B).

F One of many ways to construct S

Following the previous section, to properly construct S that would satisfy the proof of Theorem 3, we discuss two possible constructions:

If ˜ X is of full rank (the rank is s ϕ ), then there exists a subset S of the rows of X such that its image in the RFF space is a linearly independent subset ˜ S of the rows of ˜ X . Such a subset is of full rank as it is linearly independent. Such an algorithm can be accessed via [J23].

If ˜ X is rank deficient, i.e., r < s ϕ where r denotes the rank of ˜ X , then the problem by it self requires r independent vectors, i.e, one can represent ( ˜ X T ˜ X + λns ϕ λI s ϕ ) -1 ˜ X T y by Az where A ∈ R s ϕ × r and z ∈ R r . This is since the rank of ( ˜ X T ˜ X + λns ϕ λI s ϕ ) -1 ˜ X T is at max the minimum between the rank of ( ˜ X T ˜ X + λns ϕ λI s ϕ ) -1 and the rank of ˜ X T . Such rank is bounded from above by r since the rank of ( ˜ X T ˜ X + λns ϕ λI s ϕ ) -1 is s ϕ .

To this end, we can invoke the first case with respect to ( B,z ) . Thus, by choosing S to contain the r linearly independent rows in ˜ X and an additional row from ˜ X (not already in S ), there exists y S such that,

Connection to subset selection and label solve. Observe that according to this section, we showed that our bounds in Theorem 3 hold for a subset of the data but with different (potentially learned) labels (not necessarily from the set of input labels) which can be thought of as a hybrid between distillation and subset selection. Intuitively, this justifies the label-solve approach of [NCL20], where here we showed that a subset of the data can achieve similar bounds but with learning the labels.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides theoretical analysis towards recent errors of recent dataset distillation methods based on kernel ridge regression (KRR).  It mainly utilizes some previous results on KRR and applies them in the context of dataset distillation. Some simple simulation results verify the derived bounds.\n\nSTRENGTHS:\n1. The paper derives valid error bounds for KRR-based dataset distillation methods under some assumptions. It provides a guidance for the size of synthetic dataset that is required to result in pleasant errors. \n2. The writing is logical and consistent. The contributions of this paper and how this paper inherits previous results is very clear.\n\nWEAKNESSES:\n1. My major concern is that the analysis is not consistent with the typical setting of dataset distillation:\n    * This paper uses a distilled dataset whose size is greater than the number of features encoded by the kernel. However, in practice, the KRR-based methods in dataset distillation often rely on infinite-wide neural networks and in practice this is achieved by a large feature dimension, while the size of distilled dataset is in fact very small, e.g., only 1 image per class.\n    * In fact, the setting that the number of distilled samples is larger than the feature dimension makes the problem much simpler according my own experience. For example, we can directly solve the label $y$ without touching the samples $X$ as shown in Proof of (i) to get exactly the same KRR solution. Intuitively, if KRR solutions of real and synthetic datasets are consistent, predictions on data should also be consistent, which indicates that the results in the paper are somewhat trivial.\n    * Moreover, this paper assumes that the output dimension is only 1. However, the output dimension is in fact the number of classes for typical classification problems. It is unclear whether the error bounds derived by the current paper still useful in real cases.\n2. The technical contribution is not enough for a NeurIPS paper. The paper directly applies results of previous works. The results in the current paper can be viewed as an implication of previous results on the dataset distillation setting, since the setting is simple as mentioned previously.\n3. The error bounds are not tight as shown in the experiment section, which means the theoretical results are not very useful in practice.\n4. The experiments are too simple. The largest scale is only binary classification on MNIST dataset. For dataset distillation, at least results on CIFAR10 are expected as very standard benchmarks.\n\nQUESTIONS:\n1. The analysis for cases that the number of synthetic samples is smaller than the feature dimension.\n2. The analysis for cases that the output dimension is larger than 1.\n3. Experimental analysis on larger datasets. \n\nPlease refer to the weaknesses part for details.\n\nLIMITATIONS:\nI have not found issues related to this part.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-25"}

comments: SUMMARY:
This manuscirpt first give theoretical understanding on synthetic dataset generated in dataset distillation task. In concrete, the authors prove (1) the existance of distilled datasets and (2) the generalization error is related to the "number of effective degrees of freedom" in the random Fourier features (RFF) regime. The theoretical bounds are further verified by simple experiments.

STRENGTHS:
1. First theoretical work on the field of dataset distillation and the theoretical results are well established;

2. Theoretically show the correlation between the size of distilled dataset and the characteristics of kernels in RFF regime;

3. Theoretically show the generalization bound w.r.t. distilled datasets in kernel ridge regression (KRR) regime.

WEAKNESSES:
1. The theoretical results are biult on KRR, which has a gap to the finite-width network architectures used in dataset distillation.

2. Can the derived generalization bounds provide insights on developing novel dataset distillation algorithms. For example, imporving the kernel architecture to decrease the right-handed term in the generalization bound to reduce the risk trained on synthetic datasets. In this way, the bound can be seen to be tight and impractical.

QUESTIONS:
See weekness.

LIMITATIONS:
See weekness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper attempts to provide the first theoretical guarantees on the existence of dataset distillation, under the setup of kernel ridge regression. The proof techniques are mainly based on theory of random fourier features. They also provide experiments which are indicated to support their theoretical results.\n\nSTRENGTHS:\n1. The paper is the first attempt to theoretically guarantees the existence of dataset distillation, which is an important topic for efficient learning.\n2. Experimental results are seemingly consistent with the theoretical results, which could strength their claim if the relationship between the theoretical results and the experimental protocol was clear.\n\nWEAKNESSES:\n1. The paper seems not to pay attention to readability of its theoretical results. E.g., Theorem 2 is explained by just 2 lines (l.130-131) without any proof; the notations and statements in Theorem 2 and Theorem 3 are not sophisticated.\n2. The proof of Theorem 2 is not provided even in the supplementary materials, but just stated as \"A result of the proof of Theorem 1 and Corollary 2 of [ LTOS21]\".\n3. I cannot find the proof of the existence of the distilled dataset S in the proof of Theorem 3, which is the main claim in this paper.\n4. The relationship between the theoretical construction (which I cannot find) and the distillation method used in experiments is unclear. So I'm not confident in whether the experiments really support their theoretical results.\n\nQUESTIONS:\n1. Where is the existence of S proved?\n2. Can you give the intuition or short strategy of the proof before Theorem3? For example, why can the number of the distilled data S be $s_\\phi + 1$? How can we construct it explicitly?\n3. Where did you explain what distillation algorithm is used in experiments? How does the algorithm relate to the construction in the theoretical results?\n\nLIMITATIONS:\nLimitation is discussed in the last section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a theoretical analysis of dataset distillation, specifically focusing on the size and approximation error of distilled datasets. The authors provide bounds on the sufficient size and relative error of distilled datasets for kernel ridge regression (KRR) based methods using shift-invariant kernels. They prove the existence of small distilled datasets and show that a KRR solution can be generated using these distilled datasets that approximate the solution obtained on the full input data. The theoretical results are validated through empirical experiments on synthetic and real datasets.\n\nSTRENGTHS:\n(1) The paper addresses an important problem in dataset distillation by providing theoretical bounds on the size and approximation error of distilled datasets. This fills a gap in the literature where previous work has mainly been empirical.\n(2) The use of random Fourier features (RFF) and kernel ridge regression (KRR) provides a solid theoretical foundation for the analysis.\n(3) The paper includes both analytical proofs and empirical validation to support the theoretical results.\n\nWEAKNESSES:\n(1) The evaluation of the proposed method could be further strengthened by comparing it with other baseline methods in the field of dataset distillation.\n(2) The clarity of the exposition could be improved, as some sections of the paper are not easy to understand without prior knowledge of the topic.\n(3) In line 223, two clusters and each cluster has 5000 points. Will they conflict with 10^5 points?\n(4)The paper should provide more experiments on real and bigger datasets and the visualization of KRR predictive functions on the MNIST dataset.\n\nQUESTIONS:\n(1) Can you provide more experimental results on larger real-world datasets and make appropriate visualizations to prove effectiveness?\n(2) Can you provide more explanation about how understanding data distillation can guide the performance of data distillation?\n\nLIMITATIONS:\nNot fully discussed how a better understanding of data distillation will guide and evaluate subsequent data distillation work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-24"}