Title: Provable and Efficient Dataset Distillation for Kernel Ridge Regression
Venue: NeurIPS 2024
Accepted: yes
Keywords: Dataset Distillation

Kernel Ridge Regression
Abstract: Deep learning models are now trained on increasingly larger datasets, making it crucial to reduce computational costs and improve data quality. Dataset distillation aims to distill a large dataset into a small synthesized dataset such that models trained on it can achieve similar performance to those trained on the original dataset. While there have been many empirical efforts to improve dataset distillation algorithms, a thorough theoretical analysis and provable, efficient algorithms are still lacking. In this paper, by focusing on dataset distillation for kernel ridge regression (KRR), we show that one data point per class is already necessary and sufficient to recover the original model's performance in many settings. For linear ridge regression and KRR with surjective feature mappings, we provide necessary and sufficient conditions for the distilled dataset to recover the original model's parameters. For KRR with injective feature mappings of deep neural networks, we show that while one data point per class is not sufficient in general, $k+1$ data points can be sufficient for deep linear neural networks, where $k$ is the number of classes. Our theoretical results enable directly constructing analytical solutions for distilled datasets, resulting in a provable and efficient dataset distillation algorithm for KRR. We verify our theory experimentally and show that our algorithm outperforms previous work such as KIP while being significantly more efficient, e.g. 15840$\times$ faster on CIFAR-100. Our code is available at \href{https://github.com/Trustworthy-ML-Lab/provable-efficient-dataset-distill-KRR}{GitHub}.

Provable and Efficient Dataset Distillation for Kernel Ridge Regression

Abstract

Deep learning models are now trained on increasingly larger datasets, making it crucial to reduce computational costs and improve data quality. Dataset distillation aims to distill a large dataset into a small synthesized dataset such that models trained on it can achieve similar performance to those trained on the original dataset. While there have been many empirical efforts to improve dataset distillation algorithms, a thorough theoretical analysis and provable, efficient algorithms are still lacking. In this paper, by focusing on dataset distillation for kernel ridge regression (KRR), we show that one data point per class is already necessary and sufficient to recover the original model's performance in many settings. For linear ridge regression and KRR with surjective feature mappings, we provide necessary and sufficient conditions for the distilled dataset to recover the original model's parameters. For KRR with injective feature mappings of deep neural networks, we show that while one data point per class is not sufficient in general, k +1 data points can be sufficient for deep linear neural networks, where k is the number of classes. Our theoretical results enable directly constructing analytical solutions for distilled datasets, resulting in a provable and efficient dataset distillation algorithm for KRR. We verify our theory experimentally and show that our algorithm outperforms previous work such as KIP while being significantly more efficient, e.g. 15840 × faster on CIFAR-100. Our code is available at GitHub.

1 Introduction

Deep learning models are now trained on increasingly massive datasets, incurring substantial computational costs and data quality challenges. For instance, Llama 3 was pre-trained on over 15 trillion tokens, while the training of GPT-4 exceeded $100 million. Reducing these burdens is crucial. Dataset distillation [34] aims to distill a large dataset into a small synthesized dataset such that models trained on it can achieve similar performance to those trained on the original dataset. A good small distilled dataset is not only useful in saving computational cost and improving data quality but also has various applications such as continual learning [39, 40, 35], privacy protection [24, 40, 18, 5, 1], and neural architecture search [31, 39].

While there have been many empirical efforts to improve dataset distillation algorithms [24, 39, 2, 38], a thorough theoretical analysis is still lacking. Izzo and Zou [9] show single distill data is sufficient for a class of generalized linear models with one-dimensional output, where the data is assumed to follow a generalized exponential density function and the negative log-likelihood is optimized by gradient descent. For a linear ridge regression (LRR), Izzo and Zou [9] show d data points are needed to recover the original model's parameter for all regularization at the same time, where d is the dimension of the data and is large even for small datasets like MNIST [13] and CIFAR [12] ( d = 784 and 3072) in computer vision. For kernel regression with Gaussian kernel, they show n data points are necessary, where n is the number of original data points and can be large for modern datasets, e.g.

n = 60000 and 50000 for MNIST and CIFAR (see Table 2). Maalouf et al. [21] use Random Fourier Features (RFF) to approximate shif-invariant kernels that may have an infinite-dimensional feature space, and construct p distilled data for such RFF model, where p is the dimension of the RFF model and can be Ω( √ n log n ) in general cases. The results in [9, 21], however, have a large gap compared with the empirical evidence that one data point per class can often achieve comparable performance to the original model [24, 39, 2, 38].

In this paper, by focusing on dataset distillation for kernel ridge regression (KRR), we show that one data point per class is already necessary and sufficient to recover the original model's performance in many settings , which is far less than n or p data points needed in prior works [9, 21]. Besides, our analysis is more general than prior works [9, 21] and can handle more and different models, including invertible neural networks, fully-connected neural networks (FCNN), Convolutional neural networks (CNN), and Random Fourier Features (RFF). Table 1 compares our theoretical results with previous analysis. We summarize our contributions as follows.

In Sec. 4.1 and 5, for linear ridge regression (LRR) and KRR with surjective feature mappings, we show that one distilled data point per class is necessary and sufficient to recover the original model's parameters and provide necessary and sufficient conditions for such distilled datasets. In addition, we show how to find distilled data that is close to real data in Sec. 4.2.

In Sec. 5.2, for KRR with injective feature mappings of deep neural networks (NNs), we show that one data point per class is in general not sufficient to recover the original model's parameters. However, k +1 data points can be sufficient for deep linear NNs, where k is the number of classes.

Our theoretical results enable us to directly construct analytical solutions for the distilled datasets, resulting in a provable and efficient dataset distillation algorithm for KRR in Algorithm 1. We verify our theory experimentally and show that our algorithm outperforms previous SOTA dataset distillation algorithm KIP [25] while being significantly more efficient, e.g. 15840 × faster on CIFAR-100.

In Sec.6, we show our theoretical results can be used for several applications. First, it can be used as necessary or sufficient conditions for KIP-type algorithms to converge to a global minimum even if the loss function is highly non-convex. Second, our distilled dataset for KRR can provably preserve the privacy of the original dataset while having a performance guarantee.

2 Related works

Dataset distillation. Dataset distillation aims to distill a large dataset into a small synthesized dataset such that models trained on it can achieve similar performance to those trained on the original dataset. Previous approaches can be mainly divided into four categories [29]: 1) Meta-model Matching: this category formulates the problem as a bilevel optimization problem and maximize the performance of the model trained on the distilled dataset [34]. Some recent works such as KIP [24, 25], FRePo [40], RFAD [17], and RCIG [18] approximate the inner loop optimization of training neural networks by

KRR with Neural Tangent Kernel [10] or neural network Gaussian process (NNGP) kernels [14]. 2) Gradient Matching: this category minimizes the distance between the gradients of models trained on the original dataset and distilled dataset [39, 37, 15, 11]. 3) Trajectory Matching: this category aims to match the training trajectories of models trained on the original dataset and distilled dataset [2, 4, 8, 7]. 4) Distribution Matching: this approach directly matches the distribution of the original dataset and distilled dataset via a single-level optimization [38, 33, 36]. Our work is closely related to kernel-based dataset distillation algorithms [24, 25, 40, 17, 18] in category (1). Our theoretical analysis provides theoretical foundations and implications for these kernel-based algorithms.

Theoretical analysis of dataset distillation. In addition to the papers discussed in the introduction, Maalouf et al. [19] propose an efficient algorithm to construct a d 2 +1 core set of the original dataset for least mean squares problems. Maalouf et al. [20] further propose to use the SVD of the original dataset to construct a distilled dataset of size d . Tukan et al. [32] utilize the idea of subset selection to improve the initialization and training procedure of dataset distillation. Our paper focuses on KRR and constructs k distilled data analytically, where k is usually much less than d (see Table 2).

3 Preliminaries

3.1 Dataset Distillation

For an original dataset { x i , y i } n i =1 , we denote X = [ x 1 , . . . , x n ] ∈ R d × n and Y = [ y 1 , . . . , y n ] ∈ R k × n , where d is the dimension of the data, k is the dimension of the label or the number of the classes, and n is the number of data points. The goal of dataset distillation is to create a synthetic dataset X S = [ x S 1 , . . . , x S m ] ∈ R d × m and Y S = [ y S 1 , . . . , y S m ] ∈ R k × m , with the number of distilled data points m ≪ n , such that a model trained on this synthetic dataset ( X S , Y S ) can achieve similar performance to those trained on the original dataset.

As the data dimension is usually larger than the label dimension in practice, e.g. MNIST has d = 728 , k = 10 and other datasets have even larger d , we consider d ≥ k in this paper. For a matrix A , we use A + to denote its pseudoinverse and Range( A ) to denote its range space.

3.2 Dataset Distillation for Kernel Ridge Regression (KRR)

Original model : Given a kernel K ( x , x ′ ) = ⟨ ϕ ( x ) , ϕ ( x ′ ) ⟩ , where ϕ : R d ↦→ R p is the feature mapping from input space to a feature space of dimension p , a KRR model f ( x ) = W ϕ ( x ) trained on original data set with a predefined regularization λ ≥ 0 tries to minimize following objective

where W ∈ R k × p and ϕ ( X ) = [ ϕ ( x 1 ) , . . . , ϕ ( x n )] ∈ R p × n . The solution can be computed analytically as W = Y ϕ λ ( X ) + , where

and K ( X , X ) = ϕ ( X ) ⊤ ϕ ( X ) ∈ R n × n . ϕ λ ( X ) can be considered as regularized features. Linear ridge regression is a special case of kernel ridge regression (KRR) with ϕ ( x ) = x .

KRR is used in many dataset distillation algorithms [24, 25, 40, 17, 18]. In this paper, we mainly consider a finite-dimensional ϕ . This matches the practical neural networks which are usually used in dataset distillation. For shift-invariant kernels with infinite-dimensional RKHS space, e.g. Gaussian kernel, they can be well approximated by finite-dimensional random Fourier features [27, 16].

Distilled dataset model : Similarly, a KRR trained on distilled dataset with regularization λ S ≥ 0 is f S ( x ) = W S ϕ ( x ) , where W S = Y S ϕ λ S ( X S ) + ∈ R k × d . Additionally, denote X λ S = ϕ λ S ( X S ) with ϕ ( x ) = x , i.e.

The goal of dataset distillation here is to find ( X S , Y S ) such that W S = W , where W is supposed to be given, i.e. can be computed from the original dataset ( X , Y ) .

4 Dataset Distillation for Linear Ridge Regression (LRR)

In this section, we first analyze the dataset distillation for the linear ridge regression (LRR). In Sec. 4.1, for a LRR model, we show that k distilled data points (one per class) are necessary and sufficient to guarantee W S = W . We provide analytical solutions of such X S allowing us to compute the distilled dataset analytically instead of having to learn it heuristically in existing works [24, 25, 40, 17, 18]. Then, in Sec 4.2, we show how to find distilled data that is close to real data. Lastly, for fixed data and only distilling labels, we show d points are needed in Sec 4.3.

4.1 Analytical Computation for Linear Ridge Regression

Theorem 4.1. When m < k , there is no X S can guarantee W S = W unless the columns of W are in the range space of Y S . When m ≥ k and Y S is rank k , let r = min( m,d ) and take D = Y + S W + ( I m -Y + S Y S ) Z , where Z ∈ R m × d is any matrix of the same size as X ⊤ S . Suppose the reduced SVD of D is D = V diag ( σ ′ 1 , . . . , σ ′ r ) U ⊤ with σ ′ 1 ≥ · · · ≥ σ ′ r ≥ 0 , the following results hold:

λ S > 0 : W S = W if and only if, for any D defined above, λ S ≤ 1 4 σ ′ 2 1 and X S = U diag ( σ 1 , . . . , σ r ) V ⊤ where σ i = { 0 , if σ ′ i = 0 , 1 ± √ 1 -4 λ S σ ′ 2 i 2 σ ′ , otherwise.

λ S = 0 : W S = W if and only if X S = D + for any D defined above.

Proof sketch. The proof can be found in Appendix C. The key idea is that we want to solve X S from W S = Y S X + λ S = W (Note: W is given and we can select/decide Y S ). When m < k , this is an overdetermined system for X + λ S . There is no solution for X + λ S in general therefore no solution for X S . When m ≥ k and Y S is rank k , the solutions of X + λ S are given by X + λ S = Y + S W + ( I m -Y + S Y S ) Z , where Z ∈ R m × d is any matrix of the same size as X + λ S . However, not all such X + λ S corresponds to a X S . To solve X S , we assume we have the SVD of X S and solve it from the SVD of X + λ S .

Intuitively, original dataset ( X , Y ) is compressed into X S through original model's parameter W = Y ( X ⊤ X + λ I n ) -1 X ⊤ . When m = k , i.e. one distilled data per class, D = Y + S W is deterministic and X S is fully determined by W and Y S . In this case, when λ S = 0 and W is full rank, X S can be easily computed as X S = W + Y S . As an example, Figure 1 shows the distilled data for MNIST and CIFAR-100 when m = 10 / 100 . When m > k , i.e. more than one distilled data per class, there exist infinitely many distilled datasets since Z is a free variable to choose. When m = n , one can verify that X is a distilled dataset for itself by taking Y S = Y and Z = ( X ⊤ X + λ I n ) -1 X ⊤ . When m > n , we can generate more data than original dataset. Compared with [9] that needs d data, our approach only requires k data and usually k ≤ d in practice. Our result is also more flexible to distill any m ≥ k data points.

3

apple

aquarium fish

baby

bear

beaver

bed

bee

beetle

bicycle

bottle

Discussion. The requirement for λ S can be easily satisfied by setting λ S ≤ 1 4 σ ′ 2 1 for a given D . If we want to fix a predefined λ S , e.g. λ S = λ , we need to sample different D (by sampling different Z ) so that λ S ≤ 1 4 σ ′ 2 1 is satisfied. Theorem 4.1 generally suggests that a smaller λ S is better for

constructing distilled data. Practical algorithms, e.g. KIP, FRePo, and RFAD, usually use a very small regularization, which may already satisfy the requirement.

In practice, we usually do not want the dataset to be singular. Below we show that it is easy to satisfy as long as Z is full rank and the rows of W and Z are linearly independent.

Proposition 4.1. When m ≥ k and Y S , W are rank k , the X S in Theorem 4.1 is full rank for any full-rank Z such that Range ( W ⊤ ) ∩ Range ( Z ⊤ ) = { 0 } .

4.2 Finding Realistic Distilled Data

In Theorem 4.1, any D satisfying the condition will guarantee the distilled dataset model to recover the original model's performance. For example, we can choose Z to be a random Gaussian matrix. However, to make the distilled data more realistic and generalizable, we can select m real training data ˆ X S as initialization of distilled data and find the distilled data that is closest to ˆ X S .

Corollary 4.1.1. Given fixed ˆ X S , λ S , and Y S , the D that satisfies Theorem 4.1 and minimize

where ˆ X + λ S is defined analogous to X + λ S . Taking Y S = W ˆ X λ S can further minimize the distance.

When λ S = 0 , Y S = W ˆ X S is the prediction of original model for ˆ X S . Combining this with Theorem 4.1, we summarize the computation of the distilled data in Algorithm 1 with ϕ ( x ) = x . Figure 2 shows some distilled data that is close to the real data or generated with random noise.

2345

12345

(b) CIFAR-100 with IPC=5.

4.3 Label Distillation

If we fix the X S and only distill labels, as also shown in [24], we need at least m = d to guarantee W S = W because labels have fewer learnable parameters than data.

Theorem 4.2. For any fixed X S ,

when m < d , there is no Y S can guarantee W S = W in general unless the rows of W are in the row space of X + λ S . The least square solution is Y S = WX λ S and ∥ W S -W ∥ = ∥ ∥ W ( X λ S X + λ S -I d )∥ ∥ .

when m ≥ d , if X S is rank d , then Y S = WX λ S is sufficient for W S = W .

5 Dataset Distillation for Kernel Ridge Regression (KRR)

In the last section, we analyzed the dataset distillation for LRR. However, more complex models such as KRR and neural networks (NNs) are usually used in practice for better performance. Therefore, it is crucial to extend the analysis to these practical settings. In this section, we first extend the results of LRR to KRR in the feature space and then construct distilled data from desired features by considering two cases - subjective and non-surjective feature mappings.

The results in Theorem 4.1 of LRR can be directly extended to the KRR in the feature space by replacing data X S with features ϕ ( X S ) in Theorem 4.1. For completeness, we state it below.

Theorem 5.1. When m < k , there is no ϕ ( X S ) can guarantee W S = W unless the columns of W are in the range space of Y S . When m ≥ k and Y S is rank k , let r = min( m,p ) and take D = Y + S W + ( I m -Y + S Y S ) Z , where Z ∈ R m × p is any matrix of the same size as ϕ ( X S ) ⊤ . Suppose the reduced SVD of D is D = V diag ( σ ′ 1 , . . . , σ ′ r ) U ⊤ with σ ′ 1 ≥ ·· · ≥ σ ′ r ≥ 0 , the following results hold:

λ S > 0 : W S = W if and only if, for any D defined above, λ S ≤ 1 4 σ ′ 2 1 and ϕ ( X S ) = U diag ( σ 1 , . . . , σ r ) V ⊤ where σ i = { 0 , if σ ′ i = 0 , 1 ± √ 1 -4 λ S σ ′ 2 i 2 σ ′ i , otherwise.

λ S = 0 : W S = W if and only if ϕ ( X S ) = D + for any D defined above.

This shows that in the feature space, k features are necessary and sufficient to recover the original model's parameter. However, what we get is the feature of distilled data ϕ ( X S ) instead of distilled data X S itself. To get X S , we need to construct the data from the features. To do this, we consider two cases - surjective and non-surjective ϕ . For subjective ϕ , we show that we can directly construct X S from ϕ ( X S ) . For non-surjective ϕ such as neural networks (NNs), we show one data per class is in general not sufficient, but k +1 data points can be sufficient for deep linear neural networks.

5.1 Surjective Feature Mapping

When ϕ is surjective or bijective, we can always find a X S for a desired ϕ ( X S ) . In this case, k distilled data (one data per class) is sufficient to recover the original model's performance, in contrast to [21] that needs p distilled data. We summarize the computation of the distilled data in Algorithm 1. Here we give some examples of surjective/bijective ϕ .

Example 5.1 (Invertible NN) . If ϕ is invertible such as invertible NNs used in normalizing flows, then we can directly compute x = ϕ -1 ( ϕ ( x )) .

Example 5.2 (Fully-connected NN (FCNN)) . For a ( L +1 )-layer FCNN f ( x ) = W ϕ ( x ) and

where σ is a surjective or bijective activation function such as LeakyReLU, and W ( l ) ∈ R d l × d l -1 with d = d 0 ≥ d 1 ≥ · · · ≥ d L for l ∈ [ L ] . If all W ( l ) are full rank, given ϕ ( x ) , we can compute

where σ -1 is any right inverse of σ . When some W ( l ) are not full rank, we can still compute an approximated solution.

Example 5.3 (Convolutional Neural Network (CNN)) . CNN is known as a special type of FCNN. To illustrate, we give an example of a convolution layer of 2 × 2 filter. Let w ∈ R 4 be a convolutional

filter of size 2 . Then the convolution operation with stride 1 can be represented as

If the data is three-channel images, same operation can be done for each channel followed by a matrix that sums over three channels. When the matrix equation has a solution, the data can be solved from the feature.

Example 5.4 (Random Fourier Features (RFF) [27]) . A shift-invariant kernel K ( x , x ′ ) = K ( x -x ′ ) can be approximated by random Fourier features K ( x , x ′ ) ≈ ⟨ ϕ ( x ) , ϕ ( x ′ ) ⟩ . Here ϕ ( x ) = √ 2 p [ cos( a ⊤ 1 x + b 1 ) , . . . , cos( a ⊤ p x + b p ) ] ⊤ ∈ R p where a 1 , . . . , a p ∈ R d are independent samples from a distribution P ( a ) = 1 2 π ∫ e -j a ⊤ ∆ K (∆) d ∆ (Fourier transform of K (∆) ) and b 1 , . . . , b p are i.i.d. sampled from the uniform distribution on [0 , 2 π ] . For example, Gaussian kernel K ( x , x ′ ) = e - ∥ x -x ′ ∥ 2 2 2 σ has P ( a ) = N (0 , σ -2 I d ) . Denote A = [ a 1 , . . . , a p ] ⊤ and b = [ b 1 , . . . , b p ] ⊤ , then we have ϕ ( x ) = √ 2 p cos( A x + b ) . Whenever p ≤ d and A is rank p , given ϕ ( x ) we can solve x as,

To ensure the computed √ p 2 ϕ ( X S ) ∈ [ -1 , 1] , we can normalize it by its largest absolute value, which is equivalently scaling D in Theorem 5.1 and does not affect the direction of W S .

[21] use random Fourier features to approximate shif-invariant kernels that may have an infinitedimensional feature space, and construct p distilled data for such RFF model, where p ∈ Ω( √ n log n ) in general cases. Their construction, however, only uses label distillation and the X S can be any random data. Our analysis constructs X S explicitly and shows that whenever the dimension of RFF p needs to approximate shif-invariant kernels is less than d , k distilled data suffice to recover the performance of the original RFF model and approximate the original KRR with shif-invariant kernels.

5.2 Non-surjective Feature Mapping

When ϕ is injective or non-surjective, given a ϕ ( X S ) , we may not find an exactly matched X S . However, we can find an approximated distilled data ˆ X S first and then adjust Y S to ensure W S ≈ W . As in the label distillation for LRR case, m ≥ p distilled labels can guarantee W S = W .

Below we show one data per class is in general not sufficient for non-surjective ϕ , but k +1 can be sufficient for deep linear NNs. Consider a deep NN, f ( x ) = W ϕ ( x ) with

where σ is an invertible activation function such as LeakyReLU, Sigmoid, and W ( l ) ∈ R d l × d l -1 with d = d 0 < d 1 = · · · = d L = p for l ∈ [ L ] . ϕ ( x ) is an injective function in this definition.

Theorem 5.2. For a deep nonlinear NN defined above with fixed ϕ , assume W (2) , . . . , W ( L ) are full rank. Suppose λ S = 0 and Y S is rank k . When m = k , there is no distilled data X S that can guarantee W S = W in general useless the columns of σ -1 ( ( W (2) ) -1 · · · ( W ( L ) ) -1 σ -1 ( ( Y + S W ) + ) ) are in the range space of W (1) .

When m = k , only ϕ ( X S ) = ( Y + S W ) + that can guarantee W S = W . One data per class is not sufficient in general as long as ( Y + S W ) + is not in the range space of ϕ . Although k data is not sufficient, we show k +1 data can be sufficient for deep linear neural networks, where σ ( x ) = x .

Theorem 5.3. For a deep linear NN defined above with fixed ϕ , assume W (2) , . . . , W ( L ) are full rank. Suppose λ S = 0 and Y S , W are rank k . Denote H = [ ∏ L l =1 W ( l ) ( W (1) ) + ( W + W -I p ) ] ∈ R p × 2 p .

When m = k , there is no distilled data X S that can guarantee W S = W in general useless the columns of W + Y S are in the range space of ∏ L l =1 W ( l ) .

When m>k , If H is full rank and its right singular vectors V H ∈ R 2 p × 2 p 's last p × p submatrix is full rank, then there exists a X S such that W S = W .

Proof sketch. To find the distilled data theoretically, we need to guarantee 1) the feature ϕ ( X S ) need to guarantee W S = W and 2) there are some distilled data X S corresponding to the feature. For the first condition, Theorem 5.1 gives the sufficient and necessary condition of ϕ ( X S ) . However, the formulation involves a pseudoinverse of sum of matrices, which does not have a concise formulation and therefore is hard to handle when solving X S . Instead, in Theorem C.2, we derive a sufficient condition of ϕ ( X S ) without pseudoinverse. For the second condition, ϕ ( X S ) = ϕ ∗ for a given ϕ ∗ is an overdetermined system of linear equations of X S . We find the formulation of ϕ ( X S ) such that the overdetermined system has solutions. Then combining the two conditions together, we end up with an equation that has multiple free variables. Combing the variables together and solving the equation will give us the results. In the proof, we provide the algorithm to compute the distilled data when the assumptions are satisfied.

6 Applications

In this section, we show that our theoretical results can be useful in some applications. We first show the conditions in Theorem 5.1 can be necessary or sufficient conditions for KIP-type algorithms to converge in Sec 6.1. We also show our distilled dataset for KRR can provably preserve the privacy of the original dataset in Sec 6.2.

6.1 An Implication for KIP-type Algorithms

In KIP [25], FRePo [40], RFAD [17], and RCIG [18], a loss function as follows is optimized:

Below we show our results can be sufficient conditions for the above loss function to converge to 0 even if the loss function is highly non-convex.

Theorem 6.1. Suppose ϕ ( X ) is full rank and W is computed with λ = 0 . Then

when n ≤ p , the ϕ ( X S ) that can guarantee W S = W in Theorem 5.1 is sufficient for L ( X S ) = 0 .

when n ≥ p , the ϕ ( X S ) that can guarantee W S = W in Theorem 5.1 is necessary for L ( X S ) = 0 .

From this theorem, we see that KIP-type algorithms are enforcing W S = W to some extent. While the KIP algorithm is computationally expensive, our solution for ϕ ( X S ) can be computed efficiently and directly utilized in KIP-type algorithms for efficient optimization.

6.2 Privacy Preservation of Dataset Distillation

For our dataset distillation algorithm for KRR, we show that the original dataset cannot be recovered from the distilled dataset, therefore provably preserving the privacy of the original dataset while having the performance guarantees at the same time.

Proposition 6.1. Suppose n > k and Y is rank k . Given λ S , ϕ , for a distilled dataset ( X S , Y S ) that can guarantee W S = W in Theorem 5.1, we can reconstruct W from ϕ ( X S ) . However, given W , there are infinitely many solutions for ϕ ( X ) .

Since there are infinitely many solutions for ϕ ( X ) , it is impossible to recover ϕ ( X ) without additional information. As long as ϕ does not contain any information of X , then X will not be able to recover from distilled dataset ( X S , Y S ) . Note in Sec. 4.2, we use additional information (real images as reference points) to compute X S . Therefore X S resembles original images and we may recover these original images from X S . However, if we generate the distilled data with random noise, the distilled data will contain no additional information and protect the privacy of the original dataset. In

summary, we can control whether to generate realistic data by using real data as reference points or protect privacy by generating noisy distilled data as shown in Figure 2.

More formally, we prove that the distilled data can be differential private with respect to the original dataset if we take Z to be random Gaussian with suitable variance.

Theorem 6.2. Under the same setting of Theorem 4.1, suppose that λ = 0 , all data are bounded ∥ x i ∥ 2 ≤ B , and the smallest singular value of the original datasets is bounded from below σ min ( X ) > σ 0 . Suppose Y S is independent of X and unknown to the adversary. Let [ Y + S ] i denote its i -th row. Let ϵ, δ ∈ (0 , 1) and take the elements of Z ∼ N (0 , σ 2 ) with σ ≥ max i ∈ [ m ] 2 √ ln(1 . 25 /δ ) B ∥ [ Y + S ] i Y ∥ 2 σ 2 0 ϵ ∥ [ I m -Y + S Y S ] i ∥ 2 , then each row of X S is ( ϵ, δ ) -differential private with respect to X .

7 Experiments

(I) Analytical Computation of Dataset Distillation. In Table 3, we verify our theory of dataset distillation for LRR and KRR with subjective mapping. We compute the distilled dataset for different models using Algorithm 1. The models are KRR with different feature mappings: 1) identity mapping (linear model), 2) one-hidden-layer LeakyReLU neural network, and 3) Random Fourier Features (RFF) of Gaussian kernel. The feature mappings are constructed such that the feature dimension is equal to the data dimension, i.e. p = d . For NNs, we use random initialized ones and use the activations as feature mappings. As increasing the depth of NNs does not improve the performance, we only use one hidden layer. For simplicity, we set λ S = 0 for all experiments. To choose the original model's regularization λ , we split the original training set into a training set and a validation set, and choose the λ that performs best on the validation set. The results show that our analytically computed distilled dataset can indeed recover the original models' parameters and performance. Some slight differences are caused by the numerical error in recovering the data from features and computing the KRR solutions. As the purpose of this experiment is to verify if the distilled dataset can recover the performance of a specific original model, we did not report the error bars.

(II) Comparison with KIP. In Table 4, we compare our algorithm with KIP in terms of performance and efficiency under the setting of a subjective mapping. The test accuracy of models trained on distilled datasets and averaged computational cost (GPU Seconds) are reported. The mean and standard deviation of test accuracy are computed over four independent runs. As the experiment (I), we use a randomly initialized one-hidden-layer LeakyReLU NN with p = d as the feature mapping. For KIP, we implement their algorithm where we optimize a loss function (1) and use label distillation at each training step. For our results, we compute the distilled dataset using Algorithm 1. Our algorithm performs better than KIP on CIFAR-10 and CIFAR-100 while being significantly more efficient. We did not report the result of KIP with IPC=50 on CIFAR-100 because the estimated running time is more than 110 hours.

This experiment mainly aims to show that our theoretical guarantee can be transferred to practice. As the proposed Algorithm 1 is mainly for KRR with surjective mappings, we verified it and compared it with baselines in this setting. We use a randomly initialized bijective NN in order to match previous algorithms that use a randomly initialized NN. If a pre-trained NN is used, the accuracy can be improved and may match the SOTA.

(III) Privacy Protection. In this experiment, we show our algorithm can be used to protect the privacy of the original dataset. Same as experiment (II), we use a one-hidden-layer LeakyReLU neural network with p = d as the feature mapping and train a KRR model on the original dataset. Then we distill the dataset using Algorithm 1 and generate the distilled data with random Gaussian noise. As shown in Figure 3, the distilled data for MNIST are essentially random noise, which protects the privacy of the original MNIST dataset. At the same time, the model trained on it can recover the original model's performance of 93.87% test accuracy.

Figure 3: Distilled images of MNIST generated from random noise for a two-layer neural network. m = 10 (first row) and 100 (second row).

8 Conclusion and Future Works

In this paper, by focusing on dataset distillation for KRR, we show that one data point per class is already necessary and sufficient to recover the original model's performance in many settings. For linear ridge regression and KRR with surjective feature mappings, we provide necessary and sufficient conditions for the distilled dataset to recover the original model's parameters. For KRR with injective feature mappings of deep neural networks, we show that while one data point per class is not sufficient in general, k +1 data points can be sufficient for deep linear neural networks. Our theoretical results facilitate the direct construction of analytical solutions for distilled datasets, leading to a provable and efficient dataset distillation algorithm for KRR. Additionally, we have developed applications for KIP-type algorithms and privacy protection.

Several future research directions are worth exploring. First, while the current analysis shows that k data points are generally insufficient for non-surjective deep non-linear neural networks, determining the minimum number of distilled data points required remains an open question worthy of investigation. Second, this paper focuses on KRR with fixed feature mappings, which differs from some empirical works that train all neural network parameters. Extending the analysis to learnable feature mappings would bridge this gap and provide further insights.

References

Nicholas Carlini, Vitaly Feldman, and Milad Nasr. No free lunch in" privacy for free: How does dataset condensation help privacy". arXiv preprint arXiv:2209.14987 , 2022.

George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4750-4759, 2022.

Randall E Cline. Representations for the generalized inverse of sums of matrices. Journal of the Society for Industrial and Applied Mathematics, Series B: Numerical Analysis , 2(1):99-114, 1965.

Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation to imagenet1k with constant memory. In International Conference on Machine Learning , pages 6565-6590. PMLR, 2023.

Tian Dong, Bo Zhao, and Lingjuan Lyu. Privacy for free: How does dataset condensation help privacy? In International Conference on Machine Learning , pages 5378-5396. PMLR, 2022.

Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science , 9(3-4):211-407, 2014.

Yunzhen Feng, Shanmukha Ramakrishna Vedantam, and Julia Kempe. Embarrassingly simple dataset distillation. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=PLoWVP7Mjc .

Ziyao Guo, Kai Wang, George Cazenavette, HUI LI, Kaipeng Zhang, and Yang You. Towards lossless dataset distillation via difficulty-aligned trajectory matching. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/ forum?id=rTBL8OhdhH .

Zachary Izzo and James Zou. A theoretical study of dataset distillation. In NeurIPS 2023 Workshop on Mathematics of Modern Machine Learning , 2023. URL https://openreview. net/forum?id=dq5QGXGxoJ .

Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems , 31, 2018.

Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung-Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameterization. In International Conference on Machine Learning , pages 11102-11118. PMLR, 2022.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. arXiv preprint arXiv:1711.00165 , 2017.

Saehyung Lee, Sanghyuk Chun, Sangwon Jung, Sangdoo Yun, and Sungroh Yoon. Dataset condensation with contrastive signals. In International Conference on Machine Learning , pages 12352-12364. PMLR, 2022.

Zhu Li, Jean-Francois Ton, Dino Oglic, and Dino Sejdinovic. Towards a unified analysis of random fourier features. Journal of Machine Learning Research , 22(108):1-51, 2021.

Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. Efficient dataset distillation using random feature approximation. Advances in Neural Information Processing Systems , 35: 13877-13891, 2022.

Noel Loo, Ramin Hasani, Mathias Lechner, and Daniela Rus. Dataset distillation with convexified implicit gradients. In International Conference on Machine Learning , pages 22649-22674. PMLR, 2023.

Alaa Maalouf, Ibrahim Jubran, and Dan Feldman. Fast and accurate least-mean-squares solvers. Advances in Neural Information Processing Systems , 32, 2019.

Alaa Maalouf, Ibrahim Jubran, and Dan Feldman. Fast and accurate least-mean-squares solvers for high dimensional data. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(12):9977-9994, 2022.

Alaa Maalouf, Murad Tukan, Noel Loo, Ramin Hasani, Mathias Lechner, and Daniela Rus. On the size and approximation error of distilled sets. arXiv preprint arXiv:2305.14113 , 2023.

G. Marsaglia and G. P. H. Styan. When does rank(a+b)=rank(a)+rank(b)? Canadian Mathematical Bulletin , 15(3):451-452, 1972. doi: 10.4153/CMB-1972-082-8.

Lingsheng Meng and Bing Zheng. The optimal perturbation bounds of the moore-penrose inverse under the frobenius norm. Linear algebra and its applications , 432(4):956-963, 2010.

Timothy Nguyen, Zhourong Chen, and Jaehoon Lee. Dataset meta-learning from kernel ridge-regression. arXiv preprint arXiv:2011.00050 , 2020.

Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks. Advances in Neural Information Processing Systems , 34:5186-5198, 2021.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems , 20, 2007.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) , 115(3):211-252, 2015. doi: 10.1007/s11263-015-0816-y.

Noveen Sachdeva and Julian McAuley. Data distillation: A survey. Transactions on Machine Learning Research , 2023. ISSN 2835-8856. URL https://openreview.net/forum?id= lmXMXP74TO . Survey Certification.

James R Schott. Matrix analysis for statistics . John Wiley & Sons, 2016.

Felipe Petroski Such, Aditya Rawal, Joel Lehman, Kenneth Stanley, and Jeffrey Clune. Generative teaching networks: Accelerating neural architecture search by learning to generate synthetic training data. In International Conference on Machine Learning , pages 9206-9216. PMLR, 2020.

Murad Tukan, Alaa Maalouf, and Margarita Osadchy. Dataset distillation meets provable subset selection. arXiv preprint arXiv:2307.08086 , 2023.

Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12196-12205, 2022.

Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811.10959 , 2018.

Enneng Yang, Li Shen, Zhenyi Wang, Tongliang Liu, and Guibing Guo. An efficient dataset condensation plugin and its application to continual learning. Advances in Neural Information Processing Systems , 36, 2023.

Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective. Advances in Neural Information Processing Systems , 36, 2024.

Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning , pages 12674-12685. PMLR, 2021.

Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 6514-6523, 2023.

Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. arXiv preprint arXiv:2006.05929 , 2020.

Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regression. Advances in Neural Information Processing Systems , 35:9813-9827, 2022.

Appendices

Algorithm 1 Dataset distillation for kernel ridge regression Input: Number of distilled data m , number of classes k , regularization λ S , feature mapping ϕ , original model's parameter W , generate from original data or random noise 1: if Generate from original data then 2: Sample m balanced initialized data ˆ X S from original dataset. Initialize Y S as corresponding one-hot labels 3: Compute ϕ λ S ( ˆ X S ) 4: if Rank ( W ϕ λ S ( ˆ X S ) ) = k then 5: Y S = W ϕ λ S ( ˆ X S ) 6: end if 7: Construct Z = ( I m -Y + S Y S ) ( ϕ λ S ( ˆ X S ) + -Y + S W ) . 8: else if Generate from random noise then 9: Sample Z from random noise. Initialize Y S as balanced one-hot labels 10: end if 11: Compute D = Y + S W + ( I m -Y + S Y S ) Z 12: if λ S > 0 then 13: Compute the SVD of D , D = V diag ( σ ′ 1 , . . . , σ ′ r ) U ⊤ 14: if λ S > 1 4 σ ′ 2 1 then 15: λ S = 1 4 σ ′ 2 1 16: end if 17: Construct ϕ ( X S ) = U diag ( σ 1 , . . . , σ r ) V ⊤ where σ i = 0 if σ ′ i = 0 and σ i = 1 ± √ 1 -4 λ S σ ′ 2 i 2 σ ′ i otherwise. 18: else if λ S = 0 then 19: Construct ϕ ( X S ) = D + 20: end if 21: Construct X S from ϕ ( X S ) Output: X S , λ S

A Broader Impact

Our approach can be used to protect data privacy, which may have a positive societal impact. There are no particular ethical concerns we are aware of.

B Additional Experiment Details

All the experiments are implemented with PyTorch [26] and conducted on a single 24G A5000 GPU.

C Proofs for Linear Ridge Regularization

C.1 Analytical Computation for Linear Ridge Regression

Theorem 4.1. When m < k , there is no X S can guarantee W S = W unless the columns of W are in the range space of Y S . When m ≥ k and Y S is rank k , let r = min( m,d ) and take D = Y + S W + ( I m -Y + S Y S ) Z , where Z ∈ R m × d is any matrix of the same size as X ⊤ S . Suppose the reduced SVD of D is D = V diag ( σ ′ 1 , . . . , σ ′ r ) U ⊤ with σ ′ 1 ≥ · · · ≥ σ ′ r ≥ 0 , the following results hold:

1. λ S > 0 : W S = W if and only if, for any D defined above, λ S ≤ ′ 2 and X S

2. λ S = 0 : W S = W if and only if X S = D + for any D defined above.

Proof. Recall W S = Y S X + λ S where

Let W S = W ,

When m < k , this is an overdetermined system for X + λ S . There is no solution for X + λ S in general therefore no solution for X S unless all the columns of W are in the range space of Y S . When there is a solution, we can solve it as following m ≥ k cases.

In the following, we consider the m ≥ k case. Since m ≥ k and Y S is rank k , the solutions of X + λ S are given by

where Z ∈ R m × d is any matrix of the same size as X + λ S . When k = m , the solution is unique X + λ S = Y -1 S W . However, there are solutions for X + λ S does not mean there are solutions for X S . Next, we need to solve X S from X + λ S .

1. When m ≤ d . Suppose the reduced SVD of X S is X S = UΣV ⊤ , where Σ = diag ( σ 1 , . . . , σ m ) ∈ R m × m , V ∈ R m × m is a unitary matrix and U ∈ R d × m is the first m columns of a unitary matrix. Then when λ S > 0 ,

Combining (2) and (3), we must have

Denote D = Y + S W + ( I m -Y + S Y S ) Z . Given D , we can compute its reduced SVD D = V ′ diag ( σ ′ 1 , . . . , σ ′ m ) U ′⊤ with σ ′ 1 ≥ ··· ≥ σ ′ m . Note that SVD of a matrix is unique. Since

D = V ′ diag ( σ ′ 1 , . . . , σ ′ m ) U ′⊤ = V diag ( σ 1 σ 2 1 + λ S , . . . , σ m σ 2 m + λ S ) U ⊤ , we must have V = V ′ , U = U ′ , and

That is

̸

When σ ′ i = 0 , we have σ i = 0 . When σ ′ i = 0 and λ S ≤ 1 4 σ ′ 2 1 ≤ 1 4 σ ′ 2 i , it has solutions given by

Take the above computed U , V , and Σ , we can construct X S . Above shows such X S is a necessary condition for W S = W . To show the sufficiency, take such X S into W S .

which shows it is a sufficient condition.

When λ S = 0 , we have W S = Y S X + S . Let W S = Y S X + S = W . The solution for X + S is

where Z ∈ R m × d is any matrix of the same size as X ⊤ S . Therefore

Similarly, this is a necessary condition for W S = W . To show the sufficiency, take such X S into W S .

which shows it is a sufficient condition.

2. When m>d . Suppose the reduced SVD of X S is X S = UΣV ⊤ , where Σ = diag ( σ 1 , . . . , σ d ) ∈ R d × d , U ∈ R d × d is a unitary matrix and V ∈ R m × d is the first d columns of a unitary matrix. Then when λ S > 0 ,

Then we proceed similarly to the m ≤ d case. Last, we can unify two cases by taking r = min( m,d ) .

Proposition 4.1. When m ≥ k and Y S , W are rank k , the X S in Theorem 4.1 is full rank for any full-rank Z such that Range ( W ⊤ ) ∩ Range ( Z ⊤ ) = { 0 } .

Proof. X S is computed from D = Y + S W + ( I m -Y + S Y S ) Z . It is easy to check that X S is full rank if and only if D is full rank.

When Y S and W are rank k , Rank ( Y + S W ) = k . Since Rank ( I m -Y + S Y S ) = m -k and Z is full rank, by Sylvester's rank inequality,

For D = Y + S W + ( I m -Y + S Y S ) Z , since the columns of Y + S W ∈ Range ( Y ⊤ S ) and the columns of ( I m -Y + S Y S ) Z ∈ Null ( Y S ) . By the fundamental theorem of linear algebra, Range ( Y ⊤ S ) and Null ( Y S ) are orthogonal subspaces of R m . Therefore Range ( Y + S W ) ∩ Range (( I m -Y + S Y S ) Z ) = { 0 } . This can also be seen from ( Y + S W ) ⊤ ( I m -Y + S Y S ) Z = 0 , which shows their columns are orthogonal to each other. If we have Range ( W ⊤ ) ∩ Range ( Z ⊤ ) = { 0 } , then Range ( ( Y + S W ) ⊤ ) ∩ Range ( Z ⊤ ( I m -Y + S Y S )) = { 0 } . By [22],

Rank ( D ) = Rank ( Y + S W ) + Rank ( ( I m -Y + S Y S ) Z ) ≥ k +min( m,d ) -k = min( m,d ) . Therefore D is full rank and X S is full rank.

C.2 Characterization of Distilled Data without Pseudoinverse

In the last section, we give analytical solutions for X S that can guarantee W S = W . However, the expression of X S involves some pseudoinverse calculation and the explicit expression of X S remains unclear because there is no concise formulation for the pseudoinverse of sum of matrices. In this section, we give some direct characterization for X S .

Again, supposed its reduced SVD is X S = U diag ( σ 1 , . . . , σ r ) V ⊤ , where r = min( m,d ) . When λ S > 0 , from Eq. (3) and Eq. (4), X + λ S = D = V diag ( σ 1 σ 2 + λ , . . . , σ r σ 2 + λ S ) U ⊤ and X λ S =

0 . When λ S = 0 , X λ S = X S and X λ S = X S . Given X λ S and λ S , we can easily compute X S by SVD. Below we give conditions for W = W through X .

1 S r [ ] i,i + + S λ S

Suppose the eigenvalues of X λ S are σ ′ i . Then by the definition of X λ S , σ i + λ S σ i = σ ′ i if σ i > 0 . That is

Only when σ ′ 2 i ≥ 4 λ S , there are solution(s) σ i = σ ′ i ± √ σ ′ 2 i -4 λ S 2 . Therefore, to make sure there is a X S corresponds to X λ S , the nonzero singular values of X λ S need to be larger than or equal to 2 √ λ S . When λ S = 0 , there is no requirement.

Theorem C.1. Suppose k ≤ d and W is rank k . Take

where Z ′ ∈ R d × m is any matrix of the same size as X λ S such that X λ S is full rank.

When m ≤ d , it is a necessary condition for W S = W .

When m ≥ d , it is a sufficient condition for W S = W .

Proof. Case 1. When m ≤ d , recall that W S = Y S X + λ S . Set the parameter to be the same W S = W and try to solve X λ S

Multiply X λ S on both sides. Since X λ S is full rank and X + λ S X λ S = I m , we have

Since k ≤ d and W is rank k , X S has infinite many solutions. The general solutions are

where Z ′ ∈ R d × m is any matrix of the same size as X S . Therefore, this is a necessary condition for W S = W .

Case 2. When m ≥ d , to show the sufficiency, for any X λ S = W + Y S +( I d -W + W ) Z ′ ,

Multiply X + λ S on both sides. Since X λ S X + λ S = I d ,

From these two cases, we can also conclude that when m = d , such X λ S = W + Y S + ( I d -W + W ) Z ′ is a sufficient and necessary condition for W S = W .

Below we give a sufficient condition of X λ S when m ≥ k . It will be used in the proof of Theorem 5.3. Theorem C.2. When m ≥ k , a sufficient condition for W S = W is Y S is rank k and

where Z ′ ∈ R d × m is any matrix of the same size as X λ S .

Proof. When m ≥ k , for the sufficient condition X λ S = W + Y S + ( I d -W + W ) Z ′ ( I m -Y + S Y S ) , denote A = W + Y S and B = ( I d -W + W ) Z ′ ( I m -Y + S Y S ) . Since A ⊤ B = 0 and AB ⊤ = 0 , by [3], ( A + B ) + = A + + B + . Therefore

where the last equality is because W and Y S are full rank and therefore ( W + Y S ) + = Y + S W . From this, we have

For any matrix A and B , if AB = 0 then B + A + = 0 [30]. Since ( I d -W + W ) Z ′ ( I m -Y + S Y S ) Y + S = 0 , Y S [ ( I d -W + W ) Z ′ ( I m -Y + S Y S )] + = 0 . Therefore we conclude W S = W . Note in this case, we do not require X λ S to be full rank.

C.3 Finding Realistic Distilled Data

Corollary 4.1.1. Given fixed ˆ X S , λ S , and Y S , the D that satisfies Theorem 4.1 and minimize ∥ ∥ ∥ D -ˆ X + λ S ∥ ∥ ∥ F is

where ˆ X + λ S is defined analogous to X + λ S . Taking Y S = W ˆ X λ S can further minimize the distance.

Proof. Given fixed ˆ X S , λ S , and Y S , the linear ridge regression trained on ˆ X S is

By Theorem 4.1, to ensure W S = W , we need ˆ X + λ S to be equal to some D = Y + S W + ( I m -Y + S Y S ) Z , where Z is a free variable to be determined. Therefore let

That is

Since ( I m -Y + S Y S ) is idempotent and therefore singular, Z does not have a solution in general (because the system of equations can be inconsistent). The least-squares solution is

where one can verify that ( I m -Y + S Y S ) + = I m -Y + S Y S by SVD. This least-squares solution minimize ∥ ∥ ∥ ( I m -Y + S Y S ) Z -ˆ X + λ S + Y + S W ∥ ∥ ∥ F = ∥ ∥ ∥ D -ˆ X + λ S ∥ ∥ ∥ F . Take such Z into D , we have

Then we have the difference between D and ˆ X + λ S is

To further minimize the difference, we can let Y S ˆ X + λ S = W . The least square solution is Y S = W ˆ X λ S .

C.4 Label Distillation

Theorem 4.2. For any fixed X S ,

when m < d , there is no Y S can guarantee W S = W in general unless the rows of W are in the row space of X + λ S . The least square solution is Y S = WX λ S and ∥ W S -W ∥ = ∥ ∥ W ( X λ S X + λ S -I d )∥ ∥ .

when m ≥ d , if X S is rank d , then Y S = WX λ S is sufficient for W S = W .

Proof. When m<d , let

Therefore we have

and solve Y S . Y S does not have a solution in general unless the equations are consistent, i.e. the rows of W are in the row space of X + λ S . The least-squares solution is

Then we can bound the difference between W S and W ,

When m ≥ d , let

and solve Y S . Since X S is rank d , then X + λ S = X ⊤ S ( X S X ⊤ S + λ S I d ) -1 is rank d and Y S has solutions. Take the minimum norm one,

To show the sufficiency, take Y S into W S .

D Proofs for Kernel Ridge Regression

Theorem 5.1. When m < k , there is no ϕ ( X S ) can guarantee W S = W unless the columns of W are in the range space of Y S . When m ≥ k and Y S is rank k , let r = min( m,p ) and take D = Y + S W + ( I m -Y + S Y S ) Z , where Z ∈ R m × p is any matrix of the same size as ϕ ( X S ) ⊤ . Suppose the reduced SVD of D is D = V diag ( σ ′ 1 , . . . , σ ′ r ) U ⊤ with σ ′ 1 ≥ ·· · ≥ σ ′ r ≥ 0 , the following results hold:

1.

λ

S

>

0

:

W

S

=

W

if and only if, for any

D

defined above,

λ

S

λ S = 0 : W S = W if and only if ϕ ( X S ) = D + for any D defined above.

Proof. The proof is same as Theorem 4.1 but just replace X S with ϕ ( X S ) .

D.1 Deep Nonlinear Neural Networks

Theorem 5.2. For a deep nonlinear NN defined above with fixed ϕ , assume W (2) , . . . , W ( L ) are full rank. Suppose λ S = 0 and Y S is rank k . When m = k , there is no distilled data X S that can guarantee W S = W in general useless the columns of σ -1 ( ( W (2) ) -1 · · · ( W ( L ) ) -1 σ -1 ( ( Y + S W ) + ) ) are in the range space of W (1) .

Proof. To get a distilled data X S that can guarantee W S = W , the sufficient and necessary condition is that

ϕ ( X S ) need guarantee W S = W .

There is some X S corresponds to such ϕ ( X S ) . Equivalently X S is recoverable from ϕ ( X S ) .

For the first condition, when λ S = 0 , we have shown in Theorem 5.1, ϕ ( X S ) has to be

When m = k and Y S is rank k , this reduce to ϕ ( X S ) = ( Y + S W ) + .

2. For the second condition, given ϕ ( X S ) = σ ( W ( L ) · · · σ ( W (1) X S )) , solving X S is same as solving

When m = k and combined with the first condition, it becomes

Since this is an over-determined system of linear equations and RHS is fixed, It does not have a solution in general unless The RHS is in the range space of W (1) .

D.2 Deep Linear Neural Networks

Theorem 5.3. For a deep linear NN defined above with fixed ϕ , assume W (2) , . . . , W ( L ) are full rank. Suppose λ S = 0 and Y S , W are rank k . Denote H = [ ∏ L l =1 W ( l ) ( W (1) ) + ( W + W -I p ) ] ∈ R p × 2 p .

When m = k , there is no distilled data X S that can guarantee W S = W in general useless the columns of W + Y S are in the range space of ∏ L l =1 W ( l ) .

1

σ

′

2

1

and

ϕ

(

X

S

) =

4

When m>k , If H is full rank and its right singular vectors V H ∈ R 2 p × 2 p 's last p × p submatrix is full rank, then there exists a X S such that W S = W .

Proof. To get a distilled data X S that can guarantee W S = W , the sufficient and necessary condition is that

ϕ ( X S ) need guarantee W S = W .

There is some X S corresponds to such ϕ ( X S ) . Equivalently X S is solvable from ϕ ( X S ) .

For the first condition, when λ S = 0 , we have shown in Theorem 5.1, ϕ ( X S ) has to be

When m = k and Y S , W are rank k , this reduce to ϕ ( X S ) = W + Y S . When m ≥ k , from Theorem C.2, we know ϕ ( X S ) = W + Y S +( I p -W + W ) Z ′ ( I m -Y + S Y S ) for any Z ′ ∈ R p × m is also a sufficient condition for W S = W .

For the second condition, given ϕ ( X S ) = ∏ L l =1 W ( l ) X S , solving X S is same as solving

This is an over-determined system of linear equations. A necessary and sufficient condition for any solution(s) to exist is that RHS is in the range space of W (1) or equivalently X S = ( W (1) ) + ( ∏ L l =2 W ( l ) ) -1 ϕ ( X S ) is a solution. Take this solution into equation,

Solve the equation for ( ∏ L l =2 W ( l ) ) -1 ϕ ( X S ) , we have

for any Z 1 ∈ R p × m . Therefore to guarantee X S is solvable from ϕ ( X S ) , ϕ ( X S ) have to be in the form of

In this case, X S = ( W (1) ) + Z 1 . For any Z 1 , W (1) ( W (1) ) + is a projector that projects Z 1 to the range space of W (1) .

Combing two conditions (7) and (8) , we need to solve

for Z and Z 1 .

When m = k , it becomes

Since ∏ L l =1 W ( l ) ∈ R p × d and p > d , the equation has a solution only when W + Y S is in the range space of ∏ L l =1 W ( l ) .

When m > k , from Theorem C.2, we know ϕ ( X S ) = W + Y S + ( I p -W + W ) Z ′ ( I m -Y + S Y S ) for any Z ′ ∈ R p × m is a sufficient condition for W S = W . Therefore we can instead solve

Combine the variables Z ′ and Z 1 , we have

Denote H = [ ∏ L l =1 W ( l ) ( W (1) ) + ( W + W -I p ) ] ∈ R p × 2 p . If H is full rank (rank p ), then the solutions are

where Z 2 ∈ R 2 p × m is any matrix. For any RHS, we can find a solution for Z 1 . Next, we try to find a solution for Z ′ and Z 2 such that the equation is consistent. Suppose the full SVD of H = UΣV ⊤ , where U ∈ R p × p , Σ ∈ R p × 2 p , V ∈ R 2 p × 2 p . Then I 2 p -H + H = V diag (0 , . . . , 0 ︸ ︷︷ ︸ p , 1 , . . . , 1 ︸ ︷︷ ︸ p ) V ⊤ = [0 . . . 0 V p +1 . . . V p ] V ⊤ . Then the equation becomes

Denote B = V ⊤ Z 2 =    b ⊤ 1 . . . b ⊤ 2 p    ∈ R 2 p × m . Since Z 2 is solvable from any B , we can instead solve

B . Then the equation is

Denote H + W + Y S = [ C 1 C 2 ] where C 1 , C 2 ∈ R p × m are the first and last p rows. Then the equation can be partitioned into two parts

where [ V p +1 . . . V 2 p ] 1: p denotes its first p rows and [ V p +1 . . . V 2 p ] p +1:2 p denotes its last p rows. For the first equation, there is always a solution for Z 1 . So we will mainly

care about if there is a solution for the second equation. For the second equation, when [ V p +1 . . . V 2 p ] p +1:2 p is full rank, then there is always a solution for    b ⊤ p +1 . . . b ⊤ 2 p    and therefore solutions for B and Z 2 . The equations do not depend on b 1 , . . . , b p so they can be anything. In conclusion, when [ V p +1 . . . V 2 p ] p +1:2 p is full rank, there is a solution for Z 1 and X S . To construct X S , take any Z ′ and ϕ ( X S ) = W + Y S + ( I p -W + W ) Z ′ ( I m -Y + S Y S ) . Then construct H = [ ∏ L l =1 W ( l ) ( W (1) ) + ( W + W -I p ) ] ∈ R p × 2 p and its SVD H = UΣV ⊤ . If [ V p +1 . . . V 2 p ] p +1:2 p is full rank, solve (10),

Then we get Z 1 from (9)

Then we can construct X S = ( W (1) ) + Z 1 .

D.3 Additional Trainable Layer

Here we consider whether adding an additional trainable layer to the distilled dataset model will help dataset distillation. Suppose original model is f ( x ) = W ϕ ( x ) and distilled dataset model is f S ( x ) = W S A ϕ ( x ) where A ∈ R p × p is the additional trainable layer. In this case, the feature of distilled dataset model is A ϕ ( x ) instead of ϕ ( x ) and the analytical solution for W S becomes W S =

Theorem D.1. When k ≤ m<p and λ S > 0 , suppose Y S is rank k , there exists a distilled dataset ( X S , Y S ) can guarantee W S A = W if below equation has a solution for some Z ∈ R m × p such that ϕ ( X S ) is full rank and some c > 0 :

Proof. What we want now is W S A = W . That is

For a given Y S , since k ≤ m , the solution of LHS is

for any Z ∈ R m × p . Denote RHS as D = Y + S W + ( I m -Y + S Y S ) Z and multiply the inverse on both sides,

Arrange the terms,

If there exists a D such that I p -ϕ ( X S ) D is rank p ,

Here ϕ ( X S ) ⊤ ∈ R m × p . Since m<p , there are solutions for A ⊤ A . Take the minimum norm one:

Since A ⊤ A is symmetric and positive semidefinite, the RHS also needs to be positive semidefinite. A sufficient condition is that D ( I p -ϕ ( X S ) D ) -1 = cϕ ( X S ) + for some constant c > 0 . Solve it we get D = c c +1 ϕ ( X S ) + . In this case, I p -ϕ ( X S ) D = I p -c c +1 ϕ ( X S ) ϕ ( X S ) + is indeed full rank.

By the definition of D and D = c c +1 ϕ ( X S ) + , the problem boils down to

Compared with Eq. (7), adding one additional trainable layer only relaxes the original equation with constant scaling and does not help too much.

E Applications

E.1 An Implication for KIP-type Algorithms

Theorem 6.1. Suppose ϕ ( X ) is full rank and W is computed with λ = 0 . Then

when n ≤ p , the ϕ ( X S ) that can guarantee W S = W in Theorem 5.1 is sufficient for L ( X S ) = 0 .

when n ≥ p , the ϕ ( X S ) that can guarantee W S = W in Theorem 5.1 is necessary for L ( X S ) = 0 .

Proof. The ϕ ( X S ) in Theorem 5.1 guarantees W S = W . Since λ = 0 and ϕ ( X ) is full rank, W = Y ϕ ( X ) + . Therefore we have

When n ≤ p , multiply ϕ ( X ) on both sides,

which implies L ( X S ) = ∥ W S ϕ ( X ) -Y ∥ 2 = 0 . This shows that ϕ ( X S ) in Theorem 5.1 is sufficient for L ( X S ) = 0 .

When n ≥ p , the L ( X S ) = 0 means

Multiply ϕ ( X ) + on both sides, we have

This implies that W S = W is a necessary condition for L ( X S ) = 0 . Since the ϕ ( X S ) in Theorem 5.1 is sufficient and necessary for W S = W . Therefore ϕ ( X S ) in Theorem 5.1 is necessary for L ( X S ) = 0 .

E.2 Privacy Preservation of Dataset Distillation

Proposition 6.1. Suppose n > k and Y is rank k . Given λ S , ϕ , for a distilled dataset ( X S , Y S ) that can guarantee W S = W in Theorem 5.1, we can reconstruct W from ϕ ( X S ) . However, given W , there are infinitely many solutions for ϕ ( X ) .

Proof. Since W = W S , we can reconstruct W by simply compute W S .

When n > k , since W = Y ϕ λ ( X ) + and Y is rank k , given W , there are infinitely many solutions for ϕ λ ( X ) + ,

for any Z ∈ R n × p . Using a similar approach as the proof of Theorem 4.1, we can solve ϕ ( X ) by SVD and there are infinitely many solutions for ϕ ( X ) . Therefore it is impossible to recover ϕ ( X ) without additional information.

Theorem 6.2. Under the same setting of Theorem 4.1, suppose that λ = 0 , all data are bounded ∥ x i ∥ 2 ≤ B , and the smallest singular value of the original datasets is bounded from below σ min ( X ) > σ 0 . Suppose Y S is independent of X and unknown to the adversary. Let [ Y + S ] i denote its i -th row. Let ϵ, δ ∈ (0 , 1) and take the elements of Z ∼ N (0 , σ 2 ) with σ ≥ max i ∈ [ m ] 2 √ ln(1 . 25 /δ ) B ∥ [ Y + S ] i Y ∥ 2 σ 2 0 ϵ ∥ [ I m -Y + S Y S ] i ∥ 2 , then each row of X S is ( ϵ, δ ) -differential private with respect to X .

Proof. We prove that D = Y + S W + ( I m -Y + S Y S ) Z is ( ϵ, δ ) -differential private with respect to X using the Gaussian mechanism [6]. Then it follows that X S is ( ϵ, δ ) -differential private since the computation from D to X S is deterministic and independent with X .

Wefirst show the sensitivity of Y + S W is bounded and then use ( I m -Y + S Y S ) Z as random Gaussian to apply the Gaussian mechanism. Without loss of generality, suppose we have two datasets X = [ x 1 , x 2 , . . . , x n ] , X ′ = [ x ′ 1 , x 2 , . . . , x n ] ∈ R d × n that differ only in the first point and their resulting parameters are W and W ′ . Since λ = 0 , W = YX + . For each row of Y + S W , we have

where the third inequality is due to Theorem 2.2 in Meng and Zheng [23]. Therefore the sensitivity of [ Y + S ] i W is bounded. Suppose the elements of Z ∼ N (0 , σ 2 ) , the elements of ( I m -Y + S Y S ) Z are also Gaussian. The i -th row of ( I m -Y + S Y S ) Z is

whose elements are independent Gaussian with the variance ∥ ∥ [ I m -Y + S Y S ] i ∥ ∥ 2 2 σ 2 . Therefore, for each row [ Y + S ] i W + [ I m -Y + S Y S ] i Z , we can apply the Gaussian mechanism. Let ϵ ∈ (0 , 1) , by Theorem 3.22 in [6], as long as

[ Y + S ] i W + [ I m -Y + S Y S ] i Z is ( ϵ, δ ) -differential private. Take σ to be the maximum one so that all the rows are ( ϵ, δ ) -differential private.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims made match the theoretical and experimental results

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See for example the last paragraph in Section 8.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The assumptions are stated in the Theorems and all the proofs are in the Appendix. Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The experiment details are provided in Section 7.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: The experiment details are provided in the paper. We will release the code once the paper is published.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experiment details are provided in Section 7.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: See for example Table 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See Section 7 and Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: There are no particular ethical concerns we are aware of.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Our approach can be used to protect data privacy, which may have a positive societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper will not release models and datasets that have a high risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cited the datasets and packages used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents provable algorithms for computing distilled sets for various problems.\n\nStarting with LRR, they prove that it is enough to have m = number of classes to compute an exact solution as on the whole data.  Then they directly show how to make this distilled point \"Realistic\" by initializing m points from the data and computing matrix D (the distilled data) to satisfy the conditions required for the distilled set while being close to these m points.\n\nThen they prove that distilling the labels only can yield the same performance with d points (which is notably immediate from [16] which basically proved the same for the more generic case of KRR via RFF).\n\nSection 5 is an extension of the previous results from LRR to KRR which is done in the same way after swapping the original feature with the mapped features. However, what was missing was how to construct the distilled set of points in the original space from the mapped distilled points -- which is easy if \\phi is bijective. But when it is not, the authors show how to at least compute an approximation for these points.  \n\nSection 6 is crucial to draw a connection as to why practical algorithms were working in the first place!\n\nMy summary:\n\n* I like the contribution of this paper, and I think it is super crucial to have theoretical results in this practical field.\n* However, I have some major and minor concerns and questions I wish to solve in order to accept that paper, specifically about the correctness of the theorems and experiments.\n* I am willing to increase my score once my questions/concerns are addressed.\n\nSTRENGTHS:\n* Provides important provable guarantees and algorithms for the distillation work which was mainly practical.\n\n* It is very important to provide a better understanding of distillation and why it works in the first place -- a gap that has been missing for so long.\n\n* The method is very efficient which is very useful.\n\nWEAKNESSES:\nMy main concerns can be split into three:\n\n*a. The framing and guarantees:*\n\n1. why is \\lamba_s not equal to \\lamda? How do you formulate the distillation with both \\lambda and \\lambda_S?\n\n2. For theorem 4.1 (line 118), can you provide an explanation why \\lambda_S \\leq 1/(4\\alpha'^2). This seems to be a very rare case not related to practice. Or do you suggest solving the problem of LRR on the small data with another lambda? if so, how can it be determined?\n\n3. Same question as before for theorem 5.1. How do you find these lambda_s?\n\n4. Section 5.2, I did not fully understand the result, if \\phi is injective, can you guarantee W_s =W with k+1 points? if yes do you prove the existence of such a set or also provide an algorithm to compute it?\n\nb. *Experimental results:*\n\n5. Table 4 in the experimental results is not clear and might have some issues. For example \"https://proceedings.neurips.cc/paper_files/paper/2022/file/5a28f46993c19f428f482cc59db40870-Paper-Conference.pdf\" and https://arxiv.org/pdf/2307.08086 conducted similar experiments and got much higher results both for KIP and their methods. Could you elaborate on why the results are different or at least why did you use different architectures?\n\n\n*c. Simple comment but important:*\n\n6. Related work is missing specifically, for LRR:  \"https://proceedings.neurips.cc/paper/2019/hash/475fbefa9ebfba9233364533aafd02a3-Abstract.html\"  showed how to construct a set of d^2 points with the same exact solution and is a subset of the input,*  then, https://ieeexplore.ieee.org/abstract/document/9677460 extended the result and showed how to compute a distilled set (not subset) with only d point and yield the same solution.  \n* For KRR https://arxiv.org/pdf/2307.08086 also showed the first basic theoretical guarantees for full network distillation.\n* https://arxiv.org/pdf/2406.04284 provided research on the information and dynamics of dis-tilled data post-distillation process\n\n________________________\nFollowing the rebuttal (response) from the authors, my concerns have been addressed, I am raising my score.\n\nQUESTIONS:\nPlease see all my questions in the *Weaknesses* section\n\nLIMITATIONS:\nI suggest the authors to clearly frame the distillation problem and what is the goal of it-- explaining the confusing \\lambda_s and stating the main results clearer.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study theoretical aspects of dataset distillation for kernel ridge regression. They first provide analytical results for KRR with a k-dimensional label and linear features. They then extend these results to certain types of finite-dimensional feature maps, including feature maps which can be computed by a certain class of neural networks. They also study the privacy properties of their distilled datasets. They conduct experiments on standard benchmarks to verify their theory and show the computational advantage over another DD method designed for kernel regression (KIP).\n\nSTRENGTHS:\nDataset distillation (DD) is a relevant topic for the community, as it offers the potential to provide significant cost reductions in terms of both model training and data storage. This seems especially crucial as the size of both models and training data continues to grow. Furthermore, most work on DD has been empirical leaving the theoretical aspect under-studied. Theoretical insights may be helpful in moving towards more effective DD methods, as well as understanding the limits of what can be expected from DD.\n\nI generally found the writing clear and easy to follow.\n\nWEAKNESSES:\nThe main claims of the paper seem to be somewhat trivial extensions of known results [1] (reference [6] in the submitted manuscript) for linear regression. There are three extensions as compared to [1]:\n1. The present paper consider a k-dimensional label instead of a 1-dimensional real-valued label. The k-dimensional case can be derived with essentially the same techniques as the 1-dimensional case, as all the same formulas for ridge-regularized linear regression still hold.\n2. The present paper studies additional possibilities for distilled datasets when more than the minimum number of points are used. Most of these results are again direct consequences of applying the formulas used in point 1. The authors also motivate the use of more than the minimum number of points in the distilled dataset as a way of enforcing privacy, but there are problems with this claim as well (see below). If privacy is not preserved, then it's not clear what is the value in studying a larger-than-necessary distilled dataset.\n3. The present paper gives positive results for kernel ridge regression with finite-dimensional feature maps, whereas [1] only gave a negative result for kernel regression. However, these results are again a trivial extension of the linear case: the results of the linear setting hold directly in the feature space, and then some additional assumptions are made (e.g., surjectivity of the feature map) which allow one to recover points in the original data space mapping to the required features. This doesn't seem to provide much additional insight.\n\nOn the other hand, the present paper only studies the problem of distilling a single model from the data. As some of the primary motivating examples for the usefulness of DD are neural architecture search/hyperparameter tuning or continual learning, some theoretical results related to these problems would be of greater interest (and these questions were also studied by [1]).\n\nThe claims that the proposed method preserves the privacy of the original dataset have also not been adequately supported. Specifically, the authors show in Theorem 6.2 that there are infinitely many possible original datasets which could have led to the learned weights W. This fact is not sufficient to claim that privacy is preserved. For instance, in an image classification setting, Theorem 6.2 does not rule out the possibility that if we restrict the original dataset to contain only realistic images, then only certain images can lead to the final model weights W. This would then constitute a privacy breach. Put another way, Theorem 6.2 is essentially claiming that model training is inherently private since the map from data to model weights is not injective. This is at odds with the consensus of the ML community, since there is a great deal of literature dedicated to making the training of ML models private (e.g. [2] has over 6000 citations).\n\n**References:**\n\n[1] Izzo, Zachary and James Zou. \"A theoretical study of dataset distillation.\" NeurIPS Workshop on Mathematics of Modern Machine Learning. 2023.\n\n[2] Abadi, Martin, et al. \"Deep learning with differential privacy.\" Proceedings of the ACM SIGSAC conference on computer and communications security. 2016.\n\nQUESTIONS:\nDo the results in the paper have implications for practical uses of DD, e.g. hyperparameter tuning/NAS or continual learning?\n\nIs there a formal privacy definition (e.g., differential privacy) which the proposed DD method obeys?\n\nLIMITATIONS:\nSome limitations are discussed in the Future Work section. Potential negative societal impact is N/A.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presented a theoretical framework of dataset distillation for KRR, showing that (slightly more than) one data point per class is often necessary and sufficient to recover the performance of the original model. The theory led to a provable and efficient algorithm for dataset distillation, whose effectiveness was verified through experiments.\n\nSTRENGTHS:\n- The introduction provides a concise and clear view of the related work on dataset distillation and the contributions of the theoretical findings. The following setup and results are well-organized.\n- The theoretical results for the linear and surjective kernel cases are strong, both in terms of improving the existing theories and in terms of providing insights for designing efficient algorithms. \n- The visualizations are straightforward and helpful for understanding, e.g., Figure 2 provides good support for the choice of $Z$ in Corollary 4.1.1.\n\nWEAKNESSES:\n- When comparing with the existing theories, in addition to the direct comparison of the required number of distilled data (e.g., $k$ v.s. $d$), it would be helpful to provide more intuitions on where such significant improvement comes from. Whether it is due to specific settings or assumptions or due to new techniques or insights?\n- It is slightly confusing to refer to the feature map $\\phi$ as \"surjective\", especially in the introduction without clear definitions. It would be helpful to provide more intuitions why a surjective feature map matters.\n- It seems that the analysis is relatively limited to the linear and surjective kernel cases, whereas the non-surjective kernel (or neural network) case is a direct extension of the surjective kernel results and relies on several strong assumptions like invertible activation (which excludes ReLU) and full-rank weight matrices.\n\nQUESTIONS:\n- While Theorem 6.2 implies that it's impossible to *exactly* recover $\\phi(X)$ from $\\phi(X_S)$ and $W$, from Figure 2, it seems that when $Z$ is chosen to be (an affine transform of) real trained data $\\hat{X}_S$, $X_S$ can still reveal *most semantic information* of $X$. This is slightly confusing and would be helpful to clarify.\n- Figure 3 shows that changing the arbitrary matrix $Z$ from (an affine transform of) real trained data $\\hat{X}_S$ to a Gaussian random matrix leads to a private dataset distillation, at least to human eyes. But from the expression $D = Y_S^\\dagger W + (I - Y_S^\\dagger Y_S) Z$, it seems that such private distilled data are semantic information + Gaussian noise. With prior knowledge of (the distribution of) $Z$, can standard denoising techniques be applied to recover the semantic information?\n\nLIMITATIONS:\nThe major limitations are well-discussed in future works. Some further limitations are mentioned in Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the number of synthetic data required in dataset distillation to ensure that the optimal parameter learned on the synthetic dataset is the same of that learned on the original dataset.\n\nThe task considered is kernel ridge regression (KRR), where the labels contain $k$ classes. The goal is to preserve the optimal $W$ in $\\min_W \\|W\\phi(X) - Y\\|_F^2 + \\lambda \\|W\\|_F^2$.\nLet $m$ denote the number of synthetic samples in the distilled dataset.\n- For linear ridge regression (LRR), the paper finds that $m = k$ suffice to recover the parameter.\n  - The paper assumes access to $W \\in \\mathbb{R}^{k \\times d}$. Let $Y_S \\in \\mathbb{R}^{k \\times m}$ be any choice of synthetic labels such that it has rank $k$.\n  - The proof works by essentially setting $X_S = W^\\dagger Y_S$.\n  - If we want the synthetic data to be realistic, the paper suggests to set $Y_S = W\\hat{X}_S$ where $\\hat{X}_S$ are sampled from the real dataset -- this means $X_S = \\hat{X}_S$. In other words, $W$ can be preserved from a subset of $k$ samples in the real data set, as long as these samples span a rank-$k$ space.\n- For KRR with surjective feature mappings $\\phi$, $m = k$ samples also suffice to recover the parameter.\n  - Examples of surjective feature mappings include invertible NNs, fully-connected nets with invertible activation and full-rank parameters, and random Fourier features with full-rank parameters.\n  - Proof idea: first, note that recovering the linear weight on top of the feature $\\phi(X)$ is the same as LRR. The next step is then to recover $X$ from $\\phi(X)$, which is feasible when $\\phi$ is invertible.\n- For KRR with injective feature mappings (e.g. given by NNs), $k$ samples are not sufficient in general. For deep linear NNs, $k+1$ samples suffice.\n\nSTRENGTHS:\n- The paper provides a theoretical analyses on the minimal number of samples required for a synthetic dataset to preserve the parameter to a ridge regression problem.\n- The conditions proposed in this paper can be used to ensure 0 loss in some objectives proposed in prior work.\n- The paper empirically verify the number of samples required to, and is significantly faster than KIP (Kernel Inducing Points, a method in prior work).\n\nWEAKNESSES:\n- The method proposed in this paper requires knowing $W$, which is not always feasible. Moreover, the algorithm explicitly uses $W$, which means that the computational time grows with the feature dimension, which could be infinite.\n    - In contrast, KIP uses the kernel trick and hence does not incur a dependency on the feature dimension.\n- I'm concerned about the scalability of the method: although the synthetic dataset can preserve the performance on the full dataset, the accuracy is low (e.g. 42% on CIFAR10). I wonder if the method can preserve the performance of SOTA models on CIFAR10 (e.g. reaching >80% accuracy) and CIFAR100 (e.g. reaching >60% accuracy).\n- Some conditions in the theorem statements and wordings in the proofs may be problematic. Please see questions below.\n\nQUESTIONS:\n- In the proof of Thm 5.2, the second bullet point says there existing some $X_S$ corresponding to $\\phi(X_S)$ is equivalent to $X_S$ being recoverable from $\\phi(X_S)$. I don't see why the equivalency is true; for example, $\\phi$ could map all samples from the same class to a same feature embedding. What am I missing?\n- Line 578: \"for __any__ $Z_1$\" should be \"there exists some $Z_1$\"?\n- The choice of $Z$ is \"any matrix of the correct size\" at multiple places, whereas Thm C.1 additionally requires it to make $X_{\\lambda_S}$ full rank. Please fix the conditions.\n- Thm 6.1: $n$ should be $m$?\n- For completeness of the paper, please include a brief explanation of the baseline KIP.\n\nLIMITATIONS:\nThere is no direct societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}