Dataset Meta-Learning from Kernel Ridge-Regression

One of the most fundamental aspects of any machine learning algorithm is the training data used by the algorithm . We introduce the novel concept of - approximation of datasets , obtaining datasets which are much smaller than or are significant corruptions of the original training data while maintaining similar model performance . We introduce a meta-learning algorithm called Kernel Inducing Points ( KIP ) for obtaining such remarkable datasets , inspired by the recent developments in the correspondence between infinitely-wide neural networks and kernel ridge-regression ( KRR ) . For KRR tasks , we demonstrate that KIP can compress datasets by one or two orders of magnitude , significantly improving previous dataset distillation and subset selection methods while obtaining state of the art results for MNIST and CIFAR-10 classification . Furthermore , our KIP -learned datasets are transferable to the training of finite-width neural networks even beyond the lazy-training regime , which leads to state of the art results for neural network dataset distillation with potential applications to privacy-preservation . 1 INTRODUCTION . Datasets are a pivotal component in any machine learning task . Typically , a machine learning problem regards a dataset as given and uses it to train a model according to some specific objective . In this work , we depart from the traditional paradigm by instead optimizing a dataset with respect to a learning objective , from which the resulting dataset can be used in a range of downstream learning tasks . Our work is directly motivated by several challenges in existing learning methods . Kernel methods or instance-based learning ( Vinyals et al. , 2016 ; Snell et al. , 2017 ; Kaya & Bilge , 2019 ) in general require a support dataset to be deployed at inference time . Achieving good prediction accuracy typically requires having a large support set , which inevitably increases both memory footprint and latency at inference time—the scalability issue . It can also raise privacy concerns when deploying a support set of original examples , e.g. , distributing raw images to user devices . Additional challenges to scalability include , for instance , the desire for rapid hyper-parameter search ( Shleifer & Prokop , 2019 ) and minimizing the resources consumed when replaying data for continual learning ( Borsos et al. , 2020 ) . A valuable contribution to all these problems would be to find surrogate datasets that can mitigate the challenges which occur for naturally occurring datasets without a significant sacrifice in performance . This suggests the following Question : What is the space of datasets , possibly with constraints in regards to size or signal preserved , whose trained models are all ( approximately ) equivalent to some specific model ? In attempting to answer this question , in the setting of supervised learning on image data , we discover a rich variety of datasets , diverse in size and human interpretability while also robust to model architectures , which yield high performance or state of the art ( SOTA ) results when used as training data . We obtain such datasets through the introduction of a novel meta-learning algorithm called Kernel Inducing Points ( KIP ) . Figure 1 shows some example images from our learned datasets . We explore KIP in the context of compressing and corrupting datasets , validating its effectiveness in the setting of kernel-ridge regression ( KRR ) and neural network training on benchmark datasets MNIST and CIFAR-10 . Our contributions can be summarized as follows : 1.1 SUMMARY OF CONTRIBUTIONS . • We formulate a novel concept of -approximation of a dataset . This provides a theoretical framework for understanding dataset distillation and compression . • We introduce Kernel Inducing Points ( KIP ) , a meta-learning algorithm for obtaining - approximation of datasets . We establish convergence in the case of a linear kernel in Theorem 1 . We also introduce a variant called Label Solve ( LS ) , which gives a closed-form solution for obtaining distilled datasets differing only via labels . • We explore the following aspects of -approximation of datasets : 1 . Compression ( Distillation ) for Kernel Ridge-Regression : For kernel ridge regression , we improve sample efficiency by over one or two orders of magnitude , e.g . using 10 images to outperform hundreds or thousands of images ( Tables 1 , 2 vs Tables A1 , A2 ) . We obtain state of the art results for MNIST and CIFAR-10 classification while using few enough images ( 10K ) to allow for in-memory inference ( Tables A3 , A4 ) . 2 . Compression ( Distillation ) for Neural Networks : We obtain state of the art dataset distillation results for the training of neural networks , often times even with only a single hidden layer fully-connected network ( Tables 1 and 2 ) . 3 . Privacy : We obtain datasets with a strong trade-off between corruption and test accuracy , which suggests applications to privacy-preserving dataset creation . In particular , we produce images with up to 90 % of their pixels corrupted with limited degradation in performance as measured by test accuracy in the appropriate regimes ( Figures 3 , A3 , and Tables A5-A10 ) and which simultaneously outperform natural images , in a wide variety of settings . • We provide an open source implementation of KIP and LS , available in an interactive Colab notebook1 . 2 SETUP . In this section we define some key concepts for our methods . 1https : //colab.research.google.com/github/google-research/google-research/blob/master/kip/KIP.ipynb Definition 1 . A dataset in Rd is a set of n distinct vectors in Rd for some n ≥ 1 . We refer to each such vector as a datapoint . A dataset is labeled if each datapoint is paired with a label vector in RC , for some fixed C. A datapoint along with its corresponding label is a labeled datapoint . We use the notation D = ( X , y ) , where X ∈ Rn×d and y ∈ Rn×C , to denote the tuple of unlabeled datapoints X with their corresponding labels y . We henceforth assume all datasets are labeled . Next , we introduce our notions of approximation , both of functions ( representing learned algorithms ) and of datasets , which are characterized in terms of performance with respect to a loss function rather than closeness with respect to a metric . A loss function ` : RC × RC → R is one that is nonnegative and satisfies ` ( z , z ) = 0 for all z . Definition 2 . Fix a loss function ` and let f , f̃ : Rd → RC be two functions . Let ≥ 0 . 1 . Given a distribution P on Rd × RC , we say f and f̃ are weakly -close with respect to ( ` , P ) if ∣∣∣E ( x , y ) ∼P ( ` ( f ( x ) , y ) ) − E ( x , y ) ∼P ( ` ( f̃ ( x ) , y ) ) ∣∣∣ ≤ . ( 1 ) 2 . Given a distribution P on Rd we say f and f̃ are strongly -close with respect to ( ` , P ) if Ex∼P ( ` ( f ( x ) , f̃ ( x ) ) ) ≤ . ( 2 ) We drop explicit reference to ( ` , P ) if their values are understood or immaterial . Given a learning algorithm A ( e.g . gradient descent with respect to the loss function of a neural network ) , let AD denote the resulting model obtained after training A on D. We regard AD as a mapping from datapoints to prediction labels . Definition 3 . Fix learning algorithms A and Ã . Let D and D̃ be two labeled datasets in Rd with label space RC . Let ≥ 0 . We say D̃ is a weak -approximation of D with respect to ( Ã , A , ` , P ) if ÃD̃ and AD are weakly -close with respect to ( ` , P ) , where ` is a loss function and P is a distribution on Rd×RC . We define strong -approximation similarly . We drop explicit reference to ( some of ) the Ã , A , ` , P if their values are understood or immaterial . We provide some justification for this definition in the Appendix . In this paper , we will measure - approximation with respect to 0-1 loss for multiway classification ( i.e . accuracy ) . We focus on weak -approximation , since in most of our experiments , we consider models in the low-data regime with large classification error rates , in which case , sample-wise agreement of two models is not of central importance . On the other hand , observe that if two models have population classification error rates less than /2 , then ( 2 ) is automatically satisfied , in which case , the notions of weak-approximation and strong-approximation converge . We list several examples of -approximation , with = 0 , for the case when Ã = A are given by the following : Example 1 : Support Vector Machines . Given a datasetD of sizeN , train an SVM onD and obtain M support vectors . TheseM support vectors yield a dataset D̃ that is a strong 0-approximation toD in the linearly separable case , while for the nonseparable case , one has to also include the datapoints with positive slack . Asymptotic lower bounds asserting M = O ( N ) have been shown in Steinwart ( 2003 ) .2 Example 2 : Ridge Regression . Any two datasets D and D̃ that determine the same ridge-regressor are 0-approximations of each other . In particular , in the scalar case , we can obtain arbitrarily small 0-approximating D̃ as follows . Given training data D = ( X , y ) in Rd , the corresponding ridgeregressor is the predictor x∗ 7→ w · x∗ , ( 3 ) w = Φλ ( X ) y , ( 4 ) Φλ ( X ) = X T ( XXT + λI ) −1 ( 5 ) 2As a specific example , many thousands of support vectors are needed for MNIST classification ( Bordes et al . ( 2005 ) ) . where for λ = 0 , we interpret the inverse as a pseudoinverse . It follows that for any givenw ∈ Rd×1 , we can always find ( X̃ , ỹ ) of arbitrary size ( i.e . X̃ ∈ Rn×d , y ∈ Rn×1 with n arbitrarily small ) that satisfiesw = Φλ ( X̃ ) ỹ . Simply choose X̃ such thatw is in the range of Φλ ( X̃ ) . The resulting dataset ( X̃ , ỹ ) is a 0-approximation to D. If we have a C-dimensional regression problem , the preceding analysis can be repeated component-wise in label-space to show 0-approximation with a dataset of size at least C ( since then the rank of Φλ ( X̃ ) can be made at least the rank of w ∈ Rd×C ) . We are interested in learning algorithms given by KRR and neural networks . These can be investigated in unison via neural tangent kernels . Furthermore , we study two settings for the usage of -approximate datasets , though there are bound to be others : 1 . ( Sample efficiency / compression ) Fix . What is the minimum size of D̃ needed in order for D̃ to be an -approximate dataset ? 2 . ( Privacy guarantee ) Can an -approximate dataset be found such that the distribution from which it is drawn and the distribution from which the original training dataset is drawn satisfy a given upper bound in mutual information ? Motivated by these questions , we introduce the following definitions : Definition 4 . ( Heuristic ) Let D̃ and D be two datasets such that D̃ is a weak -approximation of D , with |D̃| ≤ |D| and small . We call |D|/|D̃| the compression ratio . In other words , the compression ratio is a measure of how well D̃ compresses the information available in D , as measured by approximate agreement of their population loss . Our definition is heuristic in that is not precisely quantified and so is meant as a soft measure of compression . Definition 5 . Let Γ be an algorithm that takes a dataset D in Rd and returns a ( random ) collection of datasets in Rd . For 0 ≤ ρ ≤ 1 , we say that Γ is ρ-corrupted if for any input dataset D , every datapoint3 drawn from the datasets of Γ ( D ) has at least ρ fraction of its coordinates independent of D. In other words , datasets produced by Γ have ρ fraction of its entries contain no information about the dataset D ( e.g . because they have a fixed value or are filled in randomly ) . Corrupting information is naturally a way of enhancing privacy , as it makes it more difficult for an attacker to obtain useful information about the data used to train a model . Adding noise to the inputs to neural network or of its gradient updates can be shown to provide differentially private guarantees ( Abadi et al . ( 2016 ) ) .

Summary: This paper proposes a data-driven approach to choose an informative surrogate sub-dataset,  termed "a \epsilon-approximation", from the original data set. A meta-learning algorithm called Kernel Inducing Points (KIP ) is proposed to obtain such sub-datasets for (Linear) Kernel Ridge Regression (KRR), with the potential to extend to other machine learning algorithms such as neural networks.  Some theoretical results are provided for the KRR with a linear kernel. The empirical performance of the proposed algorithm is evaluated by experiments based on synthetic data and some standard benchmark data sets.