On Learning Representations for Tabular Data Distillation
Authors: Inwon Kang
Parikshit Ram
Yi Zhou
Horst Samulowitz
Oshani Seneviratne
Venue: Accepted by TMLR ()

Abstract: Dataset distillation generates a small set of information-rich instances from a large dataset, resulting in reduced storage requirements, privacy or copyright risks, and computational costs for downstream modeling, though much of the research has focused on the image data modality. We study tabular data distillation, which brings in novel challenges such as the inherent feature heterogeneity and the common use of non-differentiable learning models (such as decision tree ensembles and nearest-neighbor predictors). To mitigate these challenges, we present $\texttt{TDColER}$, a tabular data distillation framework via column embeddings-based representation learning. To evaluate this framework, we also present a tabular data distillation benchmark, ${{\sf \small TDBench}}$. Based on an elaborate evaluation on ${{\sf \small TDBench}}$, resulting in 226,200 distilled datasets and 541,980 models trained on them, we demonstrate that $\texttt{TDColER}$ is able to boost the distilled data quality of off-the-shelf distillation schemes by 0.5-143% across 7 different tabular learning models. All of the code used in the experiments can be found in http://github.com/inwonakng/tdbench

Published in Transactions on Machine Learning Research (04/2025)
On Learning Representations for Tabular Data Distillation
Inwon Kang
kangi eduu
Deportment of Computer Science
Renaselaer Polytechnic Institute
Parikshit Ram
Porikshit Ham Bibm. com
IBM Research
Yorktown Heights, NY
Yi Zhou
Yi. Tt.Zhoo@ibb.oomm
IBM Research
San Jose, CA 95120
Horst Samulowitz
samulouiiz:@us.cc www.:huwsss.b...m ibm..com
IBM Research Yorktown Heights, NY
Oshani Seneviratne
senevo@rpic edu
Department of Computer Science
Renssclaer Polytechnic Institute.
Reviewed on OpenReview: https: http:///opppevvvveeee net/ formm id- GXLsrvOGIK
Abstract
Dataset distillation generates small set of information-rich instances from a large dataset, resulting in reduced storage requirements, privacy or copyright risks, and computational costs for downstream modeling, though much of the research has focused on the image data modality. We study tabular data distillation, which brings in novel challenges such as the inherent feature heterogeneity and the common use of non-differentiable learning models (such as decision tree ensembles and nearest-neighbor predictors). To mitigate those challenges, present TDColER, tabular data distillation framework via column embeddings-basee representation learning. To evaluate this framework, also present tabular data distillation benchmark, TDBench. Based on elaborate evaluation TDBench, resulting in 226,890 distilled datasets and 548,880 models trained on them, we demonstrate that TDColER is able to boost the distilled data quality of off-the-shelf distillation schemes by 0.5-143% across different tabular learning models. All of the code used in the experiments can be found in htttp://githii com/ wwwwnakng/ttbeen.oooo
Introduction
Dataset distillation or dataset condensation is the process creating a small set extremely informa- tive samples (usually synthetic) from a large dataset such that a model trained on this set will have predictive performance comparable to that of model trained on the original large dataset (Wang al.) 2018 Yu et al.) 2023). First, data distillation reduces data storage costs and can mitigate the privacy and copyright concerns involved in keeping around (and continuously utilizing) large amounts of raw data. Furthermore, the reduction in the data size implics lower computational cost of model train- ing, especially when multiple models need to be trained on any given dataset. The above advantages of dataset distillation also facilitate various applications. Contimual learning, where we need to learn new tasks while avoiding forgetting older tasks sequentially, often makes use of "replay buffer" of old task data to be used while learning new tasks mitigate forgetting of the older tasks (Rolnick al. 2019

---

Published in Transactions on Machine Learning Research (04/2025)
Original Dataset Nxd
Processed Datasel Nxd
Distilled Dataset nxd
XGB
KNN
1.0
1.0
Preprocess
Distill
0.5
1N0
Column Embedding
Relative Regret
0.4
0.0
175 80%
TDCoIER
50
100
50
100
Decode
MLP
0.75
Encoder
Distill
66.72%
52,650
0.25
100
Nxdxm
X
7XX
Instances Per Class
(a) Overview of TDColER. The top describes a vanilla distillation scheme that only uses standard preprocessing techniques before dis- tillation. The highlighted box describes the proposed TDColER, which column embeddings after such preprocessing and encoder-decoder architectures to generate rich compact representations.
(b) Suapshot of downstream classifier'' per- formance increase when trained on data distilled using k-means with TDColER (KM- Enc) and without. (KM) against random sampling (RND).
Figure 2: Proposed approach - TDColER: Tabular Distillation Via Column Embeddings based Representation Learning. Figure 2a shows the overview of our proposed approach, and Figure [b]shows the performance gains when using k-means clustering as the distillation method. Thronghout our experiments, a performance increase from 0.5% to as large as 143% when TDColER applied.
Dataset distillation reduces the memory over-
Adult
X-means
MLP
head of this replay buffer, allowing learning of
Agglomerativve
ReeNNet
a larger number of tasks without forgetting (Ti-
Employes Access
KIP
FT FTTamsformer
Gradiant Matching
wari et al. 2022 Rosasco et al.] 2022). In led-
Trajectory! Difficulty Matching
XGBeost
crated learning, we need train model using data spread across multiple clients without ever moving the data between clients and reducing the communication overhead. Dataset distilla- tion generates compact yet private synthetic data from the client data that can be safely model
Naive Bayas
DeepCore
KNN
Logistic Regression
Temcent CTR
Random Sampling
(1) 23 Datasets
(2) 11 Distillatinn Schernes
(3) Downstream Classifiers
Figure 1: Overview of TDBench. The benchmarking suitc allows for flexible choice of datasets, distillation schemes, and downstream models that enables for modular evaluation of any new distillation method.
exchanged for communication-cficinn training (Song et al. 2023 Goetz & Tewari, 2020 Zhou al. 2020).
While dataset distillation has been widely studied for image datasets (Cui al. 2022 Yu et al. 2023), the equally important application to other data modalities is limited. The problem of tabular data distillation has received very little attention, though many real-world learning problems and applications involve tabular data (Guo et al.] 2017 Clements et al. 2020 Borisov et al.) 2022). Various image data distillation schemes have been proposed in the literature, but their application to tabular data is not straightforward. First, all image data distillation schemes rely on the choice of a differentiable "backbone model." While differentiable neural network-based schemes are standard for images, wide variety of non-differentiable models are used with tabular data, such decision tree ensembles, nearest-neighbor models, and kernel machines. Second, almost all data distillation methods for images generate distilled data in the original pixel space. While pixels are homogencous raw features of an image, the features in tabular data can be extremely heterogencous, creating a mismatch between what the image data distillation methods are designed for and what we have as an inherent property of tabular data. Finally, it is standard to use vision-specific data augmentation schemes (such as rotation, reflection, cropping, and translation) to train the model on the distilled image data. Such standard augmentations are not available for tabular data, thus creating another discrepancy in the expectod conditions for the problem.
2

---

Published in Transactions on Machine Learning Research (04/2025)
Our contribution. In this paper, we study tabular dataset distillation and present novel scheme to enhance the distilled data quality of multiple off-the-shelf data distillation schemes across various datasets, models, and distillation sizes. Specifically, we make the following contributions:
We propose Tabular Distillation via Column Embeddings based Representation Learning or TDColER that can utilize modern neural-network architectures such as Transformers and graph neural networks generate rich compact representations. TDColER improves the quality of distilled data compared to existing distillation schemes. Figure 2a provides an overview of our proposed TDColER.
We present TDBench, Tabular Distillation Benchmark with 23 tabular datasets, 7 model classes, and 11 distillation schemes. We present an overview of TDBench, an extensible and modular framework for measuring various aspects of data distillation on tabular data, in Figure |
With the elaborate evaluation of our proposed TDColER on TDBench, resulting in over 226,890 distilled datasets and 548,880 model trainings, we show that, on aggregate across all datasets, TDColER improves upon direct application of off- the-shelf distillation method on tabular data by 0.5-143% in terms of the distilled data quality across all models at the smallest distillation of 10 instances-per-class. Figure 2b
presents snapshot of our results.
Based on our thorough evaluation, we present various insights regarding tabular dataset distillation, such as (1) k-means clustering in the learned representations make for an extremely favorable distillation scheme, (ii) transformer-based tabular data representations obtain the highest distilled data quality on aggregate, while (iii) graph neural network based tabular data representations perform slightly worse than transformers but are significantly more parameter efficient.
1.1 Related Work
Dataset distillation was introduced by Wang et al. (2018) as a bilevel optimization problem (Feng et al. 2023 and has been widely studied in the context of image data distillation. Most methods can be categorized into approaches that match the original data by (i) backbone model performance, (ii) backbone model parameters, or (iii) backbone representation distributions (Yu et al. 2023). Wang et al. (2018) minimized performance differences between the original and distilled data, while Nguyen el al. (2020) introduced kernel-induced
points (KIP) using kernel ridge regression with a neural tangent kernel (Jacot ct al. 2018 Alternatively,
methods have focused on parameter or gradient matching (Zhao et al. 2020 Lee et al. 2022 Jiang et al.,
2023: Cazenavette et al.) 2022). Gradient matching (Zhao et al.) 2020) aligns model gradients between original and synthetic data, while trajectory matching (Cazenavette et al. (022) minimizes discrepancies
between entire training trajectories. Other approaches include distribution matching (Zhao & Bilen 2023
which aligns per-class means, and cross-layer feature embedding matching (Wang et al. 2022). However, the abovementioned methods rely on differentiable backbones, limiting cross-architecture generalization (Cui ct al. 2022 Nguyen et al. 2020). As a result, research has focused primarily on images, leaving tabular data distillation largely unexplored (Medvedev & D'yakonoy 2021). We address this gap by proposing a more general distillation framework.
Datasct distillation aligns with coresct selection (Feldman 2020 which aims to reduce data size, typically selecting real data instances (potentially risking privacy). In contrast, distillation generates synthetic data beyond the real data manifold. Notably, coreset selection is subset of dataset distillation, where the synthetic data lies on the real data manifold. Generative modeling (Goodfellow et al. 2020 Kingma et al.) 2013) is another related area, usually focused on generating realistic data. In datasct distillation, the goal is to generate informative rather than realistic samples. Recently, Cazenavette et al. (2023) demonstrated how generative modcling can be used to sced the datasct distillation process, arguing that distillation methods should be applied to a latent representation instead of the pixel space. This is aligned with our proposal, in which we demonstrate that distillation in the latent space is critical to obtaining meaningful distilled data quality with tabular datasets. However, the proposed Generative Latent Distillation (GLa)))tionnGLaa) (GLaD) scheme is very focused on generative vision models, requiring a careful choice of the latent representation from within the model for trade-off in realistic distilled data or erpressivily, thus limiting cross-architecture generalization.
Cui et al. (2022) benchmarked several distillation methods and found trajectory matching (Cazenavette et al. 2022) to be most effective, followed by KIP (Nguyen et al.) (2020p. Coreset methods, like k-means clustering, also outperformed many model-basen distillation techniques, which corroborate. We focus on Gradient.
3

---

Published in Transactions on Machine Learning Research (04/2025)
Matching (GM) and Kernel Inducing Points (KIP) algorithms due to the high computational overhead of trajectory matching and omit data augmentation due to its limited applicability to tabular data. As noted before, data augmentation is not standard with tabular data, and we do not consider it in our evaluation with TDBench.
2 Table Distillation
Data distillation has been primarily studied in the context of images where each data point is composed of a homogeneous set of features pixels and the downstream models are neural networks. The two main distinctions with tabular data distillation are: (i) Feature Heterogeneity: Features in tabular data are usually heterogencous in their data types and ranges and can have vastly different meanings, making it challenging to generate appropriate feature aggregations as usually done with neural nelworks. For example, a categorical feature for car car_color cannot be treatd in the same way as feature for car _pricerieeeee in that they are represented in a completely different way (categorical vs. numerical), and also mean very different things, for example, unlike the uniform meaning of pixel values in a grid an image. While one-hot encoding is commonly used convert the categorical features into the numerical space, this approach can introduce sparsity in the resulting representation (e.g. a categorical feature with >> 100 possible values). This is further exacerbated by the common presence of missing values. (ii) Model Agnosticity: For tabular data, the downstream model that will use the distilled data can be quite varied, with decision-tree- bree- models often being quite successful (Grinsztajn et al.] 2022). while linear and nearest-neighbor models are used for interpretability. Various increasing competitive neuuralnetwork--ai
models have also been developed for tabular data (Borisov et al. 2022 Gorishniy et al.) 2021 McElfresh et al. 2023 Grinsztajn et al. 2022). ) However, in the most common cases, we cannot assume that the downstream model is differentiable and thus will be unable to perform a downstream model- specific distillation via the common bilevel formulation of the problem. The distillation has to be model-agnostic, which means that we have to retain as much of the information in the original data as possible since we do not know a priori what information the down- stream model might leverage.
Algorithm 1: Distill original data with N samples given a preprocessor P R' C I2 and distiller F: F.R.X. x Y + R y".
// Preprocess // Distill
2 R- PS))
s return R
We will consider a classification dataset S { y1), ... (#N, yN)} with N samples, r numerical features and e categorical features, and L labels, where each E IR x C and yi Y (I,...,,, LJ. Following Cui lt al. (2022), we only consider classification tasks in this work, but it should be noted that regression can be easily added into our framework. Note that features may contain missing values. After appropriate preprocessing steps to convert the categorical variables to numerical oncs and imputing the missing values, we can directly apply somee existing distillation schemes such as KIP (Nguyen et al. 2020 or G.M. (Zhao
et al. 2020). This procedure is sketched in Algorithm
2.1 Representation Learning via Column Embedding
A key ingredient in the development of neural networks for tabular data is the use of column embeddings. First developed for categorical features, the idea is to learn an embedding for each of the categories in a categorical feature (Guo & Berkhahn 2016). This embedding would replace the one-hot encoded numerical representation of the categories and be used in conjunction with the (appropriately scaled and imputed) numerical Teatures standard and custom feed-forward networks (FFNs) (Borisov et al.] (022). Column embeddings for numerical data were developed to use more standard modern architectures such as graph neural networks (GNNs) and Transformers. As with categorical data, cach value in numerical feature of the table would be converted into a learnable embedding. Thus, more precisely, a sample (row) in table with r
I For example, using data science tools such preprocesaing.Dneeiooe.e and impute. Simntee.nnneeeeptee from the scikit- seikit-learn machine learning toolkit.
4

---

Published in Transactions on Machine Learning Research (04/2025)
numerical features and categorical features is now represented a set of (r + c) embeddings in R each of size m (where m is user-specified hyperparameter), thus effectively as the (m x (r + c)) matrix
Encoder Architectures. Given the R=x((++) representation of a row (sample) using column embeddings, our goal is to learn a more compact yet faithful representation a row. One simple strategy is to concatenate all the (r + c) column embeddings into a single vector in [R(((+t) of size m(r + c) and input it into an FFN which projects it down to a lower dimensionality (Figure 13
However, one of our main motivations for using column embeddings is to leverage the capabilities of more modern architectures. For given row, the (r + c) column embeddings can be treated as initial token embeddings that are progressively updated through multiple Transformer blocks as described by Goriihnny et al.] (2021). Using dummy [CLS] token, the above process can create a m-dimensional representation of the row (Figure 16). An alternate procedure is to represent a table as a bipartite graph between columns and rows (with column values and rows as vertices) and utilize the column embeddings as representations for the column vertices (Wu et al. 2021). ). Then, the row embeddings are obtained by filling in representations for the row vertices via multiple rounds of message passing in a multi-layered GNN (Figure 14p. For our purposes, we consider all three architectures FFN, Transformer and GNN as encoders that project the R=x((++) representation of row into embedding in While calegorical column embeddings are standard, there are multiple techniques for nunerical column embeddings Gorishniy et al.) 2021 2022). In this work, we mainly consider the embeddings from Gorishniy ct al. 2021 but we discuss and ablate the effect these different schemes have in Appendix B.2
Learning Objective. Our goal is to retain as much information regarding the original data in the learned representation as possible. The need for high-fidelity learned representations is critical because we do not assume anything regording the downstream model, which will be trained with the distilled data. Thus, we try to reconstruct the original data from the learned representation as well as possible. Formally, given column embeddings C' T x C + R=x((++), and an encoder & Rmmx(++)) + RTm, we utilize a decoder tb R.m + I" x C to reconstruct the original data, and solve the following optimization problem:
min Σ C.o.v (E.y)S
(xx (((C(())))
(1)
where E(:5.) is a reconstruction error (RE) and & is implemented with nyyllyeer MLP where ny is a tunable parameter. Note that the above representation learning does not use the label information in the data, S. This representation learning framework allows us to infuse class information in the representations while ensuring no loss of original information. Thus, after obtaining the column embeddings C, encoder & and decoder to by solving Equation 1 we apply supervised fine-tuning (SFT) to the encoder by learning a classifier f I + Y on top of the learned representations while keeping the reconstruction loss low:
min C.
Σ ((I.0(((C(x))) + ally, (T,y)ES
(2)
where .(..)) is the downstream learning loss function, a >> 0 is a hyperparameter balancing the classification and reconstruction quality, and f is implemented with ny layer MLP where ny is tunable parameter. We employ 2-stage hyperparameter optimization (HPO) process for each optinization problem of each dataset to find the best architecture, A more detailed version of this procedure can be seen in Algorithm
Complete Distillation Pipeline. After the column embeddings C, an autoencoder composed of encoder à and decoder & are learned (with Equation ([) and fine-tuned (with Equation (2)), we convert the input features of the whole original dataset (with N samples) into the learned representations in I using C and & and apply the aforementioned distillation schemes to this dataset (N samples in RT) to get distilled samples in RT. At this point, we decode the distilled samples into the original representation using vb. This whole pipeline is summarized in Algorithm 2 Note that the distillation with the learned representation in R" and the availability of the decoder 2b, allows us to have two versions of the distilled data - one in the learned representation (ft in Algorithm Line 4), and one in the original representation (R in Algorithm
While each feature can have column embeddings different sizes, many neural network architectures require the column embedding size to match across all features.
5

---

Published in Transactions on Machine Learning Research (04/2025)
Line 5). Thus, using R corresponds to the bottom highlighted portion of Figure 2a while just using R corresponds to the top portion outside of the box. We can choose the appropriate distilled set. based on the downstream application: If we require the distilled data to be obfuscated with no explicit correspondence to the original features, we can use R. In this setting, we are required to have the column embeddings C and the encoder Φ during inference with the downstream trained model to map the lest points into the appropriate representation. If we require the distilled data and the model trained on it to be interpretable in terms of the original features, we should use the distilled set R in the original representation. In this case, we do not need the column embeddings the encoder during inference.
novel represenlation learning and distillation pipeline for model-agnostic tabular data distillation utilizing eristing distillation schemes, column embeddings, and network architectures such as transformers and GNNs. In our thorough empirical evaluations, we will demonstrate the distilled data qualily boost from this pipeline across mulliple dalasels and downstream models.
Remark Our contribution
3 Evaluation Benchmark
To thoroughly evaluate the various configura- lions of the proposed distillation pipeline, we establish a comprehensive benchmark suite with a varied set of datasets and downstream models, evaluating the pipeline at various levels of distillation sizes. With 3 encoder architectures, 12 distillation schernes (includ- ing variants), 20+ datasets, 7 downstream models, 10 distillation sizes, 5 repetitions per distillation pipeline, and model training, we have generated over 226,890 distilled datasets and trained over 548,88 880 individual down- stream models
Algorithm 2: TDColER: Distill dataset S with N samples given distiller F Nmm x Y mmxm y and learn- able column embeddings R R' C Rmx((+e), encoder Φ: Rm((+s) + RR",, decoder vp R + R" x C classifier
f R..
v solve Equation
// minimize
2 v. + solve Equation
2
// fine-tune
3 3 ((o(((=)) w). (z, S)
// Encode
4 At FF))
Distill in latent space
s R (x,
// Decode
s return R, C.
3.1 Datasets.
We consider 23 datasets from OpenML Vanschoren et al. 2014 with the number of samples varying from 10,000 to over 110,000, and number of leatures varying from 7 to 54. Instead of investigating a few large datasets, we choose to incorporate more datasets to generalize the findings across a wider range of datasets. The datasots are chosen to be diverse in terms of the number of samples, features, and the type of features (numerical, categorical, mixed). There are 14/23 datasets with only numerical features, 2/23 with only categorical features, and 7/23 with both numerical and categorical features. All these datasets correspond to binary classification problems. Class imbalance is a common feature of tabular datasets (Johnson & Khoshgoftaar 2019 Thabtah et al. 2020), and we focus on binary classification to carefully study the effect of class imbalance on the distilled data quality. There are 9/23 almost perfectly balanced datasets and 10/23 datasots with a ratio of close to 1:2 between the smaller and larger classcs, with the worst imbalance ratio smaller than 1:15. Note that while we only consider binary classification datasets, the distillation pipelines are natively applicable to multi-class classification problems. We handle missing values by adding new būn for categorical features and replace with 0 for numerical features. More details about the datascts used in the experiments can be found in Appendix A.1
Distillation Methods. Given our aforementioned desiderata for model-agnosticiit, we have the following existing distillation schemes available, which take as input the set S of N samples and output set R of n < N distilled samples (further details regarding implementation of each distillation method is provided in Appendix p.
The TDBench benchmarking suite (code provided in the supplement) can be extended to evaluate any new distillation method, tabular representation, and downstream model and compared against our current database of results (also provided in the supplement). More information can found Appendix C
6

---

Published in Transactions on Machine Learning Research (04/2025)
k-means Clustering (KM) finds n/L clusters for each of the L classes to produce total of n distilled samples using Lloyd's k-means algorithm (Looyd 1982). We consider two variat ions here by (i) using the Euclidean center of each cluster to generate a synthetic sample or (ii) choosing the closest real point to the Euclidean center of cach cluster. That is, R comprises n/ L cluster centers (or closest real points) for each of the I classes
again generates n/L. clusters for cach of the L classes is similar k-means. We use the Ward linkage scheme with the Euclidean distance metric. Similar to k-means, generate (i) synthetic samples by using the Euclidean center of cluster or (fi) real samples that are closest to the cluster centers.
Agglomerative Clustering (AG) (Müllner 2011
Kernel Induced Points (KIP) (gguyen et al. 2020) uses the neural tangent kernel (NTK) (Jacot et al., 2018) of a wide neural network and kernel ridge regression to produce a distilled set of samples. Given the feature matrix X E NxD and the label vector y E Y KIP learns the distilled feature matrix X E xx and label vector y E y by solving the following problem:
min C K Xxx + +)) X,y
where L is the downstream learning loss function, K Xxx E RNxx is the NTK matrix between X and X, Xxx E RR* is the NTK matrix of X with itself, and À > 0 is a regularization hyperparameter for the kernel ridge regression. Essentially, we are learning a set of synthetic samples such that the predictions made on the original dataset features using the distilled dataset via kernel ridge regression match the original labels.
Gradient Matching (GM) (Zaoo et al. 2020p produces the distilled set R for given "backbone model" Mo (parameterized by 0) by directly optimizing for R to induce model parameter gradients that are similar to the gradients obtained while training Ma on the full dataset S. Given a distance metric D(:,), and a distribution Po. over the random model parameter initializations 6g, the distillation problem tries to minimize the distance between the model gradients computed on the full and distilled datasets over the T steps of model learning as follows:
min Egg ~ Peo R
Σ D (VeC((;;)) VoC(0,
(4)
where L(0;S) S) is the loss of the model Ms the original full dataset S, [(; R) is the loss of Mo evaluated R, and the model parameters 0, are updated at 0c+1 + 0, To VoL(O; S) via gradient descent with a learning rate no using the full original dataset.
We consider KIP and GM as representatives from previous data distillation literature that are model- agnostic and model-centric, respectively. Appendix [A.3I] further discusses our choice of distillation methods considered in this work. However, based on the weaker downstream performance of these methods in our initial experiments, we later included more recent methods from computer vision, such trajectory-base matching Cazenavette et al. (2022) and its evolved version difficulti--aseed matching Guo et al. (2023). and 4 representative NN-based coreset selection methods from DeepCore Guo et al. (022)) All the above distillation schemes require the data to be preprocessed into a numerical form, and can be used in Algorithm to distill tables. But, as we will soe, this is not a very useful scheme. Our evaluation of TDCo1ER on TDBench will demonstrate how the performance of these distillation schemes are boosted via representation learning.
To study the ability of the distillation pipeline to generate really small but useful distilled datasets, we consider extremely small distilled datascts with 10-100 instances per class (IPC), corresponding to distillation fraction of the order of 0.1-1.0% on the smallest datasets, and 0.01-0.1% for the largest datasets. This is comparable to the compression ratio of 0.02-1% used in Cui et al.
2022 and Cazenavette et al. (2023).
Downstream models. We consider 7 downstream models to evaluate the distilled data quality. We consider the Nearest-Neiihhbor Classifier (KNNeighbors), Logistic Regression (LR), Gaussian Naive Bayes (GNB), and the Multi-Layered Pereeptron (MLP) from the scikit-learn library (Pedregosa et al. 2011 J. We also consider the popular XGBoost ensemble of gradient- decision trees (XGB) (Chen & Guestrin 2016). We include two recent neural network models for tabular data, the ResNet and the FTTransformer
7

---

Published in Transactions on Machine Learning Research (04/2025)
models (Gorishniy et al. 2021). Since our distillation pipeline is deliberately model-agnostic, we train these models on the distilled data using the default hyperparameters of the corresponding libraries. We also consider a hyperparameter optimization (HPO) use case using the distilled datasets in our evaluations, which can be found in Appendix 4.4
Evaluation metric. To have a standardized way to quantify the quality of the distilled data across different. models and datasets, we use the notion of relative regret which compares the each model's balanced accuracy score when trained on the full, distilled and randomly sampled data points. Precisely, the relalive regred is defined as (Ar-A)(AA--Ama)) where AF is the balanced accuracy of the model trained on the full training set, Anto is the balanced accuracy on the same test set when trained on 10 random samples per class averaged over 5 random repetitions, and A is the balanced accuracy of the model when Irained on the distilled dataset. over random 5 repetitions. This metric is used with two main assumption: 1) most methods will perform better than 10 randomly sampled points AR 2) most methods will not outperform the performance on the full dataset (Ap). A relative regret of matches the performance of random sampling at IPC=10, and a relative regret of 0 matches the performance of the model trained on the full dataset (which is usually the gold standard)- lower relative regret implies higher distilled data quality D It should be noted that while this metric is designed to provide sense of the improvement from the different pipelines, comparisons across different downstream classifiers may be misleading as the scaling depends on the AF of each classifier. In addition, it should also be noted that this metric does not have a strict lower or upper bound. If either the target method (A) or random sampling 10 points (AR1o) yields higher performance than training on the full dataset (Ar), the relative regret can be below zero. We do not find any cases where ARso > Ar, but we do rarely observe cases where A > AF by a slight margin. We do observe some instances where this metric can result in negative values, meaing that the distilled data's downstream performance is higher than that of the full dataset. There are few reasons why this may be occuring: 1) The dataset in its original (i.e. one-hot encoded categoricals) representation can be very sparse, while the learned representation transforms it into a dense form. 2) Some of the datascts are very imbalanced, and our distillation methods force balance between labels by sampling (or initializing) the same number of points for each label.
4 Results Analysis
In this section, we present the analysis of the results obtained from our benchmarking experiments. For the sake of brevity, we will use the following acronyms Instances Per Class: IPC, random sampling: RND, k-means: KM,, agglomerative: AG, gradient matching: GM, kernel inducing points: KIP, feed-forward neural network: FFN, graph neural network: GNN, transformer: TF. Additionally, the seerrvisddfine-tunee variant of the autoencoder will be marked with a + For example, the results of Algorithm 2 with transformer architecture for & as TF* whereas TF' denotes the version that skips line 2 of Algorithm to highlight the importance of the supervised fine-tuning.
4.1 How beneficial are the learned representations for distillation?
As the first step of our analysis, we examine the performance difference between pipelines that use encoder's latent space and those that do not. To fully understand the effect of our latent space projection step, we analyze our results from two angles: 1) Is it better to distill in the latent or original space? 2) If latent space is better, is better to decode the data back to the original space or stay in the latent space? Figure 3 shows the relative regret score of distillation methods under different data representation schemes. We start by examining the downstream performance difference between pipelines that use the latent space to distill in vs. ones that do not (Algorithm 2 vs. Algorithm []. The results show that using the latent space is highly beneficial in most cases with lower IPC values. This trend is most apparent in classifiers such as KNN (44.96-108.75) 79% improvement at IPC=10), Logistic Regression (22.14-62.66 .64% improvement) or MLP (32.73-68.722 improvement), while XGBoost shows the least improvement from any of the distillation hods (15.82-36.00 30.0% improvement). k-means and agglomerative clustering also show a
FFrr all the dowustream models, the aggregate (median across all datascts) relative regret of random samples at IPC=10 (smallest distillation size) is 1.0 by definition, while the aggregate relative regret of random samples at IPC=100 (largest distillation sixe) around 0.5, indicating that the benchmark challenging enough with significant room for improvement.
8

---

Published in Transactions on Machine Learning Research (04/2025)
XGB KM
XGB AG
XGB GM IPC-10
XGB KIP
IC-10
IPC-10
IFC-10
Enca +24. 75% Rec: +17.47%
Enc.: +36, 00%
Rec: +23.40%
Enc.: +18 69% Rec. +0.5%%
Enc.: +35, 02% Rec.: +5,822%
1.0
0.5
0.0
KNN-
KM
KNN- AG
KNN GM IPC-10
KNN KIP
IPC=
PPC=10
IPC=10
Enc.:
+105,80%
Rec.: +10.00%
Enc.: +108.79% Rec.: +25.35%
Enc.: +82 32% Rec. +6.89%
Enc.: +444.6% Rec. +4.04%
1.0
0.5
Relative Regret
0.0
LR- KM
LR- AG
LR GM
LR KIP
IPC=10
IPC-10
IPC-10
IPC-10
Enc.: +62 .4%%
Rec.: -4.65%
Enc:: +22.114% :
Enc.: +3%%70% Rec: -2.05%
Enc.: +32 30% Rec.:
1.0
0.5
0.0
MLP KM
MLP AG
MLP- GM
MLP KIP
IPC 10
PC-10
IPC-10
IPC-10
Enc.:
+68.7 188 Rec:: -1.333%
Enc:: +54.38%|| Rec. 141.:-11.999
Enc.: +32 33% Rec: -1.20%
Enc.: +33 85% Rec. 26%
1.0
0.5
0.0
50
75
100
25
75
100
25
50
75
100
25
50
100
KM
Rec.
Rec.
KIP
KIP
KM Enc.
RND
AG Enc.
RND
GM Enc.
AND
KIP Enc.
RAD
Instances Per Class
Figure 3: Change in relative regret of downstream classifiers when trained on distilled data over IPC E (10, 100), aggregated over datasets and encoder architectures and compared against randomly sampled points (RND) in the original space (no encoder). The text on top of each sub-plot refers to the relative performance mncrease/decreas when using the encoded encoded/roconstioee /reconstructed space when comparing to using no encoder at the lowest IPC regime (IPC=10). Lower is better. Each row corresponds to downstream classifier, and cach column corresponds to distillation method. The encoders are denoted by color and the line style refers to the encoding scheme used- - original (no encoder involved), encoded (Enc.), and reconstructed (Rec.). For the roconstructed sctting, we pick the best performance between using the oringinal data point (only applicable to clustering- methods) or using the decoder. Data distilled by clustering methods (AG, KM) in the encoded space show the best performance for all classifiers. In many cases, using the encoded representation as the final output yields a performance comparable to using the original representation. Figure 17 shows a
more detailed version of this plot that includes FTTransformer and ResNet.
Original
Enended
ITN-TT Eneaded
gnal
ITN Encoded
ETK-IT Encoded
15:
08.
-05
-10
15
20
(a) Adult dataset.
(b) Phishing Websites dataset.
Figure 5: PCA visualization of encoded points before and after applying SFT.
more apparent decrease in regret, while KIP and GM show noticeable improvements only when both the distillation and the final dataset are in the latent space. With this in mind, we examine the performance difference when training on the distilled data in the latent space or decoding to the original space before training the downstream classifier (using R or R from Algorithm 2 Figure 3 shows that training the dataset in the latent space improves the downstream performance for all distillation pipelines in fact, it is the best performer for almost every instance over classifiers and distillation methods. The change in performance is more apparent in KNN (40.92-65, (5.26655.%). Logistic Regression (33.75-67 29%) and MLP (33.93- 96.38%), while XGBoost shows a more subtle change (7.28-19. (8.19.20%). In order to verify the separability of the encoded representation, we visualize some datasets in the encoded space both before and after applying SFT.
9

---

Published in Transactions on Machine Learning Research (04/2025)
2
3
2
3
2
KM (1.6)
(4,2) KIP
AG (1.9)
33.9) KIP
KM (2
(399) KIP 3.6) GM
AG (1.7)
(4) RND
KM (1.9]
3.6) GM
AG (2.2
GM (3.5)
RND (3.6)
RND (3.4
(a) IPC==10
(b) IPC=50
(c) IPC=100
Figure Critical difference plot comparing ranks of distillation methods across datasots, encoders, and classifiers per IPC value. The x-axis denotes the average rank, and a black horizontal line connects groups of methods that are not significanlly different in the rank distribution. k-means and agglomerative are indistinguishable from each other in IPC E (10,50), but k-means gains edge in IPC=100.
Figure 5 shows one example of this visualization, clearly show- ing that SFT improves separation data points with respect the target classes. This leads us to conclude that distilla- tion methods benefit the most when both distilling and downstream training in the latent representations. It is also worth noting that decoding the distilled data from the latent space (Rec.) is also beneficial compared to random sampling in many cases.
1e5
2-
* Params
FFN
GNN
0.6
0.7
0.8
0.9
1.0
1.1
Relative Regret
Figure Scatterplot of encoder parame- ter size and downstream classifier regret at IPC=10 aggregated over datasets and classi- fiers. The dots represent the median values, and the error bars span the 25% and 75% percentile, respectively. Note that the en- coder sizes for both SFT and base versions are the same for each dataset.
4.2 How do different encoders compare?
Having observed that using the latent space is beneficial, now seek to identify which encoder architecture leads to the best performance. Table ! shows the average rank of distil- lation pipelines that use the latent space of different encoder architectures. The mean rank of each of the is calculated by comparing the performance of each pipeline where the only difference is the encoder architecture and other components are kept constant. That is, we group the experiments into sets that only differ in the encoder (i.c. same dataset d, classifier c, ipc i, distillation method dm, encoding e) and set of encoders $- {FFN, FFN* GNN, GNN*, TF, TF*), the rank of one encoder & E & is computed RS = fo o' (1 if Av > As else 0} where As is the downstream performance of the pipeline that uses Φ. We then compute the average of this rank for every possible combination of dataset, classifier, distillation method and IPC to compute the mean rank. Among the tested architectures and training objectives, the transformer architecture with supervised fine-tuning leads to the best downstream performance. We find that adding supervised fine-tuning to the autoencoder training improves the downstream performance of all encoders in general.
Another important aspect of data distillation is to improve downstream classifier efficiency providing a lightweight proxy. Thus, it is important to examine the resources required in the distillation pipeline. Specifically, one aspect of our distillation pipelines that can add an additional cost is the encoder. Tn settings that require the data be projected into latent space at inference time, the encoder can be considered part of the distilled data. Figure shows the parameter size of the different encoder architectures vs. the downstream classifier regret scores. As noted before, the transformer architecture leads to the best. downstream performance. However, it is worth noting that GNN architecture has the smallest overall porameter size while providing the second- best performance. Further discussion on the parameter size analysis of each encoder architecture can be found in Appendix A.2.2
Average rank and median relative regret of distillation pipelines that use the latent space of different encoder architec- tures evaluated at IPC=10, grouped over all datasets and classifiers.
Table 1:
Encader
Mean Rank
Median R.R.
TF FEN
4.1176
0.9439
4.3407
0.9746
GNN
4.2243
0.9695
TF.
2.3591
0.6149
FFN" 3.3652
0.8082
GNN*
2.5931
0.7135
10

---

Published in Transactions on Machine Learning Research (04/2025)
Table 2: Relative regret of pipelines that use different combinations of distill methods and encoders at IPC=10, aggregated over classifiers. The best value for each column is marked with bold, and the second best is marked with underline. The best in each distillation method group is marked with itolics. On average, k-means with SFT transformer shows the best performance, but agglomerative clustering also shows comparable performance.
Distill Method
Encoder
Regret
Min
Q1
Mean
Median
Q3
Max
TF -144499
0.0733
-0.0464
0.4056
0.7379
1.1773
KM
FFN*
-11.9912
0.2039
0.1382
0.6035
0.8389
1.5368
GNN*
-12.1045
0,0973
0.1054
0.5047
0.7887
1.0494
TF
-15.8965
0.0810
0.0187
0.4135
0. 0.6507
1.4982
AG
FFN*
-10.1288
0,2483
0.3695
0.6230
0.8823
4.1191
GNN
-13.1881
0,1397
0.2245
0.4793
0.7595
4.4801
TF
-4.1619
0.5226
1. 1.11
0.9415
1.2966
11.1034
KIP
FFX*
-5.3973
0.8053
1.6363
1,2502
1.6434
16.4137
GNN
-1.4649
0.7403
1.1957
1.0136
1.3329
10.5175
TF
-3,8002
0.4105
0. 7.7777
7952
1.0564
4.9450
GM
FFN"
-4.3269
0.5975
1,2660
0.9938
1.3827
16.5044
GNN
-1.4776
0.4626
0.8073
0.8457
0.9779
8,4566
TwoDPlanes
Pol
Jannis
TwoDPlanes
Credit
Adult
0.425
0.82
0.30
0.900
0.92
Balanced Accuracy
0.85
Distil
Balanced Accuracy
0.76
Full Distil
0.73
0.90
HomeEquityCredit
Adult
MagicTelescope Telescope
House
Pol
MiniBooNE
0.704
0.703
0.705
0.81
0.80
0.98
0.84
0.82
0.86
0.925
0.950
0.975
0.875
0.925
0.900
5
10
100
10
10a
100
250
500
10a
102
Time (s)
Time (s)
(a) Logistic Regression
(b) XGBoost
Figure 8: Comparison of HPO validation performance vs. time to train the classifier with full and distilled data. The axis denotes runtime of the HPO search procedure, while the y axis denotes the best performance encoutered so far. To better visualize the performance difference, truncate the plot for the full data run at twice the runtime of the entire distilled run.
4.3 Which distillation method leads to the best downstream performance?
We now compare the most critical piece of the distillation pipeline the distillation method. We wish to understand which method leads to the best downstream performance across datasets, encoders, representation space, and classifier configurations. To evaluate, we perform a Friedman test with post-hoc Nemenyi test to identify groups that stand out from the rest, as shown in Figure 7 The results show that dlustrring-bsse methods (k-means, agglomerative) show the strongest performance across datasets and encoder configurations, consistently placing in the top two ranks. While both methods show similar performance, find that k-means starts to outperform agglomerative the IPC increases.
4.4 Which combination leads to best performance?
Our previous analysis has revealed that. transformer encoders with SET and clustering-based distil- lation methods perform best in their respective comparisons. Now, we aim to identify which com- bination of encoder and distillation method leads to the best downstream performance. We ap- proach this question by examining the detailed statistics behind the combinations' performance and the top performers of each dataset, classifier, and combinations. Table shows detailed statis-
11

---

Published in Transactions on Machine Learning Research (04/2025)
CD
CD
10
CD
(1.6)
110 FG
KM
(10)
(10) FG
KM [2.1]
9.5] GN
AG (2.4)
(9.9) GN
AG (2.1
(9.9) GN
GM (4.3)
9.4) GL
GM (4.3)
9.8) GL
RND (4)
19.7) GL
DAT M (4.6
7.2) GC
RND (4.4)
6.8) GC
GM (4.6
1.2) GC
RND (5.6)
(599) MTT
DATM (4.7)
(6) MTT
DATM (4.8)
(6) MTT
KIP (5.9)
KIP (5.4)
KIP (5.4)
(a) IPC=10
(b) IPC=50
(c) TPC=100
Figure Critical difference plot comparing ranks of distillation methods across datascts per IPC value when applied with TF-SFT encoder for XGBoost classifier with additional baselines. The x-axis denotes the average rank, and a black horizontal line connects groups of methods that are not significantly different in the rank distribution. k-means and agglomerative are indistinguishable from cach other in IPC E {10,50), but k-means gains an edge in IPC=100. (FG: Forgetting, GN: GraNd, GL: Glister, GC: Graph Cut)
CD
CD 5
CD
10
KM (1.3)
(9.8) FG
KM (2)
(10) FG
KM (1.9)
(10) FG
AG (1.7)
(9.5) GL
AG (2.2)
(9.9) GN
AG (2.7)
(9.9) GL
GM (4.7)
9.3) GN
KIP (4.2)
997)) GL
KIP (4.5
9.9) GN
DATM (4.9)
RND 6.7) GC
GM
GC
DATM (4.5
6.7) GC
KIP (5.2)
DATM (4.6)
6.2) RND
GM (4.8
5.7) MTT
MTT (6.1)
MTT (5.8)
RND (5.3)
(a) IPC==0
(b) IPC=50
(c) IPC=100
Figure 10: Critical difference plot comparing ranks of distillation methods across datasets per IPC value when applied with TF-SFT encoder for MLP classifier with additional baselines. The x-axis denotes the average rank, and a black horizontal line connects groups of methods that are not significantly different in the rank distribution. k-means and agglomcrative are indistinguishable from cach other in IPC (10,50), but k-means gains an edge in IPC=100. (FG: Forgetting, GN: GraNd, GL: Glister, GC: Graph Cut)
lies about each distill method and encoder combination, while Table 3 shows the count of the top distillation pipelines that placed in the top 10 by performance in each comparison group. In line with previous findings, the results show that k-means clustering with weweevvisddfine-tumee transformer encoder leads to the best overall performance. All of the top performers are clustering- based methods, and all of them use the latent space, again confirming that using the latent representation from the encoder greatly benefits distillation methods. In addition, the GNN encoder shows a comparative performance to that of the transformer encoder. This is especially noteworthy, considering that GNN has the smallest parameter size among the encoder architectures.
Table 3: The best performing pipelines ranked by their appear- ance count the top 10 (out of 50 possible combinations) of each comparison at IPC-10. k- means stands out as the strongest performer in combination with a supervised-fince -iint-td--d transformer encoder ranking at the top 10 in 67 instances out of 86 rankings.
We additionally run a smaller-scale HPO experiment to consider a use case for distilled data, as seen in Figure S Specifically, we consider a case where the validation and testing data is sampled from the original data,
Count
Encoder
D.M.
Output
67
TF
KM
Enc.
and the classifier is trained on either the full or distilled data. In general,
63
GXX
KM
Enc.
we note that training on the distilled data gives comparable performance to training the full data in fraction of the time, consuming on average 21.84% of the runtime and reaching 98.37% of the performance.
61
GNN+
AG
Enc.
61
TF
AG
Enc.
42
FFN-
KM
Enc.
4.5 How do more recent data distillation methods compare?
We conduct a further comparison of more recent distillation methods against the methods compared in Section ! to verify whether these methods will show superior performance. Specifically, we incorporate four representative
NN-based coresct selection methods examined in DeepCore Guo et al. (2022) Forgetting Toneva al.
2018), GraNd Paul et al. (2021). Glister Killansetty et al. (2021] Graph Cut. Tyer & Bilmes (2013) and
12

---

Published in Transactions on Machine Learning Research (04/2025)
Original
Random Sample
k-means
0.5
0.5
0.5
0.0
0.0
0.0
-0.5
-0.5
-0.5
2
Figure 11: Feature correlation of the original, randomly sampled, and k-means distilled version of the Credit dataset 11PC=100.
MTT Cazenavette al. (2022) and DATM Guo et al. 20223 Figure 9 shows the updated critical difference plot comparing the additional methods in the west-performing setting. The raw regret scores can be found in Table [8] Consistent to our previous findings, we find that more recent distillation methods that rely on NNs do not fair woll on non-differentiable downstream classifier (XGBoost), and that clustering methods also worth noting that GM shows superior performance to MTT and DATM, suggesting that the latter two methods may actually be overfitting the teacher network's architecture.
still show dominance.
In addition to the analysis seen in Figure downstream classifier (MLP) to see if our hypothesis that gradient-bosed distillation melhods perform worse due to the downstream model's architecture will hold. Figure [0 shows the critical difference comparison of distillation methods when applied on a MLP downstream classifier and Table 19 shows the balanced regret score statistics of each method. While k-means and agglomerative clustering still show superior performance, we observe that other gradient-based methods indeed show better performance when compared to random sampling, further supporting our initial suscpicion that gradient- based distillation methods produce distilled datascts that useful only for NN-based classifiers.
conduct the same experiment
neural-network based
4.6 Does distillation preserve feature correlation?
We further investigate the presevation of feature correlation in the distilled data. Figure II shows th feature correlation heatmaps for each version of the dataset. While the randomly sampled data also preserves most of the correlation, we observe that the dataset distilled with k-means is more similar (e.g. interaction between features 3 and 7 of Credit dataset) to the origi- nal dataset. We also observe this trend for other datasets, which can be seen in Figure 22
AG
GM
4.7 How does class imbalance affect performance?
Finally, we examine the downstream performance of classifiess with respect to the label balance, or the imbalance, of the original dataset, shown Figure Compared to other methods, including random sampling, clustering-basee based methods show impressive strength when distilling datasets with high label imbalance, highlighting their robustness under challenging data distributions. One possible expla- nation behind this phenomenon is that while NN-based distillation methods may prioritize the majority class due the imbalance, the clustering methods are forced to place equal emphasis all classes, preventing overfitting the majority class.
-1
RND
Regret
KIP
KM
0.0
0.2
0.4
0.0
0.2
0.4
Minority Label Ratio
Figure 12: Average median relative re- gret of distillation methods aggregated over downstream classifiers and en- coders at IPC=10 with a least-squares linear regression. Compared to KTP and GM, k-means and agglomerative show much stronger performance in imbal- anced data.
5 Discussion
This work introduced a tabular data distillation pipeline and eval- uated it extensively leveraging various distillation methods, with
13

---

Published in Transactions on Machine Learning Research (04/2025)
focus on supporting both non-NN and NN ML classifiers. We introduce novel framework, TDColER, that leverages latent representation of tabular data in distillation, and evaluate it thoroughly in our benchmark, TDBench, which included 23 datasets, 11 distillation algorithms, 3 autoencoder architectures, and 7 down- stream classifiers, resulting in over 226,890 distilled datasets and 548,880 downstream classifier instances. Our results show that TDColER can induce superior performance distillation ion methods on tabular data, improving the quality by 0.5-143%. We also show that k-means clustering and transformer autoencoder are a particularly strong combination for tabular data distillation. We hope that this work will serve as starting point for future research in tabular data distillation and plan to extend this benchmark further to incorporate new distillation pipelines.
Acknowledgments
This work was supported by IBM through the IBM-Rensscler Future of Computing Research Collaboration.
References
James Bergstra, Brent Komer, Chris Eliasmith, Dan Yamins, and David D Cox. Hyperopt: a python library for model selection and hyperparameter optimization. Computational Science & Discovery, 8(1):014008, 2015.
Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelezyk, and Gjergji Kasncci. Deep neural networks and tabular data: A survey. IEEE transactions on neural networks and learning systems, 2022.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Eirrss, and Jun-Yan Zhu. Dataset. Distillation by Matching Training Trajectories. In Proceedings of the IEEE/CVF Conference Computer Vision and Pallern Recognition, pp. 4750 4759, 2022.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun- Zhu. Generalizing dataset distillation via deep generative prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognation, pp. 3739-37 -748, 2023.
Tianqi Chen and Carlos Guestrin. XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16, pp. 785 794, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 978-1-4503-4232-22 doi: 01444//9999622.999999
Jillian M. Clements, Di Xu, Nooshin Youseli, and Dmitry Efimov.. Sequential Deep Learning for Credit Risk Monitoring with Tabular Financial Data. arXiv preprint arXio:2012.1533 2020.
Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. DC-BENCH: Dataset Condensation Benchmark. Advances in Neural Information Processing Systems, 35:810 822, 2022.
Dan Feldman. Core-sets: Updated survey. Sampling techniques for supervised or unsupervised tasks, pp. 23-44, 2020.
Yunzhen Feng, Ramakrishna Vedantam, and Julia Kempe. Embarassingly simple dataset distillation. arXiv preprint trXit:2311.00025 2023.
Jack Goetz and Ambuj Tewari. Federated learning via synthetic data. arXiv preprint arXiw:2008.04400 2020.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11): 139-144, 2020.
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting Deep Learning Models for Tabular Data. In Advances in Neural Information Processing Systems, volume 34, pp. -8942-18943.. Curran Associates, Inc., 2021.
14

---

Published in Transactions on Machine Learning Research (04/2025)
Yury Gorishniy, Ivan Rubachev, and Artem Babenko. On Embeddings for Numerical Features in Tabular Deep Learning. Advances in Neural Information Processing Systems, 11124991-220000 2022.
Leo Grinsztajn, Edouard Oyallon, and Gael Varoquaux. Why do tree- models still outperform deep learning on typical tabular data? In Thirty-Sirth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
Cheng Guo and Felix Berkhahn. Entity embeddings of categorical variables. arViv preprint arXiv:1604.06737 2016.
Chengcheng Guo, Bo Zhao, and Yanbing Bai. Deepcore: A comprehensive library for coreset selection in deep learning. In International Conference on Database and Expert Systems Applications, pp. 181-195. Springer, 2022.
Huifong Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang Hc. DecpFM: A Factorization- Machine based Neural Network for CTR Prediction. In Proceedings of the Twenty- Sirtt International Joint Conference on Artificial Intelligence, pp- 1725-1733 1731, Melbourne, Australia, 2017. International Joint Conferences on Artificial Intelligence Organization. ISBN 978-0-9992411-0-3. doi: 10.2 1449966/jjaii20117
Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. Towards lossless datasct distillation via difficult--alggned trajectory matching. arXiv preprint arXiv:2310005775 2023.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive Representation Learning on Large Graphs. In Advances in Newral Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Xudong Han, Aili Shen, Yitong Li, Lea Frermann, Timothy Baldwin, and Trevor Cohn. Towards fair dataset. distillation for text classification. In Proceedings of The Third Workshop on Simple and Efficient Natural Language Processing (SustaiNLP), pp. 65 72, 2022.
Rishabh K Iyer and Jeff Bilmes. Submodular optimization with submodular cover and submodular knapsack constraints. Advances in newral information processing syslems, 26, 2013.
Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural Tangent Kernel: Convergence and Generalization in Neural Networks, In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Zixuan Jiang, Jiaqi Gu, Mingjic Liu, and David Z Pan. Delving into effective gradient matching for datasct condensation. In 2023 IEEE International Conference on Omni--ayee Intelligent Systems (COINS), pp- 1-6. IEEE, 2023.
Justin M Johnson and Taghi M Khoshgoftaar. Survey on deep learning with class imbalance. Journal of big data, 6(1):1-54, 2019.
Inwon Kang, Parikshit Ram, Zhou, Horst Samulowitz, and Oshani Seneviratne. Effective data distillation for tabular datasets. In AAAI Conference on Artificial Intelligence, 2024.
Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh Iyer. Glister: General- ization based data subset selection for efficient and robust learning. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 35, pp. 8110 8118, 2021.
Diederik P Kingma, Max Welling, et al. Auto-cncoding variational bayes, 2013.
Thomas N. Kipf and Max Welling. Semi-Supervised Classification with Graph Convolutional Networks. In International Conference on Learning Representations, 2016.
Sachyung Lee, Sanghyuk Chun, Sangwon Jung, Sangdoo Yun, and Sungroh Yoon. Dataset condensation with contrastive signals. In International Conference on Machine Learning, pp. 12352-12366 PMLR, 2022.
Yongqi Li and Wenjie Li. Data distillation for text classification. arXiv preprint arXiv:2104.08444 2021.
15

---

Published in Transactions on Machine Learning Research (04/2025)
Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E. Gonzalez, and lon Stoica. Tune: A Research Platform for Distributed Model Selection and Training. Xiv preprint 1rXix:1807. 0000005188, 2018.
Stuart Lloyd. Least squares quantization in pem. IEEE transactions on information theory, 28(2):129 137, 1982.
Aru Mackawa, Naoki Kobayashi, Kotaro Funakoshi, and Manabu Okumura. Dataset distillation with attention labels for fine-tuning bert. In Proceedings of the 61st Annual Meeting of the Association for Compudational Linguistics (Volume 2: Short Papers), pp. 119 127, 2023.
Aru Maekawa, Satoshi Kosugi, Kotaro Funakoshi, and Manabu Okumura. Dilm: distilling dataset into language model for text-level dataset distillation. arXiv preprint arXiv:240400266, 2024.
Duncan C. McElfresh, Sujay Khandagale, Jonathan Valverde, Vishak Prasad C, Ganesh Ramakrishnan, Micah Goldblum, and Colin White. When Do Neural Nets Outperform Boosted Trees on Tabular Data? In Thirly-Seventh Conference on Neural Information Processing Systems Dalasels and Benchmarks Track, 2023.
Dmitry Medvedev and Alexander D'yakonov. New properties of the data distillation method when working with tabular data. In Analysis of Images, Social Networks and Terts: 9th International Conference, AIST 2020, Skolkovo, Moscow, Russia, October 15 16, 2020, Revised Selected Papers pp. 379 390. Springer, 2021.
Daniel Müllner. Modern hierarchical, agglomerative clustering algorithms. arXiv preprint orXiv:1109.2375, 2011.
Timothy Nguyen, Zhourong Chen, and Jachoon Lec. Dataset meta-learning from kernel ridgo-regrossion. arXiv preprint Xrv:2201 1.00050, 2020.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. Advances in neural information processing systems, 2021.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scíkit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825 2830, 2011.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience Replay for Continual Learning, In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Tnc., 2019.
Andrea Rosasco, Antonio Carta, Andrea Cossu, Vincenzo Lomonaco, and Davide Bacciu. Distilled Replay: Overcoming Forgetting Through Synthetic Samples. In Fabio Cuzzolin, Kevin Cannons, and Vincenzo Lomonaco (eds.), Continual Supervipe Learning, pp. 104-117, Cham, 2022. Springer International Publishing. ISBN 978-3-0880 978-3-031-17587-9
doi: 10.1007/978-3 111-978--9-11--77--9911
Rui Song. Dai Liu, Dave Zhenyu Chen, Andreas Festag, Carsten Trinitis, Martin Schulz, and Alois Knoll. Federated learning via decentralized dataset distillation in resource-constraanned edge environments. In 2023 International Joint Conference on Neural Networks (LJCNN), pp. 1-10. IEEE, 2023.
Ili Sucholutsky and Matthias Schonlau. lobt-label dataset distillation and text dataset distillation. In 2021 International Joint Conference Neural Networks (IJCNN), pp. 1-8. IEFE, 2021.
Fadi Thabtah, Suhel Hammoud, Firuz Kamalov, and Amanda Gonsalves. Data imbalance in classification: Experimental evaluation. Information Sciences, 513:429 441, 2020.
Rishabh Tiwari, Krishnateja Killamsetty, Rishabh Iyer, and Pradeep Shenoy. GCR: Gradient Coreset based Replay Buffer Selection for Continual Learning. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recogrition (CVPR), pp. 99-108, New Orleans, LA, USA, 2022. IEEE. ISBN 978-1-66546-946-3. doi: 10.1 (00109/CCPR22288.22222 2000000000....2200000.2.222
16

---

Published in Transactions on Machine Learning Research (04/2025)
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. or orXiv preprinl. arXiv:1812.05159, 2018.
Joaquin Vanschoren, Jan N Van Rijn, Bernd Bischl, and Luis Torgo. Openml: networked science in machine learning. ACM SIGKDD Explorations Newsletter, 15(2):49 60, 2014.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lió, and Yoshua Bengio. Graph Attention Networks. In International Conference on Learning Representations, 2018.
Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022.
Tongshou Wang, Jan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXie preprinl. arXiv:1811.1099, 2018.
Qitian Wu, Chenxiao Yang, and Junchi Yan. Towards Open- World Feature Extrapolation: An Inductive Graph Learning Approach. In Advances in Neural Information Processing Systems, volume 34, pp. 19435-19447. Curran Associates, Inc., 2021.
Ruonan Yu, Songhua Liu, and Xinchao Wang. Dataset distillation: comprehensive review. IEEE transactions on pattern analysis and machine intelligence, 06(1):150-117) 2023.
Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching, In Proceedings of the IEEE/C /CVF Winter Conference on Applications of Computer Vision, pp. 6514 0523, 2023.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. arXiv preprint Xiv:20006. 05929, 2020.
Yanlin Zhou, George Pu, Xiyao Ma, Xiaolin Li, and Dapeng Wu. Distilled one-shot federated learning. arXiv preprint ar arXic:2009.07999 2020.
17

---

Published in Transactions on Machine Learning Research (04/2025)
Contents
Datasets
18
A.2
Autoencoder implementation details
19
A.2.1
Hyperparameler optimization for encoders
21
A.2.2
Discussion on parameter size of autoencoders
23
A.2.3
Optimization function
24
A.2.4
Supervised latent space tunituunng
25
A.2.5
Encoder architectures
25
Distill Methods
26
A.3.1
Choice of Distill Methods (KIP, GM)
26
A.3.2
Distill Method Implementaion
26
A.4
Downstream Classificr Hyperparameters
27
A.5
ResNet and FT Transformer performance
27
A.6
Performance of GM in original space
A.7
Determining the best overall performance
................................
28
A.8
Training/ Baaining/Inferen Inference setting of HPO example
29
B.
Full results of distillatinn methods by downstream classities
30
B.2
Effect of column embedding scheme on downstream performance
................................
30
B.3
Pairwise comparision of distillation methods,
30
B.4
Runtime comparison of distillation methods
................................
31
3.5
Multi-class dataset
32
................................
B.6
Affect of changing latent dimension size
32
Reproducing Results
33
C.2
Description of the workflow
33
C.3
Constructing
pipeline
33
Dataset Feature Correlation
35
D.2
Relation to Previous Work
35
D.3
Data distillation in natural language processing
37
D.4
Raw balanced Accuracy score
37
A
Appendix
A.1
Datasets
Tables and show the information about datascts used in our experiments along with their OpenML Van- schoren el al. 2014) URLs. These datasels were chosen mainly based on two characteristies: 1) the row count must be greater than 10,000 and 2) it must have more than 5 features. The main rationale for the selection is that the datasct must be big enough so that we can conduct meaningful distillation. The list of datasets gathered by searching on OpenML Vanschoren et al. (2014) and removing entries that were pncomplete/duplieeet
18

---

Published in Transactions on Machine Learning Research (04/2025)
Table 4: Dataset name and OpenML
Vanschoren et al. (2014) url
Dataset Name
Dataset URL
adult
attps://apl. openml org/d/1590
Amazon Amazon_eemllyeee access
wtt..ppmmm..o.//133.....3 org/d/4135
Bank marnk__arettin data set UCI credit
nttps://api. openm org/d/4423
http:/ommmmmoo...///0//22/00 org/d/4502
attps://api.og org/d/45020
Diabetes130US
http://pp..ooemm.rrr//o
electrcity
https://api... ope.ml...p.mmmmmm..m..mm..mm org/a/151
elevators
higgs
hedr
house 16H
https://ap3 htsps://api.opemml org/a/821
jannis
bstps://api.opant org/a/45021
achool-admission bianry
http://epp.oppmm..orr//// 0rg///33890/
Magic'Telescope
Medical-Apie Aepiral-tpppiimmmt---tee
atpps//epi..ppmmlooo////4
MiniBooNE
https:
/ttps://api.opeett openm/ org/a/44088
numerai28.6
nursery
https:
/ttps://api.opeemm..oo org///959
PhishingWebsites
pol
road-safety
stips://apl.opemmt org/d/44161
Click Click__preteeee _reeck_peeeeioo____ small
attps://ap..opem/ wrtpa//2gg//////////
2dplanes
http:://pp.oppeml..co org/a/227
A.2 Autoencoder implementation details
19

---

Published in Transactions on Machine Learning Research (04/2025)
Table 5: Metadata of each dataset seen in Table
4
Dataset
# Instances # Features
# Cont.
# Cat.
# Class 0
# Class 1
2dplanes
10,768
10
10
20,120
20,348
Amazon employeee_eeeeeeeeeeee access
32,769
1,897
30,872
Bank Bank_marketing data set UCI
45,211
16
7
39,922
5,289
Click Click__peeieeee _oii__preiction small
39,948
33,220
6,728
Diabeles130US
71,090
7
7
0
35,545
35,545
MagicTelescope
19,020
11
12,332
6,688
Pediral-Appointten--------- No-Showws
110,527
13
10
88,208
22,319
MiniBooNE
72,998
50
36,499
36,499
PhishingWebsitee
11,055
30
30
4,898
6,157
adult
48,842
14
37,155
11,687
credit
16,714
10
10
8,357
8,357
deefuut-o--crett--carrdeeeett
13,272
20
20
6,636
6,636
electrcity
45,312
8
26,075
19,237
elevators
16,599
18
18
5,130
11,469
hodr
10,000
22
5,000
5,000
higgs
98,050
28
28
46, 46,223
51,827
house 16H
22,784
6,744
16,040
jannis
57 580
54
54
28,790
28,790
hww-school-addiissooo--ann
20,800
6,694
14,106
numerai28.6
96,320
21
21
47,662
48,658
nursery
12,960
8
8,640
4,320
pol
15,000
48
48
5,041
9,959
road-safety
111,762
32
29
55,881
55,881
20

---

Published in Transactions on Machine Learning Research (04/2025)
Algorithm 3: TDColER: An expanded version of Algorithm to distill dataset S with N samples given: distiller F RN** x Y nm Y learnable column embeddings C: R x C R mmx(rto)
encoder Rm((tt) R
decoder m R² C
classifier f RD -,,
set of encoder hyperparameters θ as described in Tables 8 to
set of decoder hyperparumeterss as described in Table g
set of classifier hyperparometers as described in Table
and stage and HPO iterations and K' respectively.
our experiments,
set K 500 and K' 100.
1 Stage 1: Hyperparameter search for encoder /eabderrreederrrr for = I,..., K 2 02 ~v θ, Ça ~
// Sample encoder and decoder HP // Minimize RE
(C, vt) + minimize CR(C,o, v) from Equation 9
Record validation performance Brecon (fx, (k)
8*, 0''5* arg arngingg..s "recon (0x, Ck))
// Select the best HP Train final with best hyperparaneters
(C, v) train (rain(C,o,, Ci 0²0", entire training S.
7 Stage 2: Fine-tuning jointly with classifier head
for k ===,, =..... K' do ~ 92 0g [0.1] minimize Ln + aLc from Equation 110 Record va validation performance Uioins (wkk
// Sample classifier HP and balancing factor // Minimize joint objective
(C, t/
12 w".a" a=* arg min Vjoint (wo, (ootal a).
Select the best
(C, f) < train train(C, (C V. $:00 , a²)) on entire training set
11 discarded this step
Encoding and distillation steps $ <- ((d(C(x)) (2,,) S)
// Encode
18 R - F($)
// Distill in latent space
R- ((v((w),
(()(()) (x,
// Decode
return R, C,O,
Here we provide more details on the implementation and training procedure of the autoencoders used our experiments. Algorithm 3 shows the full procedure of the distillation pipeline, including the hyperpa- rameter optimization (HPO) steps for the encoder and decoder modules. Each of the hyperparameter and implemenation details are documented below.
A.2.1 Hyperparameter optimization for encoders
Tables 6 to 10 show the hyperparameters considered for different modules of the autoencoders. We use {x, y. 2} to denote a set of variables and (a, bj to denote an inclusive range of values. We conduct HPO for each autoencoder + dataset pair using an implementation of hyperopt Bergstra et al. (2015) from Ray Tune Liaw al. (2018) with a maximum of 500 samples for each HPO run. As noted in Section 2.1 we first train the vanilla autoencoders for each dataset using the encoder hyperparameters seen Tables to
Table 6: Hyperparameters tested for FFN encoder.
Hyperparameter
Values
hidden
(100, 200) [1,4]
o__iideen
dropout
(0, 0,,. 0.4)
d_ d_embedding
(10, 20, 50, 100, 200)
use_ @eee_mbedding
(True,False) 101-3.-1]
learning_rate
weight_decay
101-4.-1
1r schhueeduler
(None, Plateau, Cosine)
21

---

Published in Transactions on Machine Learning Research (04/2025)
Table 7: Hyperparameters tested for GNN encoder.
Hyperparameter
Values
graph_layer
(graphsage, gen, gat) (mean, softmax)
graph_aggr
n_graph
[1]
dige_direction
(bidirectional, multi multipass)
edge_dropout
(0, 0.2.
learning_rata
10
weight_decas
10 101-4,-1]
1r ir_scheduler
(None, Plateau, Cosine)
Table Hyperparameters lested for TF autoencoder.
Hyperparameter
Values
n_ blocks
[1, 10
2
n_ p__ttennton__hen
d_qkv
2/0.7]
10[-5,-1]
1ayer_norm_eee
pasr__orm_epp_
d_mlp
2
d_mlp_hidden
(100, 200)
n_mlp_hidden
[1,4]
dropout
[0, 0.4
learning rate
10 -3,-3
weight_decas
101-4,-1
1r scheduler
(None, Plateau, Cosine)
Table 9: Hyperparameters tested for decoders. The decoder architecture is kept the same for all encoders and optimized individually.
Hyperparameter
Values
d_ hidden
(100, 200) [1, 4)
n_hidden
learning_rate
101-a.-1]
weight_decay
101-4.-1]
Ir scccheuullrrr
(None, Plateau, Cosine)
22

---

Published in Transactions on Machine Learning Research (04/2025)
and decoder parameters seen in Table Once the vanilla autoencoders are trained, then conduct additional fine-tuning with classifier head with hyperperameters seen in Table 10 where is used to balance the objective functions of the decoder and classifier heads.
A.2.2 Discussion on parameter size of autoencoders
Here we expand on our parameter size of the encoder architectures of the autoencoders. This is worth noting because the distilled data is the latent space, the encoder module is required to project any new data the same space. Thus, the encoder is considered to be a part of the distilled output.
We can characterize the parameter size of each encoder architecture given D-dimensional binarized dataset with categorical features and continuous features that is projected to a d-dimensional latent space.
FFN. We used an FFN architecture with an M-dimensional embedding layer followed by H hidden layers that receive and output W-dimensional vectors. The parameter size of such an FFN is as follows:
O(DM + ()W + HW2 + Wd)
(5)
The column embeddings are of size O(DM), the input layer maps the concatenated (c r) M-dimenssioallll vector to hidden layer dimension W with (c r)M size. The hidden layers are of sizes O(W3) each for H hidden layers. The output layer maps the --dimensional hidden layer output to the desired d-dimensions.
GNN. We use a GNN encoder with H consecutive layers. The dimension of the vectors passed between the graph layers are fixed to d, meaning that M d. Thus, each graph layer maintains a by matrix to handle a d-dimensional input vector and output a d-dimensional vector.
O(Dd+ + Hd))
(6)
The column embeddings are of size O(Dd) since M = d. Each of the H GNN layers is of size O(d)).
Transformer. We consider an implementation of transformer autoencoder inspired by the architecture of FT-Transformer Gorishniy et al. (2021). The encoder has an M-dimensional embedding layer followed transformer blocks. Each transformer block takes in a sequence of M-dimensional embeddings and oututs a single d-dimensional vector. The block composed of a multiheadlattention module with m heads and FFN module to project the attention scores back the input space. We modify the architecture seen in GGorishniy et al. 2021) by allowing the dimension of the attention head to be configurable Le. instead of using M/m the dimension of single atlention head, allow the module to compute the atlention in dgkv- This choice is motivated by the fact that our encoders were trained with a latent size of 16, which may not be wide enough for the TF encoder. We then project the resulting embedding in dedemnnnnennnoo back to M-dimensionals with W.. Thus, each of Wg. Wt, W. and Wo has m. parameters. The MHA module then followed by an FFN module which takes a M-dimennional vector and projects it back M M-dimensions with a W-dimensional hidden layer.
O(H O(H(Adguu + 2MW))
(7)
Table 10: Hyperparameters tested for classifier head in SFT.
Hyperparameter
Values
d_hidden
{100, 200} [1,3]
hidden
dropout
(0,0.2, 0.4]
alpha
{0.3, 0.5, 0.7) 101-3.-1]
learning_rate
weight_decay
101-4.-1
1r scheueeuee
(None, Plateau, Cosine)
23

---

Published in Transactions on Machine Learning Research (04/2025)
Rasture Embodiings
Featrare Einbeddinngs
Row Fanheiddings
Festun
Feature
Row
Row Embedding
Festur 2
Festure
Now 3
Figure 13: FFN encoder.
Figure 14: GNN encoder.
Output
CCLS
Add
Dropout
FFN
T+1
W
LayerNorm
Adid
concat
Dropout
T
Scaled Dot Product
MHA
[CL)]
LayerNorm
W
W
W
Input
Figure 15: Modified MHA component.
Figure 16: TF encoder block.
A.2.3 Optimization function
For the decoder vi Rd - RR,, we consider multi-layered fully-connected feed-forward network. Given the encoder & and the decoder v,, we use a group-wise softmax operator a to map the output of the decoder to a per-input-fcature probability simplex: given an initial binary vector E (0, 1) constituting per-input-feature one-hot encodings (that is b [6* @ ... @ Betr] after binarizing the continuous features and one-hot. encoding the categorical features), and a decoder output Rº with per-input-feature constituents B (that is B [B'
Bet"], we apply the softmax operation each per-input-feature constituent to get b- - b' ... + [ett] E [0, 1]",, where b' softmax(B") We utilize the following per-sample reconstruction
loss:
E(b, = ट+r st
CE(b', (8),
(8)
logg 6
where CE is the standard cross-entropy loss between a one-hot vector and a softmax output, and [6*] is the length of the i-th constituent one-hot encoding in corresponding the number of categories (or bins) the i-th categorical (or numerical) feature. This loss weighted average of the per-input-feature cross-entropy loss, with weights (1/10z2 (+)) to normalize the loss across all features with varying number of categories bins.
The column embeddings, encoder and decoder are then learned by optimizing the following unsupervised loss:
Cn(C, () + Eiayes ((((6))
(9)
24

---

Published in Transactions on Machine Learning Research (04/2025)
where P is the data homogenizer, and a is the aforementioned group-wise softmax operator. Learning the latent representation in such an unsupervised manner makes this distillation pipeline agnostic to the choice of downstream model. Another advantage of this choice is that the decoder allows us to map the distilled artificial samples in the latent space to the original features, which might be necessary in some applications (for interpretability reasons).
A.2.4 Supervised latent space fine- fine-tuning
Given the alrcady learned encoder and decoder, we consider a supervised fine-tuning (FT) step where we utilize a classifier f Rdd + Y that utilizes the latent representation. f is initiallized with random weights and is updated alongside the encoder and decoder as they are fine-tuned by minimizing the following loss to ensure that the latent space is quite predictive while the reconstruction loss stays low:
Lo(l) = t Cian)es (E(y)(((P()))))
(10)
()(0,))+ale
+
(11)
where a > 0 is penalty parameter to balance the two losses, and CE is the cross-entropy loss. We consider multi-layer FFN architecture as the classifier f.
A.2.5 Encoder architectures
Fully-connected feed-forward network (FFN). This encoder first selects the column embeddings corresponding to nonzero entries in the binary representation b, concatenates them to get a (c+ + r- dimensional dense vectors (recall that 6 will only have e c r nonzeros out of the D dimensions). and inputs them to a fully-connected feod-forward network fu (R(+r)M + Rdd The encoder é : 10, - Rd can be written as:
= xb) p( i (1,..., D) b|] -])),
(12)
where b[i] is the i-th entry of the D-dimcnsional vector, and @ is the concatenation operator. The FFN μ and the column embeddings {w,, i E {1,..., D}} constitute the parameters of the encoder é. For a FFN with H
hidden layers, each of width W, the total number of parameters in this encoder is O(DM + (c+ rrWW + HW2 + Wd). Figure shows a simplified architecture of the FFN encoder.
Graph neural network (GNN) encoder. We also consider more recent encoder for tabular data proposed in Wu et al. (2021). A bipartite graph is constructed between the column embeddings (w;, (1...., D}} and the (zero-initialized)) row (sample) embeddings {2₃ E RR E NJ, with bidirectional edge between w, and 2ÿ if the by = 1, where b, E 10, is the binary representation of the j-th sample. Given the (learned) column embeddings, the row embeddings are obtained via multiple rounds of message passing through multiple GNN layors. This can be written as:
o = (n Agg(u, ,
(13)
w (n 1-1 AgB(=) NN)),
where iy is the h-th GNN layer, Agg is an aggregation, N₁ (or N2) is the neighbor set of the i-th column embedding (or j-th row embedding). We set the initial 29 = 0 (zero-inttialized row emboddings), w wi and utilize 2/ as the latent representation for distillation after H GNN layers. While Wu et al. (202)) only considerod Graph Convolutional Networks Kipf & Welling (2016) as GNN modules, we extend it GraphSage Hamilton al.] (2017) and Graph Attention Networks Veličković al. (2018). An important aspect of the GNN encoder is that the desired row embedding size d must match the column embedding size M, thus d = M. With H GNN layers, the total number of parameters in this encoder is usually O(Dd+ Hd),, which can be significantly smaller than the FFN encoder with moderately sized FFN (large enough M, W). Figure 14 shows the graph formulation (left) and the GNN encoder architeture (right).
Transformer encoder. Finally, we consider a transformer-basedd autoencoder inspired by the architecture of FT- Transsoffrerrr Gorishniy al. (2021). This encoder uses the same embedding layer as the FFN encoder, which is then followed by transformer blocks. We learn additional cls embedding, which is placed before all
25

---

Published in Transactions on Machine Learning Research (04/2025)
other tokens in every sequence. Each block takes in sequence (one row) of d embeddings, and is composed of multihead-attentionn (MHA) module and a feed-forward network (FFN) module.
For a MHA module with m attention heads, we modify the architecture seen in (Gorishniy et al. 2021 by allowing the dimension of the attention head to be separately configurable - i... instead of using d/ /mm as the dimension of a single attention head, we allow the module to compute the attention in dgkv- This choice is motivated by the fact that our encoders were trained with a latent size of 16, which may not be wide enough for the TF encoder. We then project the resulting embedding in m-dimeenon back to d-dimension with Wo. For an imput wi at the ith transformer block, the computation for the MHA module is as follows:
W (w: (wxr V dggeo
a; W.' (softmax
Ww((w))
(14)
The resulting attention score a, is then added with the original embedding and passed through an FFN module. Similarly to Gorishniy et al. (2021), the [cls] embedding is used as the final output of the encoder. Figure 15 shows our modifiod MHA component, and Figure T6 shows the TF encoder block.
Additional encoder architectures. While we do not consider any additional architectures in our experiments, note that any architecture that is capable of encoding table row to a vector can be used in place of these. One example is Variational AutoEncoder (VAE), which aims to learn more robust representation by injecting noise in the training process. Our observations suggest that if different representation learning schemes like VA do indeed result in beller embeddings, we can expect the quality of the distilled data to increase. However, it also worth noting that we already observe high sest-time roconstruction accuracy with the existing (especially TF and GNN) encoders, so the difference may not be significanl.
A.3 Distill Methods
A.3.1 Choice of Distill Methods (KIP, GM)
The clustering-based distillation schemes and KIP are not explicitly tied to specific model and thus satisfy our desiderala of model-aggoostiiil, In contrast, the Gradient Matching GM distillation scheme heavily relies on the choice of the backbone model Mo (as well as the learning algorithm parameters such as the lcarning rate), and there is no guarantcc that the distilled samples R would be useful for any other model. Thus, this scheme not model-agnostic. However, we consider GM to be representative of the model-specific distillation schemes for the sake of completeness of our evaluations. For our table distillation, we choose Mo be a multi- layered perceptron with a single hidden layer. This will pose a mismatch when we evaluate the quality of the distilled data R on standard tabular models such decision tree ensembles and nearest-neighhor models, highlighting the need for model-agnosticity in tabular data dist illation.
A.3.2 Distill Method Implementaion
k-means We use the sklearn. cluster. KMeans from Pedregosa et al. 2011 with the n_ init set "auto".
Ag Agglomerative We use sklearn. cluster .luster.Agglomeeee AgglloerrativeeChsstrrrgg from Pedregosa et al. 2011 with the linkage set to "ward". Because agglomerative clustcring docs not have a "controid", we manually calculate a euclidean centroid for each cluster by using sklearn. neirh..eeebbrrs www.ighbbrr..eeretteetttrror to compute the centroid or the closest real point.
KIP
We use the implementation provided by Nguyen et al. (020 available at https:///githu..co
GM We use the implementation provided by Zhao et al. 2020 available at https:://iithbb.com
WIGD-lor/Dattaattoodddmmtttto
26

---

Published in Transactions on Machine Learning Research (04/2025)
Table 11: Parameters of distillation methods.
Method
Hyperparameter
Value
Description
distill space
Whether to use the encoder latent space or the raw binary representation.
Common
use use_closest*
Whether to use median points instead of the euclidean center. Only applicable to clustering methods.
output_spa
Whether to keep the encoder latent/ decode raw binary space. The binary space is only applicable clustering methods when use_ pee_est set to True. Random seed for distillation algorithm. Not applicable agglomerative.
the
randon_seed
n_epochs
1000
Number of epochs to train the distilled data. Width of the neural network to compute the NTK of.
KIP
mlp_dim
1024
n_epochs
500
Number of epochs to train the distilled data. Size of the hidden layer of the target model.
mlp_dim
1024
GM
n_ n_layers
Number of hidden layers the target model.
1r 1r_mlp
0.01
Learning rate for the target model.
1r _r_data
0.1
Learning rate for the distilled data.
mom mom_data
0.5
Momentum for distilled data.
Table shows the parameters available for each distillation methods. The common parameters are used for every algorithm, with the exceptions marked on the right-most column. The method-specific parameters for KIP and GM are for the original algorithms proposed in Nguyen et al. 20200 Zhao et al. (020))
A.4
Downstream Classifier Hyperparameters
Table
12
shows the hyperparameters used for each downstream classifier. We use scikit-learn Pedregosa et al. implementation of Naive Bayes, K-Nearest-Neighhoon, Logistic Regression, and MLP, and Gorishniy
201
ct al.
2021 is implementation of FT-Transformer and ResNet.
A.5 ResNet and FT-Transformer performance
We test ResNet and FT-Transformer for 5 datasets. We found that even with early stopping, the two classifiers take significantly longer to train given the same computing resources. On average, we find that ResNet takes around 10 times longer to finish training, while FT-Transformet takes around 28 times when compared to XGBoost. We also find that the performance resnet and Tranffosfrrerr does not stand out in fact, the average test performance when trainod on the full datasct shows that both ResNet and FTTransformer show similar performance to MLP, and are outperformed by XGBoost. A similar trend is also observed by Grinsztajn et al. (0022 and McElfresh et al. 2023 who note the superior performance of GBDT models.
A.6 Performance of GM in original space
Figure X showed the performance of distillation methods in various spaces including the oringal space. We note that the performance of GM in the original space is almost shadowed by the performance of random sampling. Upon inspection, we find that the GM algorithm docs not learn any meaningful updates past the first. randomly sampled points when using the sparse original represenlation. And because we control the random seed for every experiment, the resulting points also end up being very close to the randomly sampled points, leading the overlap the results plot. In fact, this further highlights the necessity of learning dense transformation of tabular data for some distillation algorithms begin showing meaningful performance gains (over random sampling).
27

---

Published in Transactions on Machine Learning Research (04/2025)
Table 12: Hyperparameters of downstream classifiers.
Classifier
Hyperparameter
Value
d_token
128
n_blocks
2
attention fi____heaa
8
attention, attention_droph
0.15
1.25
FT-Transformer
ffn_
ffn_dropout
0.05
driidual_dropout
0
learning_rate
10
weight_decay
10-5
early_stopping
True
Naive Bayes
var ear_smoothing
10
n_neighbors
5
K-Nearest-Neighbor
leaf _size
30
P
2
penalty
12
tol
10-4
Logistic Regression
1
solver
1bfgs
d_hidden
100
n_hidden
1
MLP
learning
10
early_stopping
True
n_blocks
4
d_block
128
chidden_multiplae
1.25
dropout
0.2
ResNet
learning_rate
0.0001
weight_decay
0.00001
early_stopping
True
patience
16
Table 13: Average train/test times and test performance comparison for all downstream classifiers.
Classifier
Train Time
Test Time
Test Perf.
FTTrrasformer
281.3431
0.17934
0.7879
NB
0.0030
0.00232
0.6624
KNN
0.0007
0.54309
0.7474
LR
0.4901
0.00646
0.7709
MLP
2.4444
0.00554
0.7826
ResNet
154.9824
0.08508
0.7833
XGB
11.4055
0.01439
0.8180
A.7 Determining the best overall performance
We describe the best overall pipeline in Section 4.4 and Table Here, we provide a more detailed explanation of how we determined the best overall pipeline. The runs are grouped by their classifier, dataset and distill size n. Similar to other parts of analysis, the grouping done in order to ensure that the comparisons are fair. In this instance, we interested in only the pipeline components that lead the best classifier performance, regardless of the exact classifier kind. Thus, group every run by their non-pipeline-speefic parameters,
28

---

Published in Transactions on Machine Learning Research (04/2025)
XGBClassffer
XCBCiasaffer
XCBClessifeer
XGBClasalfer
Loglsticeerrssinn KM
LoglitisRegresssnn AG
LogliticRegresson GM
Logistic LogisticRegresset Regrossion KIP
AW cue33
KWNeighhorsClessffee KM
KNeighborsClessiin lassifierr AG
KHeighhorsClessifeee GM
WteightorsClassifeee
KIP
1.-
1.-
0.5
Reletive Regret
0.0
MUPClassifier
MLPClassifier
MLPClassifier
MUPClessiier
1.5
FTTranstormer KM
FTTrnnsormmr AG
FTTranstomer EM
FTTranstomer KIP
SR LONE:
5254 Nik C054
ResNet KM
Restiet AG
ResNet- GM
ResNet KI
1.0
0.5
0.0
20
40
60
80
100
20
60
80
100
20
40
60
50
100
20
40
60
50
100
AD
Instances Per Class
Figure 17: Full results of distillation methods by downstream classifiers.
which are the classifier, dataset and distill size n. In each group, we then count the instances the pipeline places on the top 3 in terms of the regret score and sum up the counts for each pipeline.
Following the previous findings, Table 3 shows that k-means based methods have the best performance, placing in the top 3 with all SFT encoder variants. Surprisingly, we also find pipelines that KIP and GM as the 4th and 5th best performers. While KIP and GM to place on lop, this result shows that there are exist some conditions which leads the pipelines using gradient-based based methods (KIP, GM) to be the top performer. On the other hand, the consistent rank placement of pipelines that use the autocncoder latent space shows that fine-tuned autocncoders can indood boost the performance of distillation methods significantly.
not able to determine any specific conditions that cause
A.8 Training/Inferrnce setting of HPO example
Here, discuss the detailed setting of the HPO use presented in Figure g A dataset D is split into train, validation and test sets Dte; D Dre. Only Dur is used to produce the distilled dataset R (i.c. S = D.r in Algorithm 1.). Then either Dtr or R is used to train the HP of the downstream classifier, and the HP is adjusted by the performance of the classifier on Dva. The final performance of the classifier than evaluated on Dre: which the classifier has never seen in any of the settings before. This sctting assumes that one has access to two of the splits, Dur and Dva: to produce the final classifier, and they may choose speed-up up the HPO process by reducing the size of Drr.
29

---

Published in Transactions on Machine Learning Research (04/2025)
Table 14: A comparison of relative regret scores of distillation pipelines that use the encoded space of autoencoders trained with different column embeddings, tested on 5 datasets (Adult, Amazon Employee Access, Credit, House, Phishing Websites). The center value shows the median relative regret, and smaller values each side refers to the first and third quantile, respectively. In general, PLE embeddings show the strongest performance. However, it is worth noting that PLE embeddings are not applicable to GNN encoders, and that binary embeddings also show superior performance to scaled embeddings.
Col. Emb.
KM
AG
GM
KIP
Binary
0.1082
0.5645 0,7886
0.0976
0.4633
0.7181
0.5504
0.9038 1.0083
0.6551
0.9254
1.1918
Scaled 0.7214
0.8613 1.0871
0.4908
0.6939
1.0249
1.0092
1.4412 1.8658
1.3304
1.6137 2.2985
PLE -0.2228 0.1976 0.9305
D.2699
0.2173
0.6762
000866
0.2747
1.0524
-0.0263
0.7398 1.3923
B Additional Analysis
B.1
Full results of distillation methods by downstream classifiers
B.2 Effect of column embedding scheme on downstream performance
While column embeddings are standard for categorical columns- - each category is represented with a vector, there are various ways of embedding numerical columns: (i) A numerical feature can be binned, and each bin treated as a category with an embedding w E R" corresponding to each bin. (ii) With linearly scaled column embeddings, a single column embedding w E R used for each numerical column, and the column embedding for particular numerical value V E R obtained by scaling w to e w. (iii) Piecewise linear encoding PLE /Gorishniy et al. 2022) also bin the numerical feature but use a more sophisticated way of generating the column embeddings for given numerical value. We considered binned numerical features in the main paper for a couple of reasons: (a) Binned numerical features naturally handle missing values (quile prevalent in labular data) by maintaining a "missing" bin instead of relying on a heuristic intermediate imputation step; sometimes, the fact that a value is missing is in itself a signal, and heuristic imputation schemes often lose this information. (b) The binned features can be used for all architectures we consider here- FFN, Transformer, and GNN and using common embedding scheme allows us to ablate the effect of the different architectures. The other numerical embedding schemes do not apply to GNNs.
To understand the effect of different kinds of column embeddings schemes, conduct smaller scale experiment on 5 datasets. Specifically, we compare scaled embeddings as seen in Gorishniy et al. (2021), piecewise linear encoding (PLE) as seen in Gorishniy et al. (2022), against using binary column embeddings where continuous fcatures are binarized by binning, and examine the downstream performance of distillation pipelines that use the latent space of the autoencoders trained with the corresponding column embedding scheme. Table 14shows that using the both binary column embeddings and PLE consistently leads to lower regret scores compared to scaled column embeddings. While PLE embeddings show the strongest performance, they are not applicable to the GNN autoencoder architecture. Thus, we conduct most of our experiments using binary column embeddings for fair comparison across different autoencoder architectures for a fair comparison.
B.3 Pairwise comparision of distillation methods.
In addition, we compare the downstream classifier performance with every pair of pipelines that use different distillation methods under otherwise equal settings. The left table of Figure [8| reveals that KIP had the highest tendency to underperform other distillation methods, while k-means had the highest tendency to outperform other distillation methods. This is consistent with our previous findings, where h-means outranked other distillation methods most frequently. In order to gain further insights behind the performance lag of graident- graident-based distillation methods, we conduct a pairwise comparison of the distillation methods for different classifiers as well. The center and right tables of Figure 18 shows the pairwise comparison of distillation methods for XGBoost and MLP as downstream models. This suggests that madieet--assed methods underperformance is not solely due to its kernel, but that tabular data itself may pose unique
30

---

Published in Transactions on Machine Learning Research (04/2025)
Overall
XGBoost
MLP
RS
0.51
0.72
0.48
0.61
0.69
0.83
0.68
0.76
RS
0
0.48
0.7
0.45
0.64
AG
0.48
0.7
0.4
0.64
AG
0.31
0.62
0.4
0.57
AG
0.52
0
0.73
0.39
0.68
KIP
0.28
0.27
0.22
KIP
0.17
0.36
0.28
0.34
KIP
0.3
0.26
0.21
0.34
KM
0.51
0.56
0.75
0
0.7
KM
0.32
0.56
0.7
0
0.66
KM
0.55
0.6
0.79
0.74
GM
0.38
0.33
0.67
0.27
GM
0.24
0.4
0.64
0.32
0
GM
0.36
0.32
0.65
0.25
0
R5
KIP
RS
KIP
KM
GM
RS
AG
KM
GM
Pairwise comparision of distillation methods. The relative performances distillation methods under otherwise equal sttings. Rows denote win ratio, columns denote loss ratio.
Figure 18:
1.6
Encoders
Original
1.4
TF
GNN
1.2
X FFN
Relative Regret
1.0
GNN
FFN*
0.8
Distill Methods
0.6
Original
0.4
RND
KM
0.2
AG
KIP
0.0
1000
2000
3000 4000
5000
Runtime (s)
Figure 19: Runlime us. Regrel The runtime vs. regret score for distilling the data for each pipeline. The times include the autoencoder's 's training/eneon encoding time if it is used. The colors refer to different distillation methods, while the encoders used are differentiated by the shapes. Closer to lower left corner better.
challenge in distillation that is not seen in image data. It is also worth noting that while the clustering-baseee approaches had the best overall rank, random sampling proved to be a strong baseline with a near 50% win ratio against them.
B.4 Runtime comparison of distillation methods
Figure 19 shows the runtime of distillation methods vs. regret score with different encoders. As expected, we find that the clustering methods k-means and agglomerative clustering show much lower runtime. As expected, using the encoders in addition the the distillation methods adds more overhcad, but the clustoring methods still show much shorter runtime in average. It is again worth noting that the GNN encoder provides competetive performance while taking much less time.
31

---

Published in Transactions on Machine Learning Research (04/2025)
XGBClassifier
1.0
0.9
Relative Regret
KM
KM TF
0.8
RND
18.13%
0.7
0.6
25
50
75
100
Instances Per Class
Figure 20: Multi-class dataset. The performance of distillation methods on the Eye Movements dataset.
TEaPOR Far
Figure 21: Effect of changing latent dimension size. The performance of distillation methods on the Bank Marketing and Phishing Websites datasets with different latent dimension sizes.
B.5 Multi-class dataset
This work focused on binary classification datasets. However, our proposed framework easily extendable multi-class datasets. As an example, we conduct additional analysis on the Lyye Movemente dataset that has 3 classes. Figure 20 shows the comparison of k-means distillation when applied on the dataset with and without the TF encoder. We find that the trend is the same with this dataset using the encoder shows steady improvement in performance consistently throughout the IPC values.
B.6 Affect of changing latent dimension size
In this work, we focused on using autocncoders with a fixed latent dimension of 16. However, the latent size is another hyperparameter that can be set to different values. In this subsection, we showcase a mini experiment using the Bank Marketing and Phishing Websites datasets on the TF autoencoder with SFT in increasing latent sizes 16, 32, and 128 with a XGBoost downstream classifier. The results show that the performance of the downstream classifier increases as larger latent sizes are used. However, The change between 16 and 32 is appears more significant than the change between 32 and 128. This suggests that while larger latent sizes may be beneficial to degree, the relationship may not be linear.
C Documentation of TDBench
The information in this section is also available in markdown format in the README. md file of the supple- mentary material.
http:://ppp.opemtoor//////
32

---

Published in Transactions on Machine Learning Research (04/2025)
C.1 Reproducing Results
Every plot and table in the main paper can be reconstructed using the following scripts:
Q0 jo__xperrment_scoe
Q1 [_____co_emmeds... embeeee.bpp..... py
Q1_encoding.PP py
Q2 aediitiillmmehhhtee py
03_auttonncoders py
Q4_1_runtiim..p!
v4_2_get_hpoodde
04_2_hpo.py 04_combinations py
05_class_imbal.
The scripts are organized in order of the question addressed in Section 4 and will be populated iclr-figures directory. These can be simply ran by calling python SCRIPT_NAME
The following files are included in the supplementary material and contain all the necessary information for the scripts:
settasetsstat... csv
enc enc_stats. csv
*data Blatammode___ https/mmdddssihhheeeeeeeeeeeeeee
mpo-measure/
mixedd_r tf off_results.oo csv
pple_tfrresuee fiee_tfrresutts.cmm
The files marked with asterisk (*) are not included in the repository, but can be downloaded from this url:
_lnkk
C.2 Description of the workflow
The ## Running Code section of README, md file discusses the actual commands and available options for rumning each stage in detail.
The procedure is as follows:
Train the autoencoder with the desired configuration.
(Optional) Fine-tune the autoencoder with a classifier head. Run distillation methods against specified downstream classifiers.
C.3 Constructing a new pipeline
Changing default parameters The configurations for this project are managed by hydra and can be modified by adding new fles/directories under the "config' directory.
Adding new datasets Adding new datasets is simple as adding a new config/data/ contigg/ataa/aataaee///DAAEEE HAME.yaml file. Currently, only openml datasets are supported.
The following flags must be specified for the dataset to be correctly loaded seen in Table 15
Adding new preprocessing methods The preprocessing is handled by the TabularDaaaModuule object that lives in tabdd/ /ttaa//aaaaaaaaarrtatttttllllll py. The preprocessing strategies are identified by a string, and can be configured under config/data/node The fields seen in Table [6 must be specified for the preprocessing work correctly. One can additionally define any type of scale mode or bin_strat which will be consumed by the TabularDDaaMocuule.
33

---

Published in Transactions on Machine Learning Research (04/2025)
Field
Type
dataset name
string
download_ _url
string
label
string
n_ classes int
source_type
string
Table 15: Configuration details for datasets
Field
Type
parse_mode
string
scale scale_mode mode
string
bin_strat
string
n_ b_biss
int
Table 16: Configuration details for data preprocessing
This object is configured with DatasetConf ig and DataModeConfig. The DatasetConfig is the configuration for the dataset, and the DataModeConfig is the configuration for the preprocessing method.
It's TabularDataModule. prreaparaaallllepppparee data is the method that will parse the data accordingly and save to cache. One can add arbitrary preprocessing methods in this file by adding new flags DataModeConfig and handling it inside the prepare data method.
Field
Type
is ss_drndom
string string
is_ cl.uuter
can San_use__n encoder
string
args
int
Table 17: Configuration details for distillation methods
Adding new distillation methods The distillation methods are identified by string, which should have configuration with the same name under following fields seen in Table 17
Once can characterize the method the
is random: Whether there is randomness in the method. If true, the pipeline will be ran multiple times. is_cli csccuuutr: Whether the method is clustering method. If true, an option that uses the nearest-to-center method will be included.
can_use_encodee Whether the method can be applied in the latent space.
args: any additional arguments to the act ual function.
Once the configuration is created, it will be consumed by habdd//issti11loaddddass distilleddddta.ppp data.py. This method can then be modified to include the new dis- tillation method.
data method of
Adding new encoders All encoders used in the benchmark are subelasses BaseEncoder from tabd///adell/aencdder/aaas encodere.coopeeeee py. A simple example of how to implement can be seen in hagpd///oddllsseencodddrr/aapppp autoencder...ppp The module needs to encoder the following methods: _iint__ encode, decode and forward.
The autoencoders are specified by the configuration files in config/oncoder/mooelss// The class of the encoder is specified by cls, and the hyperparameters specified by tune Lune_params.
34

---

Published in Transactions on Machine Learning Research (04/2025)
Table 18: Relative regret of pipelines that use different combinations of distill methods and encoders at IPC=10 for XGBoost downstream classifier. The best value for each column is marked with bold, and the second best is marked with underline. (FG: Forgetting, GN: GraNd, GL: Glister, GC: Graph Cut)
Distill Method
Regret
Min
Mcan
Median
Q3
Max
Random Sample
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
KM
0.0597
0.5256
0.6682
0.6654
0.8186
1.1094
AG
0.0000
0.5177
0.6301
0.6036
0.8914
0.9965
KIP
0.6728
0.8483
1.1109
1.0544
1.2523
2.2713
CM
0.4175
0.7707
0.9858
0.9377
1.1461
1.7292
FG
0.8705
1,1400
2.3837
1.4465
1.8731
16.0146
GN
0.7748
1.1498
2.0530
1.3704
2.2624
10.6670
GL
0.8376
1.1000
2.0907
1.3146
1.6823
14.1625
GC
0.6361
0.9077
1,5084
1.1031
1.5998
6.8392
MTT
0.4175
0.7707
1.0340
0.9699
1.2176
2.3026
DATM
0.4175
0.7707
1.0340
0.9699
1.2176
2.3026
Table 19: Relative regret of pipelines that different combinations of distill methods and encoders at IPC=10 for MLP downstream classifier. The best value for each column is marked with bold, and the second best is marked with underline.
Distill Method
Regret
Min
Q1
Mean
Median
Q3
Max
Random Sample
1.0000
1.0000
L.0000
1.0000
1.0000
1.0000
KM
114.4491
0.0517
-1.4423
0.3637
0.5900
0.9920
AG
-1539655
0.0597
-0.9031
0.4098
0.6199
1.4982
KIP
-4.1619
0.3722
1,4318
0.8068
1,2664
11.1034
GM
-3,8002
0.4215
0.7141
0.8443
1.1779
4.9450
FG
-2.6999
1.3400
3.3639
1.8150
3.5621
18. 18.6769
-5.4217
1.2375
2.7412
1.6194
2.5591
13.6653
GL
0,0931
1.1744
3,3400
1.5641
2.2287
14.2601
-5.7194
0.5772
L.5557
1.0039
1.4415
11,4803
MTT
-3,8002
0.4215
L.0997
0.7876
1.1894
6,2999
DATM
-3.8002
0.4215
1.0297
0.7876
1.1894
6,2999
Additional Analysis
D.1 Dataset Feature Correlation
We further investigate the presevation of feature corrclation in the distilled data. Figure 22 shows the change in feature correlation in the original, randomly sampled and distilled with k-means in the latent space of TF-SFT in 3 datasets Credit, Magic Telescope and Tencent CTR.
D.2 Relation Previous Work
Kang et al. (2024) presented a preliminary abstract on work that explores data distillation for tabular data. The authors utilize an MLP and GNN based autoencoder networks to transform the data to before distilling and show that simple clusetering-baat based methods can outperform competetive distillation algorithms proposed in computer vision (KIP [gguyen al. 2020))
Building upon this work, our work provides a comprehensive analysis of distillation methods on tabular data, and provides a detailed comparison of distillation methods across a wide range of datasets and classifiers. We also provide a detailed analysis of the effect IPC on the performance of disuillation methods, and provide insights into the effect distillation on the feature correlation of the data.
Below, we provide a detailed comparison of our work with the preliminary abstract presented by Kang et al. (2024):
35

---

Published in Transactions on Machine Learning Research (04/2025)
Original
Random Sample
k-means
1.0
1.0
1.0
0
1
7
0.5
0.5
0.5
5
0.0
0.0
0.0
6
7
8
-05
8
-05
-0.5
109
109
109
-1.0
-1.0
-1.0
4
6
8
10
2
6
8
10
0
2
6
8
10
(a) Magic Telescope
Original
Random Sample
k-means
1.0
1.0
1.0
2
0.5
0.5
0.5
3
4
5
0.0
0.0
0.0
6
7
109 8
-055
-0.5
-0.5
109
10
110
-1.0
11.0
10
(b) Tencent CTR
Figure 22: A side-by-sine comparison correlation numerical features in the training data before distillation, after random sampling@IPC-I1 and after distillation&IIP-1100 While some weaker correlations are not. entirely accurately portrayed, the distilled data preserves the stronger correlations remarkably well.
36

---

Published in Transactions on Machine Learning Research (04/2025)
We conduct comprehensive comparison of different binning methods and their effect downstream performance.
We test with a transformer-based autoencoder, and show that it outperforms MLP and GNN based autoencoders.
We additionally consider gradient matching Zhao el al. (2020) as an additional baseline to represent the
gradient-based family of distillation methods Cazenavette et al. (2022); Zhao Bilen 2023 Guo et al. (2023)
We provide complete python package, TDBench, that can be used and extended by anyone in the community.
We explore a realistic use case for data distillation in the context of HPO and show the trade-offs in utility and cost saving.
We introduce a relative regret metric to compare the performance of different distillation methods across datascts and classifiers.
D.3 Data distillation in natural language processing
While data distillation has been proposed and explored mostly in the domain of computer vision, this technique has also been studied in the domain of natural language processing. The discrete nature of lext. data, in contrast image data, is closer parallel tabular datasets. Previous works have approached by adapting tochniques from vision (Wang et al. 2018) to language domain (Li & Li 2021 Han et al. 2022 with
set of pretrained word embeddings and report success. On the other hand, (Sucholutsky & Schonlau 2021
propose an algorithm for distilling language data by first distilling the token embeddings with modified version of Wann et al. (2018) and locating the nearest real token of the distilled embeddings to produce human readable distilled text. Mackawa et al. (2023) extend this approach by using attention scores of a pre-trained language model (BERT) to produce distilled text dataset that can be used to fine-tune the important same language model. Similar to the dataset distillation techniques seen in image datascts, models, and to note that these works leverage the existence of well-trained word sereeddinnssllagguue thus are sensitive to those architectures. Maekawa et al. (0024) overcome this obstacle by using a generative model to directly generate the text data. Among these works, one core factor stands out - the existence of a well-trained latent representation of text data. However, such a universal representation scheme does not yet exist for tabular data. And in this work, we show that learning a good per-dataset representation can be leveraged to close the gap between the literature from dataset distillation in natural language.
D.4 Raw balanced Accuracy score
Below is a comparison of the raw balanced accuracy of each distillation pipelines averaged over random iterations. Table 20 shows a comparison of all 10 distillation methods that were ran with TF-SFT encoder and lested on XGB downstream classifier, and Table 21 shows the performance of the baseline methods when applied without the encoders. Tables [22] and 23 show the same comparison that with and without TF-SFT encoder for the 4 baselines methods (k-means, agglocrmative, KIP, GM) on KNN classifier, and Tables 24 and 25 show the same for MLP classifier.
The last two rows of the tables each denote the number of instances that the pipeline ranked at the top, and the number of times it outperformed random sampling. The results show that random sampling is not a trivial baseline for many methods, and that both clustering methods, AG and KM, show the strongest performance. We also see that adding the encoder to the pipeline significantly increases the downstream performer of all representative models.
37

---

Published in Transactions on Machine Learning Research (04/2025)
Table 20: Comparison of raw balancod accuracy scorcs of distillation methods applied with TF FF- on XGB classifier. Last two rows of the tables each denote the number of instances that the pipeline ranked at the top, and the number of times it outperformed random sampling. Best performance at for each dataset is marked in bold, and second- best performance is marked with underline.
20.011
0816
0.5339:0 0.22
0.5169 Lo
0. 0.5000-10 1000000000
0.62881 0088110.0000 0380
0.5181120 10112-00200 m2202
1000009
0213
0-5270-10 2:0 0213
0.6239 10.0270
0. 0.08G0 L 11.0008
0. 88000 0102
0 0.022 #0 0000
0.62043000020 000001133
0. 0.5012 5012+000000 0000
0. 0,0120 80 040002222
0.81983 0151
0:0110:0000 0201
0. 0.5008 #0.0020 02244
0.7804 0.
0.0040:00
0.7718
0.0247.50.00
MT
3/223
18/23
0.7426 0. 0930
7418-00010
0 0:00
0.01 78
0 0200
8 10
N.717700.018
0 0004
0115
0 0308 0185
0.078==0.004
0. 00265000000 0002
0. 0.00228-0 6.028-6.2226
0.8128=0.000 0.042
0. 0.5028-0 0158
0.0000
0.0000
0.0080=00000
0-0010:0.000
0. 0.
0 6377
18/23
K.M
0/223
0.717420 0017
1713840.0025
0.0009 10 00880
0.5761 1000004
0.307 10.00
0.114110.0
06111 0O.0000
0.0220 Lo. 10.09RS
6778
o SG922
o 0..602000000 0108 D8
63543000000 0073
#0022000002
0. 0.0272-0.00 0.0101
0.8483.00000 0660
0.778200.088 0810
0.5005000000
0.630740.022
0.805620.045
3/23
18/23
100004
0. 0.7344+0.00 0387
0.88211 0483
0.070310 0243
6.686 04SE
185.1...0000 00777
5000 0176
6.600 Lo 0228
0.081410.00 0001
0000110 0506
8 0533
6092 00.9
0.000000 1288
6671840.003 0733
0.6208400 0083
0.700040.01 1894
0.640
0.009
1/23
11/23
6C
0.002 0.0241:: 0000
0.0000=0 0007
0090.. 0 0007
D. 00009800000000 0356
0000 0220
A. 2017
0. 4T2N0 0303
1001
0. 00717L. 1001
8 8OTT8E20 04006
L.CT9900 1078
00900000000 0201
0. 600990000000 0110
011722000-00
0. 0.4078000000 0.009
0 0.021100.000 0.0011
1011100.044 04555
0.540=00004
0. 0.0942-0.011 00160
0. 3030000000 0,888
0.519900
0. 7526
04600
ON
0/22
2/23
0,024110 0000
0.3830+0 0025
0.000 1. 0057
0.0 0.5 11. 01
0226
1.3012100000 000202000000 000015
1..12221..00 05
0.531 D 0009
0.578810 0400S
1078
0.0769100111 1078
O. 0.64000 0.000-0..00 2663
1.808100001 0150
0.498 0.498 0042
00442
00011+00000 0601
0.6 181195400000 0669 0860
0633
16.662200000 0636
0.0362# 0016
0116
0.761850 1152
0.546420.004
0.4000:00
0.621
0/23
2/23
G1
0,7804 - 0816
0.7000==00024
0.3663 00
0436
0.0000 0.0000
D. 0978
0.5270=0 0213
02239 0 0276
0.0863 00 0100000
0.880000 01.99
03:13 0008
08844 0.0027
0.3012 - 0040
- 04102
17718860000 0786
0.5108 0108
0.80 0.807020.0020
0.0205
0.028820.00
0.3953-1
0:3000
=
0.0128
4/23
18/23
0.77920. 0468
400000000000 0855
0 6247740 22.7=000588
o 4006 Lo.007
0. 3.3661000000 0289
4887 L0.01.00000 10 0069
0.3 033355 133311 220222000000 0009
4000 10.08
o 3396 Low.ooo. 0.0264
0. 6360 16306.o.0000 10.00000
070
8246100000 1000000
o 5900 4005
o 0.8000 0.8000-000000 00001
000000
0 8326 03855 0.0088
0. 0.80000000000 1908
0.0002600 10.1177 1173
1.8330000000 60.0884
0.6219:0.028 22.000
0. 20.00 1800 16.00.0000.
1640
3000 00.66.0.0084
0/23
6/23
0.730420.00 08
0.7010+000020 0124
1.546310 T0
6.5 1.500000 04300
0 0.8288 0. 10.0300
6.5 1.5ter11..022 027777
Lo
S2roLo 0213
02329 1o 0276
01666 10062.0.0000 aro0
senozo 01.99
00 0003
6264100.013
0133
60122 00-0
1.0126-000012 12
19198-00.013
(718820.0732 07335
0.6110000000 0251
0.850800.000 0224
600910
DATM
3/223
18/25
(77409 D. 21
0-766940000 0208
0-521510 10020
0070
10.000
HG 0 none
O 0.00100 0
0
0.930810.0100 0191
0.9466-000000 0C0T
0646250-0120
0008
B0STT 0088
0.8084+0.0237 07.25
0 0037
0033
055022000000
0.7976=-0000
0.794450.00
0.7887
18/23
3/223
AG
Datavot
Vor BND
AD
4B
BM
CI
C9
DB
RL.
163
LA
MT
MA
MB
NU
PW
TC
TD
38

---

Published in Transactions on Machine Learning Research (04/2025)
Dataset
AG
GM
KIP
KM
AD
0.6078+0.088
0.7175#0.04223
0.5353 0502
0.5949::0.066
AE
0.525210.0195
0.530410.0019
0.5060110.000
0.5007:10.0126
BM
0.5599:10.0525
0.581610.0445
0.5114=:0.01
0.5842:1.00914
CR
0.5667110.440
0.539410.0334
0.5106 1. 0.083
0.5612.10.0577
CD
0.5887:10.033 0310
0.5607=10.0115
0.5309 0.1166
0.5526:4:0 0382
0.5133+0.0188
0.506340.000 0265
0.5008 01
0.51446:0.013
EL
0.5929=0.077
0.5617=0.00
0.5093 0290
0.5866
EV
0.61 20-20.06068
0. 0.50880000067
0.5730=0.0550
0.6009-10.07544
HG
0.5159.10. 10.0107
0.510010.0145
0.5028 0. 0.00
0.314110.0199
0.5909110.0372
0.591810.0377
HS
18770=10,022
0.6257:0.0567
0.64844:..0G 09956
JN
0.6076#0.001
0.6111#0.0266
0.5759
0.0644 0654
0.5755:+0 0511
0.8079 1752
0.8006=0.122 1.136
73532400. 1598
0.8101+0.15530
MT
0.8217 1813
0.958110.0228
0. 1473
0.8082+10.788
MA
0.5146+0.0185
0.5585+0.0324
0..1991000001 0.008
0,5112- 0222
MB
0.6715
0.6480110.0710
0.555944.0079
0.6476:10. 1144
0.504711.0009
0.500410.0041
NS
1.000
1.000011.000
1.000000.000
1.000010.0000
PW
0.7666 1429
0. 0.84661100011
0.675840.12111
0.79 8+00 1242
0.5966.10.0441
0.681310.0515
0.6045.10.000
0.683410.065
RS
0.6469-0.0554
0.5810=6,,077
0. 0,050
0.6469::0.054)
TC
1.5343110.0220
0.5118110.035
0.5031 0.0028
0.530110.020
TD
0.8162+0.0100
0.7790=0.0077
0.6355 0884
0.7736 0584
# Best
12/23
8/23
1/23
5/23
RND
15/23
16/23
3/23
15/23
Table 21: Comparison raw balanced accuracy scores of distillation methods applied in the original space (no encoder) on XGB classifier. Last two rows the tables each denote the number of instances that the pipcline ranked the top, and the number of times outporformed random sampling. Best performance for each dataset is marked in bold, and second-bess performance is marked with underline.
39

---

Published in Transactions on Machine Learning Research (04/2025)
Dataset
AG
GM
KIP
KM
AD
0.790420.0171
0.7609:0.0111
0.66:25:000662
0.7940=0.0007
AE
0.537110.0022
0.3246:0.0224
0.512910.0130
0.5365.10.0192
BM
0.7898=0.0052
0.7546+0 0317
0.699
0.7897 0083
CR
0.5437=0.0127
0.5337=+0.0200
0.5500=0.0170
0.5219=40.199
CD
0.6490=0.0300
0 66499000047
0.5819=0.0188
0.6674#0.0112
0.5607=00000
0.5054::0.0474
0.5408:
0.5565410.0064
EL
0.61633:0.0173
0.3758:10.0225
0,004110.0241
0.627610.0126
EV
0.7152 0.0017
0.662111.0333
1. 0.0448
0.7130110.0193
HG
0.
0.3239+0.0122
0.5205=0.0006
0.5792:00 0130
HE
0.6870=0.0061
1.65880.0017
0.6325=0.0600
0.6786 0103
HS
0.7159=0.0119
0. 0.7211=0.002 0279
0. 0.6375=0.0831
0.7721 0122
JN
O. 0.7383=00005
0.6972+0.0011
O. 0.07
0773000000033
LA
0.9979
0.965410.0225
0.0761
0.9935.10.00
MT
0.971710.000
0.9674:10.0044
0.971410.000
0.9715-10.0026
MA
0. 0096
0.5587 0252
1.506310.0188
0.5683:10.00
MB
0.6178 0156
0.6871+0.00 0132
0.6307=0.0191
0.6999+0.0211
NU
0. 0.4991:: 0060
0.4967=0.0058
0.0070+0.0000
NS
0.9944=0.0063
0.9573+0.0066
0.9716=0.0000
0,9941=00 +0000
PW
0.806410.0158
0.86201.0.010
0.6696 0.0083
0.9016
0.9016110.018 0.0158
PL
0.7829 0206
0.7505=0.0000
0.6717=0.0588
0.8277:0.0311
RS
0.715410.0211
0.6357
0.667910.044
0.7208 0.720810.011 10.0134
TC
1.5530=00.055
0.3261 0222
0.5173=0.0100
0.5609:10.0138
TD
0.9230=000012
0.9117::0.0167
0.820410.0502
0.9242:1000052
# Best
11/23
0/23
1/23
11/23
22/23
21/23
15/23
22/23
Table 22: Comparison of raw balanced accuracy scores of distillation methods with TF-SFT and KNN downstream classifier. Best performance at for each dataset is marked in bold, and second-best performance is marked with underline.
Dataset
AG
KIP
KM
AD
0.7352=0.0212
0.72924 0136
0.560020.0599
0.7216:+0.330
AE
0.5309=0.0252
0.5204:00 0109
0.5131=0.0233
0.5143::0.0162
BM
0.7111 0.635244..0m 0335
0.657420.0504
0.7210=t.0027
CR
0.5364110.0178
0.5393-10.0256
0.5161 0.0020
0.5008.10.0180
CD
0.6082-0.0252
0. 0.6005=:0.00 0292
0.552510.0075
0.6005:10.0312
DB
0.515410.0163
0.513910.00
0.504910.0211
0.5288.10.0178
EL
0.6300=0.0314
0.5630-00.0 03:16
0.527320.0163
0.6210=20.0331
EV
0. 0.68-00-0.0477
0.6380:10 0361
0.5907
0. 0.6949:10.0311
HG
0.5397=0.018
0.5121 01121
0.5118#0.000 0001
0.5281++00012
HE
0. 0.6442-0.0255
0.58371100334
0.0494
0.6546:10.018.
HS
0.695-40.00400
0.6340-20. 0532
0.6274=0.0498
0.7115=20.0277
JN
0.6555 00.0
0.6320110.0185
0.5891 10.0515
0.6597 10.0211
LA
0.8267
0.7451:10.00
0.823310.0688
0.8039:10.6672
MT
0.8070 180701100090
0.7332100.081
0.7098 7709810.000 0900
0.8236 1223610.000 0.0855
MA
0.756=0.0098
0.363240 0237
0.5111
0.5676:10.0188
0.67 0.6712
0.612240.00
0.556520.0573
0.6731#0.062+
NU
0.5065 0.0033
0.5019 0050
0.5005=0.0051
0,5035+00 0049
NS
0.9278
0.80441000122
0.9776=0.0140
0.88761:0.08
PW
0.8700 0.0175
0.8128110.011
0.6291 0.0998
0.8678.10.00
PL
0.6327 16.0750.0557
0.5675110.022
0.563411.Cm
RS
0.6350=0,00 0,0324
0.3440=20.0210
0.5213=0.0200
0.6261 1-06611-0000 0304
TC
0.5 0.5129 0285
0.515240 0240
0. 0.1953=0.0150
0.5205++00099
TD
0.7632+0.0388
0.125500.029
0.6139#0.0181
0.7814#0.0377
# Best
10/23
0/23
1/23
12/23
22/23
16/23
3/23
22/23
Table 23: Comparison of raw balanced accuracy scores of distillation methods in the original space (no encoder) KNN downstream classifier. Best performance at for each dataset marked in bold, and second-best performance marked with underline.
40

---

Published in Transactions on Machine Learning Research (04/2025)
Dataset
AG
GM
KIP
KM
AD
0762740.0088
0.740610.0168
0.731810.0212
0.7628:0.0210
AE
0.546710.
0.332411.0008 0090
0.318910.0063
0.5630:1.0022
BM
0.7891=0.0311
0.7655+0 0210
0.7632=0.0312
0.7887=0.0112
CR
0.5299=0.0242
0.5443=+000228
0.5525=0.0188
0.5349=0.01344
CD
0.6323=0. 0845
0. 1.6588#00041
0.6138=0.044 0261
0.65-12:00.0144
0.53644:0.00
0.52111:.0222
0.534840.0038
0.5405
EL
0.6432100.831
0.3690:10.053 0.326
0.6131
0.6543:0.018
EV
0.7310 0.731010.0019
0.6792.10.043)
0.67421 030447
0.7202.10.033
HG
0.6058=0.0122
0.3302+000000
0.477=0002554
0.5997+000152
HE
O. 0.6540=0.0057
0. 0.6364+0.0222 0225
0.0256=0.0870
0.6580=0.0157
HS
0.7801=0.000
0. 0.7257=00.m 0336
0.7178=0.011
0.7708+0
JN
0.7192=0.0033
0.691 169.011000100
O. 0.0952
0.7163+0.0102
LA
0.969310.0192
0.9980110.0017
MT
0.9698-0.0058
0.9627:10.003
0.969710.000
0.9733110.0033
MA
0.569420.0127
0.3571:10.026
0.516010.0100
0.5878:0.011
MB
0.6818=0.0092
0. 0.6555 0568
0.6697=0.021 0217
0.6707=+0.066
NU
U.495882.
0. 0037
0.4987=0.000G
0.5071+0.0045
NS
0.9749=0.01
0.9731
0.9838#0.0138
0.9842=0.0122
PW
0.880410.0010
0.8466.1 0.8383
0.7921 1 0.0523
0.9046 1.0018
PL
0.9010=0.0196
0.850240.0177
0.8198 0426
0.9000+ 0059
RS
O. 0.684210.0013
0.62101100058
1.662710.077
0.687710.0200
TC
0.5785=0.0231
0.3150 0307
0. 0.5366 0.0251
0.5731=#000197
TD
0.919110.0104
0.9010::00260
0.899
0.9200:1.00337
# Best
9/23
0/23
1/23
13/23
21/23
19/23
18/23
23/23
Table 24: Comparison of raw balanced accuracy scores of distillation methods with TF-SFT and MLP downstream classifier. Best performance at for each dataset is marked in bold, and second-best performance is marked with underline.
Dataset
AG
KIP
KM
AD
0.7183:0.0392
0.757620.0148
0.6756=0.0601
0.7385: 0276
AE
0.5743=0.022
0.5444+0.0158
0.5267=0.0241
0.561840 0410
BM
0.7406:0.0222
0.6573+0.0311
0.6776=0.0569
0.7351+0.0311
CR
0.5607-10.0216
0.5088110.022
0.5037
0.5618:10.027
CD
0.6146=00.076
0.592010.000 0554
0.6706
0.6040:100.00
0.5168110.0111
0.5203100022
0.5052
EL
0.6713=0.0315
050663
0.5568=0.0787
0.6573+ 03447
EV
0.6828:1000770
0.6570110.022
0.6380=0.0661
0.6900110..m 0.0255
HG
0.5:66=00.221
0.518140.00 0.0200
0.5190=0.0100
0.3-123:+0 0221
HE
0.6221
0.6213110.039
0.6369=0.0574
0.630911.022
HS
0.7514=0.0155
0.6746+0.0324
0.6970=0.080
0.7397 0396
JN
0.6352 1 0.0188
0.6328 10.0214
0.6209 00.070
0.6339 0.339.100000 0.0137
LA
0.7621+400019
0.892410.1000
0.7970.10
MT
1.0032
0.8068 1.008810.077 0..741
0.8904 0.0069
0,8839 100 05844
MA
0.5710=0.0124
0.559140.011 0198
0.5294=0.0388
0.56:10::: 0151
0.7411=0.0577
0.676840.. 0636
0.5993=0.0945
0 0656
NU
0.5076=0.002
0.5009+0.000 0057
0.500100.0028
0.3063++ 0039
NS
0.9799-10.0208
0.8150410.0002
0.9967 10.0038
0.9006:10.07
PW
0.901810.018
0.5248 0.02
0.808400.0544
0.8775 0. 0322
PL.
0..93410.0946
0.7319
RS
0.6301=0.0134
0.5567:10.0191
0.5386=0.0583
0.6305=100027
TC
0.5-104-0.028
0.517740 0141
0.5016=0.088
0.51541:0 0328
TD
0.7924=0.015A
0.716140.0507
0.6930=0.0113
0.7751+00
# Best
14/23
1/23
2/23
6/23
22/23
19/23
6/23
21/23
Table 25: Comparison of raw balanced accuracy scores of distillation methods with in the original space (no encoder) MLP downstream classifier. Best performance at for each dataset marked in bold, and second-best performance marked with underline.
41


Reviews:
[{"review_id": "w6py5HHCGR", "reviewer": "Reviewer_Wk4J", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n* Large-ish benchmark with 23 tabular datasets, 7 model classes and 11 distillation schemes.\n* Easy method (combines well-known building blocks) that tackles a novel problem (dataset distillation) in the general setting (any downstream classifier).\n\n## Weaknesses\n\n* The distillation on learned representations is not compared against distillation on raw data, or minimally processed data (using one-hot-encoding).\n* The paper could be easier to read: better structure and less references to the appendix. Please see my suggestions below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper \"On Learning Representations for Tabular Data Distillation\" presents a benchmark of methods for distilling tabular data into smaller datasets using learned representations. Representations are learned using three different kinds of neural networks, and the distillation happens on the learned representation. Downstream tasks are executed on the raw or the learned representation. The method shows that applying the proposed methods can actually yield benefits.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Major\n* Distill on raw data as a baseline. k-means can be applied on raw data, especially to datasets not containing numerical features. Otherwise, techniques such as one-hot-encoding and scaling of numerical features can be used in order to obtain a useful representation of the data for distillation.\n* The datasets section does not discuss the handling of missing values. Also, that section should contain a link to the respective section in the Appendix.\n* Discuss if the model is retrained on the full dataset after conducting HPO on the reduced dataset. Also, discuss if the distillation time is included in the HPO plots.\n* The paper should cite the recent, gradient-based dataset distillation method developed for the TabPFN classifier: https://openreview.net/forum?id=FOfU3qhcIG In an ideal world, the authors find a way to compare against the idea of TuneTables. Can the distilled data maybe also be used as an input to TabPFN, maybe even outperforming the TuneTables idea?\n* Please describe how you solve the optimization problems 1 and 2, this is not clear from the main paper. Especially, what is the decoder? Is it another neural network?\n* Writing/missing information in the main paper:\n    * Section 4 on Page 7 mentions an autoencoder, but there was no autoencoder mentioned in the sections before. This needs to be explained better.\n    * The caption of Figure 4 mentions an \"SFT\" that is not introduced in the paper before. This needs to be fixed.\n    * Subsections: please add subsections in Section 4 to better structure the paper, for example, into main experiments, hyperparameter optimization experiments and ablation studies.\n    * Page 9 mentions \"We find that adding supervised fine-tuning improves the downstream performance of all encoders in general.\", but it is not clear how this finetuning works. Towards which classifier is the representation fine-tuned?\n* Figerus 5 and 7 do not appear to be critical difference plots following the method if Demsar (JMLR 2006) is they do not show the critical difference. The paper needs to state what tests are applied in order to obtain the critical difference plots.\n\n## Minor\n\n* Figure 3: please change the legends of the plots, to the best of my understanding, the statement \"IPC=10\" is wrong as the IPC is given on the x-axis.\n* Describe exactly what numerical embedding is used - is it one of the embeddings proposed in Gorishniy et al. (2021) or in Gorishniy et al. (2022), and which one exactly?\n* Is Remark 1 necessary? Isn't this all stated in the introduction?\n* Why does the paper invent a new method similar to k-Medoids, namely choosing the medoid of the final k-Means result? Wouldn't using k-Medoids in the first place be easier?\n* It is not clear how the imbalance of the given datasets is handled in the method and the benchmarks. Is the dataset's imbalance retained in the distilled data or not?\n* The paper should explain what it means to have a relative regret below zero in Figure 3. Does this mean that using the distilled data, the final performance is improved over using the full data? If yes, why would that be the case?\n* Writing, Page 10: \"method.We\" -> \"method. We\", \"critifcal different\" -> \"critical difference\".\n* The references are not formatted well and consistent:\n    * Borisov et al., IEEE 2024: wrong year and last name of 3rd author is written wrong.\n    * Clements et al. misses the publication year and does not follow the arXiv citation guidelines.\n    * The references give URLs for a random subset of the publications. Please be consistent.\n* Questions about the Appendix: \n    * on what part of the data is alpha tuned?\n    * Figure 10: why are the multiple feature embeddings for a single feature?\n    * The findings in A7 correlate with the findings of several recent studies, this could be explicitly mentioned.", "broader_impact_concerns": "-", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DpkWP4nIUz", "reviewer": "Reviewer_vGRR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n1. The proposed framework is generic and can be instantiated with many of the existing methods for dataset distillation. This importantly includes future distillation techniques from the image and tabular distillation literature.\n2. The authors have created an important framework, dataset, and metric for the evaluation of future tabular dataset distillation techniques\n3. The paper includes a thorough Experiments section with various important experiments included\n\n**Weaknesses:**\n1. My main concern with the current version of the paper is with the writing of the experimental section. In particular, there are some missing details in the presentation of the results that I find important, and there are also a few conclusions of the experiments I want discussed and explained better.  See **Requested Changes** and **Questions** for more info.\n\n**Overall Grade:**  \nOverall, I find the work important and impactful. However, I will not recommend the paper in its current form due to the writing concerns above. I am willing to change my opinion if the authors make a reasonable effort to address my comments. One general comment I have for the overall readability of the paper is that keeping track of which encoding, which decoding, which autoencoder, and which distillation schemes are used across the experiments is tedious and very confusing at times. \n\n**Questions:**\n1. I would love for the authors to clarify on a more technical level, both in the paper and in response to me, what they mean by the tabular input features being \"heterogenous,\", especially in the context of what challenges this brings for distillation compared to images. Is it the ranges or the variables that cause the issue, the mix of discrete and continuous features, the different semantic content, the different correlations to the output, or maybe some combination of those? \n2. Did the authors forget to draw the original GM in Figure 3? Or is it that it is always under RND?\n3. \"Specifically, we consider a case where the validation and testing data is sampled from the original data\" - Why is this the case? I would have thought we wanted to evaluate the performance of the classifiers on a dedicated eval/test set where the hyperparameters haven't been optimized.\n4. Do the authors plan to make TDBench publically available after the publication of the paper? Given the importance of this contribution, I want the authors' explicit confirmation of their commitment to do this.\n5. The framework proposed is some variation of an autoencoder. To this end, there are a few big questions remaining:\n- Does adding noise to learned representation, like in VAEs, robustify and improve the embeddings produced?\n- Are other SotA methods for producing tabular data embeddings applicable in the proposed framework of Algorithm 2?\n6. In Figure 4, the authors compare the architecture based on a number of parameters. However, the number of parameters might vary based on hyperparameters. Further, the speed of inference varies across architectures. Can the authors, thus explain in more detail why  Figure 4 is relevant?\n7. The methods propose to embed the individual feature embeddings from (r+c)*m to m dimensional space. How important is it that we encode the columns into a single-row embedding? How important is it for this embedding to be of size exactly $m$? What if the final embedding size is > or < $m$?\n8. The authors claim that releasing $\\bar{R}$ instead of $R$ for the distilled dataset better preserves the privacy of the data in it. However, $\\bar{R}$ is a function of $\\bar{R}$. Thus, the private information in $\\bar{R}$ is more than in $R$ from a theoretic information perspective. Can the authors comment on this?\n\n\n**Nits:**\n1. \"We focus on GM and KIP due to the high computational......\" - At this point, you haven't introduced the GM acronym as far as I can say.\n2. Please do not split the footnote on page 5 across the pages. It can be confusing to the reader.\n3. Figure 3 - I think the description of the figure is outdated with regard to what rows and columns represent.\n4. \"To evaluate, we perform a Wilcoxon signed-rank test to identify groups that stand out from the rest, as shown in fig. 5.\" -> fig.5 is a different style than the other references. There are, in general, inconsistencies in how you use \\cref beyond this example as well.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a generic framework for tabular data distillation that can be instantiated with several different components. In particular, the framework first embeds each feature in $m$ dimensional space and then uses an encoder-decoder framework to embed the individual features into a single embedding, which is then refined using an SFT scheme involving labels. Finally, the obtained embeddings are used to distill the dataset. \nThe authors provide a large-scale ablation of the different components and a thorough comparison of their distillation framework vs. the base distillation techniques with which the framework is instantiated. The authors also provide interesting experiments to inform about the practical implications of their framework to a realistic application of table data distillation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Important:**\n1. Nowhere in Section 4 or Figure 3's description you explain what RND is. I ended up realizing this was the performance of training on the random 10 samples of the respective downstream classifier, however, it required me to read the rest of the paper to realize this. Please make sure to explain this early in the eval section, and possibly also in Figure 3's description. Further, the authors do not make clear when they define the relative regret and use it in Figure 3 that the relative regret is defined per downstream classifier and the numbers in plots for different downstream classifiers are not comparable.\n2. I think the authors need to comment on the instances in Figure 3 and Table 2 where relative regret is negative. I think in the introduction of relative regret the authors do not make it obvious the metric can go negative, and further, I think such instances need to be specially discussed and the reasons for those explained.\n3. The authors need to make clear if for the original (non-latent space) methods and the RND methods in Figure 3 they use the feature encoder $C$ or not. \n4. For the \"Rec.\" methods in Figure 3, it is not clear to me if, during test set inference, the authors use the exact features of the test points, or if they run them through both the encoder and decoder. I also would like to see an appendix experiment to compare those two methodologies.\n5. The authors should explain what exactly the mean rank is in Table 1. That is, I would love a formula + the an explanation of the set of models amongst which the ranking was computed.\n6. What do the vertical lines in Figure 4 represent? Is the case that the number of transformer parameters varies, and why? This needs to be explained in the accompanying text.\n7. For Figure 5, are the distillations in latent space or in the original space? If they are not in the latent space, can you produce the same graph in the latent space? If they are in latent space, is it Enc. or Rec.?\n8. I think a single experiment on a multi-class dataset is important to convince the reader that the proposed methods generalize beyond binary classification.\n9. In Appendix A.4.2, the authors should explain how the classifier $f$ is optimized in the context of Equation 10. That is, is $f$ optimized or static? If it is static, how was it produced? If it is dynamic, does the training of the encoder and decoder unroll the training of $f$ or is $f$ trained alternatively alongside the encoder and decoder?\n10. In FIgure 4, the number of parameters are primarily used to measure the complexity of the encoders, however in practice the more important property of the encoders is the speed of encoding. Can the authors provide numbers on the speed of encoding?\n11. The authors should explain in the paper and to me how in Appendix A.4.1 are the targets of real features handled? Are they normalized to [0,1]? Or are they instead binarized?\n12. The authors should explain, in Figure 6, what does the runtime of HPO include? Is it only training on the final optimized parameters or the runtime of the full HPO procedure?\n\n**Nice to have:**\n1. Figure 3 - I would love for the authors to have a uniform scale, if possible, for the different subplots. In particular, I would love it if you marked the 1.0, 0.5, and 0.0 relative regrets for all subplots and not only some of them.\n2. For Table 3, it will be good to give a number of total rankings so that it is known in what percentage of the rankings each model got ranked first. Alternatively, you can report the ratio of such cases.\n3. I think Figure 2 needs to be referred to more in Section 2. I think it can make understanding Section 2 easier\n4. I think Table 14, as well as, one of Figure 15 or 16 can go in the main paper. I think those results are interesting and important.\n5. Can the authors provide a comparison between GNN KM^*, TF KM^*, FNN KM^* of the relative regret for different values of the internal representation size $m$? Can they also provide inference/training speed comparison? \n6. It would be nice to have an extension to Figure 7 that shows the results on an NN downstream classifier instead of the tree-based one to corroborate the claims that the problems with SotA methods are caused by the mismatch of downstream and distillation architectures.", "broader_impact_concerns": "I do not have ethical concerns. I think the work can be very impactful if the authors release their code and they address my concerns above.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2HrIajaduf", "reviewer": "Reviewer_sjVR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper has the following advantages:\n- This paper proposes a novel framework for distillation of tabular datasets.\n- This paper builds a comprehensive benchmark for evaluating different tabular datasets, distillation methods, and network architectures.\n- This paper provides detailed experimental results and reproduction scripts.\n\nHowever, this paper still has the following weaknesses/questions:\n- The dataset distillation baseline is not novel enough. The author's baseline is mainly k-means (2011), Agglomerative (2011), GM (2021), which cannot represent the most SOTA dataset distillation schemes. Therefore, the performance of tabular data distillation under the most advanced dataset distillation methods remains to be verified. In addition, advanced Coreset may also need to be considered for verification, because they are usually more efficient than dataset distillation.\n- There is a lack of time/resource consumption comparison of different dataset distillation methods in the distillation phase. As we understand, trajectory-based distillation methods are time-consuming/GPU-intensive in image dataset distillation. Therefore, a clear explanation of the efficiency of different distillation methods will help readers choose the most appropriate table distillation method.\n- In related work, it is necessary to discuss some dataset distillation methods in the field of natural language processing, because text, like tabular data, is discrete, while images are continuous.\n- There is a lack of visualization or examples of distilled data. For example, in images, we can clearly visualize the distilled image. Can the paper provide some distillation results of tabular data? This will help to observe that the synthesized tabular data can better reflect the privacy protection effect than the original tabular data.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new tabular data distillation framework and establishes a complete benchmark for tabular dataset distillation, which will be beneficial to the dataset distillation community.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As mentioned in Weakness, in addition, some minor details are as follows:\n- The caption in Figure 2(b) is unclear, and it is unclear what KM, KM Enc., and RND stand for.\n- How is the \"0.5% to as large as 143%\" claimed in Figure 2(b) calculated, because the figure only shows 24.75%, 105.80%, 62.64%, and 68.72%, which is confusing.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: There was consensus among the reviewers that the submission should be accepted, with two weakly leaning accept and one more strongly in the camp of acceptance. This paper represents a solid contribution to TMLR.

I don't believe any of the certifications are appropriate, though -- one reviewer suggested the survey certification, but I'm not sure that the writing is appropriate for a survey. No other certifications were suggested.
Decision: Accept (poster)