Targeted synthetic data generation for tabular data via hardness characterization

## Abstract

Abstract Synthetic data generation has been proven successful in improving model performance and robustness in the context of scarce or low-quality data.
Using the data valuation framework to statistically identify beneficial and detrimental observations, we introduce a novel augmentation pipeline that generates only high-value training points based on hardness characterization.
We first demonstrate via benchmarks on real data that Shapley-based data valuation methods perform comparably with learning-based methods in hardness characterisation tasks, while offering significant theoretical and computational advantages. Then, we show that synthetic data generators trained on the hardest points outperform non-targeted data augmentation on simulated data and on a large scale credit default prediction task. In particular, our approach improves the quality of out-of-sample predictions and it is computationally more efficient compared to non-targeted methods.

## 1 Introduction

Training complex machine learning models requires large amounts of data, but in real-world applications data may be of poor quality, insufficient in amount, or subject to privacy, safety, and regulatory limitations. Such challenges have sparked an interest in synthetic data generation (SDG), representing the practice of using available data to generate realistic synthetic samples (Lu et al., ).
In this work, we argue that, when the objective is to use synthetic data to make an existing machine learning model better generalize to unseen data, augmenting only the hardest training points is more effective than augmenting the entire training dataset. In other words, we propose a scalable targeted synthetic data generation framework, focusing on binary classification tasks. The underlying intuition is that, within a dataset, some of the observations are obvious to classify, whereas others play a more significant role in determining the decision boundary of the trained model. Focusing only on these harder examples when generating additional data may allow the model to efficiently learn a more robust decision boundary.

In particular, the method proposed in this work is specifically targeted towards binary classifiers on tabular data, which often require adaptations such as oversampling, undersampling, or synthetic data generation to improve model performance (Chawla et al., ; He et al., ) under class imbalance. While deep neural networks have proven successful across domains such as images, audio or text, they are still regularly outperformed on tabular data by simpler and more interpretable tree-based architectures (Shwartz-Ziv & Armon, ). Therefore, we choose trees as the reference model for binary classification on tabular data, while deep learning models are used for synthetic data generation. In general, we will use $D=\{(\bm{x}_{i},y_{i})\}_{i=1}^{n}$ to denote the training data, where $\bm{x}_{i}\in\mathbb{R}^{d}$ is a $d$-dimensional feature vector and $y_{i}\in\{0,1\}$ is a binary label.

Our targeted synthetic data generation approach is divided into two main steps: (i) hardness characterization of training data points, followed by (ii) training of a synthetic data generator only on the hardest data points of the initial training dataset. In order to identify “hard” data points, we propose a novel hardness characterizer specific to tabular data, based on KNN data Shapleys (Jia et al., ), capable of achieving performances comparable to state-of-the-art methods in hardness detection benchmarks, while offering key advantages such as being deterministic, model agnostic and computationally efficient. Hardness characterization has already been applied in the literature with rationales similar to this work, but mainly with the objective of pruning the easier examples (Jia et al., ; Paul et al., ), rather than augmenting the harder ones.

Once the hardest points have been identified, synthetic data generation models are trained only on the most difficult points identified by our hardness characterizer. In this work, we focus on two widely used deep learning architectures for synthetic data generation for tabular data, namely Tabular Variational Autoencoders (TVAE) and Conditional Tabular Generative Adversarial Networks (CTGAN) (both introduced in Xu et al., ). To mitigate instabilities encountered during training, we devise a custom early stopping condition which allows to interrupt training when the quality of the synthetically generated samples is maximised. The best synthesizers are then found using a highly efficient routine which combines model selection and hyperparameter tuning. From a comparison of the best models we find that, given a fixed number of points to generate, augmenting only hard data results in a larger performance improvement on unseen data with respect to non-targeted data augmentation, while being computationally cheaper.

The rest of the work is structured as follows: after a recap on the related literature in Section 2, the proposed approach is discussed and empirically verified in Section 3.
Next we explore the use of KNN Shapleys as hardness characterizers via a benchmark study in Section 4.1, followed by a discussion on how to utilize them for synthetic data generation on real (Section 4.2) and simulated data (Section 4.3). The results are extensively discussed in Section 5.

## 2 Background and related literature

In this work, we aim to bridge the gap between existing methods for hardness characterization and game theoretic approaches to data valuation, with the objective of improving the performance of synthetic data generators. Therefore, our work closely aligns with existing techniques in hardness characterization, data valuation, and synthetic data generation, briefly reviewed in this section.

### 2.1 Hardness characterization

As remarked in the introduction, the first aim of this work is the identification of a subset of points that are hard for the model to learn. Existing literature on the topic of hardness characterization is often qualitative and based on different definitions of “hardness”. Following the taxonomy presented in Seedat et al. (), three main categories of hardness can be identified: (i) Mislabeling: the true label is replaced with an incorrect one. The label perturbation probability can be uniform across classes or label-dependent;
(ii) Out-of-distribution: features are transformed or shifted, leading to observations distributed differently from the main data generating process;
(iii) Atypical: observations that are compatible with the main data generating process, but located in tails or rare regions of the feature distributions.
These distinctions normally dictate different courses of actions: datapoints that are mislabeled or out of distribution are symptomatic of errors in the data collection process and should be removed from training data, while atypical datapoints are valid points and may in fact need to be augmented to train a robust model (Yuksekgonul et al., ).

While many different hardness characterizers have been proposed, most of them are based around keeping track of some statistic of interest for each datapoint during the training process, meaning they can be adapted to any model that is trained iteratively (such as XGBoost or a neural network). Examples include GraNd (Paul et al., ), which uses large gradients as an indicator of hardness, and Forget (Toneva et al., ), which counts how many times each point is learnt and then forgotten during training, with the simple intuition that hard points are forgotten more often. Based on the quantitative benchmarks reported in Seedat et al. (), Data-IQ (Seedat et al., ) emerges as one of the best performers on tabular data, and it is chosen as benchmark hardness characterizer in this work because of its interpretability and compatibility with XGBoost.

##### Data-IQ

Data-IQ (Seedat et al., ) partitions the training data into “easy” and “hard” points via the concepts of epistemic and aleatoric uncertainty, combined with a model confidence score calculated from $\mathcal{P}(\bm{x},\bm{\theta})$, representing the probability of assigning a correct label to an observation, given input features $\bm{x}$ and model parameters $\bm{\theta}$. The epistemic uncertainty is caused by the variability in the estimated model parameters, whereas the aleatoric uncertainty captures the inherent data uncertainty. Given parameter estimates $\{\bm{\theta}_{e}\}_{e=1}^{E}$ obtained during training, a Data-IQ confidence score is obtained as $\bar{\mathcal{P}}(\bm{x})=E^{-1}\sum_{e=1}^{E}\mathcal{P}(\bm{x},\bm{\theta}_{
e})$, whereas the aleatoric uncertainty is given by $v_{\mathrm{al}}(\bm{x})=E^{-1}\sum_{e=1}^{E}\mathcal{P}(\bm{x},\bm{\theta}_{e}
)[1-\mathcal{P}(\bm{x},\bm{\theta}_{e})]$. Training data can then be partitioned into (i) easy points,
corresponding to high $\bar{\mathcal{P}}(\bm{x})$ and low $v_{\mathrm{al}}(\bm{x})$, (ii) hard points,
corresponding to low $\bar{\mathcal{P}}(\bm{x})$ and low $v_{\mathrm{al}}(\bm{x})$, and (iii) ambiguous
points, with high $v_{\mathrm{al}}(\bm{x})$.

### 2.2 Data valuation

Data valuation (see Ghorbani & Zou, ) is the task of equitably quantifying the contribution of each data point to a model, via a data value $\phi_{i}(D,\mathcal{A},V)$, where $D$ is the training dataset, $\mathcal{A}$ is a model and $V$ is a performance score, where $V(S)$ is the performance score of $\mathcal{A}$ when trained on $S\subseteq D$.
Ghorbani & Zou () identifies that a suitable data valuator is the data Shapley of the $i$-th training point, which takes the following form:

$$ $\phi_{i}=C\sum_{S\subseteq D\setminus\{i\}}{\binom{n-1}{|S|}}^{-1}\left[V(S \cup\{i\})-V(S)\right],$ (1) $$

where $C$ is an arbitrary constant. The valuator $\phi_{i}$ can be interpreted as weighted sum of all the possible “marginal contributions” of the datapoint. Exact evaluation of the data Shapleys is prohibitively expensive because of the need to retrain the model $2^{n-1}$ times (once per subset) for each observation in the training data. It is possible to approximate the data Shapleys by considering only a small number of subsets $S\subseteq D\setminus\{i\}$ in equation 1, obtained via Monte-Carlo sampling of random permutations. Unfortunately, even state-of-the-art approximations based on this idea, such as TMC-Shapley (Ghorbani & Zou, ) or Beta-Shapley (Kwon & Zou, ) are computationally prohibitive for datasets with $n\gg 1000$, making them difficult to use in practice. More generally, extensive work has been developed in the literature to make the computation of data Shapleys more efficient, such as Jia et al. (), Wang et al. () and Wang et al. ().

In particular, Jia et al. () derived a recursive algorithm to compute the exact data Shapleys for KNN classifiers in $\mathcal{O}\{n\log(n)\,n_{\mathrm{test}}\}$ complexity.
For training data $\{(\bm{x}_{i},y_{i})\}_{i=1}^{n}$ and test data $\{(\bm{x}_{\mathrm{test},i},y_{\mathrm{test},i})\}_{i=1}^{n_{\mathrm{test}}}$, let
$(\alpha_{j,1},\ldots,\alpha_{j,n})$ be the indices of training data points, sorted in increasing order according to their Euclidean distance from the $j$-th test data point, for $j=1,\dots,n_{\mathrm{test}}$. The KNN Shapley for the $i$-th training data point is then obtained as the average $s_{i}=n_{\mathrm{test}}^{-1}\sum_{j=1}^{n_{\mathrm{test}}}s_{j,i}$, where $s_{j,i}$ is calculated recursively as follows:

$$ $\displaystyle s_{j,\alpha_{j,n}}=\frac{1}{n}\mathds{1}_{y_{\mathrm{test},j}}\{ y_{\alpha_{j,n}}\},$ $\displaystyle s_{j,\alpha_{j,i}}=s_{j,\alpha_{j,i+1}}+\frac{\mathds{1}_{y_{ \mathrm{test},j}}\{y_{\alpha_{j,i}}\}-\mathds{1}_{y_{\mathrm{test},j}}\{y_{ \alpha_{j,i+1}}\}}{iK\cdot\min\{K,i\}^{-1}},$ (2) $$

where $j=1,\dots,n_{\mathrm{test}}$, $i=1,\dots,n-1$, and $\mathds{1}_{\cdot}\{\cdot\}$ is the indicator function.
Jia et al. () found KNN Shapleys to be a valid alternative to data Shapleys in tasks such as data summarization or noisy labels detection, only when based on deep features extracted from image data by pre-trained architectures. A similar approach is unfortunately not possible on tabular data.
In this work we argue that KNN Shapleys computed using a test dataset large enough to cover the entire data distribution, can be used for hardness characterization on tabular data. Specifically, we argue that the lowest KNN Shapleys identify the hardest training points.

### 2.3 Synthetic data generation

Synthetic data generators (SGDs) for tabular data have been actively researched in recent years (see for example Fonseca & Bacao, ). Here, we discuss three of the most popular SDGs used in practice: SMOTE, CTGAN, and TVAE.

##### SMOTE

In Synthetic Minority Over-sampling Technique (SMOTE; Chawla et al., ), observations are picked at random among the $K$ nearest neighbours of the same class for each datapoint, and new data is generated by random sampling along the segment connecting the point to the chosen neighbour. The number of neighbours is determined based on the over-sampling ratio.

##### CTGAN

Tabular GAN (TGAN; Xu & Veeramachaneni, ) was the first attempt at using Generative Adversarial Networks (GAN; Goodfellow et al., ) to generate tabular data, which was then extended to Conditional TGANs (CTGANs; Xu et al., ). A GAN consists of two models: a generator that takes as input random noise and tries to output a point following the data distribution, and a discriminator that given observations (either from the original dataset or synthesized by the generator) outputs the probability of them being fake. Both the generator and discriminator are typically neural networks. In CTGANs, the possible non-Gaussianity and multimodality in the data is handled by fitting a variational Gaussian mixture model to each normalized numeric feature, whereas categorical features are represented as one-hot encoded vectors. Both the generator and the discriminator networks within the underlying GAN are multilayer perceptrons (MLPs). The name of the CTGAN architecture derives from the form of its conditional generator, which allows to generate data conditional on specific values of the discrete features, for better handling imbalanced datasets.

##### TVAE

A Tabular VAE (TVAE; Xu et al., ) is an adaptation for tabular data of a Variational Auto-Encoder (VAE; Kingma & Welling, ), using similar preprocessing to CTGAN.
A VAE consists in an encoder that learns to represent high-dimensional data into a low-dimensional latent space, and a decoder that reconstructs the compressed representation into the original domain.
Both the encoder and decoder networks are normally chosen to be MLPs (Kingma & Welling, ).

## 3 KNN Shapleys as hardness characterizers for SDGs

In this work, we propose to utilize KNN Shapleys as hardness characterizers, and subsequently generate synthetic data by only augmenting the hardest points. Our proposed pipeline is the following:

- 1.
KNN Shapley calculation – For a model $\mathcal{A}$ fitted on training data $D=\{(\bm{x}_{i},y_{i})\}_{i=1}^{n}$, calculate the KNN Shapleys $s_{i},\ i=1,\dots,n$, based on a test set $\{(\bm{x}_{\mathrm{test},i},y_{\mathrm{test},i})\}_{i=1}^{n_{\mathrm{test}}}$.
- 2.
Ranking by hardness – Sort the KNN Shapleys in increasing order, with points with lower values representing harder examples.
- 3.
Targeted augmentation – Given a synthetic data generator $\mathcal{G}$, perform data augmentation only on the hardest points, based on the ranking of the KNN Shapleys.

By generating synthetic data for difficult examples only, we aim to improve the performance of the model $\mathcal{A}$ specifically on the most challenging parts of the data distribution. The proposed procedure is summarized visually in Figure 1. In the next section, we provide a mathematical intuition on the interpretation of KNN Shapley as hardness charaterizers.

Figure: Figure 1: Visual representation of the proposed targeted synthetic data generation pipeline.
Refer to caption: x1.png

### 3.1 KNN Shapleys for hardness characterization: a toy example

Consider a mixture of two univariate normals with unit variance, centred at $-1$ and $1$, such that $p(x\mid y=0)=\mathcal{N}(x;-1,1)$ and $p(x\mid y=1)=\mathcal{N}(x;1,1)$. Datapoints are drawn from each distribution with probability $1/2$.
Additionally, consider two points $(x_{1},y_{1})=(-1,0)$ and $(x_{2},y_{2})=(1,1)$, representative of each component of the mixture distribution. Consider a third point $(x_{\mathrm{train}},y_{\mathrm{train}})$, with $y_{\mathrm{train}}=0$ without loss of generality. Hence, the training set takes the form $D=\{(-1,0),(1,1),(x_{\mathrm{train}},0)\}$. As $x_{\mathrm{train}}$ increases, it becomes harder to classify correctly under the true underlying data distribution, since the ratio $p(x_{\mathrm{train}}\mid y=0)\ /\ p(x_{\mathrm{train}}\mid y=1)$ decreases, consequently increasing the probability of misclassification. Note that any value of $x_{\mathrm{train}}\in\mathbb{R}$, combined with $x_{1}$ and $x_{2}$, partitions the real line into four different regions.

Any test data point $(x_{\mathrm{test}},y_{\mathrm{test}})\in\mathbb{R}\times\{0,1\}$ induces a ranking of the training data based on the distances $|x_{\mathrm{test}}-x|,\ x\in\{x_{1},x_{2},x_{\mathrm{train}}\}$.
For $x_{\mathrm{train}}=0$, the four possible regions with different rankings of the three training points are plotted in Figure 2(b). The $1$NN Shapley value $s_{\mathrm{train}}$ for the point $(x_{\mathrm{train}},y_{\mathrm{train}})=(0,0)$ can be calculated explicitly for all the possible values of $(x_{\mathrm{test}},y_{\mathrm{test}})$. The results are reported in Table 2(b).
Since the data distribution is known, the expected value $\mathbb{E}(s_{\mathrm{train}})$ of the $1$NN Shapley for $(x_{\mathrm{train}},y_{\mathrm{train}})=(0,0)$ can be explicitly calculated:

$$ $\displaystyle\mathbb{E}(s_{\mathrm{train}})$ $\displaystyle=\int_{-\infty}^{+\infty}s_{\mathrm{train}}\,p(x_{\mathrm{test}}) \ \mathrm{d}x_{\mathrm{test}}=\frac{1}{2}\sum_{y\in\{0,1\}}\int_{-\infty}^{+ \infty}s_{\mathrm{train}}\,p(x_{\mathrm{test}}\,|\,y)\ \mathrm{d}x_{\mathrm{ test}}\approx 0.209.$ (3) $$

It must be remarked that the value of $s_{\mathrm{train}}$ depends on $(x_{\mathrm{test}},y_{\mathrm{test}})$, as detailed in Table 2(b) and equation 1.
The same procedure as equation 3 can be repeated for any $x_{\mathrm{train}}\in\mathbb{R}$, with results displayed in Figure 2(c). As expected, the $1$NN Shapley decreases in the direction of increasing hardness, suggesting an association between low KNN Shapleys and hard regions of feature space.

Figure: (a) Four regions for $x_{\mathrm{test}}$ and ranking of $\{x_{1},x_{\mathrm{train}},x_{2}\}$, for $x_{\mathrm{train}}=0$.
Refer to caption: extracted/5878843/images/ranks.png

## 4 Results

In this section, the proposed hardness characterizer based on KNN Shapleys is first compared to the most common methods in the literature via comprehensive benchmarks on tabular data. Then, KNN Shapleys are calculated on a large credit default prediction dataset to identify the hardest points, which are later used to train and compare synthetic data generators. Lastly, the same pipeline is applied to a simulated dataset to verify consistency in the results.

### 4.1 Benchmark study

In Seedat et al. (), the problem of hardness characterization is approached quantitatively by comparing existing methods on how confidently they can identify different kinds of hard datapoints on a variety of OpenML datasets (Vanschoren et al., ). Their toolkit supports two tabular datasets (diabetes and cover) and it allows to perturb a chosen proportion $p$ of datapoints according to one of three hardness types (mislabeling, out-of-distribution, or atypical; cf. Section 2.1). For this study, we use $p\in\{0.05,0.1,0.15,0.2\}$ and the performance metric is the Area Under the Precision Recall Curve (AUPRC) of an MLP. Results are averaged across three independent runs per hardness type and then across the different hardness types. The KNN Shapleys-based characterizer is implemented by adding a custom class to the original GitHub repository ([https://github.com/seedatnabeel/H-CAT](https://github.com/seedatnabeel/H-CAT)). The results are plotted in Figure 3.

On the diabetes dataset, our characterizer outperforms existing methods for all choices of $p$, although based on the critical difference diagram in Figure 3(a), it can be noticed that performance is not significantly different from Data-IQ, the other method to offer native XGBoost support. On the cover dataset (see Figure 3(b)), our characterizer ranks 7th overall, although once again performance is statistically compatible with Data-IQ.
Based on the mathematical intuition discussed in Section 3.1, KNN Shapleys are expected to be lower when a datapoint lies in a region of feature space where its target is atypical (cf. Section 2.1). When $p$ is too large relative to the dataset’s difficulty, this atypicality is compromised, causing a deterioration in the performance of our characterizer.

Figure: (a) Diabetes dataset
Refer to caption: x3.png

Based on these results, low KNN Shapleys prove to be solid indentifiers of hardness. Most hardness characterization techniques in the literature were devised for neural networks, and do not natively support XGBoost. On the other hand, KNN Shapleys have the clear advantage over existing methods of not requiring model training and thus being deterministic and model agnostic. However, they require an external test set and can become expensive to calculate on very large datasets ($n>10^{6}$).

### 4.2 American Express dataset

The main real-world benchmark for the methodologies discussed in this work is the American Express credit default prediction dataset (Addison et al., ), referred to as the Amex dataset. The dataset consists of observations for $n=458\,913$ customers. For each of them, we consider 188 features available at the latest credit card statement date, in addition to a binary label $y_{i}\in\{0,1\}$ which specifies whether or not the customer was able to repay their credit card debt within 120 days of the statement.
The $n=458\,913$ available observations are split into $n_{\mathrm{valid}}=50\,000$ datapoints used to validate model performance on unseen data, $n_{\mathrm{test}}=50\,000$ observations used to calculate KNN Shapleys, and the remaining datapoints are used for training. The splits are performed randomly stratifying on the target, so that the same ratio of defaulters and non-defaulters is maintained across datasets. Performance is measured via the normalized Gini coefficient, which is $2\cdot\mathrm{AUCROC}-1$ with $\mathrm{AUCROC}$ denoting the Area Under the Receiver Operating Characteristic curve.

The Amex dataset has been first released as part of a Kaggle(^1^11See [https://www.kaggle.com/competitions/amex-default-prediction](https://www.kaggle.com/competitions/amex-default-prediction).) competition, with winning submissions typically consisting in ensembles of different models. XGBoost was found to have the best performance on its own and is thus chosen as baseline for this study. In order to benefit from GPU acceleration, training was carried out on a cluster equipped with 4 GeForce RTX 3090 Ti GPUs and 256GB of memory. The number of gradient boosting rounds was set to an arbitrarily high number, using early stopping on validation Gini with a patience of 100 iterations of gradient boosting. Because of subsampling the datapoints and the features, the training process is non-deterministic and thus retraining a large number of times with different random seeds is required to mitigate randomness. This forces to choose a set of hyperparameters that offers low training times as well as solid validation performance. Through experimenting it was found, that setting the maximum tree depth to 4, the learning rate to 0.05, the subsampling ratio of datapoints to 0.8 and of features to 0.6, gives a validation Gini of $0.91986$ in under 2 minutes of training time.

#### 4.2.1 Hardness characterization

Data-IQ’s confidence scores and aleatoric uncertainties are estimated for each training observation as per Section 2.1. Figure 4(b) displays the relationship between the two: in the top left a large amount of “easy” points, with high confidence and low aleatoric uncertainty; in the bottom left a few “hard” points, with low confidence and low aleatoric uncertainty; lastly, the “ambiguous” points are located around the elbow, with high aleatoric uncertainty. Looking at the marginal distributions in Figure 4(c), it is noticed that XGBoost is generally very confident in its predictions, causing the number of hard points to be low. Setting a low confidence threshold of $0.25$, a high confidence threshold of $0.75$ and a low aleatoric uncertainty threshold of $0.2$ allows to assign to each point a tag in $\{\textit{Easy},\textit{Hard},\textit{Ambiguous}\}$. Hard points are less than $3\%$ of training data and both hard and ambiguous points have a much higher proportion of defaulters than the full data (cf. Table 4(a)).

Figure: (a) Hardness characterization results
Refer to caption: x7.png

Beyond these initial statistics, it is interesting to know where these points lie in feature space. By looking at the total reduction in loss due to a feature in the nodes where said feature is used to split the data, we can calculate the feature importances of a trained XGBoost classifier. In the case of the Amex dataset, the first feature dominates these scores, with an importance comparable to the bottom 100 features combined. Figure 4(d) displays the distribution of this feature separated by Data-IQ tag for both defaulters and non-defaulters: hard defaulters somewhat overlap with easy non-defaulters and vice-versa, while ambiguous defaulters are almost indistinguishable from ambiguous non-defaulters. This is a common issue with tabular data, referred to as outcome heterogeneity and is known to be captured by Data-IQ (Seedat et al., ).

KNN Shapleys are then computed for $K\in\{1,5,100\}$ under the assumption that the $n_{\mathrm{test}}$ available out-of-sample data points are representative of the entire data distribution.
Figure 6 displays the distribution of 100NN Shapleys separated by Data-IQ’s tag: hard points as per Data-IQ exhibit low 100NN Shapleys, with very good adherence for the hardest points. In addition, the left tail of the ambiguous points lies in the region of low 100NN Shapleys and would thus be recognized as hard by our novel characterizer.

Figure: Figure 5: Distribution of the 100NN data Shapley scores for different Data-IQ tags.
Refer to caption: x10.png

To choose the best $K$, validation Gini is monitored as we re-train XGBoost after gradually removing the hardest datapoints. These points are expected to be the most valuable for XGBoost and thus the best hardness characterizer should be the one displaying the fastest drop in validation performance. 100NN Shapleys outperform both other choices of $K$ and Data-IQ (cf. Figure 6). This comparison highlights an issue with Data-IQ: after a sharp drop when the hard points are removed, performance decreases at a rate comparable to random as the most ambiguous points are removed. Seedat et al. () argue that the removal of ambiguous points should actually make the model more robust, as aleatoric uncertainty is data-dependent and cannot be reduced unless additional features are collected. This is however not the case for the Amex dataset, and beyond the small amount of hard points identified, Data-IQ does not provide guidance on the next most valuable points.

#### 4.2.2 Synthetic data generation

The $10\%$ hardest datapoints are chosen as training set for synthetic data generators, with the purpose of establishing whether augmenting only the hardest datapoints makes XGBoost more robust than non-targeted data augmentation. The choice of a hard threshold is imposed by the expensive computational cost of fine-tuning neural networks, and $10\%$ is justified as it is just before the elbow region where the decrease in validation performance when removing hard data slows down (cf. Figure 6). Both CTGAN and TVAE are implemented in Python using a heavily customized version of the sdv package (Patki et al., ), with extensive details about the training given in Appendix A.1.
The quality of the synthetic samples can be examined directly for the first feature: Figure 7(a) (CTGAN) displays a good but not perfect overlap with real data, whereas from Figure 7(b) (TVAE) we can see better overlap of synthetic and real data with respect to CTGAN for the first feature.

Figure: (a) CTGAN
Refer to caption: x12.png

For models trained on the $10\%$ hardest points we augment by $100\%$, while for models trained using the entire dataset we augment by $10\%$, thus guaranteeing the same amount of synthetic samples across all experiments. Notice that for each attempt synthetic data has to be generated multiple times with different seeds to mitigate randomness and quantify uncertainty on the performance metric. The proposed routine, implemented in Python with the baytune package (Smith et al., ), requires very few attempts to find good proposals, and is thus more efficient than a grid search over a hyperparameter space, reducing the computational complexity of the problem.

Figure: (a) Scores in fine-tuning routine: hard points
Refer to caption: x14.png

Figures 8(a) and 8(b) display the scores after 20 attempts: the best scores are in both cases achieved by TVAE, and augmenting only the hardest points generally leads to a more significant improvement in validation performance, both for the best attempts and on average across different sets of hyperparameters. Information on the tuning grid and best hyperparameters are detailed in Appendix A.2.

Table 8(c) reports the performance variations for the best TVAE, CTGAN and SMOTE: the latter does not work for the hardest points and is outperformed by TVAE for general augmentation. The robustness of these results for the best attempts can be verified by augmenting by different amounts ($5\%,10\%,15\%,20\%$), with results displayed in Figure 8(d): generating more hard points further increases performance with diminishing returns after $15\%$, while for non-targeted augmentation, the performance actually worsens as more data is added due to the extra noise. Finally, to quantify the magnitude of this improvement, adding back the $n_{\mathrm{test}}=50\,000$ observations used to compute KNN Shapleys into training data improves validation performance by $\approx 0.000326$, which is less than the improvement obtained from augmentation of the hard points via TVAE.

It could be argued that the improvement in validation performance achieved via data augmentation, although larger for models trained on only the hardest datapoints, is very small in absolute value.
It must be remarked that similar improvements have only been achieved in the original Kaggle competition by generating thousands of features and building large ensemble models. Additionally, XGBoost already classifies over $90\%$ of validation data correctly, and any further improvement on this already highly performing model may have business value in practice.

### 4.3 Simulation study

To verify the robustness of the results, we run the same hardness characterization and data augmentation pipeline on simulated data. Specifically, we consider four bivariate normal distributions, assigning to two of them label $y=0$, and $y=1$ to the remaining ones. We draw $n_{\mathrm{train}}=5\,000$ training datapoints, $n_{\mathrm{valid}}=2\,500$ observations for model validation and lastly $n_{\mathrm{test}}=2\,500$ datapoints to calculate $5$NN Shapleys. The training dataset can be visualized in Figure 9(a) on the left, while the $5\%$ hardest points according to $5$NN Shapleys are shown on the right: notice that they are concentrated around the decision boundary, with most of them falling on the “wrong” side, confirming the issue of outcome heterogeneity for hard data points discussed in Section 4.2.1.

We proceed by tuning TVAE using the GPEI algorithm (described in Appendix A.1) both when augmenting the hardest $5\%$ by $100\%$ and when augmenting the entire dataset by $5\%$. Figures 9(c) and 9(d) display the scores after 10 attempts: the targeted approach results in a larger performance improvement, both for the best attempt and on average across different sets of hyperparameters. Information on the tuning process can be found in Appendix A.3.
Finally, we augment by different amounts using the best attempts, with results displayed in Figure 9(b): hard points augmentation consistently outperforms the non-targeted approach and improves as more data is added.

Figure: (a) Scatter of training data (left) and hardest $5\%$ (right).
Refer to caption: x17.png

## 5 Conclusion and discussion

In this work, we verified empirically that, when it comes to data augmentation, focusing only on the most challenging points for the model can be beneficial, both in terms of performance improvement and computational efficiency. To achieve this result, we first devised a novel hardness characterizer based on KNN Shapleys, capable of achieving performances comparable to state-of-the-art methods in hardness detection benchmarks on tabular data. Then, we performed a complex fine-tuning routine on synthesizers trained either on the entire dataset or on the hardest points only, allowing to rigorously establish which approach leads to the largest performance improvement.

Methodologically, this work aims at bridging the gap between learning-based hardness characterization and game theoretic data valuation. The benchmarks reported in Section 4.1 represent the first quantitative comparison between a Shapley-based evaluator and existing hardness characterizers. While research on the two topics has often dealt with similar problems, such as mislabeling or data summarization (Paul et al., ; Jia et al., ), the two paths have never crossed.

Practically, this work attempts to change the perspective on data summarization: while existing literature has mostly focused on pruning the least valuable points from training data (Seedat et al., ; Kwon & Zou, ), here we propose augmenting the most valuable ones. We demonstrate that not all datapoints are created equal, and some play a more significant role in determining the predictive power of the final model: high-value data dictates which points we should either collect in larger amounts or, when this is not possible, synthetically generate.

#### Reproducibility statement

The Amex dataset is publicly available on the public data repository [Kaggle](https://www.kaggle.com/competitions/amex-default-prediction/data). This work uses the denoised version by user raddar, which is publicly available on Kaggle in [parquet format](https://www.kaggle.com/datasets/raddar/amex-data-integer-dtypes-parquet-format?select=train.parquet).
The code to reproduce the results in this work is available in the GitHub repository [tommaso-ferracci/Targeted_Augmentation_Amex](https://github.com/tommaso-ferracci/Targeted_Augmentation_Amex). In particular, code to reproduce the results in Section 4 is available in the analyses/ folder. The outputs/ folder contains, in addition to figures and results, the model weights for the best synthesizers. Notice how every method from the ctgan and sdv packages uses the forked versions ([tommaso-ferracci/CTGAN](https://github.com/tommaso-ferracci/CTGAN)  and [tommaso-ferracci/SDV](https://github.com/tommaso-ferracci/SDV)) with the custom early stopping. Detailed instructions on how to create a Python virtual environment and install these dependencies are available in the Makefile. Details around hyperparameter tuning are also reported in Appendices A.1, A.2 and A.3.

## Appendix A Appendix

### A.1 Training CTGAN and TVAE – Amex dataset

This section provides details about CTGAN and TVAE SDGs on the Amex dataset.
A common issue when training GANs is the instability of both generator and discriminator losses. More specifically, the two typically move in opposite ways with oscillatory patterns, making it difficult to decide when to stop training. For this reason, we introduce a novel early stopping condition which tracks epoch after epoch a weighted average of the Kolgomorov-Smirnov (KS) statistic between real and synthetic data across all individual features. In particular, we choose to use as weights the feature importances, in order to focus more on the features relevant to XGBoost. The patience is set to $50$ epochs with a maximum number of epochs of $500$. If the early stopping condition is triggered, the model is reverted to the best epoch. A large batch size of $10\,000$ is chosen to limit the number of updates per epoch and guarantee a smoother training process. Both the generator and discriminator are trained using the Adam optimizer (Kingma & Ba, ), setting the learning rate to $2\cdot 10^{-4}$, weight decay to $10^{-6}$, and the momentum parameters to $\beta_{1}=0.5$ and $\beta_{2}=0.9$.
Figure 10(a) shows the losses and the statistic epoch by epoch: we can see that the losses move against each other and then more or less converge once the model cannot further improve. We notice oscillatory patterns in the tracked statistic, symptomatic of the training dynamics of the generator and discriminator pair, and the early stopping condition kicking in after around 100 epochs when the weighted KS statistic peaks at $0.83$.

Training of the VAE relies on the Adam optimizer with a learning rate of $10^{-3}$, weight decay of $10^{-5}$, $\beta_{1}$ of $0.9$ and $\beta_{2}$ of $0.999$. To avoid overfitting of training data and the subsequent generation of exact replicas of training points, we stop training once the maximum number of epochs is reached.

The following routine integrating both model selection and hyperparameter tuning is carried out separately for models trained on only the hardest points and models trained using the entire dataset:

- •
Model selection: after training one instance of TVAE and CTGAN to gather initial data, for each candidate model $i$ the Upper Confidence Bound 1 (UCB1) score is calculated:
$\mathrm{UCB1}(i)=\bar{\psi}_{i}+\sqrt{\frac{2\ln t}{n_{i}}},$
where $\bar{\psi}_{i}$ is the average score of model $i$, $t$ is the total number of models trained, and $n_{i}$ is the number of times model $i$ is selected. We select the model with the highest UCB1 score, which results from a balance of exploration (low $n_{i}$) and exploitation (high $\bar{\psi}_{i}$).
- •
Hyperparameter tuning: once model $i$ has been proposed, the set of hyperparameters to try is chosen via Gaussian Process Expected Improvement (GPEI): a Gaussian Process (GP) is fitted to $\{(\mathbf{h}_{j},\psi_{j})\}_{j=1}^{n_{i}}$, with $\mathbf{h}_{j}$ hyperparameters and $\psi_{j}$ corresponding score, and then the new set to try is chosen to maximize over $\mathbf{h}$ the Expected Improvement $\mathrm{EI}(\mathbf{h})=\mathbb{E}[\max(0,f(\mathbf{h})-f(\mathbf{h}^{*}))]$, with $f$ mean function of the fitted GP and $\mathbf{h}^{*}$ the best set so far.

The score $\psi$ over which the optimization is carried out, is the variation in validation Gini after training XGBoost on the augmented dataset.

Figure: (a) CTGAN
Refer to caption: x22.png

### A.2 Hyperparameters - Amex dataset

For TVAE trained on the hardest $10\%$, tuning is performed on the embedding dimension and the architectures of encoder and decoder. We fix the number of hidden layers to two for the encoder and the decoder, tuning the number of units in each hidden layer. Details on the grid search and the best attempt can be found in Table 1, while a parallel coordinates plot is displayed in Figure 11.
For CTGAN trained on the hardest $10\%$, tuning is performed on the embedding dimension and the discriminator and generator architectures. We fix the number of hidden layers to two for the discriminator and the generator, tuning the number of units in each hidden layer. Further details can be found in Table 2 and Figure 12.
Similar summaries are reported for TVAE (Table 3, Figure 13) and CTGAN (Table 4, Figure 14) on the entire dataset. Notice once again the deterioration in performance with respect to the targeted case.

Figure: Table 1: TVAE, hard 10%: tuning setup.
Refer to caption: x24.png

Figure: Table 2: CTGAN, hard 10%: tuning setup.
Refer to caption: x25.png

Figure: Table 3: TVAE, full data: tuning setup.
Refer to caption: x26.png

Figure: Table 4: CTGAN, full data: tuning setup.
Refer to caption: x27.png

Figure: Table 5: TVAE, hard 5%: tuning setup.
Refer to caption: x28.png

Figure: Table 6: TVAE, full data: tuning setup.
Refer to caption: x29.png

### A.3 Hyperparameters - Simulation study

The setup is analogous to Appendix A.2, with the difference that CTGAN has been discarded due to excessive instability during training, and smaller architectures are considered to account for a much simpler dataset. Specifically, Table 5 and Figure 15 report the results for TVAE trained on the hardest 5%, while Table 6 and Figure 16 present TVAE trained on the entire dataset. Once again, we notice worse performance in the non-targeted case in the parallel coordinates plots .

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: See above in weaknesses for suggestions/questions.
soundness: 3
strengths: 1. Originality: The paper introduces a novel approach to synthetic data generation, focusing on "hard" data points identified via KNN-based Shapley values. This method augments only challenging points, enhancing model robustness and bridging data valuation with data augmentation.

2. Quality: Extensive experiments on real and simulated datasets. Detailed benchmarking against existing hardness characterizers and comprehensive hyperparameter tuning validate the method, particularly w.r.t. its computational efficiency and model robustness.

3. Clarity: The paper is well-structured and clear. The figures and precise mathematical formulations make it easier to understand the paper. 

4. Significance: This approach has broad applicability for fields like finance and healthcare, where robust tabular data predictions are essential. By focusing on challenging data points, the method not only improves model performance but also offers a cost-effective solution.

In summary, the paper makes a strong contribution to synthetic data generation and introduces novel methods and demonstrates practical benefits for high-stakes applications involving tabular data.
summary: The paper proposes a novel synthetic data augmentation approach focused on enhancing binary classification models, particularly on tabular data. It introduces a targeted synthetic data generation pipeline, using KNN-based Shapley values to identify the "hardest" points in the training data. By generating synthetic data specifically for these challenging data points, the paper aims to strengthen the model’s ability to generalize to new, unseen data while reducing computational overhead compared to non-targeted augmentation methods. Empirical evaluations demonstrate the benefits of this approach in improving out-of-sample prediction quality, particularly in applications like credit default prediction.
weaknesses: 1. The paper's focus is restricted to binary classification tasks, which limits the generalizability of the approach. Could the authors consider experimenting with multi-class tasks at least?
2. The paper relies heavily on KNN Shapley values for hardness characterization. It might be interesting to explore other methods such as gradient-based techniques (e.g., GraNd) or dropout-based uncertainty measures. Would the authors be open to trying this?
3. The method relies on specific synthetic data generators like CTGAN and TVAE, which may not perform as effectively on all types of tabular data (e.g., highly skewed or sparse datasets). 
4. The study uses a fixed threshold (e.g., augmenting 10% of the hardest points) which seems a bit hard to justify. Could the authors consider other alternatives?

---

confidence: 2
contribution: 2
initial_rating: 3
presentation: 3
questions: 0. **Redundancy?** Only for example:
  - Section 2. "Background" spans 2 pages describing in (too) much detail the already published methods (KNN Shapley, Data-IQ, SMOTE, CTGAN, TVAE), which are later applied but not modified or analyzed.
  - L315-L348 are details about training a single XGBoost model that would usually be found in the Appendix.
  - L351-369 discusses results only for the Data-IQ method (Seedat et al., 2022). Virtually Section 4.2.1 reads like a critique of Data-IQ (Seedat et al., 2022) instead of showing the applicability of the proposed method. Perhaps it can be better communicated to the reader at the beginning of this section / in its title (currently "Harndess characterization").
  - Figure 7 provides no valuable information.
1. The font size in Figures 1, 6, 7, 8(a,b), and 9(a) is too small. This could be fixed by saving the figures to smaller sizes so that all the elements will increase in comparison.
2. Please clarify: how do you label generated samples?
3. Clarify in the paper: In Sec. 3, what is model A? Does it need to be a KNN? (no)
4. Discuss in the paper: Does this method work for tabular regression tasks? (no)
5. In L300, we read "not requiring model training", but then L376 says "To choose the best K, validation Gini is monitored as we re-train XGBoost after gradually removing the hardest datapoints".
6. What is the motivation behind using the normalized Gini coefficient "2*AUCROC − 1" instead of F1?

There is a related work that came to mind when reading the paper:

- How does this relate to the concepts of dataset distillation (arXiv:1811.10959) and dataset condensation (ICLR 2021)?
- Can alternative data generators, e.g. TabDDPM (ICML 2023) or Adversarial random forests (AISTATS 2023), be used in this framework?
soundness: 2
strengths: This work proposes to change the perspective on data distillation, i.e. to augment the most valuable points in training data, instead of pruning the least valuable ones. The idea itself is original and valuable, but...
summary: This paper proposes to (1) use data valuation methods rooted in game theory to characterize samples that are difficult / hard to learn, and (2) improve model learning / performance by generating synthetic data around these hard / highly valued samples. It is an empirical study that extends our understanding of known methods (KNN Shapley, Data-IQ, SMOTE, CTGAN, TVAE).
weaknesses: ... my impression is that this work resembles a shorter workshop paper rather than a complete contribution, i.e.:

0. (For context, there are no theoretical contributions in the paper, which is not a weakness in isolation.)
1. **Empirical evaluation.** There is little signal that the proposed method works. Although I am generally in favor of interesting, simple ideas, even insightful negative results, experiments are lacking:
  - A) Overall results with only 4 tabular dataset-model pairs are rather anecdotal and insufficient to make meaningful conclusions. What holds back from doing a comprehensive benchmark on 10++ datasets/models (e.g. OpenML-CC18, OpenML-CTR23 benchmarks) that would empirically show the method's superiority?
  - B) Results presented in Sec. 4.1 are unconvincing, i.e. the method performs well on one of two datasets. The heatmap is unclear (I can't see differences between the columns/rows); why not present AUPRC in a table / lineplot / barplot, also with deviations / confidence? Moreover, it seems the mentioned conclusion about differences between the methods (L297-L302) could be derived without doing this benchmark at all.
  - C) Regarding Sec. 4.2, L468-473: I can instead argue that the chosen dataset + model (Amex + XGBoost) is not a good example to show the added value of the proposed method. Mind you, this improvement by 0.0004 seems to be the paper's main result.
  - D) The last Section 4.3 with Figure 9 is an ablation "to verify the robustness of the results" on synthetic data instead of making a convincing case on more various datasets.
2. **Computational efficiency.** There are multiple unsupported claims made about the method's efficiency, also as compared to baselines. There are no experimental results to support these claims, e.g. analyzing the computational speed of KNN Shapley, showing its impact on the XGBoost model's convergence and computational time.
3. **Redundancy in presentation?** Subjectively, the paper has overlong descriptions and too many figures, if only to fill the 10 pages. There are 9 figures containing 26 subfigures / tables, potentially obfuscating the unseen key result / contribution. See details below.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. The authors set aside a significant portion of data for hardness characterization. This does not seem ideal in a scenario where data augmentation is required since the additional data could be used for training the model instead. Do other hardness measures also require such hold-out sets?
2. Is there any downside to adding back the held out data together with the generated synthetic data when retraining the model? If not why wasn't this part of the experimental setup?
3. KNN assumes a distance metric over the input space. This is problematic for structured tabular data with mixed categorical and numerical features where a natural distance metric is not available and each feature can have different importances and scales. How was the distance chosen for KNN and how can the authors justify it?
soundness: 1
strengths: The paper is generally well-written, with a coherent presentation of the main ideas.

The authors go to great lengths to justify every decision and significant care seems to have been taken to achieve a sensible evaluation setup that appropriately accounts for the inherent variance in the methods.
summary: This paper presents two main contributions:
* Proposing the use of KNN Shapley values for hardness characterization
* Proposing a synthetic data augmentation pipeline that relies on training a generative model exclusively on hard data points

Both contributions are evaluated empirically. The effectiveness of Shapley values on a benchmark consisting of two real world datasets where it is compared to other hardness characterization baselines. 
The synthetic data augmentation pipeline is applied to a large credit default prediction dataset, demonstrating modest performance gains, with a slight advantage observed when augmenting using only hard examples.
weaknesses: **Main Points**

The main weaknesses of the paper lie with the experimental setup:
1. The main claims of the paper on data augmentation are supported by experiments on a single real-world dataset. This seems insufficient to draw meaningful conclusions about the wider applicability of the method. I would be interested to know if the conclusions hold on a wider range of datasets with different sizes and tasks.
2. Likewise, the results on KNN Shapleys as a hardness characterization are supported by evaluations on a benchmark of only 2 tabular datasets.
3. It would also be interesting if KNN Shapleys were compared to other hardness measures on the downstream synthetic data augmentation task.
4. More recent methods such as [1], [2] or [3] have shown significant improvements for synthetic data generation in tabular data over TVAE and CTGAN. It would be more relevant to evaluate the synthetic data generation pipeline with these methods.



**Minor Point**

The authors might want to be more precise with statements that can be easily misinterpreted such as (page 8):
> For models trained on the 10% hardest points we augment by 100%, while for models trained using
> the entire dataset we augment by 10%, thus guaranteeing the same amount of synthetic samples
> across all experiments. 

I am assuming the 100% refers to the size of the training set used for the generative model (10% of the original training set).

**References**

[1] Akim Kotelnikov, et al. TabDDPM: Modelling Tabular Data with Diffusion Models, 2022. https://arxiv.org/abs/2209.15421

[2] Jayoung Kim, et al. STaSy: Score-based Tabular data Synthesis, 2022. https://arxiv.org/abs/2210.04018

[3] Hengrui Zhang, et al. Mixed-Type Tabular Data Synthesis with Score-based Diffusion in Latent Space, 2023. https://arxiv.org/abs/2310.09656

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 4
questions: - Q1: As mentioned in the weaknesses how does the proposed approach to targeted synthetic data generation compare conceptually and empirically to: https://arxiv.org/abs/2310.16981

- Q2: How does the augmentation of only hard samples affect statistical fidelity?

- Q3: Would you say the findings are generalizable to any tabular dataset?
soundness: 2
strengths: Originality: The idea of augmenting the hard samples is novel as is using KNN Shapley values to target hard samples for synthetic data generation is interesting as it differs from the different hardness characterization literature. Although see weaknesses below: neither in KNN shapley novel neither is targeted synthetic data generation.

Quality: The empirical quality is good and the methodology is rigorous and well-executed, with solid empirical benchmarks.

Clarity: The paper is well written and the clear details elucidate the problem and solution well.

Significance: Targeted synthetic data generation could be impactful in improving model robustness, particularly when data is scarce or of poor quality. So overall the problem seems important
summary: The paper proposes targeted synthetic data generation for tabular data using hardness characterization. It introduces a pipeline that focuses on generating synthetic data for the hardest points within the training dataset, as identified by a KNN Shapley-based hardness characterizer. The paper shows that augmenting only the hardest points improves out-of-sample performance and computational efficiency compared to non-targeted data augmentation approaches.
weaknesses: There are three major weaknesses: Novelty, metrics and evaluation

Novelty:
- Use of KNN Shapley: the proposed approach to assess samples using KNN-Shapley isn’t novel, rather already proposed in Jia et al. (https://arxiv.org/abs/1911.07128). So while the application to synthetic data is novel, the proposed KNN-shapley assessment approach option itself is not novel. By itself this isn't an issue. But because the paper proposes to use an off-the-shelf valuation approach, it then warrants assessing why this is the best data valuation approach vs other alternative data valuation approaches (e.g. https://arxiv.org/pdf/2306.10577)

- Targeted synthetic data generation: The paper proposes KNN-shapley to do the targeting. However, the idea of using the targeted samples to condition synthetic generation also mirrors other works from data-centric literature (see https://arxiv.org/abs/2310.16981), which use the hardness ideas mentioned in the paper. It is important to contrast the different approaches in the related work for targeting synthetic data generation vs what's proposed in the paper --- both conceptually and empirically to understand strengths and weaknesses.

Metrics:
- The paper shows the effect of the synthetic data in terms of performance and Gini scores. It is important when using synthetic data to also assess how the statistical fidelity is affected. i.e. is the synthetic data good statistically with respect to real data. It would improve the paper to assess fidelity metrics like KL-Divergence, MMD, Precision and Recall 
- Examples of widely used synthetic data metrics (particularly fidelity) can be found in: (1) https://arxiv.org/abs/1806.00035, (2) https://arxiv.org/pdf/2406.13130v1, (3) https://arxiv.org/abs/2102.08921

Evaluation:
The assessment on the Amex dataset is interesting, however to claim generality of the findings a greater diversity of datasets should be explored, either across domains or with different properties (sample sizes, feature dimensionality)