Can You Rely on Your Model Evaluation? Improving Model Evaluation with Synthetic Test Data
Authors: Boris van Breugel
Nabeel Seedat
Fergus Imrie
Mihaela van der Schaar
Venue: NeurIPS 2023 poster (2023)
Keywords: model evaluation
tabular
synthetic data
TL;DR: Model evaluation in tabular data settings using synthetic data

Abstract: Evaluating the performance of machine learning models on diverse and underrepresented subgroups is essential for ensuring fairness and reliability in real-world applications. However, accurately assessing model performance becomes challenging due to two main issues: (1) a scarcity of test data, especially for small subgroups, and (2) possible distributional shifts in the model's deployment setting, which may not align with the available test data.  In this work, we introduce 3S Testing, a deep generative modeling framework to facilitate model evaluation by generating synthetic test sets for small subgroups and simulating distributional shifts. Our experiments demonstrate that 3S-Testing outperforms traditional baselines---including real test data alone---in estimating model performance on minority subgroups and under plausible distributional shifts. In addition, 3S offers intervals around its performance estimates, exhibiting superior coverage of the ground truth compared to existing approaches.  Overall, these results raise the question of whether we need a paradigm shift away from limited real test data towards synthetic test data.

Can You Rely on Your Model Evaluation? Improving Model Evaluation with Synthetic Test Data
Boris van Breugel" University of Cambridge bv2920cam. ac uk
Nabeel Seedat" University of Cambridge ns7410cam. ac uk
Fergus Imrie
Mihaela van der Schaar University of Cambridge mv4720cam ac. uk
UCLA imrie0g ucla. edu
Abstract
Evaluating the performance of machine learning models on diverse and underrep- resented subgroups is essential for ensuring fairness and reliability in real-world applications. However, accurately assessing model performance becomes chal- lenging due to two main issues: (I) scarcity of test data, especially for small subgroups, and (2) possible distributional shifts the model's deployment set- ting, which may not align with the available test data. In this work, we introduce 3S Testing, a deep generative modeling framework to facilitate model evaluation by generating synthetic test sets for small subgroups and simulating distributional shifts. Our experiments demonstrate that 3S Testing outperforms traditional baselines-inntudingg real test data alone-in estimating model performance minority subgroups and under plausible distributional shifts. addition, 3S offers intervals around its performance estimates, exhibiting superior coverage of the ground truth compared to existing approaches. Overall, these results raise the question of whether need paradigm shift away from limited real test data towards synthetic test data.
Introduction
Machine learning (ML) models increasingly deployed in high-stakes and safety- critical areas, e.g. medicine or finance-setting that demand reliable and measurable performance [1]. Failure to rigorously test systems could result in models best failing unpredictably and worst leading to silent failures. Regrettably, such failures of ML are all too common [2-9]. Many mature industries involve standardized processes evaluate performance under various testing and operating conditions [10]. For instance, automobiles use wind tunnels and crash tests to assess specific components, whilst electronic component data sheets outline conditions where reliable operation is guaranteed. Unfortunately, current evaluation approaches of supervised ML models do not have the same level of detail and rigor.
Motivation.
The prevailing testing approach in ML is to evaluate only using average prediction performance on held-out test set. This can hide undesirable performance differences more granular level, e.g. for small subgroups [2, 4, 5, 11, 12], low-density regions [13-15], and individuals [16-18]. Standard ML testing also ignores distributional shifts. In ever-evolving world where ML models are employed across borders, failing to anticipate shifts between train and deployment data can lead overestimated real-world performance [6-8, 19, 20].
Equal Contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

However, real test data alone does not always suffice for more detailed model evaluation. Indeed, testing can be done on a granular level by evaluating on individual subgroups, and theory, shifts could be tested using e.g. rejection or importance sampling of real test data. The main challenge is that insufficient amounts of test data cause inaccurate performance estimates [21]. In Sec. 3 we will further explore why this is the case, but for now let us give an example.
Consider the real example of estimating model performance on the Adult dataset, looking at race and age variables- see Fig. 1 and experimental details in Appendix B. There are limited samples of the older Black subgroup, leading compared to an oracle. Similarly, if we tried to engineer a distribution shift towards increased age using rejection sampling, the scarcity of data would yield equally imprecise estimates. Such imprecise performance estimates could mislead us into drawing false conclusions about our model's capabilities. In Sec. 5.1, we empirically demonstrate how synthetic data can rectify this shortfall.
Example 1
significantly erroneous performance estimates
Aim. Our goal is to build a model evaluation frame- work with synthetic data that allows engineers, audi- tors, business stakeholders, or policy and compliance teams to understand better when they can rely on the predictions of their trained ML models and where they can improve the model further. We desire the following properties for our evaluation framework: (P1) Granular evaluation: accurately evaluate model performance on a granular level, even for re- gions with few test samples.
Acc Extimare Gracle
Black
RACE
Do Do
A
White
Cell
DO
8
22
AGE
Figure Limitation of testing with real data (Adult dataset): Insufficient real test sam- ples result imprecise performance estimates when (i) testing underrepresented subgroups (e.g. Black) and (ii) testing effect of a shift (e.g increased age)
(P2) Distributional shifts: accurately assess how distribution shifts affect model performance.
Our primary focus is tabular data. Not only are many high-stakes applications predominately tabular, such as credit scoring and medical forecasting [22. 23).
but the ubiquity of tabular data in real-world applications also presents opportunities for broad impact. To put it in perspective, nearly 79% of data scientists work with tabular data on a daily basis, dwarfing the 14% who work with modalities such images [24].
Moreover, tabular data presents us with interpretable feature identifiers, such as ethnicity or age, instrumental in defining minority groups shifts. This contrasts with other modalities, where raw data is not interpretable and external (tabular) metadata needed.
Contributions. ! Conceptually, we show why real test data may not suffice for model evaluation on subgroups and distribution shifts and how synthetic data can help (Sec. 3). 2 Technically, we propose the framework 38- 38-Testing (s.f. Synthetic data for Subgroup and Shift Testing) (Sec. 4). 35 uses conditional deep generative models create synthetic test sets, addressing both (PI) and (P2). 3S also accounts for possible errors in the generative process itself, providing uncertainty estimates for its predictions via deep generative ensemble (DGE) [25]. 3 Empirically, we show synthetic test data provides a more accurate estimate of the true model performance on small subgroups compared to baselines, including real test data (Sec. 5. I.),,, with prediction intervals providing good coverage of the real value (Sec. 5.1. We further demonstrate how 3S can generate data with shifts which better estimate model performance on shifted data, compared to real data or baselines. 3S accommodates both minimal user input (Sec. 5.2.1) or some prior knowledge of the target domain (See. 5.2.2).
2 Related Work
This paper primarily engages with the literature on model testing and benchmarking, synthetic data, and data-centric Al-see Appendix A for an extended discussion.
Model evaluation. ML models mostly evaluated on hold-out datasets, providing measure of aggregate performance [26]. Such aggregate measures do not account for under-performan on specific subgroups [2] or assess performance under data shifts [7, 27].
The MI. community has tried to remedy these issues by creating better benchmark datasets: either manual corruptions like Imagenet-C [28] by collecting additional real data such as the Wilds

---

benchmark [8]. Benchmark datasets are labor-intensive to collect and evaluation is limited to specific benchmark tasks, hence this approach is not flexible for any dataset or task. The second approach is model behavioral testing of specified properties, e.g. see Checklist [29] or HateCheck [30]. Behavioral testing also labor-intensive, requiring humans to create or validate the tests. In contrast to both paradigms, 3S generates synthetic test sets for varying tasks and datasets.
Challenges of model evaluation. 3S aims to mitigate the challenges of model evaluation with limited real test sets, particularly estimating performance for small subgroups or under distributional shifts. We are not the first to address this issue. Subgroups: Model-based metrics (MBM [21]) model the conditional distribution of the predictive model score to enable subgroup performance estimates. Distribution shift: Prior works aim to predict model performance shifted target domain using (1) Average samples above a threshold confidence (ATC [31]), (2) difference of confidences (DOC [32]; (3) Importance Re-weighting (IM [33]). A fundamental difference to 3S is that they assume access to unlabeled data from the target domain, which is unavailable in many settings, e.g. when studying potential effects of unseen or future shifts. We note that work on robustness to distributional shifts is not directly related, the goal is to learn a model robust to the shift, rather than reliably estimating performance of an already-trained model under a shift.
Synthetic data. Improvements in deep generative models have spurred the development of synthetic data for different uses [34], including privacy (i.e. to enable data sharing. 35, 36), fairness [37, 38]. and improving downstream models [39-42]. 3S provides completely different and unexplored use of synthetic data: improving testing and evaluation of ML models. Simulated (CGI-based) and synthetic images have been used previously in computer vision (CV) applications for a variety of purposes- - often to train more robust models [43, 44). These CV-based methods require additional metadata like lighting, shape, or texture [45, 46), which may not be available in practice. Additionally, beyond the practical differences between modalities, the CV methods differ significantly from 3S in terms of (i) aim, (ii) approach, and (iii) amount of data-sce Table 4, Appendix A.
3
Why Synthetic Test Data Can Improve Evaluation
3.1 Why Real Data Fails
Let X' and y be the feature and label space, respectively. The random variable X (X, Y) is defined on this space, with distribution p( X(X, Y). We assume access to a trained black-box prediction model f X + y and test dataset Dtest./ = {æ,, yi ) Nieue. rid ~ p(X,Y), Y). Importantly, we do not
Notation.
assume access to the training data of the predictive models, Dirain.f. Lastly, let M ) x y - R be a performance metric.
Real data does not suffice for estimating granular performance (P1). In evaluating performance of on subgroups, we assume that a subgroup S X' is given. The usual approach to assess subgroup performance is simply restricting the test set Dreat,f to the subspace S:
A(f: AlfiDteat. (teat.f. Drest. N
1
Σ
M(f(x),).
(I)
This is an unbiased and consistent estimate of the true performance, i.e. for increasing | this converges to the true performance A" (J; S) = ([M()(X), Y))(X,)) E S).
However, what happens when [Dtest,f n S) is small? The variance VD Dree Aff. Dreat, S) will be large. In other words, the expected error of our performance estimates becomes large.
Example 2 To highlight how this affects Eq. 1, let us assume the very simple setting in which p( p(Y). Despite Y being independent of X, Eq. 1 is not-the smaller S, the higher the expected error in our estimate. In particular, for |S| + 0 (w.rt. p(X)), almost certainly [Drest,f S| 0, making Eq. 1 meaningless.
As a result, we find that the smaller our subgroup S, the harder it becomes to measure model f. At the same time, ML models have been known to perform less consistently on small subgroups [2-5], hence being able to measure performance on these groups would be most useful. Finally, by definition, minorities are more likely to form these small subgroups, and they are the most vulnerable
3

---

to historical bias and resulting ML unfairness. In other words, model evaluation is most inaccurate on the groups that are most vulnerable and for which the model itself is most unreliable.
Real test data fails for distributional shift (P2). If we do not take into account shifts between the test and deployment distribution, trivially the test performance will be a poor measure for performance- often leading to overestimated performance [6-8]. Nonetheless, even if we do plan to consider shifts in our evaluation, for example by using importance weighting or rejection sampling based on our shift knowledge, real test data will give poor estimates. The reason is the same as before; in the regions that we oversample or overweight, there may be few data points, leading to high variance and noisy estimates. As expected, problems are most pervasive for large shifts, because these require higher reweighting or oversampling of individual points.
3.2 Why Generative Models Can Help
We have seen there are two problems with real test data. Firstly, the more granular a metric, the higher the noise in Eq. 1-even if the distribution is well- behaved like in Example 2. Secondly, we desire way to emulate shifts, and simple reweighing or sampling of real data again leads to noisy estimates.
(a) Original data
(b) With synthetic data
Figure 2: Illustration why synthetic data can give more accurate estimates using the same real samples. Assume we want to evaluate f (decision boundary line), which aims to discriminate between Y = (green stars) and Y = 0 (red circles). Due the low num- ber of samples for Y = 1, evaluating f using the test set alone (Eq. 1) has high variance. On the other hand, a generative model can learn the manifold from Dtest, and generate additional data for Y = 1 by only learning the offset (b, green triangles). This can reduce variance of the estimated performance of f.
Generative models can provide solution to both problems. As we will detail in the next section, in- stead of using D.est.f. we use a generative model G trained on f to create large synthetic dataset D Dayn for evaluating f- We can induce shifts in the learnt distribution, thereby solving problem It also solves the first problem. A generative model aims to approximate p(X, Y). which we can regard as effectively interpolating p(Y|X) between real data points to generate more data within S.
It may seem counterintuitive that A A(J; san,, S) would ever be lower than IA* A(J;Drest,
after all G is trained
on |Dtes,yl and there is no new information added to the system. However, even though the generative process may be imperfect, we will see that the noise of the generative process can be significantly lower than the noisy real estimates (Eq. 1). Secondly, a generative model can learn implicit data representations [39], i.e. learn relationships within the data (e.g. low-dimensional manifolds) from the entire dataset and transfer this knowledge to small S. We give a toy example in Fig. 2. This motivates modeling the full data distribution p(X, Y), not just [X)
Of course, synthetic data cannot always help model evaluation, and may in fact induce noise due to an imperfect G. Through the inclusion of uncertainty estimates, we promote trustworthiness of results (Sec 4.1,, and when we combine synthetic data with real data, we observe almost consistent benefits (see Sec. 5). In Section 6 we include limitations.
4 Synthetic Data for Subgroup and Shift Testing
4.1 Using Deep Generative Models for Synthetic Test Data
We reiterate that our goal is to generate test datasets that provide insight into model performance on granular level (PI) and for shifted distributions (P2). We propose using synthetic data for testing purposes, which we refer to as 3S -testing.. This has the following workflow (Fig. 3): (1) train a (conditional) generative model on the real test set, (2) generate synthetic data conditionally on the subgroup or shift specification, and (3) evaluate model performance on the generated data, A(;;D, Dayn,, S). This procedure is flexible w.r.t. the generative model, but a conditional generative model is most suitable since it allows precise generation conditioned on subgroup or shift information out-of-the-box. Throughout this paper we use CTGAN [47] as the generative model see Appendix C for other generative model results and more details on the generative training process.

---

Pridlren Basted reall vnlggrape
Prodlee: rest Cree connot fost skyli
1(3)
modde
Skn nagaiaale
Model evaluation
(I) Train tune Q
(1) Genorate
(1) Model evaluation
(PI) Reliable granular evaluation
(P2) Sensitivity distributional shifts
Figure 3: 3S is framework for evaluating model performance using synthetic data generators. It has three phases: training the generative model, generating synthetic data, and model evaluation. Firstly. 3S enables (P1) granular evaluation when there is (i) limited real test data in small subgroups. by (ii) generating synthetic data conditional on subgroup information Xx, thereby (ii) permitting more reliable model evaluation distributional shifts when (i) the real test data does not reflect shifts, by (ii) generating synthetic data conditional marginal shift information of features Xe, thereby (iii) quantifying model sensitivity to distributional shift. Required inputs are denoted in yellow.
small subgroups. Secondly, 38 enables assessment of (P2)
Estimating uncertainty. Generative models are not perfect, leading to imperfect synthetic datasets and inaccurate estimates. To provide insight into the trustworthiness of its estimates, we quantify the uncertainty in the 3S generation process through an ensemble of generative models [25], similar in vain Deep Ensembles [48]. We (i) initialize and train K generative models independently, (ii)
generate synthetic datasets (Di (ayy === and (iii) evaluate model f on each. The final estimates are assumed Gaussian, with statistics given by the sample mean and variance:
K Σ K
K Σ k=1
((A)
A(J:D syn: and (2(A) K
(A(EDE
S) ;(A))
(2)
which can be directly used for constructing prediction interval. In Sec. 5.1.2, we show this provides high empirical coverage of the true value compared to alternatives.
Defining subgroups. The actual definition of subgroups is flexible. Examples include a specific category of one feature (e.g. female), intersectional subgroups [49] (e.g. black, female), slices from continuous variables (e.g. over 75 years old), particular points of interest (e.g. people similar to patient X), and outlier groups In Appendix E, we elaborate some of these further.
4.2 Generating Synthetic Test Sets with Shifts
Distributional shifts between training and test sets are not unusual in practice 50, 51] and have been shown to degrade model performance [6. 8, 15, 52]. Unfortunately, often there may be or insufficient data available from the shifted target domain.
Defining shifts. In some cases, there is prior knowledge to define shifts. For example, covariate shift [53, 54] focuses on a changing covariate distribution p(X), ). but a constant label distribution p( pX) conditional on the features. Label (prior probability) shift [54, 55] is defined vice versa, with fixed p(X(Y) and changing p(Y).
Generalizing this slightly, we assume only the marginal of some variables changes, while the distribution of the other variables conditional on these variables does not. Specifically, let e c {1...., [X] denote the indices of the features or targets in X of which the marginal distribution may shift. Equivalent to the covariate and label shift literature, we assume the distribution p(X.(X.) remains fixed (č denoting the complement of e).² Let us denote the marginal's 's shifted distribution by y²( (xx)) with the shift parameterisation, with p" (X.) having generated the original data. The full shifted distribution is p(Xx|\Xc)p" (X..).
Example: single marginal shift. Without further knowledge, we study the simplest such shifts first: only a single X, 's marginal is shifted. Letting pXX,) denote the original marginal, we
*Concept drifts are beyond the scope of this work.
'This reduces to label shift-p(( YY) constant but p(Y) changed) and covariate shift - constant but p(X) changed) for X. Y and X. X. respectively.
5

---

define a family of shifts p p'(X;) with E R the shift magnitude. To illustrate, we choose mean shift for continuous variables, p²(X;) = p²(X; - s), and logistic shift for any binary variable, logit (X; 1) logit (p" (X,)) s.' As before, we assume p(X--lX,) remains constant. This can be repeated for all i and multiple to characterize the sensitivity of the model performance to distributional shifts. The actual shift can be achieved using any conditional generative model, with the condition given by X,,
Incorporating prior knowledge on shift. In many scenarios, we may want to make stronger assumptions about the types of shift to consider. Let give two use cases. First, we may acquire high-level statistics of some variables in the target domain e.g. we may know that the age in the target domain approximately follows normal distribution N (50, 10). In other cases, we may actually acquire data in the target domain for some basic variables (e-g- age and gender), but not all variables. In both cases, we can explicitly use this knowledge for sampling the shifted variables Xo. and subsequently generating X e.g. sample (case 1) age from N (50, 10) (case 2) (age, gender) from the target dataset. Variables X X are generated using the original generator G, trained on D.est.f-
Characterizing sensitivity to shifts. This gives the following recipe for testing models under shift. Given some conditional generative model G, we (i) train G to approximate p( xx)) (2) choose a shifted distribution pa px(X))=e.8 a marginal mean shift of the original y (X.) (Section 5.2.1), or drawing Te samples from secondary dataset (Section 5.2.2); (3) draw samples x and subsequently use G to generate the rest of the variables Xa conditional on these drawn samples- together givingDiyn* and (4) evaluate downstream models; (5) Repeat (2-4) for different shifts (e.g. shift magnitudes s) to characterize the sensitivity of the model to distributional shifts.
More general shifts. Evidently, marginal shifts can be generalised. We can consider family of shifts T and test how model would behave under different shifts in the family. Let P be the space of distributions defined on &. We test models on data from (X)) ). for all T T, with T For example, for single marginal shifts this corresponds to T (p) general recipe for testing models under general shifts then becomes
P. The follows. Let G' be some
= (X.))((Xx..x))))
generative model, we (1) Train generator G on Drrain,G to fit (X); ;);)))))))) (2) Define family of possible shifts T, either with or without background knowledge; Denote shift with magnitude by T": (3) Set
s and generate data D syn from T (p); (4) Evaluate model on D (5) Repeat steps 2-4 for different
families of shifts and magnitudes
5 Use Cases of 3S Testing
We now demonstrate how 3S satisfies (P1) Granular evaluation and (P2) Distributional shifts. We re- iterate that the aim throughout is to estimate the true prediction performance of the model / as closely as possible. We tune and select the generative model based on Maximum Mcan Discrepancy [56]. see Appendix C. We describe the experimental details, baselines, and datasets for each experiment further in Appendix B 5.
5.1
(P1) Granular Evaluation
5.1.1
Correctness of Subgroup Performance Estimates
Goal. This experiment assesses the value of synthetic data when evaluating model performance on minority subgroups. The challenge with small subgroups is that the conventional paradigm of using hold-out evaluation set might result in high variance estimates due to the small sample size.
Datasets. We use the following five real-world medical and finance datasets: Adult [57], Covid- 19 cases in Brazil [58], Support [59], Bank [60], and Drug [61]. These datasets have varying characteristics, from sample size number of features. They also possess representational imbalance and biases, pertinent to 3S [4, 5]: ! Minority subgroups: we evaluate the following groups which
"We consider any categorical variable with classes using different shifts of the individual probabilities. scaling the other probabilities appropriately.
*Code for use cases found at: https: //tbthhttttu com/ htmm//eedatabbee///50 wtwteeaaabeee//S-eeeeeeeeeee -es--eening or https:/ //gg whw. com/ http///vderrsaaaallalaee Testinng

---

35 Drug
3S+
MBM
Dtest Support
Adult
Covid
Bank
15
40
Estrain (MNE)
15
30
30
20
10
(29) (29)
(29)
(1.5%)
(19)
(70HU (324g)
GIOUD
Group
Group
Group
Group
Figure 4: Assessing the reliability of performance estimates based on mean absolute error of
the predicted performance estimate the performance evaluated by oracle (L better). 3S better approximates true estimates compared to both baselines. 3S+ enjoys the best of both worlds by combining synthetic and real data. We evaluate RF model, with other model classes shown in the Appendix.
differ in proportional representation - Adulr: Race; Covid: Ethnicity; Bank: Employment: Support: Race; Drug: Ethnicity. ② Intersectional subgroups: we evaluate intersectional subgroups [49] (e.g. black males or young females)- see Appendix F, intersectional model performance matrix.
Set-up. We evaluate the estimates of subgroup performance for trained model f using different evaluation sets. We consider two baselines: (1) f: a typical hold-out test dataset and (2) Model-based metries (MBM) [21]. MBM uses bootstrapping approach for obtaining multiple test sets. We compare the baselines to 3S testing datasets, which generate data to balance the subgroup samples: (i) 38 (Dsya): synthetic data generated by G, which is trained Diest, and (ii) 3S+ (Dsym U f)) test data augmented with the synthetic dataset.
For some subgroup S, each test set gives an estimated model performance A(;; D., S), which we compare to a pseudo-oracle performance A(f; ; Doracte, S): the oracle the performance of evaluuted large unseen real dataset Doracle p(X, Y), where [D (Doractel > [Dtest, As outlined above the subgroups are as follows: (i) Adult: Race, (ii) Drug: Ethnicity, (iii) Covid: Ethnicity (Region), (iv) Support: Race, (v) Bank: Employment status.
We evaluate the reliability of the different performance estimates based on their Mean Absolute Error (MAE) relative to the Oracle predictive accuracy estimates. We desire low MAE such that our estimates match the oracle.
Analysis. Fig. 4 illustrates across the datasets that the 3S synthetic data (red, green) closely matches estimates on the Oracle data. i.e. lower MAE baselines. In particular, for small subgroups (c.g. racial minorities), 3S provides a more accurate evaluation of model performance (i.e. with estimates closer to the oracle) compared
conventional hold-out dataset (Dtest, and MBM.
In addition, 3S estimates have reduced standard deviation. Thus, despite 3S using the same (randomly drawn test set) / to train its generator, its estimates more robust to this randomness. The results highlight an evaluation pitfall of the standard hold-out test set paradigm: the estimate's ' high variance w.r.t. the drawn Dtest.f could lead to potentially misleading conclusions about model performance in the wild, since an end-user only has access to a single draw of Dtest, e.g., we might incorrectly overestimate the true performance of minorities. The use of synthetic data solves this.
Next, we move beyond single-feature minority subgroups and show that synthetic data can also be used to evaluate performance on intersectional groups subgroups with even smaller sample sizes due to the intersection. 3S performance estimates on 2-feature intersections are shown in Appendix Intersectional performance matrices provide model developers more granular insight into where they can improve their model most, as well as inform users how a model may perform on intersections of groups (especially important to evaluate sensitive intersectional subgroups). Appendix F further illustrates how these intersectional performance matrices can be used as part of model reports.
We evaluate the intersectional performance estimates of 3S and baseline Drest,f using the MAE of the performance matrices w.r.t. the oracle, averaged across 3 models (i.e, RF, GBDT, MLP). The error of 3S (11.90 + 0.19) is significantly lower than Drest,f (20.29 + 0.14), hence demonstrating 3S provides more reliable intersectional estimates.
'N.B, low- www-performance estimates by 3S only indicate poor model performance; this does not necessarily imply that the data itself is biased for these subgroups. However, it could warrant investigating potential data bias and how improve the model.
7

---

Takeaway. Synthetic data provides more accurate performance estimates on small subgroups compared to evaluation on standard test set. This is especially relevant from representational bias and fairness aelloective-alloinng more accurate performance estimates on minority subgroups.
5.1.2 Reliability through Confidence Intervals
Goal. In Fig. 4, we see that all methods, including 3S, have errors in some cases, which warrants the desire to have confidence intervals at test time. 3S uses a deep generative ensemble to provide uncertainty estimates at test-time- see Sec. 4.1.
Bontsiap
Mcan Coverage
MBM
0.8
35 [K-1]
35 [K-5]
0.8
(K-10)
Set-up. We assess coverage over 20 random splits and seeds for 3S vs the baselines (B1) bootstrapping [62] confidence intervals for D and (B2) MBM: which itself uses bootstrapping for the predictive distribution. For 3S, we assess a Deep Genera- tive Ensemble with K = 1,5, 5, 10 randomly initialized generative models. For each method, we take the average estimate + stan- dard deviations. We evaluate the intervals based on the following two metrics defined in [63-65]] (i) Coverage E [ [.. (ii) Width El[r: - 1.]l- Coverage measures how often the true label is in the prediction region, while width measures how specific that prediction region In the ideal case, have high coverage with low width. See Appendix B for more details.
0.2
0.2
0.3 0.4
Mean Interval Width
Figure 5: 3S is more reliable than baselines with higher coverage, lower widths with less variabil- ity.
Analysis. Fig. 5 shows the mean test set coverage and width averaged over the five datasets. 3S (with K=5 and K=10) is more reliable, attaining higher coverage rates with lower width compared to baselines. In addition, the variability with 3S is much lower for both coverage and width. We note that this comes at a price: computational cost scales linearly with K. For fair comparison, we set K in the of the paper.
Takeaway. 35 includes uncertainty estimates at test time that cover the true value much better than baselines, allowing practitioners to decide when (not) to trust 3S performance estimates.
5.2 (P2) Sensitivity Distributional Shifts
ML models deployed in the wild often encounter data distributions differing from the training set. We simulate distributional shifts to evaluate model performance under different potential post-deployment conditions. We examine two setups with varying knowledge of the potential shift.
5.2.1 No Prior Information
Goal. Assume we have no prior information for the (future) model deployment environment. In this case, we might still wish to stress test the sensitivity for different potential operating conditions, such that practitioner understands model behavior under different conditions, which can guide as to when the model can and cannot be used. We wish to simulate distribution shifts using synthetic data and assess it captures true performance.
Set-up. We consider shifts in the marginal of some feature X;, keeping p(X x/X;) fixed (see Sec. 3). For instance, a shift in the marginal X,'s mean (see Sec. 4.2). To assess performance for different degrees of shift, we compute three shift buckets around the mean of the original feature distribution: large negative shift from the mean (-), small negative/positive shift from the mean (+), and large positive shift from the mean (+).
We define each in terms the feature quantiles. We generate uniformly distributed shifts (between min(feature) and max(feature)). Any shift that shifts the mean to less than Q1 is (-), any shift that shifts the mean to more than Q3 is (+) and any shift in between (=).
As before, we compare estimated accuracy w.r.t. pseudo-oracle test set. We compare two baselines: (i) Mean-shift (MS) and (ii) Rejection sampling (RS); both applied the real data.
Analysis. Table shows the potential utility of synthetic data to more accurately estimate performance for unknown distribution shifts compared real data alone. This seen both with an average lower mean error of estimates, but also across all three buckets. This implies that the synthetic data is able closely capture the true performance across the range of feature shifts.

---

Table Mean error in estimated accuracy across shift quantile buckets for all 5 datasets. The results show 3S indeed provides more reliable estimates even for distribution shift. is better.
Adult
Support
Bank
Drug
SEER
Mean
Mcan
+
Mcan
Mean
Mcan
38
2.6 2.2 1.8 3.9
2.0 2.6 2.0
1.1
5,4
3.7 3.6
6.7
5.6 5.7
4.4
7.8
2.7
5,5
3.0
2.0
MS
5.9
5.2
5.6
6.9
19.3 22.9
18.5
15.2
18.6
18.6
19,9
17.9
18.5 19.6
19.0
16.3
3.3
2.6
3.9
3.2
RS
15.9
10.5
17.9
18.6
25.1
27.8
24.2
22.3
18.9
18.9
20.1
18.3
20.1
21.7
21.0
18.2
20.0
20.3
23.6
18.6
Takeaway. Synthetic data can be used to more accurately characterize model performance across a range of possible distributional shifts.
5.2.2 Incorporating Prior Knowledge on Shift
Goal. Consider the scenario where we have some knowledge of the shifted distribution and wish to estimate target domain performance. Specifically, here we assume we only have access to the feature marginals in the form of high-level info from the target domain, e.g. age (mean, std) or gender (proportions). We sample from this marginal and generate the other features con- ditionally (Sec. 4.2). Set-up.
Model performance rank
Target
35
Datases MLP
GRDT
RF
1
vi+pta
#2-treat
*3-stage
Tarpet
0.0
38
0.8
Acqurryy
ScKEr (R5)
features (Price knowlddge)
Model performance rank com- pared.
(b) Average accuracy increased prior knowledge
Figure 6: Incorporating prior knowledge of the shift. (a) D syn matches the performance rank of the true target domain, which can help to select the best model use in the target domain, and (b) D Deym better estimates target domain performance compared to baselines. Performance improves and gets closer as more prior knowledge is incorporated via added features. Points are connected to highlight trends.
We use datasets SEER (US) [66] and CUTRACT (UK) [67],
two real cancer datasets with
the same features, but with shifted distributions due to com-
ing from different countries. We train models f and G on the source domain (USA). We then wish to estimate likely model performance in the shifted target domain (UK). We assume access to information from n features in the target domain (features X.), sample X. from this marginal, conditionally generate X.. We estimate performance with Dayn-
We use the CUTRACT dataset (Target) as the ground truth to validate our estimate. As baselines. we use estimates on the source test set, along with Source Rejection Sampling (RS), which achieves a distributional shift through rejection sampling the source data using the observed target features. We also compare to baselines which assume access to more information than 3S, i.c. access to full unlabeled data from the target domain and hence have an advantage over 3S when predicting target domain performance. We benchmark ATC [31], DOC [32] and IM [33]. Details on all baselines are in Appendix B. Note, we repeat this experiment in Appendix for Covid- 19 data [58], where there is a shift between Brazil's north and south patients.
Analysis. In Fig. 6a, we show the model ranking of the dif- ferent predictive models based on performance estimates of the different methods. Using the synthetic data from 3S, we de- termine the same model rank- ing as the true ranking on the target-showcasing how 3S can be used for model selection with distributional shifts. On the
Table 2: 35 has lower performance estimate error in target domain for dif- ferent downstream models. Rows yellow have access to more information than 3S, in the form of unlabeled data from the target domain. is better
mean ada
bug
gbc
mlp
rf
knn
Ir
3S-Testing
0.023
0.051
0.012
0.030
0.009
0.015
0.020
0.029
(Source)
0.258
0.207
0.327
0.207
0.170
0.346
0.233
0.211
RS (Source)
0.180
0.028
0.298
0.096
0.014
0.373
0.213
0.094
ATC
0.249
0.253
0.288
0.162
0.140
0.214
0.369
0.165
IM [33] DOC[32]
0.215
0.206
0.278
0.156
0.126
0.268
0.131
0.163
0.201
0.207
0.211
0.162
0.116
0.223
0.148
0.161
other hand, baselines provide incorrect rankings.

---

Fig. 6b shows the average estimated performance of f as a function of the number of observed features. We see that the 3S estimates are closer to the oracle across the board compared baselines. Furthermore, for an increasing number of features (i.e. increasing prior knowledge), we observe that 3S estimates converge to the oracle. This is unsurprising: the more features we acquire target statistics of, the better we can model the true shifted distribution. Source RS does so too, but more slowly and with major variance issues.
We also assess raw estimate errors in Table 2. 3S clearly has lower performance estimate errors for the numerous downstream models. Beyond having reduced error compared to rejection sampling, is interesting that 3S generally outperforms highly specialized methods (ATC, IM, DOC), which not only have access to more information but are also developed specifically to predict target domain performance. A likely rationale for this is that these methods rely on probabilities and hence do not translate well to the non-neural methods widely used the tabular domain.
Takeaway: High-level information about potential shifts can be translated into realistic synthetic data, better estimate target domain model performance and select the best model use.
Discussion
Synthetic data for model evaluation. Accurate model evaluation is of vital importance to ML, but this is challenging when there is limited test data. We have shown in Sec. 3.1 that is hard to accurately evaluate performance for small subgroups (e.g, minority race groups) and to understand how models would perform under distributional shifts using real data alone. We have investigated the potential of synthetic data for model evaluation and found that 38 can accurately evaluate the performance a prediction model, even when the generative model is trained on the same test set. A deep generative ensemble approach can be used to quantify the uncertainty in 3S estimates, which have shown provides reliable coverage of the true model performance. Furthermore, we explored synthetic test sets with shifts, which provide practitioners with insight into how their model may perform in other populations or future scenarios.
Model reports. We envision evaluations using synthetic data could be published alongside models to give insight into when a model should and should not be used-c.g. to complete model evaluation templates such Model Cards for Model Reporting [10]. Appendix F illustrates example model report using 3S.
Practical considerations. We discuss and explore limitations in detail in Appendix D. Let us highlight three practical considerations to the application of synthetic data for testing. Firstly, evaluating the performance under distributional shifts requires assumptions on the shift. These assumptions affect model evaluation and require careful consideration from the end-user. This is especially true for large shifts scenarios where we do not have enough training data to describe the shifted distribution well enough. However, even if absolute estimates are inaccurate, we can still provide insight into trends of different scenarios. Secondly, synthetic data might have failure modes or limitations in certain settings, such as cases where there are only a handful of samples or with many samples. Thirdly, training and tuning generative model is non-trivial. 3S' DGE mechanism mitigates this by offering an uncertainty estimate that provides insight into the error of the generative learning process. The computational cost of such generative ensemble may be justified by the cost of untrustworthy model evaluation.
Acknowledgements
Boris van Bruegel is supported by ONR-UK. Nabeel Seedat is supported by the Cystic Fibrosis Trust. We would like to warmly thank the reviewers for their time and useful feedback. The authors also thank Alicia Curth, Alex Chan, Zhaozhi Qian, Daniel Jarrett, Tennison Liu and Andrew Rashbass for their comments and feedback on an earlier manuscript.
References
[1] Murtuza N Shergadwala, Himabindu Lakkaraju, and Krishnaram Kenthapadi. A human- centric perspective on model monitoring. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 10, pages 173-183, 2022.
10

---

[2] Luke Oakden-Rayner, Jared Dunnmon, Gustavo Carneiro, and Christopher Ré. Hidden strat- ification causes clinically meaningful failures in machine learning for medical imaging. In Proceedings of the ACM Conference on Health, Inference, and Learning, pages 151-159, 2020.
[3] Harini Suresh and John V Guttag. A framework for understanding unintended consequences of machine learning, arXiv preprint arXiv:1901. 2:8, 2019.
[4] Âneel Alexander Cabrera, Will Epperson, Fred Hohman, Minsuk Kahng, Jamie Morgenstern, and Duen Horng Chau. FairVis: Visual analyties for discovering intersectional bias in machine learning. In 2019 IEEE Conference on Visual Analytics Science and Technology (VAST), pages 46-56 IEEE, 2019.
[5] Angel Alexander Cabrera, Minsuk Kahng, Fred Hohman, Jamie Morgenstern, and Duen Horng Chau. Discovery of intersectional bias in machine learning using automatic subgroup generation. In ICLR Debugging Machine Learning Models Workshop, 2019.
[6] Oleg S Pianykh, Georg Langs, Mare Dewey, Dieter R Enzmann, Christian J Herold, Stefan O Schoenberg, and James Brink. Continuous learning AI radiology: Implementation principles and early applications. Radiology, 6-,,,, 2020.
[7] Joaquin Mainonero-Candedd Masashi Sugiyama, Anton Schwaighofer, and Neil Lawrence. Dataset shift in machine learning. MIT Press, 2008
[8] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienme David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A benchmark of wi-the-wild distribution shifts. In International Conference on Machine Learning, pages 5637-5664. PMLR, 2021.
[9] Nabeel Seedat, Fergus Imrie, and Mihaela van der Schaar. De-check: data-centric ai checklist guide the development of reliable machine learning systems. arXiv preprinl 4rX1v:2211.05764, 2022.
[10] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé lii, and Kate Crawford. Datasheets for datasets. Communications of the ACM, 64(1 (4(11):86-99 2021.
[11] Harini Suresh, Jen J Gong, and John V Guttag. Learning tasks for multitask learning: Heteroge- nous patient populations in the ICU, In Proceedings the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 802-810, 2018.
[12] Karan Goel, Albert Gu, Yixuan Li, and Christopher Re Model patching: Closing the sub- group performance gap with data augmentation. In International Conference on Learning Representations, 2020.
[13] Suchi Saria and Adarsh Subbaswamy. Tutorial: safe and reliable machine learning. arXiv preprint
2019.
[14] Alexander D' Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D Hoffman, et al. Under- specification presents challenges for credibility modern machine learning. arXiv preprint arXiv:2011.03395. 2020.
[15] Joseph Paul Cohen, Tianshi Cao, Joseph D Viviano, Chin- Huang, Michael Fralick, Marzych Ghassemi, Muhammad Mamdani, Russell Greiner, and Yoshua Bengio. Problems in the deployment of machine-learned models in health care. CMAJ, 193(55):E1391-E1334 2021.
[16] Rahul C Deo. Machine learning in medicine. Circulation, 1920-1919 2015.
[17] Neil Savage. Better medicine through machine learning. Communications of the ACM, 55(1):17-
19, 2012.
11

---

[18] Milad Zafar Nezhad. Dongxiao Zhu, Najibesadat Sadati, Kai Yang. and Phillip Levi. SUBIC: supervised bi-clustering upproach for precision medicine. In 2017 16th IEEE International Conference Machine Learning and Applications (ICMIA), pages 755-760. IEEE, 2017.
[19] Kayur Patel, James Fogarty, James A Landay, and Beverly Harrison. Investigating statistical machine learning as a tool for software development. Proceedings of the SIGCHI Conference on Human Factors in Compuling Systems, pages 667-676, 2008
[20] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? In International Conference on Machine Learning, pages PMLR, 2019.
[21] Andrew C Miller, Leon A Gatys, Joseph Futoma, and Emily Fox. Model-based metries: Sample- efficient estimates of predictive model subpopulation performance. In Machine Learning for Healthcare Conference, pages 308-336. PMLR, 2021.
[22] Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug. Martin Pawelezyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey. arXiv preprint arXiv:2110.01800 2021.
[23] Ravid Shwartz-Zir and Amitai Armon. Tabular data: Deep learning not all you need. Information Fusion, 81:84-90, 2022.
[24] Kaggle. Kaggle machine learning and data science survey, 2017.
[25] Boris van Breugel, Zhaozhi Qian, and Mihaela van der Schaar. Synthetic data, real errors: how (not) to publish and use synthetic data, In International Conference on Machine Learning. PMLR, 2023.
[26] Peter Flach. Performance evaluation in machine learning: the good, the bad, the ugly, and the way forward. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9808 9808-9814, 2019.
[27] Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre-Alvise Rebuffi, Ira Ktena, Krishna- murthy Dj Dvijotham, and Ali Taylan Cemgil. A fine-grained analysis on distribution shift. In International Conference on Learning Representations, 2021.
[28] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2018.
[29] Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy: Behavioral testing of nip models with checklist. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902-4912, 2020.
[30] Paul Röttger, Bertie Vidgen, Dong Nguyen, Zeerak Waseem, Helen Margetts, Janet Pierre- humbert, et al. Hatecheck: Functional tests for hate speech detection models. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguisties and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), page 41. Association for Computational Linguistics, 2021.
[31] Saurabh Garg, Sivaraman Balakrishnan, Zachary Chase Lipton, Behnam Neyshabur, and Hanie Sedghi. Leveraging unlabeled data to predict out-of-distribution performance. In International Conference on Learning Representations, 2022.
[32] Devin Guillory, Vaishaal Shankar, Sayna Ebrahimi, Trevor Darrell, and Ludwig Schmidt. Pre- dicting with confidence on unseen distributions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1144-1144, 2021.
[33] Mayee Chen, Karan Goel, Nimit S Sohoni, Fait Poms, Kayvon Fatahalian, and Christopher Ré. Mandoline: Model evaluation under distribution shift. In International Conference on Machine Learning, pages 1617-1629. PMLR, 2021.
[34] Boris van Breugel and Mihaela van der Schaar. Beyond privacy: Navigating the opportunities and challenges of synthetic data, arXiv preprint arXiv:2304.03722. 2023.
12

---

[35] James Jordon, Jinsung Yoon, and Mihacla Van Der Schaar. PATE-GAN:
Generating synthetic data with differential privacy guarantees. In International Conference on Learning Representa- lions; 2018.
[36] Samuel Assefa, Danial Dervovic. Mahmoud Mahfouz, Robert E Tillman, Prashant Reddy, and Manuela Veloso. Generating synthetic data in finance: opportunities, challenges and pitfalls. In Proceedings of the First ACM International Conference on Al Finance, pages 1-8, 2020.
[37] Depeng Xu, Yongkai Wu, Shuhan Yuan, Lu Zhang, and Xintao Wu. Achieving causal fairness through generative adversarial networks. Proceedings of the Twenty-Fightn International Joint Conference on Artificial Intelligence, 2019.
[38] Boris van Breugel, Trent Kyono, Jeroen Berrevoets, and Mihaela van der Schaar. DECAF: Generating fair synthetic data using causally-aware generative networks. Advances in Neural Information Processing Systems, 34:22221 -22233,,,,, 2021.
[39] Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data augmentation generative adversarial networks. arXiv preprint arXiv:
2017.
[40] Ayesha S Dina, AB Siddique, and D) Manivannan. Effect of halancing data using synthetic data on the performance of machine learning classifiers for intrusion detection in computer networks. arXiv preprint arXiv:2204.00144 2022.
[41] Hari Prasanna Das, Ryan Tran, Japjot Singh, Xiangyu Yue, Geoffrey Tison, Alberto Sangiovanni- Vincentelli, and Costas J Spanos. Conditional synthetic data generation for robust machine learning applications with limited pandemic data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11 11792-11 1122-1800, 1800, 2022.
[42] Simon Bing, Andrea Dittadi, Stefan Bauer, and Patrick Schwab. Conditional generation of medical time series for extrapolation to underrepresented populations. PLOS Digital Health, 1(7):1-26, 07 2022.
[43] Qi Wang, Junyu Gao, Wei Lin, and Yuan Yuan. Learning from synthetic data for crowd counting the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 8198-8207, 2019.
[44] Daniel Sáez Trigueros, Li Meng, and Margaret Hartnett. Generating photo- realistic training data to improve face recognition accuracy. Neural Networks, 134:86-94, 2021.
[45] Nataniel Ruiz, Adam Kortylewski, Weichao Qiu, Cihang Xie, Sarah Adel Bargal, Alan Yuille, and Stan Sclaroff. Simulated adversarial testing of face recognition models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4145-4155, 2022.
[46] Samin Khan, Buu Phan, Rick Salay, and Krzysztof Czarnecki. Procsy: Procedural synthetic dataset generation towards influence factor studies of semantic segmentation networks. In CVPR workshops, pages 88-96, 2019.
[47] Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni. Modeling tabular data using conditional GAN. Advances in Neural Information Processing Systems, 32, 2019.
[48] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems, 30, 2017.
[49] Kimberley Crenshaw. Demarginalizing the intersection of race and sex: A black feminist critique of antidiscrimination doctrine, feminist theory, and antiracist polities. the university of chicago legal forum, 1989 (1), 139-167. Chicago, IL, 1989.
[50] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety. arXiv preprint arXiv:1606.06550 2016.
13

---

[51] Ruihan Wu, Chuan Guo, Yi Su, and Kilian Q Weinberger. Online adaptation to label distribution shift. Advances Neural Information Processing Systems, 34:1 1340-11351, 2021.
[52] Christina X Ji, Ahmed M Alaa, and David Sontag. Large-scale study of temporal shift in health insurance claims. Conference on Health, Inference, and Learning (CHIL), 2023.
[53] Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log-liielihood function. Journal of Statistical Planning and Inference, 90(2):227- 2000.
[54] Jose G Moreno-Torres Troy Raeder, Rocío Alaiz-Rodríguez, Nitesh V Chawla, and Francisco Herrera. A unifying view on dataset shift in classification. Pattern Recognition, 45(1):521-555) 2012.
[55] Marco Saerens, Patrice Latinne, and Christine Decaestecker. Adjusting the outputs of a classifier new priori probabilities: simple procedure. Neural Computation. 14(1):21-41, 2002.
[56] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
[57] Arthur Asuncion and David Newman. UCI machine learning repository, 2007.
[58] Pedro Baqui, Ioana Bica, Valerio Marra, Ari Ercole, and Mihaela van Der Schaar. Ethnic and regional variations in hospital mortality from COVID-19 in Brazil: a cross-sectional observational study. The Lancet Global Health, ((8)e1018-e1026 2020.
(59] William A Knaus, Frank E Harrell, Joanne Lynn, Lee Goldman, Russell S Phillips, Alfred Connors, Neal V Dawson, William J Fulkerson, Robert M Califf, Norman Desbiens, Peter Layde, Robert K Oye, Paul E Bellamy, Rosemarie B Hakim, and Douglas P Wagner. The SUPPORT prognostic model: Objective estimates of survival for seriously ill hospitalized adults. Annals of Internal Medicine, 122(3):191-203 1995.
[60] Sérgio Jesus, José Pombal, Duarte Alves, André Cruz, Pedro Saleiro, Rita Ribeiro, João Gama, and Pedro Bizarro. Turning the tables: Biased, imbalanced, dynamic tabular datasets for ML. evaluation. Advances in Neural Information Processing Systems, 35:33563 335575, 2022.
[61] Elaine Fehrman, Awaz K Muhammad, Evgeny M Mirkes, Vincent Egan, and Alexander N Gorban. The five factor model of personality and evaluation of drug consumption risk. In Data Science: Innovative Developments in Data Analysis and Clustering, pages 231-242. Springer, 2017.
[62] Bradley Efron. Bootstrap methods: another look at the jackknife. Springer, 1992.
[63] Benjamin Kompa, Jasper Snoek, and Andrew L. Beam. Empirical frequentist coverage of deep learning uncertainty quantification procedures. Entropy, 23(12):1608, 2021.
[64] Nabeel Seedat, Jonathan Crabbé, and Mihacla van der Schaar. Data-SUITE: Data-centric identification of in-distribution incongruous examples. In International Conference on Machine Learning, pages 19467-19496. PMLR, 2022.
[65] Jiri Navratil, Matthew Arnold, and Benjamin Elder. Uncertainty prediction for deep sequential regression using meta models. arXiv preprint arXiv:2007.01350 2020.
[66] Máire A Duggan, William F Anderson, Sean Altekruse, Lynne Penberthy, and Mark E Sherman. The surveillance, epidemiology and end results (SEER) program and pathology: towards strengthening the critical relationship. The American Journal of Surgical Pathology, 40(12):994 2016.
[67] Prostate Cancer UK. Cutract. htpp::/ppostaaeccneeeukk.ogg 2019.
[68] Marco Tulio Ribeiro and Scott Lundberg. Adaptive testing and debugging of nlp models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3253-3267 2022.
14

---

[69] Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M Aroyo. "Everyone wants do the model work, not the data work": Data cascades in high-stakes ai. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1-15, 2021.
[70] Nabeel Seedat, Jonathan Crabbé, loana Bica, and Mihaela van der Schaar. Data-iq: Characteriz- ing subgroups with heterogeneous outcomes in tabular data, Advances in Neural Information Processing Systems, 35:23660-23674, 2022.
[71] Andrew Ng, Lora Aroyo, Cody Coleman, Greg Diamos, Vijay Janapa Reddi, Joaquin Van- schoren, Carole-Jean Wu, and Sharon Zhou. NeurIPS data-centric Al workshop, 2021.
[72] Neoklis Polyzotis and Matci Zaharia. What can data-centric AI learn from data and ML engineering? arXiv preprint
2021.
[73] Alex Chan, Ioana Bica, Alihan Hüyük, Daniel Jarrett, and Mihaela van der Schaar. The Medkit-Learn(nng) environment: Medical decision modelling through simulation. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). 2021.
[74] Brady Neal, Chin-Wei Huang, and Sunand Raghupathi. RealCause: Realistic causal inference benchmarking. arXiv preprint arXiv:2011.15007 2020.
[75] Susan Athey, Guido W. Imbens, Jonas Metzger, and Evan Munro. Using Wasserstein generative adversarial networks for the design of Monte Carlo simulations. Journal of Econometrics, 2021.
[76] Harsh Parikh, Carlos Varjao, Louise Xu, and Eric Tchetgen Tchetgen. Validating causal inference methods. In International Conference on Machine Learning, pages 17346-1777 17358. PMLR, 2022.
[77] Adam Kortylewski, Bernhard Egger, Andreas Schneider, Thomas Gerig, Andreas Morel- Forst-Forster, and Thomas Vetter. Analyzing and reducing the damage of dataset bias to face recognition with synthetic data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition Workshops, 2019.
[78] Zhiheng and Chenliang Xu. Discover the unknown biased attribute of an image classifier. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 14970- 14979, 2021.
[79] Daniel McDuff, Shuang Ma. Yale Song. and Ashish Kapoor. Characterizing bias in classifiers using generative models. Advances in Neural Information Processing Systems, 32, 2019.
[80] Bill Howe, Julia Stoyanovich, Haoyue Ping, Bernease Herman, and Matt Gee. Synthetic data for social good. arXiv preprint arXiv:1710.0880 1710.08874, 2017.
[81] Diptikalyan Saha, Aniya Aggarwal, and Sandeep Hans. Data synthesis for testing black-box machine learning models. In 5th Joint International Conference on Data Science & Management of Data (9th ACM IKDD CODS and 27th COMAD), pages 110-114, 2022.
[82] SIVEP Brazil Ministry of Health. Ministry of Health Evipe-Gripe public dataset.
[83] Bradley Efron. Estimating the enror rate of a prediction rule: improvement on cross-validation. Journal of the American Statistical Association, 78(38)):316333) 16-331, 1983.
[84] Thomas DiCiccio and Bradley Efron. Bootstrap confidence intervals. Statistical science,
1 11(3):189-228, 1996.
[85] James Carpenter and John Bithell. Bootstrap confidence intervals: when, which, what? practical guide for medical statisticians. Statistics in medicine, 19(9):1 141-1164, 2000.
[86] Ali Borji. Pros and cons of GAN evaluation measures. Computer Vision and Image Under- standing, 179:41-65, 2019.
15

---

[87] Dougal J Sutherland, Hsiao- Yu Tung, Heiko Strathmann, Soumyajit De, Aaditya Ramdas, Alexander J Smola, and Arthur Gretton. Generative models and model criticism via optimized maximum mean discrepancy. In International Conference Learning Representations, 2017.
[88] Wacha Bounliphone, Eugene Belilovsky, Matthew B Blaschko, loannis Antonoglou, and Arthur Gretton. A test of relative similarity for model selection in generative models. International Conference Learning Representations, 2016.
[89] Kush R Varshney. Trustworthy machine learning. Chappaqua, NY, 2021. Section 9.2.1.
[90] Ahmed Alaa, Boris Van Breugel. Evgeny Saveliev, and Mihacla van der Schaar. How faithful is your synthetic data? sample-level metrics for evaluating and auditing generative models, In International Conference Machine Learning, pages 290-306. PMLR, 2022.
16


Reviews:
[{"review_id": "Q0PMkAghg8", "reviewer": "Reviewer_DA3a", "summary": "This paper proposes a model evaluation framework, generating the synthetic test to mitigate the challenges of model evaluation with limited real test sets, such as unbalanced subgroups and distribution shifts.", "questions": "Please check the weakness.", "limitations": "Please check the weakness.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea of using synthetic data to improve the testing and evaluation of machine learning models is impressive. \n\nThis paper analyzes clearly and reasonably the failure of real test data and its corresponding challenge for reliable model evaluation.\n", "weaknesses": "**1. Counterintuition.** Although the empirical benefits of syntenic data are observed in the experimental parts, the inequation of Line 155 is not evaluated theoretically. More precise statements are needed here. \n\n**2. Overfitting of deep generative models.** With limited test data in the small subgroup, deep generative models tend to perform overfitting. In this case, the learned manifold could have a huge gap from the real manifold. \n\nIt could be better to show more visualizations in Figure 2. In detail, if we change the test samples in the green subgroup, what will happen when we compare synthetic manifolds and the real manifold?\n\n**3. Failure case.** As a model evaluation framework with synthetic data, which makes a lot of sense in the real world, this paper lacks failure cases to show the limitations of their works. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThank you for your time and feedback. We would like to address each of your comments in turn.\n\n## 1. Counterintuitive [line 155]\n\nWe do not want to give the impression that the equation in line 155 holds in general—though we hope the experiments are convincingly consistent and argue it very often will. To mitigate us seeming to make any theoretical claims here, we have changed line 155 to \"It may seem counterintuitive that $|A^*-A(f;D_{syn}, S)|$ **would ever be lower than** $|A^*-A(f;D_{test,f}, S)|$, ...” and we will link to an extended limitation section with failure cases—see point 3 below.\n\n## 2. Overfitting of deep generative models and learnt synthetic data manifold\n\nWe agree that overfitting in generative models is a serious problem, possibly leading to serious errors (e.g. in learnt manifold). In practice however, we have seen that the real manifold seems very well approximated. In the response pdf, Figure 1 we follow your suggestion and include a T-SNE of the real test data, an oracle set, and synthetic data. We observe that the synthetic data displays the same microstructures/manifold as the oracle set.\n\n*DGE*. To mitigate (and provide insight into) generative error, 3S uses a deep generative ensemble (DGE) [24] to model the uncertainty over the generative model parameters. Putting this in the context of Figure 2b: we agree that the learnt manifold may vary in practice, because there is a lack of evidence for preferring one manifold over another—e.g. a different member of the DGE would have converged to a slightly different manifold. By using DGE, we aim to implicitly model our uncertainty over the “correct” manifold. \n\n\n## 3. Failure cases\n\nWe agree that the paper would benefit from an extended limitation section and specific failure cases. We discuss these in the general response under “Limitations and Failure Cases”, as we think this discussion will be of interest to some of the other reviewers too."}, {"review_id": "YmMXLIH4D9", "reviewer": "Reviewer_fN4G", "summary": "In this paper, the authors propose the utilization of synthetic data for evaluating models and introduce an automated suite of synthetic data generators called 3S. 3S offers two key advantages: it enables reliable and detailed evaluation, and it measures model sensitivity to distributional shifts. The paper explores different scenarios involving 3S, providing valuable insights into the application of synthetic data for model evaluation.", "questions": "No questions.", "limitations": "As mentioned, further discussion on method limitation is needed. ", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper exhibits a clear and easily comprehensible writing style. The authors conduct a comprehensive examination of relevant literature, effectively summarizing its advantages.\n2. The issue investigated in this paper, namely the utilization of synthetic data for evaluating models, is both intriguing and significant.\n3. The authors effectively discuss various use cases of 3S, offering insights into the practical utilization of synthetic data for model evaluation.\n\n\n\n\n\n", "weaknesses": "1. Further discussion on method limitation is needed. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nMany thanks for your time and positive feedback. \n\nWe agree that the paper would benefit from an extended limitation section and specific failure cases. We discuss these in the general response under “Limitations and Failure Cases”, as we think this discussion will be of interest to some of the other reviewers too."}, {"review_id": "mN1RFAIoB4", "reviewer": "Reviewer_zuki", "summary": "This paper proposes to use synthetic test data to improve the estimation of model performance for tabular datasets when insufficient test data is available. Their approach of generating synthetic test data conditioned on subgroups improves performance estimation for underrepresented subgroups and can accurately estimate the model performance under distributional shift towards the underrepresented subgroups. ", "questions": "Q1. Extreme underrepresentation of the data in some regions, i.e., small subgroups, could affect the training of the generative model as well. At what point does your method fail, i.e., it is simply as good as Dtest strategy? Can you show this for different datasets studied in Figure4?\n\nQ2. How do you justify the 3S+ performing worse than 3S for the bank dataset especially for decision tree and random forest in Appendix Figure 10 and Figure 4?\n", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is well-written, and the details are covered by the Appendix. \n\nThey have done a good job in showcasing the advantage of their synthetic strategy.\n", "weaknesses": "Their proposed method is limited to the tabular data and data synthesis has been looked at in prior studies, but I think this is a good paper with nice experiments. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThank you for your time and feedback. We would like to address each of your comments in turn.\n\n## 1. Extreme underrepresentation failure case.\n\nWe agree that the paper would benefit from the inclusion of specific failure cases, including very small subgroups. We discuss this failure case in the general response under “Limitations and Failure Cases”, as we think this will be of interest to some of the other reviewers. In particular for small subgroups, we see that a practitioner can use the 3S uncertainty bounds to decide whether to trust an estimate. For extremely small subgroups the uncertainty bounds of 3S are usually larger and may urge a practitioner to gather additional data. \n\n## 2. Bank dataset 3S and 3S+ performance comparison.\n\nIndeed, in Figure 4, 3S+ (pink) has lower MAE vs the Oracle compared to 3S (green) for the small groups for the random forest classifier. In the Appendix, we see that real data ($D_{test}$) has a very large error when estimating the performance of the decision tree—only implying this particular real test data is ill-suited to probe this particular shallow decision tree. Because 3S+ consists of 3S and real test data, it partly inherits the poor performance from the $D_{test}$ estimate and thus leads to poorer estimates than 3S for this particular instance.\n"}, {"review_id": "f5NkaA5zdV", "reviewer": "Reviewer_vjS3", "summary": "The authors propose to use synthetic data to evaluate models, especially under distribution shifts or in areas of the input space with low coverage. The authors use CTGAN to empirically validate their idea, and apply it to tabular data.\n\nThe paper is a resubmission from ICLR 2023 ( https://openreview.net/forum?id=J7CTp-jNyJ ).", "questions": "Figure 4: It is unclear what the x-axis actually represents. It would be nice to get an explanation of what the Different Groups mean. The way it is presented, I don't understand what I am looking at.\n\nLine 180: \"similar in vain to Deep Ensembles\"  => similar in vein", "limitations": "The authors mention limiations in passing, but have not gone out of their way to find out failure modes of their method.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: While the general idea of using synthetic data for evaluation does not appear new, to the best of my knowledge this is the first thorough evaluation of the idea for tabular data. (I went out of my way to find published studies on this, and the only works I found were on relatively low-quality journals, or very different in scope).\n\nQuality: Experiments were done on 6 datasets of relatively small scale. This gives a first idea that the idea might work, but mention two possible extensions under \"Weaknesses\".\n\nClarity: the paper clarity was fine.\n\nSignificance: I think this is an important topic that deserves more study, the results of which will be of interest primarily to practitioners, but might also encourage further research on the topic.", "weaknesses": "* The paper mostly focuses on 6 small datasets. While the approach itself is probably interesting in cases where data is scarce, it would be interesting to see this applied to harder datasets that have more than just a couple of dozen features, or where there are millions of samples involved. I personally remain unconvinced the method scales to larger data, and would appreciate if the authors could report results on larger datasets.\n\n* The authors write \"* \"an end-user only has access to a single draw of Dtest,f . e.g., we might incorrectly overestimate 265 the true performance of minorities. The use of synthetic data solves this.\". I'd like to challenge that statement: I think it's very typical in real life (especially given the relatively small data sets that are the focus in the publication) to at least use cross validation to get multiple test sets, potentially even Leave-1-Out CV. This should give a much better estimate of the loss landscape. I'd appreciate if the authors could add such experiments as baselines.\n\n* It would be nice if the authors gave an indication of run times of the various methods they compare too, as this might be useful to practictioners. E.g. They mention in Appendix section C that the GAN was trained with a fairly large hyperparameter search, which I expect is not cheap.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThank you for your time and feedback. We would like to address each of your comments in turn.\n\n## 1. Large datasets and other limitations\n\nWe agree that the paper would benefit from the inclusion of specific failure cases, including very large datasets for which 3S may not be necessary. We discuss this failure case in the general response under “Limitations and Failure Cases”, as we think this discussion will be of interest to some of the other reviewers too. In a nutshell, we include a new experiment with the already-used Bank dataset (test set 75k+) and see that the benefit of 3S is determined primarily by the size of the subgroup we are trying to measure. If this subgroup is very small, we benefit from 3S; when subgroups get larger, real data evaluation is sufficient. We can use the bootstrap-estimated variance of the real dataset estimate to decide when to use 3S or not.\n\n\n\n## 2. Cross-validation — why its not applicable to our setting & alternatives\n\nWe are in the setting where we have a trained black box predictive model $f$, which could have been trained by someone else. For instance, the increasingly common scenario where a trained model is behind an API, yet we still wish to test the model’s capabilities. An example of this was during the pandemic when prognostic models were built in certain countries and were being tested by external parties who did not have access to the original medical training set for data sharing and patient privacy reasons. In this setting, we can only access the model and not the training dataset (see L112-113) — with the test dataset ($D_{test,f}$) our only available data. Of course, this is a challenging setting for 3S as well, since we train the generative model $G$ on $D_{test}$—as we have no access to a (potentially large) $D_{train}$. Since we do not have access to $D_{train}$, we cannot perform cross-validation to obtain multiple train-test splits. \nNonetheless, we agree with your suggestion and there are alternative ways to obtain multiple “test sets”. A common approach which would fit our problem setting and help get multiple test sets is **bootstrapping** — which allows us to obtain multiple datasets. We have included a bootstrapping baseline in the form of Model-Based Metrics (MBM) [20] — which does a computationally efficient version of nonparametric bootstrap. We outperform MBM both on performance vs Oracle in Figure 4 and coverage of the true value with intervals in Figure 5. To make this more clear to the reader we will update Sec 5.1.1., clarifying that MBM does bootstrapping and that our rationale for inclusion as a baseline is to be an alternative statistical mechanism for obtaining multiple test sets. \n\n\n## 3. Run times.\n\nWe agree that point made about showing run times, especially since we tune a generative model. Please refer to point 4 in the general response where we outline the computational cost. As a summary,  for typical tabular datasets, the computational requirements are often still very manageable — under 5 min for most datasets and under 40 min for the largest dataset Bank (see Response pdf). Of course, tuning will depend on dataset size and the numbers we report in the table in the response pdf are with respect to the dataset sizes in the main manuscript.\n\n## 4. Unclear subgroups Figure 4.\n\nWe regret that the groups in Figure 4 were unclear. The subgroups correspond to possible values one particular categorical feature can take.. This differs per dataset: for Adult, it is different Race groups; for Drug, Ethnicity groups; for Covid, the Region the patient comes from; for Support; race groups; for Bank, different Employment statuses. Since the specifics of each group are challenging to put in the figure without harming readability, we will add an Appendix that specifies the different groups for each dataset.\n"}, {"review_id": "inHMZKPpxz", "reviewer": "Reviewer_6cRw", "summary": "This paper contributes a new model evaluation framework called 3S-Testing, which uses (conditional) deep generative models to create synthetic test sets. To provide uncertainty estimations, 3S uses deep generative ensemble method. It is empirically confirmed that the better performance on small subgroups (with real data), and distributional shifted data. ", "questions": "$\\bullet$ How to estimate the biases of the generative models on the subgroup, besides providing uncertainty estimation? One possible case is mentioned in weakness, where the generative model generates new samples based on some its own bias.   \n\n$\\bullet$ Lines 195-198 is confusing to me. After G is trained, how exactly is it used? Steps (2) to (4) are not clear about this. I understand it is talked about later, but it is important to make it clear in the summary paragraph. \n\n$\\bullet$ Lines 207-208, Where does that assumption come from? \n\n$\\bullet$ Can the author(s) please briefly summarize the discussion (with minimum information/notation/assumptions needed) for lines 204-218? More specifically, does the final recipe proposed in lines 217 to 218 rely on the assumption “simplest such shift” (line 212)? \n", "limitations": "Fully discussed with explanation. ", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The motivation is clear by targeting the subdominant class. Using generative models (and generative ensembles) to evaluate model is an encouraging attempt, especially for uncertainty estimation. The experiment section is clearly organized and easily understood.", "weaknesses": "$\\bullet$ The major concern is to evaluate a model, more uncertified models are introduced even though they are used for uncertainty estimation. The author(s) gave reasoning from line 155 to 161. The first is reasonable, while the second is not convincing. For example, in figure 2, the samples of green star class are sparse in the test set by nature. Generating samples of green star class becomes (b) is a “bias” introduced by the generative model. \n\n$\\bullet$ 4.2 is hard to read. While the intention was to keep the discussion for shift as general, more and more notations and assumptions for simplifications are continuously added, sometimes without explanation (see below in questions).\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThank you for your time and constructive feedback. We wish to clarify each point in turn.\n\n## 1. Reasons for using generative models.\nThe paper only uses a deep generative ensemble (DGE) [24] to model the uncertainty over the generative model parameters and estimate generative errors. We acknowledge that estimating errors is hard, and this ensemble approach does not come with guarantees—to do so we would instead need to model the intractable posterior over the generative model parameters. Nonetheless, though the 3S estimate may or may not be unbiased, we believe our consistently better evaluation results give strong evidence it can provide a significantly better bias-variance trade-off than real data evaluation, and in turn we hope to have shown the value of synthetic data for evaluation purposes.\n\n**Figure 2b**. Putting this in the context of Figure 2b: we agree that just because the manifold seems plausible, this does not mean it is the true underlying distribution. A different member of the DGE would have converged to a slightly different manifold, caused by a lack of evidence for preferring one manifold over another. By using DGE, we aim to implicitly model our uncertainty over the “correct” manifold. \n\n**Generative models for implicit interpolation/weighting of real data**. At last, the paper’s second argument (L158) for using generative models for evaluation (for which you raise concerns) extends further than learning manifolds. For example, assume you want to estimate the expected performance of model $f$ for a specific person with features $x_c^*$ (a subset of all features). This is hard with real data alone: for continuous variables $X_c$, the probability that there exist points in $D_{test,f}$ with exactly $x_c^*$ is zero, hence $A(f; D_{test,f}, (x:x_c=x_c^*))$ is undefined. There may be many points in $D_{test,f}$ with $x_c$ close to $x^*_c$, but which metric do we use to define “close”, and given this metric, how do we weigh close points to compute $A$? This is non-trivial, because the target distribution may be very sensitive to some feature changes, while being independent of others (hence an Euclidean metric may not be good). Here too, we may expect that a generative model can actually do better. The generative model can “interpolate” the distribution over the whole space, and hence allows generating points with $x_c = x^*_c$. Of course, this may not be perfect, but again the DGE (where each member interpolates differently) gives insight into the uncertainty.\n\n## 2. Section 4.2 unclarities\n### L195-198\n**To ensure readers are not confused, we will include the simplified recipe first and move the general shift to the end of the section.**\n\n**General shift**. We can envision many types of shifts that we do not cover in the experiments, but which are also supported by the recipe on line 195 (e.g. concept drifts, see footnote 1 and [Varshney, 2021]). Regardless, we agree that the paper would be clearer if we are not too general when starting Section 4.2 and instead focus on our main setting of interest—i.e. where we make the assumptions of paragraph “Defining shifts” (specifically L204) and where the trained $G$ approximates $p(X_\\bar{c}|X_c)$. \n\n**How is $G$ used in practice**. In this case, the 3S evaluation recipe simplifies to: (i) train $G$ to approximate $p(X_\\bar{c}|X_c)$, (2) choose a shifted distribution $p(^\\sim X_c)$---e.g. a marginal mean shift of the original $p(X_c)$ (Section 5.2.1), or drawing $x_c$ samples from a secondary dataset (Section 5.2.2); (3) draw samples $x_c$, and subsequently *use $G$ to generate the rest of the variables $X_{\\bar{c}}$ conditional on these drawn samples*; and (4) evaluate downstream models.\n\n### L207-208\nWe use this assumption because it captures two of the most studied assumptions in distributional shift literature; $p(X|Y)$ is fixed and $p(Y)$ changes, or $p(X)$ changes and $p(Y|X)$ is fixed (see lines 201-203, and [Varshney, 2021; Section 9.2.1]). We have rewritten footnote 2 to make this link more explicit.\n\n### L204-218\nIn summary, we focus on shifts in which some variables’ distribution changes, but the other variables’ distribution conditional on these variables does not. From L211 we discuss the simplest such shift (where $X_c$ is a single variable), because it leads to easily-graspable insight for users. The final recipe (L217) does not depend on this simplest shift. Let us elaborate.\n\nIn paragraph L211 we focus on simple one-dimensional shifts (in a single variable $X_c=X_i$), because it is easier to define, visualise, and understand the effects of these shifts hence get meaningful insight into the model. For example, in 5.2.1 we shift the mean of $X_i$, use $G$ to generate data conditional on the shifted $X_i$, and study the effect on downstream model scores; if we had done this with two shifted variables instead, this would require generating and testing over a 2D set of shifted distributions—possible, but not as easy to understand.\n\nChoosing a shift in a multi-dimensional $X_c$’s distribution is not always hard however. For example, as shown in Section 5.2.2, if we observe some of the same features in another dataset, we can simply draw $X_c$ from this dataset and avoid having to define the shifted distribution explicitly. To do so, we assume the variables we observe ($X_c$) accurately describe the shift between our original domain and the new domain, such that the generated data will indeed resemble the target domain.\n\nIn any case, the recipe in L217 is independent of the shift distribution $p(^\\sim X_c)$. A reasonable constraint is that the shifted distribution “fails within” the old distribution (i.e. $p(X_c)$ dominates $p(^\\sim X_c)$). E.g., if $G$ generates medical data based on age and it has only ever seen patients up to 100 years old, it will yield unexpected behaviour when generating 110-year-old patients.\n\n## References\nVarshney KR. Trustworthy machine learning. 2021. Chappaqua, NY. p. 118."}]

Author rebuttal:
Dear reviewers,

Some reviewers suggested a longer limitations discussion, others were interested in specific failure cases. We address these points here. In the camera-ready paper we will extend the discussion to reflect these. 

## Limitations

We summarise 3S’s main limitations (where applicable referencing limitations already outlined in the paper).

1. **Subgroups in real test set are large**. When the aim is subgroup evaluation and there is sufficient real data for this subgroup, the real data estimate on $\mathcal{D}_{test,f}$ will be sufficiently accurate and there is no need to use 3S given the computational overhead. See *Failure Case 1* for new experiments and further discussion. Note, even for very large datasets, there can be very sparse regions or small subgroups for which using 3S is still beneficial. Also note that this limitation mostly applies to subgroup evaluation and less to Generating Synthetic Data with Shifts (Sec. 4.2), because performance estimates for possible shifts is less trivial using real data alone (e.g. require reweighting or resampling of test data) and we show (Sec. 5.2, Table 1 & Fig. 6) that 3S beats real data baselines consistently.

2. **Possible errors in the generative process** (L177, L402 & Fig. 7). Errors in the generative process can affect downstream evaluation. This is especially relevant for groups or regions with few samples, as it is more likely a generative model does not fit the distribution perfectly here. By replacing the generative model by a deep generative ensemble [24], 3S provides insight into its own generative uncertainty. When there is very little data and 3S’s uncertainty bounds are too large, practitioners should consider gathering additional data. See *Failure Case 2* below.

3. **Not enough knowledge or data to generate realistic shifts** (L398, Sec. 5.2.2). The success of modelling data with distributional shifts relies on the validity of the distributional shift’s assumptions. This is true for 3S as much as for supervised works on distributional shifts (e.g. see [Varshney, 2021; Sec. 9.2.1]). In Sec. 5.2.2 we show how a lack of shift knowledge may affect evaluation. We aimed to generate data to test a model on UK patients, using mostly data from US patients. We do not define the shift explicitly—we only assume that we observe a small number (1 to 4) of features $X_c$ (hence the conditional distribution of the rest of the features conditional on these features is the same across countries). In Fig. 6b, we show the assumption does not hold when we only see one feature—this lack of shift knowledge is a failure case. When we add more features, the assumption is more likely to approximately hold and 3S converges to a better estimate. We reiterate that invalid shift assumptions are a limitation of any distributional shift method, e.g. the rejection sampling (RS) baseline using only real data is worse than 3S overall.

4. **Computational cost** (L404). The computational cost & complexity of using generative models is always higher than using real test data directly. For typical tabular datasets, the computational requirements are often very manageable: under 5 min for most datasets and under 40 min for the largest dataset Bank (see Response pdf). The reported times correspond to the dataset sizes in the paper.  We would like to emphasise that the cost at deployment time for a poorly evaluated model can be unexpectedly high, which warrants 3S’s higher evaluation cost. 
Additionally, pre-implemented generative libraries (e.g. Patki 2016, Qian 2023) can accelerate the generative process, automating generator training with minimal user input.

## Failure cases

For limitations 1 & 2 mentioned above, we include two new experiments highlighting failure cases on two extreme settings:

### F1. Subgroups in real test set are large 
With sufficiently large real data, 3S provides no improvement despite higher complexity. We can determine sufficient size by estimating the variance Var$(A)$ of the performance metric $A(f;D,S)$ w.r.t. _the random variable_ denoting the test data $D$. With only access to one real dataset $D_{test}$ however, we can approximate $Var(A)$ via bootstrapping (Fig. 5). If Var$(A)$ falls below a small threshold $\alpha$ (e.g. 0.05), practitioners may for example decide to trust their real data estimate and not use 3S, as further evaluation improvements are unlikely. In our Bank dataset experiment (500k examples), we vary age & credit risk thresholds, retaining samples above each cut-off to shrink the test set (**see Fig. 2 - response pdf**). Note that for large datasets but very small subgroups, the $D_{test,f}$ estimate still has a high variance, (reflected in the large bootstrapped Var$(A)$ ), hence this should urge a practitioner to use 3S (see Fig. 2 response pdf).

### F2. Large uncertainty for very small test sets 
At the other extreme, when there are _too few_ samples the uncertainty of 3S (quantified through a DGE [24]) can become too large. We include a new experiment (**Fig. 3, response pdf**) in which we reduce subgroups to fewer than 10 samples in the test data. We train $G$ in 3S on the overall test set (which includes the small subgroup) with $n_{samples}$. Despite good performance versus an oracle, the uncertainty intervals from 3S's ensemble span 0.1-0.2.  These wide intervals make the 3S estimates unreliable and less useful, and would urge a practitioner to consider gathering additional data. The key takeaway: With extremely sparse subgroups, the large uncertainties signal that more data should be gathered before relying on 3S's uncertain estimates. 

## References
Patki, N., Wedge, R., & Veeramachaneni, K. (2016, October). The Synthetic Data Vault. IEEE.

Varshney KR. Trustworthy machine learning. 2021. p. 118.

Qian, Z., Cebere, B. C., & van der Schaar, M. (2023). Synthcity: facilitating innovative use cases of synthetic data in different data modalities. arXiv preprint arXiv:2301.07573.

Meta-review: The paper addresses the evaluation of generative models by the use of synthetic data. Specifically, it focuses on the problem of underrepresented groups and domain shift which are important topics. I believe these topics are important and having a proper evaluation of small subpopulations is important for fairness. The work also addresses some of the limitations of using a generative model by taking care to estimate the generative uncertainty. I believe this work is of value to the research community and should be accepted to this conference.
Decision: Accept (poster)