SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We now demonstrate how 3S satisfies (P1) Granular evaluation and (P2) Distributional shifts. We reiterate that the aim throughout is to estimate the true prediction performance of the model  $f$  as closely as possible. We tune and select the generative model based on Maximum Mean Discrepancy [56], see Appendix C. We describe the experimental details, baselines, and datasets for each experiment further in Appendix B<sup>5</sup>.
5.1 (P1) Granular Evaluation
5.1.1 Correctness of Subgroup Performance Estimates
Goal. This experiment assesses the value of synthetic data when evaluating model performance on minority subgroups. The challenge with small subgroups is that the conventional paradigm of using a hold-out evaluation set might result in high variance estimates due to the small sample size.
Datasets. We use the following five real-world medical and finance datasets: Adult [57], Covid-19 cases in Brazil [58], Support [59], Bank [60], and Drug [61]. These datasets have varying characteristics, from sample size to number of features. They also possess representational imbalance and biases, pertinent to 3S [4, 5]:  $①$  Minority subgroups: we evaluate the following groups which
<sup>4</sup>We consider any categorical variable with  $m$  classes using  $m$  different shifts of the individual probabilities, scaling the other probabilities appropriately.
$^{5}$ Code for use cases found at: https://github.com/seedatnabeel/3S-Testing or https://github.com/vanderschaarlab/3S-Testing
Figure 4: Assessing the reliability of performance estimates based on mean absolute error of the predicted performance estimate to the performance evaluated by oracle  $(\downarrow$  better). 3S better approximates true estimates compared to both baselines.  $3S+$  enjoys the best of both worlds by combining synthetic and real data. We evaluate an RF model, with other model classes shown in the Appendix.
differ in proportional representation - Adult: Race; Covid: Ethnicity; Bank: Employment; Support: Race; Drug: Ethnicity. ② Intersectional subgroups: we evaluate intersectional subgroups [49] (e.g. black males or young females)—see Appendix F, intersectional model performance matrix.
Set-up. We evaluate the estimates of subgroup performance for trained model  $f$  using different evaluation sets. We consider two baselines: (1)  $\mathcal{D}_{test,f}$ : a typical hold-out test dataset and (2) Model-based metrics (MBM) [21]. MBM uses a bootstrapping approach for obtaining multiple test sets. We compare the baselines to 3S testing datasets, which generate data to balance the subgroup samples: (i) 3S ( $\mathcal{D}_{syn}$ ): synthetic data generated by  $G$ , which is trained on  $\mathcal{D}_{test,f}$  and (ii) 3S+ ( $\mathcal{D}_{syn} \cup \mathcal{D}_{test,f}$ ): test data augmented with the synthetic dataset.
For some subgroup  $S$ , each test set gives an estimated model performance  $A(f; \mathcal{D}_{,S})$ , which we compare to a pseudo-oracle performance  $A(f; \mathcal{D}_{oracle}, S)$ : the oracle is the performance of  $f$  evaluated on a large unseen real dataset  $\mathcal{D}_{oracle} \sim p(X, Y)$ , where  $|\mathcal{D}_{oracle}| \gg |\mathcal{D}_{test,f}|$ . As outlined above the subgroups are as follows: (i) Adult: Race, (ii) Drug: Ethnicity, (iii) Covid: Ethnicity (Region), (iv) Support: Race, (v) Bank: Employment status.
We evaluate the reliability of the different performance estimates based on their Mean Absolute Error (MAE) relative to the Oracle predictive accuracy estimates. We desire low MAE such that our estimates match the oracle.
Analysis. Fig. 4 illustrates across the 5 datasets that the 3S synthetic data (red, green) closely matches estimates on the Oracle data. i.e. lower MAE vs baselines. In particular, for small subgroups (e.g. racial minorities), 3S provides a more accurate evaluation of model performance (i.e. with estimates closer to the oracle) compared to a conventional hold-out dataset  $(\mathcal{D}_{test,f})$  and MBM.
In addition, 3S estimates have reduced standard deviation. Thus, despite 3S using the same (randomly drawn test set)  $\mathcal{D}_{test,f}$  to train its generator, its estimates are more robust to this randomness. The results highlight an evaluation pitfall of the standard hold-out test set paradigm: the estimate's high variance w.r.t. the drawn  $\mathcal{D}_{test,f}$  could lead to potentially misleading conclusions about model performance in the wild, since an end-user only has access to a single draw of  $\mathcal{D}_{test,f}$ . e.g., we might incorrectly overestimate the true performance of minorities. The use of synthetic data solves this.
Next, we move beyond single-feature minority subgroups and show that synthetic data can also be used to evaluate performance on intersectional groups — subgroups with even smaller sample sizes due to the intersection. 3S performance estimates on 2-feature intersections are shown in Appendix F. Intersectional performance matrices provide model developers more granular insight into where they can improve their model most, as well as inform users how a model may perform on intersections of groups (especially important to evaluate sensitive intersectional subgroups).<sup>6</sup> Appendix F further illustrates how these intersectional performance matrices can be used as part of model reports.
We evaluate the intersectional performance estimates of 3S and baseline  $\mathcal{D}_{test,f}$  using the MAE of the performance matrices w.r.t. the oracle, averaged across 3 models (i.e, RF, GBDT, MLP). The error of 3S  $(11.90 \pm 0.19)$  is significantly lower than  $\mathcal{D}_{test,f}$ $(20.29 \pm 0.14)$ , hence demonstrating 3S provides more reliable intersectional estimates.
$^{6}$ N.B. low-performance estimates by 3S only indicate poor model performance; this does not necessarily imply that the data itself is biased for these subgroups. However, it could warrant investigating potential data bias and how to improve the model.
Takeaway. Synthetic data provides more accurate performance estimates on small subgroups compared to evaluation on a standard test set. This is especially relevant from a representational bias and fairness perspective—allowing more accurate performance estimates on minority subgroups.
5.1.2 Reliability through Confidence Intervals
Goal. In Fig. 4, we see that all methods, including 3S, have errors in some cases, which warrants the desire to have confidence intervals at test time. 3S uses a deep generative ensemble to provide uncertainty estimates at test-time—see Sec. 4.1.
Set-up. We assess coverage over 20 random splits and seeds for 3S vs the baselines (B1) bootstrapping [62] confidence intervals for  $\mathcal{D}_{test,f}$  and (B2) MBM: which itself uses bootstrapping for the predictive distribution. For 3S, we assess a Deep Generative Ensemble with  $K = 1,5,10$  randomly initialized generative models. For each method, we take the average estimate  $\pm 2$  standard deviations. We evaluate the intervals based on the following two metrics defined in [63-65]: (i) Coverage  $= \mathbb{E}\left[1_{x_i\in [l_i,r_i]}\right]$  (ii)  $\mathrm{Width} = \mathbb{E}[|r_i - l_i|]$ . Coverage measures how often the true label is in the prediction region, while width measures how specific
Figure 5: 3S is more reliable than baselines with higher coverage, lower widths — with less variability.
that prediction region is. In the ideal case, we have high coverage with low width. See Appendix B for more details.
Analysis. Fig. 5 shows the mean test set coverage and width averaged over the five datasets. 3S (with  $\mathrm{K} = 5$  and  $\mathrm{K} = 10$ ) is more reliable, attaining higher coverage rates with lower width compared to baselines. In addition, the variability with 3S is much lower for both coverage and width. We note that this comes at a price: computational cost scales linearly with  $K$ . For fair comparison, we set  $K = 1$  in the rest of the paper.
Takeaway. 3S includes uncertainty estimates at test time that cover the true value much better than baselines, allowing practitioners to decide when (not) to trust 3S performance estimates.
5.2 (P2) Sensitivity to Distributional Shifts
ML models deployed in the wild often encounter data distributions differing from the training set. We simulate distributional shifts to evaluate model performance under different potential post-deployment conditions. We examine two setups with varying knowledge of the potential shift.
5.2.1 No Prior Information
Goal. Assume we have no prior information for the (future) model deployment environment. In this case, we might still wish to stress test the sensitivity for different potential operating conditions, such that a practitioner understands model behavior under different conditions, which can guide us to when the model can and cannot be used. We wish to simulate distribution shifts using synthetic data and assess if it captures true performance.
Set-up. We consider shifts in the marginal of some feature  $X_{i}$ , keeping  $p(\hat{X}_{\neg i}|\hat{X}_i)$  fixed (see Sec. 3). For instance, a shift in the marginal  $X_{i}$ 's mean (see Sec. 4.2). To assess performance for different degrees of shift, we compute three shift buckets around the mean of the original feature distribution: large negative shift from the mean (-), small negative/positive shift from the mean (±), and large positive shift from the mean (+).
We define each in terms of the feature quantiles. We generate uniformly distributed shifts (between min feature) and max (feature)). Any shift that shifts the mean to less than Q1 is  $(-)$ , any shift that shifts the mean to more than Q3 is  $(+)$  and any shift in between is  $(\pm)$ .
As before, we compare estimated accuracy w.r.t. a pseudo-oracle test set. We compare two baselines: (i) Mean-shift (MS) and (ii) Rejection sampling (RS); both applied to the real data.
Analysis. Table 1 shows the potential utility of synthetic data to more accurately estimate performance for unknown distribution shifts compared to real data alone. This is seen both with an average lower mean error of estimates, but also across all three buckets. This implies that the synthetic data is able to closely capture the true performance across the range of feature shifts.
Table 1: Mean error in estimated accuracy across shift quantile buckets for all 5 datasets. The results show 3S indeed provides more reliable estimates even for distribution shift.  $\downarrow$  is better.
Adult	Support	Bank	Drug	SEER
Mean	-	±	+	Mean	-	±	+	Mean	-	±	+	Mean	-	±	+	Mean	-	±	+
3S	2.6	2.2	1.8	3.9	2.0	2.6	2.0	1.1	5.4	3.7	3.6	6.7	5.6	5.7	4.4	7.8	2.7	5.5	3.0	2.0
MS	5.9	5.2	5.6	6.9	19.3	22.9	18.5	15.2	18.6	18.6	19.9	17.9	18.5	19.6	19.0	16.3	3.3	2.6	3.9	3.2
RS	15.9	10.5	17.9	18.6	25.1	27.8	24.2	22.3	18.9	18.9	20.1	18.3	20.1	21.7	21.0	18.2	20.0	20.3	23.6	18.6
Takeaway. Synthetic data can be used to more accurately characterize model performance across a range of possible distributional shifts.
5.2.2 Incorporating Prior Knowledge on Shift
Goal. Consider the scenario where we have some knowledge of the shifted distribution and wish to estimate target domain performance. Specifically, here we assume we only have access to the feature marginals in the form of high-level info from the target domain, e.g. age (mean, std) or gender (proportions). We sample from this marginal and generate the other features conditionally (Sec. 4.2). Set-up.
We use datasets SEER (US) [66] and CUTRACT (UK) [67], two real cancer datasets with the same features, but with shifted distributions due to com
(a) Model performance rank compared.
(b) Average accuracy vs increased prior knowledge
Figure 6: Incorporating prior knowledge of the shift. (a)  $\mathcal{D}_{syn}$  matches the performance rank of the true target domain, which can help to select the best model to use in the target domain, and (b)  $\mathcal{D}_{syn}$  better estimates target domain performance compared to baselines. Performance improves and gets closer as more prior knowledge is incorporated via added features. Points are connected to highlight trends.
ing from different countries. We train models  $f$  and  $G$  on the source domain (USA). We then wish to estimate likely model performance in the shifted target domain (UK). We assume access to information from  $n$  features in the target domain (features  $X_{c}$ ), sample  $X_{c}$  from this marginal, conditionally generate  $X_{\bar{c}}|X_{c}$ . We estimate performance with  $\mathcal{D}_{syn}$ .
We use the CUTRACT dataset (Target) as the ground truth to validate our estimate. As baselines, we use estimates on the source test set, along with Source Rejection Sampling (RS), which achieves a distributional shift through rejection sampling the source data using the observed target features. We also compare to baselines which assume access to more information than 3S, i.e. access to full unlabeled data from the target domain and hence have an advantage over 3S when predicting target domain performance. We benchmark ATC [31], DOC [32] and IM [33]. Details on all baselines are in Appendix B. Note, we repeat this experiment in Appendix E for Covid-19 data [58], where there is a shift between Brazil's north and south patients.
Analysis. In Fig. 6a, we show the model ranking of the different predictive models based on performance estimates of the different methods. Using the synthetic data from 3S, we determine the same model ranking as the true ranking on the target—showcasing how 3S can be used for model selection with distributional shifts. On the
Table 2: 3S has lower performance estimate error in target domain for different downstream models. Rows yellow have access to more information than 3S, in the form of unlabeled data from the target domain.  $\downarrow$  is better
mean	ada	bag	gbc	mlp	rf	knn	lr
3S-Testing	0.023	0.051	0.012	0.030	0.009	0.015	0.020	0.029
All (Source)	0.258	0.207	0.327	0.207	0.170	0.346	0.233	0.211
RS (Source)	0.180	0.028	0.298	0.096	0.014	0.373	0.213	0.094
ATC [31]	0.249	0.253	0.288	0.162	0.140	0.214	0.369	0.165
IM [33]	0.215	0.206	0.278	0.156	0.126	0.268	0.131	0.163
DOC[32]	0.201	0.207	0.211	0.162	0.116	0.223	0.148	0.161
other hand, baselines provide incorrect rankings.
Fig. 6b shows the average estimated performance of  $f$  as a function of the number of observed features. We see that the 3S estimates are closer to the oracle across the board compared to baselines. Furthermore, for an increasing number of features (i.e. increasing prior knowledge), we observe that 3S estimates converge to the oracle. This is unsurprising: the more features we acquire target statistics of, the better we can model the true shifted distribution. Source RS does so too, but more slowly and with major variance issues.
We also assess raw estimate errors in Table 2. 3S clearly has lower performance estimate errors for the numerous downstream models. Beyond having reduced error compared to rejection sampling, it is interesting that 3S generally outperforms highly specialized methods (ATC, IM, DOC), which not only have access to more information but are also developed specifically to predict target domain performance. A likely rationale for this is that these methods rely on probabilities and hence do not translate well to the non-neural methods widely used in the tabular domain.
Takeaway: High-level information about potential shifts can be translated into realistic synthetic data, to better estimate target domain model performance and select the best model to use.
Synthetic data for model evaluation. Accurate model evaluation is of vital importance to ML, but this is challenging when there is limited test data. We have shown in Sec. 3.1 that it is hard to accurately evaluate performance for small subgroups (e.g. minority race groups) and to understand how models would perform under distributional shifts using real data alone. We have investigated the potential of synthetic data for model evaluation and found that 3S can accurately evaluate the performance of a prediction model, even when the generative model is trained on the same test set. A deep generative ensemble approach can be used to quantify the uncertainty in 3S estimates, which we have shown provides reliable coverage of the true model performance. Furthermore, we explored synthetic test sets with shifts, which provide practitioners with insight into how their model may perform in other populations or future scenarios.
Model reports. We envision evaluations using synthetic data could be published alongside models to give insight into when a model should and should not be used—e.g. to complete model evaluation templates such as Model Cards for Model Reporting [10]. Appendix F illustrates an example model report using 3S.
Practical considerations. We discuss and explore limitations in detail in Appendix D. Let us highlight three practical considerations to the application of synthetic data for testing. Firstly, evaluating the performance under distributional shifts requires assumptions on the shift. These assumptions affect model evaluation and require careful consideration from the end-user. This is especially true for large shifts or scenarios where we do not have enough training data to describe the shifted distribution well enough. However, even if absolute estimates are inaccurate, we can still provide insight into trends of different scenarios. Secondly, synthetic data might have failure modes or limitations in certain settings, such as cases where there are only a handful of samples or with many samples. Thirdly, training and tuning a generative model is non-trivial. 3S' DGE mechanism mitigates this by offering an uncertainty estimate that provides insight into the error of the generative learning process. The computational cost of such a generative ensemble may be justified by the cost of untrustworthy model evaluation.
Boris van Bruegel is supported by ONR-UK. Nabeel Seedat is supported by the Cystic Fibrosis Trust. We would like to warmly thank the reviewers for their time and useful feedback. The authors also thank Alicia Curth, Alex Chan, Zhaozhi Qian, Daniel Jarrett, Tennison Liu and Andrew Rashbass for their comments and feedback on an earlier manuscript.
[1] Murtuza N Shergadwala, Himabindu Lakkaraju, and Krishnamaram Kenthapadi. A human-centric perspective on model monitoring. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 10, pages 173-183, 2022.
[87] Dougal J Sutherland, Hsiao-Yu Tung, Heiko Strathmann, Soumyajit De, Aaditya Ramdas, Alexander J Smola, and Arthur Gretton. Generative models and model criticism via optimized maximum mean discrepancy. In International Conference on Learning Representations, 2017.
[88] Wacha Bounliphone, Eugene Belilovsky, Matthew B Blaschko, Ioannis Antonoglou, and Arthur Gretton. A test of relative similarity for model selection in generative models. In International Conference on Learning Representations, 2016.
[89] Kush R Varshney. Trustworthy machine learning. Chappaqua, NY, 2021. Section 9.2.1.
[90] Ahmed Alaa, Boris Van Breugel, Evgeny S Saveliev, and Mihaela van der Schaar. How faithful is your synthetic data? sample-level metrics for evaluating and auditing generative models. In International Conference on Machine Learning, pages 290–306. PMLR, 2022.