Title: Post-processing Private Synthetic Data for Improving Utility on Selected Measures
Venue: NeurIPS 2023
Accepted: yes
Keywords: differential privacy

synthetic data
Abstract: Existing private synthetic data generation algorithms are agnostic to downstream tasks. However, end users may have specific requirements that the synthetic data must satisfy. Failure to meet these requirements could significantly reduce the utility of the data for downstream use. We introduce a post-processing technique that improves the utility of the synthetic data with respect to measures selected by the end user, while preserving strong privacy guarantees and dataset quality. Our technique involves resampling from the synthetic data to filter out samples that do not meet the selected utility measures, using an efficient stochastic first-order algorithm to find optimal resampling weights. Through comprehensive numerical experiments, we demonstrate that our approach consistently improves the utility of synthetic data across multiple benchmark datasets and state-of-the-art synthetic data generation algorithms.

Post-processing Private Synthetic Data for Improving Utility on Selected Measures

Abstract

Existing private synthetic data generation algorithms are agnostic to downstream tasks. However, end users may have specific requirements that the synthetic data must satisfy. Failure to meet these requirements could significantly reduce the utility of the data for downstream use. We introduce a post-processing technique that improves the utility of the synthetic data with respect to measures selected by the end user, while preserving strong privacy guarantees and dataset quality. Our technique involves resampling from the synthetic data to filter out samples that do not meet the selected utility measures, using an efficient stochastic first-order algorithm to find optimal resampling weights. Through comprehensive numerical experiments, we demonstrate that our approach consistently improves the utility of synthetic data across multiple benchmark datasets and state-of-the-art synthetic data generation algorithms.

1 Introduction

The advancement of machine learning (ML) techniques relies on large amounts of training data. However, data collection also poses a significant risk of exposing private information. In recent years, several instances of privacy breaches have surfaced [NS06, Con18], making it urgent to find a reliable way to share data. Today, the de facto standard for privacy protection is differential privacy (DP) [DR14]. DP ensures that useful information of a private dataset can be released while simultaneously preventing adversaries from identifying individuals' personal data. DP has been utilized effectively in a wide variety of settings, by actors including the US Census [Abo18] and various large corporations [App17, Fac20, RE19, HBMAL19, IBM23].

Generating private synthetic data is a crucial application of DP. It allows data scientists to train their ML models on the synthetic data while preserving a certain level of utility when deploying these models on real test data. The U.S. National Institute of Standards and Technology (NIST) has emphasized its significance by hosting a series of competitions [RTMT21, MMS21]. There is also significant work introducing new DP synthetic data generation mechanisms, including GAN-based [XLW + 18, BJWW + 19, JYVDS19, TWB + 19], marginal-based [ZCP + 17, MSM19, MMS21], and workload-based [VTB + 20, ABK + 21, LVW21, MMSM22, VAA + 22] methods. Existing methods for generating private synthetic data are task-agnostic-they do not take into account the downstream use cases of the synthetic data in the data generation process. However, end users often have specific requirements for synthetic datasets to be successfully analyzed by their existing data science pipelines, which are often well-established, well-understood, heavily vetted, and difficult or expensive to change. Unfortunately, synthetic data generated by existing methods may not always meet these requirements, resulting in reduced utility for their downstream use cases. This raises a fundamental question:

Is it possible to improve a synthetic dataset's utility on a set of selected measures while preserving its differential privacy guarantees?

In this paper, we introduce a post-processing procedure that enhances the quality of the synthetic data on a set of utility measures. These measures are usually based on the end user's requirements for the synthetic data. Our method solves an optimization problem to calculate the optimal resampling weights for each data point. We then use these weights to resample the synthetic dataset, eliminating data points that do not accurately reflect the real data based on the selected utility measures. This postprocessing procedure has many remarkable properties. Firstly, it is model-agnostic, meaning that it does not depend on any assumptions about how the synthetic data were generated. The synthetic data could be generated using either GAN-based, marginal-based, or workload-based methods. Second, the post-processed data preserve the DP guarantees while maintaining the performance of downstream models on real data compared to models trained on the original synthetic data. Finally, our algorithm is highly scalable, especially for high-dimensional data. It only requires a convex optimization whose number of variables is equal to the number of desired utility measures. We develop a stochastic first-order method that can efficiently solve this optimization problem. As a reference, we apply our algorithm to the home-credit dataset [MOK18] which consists of 307,511 data points and 104 features, and it only takes around 4 mins to finish running.

We conduct comprehensive numerical experiments to demonstrate the effectiveness of our postprocessing techniques. Our results show that this algorithm consistently improves the utility of the synthetic data on different selected measures, across multiple datasets, and for synthetic data generated using various state-of-the-art privacy mechanisms. Additionally, the downstream models trained on the post-processed data can achieve comparable (or even better) performance on real data compared to the models trained on the original synthetic data. We demonstrate an application of our technique by using it to align the correlation matrix of the synthetic data with that of the real data. The correlation matrix shows the correlation coefficients between pairs of features and is often used as a reference for feature selection. By applying our post-processing technique, we can ensure that the correlation matrix of the synthetic data closely matches that of the real data. This, for instance, can help downstream data scientists more accurately select a task-appropriate set of features for fitting their ML models, e.g. by thresholding the correlation with the desired target variable or using Lasso techniques.

Our proof techniques rely on an information-theoretic formulation known as information projection, which dates back to Csiszár's seminal work [Csi75]. This formulation has recently been successfully applied in several domains, such as large deviation theory [DZ96], hypothesis testing [Csi84], and fair machine learning [AHJ + 22]. Here we use information projection to 'project' the empirical distribution of the synthetic data onto a set of distributions whose utility measures match those of the real data. We show that the optimal projected distribution can be expressed as an exponential tilting of the original distribution and determine the tilting parameters via a convex program in the dual space. To solve this optimization problem, we introduce a compositional proximal gradient algorithm, which is a stochastic first-order method and highly scalable for large dataset. Notably, this exponential tilting takes the form of a simple reweighting of the data points in the original synthetic dataset. Hence, our approach has the added benefit that it only reweights synthetic data points without changing their values, ensuring that the quality of the data points is not compromised and the support set of the overall dataset is not expanded.

In summary, our main contributions are:

We present a post-processing technique that enables end users to customize synthetic data according to their specific requirements.

Our technique is model-agnostic-it is applicable regardless of the methods used to generate private synthetic data. It is scalable to high-dimensional data and maintains strong privacy guarantees.

We conduct comprehensive numerical experiments, showing that our technique can consistently improve the utility of synthetic data across multiple benchmark datasets and for synthetic data generated using state-of-the-art DP mechanisms.

The supplementary material of this paper includes: (i) omitted proofs of all theoretical results, (ii) details on algorithm implementation, and (iii) supporting experimental results.

1.1 Related Work

DP synthetic data generation mechanisms. Generating synthetic data using DP mechanisms is an active area of research [see e.g., HLM12, BLR13, GAH + 14, CXZX15, BSG17, AZK + 19, UV20, GMHI20, TMH + 21, VAA + 22, BSV22]. Along this line of work, the closest ones to ours are workload-aware methods [VTB + 20, ABK + 21, MMSM22, LVW21], which aim to ensure that the synthetic data perform well on a collection of queries. Our work differs in three key features. First, all existing approaches focus on generating private synthetic data from scratch, while we aim to post-process synthetic data to downweight samples that do not accurately represent the real data under the specified utility measures. As a result, our approach is more efficient as it does not need to fit a graphical model [MMSM22] or a neural network [LVW21] to represent the data distribution, and it is more extendable, allowing a single synthetic dataset to be quickly post-processed multiple times for different sets of utility measures as needed by downstream users. Second, some existing work may not scale well to large datasets as they either require solving an integer program multiple times [VTB + 20] or need to solve a large-scale optimization problem [ABK + 21]. In contrast, our approach is highly scalable, as it only requires solving a convex program whose number of variables is equal to the number of specified utility measures. Third, existing work often evaluates the quality of the synthetic data by how well it preserves key statistics (e.g., 3-way marginals) of the real data. In contrast, in our experiments, we evaluate our approach on the more stringent and realistic test of training various downstream ML models on the synthetic data and measuring their performance on real test data. The experimental results demonstrate that our approach can enhance the utility of synthetic data on selected measures without compromising their downstream quality.

Public data assisted methods/Post-processing methods. Our work introduces a post-processing procedure for improving the utility of a given synthetic dataset based on selected measures. As a special case, our approach can be applied to post-process publicly available datasets. In this regard, this work is related to public-data-assisted methods [see e.g., LVS + 21, LVW21], which leverage public data for saving privacy budgets. We extend Algorithm 1 in [LVS + 21] by formalizing the preservation of (noisy) utility measures as a constrained optimization problem, rather than minimizing the corresponding Lagrangian function. We further establish strong duality and propose a stochastic first-order method for solving this constrained optimization efficiently. We extend Algorithm 4 in [LVW21] by allowing any non-negative violation tolerance (i.e., any γ ≥ 0 in (3b)) compared with γ = 0 in [LVW21]. This extension offers increased flexibility, as users can now select various values of γ to navigate the trade-off between minimizing the distortion of the synthetic data and enhancing their utility on selected measures. Moreover, our experiments show that setting γ to be a small positive number (e.g., γ = 1 e -5 ) consistently outperforms when γ = 0 . Finally, our work is related with [NWD20], which proposes to post-process outputs from differentially private GAN-based models for improving the quality of the generated synthetic data. However, their method is tailored to GAN-based privacy mechanisms while our approach is model-agnostic. This versatility is crucial, given that marginal-based and workload-based mechanisms often yield higher quality synthetic tabular data, as evidenced by benchmark experiments in [TMH + 21]. Our experiments indicate that our method consistently improves the utility of synthetic data produced by all kinds of privacy mechanisms, even when the initial synthetic data are of high quality.

2 Preliminaries and Problem Formulation

In this section, we review differential privacy and provide an overview of our problem setup.

2.1 Differential Privacy

We first recall the definition of differential privacy (DP) [DR14].

Definition 1. A randomized mechanism M : X n →R satisfies ( ϵ, δ ) -differential privacy, if for any adjacent datasets D and D ′ , which only differ in one individual's record, and all possible outputs from the mechanism O ⊆ R , we have

DP has two important properties: post-processing immunity and composition rule. Specifically, if M : X n →R satisfies ( ϵ, δ ) -DP and g : R→R ′ is any randomized (or deterministic) function, then

g ◦ M : X n →R ′ is ( ϵ, δ ) -DP; if M = ( M 1 , · · · , M k ) is a sequence of randomized mechanisms, where M i is ( ϵ i , δ i ) -DP, then M is ( ∑ k i =1 ϵ i , ∑ k i =1 δ i ) -DP. There are also advanced composition theorems [see e.g., DRV10, KOV15].

The Laplace and Gaussian mechanisms are two fundamental DP mechanisms. For a function f : X n → R K , we define its L p sensitivity for p ∈ { 1 , 2 } as

where D and D ′ are adjacent datasets. The Laplace mechanism adds Laplace noise N ∼ Lap ( 0 , b I ) to the output of f :

In particular, if b = ∆ 1 ( f ) /ϵ , the Laplace mechanism satisfies ( ϵ, 0) -DP. Similarly, the Gaussian mechanism adds Gaussian noise N ∼ N ( 0 , σ 2 I ) to the output of f . We recall a result from [BW18] that establishes a DP guarantee for the Gaussian mechanism and refer the reader to their Algorithm 1 for a way to compute σ from ( ϵ, δ ) and ∆ 2 ( f ) .

Lemma 1 ([BW18]) . For any ϵ ≥ 0 and δ ∈ [0 , 1] , the Gaussian mechanism is ( ϵ, δ ) -DP if and only if

where Φ is the Gaussian CDF: Φ( t ) ≜ 1 √ 2 π ∫ t -∞ e -y 2 / 2 d y .

The function f can often be determined by a query q : X → R : f ( D ) = 1 n ∑ x i ∈D q ( x i ) . In this case, we denote ∆( q ) ≜ sup x,x ′ ∈X | q ( x ) -q ( x ′ ) | ; q ( P ) ≜ E X ∼ P [ q ( X )] for a probability distribution P ; and q ( D ) ≜ 1 n ∑ x i ∈D q ( x i ) for a dataset D .

2.2 Problem Formulation

Consider a dataset D comprising private data from n individuals, where each individual contributes a single record x = ( x 1 , · · · , x d ) ∈ X . Suppose that a synthetic dataset D syn has been generated from the real dataset D , but it may not accurately reflect the real data on a set of utility measures (also known as workload). We express the utility measures of interest as a collection of queries, denoted by Q ≜ { q k : X → R } K k =1 , and upper bound their L p sensitivity by ∆ p ( Q ) ≜ 1 n ( ∑ k ∈ [ K ] ∆( q k ) p ) 1 /p .

Our goal is to enhance the utility of the synthetic dataset D syn w.r.t. the utility measures Q while maintaining the DP guarantees. We do not make any assumption about how these synthetic data have been generated-e.g., they could be generated by either GAN-based, marginal-based, or workload-based methods. We evaluate the utility measures from the real data via a ( ϵ post , δ post ) -DP mechanism and denote their answers by a = ( a 1 , · · · , a K ) . Next, we present a systematic approach to post-process the synthetic data, aligning their utility with the (noisy) answers a .

3 Post-processing Private Synthetic Data

In this section, we present our main result: a post-processing procedure for improving the utility of a synthetic dataset. We formulate this problem as an optimization problem that projects the probability distribution of the synthetic data onto a set of distributions that align the utility measures with the (noisy) real data. We prove that the optimal projected distribution can be expressed in a closed form, which is an exponentially tilted distribution of the original synthetic data distribution. To compute the tilting parameters, we introduce a stochastic first-order method and leverage the tilting parameters to resample from the synthetic data. By doing so, we filter out synthetic data that inaccurately represent the real data on the selected set of utility measures.

We use P syn and P post to represent the (empirical) probability distribution of the synthetic data and the generating distribution of post-processed data, respectively. We obtain the post-processed distribution

by 'projecting' P syn onto the set of distributions that align with the real data utility measures up to γ ≥ 0 :

Here the 'distance' between two probability distributions is measured using the KL-divergence. Solving this problem directly suffers from the curse of dimensionality, as the number of variables grows exponentially with the number of features. Below, we leverage strong duality and prove that the optimal distribution P ∗ post can be achieved by tilting P syn and the tilting parameters can be obtained from a convex program. This approach involves fewer variables, making it more efficient and practical.

Theorem 1. If (3) is feasible, then its optimal solution has a closed-form expression:

where the dual variables λ ∗ ≜ ( λ ∗ 1 , · · · , λ ∗ K ) are the optimal solution of the following optimization:

Theorem 1 only requires the feasibility of (3). To ensure feasibility, we denoise the (noisy) answers a computed from real data. This denoising step could also increase the accuracy of the released answers while preserving DP guarantees due to the post-processing property [BW18, LWK15]. Specifically, suppose that D syn has S unique values. We denote the collection of these unique values by supp ( D syn ) = { x 1 , · · · , x S } . We construct a matrix Q ∈ R K × S such that Q j,s = q j ( x s ) and solve the following linear or quadratic programs (see Appendix B for more details):

Here ∆ + S = { p ∈ R S | 1 T p = 1 , p > 0 } is the interior of the probability simplex in R S . We obtain the denoised answers as a ∗ = Qp ∗ = ( a ∗ 1 , · · · , a ∗ K ) . Finally, if we solve (3) with a ∗ instead, then it is guaranteed to be feasible.

Although (5) is a convex program, its form differs from the standard empirical risk minimization as the expected loss is composed with a logarithmic function. Thus, one cannot directly apply stochastic gradient descent (SGD) to solve this problem as computing an unbiased (sub)-gradient estimator for (5) is challenging. Instead, we apply a stochastic compositional proximal gradient algorithm to solve (5). Note that the partial derivative of f in (5) has a closed-form expression:

We estimate this partial derivative by using a mini-batch of data to calculate the expectations in numerator and denominator separately, and then computing their ratio. With this estimator, we can apply the stochastic proximal gradient algorithm [PB14] to solve (5). Although this approach uses a biased gradient estimator, it has favorable convergence guarantees [see e.g., WLF16, HZCH20]. We provide more details about the updating rule in Algorithm 1.

After obtaining the optimal dual variables from Algorithm 1, we generate the post-processed dataset by resampling from the synthetic data with weights: exp( -∑ K k =1 λ ∗ k ( q k ( x ) -a k )) for each point x . This ensures that the resampled data follow the optimal distribution P ∗ post in (4). As our approach only reweights synthetic data without altering their values, the quality of the data points is preserved and the support set of the overall dataset is not expanded. Moreover, our numerical experiments

Algorithm 1 Dual variables computation.

Input: D = { ¯ q i } n i =1 where ¯ q i = ( q 1 ( x syn i ) -a 1 , · · · , q K ( x syn i ) -a K ) ; maximal number of iterations T ; mini-batch index B t ⊆ [ n ] ; step size α t for

t

= 1

,

· · ·

, T

do

Dual variables update:

end for Output: λ ( T +1)

Function: prox α ∥·∥ 1 ( x ) = sign ( x ) max {| x | -α, 0 }

demonstrate that downstream predictive models trained on the post-processed synthetic data can achieve comparable or even better performance on real data than those trained on the original synthetic data.

Finally, we establish a DP guarantee for the post-processing data. Since our algorithm only accesses the real data when computing the utility measures, the composition rule yields the following proposition.

Proposition 1. Suppose the synthetic data are generated from any ( ϵ, δ ) -DP mechanism and the utility measures are evaluated from the real data from any ( ϵ post , δ post ) -DP mechanism. Then the post-processed data satisfy ( ϵ + ϵ post , δ + δ post ) -DP.

So far, we have introduced a general framework for post-processing synthetic data to improve their utility on selected measures. Next, we present a concrete use case of our framework: aligning the correlation matrix. This can be achieved by choosing a specific set of utility measures in our framework.

Aligning the correlation matrix. The correlation matrix comprises the correlation coefficients between pairs of features and is commonly used by data scientists to select features before fitting their predictive models. Aligning the correlation matrix of synthetic data with real data can assist data scientists in identifying the most suitable set of features and developing a more accurate model. Recall the definition of the correlation coefficient between features X i and X j for i, j ∈ [ d ] :

Since the correlation coefficient only depends on the first and second order moments, we can select the following set of ( d +3) d/ 2 queries as our utility measures:

Assume that each feature x i is normalized and takes values within [0 , 1] . As a result, the L 1 and L 2 sensitivity of Q can be upper bounded:

We can compute these utility measures using the Laplace or Gaussian mechanism, denoise the answers, compute the optimal dual variables, and resample from the synthetic data.

4 Numerical Experiments

We now present numerical experiments to demonstrate the effectiveness of our post-processing algorithm. Our results show that this algorithm consistently improves the utility of the synthetic

data across multiple datasets and state-of-the-art privacy mechanisms. Additionally, it achieves this without degrading the performance of downstream models or statistical metrics. We provide additional experimental results and implementation details in Appendix C.

Benchmark. We evaluate our algorithm on four benchmark datasets: Adult , Bank , Mushroom , and Shopping , which are from the UCI machine learning repository [DG17]. All the datasets have a target variable that can be used for a downstream prediction task. We use four existing DP mechanismsAIM [MMSM22], MST [MMS21], DPCTGAN [RLP + 20], and PATECTGAN [RLP + 20]to generate private synthetic data. Among these DP mechanisms, AIM is a workload-based method; MST is a marginal-based method; DPCTGAN and PATECTGAN are GAN-based methods. All of their implementations are from the OpenDP library [Sma23]. To demonstrate the scalability of our approach, we conduct an additional experiment on the home-credit dataset [MOK18]. This dataset has 307,511 data points and 104 features. We apply GEM [LVW21] to this high-dimensional dataset for generating synthetic data. We pre-process each dataset to convert categorical columns into numerical features and standardize all features to the range [0 , 1] .

We clarify that our objective is not to compare the quality of synthetic data generated by these DP mechanisms (for a benchmark evaluation, please refer to [TMH + 21]), but to demonstrate how our post-processing method can consistently improve the utility of synthetic data generated from these different DP mechanisms.

Setup. We compare synthetic data produced by different privacy mechanisms with ϵ ∈ { 2 , 4 } against the post-processed synthetic data that are generated by applying our post-processing technique with ϵ post = 1 to synthetic data generated from privacy mechanisms with ϵ ∈ { 1 , 3 } . For UCI datasets, we select 5 features, including the target variable, from the synthetic data that have the highest absolute correlation with the target variable. These features are chosen as they have a higher influence on downstream prediction tasks. The set of these features is denoted by S ⊆ [ d ] . Next, we define the utility measures as the first-order and second-order moment queries among the features in S : Q = { q i ( x ) = x i } i ∈S ∪ { q i,j ( x ) = x i x j } i,j ∈S . For home-credit dataset, we follow the same setup but choosing 10 features.

We apply the Gaussian mechanism with ( ϵ post = 1 , δ post = 1 /n 2 ) to estimate utility measures from the real data, where n denotes the number of real data points. Finally, we apply Algorithm 1 with γ = 1 e -5 , a batch size of 256 for UCI datasets and 4096 for home-credit , and 200 epochs to compute the optimal resampling weights, which are then used to resample from the synthetic data with the same sample size. We repeat our experiment 5 times to obtain an error bar for each evaluation measure.

Evaluation measure. We provide evaluation metrics for assessing the quality of synthetic data after post-processing. Our first metric, utility improvement , is defined as:

The range of this metric is ( -∞ , 1] , where a value of 1 indicates a perfect alignment between the correlation matrix of the post-processed data and the real data.

To assess the downstream quality of synthetic data, we train predictive models using the synthetic data and then test their performance on real data. Specifically, we split the real data, using 80% for generating synthetic data and setting aside 20% to evaluate the performance of predictive models. We use the synthetic data to train a logistic regression classifier to predict the target variable based on other features. Then we apply this classifier to real data and calculate its F1 score. We use the function BinaryLogisticRegression from SDMetrics library [Dat23] to implement this process.

We compute two statistical measures to quantify the similarity between synthetic and real data distributions. The first measure is the average Jensen-Shannon distance between the marginal distributions of synthetic and real data, which ranges from 0 (identical distribution) to 1 (totally different distributions). The second measure is the average inverse KL-divergence between the marginal distributions of synthetic and real data, which ranges from 0 (totally different distributions) to 1 (identical distribution). These statistical measures are implemented by using synthcity library [QCvdS23].

Result. We present the experimental results on the UCI datasets in Table 1 (and Table 3 in Appendix C with a higher privacy budget). We report utility improvement , F1 score , and statistical measures . As shown, our post-processing technique consistently enhances the utility of synthetic data on selected metrics across all benchmark datasets and all privacy mechanisms. Moreover, the utility improvements are achieved without degrading the performance of downstream models or statistical properties of synthetic data. In other words, our post-processing procedure ensures that the logistic regression classifier trained on synthetic data can achieve comparable or even higher performance on real test data, while simultaneously reducing or maintaining the statistical divergences between synthetic and real data.

We present the experimental results on the home-credit dataset in Table 2 and illustrate the misalignment of the correlation matrix with and without applying our post-processing procedure in Figure 1. The results demonstrate that our algorithm consistently reduces the overall correlation misalignment. Additionally, our procedure, which includes computing the utility measures from real data, denoising the noisy answers, and computing optimal resampling weights, only takes around 4 mins on 1x NVIDIA GeForce RTX 3090 GPU.

Without post-proc

With post-proc

0.00

0.00

FLAG_DOCUMENT_3

0.000.01

0.000.01

0.060

DAYS_BIRTH

0.000.050.03

0.000.000.01

REG_CITY_NOT_LIVE_CITY

0.000.07

0.020.01

0.000.010.000.01

0.045

LIVE_CITY_NOT_WORK_CITY

0.000.020.00 0.000.02

0.000.000.000.000.00

FLAG_EMP_PHONE

0.000.010.000.02

0.000.01

0.030

REGION_RATING_CLIENT_W_CITY

0.000.020.010.010.000.010.01

000000000000000100000

REGION_RATING_CLIENT

0.000.030.010.020.01 0.01 0.01 0.01

0.000.020.000.000.000.010.000.00

0.015

REG_CITY_NOT_WORK_CITY

0.000.010.000.000.010.000.030.010.01

0.000.000.010.000.000.010.010.000.00

TARGET

0.00

0.02

0.000.000.010.020.000.03

0.000.01

0.000.000.000.000.000.000.000.010.000.01

TARGET

TARGET

DAYS_BIRTH

0.000

5 Conclusion and Limitations

The use of private synthetic data allows for data sharing without compromising individuals' private information. In practice, end users typically have specific requirements that the synthetic data must meet, which are often derived from their standard data science pipeline. Failure to meet these requirements can diminish the usefulness of the synthetic data. To address this issue, we introduce a theoretically-grounded post-processing procedure for improving a synthetic dataset's utility on a selected set of measures. This procedure filters out samples that do not accurately reflect the real data, while still maintaining DP guarantees. We also present a scalable algorithm for computing resampling weights and demonstrate its effectiveness through extensive experiments.

We believe there is a crucial need for future research to comprehend the capabilities and limitations of private synthetic data from various perspectives [see e.g., JSH + 22, GODC22, BDI + 23, for further discussions]. It should be noted that synthetic data generation is not intended to replace real data, and any model trained on synthetic data must be thoroughly evaluated before deployment on real data. In cases a ML model trained on synthetic data exhibits algorithmic bias upon deployment on real data, it becomes imperative to identify the source of bias and repair the model by querying the real data with a privacy mechanism.

When generating synthetic data using DP mechanisms, outliers may receive higher noise, which can have a greater impact on minority groups. Consequently, ML models trained on DP-synthetic data by downstream users may demonstrate a disparate impact when deployed on real data [GODC22]. This is true even when using fairness-intervention algorithms [HCZ + 22] during model training because the synthetic data follows a different probability distribution than real data. One can potentially apply our post-processing techniques to resample synthetic data and eliminate samples that do not reflect the underlying distribution of the population groups, while ensuring sample diversity. By doing so, they can achieve a higher fairness-accuracy curve when deploying downstream ML models trained on the post-processed synthetic data on real data.

Another promising avenue of research is to investigate the data structure in cases where a user or a company can provide multiple records to the dataset. Financial data, for instance, is typically collected over time and presented as a time series. While it is possible to combine all of a user's data into a single row, this approach can lead to an unmanageably unbounded feature domain and limit the effectiveness of the synthetic data. Therefore, it is worthwhile to explore other privacy notions [see e.g., ZXLZ14, DWCS19, LSA + 21] and develop strategies for optimizing higher privacy-utility trade-offs under special data structures.

References

[ABK + 21] Sergul Aydore, William Brown, Michael Kearns, Krishnaram Kenthapadi, Luca Melis, Aaron Roth, and Ankit A Siva. Differentially private query release through adaptive projection. In International Conference on Machine Learning , pages 457-467. PMLR, 2021.

[Abo18] John M Abowd. The US census bureau adopts differential privacy. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 2867-2867, 2018.

[AHJ + 22] Wael Alghamdi, Hsiang Hsu, Haewon Jeong, Hao Wang, Peter Michalak, Shahab Asoodeh, and Flavio Calmon. Beyond adult and compas: Fair multi-class prediction via information projection. Advances in Neural Information Processing Systems , 35:38747-38760, 2022.

[App17] Apple. Learning with privacy at scale. https://machinelearning.apple.com/ research/learning-with-privacy-at-scale , 2017.

[AZK + 19] Nazmiye Ceren Abay, Yan Zhou, Murat Kantarcioglu, Bhavani Thuraisingham, and Latanya Sweeney. Privacy preserving synthetic data release using deep learning. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2018, Dublin, Ireland, September 10-14, 2018, Proceedings, Part I 18 , pages 510-526. Springer, 2019.

[BDI + 23] Brian Belgodere, Pierre Dognin, Adam Ivankay, Igor Melnyk, Youssef Mroueh, Aleksandra Mojsilovic, Jiri Navartil, Apoorva Nitsure, Inkit Padhi, Mattia Rigotti, Jerret Ross, Yair Schiff, Radhika Vedpathak, and Richard Young. Auditing and generating synthetic data with controllable trust trade-offs. arXiv preprint arXiv:2304.10819 , 2023.

[BJWW + 19] Brett K Beaulieu-Jones, Zhiwei Steven Wu, Chris Williams, Ran Lee, Sanjeev P Bhavnani, James Brian Byrd, and Casey S Greene. Privacy-preserving generative deep neural networks support clinical data sharing. Circulation: Cardiovascular Quality and Outcomes , 12(7):e005122, 2019.

[BLR13] Avrim Blum, Katrina Ligett, and Aaron Roth. A learning theory approach to noninteractive database privacy. Journal of the ACM (JACM) , 60(2):1-25, 2013.

[BSG17] Vincent Bindschaedler, Reza Shokri, and Carl A Gunter. Plausible deniability for privacy-preserving data synthesis. arXiv preprint arXiv:1708.07975 , 2017.

[BSV22] March Boedihardjo, Thomas Strohmer, and Roman Vershynin. Privacy of synthetic data: A statistical framework. IEEE Transactions on Information Theory , 69(1):520527, 2022.

[BV04] Stephen P Boyd and Lieven Vandenberghe. Convex optimization . Cambridge university press, 2004.

[BW18] Borja Balle and Yu-Xiang Wang. Improving the gaussian mechanism for differential privacy: Analytical calibration and optimal denoising. In International Conference on Machine Learning , pages 394-403. PMLR, 2018.

[Con18] Nicholas Confessore. Cambridge Analytica and Facebook: The scandal and the fallout so far. The New York Times , 2018.

[Csi75] Imre Csiszár. I-divergence geometry of probability distributions and minimization problems. The Annals of Probability , pages 146-158, 1975.

[Csi84] Imre Csiszár. Sanov property, generalized I-projection and a conditional limit theorem. The Annals of Probability , pages 768-793, 1984.

[CXZX15] Rui Chen, Qian Xiao, Yu Zhang, and Jianliang Xu. Differentially private highdimensional data publication via sampling-based inference. In ACM SIGKDD international conference on knowledge discovery and data mining , pages 129-138, 2015.

[Dat23] DataCebo. Synthetic data metrics. https://docs.sdv.dev/sdmetrics/ , 2023.

[DG17] Dheeru Dua and Casey Graff. UCI machine learning repository. http://archive. ics.uci.edu/ml , 2017.

[DR14] Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science , 9(3-4):211-407, 2014.

[DRV10] Cynthia Dwork, Guy N Rothblum, and Salil Vadhan. Boosting and differential privacy. In 2010 IEEE 51st Annual Symposium on Foundations of Computer Science , pages 51-60, 2010.

[DWCS19] Mario Diaz, Hao Wang, Flavio P Calmon, and Lalitha Sankar. On the robustness of information-theoretic privacy measures and mechanisms. IEEE Transactions on Information Theory , 66(4):1949-1978, 2019.

[DZ96] Amir Dembo and Ofer Zeitouni. Refinements of the Gibbs conditioning principle. Probability Theory and Related Fields , 104:1-14, 1996.

[Fac20] Facebook AI. Introducing Opacus: A high-speed library for training PyTorch models with differential privacy. https://ai.facebook.com/blog/ introducing-opacus-a-high-speed-library-for-training-pytorch-\ models-with-differential-privacy/ , 2020.

[GAH + 14] Marco Gaboardi, Emilio Jesús Gallego Arias, Justin Hsu, Aaron Roth, and Zhiwei Steven Wu. Dual query: Practical private query release for high dimensional data. In International Conference on Machine Learning , pages 1170-1178. PMLR, 2014.

[GMHI20] Chang Ge, Shubhankar Mohapatra, Xi He, and Ihab F Ilyas. Kamino: Constraint-aware differentially private data synthesis. arXiv preprint arXiv:2012.15713 , 2020.

[GODC22] Georgi Ganev, Bristena Oprisanu, and Emiliano De Cristofaro. Robin Hood and Matthew effects: Differential privacy has disparate impact on synthetic data. In International Conference on Machine Learning , pages 6944-6959. PMLR, 2022.

[HBMAL19] Naoise Holohan, Stefano Braghin, Pól Mac Aonghusa, and Killian Levacher. Diffprivlib: the IBM differential privacy library. arXiv preprint arXiv:1907.02444 , 2019.

[HCZ + 22] Max Hort, Zhenpeng Chen, Jie M Zhang, Federica Sarro, and Mark Harman. Bias mitigation for machine learning classifiers: A comprehensive survey. arXiv preprint arXiv:2207.07068 , 2022.

[HLM12] Moritz Hardt, Katrina Ligett, and Frank McSherry. A simple and practical algorithm for differentially private data release. Advances in neural information processing systems , 25, 2012.

[HZCH20] Yifan Hu, Siqi Zhang, Xin Chen, and Niao He. Biased stochastic first-order methods for conditional stochastic optimization and applications in meta learning. Advances in Neural Information Processing Systems , 33:2759-2770, 2020.

[IBM23] IBM Research. Synderella. https://research.ibm.com/projects/ synderella , 2023.

[JSH + 22] James Jordon, Lukasz Szpruch, Florimond Houssiau, Mirko Bottarelli, Giovanni Cherubin, Carsten Maple, Samuel N Cohen, and Adrian Weller. Synthetic data-what, why and how? arXiv preprint arXiv:2205.03257 , 2022.

[JYVDS19] James Jordon, Jinsung Yoon, and Mihaela Van Der Schaar. PATE-GAN: Generating synthetic data with differential privacy guarantees. In International conference on learning representations , 2019.

[KOV15] Peter Kairouz, Sewoong Oh, and Pramod Viswanath. The composition theorem for differential privacy. In International Conference on Machine Learning , pages 1376-1385. PMLR, 2015.

[LSA + 21] Daniel Levy, Ziteng Sun, Kareem Amin, Satyen Kale, Alex Kulesza, Mehryar Mohri, and Ananda Theertha Suresh. Learning with user-level privacy. Advances in Neural Information Processing Systems , 34:12466-12479, 2021.

[LVS + 21] Terrance Liu, Giuseppe Vietri, Thomas Steinke, Jonathan Ullman, and Steven Wu. Leveraging public data for practical private query release. In International Conference on Machine Learning , pages 6968-6977. PMLR, 2021.

[LVW21] Terrance Liu, Giuseppe Vietri, and Steven Z Wu. Iterative methods for private synthetic data: Unifying framework and new methods. Advances in Neural Information Processing Systems , 34:690-702, 2021.

[LWK15] Jaewoo Lee, Yue Wang, and Daniel Kifer. Maximum likelihood postprocessing for differential privacy under consistency constraints. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 635-644, 2015.

[MMS21] Ryan McKenna, Gerome Miklau, and Daniel Sheldon. Winning the NIST contest: A scalable and general approach to differentially private synthetic data. arXiv preprint arXiv:2108.04978 , 2021.

[MMSM22] Ryan McKenna, Brett Mullins, Daniel Sheldon, and Gerome Miklau. Aim: An adaptive and iterative mechanism for differentially private synthetic data. arXiv preprint arXiv:2201.12677 , 2022.

[MOK18] Anna Montoya, Kirill Odintsov, and Martin Kotek. Home credit default risk. https: //kaggle.com/competitions/home-credit-default-risk , 2018.

[MSM19] Ryan McKenna, Daniel Sheldon, and Gerome Miklau. Graphical-model based estimation and inference for differential privacy. In International Conference on Machine Learning , pages 4435-4444. PMLR, 2019.

[NS06] Arvind Narayanan and Vitaly Shmatikov. How to break anonymity of the Netflix prize dataset. arXiv preprint cs/0610105 , 2006.

[NWD20] Marcel Neunhoeffer, Zhiwei Steven Wu, and Cynthia Dwork. Private post-gan boosting. arXiv preprint arXiv:2007.11934 , 2020.

[PB14] Neal Parikh and Stephen Boyd. Proximal algorithms. Foundations and trends® in Optimization , 1(3):127-239, 2014.

[QCvdS23] Zhaozhi Qian, Bogdan-Constantin Cebere, and Mihaela van der Schaar. Synthcity: facilitating innovative use cases of synthetic data in different data modalities. https: //arxiv.org/abs/2301.07573 , 2023.

[RE19] Carey Radebaugh and Ulfar Erlingsson. Introducing TensorFlow privacy: Learning with differential privacy for training data. https://blog.tensorflow.org/2019/ 03/introducing-tensorflow-privacy-learning.html , 2019.

[RLP + 20] Lucas Rosenblatt, Xiaoyan Liu, Samira Pouyanfar, Eduardo de Leon, Anuj Desai, and Joshua Allen. Differentially private synthetic data: Applied evaluations and enhancements. arXiv preprint arXiv:2011.05537 , 2020.

[RTMT21] Diane Ridgeway, Mary F Theofanos, Terese W Manley, and Christine Task. Challenge design and lessons learned from the 2018 differential privacy challenges. NIST Technical Note 2151 , 2021.

[Sma23] SmartNoise. Smartnoise sdk: Tools for differential privacy on tabular data. https: //github.com/opendp/smartnoise-sdk , 2023.

[TMH + 21] Yuchao Tao, Ryan McKenna, Michael Hay, Ashwin Machanavajjhala, and Gerome Miklau. Benchmarking differentially private synthetic data generation algorithms. arXiv preprint arXiv:2112.09238 , 2021.

[TWB + 19] Uthaipon Tantipongpipat, Chris Waites, Digvijay Boob, Amaresh Ankit Siva, and Rachel Cummings. Differentially private mixed-type data generation for unsupervised learning. arXiv preprint arXiv:1912.03250 , 1:13, 2019.

[UV20] Jonathan Ullman and Salil Vadhan. PCPs and the hardness of generating synthetic data. Journal of Cryptology , 33(4):2078-2112, 2020.

[VAA + 22] Giuseppe Vietri, Cedric Archambeau, Sergul Aydore, William Brown, Michael Kearns, Aaron Roth, Ankit Siva, Shuai Tang, and Steven Z Wu. Private synthetic data for multitask learning and marginal queries. Advances in Neural Information Processing Systems , 35:18282-18295, 2022.

[VTB + 20] Giuseppe Vietri, Grace Tian, Mark Bun, Thomas Steinke, and Steven Wu. New oracleefficient algorithms for private synthetic data release. In International Conference on Machine Learning , pages 9765-9774. PMLR, 2020.

[WLF16] Mengdi Wang, Ji Liu, and Ethan Fang. Accelerating stochastic composition optimization. Advances in Neural Information Processing Systems , 29, 2016.

[XLW + 18] Liyang Xie, Kaixiang Lin, Shu Wang, Fei Wang, and Jiayu Zhou. Differentially private generative adversarial network. arXiv preprint arXiv:1802.06739 , 2018.

[ZCP + 17] Jun Zhang, Graham Cormode, Cecilia M Procopiuc, Divesh Srivastava, and Xiaokui Xiao. PrivBayes: Private data release via Bayesian networks. ACM Transactions on Database Systems (TODS) , 42(4):1-41, 2017.

[ZXLZ14] Tianqing Zhu, Ping Xiong, Gang Li, and Wanlei Zhou. Correlated differential privacy: Hiding information in non-IID data set. IEEE Transactions on Information Forensics and Security , 10(2):229-242, 2014.

A Omitted Proofs

A.1 Proof of Theorem 1

Recall that Slater's condition is a sufficient condition for ensuring strong duality in the context of convex program [see Section 5.2.3 in BV04]: it states that strong duality holds for a convex program if there exists a strictly feasible solution (i.e., a solution that satisfies all constraints and ensures that nonlinear constraints are satisfied with strict inequalities).

Proof. Since (3) only has linear constraints and is feasible, Slater's condition holds. Recall that P syn represents the empirical distribution of the synthetic data, supported on X . Without loss of generality, we assume X is a finite set and denote p post ≜ ( P post ( x )) x ∈X and p syn ≜ ( P syn ( x )) x ∈X where p post x ≜ P post ( x ) , p syn x ≜ P syn ( x ) . For each query, we denote q k ≜ ( q k ( x )) x ∈X . Now we introduce the dual variables µ , λ + ≜ ( λ + 1 , · · · , λ + K ) , and λ -≜ ( λ -1 , · · · , λ -K ) . Then the Lagrangian function of (3) can be written as

The Lagrangian function is strictly convex w.r.t. p post so it has a unique solution if exists. We compute the partial derivative w.r.t. p post and set it to be zero:

We denote λ ≜ λ + , ∗ -λ -, ∗ . Then (6) yields a closed-form expression of the optimal solution:

Substitute this optimal solution into the Lagrangian function and rewrite it as

where we omit the terms that do not depend on µ . Therefore, the µ that maximizes L ( p post , ∗ , λ + , λ -, µ ) satisfies:

We simplify the above expression and obtain

Now we substitute µ ∗ into the Lagrangian function and get

Note that the ( λ + , λ -) ≥ 0 that maximizes L ( p post , ∗ , λ + , λ -, µ ∗ ) must satisfy λ + k λ -k = 0 since, otherwise, we can replace them with

and the Lagrangian function increases. Hence, we have ∥ λ ∥ 1 = ∑ K k =1 ( λ + , ∗ k + λ -, ∗ k ) . Now we simplify the Lagrangian function:

Finally, we rewrite the optimal primal solution as

B More Details on Algorithm Implementation

Denoising output from Laplace mechanism. For a given matrix Q ∈ R K × S and a vector a ∈ R K , we can denoise the outputs from Laplace mechanism by solving a linear program (LP):

Denoising output from Gaussian mechanism. For a given matrix Q ∈ R K × S and a vector a ∈ R K , we can solve the following quadratic program to denoise the outputs from Gaussian mechanism:

Implementing Algorithm 1 in log-domain. Recall that we introduced a stochastic first-order method for computing the dual variables in Algorithm 1. To avoid underflow and overflow problems, we use log-domain computations to implement this algorithm. The details are included in Algorithm 2.

C Additional Experimental Results

In Table 3, we reproduce our experiment in Table 1 with a higher privacy budget. Our observation is consistent with prior experimental results: our post-processing technique consistently enhances the utility of synthetic data on selected metrics across all benchmark datasets and all privacy mechanisms.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers how to obtain improved utility from synthetic data under differential privacy.  Prior work has tried to achieve utility by incorporating utility directly into the generation of the synthetic data e.g., by making use of workload data.  Here, the proposal is to perform post-hoc reweighting of the synthetic data, based on measuring a difference in values between the final data under a set of queries with known answers.  This measurement/reweighting step is also done under privacy.  The paper shows that the optimization can be done using standard solvers, and so can be performed effectively via linear or quadratic programming (for pure or approximate DP). A specific use case based on aligning the correlation matrix is used to demonstrate that the sensitivity calculations can be quite simple.  Numerical experiments demonstrate that the approach can be quite effective in terms of absolute error ratio, and F1 score.\n\nSTRENGTHS:\nSynthetic data is widely promoted as a practical way to gain from private data, without running into issues around exhaustion of privacy budget.  Ensuring that synthetic data is usable for the target queries is therefore an important challenge, particularly if the workload may not be known at data generation time.  This approach shows a promising way to overcome these issues, provided we still have access to the data to perform the private optimization step. \n\nThe approach is very clean, and shows good promise in terms of utility and performance.\n\nWEAKNESSES:\nEvaluation is on four benchmark data sets that may be considered \"easy\".  It would be interesting to test this out on additional data, such as that used in recent synthetic data challenges by NIST.\n\nQUESTIONS:\nApart from correlation matrix, can you generate more examples of query families to apply that have bounded sensitivity?\n\nLIMITATIONS:\nNo need to address societal impact.  \n\nTechnical limitations are outlined in Section 5 in broad strokes, but it would be interesting to hear more specific suggestions about how this approach could be extended to handle more diverse data.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a DP post-processing method for synthetic data that weights the synthetic data to match user-selected utility measures on the real dataset. The utility measures on real data are measured with noise to make this post-processing DP. To find the synthetic data weights, the paper derives a closed-form expression, and develops an optimisation algorithm to find optimal dual variables required to evaluate the closed-form optimum. To evaluate the method, the paper conducts experiments on 5 real datasets, testing how much the method is able to improve the synthetic data produced by 5 existing methods of generating DP synthetic data.\n\nSTRENGTHS:\nThe paper is written well, and the main points are easy to understand. The idea behind the proposed method is interesting, and it should, at least in principle, be applicable to any kind of synthetic data and utility measure.\n\nWEAKNESSES:\nSome important experimental details are missing. See my questions.\n\nThe Private PGB method of Neunhoeffer et al. (2021) is fairly similar to the proposed method, and their differences should be discussed. Currently the paper is not even cited.\n\nMinor comments:\n- Slater's condition should be introduced before the proof of Theorem 1.\n- Seeing the raw utility and F1 scores from the experiment in Table 1 would be useful, as it would show whether the large improvement from post-processing on the GANs is just caused by the GANs generating poor synthetic data\n- References [App17], [Dat23], [DG17], [MOK18], [Sma23] should have URLs\n\nQUESTIONS:\nImportant experimental details:\n- Is the plain synthetic data baseline $\\epsilon$ or $(\\epsilon + \\epsilon')$-DP? It seems to be the former, which is not a fair comparison, as the generation + post-processing is $(\\epsilon + \\epsilon')$-DP.\n- Are the categorical columns also converted to numerical features for AIM and MST? They should not be, as both methods handle categorical values natively.\n- What is the workload given to AIM? Does it contain the marginal queries with the variables the utility measures are looking at?\n\nMinor questions:\n- How is the real data split into training and test data?\n- Why were MST and AIM not run on the home-credit dataset? The original paper on Private-PGM (McKenna et al. 2019) that both methods are based on experiments on a dataset with a similar number of features.\n- Is there a difference between $\\lambda^*$ in (4) and $\\lambda$ in (5)?\n\nReferences:\n- Neunhoeffer et al. \"Private Post-GAN Boosting\" ICLR 2021\n- McKenna et al. \"Graphical-model based estimation and inference for differential privacy\" ICML 2019\n\nLIMITATIONS:\nThe paper mentions that they assume the real data is normalised so each feature lies in $[0, 1]$. This is a fairly large limitation, as normalising data under DP while retaining the possibility of undoing the normalisation is not trivial, and should be discussed further..", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-21"}

comments: SUMMARY:
The paper introduces a post-processing technique that enhances the utility of synthetic DP data with respect to selected measures. 
The proposed technique involves resampling from the synthetic data to filter out samples that do not meet the selected utility measures, using a stochastic first-order algorithm to find optimal resampling weights.

STRENGTHS:
Strengths:

- The post-processing technique discussed appears to be novel and significant.

- The method is model-agnostic. This makes the technique highly versatile.

- The authors provide a good set of numerical experiments to validate their approach. These results are strong and show improvements across multiple benchmark datasets and synthetic data generation algorithms.

- The paper is well-structured and the methodology is clearly explained.

WEAKNESSES:
Weaknesses:

- The paper could benefit from a more detailed discussion on the trade-offs involved in using the proposed post-processing technique. For instance, it would be helpful to understand the impact of the technique on the computational complexity of the data synthesis process.

- The authors could provide more examples or case studies to illustrate the practical applications and benefits of their proposed technique, beside the correlation example illustrated.

QUESTIONS:
- Could you elaborate on the computational complexity of the proposed post-processing technique? How does it compare to the complexity of existing synthetic data generation algorithms?

- How does the proposed technique handle high-dimensional data? Are there any limitations or challenges in this regard?

- Could you provide additional examples to illustrate the practical applications and benefits of the proposed technique?

LIMITATIONS:
See my questions above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper under review introduces a technique aimed at enhancing the quality of differentially private synthetic data. Specifically, this approach is applicable when a private synthetic data set, generated by any available privacy-preserving mechanism, does not align with the original data set on certain key measures or queries. The proposed solution involves adjusting the weights of synthetic data samples to ensure their alignment with the original data based on the specified objective. The paper specifically applies this technique to post-process synthetic data such that the resultant synthetic data aligns with the true data's correlation matrix.\n\nThe proposed algorithm commences by estimating the empirical correlation matrix using the Gaussian mechanism with privacy parameters epsilon=1 and epsilon=3. This involves the addition of independent Gaussian noise to each entry of the correlation matrix. Subsequently, the algorithm resolves a convex optimization problem using a first-order method to determine the optimal sample weights that best conform to the noisy correlation matrix.\nThe research findings demonstrate that the post-processed synthetic data offers a more accurate approximation of the correlation matrix compared to the original synthetic data (prior to the reweighting operation), thereby underscoring the effectiveness of the proposed method.\n\nSTRENGTHS:\nThe idea of post-processing synthetic data presented in this paper is a compelling approach that boasts potential applicability to a multitude of problems. While the primary objective here is the alignment of pair-wise correlations within the data, theoretically, the objective could be a more intricate query that existing synthetic data methods are not designed to tackle.\n\nWEAKNESSES:\nIn terms of originality, the proposed solution bears similarity to the Private Entropy Projection (PEP) mechanism referenced in [LVW21]. The PEP mechanism functions over the entirety of the data domain, assumed to be sufficiently small for its computational limitations. Nevertheless, it could, in theory, operate over any support, including samples from synthetic datasets, as suggested in this paper. Thus, it would be advantageous to explain the distinctions between these methods. Further, if they indeed differ, the paper should clarify why the PEP method is ill-suited for addressing the problem at hand.\n\nRegarding the experimental setup, it appears to have some potential flaws. The paper compares the utility of a synthetic dataset D_syn, presumably generated using an original privacy budget (termed epsilon_original), with the utility of a post-processed dataset D_post, created with an additional privacy budget of epsilon=1. Consequently, the post-processed dataset D_post possesses less privacy than the pre-processed D_syn. Hence, it remains ambiguous whether the superiority of D_post over D_syn can be attributed to the proposed resampling method or merely to the fact that a larger privacy budget was utilized to generate D_post. A more transparent comparison would involve assessing the utility of D_post against a synthetic dataset generated using a budget of (epsilon_original + 1).\n\nLastly, while the post-processing operation enhances the synthetic data in relation to the correlation matrix, it remains uncertain whether this operation compromises other essential data properties. For instance, if the original synthetic data was trained on 3-way marginals, the post-processing step could cause a deviation from these queries, thus leading to a poorer approximation. This possibility could account for the machine learning results in Figure 1, where the post-processed dataset exhibits a lower F1 score under certain conditions.\n\nQUESTIONS:\nWhy was the algorithm GEM only used for the home-credit dataset ?\nWhat is the effect of the post-processes dataset on the measures that the original data was optimized for? That is, does the utility degrade?\nHow does the quality of the original synthetic data affect the absolute performance of the post-process synthetic data ?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}