SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Function:  $\operatorname{prox}_{\alpha \| \cdot \|_1}(x) = \operatorname{sign}(x) \max \{|x| - \alpha, 0\}$
demonstrate that downstream predictive models trained on the post-processed synthetic data can achieve comparable or even better performance on real data than those trained on the original synthetic data.
Finally, we establish a DP guarantee for the post-processing data. Since our algorithm only accesses the real data when computing the utility measures, the composition rule yields the following proposition.
Proposition 1. Suppose the synthetic data are generated from any  $(\epsilon, \delta)$ -DP mechanism and the utility measures are evaluated from the real data from any  $(\epsilon_{post}, \delta_{post})$ -DP mechanism. Then the post-processed data satisfy  $(\epsilon + \epsilon_{post}, \delta + \delta_{post})$ -DP.
So far, we have introduced a general framework for post-processing synthetic data to improve their utility on selected measures. Next, we present a concrete use case of our framework: aligning the correlation matrix. This can be achieved by choosing a specific set of utility measures in our framework.
Aligning the correlation matrix. The correlation matrix comprises the correlation coefficients between pairs of features and is commonly used by data scientists to select features before fitting their predictive models. Aligning the correlation matrix of synthetic data with real data can assist data scientists in identifying the most suitable set of features and developing a more accurate model. Recall the definition of the correlation coefficient between features  $\mathrm{X}_i$  and  $\mathrm{X}_j$  for  $i, j \in [d]$ :
$$
\operatorname {c o r r} \left(\mathbf {X} _ {i}, \mathbf {X} _ {j}\right) = \frac {\mathbb {E} \left[ \mathbf {X} _ {i} \mathbf {X} _ {j} \right] - \mathbb {E} \left[ \mathbf {X} _ {i} \right] \mathbb {E} \left[ \mathbf {X} _ {j} \right]}{\sqrt {\mathbb {E} \left[ \mathbf {X} _ {i} ^ {2} \right] - \mathbb {E} \left[ \mathbf {X} _ {i} \right] ^ {2}} \cdot \sqrt {\mathbb {E} \left[ \mathbf {X} _ {j} ^ {2} \right] - \mathbb {E} \left[ \mathbf {X} _ {j} \right] ^ {2}}}.
$$
Since the correlation coefficient only depends on the first and second order moments, we can select the following set of  $(d + 3)d / 2$  queries as our utility measures:
$$
\mathcal {Q} = \left\{q _ {i} (\boldsymbol {x}) = x _ {i} \right\} _ {i \in [ d ]} \cup \left\{q _ {i, j} (\boldsymbol {x}) = x _ {i} x _ {j} \right\} _ {i \leq j}.
$$
Assume that each feature  $x_{i}$  is normalized and takes values within [0, 1]. As a result, the  $L_{1}$  and  $L_{2}$  sensitivity of  $\mathcal{Q}$  can be upper bounded:
$$
\Delta_ {1} (\mathcal {Q}) \leq \frac {(d + 3) d}{2 n} \quad \text {a n d} \quad \Delta_ {2} (\mathcal {Q}) \leq \frac {\sqrt {(d + 3) d}}{\sqrt {2} n}.
$$
We can compute these utility measures using the Laplace or Gaussian mechanism, denoise the answers, compute the optimal dual variables, and resample from the synthetic data.
We now present numerical experiments to demonstrate the effectiveness of our post-processing algorithm. Our results show that this algorithm consistently improves the utility of the synthetic
data across multiple datasets and state-of-the-art privacy mechanisms. Additionally, it achieves this without degrading the performance of downstream models or statistical metrics. We provide additional experimental results and implementation details in Appendix C.
Benchmark. We evaluate our algorithm on four benchmark datasets: Adult, Bank, Mushroom, and Shopping, which are from the UCI machine learning repository [DG17]. All the datasets have a target variable that can be used for a downstream prediction task. We use four existing DP mechanisms—AIM [MMSM22], MST [MMS21], DPCTGAN [RLP+20], and PATECTGAN [RLP+20] to generate private synthetic data. Among these DP mechanisms, AIM is a workload-based method; MST is a marginal-based method; DPCTGAN and PATECTGAN are GAN-based methods. All of their implementations are from the OpenDP library [Sma23]. To demonstrate the scalability of our approach, we conduct an additional experiment on the home-credit dataset [MOK18]. This dataset has 307,511 data points and 104 features. We apply GEM [LVW21] to this high-dimensional dataset for generating synthetic data. We pre-process each dataset to convert categorical columns into numerical features and standardize all features to the range  $[0,1]$ .
We clarify that our objective is not to compare the quality of synthetic data generated by these DP mechanisms (for a benchmark evaluation, please refer to  $\mathrm{[TMH^{+}21]}$ ), but to demonstrate how our post-processing method can consistently improve the utility of synthetic data generated from these different DP mechanisms.
Setup. We compare synthetic data produced by different privacy mechanisms with  $\epsilon \in \{2,4\}$  against the post-processed synthetic data that are generated by applying our post-processing technique with  $\epsilon_{\mathrm{post}} = 1$  to synthetic data generated from privacy mechanisms with  $\epsilon \in \{1,3\}$ . For UCI datasets, we select 5 features, including the target variable, from the synthetic data that have the highest absolute correlation with the target variable. These features are chosen as they have a higher influence on downstream prediction tasks. The set of these features is denoted by  $S \subseteq [d]$ . Next, we define the utility measures as the first-order and second-order moment queries among the features in  $S$ :  $\mathcal{Q} = \{q_i(\boldsymbol{x}) = x_i\}_{i \in S} \cup \{q_{i,j}(\boldsymbol{x}) = x_i x_j\}_{i,j \in S}$ . For home-credit dataset, we follow the same setup but choosing 10 features.
We apply the Gaussian mechanism with  $(\epsilon_{\mathrm{post}} = 1, \delta_{\mathrm{post}} = 1 / n^2)$  to estimate utility measures from the real data, where  $n$  denotes the number of real data points. Finally, we apply Algorithm 1 with  $\gamma = 1e - 5$ , a batch size of 256 for UCI datasets and 4096 for home-credit, and 200 epochs to compute the optimal resampling weights, which are then used to resample from the synthetic data with the same sample size. We repeat our experiment 5 times to obtain an error bar for each evaluation measure.
Evaluation measure. We provide evaluation metrics for assessing the quality of synthetic data after post-processing. Our first metric, utility improvement, is defined as:
$$
1 - \frac {\operatorname {e r r o r} _ {\operatorname {c o r r}} (\mathcal {D} _ {\text {p o s t}})}{\operatorname {e r r o r} _ {\operatorname {c o r r}} (\mathcal {D} _ {\text {s y n}})}, \quad \text {w h e r e e r r o r} _ {\operatorname {c o r r}} (\mathcal {D}) = \| \operatorname {c o r r} (\mathcal {D} _ {\text {r e a l}}) - \operatorname {c o r r} (\mathcal {D}) \| _ {1} \text {f o r} \mathcal {D} \in \left\{\mathcal {D} _ {\text {p o s t}}, \mathcal {D} _ {\text {s y n}} \right\}.
$$
The range of this metric is  $(- \infty, 1]$ , where a value of 1 indicates a perfect alignment between the correlation matrix of the post-processed data and the real data.
To assess the downstream quality of synthetic data, we train predictive models using the synthetic data and then test their performance on real data. Specifically, we split the real data, using  $80\%$  for generating synthetic data and setting aside  $20\%$  to evaluate the performance of predictive models. We use the synthetic data to train a logistic regression classifier to predict the target variable based on other features. Then we apply this classifier to real data and calculate its F1 score. We use the function BinaryLogisticRegression from SDMetrics library [Dat23] to implement this process.
We compute two statistical measures to quantify the similarity between synthetic and real data distributions. The first measure is the average Jensen-Shannon distance between the marginal distributions of synthetic and real data, which ranges from 0 (identical distribution) to 1 (totally different distributions). The second measure is the average inverse KL-divergence between the marginal distributions of synthetic and real data, which ranges from 0 (totally different distributions) to 1 (identical distribution). These statistical measures are implemented by using synthcity library [QCvdS23].
Dataset	DP Mechanism	Utility Improv.	F1 score	JS distance (marginal)	Inverse KL (marginal)
w/o post-proc.	w/ post-proc.	w/o post-proc.	w/ post-proc.	w/o post-proc.	w/ post-proc.
Adult	AIM	0.13 ±0.03	0.61	0.61 ±0.0	0.01	0 ±0.0	0.99	1 ±0.0
MST	0.22 ±0.02	0.55	0.56 ±0.0	0.01	0 ±0.0	1	1 ±0.0
DPCTGAN	0.81 ±0.09	0.22	0.33 ±0.02	0.07	0.03 ±0.02	0.85	0.91 ±0.0
PATECTGAN	0.6 ±0.04	0.37	0.5 ±0.03	0.04	0.01 ±0.0	0.72	0.87 ±0.0
Mushroom	AIM	0.12 ±0.0	0.93	0.93 ±0.0	0.01	0 ±0.0	1	1 ±0.0
MST	0.58 ±0.0	0.83	0.83 ±0.02	0.01	0 ±0.0	1	1 ±0.0
DPCTGAN	0.69 ±0.04	0.47	0.68 ±0.01	0.05	0.02 ±0.0	0.83	0.92 ±0.0
PATECTGAN	0.83 ±0.05	0.6	0.86 ±0.04	0.03	0.02 ±0.0	0.87	0.95 ±0.0
Shopper	AIM	0.1 ±0.02	0.48	0.48 ±0.02	0.02	0.01	0.84	0.92 ±0.0
MST	0.5 ±0.02	0.42	0.47 ±0.02	0.01	0 ±0.0	0.99	1 ±0.0
DPCTGAN	0.36 ±0.05	0.27	0.3 ±0.02	0.03	0.01 ±0.0	0.74	0.85 ±0.0
PATECTGAN	0.11 ±0.04	0.25	0.31 ±0.05	0.04	0.01 ±0.0	0.8	0.89 ±0.0
Bank	AIM	0.18 ±0.01	0.45	0.46 ±0.01	0.04	0.02 ±0.01	0.83	0.9 ±0.0
MST	0.32 ±0.02	0.43	0.44 ±0.02	0.02	0.01 ±0.0	0.98	1 ±0.0
DPCTGAN	0.2 ±0.02	0.22	0.24 ±0.07	0.04	0.02 ±0.01	0.71	0.88 ±0.0
PATECTGAN	0.25 ±0.03	0.2	0.23 ±0.05	0.03	0.01 ±0.0	0.83	0.9 ±0.0
Table 1: We compare synthetic data generated without and with our post-processing technique, all under the same privacy budget  $\epsilon = 2$ . We demonstrate utility improvement (higher is better, positive numbers imply improvement) and F1 score for downstream models trained on synthetic data and tested on real data. For reference, when training the same downstream model using real data, the F1 scores are: (Adultt, 0.61), (Mushroom, 0.95), (Shopper, 0.54), and (Bank, 0.47). Additionally, we measure the average Jensen-Shannon (JS) distance between the marginal distributions of synthetic and real data (0: identical distribution; 1: totally different distributions) and the average inverse KL-divergence (0: totally different distributions; 1: identical distribution). As shown, our technique consistently improves the utility of the synthetic data across all datasets and all DP mechanisms without degrading the performance of downstream models or statistical metrics.
DP Mechanism	Utility Impro.	F1 score w/o post-proc.	F1 score w/ post-proc.	JS distance w/o post-proc.	JS distance w/ post-proc.	Inverse KL w/o post-proc.	Inverse KL w/ post-proc.
GEMε=2	0.57 ±0.03	0.19	0.19 ±0.02	0.01	0.01 ±0.0	0.93	0.97 ±0.01
GEMε=4	0.68 ±0.01	0.21	0.21 ±0.01	0.01	0.01 ±0.0	0.95	0.99 ±0.01
Table 2: Experimental results on the home-credit dataset. We compare synthetic data produced by GEM with  $\epsilon \in \{2,4\}$  against the post-processed synthetic data that are generated by applying our post-processing technique with  $\epsilon_{\mathrm{post}} = 1$  to synthetic data generated from GEM with  $\epsilon \in \{1,3\}$ . For reference, the downstream model trained and tested on real data has an F1 score of 0.24.
Result. We present the experimental results on the UCI datasets in Table 1 (and Table 3 in Appendix C with a higher privacy budget). We report utility improvement,  $F1$  score, and statistical measures. As shown, our post-processing technique consistently enhances the utility of synthetic data on selected metrics across all benchmark datasets and all privacy mechanisms. Moreover, the utility improvements are achieved without degrading the performance of downstream models or statistical properties of synthetic data. In other words, our post-processing procedure ensures that the logistic regression classifier trained on synthetic data can achieve comparable or even higher performance on real test data, while simultaneously reducing or maintaining the statistical divergences between synthetic and real data.
We present the experimental results on the home-credit dataset in Table 2 and illustrate the misalignment of the correlation matrix with and without applying our post-processing procedure in Figure 1. The results demonstrate that our algorithm consistently reduces the overall correlation misalignment. Additionally, our procedure, which includes computing the utility measures from real data, denoising the noisy answers, and computing optimal resampling weights, only takes around 4 mins on 1x NVIDIA GeForce RTX 3090 GPU.
Figure 1: Correlation matrix misalignment with and without applying our post-processing procedure. Left: absolute difference between the real and synthetic data correlation matrices. Right: absolute difference between the real and post-processed synthetic data correlation matrices. As shown, our procedure effectively aligns the correlation matrix of the synthetic data with that of the real data.
The use of private synthetic data allows for data sharing without compromising individuals' private information. In practice, end users typically have specific requirements that the synthetic data must meet, which are often derived from their standard data science pipeline. Failure to meet these requirements can diminish the usefulness of the synthetic data. To address this issue, we introduce a theoretically-grounded post-processing procedure for improving a synthetic dataset's utility on a selected set of measures. This procedure filters out samples that do not accurately reflect the real data, while still maintaining DP guarantees. We also present a scalable algorithm for computing resampling weights and demonstrate its effectiveness through extensive experiments.
We believe there is a crucial need for future research to comprehend the capabilities and limitations of private synthetic data from various perspectives [see e.g., JSH+22, GODC22, BDI+23, for further discussions]. It should be noted that synthetic data generation is not intended to replace real data, and any model trained on synthetic data must be thoroughly evaluated before deployment on real data. In cases a ML model trained on synthetic data exhibits algorithmic bias upon deployment on real data, it becomes imperative to identify the source of bias and repair the model by querying the real data with a privacy mechanism.
When generating synthetic data using DP mechanisms, outliers may receive higher noise, which can have a greater impact on minority groups. Consequently, ML models trained on DP-synthetic data by downstream users may demonstrate a disparate impact when deployed on real data [GODC22]. This is true even when using fairness-intervention algorithms  $\left[\mathrm{HCZ}^{+}22\right]$  during model training because the synthetic data follows a different probability distribution than real data. One can potentially apply our post-processing techniques to resample synthetic data and eliminate samples that do not reflect the underlying distribution of the population groups, while ensuring sample diversity. By doing so, they can achieve a higher fairness-accuracy curve when deploying downstream ML models trained on the post-processed synthetic data on real data.
Another promising avenue of research is to investigate the data structure in cases where a user or a company can provide multiple records to the dataset. Financial data, for instance, is typically collected over time and presented as a time series. While it is possible to combine all of a user's data into a single row, this approach can lead to an unmanageably unbounded feature domain and limit the effectiveness of the synthetic data. Therefore, it is worthwhile to explore other privacy notions [see e.g., ZXLZ14, DWCS19,  $\mathrm{LSA}^{+}21$ ] and develop strategies for optimizing higher privacy-utility trade-offs under special data structures.
H. Wang would like to thank Rui Gao (UT Austin) for his valuable input in the early stage of this project.
[ABK+21] Sergul Aydore, William Brown, Michael Kearns, Krishnamaram Kenthapadi, Luca Melis, Aaron Roth, and Ankit A Siva. Differentially private query release through adaptive projection. In International Conference on Machine Learning, pages 457-467. PMLR, 2021.
[Ab018] John M Abowd. The US census bureau adopts differential privacy. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2867-2867, 2018.
$\left[\mathrm{AHJ}^{+}22\right]$  Wael Alghamdi, Hsiang Hsu, Haewon Jeong, Hao Wang, Peter Michalak, Shahab Asoodeh, and Flavio Calmon. Beyond adult and compas: Fair multi-class prediction via information projection. Advances in Neural Information Processing Systems, 35:38747-38760, 2022.
[App17] Apple. Learning with privacy at scale. https://machinelearning.apple.com/research/learning-with-privacy-at-scale, 2017.
[AZK+19] Nazmiye Ceren Abay, Yan Zhou, Murat Kantarcioglu, Bhavani Thuraisingham, and Latanya Sweeney. Privacy preserving synthetic data release using deep learning. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2018, Dublin, Ireland, September 10–14, 2018, Proceedings, Part I 18, pages 510–526. Springer, 2019.
$\left[\mathrm{BDI}^{+}23\right]$  Brian Belgodere, Pierre Dognin, Adam Ivankay, Igor Melnyk, Youssef Mroueh, Aleksandra Mojsilovic, Jiri Navartil, Apoorva Nitsure, Inkit Padhi, Mattia Rigotti, Jerret Ross, Yair Schiff, Radhika Vedpathak, and Richard Young. Auditing and generating synthetic data with controllable trust trade-offs. arXiv preprint arXiv:2304.10819, 2023.
[BJWW+19] Brett K Beaulieu-Jones, Zhiwei Steven Wu, Chris Williams, Ran Lee, Sanjeev P Bhavnani, James Brian Byrd, and Casey S Greene. Privacy-preserving generative deep neural networks support clinical data sharing. Circulation: Cardiovascular Quality and Outcomes, 12(7):e005122, 2019.
[BLR13] Avrim Blum, Katrina Ligett, and Aaron Roth. A learning theory approach to noninteractive database privacy. Journal of the ACM (JACM), 60(2):1-25, 2013.
[BSG17] Vincent Bindschaedler, Reza Shokri, and Carl A Gunter. Plausible deniability for privacy-preserving data synthesis. arXiv preprint arXiv:1708.07975, 2017.
[BSV22] March Boedihardjo, Thomas Strohmer, and Roman Vershynin. Privacy of synthetic data: A statistical framework. IEEE Transactions on Information Theory, 69(1):520-527, 2022.
[BV04] Stephen P Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[BW18] Borja Balle and Yu-Xiang Wang. Improving the gaussian mechanism for differential privacy: Analytical calibration and optimal denoising. In International Conference on Machine Learning, pages 394-403. PMLR, 2018.
[Con18] Nicholas Confessore. Cambridge Analytica and Facebook: The scandal and the fallout so far. The New York Times, 2018.
[Csi75] Imre Csiszár. I-divergence geometry of probability distributions and minimization problems. The Annals of Probability, pages 146-158, 1975.
[Sma23] SmartNoise. Smartnoise sdk: Tools for differential privacy on tabular data. https://github.com/opendp/smartnoise-sdk, 2023.
$\left[\mathrm{TMH}^{+}21\right]$  Yuchao Tao, Ryan McKenna, Michael Hay, Ashwin Machanavajjhala, and Gerome Miklau. Benchmarking differentially private synthetic data generation algorithms. arXiv preprint arXiv:2112.09238, 2021.
$\left[\mathrm{TWB}^{+}19\right]$  Uthaipon Tantipongpipat, Chris Waites, Digvijay Boob, Amaresh Ankit Siva, and Rachel Cummings. Differentially private mixed-type data generation for unsupervised learning. arXiv preprint arXiv:1912.03250, 1:13, 2019.
[UV20] Jonathan Ullman and Salil Vadhan. PCPs and the hardness of generating synthetic data. Journal of Cryptology, 33(4):2078-2112, 2020.
$\left[\mathrm{VAA}^{+}22\right]$  Giuseppe Vietri, Cedric Archambeau, Sergul Aydore, William Brown, Michael Kearns, Aaron Roth, Ankit Siva, Shuai Tang, and Steven Z Wu. Private synthetic data for multitask learning and marginal queries. Advances in Neural Information Processing Systems, 35:18282-18295, 2022.
$\left[\mathrm{VTB}^{+}20\right]$  Giuseppe Vietri, Grace Tian, Mark Bun, Thomas Steinke, and Steven Wu. New oracle-efficient algorithms for private synthetic data release. In International Conference on Machine Learning, pages 9765-9774. PMLR, 2020.
[WLF16] Mengdi Wang, Ji Liu, and Ethan Fang. Accelerating stochastic composition optimization. Advances in Neural Information Processing Systems, 29, 2016.
$\left[\mathrm{XLW}^{+}18\right]$  Liyang Xie, Kaixiang Lin, Shu Wang, Fei Wang, and Jiayu Zhou. Differentially private generative adversarial network. arXiv preprint arXiv:1802.06739, 2018.
$\left[\mathrm{ZCP}^{+}17\right]$  Jun Zhang, Graham Cormode, Cecilia M Procopioc, Divesh Srivastava, and Xiaokui Xiao. PrivBayes: Private data release via Bayesian networks. ACM Transactions on Database Systems (TODS), 42(4):1-41, 2017.
[ZXLZ14] Tianqing Zhu, Ping Xiong, Gang Li, and Wanlei Zhou. Correlated differential privacy: Hiding information in non-IID data set. IEEE Transactions on Information Forensics and Security, 10(2):229-242, 2014.
Recall that Slater's condition is a sufficient condition for ensuring strong duality in the context of convex program [see Section 5.2.3 in BV04]: it states that strong duality holds for a convex program if there exists a strictly feasible solution (i.e., a solution that satisfies all constraints and ensures that nonlinear constraints are satisfied with strict inequalities).
Proof. Since (3) only has linear constraints and is feasible, Slater's condition holds. Recall that  $P_{\mathrm{syn}}$  represents the empirical distribution of the synthetic data, supported on  $\mathcal{X}$ . Without loss of generality, we assume  $\mathcal{X}$  is a finite set and denote  $p^{\mathrm{post}} \triangleq (P_{\mathrm{post}}(\pmb{x}))_{\pmb{x} \in \mathcal{X}}$  and  $p^{\mathrm{syn}} \triangleq (P_{\mathrm{syn}}(\pmb{x}))_{\pmb{x} \in \mathcal{X}}$  where  $p_{\pmb{x}}^{\mathrm{post}} \triangleq P_{\mathrm{post}}(\pmb{x})$ ,  $p_{\pmb{x}}^{\mathrm{syn}} \triangleq P_{\mathrm{syn}}(\pmb{x})$ . For each query, we denote  $q_k \triangleq (q_k(\pmb{x}))_{\pmb{x} \in \mathcal{X}}$ . Now we introduce the dual variables  $\mu, \lambda^+ \triangleq (\lambda_1^+, \dots, \lambda_K^+)$ , and  $\lambda^- \triangleq (\lambda_1^-, \dots, \lambda_K^-)$ . Then the Lagrangian function of (3) can be written as
$$
L (\boldsymbol {p} ^ {\text {p o s t}}, \boldsymbol {\lambda} ^ {+}, \boldsymbol {\lambda} ^ {-}, \mu)
$$
$$
\triangleq \sum_ {\boldsymbol {x} \in \mathcal {X}} p _ {\boldsymbol {x}} ^ {\text {p o s t}} \log \frac {p _ {\boldsymbol {x}} ^ {\text {p o s t}}}{p _ {\boldsymbol {x}} ^ {\text {s y n}}} + \mu (\mathbf {1} ^ {T} \boldsymbol {p} ^ {\text {p o s t}} - 1) + \sum_ {k = 1} ^ {K} \lambda_ {k} ^ {+} (\boldsymbol {q} _ {k} ^ {T} \boldsymbol {p} ^ {\text {p o s t}} - a _ {k} - \gamma) + \lambda_ {k} ^ {-} (- \boldsymbol {q} _ {k} ^ {T} \boldsymbol {p} ^ {\text {p o s t}} + a _ {k} - \gamma).
$$
The Lagrangian function is strictly convex w.r.t.  $\pmb{p}^{\mathrm{post}}$  so it has a unique solution if exists. We compute the partial derivative w.r.t.  $\pmb{p}^{\mathrm{post}}$  and set it to be zero:
$$
\frac {\partial L \left(\boldsymbol {p} ^ {\text {p o s t}} , \boldsymbol {\lambda} ^ {+} , \boldsymbol {\lambda} ^ {-} , \mu\right)}{\partial p _ {\boldsymbol {x}} ^ {\text {p o s t}}} = \log \frac {p _ {\boldsymbol {x}} ^ {\text {p o s t}}}{p _ {\boldsymbol {x}} ^ {\text {s y n}}} + 1 + \mu + \sum_ {k = 1} ^ {K} \left(\lambda_ {k} ^ {+} - \lambda_ {k} ^ {-}\right) q _ {k, \boldsymbol {x}} = 0. \tag {6}
$$
We denote  $\lambda \triangleq \lambda^{+,*} - \lambda^{-,*}$ . Then (6) yields a closed-form expression of the optimal solution:
$$
p _ {\boldsymbol {x}} ^ {\text {p o s t , *}} = p _ {\boldsymbol {x}} ^ {\text {s y n}} \exp \left(- \sum_ {k = 1} ^ {K} \lambda_ {k} q _ {k, \boldsymbol {x}} - 1 - \mu^ {*}\right).
$$
Substitute this optimal solution into the Lagrangian function and rewrite it as