SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} L (\boldsymbol {p} ^ {\text {p o s t}}, *, \boldsymbol {\lambda} ^ {+}, \boldsymbol {\lambda} ^ {-}, \mu) \\ = \sum_ {\boldsymbol {x} \in \mathcal {X}} p _ {\boldsymbol {x}} ^ {\text {p o s t , *}} \left(- \sum_ {k = 1} ^ {K} \lambda_ {k} q _ {k, \boldsymbol {x}} - 1 - \mu\right) + \mu \left(\sum_ {\boldsymbol {x} \in \mathcal {X}} p _ {\boldsymbol {x}} ^ {\text {p o s t , *}} - 1\right) + \sum_ {k = 1} ^ {K} \sum_ {\boldsymbol {x} \in \mathcal {X}} \lambda_ {k} q _ {k, \boldsymbol {x}} p _ {\boldsymbol {x}} ^ {\text {p o s t , *}} + \dots \\ = - \sum_ {\boldsymbol {x} \in \mathcal {X}} p _ {\boldsymbol {x}} ^ {\text {p o s t}, *} - \mu + \dots \\ = - \sum_ {\boldsymbol {x} \in \mathcal {X}} p _ {\boldsymbol {x}} ^ {\text {s y n}} \exp \left(- \sum_ {k = 1} ^ {K} \lambda_ {k} q _ {k, \boldsymbol {x}} - 1 - \mu\right) - \mu + \dots \\ \end{array}
$$
where we omit the terms that do not depend on  $\mu$ . Therefore, the  $\mu$  that maximizes  $L(\pmb{p}^{\mathrm{post}},*,\pmb{\lambda}^{+},\pmb{\lambda}^{-},\mu)$  satisfies:
$$
\sum_ {\boldsymbol {x} \in \mathcal {X}} p _ {\boldsymbol {x}} ^ {\text {s y n}} \exp \left(- \sum_ {k = 1} ^ {K} \lambda_ {k} q _ {k, \boldsymbol {x}} - 1 - \mu^ {*}\right) - 1 = 0.
$$
We simplify the above expression and obtain
$$
\exp \left(1 + \mu^ {*}\right) = \sum_ {\boldsymbol {x} \in \mathcal {X}} p _ {\boldsymbol {x}} ^ {\text {s y n}} \exp \left(- \sum_ {k = 1} ^ {K} \lambda_ {k} q _ {k, \boldsymbol {x}}\right).
$$
Now we substitute  $\mu^{*}$  into the Lagrangian function and get
$$
L \left(\boldsymbol {p} ^ {\text {p o s t}, *}, \boldsymbol {\lambda} ^ {+}, \boldsymbol {\lambda} ^ {-}, \mu^ {*}\right) = - \log \sum_ {\boldsymbol {x} \in \mathcal {X}} p _ {\boldsymbol {x}} ^ {\text {s y n}} \exp \left(- \sum_ {k = 1} ^ {K} \lambda_ {k} q _ {k, \boldsymbol {x}}\right) - \sum_ {k = 1} ^ {K} a _ {k} \lambda_ {k} - \gamma \sum_ {k = 1} ^ {K} \left(\lambda_ {k} ^ {+} + \lambda_ {k} ^ {-}\right).
$$
Note that the  $(\lambda^{+},\lambda^{-})\geq 0$  that maximizes  $L(\pmb{p}^{\mathrm{post},*},\pmb{\lambda}^{+},\pmb{\lambda}^{-},\mu^{*})$  must satisfy  $\lambda_k^+\lambda_k^- = 0$  since, otherwise, we can replace them with
$$
\left(\lambda_ {k} ^ {+} - \min  \{\lambda_ {k} ^ {+}, \lambda_ {k} ^ {-} \}, \lambda_ {k} ^ {+} - \min  \{\lambda_ {k} ^ {+}, \lambda_ {k} ^ {-} \}\right)
$$
and the Lagrangian function increases. Hence, we have  $\| \pmb{\lambda}\| _1 = \sum_{k = 1}^K\left(\lambda_k^{+,*} + \lambda_k^{-,*}\right)$ . Now we simplify the Lagrangian function:
$$
\begin{array}{l} L \left(\boldsymbol {p} ^ {\text {p o s t}, *}, \boldsymbol {\lambda} ^ {+}, \boldsymbol {\lambda} ^ {-}, \mu^ {*}\right) = - \log \sum_ {\boldsymbol {x} \in \mathcal {X}} p _ {\boldsymbol {x}} ^ {\text {s y n}} \exp \left(- \sum_ {k = 1} ^ {K} \lambda_ {k} q _ {k, \boldsymbol {x}}\right) - \sum_ {k = 1} ^ {K} a _ {k} \lambda_ {k} - \gamma \| \boldsymbol {\lambda} \| _ {1} \\ = - \log \left(\mathbb {E} \left[ \exp \left(- \sum_ {k = 1} ^ {K} \lambda_ {k} q _ {k} (\mathbf {X})\right) \right]\right) - \sum_ {k = 1} ^ {K} a _ {k} \lambda_ {k} - \gamma \| \boldsymbol {\lambda} \| _ {1} \\ = - \log \left(\mathbb {E} \left[ \exp \left(\sum_ {k = 1} ^ {K} - \lambda_ {k} (q _ {k} (\mathbf {X}) - a _ {k})\right) \right]\right) - \gamma \| \boldsymbol {\lambda} \| _ {1}. \\ \end{array}
$$
Finally, we rewrite the optimal primal solution as
$$
\begin{array}{l} P ^ {\text {p o s t}, *} (\boldsymbol {x}) \propto P ^ {\text {s y n}} (\boldsymbol {x}) \exp \left(- \sum_ {k = 1} ^ {K} \lambda_ {k} ^ {*} q _ {k} (\boldsymbol {x})\right) \\ \propto P ^ {\mathrm {s y n}} (\boldsymbol {x}) \exp \left(- \sum_ {k = 1} ^ {K} \lambda_ {k} ^ {*} (q _ {k} (\boldsymbol {x}) - a _ {k})\right). \\ \end{array}
$$
B More Details on Algorithm Implementation
Denoising output from Laplace mechanism. For a given matrix  $\mathbf{Q} \in \mathbb{R}^{K \times S}$  and a vector  $\mathbf{a} \in \mathbb{R}^K$ , we can denoise the outputs from Laplace mechanism by solving a linear program (LP):
$$
\begin{array}{l} \boldsymbol{p}^{*} = \operatorname *{argmin}_{\boldsymbol {p}\in \Delta_{S}^{+}}\| \boldsymbol {Q}\boldsymbol {p} - \boldsymbol {a}\|_{1} \\ = \operatorname * {a r g m i n} _ {\boldsymbol {p} \in \mathbb {R} ^ {S}} \boldsymbol {1} ^ {T} \boldsymbol {t} \\ Q p - a \leq t \\ Q p - a \geq - t \\ \mathbf {1} ^ {T} \boldsymbol {p} = 1 \\ \boldsymbol {p} > 0. \\ \end{array}
$$
Denoising output from Gaussian mechanism. For a given matrix  $\mathbf{Q} \in \mathbb{R}^{K \times S}$  and a vector  $\mathbf{a} \in \mathbb{R}^K$ , we can solve the following quadratic program to denoise the outputs from Gaussian mechanism:
$$
\boldsymbol{p}^{*} = \operatorname *{argmin}_{\boldsymbol {p}\in \Delta_{S}^{+}}\frac{1}{2}\| \boldsymbol {Q}\boldsymbol {p} - \boldsymbol {a}\|_{2}^{2}.
$$
Implementing Algorithm 1 in log-domain. Recall that we introduced a stochastic first-order method for computing the dual variables in Algorithm 1. To avoid underflow and overflow problems, we use log-domain computations to implement this algorithm. The details are included in Algorithm 2.
In Table 3, we reproduce our experiment in Table 1 with a higher privacy budget. Our observation is consistent with prior experimental results: our post-processing technique consistently enhances the utility of synthetic data on selected metrics across all benchmark datasets and all privacy mechanisms.
Algorithm 2 Dual variables computation in log scale.
Input:  $\mathcal{D} = \{\bar{q}_i\}_{i=1}^n$  where  $\bar{q}_i = (q_1(\pmb{x}_i) - a_1, \dots, q_K(\pmb{x}_i) - a_K)$ ; maximal number of iterations  $T$ ; mini-batch index  $\mathcal{B}_t \subseteq [n]$ ; step size  $\alpha_t$
Dual variables update (in log-scale):
$$
\begin{array}{l} l l _ {j} ^ {(t + 1)} = \exp \left(\operatorname {p r o x} _ {\alpha_ {t} \gamma \| \cdot \| _ {1}} \left(\log \left(l l _ {j} ^ {(t)}\right) + \frac {\alpha_ {t}}{\tau^ {(t)}} \frac {1}{| \mathcal {B} _ {t} |} \sum_ {i \in \mathcal {B} _ {t}} \prod_ {k = 1} ^ {K} l l _ {k} ^ {(t) (- \bar {q} _ {i, k})} \bar {q} _ {i, j}\right)\right) \quad \text {f o r} j \in [ K ] \\ \tau^ {(t + 1)} = \frac {1}{| \mathcal {B} _ {t} |} \sum_ {i \in \mathcal {B} _ {t}} \prod_ {k = 1} ^ {K} l l _ {k} ^ {(t + 1) (- \bar {q} _ {i, k})} \\ \end{array}
$$
Output:  $\lambda^{(T + 1)} = \log (ll^{(T + 1)})$
Dataset	DP Mechanism	Utility Improv.	F1 score	JS distance (marginal)	Inverse KL (marginal)
w/o post-proc.	w/ post-proc.	w/o post-proc.	w/ post-proc.	w/o post-proc.	w/ post-proc.
Adult	AIM	0.29 ±0.04	0.61	0.61 ±0.0	0.01	0 ±0.0	0.99	1 ±0.0
MST	0.1 ±0.09	0.55	0.55 ±0.0	0.01	0 ±0.0	1	1 ±0.0
DPCTGAN	0.52 ±0.02	0.56	0.58 ±0.03	0.07	0.02 ±0.0	0.81	0.87 ±0.0
PATECTGAN	0.7 ±0.01	0.16	0.49 ±0.01	0.02	0.01 ±0.0	0.88	0.93 ±0.0
Mushroom	AIM	0.28 ±0.0	0.93	0.95 ±0.0	0.01	0 ±0.01	0.96	1 ±0.0
MST	0.55 ±0.0	0.83	0.84 ±0.0	0.01	0 ±0.0	1	1 ±0.0
DPCTGAN	0.74 ±0.04	0.61	0.73 ±0.02	0.07	0.02 ±0.0	0.72	0.85 ±0.0
PATECTGAN	0.8 ±0.09	0.29	0.78 ±0.08	0.02	0.01 ±0.0	0.94	0.99 ±0.0
Shopper	AIM	0.17 ±0.01	0.5	0.5 ±0.0	0.01	0.01 ±0.0	0.89	0.97 ±0.0
MST	0.47 ±0.01	0.47	0.52 ±0.0	0.01	0 ±0.0	0.98	1 ±0.0
DPCTGAN	0.1 ±0.09	0.19	0.25 ±0.01	0.06	0.02 ±0.01	0.8	0.89 ±0.0
PATECTGAN	0.23 ±0.06	0.22	0.26 ±0.05	0.02	0.01 ±0.0	0.83	0.96 ±0.02
Bank	AIM	0.21 ±0.05	0.47	0.47 ±0.0	0.02	0.02 ±0.0	0.93	0.98 ±0.0
MST	0.21 ±0.02	0.41	0.44 ±0.0	0.01	0 ±0.0	0.98	1 ±0.0
DPCTGAN	0.15 ±0.03	0.23	0.27 ±0.02	0.03	0.01 ±0.0	0.77	0.92 ±0.0
PATECTGAN	0.23 ±0.03	0.16	0.19 ±0.05	0.02	0.01 ±0.0	0.94	0.97 ±0.01
Table 3: We compare synthetic data generated without and with our post-processing technique, all under the same privacy budget  $\epsilon = 4$ .