SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For all experiments, we require all mechanisms to have privacy cost  $pcost(\mathcal{M}) = 1$ . By definition 2.3,  $\mathcal{M}$  satisfies  $\rho$ -zCDP with  $\rho = 1/2$  [46] and satisfies  $\mu$ -Gaussian DP with  $\mu = 1$  [15, 46].
Each experiment is repeated 5 times, we report the mean value of these 5 results and a confidence interval consisting of  $\pm 2$  standard deviations. This is most useful for running time, as the variance loss metrics have negligible variance across all algorithms.
I.1 Scalability
In this section, we study the scalability of ResidualPlanner. This is done using the  $\mathrm{Synth} - n^d$  dataset, where  $d$  is the number of attributes and  $n$  is the domain size of each attribute. We use all  $\leq 3$ -way marginals as a fixed workload and vary  $n$  or  $d$  to get the computation time for HDMM and ResidualPlanner.
I.1.1 Varying Attribute Domain Size  $n$  in the Selection Step.
This experiment considers what happens when the attribute domain size  $n$  gets larger. We fix the number of attributes  $d = 5$  and vary the domain size  $n$  for each attribute, where  $n$  ranges from 2 to 1024. We evaluate the running time and accuracy of the selection step
Table 9 shows the running time for the selection step of HDMM and ResidualPlanner. The RMSE on the workload that the selection step guarantees is also measured. Both HDMM and ResidualPlanner
have no trouble here. HDMM is nearly optimal in RMSE and ResidualPlanner is optimal, as shown by agreement with the SVD Bound. ResidualPlanner is faster, but both methods are fast in this experiment setting.
Table 9: Selection step on Synth- $n^d$  dataset where  $d = 5$  and  $n$  varies. The workload is all  $\leq 3$ -way margins. Metrics are running time and RMSE.
n	TimeHDMM	TimeResPlan	RMSEHDMM	RMSEResPlan	SVDB
2	0.069 ± 0.018	0.001 ± 0.000	1.903	1.890	1.890
4	0.064 ± 0.006	0.001 ± 0.000	2.685	2.681	2.681
8	0.070 ± 0.021	0.001 ± 0.000	3.156	3.156	3.156
16	0.076 ± 0.020	0.001 ± 0.000	3.367	3.366	3.366
32	0.105 ± 0.020	0.001 ± 0.000	3.422	3.423	3.423
64	0.114 ± 0.033	0.001 ± 0.000	3.408	3.407	3.407
128	0.137 ± 0.048	0.001 ± 0.000	3.371	3.367	3.367
256	0.187 ± 0.050	0.001 ± 0.000	3.331	3.322	3.322
512	0.183 ± 0.020	0.001 ± 0.000	3.294	3.283	3.283
1024	0.353 ± 0.058	0.001 ± 0.000	3.328	3.251	3.251
Table 10 shows the running time and Max Variance comparison for the selection step. HDMM can only optimize for RMSE, not max variance, so this table shows that RMSE is not a good substitute when one needs to optimize for Max Variance.
Table 10: Selection step on  $\mathrm{Synth} - n^d$  dataset where  $d = 5$  and  $n$  varies. The workload is all  $\leq$  3-way marginals. Metrics are running time and Max Variance.
n	TimeHDMM	TimeResPlan	MaxVarHDMM	MaxVarResPlan
2	0.069 ± 0.018	0.008 ± 0.001	8.091	4.148
4	0.064 ± 0.006	0.008 ± 0.001	44.693	9.760
8	0.070 ± 0.021	0.008 ± 0.001	180.343	15.643
16	0.076 ± 0.020	0.008 ± 0.001	588.115	20.067
32	0.105 ± 0.020	0.008 ± 0.001	1649.341	22.811
64	0.114 ± 0.033	0.008 ± 0.001	5560.807	24.345
128	0.137 ± 0.048	0.008 ± 0.001	12229.480	25.157
256	0.187 ± 0.050	0.008 ± 0.001	8168.716	25.574
512	0.183 ± 0.020	0.008 ± 0.001	32159.958	25.786
1024	0.353 ± 0.058	0.008 ± 0.001	277825.955	25.893
I.1.2 Impact of varying the number of attributes in the Selection Step.
Next, we fix the domain size of each attribute to be  $n = 10$  and vary the number of attributes  $d$ , where  $d$  ranges from 2 to 200. This experiment can test some of the limits of ResidualPlanner. While HDMM cannot perform selection when the number of attributes is 20 or larger, ResidualPlanner has no trouble optimizing RMSE even for 200 attributes. However, optimizing for Max Variance is much more difficult. ResidualPlanner can do this for  $d = 100$  but the underlying optimization took more than 1 hour for  $d = 200$  and we killed the process.
Table 11 shows the running time and RMSE comparison for the selection step. The running time of HDMM increases sharply and it quickly runs out of memory. At the same point, the SVD Bound can no longer be computed. Meanwhile, ResidualPlanner continues to run efficiently.
Table 12 shows the running time and Max Variance comparison on the Selection step. Optimizing for Max Variance is much harder for ResidualPlanner compared to RMSE and we killed the process for  $d = 200$ . Meanwhile, HDMM is not able to run at  $d = 20$  (we emphasize again, it optimizes for RMSE even if one cares about Max Variance). There is an interesting phenomenon with HDMM that takes place for  $d$  between 8 and 15. In this case, HDMM always produces a max variance of 1000. This maximum is always achieved for the sum query (a zero-dimensional marginal) for the following reason. For  $d$  between 8 and 15, HDMM decides to add noise to all 3-way marginals and nothing else (even though the workload is all  $\leq 3$  marginals). The privacy loss budget is split equally among them. Thus, each of the  $\binom{d}{3}$  marginals it measures gets  $N(0,\binom{d}{3})$  noise. The sum query gets reconstructed
Table 11: Selection step on  $\mathrm{Synth} - n^d$  dataset where  $n = 10$  and  $d$  varies. The workload is all  $\leq$  3-way marginals. Metrics are running time and RMSE.
d	TimeHDMM	TimeResPlan	RMSEHDMM	RMSEResPlan	SVDB
2	0.013 ± 0.003	0.001 ± 0.0008	1.379	1.379	1.379
4	0.028 ± 0.007	0.002 ± 0.001	2.346	2.345	2.345
6	0.065 ± 0.012	0.002 ± 0.0008	4.278	4.275	4.275
8	0.167 ± 0.019	0.004 ± 0.001	6.726	6.638	6.638
10	0.639 ± 0.059	0.009 ± 0.001	9.629	9.348	9.348
12	4.702 ± 0.315	0.015 ± 0.001	12.904	12.359	12.359
14	46.054 ± 12.735	0.025 ± 0.002	16.506	15.642	15.642
15	201.485 ± 13.697	0.030 ± 0.017	18.421	17.378	17.378
20	Out of memory	0.079 ± 0.017	Out of memory	26.916	Out of memory
30	Out of memory	0.247 ± 0.019	Out of memory	49.713	Out of memory
50	Out of memory	1.207 ± 0.047	Out of memory	107.258	Out of memory
100	Out of memory	9.913 ± 0.246	Out of memory	303.216	Out of memory
200	Out of memory	80.120 ± 1.502	Out of memory	855.330	Out of memory
as follows. For any single noisy 3-way marginal, one can estimate the sum by adding up the cells in the marginal. Since each cell has variance  $\binom{d}{3}$  and there are  $n^3 = 1,000$  cells, the sum estimate from a single 3-way marginal has a variance of  $1000\binom{d}{3}$ . But one can obtain an independent estimate to the sum query from each of the  $\binom{d}{3}$  noisy 3-way marginals. By averaging these noisy estimates, one can obtain an estimate of the sum query with variance 1,000.
Table 12: Selection step on  $\mathrm{Synth} - n^d$  dataset where  $n = 10$  and  $d$  varies. The workload is all  $\leq$  3-way marginals. Metrics are running time and Max Variance.
d	TimeHDMM	TimeResPlan	MaxVarHDMM	MaxVarResPlan
2	0.013 ± 0.003	0.007 ± 0.001	13.745	3.306
4	0.028 ± 0.007	0.010 ± 0.005	132.620	10.480
6	0.065 ± 0.012	0.009 ± 0.001	461.132	26.904
8	0.167 ± 0.019	0.015 ± 0.003	1000.000	56.961
10	0.639 ± 0.059	0.018 ± 0.001	1000.000	105.031
12	4.702 ± 0.315	0.028 ± 0.001	1000.000	175.496
14	46.054 ± 12.735	0.041 ± 0.001	1000.000	272.738
15	201.485 ± 13.697	0.050 ± 0.001	1000.000	332.769
20	Out of memory	0.123 ± 0.023	Out of memory	768.941
30	Out of memory	0.461 ± 0.024	Out of memory	2540.440
50	Out of memory	4.011 ± 0.112	Out of memory	11597.037
100	Out of memory	121.224 ± 3.008	Out of memory	91960.917
I.1.3 Scalability of the Reconstruction Step.
We conduct similar experiments, but now we measure the time in the reconstruction step. To complement the reconstruction scalability experiments from the main paper on the  $\mathrm{Synth} - n^d$  synthetic dataset, we first fix the number of attributes  $d = 5$  and vary the domain size  $n$  for each attribute, where  $n$  ranges from 2 to 512. The reconstruction time for ResidualPlanner does not depend on the metric that the select step was optimized for. Again we compare with HDMM [38] and a version of HDMM with improved reconstruction scalability called HDMM+PGM [38, 41] (the PGM settings used 50 iterations of its Local-Inference estimator, as the default 1000 was too slow). Table 13 shows the results. Again, at some point HDMM runs out of memory while ResidualPlanner runs efficiently. HDMM runs of out memory because of choices it had made in the selection step. When  $n = 128$  it decided to measure a 5-way marginal, which is so large (requiring  $128^5$  space) that it caused HDMM and HDMM+PGM to have memory issues.
We next fix  $n = 3$  and vary  $d$ . Table 14 shows ResidualPlanner is clearly faster. Furthermore, HDMM and HDMM+PGM are hampered by the failure of the selection step (when selection fails, there is nothing to reconstruct). It is interesting to compare HDMM+PGM behavior when  $n = 3$  in Table
Table 13: Running time (in seconds) of the reconstruction step on  $\mathrm{Synth} - n^d$  dataset where  $d = 5$  and  $n$  varies. The workload is all  $\leq 3$ -way marginals.
n	HDMM	HDMM + PGM	ResPlan
2	0.005 ± 0.002	2.466 ± 0.278	0.008 ± 0.002
4	0.005 ± 0.000	1.894 ± 0.146	0.011 ± 0.008
8	0.008 ± 0.000	1.871 ± 0.122	0.011 ± 0.008
16	0.064 ± 0.036	1.936 ± 0.131	0.016 ± 0.001
32	1.924 ± 0.060	3.211 ± 0.220	0.045 ± 0.007
64	56.736 ± 1.460	12.574 ± 0.512	0.217 ± 0.021
128	Out of memory	Out of memory	1.244 ± 0.059
256	Out of memory	Out of memory	12.090 ± 0.504
512	Out of memory	Out of memory	166.045 ± 13.803
14 with  $n = 10$  in Table 2 from the main paper. Clearly HDMM+PGM is faster for  $n = 10$  than  $n = 3$ . This counterintuitive result can be explained by the complex workings of HDMM as follows. When  $n = 3$ , the selection step in HDMM returns some 4-way marginals. But when  $n = 10$ , HDMM only returns  $\leq 3$ -way marginals. The 4-way marginals make the reconstruction step harder for both HDMM and HDMM + PGM.
Table 14: Time for Reconstruction Step in seconds on  $\mathrm{Synth} - n^d$  dataset.  $n = 3$  and the number of attributes  $d$  varies. The workload consists of all marginals on  $\leq 3$  attributes each. Times are reported with  $\pm 2$  standard deviations. Reconstruction can only be performed if the select step completed.
d	HDMM	HDMM + PGM	ResidualPlanner
2	0.001 ± 0.0001	0.256 ± 0.030	0.005 ± 0.002
6	0.009 ± 0.001	3.293 ± 0.253	0.020 ± 0.004
10	0.334 ± 0.010	51.568 ± 3.391	0.086 ± 0.004
12	3.882 ± 0.101	180.708± 5.437	0.153 ± 0.002
14	55.856 ± 0.361	314.252 ± 3.991	0.280 ± 0.072
15	231.283 ± 0.554	713.526 ± 4.957	0.307 ± 0.005
20	Unavailable (select step failed)	Unavailable (select step failed)	0.758 ± 0.023
30	Unavailable (select step failed)	Unavailable (select step failed)	2.700 ± 0.200
50	Unavailable (select step failed)	Unavailable (select step failed)	12.480 ± 0.208
100	Unavailable (select step failed)	Unavailable (select step failed)	99.787 ± 2.113
I.2 Comparison on Real Datasets.
In this section, we compare RMSE and Max Variance on the real datasets: CPS, Adult, and Loans. The different workloads are 1-way, 2-way, 3-way, 4-way, 5-way marginals, all  $\leq 3$ -way marginals, and Small Marginals.
I.2.1 RMSE Comparisons
We provide an expanded comparison of RMSE on the 3 real datasets from the main paper. Here we add more workloads. Table 15, 16 and 17 show the comparison of RMSE on the CPS, Adult, and Loans datasets respectively.
We notice that ResidualPlanner matches the theoretical SVD Bound while HDMM is slightly worse, but still accurate. We conclude that when optimizing RMSE, the main advantage of ResidualPlanner is superior scalability.
Since ResidualPlanner is optimal, the purpose of the accuracy comparisons is a sanity check. For RMSE, we compare the quality of ResidualPlanner to the theoretically optimal lower bound known as the SVD bound [31] (they match, as shown in Table 18).
Table 15: Comparison of RMSE on CPS(5D) dataset.
Workload	HDDM	ResPlan	SVDB
1-way Marginals	1.756	1.744	1.744
2-way Marginals	2.103	2.035	2.035
3-way Marginals	2.089	2.048	2.048
4-way Marginals	1.648	1.627	1.627
5-way Marginals	1.000	1.000	1.000
≤ 3-way Marginals	2.301	2.276	2.276
Small Marginals	2.525	2.525	2.525
Table 16: Comparison of RMSE on Adult(14D) dataset.
Workload	HDDM	ResPlan	SVDB
1-way Marginals	3.081	3.047	3.047
2-way Marginals	6.504	6.359	6.359
3-way Marginals	11.529	10.515	10.515
4-way Marginals	16.618	14.656	14.656
5-way Marginals	20.240	17.844	17.844
≤ 3-way Marginals	11.555	10.665	10.665
Small Marginals	10.006	9.945	9.945
I.2.2 Max Variance
The next comparison is on optimization for Max Variance. We repeat that HDMM only optimizes for RMSE and this shows that optimizing for RMSE is highly suboptimal when one cares about max variance.
In contrast to RMSE, where the optimization problem generated by ResidualPlanner's selection step can be solved in closed form, for Max Variance, the optimization needs a convex solver. Hence we include comparisons between the open source ECOS [14] optimizer to the commercial Gurobi optimizer [21]. Thus, our results have columns labeled ResidualPlanner+ECOS and ResidualPlanner+Gurobi.
Tables 19, 20 and 21 show the results for the CPS, Adult, and Loans datasets, respectively. There is one item to note about numerical stability. Although Gurobi is generally faster and more numerically stable, the differences do not matter much. Situations where EOCS was worse are highlighted in red. For example, in Table 19 for the CPS dataset, the dataset has only 5 attributes, so a 5-way marginal is basically the entire dataset. The optimal mechanism for 5-way marginals simply adds  $N(0,1)$  noise to each cell and optimizing for RMSE is equal to optimizing Max Variance for this special case. As we see, the Max Variance for ResidualPlanner+ECOS is 1.008 which is  $0.8\%$  worse than optimal. The reason for this is the numerical precision with which ECOS can solve the optimization problem that ResidualPlanner gives it. In general, however, it looks like open source optimizers should work fairly reliably for them to be used in real applications of ResidualPlanner.
Table 17: Comparison of RMSE on Loans(12D) dataset.
Workload	HDMM	ResPlan	SVDB
1-way Marginals	2.903	2.875	2.875
2-way Marginals	5.747	5.634	5.634
3-way Marginals	9.478	8.702	8.702
4-way Marginals	12.537	11.267	11.267
5-way Marginals	14.872	12.678	12.678
≤ 3-way Marginals	9.406	8.876	8.876
Small Marginals	8.262	8.206	8.206
Table 18: RMSE Comparisons to the theoretical lower bound SVD Bound [31]
Adult Dataset	CPS Dataset	Loans Dataset
Workload	ResPlan	SVDB	ResPlan	SVDB	ResPlan	SVDB
1-way Marginals	3.047	3.047	1.744	1.744	2.875	2.875
2-way Marginals	6.359	6.359	2.035	2.035	5.634	5.634
3-way Marginals	10.515	10.515	2.048	2.048	8.702	8.702
≤ 3-way Marginals	10.665	10.665	2.276	2.276	8.876	8.876
Table 19: Comparison of Max Variance on CPS(5D) dataset.
Workload	HDDM	ResPlan + ECOS	ResPlan + Gurobi
1-way Marginals	13.672	4.346	4.346
2-way Marginals	47.741	7.897	7.897
3-way Marginals	71.549	7.706	7.706
4-way Marginals	15.538	4.142	4.141
5-way Marginals	1.000	1.008	1.000
≤ 3-way Marginals	415.073	13.216	13.216
Small Marginals	223.579	11.774	11.774
Table 20: Comparison of Max Variance on Adult(14D) dataset.
Workload	HDDM	ResPlan + ECOS	ResPlan + Gurobi
1-way Marginals	41.772	12.047	12.047
2-way Marginals	599.843	67.802	67.802
3-way Marginals	5675.238	236.843	236.843
4-way Marginals	26959.322	575.213	575.213
5-way Marginals	79817.002	1030.948	1030.948
≤ 3-way Marginals	6677.253	253.605	253.605
Small Marginals	2586.980	126.902	126.902
Table 21: Comparison of Max Variance on Loans(12D) dataset.
Workload	HDDM	ResPlan + ECOS	ResPlan + Gurobi
1-way Marginals	33.256	10.640	10.640
2-way Marginals	437.478	52.217	52.217
3-way Marginals	3095.997	156.638	156.638
4-way Marginals	13776.417	320.778	320.778
5-way Marginals	26056.289	474.244	474.243
≤ 3-way Marginals	4317.709	180.817	180.817
Small Marginals	2330.883	89.873	89.873