SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In our initial pass through the data, for each data point  $x_{j}$  we will find the index  $i$  such that  $\beta^{i} \leq x_{j} - \ell + 1 < \beta^{i + 1}$ , which can be done by simply computing  $\lfloor \log_{\beta}(x_j - \ell + 1) \rfloor$  as our lower bound ensures  $x_{j} - \ell + 1 \geq 1$ . Using a dictionary or similar data structure we can efficiently store  $|\{x_{j} \in x | \beta^{i} \leq x_{j} - \ell + 1 < \beta^{i + 1}\}|$  for each  $i$  with the default being 0. This preprocessing does not
2Note that the process can also be terminated at any time without affecting the privacy guarantees.
require sorted data and takes  $O(n)$  arithmetic time, where we note that the previous algorithms also measure runtime arithmetically.
Finally, for each query if we already have  $|\{x_{j} \in x | x_{j} - \ell + 1 < \beta^{i}\}|$ , then we can add  $|\{x_{j} \in x | \beta^{i} \leq x_{j} - \ell + 1 < \beta^{i+1}\}|$  in O(1) time to get  $|\{x_{j} \in x | x_{j} - \ell + 1 < \beta^{i+1}\}|$ . Inductively, each query will take O(1) time. We provide the code in the appendix for easier reproducibility.
4.3 Extension to multiple quantiles
The framework for computing multiple quantiles set up in Kaplan et al. (2022) is agnostic to the technique used for computing a single quantile. Their method will first compute the middle quantile and split the data according to the result. Through recursive application the number of levels of computation will be logarithmic. Furthermore, at each level we can see that at most one partition of the data will differ between neighbors, allowing for instead a logarithmic number of compositions. As such our approach can easily be applied to this recursive splitting framework to achieve the same improvements in composition. This will require some minor updating of their proofs to the swap definition that we will do in the appendix.
In this section we empirically evaluate our approach compared to the previous approaches. We give further detail of the previous approaches, particularly EMQ, in Appendix A along with strong intuition upon why our approach will better handle maximal quantile estimation for data clipping. We first go over the datasets and settings used for our experiments which will follow recent related work Gillenwater et al. (2021); Kaplan et al. (2022). Next we evaluate how accurately our method estimates quantiles for the different datasets in the bounded setting. Finally, we will consider the application of computing differentially private sum, which also gives mean computation, and show how our algorithm allows for a significantly more robust and accurate method when tight bounds are not known for the dataset.
We borrow the same setup and datasets as Gillenwater et al. (2021); Kaplan et al. (2022). We test our algorithm compared to the state-of-the-art on six different datasets. Two datasets will be synthetic. One draws 10,000 data points from the uniform distribution in the range  $[-5, 5]$  and the other draws 10,000 data points from the normal distribution with mean zero and standard deviation of five. Two datasets will come from Soumik (2019) with 11,123 data points, where one has book ratings and the other has book page counts. Two datasets will come from Dua and Graf (2019) with 48,842 data points, where one has the number of hours worked per week and the other has the age for different people. We provide histograms of our datasets for better understanding in Figure 1.
5.2 Quantile estimation experiments
For our quantile estimation experiments, for a given quantile  $q \in [0,1]$  we consider the error of outcome  $o_{q}$  from one of the private methods to be  $|o_{q} - t_{q}|$  where  $t_q$  is the true quantile value. We use the in-built quantile function in the numpy library with the default settings to get the true quantile value. As in previous related works, we randomly sample 1000 datapoints from each dataset and run the quantile computation on each method. This process is then iterated upon 100 times and the error is averaged. We set  $\varepsilon = 1$  as in the previous works, which will require setting  $\varepsilon_{1} = \varepsilon_{2} = 1 / 2$  for the call to AboveThreshold in our method.
We will also tighten the ranges to the following,  $[-5, 5]$  for the uniform dataset,  $[-25, 25]$  for the normal dataset,  $[0, 10]$  for the ratings dataset,  $[0, 10000]$  for the pages dataset,  $[0, 100]$  for the hours dataset, and  $[0, 100]$  for the ages dataset. Given that EMQ suffers performance when many datapoints are equal we add small independent noise to our non-synthetic datasets. This noise will be from the normal distribution with standard deviation 0.001 for the ratings dataset and 0.1 for the other three that have integer values. Our method does not require the noise addition but we will use the perturbed dataset for fair comparison. True quantiles are still computed upon the original data. For the datasets with integer values we rounded each output to the nearest integer. For our method we set  $\beta = 1.001$  for all datasets.
(a) Uniform
(d) Book pages
Figure 1: Histograms for each of our datasets.
(b) Gaussian
(e) Census ages
(c) Book ratings
(f) Census hours
For these experiments we only compare our method UQE and the previous method EMQ, using the implementations from Gillenwater et al. (2021). The other procedures, discussed in the appendix are more generalized and thus for this specific setting do not perform nearly as well which can be seen in the previous experiments Gillenwater et al. (2021); Kaplan et al. (2022), so we omit them from our results. For this experiment we consider estimating each quantile from  $5\%$  to  $95\%$  at a  $1\%$  interval. In Figure 2 we plot the mean absolute error of each normalized by the mean absolute error of UQE to make for an easier visualization.
Figure 2: Plots of UQE = (mean absolute error UQE) / (mean absolute error UQE) and EMQ = (mean absolute error EMQ) / (mean absolute error UQE). Normalizing in this way will make for an easier visualization. When EMQ is below UQE then it's error is lower, and when EMQ is above UQE then it's error is higher
As we can see in Figure 2, EMQ consistently performs better on synthetic data, and UQE tends to perform better on the non-synthetic data. This fits with our intuition that UQE will be best suited to situations where the data is unstructured and less is known about the dataset beforehand because our guess-and-check methodology is designed to better handle ill-behaving datasets.
5.3 Sum estimation experiments
As the primary application of our method we will also be considering differentially private sum computation, which can thereby compute mean as well. We will be using the following  $2\varepsilon$ -DP general procedure for computing the sum of non-negative data:
1. Let  $\Delta = \text{PrivateQuantile}(x, q, \varepsilon)$  where  $\text{PrivateQuantile}$  is any differentially private computation algorithm and  $q \approx 1$ .
2. Output  $\mathrm{Lap}(\Delta /\varepsilon) + \sum_{j = 1}^{n}\min (x_j,\Delta)$
We further test this upon the non-synthetic datasets. For large-scale privacy systems that provide statistical analysis for a wide variety of datasets, if we use a PrivateQuantile that requires an upper bound then we ideally want this bound to be agnostic to the dataset. The is particularly true for sum computations upon groupby queries as the range and size can differ substantially amongst groups. As such, we fix the range at [0, 10000] to encompass all the datasets. We will otherwise use the same general setup as in Section 5.2.
We will measure the error of this procedure as the absolute error of the output and the true sum of the dataset. For each of the 100 iterations of choosing 1000 samples randomly from the full dataset, we also add Lap noise 100 times. Averaging over all these iterations gives our mean absolute error.
Privacy	Method	Ratings data	Pages data	Ages data	Work hours data
ε = 1	UQE	4.780.21	4385.232077.16	103.0516.04	180.4844.92
EMQ	5.730.54	4324.382343.25	187.0633.55	339.0675.82
AT	8.750.31	4377.452340.93	293.111117.32	471.98174.07
ε = 0.5	UQE	9.220.31	7102.343093.13	180.6127.03	277.8977.60
EMQ	6906.137123.36	7601.473963.52	678.221931.11	2131.804669.11
AT	29.01115.04	7491.974367.31	473.19238.19	582.58369.29
ε = 0.1	UQE	44.591.79	21916.376423.75	821.77157.68	981.10219.66
EMQ	45861.7928366.46	46552.0927944.18	50558.3228843.72	47185.5829437.22
AT	11351.7721423.40	30830.4920422.05	14490.0625268.71	9928.1820159.17
Table 1: Mean absolute error for differentially private sum estimation. The standard deviation over the 100 iterations is also provided for each in the subscript. UQE = Our unbounded quantile estimation method. EMQ = The exponential mechanism based quantile estimation method. AT = The aggregate tree method for quantile estimation. For our method we only use  $q = 0.99$ . For the others we use the best performance for  $q \in \{0.95, 0.96, 0.97, 0.98, 0.99\}$ .
We will run this procedure with  $\varepsilon \in \{0.1, 0.5, 1\}$ . Further we will use  $q = 0.99$  always for our method, but give the absolute error for the best performing  $q \in \{0.95, 0.96, 0.97, 0.98, 0.99\}$  for the other methods. It is important to note that this value would have to be chosen ahead of time, which would add more error to the other methods. The previous methods we consider here are again the EMQ, but also the aggregate tree (AT) methods, were we use both the implementation along with generally best performing height (3) and branching factor (10) from Gillenwater et al. (2021). We also implemented the bounding technique using inner quartile range within Algorithm 1 of Smith (2011), but this performed notably worse than the others so we omitted the results from our table.
As we can see in Table 1, our method is far more robust and accurate. Furthermore, for our method the choice of  $q$  remained constant and we can see that our results still stayed consistently accurate when  $\varepsilon$  changed. Note that the noise added to the clipped sum is also scaled proportional to  $\varepsilon$  so the amount the error increased as  $\varepsilon$  decreased for our method is what would be expected proportionally. Once again these findings are consistent with our intuition. Our technique is more robust to differing datasets and privacy parameters, and especially better performing for this important use case.
Recall that the sampled data had size 1000 so dividing accordingly can give the error on mean estimates. There is a long line of literature on differentially private mean estimation. To our knowledge, all of these more complex algorithms either require assumptions upon the data distribution,
3See Kamath et al. (2022) for an extensive review of this literature
such as sub-Gaussian or bounded moments, or bounds upon the data range or related parameters, and most often require both. These results also focus upon proving strong theoretical guarantees of accuracy with respect to asymptotic sample complexity. We first note that our approach will provide better initial bounds upon the data as seen in our experiments, which directly improve the theoretical guarantees in the results that require a data range. But also our focus here is upon practical methods that are agnostic to data distributions and more widely applicable to real-world data. Consequently, a rigorous comparison among all of these methods would be untenable and outside the scope of this work.
5.4 Parameter tuning
We kept our  $\beta$  parameter fixed in all experiments for consistency but also to make our method data agnostic. However, our choice was aggressively small in order to achieve higher precision in the inner quantile estimation comparison in Section 5.2. This choice was still highly resilient to changes in  $\varepsilon$  for our sum experiments as we see our error only scaled proportional to the increase in noise. But for more significant decreases in  $\varepsilon$  or in the data size, i.e. the conditions under which all private algorithms suffer substantial accuracy loss, this choice of  $\beta$  could be too small. Those settings imply that the noise added is larger and the distance between queries and threshold shrinks, so our method is more likely to terminate earlier than desired. Smaller  $\beta$  values will then intensify this issue as the candidate values increase more slowly. For the clipping application, we generally think using a value of  $\beta = 1.01$  would be a more stable choice. In fact, replicating our empirical testing with this value actually improves our results in Table 1. Furthermore, increasing  $\beta$  will also reduce the number of queries and thus the computational cost. In general, we find that setting  $\beta \in [1.01, 1.001]$  gives good performance with  $\beta = 1.01$  as a default for computational efficiency. We leave to future work a more thorough analysis of this parameter to fully optimize setting it with relation to data size and privacy parameter. Additionally, all our methods and proofs only require an increasing sequence of candidate values and it's possible that other potential sequences would be even more effective. For example, if tight upper and lower bounds are known on the data, such as within the recursive computation of multiple quantiles, then it likely makes more sense to simply uniformly partition the interval and check in increasing order. But we leave more consideration upon this to future work as well.
Our choice of  $q = 0.99$  in the differentially private sum experiments was also to maintain consistency with the choices for the previous method but also to keep variance of the estimates lower. This will create some negative bias as we then expect to clip the data. As we can see in our illustrative example Figure 3, the PDF will exponentially decrease once we pass the true quantile value, but will do so less sharply once all the queries have value  $n$ . Accordingly, setting  $q = 1.0$  would add slightly more variance to the estimation but initial testing showed improvement in error. However, if the user would prefer slightly higher variance to avoid negative bias, then setting the threshold at  $n$  or even  $n + 1 / \varepsilon$ , would make it far more likely that the process terminates with a value slightly above the maximum. This is particularly useful for heavy-tailed data, where clipping at the 99th percentile can have an out-sized impact on the bias.
We thank our colleagues Matthew Clegg, James Honaker, and Kenny Leftin for helpful discussions and feedback. We also thank anonymous reviewers for their helpful feedback.
Alabi, D., Ben-Eliezer, O., and Chaturvedi, A. (2022). Bounded space differentially private quantiles. arXiv preprint arXiv:2201.03380.
Barthe, G., Fong, N., Gaboardi, M., Gregoire, B., Hsu, J., and Strub, P.-Y. (2016a). Advanced probabilistic couplings for differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pages 55-67.
Barthe, G., Gaboardi, M., Gregoire, B., Hsu, J., and Strub, P.-Y. (2016b). Proving differential privacy via probabilistic couplings. In Proceedings of the 31st Annual ACM/IEEE Symposium on Logic in Computer Science, pages 749-758.
Nissim, K., Stemmer, U., and Vadhan, S. (2016). Locating a small cluster privately. In Proceedings of the 35th ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems, pages 413-427.
Roth, A. and Roughgarden, T. (2010). Interactive privacy via the median mechanism. In Proceedings of the forty-second ACM symposium on Theory of computing, pages 765-774.
Sajed, T. and Sheffet, O. (2019). An optimal private stochastic-mab algorithm based on optimal private stopping rule. In International Conference on Machine Learning, pages 5579-5588. PMLR.
Shokri, R. and Shmatikov, V. (2015). Privacy-preserving deep learning. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pages 1310-1321.
Smith, A. (2011). Privacy-preserving statistical estimation with optimal convergence rates. In Proceedings of the forty-third annual ACM symposium on Theory of computing, pages 813-822.
Soumik (2019). Goodreads-books dataset. In https://www.kaggle.com/jealousleopard/ goodreadsbooks.
Steinke, T. and Ullman, J. (2016). Between pure and approximate differential privacy. *Journal of Privacy and Confidentiality*.
Ullman, J. (2015). Private multiplicative weights beyond linear queries. In Proceedings of the 34th ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems, pages 303-312.
A Maximal Quantiles Estimation
In this section, we specifically consider applying our unbounded quantile estimation to computing the highest quantiles. This is commonly needed for clipping the dataset to privately compute the sum and mean, which we empirically test in Section 5.3. Our primary goal here is to build intuition upon why our approach fundamentally gives more accurate and robust estimations of the highest quantiles when only loose upper bounds on the data are known. We first give more details upon the previous methods, particularly the EMQ method. Then we give a more detailed look at how these methods are negatively affected by loose upper bounds and why ours performs well in comparison.
A.1 Previous techniques
The most effective and practical previous method for quantile estimation, EMQ, builds a distribution over an assumed bounded range  $[a,b]$  through an invocation of the exponential mechanism. Assuming the data is sorted, it will partition the range based upon the data and select an interval  $[x_j,x_{j + 1}]$  with probability proportional to
$$
\exp \left(- \frac {\varepsilon | j - q n |}{2}\right) \left(x _ {j + 1} - x _ {j}\right)
$$
where the intervals  $[a, x_1]$  and  $[x_n, b]$  are also considered. A uniformly random point is then drawn from within the selected interval. Note that this utility function is not monotonic and will not enjoy the same improved privacy bounds.
There are other approaches that recursively partition the data range while computing statistics on each partition, then aggregate these statistics across logarithmic levels to reduce the noise for quantile computation Dwork et al. (2010); Chan et al. (2011). Another technique is to instead utilize the local sensitivity while maintaining privacy by using a smoothing procedure that can also be used for computing quantiles Nissim et al. (2007). There is also a similar approach to ours within Chen et al. (2016) for bounding an unbounded dataset that increases the bounds by a factor of 2 each iteration but we will discuss in Section A.2 why this variant performs substantially worse.
A.2 Effect of bounds upon maximum quantiles
As mentioned, EMQ will construct a probability distribution over the range  $[a,b]$ . The corresponding PDF is a unimodal step function of the intervals defined by the data with the peak interval  $[x_j,x_{j + 1}]$  being such that  $j$  closest to  $qn$ . Note then that if  $b >> x_{n}$  then the probability of selecting  $[x_n,b]$  increases dramatically. The exponential decay as the PDF moves away from the peak interval diminishes the impact of  $[x_n,b]$  significantly if  $n$  is far away from  $qn$ . However, for computing the highest quantiles, we want  $q$  close to 1 by definition, and the looseness of the upper bound drastically effects the accuracy.
In contrast, as soon as the candidate value for our method exceeds the true quantile value, each successive query will have an output of at least  $qn$  which is the threshold. The probability of continuing for  $\lambda$  more queries then decreases in  $\lambda$ . For ease of comparison, we can modify our Algorithm 2 such that in the last step, the output is drawn uniformly from  $[\beta^{k-1}, \beta^k]$ , assuming  $\ell = 1$  for simplicity. This would then create a continuous probability distribution that would similarly be a step function with each interval  $[\beta^{k-1}, \beta^k]$  being a step.
For better intuition we plot the approximate PDF of each in Figure 3. We use the Gumbel noise in the call to AboveThreshold to take advantage of the closed form expression in Lemma B.2 for easier PDF computation.
As we can see in Figure 3, the upper bound,  $b$ , increasing from 10 to 20 dramatically increases the probability of selecting a point in  $[x_n, b]$  which changes the normalization for the other intervals, significantly altering the PDF of EMQ. Given that our method is unbounded, the PDF for UQE stays
<sup>4</sup>We expect  $\lambda$  to most likely be small and this only adds a factor of  $\beta^{\lambda}$  additional error. Note that setting  $\beta$  too high, such as  $\beta = 2$  which gives a similar algorithm to Chen et al. (2016), implies that even taking five additional queries will lead to a value that is over 32 times too large.
Figure 3: Illustrative example of how the approximate PDF of the previous method, EMQ is affected by looser upper bounds compared to our unbounded method UQE. A small amount of data was drawn uniformly from [0, 10] and we set  $q = 0.9$ , so accurate output would be about 9. The left and right side assumed a range of [0, 20] and [0, 10] respectively for EMQ.
the same in both figures and sees the expected exponential decay once the candidate quantile passes the true quantile.
There are also alternative methods for bounding the data range by computing the interquartile range more accurately and scaling up. However these methods make strong assumptions upon the data distribution being close to the normal distribution as well as bounds upon the third moment Smith (2011). For real-world datasets, the tails of the data can vary significantly, and scaling up in a data-agnostic manner can often be similarly inaccurate. We also found this to be true in our experiments.
The smooth sensitivity framework will run into similar issues when  $q$  is closer to 1 because fewer data points need to be changed to push the quantile value to the upper bound. If this upper bound is large, then the smooth local sensitivity will still be substantial. Aggregate tree methods are slightly more robust to loose upper bounds as they aren't partitioning specific to the data. However, for accurate estimates the partitioning of the range still requires the data be well-distributed across the evenly split partitions, which is significantly affected by loose upper bounds.
B Improved analysis for Sparse Vector Technique
In this section we complete the analysis for improving the privacy bounds for sparse vector technique with monotonic queries. We will first give a more generalized characterization of query properties by which we can improve the privacy bounds for AboveThreshold. While this characterization is more complex, we will immediately utilize it in Section C to improve the privacy bounds of our methods under an alternate common definition of neighboring. Additionally, the class of queries that we apply it to are not monotonic, so improvements can be made in a multitude of settings. Next we will review the definition of range-bounded, a property of privacy mechanisms that can be used to improve composition bounds, and apply it to our setting. Finally we will utilize these results to show that the privacy analysis of the sparse vector technique can be improved more generally, but also specifically for monotonic queries.
B.1 Generalized characterization
We provide a more general characterization for the privacy loss of AboveThreshold that is specific to the input stream of queries. This will be achieved by providing a one-sided privacy parameter for each pair of neighboring datasets where order matters. For a given pair  $x, x'$ , our goal will be to upper bound the output distribution of  $x$  by the output distribution of  $x'$  up to an exponential factor of the one-sided privacy parameter. The specificity of the parameter to the neighboring datasets will reduce conciseness but it is for this reason that we will be able to further improve privacy analysis. This precise characterization will also help improve bounds for our method in Section C without requiring onerous analysis.
Definition B.1. For a stream of queries  $\{f_i\}$  with sensitivity  $\Delta$ , we define our one-sided privacy loss for neighboring data sets  $x, x'$  as