Title: Unbounded Differentially Private Quantile and Maximum Estimation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Differential privacy

Theory

Spars Vector Technique

Quantile
Abstract: In this work we consider the problem of differentially private computation of
quantiles for the data, especially the highest quantiles such as maximum, but
with an unbounded range for the dataset. We show that this can be done
efficiently through a simple invocation of $\texttt{AboveThreshold}$, a
subroutine that is iteratively called in the fundamental Sparse Vector
Technique, even when there is no upper bound on the data. In particular, we
show that this procedure can give more accurate and robust estimates on the
highest quantiles with applications towards clipping that is essential for
differentially private sum and mean estimation. In addition, we show how two
invocations can handle the fully unbounded data setting. Within our study, we
show that an improved analysis of $\texttt{AboveThreshold}$ can improve the
privacy guarantees for the widely used Sparse Vector Technique that is of
independent interest. We give a more general characterization of privacy loss
for $\texttt{AboveThreshold}$ which we immediately apply to our method for
improved privacy guarantees. Our algorithm only requires one $O(n)$ pass
through the data, which can be unsorted, and each subsequent query takes $O(1)$
time. We empirically compare our unbounded algorithm with the state-of-the-art
algorithms in the bounded setting. For inner quantiles, we find that our method
often performs better on non-synthetic datasets. For the maximal quantiles,
which we apply to differentially private sum computation, we find that our
method performs significantly better.

Unbounded Differentially Private Quantile and Maximum Estimation

Abstract

In this work we consider the problem of differentially private computation of quantiles for the data, especially the highest quantiles such as maximum, but with an unbounded range for the dataset. We show that this can be done efficiently through a simple invocation of AboveThreshold , a subroutine that is iteratively called in the fundamental Sparse Vector Technique, even when there is no upper bound on the data. In particular, we show that this procedure can give more accurate and robust estimates on the highest quantiles with applications towards clipping that is essential for differentially private sum and mean estimation. In addition, we show how two invocations can handle the fully unbounded data setting. Within our study, we show that an improved analysis of AboveThreshold can improve the privacy guarantees for the widely used Sparse Vector Technique that is of independent interest. We give a more general characterization of privacy loss for AboveThreshold which we immediately apply to our method for improved privacy guarantees. Our algorithm only requires one /u1D442 (/u1D45B) pass through the data, which can be unsorted, and each subsequent query takes /u1D442 (1) time. We empirically compare our unbounded algorithm with the state-of-the-art algorithms in the bounded setting. For inner quantiles, we find that our method often performs better on non-synthetic datasets. For the maximal quantiles, which we apply to differentially private sum computation, we find that our method performs significantly better.

1 Introduction

In statistics, quantiles are values that divide the data into specific proportions, such as median that divides the data in half. Quantiles are a central statistical method for better understanding a dataset. However, releasing quantile values could leak information about specific individuals within a sensitive dataset. As a result, it becomes necessary to ensure that individual privacy is ensured within this computation. Differential privacy offers a rigorous method for measuring the amount that one individual can change the output of a computation. Due it's rigorous guarantees, differential privacy has become the gold standard for measuring privacy. This measurement method then offers an inherent tradeoff between accuracy and privacy with outputs of pure noise achieving perfect privacy. Thus, the goal of designing algorithms for differentially private quantile computation is to maximize accuracy for a given level of privacy.

There are a variety of previous methods for computing a given quantile of the dataset that we will cover in Section 1.2 , but each of these requires known bounds on the dataset. The most effective and practical method invokes the exponential mechanism Smith (2011). For computing multiple quantiles this method can be called iteratively. Follow-up work showed that it could be called recursively by splitting the dataset at each call to reduce the privacy cost of composition Kaplan et al. (2022). Further, a generalization can be called efficiently in one shot Gillenwater et al. (2021).

1.1 Our contributions

In this work, we offer an alternative practical and accurate approach, Unbounded Quantile Estimation (UQE), that also invokes a well-known technique and can additionally be applied to the unbounded setting. While the commonly-used technique designs a distribution to draw from that is specific to the dataset, our method will simply perform a noisy guess-and-check. Initially we assume there is only a lower bound on the data, as non-negative data is common in real world datasets with sensitive individual information. Our method will simply iteratively increase the candidate value by a small percentage and halt when the number of data points below the value exceeds the desired amount dictated by the given quantile. While the relative increase will be small each iteration, the exponential nature still implies that the candidate value will become massive within a reasonable number of iterations. As a consequence, our algorithm can handle the unbounded setting where we also show that two calls to this procedure can handle fully unbounded data. Computing multiple quantiles can be achieved by applying the recursive splitting framework from Kaplan et al. (2022).

Performing our guess-and-check procedure with differential privacy exactly fits AboveThreshold , a method that is iteratively called in the Sparse Vector Technique Dwork et al. (2009). We also take a deeper look at AboveThreshold and unsurprisingly show that similar to report noisy max algorithms, the noise addition can come from the Laplace, Gumbel or Exponential distributions. We further push this analysis to show that for monotonic queries, a common class of queries which we will also utilize in our methods, the privacy bounds for composition within the Sparse Vector Technique can be further improved. Given the widespread usage of this technique, 1 we believe this result is of independent interest. Furthermore, we give a more general characterization of query properties that can improve the privacy bounds of AboveThreshold . We immediately utilize this characterization in our unbounded quantile estimation algorithm to improve privacy guarantees.

While the commonly used algorithms for quantile estimation can still apply incredibly loose bounds to ensure the data is contained within, this can have a substantial impact upon the accuracy for estimating the highest quantiles such as maximum. This leads to an especially important application for our algorithm, differentially private sum computation, which can thereby be used to compute mean as well. Performing this computation practically without assumptions upon the distribution often requires clipping the data and adding noise proportionally. Clipping too high adds too much noise, and clipping too low changes the sum of the data too much. The highest quantiles of the data are used for clipping to optimize this tradeoff. The unbounded nature of our approach fundamentally allows us to estimate the highest quantiles more robustly and improve the accuracy of differentially private sum computation.

This improvement in differentially private sum computation is further evidenced by our empirical evaluation, with significant improvements in accuracy. Our empericial comparison will be upon the same datasets from previous work in the bounded setting. We also compare private computation of the inner quantiles on these datasets. For synthetic datasets generated from uniform or guassian distributions, we see that the more structured approach of designing a distribution for the data from the exponential mechanism consistently performs better. However, for the real-world datasets, we see that our unstructured approach tends to perform better even within this bounded setting. By design our algorithm is less specific to the data, so our alternative approach becomes advantageous when less is known about the structure and bounds of the data a priori . As such, for large-scale privacy systems that provide statistical analysis for a wide variety of datasets, our methods will be more flexible to handle greater generality accurately.

1.2 Background literature

Theprimary algorithm for privately computing a given quantile, by which we compare our technique, applies the exponential mechanism with a utility function based upon closeness to the true quantile Smith (2011). We will discuss this algorithm, which we denote as Exponential Mechanism Quantile (EMQ), in greater detail in Appendix A. This approach was then extended to computing multiple quantiles more cleverly by recursively splitting the data and establishing that only one partition of

the dataset can change between neighbors thereby reducing the composition costs Kaplan et al. (2022). Additional follow-up work showed that a generalization of the utility function to multiple quantiles could be efficiently drawn upon in one shot Gillenwater et al. (2021). Another recent result examined this problem in the streaming data setting and gave a method that only uses strongly sub-linear space complexity Alabi et al. (2022).

Quantile computation can also be achieved through CDF estimation Bun et al. (2015); Kaplan et al. (2020a). However these techniques offer limited practicality as they rely upon several reductions and parameter tuning. Recursively splitting the data is also done for CDF estimation algorithms where the statistics from each split can be aggregated for quantile computation Dwork et al. (2010); Chan et al. (2011). These techniques tend to be overkill for quantile estimation and thus suffer in accuracy comparatively.

We will also give improved privacy analysis of the Sparse Vector Technique which was originally introduced in Dwork et al. (2009). A more detailed analysis of the method can be found in Lyu et al. (2017). Additional recent work has shown that more information can be output from the method at no additional privacy cost Kaplan et al. (2021); Ding et al. (2023).

1.3 Organization

We provide the requisite notation and definitions in Section 2. In Section 3, we review the AboveThreshold algorithm from the literature and show that privacy analysis can be further improved. In Section 4, we provide our unbounded quantile estimation method. In Section 5, we test our method compared to the previous techniques on synthetic and real world datasets. In Appendix A, we consider the estimation of the highest quantiles which has immediate application to differentially private sum and mean estimation. In Appendix B, we give further results on the AboveThreshold algorithm and provide the missing proofs from Section 3. In Appendix C, we provide further variants and extensions of our unbounded quantile estimation technique.

2 Preliminaries

We will let /u1D465 , /u1D465 ′ denote datasets in our data universe  .

Definition 2.1. Datasets /u1D465 , /u1D465 ′ ∈  are neighboring if at most one individual's data has been changed.

Note that we use the swap definition, but our analysis of the AboveThreshold algorithm will be agnostic to the definition of neighboring. Using this definition as opposed to the add-subtract definition is necessary to apply the same experimental setup as in Gillenwater et al. (2021). Our differentially private quantile estimation will apply to either and we will give the privacy guarantees if we instead use the add-subtract definition in Appendix C.2.

Definition 2.2. A function /u1D453 ∶  → ℝ has sensitivity Δ if for any neighboring datasets |/u1D453 (/u1D465 ) /u1D453 (/u1D465 ′ )| ≤ Δ

Definition 2.3. Dwork et al. (2006b,a) A mechanism /u1D440 ∶  →  is (/u1D700 , /u1D6FF ) -differentially-private (DP) if for any neighboring datasets /u1D465 , /u1D465 ′ ∈  and /u1D446 ⊆  :

We will primarily work with pure differential privacy in this work where /u1D6FF = 0 . We will also be considering the composition properties of the Sparse Vector Technique, and the primary method for comparison will be Concentrated Differential Privacy that has become widely used in practice due to it's tighter and simpler advanced composition properties Bun and Steinke (2016). This definition is instead based upon Reny divergence where for probability distributions /u1D443 , /u1D444 over the same domain and /u1D6FC > 1

Definition 2.4. Bun and Steinke (2016) A mechanism /u1D440 ∶  →  is /u1D70C -zero-concentrateddifferentially-private (zCDP) if for any neighboring datasets /u1D465 , /u1D465 ′ ∈  and all /u1D6FC ∈ (1, ∞) :

We can translate DP into zCDP in the following way.

Proposition 1. Bun and Steinke (2016) If /u1D440 satisfies /u1D700 -DP then /u1D440 satisfies 1 2 /u1D700 2 -zCDP

In our examination of AboveThreshold we will add different types of noise, similar to the report noisy max algorithms Ding et al. (2021). Accordingly, we will consider noise from the Laplace, Gumbel and Exponential distributions where Lap (/u1D44F) has PDF /u1D45D Lap (/u1D467; /u1D44F) , Gumbel (/u1D44F) has PDF /u1D45D Gumbel (/u1D467; /u1D44F) , and Expo (/u1D44F) has PDF /u1D45D Expo (/u1D467; /u1D44F) where

We let Noise (/u1D44F) denote noise addition from any of Lap (/u1D44F), Gumbel (/u1D44F) , or Expo (/u1D44F) . We will also utilize the definition of the exponential mechanism to analyze the addition of Gumbel noise.

Definition 2.5. McSherry and Talwar (2007) The Exponential Mechanism is a randomized mapping /u1D440 ∶  →  such that

where /u1D45E ∶  ×  →ℝ has sensitivity Δ .

3 Improved Analysis for Sparse Vector Technique

In this section, we review the AboveThreshold algorithm from the literature. To our knowledge, this technique has only been used with Laplace noise in the literature. Unsurprisingly, we show that Gumbel and Exponential noise can also be applied, with the former allowing for a closed form expression of each output probability. We further show that for monotonic queries the privacy analysis of the Sparse Vector Technique, which iteratively applies AboveThreshold , can be improved. All proofs are be pushed to Appendix B where we also give a more general characterization of query properties that can improve the privacy bounds of AboveThreshold .

3.1 Above Threshold Algorithm

We first provide the algorithm for AboveThreshold where noise can be applied from any of the Laplace, Gumbel or Exponential distributions.

Algorithm 1 AboveThreshold

Require: Input dataset /u1D465 , a stream of queries {/u1D453 /u1D456 ∶  →ℝ} with sensitivity Δ , and a threshold /u1D447

We will also define a common class of queries within the literature that is often seen to achieve a factor of 2 improvement in privacy bounds.

Definition 3.1. We say that stream of queries {/u1D453 /u1D456 ∶  →ℝ} with sensitivity Δ is monotonic if for any neighboring /u1D465 , /u1D465 ′ ∈  we have either /u1D453 /u1D456 (/u1D465 ) ≤ /u1D453 /u1D456 (/u1D465 ′ ) for all /u1D456 or /u1D453 /u1D456 (/u1D465 ) ≥ /u1D453 /u1D456 (/u1D465 ′ ) for all /u1D456 .

To our knowledge all previous derivations of AboveThreshold in the literature apply Lap noise which gives the following privacy guarantees.

Lemma 3.1 (Theorem 2 and 3 of Lyu et al. (2017)) . If the noise addition is Lap then Algorithm 1 is (/u1D700 1 + 2/u1D700 2 ) -DP for general queries and is (/u1D700 1 + /u1D700 2 ) -DP for monotonic queries.

Given that Expo noise is one-sided Lap noise, it can often be applied for comparative algorithms such as this one and report noisy max as well. We will show this extension in the appendix for completeness.

Corollary 3.1. If the noise addition is Expo then Algorithm 1 is (/u1D700 1 + 2/u1D700 2 ) -DP for general queries and (/u1D700 1 + /u1D700 2 ) -DP for monotonic queries.

While the proofs for Expo noise generally follow from the Lap noise proofs, it will require different techniques to show that Gumbel noise can be applied as well. In particular, we utilize the known connection between adding Gumbel noise and the exponential mechanism.

Lemma 3.2. If the noise addition is Gumbel and /u1D700 1 = /u1D700 2 then Algorithm 1 is (/u1D700 1 + 2/u1D700 2 ) -DP for general queries and (/u1D700 1 + /u1D700 2 ) -DP for monotonic queries.

We defer the proof of this to the appendix. In all of our empirical evaluations we will use Expo noise in our calls to AboveThreshold because it has the lowest variance for the same parameter. While we strongly believe that Expo noise will be most accurate under the same noise parameters, we leave a more rigorous examination to future work. We also note that this examination was implicitly done for report noisy max between Gumbel and Expo noise in McKenna and Sheldon (2020), where their algorithm is equivalent to adding Expo noise Ding et al. (2021), and Expo noise was shown to be clearly superior.

3.2 Improved privacy analysis for Sparse Vector Technique

In this section, we further consider the iterative application of AboveThreshold which is known as the sparse vector technique. We show that for monotonic queries, we can improve the privacy analysis of sparse vector technique to obtain better utility for the same level of privacy. Our primary metric for measuring privacy through composition will be zCDP which we defined in Section 2 and has become commonly used particularly due to the composition properties. We further show in the appendix that our analysis also enjoys improvement under the standard definition of differential privacy. These improved properties immediately apply to our unbounded quantile estimation algorithm as our queries will be monotonic.

Theorem 1. If the queries are monotonic, then for any noise addition of Lap , Gumbel , or Expo we have that Algorithm 1 is 1 2 /u1D700 2 -zCDP where /u1D700 = /u1D700 1 2 + /u1D700 2 . If the noise addition is Gumbel then we further require /u1D700 1 = /u1D700 2

Note that applying Proposition 1 will instead give /u1D700 = /u1D700 1 + /u1D700 2 . It will require further techniques to reduce this by /u1D700 1 /2 which will immediately allow for better utility with the same privacy guarantees. This bound also follows the intuitive factor of 2 improvement that is often expected for monotonic queries. The analysis will be achieved through providing a range-bounded property, a definition that was introduced in Durfee and Rogers (2019). This definition is ideally suited to characterizing the privacy loss of selection algorithms, by which we can view AboveThreshold . As such it will also enjoy the improved composition bounds upon the standard differential privacy definition shown in Dong et al. (2020). This range bounded property was then unified with zCDP with improved privacy guarantees in Cesar and Rogers (2021).

We give a proof of this theorem along with further discussion in Appendix B.3. We will also give a generalized characterization of when we can take advantage of properties of the queries to tighten the privacy bounds in AboveThreshold . We will immediately utilize this characterization to improve the privacy guarantees for our method in Appendix C.

4 Unbounded Quantile Estimation

In this section we give our method for unbounded quantile estimation. We focus upon the lower bounded setting which we view as most applicable to real-world problems, where non-negative

data is incredibly common, particularly for datasets that contain information about individuals. This method can be symmetrically apply to upper bounded data, and in the appendix we will show how this approach can be extended to the fully unbounded setting.

For the quantile problem we will assume our data /u1D465 ∈ ℝ /u1D45B . Given quantile /u1D45E ∈ [0, 1] and dataset /u1D465 ∈ ℝ /u1D45B the goal is to find /u1D461 ∈ ℝ such that |{/u1D465 /u1D457 ∈ /u1D465 |/u1D465 /u1D457 < /u1D461 }| is as close to /u1D45E /u1D45B as possible.

4.1 Unbounded quantile mechanim

The idea behind unbounded quantile estimation will be a simple guess-and-check method that will just invoke AboveThreshold . In particular, we will guess candidate values /u1D461 such that |{/u1D465 /u1D457 ∈ /u1D465 |/u1D465 /u1D457 < /u1D461 }| is close to /u1D45E /u1D45B . We begin with the smallest candidate, recall that we assume lower bounded data here and generalize later, and iteratively increase by a small percentage. At each iteration we check if |{/u1D465 /u1D457 ∈ /u1D465 |/u1D465 /u1D457 < /u1D461 }| has exceeded /u1D45E /u1D45B , and terminate when it does, outputting the most recent candidate. In order to achieve this procedure privately, we will simply invoke AboveThreshold . We also discuss how this procedure can be achieved efficiently in Section 4.2.

Thus we give our algorithm here where the lower bound of the data, /u1D4C1.alt , is our starting candidate and /u1D6FD is the scale at which the threshold increases. Given that we want our candidate value to increase by a small percentage it must start as a positive value. As such we will essentially just shift the data such that the lower bound is instead 1. In all our experiments we set /u1D6FD = 1.001 , so the increase is by 0.1% each iteration.

Algorithm 2 Unbounded quantile mechanism

Require: Input dataset /u1D465 , a quantile /u1D45E , a lower bound /u1D4C1.alt , and parameter /u1D6FD > 1

1: Run AboveThreshold with /u1D465 , /u1D447 = /u1D45E /u1D45B and /u1D453 /u1D456 (/u1D465 ) = |{/u1D465 /u1D457 ∈ /u1D465 |/u1D465 /u1D457 - /u1D4C1.alt + 1 < /u1D6FD /u1D456 }|

2: Output /u1D6FD /u1D458 + /u1D4C1.alt - 1 where /u1D458 is the query that AboveThreshold halted at

Given that our method simply calls AboveThreshold it will enjoy all the privacy guarantees from Section 3.1. Furthermore we will show that our queries are monotonic.

Lemma 4.1. For any sequence of thresholds {/u1D461 /u1D456 ∈ ℝ} let /u1D453 /u1D456 (/u1D465 ) = |{/u1D465 /u1D457 ∈ /u1D465 |/u1D465 /u1D457 < /u1D461 /u1D456 }| for all /u1D456 . For any neighboring dataset under Definition 2.1, we have that {/u1D453 /u1D456 } are monotonic queries with sensitivity 1.

Proof. Let /u1D465 /u1D457 be the value that differs between neighbors /u1D465 , /u1D465 ′ . Define  /u1D465 ,/u1D461 = {/u1D465 /u1D457 ∈ /u1D465 |/u1D465 /u1D457 < /u1D461 } . We consider the case /u1D465 ′ /u1D457 > /u1D465 /u1D457 and the other will follow symmetrically. For all thresholds /u1D461 /u1D456 ∈ (-∞, /u1D465 /u1D457 ] we have /u1D465 /u1D457 ∉  /u1D465 ,/u1D461 /u1D456 and /u1D465 ′ /u1D457 ∉  /u1D465 ′ ,/u1D461 /u1D456 , so /u1D453 /u1D456 (/u1D465 ) = /u1D453 /u1D456 (/u1D465 ′ ) . For all thresholds /u1D461 /u1D456 ∈ (/u1D465 ′ /u1D457 , ∞) we have /u1D465 /u1D457 ∈  /u1D465 ,/u1D461 /u1D456 and /u1D465 ′ /u1D457 ∈  /u1D465 ′ ,/u1D461 /u1D456 , so /u1D453 /u1D456 (/u1D465 ) = /u1D453 /u1D456 (/u1D465 ′ ) . Finally, for all thresholds /u1D461 /u1D456 ∈ (/u1D465 /u1D457 , /u1D465 ′ /u1D457 ] we have /u1D465 /u1D457 ∈  /u1D465 ,/u1D461 /u1D456 and /u1D465 ′ /u1D457 ∉  /u1D465 ′ ,/u1D461 /u1D456 , so /u1D453 /u1D456 (/u1D465 ) = /u1D453 /u1D456 (/u1D465 ′ ) + 1 . Therefore, /u1D453 /u1D456 (/u1D465 ) ≥ /u1D453 /u1D456 (/u1D465 ′ ) for all /u1D456 , and the sensitivity is 1.

Note that for the swap definition of neighboring the threshold remains constant. We will discuss how to extend our algorithm and further improve the privacy bounds for the add-subtract definition of neighboring in the appendix.

4.2 Simple and scalable implementation

In this section we show how our call to AboveThreshold can be done with a simple linear time pass through the data, and each subsequent query takes /u1D442 (1) time. While the running time could potentially be infinite, if we set /u1D6FD = 1.001 , then after 50,000 iterations our threshold is already over 10 21 and thus highly likely to have halted. Unless the scale of the data is absurdly high or the /u1D6FD value chosen converges to 1, our guess-and-check process will finish reasonably quickly. 2

In our initial pass through the data, for each data point /u1D465 /u1D457 we will find the index /u1D456 such that /u1D6FD /u1D456 ≤ /u1D465 /u1D457 - /u1D4C1.alt + 1 < /u1D6FD /u1D456+1 , which can be done by simply computing ⌊log /u1D6FD (/u1D465 /u1D457 - /u1D4C1.alt + 1)⌋ as our lower bound ensures /u1D465 /u1D457 - /u1D4C1.alt + 1 ≥ 1 . Using a dictionary or similar data structure we can efficiently store |{/u1D465 /u1D457 ∈ /u1D465 |/u1D6FD /u1D456 ≤ /u1D465 /u1D457 - /u1D4C1.alt + 1 < /u1D6FD /u1D456+1 }| for each /u1D456 with the default being 0. This preprocessing does not

require sorted data and takes /u1D442 (/u1D45B) arithmetic time, where we note that the previous algorithms also measure runtime arithmetically.

Finally, for each query if we already have |{/u1D465 /u1D457 ∈ /u1D465 |/u1D465 /u1D457 - /u1D4C1.alt + 1 < /u1D6FD /u1D456 }| , then we can add |{/u1D465 /u1D457 ∈ /u1D465 |/u1D6FD /u1D456 ≤ /u1D465 /u1D457 - /u1D4C1.alt + 1 < /u1D6FD /u1D456+1 }| in O(1) time to get |{/u1D465 /u1D457 ∈ /u1D465 |/u1D465 /u1D457 - /u1D4C1.alt + 1 < /u1D6FD /u1D456+1 }| . Inductively, each query will take O(1) time. We provide the code in the appendix for easier reproducibility.

4.3 Extension to multiple quantiles

The framework for computing multiple quantiles set up in Kaplan et al. (2022) is agnostic to the technique used for computing a single quantile. Their method will first compute the middle quantile and split the data according to the result. Through recursive application the number of levels of computation will be logarithmic. Furthermore, at each level we can see that at most one partition of the data will differ between neighbors, allowing for instead a logarithmic number of compositions. As such our approach can easily be applied to this recursive splitting framework to achieve the same improvements in composition. This will require some minor updating of their proofs to the swap definition that we will do in the appendix.

5 Empirical Evaluation

In this section we empirically evaluate our approach compared to the previous approaches. We give further detail of the previous approaches, particularly EMQ, in Appendix A along with strong intuition upon why our approach will better handle maximal quantile estimation for data clipping. We first go over the datasets and settings used for our experiments which will follow recent related work Gillenwater et al. (2021); Kaplan et al. (2022). Next we evaluate how accurately our method estimates quantiles for the different datasets in the bounded setting. Finally, we will consider the application of computing differentially private sum, which also gives mean computation, and show how our algorithm allows for a significantly more robust and accurate method when tight bounds are not known for the dataset.

5.1 Datasets

We borrow the same setup and datasets as Gillenwater et al. (2021); Kaplan et al. (2022). We test our algorithm compared to the state-of-the-art on six different datasets. Two datasets will be synthetic. One draws 10,000 data points from the uniform distribution in the range [-5, 5] and the other draws 10,000 data points from the normal distribution with mean zero and standard deviation of five. Two datasets will come from Soumik (2019) with 11,123 data points, where one has book ratings and the other has book page counts. Two datasets will come from Dua and Graf (2019) with 48,842 data points, where one has the number of hours worked per week and the other has the age for different people. We provide histograms of our datasets for better understanding in Figure 1.

5.2 Quantile estimation experiments

For our quantile estimation experiments, for a given quantile /u1D45E ∈ [0, 1] we consider the error of outcome /u1D45C /u1D45E from one of the private methods to be |/u1D45C /u1D45E -/u1D461 /u1D45E | where /u1D461 /u1D45E is the true quantile value. We use the in-built quantile function in the numpy library with the default settings to get the true quantile value. As in previous related works, we randomly sample 1000 datapoints from each dataset and run the quantile computation on each method. This process is then iterated upon 100 times and the error is averaged. We set /u1D700 = 1 as in the previous works, which will require setting /u1D700 1 = /u1D7002 = 1/2 for the call to AboveThreshold in our method.

We will also tighten the ranges to the following, [-5, 5] for the uniform dataset, [-25, 25] for the normal dataset, [0, 10] for the ratings dataset, [0, 10000] for the pages dataset, [0, 100] for the hours dataset, and [0, 100] for the ages dataset. Given that EMQ suffers performance when many datapoints are equal we add small independent noise to our non-sythetic datasets. This noise will be from the normal distribution with standard deviation 0.001 for the ratings dataset and 0.1 for the other three that have integer values. Our method does not require the noise addition but we will use the perturbed dataset for fair comparison. True quantiles are still computed upon the original data. For the datasets with integer values we rounded each output to the nearest integer. For our method we set /u1D6FD = 1.001 for all datasets.

(a) Uniform

(c) Book ratings

(d) Book pages

(e) Census ages

(f) Census hours

350

120

300

800

100

250

80

600

60

150

400

40

100

200

20

50

20

15

-10

15

20

1400

1750

1500

1200

20000

1000

1250

15000

800

1000

750

600

10000

500

400

5000

250

200

1000

2000

3000

4000

5000

6000

0

20

40

60

80

100

20

40

60

80

100

For these experiments we only compare our method UQE and the previous method EMQ, using the implementations from Gillenwater et al. (2021). The other procedures, discussed in the appendix are more generalized and thus for this specific setting do not perform nearly as well which can be seen in the previous experiments Gillenwater et al. (2021); Kaplan et al. (2022), so we omit them from our results. For this experiment we consider estimating each quantile from 5% to 95% at a 1% interval. In Figure 2 we plot the mean absolute error of each normalized by the mean absolute error of UQE to make for an easier visualization.

Uniform data

Gaussiandata

Bookratingsdata

1.00

1.05

30n

EMQ

1.4

UQE

EMQ

1.00

S60

S60

1.3

Relative Error

060

1.2

Relative

0.85

S80

1.1

08°0

0.80

0.75

1.0

0.75

UQE

0.70

60

EMQ

0.65

80

20

40

60

80

20

40

60

80

20

40

60

80

saqueno

Bookpagesdata

Agedata

Workhoursdata

UQE

UQE

UQE

EMQ

1.3

EMQ

12

EMQ

1.4

1.2

10

Relative Error

1.2

Relative Error

Relative Error

8

1.1

1.0

1.0

0.8

60

0.6

8'0

20

40

60

80

20

40

60

80

20

40

60

80

As we can see in Figure 2, EMQ consistently performs better on synthetic data, and UQE tends to performs better on the non-synthetic data. This fits with our intuition that UQE will be best suited to situations where the data is unstructured and less is known about the dataset beforehand because our guess-and-check methodology is designed to better handle ill-behaving datasets.

5.3 Sum estimation experiments

As the primary application of our method we will also be considering differentially private sum computation, which can thereby compute mean as well. We will be using the following 2/u1D700 -DP general procedure for computing the sum of non-negative data:

Let Δ = PrivateQuantile (/u1D465 , /u1D45E , /u1D700 ) where PrivateQuantile is any differentially private computation algorithm and /u1D45E ≈ 1 .

Output Lap (Δ//u1D700) + ∑ /u1D45B /u1D457=1 min(/u1D465 /u1D457 , Δ)

We further test this upon the non-synthetic datasets. For large-scale privacy systems that provide statistical analysis for a wide variety of datasets, if we use a PrivateQuantile that requires an upper bound then we ideally want this bound to be agnostic to the dataset. The is particularly true for sum computations upon groupby queries as the range and size can differ substantially amongst groups. As such, we fix the range at [0, 10000] to encompass all the datasets. We will otherwise use the same general setup as in Section 5.2.

We will measure the error of this procedure as the absolute error of the output and the true sum of the dataset. For each of the 100 iterations of choosing 1000 samples randomly from the full dataset, we also add Lap noise 100 times. Averaging over all these iterations gives our mean absolute error.

Table 1: Mean absolute error for differentially private sum estimation. The standard deviation over the 100 iterations is also provided for each in the subscript. UQE = Our unbounded quantile estimation method. EMQ = The exponential mechanism based quantile estimation method. AT = The aggregate tree method for quantile estimation. For our method we only use /u1D45E = 0.99 . For the others we use the best performance for /u1D45E ∈ {0.95, 0.96, 0.97, 0.98, 0.99} .

We will run this procedure with /u1D700 ∈ {0.1, 0.5, 1} . Further we will use /u1D45E = 0.99 always for our method, but give the absolute error for the best performing /u1D45E ∈ {0.95, 0.96, 0.97, 0.98, 0.99} for the other methods. It is important to note that this value would have to be chosen ahead of time, which would add more error to the other methods. The previous methods we consider here are again the EMQ, but also the aggregate tree (AT) methods, were we use both the implementation along with generally best performing height (3) and branching factor (10) from Gillenwater et al. (2021). We also implemented the bounding technique using inner quartile range within Algorithm 1 of Smith (2011), but this performed notably worse than the others so we omitted the results from our table.

As we can see in Table 1, our method is far more robust and accurate. Furthermore, for our method the choice of /u1D45E remained constant and we can see that our results still stayed consistently accurate when /u1D700 changed. Note that the noise added to the clipped sum is also scaled proportional to /u1D700 so the amount the error increased as /u1D700 decreased for our method is what would be expected proportionally. Once again these findings are consistent with our intuition. Our technique is more robust to differing datasets and privacy parameters, and especially better performing for this important use case.

Recall that the sampled data had size 1000 so dividing accordingly can give the error on mean estimates. There is a long line of literature on differentially private mean estimation. 3 To our knowledge, all of these more complex algorithms either require assumptions upon the data distribution,

such as sub-Gaussian or bounded moments, or bounds upon the data range or related parameters, and most often require both. These results also focus upon proving strong theoretical guarantees of accuracy with respect to asymptotic sample complexity. We first note that our approach will provide better initial bounds upon the data as seen in our experiments, which directly improve the theoretical guarantees in the results that require a data range. But also our focus here is upon practical methods that are agnostic to data distributions and more widely applicable to real-world data. Consequently, a rigorous comparison among all of these methods would be untenable and outside the scope of this work.

5.4 Parameter tuning

We kept our /u1D6FD parameter fixed in all experiments for consistency but also to make our method data agnostic. However, our choice was aggressively small in order to achieve higher precision in the inner quantile estimation comparison in Section 5.2. This choice was still highly resilient to changes in /u1D700 for our sum experiments as we see our error only scaled proportional to the increase in noise. But for more significant decreases in /u1D700 or in the data size, i.e. the conditions under which all private algorithms suffer substantial accuracy loss, this choice of /u1D6FD could be too small. Those settings imply that the noise added is larger and the distance between queries and threshold shrinks, so our method is more likely to terminate earlier than desired. Smaller /u1D6FD values will then intensify this issue as the candidate values increase more slowly. For the clipping application, we generally think using a value of /u1D6FD = 1.01 would be a more stable choice. In fact, replicating our empirical testing with this value actually improves our results in Table 1. Furthermore, increasing /u1D6FD will also reduce the number of queries and thus the computational cost. In general, we find that setting /u1D6FD ∈ [1.01, 1.001] gives good performance with /u1D6FD = 1.01 as a default for computational efficiency. We leave to future work a more thorough analysis of this parameter to fully optimize setting it with relation to data size and privacy parameter. Additionally, all our methods and proofs only require an increasing sequence of candidate values and it's possible that other potential sequences would be even more effective. For example, if tight upper and lower bounds are known on the data, such as within the recursive computation of multiple quantiles, then it likely makes more sense to simply uniformly partition the interval and check in increasing order. But we leave more consideration upon this to future work as well.

Our choice of /u1D45E = 0.99 in the differentially private sum experiments was also to maintain consistency with the choices for the previous method but also to keep variance of the estimates lower. This will create some negative bias as we then expect to clip the data. As we can see in our illustrative example Figure 3, the PDF will exponentially decrease once we pass the true quantile value, but will do so less sharply once all the queries have value /u1D45B . Accordingly, setting /u1D45E = 1.0 would add slightly more variance to the estimation but initial testing showed improvement in error. However, if the user would prefer slightly higher variance to avoid negative bias, then setting the threshold at /u1D45B or even /u1D45B +1//u1D700 , would make it far more likely that the process terminates with a value slightly above the maximum. This is particularly useful for heavy-tailed data, where clipping at the 99th percentile can have an out-sized impact on the bias.

Acknowledgments and Disclosure of Funding

We thank our colleagues Matthew Clegg, James Honaker, and Kenny Leftin for helpful discussions and feedback. We also thank anonymous reviewers for their helpful feedback.

References

Alabi, D., Ben-Eliezer, O., and Chaturvedi, A. (2022). Bounded space differentially private quantiles. arXiv preprint arXiv:2201.03380 .

Barthe, G., Fong, N., Gaboardi, M., Grégoire, B., Hsu, J., and Strub, P.-Y. (2016a). Advanced probabilistic couplings for differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security , pages 55-67.

Barthe, G., Gaboardi, M., Grégoire, B., Hsu, J., and Strub, P.-Y. (2016b). Proving differential privacy via probabilistic couplings. In Proceedings of the 31st Annual ACM/IEEE Symposium on Logic in Computer Science , pages 749-758.

Bassily, R., Thakkar, O., and Guha Thakurta, A. (2018). Model-agnostic private learning. Advances in Neural Information Processing Systems , 31.

Blum, A., Morgenstern, J., Sharma, A., and Smith, A. (2015). Privacy-preserving public information for sequential games. In Proceedings of the 2015 Conference on Innovations in Theoretical Computer Science , pages 173-180.

Bun, M., Nissim, K., Stemmer, U., and Vadhan, S. (2015). Differentially private release and learning of threshold functions. In 2015 IEEE 56th Annual Symposium on Foundations of Computer Science , pages 634-649. IEEE.

Bun, M. and Steinke, T. (2016). Concentrated differential privacy: Simplifications, extensions, and lower bounds. In Theory of Cryptography: 14th International Conference, TCC 2016-B, Beijing, China, October 31-November 3, 2016, Proceedings, Part I , pages 635-658. Springer.

Bun, M., Steinke, T., and Ullman, J. (2017). Make up your mind: The price of online queries in differential privacy. In Proceedings of the twenty-eighth annual ACM-SIAM symposium on discrete algorithms , pages 1306-1325. SIAM.

Cesar, M. and Rogers, R. (2021). Bounding, concentrating, and truncating: Unifying privacy loss composition for data analytics. In Algorithmic Learning Theory , pages 421-457. PMLR.

Chan, T.-H. H., Shi, E., and Song, D. (2011). Private and continual release of statistics. ACM Transactions on Information and System Security (TISSEC) , 14(3):1-24.

Chen, Y., Machanavajjhala, A., Reiter, J. P., and Barrientos, A. F. (2016). Differentially private regression diagnostics. In ICDM , pages 81-90.

Cummings, R., Kearns, M., Roth, A., and Wu, Z. S. (2015). Privacy and truthful equilibrium selection for aggregative games. In Web and Internet Economics: 11th International Conference, WINE 2015, Amsterdam, The Netherlands, December 9-12, 2015, Proceedings 11 , pages 286-299. Springer.

Cummings, R., Krehbiel, S., Lut, Y., and Zhang, W. (2020). Privately detecting changes in unknown distributions. In International Conference on Machine Learning , pages 2227-2237. PMLR.

Ding, Z., Kifer, D., Steinke, T., Wang, Y., Xiao, Y., Zhang, D., et al. (2021). The permute-and-flip mechanism is identical to report-noisy-max with exponential noise. arXiv preprint arXiv:2105.07260 .

Ding, Z., Wang, Y., Xiao, Y., Wang, G., Zhang, D., and Kifer, D. (2023). Free gap estimates from the exponential mechanism, sparse vector, noisy max and related algorithms. The VLDB Journal , 32(1):23-48.

Dong, J., Durfee, D., and Rogers, R. (2020). Optimal differential privacy composition for exponential mechanisms. In International Conference on Machine Learning , pages 2597-2606. PMLR.

Dua, D. and Graf, C. (2019). Uci machine learning repository. In http://archive.ics.uci.edu/ ml .

Durfee, D. and Rogers, R. M. (2019). Practical differentially private top-k selection with pay-whatyou-get composition. Advances in Neural Information Processing Systems , 32.

Dwork, C., Feldman, V., Hardt, M., Pitassi, T., Reingold, O., and Roth, A. L. (2015). Preserving statistical validity in adaptive data analysis. In Proceedings of the forty-seventh annual ACM symposium on Theory of computing , pages 117-126.

Dwork, C., Kenthapadi, K., McSherry, F., Mironov, I., and Naor, M. (2006a). Our data, ourselves: Privacy via distributed noise generation. In Advances in Cryptology-EUROCRYPT 2006: 24th Annual International Conference on the Theory and Applications of Cryptographic Techniques, St. Petersburg, Russia, May 28-June 1, 2006. Proceedings 25 , pages 486-503. Springer.

Dwork, C., McSherry, F., Nissim, K., and Smith, A. (2006b). Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3 , pages 265-284. Springer.

Dwork, C., Naor, M., Pitassi, T., and Rothblum, G. N. (2010). Differential privacy under continual observation. In Proceedings of the forty-second ACM symposium on Theory of computing , pages 715-724.

Dwork, C., Naor, M., Reingold, O., Rothblum, G. N., and Vadhan, S. (2009). On the complexity of differentially private data release: efficient algorithms and hardness results. In Proceedings of the forty-first annual ACM symposium on Theory of computing , pages 381-390.

Feldman, V. and Steinke, T. (2017). Generalization for adaptively-chosen estimators via stable median. In Conference on learning theory , pages 728-757. PMLR.

Gillenwater, J., Joseph, M., and Kulesza, A. (2021). Differentially private quantiles. In International Conference on Machine Learning , pages 3713-3722. PMLR.

Hardt, M. and Rothblum, G. N. (2010). A multiplicative weights mechanism for privacy-preserving data analysis. In 2010 IEEE 51st annual symposium on foundations of computer science , pages 61-70. IEEE.

Hasidim, A., Kaplan, H., Mansour, Y., Matias, Y., and Stemmer, U. (2020). Adversarially robust streaming algorithms via differential privacy. Advances in Neural Information Processing Systems , 33:147-158.

Hsu, J., Roth, A., and Ullman, J. (2013). Differential privacy for the analyst via private equilibrium computation. In Proceedings of the forty-fifth annual ACM symposium on Theory of computing , pages 341-350.

Kamath, G., Mouzakis, A., Singhal, V., Steinke, T., and Ullman, J. (2022). A private and computationally-efficient estimator for unbounded gaussians. In Conference on Learning Theory , pages 544-572. PMLR.

Kaplan, H., Ligett, K., Mansour, Y., Naor, M., and Stemmer, U. (2020a). Privately learning thresholds: Closing the exponential gap. In Conference on Learning Theory , pages 2263-2285. PMLR.

Kaplan, H., Mansour, Y., and Stemmer, U. (2021). The sparse vector technique, revisited. In Conference on Learning Theory , pages 2747-2776. PMLR.

Kaplan, H., Schnapp, S., and Stemmer, U. (2022). Differentially private approximate quantiles. In International Conference on Machine Learning , pages 10751-10761. PMLR.

Kaplan, H., Sharir, M., and Stemmer, U. (2020b). How to find a point in the convex hull privately. SoCG .

Ligett, K., Neel, S., Roth, A., Waggoner, B., and Wu, S. Z. (2017). Accuracy first: Selecting a differential privacy level for accuracy constrained erm. Advances in Neural Information Processing Systems , 30.

Lyu, M., Su, D., and Li, N. (2017). Understanding the sparse vector technique for differential privacy. Proceedings of the VLDB Endowment , 10(6).

McKenna, R. and Sheldon, D. R. (2020). Permute-and-flip: A new mechanism for differentially private selection. Advances in Neural Information Processing Systems , 33:193-203.

McSherry, F. and Talwar, K. (2007). Mechanism design via differential privacy. In 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS'07) , pages 94-103. IEEE.

Nandi, A. and Bassily, R. (2020). Privately answering classification queries in the agnostic pac model. In Algorithmic Learning Theory , pages 687-703. PMLR.

Nissim, K., Raskhodnikova, S., and Smith, A. (2007). Smooth sensitivity and sampling in private data analysis. In Proceedings of the thirty-ninth annual ACM symposium on Theory of computing , pages 75-84.

Nissim, K. and Stemmer, U. (2018). Clustering algorithms for the centralized and local models. In Algorithmic Learning Theory , pages 619-653. PMLR.

Nissim, K., Stemmer, U., and Vadhan, S. (2016). Locating a small cluster privately. In Proceedings of the 35th ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems , pages 413-427.

Roth, A. and Roughgarden, T. (2010). Interactive privacy via the median mechanism. In Proceedings of the forty-second ACM symposium on Theory of computing , pages 765-774.

Sajed, T. and Sheffet, O. (2019). An optimal private stochastic-mab algorithm based on optimal private stopping rule. In International Conference on Machine Learning , pages 5579-5588. PMLR.

Shokri, R. and Shmatikov, V. (2015). Privacy-preserving deep learning. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security , pages 1310-1321.

Smith, A. (2011). Privacy-preserving statistical estimation with optimal convergence rates. In Proceedings of the forty-third annual ACM symposium on Theory of computing , pages 813-822.

Soumik (2019). Goodreads-books dataset. In https://www.kaggle.com/jealousleopard/ goodreadsbooks .

Steinke, T. and Ullman, J. (2016). Between pure and approximate differential privacy. Journal of Privacy and Confidentiality .

Ullman, J. (2015). Private multiplicative weights beyond linear queries. In Proceedings of the 34th ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems , pages 303-312.

A Maximal Quantiles Estimation

In this section, we specifically consider applying our unbounded quantile estimation to computing the highest quantiles. This is commonly needed for clipping the dataset to privately compute the sum and mean, which we empirically test in Section 5.3. Our primary goal here is to build intuition upon why our approach fundamentally gives more accurate and robust estimations of the highest quantiles when only loose upper bounds on the data are known. We first give more details upon the previous methods, particularly the EMQ method. Then we give a more detailed look at how these methods are negatively affected by loose upper bounds and why ours performs well in comparison.

A.1 Previous techniques

The most effective and practical previous method for quantile estimation, EMQ, builds a distribution over an assumed bounded range [/u1D44E, /u1D44F] through an invocation of the exponential mechanism. Assuming the data is sorted, it will partition the range based upon the data and select an interval [/u1D465 /u1D457 , /u1D465 /u1D457+1 ] with probability proportional to

where the intervals [/u1D44E, /u1D465 1 ] and [/u1D465 /u1D45B , /u1D44F] are also considered. A uniformly random point is then drawn from within the selected interval. Note that this utility function is not monotonic and will not enjoy the same improved privacy bounds.

There are other approaches that recursively partition the data range while computing statistics on each partition, then aggregate these statistics across logarithmic levels to reduce the noise for quantile computation Dwork et al. (2010); Chan et al. (2011). Another technique is to instead utilize the local sensitivity while maintaining privacy by using a smoothing procedure that can also be used for computing quantiles Nissim et al. (2007). There is also a similar approach to ours within Chen et al. (2016) for bounding an unbounded dataset that increases the bounds by a factor of 2 each iteration but we will discuss in Section A.2 why this variant performs substantially worse.

A.2 Effect of bounds upon maximum quantiles

As mentioned, EMQ will construct a probability distribution over the range [/u1D44E, /u1D44F] . The corresponding PDF is a unimodal step function of the intervals defined by the data with the peak interval [/u1D465 /u1D457 , /u1D465 /u1D457+1 ] being such that /u1D457 closest to /u1D45E /u1D45B . Note then that if /u1D44F >> /u1D465 /u1D45B then the probability of selecting [/u1D465 /u1D45B , /u1D44F] increases dramatically. The exponential decay as the PDF moves away from the peak interval diminishes the impact of [/u1D465 /u1D45B , /u1D44F] significantly if /u1D45B is far away from /u1D45E /u1D45B . However, for computing the highest quantiles, we want /u1D45E close to 1 by definition, and the looseness of the upper bound drastically effects the accuracy.

In contrast, as soon as the candidate value for our method exceeds the true quantile value, each successive query will have an output of at least /u1D45E /u1D45B which is the threshold. The probability of continuing for /u1D706 more queries then decreases in /u1D706 . 4 For ease of comparison, we can modify our Algorithm 2 such that in the last step, the output is drawn uniformly from [/u1D6FD /u1D458 -1 , /u1D6FD /u1D458 ] , assuming /u1D4C1.alt = 1 for simplicity. This would then create a continuous probability distribution that would similarly be a step function with each interval [/u1D6FD /u1D458 -1 , /u1D6FD /u1D458 ] being a step.

For better intuition we plot the approximate PDF of each in Figure 3. We use the Gumbel noise in the call to AboveThreshold to take advantage of the closed form expression in Lemma B.2 for easier PDF computation.

As we can see in Figure 3, the upper bound, /u1D44F , increasing from 10 to 20 dramatically increases the probability of selecting a point in [/u1D465 /u1D45B , /u1D44F] which changes the normalization for the other intervals, significantly altering the PDF of EMQ. Given that our method is unbounded, the PDF for UQE stays

Loose bounds

Tight bounds

UQE PDF

UQE PDF

0.30

EMQ PDF

0.4

EMQ PDF

0.25

Probability density

Probability density

0.3

0.20

0.15

0.2

0.10

0.05

0.00

0.0

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Quantile output

Quantile output

the same in both figures and sees the expected exponential decay once the candidate quantile passes the true quantile.

There are also alternative methods for bounding the data range by computing the interquartile range more accurately and scaling up. However these methods make strong assumptions upon the data distribution being close to the normal distribution as well as bounds upon the third moment Smith (2011). For real-world datasets, the tails of the data can vary significantly, and scaling up in a data-agnostic manner can often be similarly inaccurate. We also found this to be true in our experiments.

The smooth sensitivity framework will run into similar issues when /u1D45E is closer to 1 because fewer data points need to be changed to push the quantile value to the upper bound. If this upper bound is large, then the smooth local sensitivity will still be substantial. Aggregate tree methods are slightly more robust to loose upper bounds as they aren't partitioning specific to the data. However, for accurate estimates the partitioning of the range still requires the data be well-distributed across the evenly split partitions, which is significantly affected by loose upper bounds.

B Improved analysis for Sparse Vector Technique

In this section we complete the analysis for improving the privacy bounds for sparse vector technique with monotonic queries. We will first give a more generalized characterization of query properties by which we can improve the privacy bounds for AboveThreshold . While this characterization is more complex, we will immediately utilize it in Section C to improve the privacy bounds of our methods under an alternate common definition of neighboring. Additionally, the class of queries that we apply it to are not monotonic, so improvements can be made in a multitude of settings. Next we will review the definition of range-bounded, a property of privacy mechanisms that can be used to improve composition bounds, and apply it to our setting. Finally we will utilize these results to show that the privacy analysis of the sparse vector technique can be improved more generally, but also specifically for monotonic queries.

B.1 Generalized characterization

We provide a more general characterization for the privacy loss of AboveThreshold that is specific to the input stream of queries. This will be achieved by providing a one-sided privacy parameter for each pair of neighboring datasets where order matters. For a given pair /u1D465 , /u1D465 ′ , our goal will be to upper bound the output distribution of /u1D465 by the output distribution of /u1D465 ′ up to an exponential factor of the one-sided privacy parameter. The specificity of the parameter to the neighboring datasets will reduce conciseness but it is for this reason that we will be able to further improve privacy analysis. This precise characterization will also help improve bounds for our method in Section C without requiring onerous analysis.

Definition B.1. For a stream of queries {/u1D453 /u1D456 } with sensitivity Δ , we define our one-sided privacy loss for neighboring data sets /u1D465 , /u1D465 ′ as

where Δ/u1D458 (/u1D465 , /u1D465 ′ ) = max/u1D456</u1D458 max{0, /u1D453 /u1D456 (/u1D465 ′ ) - /u1D453 /u1D456 (/u1D465 )} .

Given that our definition is meant to encompass the one-sided privacy loss for AboveThreshold we will now prove that fact for Expo and Gumbel noise (and Lap follows equivalently to Expo ). We first prove this conjecture for Expo noise.

Lemma B.1. For a stream of queries {/u1D453 /u1D456 } with sensitivity Δ , threshold /u1D447 and neighboring data sets /u1D465 , /u1D465 ′ , if we run Algorithm 1 with Expo noise, then for any given outcome {⊥ /u1D458-1 , ⊤} we have

Proof. Let /u1D708 /u1D456 ∼ Expo (Δ//u1D700 2 ) denote the noise drawn for query /u1D453 /u1D456 , and let /u1D708 ∼ Expo (Δ//u1D700 1 ) denote the noise drawn for the threshold. Further let /u1D708 /u1D456≤/u1D458 denote all /u1D708 /u1D456 such that /u1D456 ≤ /u1D458 . By construction, we have

We will fix the randomness of /u1D708 /u1D456</u1D458 , denoting /u1D708 /u1D456 such that /u1D456 < /u1D458 , for datasets /u1D465 and /u1D465 ′ , and let /u1D70F = max/u1D456</u1D458 /u1D453 /u1D456 (/u1D465 ) + /u1D708 /u1D456 and /u1D70F ′ = max/u1D456</u1D458 /u1D453 /u1D456 (/u1D465 ′ ) + /u1D708 /u1D456 . It suffices then to show that for any fixed randomness /u1D708 /u1D456</u1D458 we have

Let Δ1 = max{0, /u1D70F ′ - /u1D70F} and Δ2 = max{0, Δ1 - (/u1D453 /u1D458 (/u1D465 ′ ) - /u1D453 /u1D458 (/u1D465 ))} and let /u1D708 ′ = /u1D708 + Δ1 and /u1D708 ′ /u1D458 = /u1D708 /u1D458 + Δ2 . It suffices to show that for every pair of draws /u1D708, /u1D708 /u1D458 that satisfies /u1D70F < /u1D447 + /u1D708 < /u1D453 /u1D458 (/u1D465 ) + /u1D708 /u1D458 , then the corresponding /u1D708 ′ , /u1D708 ′ /u1D458 are i) non-negative, ii) satisfy /u1D70F ′ < /u1D447 + /u1D708 ′ < /u1D453 /u1D458 (/u1D465 ′ ) + /u1D708 ′ /u1D458 and iii)

We note that condition iii) does not require any scalar on the RHS as our change of variable is a shift for each so the Jacobian of the mapping is the identity matrix with determinant of 1.

Condition i) follows from the fact that /u1D708, /u1D708 /u1D458 are non-negative because they're drawn from the exponential distribution, and Δ1, Δ2 are non-negative by construction.

For condition ii), if /u1D70F < /u1D447 + /u1D708 we must have /u1D70F ′ < /u1D447 + /u1D708 + Δ1 by definition of Δ1 . Similarly, if /u1D447 + /u1D708 < /u1D453 /u1D458 (/u1D465 ) + /u1D708 /u1D458 then /u1D447 + /u1D708 + Δ1 < /u1D453 /u1D458 (/u1D465 ′ ) + /u1D708 /u1D458 + Δ2 because Δ2 ≥ Δ1 - (/u1D453 /u1D458 (/u1D465 ′ ) - /u1D453 /u1D458 (/u1D465 )) .

For condition iii), the PDF of the exponential distribution implies /u1D45D Expo (/u1D708; Δ//u1D700 1 ) ≤ exp( /u1D700 1 Δ 1 Δ )/u1D45D Expo (/u1D708 ′ ; Δ//u1D700 1 ) and /u1D45D Expo (/u1D708 /u1D458 ; Δ//u1D700 2 ) ≤ exp( /u1D700 2 Δ 2 Δ )/u1D45D Expo (/u1D708 ′ /u1D458 ; Δ//u1D700 2 ) . Due to the fixing of randomness of /u1D708 /u1D456</u1D458 , we have that /u1D70F ′ - /u1D70F ≤ max/u1D456</u1D458 (/u1D453 /u1D456 (/u1D465 ′ ) - /u1D453 /u1D456 (/u1D465 )) so Δ1 ≤ Δ/u1D458 (/u1D465 , /u1D465 ′ ) from Definition B.1 and Δ2 ≤ max{0, Δ/u1D458 (/u1D465, /u1D465 ′ ) - (/u1D453 /u1D458 (/u1D465 ′ ) - /u1D453 /u1D458 (/u1D465 ))} which implies condition iii).

This proof then easily applies to Lap as well. The proof for Gumbel will differ though as we no longer have similar closeness of PDF properties, but it will follow from the fact that we can give a closed form expression for each probability.

Lemma B.2. Given a dataset /u1D465 , a stream of queries {/u1D453 /u1D456 } and threshold /u1D447 , for any outcome {⊥ /u1D458-1 , ⊤} from running AboveThreshold with Gumbel noise and /u1D700 = /u1D700 1 = /u1D700 2 , then

Proof. We add noise Gumbel (Δ//u1D700) to all /u1D453 /u1D456 (/u1D465 ) and the threshold /u1D447 , and in order to output {⊥ /u1D458-1 , ⊤} we must have that for the first /u1D458 queries, the noisy value of /u1D453 /u1D458 (/u1D465 ) is the largest and the noisy threshold is the second largest. It is folklore in the literature that adding Gumbel noise (with the same noise parameter) and choosing the largest index is equivalent to the exponential mechanism. This can also be extended to showing that adding Gumbel noise and choosing the top-k indices in order is equivalent to the peeling exponential mechanism. The peeling exponential mechanism first selects the top index using the exponential mechanism and removes it from the candidate set, repeating iteratively until accumulating the top-k indices. A formal proof of this folklore result is also provided in Lemma 4.2 of Durfee and Rogers (2019). Applying this result and the definition of the exponential mechanism gives the desired equality.

We then utilize this closed form expression to get our desired one-sided privacy bounds when using Gumbel noise.

Lemma B.3. For a stream of queries {/u1D453 /u1D456 } with sensitivity Δ , threshold /u1D447 and neighboring data sets /u1D465 , /u1D465 ′ , if we run Algorithm 1 with Gumbel noise with /u1D700 1 = /u1D700 2 , then for any outcome {⊥ /u1D458-1 , ⊤} we have

Proof. Let /u1D700 = /u1D700 1 = /u1D700 2 . From Lemma B.2 we know that

Similar to the proof of Lemma B.1, we let Δ1 = max{0, max/u1D456</u1D458 (/u1D453 /u1D456 (/u1D465 ′ ) - /u1D453 /u1D456 (/u1D465 ))} and Δ2 = max{0, Δ1 + (/u1D453 /u1D458 (/u1D465 ) - /u1D453 /u1D458 (/u1D465 ′ ))} . We first want to show that

This inequality reduces to showing exp( /u1D716 Δ /u1D447 )+∑ /u1D458-1 /u1D456=1 exp( /u1D716 Δ /u1D453 /u1D456 (/u1D465 ′ )) ≤ /u1D452 /u1D716Δ 1 Δ (exp( /u1D716 Δ /u1D447 )+∑ /u1D458-1 /u1D456=1 exp( /u1D716 Δ /u1D453 /u1D456 (/u1D465 ))) . This holds from the fact that exp( /u1D716 Δ /u1D447 ) ≤ exp(/u1D716Δ 1 /Δ) exp( /u1D716 Δ /u1D447 ) because /u1D716Δ1 /Δ ≥ 0 and exp( /u1D716 Δ /u1D453 /u1D456 (/u1D465 ′ )) ≤ exp(/u1D716Δ 1 /Δ) exp( /u1D716 Δ /u1D453 /u1D456 (/u1D465 )) for all /u1D456 < /u1D458 by our definition of Δ1 .

First we let Δ ′ 1 = max{0, max/u1D4C1.alt≤/u1D458 (/u1D453 /u1D4C1.alt (/u1D465 ′ ) - /u1D453 /u1D4C1.alt (/u1D465 ))} and Δ ′ 2 = /u1D453 /u1D458 (/u1D465 ) - /u1D453 /u1D458 (/u1D465 ′ ) , and we will achieve our desired inequality (2) by bounding the numerator with respect to Δ ′ 2 and the denominator with respect to Δ ′ 1 . We have exp( /u1D716 Δ /u1D453 /u1D458 (/u1D465 )) = exp(/u1D716Δ ′ 2 /Δ) exp( /u1D716 Δ /u1D453 /u1D458 (/u1D465 ′ )) by definition of Δ ′ 2 . Further exp( /u1D716 Δ /u1D453 /u1D456 (/u1D465 ′ )) ≤ exp(/u1D716Δ ′ 1 /Δ) exp( /u1D716 Δ /u1D453 /u1D456 (/u1D465 )) by definition of Δ ′ 1 , so

because /u1D716Δ ′ 1 /Δ ≥ 0 . In order to show our desired inequality (2) it then suffices to show Δ2 ≥ Δ ′ 1 +Δ ′ 2 . By definition we know Δ2 ≥ Δ1 + Δ ′ 2 . Furthermore, if Δ ′ 1 > Δ1 then Δ ′ 1 = /u1D453 /u1D458 (/u1D465 ′ ) - /u1D453 /u1D458 (/u1D465 ) = -Δ ′ 2 which implies Δ ′ 1 + Δ ′ 2 = 0 , and we know Δ2 ≥ 0 by construction.

Combining inequality (1) and (2) along with the fact that Δ1 = Δ/u1D458 (/u1D465 , /u1D465 ′ ) from Definition B.1 completes the proof.

Now that we've bounded the one-sided privacy loss for neighboring datasets for each type of noise, this will immediately imply an overall privacy bound over all neighbors that utilizes this general characterization.

Corollary B.1. For a stream of queries {/u1D453 /u1D456 } with sensitivity Δ and threshold /u1D447 , Algorithm 1 with Lap , Gumbel , or Expo noise is /u1D700 -DP where

and for Gumbel noise we must have /u1D700 1 = /u1D700 2 .

B.2 Range-bounded definition and properties

The definition of range-bounded was originally introduced in Durfee and Rogers (2019) to tighten the privacy bounds on the composition of exponential mechanisms. The analysis was further extended in Dong et al. (2020) to give the optimal composition of range-bounded mechanisms. This definition was then unified with other definitions in Cesar and Rogers (2021).

Definition B.2 (Durfee and Rogers (2019)) . A mechanism /u1D440 ∶  →  is /u1D700 -range-bounded if for any neighboring datasets /u1D465 , /u1D465 ′ ∈  and outcomes /u1D466 , /u1D466 ′ ∈  :

If we have bounds upon this property that are stronger than those immediately implied by DP, then it was shown in Dong et al. (2020) that substantial improvements can be made in bounding the overall DP over the composition of these mechanisms. While these improvements can be applied to our results here as well, we focus upon the privacy bounds with respect to zCDP as it is much cleaner to work with in providing strong composition bounds. From the proof of Lemma 3.4 in Cesar and Rogers (2021) we see that the range-bounded property can give a significant improvement in zCDP properties compared to similar DP guarantees.

Proposition 2 (Cesar and Rogers (2021)) . If a mechanism /u1D440 is /u1D700 -range-bounded then it is 1 8 /u1D700 2 -zCDP.

Our general characterization is already set up to provide range-bounded properties of AboveThreshold . This will allow us to give tighter guarantees on the range-boundedness which can be translated to better privacy composition bounds.

Corollary B.2. For a stream of queries {/u1D453 /u1D456 } and threshold /u1D447 , Algorithm 1 with Lap , Gumbel , or Expo noise is /u1D700 -range-bounded where

and for Gumbel noise we must have /u1D700 1 = /u1D700 2 .

Proof. We can rewrite the inequality from Definition B.2 as equivalently

Our claim then follows immediately by applying Lemma B.1 and Lemma B.3

B.3 Improved composition analysis of SVT

In this section we complete our analysis for improving the privacy bounds for sparse vector technique. While our generalized characterization was in a more complex form, we simplify it here for general queries and monotonic queries. This will then match up to the privacy guarantees that we gave in Section 3 for AboveThreshold .

Lemma B.4. Given a stream of queries {/u1D453 /u1D456 } with sensitivity Δ then for any neighboring datasets /u1D700 (/u1D465 , /u1D465 ′ ) ≤ /u1D700 1 + 2/u1D700 2 . If the queries are monotonic then for any neighboring datasets /u1D700 (/u1D465 , /u1D465 ′ ) ≤ /u1D700 1 + /u1D700 2 , and furthermore, /u1D700 (/u1D465 , /u1D465 ′ ) + /u1D700(/u1D465 ′ , /u1D465 ) ≤ /u1D700 1 + 2/u1D700 2 .

Proof. By Definition 2.2 we must always have Δ/u1D458 (/u1D465 , /u1D465 ′ ) ≤ Δ for any neighboring datasets. Furthermore, we must also have /u1D453 /u1D458 (/u1D465 ′ ) - /u1D453 /u1D458 (/u1D465 ) ≥ -Δ for any neighboring datasets. This implies /u1D700 (/u1D465 , /u1D465 ′ ) ≤ /u1D700 1 + 2/u1D700 2 for any neighboring datasets.

Now assume the queries are monotonic. Without loss of generality assume /u1D453 /u1D458 (/u1D465 ) ≥ /u1D453 /u1D458 (/u1D465 ′ ) for all /u1D458 . Therefore Δ/u1D458 (/u1D465 , /u1D465 ′ ) = 0 and /u1D453 /u1D458 (/u1D465 ′ ) - /u1D453 /u1D458 (/u1D465 ) ≥ -Δ , which implies /u1D700 (/u1D465 , /u1D465 ′ ) ≤ /u1D700 2 . Similarly we have Δ/u1D458 (/u1D465 ′ , /u1D465 ) ≤ Δ but also /u1D453 /u1D458 (/u1D465 ) - /u1D453 /u1D458 (/u1D465 ′ ) ≥ 0 , which implies /u1D700 (/u1D465 ′ , /u1D465 ) ≤ /u1D700 1 + /u1D700 2 . Combining these implies /u1D700 (/u1D465 ′ , /u1D465 ) ≤ /u1D700 1 + /u1D700 2 and /u1D700 (/u1D465 ′ , /u1D465 ) + /u1D700(/u1D465 , /u1D465 ′ ) ≤ /u1D700 1 + 2/u1D700 2 for any neighboring datasets.

This lemma along with Corollary B.1 immediately imply Corollary 3.1 and Lemma 3.2

Proof of Theorem 1. From Lemma B.4 and Corollary B.2 we have that Algorithm 1 is /u1D700 1 + 2/u1D700 2 -rangebounded. Applying Proposition 2 then gives the zCDP guarantees.

The sparse vector technique is simply an iterative call to AboveThreshold and as such the privacy bounds will come from the composition. By analyzing the composition through zCDP which has become commonplace in the privacy community, we can immediately improve the privacy guarantees for monotonic queries in the sparse vector technique. Furthermore we can also improve the privacy guarantees under the standard definition using the improved composition bounds for range-bounded mechanisms from Dong et al. (2020).

We also note here that the generalized Sparse Vector Technique given in Lyu et al. (2017) only adds noise once to the threshold and only has to scale the noise to the number of calls to AboveThreshold for the queries and not the threshold. Our analysis can extend to give the same properties for Lap and Expo noise, but utilizing advanced composition properties will give far more accuracy. More specifically, the threshold and queries having noise proportional to √ /u1D450//u1D700 , is preferable to all the queries having noise proportional to /u1D450 //u1D700 , and only the threshold proportional to 1//u1D700 , where /u1D450 is the number of calls to AboveThreshold .

B.4 Iterative exponential mechanism

It is folklore that the peeling exponential mechanism is equivalent to taking the top/u1D458 after adding Gumbel noise, as formally shown in Lemma 4.2 of Durfee and Rogers (2019). We show a similar property here for AboveThreshold with Gumbel noise that it is equivalent to iteratively running the exponential mechanism.

Algorithm 3 Iterative Exponential Mechanism

Require: Input dataset /u1D465 , a stream of queries {/u1D453 /u1D456 ∶  →ℝ} with sensitivity Δ , and a threshold /u1D447 1: for each query /u1D456 do 2: Run exponential mechanism with  = {0, ..., /u1D456} where /u1D45E (/u1D465 , 0) = /u1D447 and /u1D45E (/u1D465 , /u1D457 ) = /u1D453 /u1D457 (/u1D465 ) for all /u1D457 > 0 3: if /u1D456 is selected then 4: Output ⊤ and halt 5: else 6: Output ⊥ 7: end if 8: end for

Lemma B.5. If /u1D700 1 = /u1D700 2 = /u1D700/2 then Algorithm 1 with Gumbel noise gives the equivalent output distribution to Algorithm 3.

Proof. We first define

By construction, the probability of Algorithm 3 outputting {⊥ /u1D458-1 , ⊤} is equal to /u1D45D /u1D458 ∏ /u1D458-1 /u1D456=1 (1 - /u1D45D /u1D456 ) . Furthermore, by our definition of /u1D45D /u1D458 we have

Through telescoping cancellation

From Lemma B.2 we then see that the output probabilities are equivalent.

C Additional unbounded quantile estimation results

In this section we first extend our method to the fully unbounded setting by simply making two calls to AboveThreshold and essentially searching through both positive and negative numbers in each respective call. Next we show how our methods can also extend to the add-subtract definition of neighboring. Further, we'll apply our approach to the framework set up in Kaplan et al. (2022) for computing multiple quantiles and extend it to the swap definition.

C.1 Fully unbounded quantile estimation

Recall that our unbounded quantile estimation algorithm assumed that the data was lower bounded. It then slowly increased that bound by a small percentage until the appropriate amount of data fell below the threshold for the given quantile. In order to extend to the fully unbounded setting, we will simply first apply this guess and check method to the positive numbers and then apply it to the negative numbers.

Algorithm 4 Fully unbounded quantile mechanism

Require: Input dataset /u1D465 , a quantile /u1D45E , and parameter /u1D6FD > 1

1: Run AboveThreshold with /u1D465 , /u1D447 = /u1D45E /u1D45B and /u1D453 /u1D456 (/u1D465 ) = |{/u1D465 /u1D457 ∈ /u1D465 |/u1D465 /u1D457 + 1 < /u1D6FD /u1D456 }|

2: Run AboveThreshold with /u1D465 , /u1D447 = (1 - /u1D45E)/u1D45B and /u1D453 /u1D456 (/u1D465 ) = |{/u1D465 /u1D457 ∈ /u1D465 |/u1D465 /u1D457 - 1 > -/u1D6FD /u1D456 }|

3: If the first AboveThreshold halts at /u1D458 > 0 then output /u1D6FD /u1D458 - 1

4: If the second AboveThreshold halts at /u1D458 > 0 then output -/u1D6FD /u1D458 + 1

5: Otherwise return 0

Note that the second call could equivalently be achieved by flipping the sign of all the datapoints and again applying queries /u1D453 /u1D456 (/u1D465 ) = |{/u1D465 /u1D457 ∈ /u1D465 |/u1D465 /u1D457 + 1 < /u1D6FD /u1D456 }| . Therefore all our privacy guarantees from Section 4 will still apply, but composing over two calls to AboveThreshold , which is the Sparse Vector Technique.

In the first call to AboveThreshold we are assuming a lower bound of 0, and searching the positive numbers. If it terminates immediately then it is likely that more than a /u1D45E th fraction of the data is below 0. We then symmetrically search through the negative numbers by assuming an upper bound of 0. If this halts immediately then it is likely that the quantile is already near 0. We could also apply other variants of this, such as three total calls to get the maximum and minimum of the data if we wanted full data bounds.

Further note that computing the smallest quantiles will be challenging for our algorithm because it may take many queries to get to the appropriate threshold, but each query will have a reasonable chance of terminating if /u1D45E is very small. To account for these queries in the lower-bounded setting, we can invert all the datapoints and instead search for quantile 1 - /u1D45E on this transformed data, then invert our resulting estimate. We would also need to reduce our parameter /u1D6FD a reasonable amount in this setting.

C.2 Extension to add-subtract neighbors

We also extend our results to the add-subtract definition of neighboring datasets.

Definition C.1. Datasets /u1D465 , /u1D465 ′ ∈  are neighboring if one of them can be obtained from the other by adding or removing one individual's data.

Under this definition we see that our threshold /u1D447 = /u1D45E /u1D45B is no longer fixed, so we will instead set each query to be /u1D453 /u1D456 (/u1D465 ) = |{/u1D465 /u1D457 ∈ /u1D465 |/u1D465 /u1D457 - /u1D4C1.alt + 1 < /u1D6FD /u1D456 }| - /u1D45E/u1D45B . Unfortunately this query will no longer be monotonic, so we will instead take advantage of our more general characterization in Section B.1 to further tighten the privacy bounds in this setting.

Lemma C.1. Given the stream of queries /u1D453 /u1D456 (/u1D465 ) = |{/u1D465 /u1D457 ∈ /u1D465 |/u1D465 /u1D457 - /u1D4C1.alt + 1 < /u1D6FD /u1D456 }| - /u1D45E /u1D45B with sensitivity Δ = 1 , for any neighboring datasets /u1D465 , /u1D465 ′ under Definition C.1 and quantile /u1D45E ∈ [0, 1] we have /u1D700 (/u1D465 , /u1D465 ′ ) ≤ max{(1 - /u1D45E)/u1D700 1 , /u1D45E /u1D700 1 + /u1D700 2 } .

Proof. Without loss of generality, assume that /u1D465 has one more individuals data, so /u1D465 ∈ ℝ /u1D45B and /u1D465 ′ ∈ ℝ /u1D45B-1 . Let /u1D454 /u1D456 (/u1D465 ) = |{/u1D465 /u1D457 ∈ /u1D465 |/u1D465 /u1D457 - /u1D4C1.alt + 1 < /u1D6FD /u1D456 }| . By construction we must have /u1D454 /u1D456 (/u1D465 ) - /u1D454 /u1D456 (/u1D465 ′ ) ∈ {0, 1} because /u1D465 has an additional datapoint. Furthermore, because the thresholds are increasing, if /u1D454 /u1D458 (/u1D465 ) - /u1D454 /u1D458 (/u1D465 ′ ) = 1 then /u1D454 /u1D456 (/u1D465 ) - /u1D454 /u1D456 (/u1D465 ′ ) = 1 for all /u1D456 > /u1D458 . Similarly if /u1D454 /u1D458 (/u1D465 ) - /u1D454 /u1D458 (/u1D465 ′ ) = 0 then /u1D454 /u1D456 (/u1D465 ) - /u1D454 /u1D456 (/u1D465 ′ ) = 0 for all /u1D456 < /u1D458 . We further see that /u1D453 /u1D456 (/u1D465 ) - /u1D453 /u1D456 (/u1D465 ′ ) = /u1D454 /u1D456 (/u1D465 ) - /u1D454 /u1D456 (/u1D465 ′ ) - /u1D45E for all /u1D456 .

First consider the case when /u1D454 /u1D458 (/u1D465 ) - /u1D454 /u1D458 (/u1D465 ′ ) = 0 . We therefore have /u1D454 /u1D456 (/u1D465 ) - /u1D454 /u1D456 (/u1D465 ′ ) = 0 for all /u1D456 < /u1D458 so Δ/u1D458 (/u1D465 , /u1D465 ′ ) = /u1D45E , and also Δ/u1D458 (/u1D465 , /u1D465 ′ ) - (/u1D453 /u1D458 (/u1D465 ′ ) - /u1D453 /u1D458 (/u1D465 )) = 0 , so /u1D700 (/u1D465 , /u1D465 ′ ) ≤ /u1D45E /u1D700 1 . Similarly, we have Δ/u1D458 (/u1D465 ′ , /u1D465 ) = 0 and Δ/u1D458 (/u1D465 , /u1D465 ′ ) - (/u1D453 /u1D458 (/u1D465 ′ ) - /u1D453 /u1D458 (/u1D465 )) = /u1D45E , so /u1D700 (/u1D465 ′ , /u1D465 ) ≤ /u1D45E /u1D700 2 .

Next consider the case when /u1D454 /u1D458 (/u1D465 ) - /u1D454 /u1D458 (/u1D465 ′ ) = 1 . Therefore we have Δ/u1D458 (/u1D465 , /u1D465 ′ ) ≤ /u1D45E and also /u1D453 /u1D458 (/u1D465 ′ ) - /u1D453 /u1D458 (/u1D465 ) = /u1D45E -1 which implies Δ/u1D458 (/u1D465 , /u1D465 ′ ) - (/u1D453 /u1D458 (/u1D465 ′ ) - /u1D453 /u1D458 (/u1D465 )) ≤ 1 , so /u1D700 (/u1D465 , /u1D465 ′ ) ≤ /u1D45E/u1D700 1 +/u1D700 2 . Similarly, we have that Δ/u1D458 (/u1D465 ′ , /u1D465 ) ≤ 1 - /u1D45E , and thus Δ/u1D458 (/u1D465 ′ , /u1D465 ) - (/u1D453 /u1D458 (/u1D465 ) - /u1D453 /u1D458 (/u1D465 ′ )) = 0 , so /u1D700 (/u1D465 ′ , /u1D465 ) ≤ (1 - /u1D45E)/u1D700 1 .

Combining these bounds gives our desired result.

With these improved bounds we can then show that our methods also extend to the add-subtract definition of neighboring with tighter privacy guarantees.

Corollary C.1. If we run Algorithm 2 with /u1D700 1 = /u1D700 2 for a quantile /u1D45E under Definition C.1 then it is (/u1D45E /u1D700 1 + /u1D700 2 ) -DP.

Proof. Combining Lemma C.1 and Corollary B.1 immediately imply the desired result

C.3 Extension to multiple quantile estimation

As previously established, the framework in Kaplan et al. (2022) is agnostic to the single quantile estimation method. However their proof is for the add-subtract neighbors, although they note that it can be extended easily to the swap neighbors. We also discuss here how it can be extended for completeness.

For the swap neighboring definition, at each level of the recursive partitioning scheme either two partitions differ under the add-subtract definition, or one partition differs under the swap definition. Applying their framework to our algorithm, we will instead compute all the thresholds in advance of the splitting. Accordingly these thresholds will remain fixed. If the thresholds are fixed then we can actually further improve our privacy bounds for the add-subtract definition. Once again we will use our more general characterization from Section B.1.

Lemma C.2. Given the stream of queries /u1D453 /u1D456 (/u1D465 ) = |{/u1D465 /u1D457 ∈ /u1D465 |/u1D465 /u1D457 - /u1D4C1.alt + 1 < /u1D6FD /u1D456 }| with sensitivity Δ = 1 , for any neighboring datasets /u1D465 , /u1D465 ′ under Definition C.1 we have /u1D700 (/u1D465 , /u1D465 ′ ) ≤ max{/u1D700 1 , /u1D700 2 }

Proof. Without loss of generality, assume that /u1D465 has one more individuals data, so /u1D465 ∈ ℝ /u1D45B and /u1D465 ′ ∈ ℝ /u1D45B-1 . By construction we must have /u1D453 /u1D456 (/u1D465 ) -/u1D453 /u1D456 (/u1D465 ′ ) ∈ {0, 1} because /u1D465 has an additional datapoint. Furthermore, because the thresholds are increasing, if /u1D453 /u1D458 (/u1D465 ) - /u1D453 /u1D458 (/u1D465 ′ ) = 0 then /u1D453 /u1D456 (/u1D465 ) - /u1D453 /u1D456 (/u1D465 ′ ) = 0 for all /u1D456 < /u1D458 . Thus the case of /u1D453 /u1D458 (/u1D465 ) - /u1D453 /u1D458 (/u1D465 ′ ) = 0 is easy.

Instead consider /u1D453 /u1D458 (/u1D465 ) - /u1D453 /u1D458 (/u1D465 ′ ) = 1 . We know Δ/u1D458 (/u1D465 , /u1D465 ′ ) = 0 so Δ/u1D458 (/u1D465 , /u1D465 ′ ) - (/u1D453 /u1D458 (/u1D465 ′ ) - /u1D453 /u1D458 (/u1D465 )) = 1 , so /u1D700 (/u1D465 , /u1D465 ′ ) ≤ /u1D700 2 . Further we must have Δ/u1D458 (/u1D465 ′ , /u1D465 ) ≤ 1 and Δ/u1D458 (/u1D465 ′ , /u1D465 ) - (/u1D453 /u1D458 (/u1D465 ) - /u1D453 /u1D458 (/u1D465 ′ )) = 0 , so /u1D700 (/u1D465 ′ , /u1D465 ) ≤ /u1D700 1 . Combining these implies /u1D700 (/u1D465 ′ , /u1D465 ) ≤ max{/u1D700 1 , /u1D700 2 } for any neighboring datasets.

Applying these bounds we see that applying the framework of Kaplan et al. (2022) to the swap definition either leads to one composition of (/u1D700 1 +/u1D700 2 ) -DP for the swap definition or two compositions of max{/u1D700 1 , /u1D700 2 } for the add-subtract definition at each level. Setting /u1D700 1 = /u1D700 2 we then have that the privacy cost of computing /u1D45A quantiles with this framework will be equivalent to log(/u1D45A + 1) compositions of (/u1D700 1 + /u1D700 2 ) -DP.

D Implementation code

For ease of implementation, we provide some simple python code to run our method with access to the respective Noise generator of the users choosing. In our experiments we used exponential noise from the numpy library.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problem of differentially private computation of quantiles for the data. The core idea is to leverage the ‘AboveThreshold’ algorithm  as a subroutine to guess the desired item value such that the rank of the item value is about qn for target quantile q. Authors also discuss the application of this method for mean and sum estimation where the proposed algorithm can help clipping the item values.Experimental results indicate improvement for inner quantiles over Page, Age, Workhours dataset.\n\nSTRENGTHS:\nQuantile estimation is very important problem with real-world applications. This paper presents a straightforward application Abovethreshold differential privacy techniques. For applications, the simplicity in algorithms is greatly valued. The paper is well written and very accessible for the general audience.\n\nWEAKNESSES:\nOne major concern is the requirement of O(n) passes through the dataset. With a large volume of data, n passes seem to be formidable. The experimental section can be strengthened by using more data points and considering query latency. \n\nRelated works on quantile approximation are missing.\n     Gillenwater, Jennifer, Matthew Joseph, and Alex Kulesza. \"Differentially private quantiles.\"  They present a DP-algorithm with exponential mechanism that estimates all m quantiles in $O(mnlog(n)+m^2n)$ time. Zhao, Fuheng, et al. \"Differentially private linear sketches: Efficient implementations and applications.\"  && Pagh, Rasmus, and Mikkel Thorup. \"Improved utility analysis of private countsketch.\" They present a DP-algorithm using count sketch to estimate all quantiles in one pass with gaussian mechanism.\n\nQUESTIONS:\n1) Can the algorithm improve to O(logN) pass over the data with something like exponential search? \n\n2) 1000 data points in the experiments are very small. Also maybe consider including some results on the query latency. For example the time to find the median.\n\nLIMITATIONS:\nNo potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tries to improve upon Sparse Vector Techniques for differential privacy to obtain better estimates of the maximum and quantiles.\n\nSTRENGTHS:\nThe paper covers an important problem, finding a differentially private estimate of the max. This  is often needed in one of the most common aggregations, i.e. sums, to clip and set a sensitivity.\n\n It promises less noise for the same privacy through an improvement to the sparse vector technique. \n\nI would be interested in reading it again if the proofs were correct. However,...\n\nWEAKNESSES:\nI unfortunately cannot trust the results in the paper due to bad proofs.\n\nThe proof for Lemma B.1 is incorrect though the conclusion may not be. For example, the statement in Line 557 of the supplement is incorrect. The statement says you will get the same answer _with the exact same noise_. You’ve fixed the randomness and there is no remaining randomness for the probabilities. You end up using v’ \\neq v and v_k’ \\neq v_k later.\n\nFurthermore, injectivity of the mapping v, v_k -> v’, v_k’ is not enough to show that the probabilities P(AboveThreshold(x) = O) and P(AboveThreshold(x’) = O) are the same for some output O. For example x -> x/2 is an injective mapping from [0,1]. But for a uniform distribution, the probability of the image is ½, not 1. In this case, the mapping happens to be a shift, so the Jacobian of the map is the identity, which actually is enough since the determinant is 1. But the current proof is too much of a mess to know if I can trust the theoretical claims in the paper.\n\nPlease fix the reference in Lemma 3.1. The Thm 4, 5 in the VLDB version are show that algorithms are _not_ DP.\n\nQUESTIONS:\nPlease fix the proofs!\n\nIt is hard to tease out your contributions from the contributions section. Most of the contributions section belongs in the introduction. The contributions should be a succinct description of what _you_ contributed, not about where it can be used.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a differentially private algorithm for differentially private quantile estimation that can handle unbounded data setting. The authors revisited the AboveThreshold subroutine in Sparse Vector Technique and proved new privacy guarantee results. The authors can verify the asymptotic runtime complexity and demonstrate the utility through experiments. The method can have successful application to differentially private sum estimation.\n\nSTRENGTHS:\n**Originality** The proposed method for finding private quantile is a creative use of an existing method for the application to a new domain. The authors proved new privacy results for the existing method.\n\n**Quality**  The method can solve the problem. The theoretical analysis can verify the privacy and complexity of the proposed method. The authors can demonstrate the accuracy of the method through numerical studies.\n\n**Clarity** The problem is well-defined and clearly stated. The method can be clearly understood.\n\n**Significance** The problem being solved is important and as verified by the experiment the proposed method can improve accuracy for real-world data . The new privacy analysis to the Sparse Vector Technique can be of independent interest.\n\nWEAKNESSES:\nThe authors do not provide theoretical utility guarantee; though the experiment results look good.\n\nQUESTIONS:\n1. You provide theoretical runtime complexity analysis. It would be of interest to compare the runtime in experiments to verify your asymptotic analysis. \n\n2. From a quick search, there seems to be little literature in quantile estimation that addresses the unbounded setting.  It would be great to highlight this by citing some papers that can handle unbounded data in Introduction/Related Work.  You briefly mentioned this in your contributions, saying “commonly used algorithms can apply loose bounds to ensure data is contained within” and analyzing their drawbacks, but no citation is included.\n\nLIMITATIONS:\nThe proposed method works worse than the existing EMQ method for synthetic data. Can you offer any suggestions on how to improve your method’s accuracy for these well-behaving datasets?", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problem of computing the quantile q of a dataset with differential privacy. The prior standard technique for this problem was the exponential mechanism with the loss of an output T being the difference between the number of examples less than T and qn. An issue with the exponential mechanism (and most mechanisms for quantile estimation) is that it requires the user to bound the range of possible values in the dataset. The authors propose an alternate algorithm which does not require a prior bound, and takes advantage of the AboveThreshold algorithm. They propose using the AboveThreshold on queries of the form \"how many examples in the dataset are less than T?\", where T is exponentially increasing, and return the value T that AboveThreshold reports has more than qn elements less than T. As part of the analysis, the authors give an improved analysis of AboveThreshold, showing it has improved privacy guarantees for monotonic queries when using Exponential or Gumbel noise; previously, this was shown for Laplace noise. The Exponential distribution has lower variance, so this allows them to get a lower-variance estimate without sacrificing privacy. The authors perform experiments to compare to the exponential mechanism, as well as the aggregate tree method. On synthetic datasets, the exponential mechanism performs slightly better, but otherwise the authors' result is better for quantile estimation. For sum estimation, when using parameters agnostic to the dataset, the authors' algorithm by far outperforms the baselines, due to not relying on a good prior range estimate.\n\nSTRENGTHS:\nOverall I enjoyed reading the paper and feel it makes a substantial contribution to the field. Private quantile selection is an extremely important problem with a number of applications, including any setting where we wish to compute a private sum. The problem of needing a bounded range for these problems has always been a troublesome part of DP research and application, and having a seemingly robust approach that at least partially gets around this issue is thus of significant impact. The main idea in the paper is simple to describe, which makes it more likely to see adoption and further development in follow-up works, but at the same time requires new analyses to achieve competitive privacy-utility tradeoffs. Finally, I felt the paper was overall a pretty easy read, in part due to the elegance of the algorithmic ideas but also the quality and clarity of the writing.\n\nWEAKNESSES:\nI think the only major weakness is that one still needs to choose beta appropriately. The authors mention this in the discussion in Appendix D, but this discussion is somewhat incomplete. Still, I believe that beta is a much easier parameter for a practitioner to choose appropriately than the range of the dataset, especially since the experiments show beta is somewhat dataset-agnostic, so this is a minor weakness.\n\nQUESTIONS:\nIntuitively it seems like smaller beta is more likely to underestimate the quantile and larger beta is more likely to overestimate, is this correct? Also, for e.g. a bimodal dataset such as the Gaussian mixture 0.5 * N(0, 1) + 0.5 * N(Delta, 1), Delta >> 1, for estimating the 75th percentile, how would you expect UQE to compare to EMQ? (with a good or bad range estimate)\n\nLIMITATIONS:\nYes, in the appendix the authors discuss the main limitation/weakness. N/A on negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}