SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Edith Cohen
Google Research and Tel Aviv University
edith@cohenwang.com
Xin Lyu
lyuxin1999@gmail.com
We propose the Target Charging Technique (TCT), a unified privacy analysis framework for interactive settings where a sensitive dataset is accessed multiple times using differentially private algorithms. Unlike traditional composition, where privacy guarantees deteriorate quickly with the number of accesses, TCT allows computations that don't hit a specified target, often the vast majority, to be essentially free (while incurring instead a small overhead on those that do hit their targets). TCT generalizes tools such as the sparse vector technique and top- $k$  selection from private candidates and extends their remarkable privacy enhancement benefits from noisy Lipschitz functions to general private algorithms.
In many practical settings of data analysis and optimization, the dataset  $D$  is accessed multiple times interactively via different algorithms  $(\mathcal{A}_i)$ , so that  $\mathcal{A}_i$  depends on the transcript of prior responses  $(\mathcal{A}_j(D))_{j < i}$ . When each  $\mathcal{A}_i$  is privacy-preserving, we are interested in tight end-to-end privacy analysis. We consider the standard statistical framework of differential privacy introduced in [11]. Composition theorems [14] are a generic way to do that and achieve overall privacy cost that scales linearly or (via "advanced" composition) with square-root dependence in the number of private computations. We aim for a broad understanding of scenarios where the overall privacy bounds can be lowered significantly via the following paradigm: Each computation is specified by a private algorithm  $\mathcal{A}_i$  together with a target  $\top_i$ , that is a subset of its potential outputs. The total privacy cost depends only on computations where the output hits its target, that is  $\mathcal{A}_i(D) \in \top_i$ . This paradigm is suitable and can be highly beneficial when (i) the specified targets are a good proxy for the actual privacy exposure and (ii) we expect the majority of computations to not hit their target, and thus essentially be "free" in terms of privacy cost.
The Sparse Vector Technique (SVT) [12, 30, 18, 34] is a classic special case. SVT is designed for computations that have the form of approximate threshold tests applied to Lipschitz functions. Concretely, each such AboveThreshold test is specified by a 1-Lipschitz function  $f$  and a threshold value  $t$  and we wish to test whether  $f(D) \gtrsim t$ . The textbook SVT algorithm compares a noisy value with a noisy threshold (independent Laplace noise for the values and threshold noise that can be updated only after positive responses). Remarkably, the overall privacy cost depends only on positive responses: Roughly, composition is applied to twice the number of positive responses instead of to the total number of computations. In our terminology, the target of each test is a positive response. SVT privacy analysis benefits when the majority of AboveThreshold test results are negative (and hence "free"). This makes SVT a key ingredient in a range of methods [13]: private multiplicative weights [18], Propose-Test-Release [10], fine privacy analysis via distance-to-stability [33], model-agnostic private learning [1], and designing streaming algorithms that are robust to adaptive inputs [19, 6].
We aim to extend such target-hits privacy analysis to interactive applications of general private algorithms (that is, algorithms that provide privacy guarantees but have no other assumptions): private tests, where we would hope to incur privacy cost only for positive responses, and private algorithms that return more complex outputs, e.g., vector average, cluster centers, a sanitized dataset, or a trained ML model, where the goal is to incur privacy cost only when the output satisfies some criteria. Textbook SVT, however, is less amenable to such extensions: First, SVT departs from the natural paradigm of applying private algorithms to the dataset and reporting the output. A natural implementation of private AboveThreshold tests would add Laplace noise to the value and compare with the threshold. Instead, SVT takes as input the Lipschitz output of the non-private algorithms with threshold value and the privacy treatment is integrated (added noise both to values and threshold). The overall utility and privacy of the complete interaction are analyzed with respect to the non-private values, which is not suitable when the algorithms are already private. Moreover, the technique of using a hidden shared threshold noise across multiple AboveThreshold tests is specific for Lipschitz functions, introduces dependencies between responses, and more critically, results in separate privacy costs for reporting noisy values (that is often required by analytics tasks [22]).
Consider private tests. The natural paradigm is to sequentially choose a test, apply it, and report the result. The hope is to incur privacy loss only on positive responses. Private testing was considered in prior works [21, 7] but in ways that departed from this paradigm: [21] processed the private tests so that a positive answer is returned only when the probability  $p$  of a positive response by the private test is very close to 1. This seems unsatisfactory: If the design goal of the private testing algorithm was to report only very high probabilities, then this could have been more efficiently integrated into the design, and if otherwise, then we miss out on acceptable positive responses with moderately high probabilities (e.g.  $95\%$ ).
Consider now Top- $k$  selection, which is a basic subroutine in data analysis, where input algorithms  $(\mathcal{A}_i)_{i\in [m]}$  (aka candidates) that return results with quality scores are provided in a batch (i.e., non interactively). The selection returns the  $k$  candidates with highest quality scores on our dataset. The respective private construct, where the data is sensitive and the algorithms are private, had been intensely studied [23, 17, 32]. The top- $k$  candidates can be viewed as target-hits and we might hope for privacy cost that is close to a composition over  $k$  private computations, instead of over  $m\gg k$ . The natural approach for top- $k$  is one-shot (Algorithm 3), where each algorithm is applied once and the responses with top- $k$  scores are reported. Prior works on private selection that achieve this analysis goal include those [9, 29] that use the natural one-shot selection but are tailored to Lipschitz functions (apply the Exponential Mechanism [24] or the Report-Noise-Max paradigm [13]) and works [21, 28, 7] that do apply with general private algorithms but significantly depart from the natural one-shot approach: They make a randomized number of computations that is generally much larger than  $m$ , with each  $\mathcal{A}_i$  invoked multiple times or none. The interpretation of the selection deviates from top-1 and does not naturally extend to top- $k$ . We seek privacy analysis that applies to one-shot top- $k$  selection with candidates that are general private algorithms.
The natural interactive paradigm and one-shot selection are simple, interpretable, and general. The departures made in prior works were made for a reason: Simple arguments (that apply with both top-1 one-shot private selection [21] and AboveThreshold tests) seem to preclude efficient target-charging privacy analysis: With pure-DP, if we perform  $m$  computations that are  $\varepsilon$ -DP (that is,  $m$  candidates or  $m$  tests), then the privacy parameter value for a pure DP bound is  $\Omega(m)\varepsilon$ . With approximate-DP, and even a single "hit," the parameter values are  $(\Omega(\varepsilon\log(1/\delta)), \delta)$ . The latter suggests a daunting overhead of  $O(\log(1/\delta))$  instead of  $O(1)$  per "hit." We circumvent the mentioned limitations by taking approximate DP to be a reasonable relaxation and additionally, aim for application regimes where many private computations are performed on the same dataset and we expect multiple, say  $\Omega(\log(1/\delta))$ , "target hits" (e.g. positive tests and sum of the  $k$ -values of selections). With these relaxations in place, we seek a unified target-charging analysis (e.g. privacy charge that corresponds to  $O(1)$  calls per "target hit") that applies with the natural paradigm across interactive calls and top- $k$  selections.
2 Overview of Contributions
We overview our contributions (proofs and details are provided in the Appendix). We introduce the Target-Charging Technique (TCT) for privacy analysis over interactive private computations (see Algorithm 1). Each computation performed on the sensitive dataset  $D$  is specified by a pair of
private algorithm  $\mathcal{A}_i$  and target  $\top_{i}$ . The interaction is halted after a pre-specified number  $\tau$  of computations satisfy  $\mathcal{A}_i(D)\in \top_i$ . We define targets as follows:
Definition 2.1 ( $q$ -Target). Let  $\mathcal{M}: X^n \to \mathcal{V}$  be a randomized algorithm. For  $q \in (0,1]$  and  $\varepsilon > 0$ , we say that a subset  $\top \subseteq \mathcal{V}$  of outcomes is a  $q$ -Target of  $\mathcal{M}$ , if the following holds: For any pair  $D^0$  and  $D^1$  of neighboring data sets, there exist  $p \in [0,1]$ , and three distributions  $\mathbf{C}$ ,  $\mathbf{B}^0$  and  $\mathbf{B}^1$  such that
1. The distributions  $\mathcal{M}(D^0)$  and  $\mathcal{M}(D^1)$  can be written as the following mixtures:
$$
\mathcal {M} (D ^ {0}) \equiv p \cdot \mathbf {C} + (1 - p) \cdot \mathbf {B} ^ {0},
$$
$$
\mathcal {M} (D ^ {1}) \equiv p \cdot \mathbf {C} + (1 - p) \cdot \mathbf {B} ^ {1}.
$$
2.  $\mathbf{B}^0, \mathbf{B}^1$  are  $(\varepsilon, 0)$ -indistinguishable,
3.  $\min (\operatorname {Pr}[\mathbf{B}^0\in \top ],\operatorname {Pr}[\mathbf{B}^1\in \top ])\geq q.$
The effectiveness of a target as a proxy of the actual privacy cost is measured by its  $q$ -value where  $q \in (0,1]$ . We interpret  $1 / q$  as the overhead factor of the actual privacy exposure per target hit, that is, the number of private accesses that correspond to a single target hit. Note that an algorithm with a  $q$ -target for  $\varepsilon > 0$  must be  $(\varepsilon, 0)$ -DP and that any  $(\varepsilon, 0)$ -DP algorithm has a 1-target, as the set of all outcomes  $\top = \mathcal{V}$  is a 1-target (and hence also a  $q$ -target for any  $q \leq 1$ ). More helpful targets are "smaller" (so that we are less likely to be charged) with a larger  $q$  (so that the overhead per charge is smaller). We establish the following privacy bounds.
Lemma 2.2 (simplified meta privacy cost of target-charging). The privacy parameters of Algorithm 1 are  $(\varepsilon', \delta)$  where  $\varepsilon' \approx \frac{\tau}{q} \varepsilon$  and  $\delta = e^{-\Omega(\tau)}$ .
Alternatively, we obtain parameter values  $(\varepsilon', \delta') = (f_{\varepsilon}(r, \varepsilon), f_{\delta}(r, \varepsilon) + e^{-\Omega(\tau)})$  where  $r \approx \tau/q$  and  $(f_{\varepsilon}(r, \varepsilon), f_{\delta}(r, \varepsilon))$  are privacy parameter values for advanced composition [14] of  $r$ $\varepsilon$ -DP computations.
Proof details for a more general statement that also applies with approximate DP algorithms are provided in Section B (in which case the  $\delta$  parameters of all calls simply add up). The idea is simple: We compare the execution of Algorithm 4 on two neighboring data sets  $D^0, D^1$ . Given a request  $(\mathcal{A}, \top)$ , let  $p, \mathbf{C}, \mathbf{B}, \mathbf{B}^0, \mathbf{B}^1$  be the decomposition of  $\mathcal{A}$  w.r.t.  $D^0, D^1$  given by Definition 2.1. Then, running  $\mathcal{A}$  on  $D^0, D^1$  can be implemented in the following equivalent way: we flip a  $p$ -biased coin. With probability  $p$ , the algorithm samples from  $\mathbf{C}$  and returns the result, without accessing  $D^0, D^1$  at all (!). Otherwise, the algorithm needs to sample from  $\mathbf{B}^0$  or  $\mathbf{B}^1$ , depending on whether the private data is  $D^0$  or  $D^1$ . However, by Property 3 in Definition 2.1, there is a probability of at least  $q$  that Algorithm 1 will "notice" the privacy-leaking computation by observing a result in the target set  $\top$ . If this indeed happens, the algorithm increments the counter. On average, each counter increment corresponds to  $\frac{1}{q}$  accesses to the private data. Therefore we use the number of target hits (multiplied by  $1/q$ ) as a proxy for the actual privacy leak. Finally, we apply concentration inequalities to obtain high confidence bounds on the probability that the actual number of accesses significantly exceeds its expectation of  $\tau/q$ . The multiplicative error decreases when the number  $\tau$  of target hits is larger. In the regime  $\tau > \ln(1/\delta)$ , we amortize the mentioned  $O(\log(1/\delta))$  overhead of the natural paradigm so that each target hit results in privacy cost equivalent to  $O(1/q)$  calls. In the regime of very few target hits (e.g., few private tests or private selections), we still have to effectively "pay" for the larger  $\tau = \Omega(\ln(1/\delta))$ , but TCT still has some advantages over alternative approaches, due to its use of the natural paradigm and its applicability with general private algorithms.
TCT is simple but turns out to be surprisingly powerful due to natural targets with low overhead. We present an expansive toolkit that is built on top of TCT and describe application scenarios.
2.1 NotPrior targets
A NotPrior target of an  $\varepsilon$ -DP algorithm is specified by any outcome of our choice (the "prior") that we denote by  $\perp$ . The NotPrior target is the set of all outcomes except  $\perp$ . Surprisingly perhaps, this is an effective target (See Section C for the proof that applies also with approximate-DP):
Lemma 2.3 (Property of a NotPrior target). Let  $\mathcal{A}: X \to \mathcal{V} \cup \{\bot\}$ , where  $\bot \notin \mathcal{V}$ , be an  $\varepsilon$ -DP algorithm. Then the set of outcomes  $\mathcal{V}$  constitutes an  $\frac{1}{e^{\varepsilon} + 1}$ -target for  $\mathcal{A}$ .
Algorithm 1: Target Charging
Input: Dataset  $D = \{x_{1},\dots ,x_{n}\} \in X^{n}$  . Integer  $\tau \geq 1$  (Upper limit on the number of target hits). Fraction  $q\in [0,1]$ $C\gets 0$  // Initialize target hit counter   
while  $C <   \tau$  do // Main loop   
Receive  $(A,\top)$  where  $\mathcal{A}$  is an  $\varepsilon$  -DP mechanism, and  $\top$  is a  $q$  -target for  $\mathcal{A}$ $r\gets A(D)$    
Publish  $r$    
if  $r\in \top$  then  $C\leftarrow C + 1$  // outcome is a target hit
Note that for small  $\varepsilon$ , we have  $q$  approaching  $1/2$  and thus the overhead factor is close to 2. The TCT privacy analysis is beneficial over plain composition when the majority of all outcomes in our interaction match their prior  $\perp$ . We describe application scenarios for NotPrior targets. For most of these scenarios, TCT is the only method we are aware of that provides the stated privacy guarantees in the general context.
Private testing A private test is a private algorithm with a Boolean output. By specifying our prior to be a negative response, we obtain (for small  $\varepsilon$ ) an overhead of 2 for positive responses, which matches SVT. TCT is the only method we are aware of that provides SVT-like guarantees with general private tests.
Pay-only-for-change When we have a prior on the result of each computation and expect the results of most computations to agree with their respective prior, we set  $\perp$  to be our prior. We report all results but pay only for those that disagree with the prior. We describe some use cases where paying only for change can be very beneficial (i) the priors are results of the same computations on an older dataset, so they are likely to remain the same (ii) In streaming or dynamic graph algorithms, the input is a sequence of updates where typically the number of changes to the output is much smaller than the number of updates. Differential privacy was used to obtain algorithms that are robust to adaptive inputs [19, 2] by private aggregation of non-robust copies. The pay-only-for-change allows for number of changes to output (instead of the much larger number of updates) that is quadratic in the number of copies. Our result enables such gain with any private aggregation algorithm (that is not necessarily in the form of AboveThreshold tests).
2.2 Conditional Release
We have a private algorithm  $\mathcal{A}: X \to \mathcal{Y}$  but are interested in the output  $\mathcal{A}(D)$  only when a certain condition holds (i.e., when the output is in  $\top \subseteq \mathcal{Y}$ ). The condition may depend on the interaction transcript thus far (depend on prior computations and outputs). We expect most computations not to meet their release conditions and want to be "charged" only for the ones that do. Recall that with differential privacy, not reporting a result also leaks information on the dataset, so this is not straightforward. We define  $A_{\top} := \text{ConditionalRelease}(\mathcal{A}, \top)$  as the operation that inputs a dataset  $D$ , computes  $y \gets A(D)$ . If  $y \in \top$ , then publish  $y$  and otherwise publish  $\bot$ . We show that this operation can be analysed in TCT as a call with the algorithm and NotPrior target pair  $(\mathcal{A}_{\top}, \top)$ , that is, a target hit occurs if and only if  $y \in \top$ :
Lemma 2.4 (ConditionalRelease privacy analysis).  $\mathcal{A}_{\top}$  satisfies the privacy parameters of  $\mathcal{A}$  and  $\top$  is a NotPrior target of  $\mathcal{A}_{\top}$ .
Proof.  $\mathcal{A}_{\top}$  processes the output of the private algorithm  $\mathcal{A}$  and thus from post processing property is also private with the same privacy parameter values. Now note that  $\top$  is a NotPrior target of  $\mathcal{A}$ , with respect to prior  $\bot$ .
We describe some example use-cases:
(i) Private learning of models from the data (clustering, regression, average, ML model) but we are interested in the result only when its quality is sufficient, say above a specified threshold, or when some other conditions hold.
(ii) Greedy coverage or representative selection type applications, where we incur privacy cost only for selected items. To do so, we condition the release on the "coverage" of past responses. For example, when greedily selecting a subset of features that are most relevant or a subset of centers that bring most value.
(iii) Approximate AboveThreshold tests on Lipschitz functions, with release of above-threshold noisy values: As mentioned, SVT incurs additional privacy cost for the reporting whereas TCT (using ConditionalRelease) does not, so TCT benefits in the regime of sufficiently many target hits.
(iv) AboveThreshold tests with sketch-based approximate distinct counts: Distinct counting sketches [16, 15, 5] meet the privacy requirement by the built-in sketch randomness [31]. We apply ConditionalRelease and set  $\top$  to be above threshold values. In comparison, despite the function (distinct count) being 1-Lipschitz, the use of SVT for this task incurs higher overheads in utility (approximation quality) and privacy: Even for the goal of just testing, a direct use of SVT treats the approximate value as the non-private input, which reduces accuracy due to the additional added noise. Treating the reported value as a noisy Lipschitz still incurs accuracy loss due to the threshold noise, threshold noise introduces bias, and analysis is complicated by the response not following a particular noise distribution. For releasing values, SVT as a separate distinct-count sketch is needed to obtain an independent noisy value [22], which increases both storage and privacy costs.
2.3 Conditional Release with Revisions
We present an extension of Conditional Release that allows for followup revisions of the target. The initial ConditionalRelease and the followup ReviseCR calls are described in Algorithm 2. The ConditionalRelease call specifies a computation identifier  $h$  for later reference, an algorithm and a target pair  $(\mathcal{A},\top)$ . It draws  $r_h \sim \mathcal{A}(D)$  and internally stores  $r_h$  and a current target  $\top_h \leftarrow \top$ . When  $r_h \in \top$  then  $r_h$  is published and a charge is made. Otherwise,  $\bot$  is published. Each (followup) ReviseCR call specifies an identifier  $h$  and a disjoint extension  $\top'$  to its current target  $\top_h$ . If  $r_h \in \top'$ , then  $r_h$  is published and a charge is made. Otherwise,  $\bot$  is published. The stored current target for computation  $h$  is augmented to include  $\top'$ . Note that a target hit occurs at most once in a sequence of (initial and followup revise) calls and if and only if the result of the initial computation  $r_h$  is in the final target  $\top_h$ .
Algorithm 2: Conditional Release and Revise Calls
// Initial Conditional Release call: Analysed in TCT as a  $(\varepsilon, \delta)$ -DP algorithm  $\mathcal{A}_{\top}$  and NotPrior target  
Function ConditionalRelease(h,  $\mathcal{A}, \top$ ): // unique identifier  $h$ , an  $(\varepsilon, \delta)$ -DP algorithm  $\mathcal{A} \to \mathcal{Y}$ ,  
 $\top \subset \mathcal{Y}$ $\top_{h} \gets \top$  // Current target for computation  $h$   
TCT Charge for  $\delta$  // If  $\delta > 0$ , see Section B  
 $r_{h} \gets \mathcal{A}(D)$  // Result for computation  $h$   
if  $r_{h} \in \top_{h}$  then // publish and charge only if outcome is in  $\top_{h}$   
Publish  $r_{h}$   
TCT Charge for a NotPrior target hit of an  $\varepsilon$ -DP algorithm  
else  
Publish  $\bot$   
// Revise call: Analysed in TCT as a  $2\varepsilon$ -DP Algorithm  $(\mathcal{A} \mid \neg \top_{h})_{\top'}$  and NotPrior target  $\top'$   
Function ReviseCR(h,  $\top'$ ): // Revise target to include  $\top'$   
Input: An identifier  $h$  of a prior ConditionalRelease call, target extension  $\top'$  where  $\top' \cap \top_{h} = \emptyset$   
if  $r_{h} \in \top'$  then // Result is in current target, publish and charge  
Publish  $r_{h}$   
TCT Charge for a NotPrior target hit of an  $2\varepsilon$ -DP algorithm  
else  
Publish  $\bot$ $\top_{h} \gets \top_{h} \cup \top'$  // Update the target to include extension
We show the following (Proof provided in Section D):
Lemma 2.5 (Privacy analysis for Algorithm 2). Each ReviseCR call can be analysed in TCT as a call to a  $2\varepsilon$ -DP algorithm with a NotPrior target  $\top'$ .
Thus, the privacy cost of conditional release followed by a sequence of revise calls is within a factor of 2 (due to the doubled privacy parameter on revise calls) of a single ConditionalRelease call made with the final target. The revisions extension of conditional release facilitates our results for private selection, which are highlighted next.
2.4 Private Top- $k$  Selection
Consider the nature one-shot top- $k$  selection procedure as shown in Algorithm 3: We call each algorithm once and report the  $k$  responses with the highest quality scores. We establish the following:
Lemma 2.6 (Privacy of One-Shot Top- $k$  Selection). Consider one-shot top- $k$  selection (Algorithm 3) on a dataset  $D$  where  $\{\mathcal{A}_i\}$  are  $(\varepsilon, \delta_i)$ -DP. This selection can be simulated exactly in TCT by a sequence of calls to  $(2\varepsilon, \delta)$ -DP algorithms with NotPrior targets that has  $k$  target hits.
As a corollary, assuming  $\varepsilon < 1$ , Algorithm 3 is  $(O(\varepsilon \sqrt{k \log(1 / \delta)}), 2^{-\Omega(k)} + \delta + \sum_{i} \delta_{i})$ -DP for every  $\delta > 0$ .
To the best of our knowledge, our result is the first such bound for one-shot selection from general private candidates. For the case when the only computation performed on  $D$  is a single top-1 selection, we match the "bad example" in [21] (see Theorem J.1). In the regime where  $k > \log(1/\delta)$  our bounds generalize those specific to Lipschitz functions in [9, 29] (see Section J). Moreover, Lemma 2.6 allows for a unified privacy analysis of interactive computations that are interleaved with one-shot selections. We obtain  $O(1)$  overhead per target hit when there are  $\Omega(\log(1/\delta))$  hits in total.