SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ryan Rogers
Gennady Samorodnitsky
Aaditya Ramdas
In many practical applications of differential privacy, practitioners seek to provide the best privacy guarantees subject to a target level of accuracy. A recent line of work by Ligett et al. [2017], Whitehouse et al. [2022] has developed such accuracy-first mechanisms by leveraging the idea of noise reduction that adds correlated noise to the sufficient statistic in a private computation and produces a sequence of increasingly accurate answers. A major advantage of noise reduction mechanisms is that the analysts only pay the privacy cost of the least noisy or most accurate answer released. Despite this appealing property in isolation, there has not been a systematic study on how to use them in conjunction with other differentially private mechanisms. A fundamental challenge is that the privacy guarantee for noise reduction mechanisms is (necessarily) formulated as ex-post privacy that bounds the privacy loss as a function of the released outcome. Furthermore, there has yet to be any study on how ex-post private mechanisms compose, which allows us to track the accumulated privacy over several mechanisms. We develop privacy filters [Rogers et al., 2016, Feldman and Zrnic, 2021, Whitehouse et al., 2023] that allow an analyst to adaptively switch between differentially private and ex-post private mechanisms subject to an overall differential privacy guarantee.
Although differential privacy has been recognized by the research community as the de-facto standard to ensure the privacy of a sensitive dataset while still allowing useful insights, it has yet to become widely applied in practice despite its promise to ensure formal privacy guarantees. There are notable applications of differential privacy, including the U.S. Census [Abowd et al., 2022], yet few would argue that differential privacy has become quite standard in practice.
One common objection to differential privacy is that it injects noise and can cause spurious results for data analyses. A recent line of work in differential privacy has focused on developing accuracy-first mechanisms that aim to ensure a target accuracy guarantee while achieving the best privacy guarantee [Ligett et al., 2017, Whitehouse et al., 2022]. In particular, these accuracy-first mechanisms do not ensure a predetermined level of privacy, but instead provide ex-post privacy, which allows the resulting privacy loss to depend on the outcome of the mechanism. This is in contrast to the prevalent paradigm of differential privacy that fixes the scale of privacy noise in advance and hopes the result is accurate. With accuracy-first mechanisms, practitioners instead specify the levels of accuracy that would ensure useful data analyses and then aim to achieve such utility with the strongest privacy guarantee.
However, one of the limitations of this line of work is that it is not clear how ex-post privacy mechanisms compose, so if we combine multiple ex-post privacy mechanisms, what is the overall
privacy guarantee? Composition is one of the key properties of differential privacy (when used in a privacy-first manner), so it is important to develop a composition theory for ex-post privacy mechanisms. Moreover, how do we analyze the privacy guarantee when we compose ex-post privacy mechanisms with differentially private mechanisms?
Our work seeks to answer these questions by connecting with another line work in differential privacy on fully adaptive privacy composition. Traditional differential privacy composition results would require the analyst to fix privacy loss parameters, which is then inversely proportional to the scale of noise, for each analysis in advance, prior to any interaction. Knowing that there will be noise, the data scientist may want to select different levels of noise for different analyses, subject to some overall privacy budget. Privacy filters and odometers, introduced in Rogers et al. [2016], provide a way to bound the overall privacy guarantee despite adaptively selected privacy loss parameters. There have since been other works that have improved on the privacy loss bounds in this adaptive setting, to the point of matching (including constants) what one achieves in a nonadaptive setting [Feldman and Zrnic, 2021, Whitehouse et al., 2023].
A natural next step would then be to allow an analyst some overall privacy loss budget to interact with the dataset and the analyst can then determine the accuracy metric they want to set with each new query. As a motivating example, consider some accuracy metric of  $\alpha\%$  relative error to different counts with some overall privacy loss parameters  $\epsilon, \delta$ , so that the entire interaction will be  $(\varepsilon, \delta)$ -differentially private. The first true count might be very large, so the amount of noise that needs to be added to ensure the target  $\alpha\%$  relative error can be huge, and hence very little of the privacy budget should be used for that query, allowing for potentially more results to be returned than an approach that sets an a priori noise level.
A baseline approach to add relative noise would be to add a large amount of noise and then check if the noisy count is within some tolerance based on the scale of noise added, then if the noisy count is deemed good enough we stop, otherwise we scale the privacy loss up by some factor and repeat. We refer to this approach as the doubling approach (see Section 7.1 for more details), which was also used in Ligett et al. [2017]. The primary issue with this approach is that the accumulated privacy loss needs to combine the privacy loss each time we add noise, even though we are only interested in the outcome when we stopped. Noise reduction mechanisms from Ligett et al. [2017], Whitehouse et al. [2022] show how it is possible to only pay for the privacy of the last noise addition. However, it is not clear how the privacy loss will accumulate over several noise reduction mechanisms, since each one ensures ex-post privacy, not differential privacy.
We make the following contributions in this work:
- We present a general (basic) composition result for ex-post privacy mechanisms that can be used to create a privacy filter when an analyst can select arbitrary ex-post privacy mechanisms and (concentrated) differentially private mechanisms.
- We develop a unified privacy filter that combines noise reduction mechanisms — specifically the Brownian Mechanism [Whitehouse et al., 2022] — with traditional (concentrated) differentially private mechanisms.
- We apply our results to the task of releasing counts from a dataset subject to a relative error bound comparing the unified privacy filter and the baseline doubling approach, which uses the privacy filters from Whitehouse et al. [2023].
Our main technical contribution is in the unified privacy filter for noise reduction mechanisms and differentially private mechanisms. Prior work [Whitehouse et al., 2022] showed that the privacy loss of the Brownian noise reduction can be written in terms of a scaled Brownian motion at the time where the noise reduction was stopped. We present a new analysis for the ex-post privacy guarantee of the Brownian mechanism that considers a reverse time martingale, based on a scaled standard Brownian motion. Composition bounds for differential privacy consider a forward time martingale and apply a concentration bound, such as Azuma's inequality [Dwork et al., 2010], so we show how we can construct a forward time martingale from the stopped Brownian motions, despite the stopping times being adaptive, not predictable, at each time. See Figure 1 for a sketch of how we combine reverse time martingales into an overall forward time filtration.
There have been other works that have considered adding relative noise subject to differential privacy. In particular, iReduct [Xiao et al., 2011] was developed to release a batch of queries subject to a relative error bound. The primary difference between that work and our setting here is that we do not
Figure 1: Our privacy filter tracks an accumulated privacy loss over many different mechanisms (forward filtration, X-axis) and stopping when it exceeds a predefined  $\epsilon$  (dotted line). Each mechanism satisfies approximate zCDP (blue dot) or is a noise reduction mechanism (black dots). The latter itself involves several rounds of interaction in a reverse filtration (red arrow) until a stopping criterion based on utility is met (red box). Later queries/mechanisms can depend on past ones.
want to fix the number of queries in advance and we want to allow the queries to be selected in an adaptive way. Xiao et al. [2011] consider a batch of  $m$  queries and initially adds a lot of noise to each count and iteratively checks whether the noisy counts are good enough. The counts that should have smaller noise are then identified and a Laplace based noise reduction algorithm is used to decrease the noise on the identified set. They continue in this way until either all counts are good enough or a target privacy loss is exhausted. There are two scenarios that might arise: (1) all counts satisfy the relative error condition and we should have tried more counts because some privacy loss budget remains, (2) the procedure stopped before some results had a target relative error, so we should have selected fewer queries. In either case, selecting the number of queries becomes a parameter that the data analyst would need to select in advance, which would be difficult to do a priori. In our setting, no such parameter arises. Furthermore, they add up the privacy loss parameters for each count to see if it is below a target privacy loss bound at each step (based on the  $\ell_1$ -general sensitivity), however we show that adding up the privacy loss parameters can be significantly improved on. Other works have modified the definition of differential privacy to accommodate relative error, e.g. Wang et al. [2022].
We start with some basic definitions from differential privacy, beginning with the standard definition from Dwork et al. [2006b,a]. We first need to define what we mean by neighboring datasets, which can mean adding or removing one record to a dataset or changing an entry in a dataset. We will leave the neighboring relation arbitrary, and write  $x, x' \in \mathcal{X}$  to be neighbors as  $x \sim x'$ .
Definition 2.1. An algorithm  $A: \mathcal{X} \to \mathcal{Y}$  is  $(\epsilon, \delta)$ -differentially private if, for any measurable set  $E \subset \mathcal{Y}$  and any neighboring inputs  $x \sim x'$ ,  $\operatorname*{Pr}[A(x) \in E] \leq e^{\epsilon} \operatorname*{Pr}[A(x') \in E] + \delta$ . If  $\delta = 0$ , we say  $A$  is  $\varepsilon$ -DP or simply pure DP.
We define a statistic's sensitivity as the following:  $\Delta_p(f) \coloneqq \max_{x, x' : x \sim x'} \{||f(x) - f(x')||_p\}$ .
A central actor in our analysis will be the privacy loss of an algorithm, which will be a random variable that depends on the outcomes of the algorithm under a particular dataset.
Definition 2.2 (Privacy Loss). Let  $A: \mathcal{X} \to \mathcal{Y}$  be an algorithm and fix neighbors  $x \sim x'$  in  $\mathcal{X}$ . Let  $p^x$  and  $p^{x'}$  be the respective densities of  $A(x)$  and  $A(x')$  on the space  $\mathcal{Y}$  with respect to some reference measure. Then, the privacy loss between  $A(x)$  and  $A(x')$  evaluated at point  $y \in \mathcal{Y}$  is:
$$
\mathcal {L} _ {A} (y; x, x ^ {\prime}) := \log \left(\frac {p ^ {x} (y)}{p ^ {x ^ {\prime}} (y)}\right).
$$
Further, we refer to the privacy loss random variable to be  $\mathcal{L}_A(x,x')\coloneqq \mathcal{L}_A(A(x);x,x')$ . When the algorithm  $A$  and the neighboring datasets are clear from context, we drop them from the privacy loss, i.e.  $\mathcal{L}(y) = \mathcal{L}_A(y;x,x')$  and  $\mathcal{L} = \mathcal{L}_A(x,x')$ .
Many existing privacy composition analyses leverage a variant of DP called (approximate) zero-concentrated  $DP$  (zCDP), introduced by Bun and Steinke [2016]. Recall that the Rényi Divergence of order  $\lambda \geq 1$  between two distributions  $P$  and  $Q$  on the same domain is written as the following where  $p(\cdot)$  and  $q(\cdot)$  are the respective probability mass/density functions,
$$
D _ {\lambda} (P | | Q) := \frac {1}{\lambda - 1} \log \left(\mathbb {E} _ {y \sim P} \left[ \left(\frac {p (y)}{q (y)}\right) ^ {\lambda - 1} \right]\right).
$$
Since we study fully adaptive privacy composition (where privacy parameters can be chosen adaptively), we will use the following conditional extension of approximate zCDP, in which the zCDP parameters of a mechanism  $A$  can depend on prior outcomes.
Definition 2.3 (Approximate zCDP [Whitehouse et al., 2023]). Suppose  $A: \mathcal{X} \times \mathcal{Z} \to \mathcal{Y}$  with outputs in a measurable space  $(\mathcal{V}, \mathcal{G})$ . Suppose  $\delta, \rho: \mathcal{V} \to \mathbb{R}_{\geq 0}$ . We say the algorithm  $A$  satisfies conditional  $\delta(z)$ -approximate  $\rho(z)$ -zCDP if, for all  $z \in \mathcal{Z}$  and any neighboring datasets  $x, x'$ , there exist probability transition kernels  $P', P'', Q', Q'': \mathcal{Z} \times \mathcal{G} \to [0,1]$  such that the conditional outputs are distributed according to the following mixture distributions:
$$
A (x; z) \sim (1 - \delta (z)) P ^ {\prime} (\cdot | z) + \delta (z) P ^ {\prime \prime} (\cdot | z)
$$
$$
A (x ^ {\prime}; z) \sim (1 - \delta (z)) Q ^ {\prime} (\cdot | z) + \delta (z) Q ^ {\prime \prime} (\cdot | z),
$$
where for all  $\lambda \geq 1$ ,  $D_{\lambda}(P'(\cdot | z) \| Q'(\cdot | z)) \leq \rho(z) \lambda$  and  $D_{\lambda}(Q'(\cdot | z) \| P'(\cdot | z)) \leq \rho(z) \lambda$ ,  $\forall z \in \mathcal{Z}$ .
The following results establish the relationship between zCDP and DP and the composition of zCDP.
Lemma 2.1 (Bun and Steinke [2016]). If  $M: \mathcal{X} \to \mathcal{Y}$  is  $(\varepsilon, \delta)$ -DP, then  $M$  is  $\delta$ -approximate  $\varepsilon^2 / 2 - zCDP$ . If  $M$  is  $\delta$ -approximate  $\rho$ -zCDP, then  $M$  is  $(\rho + 2\sqrt{\rho\log(1 / \delta'')}$ ,  $\delta + \delta'')$ -DP for  $\delta'' > 0$ .
Lemma 2.2 (Bun and Steinke [2016]). If  $M_1: \mathcal{X} \to \mathcal{Y}$  is  $\delta_1$ -approximate  $\rho_1$ -zCDP and  $M_2: \mathcal{X} \times \mathcal{Y} \to \mathcal{Y}'$  is  $\delta_2$ -approximate  $\rho_2$ -zCDP in its first coordinate, then  $M: \mathcal{X} \to \mathcal{Y}'$  where  $M(x) = (M_1(x), M_2(x, M_1(x)))$  is  $(\delta_1 + \delta_2)$ -approximate  $(\rho_1 + \rho_2)$ -zCDP.
3 Privacy Filters
In order for us to reason about the overall privacy loss of an interaction with a sensitive dataset, we will use the framework of privacy filters, introduced in Rogers et al. [2016]. Privacy filters allow an analyst to adaptively select privacy parameters as a function of previous outcomes until some stopping time that determines whether a target privacy budget has been exhausted. To denote that privacy parameters may depend on previous outcomes, we will write  $\rho_{n}(x)$  to mean the privacy parameter selected at round  $n$  that could depend on the previous outcomes  $A_{1:n - 1}(x)$ , and similarly for  $\delta_n(x)$ . We now state the definition of privacy filters in the context of approximate zCDP mechanisms.
Definition 3.1 (Privacy Filter). Let  $(A_{n}:\mathcal{X}\to \mathcal{Y})_{n\geq 1}$  be an adaptive sequence of algorithms such that, for all  $n\geq 1$ ,  $A_{n}(\cdot ;y_{1:n - 1})$  is  $\delta_n(y_{1:n - 1})$ -approximate  $\rho_{n}(y_{1:n - 1})$ -zCDP for all  $y_{1:n - 1}\in \mathcal{Y}^{n - 1}$ . Let  $\epsilon >0$  and  $\delta \geq 0$  be fixed privacy parameters. Then, a function  $N:\mathcal{Y}^{\infty}\to \mathbb{N}$  is an  $(\epsilon ,\delta)$ -privacy filter if
1. for all  $(y_{1},y_{2},\dots)\in \mathcal{V}^{\infty}$ ,  $N(y_{1},y_{2},\dots)$  is a stopping time with respect to the natural filtration generated by  $(A_{n}(x))_{n\geq 1}$ , and
2.  $A_{1:N(\cdot)}(\cdot)$  is  $(\epsilon, \delta)$ -differentially private where  $N(x) = N(A_1(x), A_2(x), \dots)$ .
Whitehouse et al. [2023] showed that we can use composition bounds from traditional differential privacy (which required setting privacy parameters for the interaction prior to any interaction) in the more adaptive setting, where privacy parameters can be set before each query.
Theorem 1. Suppose  $(A_{n}:\mathcal{X}\to \mathcal{Y})_{n\geq 1}$  is a sequence of algorithms such that, for any  $n\geq 1$ $A_{n}(\cdot ;y_{1:n - 1})$  is  $\delta_n(y_{1:n - 1})$  -approximate  $\rho_{n}(y_{1:n - 1})$  -zCDP for all  $y_{1:n - 1}$ . Let  $\epsilon >0$  and  $\delta \geq 0$
$\delta'' > 0$  be fixed privacy parameters. Consider the function  $N: \mathbb{R}_{\geq 0}^{\infty} \to \mathbb{N}$  given by the following where  $\delta < \sum_{m \leq n+1} \delta_m(y_{1:m-1})$  for all  $y_1, y_2, \dots$  and
$$
N (y _ {1}, y _ {2}, \dots) := \inf  \left\{n: \epsilon <   2 \sqrt {\log \left(\frac {1}{\delta^ {\prime \prime}}\right) \sum_ {m \leq n + 1} \rho_ {m} (y _ {1 : m - 1})} + \sum_ {m \leq n + 1} \rho_ {m} (y _ {1: m - 1}) \right\}.
$$
Then, the algorithm  $A_{1:N(\cdot)}(\cdot)$  is  $(\epsilon, \delta + \delta'')$ -DP, where  $N(x) := N((A_n(x))_{n \geq 1})$ . In other words,  $N$  is an  $(\epsilon, \delta)$ -privacy filter.
4 Ex-post Private Mechanisms
Although privacy filters allow a lot of flexibility to a data analyst in how they interact with a sensitive dataset while still guaranteeing a fixed privacy budget, there are some algorithms that ensure a bound on privacy that is adapted to the dataset. Ex-post private mechanisms define privacy loss as a probabilistic bound which can depend on the algorithm's outcomes, so some outcomes might contain more information about an individual than others [Ligett et al., 2017, Whitehouse et al., 2022]. Note that ex-post private mechanisms do not have any fixed a priori bound on the privacy loss, so by default they cannot be composed in a similar way to differentially private mechanisms.
Definition 4.1. Let  $A: \mathcal{X} \times \mathcal{Y} \to \mathcal{Z}$  be an algorithm and  $\mathcal{E}: \mathcal{Z} \times \mathcal{Y} \to \mathbb{R}_{\geq 0}$  a function. We say  $A(\cdot; y)$  is  $(\mathcal{E}(\cdot; y), \delta(y))$ -ex-post private for all  $y \in \mathcal{Y}$  if, for any neighboring inputs  $x \sim x'$ , we have  $\operatorname*{Pr}\left[\mathcal{L}(A(x; y)) > \mathcal{E}(A(x; y); y)\right] \leq \delta(y)$  for all  $y \in \mathcal{Y}$ .
We next define a single noise reduction mechanism, which will interactively apply sub-mechanisms and stop at some time  $k$ , which can be random. Each iterate will use a privacy parameter from an increasing sequence of privacy parameters  $(\varepsilon^{(n)}: n \geq 1)$  set in advance and the overall privacy will only depend on the last privacy parameter  $\epsilon^{(k)}$ , despite releasing noisy values with parameters  $\varepsilon^{(i)}$  for  $i \leq k$ . Noise reduction algorithms will allow us to form ex-post private mechanisms because the privacy loss will only depend on the final outcome. We will write  $M: \mathcal{X} \to \mathcal{Y}^*$  to be any algorithm mapping databases to sequences of outputs in  $\mathcal{Y}$ , with intermediate mechanisms written as  $M^{(k)}: \mathcal{X} \to \mathcal{Y}$  for the  $k$ -th element of the sequence and  $M^{(1:k)}: \mathcal{X} \to \mathcal{Y}^k$  for the first  $k$  elements. Let  $\mu$  be a probability measure on  $\mathcal{Y}$ , for each  $k \geq 1$  let  $\mu_k$  be a probability measure on  $\mathcal{Y}^k$ , and let  $\mu^*$  be a probability measure on  $\mathcal{Y}^*$ . We assume that the law of  $M^{(k)}(x)$  on  $\mathcal{Y}$  is equivalent to  $\mu$ , the law of  $M(x)$  on  $\mathcal{Y}^*$  is equivalent to  $\mu^*$ , and the law of  $M^{(1:k)}(x)$  on  $\mathcal{Y}^k$  is equivalent to  $\mu_k$  for every  $k$  and every  $x$ . Furthermore, we will write  $\mathcal{L}$  to be the privacy loss of  $M$ ,  $\mathcal{L}^{(k)}$  to be the privacy loss of  $M^{(k)}$ , and  $\mathcal{L}^{(1:k)}$  to be the privacy loss of the sequence of mechanisms  $M^{(1:k)}$ . We then define noise reduction mechanisms more formally.
Definition 4.2 (Noise Reduction Mechanism). Let  $M: \mathcal{X} \to \mathcal{Y}^{\infty}$  be a mechanism mapping sequences of outcomes and  $x, x'$  be any neighboring datasets. We say  $M$  is a noise reduction mechanism if for any  $k \geq 1$ ,  $\mathcal{L}^{(1:k)} = \mathcal{L}^{(k)}$ .
We will assume there is a fixed grid of time values  $t^{(1)} > t^{(2)} > \dots > t^{(k)} > 0$ . We will typically think of the time values as being inversely proportional to the noise we add, i.e.  $t^{(i)} = \left(1 / \varepsilon^{(i)}\right)^{2}$  where  $\varepsilon^{(1)} < \varepsilon^{(2)} < \dots$ . An analyst will not have a particular stopping time set in advance and will instead want to stop interacting with the dataset as a function of the noisy answers that have been released. It might also be the case that the analyst wants to stop based on the outcome and the privatized dataset, but in this work we consider stopping times that can only depend on the noisy outcomes or possibly some public information, not the underlying dataset.
Definition 4.3 (Stopping Function). Let  $A: \mathcal{X} \to \mathcal{Y}^{\infty}$  be a noise reduction mechanism. For  $x \in \mathcal{X}$ , let  $(\mathcal{F}^{(k)}(x))_{k \in \mathbb{N}}$  be the filtration given by  $\mathcal{F}^{(k)}(x) \coloneqq \sigma(A^{(i)}(x) : i \leq k)$ . A function  $T: \mathcal{Y}^{\infty} \to \mathbb{N}$  is called a stopping function if for any  $x \in \mathcal{X}$ ,  $T(x) \coloneqq T(A(x))$  is a stopping time with respect to  $(\mathcal{F}^{(k)}(x))_{k \geq 1}$ . Note that this property does not depend on the choice of measures  $\mu, \mu^{*}$  and  $\mu_k$ .
We now recall the noise reduction mechanism with Brownian noise [Whitehouse et al., 2022].
Definition 4.4 (Brownian Noise Reduction). Let \( f: \mathcal{X} \to \mathbb{R}^d \) be a function and \( (t^{(k)})_{k \geq 1} \) be a sequence of time values. Let \( (B^{(t)})_{t \geq 0} \) be a standard \( d \)-dimensional Brownian motion and \( T:
$(\mathbb{R}^d)^\infty \to \mathbb{N}$  be a stopping function. The Brownian mechanism associated with  $f$ , time values  $(t^{(k)})_{k\geq 1}$ , and stopping function  $T$  is the algorithm  $\mathcal{BM}:\mathcal{X}\rightarrow ((0,t^{(1)}]\times \mathbb{R}^d)^*$  given by
$$
B M (x) := \left(t ^ {(k)}, f (x) + B ^ {(t ^ {(k)})}\right) _ {k \leq T (x)}.
$$
We then have the following result.
Lemma 4.1 (Whitehouse et al. [2022]). The Brownian Noise Reduction mechanism  $BM$  is a noise reduction algorithm for a constant stopping function  $T(x) = k$ . Furthermore, we have for any stopping function  $T: \mathcal{Y}^{\infty} \to \mathbb{N}$ , the noise reduction property still holds, i.e.  $\mathcal{L}^{(1:T(x))} = \mathcal{L}^{(T(x))}$ .
Another noise reduction mechanism uses Laplace noise from Koufogiannis et al. [2017], Ligett et al. [2017], which we consider in Appendix C.
5 General Composition of Ex-post Private Mechanisms
We consider combining zCDP mechanisms with mechanisms that satisfy ex-post privacy. We consider a sequence of mechanisms  $(A_{n}:\mathcal{X}\to \mathcal{Y})_{n\geq 1}$  where each mechanism may depend on the previous outcomes. At each round, an analyst will use either an ex-post private mechanism or an approximate zCDP mechanism, in either case the privacy parameters may depend on the previous results as well. We will write  $\delta_n(x)\coloneqq \delta_n(A_1(x),\dots ,A_{n - 1})(x))$ $\rho_{n}(x)\coloneqq \rho (A_{1}(x),\dots ,A_{n - 1}(x))$  , and  $\mathcal{E}_n(A_n(x);x)\coloneqq \mathcal{E}_n(A_n(x);A_1(x),\dots ,A_{n - 1}(x))$