SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Palak Jain
palakj@bu.edu
Iden Kalemaj
ikalemaj@bu.edu
Sofya Raskhodnikova
sofya@bu.edu
Satchit Sivakumar
satchit@bu.edu
ads22@bu.edu
Privacy is a central challenge for systems that learn from sensitive data sets, especially when a system's outputs must be continuously updated to reflect changing data. We consider the achievable error for differentially private continual release of a basic statistic—the number of distinct items—in a stream where items may be both inserted and deleted (the turnstile model). With only insertions, existing algorithms have additive error just polylogarithmic in the length of the stream  $T$ . We uncover a much richer landscape in the turnstile model, even without considering memory restrictions. We show that every differentially private mechanism that handles insertions and deletions has worst-case additive error at least  $T^{1/4}$  even under a relatively weak, event-level privacy definition. Then, we identify a parameter of the input stream, its maximum flippancy, that is low for natural data streams and for which we give tight parameterized error guarantees. Specifically, the maximum flippancy is the largest number of times that the contribution of a single item to the distinct elements count changes over the course of the stream. We present an item-level differentially private mechanism that, for all turnstile streams with maximum flippancy  $w$ , continually outputs the number of distinct elements with an  $O(\sqrt{w} \cdot \text{poly log } T)$  additive error, without requiring prior knowledge of  $w$ . We prove that this is the best achievable error bound that depends only on  $w$ , for a large range of values of  $w$ . When  $w$  is small, the error of our mechanism is similar to the polylogarithmic in  $T$  error in the insertion-only setting, bypassing the hardness in the turnstile model.
Machine learning algorithms are frequently run on sensitive data. In this context, a central challenge is to protect the privacy of individuals whose information is contained in the training set. Differential privacy [27] provides a rigorous framework for the design and analysis of algorithms that publish aggregate statistics, such as parameters of machine learning models, while preserving privacy. In this work, we focus on the model of differential privacy interchangeably called continual observation and continual release that was introduced by Dwork et al. [25] and Chan et al. [13] to study privacy in settings when both the data and the published statics are constantly updated. One of the most fundamental statistics about a data stream is the number of distinct elements it contains (see, e.g., the book by Leskovec et al. [47]). The problem of counting distinct elements has been widely studied, starting with the work of Flajolet and Martin [32], and has numerous applications [2, 30, 49, 42, 4], including monitoring the number of logins from distinct accounts to a streaming service, tracking
the number of different countries represented by people in a chat room, and tracking the number of students signed up for at least one club at a university. Algorithms for this problem are also used as basic building blocks in more complicated data analyses.
We investigate the problem of privately counting the number of distinct elements under continual observation in the turnstile model, which allows both element insertions and deletions. In the continual release model, a data collector receives a sensitive dataset as a stream of inputs and produces, after receiving each input, an output that is accurate for all inputs received so far. The input stream is denoted  $x$  and its length (also called the time horizon) is denoted  $T$ . The elements come from a universe  $\mathcal{U}$ . Each entry in the stream is an insertion (denoted by  $+u$ ) or a deletion (denoted by  $-u$ ) of some element  $u \in \mathcal{U}$  or, alternatively, a no-op (denoted by  $\bot$ ), representing that no update occurred in the current time step. More formally, for a universe  $\mathcal{U}$ , let  $\mathcal{U}_{\pm}$  denote the set  $\{+, -\} \times \mathcal{U} \cup \{\bot\}$  of possible stream entries. The shorthand  $+u$  and  $-u$  is used for the pairs  $(+, u)$  and  $(-, u)$ . Given a vector  $x$  of length  $T$  and an integer  $t \in [T]$ , the vector  $x[1:t]$  denotes the prefix of  $x$  consisting of the first  $t$  entries of  $x$ .
Next, we define the function CountDistinct in the (turnstile) continual release model.
Definition 1.1 (Existence vector, CountDistinct). Fix a universe  $\mathcal{U}$  and a time horizon  $T\in \mathbb{N}$ . For an element  $u\in \mathcal{U}$  and a stream  $x\in \mathcal{U}_{\pm}^{T}$ , the existence vector  $f_{u}(x)\in \{0,1\}^{T}$  is an indicator vector that tracks the existence of element  $u$  in  $x$ : specifically, for each  $t\in [T]$ , the value  $f_{u}(x)[t] = 1$  if and only if there are strictly more insertions than deletions of element  $u$  in  $x[1:t]$ . The function  $\text{CountDistinct}:\mathcal{U}_{\pm}^{T}\to \mathbb{N}^{T}$  returns a vector of the same length as its input, where  $\text{CountDistinct}(x)[t] = \sum_{u\in \mathcal{U}}f_u(x)[t]$  for all  $t\in [T]$ .
The focus of our investigation is the best achievable error in the continual release model for a given time horizon  $T$  and privacy parameters. We study the worst-case (over all input streams and time steps  $t$ ) additive error of privately approximating the distinct elements counts under continual release.
Definition 1.2 (Error of an answer vector and error of a mechanism for CountDistinct). Given an answer vector  $a \in \mathbb{R}^T$ , the error of this vector with respect to the desired function value  $f(x) \in \mathbb{R}^T$  computed on dataset  $x$  is defined as  $\mathsf{ERR}_f(x, a) = \| f(x) - a\|_{\infty}$ . A mechanism for CountDistinct in the continual release model is  $\alpha$ -accurate if it outputs a vector of answers  $a$  with error  $\mathsf{ERR}_{\mathsf{CountDistinct}}(x, a) \leq \alpha$  with probability at least 0.99.
Next, we discuss privacy. Originally, differential privacy [27] was defined in a setting where a data collector outputs the desired information about an entire dataset all at once. We call this the batch model to contrast it with continual release. In the batch model, two datasets are called neighbors if they differ in the data of one individual. There are two natural ways to adapt this definition to the continual release model [25, 13], depending on the desired privacy guarantees.
Definition 1.3 (Neighboring streams). Let  $x, x' \in \mathcal{U}_{\pm}^{T}$  be two streams of length  $T$ . Streams  $x$  and  $x'$  are event-neighbors if one can be obtained from the other by replacing a stream entry with  $\bot$ . Streams  $x$  and  $x'$  are item-neighbors if one can be obtained from the other by replacing a subset of stream entries pertaining to one specific element of  $\mathcal{U}$  with symbol  $\bot$ .
Differential privacy can be defined with respect to any notion of neighboring datasets. There are two privacy parameters:  $\varepsilon > 0$  and  $\delta \in [0,1)$ . An algorithm  $\mathcal{A}$  is  $(\varepsilon, \delta)$ -differentially private (DP) if for all pairs of neighboring datasets  $x, x'$  and all events  $S$  in the output space of  $\mathcal{A}$ ,
$$
\Pr [ \mathcal {A} (x) \in S ] \leq e ^ {\varepsilon} \Pr [ \mathcal {A} (x ^ {\prime}) \in S ] + \delta .
$$
The case when  $\delta = 0$  is referred to as pure differential privacy, and the general case as approximate differential privacy. For event-neighboring (respectively, item-neighboring) streams  $x, x' \in \mathcal{U}_{\pm}^{T}$ , we say that  $\mathcal{A}$  is  $(\varepsilon, \delta)$ -event-level-  $DP$  (respectively, item-level-  $DP$ ). Item-level differential privacy imposes a more stringent requirement than event-level, since it guards against larger changes in the input stream. To contrast with the batch setting, we refer to continual release algorithms as mechanisms.
In the batch setting, where only CountDistinct  $(x)[T]$  is released, there is an  $\varepsilon$ -DP algorithm for counting distinct elements with expected error  $O(1 / \varepsilon)$  since the function CountDistinct  $(x)[T]$  has sensitivity 1—regardless of whether we consider event-level or item-level privacy. Privacy is more challenging in the continual release setting, where we aim to release a sequence of estimates, one for each time  $t$ , and we require that the privacy guarantee hold for the entire sequence of outputs.
Bounds	Item-Level Privacy	Event-Level Privacy
Upper	\(\tilde{O}\left(\min \left(\left(\sqrt{w_x} \log T + \log^3 T\right) \cdot \frac{\sqrt{\log 1/\delta}}{\varepsilon}, \frac{(T \log 1/\delta)^{1/3}}{\varepsilon^{2/3}}, T\right)\right)\)	(Thm. 1.5)
Lower	\(\tilde{\Omega}\left(\min \left(\frac{\sqrt{w_x}}{\varepsilon}, \frac{T^{1/3}}{\varepsilon^{2/3}}, T\right)\right)\)	(Thm. 1.7)
(Thm. 1.6)
Table 1: Summary of our results: bounds on the worst-case additive error for CountDistinct under event-level and item-level  $(\varepsilon, \delta)$ -differential privacy, with  $\varepsilon \leq 1$  and  $\delta = o\left(\frac{\varepsilon}{T}\right)$ . Upper bounds depend on the maximum flippancy  $w_x$  of the input  $x$ , for every  $x$ . Lower bounds apply to the worst-case error of an algorithm taken over all inputs with a given maximum flippancy.
Prior work on privately estimating distinct elements in this setting considered the insertion-only model, exclusively: Bolot et al. [9] show that one can get a sequence of estimates, all of which are within additive error  $\text{poly}(\log T) / \varepsilon$ . Their result holds for both item-level and event-level privacy (which are essentially equivalent for counting distinct elements with only insertions). Follow-up work generalized their mechanism but, again, considered only insertions [35, 29].
We uncover a much richer landscape in the turnstile model, even without considering memory restrictions. We show that any differentially private mechanism that handles insertions and deletions has worst-case additive error at least  $T^{1/4}$  even under event-level privacy, the weaker of the two privacy notions. To overcome this lower bound, we identify a property of the input stream, its maximum flippancy, that is low for natural data streams and for which one can give tight parameterized error guarantees. To define flippancy, recall the notion of the existence vector from Definition 1.1.
Definition 1.4 (Flippancy). Given a stream  $x$  of length  $T$  and an element  $u \in \mathcal{U}$ , the flippancy of  $u$  in  $x$ , denoted by  $\text{flip}(u, x)$ , is the number of pairs of adjacent entries in the existence vector  $f_u(x)$  with different values. That is,  $\text{flip}(u, x) = |\{j \in [T - 1] : f_u(x)[j] \neq f_u(x)[j + 1]\}|$ . The maximum flippancy of a stream  $x$ , denoted  $w_x$ , is  $\max_{u \in \mathcal{U}} \text{flip}(u, x)$ .
In other words, the maximum flippancy is the largest number of times the contribution of a single item to the distinct elements count changes over the course of the stream. We design item-level private mechanisms whose error scales with the maximum flippancy of the stream, even though the maximum flippancy is not an input to the mechanism. We show matching lower bounds for item-level privacy that hold in all parameter regimes. For a large range of the flippancy parameter, we also show a matching lower bound for event-level privacy, via a different argument. This leaves a range with an intriguing gap between item-level and event-level bounds.
1.1 Our results
Our results are summarized in Table 1. Our first result is a mechanism for privately approximating CountDistinct for turnstile streams. For a stream  $x$  of length  $T$  with maximum flippancy  $w_{x}$ , this mechanism is item-level-DP and has error  $O\left(\min (\sqrt{w_x}\cdot \mathrm{polylog}T,T^{1 / 3})\right)$ . Crucially, the mechanism is not given the maximum flippancy upfront.
Theorem 1.5 (Upper bound). For all  $\varepsilon, \delta \in (0,1]$  and sufficiently large  $T \in \mathbb{N}$ , there exists an  $(\varepsilon, \delta)$ -item-level-DP mechanism for CountDistinct that is  $\alpha$ -accurate for all turnstile streams  $x$  of length  $T$ , where
$$
\alpha = \tilde {O} \left(\min  \left(\left(\sqrt {w _ {x}} \log T + \log^ {3} T\right) \cdot \frac {\sqrt {\log 1 / \delta}}{\varepsilon}, \frac {(T \log 1 / \delta) ^ {1 / 3}}{\varepsilon^ {2 / 3}}, T\right)\right),
$$
and  $w_{x}$  is the maximum flippancy of the stream  $x$ .
Since this mechanism is item-level-DP, it is also event-level-DP with the same privacy parameters. The error it achieves is the best possible in terms of dependence only on  $w_{x}$  for item-level privacy, and this error is nearly tight for event-level privacy. When  $w_{x}$  is small, as is the case for many natural streams, our mechanism has error  $O(\mathrm{polylog}T)$ , similar to mechanisms for the insertion-only setting.
Theorem 1.5 can be easily extended to  $\varepsilon$  bounded by any constant larger than 1. We fixed the bound to be 1 to simplify the presentation. Our mechanism has polynomial time and space complexity in the
input parameters, although it does not achieve the typically sublinear space guarantees of streaming algorithms. (See "Bounded Memory" in Section 1.4 for discussion.)
Our lower bounds on the accuracy for CountDistinct for turnstile streams are parametrized by a flippancy bound  $w$ , and apply for streams with maximum flippancy  $w_{x} \leq w$ . For event-level DP, our lower bound shows that for all mechanisms with error guarantees expressed solely in terms of the maximum flippancy  $w_{x}$ , time horizon  $T$ , and privacy parameter  $\varepsilon$ , our CountDistinct mechanism is asymptotically optimal for a large range of values of  $w_{x}$ , namely, for all  $w_{x} \leq T^{1/2}$  and  $w_{x} \geq T^{2/3}$ . The best achievable error for  $w_{x} \in (T^{1/2}, T^{2/3})$  for event-level DP remains an open problem.
Theorem 1.6 (Event-level lower bound). Let  $\varepsilon, \delta \in (0,1]$ , and sufficiently large  $w, T \in \mathbb{N}$  such that  $w \leq T$ . For all  $(\varepsilon, \delta)$ -event-level-  $DP$  mechanisms that are  $\alpha$ -accurate for CountDistinct on turnstile streams of length  $T$  with maximum flippancy at most  $w$ , if  $\delta = o\left(\frac{\varepsilon}{T}\right)$ ,
$$
\alpha = \Omega \left(\min  \left(\frac {\sqrt {w}}{\varepsilon}, \frac {T ^ {1 / 4}}{\varepsilon^ {3 / 4}}, T\right)\right).
$$
In particular, any accuracy bound for event-level algorithms depending only on  $w_{x}$ , as in Theorem 1.5, must grow as  $\Omega \left( \min \left( \frac{\sqrt{w_{x}}}{\varepsilon}, \frac{T^{1/4}}{\varepsilon^{3/4}}, T \right) \right)$ , in all ranges of  $w_{x}$ . This is reflected in Table 1.
For item-level DP, our lower bound on the error matches our upper bound for all regimes of  $w_{x}$  up to polylogarithmic factors.
Theorem 1.7 (Item-level lower bound). Let  $\varepsilon \in (0,1], \delta \in (0,1]$ , and sufficiently large  $w, T \in \mathbb{N}$  such that  $w \leq T$ . For all  $(\varepsilon, \delta)$ -item-level-  $DP$  mechanisms that are  $\alpha$ -accurate for CountDistinct on turnstile streams of length  $T$  with maximum flippancy at most  $w$ ,
1 If  $\delta = o(\varepsilon / T)$ , then  $\alpha = \tilde{\Omega}\Big(\min\left(\frac{\sqrt{w}}{\varepsilon}, \frac{T^{1/3}}{\varepsilon^{2/3}}, T\right)\Big)$ .
2 If  $\delta = 0$ , then  $\alpha = \Omega \left(\min \left(\frac{w}{\varepsilon}, \sqrt{\frac{T}{\varepsilon}}, T\right)\right)$ .
In particular, any accuracy bounds depending only on  $w_{x}$  must grow at least as quickly as the expressions in Table 1.
Variants of the model. All our lower bounds also hold in the strict turnstile model, where element counts never go below 0. They also apply to offline mechanisms that receive the entire input stream before producing output; they do not rely on the mechanism's uncertainty about what comes later in the stream. Furthermore, our item-level lower bounds hold even in the model where each element can be inserted only when it is absent and deleted only when it is present (as is the case, for example, with the "like" counts on social media websites).
1.2 Our techniques
Upper bound techniques: tracking the maximum flippancy. Before describing our algorithmic ideas, we explain the main obstacle to using the techniques previously developed for insertion-only streams [9, 29] in the turnstile setting. Bolot et al. [9] and Epasto et al. [29] used a reduction from CountDistinct to the summation problem. A mechanism for the summation problem outputs, at every time step  $t \in [T]$ , the sum of the first  $t$  elements of the stream. Dwork et al. [27] and Chan et al. [13] designed the binary-tree mechanism to obtain a  $O(\mathrm{polylog} T)$ -accurate mechanism for summation. Given an input stream  $x$  of length  $T$  (to the CountDistinct problem), define a corresponding summation stream  $s_x \in \{-1, 0, 1\}^T$ . At time step  $t \in [T]$ , the entry  $s_x[t]$  equals the difference in the count of distinct elements between time steps  $t - 1$  and  $t$ , i.e.,  $s_x[t] = \mathrm{CountDistinct}(x)[t] - \mathrm{CountDistinct}(x)[t - 1]$ . Then  $\mathrm{CountDistinct}(x)[t]$  is precisely the sum of the first  $t$  elements of  $s_x$ . In the insertion-only model, changing one entry of  $x$  changes at most 2 entries of  $s_x$ , and thus, by group privacy, the binary-tree mechanism has  $O(\mathrm{polylog} T)$  additive error for CountDistinct. For turnstile streams, even under the weaker notion of event-level privacy, a change in the stream  $x$  can cause  $\Omega(T)$  changes to  $s_x$ . To see this, consider the stream consisting of alternating insertions (+u) and deletions (-u) of a single element  $u \in \mathcal{U}$ , and its event-neighboring stream where the first occurrence of +u is replaced with ⊥. This example illustrates that one of the difficulties of the CountDistinct problem for turnstile streams lies with items that switch from being present to absent multiple times over the course of the stream, that is, have high flippancy. We present
a private mechanism that outputs estimates of the count of distinct elements in a turnstile stream with optimal accuracy in terms of maximum flippancy.
Our first key idea allows us to obtain a mechanism, Algorithm 1, that is given as input a flippancy upper bound  $w$ . For a stream  $x$  whose maximum flippancy is bounded by  $w$ , changing to an item-neighbor of  $x$  causes at most  $2w$  changes to the corresponding summation stream  $s_x$ . This observation, combined with a group privacy argument, gives a mechanism with error  $O(w \cdot \text{polylog } T)$  directly from the accuracy guarantee of the binary-tree mechanism for summation. Previous works in the insertion-only model [9, 29] used precisely this approach for the special case  $w = 1$ . To obtain the better  $\sqrt{w}$  dependence on  $w$  in our upper bound, we "open up" the analysis of the binary-tree mechanism. By examining the information stored in each node of the binary tree for the summation stream, we show that changing the occurrences of one item in a stream  $x$  with maximum flippancy at most  $w$  can change the values of at most  $w$  nodes in each level of the binary tree. The  $\sqrt{w}$  dependence in the error then follows from the privacy guarantees of the Gaussian mechanism (used inside the binary-tree mechanism) for approximate differential privacy. This type of noise reduction makes crucial use of the binary tree approach: there are optimized noise addition schemes for prefix sums that improve quantitatively over the binary-tree mechanism (see, e.g., [18, 41]), but it is unclear if they allow the same noise reduction. While our mechanism is only accurate for streams with maximum flippancy at most  $w$ , it is private even for streams that violate this condition. To achieve this, our mechanism ignores stream elements after their flippancy exceeds  $w$ .
The second key idea allows our algorithms to adapt automatically to the maximum flippancy  $w_{x}$  of the input, without the need for an a-priori bound  $w$ . We design a private mechanism, Algorithm 3, that approximately keeps track of the maximum flippancy of the prefix of the stream seen so far and invokes our first mechanism (Algorithm 1) with the current estimated maximum flippancy  $\hat{w}$  as an input. Our main innovation lies in the careful application of the sparse vector algorithm [24] to track the maximum flippancy of the stream. We cannot do this directly, since the sparse vector algorithm achieves good utility only for queries of low sensitivity, and maximum flippancy has global sensitivity  $\Omega(T)$  under item-level changes.
Instead, we track a low sensitivity proxy that indirectly monitors the maximum flippancy  $w_{x}$ : given the current estimate  $\hat{w}$  of the flippancy, we use the sparse vector algorithm to continuously query the number of items in the stream with flippancy above  $\hat{w}$ . This query has sensitivity one for item-level neighbors, as desired, but it is not a priori clear how to use it to upper bound the maximum flippancy of the stream. This is remedied by observing that Algorithm 1, invoked with a flippancy bound  $\hat{w}$ , has the same error (and privacy) guarantee even if at most  $\sqrt{\hat{w}}$  items in the stream have flippancy higher than  $\hat{w}$ . That is, an exact upper bound on the maximum flippancy is not needed to design an accurate mechanism. Items that violate the flippancy bound are safely ignored by Algorithm 1 and do not contribute to the distinct elements count.
When the number of high-flippancy items gets large, we adjust the estimate  $\hat{w}$  and invoke a new instantiation of Algorithm 1. By doubling  $\hat{w}$  each time this happens, we ensure that it remains at most twice the actual maximum flippancy  $w_{x}$ , and that we need only invoke  $\log T$  different copies of Algorithm 1 and the sparse vector algorithm<sup>1</sup>. With these ideas, we obtain an item-level private mechanism that, for all streams  $x$ , has error that scales with  $\sqrt{w_x}$ .
Lower bound techniques. Our lower bounds use the embedding technique introduced by Jain et al. [44] to obtain strong separations between the batch and continual release models of differential privacy. The approach of Jain et al. embeds multiple separate instances of an appropriately chosen base problem on the same sensitive dataset in the batch model into a single instance of a continual release problem. Then, the continual release mechanism can be used to solve multiple instances of the base problem in the batch model. The hardness results in the continual release model follow from lower bounds for the batch model.
A key idea in our event-level lower bound is a connection between the inner product of two binary vectors and the count of distinct elements in the union of those indices where the vector bits equal 1. Estimates of distinct elements counts can thus be used to estimate inner products on a sensitive dataset of binary bits. Lower bounds on the accuracy of private algorithms for estimating inner product queries have been previously established in the batch model through the reconstruction attack of Dinur and Nissim [21]. This connection was used by Mir et al. [50] to provide lower bounds for
<sup>1</sup>All log expressions in this article are base 2.
pan-private algorithms for counting distinct elements. However, continual release and pan-privacy are orthogonal notions, and their results don't imply any lower bounds in our setting. We crucially use deletions to embed multiple instances of inner product queries into a stream: once a query is embedded and the desired estimate is received, the elements inserted to answer that query can be entirely deleted from the stream to obtain a "clean slate" for the next query. We obtain a lower bound of  $T^{1/4}$  on the error of event-level private mechanisms for CountDistinct in turnstile streams.