SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Joel Daniel Andersson
Basic Algorithms Research Copenhagen
jda@di.ku.dk
Rasmus Pagh
pagh@di.ku.dk
In privacy under continual observation we study how to release differentially private estimates based on a dataset that evolves over time. The problem of releasing private prefix sums of  $x_{1}, x_{2}, x_{3}, \dots \in \{0, 1\}$  (where the value of each  $x_{i}$  is to be private) is particularly well-studied, and a generalized form is used in state-of-the-art methods for private stochastic gradient descent (SGD). The seminal binary mechanism privately releases the first  $t$  prefix sums with noise of variance polylogarithmic in  $t$ . Recently, Henzinger et al. and Denisov et al. showed that it is possible to improve on the binary mechanism in two ways: The variance of the noise can be reduced by a (large) constant factor, and also made more even across time steps. However, their algorithms for generating the noise distribution are not as efficient as one would like in terms of computation time and (in particular) space. We address the efficiency problem by presenting a simple alternative to the binary mechanism in which 1) generating the noise takes constant average time per value, 2) the variance is reduced by a factor about 4 compared to the binary mechanism, and 3) the noise distribution at each step is identical. Empirically, a simple Python implementation of our approach outperforms the running time of the approach of Henzinger et al., as well as an attempt to improve their algorithm using high-performance algorithms for multiplication with Toeplitz matrices.
There are many actors in society that wish to publish aggregate statistics about individuals, be it for financial or social utility. Netflix might recommend movies based on other users' preferences, and policy might be driven by information on average incomes across groups. Whatever utility one has in mind however, it should be balanced against the potential release of sensitive information. While it may seem anodyne to publish aggregate statistics about users, doing it without consideration to privacy may expose sensitive information of individuals (Dinur & Nissim, 2003). Differential privacy offers a framework for dealing with these problems in a mathematically rigorous way.
A particular setting is when statistics are updated and released continually, for example a website releasing its number of visitors over time. Studying differential privacy in this setup is referred to as differential privacy under continual observation (Dwork et al., 2010; Dwork & Roth, 2013). A central problem in this domain is referred to as differentially private counting under continual observation (Chan et al., 2011; Dwork et al., 2010), continual counting for short. It covers the following problem: a binary stream  $x_{1}, x_{2}, x_{3}, \ldots$  is received one element at a time such that  $x_{t}$  is received in round  $t$ . The objective is to continually output the number of 1s encountered up to each time step while maintaining differential privacy. We consider two streams  $x$  and  $x'$  as neighboring if they are identical except for a single index  $i$  where  $x_{i} \neq x_{i}'$ . It suffices to study the setting in which there is a known upper bound  $T$  on the number of prefix sums to release — algorithms for the case of unbounded streams then follow by a general reduction (Chan et al., 2011).
Table 1: Comparison between different  $\rho$  -zCDP mechanisms for continual counting.
Mechanism M	Var[M(t)]·2ρ/ log2T	Time to produce all T outputs	Space	Matrix type
Binary mech.	1	O(T)	O(log T)	sparse
Honaker Online	0.5 + o(1)	O(T log T)	O(log T)	sparse
Denisov et al.	0.0487... + o(1)*	O(T2)	O(T2)	dense
Henzinger et al.	0.0487... + o(1)	O(T log T)**	O(T)	Toeplitz
Our mechanism	0.25 + o(1)	O(T)	O(log T)	sparse
"Honaker Online" refers to the "Estimation from below" variant in Honaker (2015) as implemented in Kairouz et al. (2021). There is no explicit bound on variance in Denisov et al. (2022), but the method finds an optimal matrix factorization so it should achieve same variance \* as Henzinger et al. (2023) up to lower order terms. As to their efficiency, Denisov et al. (2022) contains empirical work for reducing the time and space usage by approximating the matrices involved as a sum of a banded matrix and a low-rank approximation, but without formal guarantees. The time usage in \*\* assumes implementing the matrix-vector product using FFT. Leveraging FFT for continual observation is not a novel approach (Choquette-Choo et al., 2023). All sparse matrices have  $O(\log T)$  nonzero entries per row or column, and all dense matrices have  $\Omega (T^2)$  nonzero entries - a Toeplitz matrix can be seen as intermediate in the sense of having  $O(T)$  unique diagonals, allowing for efficient storage and multiplication.
Aside from the natural interpretation of continual counting as the differentially private release of user statistics over time, mechanisms for continual counting (and more generally for releasing prefix sums) are used as a subroutine in many applications. Such a mechanism is for example used in Google's privacy-preserving federated next word prediction model (McMahan & Thakurta, 2022; Kairouz et al., 2021; Choquette-Choo et al., 2023), in non-interactive local learning (Smith et al., 2017), in stochastic convex optimization (Han et al., 2022) and in histogram estimation (Cardoso & Rogers, 2022; Chan et al., 2012; Huang et al., 2022; Upadhyay, 2019) among others.
Given the broad adoption of continual counting as a primitive, designing algorithms for continual counting that improve constants in the error while scaling well in time and space is of practical interest.
1.1 Our contributions
In this paper we introduce the Smooth Binary Mechanism, a differentially private algorithm for the continual counting problem that improves upon the original binary mechanism by Chan et al. (2011); Dwork et al. (2010) in several respects, formalized in Theorem 1.1 and compared to in Table 1.
Theorem 1.1 (Smooth Binary Mechanism). For any  $\rho >0$ ,  $T > 1$ , there is an efficient  $\rho -zCDP$  continual counting mechanism  $\mathcal{M}$ , that on receiving a binary stream of length  $T$  satisfies
$$
\operatorname {V a r} [ \mathcal {M} (t) ] = \frac {1 + o (1)}{8 \rho} \log_ {2} (T) ^ {2}
$$
where  $\mathcal{M}(t)$  is the output prefix sum at time  $t$ , while only requiring  $O(\log T)$  space,  $O(T)$  time to output all  $T$  prefix sums, and where the error is identically distributed for all  $1 \leq t \leq T$ .
Our mechanism retains the scalability in time and space of the binary mechanism while offering an improvement in variance by a factor of  $4 - o(1)$ . It also has the same error distribution in every step by design, which could make downstream applications easier to analyze.
Sketch of technical ideas. Our starting point is the binary mechanism which, in a nutshell, uses a complete binary tree with  $\geq T + 1$  leaves (first  $T$  leaves corresponding to  $x_{1},\ldots ,x_{T}$ ) in which each node contains the sum of the leaves below, made private by adding random noise (e.g. from a Gaussian distribution). To estimate a prefix sum  $\sum_{i = 1}^{t}x_{i}$  we follow the path from the root to the leaf storing  $x_{t + 1}$ . Each time we go to a right child the sum stored in its sibling node is added to a counter. An observation, probably folklore, is that it suffices to store sums for nodes that are left children, so
suppose we do not store any sum in nodes that are right children. The number of terms added when computing the prefix sum is the number of 1s in the binary representation  $\mathrm{bin}(t)$  of  $t$ , which encodes the path to the leaf storing  $x_{t+1}$ . The sensitivity of the tree with respect to  $x_t$ , i.e., the number of node counts that change by 1 if  $x_t$  changes, is the number of 0s in  $\mathrm{bin}(t-1)$ . Our idea is to only use leaves that have balanced binary representation, i.e. same number of 0s and 1s (assuming the height  $h$  of the tree is an even integer). To obtain  $T$  useful leaves we need to make the tree slightly deeper — it turns out that height  $h$  slightly more than  $\log_2(T)$  suffices. This has the effect of making the sensitivity of every leaf  $h/2$ , and the noise in every prefix sum a sum of  $h/2$  independent noise terms.
Limitations. As shown in Table 1 and Section 4, the smooth binary mechanism, while improving on the original binary mechanism, cannot achieve as low variance as matrix-based mechanisms such as Henzinger et al. (2023). However, given that the scaling of such methods can keep them from being used in practice, our variant of the binary mechanism has practical utility in large-scale settings.
All good methods for continual counting that we are aware of can be seen as instantiations of the factorization mechanism (Li et al., 2015), sometimes also referred to as the matrix mechanism. These methods perform a linear transformation of the data, given by a matrix  $R$ , then add noise according to the sensitivity of  $Rx$ , and finally obtain the sequence of estimates by another linear transformation given by a matrix  $L$ . To obtain unbiased estimates, the product  $LRx$  needs to be equal to the vector of prefix sums, that is,  $LR$  must be the lower triangular all 1s matrix.
Seminal works introducing the first variants of the binary mechanism are due to Chan, Shi, and Song (2011) and Dwork, Naor, Pitassi, and Rothblum (2010), but similar ideas were proposed independently in Hay, Rastogi, Miklau, and Suciu (2010) and Gehrke, Wang, and Xiao (2010). Honaker (2015) noticed that better estimators are possible by making use of all information in the tree associated with the binary tree method. A subset of their techniques can be leveraged to reduce the variance of the binary mechanism by up to a factor 2, at some cost of efficiency, as shown from their implementation in Kairouz, Mcmahan, Song, Thakkar, Thakurta, and Xu (2021).
A number of recent papers have studied improved choices for the matrices  $L, R$ . Denisov, McMahan, Rush, Smith, and Thakurta (2022) treated the problem of finding matrices leading to minimum largest variance on the estimates as a convex optimization problem, where (at least for  $T$  up to 2048) it was feasible to solve it. To handle a larger number of time steps they consider a similar setting where a restriction to banded matrices makes the method scale better, empirically with good error, but no theoretical guarantees are provided.
Fichtenberger, Henzinger, and Upadhyay (2023) gave an explicit decomposition into lower triangular matrices, and analyzed its error in the  $\ell_{\infty}$  metric. The matrices employed are Toeplitz (banded) matrices. Henzinger, Upadhyay, and Upadhyay (2023) analyzed the same decomposition with respect to mean squared error of the noise, and showed that it obtains the best possible error among matrix decompositions where  $L$  and  $R$  are square matrices, up to a factor  $1 + o(1)$  where the  $o(1)$  term vanishes when  $T$  goes to infinity.
A breakdown of how our mechanism compares to existing ones is shown in Table 1. While not achieving as small an error as the factorization mechanism of Henzinger et al. (2023), its runtime and small memory footprint allows for better scaling for longer streams. For concreteness we consider privacy under  $\rho$ -zero-Concentrated Differential Privacy (zCDP) (Bun & Steinke, 2016), but all results can be expressed in terms of other notions of differential privacy.
Binary representation of numbers. We will use the notation  $\mathrm{bin}(n)$  to refer to the binary representation of a number  $n\in [2^h]$ , where  $h > 0$  is an integer, and let  $\mathrm{bin}(n)$  be padded with leading zeros to  $h$  digits. For example,  $\mathrm{bin}(2) = 0\mathrm{b}010$  for a tree of height  $h = 3$ . When indexing such a number we let index  $i$  refer to the  $i^{th}$  least significant bit, e.g.  $\mathrm{bin}(2)_1 = 1$ . We will also refer to prefixes and suffixes of binary strings, and we use the convention that a prefix of a binary string includes its most significant bits.
Partial sums (p-sums). To clear up the notation we use the concept of p-sums  $\Sigma[i, j]$  where  $\Sigma[i, j] = \sum_{t=i}^{j} x_t$ . We will furthermore use the concept of noisy p-sums
$$
\widehat {\Sigma} [ i, j ] = \Sigma [ i, j ] + X [ i, j ], X [ i, j ] \sim \mathcal {F}
$$
where  $\mathcal{F}$  is a suitable distribution for the DP paradigm, e.g. Laplacian or Gaussian. For convenience we also define  $\hat{x}_t = \widehat{\Sigma}[t,t]$ , i.e.  $\hat{x}_t$  is the single stream element with noise added.
2.1 Continual observation of bit stream
Given an integer  $T > 1$  we consider a finite length binary stream  $x = (x_{1}, x_{2}, \ldots, x_{T})$ , where  $x_{t} \in \{0, 1\}$ ,  $1 \leq t \leq T$ , denotes the bit appearing in the stream at time  $t$ .
Definition 2.1 (Continual Counting Query). Given a stream  $x \in \{0,1\}^T$ , the count for the stream is a mapping  $c: \{1,\ldots,T\} \to \mathbb{Z}$  such that  $c(t) = \sum_{i=1}^{t} x_i$ .
Definition 2.2 (Counting Mechanism). A counting mechanism  $\mathcal{M}$  takes a stream  $x\in \{0,1\} ^T$  and produces a (possibly random) vector  $\mathcal{M}_x\in \mathbb{R}^T$  where  $(\mathcal{M}_x)_t$  is a function of the first  $t$  elements of the stream. For convenience we will write  $\mathcal{M}(t)$  for  $(\mathcal{M}_x)_t$  when there is little chance for ambiguity.
To analyze a counting mechanism from the perspective of differential privacy, we also need a notion of neighboring streams.
Definition 2.3 (Neighboring Streams). Streams  $x, x' \in \{0,1\}^T$  are said to be neighboring, denoted  $x \sim x'$ , if  $|\{i | x_i \neq x_i'\}| = 1$ .
Intuitively, for a counting mechanism to be useful at a given time  $t$ , we want it to minimize  $|\mathcal{M}(t) - c(t)|$ . We consider unbiased mechanisms that return the true counts in expectation and we focus on minimizing  $\mathrm{Var}[\mathcal{M}(t) - c(t)]$ .
2.2 Differential privacy
For a mechanism to be considered differentially private, we require that the outputs for any two neighboring inputs are are indistinguishable. We will state our results in terms of  $\rho$ -zCDP:
Definition 2.4 (Concentrated Differential Privacy (zCDP) (Bun & Steinke, 2016)). For  $\rho > 0$ , a randomized algorithm  $\mathcal{A} : \mathcal{X}^n \to \mathcal{Y}$  is  $\rho$ -zCDP if for any  $\mathcal{D} \sim \mathcal{D}'$ ,  $D_{\alpha}(\mathcal{A}(\mathcal{D})||\mathcal{A}(\mathcal{D}')|) \leq \rho \alpha$  for all  $\alpha > 1$ , where  $D_{\alpha}(\mathcal{A}(\mathcal{D})||\mathcal{A}(\mathcal{D}')|$  is the  $\alpha$ -Rényi divergence between  $\mathcal{A}(\mathcal{D})$  and  $\mathcal{A}(\mathcal{D}')$ .
In the scenario where we are looking to release a real-valued function  $f(\mathcal{D})$  taking values in  $\mathbb{R}^d$ , we can achieve zCDP by adding Gaussian noise calibrated to the  $\ell_2$ -sensitivity of  $f$ .
Lemma 2.5 (Gaussian Mechanism (Bun & Steinke, 2016)). Let  $f: \mathcal{X}^n \to \mathbb{R}^d$  be a function with global  $\ell_2$ -sensitivity  $\Delta := \max_{\mathcal{D} \sim \mathcal{D}'} \| f(\mathcal{D}) - f(\mathcal{D}') \|_2$ . For a given data set  $\mathcal{D} \in \mathcal{X}^n$ , the mechanism that releases  $f(\mathcal{D}) + \mathcal{N}(0, \frac{\Delta^2}{2\rho})^d$  satisfies  $\rho$ -zCDP.
It is known that  $\rho$ -zCDP implies  $(\rho + 2\sqrt{\rho \ln(1 / \delta)}, \delta)$ -differential privacy for every  $\delta > 0$  (Bun & Steinke, 2016).
Lastly, when comparing counting mechanisms based on Gaussian noise, we note that it is sufficient to look at the variance. For a single output  $\mathcal{M}(t)$ , the variance  $\mathrm{Var}[\mathcal{M}(t)]$  is the only relevant metric, as it completely describes the output distribution. For a series of outputs  $\mathcal{M}_x$ , and related norms  $\| \mathcal{M}_x\| _p$ , we note that a mechanism with lower variance will be more concentrated in each coordinate and have lower  $p^{th}$  moment, allowing for tighter bounds on the norm.
2.3 Differentially private continual counting
We next describe two approaches to continual counting.
Binary mechanism. The binary mechanism (Chan et al., 2011; Dwork et al., 2010; Gehrke et al., 2010; Hay et al., 2010) can be considered the canonical mechanism for continual counting. In this section we present a variant of it where only left subtrees are used. The mechanism derives its name
from the fact that a binary tree is built from the input stream. Each element from the stream is assigned a leaf in the binary tree, and each non-leaf node in the tree represents a p-sum of all elements in descendant leaves. All values are stored noisily in nodes, and nodes are added together to produce a given prefix sum. Such a binary tree is illustrated in Figure 1(a).
(a) Binary mechanism tree for  $T = 7$ .
(b) Computation of  $\mathcal{M}(6)$  using Algorithm 1.
Figure 1: Binary trees for a sequence of length  $T = 7$ . In Figure 1(b) each leaf is labeled by  $\mathrm{bin}(t - 1)$ , and it illustrates how the prefix sum up to  $t = 6$  can be computed from  $\mathrm{bin}(t)$ . Blue nodes describe the path taken by Algorithm 1, and the sum of red nodes form the desired output  $\mathcal{M}(6)$ .
We will return for a closer analysis of the binary mechanism in Section 3.1. For now we settle for stating that the  $\rho$ -zCDP binary mechanism achieves  $\mathrm{Var}[\mathcal{M}(t)] = O(\log(T)^2)$  and is known to be computationally efficient: to release all prefix sums up to a given time  $t \leq T$  requires only  $O(\log(T))$  space and  $O(t)$  time.
Factorization mechanism. The binary mechanism belongs to a more general class of mechanisms called factorization mechanisms (Li et al., 2015), sometimes also referenced as matrix mechanisms. Computing a prefix sum is a linear operation on the input stream  $x \in \mathbb{R}^T$ , and computing all prefix sums up to a given time  $T$  can therefore be represented by a matrix  $A \in \mathbb{R}^{T \times T}$ , where  $c(t) = (Ax)_t$ .  $A$  is here a lower-triangular matrix of all 1s. The factorization mechanism characterizes solutions to the continual counting problem by factorizing  $A$  as  $A = LR$  with corresponding mechanism  $\mathcal{M}_x = L(Rx + z)$ ,  $z \sim \mathcal{F}^n$  where  $n$  is the dimension of  $Rx$ .
Intuitively  $Rx$  represents linear combinations of the stream, which are made private by adding noise  $z$ , and which then are aggregated by  $L$ . To achieve  $\rho$ -zCDP for this mechanism, we let  $z \sim \mathcal{N}(0, \frac{\Delta^2}{2\rho})^n$  where  $\Delta = \max_i \|Re_i\|_2$ ,  $e_i$  being the  $i^{th}$  unit vector. It follows that the corresponding output noise becomes  $Lz$  with coordinates  $(Lz)_i \sim \mathcal{N}(0, \frac{\Delta^2}{2\rho} \|L_i\|_2^2)$  where  $L_i$  is the  $i^{th}$  row in  $L$ .
Extension to multidimensional input. While we have so far assumed one-dimensional input, any one-dimensional  $\rho$ -zCDP factorization mechanism naturally generalizes to higher dimensions. This fact enables factorization mechanisms to be applied to the problem of private learning where gradients are summed. We sketch an argument for this fact in Appendix B.
3 Our mechanism
To introduce our variant of the binary mechanism, we need to return to the original mechanism.
3.1 Closer analysis of the binary mechanism
As has been pointed out in earlier work (Denisov et al., 2022; Henzinger et al., 2023), a naive implementation of the binary mechanism will lead to variance that is non-uniform with respect to time: the number of 1s in the bitwise representation of  $t$  determines the variance. To underline this, consider the pseudo-code in Algorithm 1 for computing a prefix sum given a binary mechanism tree structure. The code assumes that the tree has  $\geq t + 1$  leaves, a detail that only matters when  $t$  is a power of 2 and allows us to never use the root node. To illustrate a simple case, see Figure 1(b).
Algorithm 1 Prefix Sum for Binary Mechanism
1: Input: binary tree of height  $h$  storing  $\widehat{\Sigma}[a, b]$  for  $b \leq t$ , time  $t \leq 2^h - 1$   
2: output  $\gets 0$   
3:  $s \gets \mathrm{bin}(t)$  {padded to  $h$  bits by adding zeros}  
4:  $a \gets 1; b \gets 2^h$   
5: for  $i = h - 1$  to 0 do  
6:  $d = \lfloor \frac{a + b}{2} \rfloor$   
7: if  $s_i = 1$  then  
8: output  $\gets \mathrm{output} + \widehat{\Sigma}[a, d]$   
9:  $a \gets d + 1$   
10: else  
11:  $b \gets d$   
12: end if  
13: end for  
14: return output
We can make the following two observatons:
-  $\operatorname{bin}(t - 1)$  encodes a path from the root to the leaf where  $x_{t}$  is stored.
- To compute prefix sums using Algorithm 1, we only need to store values in "left children".
Combining these observations, we get the following result:
Proposition 3.1. The squared  $\ell_2$ -sensitivity  $\Delta^2$  of  $x_{t}$  is equal to the number of 0s in  $\mathrm{bin}(t - 1)$ .
To see this, note that the number of 0s in  $\mathrm{bin}(t - 1)$  is equal to the number of left-children that are passed through to reach the given node. Changing the value of  $x_{t}$  impacts all its ancestors, and since only left-children are used for prefix sums, the result immediately follows. This does not address the volatility of variance with respect to time. However, studying Algorithm 1 gives the reason:
Proposition 3.2.  $\operatorname{Var}[\mathcal{M}(t)]$  is proportional to the number of  $1s$  in  $\mathrm{bin}(t)$ .
The result follows from the fact that the number of terms added together for a given prefix sum  $c(t)$  is equal to the number of 1s in  $\mathrm{bin}(t)$ , since Line 8 is executed for each such 1. In this view, each node that is used for the prefix sum at time  $t$  can be identified as a prefix string of  $\mathrm{bin}(t)$  that ends with a 1.
We will return to Proposition 3.1 and Proposition 3.2 when constructing our smooth mechanism, but for now we settle for stating that combined they give the exact variance at each time step for the binary mechanism. Following Li et al. (2015), to make the mechanism private we have to accommodate for the worst sensitivity across all leaves, which yields Theorem 3.3.
Theorem 3.3 (Exact Variance for Binary Mechanism (Chan et al., 2011; Dwork et al., 2010)). For any  $\rho > 0$ ,  $T > 1$ , the  $\rho$ -zCDP binary mechanism  $\mathcal{M}$  based on Algorithm 1 achieves variance
$$
\operatorname {V a r} [ \mathcal {M} (t) ] = \frac {\lceil \log (T + 1) \rceil}{2 \rho} \| \mathrm {b i n} (t) \| _ {1}
$$
for all  $1 \leq t \leq T$ , where  $\| \mathrm{bin}(t) \|_1$  is equal to the number of  $1s$  in  $\mathrm{bin}(t)$ .
3.2 A smooth binary mechanism
Based on the analysis, a naive idea to improve the binary mechanism would be to only consider leaves with "favorable" time indices. To make this a bit more precise, we ask the following question: could there be a better mechanism in which we store elements in only a subset of the leaves in the original binary tree, and then use Algorithm 1 to compute the prefix sums? We give an affirmative answer.
Consider a full binary tree of height  $h$  where we let the leaves be indexed by  $i$  (1-indexed). Given a sequence of elements  $x \in \{0, 1\}^T$  (we assume a great enough height  $h$  to accommodate for all elements) and an integer  $0 \leq k \leq h$ , we conceptually do the following for each leaf with index  $i$ :
- If  $\mathrm{bin}(i - 1)$  has  $k$  0s, store the next element of the stream in the leaf.
- Otherwise store a token 0 in the leaf.
More rigorously stated, we are introducing an injective mapping from time to leaf-indices  $m: \{1, \dots, T + 1\} \to [2^h + 1]$ , such that  $m(t)$  is the  $(t - 1)^{st}$  smallest  $h$ -bit integer with  $k$  0s in its binary representation.  $x_t$  gets stored in the leaf with index  $m(t)$ , and to compute  $\mathcal{M}(t)$ , we would add  $p$ -sums based on  $\mathrm{bin}(m(t + 1))$ . The resulting algorithm is listed as Algorithm 2, where  $\widehat{\Sigma}[i,j]$  is the noisy count of leaf  $i$  through  $j$ .