SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 2 Prefix Sum for Smooth Binary Mechanism
1: Input: binary tree of height  $h$  storing  $\widehat{\Sigma}[a, b]$  for  $b \leq m(t)$ , time  $t$  where  $m(t + 1) \leq 2^h$ 
2: output  $\leftarrow 0$ 
3:  $s \gets \mathrm{bin}(m(t + 1))$  {padded to  $h$  bits by adding zeros}
4:  $a \gets 1; b \gets 2^h$ 
5: for  $i = h - 1$  to 0 do
6:  $d = \lfloor \frac{a + b}{2} \rfloor$ 
7: if  $s_i = 1$  then
8: output  $\leftarrow$  output +  $\widehat{\Sigma}[a, d]$ 
9:  $a \gets d + 1$ 
10: else
11:  $b \gets d$ 
12: end if
13: end for
14: return output
It follows that the  $\ell_2$ -sensitivity is equal to  $\sqrt{k}$ , and that any prefix sum will be a sum of  $h - k$  nodes. Importantly, the latter fact removes the dependence on  $t$  for the variance.
Choosing a k. The optimal choice of  $k$  depends on the differential privacy paradigm. Here we only consider  $\rho$ -zCDP where  $\Delta = \sqrt{k}$ . Since each prefix sum is computed as a sum of  $h - k$  nodes, the variance for a given prefix sum becomes  $\mathrm{Var}[\mathcal{M}(t)] = (h - k) \cdot \frac{k}{2\rho}$ , which for  $k = h/2$  gives a leading constant of  $1/4$  compared to the maximum variance of the binary mechanism. This choice of  $k$  is valid if the tree has an even height  $h$ , and it maximizes the number of available leaves. Such a tree together with a computation is shown in Figure 2.
Figure 2: Computation of  $\mathcal{M}(5)$  using the smooth binary mechanism where  $T = 5$ . The figure illustrates how the prefix sum up to  $t$  can be computed from  $\mathrm{bin}(m(t + 1))$ . Blue nodes describe the path taken by Algorithm 2, and the sum of red nodes form the desired output  $\mathcal{M}(t)$ . All values shown in nodes are stored noisily, and  $\Sigma[i,j]$  is here defined as the sum of leaves  $i$  through  $j$ . Observe that the noise in each node is drawn from the same distribution and that each query is formed by adding together  $h/2 = 2$  noisy nodes, implying an identical distribution of the error at each step.
A penalty in height. The analysis above assumes that we have a tree of sufficient height. If we before had a tree of height  $\lceil \log (T + 1)\rceil$ , we now need a tree of height  $h\geq \lceil \log (T + 1)\rceil$  to have enough leaves with the right ratio of 1s in their index. To account for this, we let  $h$  be the smallest even integer such that  $h\geq \lceil \log (T)\rceil +a\log \log T$ , where  $a$  is a constant. For our new tree to support as many prefix sums, we need that  $\binom{h}{h/2}\geq T+1$ . This holds for  $a > 1/2$  and sufficiently large  $T$ , but we show it for  $a = 1$  next. Using Stirling's approximation in the first step, we can establish that
$$
\binom {h} {h / 2} \geq \frac {2 ^ {h}}{\sqrt {2 h}} \geq \frac {2 ^ {\log T + \log \log T}}{\sqrt {2} \sqrt {\lceil \log T \rceil + \log \log T}} = \frac {\log T}{\sqrt {2} \sqrt {\lceil \log T \rceil + \log \log T}} \cdot T ,
$$
which is at least  $T + 1$  for  $T \geq 13$ .
Resulting variance. This height penalty makes  $\operatorname{Var}[\mathcal{M}(t)]$  no longer scale as  $\log (T + 1)^2$ , but  $(\log (T) + a\log \log (T))^2$ . Nevertheless, we can state the following:
Lemma 3.4. For any  $\rho >0$ ,  $T\geq 13$ , the  $\rho$ -zCDP smooth binary mechanism  $\mathcal{M}$  achieves variance
$$
\operatorname {V a r} [ \mathcal {M} (t) ] = \frac {1 + o (1)}{8 \rho} \log (T) ^ {2}
$$
where  $1 \leq t \leq T$ , and the  $o(1)$  term is at most  $\frac{2 \log \log(T)}{\log(T)} + \left(\frac{\log \log(T)}{\log(T)}\right)^2$ .
which is an improvement over the original binary mechanism by a factor of  $1/4$  with regard to the leading term. This improvement is shown empirically in Section 4.
Constant average time per output. When outputting  $T$  prefix sums continuously while reading a stream, we only have to store the noise of the nodes, not the p-sums themselves. To make this more explicit, let  $S_{t}$  describe the set of nodes (p-sum indices) that the smooth mechanism adds together to produce the output at time  $t$ . To produce  $\mathcal{M}(t + 1)$  given  $\mathcal{M}(t)$ , we effectively do:
$$
\mathcal {M} (t + 1) = \mathcal {M} (t) + x _ {t + 1} + \sum_ {(i, j) \in S _ {t + 1} \backslash S _ {t}} X [ i, j ] - \sum_ {(i, j) \in S _ {t} \backslash S _ {t + 1}} X [ i, j ].
$$
To quantify the cost, we need to deduce how many nodes are replaced from  $t$  to  $t + 1$ , which means reasoning about  $S_{t}$  and  $S_{t + 1}$ . Recalling that each element in  $S_{t}$  can be identified by a prefix string of  $\mathrm{bin}(m(t + 1))$  terminating with a 1, consider Figure 3. Based on the pattern shown in Figure 3, where the leaf indices only differ in their least significant bits, we get that  $|S_{t + 1} \setminus S_t| = |S_t \setminus S_{t + 1}| = n$ , where  $n$  is the number of 1s in the least significant block of 1s. We formalize this observation next to give a bound on the average cost when outputting a sequence of prefix sums.
Lemma 3.5. Assuming the cost of replacing a node in a prefix sum is 1, then the cost to release all  $\binom{2k}{k}-1$  prefix sums in the tree of height  $2k$  using the smooth binary mechanism is at most  $2\binom{2k}{k}$ .
Proof. As argued before, to compute  $\mathcal{M}(t + 1)$  given  $\mathcal{M}(t)$  we need to replace a number of nodes equal to the size  $n$  of the least significant block of 1s in  $\mathrm{bin}(m(t + 1))$ . We can therefore directly compute the total cost by enumerating all valid indices with different block sizes as
$$
\begin{array}{l} \operatorname {c o s t} = - k + \sum_ {n = 1} ^ {k} \sum_ {i = k - n} ^ {2 k - n - 1} n \binom {i} {k - n} = - k + \sum_ {n = 1} ^ {k} n \binom {2 k - n} {k - n + 1} \\ = - k + \sum_ {i = 1} ^ {k} \sum_ {j = 1} ^ {i} \binom {k - 1 + j} {k - 1} = - 2 k + \sum_ {i = 1} ^ {k} \binom {k + i} {k} = \binom {2 k + 1} {k + 1} - (2 k + 1) \leq 2 \binom {2 k} {k} \\ \end{array}
$$
where the initial  $-k$  term comes from excluding the last balanced leaf index in the tree.
$$
\operatorname {b i n} (m (t + 1)) = \boxed {\dots \quad 0 \quad 1 \dots 1 \quad 0 \dots 0} ; \operatorname {b i n} (m (t + 2)) = \boxed {\dots \quad 1 \quad 0 \dots 0 \quad 1 \dots 1}
$$
Figure 3: The least significant bits of two leaf indices in a full binary tree that are neighboring with respect to time when used in the smooth binary mechanism. If the first cluster of  $1s$  in  $\mathrm{bin}(m(t + 1))$ , counted from the least significant bit, has  $n$ $1s$  then  $n$  nodes in total will be replaced from  $t$  to  $t + 1$ .
In particular, this implies that the average cost when releasing all prefix sums in a full tree is  $\leq 2$ . For  $T$  that does not use all leaves of a tree, comparing to the closest  $T'$  where  $T \leq T' = \binom{2k}{k} - 1$  also implies an average cost of  $\leq 4$  for arbitrary  $T$ . For a single output the cost is  $O(\log(T))$ . It is not hard to check that computing  $\mathrm{bin}(m(t + 1))$  from  $\mathrm{bin}(m(t))$  can be done in constant time using standard arithmetic and bitwise operations on machine words.
Logarithmic space. The argument for the binary mechanism only needing  $O(\log (T))$  space extends to our smooth variant. We state this next in Lemma 3.6, and supply a proof in the appendix.
Lemma 3.6. The smooth binary mechanism computing prefix sums runs in  $O(\log (T))$  space.
Finishing the proof for Theorem 1.1. The last, and more subtle, property of our mechanism is that the error at each time step not only has constant variance, but it is identically distributed. By fixing the number of 1s in each leaf index to  $k$ , the output error at each step becomes the  $k$ -fold convolution of the noise distribution used in each node, independently of what that distribution is. Our mechanism achieves this by design, but note that this can be achieved for any other mechanism (e.g. the regular binary mechanism) by adding fresh noise in excess of what gives privacy (at the expense of utility). Combining Lemmas 3.4, 3.5 and 3.6 together with this last property, we arrive at Theorem 1.1.
4 Comparison of mechanisms
In this section we review how the smooth binary mechanism compares to the original binary mechanism, and the factorization mechanism of Henzinger et al. (2023). A Python implementation of our smooth binary mechanism (and the classic binary mechanism) can be found on https://github.com/jodander/smooth-binary-mechanism. We do not compare to Denisov et al. (2022) since their method is similar to that of Henzinger et al. (2023) in terms of error, and less efficient in terms of time and space usage. We also do not compare to the online version of Honaker (2015) as it is less efficient in time, and its improvement in variance over the binary mechanism is at most a factor of 2, and can therefore be estimated from the figures.
Variance comparison. To demonstrate how the variance behaves over time for our smooth binary mechanism, see Figure 4. Given a fix  $T$ , the tree-based mechanisms compute the required tree height
(a) Running variance for  $T = 250$ .
(b) Maximum variance vs. upper bound on time.
Figure 4: Comparison of variance between the mechanism in Henzinger et al. (2023), the standard binary mechanism and our mechanism. Figure 4(a) shows  $\mathrm{Var}[\mathcal{M}(t)]$  for  $1\leq t\leq T$  for  $T = 250$ , whereas Figure 4(b) shows the maximum variance that each mechanism would attain for a given upper bound on time. At the last time step in Figure 4(b), our mechanism reduces the variance by a factor of 3.27 versus the binary mechanism.
to support all elements, and the factorization mechanism a sufficiently large matrix. The volatility of the error in the regular binary mechanism is contrasted by the stable noise distribution of our smooth binary mechanism, as demonstrated in Figure 4(a). In terms of achieving the lowest variance, the factorization mechanism in Henzinger et al. (2023) is superior, as expected. This result is replicated in Figure 4(b) where our mechanism offers a substantial improvement in terms of maximum variance over the original binary mechanism, but does not improve on Henzinger et al. (2023).
Computational efficiency comparison. While our mechanism does not achieve as low noise as the mechanism of Henzinger et al. (2023), it scales well in time and space, and with respect to the dimensionality of the stream elements. This is empirically demonstrated in Figure 5. Since the matrix
(a) Average computational time per output.
(b) Maximum space needed vs upper bounds on time.
Figure 5: Comparison of computational efficiency between the mechanism in Henzinger et al. (2023), the binary mechanism and our mechanism. Figure 5(a) shows the computation time spent per d-dimensional input. The simulation was run 5 times for each method, meaning each method has 5 data points in the plot per time step. The computation was performed for elements of dimension  $d = 10^4$ , was run on a Macbook Pro 2021 with Apple M1 Pro chip and 16 GB memory using Python 3.9.6, scipy version 1.9.2, and numpy version 1.23.3. Figure 5(b) shows the maximum number of floats that has to be stored in memory when outputting all prefix sums up to a given time, assuming binary input.
used in Henzinger et al. (2023) is a Toeplitz matrix, the scipy method "matmulToeplitz" (based on FFT and running in time  $O(dT\log T)$  to produce a  $d$ -dimensional output) was used to speed up the matrix multiplication generating the noise in Figure 5(a).
The discrepancy in the computation time scaling can largely be attributed to space: the needed space for these tree-based methods scales logarithmically with  $T$ , and linearly with  $T$  for matrix-multiplication based methods. This is demonstrated in Figure 5(b). As to the difference in computation time between the tree-based methods, the smooth binary mechanism generates twice as much fresh noise per time step on average, which likely is the dominating time sink in this setup.
5 Conclusion and discussion
We presented an improved "smooth" binary mechanism that retains the low time complexity and space usage while improving the additive error and achieving stable noise at each time step. Our mechanism was derived by performing a careful analysis of the original binary mechanism, and specifically the influence of the binary representation of leaf indices in the induced binary tree. Our empirical results demonstrate the stability of the noise and its improved variance compared to the binary mechanism. The factorization mechanism of Henzinger et al. (2023) offers better variance, but is difficult to scale to a large number of time steps, especially if we need high-dimensional noise.
We note that the smooth binary mechanism can be extended to  $\varepsilon$ -DP. The optimal fraction of 1s in the leaves of the binary tree would no longer be  $1/2$ . An interesting problem is to find mechanisms that have lower variance and attractive computational properties. It is possible that the dependence on  $T$  can be improved by leaving the factorization mechanism framework, but in absence of such a result the best we can hope is to match the variance obtained by Henzinger et al. (2023).
Acknowledgments and disclosure of funding
We would like to thank Jalaj Upadhyay for useful discussions about our results and their relation to previous work. We would also like to thank the reviewers for their comments and suggestions which have contributed to a clearer paper. The authors are affiliated with Basic Algorithms Research Copenhagen (BARC), supported by the VILLUM Foundation grant 16582, and are also supported by Providentia, a Data Science Distinguished Investigator grant from Novo Nordisk Fonden.
A Proof of Lemma 3.6
The proof of Lemma 3.6 was omitted from the main text, it is given next.
Proof. The critical observation to make is that once a given p-sum is removed from a prefix sum, then it will never re-appear. Let its associated prefix string at time  $t$  be  $s$  of length  $l$ . If we look at the same  $l$  bit positions in  $\mathrm{bin}(m(t))$  as  $t$  increases and interpret it as a number, then it is monotonously increasing with  $t$ . This implies that once a given prefix-string  $s$  in  $\mathrm{bin}(m(t))$  disappears at  $t' > t$  then it will not be encountered again. We can therefore free up the memory used for storing any p-sum the moment it is no longer used. Because of this, it suffices to only store the p-sums in the active prefix sum at any time, of which there are at most  $O(\log(T))$ , proving the statement.
B Factorization mechanisms on multidimensional input
Assuming a  $d$ -dimensional input stream with elements  $x_{i} \in \mathbb{R}^{d}$ , let two streams  $x, x'$  be neighboring if they differ at exactly one time step  $t$  where  $\| x_{t} - x_{t}' \|_{2} \leq 1$ . Given a one-dimensional  $\rho$ -zCDP factorization mechanism for continual counting with factorization  $A = LR$ , we want to argue that running it along each dimension separately gives a  $\rho$ -zCDP factorization mechanism for releasing prefix sums on  $x$ .
To see this, consider the new, flattened vector input  $\tilde{x} \in \mathbb{R}^{d \cdot T}$  where  $\tilde{x} = [(x_1)_1, (x_1)_2, \ldots, (x_1)_d, (x_2)_1, \ldots, (x_T)_d]$ , i.e.,  $(x_t)_j = \tilde{x}_{(t-1) \cdot d+j}$ . Analogously for these new inputs, we can define a counting matrix  $\tilde{A} = A \oplus I_{d \times d} \in \mathbb{R}^{d \cdot T \times d \cdot T}$  that sums each dimension separately, where  $\oplus$  is the Kronecker product and  $I_{d \times d}$  is the  $d$ -dimensional identity matrix. We get a corresponding factorization  $\tilde{A} = \tilde{L}\tilde{R}$  where  $\tilde{L} = L \oplus I_{d \times d}$  and  $\tilde{R} = R \oplus I_{d \times d}$ .
It follows immediately that,  $\tilde{L},\tilde{R}$  gives a factorization mechanism for privately releasing  $\tilde{A}\tilde{x}$ , and from it we can extract all private prefix sums on the original stream  $x$ . Letting  $\tilde{x}^{\prime}$  be the vector representation of the stream  $x^{\prime}\sim x$ , we will reason about the  $\ell_2$ -sensitivity  $\Delta$  next. First observe that for neighbouring inputs  $x$  and  $x^{\prime}$  that differ at time  $t$ , we have:
$$
\left\| \tilde {R} (\tilde {x} - \tilde {x} ^ {\prime}) \right\| _ {2} = \| R e _ {t} \| _ {2} \cdot \| \tilde {x} - \tilde {x} ^ {\prime} \| _ {2}
$$
$$
\Delta = \sup _ {\tilde {x} \sim \tilde {x} ^ {\prime}} \left\| \tilde {R} (\tilde {x} - \tilde {x} ^ {\prime}) \right\| _ {2} = \max _ {i} \| R e _ {i} \| _ {2} \cdot \sup _ {\tilde {x} \sim \tilde {x} ^ {\prime}} \| \tilde {x} - \tilde {x} ^ {\prime} \| _ {2} = \max _ {i} \| R e _ {i} \| _ {2},
$$
which equates the  $\ell_2$ -sensitivity of the one-dimensional case. Recapitulating, any one-dimensional  $\rho$ -zCDP factorization mechanism for continual counting is also a  $\rho$ -zCDP counting mechanism for inputs of higher dimension, as long as the neighboring relation is appropriately extended.