Title: A Smooth Binary Mechanism for Efficient Private Continual Observation
Venue: NeurIPS 2023
Accepted: yes
Keywords: differential privacy

continual observation

binary mechanism
Abstract: In privacy under continual observation we study how to release differentially private estimates based on a dataset that evolves over time. The problem of releasing private prefix sums of $x_1, x_2, x_3,\dots\in${$0,1$} (where the value of each $x_i$ is to be private) is particularly well-studied, and a generalized form is used in state-of-the-art methods for private stochastic gradient descent (SGD).

The seminal binary mechanism privately releases the first $t$ prefix sums with noise of variance polylogarithmic in $t$. Recently, Henzinger et al. and Denisov et al. showed that it is possible to improve on the binary mechanism in two ways: The variance of the noise can be reduced by a (large) constant factor, and also made more even across time steps. However, their algorithms for generating the noise distribution are not as efficient as one would like in terms of computation time and (in particular) space.

We address the efficiency problem by presenting a simple alternative to the binary mechanism in which 1) generating the noise takes constant average time per value, 2) the variance is reduced by a factor about 4 compared to the binary mechanism, and 3) the noise distribution at each step is identical. Empirically, a simple Python implementation of our approach outperforms the running time of the approach of Henzinger et al., as well as an attempt to improve their algorithm using high-performance algorithms for multiplication with Toeplitz matrices.

A Smooth Binary Mechanism for Efficient Private Continual Observation

Abstract

In privacy under continual observation we study how to release differentially private estimates based on a dataset that evolves over time. The problem of releasing private prefix sums of x 1 , x 2 , x 3 , · · · ∈ { 0 , 1 } (where the value of each x i is to be private) is particularly well-studied, and a generalized form is used in state-of-the-art methods for private stochastic gradient descent (SGD). The seminal binary mechanism privately releases the first t prefix sums with noise of variance polylogarithmic in t . Recently, Henzinger et al. and Denisov et al. showed that it is possible to improve on the binary mechanism in two ways: The variance of the noise can be reduced by a (large) constant factor, and also made more even across time steps. However, their algorithms for generating the noise distribution are not as efficient as one would like in terms of computation time and (in particular) space. We address the efficiency problem by presenting a simple alternative to the binary mechanism in which 1) generating the noise takes constant average time per value, 2) the variance is reduced by a factor about 4 compared to the binary mechanism, and 3) the noise distribution at each step is identical . Empirically, a simple Python implementation of our approach outperforms the running time of the approach of Henzinger et al., as well as an attempt to improve their algorithm using high-performance algorithms for multiplication with Toeplitz matrices.

1 Introduction

There are many actors in society that wish to publish aggregate statistics about individuals, be it for financial or social utility. Netflix might recommend movies based on other users' preferences, and policy might be driven by information on average incomes across groups. Whatever utility one has in mind however, it should be balanced against the potential release of sensitive information. While it may seem anodyne to publish aggregate statistics about users, doing it without consideration to privacy may expose sensitive information of individuals (Dinur & Nissim, 2003). Differential privacy offers a framework for dealing with these problems in a mathematically rigorous way.

̸

A particular setting is when statistics are updated and released continually, for example a website releasing its number of visitors over time. Studying differential privacy in this setup is referred to as differential privacy under continual observation (Dwork et al., 2010; Dwork & Roth, 2013). A central problem in this domain is referred to as differentially private counting under continual observation (Chan et al., 2011; Dwork et al., 2010), continual counting for short. It covers the following problem: a binary stream x 1 , x 2 , x 3 , . . . is received one element at a time such that x t is received in round t . The objective is to continually output the number of 1s encountered up to each time step while maintaining differential privacy. We consider two streams x and x ′ as neighboring if they are identical except for a single index i where x i = x ′ i . It suffices to study the setting in which there is a known upper bound T on the number of prefix sums to release - algorithms for the case of unbounded streams then follow by a general reduction (Chan et al., 2011).

Rasmus Pagh

'Honaker Online' refers to the 'Estimation from below' variant in Honaker (2015) as implemented in Kairouz et al. (2021). There is no explicit bound on variance in Denisov et al. (2022), but the method finds an optimal matrix factorization so it should achieve same variance * as Henzinger et al. (2023) up to lower order terms. As to their efficiency, Denisov et al. (2022) contains empirical work for reducing the time and space usage by approximating the matrices involved as a sum of a banded matrix and a low-rank approximation, but without formal guarantees. The time usage in ** assumes implementing the matrix-vector product using FFT. Leveraging FFT for continual observation is not a novel approach (Choquette-Choo et al., 2023). All sparse matrices have O (log T ) nonzero entries per row or column, and all dense matrices have Ω( T 2 ) nonzero entries - a Toeplitz matrix can be seen as intermediate in the sense of having O ( T ) unique diagonals, allowing for efficient storage and multiplication.

Aside from the natural interpretation of continual counting as the differentially private release of user statistics over time, mechanisms for continual counting (and more generally for releasing prefix sums) are used as a subroutine in many applications. Such a mechanism is for example used in Google's privacy-preserving federated next word prediction model (McMahan & Thakurta, 2022; Kairouz et al., 2021; Choquette-Choo et al., 2023), in non-interactive local learning (Smith et al., 2017), in stochastic convex optimization (Han et al., 2022) and in histogram estimation (Cardoso & Rogers, 2022; Chan et al., 2012; Huang et al., 2022; Upadhyay, 2019) among others.

Given the broad adoption of continual counting as a primitive, designing algorithms for continual counting that improve constants in the error while scaling well in time and space is of practical interest.

1.1 Our contributions

In this paper we introduce the Smooth Binary Mechanism , a differentially private algorithm for the continual counting problem that improves upon the original binary mechanism by Chan et al. (2011); Dwork et al. (2010) in several respects, formalized in Theorem 1.1 and compared to in Table 1.

Theorem 1.1 (Smooth Binary Mechanism) . For any ρ > 0 , T > 1 , there is an efficient ρ -zCDP continual counting mechanism M , that on receiving a binary stream of length T satisfies

where M ( t ) is the output prefix sum at time t , while only requiring O (log T ) space, O ( T ) time to output all T prefix sums, and where the error is identically distributed for all 1 ≤ t ≤ T .

Our mechanism retains the scalability in time and space of the binary mechanism while offering an improvement in variance by a factor of 4 -o (1) . It also has the same error distribution in every step by design, which could make downstream applications easier to analyze.

Sketch of technical ideas. Our starting point is the binary mechanism which, in a nutshell, uses a complete binary tree with ≥ T +1 leaves (first T leaves corresponding to x 1 , . . . , x T ) in which each node contains the sum of the leaves below, made private by adding random noise (e.g. from a Gaussian distribution). To estimate a prefix sum ∑ t i =1 x i we follow the path from the root to the leaf storing x t +1 . Each time we go to a right child the sum stored in its sibling node is added to a counter. An observation, probably folklore, is that it suffices to store sums for nodes that are left children , so

suppose we do not store any sum in nodes that are right children. The number of terms added when computing the prefix sum is the number of 1s in the binary representation bin( t ) of t , which encodes the path to the leaf storing x t +1 . The sensitivity of the tree with respect to x t , i.e., the number of node counts that change by 1 if x t changes, is the number of 0s in bin( t -1) . Our idea is to only use leaves that have balanced binary representation, i.e. same number of 0s and 1s (assuming the height h of the tree is an even integer). To obtain T useful leaves we need to make the tree slightly deeper -it turns out that height h slightly more than log 2 ( T ) suffices. This has the effect of making the sensitivity of every leaf h/ 2 , and the noise in every prefix sum a sum of h/ 2 independent noise terms.

Limitations. As shown in Table 1 and Section 4, the smooth binary mechanism, while improving on the original binary mechanism, cannot achieve as low variance as matrix-based mechanisms such as Henzinger et al. (2023). However, given that the scaling of such methods can keep them from being used in practice, our variant of the binary mechanism has practical utility in large-scale settings.

1.2 Related work

All good methods for continual counting that we are aware of can be seen as instantiations of the factorization mechanism (Li et al., 2015), sometimes also referred to as the matrix mechanism . These methods perform a linear transformation of the data, given by a matrix R , then add noise according to the sensitivity of Rx , and finally obtain the sequence of estimates by another linear transformation given by a matrix L . To obtain unbiased estimates, the product LRx needs to be equal to the vector of prefix sums, that is, LR must be the lower triangular all 1s matrix.

Seminal works introducing the first variants of the binary mechanism are due to Chan, Shi, and Song (2011) and Dwork, Naor, Pitassi, and Rothblum (2010), but similar ideas were proposed independently in Hay, Rastogi, Miklau, and Suciu (2010) and Gehrke, Wang, and Xiao (2010). Honaker (2015) noticed that better estimators are possible by making use of all information in the tree associated with the binary tree method. A subset of their techniques can be leveraged to reduce the variance of the binary mechanism by up to a factor 2 , at some cost of efficiency, as shown from their implementation in Kairouz, Mcmahan, Song, Thakkar, Thakurta, and Xu (2021).

A number of recent papers have studied improved choices for the matrices L , R . Denisov, McMahan, Rush, Smith, and Thakurta (2022) treated the problem of finding matrices leading to minimum largest variance on the estimates as a convex optimization problem, where (at least for T up to 2048) it was feasible to solve it. To handle a larger number of time steps they consider a similar setting where a restriction to banded matrices makes the method scale better, empirically with good error, but no theoretical guarantees are provided.

Fichtenberger, Henzinger, and Upadhyay (2023) gave an explicit decomposition into lower triangular matrices, and analyzed its error in the ℓ ∞ metric. The matrices employed are Toeplitz (banded) matrices. Henzinger, Upadhyay, and Upadhyay (2023) analyzed the same decomposition with respect to mean squared error of the noise, and showed that it obtains the best possible error among matrix decompositions where L and R are square matrices, up to a factor 1 + o (1) where the o (1) term vanishes when T goes to infinity.

A breakdown of how our mechanism compares to existing ones is shown in Table 1. While not achieving as small an error as the factorization mechanism of Henzinger et al. (2023), its runtime and small memory footprint allows for better scaling for longer streams. For concreteness we consider privacy under ρ -zero-Concentrated Differential Privacy (zCDP) (Bun & Steinke, 2016), but all results can be expressed in terms of other notions of differential privacy.

2 Preliminaries

Binary representation of numbers. We will use the notation bin( n ) to refer to the binary representation of a number n ∈ [2 h ] , where h > 0 is an integer, and let bin( n ) be padded with leading zeros to h digits. For example, bin(2) = 0b010 for a tree of height h = 3 . When indexing such a number we let index i refer to the i th least significant bit, e.g. bin(2) 1 = 1 . We will also refer to prefixes and suffixes of binary strings, and we use the convention that a prefix of a binary string includes its most significant bits.

Partial sums (p-sums). To clear up the notation we use the concept of p-sums Σ[ i, j ] where Σ[ i, j ] = ∑ j t = i x t . We will furthermore use the concept of noisy p-sums

where F is a suitable distribution for the DP paradigm, e.g. Laplacian or Gaussian. For convenience we also define ˆ x t = ̂ Σ[ t, t ] , i.e. ˆ x t is the single stream element with noise added.

2.1 Continual observation of bit stream

Given an integer T > 1 we consider a finite length binary stream x = ( x 1 , x 2 , . . . , x T ) , where x t ∈ { 0 , 1 } , 1 ≤ t ≤ T , denotes the bit appearing in the stream at time t .

Definition 2.1 (Continual Counting Query) . Given a stream x ∈ { 0 , 1 } T , the count for the stream is a mapping c : { 1 , . . . , T } → Z such that c ( t ) = ∑ t i =1 x i .

Definition 2.2 (Counting Mechanism) . A counting mechanism M takes a stream x ∈ { 0 , 1 } T and produces a (possibly random) vector M x ∈ R T where ( M x ) t is a function of the first t elements of the stream. For convenience we will write M ( t ) for ( M x ) t when there is little chance for ambiguity.

To analyze a counting mechanism from the perspective of differential privacy, we also need a notion of neigboring streams.

̸

Definition 2.3 (Neighboring Streams) . Streams x, x ′ ∈ { 0 , 1 } T are said to be neighboring , denoted x ∼ x ′ , if |{ i | x i = x ′ i }| = 1 .

Intuitively, for a counting mechanism to be useful at a given time t , we want it to minimize |M ( t ) -c ( t ) | . We consider unbiased mechanisms that return the true counts in expectation and we focus on minimizing Var[ M ( t ) -c ( t )] .

2.2 Differential privacy

For a mechanism to be considered differentially private, we require that the outputs for any two neighboring inputs are are indistinguishable. We will state our results in terms of ρ -zCDP:

Definition 2.4 (Concentrated Differential Privacy (zCDP) (Bun & Steinke, 2016)) . For ρ > 0 , a randomized algorithm A : X n →Y is ρ -zCDP if for any D ∼ D ′ , D α ( A ( D ) ||A ( D ′ )) ≤ ρα for all α > 1 , where D α ( A ( D ) ||A ( D ′ )) is the α -R´ enyi divergence between A ( D ) and A ( D ′ ) .

In the scenario where we are looking to release a real-valued function f ( D ) taking values in R d , we can achieve zCDP by adding Gaussian noise calibrated to the ℓ 2 -sensitivity of f .

Lemma 2.5 (Gaussian Mechanism (Bun & Steinke, 2016)) . Let f : X n → R d be a function with global ℓ 2 -sensitivity ∆ := max D∼D ′ ∥ f ( D ) -f ( D ′ ) ∥ 2 . For a given data set D ∈ X n , the mechanism that releases f ( D ) + N (0 , ∆ 2 2 ρ ) d satisfies ρ -zCDP.

It is known that ρ -zCDP implies ( ρ +2 √ ρ ln(1 /δ ) , δ ) -differential privacy for every δ > 0 (Bun & Steinke, 2016).

Lastly, when comparing counting mechanisms based on Gaussian noise, we note that it is sufficient to look at the variance. For a single output M ( t ) , the variance Var[ M ( t )] is the only relevant metric, as it completely describes the output distribution. For a series of outputs M x , and related norms ∥M x ∥ p , we note that a mechanism with lower variance will be more concentrated in each coordinate and have lower p th moment, allowing for tighter bounds on the norm.

2.3 Differentially private continual counting

We next describe two approaches to continual counting.

Binary mechanism. The binary mechanism (Chan et al., 2011; Dwork et al., 2010; Gehrke et al., 2010; Hay et al., 2010) can be considered the canonical mechanism for continual counting. In this section we present a variant of it where only left subtrees are used. The mechanism derives its name

from the fact that a binary tree is built from the input stream. Each element from the stream is assigned a leaf in the binary tree, and each non-leaf node in the tree represents a p-sum of all elements in descendant leaves. All values are stored noisily in nodes, and nodes are added together to produce a given prefix sum. Such a binary tree is illustrated in Figure 1(a).

̂

Σ[1

,

8]

̂

Σ[1

,

4]

̂

Σ[1

,

2]

ˆ

x

1

ˆ

x

2

̂

Σ[3

,

4]

ˆ

x

3

ˆ

x

4

̂

Σ[5

,

8]

̂

Σ[5

,

6]

ˆ

x

5

ˆ

x

6

̂

,

8]

ˆ

x

7

0

(a) Binary mechanism tree for T = 7 .

̂

Σ[1

,

8]

̂

Σ[1

,

4]

̂

Σ[1

,

2]

ˆ

x

1

ˆ

x

2

̂

Σ[3

,

4]

ˆ

x

3

ˆ

x

4

̂

Σ[5

,

8]

̂

Σ[5

,

6]

ˆ

x

5

ˆ

x

6

̂

,

8]

ˆ

x

7

0

0b000

0b001

0b010

0b011

0b100

0b101

0b110

0b111

(6) =

̂

Σ[1

,

4] +

̂

Σ[5

,

6]

M

(b) Computation of M (6) using Algorithm 1.

We will return for a closer analysis of the binary mechanism in Section 3.1. For now we settle for stating that the ρ -zCDP binary mechanism achieves Var[ M ( t )] = O (log( T ) 2 ) and is known to be computationally efficient: to release all prefix sums up to a given time t ≤ T requires only O (log( T )) space and O ( t ) time.

Factorization mechanism. The binary mechanism belongs to a more general class of mechanisms called factorization mechanisms (Li et al., 2015), sometimes also referenced as matrix mechanisms . Computing a prefix sum is a linear operation on the input stream x ∈ R T , and computing all prefix sums up to a given time T can therefore be represented by a matrix A ∈ R T × T , where c ( t ) = ( Ax ) t . A is here a lower-triangular matrix of all 1s. The factorization mechanism characterizes solutions to the continual counting problem by factorizing A as A = LR with corresponding mechanism M x = L ( Rx + z ) , z ∼ F n where n is the dimension of Rx .

Intuitively Rx represents linear combinations of the stream, which are made private by adding noise z , and which then are aggregated by L . To achieve ρ -zCDP for this mechanism, we let z ∼ N (0 , ∆ 2 2 ρ ) n where ∆ = max i ∥ Re i ∥ 2 , e i being the i th unit vector. It follows that the corresponding output noise becomes Lz with coordinates ( Lz ) i ∼ N (0 , ∆ 2 2 ρ ∥ L i ∥ 2 2 ) where L i is the i th row in L .

Extension to multidimensional input. While we have so far assumed one-dimensional input, any one-dimensional ρ -zCDP factorization mechanism naturally generalizes to higher dimensions. This fact enables factorization mechanisms to be applied to the problem of private learning where gradients are summed. We sketch an argument for this fact in Appendix B.

3 Our mechanism

To introduce our variant of the binary mechanism, we need to return to the original mechanism.

3.1 Closer analysis of the binary mechanism

As has been pointed out in earlier work (Denisov et al., 2022; Henzinger et al., 2023), a naive implementation of the binary mechanism will lead to variance that is non-uniform with respect to time: the number of 1s in the bitwise representation of t determines the variance. To underline this, consider the pseudo-code in Algorithm 1 for computing a prefix sum given a binary mechanism tree structure. The code assumes that the tree has ≥ t +1 leaves, a detail that only matters when t is a power of 2 and allows us to never use the root node. To illustrate a simple case, see Figure 1(b).

Algorithm 1 Prefix Sum for Binary Mechanism

We can make the following two observatons:

bin( t -1) encodes a path from the root to the leaf where x t is stored.

To compute prefix sums using Algorithm 1, we only need to store values in 'left children'.

Combining these observations, we get the following result:

Proposition 3.1. The squared ℓ 2 -sensitivity ∆ 2 of x t is equal to the number of 0s in bin( t -1) .

To see this, note that the number of 0s in bin( t -1) is equal to the number of left-children that are passed through to reach the given node. Changing the value of x t impacts all its ancestors, and since only left-children are used for prefix sums, the result immediately follows. This does not address the volatility of variance with respect to time. However, studying Algorithm 1 gives the reason:

Proposition 3.2. Var[ M ( t )] is proportional to the number of 1s in bin( t ) .

The result follows from the fact that the number of terms added together for a given prefix sum c ( t ) is equal to the number of 1s in bin( t ) , since Line 8 is executed for each such 1. In this view, each node that is used for the prefix sum at time t can be identified as a prefix string of bin( t ) that ends with a 1.

We will return to Proposition 3.1 and Proposition 3.2 when constructing our smooth mechanism, but for now we settle for stating that combined they give the exact variance at each time step for the binary mechanism. Following Li et al. (2015), to make the mechanism private we have to accommodate for the worst sensitivity across all leaves, which yields Theorem 3.3.

Theorem 3.3 (Exact Variance for Binary Mechanism (Chan et al., 2011; Dwork et al., 2010)) . For any ρ > 0 , T > 1 , the ρ -zCDP binary mechanism M based on Algorithm 1 achieves variance

for all 1 ≤ t ≤ T , where ∥ bin( t ) ∥ 1 is equal to the number of 1s in bin( t ) .

3.2 A smooth binary mechanism

Based on the analysis, a naive idea to improve the binary mechanism would be to only consider leaves with 'favorable' time indices. To make this a bit more precise, we ask the following question: could there be a better mechanism in which we store elements in only a subset of the leaves in the original binary tree, and then use Algorithm 1 to compute the prefix sums? We give an affirmative answer.

Consider a full binary tree of height h where we let the leaves be indexed by i ( 1 -indexed). Given a sequence of elements x ∈ { 0 , 1 } T (we assume a great enough height h to accommodate for all elements) and an integer 0 ≤ k ≤ h , we conceptually do the following for each leaf with index i :

If bin( i -1) has k 0 s, store the next element of the stream in the leaf.

Otherwise store a token 0 in the leaf.

More rigorously stated, we are introducing an injective mapping from time to leaf-indices m : { 1 , . . . , T +1 } → [2 h +1] , such that m ( t ) is the ( t -1) st smallest h -bit integer with k 0 s in its binary representation. x t gets stored in the leaf with index m ( t ) , and to compute M ( t ) , we would add p-sums based on bin( m ( t +1)) . The resulting algorithm is listed as Algorithm 2, where ̂ Σ[ i, j ] is the noisy count of leaf i through j .

Algorithm 2 Prefix Sum for Smooth Binary Mechanism

It follows that the ℓ 2 -sensitivity is equal to √ k , and that any prefix sum will be a sum of h -k nodes. Importantly, the latter fact removes the dependence on t for the variance.

Choosing a k. The optimal choice of k depends on the differential privacy paradigm. Here we only consider ρ -zCDP where ∆ = √ k . Since each prefix sum is computed as a sum of h -k nodes, the variance for a given prefix sum becomes Var[ M ( t )] = ( h -k ) · k 2 ρ , which for k = h/ 2 gives a leading constant of 1 / 4 compared to the maximum variance of the binary mechanism. This choice of k is valid if the tree has an even height h , and it maximizes the number of available leaves. Such a tree together with a computation is shown in Figure 2.

Σ[1

,

8]

Σ[1

,

4]

Σ[1

,

2]

0

0

0

x

1

Σ[5

,

6]

0

x

2

x

3

0

Σ[9

,

12]

Σ[9

,

10]

0

x

4

x

5

0

Σ[13

,

14]

0

0

0

0

0b0000

0b0001

0b0010

m

(1)) =

0b0011

0b0100

m

(2)) =

0b0101

m

(3)) =

0b0110

0b0111

0b1000

m

(4)) =

0b1001

m

(5)) =

0b1010

0b1011

m

(6)) =

0b1100

0b1101

0b1110

0b1111

(5) =

̂

Σ[1

,

8] +

̂

Σ[9

,

12]

M

A penalty in height. The analysis above assumes that we have a tree of sufficient height. If we before had a tree of height ⌈ log( T +1) ⌉ , we now need a tree of height h ≥ ⌈ log( T +1) ⌉ to have enough leaves with the right ratio of 1s in their index. To account for this, we let h be the smallest even integer such that h ≥ ⌈ log( T ) ⌉ + a log log T , where a is a constant. For our new tree to support as many prefix sums, we need that ( h h/ 2 ) ≥ T +1 . This holds for a > 1 / 2 and sufficiently large T , but we show it for a = 1 next. Using Stirling's approximation in the first step, we can establish that

which is at least T +1 for T ≥ 13 .

Resulting variance. This height penalty makes Var[ M ( t )] no longer scale as log( T +1) 2 , but (log( T ) + a log log( T )) 2 . Nevertheless, we can state the following:

Lemma 3.4. For any ρ > 0 , T ≥ 13 , the ρ -zCDP smooth binary mechanism M achieves variance

where 1 ≤ t ≤ T , and the o (1) term is at most 2 log log( T ) log( T ) + ( log log( T ) log( T ) ) 2 .

which is an improvement over the original binary mechanism by a factor of 1 / 4 with regard to the leading term. This improvement is shown empirically in Section 4.

Constant average time per output. When outputting T prefix sums continuously while reading a stream, we only have to store the noise of the nodes, not the p-sums themselves. To make this more explicit, let S t describe the set of nodes (p-sum indices) that the smooth mechanism adds together to produce the output at time t . To produce M ( t +1) given M ( t ) , we effectively do:

To quantify the cost, we need to deduce how many nodes are replaced from t to t +1 , which means reasoning about S t and S t +1 . Recalling that each element in S t can be identified by a prefix string of bin( m ( t +1)) terminating with a 1, consider Figure 3. Based on the pattern shown in Figure 3, where the leaf indices only differ in their least significant bits, we get that | S t +1 \ S t | = | S t \ S t +1 | = n , where n is the number of 1s in the least significant block of 1s. We formalize this observation next to give a bound on the average cost when outputting a sequence of prefix sums.

Lemma 3.5. Assuming the cost of replacing a node in a prefix sum is 1 , then the cost to release all ( 2 k k ) -1 prefix sums in the tree of height 2 k using the smooth binary mechanism is at most 2 ( 2 k k ) .

Proof. As argued before, to compute M ( t +1) given M ( t ) we need to replace a number of nodes equal to the size n of the least significant block of 1s in bin( m ( t +1)) . We can therefore directly compute the total cost by enumerating all valid indices with different block sizes as

where the initial -k term comes from excluding the last balanced leaf index in the tree.

In particular, this implies that the average cost when releasing all prefix sums in a full tree is ≤ 2 . For T that does not use all leaves of a tree, comparing to the closest T ′ where T ≤ T ′ = ( 2 k k ) -1 also implies an average cost of ≤ 4 for arbitrary T . For a single output the cost is O (log( T )) . It is not hard to check that computing bin( m ( t +1)) from bin( m ( t )) can be done in constant time using standard arithmetic and bitwise operations on machine words.

Logarithmic space. The argument for the binary mechanism only needing O (log( T )) space extends to our smooth variant. We state this next in Lemma 3.6, and supply a proof in the appendix.

Lemma 3.6. The smooth binary mechanism computing prefix sums runs in O (log( T )) space.

Finishing the proof for Theorem 1.1. The last, and more subtle, property of our mechanism is that the error at each time step not only has constant variance, but it is identically distributed. By fixing the number of 1s in each leaf index to k , the output error at each step becomes the k -fold convolution of the noise distribution used in each node, independently of what that distribution is. Our mechanism achieves this by design, but note that this can be achieved for any other mechanism (e.g. the regular binary mechanism) by adding fresh noise in excess of what gives privacy (at the expense of utility). Combining Lemmas 3.4, 3.5 and 3.6 together with this last property, we arrive at Theorem 1.1.

4 Comparison of mechanisms

In this section we review how the smooth binary mechanism compares to the original binary mechanism, and the factorization mechanism of Henzinger et al. (2023). A Python implementation of our smooth binary mechanism (and the classic binary mechanism) can be found on https://github.com/jodander/smooth-binary-mechanism . We do not compare to Denisov et al. (2022) since their method is similar to that of Henzinger et al. (2023) in terms of error, and less efficient in terms of time and space usage. We also do not compare to the online version of Honaker (2015) as it is less efficient in time, and its improvement in variance over the binary mechanism is at most a factor of 2 , and can therefore be estimated from the figures.

Variance comparison. To demonstrate how the variance behaves over time for our smooth binary mechanism, see Figure 4. Given a fix T , the tree-based mechanisms compute the required tree height

0

50

100

150

200

250

time

5

10

15

20

25

Var[

M

(

t

)]

×

ρ

Binary Mechanism

Our Mechanism

(a) Running variance for T = 250 .

10

0

10

1

10

2

10

3

10

4

10

5

10

6

10

7

upper bound on time

0

50

100

150

200

250

t

T

Var[

M

(

t

)]

×

ρ

Binary Mechanism

Our Mechanism

(b) Maximum variance vs. upper bound on time.

to support all elements, and the factorization mechanism a sufficiently large matrix. The volatility of the error in the regular binary mechanism is contrasted by the stable noise distribution of our smooth binary mechanism, as demonstrated in Figure 4(a). In terms of achieving the lowest variance, the factorization mechanism in Henzinger et al. (2023) is superior, as expected. This result is replicated in Figure 4(b) where our mechanism offers a substantial improvement in terms of maximum variance over the original binary mechanism, but does not improve on Henzinger et al. (2023).

Computational efficiency comparison. While our mechanism does not achieve as low noise as the mechanism of Henzinger et al. (2023), it scales well in time and space, and with respect to the dimensionality of the stream elements. This is empirically demonstrated in Figure 5. Since the matrix

10

2

10

3

10

4

number of d-dimensional vectors

0

1

2

3

4

5

ms/output

Binary Mechanism

Our Mechanism

10

0

10

1

10

2

10

3

10

4

10

5

10

6

upper bound on time

10

0

10

1

10

2

10

3

10

4

10

5

10

6

max # of floats needed to be stored

Binary Mechanism

Our Mechanism

(a) Average computational time per output.

(b) Maximum space needed vs upper bounds on time.

used in Henzinger et al. (2023) is a Toeplitz matrix, the scipy method 'matmul toeplitz' (based on FFT and running in time O ( dT log T ) to produce a d -dimensional output) was used to speed up the matrix multiplication generating the noise in Figure 5(a).

The discrepancy in the computation time scaling can largely be attributed to space: the needed space for these tree-based methods scales logarithmically with T , and linearly with T for matrixmultiplication based methods. This is demonstrated in Figure 5(b). As to the difference in computation time between the tree-based methods, the smooth binary mechanism generates twice as much fresh noise per time step on average, which likely is the dominating time sink in this setup.

5 Conclusion and discussion

We presented an improved 'smooth' binary mechanism that retains the low time complexity and space usage while improving the additive error and achieving stable noise at each time step. Our mechanism was derived by performing a careful analysis of the original binary mechanism, and specifically the influence of the binary representation of leaf indices in the induced binary tree. Our empirical results demonstrate the stability of the noise and its improved variance compared to the binary mechanism. The factorization mechanism of Henzinger et al. (2023) offers better variance, but is difficult to scale to a large number of time steps, especially if we need high-dimensional noise.

We note that the smooth binary mechanism can be extended to ε -DP. The optimal fraction of 1s in the leaves of the binary tree would no longer be 1 / 2 . An interesting problem is to find mechanisms that have lower variance and attractive computational properties. It is possible that the dependence on T can be improved by leaving the factorization mechanism framework, but in absence of such a result the best we can hope is to match the variance obtained by Henzinger et al. (2023).

Acknowledgments and disclosure of funding

We would like to thank Jalaj Upadhyay for useful discussions about our results and their relation to previous work. We would also like to thank the reviewers for their comments and suggestions which have contributed to a clearer paper. The authors are affiliated with Basic Algorithms Research Copenhagen (BARC), supported by the VILLUM Foundation grant 16582, and are also supported by Providentia, a Data Science Distinguished Investigator grant from Novo Nordisk Fonden.

References

Bun, M. and Steinke, T. Concentrated differential privacy: Simplifications, extensions, and lower bounds. In Hirt, M. and Smith, A. D. (eds.), Theory of Cryptography - 14th International Conference, TCC 2016-B, Beijing, China, October 31 - November 3, 2016, Proceedings, Part I , volume 9985 of Lecture Notes in Computer Science , pp. 635-658, 2016. doi: 10.1007/978-3-662-53641-4 \ 24. URL https://doi.org/10.1007/978-3-662-53641-4_24 .

Cardoso, A. R. and Rogers, R. Differentially private histograms under continual observation: Streaming selection into the unknown. In Camps-Valls, G., Ruiz, F. J. R., and Valera, I. (eds.), International Conference on Artificial Intelligence and Statistics, AISTATS 2022, 28-30 March 2022, Virtual Event , volume 151 of Proceedings of Machine Learning Research , pp. 2397-2419. PMLR, 2022. URL https://proceedings.mlr.press/v151/rivera-cardoso22a.html .

Chan, T.-H. H., Shi, E., and Song, D. Private and Continual Release of Statistics. ACM Transactions on Information and System Security , 14(3):26:1-26:24, November 2011. doi: 10.1145/2043621. 2043626. URL https://doi.org/10.1145/2043621.2043626 .

Chan, T. H. H., Li, M., Shi, E., and Xu, W. Differentially Private Continual Monitoring of Heavy Hitters from Distributed Streams. In Fischer-H¨ ubner, S. and Wright, M. (eds.), Privacy Enhancing Technologies , Lecture Notes in Computer Science, pp. 140-159, Berlin, Heidelberg, 2012. Springer. doi: 10.1007/978-3-642-31680-7 8. URL https://doi.org/10.1007/ 978-3-642-31680-7_8 .

Choquette-Choo, C. A., Ganesh, A., McKenna, R., McMahan, H. B., Rush, K., Thakurta, A. G., and Xu, Z. (Amplified) banded matrix factorization: A unified approach to private training, 2023. URL https://arxiv.org/abs/2306.08153 . Appears at NeurIPS 2023.

Choquette-Choo, C. A., McMahan, H. B., Rush, J. K., and Thakurta, A. G. Multi-epoch matrix factorization mechanisms for private machine learning. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceedings of Machine Learning Research , pp. 5924-5963. PMLR, 2023. URL https://proceedings.mlr.press/ v202/choquette-choo23a.html .

Denisov, S., McMahan, H. B., Rush, J., Smith, A. D., and Thakurta, A. G. Improved differential privacy for SGD via optimal private linear operators on adaptive streams. In NeurIPS , 2022. URL http://papers.nips.cc/paper_files/paper/2022/hash/ 271ec4d1a9ff5e6b81a6e21d38b1ba96-Abstract-Conference.html .

Dinur, I. and Nissim, K. Revealing information while preserving privacy. In Proceedings of the twenty-second ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems , PODS '03, pp. 202-210, New York, NY, USA, June 2003. Association for Computing Machinery. doi: 10.1145/773153.773173. URL https://doi.org/10.1145/773153.773173 .

Dwork, C. and Roth, A. The Algorithmic Foundations of Differential Privacy. Foundations and Trends® in Theoretical Computer Science , 9(3-4):211-407, 2013. doi: 10.1561/0400000042. URL https://doi.org/10.1561/0400000042 .

Dwork, C., Naor, M., Pitassi, T., and Rothblum, G. N. Differential privacy under continual observation. In Proceedings of the forty-second ACM symposium on Theory of computing , STOC '10, pp. 715724, New York, NY, USA, June 2010. Association for Computing Machinery. doi: 10.1145/ 1806689.1806787. URL https://doi.org/10.1145/1806689.1806787 .

Fichtenberger, H., Henzinger, M., and Upadhyay, J. Constant matters: Fine-grained error bound on differentially private continual observation. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceedings of Machine Learning Research , pp. 10072-10092. PMLR, 2023. URL https://proceedings.mlr.press/v202/ fichtenberger23a.html .

Gehrke, J., Wang, G., and Xiao, X. Differential privacy via wavelet transforms. In 2010 IEEE 26th International Conference on Data Engineering (ICDE 2010) , pp. 225-236, Los Alamitos, CA, USA, mar 2010. IEEE Computer Society. doi: 10.1109/ICDE.2010.5447831. URL https: //doi.ieeecomputersociety.org/10.1109/ICDE.2010.5447831 .

Han, Y., Liang, Z., Liang, Z., Wang, Y., Yao, Y., and Zhang, J. Private Streaming SCO in ℓ p geometry with Applications in High Dimensional Online Decision Making. In Proceedings of the 39th International Conference on Machine Learning , pp. 8249-8279. PMLR, June 2022. URL https://proceedings.mlr.press/v162/han22d.html .

Hay, M., Rastogi, V., Miklau, G., and Suciu, D. Boosting the accuracy of differentially private histograms through consistency. Proc. VLDB Endow. , 3(1-2):1021-1032, sep 2010. doi: 10.14778/ 1920841.1920970. URL https://doi.org/10.14778/1920841.1920970 .

Henzinger, M., Upadhyay, J., and Upadhyay, S. Almost tight error bounds on differentially private continual counting. In Proceedings of the 2023 Annual ACM-SIAM Symposium on Discrete Algorithms (SODA) , pp. 5003-5039, 2023. doi: 10.1137/1.9781611977554.ch183. URL https: //epubs.siam.org/doi/abs/10.1137/1.9781611977554.ch183 .

Honaker, J. Efficient use of differentially private binary trees. Theory and Practice of Differential Privacy (TPDP 2015), London, UK , 2:26-27, 2015.

Huang, Z., Qiu, Y., Yi, K., and Cormode, G. Frequency estimation under multiparty differential privacy: one-shot and streaming. Proceedings of the VLDB Endowment , 15(10):2058-2070, September 2022. doi: 10.14778/3547305.3547312. URL https://doi.org/10.14778/3547305. 3547312 .

Kairouz, P., Mcmahan, B., Song, S., Thakkar, O., Thakurta, A., and Xu, Z. Practical and Private (Deep) Learning Without Sampling or Shuffling. In Proceedings of the 38th International Conference on Machine Learning , pp. 5213-5225. PMLR, July 2021. URL https://proceedings.mlr. press/v139/kairouz21b.html .

Li, C., Miklau, G., Hay, M., McGregor, A., and Rastogi, V. The matrix mechanism: optimizing linear counting queries under differential privacy. The VLDB Journal , 24(6):757-781, December 2015. doi: 10.1007/s00778-015-0398-x. URL https://doi.org/10.1007/s00778-015-0398-x .

McMahan, H. B. and Thakurta, A. Federated Learning with Formal Differential Privacy Guarantees, 2022. URL https://ai.googleblog.com/2022/02/federated-learning-with-formal. html . Accessed online 2023-05-15.

Smith, A., Thakurta, A., and Upadhyay, J. Is Interaction Necessary for Distributed Private Learning? In 2017 IEEE Symposium on Security and Privacy (SP) , pp. 58-77, May 2017. doi: 10.1109/SP. 2017.35. URL https://doi.org/10.1109/SP.2017.35 .

Upadhyay, J. Sublinear Space Private Algorithms Under the Sliding Window Model. In Proceedings of the 36th International Conference on Machine Learning , pp. 6363-6372. PMLR, May 2019. URL https://proceedings.mlr.press/v97/upadhyay19a.html .

A Proof of Lemma 3.6

The proof of Lemma 3.6 was omitted from the main text, it is given next.

Proof. The critical observation to make is that once a given p-sum is removed from a prefix sum, then it will never re-appear. Let its associated prefix string at time t be s of length l . If we look at the same l bit positions in bin( m ( t )) as t increases and interpret it as a number, then it is monotonously increasing with t . This implies that once a given prefix-string s in bin( m ( t )) disappears at t ′ > t then it will not be encountered again. We can therefore free up the memory used for storing any p-sum the moment it is no longer used. Because of this, it suffices to only store the p-sums in the active prefix sum at any time, of which there are at most O (log( T )) , proving the statement.

B Factorization mechanisms on multidimensional input

Assuming a d -dimensional input stream with elements x i ∈ R d , let two streams x, x ′ be neigboring if they differ at exactly one time step t where ∥ x t -x ′ t ∥ 2 ≤ 1 . Given a one-dimensional ρ -zCDP factorization mechanism for continual counting with factorization A = LR , we want to argue that running it along each dimension separately gives a ρ -zCDP factorization mechanism for releasing prefix sums on x .

To see this, consider the new, flattened vector input ˜ x ∈ R d · T where ˜ x = [( x 1 ) 1 , ( x 1 ) 2 , . . . , ( x 1 ) d , ( x 2 ) 1 , . . . , ( x T ) d ] , i.e., ( x t ) j = ˜ x ( t -1) · d + j . Analogously for these new inputs, we can define a counting matrix ˜ A = A ⊕ I d × d ∈ R d · T × d · T that sums each dimension separately, where ⊕ is the Kronecker product and I d × d is the d -dimensional identity matrix. We get a corresponding factorization ˜ A = ˜ L ˜ R where ˜ L = L ⊕ I d × d and ˜ R = R ⊕ I d × d .

It follows immediately that, ˜ L, ˜ R gives a factorization mechanism for privately releasing ˜ A ˜ x , and from it we can extract all private prefix sums on the original stream x . Letting ˜ x ′ be the vector representation of the stream x ′ ∼ x , we will reason about the ℓ 2 -sensitivity ∆ next. First observe that for neighbouring inputs x and x ′ that differ at time t , we have:

and therefore

which equates the ℓ 2 -sensitivity of the one-dimensional case. Recapitulating, any one-dimensional ρ -zCDP factorization mechanism for continual counting is also a ρ -zCDP counting mechanism for inputs of higher dimension, as long as the neigboring relation is appropriately extended.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe binary mechanism is a simple, efficient and well-known method to answer linear prefix queries (like counting) when a database of size T is not given in advanced, but rather in each time point $t \\in [T]$ we are given a new row $x_t$, and we are required to update our result based on the new row (while taking into account that the previous estimations were already released). For the prefix sum (or counting) problem where each row is 0 or 1, the binary mechanism in the zCDP model results with an additive error of Gaussian noise for each observation, where the maximal noise variance is $0.5 \\cdot \\log_2^2 T$. The noise variance is not identical for every observation, as it is a function of the number of zeros in the binary representation of $t-1$ (the observation index). This can also be change to be depend on the number of ones instead of the number of zeros.\nThe main contribution of this work is to develop a (simple but clever) variant of the binary mechanism that balances between the different noises such that each observation will get the same noise variance, and this variance will be only $(0.125 + o(1)) \\cdot \\log_2^2 T$, which is almost $4$ times better than the maximal variance of the binary mechanism (which translates to factor $2$ in the resulting noise magnitude).\nThis work does not improve upon the best-known variance, as Henzinger et al. (SODA 2023) and Denisov et al. (NeurIPS 2022) have managed to get variance of $0.04865 \\cdot \\log_2^2 T$, which is $10.2$ times better than the standard binary mechanism and $2.57$ times better than this work. However, these works are much more complex, and more importantly, much less efficient in terms of running time and space.\n\nSTRENGTHS:\nAt the bottom line, this paper improves a factor of $4$ in the variance of the Binary Mechanism with almost no additional cost in its complexity. Therefore, it potentially can be useful in practice. It does not achieve the best accuracy, but definitely provides a simpler and more efficient alternative to the more accurate but complex algorithms of Henzinger et al. (SODA 2023) and Denisov et al. (NeurIPS 2022).\n\nWEAKNESSES:\nThe improvement is clever and simple, but perhaps too simple for a NeurIPS submission. I lean towards accepting this paper mainly because it might be useful in practice.\n\nQUESTIONS:\nNo questions to the authors.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the continual counting with differential privacy problem. Here there are $T$ time steps; at time step $t \\\\in [T]$, we receive a $x_t \\\\in \\\\{0, 1\\\\}$ and we have to output an estimate to the sum so far $x_1 + \\\\cdots + x_t$. We would like to entire stream of output to be differentially private (DP) w.r.t. changing a single entry $x_t$. Below we will discuss the privacy guarantees in terms of concentrated DP (CDP) but this can be easily translated to standard approximate-DP guarantee.\n\nThe classic binary tree mechanism [Chan et al., 2011; Dwork et al., 2010] solves this problem by building a complete binary tree with $T$ leaves, each representing $x_t$ for $t \\in [T]$. Each node in the tree represents the sum of all leaves below. Appropriate noise are added to each node to make sure this data structure is private. When we want to output an estimate, we take the sum of the appropriate nodes; it is not hard to see that then number of nodes required are at most $O(\\\\log T)$. This gives error (standard deviation) for each estimate that is $O(\\\\log T)$, a significant improvement over the trivial algorithm (that release each estimate with independent noise)--which has error $O(\\sqrt{T})$. The binary tree mechanism belongs to a class of mechanism called matrix mechanism based on a factorization of the \"workload\" matrix, which is the lower triangular all-1 matrix in the continual counting case. [Denisov et al., 2022] experimented with matrix mechanism and showed significant error improvement (more than 3x) over the binary tree mechanism. [Henzinger et al., 2023] gave an elegant factorization based on Toeplitz matrices and prove that it is asymptotically optimal.\n\nDespite the (asymptotic) optimality of [Denisov et al., 2022, Henzinger et al., 2023], the algorithms are harder to implement than the binary tree mechanism. In particular, the noise generation requires $O(T)$ time and $O(T)$ space (for each update), whereas the binary tree mechanism (by keeping the noise only of the \"active\" node) requires only $O(1)$ amortized time (or $O(\\\\log T)$ worst case time) and $O(T)$ space. On the other hand, as stated above, the error of the binary tree mechanism is more than 3x larger than those of [Denisov et al., 2022, Henzinger et al., 2023]. This brings us to the contribution of this work: **the authors give an algorithm with the same time / space complexity as binary tree mechanism but with error reduced by a half** (so it is now only ~1.6x more than [Denisov et al., 2022, Henzinger et al., 2023]).\n\nTo understand the techniques, we need to discuss the binary tree mechanism in more detail. There are two important parameters that affect the error:\n1. *Sensitivity*: This is the amount of contribution from each $x_t$ to the datastructure (measured in $\\ell_2$ norm), which directly determines the scale of the noise needed to provide privacy.\n2. *Number of nodes to combine*: This is the number of nodes that needs to be summed together to produce an estimate; the larger this number is, the larger number of noises get added together--leading to a larger error.\n\nFor 1., the observation (which is perhaps folklore) to start with is that we never use the right children when computing the estimates anyway, so we can remove these from the trees. This leaves us with an interesting situation: each $x_t$ does not have the same sensitivity. In particular, the number of nodes $x_t$ contributes to is now the number zeros in the binary representation of $t - 1$. For 2., it can be seen that different estimates requires adding up different number of nodes: specifically, the number of nodes needed for $x_1 + \\\\cdots + x_t$ is the number of ones in the binary representation of $t$. This leads to a rather natural and elegant strategy that the authors propose: instead of using all leaves, only use those that have equal number of zeros and ones in their binary representations. This reduces the sensitivity by $\\\\sqrt{2}$ and reduces the number of node to combined by a factor of $2$, leading to a total of $2$ saving in the standard deviation of the estimates compared to the vanilla binary tree mechanism. Note that since there are only $\\\\Theta(1/\\\\sqrt{T})$ fraction of leaves with equal zero-one in their binary representations, we actually have to look at a slightly higher tree than before, but this only contributes to $O(\\log \\log T)$--which is a lower order term--in the error.\n\n## Post-Rebuttal Comments\n\nThank you the authors for the responses. As stated in my reply, I'd still strongly encourage the authors to clarify W2 and Q1 in more detail in the revision.\n\nSTRENGTHS:\n- Continual counting is a basic problem in literature and this paper provides a significant improvement over the classic binary tree mechanism.\n\n- The approach is very elegant is likely practical.\n\nWEAKNESSES:\n- Although this paper's algorithm is more efficient than those in [Denisov et al., 2022, Henzinger et al., 2023], the error is still quite noticeably larger (by ~1.6x factor) for the same privacy level. Therefore, this is a tradeoff between time-space efficiency and privacy-utility tradeoff. In my opinion, the latter is typically more important in practice.\n\n- The writing is not ideal:\n  - Main algorithm (Algorithm 2) is not presented in the main body.\n  - Despite discussing and using the high-dimensional version (where each $x_i$ is a $d$-dimensional vector) in the experiments, the main proof sections only consider the case where $x_i$ is a binary scalar.\n  - \"Streaming support\" and \"same error distribution\" is discussed in a possibly misleading manner (see the first two questions below in more detail).\n\nQUESTIONS:\n- Line 57-58: \"Unlike existing mechanisms it has the same error distribution in every step, which could make downstream applications easier to analyze.\" I am not sure why this is a big deal, since the noise distribution is just a Gaussian we can always just add another independent Gaussian to make it the same, right? (For example, if at the $t$-time step, my variance is $\\\\sigma_t^2$ whereas my worst case variance is $\\\\sigma_{\\\\max}^2$, then I can just add a noise drawn from $\\\\mathcal{N}(0, \\\\sigma_{\\\\max}^2 - \\\\sigma_t^2)$ to get the error distribution to be the same, right?)\n\n- In Table 1, it is said that if we precompute the noise for Henzinger et al.'s algorithm, then we would not be able to support streaming setting. I do not understand why this is the case. Can you please elaborate? (Note that we have to be talking about fix time horizon $T$ anyway, since even your algorithm would need this to be able to set the noise magnitude.)\n\n- Since Henzinger et al.'s factorization is pretty explicit (and has very nice form), is there any (informal) reason why one should think that it is hard to implement their algorithm using say polylogarithmic space and time?\n\n- It seems like for your algorithm we only need the binary representation of every contributed time step minus one to have *at most* $h/2$ zeros and the binary representation of every queried time step to have *at most* $h/2$ ones. Since about half of the leaves have at most $h/2$ zeros (or $h/2$ ones), can you use this, instead of *exactly* $h/2$ to reduce the extra height to just $O(1)$ instead of the current $O(\\\\log \\\\log T)$? (Note that I think you do not need the contributed / queried leaves to be exactly equal; you only need a sequence $t_1, \\dots, t_T$ for contribution and $q_1, \\dots, q_T$ for queries that are interleaved, i.e. $t_1 \\leq q_1 < t_2 \\leq q_2 < \\cdots < t_T \\leq q_T$.)\n\n## Minor comments\n\nBelow are minor comments to help the authors improve their revision. Please do *not* reply to them during rebuttal.\n\n- Line 49: \"algoritm\" -> \"algorithm\"\n\n- Line 91 - 94: I believe the fact that finding the optimal matrix mechanism can be formulated as a convex optimization problem was well known before [Denisov et al.]. See for example, the paper \"Factorization Norms and Hereditary Discrepancy\" by Matousek et al. and references therein.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the important problem of optimizing continual observation algorithms while also making them efficient.  There has been some recent advances in the classical binary mechanism for releasing a streaming counter subject to differential privacy.  One of the main issues with the binary mechanism is that is add varying levels of noise to each count, since the number of noise terms that are added at each round depends on the number of 1’s in the binary representation of the length of the stream.  Work from Henzinger et al. shows how the variance of the noise need not change drastically at each round and overall the variance can be significantly reduced.  However, this approach has runtime that scales linearly with the length of the stream, which can be prohibitive, especially when wanting to release counts in a streaming setting.  This work shows that it is possible to get closer to the benefits of Henzinger et al. while also having runtime that is similar to the binary mechanism, making it a great replacement for the classical binary mechanism.\n\nSTRENGTHS:\nThere are impressive results with this work, where variance is almost constant across rounds and the overall variance is similar to the optimal approaches.  Table 1 nicely summarizes the results and how they compare to existing approaches.\n\nWEAKNESSES:\nThe new algorithm in Section 3.2 is not clearly presented.  As this is the main contribution of this work, I would expect the algorithm to be clearly presented, but it is pushed to the supplementary file.  There is a nice figure to explain the binary mechanism, but not the new algorithm.  Providing an example here would help, as is done with the binary mechanism.  I even went to the supplementary code file to better understand what the algorithm is doing, but there are few comments so it is not very readable.  Will increase my score if the paper can be updated with a better description of the main algorithm.\n\nQUESTIONS:\nCan you give a better presentation for the algorithm and how it works?  The whole algorithm seems to only be two itemized bullets in lines 214 and 215.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of releasing private prefix sums in the continual release model of differential privacy. A continual release mechanism for prefix sums receives a binary stream $x_1, x_2, x_3, ...$ one element at a time and continually outputs, at each time step, the number of ones received so far.  This problem can be solved by deploying the binary tree mechanism introduced by the concurrent works of Dwork et. al. and Chan et al. from 2010. In this manuscript the authors provide a variant of the binary tree mechanism that preserves its asymptotic space complexity while achieving better variance than the base mechanism. While the variance of the proposed smooth binary tree mechanism is higher than that of the matrix mechanism proposed by Henzinger et. al., the smooth binary tree scales better in time and space, with respect to the dimensionality of the stream elements. Finally, the authors claim that the error of this mechanism is identically distributed for all outputs of the mechanism.\n\nSTRENGTHS:\nThe problem of private prefix sums is a fundamental problem which is used as a subroutine in various private algorithms. Therefore the smooth binary tree mechanism with improved variance and similar asymptotic space and time complexity is an interesting contribution. The paper provides a clear explanation of the proposed mechanism as well as the naive binary tree mechanism. These explanations are supported by helpful diagrams.\n\nWEAKNESSES:\n-The text in the paper is very clear but the formal portions of the paper are (in several instances) not well specified.\n\n-While the proposed mechanism is nice, the analysis of the mechanism could use much more detail.\n\n-In particular, the paper makes several claims about the error of the proposed mechanism in the text of the paper, even adding this value to a table in the introduction. However the manuscript does not specifically define their notion of error or make any formal verifiable statements about the error of the proposed mechanism in either the main paper or the supplementary material. While the notion of error they are talking about can be inferred from the text, the distribution of the error of the mechanism is key selling point that the authors mention several times. This omission and the lack of detail in the analysis would make the paper fairly inaccessible for many readers.\n\nQUESTIONS:\nIs there a reason that you do not consider any specific notions or provide any formal statements and proofs about the error of the smooth mechanism? \n\nIn line 221 you say that the maximum sensitivity is proportional to $k$. I understand that this is true for your choice of $k=h/2$ but as far as I can tell, it need not be true in general, right? (Since the maximum sensitivity of a particular node is technically proportional to $h-k$, which need not be proportional to $k$?)\n\n-------------------\n\n-------------------\n\nTypo(s) and minor comments:\n\nOn line 112 you say “let $\\mathsf{bin}(n)$ be padded with leading to $h$ digits.” This sentence is missing the word zeroes.\n\nThere seems to be a mistake in line 68: It says that the sensitivity of some node $x_t$ is the number of zeroes in  $\\mathsf{bin}(t-1)$. Isn't it actually $\\mathsf{bin}(t-1) + 1$? (Also, line 68 occurs prior to the definition of $\\mathsf{bin}(\\cdot)$ which doesn’t appear till line 113.) A similar problem to that in line 68 occurs in the statement of proposition 3.1 on line 190, which is a formal statement of line 68.\n\nIn table 1, the column ‘identically distributed’ does not indicate that it is about error, or which error notion(s) it is about.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-13"}