Title: Private Federated Frequency Estimation: Adapting to the Hardness of the Instance
Venue: NeurIPS 2023
Accepted: yes
Keywords: sketch

federated analytics

privacy
Abstract: In federated frequency estimation (FFE), multiple clients work together to estimate the frequency of their local data by communicating with a server, while maintaining the security constraint of $\mathtt{secsum}$ where the server can only access the sum of client-held vectors. For FFE with a single communication round, it is known that count sketch is nearly information-theoretically optimal [Chen et al., 2022]. However, when multiple communication rounds are allowed, we propose a new sketch algorithm that is provably more accurate than a naive adaptation of count sketch. Furthermore, we show that both our sketch algorithm and count sketch can achieve better accuracy when the problem instance is simpler. Therefore, we propose a two-phase approach to enable the use of a smaller sketch size for simpler problems. Finally, we provide mechanisms to make our proposed algorithm differentially private. We verify the performance of our methods through experiments conducted on real datasets.

Private Federated Frequency Estimation: Adapting to the Hardness of the Instance

Abstract

In federated frequency estimation (FFE), multiple clients work together to estimate the frequencies of their collective data by communicating with a server that respects the privacy constraints of Secure Summation ( SecSum ), a cryptographic multiparty computation protocol that ensures that the server can only access the sum of client-held vectors. For single-round FFE, it is known that count sketching is nearly information-theoretically optimal for achieving the fundamental accuracycommunication trade-offs [Chen et al., 2022]. However, we show that under the more practical multi-round FEE setting, simple adaptations of count sketching are strictly sub-optimal, and we propose a novel hybrid sketching algorithm that is provably more accurate. We also address the following fundamental question: how should a practitioner set the sketch size in a way that adapts to the hardness of the underlying problem ? We propose a two-phase approach that allows for the use of a smaller sketch size for simpler problems (e.g. near-sparse or light-tailed distributions). We conclude our work by showing how differential privacy can be added to our algorithm and verifying its superior performance through extensive experiments conducted on large-scale datasets.

1 Introduction

In many distributed learning applications, a server seeks to compute population information about data that is distributed across multiple clients (users). For example, consider a distributed frequency estimation problem where there are n clients, each holding a local data from a domain of size d , and a server that aims to estimate the frequency of the items from the n clients with the minimum communication cost. This task can be done efficiently by letting each client binary encode their data and send the encoding to the server, at a local communication bandwidth cost of log( d ) bits. With the binary encoding, the server can faithfully decode each local data and compute the global frequency vector (i.e., the normalized histogram vector).

However, the local data could be sensitive or private, and the clients may wish to keep it hidden from the server. The above binary encoding communication method, unfortunately, allows the server to observe each individual local data, and therefore may not satisfy the users' privacy concerns. Federated Analytics (FA) [Ramage and Mazzocchi, 2020, Zhu et al., 2020] addresses this issue by developing new methods that enable the server to learn population information about the clients while preventing the server from prying on any individual local data. In particular, a cryptographic

multi-party computation protocol, Secure Summation ( SecSum ) [Bonawitz et al., 2016], has become a widely adopted solution to provide data minimization guarantees for FA [Bonawitz et al., 2021]. Specifically, SecSum sets up a communication protocol between clients and the server, which injects carefully designed additive noise to each data that cancels out when all of the local data are summed together , but blurs out (information theoretically) each individual local data. Under SecSum , the server is able to faithfully obtain the correct summation of the data from all clients but is unable to read a single local data. The federated frequency estimation (FFE) problem refers to the distributed frequency estimation problem under the constraint of SecSum . Clearly, the binary encoding method is not compatible with SecSum , because when the binary encoding is passed to the server through SecSum , the server only gets the summation of the binary encodings of the users' data, which does not provide sufficient information for computing the global frequency vector.

Anaive approach to FFE can be accomplished by employing one-hot encoding : each client encodes its local data into a d -dimensional one-hot vector that represents the local frequency vector and sends it to the server through SecSum . Then the server observes the summation of the local frequency vectors using SecSum and scales it by the number of clients to obtain the true frequency vector. However, this one-hot encoding approach costs Θ( d log( n )) bits of communication per client. This is because SecSum adds noise from a finite group of size Θ(log n ) to each component of the d -dimensional local frequency vector [Bonawitz et al., 2016] to avoid overflows. With a linear dependence on domain size d , the one-hot encoding approach is inefficient for large domain problems, which is a common setting in practice. In what follows, we focus on the regime where d > n .

Recently, linear compression methods were applied to mitigate the high communication cost issue for FFE with large domains [Chen et al., 2021, 2022]. The idea is to first linearly compress the local frequency vector into a lower dimensional vector before sending it to the server through SecSum ; as linear compression operators commute with the summation operator, the server equivalently observes a linearly compressed global frequency vector though SecSum (after rescaling by the number of clients). The server then applies standard decoding methods to approximately recover the global frequency vector from the linearly compressed one. In particular, Chen et al. [2022] show that CountSketch [Charikar et al., 2002] (among other sparse recovery methods) can be used as a linear compressor for the above purpose, which leads to a communication bandwidth cost of O ( n log( d ) log( n )) bits. Therefore when d > n , CountSketch achieves a saving in local communication bandwidth compared to the one-hot encoding method that requires Θ( d log( n )) bits. Moreover, Chen et al. [2022] show that for FFE with a single communication round, an Ω( n log( d )) local communication cost is information-theoretically unavoidable for worst-case data distributions, i.e., we cannot do better without making additional assumptions on the global frequency vector.

Contributions. In this work, we make three notable extensions to CountSketch for FFE problems.

We show that the way Chen et al. [2022] set up the sketch size (linear in the number of clients n ) is often pessimistic (see Corollary 2.4). In fact, in the streaming literature, the estimation error induced by CountSketch is known to adapt to the tail norm of the global frequency vector [Minton and Price, 2014], which is often sub-linear in n . Motivated by this, we provide an easy-to-use, two-phase approach that allows practitioners to determine the necessary sketch size by automatically adapting to the hardness of the FFE problem instance.

We consider FFE with multiple communication rounds, which better models practical deployments of FA where aggregating over (hundreds of) millions of clients in a single round is not possible due to device availability and limited server bandwidth. We propose a new multi-round sketch algorithm called HybridSketch that provably performs better than simple adaptations of CountSketch in the multi-round setting, leading to further improvements in the communication cost. Surprisingly, we show that HybridSketch adapts to the tail norm of a heterogeneity vector (see Theorem 3.2). Moreover, the tail of the heterogeneity vector is always no heavier, and could be much lighter, than that of the global frequency vector, explaining the advantage of HybridSketch . For instance, on the C4 dataset [Bowman et al., 2020] with a domain size of d = 150 , 868 and 150 , 000 users, we show that our method can reduce the sketch size by 83% relative to simple sketching methods when the number of sketch rows is not very large.

We extend the Gaussian mechanism for CountSketch proposed by Pagh and Thorup [2022], Zhao et al. [2022] to the multi-round FFE setting to show how our sketching methods can be made differentially private [Dwork et al., 2006]. We also characterize the trade-offs between accuracy and privacy for our proposed method.

We conclude by verifying the performance of our methods through experiments conducted on several large-scale datasets. All proofs and additional experimental results are differed to the appendices.

2 Adapting CountSketch to the Hardness of the Instance

In this part, we focus on single-round FFE and show how CountSketch can achieve better results when the underlying problem is simpler. Motivated by this, we also provide a two-phase method for auto-tuning the hyperparameters of CountSketch , allowing it to automatically adapt to the hardness of the instance.

Single-Round FFE. Consider n clients, each holding an item from a discrete domain of size d . The items are denoted by x t ∈ [ d ] for t = 1 , . . . , n . Then the frequency of item j is denoted by

We use x t to denote the one-hot representation of x t , i.e., x t = e x t where ( e t ) d t =1 refers to the canonical basis. Then the frequency vector can be denoted by

In single-round FFE, the n clients communicate with a server once under the constraint of SecSum , and aim to estimate the frequency vector f . Note that SecSum ensures that the server can only observe the sum of the local data.

Algorithm 1 COUNT SKETCH FOR FEDERATED FREQUENCY ESTIMATION

Require: n clients with local data x t ∈ [ d ] for t = 1 , . . . , n . Sketch length L and width W .

1: The server prepares independent hash functions and broadcasts them to each client:

2: for Client t = 1 , . . . , n in parallel do

3: Client t encodes the local data x t ∈ [ d ] to enc ( x t ) ∈ R L × W where

4: Client t sends enc ( x t ) ∈ R L × W to SecSum .

5: end for

7: for Item j = 1 , . . . , d in parallel do

6: SecSum receives ( enc ( x t ) ) n t =1 and only reveals the summation ∑ n t =1 enc ( x t ) to the server.

8: Server produces L estimators for f j :

9: Server computes the median of the L estimators:

10: end for

11: return ( dec ( j )) d j =1 as estimate to ( f j ) d j =1 .

Count Sketch. CountSketch is a classic streaming algorithm that dates back to [Charikar et al., 2002]. In the literature of streaming algorithms, CountSketch has been extensively studied and is known to be able to adapt to the hardness of the problem instance. Specifically, CountSketch of a fixed size induces an estimation error adapting to the tail norm of the global frequency vector [Minton and Price, 2014].

A recent work by Chen et al. [2022] apply CountSketch to single-round FFE. See Algorithm 1 for details. They show that CountSketch approximately solves single-round FFE with a communication cost of O ( n log( d ) log( n )) bits per client. Moreover, they show Ω( n log( d )) bits of communication per client is unavoidable for worst-case data distributions (unless additional assumptions are made),

confirming its near optimality. However, the results by Chen et al. [2022] are pessimistic as they ignore the ability of CountSketch to adapt to the hardness of the problem instance. In what follows, we show how the performance of CountSketch can be improved when the underlying problem becomes simpler.

We first present a problem-dependent accuracy guarantee for CountSketch of a fixed size, L × W , that gives the sharpest bound to our knowledge. The bound is due to Minton and Price [2014] and is restated for our purpose.

Proposition 2.1 (Restated Theorem 4.1 in Minton and Price [2014]) . Let ( ˆ f j ) d j =1 be estimates produced by CountSketch (see Algorithm 1). Then for each p ∈ (0 , 1) , W ≥ 2 and L ≥ log(1 /p ) , it holds that: for each j ∈ [ d ] , with probability at least 1 -p ,

where ( f ∗ i ) i ≥ 1 refers to ( f i ) i ≥ 1 sorted in non-increasing order, and C > 0 is an absolute constant.

For the concreteness of discussion, we will focus on glyph[lscript] ∞ as a measure of estimation error in the remainder of the paper. Our discussions can be easily extended to glyph[lscript] 2 or other types of error measures. Proposition 2.1 directly implies the following glyph[lscript] ∞ -error bounds for CountSketch (by an application of union bound).

Corollary 2.2 ( glyph[lscript] ∞ -error bounds for CountSketch ) . Consider Algorithm 1. Then for each p ∈ (0 , 1) , L = log( d/p ) and W ≥ 2 , it holds that: with probability at least 1 -p ,

where C > 0 is an absolute constant. In particular, (1) implies that

According to Corollary 2.2, the estimation error is smaller when the underlying frequency vector ( f ∗ i ) i ≥ 1 has a lighter tail. In other words, CountSketch requires a smaller communication bandwidth when the global frequency vector has a lighter tail. Our next Corollary 2.3 precisely characterizes this adaptive property in terms of the required communication bandwidth. To show this, we will need the following definition on the probable approximate correctness of an estimate.

Definition 1 ( ( τ, p ) -correctness) . An estimate ˆ f := ( ˆ f i ) d i =1 of the global frequency vector f := ( f i ) d i =1 is ( τ, p ) -correct if

Corollary 2.3 (Oracle sketch size) . Fix parameters τ, p ∈ (0 , 1) . Then for CountSketch (see Algorithm 1) to produce an ( τ, p ) -correct estimate, it suffices to set the sketch size to L = log( d/p ) and

where C > 0 is an absolute constant. In particular, the width W in (2) satisfies

Corollary 2.3 suggests that the sketch size can be set smaller if the underlying frequency vector has a lighter tail. When translated to the communication bits per client (that is O ( L · W · log( n )) , where log( n ) accounts for the cost of SecSum ), Corollary 2.3 implies that CountSketch requires

bits of communication per client to be ( τ, p ) -correct. In the worst case where ( f i ) d i =1 is Θ( n ) -sparse and τ = O (1 /n ) , (4) nearly matches the Ω( n log( d )) information-theoretic worst-case communication cost shown in Chen et al. [2022], ignoring the log( n ) factor from SecSum . However, in practice, ( f i ) d i =1 has a fast-decaying tail, and (4) suggests that CountSketch can use less communication to solve the problem. We provide the following examples for a better illustration of the sharp contrast between the worst and typical cases.

Corollary 2.4 (Examples) . Fix parameters τ, p ∈ (0 , 1) . Consider Algorithm 1 with sketch length L = log( d/p ) . Then in each case for Algorithm 1 to produce an ( τ, p ) -correct estimate for τ > 1 /n :

When f i ∝ 2 -i , it suffices to set W = Θ(log(1 /τ )) .

When f i ∝ i -a for a > 1 , it suffices to set W = Θ( τ -1 /a ) .

When f i ∝ i -1 log -b ( i ) for b > 1 , it suffices to set W = Θ( τ -1 log -b (1 /τ )) .

When f i = 10 /n for i = 1 , . . . , n/ 10 , it suffices to set W = Θ(1 /τ ) .

ATwo-Phase Method for Hyperparameter Setup. Corollary 2.3 allows to use CountSketch with a smaller width for an easier single-round FFE problem, saving communication bandwidth. However, the sketch size formula given by (2) in Corollary 2.3 relies on crucial information of the frequency ( f i ) i ≥ 1 , i.e., # { f i : f i ≥ τ } and ∑ f i <τ f 2 i , which are unknown to who sets the sketch size. Thus, it is unclear if and how these gains can be realized in practical deployments.

We resolve this quandary by observing that in practice, the frequency vector often follows Zipf's law [Cevher, 2009, Powers, 1998]. This motives us to conservatively model the global frequency vector by a polynomial with parameters. By doing so, we can first run a small CountSketch to collect data from a (randomly sampled) fraction of the clients for estimating the parameters. Then based on the estimated parameter, we can set up an appropriate sketch size for a CountSketch to solve the FFE problem. This two-phase method is formally stated as follows.

We approximate the (sorted) global frequency vector ( f ∗ i ) d i =1 by a polynomial [Cevher, 2009] with two parameters α > 0 and β > 0 , such that

where i ∗ := max { i : ∑ i j =1 β · j -α ≤ 1 } is set such that poly ( i ; α, β ) is a valid frequency vector. Here's an executive summary of the proposed approach for setting the sketch size.

Randomly select a subset of clients (e.g., 5 , 000 out of 10 6 .)

Fix a small sketch (e.g., 16 × 100 ) and run Algorithm 1 with the subset of clients to obtain an estimate ( ˜ f i ).

Use the topk values (e.g., top 20 ) from ˜ f i to fit a polynomial with parameter α and β (under squared error).

Solve Equation (4) under the approximation that f ∗ i ≈ β · i α and output W according to the result.

Experiments. We conduct three sets of experiments to verify our methods. In the first set of experiments, we simulate a single-round FFE problem with the Gowalla dataset [Cho et al., 2011]. The dataset contains 6 , 442 , 892 lists of location information. We first construct a domain of size d = 175 , 000 , which corresponds to a grid over the US map. Then we sample n = d/ 10 = 17 , 500 lists of the location information (that all belong to the domain created) to represent the data of n clients, uniformly at random. This way, we set up a single-round FFE problem with n = 17 , 500 clients in a domain of size d = 175 , 000 . In the experiments, we fix the confidence parameter to be p = 0 . 1 and the sketch length to be L = ln(2 d/p ) ≈ 16 . The targeted glyph[lscript] ∞ -error τ is chosen evenly from (10 -3 , 10 -1 ) . We only test τ > 20 /n because it is less important to estimate frequencies over items with small counts (say, 20 ). For CountSketch , we compute sketch width with three strategies, using (2) (called 'instance optimal'), using (3) (called 'minimax optimal'), and using the two-phase method. We emphasize that the 'instance optimal' method is not a practical algorithm as it requires access to unknown information about the frequency; we use it only for demonstrating the correctness of our theory. We set all constant factors to be 2 . The results are presented in Figures 1(a) and (b). We observe that the 'minimax optimal' way of hyperparameter choice is in fact suboptimal in practice, and is improved by the 'instance optimal' and the two-phase strategies.

In the second set of experiments, we run simulations on the 'Colossal Clean Crawled Corpus' (C4) dataset [Bowman et al., 2020], which consists of clean English text scraped from the web. We treat each domain in the dataset as a user and calculate the number of examples each user has. The domain size d = 150 , 868 , which is the maximum example count per user. We randomly sample n = 150 , 000 users from the dataset. We fix the sketch length to be L = 5 . Other parameters are the same as the Gowalla dataset. The results are presented in Figures 1(d) and (e), and are consistent with what we have observed in the Gowalla simulations.

10

3

10

2

10

1

target

error

10

3

10

2

10

1

achieved

error

y=x

minimax optimal

instance optimal

two-phase-method

10

3

10

2

10

1

target

error

10 0

10 1

10 2

10 3

sketch width

minimax optimal

instance optimal

two-phase-method

0

200

400

600

800

1000

1200

sketch width

0.000

0.005

0.010

0.015

0.020

0.025

0.030

0.035

achieved

error

shared design

fresh design

hybrid design

(a) Gowalla, single round

(b) Gowalla, single round

(c) Gowalla, multi-round

10

3

10

2

10

1

target

error

10

4

10

3

10

2

10

1

achieved

error

y=x

minimax optimal

instance optimal

two-phase-method

10

3

10

2

10

1

target

error

10 2

10 3

sketch width

minimax optimal

instance optimal

two-phase-method

0

200

400

600

800

1000

1200

sketch width

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

achieved

error

shared design

fresh design

hybrid design

(d) C4, single round

(e) C4, single round

(f) C4, multi-round

10

3

10

2

10

1

target

error

10

3

10

2

10

1

achieved

error

y=x

minimax optimal

instance optimal

two-phase-method

10

3

10

2

10

1

target

error

10 0

10 1

10 2

10 3

sketch width

minimax optimal

instance optimal

two-phase-method

0

100

200

300

400

500

600

sketch width

0.000

0.005

0.010

0.015

0.020

achieved

error

shared design

fresh design

hybrid design

(g) Setiment-140, single round

(h) Setiment-140, single round

(i) Setiment-140, multi-round

In the third set of experiments, we run simulations on a Twitter dataset Sentiment-140 [Go et al., 2009]. The dataset contains d = 739 , 972 unique words from N = 659 , 497 users. We randomly sample one word from each user to construct our experiment dataset. The number of rounds M = 10 , and in each round, n = N/ 10 = 65 , 949 clients participate. The algorithm setup is the same as in the Gowalla experiments. Results are provided in Figures 1(g) and (h), and are consistent with our prior understandings.

3 Sketch Methods for Multi-Round Federated Frequency Estimation

In practice, having all clients participate in a single communication round is usually infeasible due to the large number of devices, their unpredictable availability, and limited server bandwidth [Bonawitz et al., 2019]. This motivates us to consider a multi-round FFE setting.

Multi-Round FFE. Consider a FFE problem with M rounds of communication. In each round, n clients participate, each holding an item from a universe of size d . The items are denoted by x ( m ) t ∈ [ d ] , where t ∈ [ n ] denotes the client index and m ∈ [ M ] denotes the round index. For simplicity, we assume in each round a new set of clients participate. So in total there are N = Mn clients. Then the frequency of item j is now denoted by

For the m -th round, the local frequency is denoted by f ( m ) j := 1 n ∑ n t =1 1 [ x ( m ) t = j ] . Clearly, we have f j = 1 M ∑ M m =1 f ( m ) j . Similarly, we use x ( m ) t to denote the one-hot representation of x ( m ) t , i.e.,

0

100

200

300

400

500

600

700

800

sketch width

10

3

10

2

10

1

error in expectation

shared design

fresh design

hybrid design

(a)

f

i

∝

i

-

1

.

1

0

50

100

150

200

250

300

350

400

sketch width

10

3

10

2

10

1

10 0

error in expectation

shared design

fresh design

hybrid design

(b)

f

i

∝

i

-

3

25

50

75

100

125

150

175

200

sketch width

10

3

10

2

10

1

10 0

error in expectation

shared design

fresh design

hybrid design

(c)

f

i

∝

i

-

5

x ( m ) t = e x ( m ) t where ( e t ) d t =1 refers to the canonical basis. Then the frequency vector can be denoted by f := ( f 1 , . . . , f d ) glyph[latticetop] . The aim is to estimate the frequency vector f in a manner that is compatible with SecSum .

Baseline Method 1: Shared Sketch. A multi-round FFE problem can be reduced to a single-round FFE problem with a large communication. Specifically, one can apply the CountSketch with the same randomness for every round; after collecting all the sketches from the M round, one simply averages them. Due to the linearity of the sketching compress method, this is equivalent to a single round setting with N = Mn clients. We refer to this method as count sketch with shared hash design ( SharedSketch ).

Thanks to the reduction idea, we can obtain the error and sketch size bounds for SharedSketch via applying Corollaries 2.2 and 2.3 to SharedSketch by replacing n by N = Mn ,

Baseline Method 2: Fresh Sketch. A multi-round FFE problem can also be broken down to M independent single-round FFE problems. Specifically, one can apply independent CountSketch in each round, and decode M local estimators for the M local frequency vectors. As the CountSketch produces an unbiased estimator, one can show that the average of the M local estimators is an unbiased estimator for the global frequency vector. We call this method count sketch with fresh hash design ( FreshSketch ). We provide the following bound for FreshSketch . The proof of which is motivated by Huang et al. [2021].

Theorem 3.1 (Instance-specific bound for FreshSketch ) . Let ( ˆ f j ) d j =1 be estimates produced by FreshSketch . Then for each p ∈ (0 , 1) , W ≥ 1 and L ≥ log(1 /p ) , it holds that: for each j ∈ [ d ] , with probability at least 1 -p ,

where C is an absolute constant, and ( F ∗ i ) d i =1 are defined as in Theorem 3.2.

Hybrid Sketch. Both SharedSketch and FreshSketch reduce a multi-round FFE problem into single-round FFE problem(s). In contrast, we show a more comprehensive sketching method, called count sketch with hybrid hash design ( HybridSketch ), that solves a multi-round FFE problem as a whole. HybridSketch is presented as Algorithm 2. Specifically, HybridSketch generates M sketches that share a set of bucket hashes but use independent sets of sign hashes. Then in the m -th communication round, participating clients and the server communicate by the CountSketch algorithm based on the m -th sketch, so the server observes the summation of the sketched data through SecSum . After collecting M summations of the sketched local data, the server first computes averages over different rounds for variance reduction , then computes the median over different repeats

Algorithm 2 HYBRID SKETCH FOR FEDERATED FREQUENCY ESTIMATION

Require: The number of rounds M . N = Mn clients with local data x ( m ) t ∈ [ d ] for m ∈ [ M ] and t ∈ [ n ] . Sketch length L and width W .

1: The server prepares independent hash functions and broadcasts them to each client:

2: for Round m = 1 , . . . , M in parallel do

3: for Client t = 1 , . . . , n in parallel do

4: Client ( m,t ) encodes the local data x ( m ) t to enc ( m ) ( x ( m ) t ) ∈ R L × W where

5: Client ( m,t ) sends enc ( m ) ( x ( m ) t ) to SecSum .

6: end for

8: end for

7: SecSum receives ( enc ( m ) ( x ( m ) t ) ) n t =1 and reveals the sum ∑ n t =1 enc ( m ) ( x ( m ) t ) to the server.

9: for Item j = 1 , . . . , d in parallel do

10: Server produces M × L estimators for f j :

11: Server computes the median over glyph[lscript] ∈ [ L ] of the averages over m ∈ [ M ] of the estimators:

12: end for

13: return ( dec ( j ) ) j =1 as estimate to ( f j ) d j =1 .

d

(or sketch rows) for success probability amplification . We provide the following problem-dependent bound for HybridSketch .

Theorem 3.2 (Instance-specific bound for HybridSketch ) . Let ( ˆ f j ) d j =1 be estimates produced by HybridSketch (see Algorithm 2). Define a heterogeneity vector ( F i ) d i =1 by

Clearly, it holds that F i ≤ f i for every i ∈ [ d ] . Let ( F ∗ i ) i ≥ 1 be ( F i ) i ≥ 1 sorted in non-increasing order. Then for each p ∈ (0 , 1) , W ≥ 1 and L ≥ log(1 /p ) , it holds that: for each j ∈ [ d ] , with probability at least 1 -p ,

where C is an absolute constant.

We would like to point out that, although our HybridSketch algorithm is developed for multi-round frequency estimation problems, it can be adapted to multi-round vector recovery problems as well. Hence it could have broader applications in other federated learning scenarios.

Hybrid Sketch vs. Fresh Sketch. By comparing Theorem 3.2 with Theorem 3.1, we see that, with the same sketch size, the estimation error of HybridSketch is smaller than that of FreshSketch by a factor of √ log( M/p ) . This provides theoretical insights that HybridSketch is superior to FreshSketch in terms of adapting to the instance hardness in multi-round FFE settings. This is also verified empirically by Figure 2.

Hybrid Sketch vs. Shared Sketch. We now compare the performance of HybridSketch and SharedSketch by comparing Theorem 3.2 and Proposition 2.1 (under a revision of replacing n with N = Mn ). Note that

So with the same sketch size, HybridSketch achieves an error that is no worse than csc in every case. Moreover, in the homogeneous case where all local frequency vectors are equivalent to the global frequency vector, i.e., f ( m ) ≡ f for all m , then it holds that F i = f i / √ M. So in the homogeneous case, HybridSketch achieves an error that is smaller than that of csc by a factor of 1 / √ M . In the general cases, the local frequency vectors are not perfectly homogeneous, then the improvement of HybridSketch over

0

200

400

600

800

1000

1200

sketch width

10 2

10 3

10 4

10 5

# items w/ error  > 0.1 /

width

shared design

fresh design

hybrid design

SharedSketch will depend on the heterogeneity of these local frequency vectors.

Experiments. Weconduct three sets of experiments to verify our understandings about these sketches methods for multi-round FFE.

In the first sets of experiments, we simulate a multi-round FFE problem in homogeneous settings, where in every round the local frequency vectors are exactly the same. More specially, we set a domain size d = 10 5 , a number of rounds M = 10 and test three different cases, where all the local frequency vectors are the same and (hence also the global frequency vector) are proportional to ( i -1 . 1 ) d i =1 , ( i -2 ) d i =1 and ( i -5 ) d i =1 , respectively. In all the settings, we fix the sketch length to L = 5 . In each experiment, we measure the expected glyph[lscript] ∞ -error of each method with the averaging over 1 , 000 independent repeats. The results are plotted in Figure 2. We can observe that: for low-degree polynomials, HybridSketch is nearly as good as FreshSketch and both are better than SharedSketch . But for slightly high degree polynomials (with a degree of 3 ), HybridSketch already outperforms both FreshSketch and SharedSketch . The numerical results are consistent with our theoretical analysis.

In the second sets of experiments, we simulate a multi-round FFE problem with the Gowalla dataset [Cho et al., 2011]. Similar to previously, we construct a domain of size d = 175 , 000 , which corresponds to a grid over the US map. Then we sample N = d = 175 , 000 lists of the location information (that all belong to the domain created) to represent the data of N clients, uniformly at random. We set the number of rounds to be M = 10 . In each round, n = N/M = 17 , 500 clients participate. The results are presented in Figure 1(c). Here, the frequency and heterogeneity vectors have heavy tails, so HybridSketch and FreshSketch perform similarly and both are better than SharedSketch . This is consistent with our theoretical understanding.

In the third sets of experiments, we run simulations on the C4 [Bowman et al., 2020] dataset. Similar to the single round simulation, the domain size d = 150 , 868 . We randomly sample N = 150 , 000 users from the dataset. The number of rounds M = 10 , and in each round, n = N/ 10 = 15 , 000 clients participate. The results are provided in Figures 1(f) and 3. Here, the frequency and heterogeneity vectors have moderately light tails, and Figure 3 already suggests that HybridSketch produces an estimate that has a better shape than that produced by FreshSketch and SharedSketch , verifying the advantages of HybridSketch .

4 Differentially Private Sketches

While SecSum provides security guarantees, it does not provide differential privacy guarantees. In this part, we discuss a simple modifications to the sketching algorithms to make them provably differentially private (DP).

Definition 2 ( ( glyph[epsilon1], δ ) -DP [Dwork et al., 2006]) . Let alg ( · ) be a randomized algorithm that takes a dataset D as its input. Let P be its probability measure. alg ( · ) is ( glyph[epsilon1], δ ) -DP if: for every pair of neighboring datasets D and D ′ , it holds that

In our case, a dataset corresponds to all participated clients (or their data), and two neighboring datasets should be regarded as two sets of clients (local data) that only differ in a single client (local data). The algorithm refers to all procedures before releasing the final frequency estimate, and all the intermediate computation is considered private and is not released.

We work with central DP , that is, server releases data in a differentially private way while clients do not release data. We focus on HybridSketch as a representative algorithm. The DP mechanism can also be extended to the other sketching algorithms. Specifically, we use a DP mechanism that adds independent Gaussian noise to each entry of the sketching matrix, which is initially proposed for making CountSketch differentially private by Pagh and Thorup [2022], Zhao et al. [2022].

We provide the following theorem characterizing the trade-off between privacy and accuracy.

Theorem 4.1 (DP-hybrid sketch) . Consider a modified Algorithm 2, where we add to each entry of the sketching matrix an independent Gaussian noise, N (0 , c 0 · √ L log(1 /δ ) /glyph[epsilon1] ) , where c 0 > 0 is a known constant. Suppose that L = log( d/p ) and W ≥ 2 . Then the final output of the modified Algorithm 2, denoted by ( ˆ f j ) d j =1 , is ( glyph[epsilon1], δ ) -DP for glyph[epsilon1] < 1 and δ < 0 . 1 . Moreover, with probability at most 1 -p , it holds that

where C > 0 is an absolute constant and ( F ∗ i ) d i =1 are as defined in Theorem 3.2.

It is worth noting that if the number of clients per round ( n ) is fixed, then a larger number of rounds M improves both the estimation error and the DP error in non-worst cases, e.g., when the local frequency vectors are nearly homogenous. However, if the total number of clients ( N = Mn ) is fixed, then a larger number of rounds M improves the estimation error but makes the DP error worse.

When M = 1 , Theorem 4.1 recovers the bounds for differentially private CountSketch in Pagh and Thorup [2022], Zhao et al. [2022] and Theorem 5.1 in Chen et al. [2022]. Moreover, Chen et al. [2022] shows that in single-round FFE, for any algorithm that achieves an glyph[lscript] ∞ -error smaller than τ := O ( √ log( d ) log(1 /δ ) / ( nglyph[epsilon1] )) , in the worse case, each client must communicate Ω( n · min { √ log( d ) / log(1 /δ ) , log( d ) } ) bits (see Their Corollary 5.1). In comparison, According to Theorem 4.1 and Corollary 2.3, the differentially private CountSketch can achieve an glyph[lscript] ∞ -error smaller than τ with length L glyph[equalorsimilar] log( d ) and width

resulting in a per-client communication of O ( WL log( n )) bits, which matches the minimax lower bound in Chen et al. [2022] ignoring a log( n ) factor, but could be much smaller in non-worst cases where ( f i ) d i =1 decays fast.

5 Concluding Remarks

We make several novel extensions to the count sketch method for federated frequency estimation with one or more communication rounds. In the single round setting, we show that count sketch can achieve better communication efficiency when the underlying problem is simpler. We provide a two-phase approach to automatically select a sketch size that adapts to the hardness of the problem. In the multiple rounds setting, we show a new sketching method that provably achieves better accuracy than simple adaptions of count sketch. Finally, we adapt the Gaussian mechanism to make the hybrid sketching method differentially private.

We remark that the improvement of the instance-dependent method relies on the assumptions that the underlying frequency has a lighter tail, which might be unverifiable a priori due to constraints, e.g., limited communication and privacy budget. Finally, this work focuses on an offline setting where the frequency is considered to be fixed. Extending our results to an online setting where the frequency is varying is an interesting future direction.

Acknolwdgement

We thank the anonymous reviewers for their helpful comments. We thank Brendan McMahan for insightful discussions during the project. VB has been partially supported by National Science Foundation Awards 2244899 and 2333887 and the ONR award N000142312737.

References

Kallista Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for federated learning on user-held data. arXiv preprint arXiv:1611.04482 , 2016.

Kallista Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloé Kiddon, Jakub Koneˇ cný, Stefano Mazzocchi, Brendan McMahan, Timon Van Overveldt, David Petrou, Daniel Ramage, and Jason Roselander. Towards federated learning at scale: System design. In MLSys . mlsys.org, 2019.

Kallista Bonawitz, Peter Kairouz, Brendan McMahan, and Daniel Ramage. Federated learning and privacy: Building privacy-preserving systems for machine learning and data science on decentralized data. Queue , 19(5):87-114, 2021.

Samuel R. Bowman, Gabriel Angeli, Siddharth Jain, Jared Kaplan, Prafulla Dhariwal, Saurabh Neelakantan, Jonathon Shlens, and Dario Amodei. C4: Colossal clean crawled corpus. arXiv preprint arXiv:2005.14165 , 2020.

Volkan Cevher. Learning with compressible priors. Advances in Neural Information Processing Systems , 22, 2009.

Moses Charikar, Kevin Chen, and Martin Farach-Colton. Finding frequent items in data streams. In International Colloquium on Automata, Languages, and Programming , pages 693-703. Springer, 2002.

Wei-Ning Chen, Christopher A Choquette-Choo, and Peter Kairouz. Communication efficient federated learning with secure aggregation and differential privacy. In NeurIPS 2021 Workshop Privacy in Machine Learning , 2021.

Wei-Ning Chen, Ayfer Özgür, Graham Cormode, and Akash Bharadwaj. The communication cost of security and privacy in federated frequency estimation. arXiv preprint arXiv:2211.10041 , 2022.

Eunjoon Cho, Seth A Myers, and Jure Leskovec. Friendship and mobility: user movement in locationbased social networks. In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 1082-1090, 2011.

Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3 , pages 265-284. Springer, 2006.

Alec Go, Richa Bhayani, and Lei Huang. Twitter sentiment classification using distant supervision. CS224N Project Report, Stanford , 1(12), 2009.

Ziyue Huang, Yuan Qiu, Ke Yi, and Graham Cormode. Frequency estimation under multiparty differential privacy: One-shot and streaming. arXiv preprint arXiv:2104.01808 , 2021.

Gregory T Minton and Eric Price. Improved concentration bounds for count-sketch. In Proceedings of the twenty-fifth annual ACM-SIAM symposium on Discrete algorithms , pages 669-686. SIAM, 2014.

Rasmus Pagh and Mikkel Thorup. Improved utility analysis of private countsketch. arXiv preprint arXiv:2205.08397 , 2022.

David M. W. Powers. Applications and explanations of Zipf's law. In New Methods in Language Processing and Computational Natural Language Learning , 1998. URL https://aclanthology. org/W98-1218 .

Daniek Ramage and Stefano Mazzocchi. Federated analytics: Collaborative data science without data collection. https://ai.googleblog.com/2020/05/ federated-analytics-collaborative-data.html , 2020.

Fuheng Zhao, Dan Qiao, Rachel Redberg, Divyakant Agrawal, Amr El Abbadi, and Yu-Xiang Wang. Differentially private linear sketches: Efficient implementations and applications. arXiv preprint arXiv:2205.09873 , 2022.

Wennan Zhu, Peter Kairouz, Brendan McMahan, Haicheng Sun, and Wei Li. Federated heavy hitters discovery with differential privacy. In International Conference on Artificial Intelligence and Statistics , pages 3837-3847. PMLR, 2020.

A Missing Proofs for Section 2

A.1 Proof of Proposition 2.1

Proof of Proposition 2.1. We refer the reader to Theorem 4.1 in Minton and Price [2014].

A.2 Proof of Corollary 2.2

Proof of Corollary 2.2. From Proposition 1 we know that

By union bound we have

Replacing δ with δ/d , setting L = log( d/δ ) , and using the definition of glyph[lscript] ∞ -norm, we obtain

We next show that:

To this end, we first show that f ∗ W ≤ 1 W . If not, we must have for i = 1 , . . . , W , f ∗ i ≥ f ∗ W > 1 W , as ( f ∗ i ) d i =1 is sorted in non-increasing order. Then ∑ d i f ∗ i ≥ ∑ W i =1 f ∗ i > 1 , which contradicts to the fact that ( f ∗ i ) d i =1 is a frequency vector. We have shown that f ∗ W ≤ 1 W , and this further implies that for any i ≥ W , f ∗ i ≤ f ∗ W ≤ 1 W . Then we can obtain

since ( f ∗ ) d i i =1 is a frequency vector. We have completed all the proof.

A.3 Proof of Corollary 2.3

Proof of Corollary 2.3. Define

We will show the following:

If W ≥ # { f i ≥ τ } + 1 τ 2 ∑ f i <τ f 2 i , then E ( W ) ≤ τ .

Moreover, if E ( W ) ≤ τ , then W ≥ 1 2 ( # { f i ≥ τ } + 1 τ 2 ∑ f i <τ f 2 i ) .

Then Corollary 2.3 follows by combining Corollary 2.2 with the above claims.

We first show the first part. First note that W ≥ # { f i ≥ τ } and that ( f ∗ i ) d i =1 is sorted in nonincreasing order, so for all i ≥ W it holds that f ∗ i < τ . Therefore,

Moreover, note that W ≥ 1 τ 2 ∑ f i <τ f 2 i , so we further have E ( W ) ≤ τ .

To show that second part, we first note that, by definition, E ( W ) ≤ τ is equivalent to

Consider the following function

one can directly verify that F ( k ) is minimized at k ∗ := # { i : f i ≥ τ } ; moreover,

Therefore, we have

This completes our proof.

B Missing Proofs for Section 3

B.1 Proof of Theorem 3.1

Proof of Theorem 3.1. The proof is motivated by Huang et al. [2021]. Define the following events

Then by Proposition 2.1 we have

Then by union bound, we have where

Conditional on the event of ⋂ M m =1 E ( m ) j , we know that every random variable ˆ f ( m ) j -f ( m ) j is bounded within

So by Hoeffding inequality, we have

Then we have

Note that

So we have

Note replace p 1 = p ′ / 2 and p = p ′ / (2 M ) , we have that

B.2 Proof of Theorem 3.2

Proof of Theorem 3.2. Let us consider the hybrid sketch approach in Algorithm 2. Recall that within a round, clients use the same set of hash functions to construct their sketching matrices. Across different rounds, clients use the same set of location hashes but a fresh set of sign hashes. Denote the hash functions by:

Recall the local frequency in each round is defined by

And the global frequency vector is defined by

Then according to the communication protocol, the server receives M sketching matrices (each corresponds to a summation of clients' sketches within the same round). From the m -th sketch, we can extract L estimators for each index j ∈ [ d ] , i.e., glyph[negationslash]

For each index, we will first average the estimators from different rounds to reduce the variance, then take the median over different rows to amplify the success probability. In particular, denote the round-wise averaging by glyph[negationslash]

These imply that we then have

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

Then we take the median over these estimators to obtain

Head Noise. The only randomness comes from the algorithm. Note that the head noise contains at most | W | ≤ 0 . 1 W independent terms, and each is zero with probability 1 -1 /W . Thus the head noise is zero with probability at least (1 -1 /W ) | W | ≥ (1 -1 /W ) 0 . 1 W ≥ 0 . 9 provided that W > 10 .

Tail Noise. Now consider the second noise term in (5). Fixing glyph[lscript] and j . Define glyph[negationslash]

First notice that ( ξ ( m ) i ) M m =1 are independent random variables and

Moreover, notice that ( η i , ξ i ) i = j are independent random variables, and

Therefore we conclude that

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

Then by Chebyshev we see that: for fixed j ∈ [ d ] and glyph[lscript] ∈ [ L ] it holds that

By a union bound we see that: for fixed j ∈ [ d ] and glyph[lscript] ∈ [ L ] it holds that

Probability Amplification. Fixing j . Recall that ( ˜ f ( glyph[lscript] ) j ) L glyph[lscript] =1 are i.i.d. random variables and that ˜ f j := median { ˜ f ( glyph[lscript] ) j : glyph[lscript] ∈ [ L ] } . By Chernoff over glyph[lscript] and union bound over j we see that:

By choosing L = Θ(log(2 d/δ )) we obtain that, with probability at least 1 -δ ,

C Missing Proofs for Section 4

C.1 Proof of Theorem 4.1

Proof of Theorem 4.1. We follow the method of Pagh and Thorup [2022], Zhao et al. [2022] to add DP noise to all M sketches. Suppose F = ( f ( m ) ) M m =1 and ˚ F = ( ˚ f ( m ) ) M m =1 are the sets of local frequencies for two neighboring datasets respectively, then

Denote the sketches to be released by S ◦ F := ( S ( m ) ◦ f ( m ) ) M m =1 . One can then calculate the glyph[lscript] 2 -sensitivity:

where L glyph[equalorsimilar] log( d/δ ) is the sketch length. Therefore the sketching will be ( glyph[epsilon1], δ ) -DP by adding Gaussian noise N (0 , σ 2 ) to each bucket of each sketch, where

The final released frequency estimator is obtained by post-processing the sketch, so it is also ( glyph[epsilon1], δ ) -DP.

We then calculate the error for the noisy sketch matrix. For each row estimator, we have that with probability at least 2 / 3 :

By taking median over L glyph[equalorsimilar] log( d/δ ) repeats, we see that with probability at least 1 -δ , it holds that

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn the nascent area of 'federated analytics', the canonical problem is frequency estimation: each client holds a label (or a collection of labels), and the aim is to recover the frequency distribution of these labels, or at least information on the most frequent labels (heavy hitters).  Prior work has demonstrated that this problem can be tackled by making use of 'sketch' data structures designed over the last few years.  The question is now to understand how to optimize the accuracy-communication tradeoff, while meeting various security and privacy guarantees.  This paper builds on prior work in this direction, by arguing that better bounds can be achieved due to the skewed nature of the input data: essentially, the worst case for sketches is when the data is uniform, and better accuracy is seen when most mass is in the head of the label frequency distribution.  Results for this case follow in part due to prior analysis of such data structures, adapted to the parameters of the federated setting. The second results apply to the case when data is collected in multiple rounds, by comparing the effect of using the same randomly chosen sketch parameters versus varying some.  The results show that this achieves improved bounds in the case that the frequency distribution changes between rounds.  Experiments based on simulated data allocations quantify this further.\n\nSTRENGTHS:\nThis is a foundational problem for the area of federated computing, and the paper shows some improved results which could be of use to practitioners. \n\nThe algorithms proposed are clear and suitable to be implemented.  It is straightforward to achieve differential privacy by noise addition to the sketch (under secure aggregation), with bounded impact on accuracy.\n\nWEAKNESSES:\nThe novel contribution is not extremely high.  The first set of results are shown by plugging parameters into theorems from prior work, and some manipulation of probabilities. The second set of results are more involved, but can still be viewed as adaptations of prior proofs.  From a technical perspective, there is not much excitement.  The potential practical implications could elevate this work, but this would require more effort to demonstrate that the real world scenarios where this is needed map on to the assumptions in this work. \n\nThe results separate best from the single sketch approach when the data is 'heterogenous'.  The paper could do more to define the exact model of heterogeneity assumed, and to test these on real data.  The main opportunity for hybrid sketch to shine (P8) is when the frequency vectors are heterogenous -- that is, when the frequency distribution is different during each round of data collection.  (Note that most commonly, in the federated setting, heterogenous is used to refer to the case when the allocating of data to each client can vary a lot.  However, the definition of the frequency estimation problem is oblivious to the allocation of labels to clients: it just asks for the global frequency distribution). It would be helpful to know what kind of scenarios expect to have different frequency distributions per round, and what are the implications of this for the frequency estimation problem -- when the data is constantly shifting, what should the ground truth frequency distribution be?\n\nQUESTIONS:\nAre there any good data sets or references that can help to argue that the heterogeneity vectors F are substantially smaller (in Euclidean norm) than the frequency vectors?\n\nLIMITATIONS:\nNo limitations in the send of negative societal impact\nCould say more about technical limitations of the work, or future work -- the conclusion just summarizes the results.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-17"}

comments: SUMMARY:
The paper considers the problem of federated frequency estimation under the Secure Summation constraint. Motivated by the tail-bound analysis of CountSketch, the authors propose a two-round communication approach that can significantly reduce the sketch size and improve the overall communication size. Later, the authors extend the algorithm to a multi-round HybridSketch algorithm and improve the protocol's scalability. This paper also carefully analyzed their proposed method's trade-offs between accuracy and privacy. The proposed algorithm shows performance improvements in real-world datasets (Gowalla and Colossal Clean Crawled Corpus).

STRENGTHS:
Federated frequency estimation is a very important task in federated analytic and has many applications. The main idea is intuitive. Many real-world data distributions exhibit skewness, while the chen et al. showcases the lower bound for this task is $nlogd$ bits of communication per client, this work takes the distribution into account to reduce the communication costs.

WEAKNESSES:
The experiments showcase good estimation accuracy improvement. One may wonder how the communication latency changes between the baselines and hybrid sketch.

QUESTIONS:
What is the difference in communication latency between the proposed algorithm vs. baselines?

Use different ticks or line styles to better distinguish between the lines in figures.

LIMITATIONS:
No limitations
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of Federated Frequency Estimation (FFE) by leveraging the Federated CountSketch algorithm (Algorithm 2 in this paper). Compared to previous FFE methods the authors argue that if the tail of the underlying frequencies is light or small (\\emph{i.e., } the smaller frequencies have considerably smaller values). Furthermore, such a lighter tail provides two benefits, (1) The FFE error decreases and (2) A smaller width can be afforded in the Federated CountSketch algorithm that leads to a direct reduction in the client communication costs. Under a Federated setting, both advantages are meaningful. In the first half of the paper, the authors provide ways to leverage this lighter tail in the Federated setting. While in the second half, the authors provide FFE algorithms that work for a large number of clients by running CountSketch over multiple groups of clients. Finally, the paper also provides a Differential Privacy (DP) guarantee and estimation error while running the FFE algorithm over multiple rounds.\n\nOverall, the authors provide a clear and unique analysis that furthers the FFE literature by reduction of the estimation error and sketch width when leveraging the (almost optimal) CountSketch algorithm. The empirical evaluations provide a complete picture of the impact of the suggested method. Minor comments regarding the two-phase method, empirical evaluations, and DP section have been provided below.\n\nSTRENGTHS:\nThe paper tackles several challenges in the FFE literature. Primarily the authors demonstrate FFE algorithms for the cases when (1) the number of clients is low to moderate and (2) for a high number of clients. They also tackle the case when Differential Privacy (DP) is introduced in FFE deployments.\n\n1. The core idea that the authors leverage is the reduction in the estimation error while using the CountSketch algorithm when the underlying frequencies have a lighter tail. The authors argue that when the data is distributed in such a (lighter-tail) fashion, the previous estimation error bounds are unable to leverage these tighter bounds to provide real-world benefits. Using Corollaries 2.2-2.4 the authors demonstrate a reduction in the estimation error and the sketch width and thus the communication cost of the clients.\n2. Furthermore, the paper suggests that when the clients are high in number, simply computing CountSketch over a single round might not be feasible. They demonstrate that the naive application of CountSketch in the multi-round setup is not enough. Thus, they suggest a new algorithm “Hybrid Sketch, \" extending CountSketch for the multi-round setup. The Hybrid Sketch algorithm by maintaining the same hash buckets can combine the analysis of frequency estimation across rounds rather than analyze each separately. \n3. Finally, a DP alternative of Hybrid Sketch is provided that follows similarly to previous approaches.\n\nWEAKNESSES:\n1. For the two-phase method in section 2, the authors suggest that frequencies follow the Zipf law; thus, they estimate the polynomial coefficients by a kind of pilot study on a small number of clients. It is however unclear how the instance optimal method can leverage Eqn. (2) with the unknown frequencies to estimate the sketch width. Is the two-phase method employed for the instance optimal method as well? Please consider adding relevant clarification for all three methods.\n2. For the DP analysis consider mentioning the exact mechanism used to add noise. At first glance, it might not be clear if the method employs local DP or central DP.\n3. In certain cases, the idea that the frequencies have a lighter tail might not be verifiable (before running the sketching algorithm). Such an instance may occur if the communication or DP budget is constrained or if there are not enough clients to handle both the pilot and the actual sketching algorithm under DP constraints. Thus, adding this limitation clearly toward the end of the article will add clarity.\n\nQUESTIONS:\nThe paper is well-written and conveys interesting results for performing Federated Frequency estimation over various scenarios. I do not have major concerns regarding the paper. A few minor suggestions have been pointed out in the weaknesses section.\n\nLIMITATIONS:\nThe primary limitation of the proposed method has been provided in point 3 of the weaknesses section. The authors can consider including a limitations/discussion section that highlights such challenges.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores several variants of the count sketch method for federated frequency estimation (FFE).\n- With only one communication, they provide a refined instance-dependent analysis for CountSketch and find that the sketch size depends on unknown problem-dependent quantities.\nThey then propose a two-phase approach to first learn these quantities first and then perform FFE.\n- They also consider the case where multiple communication is available, they explore several variants of count sketch methods to utilize the multiple communication rounds, provide theoretical analysis, and conduct numerical experiments to verify their findings.\n- They also explore a differential private extension.\n\n======= after rebuttal =======\n\nI have read the author's rebuttal which addresses most of my concerns.\n\nThe remaining issue from my perspective is the lack of an instance-dependent lower bound which is left as a future work, as claimed in the rebuttal.\n\nHence, I tend to increase the point.\n\nSTRENGTHS:\nThey conduct a systematic investigation of FFE problems and provide theoretical analysis.\n\nThe paper is well-written and easy to follow.\n\nWEAKNESSES:\nThe paper tries to study a lot of stuff, which, however, left many questions untouched. See the Questions part.\n\nThere seems not much novelty in the theoretical analysis. \n\nThe benefit of instance dependence on the estimation guarantee is not well illustrated.\n\nThough the author provides a lot of numerical experiments to validate their theoretical predictions, it is still unclear how the proposed method would affect practical training where (overparameterized) neural networks are used.\n\nQUESTIONS:\n1. From Figure 1 (a)(d), I find that given a target $\\ell_{\\infty}$ error, the minimax optimal curve often achieves a smaller $\\ell_{\\infty}$ error than the instance optimal one, though the latter aligns with the straight line $y=x$ better. It seems that the minimax optimal one is better than the instance optimal one in the achieved accuracy, which is quite counterintuitive. Is there anything I missed? Could the author provide some explanation?\n2. Could the author provide some intuitive explanation about why Hybrid Sketch could be better than Fresh Sketch? Note that Hybrid Sketch shares a set of bucket hashes but uses independent sets of sign hashes. Why this dependence on shared bucket hashes is better than independent bucket hashes? What about independent bucket hashes and shared sign hashes?\n3. Is it possible to provide any lower bound to show the instance-dependent upper bounds are tight?\n4. Line 316, when $n$ is fixed, why increasing $M$ improves the estimation error? In Theorem 4.1, the estimation error $\\sqrt{\\frac{\\sum_{i>W}\\left(F_i^*\\right)^2}{W}}$ seems to have nothing to do with $M$ in the worst case.\n5. Is it possible to extend the methods developed in this paper to empirical risk minimization where gradients of nonconvex models (such as neural networks) are used?\n\nLIMITATIONS:\nN/A.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}