SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(b)  $f_{i}\propto i^{-3}$
Figure 2: Shared vs. Hybrid vs. Fresh Sketches. We refer the reader to Section 3 for the definitions of the three methods. We compute the expected  $\ell_{\infty}$ -error for shared/hybrid/fresh sketches for a homogeneous, multi-round FFE problem. The domain size is  $d = 10^{5}$ . The number of rounds is  $M = 10$ . In all setups, the sketch length is fixed to  $L = 5$ . In every setting, the  $\ell_{\infty}$  error is averaged with 1,000 random repeats for simulating the expectation. In the case when the global frequency vector is a low-degree polynomial, hybrid sketch performs similarly to fresh sketch, and both are better than shared sketch. As long as the global frequency vector is a slightly higher degree polynomial (e.g., with a degree higher than 3), then hybrid sketch is significantly better than both shared and fresh sketches.
(c)  $f_{i}\propto i^{-5}$
$\mathbf{x}_t^{(m)} = \mathbf{e}_{x_t^{(m)}}$  where  $(\mathbf{e}_t)_{t=1}^d$  refers to the canonical basis. Then the frequency vector can be denoted by  $\mathbf{f} := (f_1, \ldots, f_d)^\top$ . The aim is to estimate the frequency vector  $\mathbf{f}$  in a manner that is compatible with  $\operatorname{SecSum}$ .
Baseline Method 1: Shared Sketch. A multi-round FFE problem can be reduced to a single-round FFE problem with a large communication. Specifically, one can apply the CountSketch with the same randomness for every round; after collecting all the sketches from the  $M$  round, one simply averages them. Due to the linearity of the sketching compress method, this is equivalent to a single round setting with  $N = Mn$  clients. We refer to this method as count sketch with shared hash design (SharedSketch).
Thanks to the reduction idea, we can obtain the error and sketch size bounds for SharedSketch via applying Corollaries 2.2 and 2.3 to SharedSketch by replacing  $n$  by  $N = Mn$ ,
Baseline Method 2: Fresh Sketch. A multi-round FFE problem can also be broken down to  $M$  independent single-round FFE problems. Specifically, one can apply independent CountSketch in each round, and decode  $M$  local estimators for the  $M$  local frequency vectors. As the CountSketch produces an unbiased estimator, one can show that the average of the  $M$  local estimators is an unbiased estimator for the global frequency vector. We call this method count sketch with fresh hash design (FreshSketch). We provide the following bound for FreshSketch. The proof of which is motivated by Huang et al. [2021].
Theorem 3.1 (Instance-specific bound for FreshSketch). Let  $(\hat{f}_j)_{j=1}^d$  be estimates produced by FreshSketch. Then for each  $p \in (0,1)$ ,  $W \geq 1$  and  $L \geq \log(1/p)$ , it holds that: for each  $j \in [d]$ , with probability at least  $1 - p$ ,
$$
| \hat {f} _ {j} - f _ {j} | <   C \cdot \sqrt {\frac {\log (1 / p) \log (M / p)}{L} \cdot \frac {1}{W} \cdot \sum_ {i > W} (F _ {i} ^ {*}) ^ {2}},
$$
where  $C$  is an absolute constant, and  $(F_{i}^{*})_{i = 1}^{d}$  are defined as in Theorem 3.2.
Hybrid Sketch. Both SharedSketch and FreshSketch reduce a multi-round FFE problem into single-round FFE problem(s). In contrast, we show a more comprehensive sketching method, called count sketch with hybrid hash design (HybridSketch), that solves a multi-round FFE problem as a whole. HybridSketch is presented as Algorithm 2. Specifically, HybridSketch generates  $M$  sketches that share a set of bucket hashes but use independent sets of sign hashes. Then in the  $m$ -th communication round, participating clients and the server communicate by the CountSketch algorithm based on the  $m$ -th sketch, so the server observes the summation of the sketched data through SecSum. After collecting  $M$  summations of the sketched local data, the server first computes averages over different rounds for variance reduction, then computes the median over different repeats
Algorithm 2 HYBRID SKETCH FOR FEDERATED FREQUENCY ESTIMATION
Require: The number of rounds  $M$ .  $N = Mn$  clients with local data  $x_{t}^{(m)} \in [d]$  for  $m \in [M]$  and  $t \in [n]$ . Sketch length  $L$  and width  $W$ .
$$
h _ {\ell}: [ d ] \to [ W ], \sigma_ {\ell} ^ {(m)}: [ d ] \to \{\pm 1 \} \text {f o r} \ell \in [ L ], m \in [ M ].
$$
2: for Round  $m = 1,\dots ,M$  in parallel do
3: for Client  $t = 1, \dots, n$  in parallel do
4: Client  $(m,t)$  encodes the local data  $x_{t}^{(m)}$  to  $\mathsf{enc}^{(m)}\big(x_t^{(m)}\big)\in \mathbb{R}^{L\times W}$  where
$$
\left(\mathsf {e n c} ^ {(m)} \big (x _ {t} ^ {(m)} \big)\right) _ {\ell , k} = \mathbb {1} \left[ h _ {\ell} \left(x _ {t} ^ {(m)}\right) = k \right] \cdot \sigma_ {\ell} ^ {(m)} \left(x _ {t} ^ {(m)}\right) \text {f o r} \ell \in [ L ], k \in [ W ].
$$
5: Client  $(m, t)$  sends  $\mathsf{enc}^{(m)}\left(x_{t}^{(m)}\right)$  to SecSum.
6: end for
7: SecSum receives  $\left(\mathsf{enc}^{(m)}(x_t^{(m)})\right)_{t = 1}^n$  and reveals the sum  $\sum_{t = 1}^{n}\mathsf{enc}^{(m)}\big(x_t^{(m)}\big)$  to the server.
8: end for
9: for Item  $j = 1, \ldots, d$  in parallel do
10: Server produces  $M \times L$  estimators for  $f_{j}$ :
$$
\mathbf {d e c} (j; m, l) := \sigma_ {\ell} ^ {(m)} (j) \cdot \left(\frac {1}{n} \sum_ {t = 1} ^ {n} \mathbf {e n c} ^ {(m)} \left(x _ {t} ^ {(m)}\right)\right) _ {\ell , h _ {\ell} (j)} \text {f o r} m \in [ M ], \ell \in [ L ].
$$
11: Server computes the median over  $\ell \in [L]$  of the averages over  $m\in [M]$  of the estimators:
$$
\operatorname {d e c} (j) := \operatorname {m e d i a n} \left\{\frac {1}{M} \sum_ {m = 1} ^ {M} \operatorname {d e c} (j; m, l), \ell \in [ L ] \right\}.
$$
12: end for
13: return  $\left(\operatorname{dec}(j)\right)_{j=1}^{d}$  as estimate to  $(f_j)_{j=1}^d$ .
(or sketch rows) for success probability amplification. We provide the following problem-dependent bound for HybridSketch.
Theorem 3.2 (Instance-specific bound for HybridSketch). Let  $(\hat{f}_j)_{j=1}^d$  be estimates produced by HybridSketch (see Algorithm 2). Define a heterogeneity vector  $(F_i)_{i=1}^d$  by
$$
F _ {i} := \frac {1}{M} \sqrt {\sum_ {m = 1} ^ {M} \left(f _ {i} ^ {(m)}\right) ^ {2}}, \quad i = 1, \dots , d.
$$
Clearly, it holds that  $F_{i} \leq f_{i}$  for every  $i \in [d]$ . Let  $(F_{i}^{*})_{i \geq 1}$  be  $(F_{i})_{i \geq 1}$  sorted in non-increasing order. Then for each  $p \in (0,1)$ ,  $W \geq 1$  and  $L \geq \log(1/p)$ , it holds that: for each  $j \in [d]$ , with probability at least  $1 - p$ ,
$$
| \hat {f} _ {j} - f _ {j} | <   C \cdot \sqrt {\frac {\log (1 / p)}{L} \cdot \frac {1}{W} \cdot \sum_ {i > W} \left(F _ {i} ^ {*}\right) ^ {2}},
$$
where  $C$  is an absolute constant.
We would like to point out that, although our HybridSketch algorithm is developed for multi-round frequency estimation problems, it can be adapted to multi-round vector recovery problems as well. Hence it could have broader applications in other federated learning scenarios.
Hybrid Sketch vs. Fresh Sketch. By comparing Theorem 3.2 with Theorem 3.1, we see that, with the same sketch size, the estimation error of HybridSketch is smaller than that of FreshSketch by a factor of  $\sqrt{\log(M / p)}$ . This provides theoretical insights that HybridSketch is superior to FreshSketch in terms of adapting to the instance hardness in multi-round FFE settings. This is also verified empirically by Figure 2.
Hybrid Sketch vs. Shared Sketch. We now compare the performance of HybridSketch and SharedSketch by comparing Theorem 3.2 and Proposition 2.1 (under a revision of replacing  $n$  with  $N = Mn$ ). Note that
$$
F _ {i} = \frac {1}{M} \sqrt {\sum_ {m = 1} ^ {M} \left(f _ {i} ^ {(m)}\right) ^ {2}} \leq \frac {1}{M} \sum_ {m = 1} ^ {M} f _ {i} ^ {(m)} = f _ {i}.
$$
So with the same sketch size, HybridSketch achieves an error that is no worse than csc in every case. Moreover, in the homogeneous case where all local frequency vectors are equivalent to the global frequency vector, i.e.,  $\mathbf{f}^{(m)}\equiv \mathbf{f}$  for all  $m$  then it holds that  $F_{i} = f_{i} / \sqrt{M}$ . So in the homogeneous case, HybridSketch achieves an error that is smaller than that of csc by a factor of  $1 / \sqrt{M}$ . In the general cases, the local frequency vectors are not perfectly homogeneous, then the improvement of HybridSketch over
Figure 3: The number of items with error greater than 0.1/width for Shared, Hybrid, and Fresh Sketches with C4 dataset. HybridSketch with a width of 200 achieves roughly the same error as SharedSketch with a width of 1200 and Fresh sketch with a width of 600.
SharedSketch will depend on the heterogeneity of these local frequency vectors.
Experiments. We conduct three sets of experiments to verify our understandings about these sketches methods for multi-round FFE.
In the first sets of experiments, we simulate a multi-round FFE problem in homogeneous settings, where in every round the local frequency vectors are exactly the same. More specially, we set a domain size  $d = 10^5$ , a number of rounds  $M = 10$  and test three different cases, where all the local frequency vectors are the same and (hence also the global frequency vector) are proportional to  $(i^{-1.1})_{i=1}^{d}$ ,  $(i^{-2})_{i=1}^{d}$  and  $(i^{-5})_{i=1}^{d}$ , respectively. In all the settings, we fix the sketch length to  $L = 5$ . In each experiment, we measure the expected  $\ell_{\infty}$ -error of each method with the averaging over 1,000 independent repeats. The results are plotted in Figure 2. We can observe that: for low-degree polynomials, HybridSketch is nearly as good as FreshSketch and both are better than SharedSketch. But for slightly high degree polynomials (with a degree of 3), HybridSketch already outperforms both FreshSketch and SharedSketch. The numerical results are consistent with our theoretical analysis.
In the second sets of experiments, we simulate a multi-round FFE problem with the Gowalla dataset [Cho et al., 2011]. Similar to previously, we construct a domain of size  $d = 175,000$ , which corresponds to a grid over the US map. Then we sample  $N = d = 175,000$  lists of the location information (that all belong to the domain created) to represent the data of  $N$  clients, uniformly at random. We set the number of rounds to be  $M = 10$ . In each round,  $n = N / M = 17,500$  clients participate. The results are presented in Figure 1(c). Here, the frequency and heterogeneity vectors have heavy tails, so HybridSketch and FreshSketch perform similarly and both are better than SharedSketch. This is consistent with our theoretical understanding.
In the third sets of experiments, we run simulations on the C4 [Bowman et al., 2020] dataset. Similar to the single round simulation, the domain size  $d = 150,868$ . We randomly sample  $N = 150,000$  users from the dataset. The number of rounds  $M = 10$ , and in each round,  $n = N / 10 = 15,000$  clients participate. The results are provided in Figures 1(f) and 3. Here, the frequency and heterogeneity vectors have moderately light tails, and Figure 3 already suggests that HybridSketch produces an estimate that has a better shape than that produced by FreshSketch and SharedSketch, verifying the advantages of HybridSketch.
4 Differentially Private Sketches
While SecSum provides security guarantees, it does not provide differential privacy guarantees. In this part, we discuss a simple modifications to the sketching algorithms to make them provably differentially private (DP).
Definition 2  $((\epsilon, \delta)$ -DP [Dwork et al., 2006]). Let  $\mathrm{alg}(\cdot)$  be a randomized algorithm that takes a dataset  $\mathcal{D}$  as its input. Let  $\mathbb{P}$  be its probability measure.  $\mathrm{alg}(\cdot)$  is  $(\epsilon, \delta)$ -DP if: for every pair of neighboring datasets  $\mathcal{D}$  and  $\mathcal{D}'$ , it holds that
$$
\mathbb {P} \{\operatorname {a l g} (\mathcal {D}) \in \mathcal {E} \} <   e ^ {\epsilon} \cdot \mathbb {P} \{\operatorname {a l g} \left(\mathcal {D} ^ {\prime}\right) \in \mathcal {E} \} + \delta .
$$
In our case, a dataset corresponds to all participated clients (or their data), and two neighboring datasets should be regarded as two sets of clients (local data) that only differ in a single client (local data). The algorithm refers to all procedures before releasing the final frequency estimate, and all the intermediate computation is considered private and is not released.
We work with central  $DP$ , that is, server releases data in a differentially private way while clients do not release data. We focus on HybridSketch as a representative algorithm. The DP mechanism can also be extended to the other sketching algorithms. Specifically, we use a DP mechanism that adds independent Gaussian noise to each entry of the sketching matrix, which is initially proposed for making CountSketch differentially private by Pagh and Thorup [2022], Zhao et al. [2022].
We provide the following theorem characterizing the trade-off between privacy and accuracy.
Theorem 4.1 (DP-hybrid sketch). Consider a modified Algorithm 2, where we add to each entry of the sketching matrix an independent Gaussian noise,  $\mathcal{N}(0,c_0\cdot \sqrt{L\log(1 / \delta)} /\epsilon)$ , where  $c_{0} > 0$  is a known constant. Suppose that  $L = \log (d / p)$  and  $W\geq 2$ . Then the final output of the modified Algorithm 2, denoted by  $(\hat{f}_j)^{\underline{d}}_{j = 1}$ , is  $(\epsilon ,\delta)$ -DP for  $\epsilon < 1$  and  $\delta < 0.1$ . Moreover, with probability at most  $1 - p$ , it holds that
$$
\max _ {j} | \hat {f} _ {j} - f _ {j} | <   C \cdot \left(\sqrt {\frac {\sum_ {i > W} (F _ {i} ^ {*}) ^ {2}}{W}} + \frac {\sqrt {\log (d / p) \log (1 / \delta)}}{n \sqrt {M} \epsilon}\right),
$$
where  $C > 0$  is an absolute constant and  $(F_i^*)_{i=1}^d$  are as defined in Theorem 3.2.
It is worth noting that if the number of clients per round  $(n)$  is fixed, then a larger number of rounds  $M$  improves both the estimation error and the DP error in non-worst cases, e.g., when the local frequency vectors are nearly homogenous. However, if the total number of clients  $(N = Mn)$  is fixed, then a larger number of rounds  $M$  improves the estimation error but makes the DP error worse.
When  $M = 1$ , Theorem 4.1 recovers the bounds for differentially private CountSketch in Pagh and Thorup [2022], Zhao et al. [2022] and Theorem 5.1 in Chen et al. [2022]. Moreover, Chen et al. [2022] shows that in single-round FFE, for any algorithm that achieves an  $\ell_{\infty}$ -error smaller than  $\tau := \mathcal{O}(\sqrt{\log(d)\log(1 / \delta)} /(n\epsilon))$ , in the worse case, each client must communicate  $\Omega (n\cdot \min \{\sqrt{\log(d) / \log(1 / \delta)},\log (d)\})$  bits (see Their Corollary 5.1). In comparison, According to Theorem 4.1 and Corollary 2.3, the differentially private CountSketch can achieve an  $\ell_{\infty}$ -error smaller than  $\tau$  with length  $L\approx \log (d)$  and width
$$
W = C \cdot \min  \left\{\left(\# \left\{f _ {i}: f _ {i} \geq \tau \right\} + \frac {1}{\tau^ {2}} \cdot \sum_ {f _ {i} <   \tau} f _ {i} ^ {2}\right), n \right\} \leq C \cdot \min  \{2 / \tau , n \},
$$
resulting in a per-client communication of  $\mathcal{O}(WL\log (n))$  bits, which matches the minimax lower bound in Chen et al. [2022] ignoring a  $\log (n)$  factor, but could be much smaller in non-worst cases where  $(f_i)_{i = 1}^d$  decays fast.
We make several novel extensions to the count sketch method for federated frequency estimation with one or more communication rounds. In the single round setting, we show that count sketch can achieve better communication efficiency when the underlying problem is simpler. We provide a two-phase approach to automatically select a sketch size that adapts to the hardness of the problem. In the multiple rounds setting, we show a new sketching method that provably achieves better accuracy than simple adoptions of count sketch. Finally, we adapt the Gaussian mechanism to make the hybrid sketching method differentially private.
We remark that the improvement of the instance-dependent method relies on the assumptions that the underlying frequency has a lighter tail, which might be unverifiable a priori due to constraints, e.g., limited communication and privacy budget. Finally, this work focuses on an offline setting where the frequency is considered to be fixed. Extending our results to an online setting where the frequency is varying is an interesting future direction.
We thank the anonymous reviewers for their helpful comments. We thank Brendan McMahan for insightful discussions during the project. VB has been partially supported by National Science Foundation Awards 2244899 and 2333887 and the ONR award N000142312737.
Kallista Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for federated learning on user-held data. arXiv preprint arXiv:1611.04482, 2016.
Kallista Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloé Kiddon, Jakub Konečný, Stefano Mazzocchi, Brendan McMahan, Timon Van Overveldt, David Petrou, Daniel Ramage, and Jason Roselander. Towards federated learning at scale: System design. In MLSys. mlsys.org, 2019.
Kallista Bonawitz, Peter Kairouz, Brendan McMahan, and Daniel Ramage. Federated learning and privacy: Building privacy-preserving systems for machine learning and data science on decentralized data. Queue, 19(5):87-114, 2021.
Samuel R. Bowman, Gabriel Angeli, Siddharth Jain, Jared Kaplan, Prafulla Dhariwal, Saurabh Neelakantan, Jonathon Shlens, and Dario Amodei. C4: Colossal clean crawled corpus. arXiv preprint arXiv:2005.14165, 2020.
Volkan Cevher. Learning with compressible priors. Advances in Neural Information Processing Systems, 22, 2009.
Moses Charikar, Kevin Chen, and Martin Farach-Colton. Finding frequent items in data streams. In International Colloquium on Automata, Languages, and Programming, pages 693-703. Springer, 2002.
Wei-Ning Chen, Christopher A Choquette-Choo, and Peter Kairouz. Communication efficient federated learning with secure aggregation and differential privacy. In NeurIPS 2021 Workshop Privacy in Machine Learning, 2021.
Wei-Ning Chen, Ayfer Özgür, Graham Cormode, and Akash Bharadwaj. The communication cost of security and privacy in federated frequency estimation. arXiv preprint arXiv:2211.10041, 2022.
Eunjoon Cho, Seth A Myers, and Jure Leskovec. Friendship and mobility: user movement in location-based social networks. In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 1082-1090, 2011.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3, pages 265-284. Springer, 2006.
Alec Go, Richa Bhayani, and Lei Huang. Twitter sentiment classification using distant supervision. CS224N Project Report, Stanford, 1(12), 2009.
Ziyue Huang, Yuan Qiu, Ke Yi, and Graham Cormode. Frequency estimation under multiparty differential privacy: One-shot and streaming. arXiv preprint arXiv:2104.01808, 2021.
Gregory T Minton and Eric Price. Improved concentration bounds for count-sketch. In Proceedings of the twenty-fifth annual ACM-SIAM symposium on Discrete algorithms, pages 669-686. SIAM, 2014.
Rasmus Pagh and Mikkel Thorup. Improved utility analysis of private countsketch. arXiv preprint arXiv:2205.08397, 2022.
David M. W. Powers. Applications and explanations of Zipf's law. In New Methods in Language Processing and Computational Natural Language Learning, 1998. URL https://aclanthology.org/W98-1218.
Daniek Ramage and Stefano Mazzocchi. Federated analytics: Collaborative data science without data collection. https://ai.googleblog.com/2020/05/federated-analytics-collaborative-data.html, 2020.
Fuheng Zhao, Dan Qiao, Rachel Redberg, Divyakant Agrawal, Amr El Abbadi, and Yu-Xiang Wang. Differentially private linear sketches: Efficient implementations and applications. arXiv preprint arXiv:2205.09873, 2022.
Wennan Zhu, Peter Kairouz, Brendan McMahan, Haicheng Sun, and Wei Li. Federated heavy hitters discovery with differential privacy. In International Conference on Artificial Intelligence and Statistics, pages 3837-3847. PMLR, 2020.
A Missing Proofs for Section 2
A.1 Proof of Proposition 2.1
Proof of Proposition 2.1. We refer the reader to Theorem 4.1 in Minton and Price [2014].
A.2 Proof of Corollary 2.2
Proof of Corollary 2.2. From Proposition 1 we know that
$$
\text {f o r e v e r y} j \in [ d ], \quad \mathbb {P} \left\{\left| \operatorname {d e c} (j) - f _ {j} \right| > C \cdot \sqrt {\frac {\log (1 / \delta)}{L} \cdot \frac {1}{W} \cdot \sum_ {i > W} \left(f _ {i} ^ {*}\right) ^ {2}} \right\} <   \delta .
$$
By union bound we have
$$
\mathbb {P} \left\{\text {t h e r e e x i s t s} j \in [ d ], | \mathrm {d e c} (j) - f _ {j} | > C \cdot \sqrt {\frac {\log (1 / \delta)}{L} \cdot \frac {1}{W} \cdot \sum_ {i > W} \left(f _ {i} ^ {*}\right) ^ {2}} \right\} <   d \delta .
$$
Replacing  $\delta$  with  $\delta /d$ , setting  $L = \log (d / \delta)$ , and using the definition of  $\ell_{\infty}$ -norm, we obtain
$$
\mathbb {P} \left\{\| \operatorname {d e c} (\cdot) - \mathbf {f} \| _ {\infty} > C \cdot \sqrt {\frac {1}{W} \cdot \sum_ {i > W} \left(f _ {i} ^ {*}\right) ^ {2}} \right\} <   \delta .
$$
We next show that:
$$
\sqrt {\frac {1}{W} \cdot \sum_ {i > W} \left(f _ {i} ^ {*}\right) ^ {2}} \leq \frac {1}{W}.
$$
To this end, we first show that  $f_W^* \leq \frac{1}{W}$ . If not, we must have for  $i = 1, \dots, W$ ,  $f_i^* \geq f_W^* > \frac{1}{W}$  as  $(f_i^*)_{i=1}^d$  is sorted in non-increasing order. Then  $\sum_i^d f_i^* \geq \sum_{i=1}^W f_i^* > 1$ , which contradicts to the fact that  $(f_i^*)_{i=1}^d$  is a frequency vector. We have shown that  $f_W^* \leq \frac{1}{W}$ , and this further implies that for any  $i \geq W$ ,  $f_i^* \leq f_W^* \leq \frac{1}{W}$ . Then we can obtain
$$
\sqrt {\frac {1}{W} \cdot \sum_ {i > W} \left(f _ {i} ^ {*}\right) ^ {2}} \leq \sqrt {\frac {1}{W ^ {2}} \cdot \sum_ {i > W} f _ {i} ^ {*}} \leq \frac {1}{W},
$$
since  $(f_i^*)_{i=1}^d$  is a frequency vector. We have completed all the proof.
A.3 Proof of Corollary 2.3
Proof of Corollary 2.3. Define
$$
E (W) := \sqrt {\frac {1}{W} \sum_ {i > W} \left(f _ {i} ^ {*}\right) ^ {2}}.
$$
We will show the following:
1. If  $W \geq \# \{f_i \geq \tau\} + \frac{1}{\tau^2} \sum_{f_i < \tau} f_i^2$ , then  $E(W) \leq \tau$ .
2. Moreover, if  $E(W) \leq \tau$ , then  $W \geq \frac{1}{2}\big(\# \{f_i \geq \tau\} + \frac{1}{\tau^2} \sum_{f_i < \tau} f_i^2\big)$ .
Then Corollary 2.3 follows by combining Corollary 2.2 with the above claims.
We first show the first part. First note that  $W \geq \# \{f_i \geq \tau\}$  and that  $(f_i^*)_{i=1}^d$  is sorted in nonincreasing order, so for all  $i \geq W$  it holds that  $f_i^* < \tau$ . Therefore,
$$
E (W) := \sqrt {\frac {1}{W} \sum_ {i > W} \left(f _ {i} ^ {*}\right) ^ {2}} \leq \sqrt {\frac {1}{W} \sum_ {f _ {i} <   \tau} f _ {i} ^ {2}}.
$$
Moreover, note that  $W \geq \frac{1}{\tau^2} \sum_{f_i < \tau} f_i^2$ , so we further have  $E(W) \leq \tau$ .
To show that second part, we first note that, by definition,  $E(W) \leq \tau$  is equivalent to
$$
2 W \geq W + \frac {1}{\tau^ {2}} \sum_ {i > W} \left(f _ {i} ^ {*}\right) ^ {2}.
$$
Consider the following function
$$
F (k) := k + \frac {1}{\tau^ {2}} \sum_ {i > k} \left(f _ {i} ^ {*}\right) ^ {2}, \quad k \geq 1,
$$
one can directly verify that  $F(k)$  is minimized at  $k^{*} := \# \{i : f_{i} \geq \tau\}$ ; moreover,
$$
F \left(k ^ {*}\right) = k ^ {*} + \frac {1}{\tau^ {2}} \sum_ {i > k ^ {*}} \left(f _ {i} ^ {*}\right) ^ {2} = \# \left\{f _ {i} \geq \tau \right\} + \frac {1}{\tau^ {2}} \sum_ {f _ {i} <   \tau} f _ {i} ^ {2}.
$$
$$
2 W \geq F (W) \geq F (k ^ {*}) = \# \left\{f _ {i} \geq \tau \right\} + \frac {1}{\tau^ {2}} \sum_ {f _ {i} <   \tau} f _ {i} ^ {2}.
$$
B Missing Proofs for Section 3
Proof of Theorem 3.1. The proof is motivated by Huang et al. [2021].
Define the following events