Title: A Huber Loss Minimization Approach to Mean Estimation under User-level Differential Privacy
Venue: NeurIPS 2024
Accepted: yes
Keywords: differential privacy

robustness

huber loss
Abstract: Privacy protection of users' entire contribution of samples is important in distributed systems. The most effective approach is the two-stage scheme, which finds a small interval first and then gets a refined estimate by clipping samples into the interval. However, the clipping operation induces bias, which is serious if the sample distribution is heavy-tailed. Besides, users with large local sample sizes can make the sensitivity much larger, thus the method is not suitable for imbalanced users. Motivated by these challenges, we propose a Huber loss minimization approach to mean estimation under user-level differential privacy. The connecting points of Huber loss can be adaptively adjusted to deal with imbalanced users. Moreover, it avoids the clipping operation, thus significantly reducing the bias compared with the two-stage approach. We provide a theoretical analysis of our approach, which gives the noise strength needed for privacy protection, as well as the bound of mean squared error. The result shows that the new method is much less sensitive to the imbalance of user-wise sample sizes and the tail of sample distributions. Finally, we perform numerical experiments to validate our theoretical analysis.

A Huber Loss Minimization Approach to Mean Estimation under User-level Differential Privacy

Abstract

Privacy protection of users' entire contribution of samples is important in distributed systems. The most effective approach is the two-stage scheme, which finds a small interval first and then gets a refined estimate by clipping samples into the interval. However, the clipping operation induces bias, which is serious if the sample distribution is heavy-tailed. Besides, users with large local sample sizes can make the sensitivity much larger, thus the method is not suitable for imbalanced users. Motivated by these challenges, we propose a Huber loss minimization approach to mean estimation under user-level differential privacy. The connecting points of Huber loss can be adaptively adjusted to deal with imbalanced users. Moreover, it avoids the clipping operation, thus significantly reducing the bias compared with the two-stage approach. We provide a theoretical analysis of our approach, which gives the noise strength needed for privacy protection, as well as the bound of mean squared error. The result shows that the new method is much less sensitive to the imbalance of user-wise sample sizes and the tail of sample distributions. Finally, we perform numerical experiments to validate our theoretical analysis.

1 Introduction

Privacy is one of the major concerns in modern data analysis. Correspondingly, differential privacy (DP) [1] has emerged as a standard framework of privacy protection. Various statistical problems have been analyzed with additional DP requirements [2-5]. Among all these problems, mean estimation is a fundamental one [6-9], which is not only useful in its own right [10], but also serves as a building block of many other tasks relying on estimating gradients, such as private stochastic optimization [11-16] and machine learning [17-23]. Existing research on DP mean estimation focuses primarily on item-level cases, i.e. each user contributes only one sample. However, in many practical scenarios, especially in recommendation systems [24-26] and federated learning [27-32], a user has multiple samples. We hope to regard them as a whole for privacy protection.

In recent years, a flurry of works focus on user-level DP [33-36]. The most popular one is the Winsorized Mean Estimator (WME) proposed in [34], which takes a two-stage approach. In the first stage, WME identifies an interval, which is small but contains the ground truth µ with high probability. In the second stage, WME clips user-wise averages to control the sensitivity and then calculates the final average with appropriate noise. This method can be extended to high dimensionality by Hadamard transform [37]. The convergence rate has been established in [34] under some ideal

Zhe Liu ∗

③❤❡✳❧✐✉❅③❤❡❥✐❛♥❣❧❛❜✳❝♦♠

assumptions. Despite the merit of the two-stage approach from the theoretical perspective, this method may face challenges in many realistic settings. Firstly, [34] assumes that users are balanced, which means that users have the same number of items. Nevertheless, in federated learning applications, it is common for clients (each client is regarded as a user here) to possess different numbers of samples [38-40]. Secondly, this method is not suitable for heavy-tailed distributions, which is also common in reality [41-46]. For heavy-tailed distributions, the interval generated in the first stage needs to be large enough to prevent clipping bias, which results in large sensitivity. As a result, stronger additive noise is needed for privacy protection, which significantly increases the estimation error. These drawbacks hinder the practical application of user-level DP. We aim to propose new solutions to address these challenges.

Towards this goal, in this paper, we propose a new method, which estimates the mean using Huber loss minimizer [47], and then adds noise for privacy protection. A challenge is that to determine an appropriate noise strength, it is necessary to conduct a thorough analysis of the local sensitivity that considers all possible datasets. To overcome this challenge, we divide datasets into three types, including those with no outliers, a few outliers, and many outliers, and analyze these cases separately. Based on the sensitivity analysis, we then use the smooth sensitivity framework [48] to determine the noise strength carefully.

Our method has the following advantages. Firstly, our method adapts well to imbalanced datasets, since the threshold T i of Huber loss are selected adaptively according to the sample size per user, which leads to a better tradeoff between sensitivity and bias. Secondly, our method performs better for heavy-tailed distributions, since we control sensitivity by penalizing large distances using Huber loss, which yields a smaller bias than the clipping operation. Apart from solving these practical issues, it worths mentioning that our method solves robustness (to model poisoning attacks) and privacy issues simultaneously. In modern data analysis, it is common for a system to suffer from both poisoning and inference attacks at the same time [49-51]. Consequently, many recent works focus on unified methods for item-level DP and robustness to cope with both attacks simultaneously [52-55]. To the best of our knowledge, our method is the first attempt to unify robustness and DP at user-level.

The main contributions are summarized as follows.

We propose the Huber loss minimization approach, which finds the point with minimum Huber distance to all samples. Our method is convenient to implement and only requires linear time complexity.

For the simplest case with balanced users, we provide a theoretical analysis, which shows that our method makes a slight improvement for bounded distributions and a significant improvement for heavy-tailed distributions over the two-stage approach.

For imbalanced users, we design an adaptive strategy to select weights and connecting points in Huber loss, which makes our method much less sensitive to the imbalance of local sample sizes of users.

We conduct experiments using both synthesized and real data, which also verify the effectiveness of the proposed method for imbalanced users and heavy-tailed distributions.

2 Related Work

User-level DP. [27] applies a brute-force clipping method [18] for user-level DP in federated learning. [33] made the first step towards optimal rates under user-level DP, which analyzed discrete distribution estimation problems. The popular method WME was proposed in [34], which uses the idea of twostage approaches [55-57]. The two-stage method for user-level DP has also been extended to stochastic optimization problems [58,59]. [60] analyzes mean estimation for boolean signals under user-level DP in heterogeneous settings. There are also some works focusing on black-box conversion from item-level DP to user-level counterparts. [35] analyzed general statistical problems, which shows that a class of algorithms for item-level DP problems having the pseudo-globally stable property can be converted into user-level DP algorithms. Following [35], [61] expanded such transformation for any item-level algorithms. [36] extends the works to smaller m . It is discussed in [59] that these black-box methods have suboptimal dependence on ϵ . [62-64] studies user-level DP under local model.

From robustness to DP. Robustness and DP have close relationships since they both require the outputs to be insensitive to minor changes in input samples. There are three types of methods for conversion from robust statistics to DP. The first one is propose-test-release (PTR), which was first proposed in [65], and was extended into high dimensional cases in [66]. The second choice is smooth sensitivity [48], which calculates the noise based on the "smoothed" local sensitivity. For example, [67] designed a method to protect trimmed mean with smooth sensitivity. The third solution is inverse sensitivity [6, 68, 69], which can achieve pure differential privacy (i.e. δ = 0 ). All these methods require a detailed analysis of the sensitivity. For some recently proposed high dimensional estimators [70-72], the sensitivity is usually large and hard to analyze. As a common method for robust statistics [47], Huber loss minimization has been widely applied in robust regression [73, 74], denoising [75] and robust federated learning [76]. Huber loss has also been used in DP [77,78] for linear regression problems.

Concurrent work. After the initial submission of this paper, we notice an independent work [79], which also studies mean estimation under user-level DP (which is called person-level DP in [79]). [79] considers directional bound, which requires that the moment is bounded in every direction. However, we consider non-directional bound, which bounds the ℓ 2 norm of a random vector. We refer to Section 1.3.1 in [79] for further discussion.

Our work is the first attempt to use the Huber loss minimization method in user-level DP. With an adaptive selection of weights and connecting points between quadratic and linear parts, our method achieves a significantly better performance for imbalanced users and heavy-tailed distributions.

3 Preliminaries

In this section, we introduce definitions and notations. To begin with, we recall some concepts of DP and introduce the notion of user-level DP. Denote Ω as the space of all datasets, and Θ as the space of possible outputs of an algorithm.

Definition 1. (Differential Privacy (DP) [1]) Let ϵ, δ ≥ 0 . A function A : Ω → Θ is ( ϵ, δ ) -DP if for any measurable subset O ⊆ Θ and any two adjacent datasets D and D ′ ,

in which D and D ′ are adjacent if they differ only on a single sample. Moreover, A is ϵ -DP if (1) holds with δ = 0 .

Definition 1 is about item-level DP. In this work, we discuss the case where the dataset contains multiple users, i.e. D = { D 1 , . . . , D n } , with the i -th user having m i samples. Considering that the sample sizes of users are usually much less sensitive [80], throughout this work, we assume that the local sample sizes m i are public information. Under this setting, user-level DP is defined as follows.

̸

Definition 2. (User-level DP [34]) Two datasets D , D ′ are user-level adjacent if they differ in items belonging to only one user. In particular, if D = { D 1 , . . . , D n } , D ′ = { D ′ 1 , . . . , D ′ n } , in which | D i | = | D ′ i | = m i for all i , and there is only one i ∈ [ n ] such that D i = D ′ i , then D and D ′ are user-level adjacent. A function A is user-level ( ϵ, δ ) -DP if (1) is satisfied for any two user-level adjacent datasets D and D ′ .

Since m i , i = 1 , . . . , N are public information, in Definition 2, it is required that | D i | = | D ′ i | , which means that two adjacent datasets need to have the same sample sizes for all users. We then state some concepts related to sensitivity, which describes the maximum change of the output after replacing a user with another one:

Definition 3. (Sensitivity) Define the local sensitivity of function f as

̸

in which d H ( D , D ′ ) = ∑ n i =1 1 ( D i = D ′ i ) denotes the Hamming distance. The global sensitivity of f is GS f = sup D LS f ( D ) .

Adding noise proportional to the global sensitivity can be inefficient, especially for user-level problems. In this work, we use the smooth sensitivity framework [48].

Definition 4. (Smooth sensitivity) S f is a β -smooth sensitivity of f , if (1) for any D , S f ( D ) ≥ LS f ( D ) ; (2) for any neighboring D and D ′ , S f ( D ) ≤ e β S f ( D ′ ) .

The smooth sensitivity can be used to determine the scale of noise. In this work, the noise follows Gaussian distribution. It has been shown in [48] that if W ∼ N (0 , ( S 2 ( D ) /α 2 ) I ) , then the final output f ( D ) + W is ( ϵ, δ ) -DP for the following ( α, β ) pair:

Notations. Throughout this paper, ∥·∥ denotes the ℓ 2 norm by default. a ≲ b means that a ≤ Cb for some absolute constant C , and ≳ is defined conversely. a ∼ b if a ≲ b and b ≲ a .

4 The Proposed Method

This section introduces the algorithm structures. Details about parameter selection are discussed together with the theoretical analysis in Section 5 and 6, respectively. For a dataset D = { D 1 , . . . , D n } , in which D i ⊂ R d is the local dataset of the i -th user, denote m i = | D i | as the sample size of the i -th user. Denote N as the total number of samples, then N = ∑ n i =1 m i . We calculate the user-wise mean first, i.e. y i ( D ) = (1 /m i ) ∑ x ∈ D i x . The new proposed estimator (before adding noise) is

in which w i is the normalized weight of user i , i.e. ∑ n i =1 w i = 1 . If users are balanced, then w i are the same for all i . ϕ i is the Huber loss function:

T i is the connecting point between quadratic and linear parts of Huber loss. For balanced users, w i and T i are the same for all users. For imbalanced users, w i and T i are set differently depending on the per-user sample sizes. The general guideline is that w i increases with m i , while T i decreases with m i . The final output needs to satisfy user-level ( ϵ, δ ) -DP requirement. Hence, we set

in which Clip( v , R c ) = v min(1 , R c / ∥ v ∥ ) is the function that clips the result into B d ( 0 , R c ) . The clipping operation is used to control the worst case sensitivity. W denotes the noise added to the estimated value. In this work, we use Gaussian noise W ∼ N (0 , σ 2 I ) . The clipping radius R c is determined by the knowledge of the range of µ . Given a prior knowledge ∥ µ ∥ ≤ R , then we can set R c = R . Actually, similar to [55], our analysis shows that R c can grow exponentially with n without significantly compromising the accuracy. The noise parameter σ 2 needs to be determined carefully through a detailed sensitivity analysis.

Now we comment on the implementation. As has been discussed in [76], minimizing multidimensional Huber loss can be implemented by a modification of an iterative Weiszfeld's algorithm [81, 82]. The overall worst-case time complexity is O ( nd/ξ ) , in which ξ is the desired precision. Moreover, for bounded support, with high probability, the algorithm requires only one iteration with time complexity O ( nd ) . Details can be found in Appendix A.

5 Analysis: Balanced Users

In this section, to gain some insights, we focus on the relatively simpler case and assume that all users have the same number of items, i.e. m i are equal for all i . Therefore, throughout this section, we omit the subscript and use m to denote the local sample size. Now w i = 1 /n for all i . T i are also the same for all users, denoted as T in this section. Let X be a random vector, whose statistical mean µ := E [ X ] is unknown. Given a dataset D = { D 1 , . . . , D n } , the goal is to estimate µ , while

satisfying user-level ( ϵ, δ ) -DP. We present the sensitivity analysis for the general dataset D first, and then analyze the estimation error for a randomly generated dataset. To begin with, define

in which ¯ y ( D ) = (1 /n ) ∑ n i =1 y i ( D ) is the overall average. A small Z ( D ) indicates that the userwise means y i are concentrated within a small region. If Z ( D ) is large, then there are some outliers. The sensitivity is bounded separately depending on whether the user-wise means are well concentrated. Throughout this section, we use LS ( D ) to denote the local sensitivity of Clip(ˆ µ 0 ( D ) , R c ) , in which the subscript f in (2) is omitted.

1) No outliers. We first bound the local sensitivity for the case with Z ( D ) < (1 -2 /n ) T , in which T represents T i in (5) for all i . It requires that all y i 's are not far away from their average.

Lemma 1. If Z ( D ) < (1 -2 /n ) T , then

The proof of Lemma 1 is shown in Appendix B. Here we provide some intuition. From the definition (2), local sensitivity is the maximum change of ˆ µ 0 ( D ) after replacing some y i with y ′ i . To achieve such maximum change, the optimal choice is to move y i sufficiently far away in the direction of ˆ µ 0 ( D ) -y i . The impact of y i on ˆ µ 0 ( D ) is roughly Z ( D ) / ( n -1) , while the impact of y ′ i is roughly T/ ( n -1) . Since y i and y ′ i are at opposite direction with respect to ˆ µ 0 ( D ) , the overall effect caused by replacing y i with y ′ i is upper bounded by ( T + Z ( D )) / ( n -1) .

2) A few outliers. Now we consider a more complex case: Z ( D ) is large, and the dataset is not well concentrated, but the number of outliers is not too large. Formally, assume that there exists another dataset D ∗ whose Hamming distance to D is bounded by k , and D ∗ is well concentrated. Then we have the following lemma to bound the local sensitivity.

Lemma 2. For a dataset D , if there exists a dataset D ∗ such that d H ( D , D ∗ ) ≤ k , in which d H is the Hamming distance (see Definition 3), and Z ( D ∗ ) < (1 -2( k +1) /n ) T , then LS ( D ) ≤ 2 T/ ( n -k ) .

The proof of Lemma 2 is shown in Appendix C. The intuition is that since there exists a well concentrated dataset D ∗ with d H ( D , D ∗ ) ≤ k , D contains no more than k outliers. At least n -k other user-wise mean values fall in a small region. To achieve the maximum change of ˆ µ 0 ( D ) , the optimal choice is to replace an outlier y i with y ′ i , such that y i -ˆ µ 0 ( D ) and y ′ i -ˆ µ 0 ( D ) have opposite directions. Each of them has an effect of roughly T/ ( n -k ) on ˆ µ 0 ( D ) , thus the overall change is 2 T/ ( n -k ) .

3) Other cases. For all other cases, since ∥ Clip(ˆ µ 0 , R c ) ∥ ≤ R c always hold, the local sensitivity can be bounded by LS ( D ) ≤ 2 R c .

From the analysis above, we now construct a valid smooth sensitivity. Define

∆( D ) can be viewed as the number of outliers. From (9), if D is well concentrated, with Z ( D ) < T/ 2 , then ∆( D ) = 0 . Now we define G ( D , k ) as follows.

(b) If conditions in (a) are not satisfied, and ∆( D ) exists, if k ≤ n/ 4 -1 -∆( D ) ,

(c) If conditions in (a) and (b) are not satisfied, then G ( D , k ) = 2 R c .

Based on Definition 5, the smooth sensitivity is given by

in which β is determined in (3). Then we show that S ( D ) is a valid smooth sensitivity, and the privacy requirement is satisfied.

Theorem 1. With σ = S ( D ) /α , in which α is determined in (3) , W ∼ N (0 , σ 2 I ) , the estimator ˆ µ defined in (6) is ( ϵ, δ ) -DP.

To prove Theorem 1, we need to show that S is a valid smooth sensitivity, i.e. two conditions in Definition 4 are satisfied. The detailed proof is provided in Appendix D.

In the analysis above, all results are derived for a general dataset D . In the remainder of this section, we analyze the performance of estimator (6) for randomly generated samples.

5.1 Bounded Support

Let X be a random vector generated from distribution P with an unknown statistical mean µ := E [ X ] . We make the following assumption:

Assumption 1. X is supported on B d (0 , R ) = { u | ∥ u ∥ ≤ R } ⊂ R d .

The mean squared error is analyzed in the following theorem.

Theorem 2. Let R c = R , and T = C T R ln( mn 3 ( d + 1)) / √ m with C T > 16 √ 2 / 3 . If n > (4 /β ) ln( nR c /T ) , then

The proof of Theorem 2 is shown in Appendix E. With the selection rule of T in Theorem 2, it is shown that with high probability, ∆( D ) = 0 , indicating that D is expected to be well concentrated around the population mean µ . The smooth sensitivity S ( D ) can then be bounded. The first term in the right hand side of (12) is the non-private estimation error, i.e. the error of ˆ µ 0 ( D ) , while the second term is the error caused by noise W . The condition n > (4 /β ) ln( nR c /T ) is necessary, since it ensures that G ( D , k ) = 2 R c (Definition 5 (c)) occurs only for sufficiently large k , thus e -βk is small, and does not affect the calculation of S ( D ) in (11). A lower bound on the number of users n has also been imposed for the two-stage method [34].

For the simplest case with bounded support and balanced users, the two-stage approach in [34] is already nearly optimal (Corollary 1 in [34]). Therefore, improvement in polynomial factors is impossible. Nevertheless, we still improve on the logarithm factor. The main purpose of Theorem 2 is to show that our improvement on heavy-tailed distributions and imbalanced users is not at the cost of hurting the performance under the simplest case with bounded distributions and balanced users.

5.2 Unbounded Support

Now we analyze the heavy-tailed case. Instead of requiring X ∈ B d (0 , R ) , we now assume that X has p -th bounded moment.

Assumption 2. Suppose that µ ∈ B ( 0 , R ) , and the p -th ( p ≥ 2 ) moment of X -µ is bounded, i.e. E [ ∥ X ∥ p ] ≤ M p .

In Assumption 2, higher p indicates a lighter tail and vice versa. We then show the convergence rate of mean squared error in Theorem 3.

Theorem 3. Let R c = R , and

with ν = √ d/ ( nϵ ) and C T > 8 M 1 p p . If n > 8(1 + (1 /β ) ln( n/ 2 T )) , then under Assumption 2,

The proof of Theorem 3 is shown in Appendix F. Here we provide an intuitive understanding. From the central limit theorem, each user-wise mean y i ( D ) is the average of m i.i.d variables, thus it has

a Gaussian tail around the population µ . However, since X is only required to have p -th bounded moment, the tail probability away from µ is still polynomial. The formal statement of the tail bound is shown in Lemma 13 in the appendix. Then the threshold T is designed based on the high probability upper bound of Z ( D ) to ensure that with high probability, ∆( D ) is small. Regarding the result, we have the following remarks.

Remark 1. Here we comment on small and large m limits. If m = 1 , the right hand side of (14) becomes O (1 /n +( d/n 2 ϵ 2 ) 1 -1 /p ) , which matches existing analysis on item-level DP for heavy-tailed random variables [57]. For the opposite limit, with m 1 -2 /p ≳ n 2 /p ln( nd ) , then the convergence rate is the same as the case with bounded support, indicating that the tail of sample distribution does not affect the error more than a constant factor.

Remark 2. Now we compare (14) with the two-stage approach. Following the analysis in [34], it can be shown that the bound of mean squared error in [34] is ˜ O (( d/ ( n 2 ϵ 2 ))(1 /m + m 4 /p -2 n 6 /p )) (we refer to Appendix G for details). Therefore, we have achieved an improved rate in (14) .

The theoretical results in this section are summarized as follows. If the support is bounded, our method has the same convergence rate as the existing method. For heavy-tailed distributions, our approach significantly reduces the error, since our method avoids the clipping process. In federated learning applications, it is common for gradients to have heavy-tailed distributions [41-44], thus our method has the potential of improving the performance of federated learning under DP requirements. Apart from heavy-tailed distributions, another common characteristic in reality is that users are usually imbalanced. We analyze it in the next section.

6 Analysis: Imbalanced Users

Now we analyze the general case where m i , i = 1 , . . . , n are different. Recall that for balanced users, we have defined Z ( D ) in (7) that finds the maximum distance from y i ( D ) to their average ¯ y ( D ) . For imbalanced users, instead of taking the maximum, we define Z i separately for each i :

in which ¯ y ( D ) = ∑ n i =1 w i y i ( D ) is the average of samples all over the dataset.

From now on, without loss of generality, suppose that users are arranged in ascending order of m i , i.e. m 1 ≤ . . . ≤ m n . Define

Similar to the case with balanced users, we analyze the sensitivity for datasets with no outliers, a few outliers, and other cases separately.

1) No outliers. We show the following lemma.

The general idea of the proof is similar to Lemma 1. However, the details become more complex since now the samples are unbalanced. The detailed proof is shown in Appendix H.

2) A few outliers. Similar to Lemma 2, we find a neighboring dataset D ∗ that is well concentrated and then bounds the local sensitivity. The formal statement is shown in the following lemma.

Lemma 4. For a dataset D , if there exists another dataset D ∗ such that h ( D ∗ , k +1) < min i ( T i -Z i ( D ∗ )) , then LS ( D ) ≤ 2 max i ( w i T i ) / ∑ n -k -1 i =1 w i .

The proof of Lemma 4 is shown in Appendix I, which just follows the proof of Lemma 2.

3) Other cases. Finally, for all cases not satisfying the conditions in Lemma 3 and 4, we can just bound the local sensitivity with 2 R c , i.e. LS ( D ) ≤ 2 R c .

Similar to the case with balanced users, now we define

in which k 0 is any integer, and can be viewed as a design parameter. Correspondingly, the smooth sensitivity G ( D , k ) is defined as follows.

Definition 6. (a) If h ( D , 1) ≤ min i ( T i -Z i ( D )) , then G ( D , 0) = h ( D , 1) ;

(b) If the conditions in (a) are not satisfied, and ∆( D ) exists, then for all k ≤ k 0 -∆( D ) -1 ,

(c) If the conditions in both (a) and (b) are not satisfied, then G ( D , k ) = 2 R c .

We still use the same settings of α and β as in the case with balanced samples. With smooth sensitivity calculated using (11), the privacy requirement is satisfied:

Theorem 4. Let W ∼ N (0 , ( S ( D ) 2 /α 2 ) I ) , in which S ( D ) = max k e -βk G ( D , k ) , then the estimator ˆ µ is ( ϵ, δ ) -DP.

Now we analyze the convergence of the algorithm. We begin with Assumption 3. Intuitively, this assumption requires that the users can not be too unbalanced. At least half samples belong to users whose sample sizes are not very large.

Assumption 3. Suppose there exists a constant γ ≥ 1 . Let k c = min { i | m i > γN/n } , then ∑ n i = k c m i ≤ N/ 2 .

In Assumption 3, γ can be viewed as the degree of imbalance. For a better explanation, we provide the following examples:

If users are balanced, then γ = 1 ;

If the i -th user has ki samples (which means that the number of items belonging to each user is linear in its order), then for large n , γ is approximately √ 2 .

In general, γ is large if users are highly imbalanced. Under Assumption 3, the convergence of mean squared error is shown in Theorem 5.

Theorem 5. Let the weights of users in (4) be w i = m i ∧ m c / ( ∑ n j =1 m j ∧ m c ) , in which m c = γN/n . Moreover, let

in which C T > 16 √ 2 / 3 . In (17) , k 0 = ⌊ n/ 8 γ ⌋ .

With the parameters above, if n > 8 γ (1 + (1 / 2 β ) ln( Nn )) , then under Assumption 1 and 3,

The proof of Theorem 5 is shown in Appendix J. If users are balanced, then γ = 1 , with N = nm , (20) reduces to (12). From (20) and Assumption 3 it can be observed that our method is much less sensitive to a single large m i . As long as m i are not very large for most users, the convergence rate of mean squared error is not affected. There are two intuitive reasons. Firstly, w i is upper bounded by m c / ( ∑ n j =1 m j ∧ m c ) , thus the worst-case sensitivity is controlled. Secondly, T i are set adaptively to achieve a good tradeoff between sensitivity and bias. With larger m i , smaller T i is used, and vice versa. We refer to Appendix G for comparison with the two-stage approach.

7 Numerical Examples

In this section, we show numerical experiments. We compare the performance of our new Huber loss minimization approach (denoted as HLM) versus the two-stage approach proposed in [34], called Winsorized Mean Estimator (denoted as WME).

7.1 Balanced Users

Here all users have the same sizes of local datasets. In the following experiments, we fix ϵ = 1 and δ = 10 -5 . For a fair comparison, the parameter T for our method as well as τ in [34] are both tuned optimally for each case. Figure 1 shows the curve of mean squared error. Let the number of users n be either 1 , 000 and 10 , 000 . In each curve, n is fixed, while the number of samples per user m varies from 1 to 1 , 000 . The results are plotted in logarithm scale. Figure 1(a)-(c) show the results of uniform distribution in [ -1 , 1] , Gaussian distribution N (0 , 1) , and the Lomax distribution, whose pdf is f ( x ) = a/ (1 + x ) a +1 (we use a = 4 in Figure 1(c)). Figure 1 (d)-(f) shows the corresponding experiments with dimensionality d = 3 . Finally, Figure 1 (g) and (h) show the results using the IPUMS dataset [83] for total income and salary, respectively, which are typical examples of data following heavy-tailed distributions.

10 0

10 1

10 2

10 3

m

10

7

10

6

10

5

10

4

10

3

mean squared error

HLM, n=1000

WME, n=1000

HLM, n=10000

WME, n=10000

(a) Uniform,

d

= 1

10 0

10 1

10 2

10 3

m

10

7

10

6

10

5

10

4

10

3

10

2

mean squared error

HLM, n=1000

WME, n=1000

HLM, n=10000

WME, n=10000

d

= 1

10 0

10 1

10 2

10 3

m

10

7

10

6

10

5

10

4

10

3

10

2

10

1

mean squared error

HLM, n=1000

WME, n=1000

HLM, n=10000

WME, n=10000

(c) Lomax,

d

= 1

10 0

10 1

10 2

10 3

m

10

5

10

4

10

3

10

2

10

1

10 0

mean squared error

HLM, n=1000

WME, n=1000

HLM, n=10000

WME, n=10000

(d) Uniform,

d

= 3

10 0

10 1

10 2

10 3

m

10

5

10

4

10

3

10

2

10

1

10 0

10 1

mean squared error

HLM, n=1000

WME, n=1000

HLM, n=10000

WME, n=10000

d

= 3

10 0

10 1

10 2

10 3

m

10

6

10

5

10

4

10

3

10

2

10

1

10 0

mean squared error

HLM, n=1000

WME, n=1000

HLM, n=10000

WME, n=10000

(f) Lomax,

d

= 3

10 0

10 1

10 2

10 3

m

10 2

10 3

10 4

10 5

10 6

10 7

10 8

10 9

mean squared error

HLM, n=1000

WME, n=1000

HLM, n=10000

WME, n=10000

(g) IPUMS, total income

10 0

10 1

10 2

10 3

m

10 2

10 3

10 4

10 5

10 6

10 7

10 8

10 9

mean squared error

HLM, n=1000

WME, n=1000

HLM, n=10000

WME, n=10000

(h) IPUMS, salary

Figure 1 (a) and (b) show that for one-dimensional uniform and Gaussian distribution, the Huber loss minimization approach has nearly the same performance as the two-stage method. Our explanation is that uniform and Gaussian distributions are symmetric, with no tails or light tails, thus the clipping operation does not introduce additional bias. However, for heavy-tailed and skewed distribution, such as Lomax distribution, our new method has a significantly faster convergence rate than the two-stage method. These results agree with the theoretical analysis, which shows that our method reduces the clipping bias. With higher dimensionality, Figure 1(d)-(f) show that the advantage of the practical performance of our method becomes more obvious.

7.2 Imbalanced Users

Now we show the performance with unbalanced users. For some γ ≥ 1 , let s i = ⌈ N ( i/n ) γ ⌉ for i = 0 , . . . , n , and m i = s i -s i -1 for i = 1 , . . . , n . It can be shown that Assumption 3 is satisfied with γ . Therefore, according to the analysis in Section 6, we let w i = m i ∧ m c / ∑ j m j ∧ m c , with m c = γN/n , and T i = A/ √ m i ∧ m c , in which A is tuned optimally for each case. The selection of T i may be slightly different from (19) in Theorem 5. In Theorem 5, T i is selected to minimize the theoretical upper bound. To ensure that the analysis is mathematically rigorous, the upper bound of estimation error is larger than the truth. Therefore, the optimal value of T i in practice is slightly different from that derived in theories. Note that such parameter tuning does not require additional privacy budget since in each experiment, T i are hyperparameters that is fixed before knowing the value of each sample. They are not determined adaptively based on the data. Figure 2 shows the growth curve of mean squared error with respect to γ .

From Figure 2, it can be observed that with the increase of γ , the two-stage method degrades, while the Huber loss minimization approach performs significantly more stable.

2

4

6

8

0.0000

0.0001

0.0002

0.0003

0.0004

mean squared error

HLM, n=2000

WME, n=2000

HLM, n=10000

WME, n=10000

(a) Uniform.

2

4

6

8

0.0000

0.0001

0.0002

0.0003

0.0004

0.0005

0.0006

0.0007

mean squared error

HLM, n=2000

WME, n=2000

HLM, n=10000

WME, n=10000

2

4

6

8

0.00000

0.00025

0.00050

0.00075

0.00100

0.00125

0.00150

0.00175

mean squared error

HLM, n=2000

WME, n=2000

HLM, n=10000

WME, n=10000

(c) Exponential.

2

4

6

8

0

100000

200000

300000

mean squared error

HLM, n=2000

WME, n=2000

HLM, n=10000

WME, n=10000

(d) IPUMS total income.

8 Conclusion

In this paper, we have proposed a new approach to mean estimation under user-level DP based on Huber loss minimization. The sensitivity is bounded for all possible datasets. Based on the sensitivity analysis, we use the smooth sensitivity framework to determine the noise added to the result. We have also derived the bound on the mean squared error for various cases. The result shows that our method reduces the error for heavy-tailed distributions, and is more suitable to imbalanced users. It is promising to extend our approach to more learning problems, such as calculating average gradients in federated learning.

Limitations: The limitations of our work include: (1) There are some requirements on the minimum number of users n . while entirely removing this condition is impossible, to make our method more practical, we expect that it can be somewhat weakened. (2) The case with local sample sizes m i also being private has not been analyzed. We will leave these two points as future works.

References

Dwork, C., F. McSherry, K. Nissim, et al. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3 , pages 265-284. Springer, 2006.

Dwork, C., G. N. Rothblum, S. Vadhan. Boosting and differential privacy. In 2010 IEEE 51st Annual Symposium on Foundations of Computer Science , pages 51-60. IEEE, 2010.

Kasiviswanathan, S. P., H. K. Lee, K. Nissim, et al. What can we learn privately? SIAM Journal on Computing , 40(3):793-826, 2011.

Dwork, C., A. Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science , 9(3-4):211-407, 2014.

Wang, S., L. Huang, Y. Nie, et al. Local differential private data aggregation for discrete distribution estimation. IEEE Transactions on Parallel and Distributed Systems , 30(9):20462059, 2019.

Huang, Z., Y. Liang, K. Yi. Instance-optimal mean estimation under differential privacy. Advances in Neural Information Processing Systems , 34:25993-26004, 2021.

Asi, H., V. Feldman, K. Talwar. Optimal algorithms for mean estimation under local differential privacy. In International Conference on Machine Learning , pages 1046-1056. PMLR, 2022.

Hopkins, S. B., G. Kamath, M. Majid. Efficient mean estimation with pure differential privacy via a sum-of-squares exponential mechanism. In Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing , pages 1406-1417. 2022.

Nguyên, T. T., X. Xiao, Y. Yang, et al. Collecting and analyzing data from smart device users with local differential privacy. arXiv preprint arXiv:1606.05053 , 2016.

Sun, C., Y. Fu, J. Zhou, et al. Personalized privacy-preserving frequent itemset mining using randomized response. The Scientific World Journal , 2014, 2014.

Chaudhuri, K., C. Monteleoni, A. D. Sarwate. Differentially private empirical risk minimization. Journal of Machine Learning Research , 12(3), 2011.

Bassily, R., A. Smith, A. Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 2014 IEEE 55th annual symposium on foundations of computer science , pages 464-473. IEEE, 2014.

Bassily, R., V. Feldman, K. Talwar, et al. Private stochastic convex optimization with optimal rates. Advances in neural information processing systems , 32, 2019.

Feldman, V., T. Koren, K. Talwar. Private stochastic convex optimization: optimal rates in linear time. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing , pages 439-449. 2020.

Asi, H., V. Feldman, T. Koren, et al. Private stochastic convex optimization: Optimal rates in l1 geometry. In International Conference on Machine Learning , pages 393-403. PMLR, 2021.

Cheu, A., M. Joseph, J. Mao, et al. Shuffle private stochastic convex optimization. In International Conference on Learning Representations . 2022.

Shokri, R., V. Shmatikov. Privacy-preserving deep learning. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security , pages 1310-1321. 2015.

Abadi, M., A. Chu, I. Goodfellow, et al. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security , pages 308-318. 2016.

McMahan, H. B., D. Ramage, K. Talwar, et al. Learning differentially private recurrent language models. arXiv preprint arXiv:1710.06963 , 2017.

Dan, H.-C., B. Lu, M. Li. Evaluation of asphalt pavement texture using multiview stereo reconstruction based on deep learning. Construction and Building Materials , 412:134837, 2024.

Dan, H.-C., P. Yan, J. Tan, et al. Multiple distresses detection for asphalt pavement using improved you only look once algorithm based on convolutional neural network. International Journal of Pavement Engineering , 25(1):2308169, 2024.

Wang, X., Y. Wang, Y. Su, et al. Ensguard: A novel acceleration framework for adversarial ensemble learning. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems , 43(10):3088-3101, 2024.

Mo, K., P. Ye, X. Ren, et al. Security and privacy issues in deep reinforcement learning: Threats and countermeasures. ACM Computing Surveys , 56(6):1-39, 2024.

Huang, R., H. Zhang, L. Melis, et al. Federated linear contextual bandits with user-level differential privacy. In International Conference on Machine Learning , pages 14060-14095. PMLR, 2023.

Li, T., L. Song, C. Fragouli. Federated recommendation system via differential privacy. In 2020 IEEE international symposium on information theory (ISIT) , pages 2592-2597. IEEE, 2020.

McSherry, F., I. Mironov. Differentially private recommender systems: Building privacy into the netflix prize contenders. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 627-636. 2009.

Geyer, R. C., T. Klein, M. Nabi. Differentially private federated learning: A client level perspective. arXiv preprint arXiv:1712.07557 , 2017.

McMahan, H. B., G. Andrew, U. Erlingsson, et al. A general approach to adding differential privacy to iterative training procedures. arXiv preprint arXiv:1812.06210 , 2018.

Wei, K., J. Li, M. Ding, et al. Federated learning with differential privacy: Algorithms and performance analysis. IEEE transactions on information forensics and security , 15:3454-3469, 2020.

Kairouz, P., H. B. McMahan, B. Avent, et al. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning , 14(1-2):1-210, 2021.

Fu, J., Y . Hong, X. Ling, et al. Differentially private federated learning: A systematic review. arXiv preprint arXiv:2405.08299 , 2024.

Feng, J., Y. Wu, H. Sun, et al. Panther: Practical secure 2-party neural network inference. IEEE Transactions on Information Forensics and Security , 2025.

Liu, Y., A. T. Suresh, F. X. X. Yu, et al. Learning discrete distributions: user vs item-level privacy. Advances in Neural Information Processing Systems , 33:20965-20976, 2020.

Levy, D., Z. Sun, K. Amin, et al. Learning with user-level privacy. Advances in Neural Information Processing Systems , 34:12466-12479, 2021.

Ghazi, B., R. Kumar, P. Manurangsi. User-level differentially private learning via correlated sampling. Advances in Neural Information Processing Systems , 34:20172-20184, 2021.

Ghazi, B., P. Kamath, R. Kumar, et al. User-level differential privacy with few examples per user. Advances in Neural Information Processing Systems , 36, 2023.

Yarlagadda, R. K., J. E. Hershey. Hadamard matrix analysis and synthesis: with applications to communications and signal/image processing , vol. 383. Springer Science & Business Media, 2012.

Li, T., M. Sanjabi, A. Beirami, et al. Fair resource allocation in federated learning. In International Conference on Learning Representations . 2019.

Duan, M., D. Liu, X. Chen, et al. Self-balancing federated learning with global imbalanced data in mobile systems. IEEE Transactions on Parallel and Distributed Systems , 32(1):59-71, 2020.

Gong, B., T. Xing, Z. Liu, et al. Adaptive client clustering for efficient federated learning over non-iid and imbalanced data. IEEE Transactions on Big Data , 2022.

Gurbuzbalaban, M., U. Simsekli, L. Zhu. The heavy-tail phenomenon in sgd. In International Conference on Machine Learning , pages 3964-3975. PMLR, 2021.

Simsekli, U., L. Sagun, M. Gurbuzbalaban. A tail-index analysis of stochastic gradient noise in deep neural networks. In International Conference on Machine Learning , pages 5827-5837. PMLR, 2019.

Zhang, J., S. P. Karimireddy, A. Veit, et al. Why are adaptive methods good for attention models? Advances in Neural Information Processing Systems , 33:15383-15393, 2020.

Nguyen, T. D., T. H. Nguyen, A. Ene, et al. Improved convergence in high probability of clipped gradient methods with heavy tailed noise. Advances in Neural Information Processing Systems , 36:24191-24222, 2023.

Ke, Z., Y. Yin. Tail risk alert based on conditional autoregressive var by regression quantiles and machine learning algorithms. arXiv preprint arXiv:2412.06193 , 2024.

Zhang, J., Z. Xu. Exponential calibration for correlation coefficient with additive distortion measurement errors. Statistical Analysis and Data Mining: The ASA Data Science Journal , 14(3):271-289, 2021.

Huber, P. J. Robust statistics , vol. 523. John Wiley & Sons, 2004.

Nissim, K., S. Raskhodnikova, A. Smith. Smooth sensitivity and sampling in private data analysis. In Proceedings of the thirty-ninth annual ACM symposium on Theory of computing , pages 75-84. 2007.

Wang, Z., Y. Huang, M. Song, et al. Poisoning-assisted property inference attack against federated learning. IEEE Transactions on Dependable and Secure Computing , 2022.

Lyu, W., S. Zheng, L. Pang, et al. Attention-enhancing backdoor attacks against bert-based models. In Findings of the Association for Computational Linguistics: EMNLP 2023 , pages 10672-10690. 2023.

Lyu, W., L. Pang, T. Ma, et al. Trojvlm: Backdoor attack against vision language models. European Conference on Computer Vision, 2024.

Liu, X., W. Kong, S. Kakade, et al. Robust and differentially private mean estimation. Advances in neural information processing systems , 34:3887-3901, 2021.

Liu, X., P. Jain, W. Kong, et al. Label robust and differentially private linear regression: Computational and statistical efficiency. Advances in Neural Information Processing Systems , 36, 2023.

Qi, T., H. Wang, Y. Huang. Towards the robustness of differentially private federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence , vol. 38, pages 19911-19919. 2024.

Li, M., T. B. Berrett, Y . Yu. On robustness and local differential privacy. The Annals of Statistics , 51(2):717-737, 2023.

Smith, A. Privacy-preserving statistical estimation with optimal convergence rates. In Proceedings of the forty-third annual ACM symposium on Theory of computing , pages 813-822. 2011.

Kamath, G., V. Singhal, J. Ullman. Private mean estimation of heavy-tailed distributions. In Conference on Learning Theory , pages 2204-2235. PMLR, 2020.

Bassily, R., Z. Sun. User-level private stochastic convex optimization with optimal rates. In International Conference on Machine Learning , pages 1838-1851. PMLR, 2023.

Liu, D., H. Asi. User-level differentially private stochastic convex optimization: Efficient algorithms with optimal rates. In International Conference on Artificial Intelligence and Statistics , pages 4240-4248. PMLR, 2024.

Cummings, R., V. Feldman, A. McMillan, et al. Mean estimation with user-level privacy under data heterogeneity. Advances in Neural Information Processing Systems , 35:29139-29151, 2022.

Bun, M., M. Gaboardi, M. Hopkins, et al. Stability is stable: Connections between replicability, privacy, and adaptive generalization. In Proceedings of the 55th Annual ACM Symposium on Theory of Computing , pages 520-527. 2023.

Acharya, J., Y. Liu, Z. Sun. Discrete distribution estimation under user-level local differential privacy. In International Conference on Artificial Intelligence and Statistics , pages 8561-8585. PMLR, 2023.

Zhao, P., L. Shen, R. Fan, et al. Learning with user-level local differential privacy. arXiv preprint arXiv:2405.17079 , 2024.

Ma, Y., K. Jia, H. Yang. Better locally private sparse estimation given multiple samples per user. arXiv preprint arXiv:2408.04313 , 2024.

Dwork, C., J. Lei. Differential privacy and robust statistics. In Proceedings of the forty-first annual ACM symposium on Theory of computing , pages 371-380. 2009.

Liu, X., W. Kong, S. Oh. Differential privacy and robust statistics in high dimensions. In Conference on Learning Theory , pages 1167-1246. PMLR, 2022.

Bun, M., T. Steinke. Average-case averages: Private algorithms for smooth sensitivity and mean estimation. Advances in Neural Information Processing Systems , 32, 2019.

Asi, H., J. Ullman, L. Zakynthinou. From robustness to privacy and back. In International Conference on Machine Learning , pages 1121-1146. PMLR, 2023.

Asi, H., J. C. Duchi. Instance-optimality in differential privacy via approximate inverse sensitivity mechanisms. Advances in neural information processing systems , 33:14106-14117, 2020.

Diakonikolas, I., G. Kamath, D. Kane, et al. Robust estimators in high-dimensions without the computational intractability. SIAM Journal on Computing , 48(2):742-864, 2019.

Diakonikolas, I., G. Kamath, D. M. Kane, et al. Being robust (in high dimensions) can be practical. In International Conference on Machine Learning , pages 999-1008. PMLR, 2017.

Diakonikolas, I., D. M. Kane. Algorithmic high-dimensional robust statistics . Cambridge university press, 2023.

Hall, P., M. Jones. Adaptive m-estimation in nonparametric regression. The annals of statistics , pages 1712-1728, 1990.

Zhao, P., Z. Wan. Robust nonparametric regression under poisoning attack. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 17007-17015. 2024.

Sardy, S., P. Tseng, A. Bruce. Robust wavelet denoising. IEEE transactions on signal processing , 49(6):1146-1152, 2001.

Zhao, P., F. Yu, Z. Wan. A huber loss minimization approach to byzantine robust federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 21806-21814. 2024.

Avella-Medina, M., C. Bradshaw, P.-L. Loh. Differentially private inference via noisy optimization. The Annals of Statistics , 51(5):2067-2092, 2023.

Song, S., T. Steinke, O. Thakkar, et al. Evading the curse of dimensionality in unconstrained private glms. In International Conference on Artificial Intelligence and Statistics , pages 26382646. PMLR, 2021.

Agarwal, S., G. Kamath, M. Majid, et al. Private mean estimation with person-level differential privacy. arXiv preprint arXiv:2405.20405 , 2024.

Wei, K., J. Li, M. Ding, et al. User-level privacy-preserving federated learning: Analysis and performance optimization. IEEE Transactions on Mobile Computing , 21(9):3388-3401, 2021.

Weiszfeld, E., F. Plastria. On the point for which the sum of the distances to n given points is minimum. Annals of Operations Research , 167:7-41, 2009.

Beck, A., S. Sabach. Weiszfeld's method: Old and new results. Journal of Optimization Theory and Applications , 164:1-40, 2015.

Ruggles, S., S. Flood, M. Sobek, et al. IPUMS USA: Version 15.0 [dataset], 2024.

Sun, L., J. Tian, G. Muhammad. Fedkc: Personalized federated learning with robustness against model poisoning attacks in the metaverse for consumer health. IEEE Transactions on Consumer Electronics , 2024.

Sun, L., J. He. An extensible framework for ecg anomaly detection in wireless body sensor monitoring systems. International Journal of Sensor Networks , 29(2):101-110, 2019.

Zhang, Y., L. O. Wijeratne, S. Talebi, et al. Machine learning for light sensor calibration. Sensors , 21(18):6259, 2021.

Lary, D. J., L. O. H. Wijeratne, G. K. Zewdie, et al. Machine learning, big data, and spatial tools: A combination to reveal complex facts that impact environmental health. In Geospatial Technology for Human Well-Being and Health , pages 219-241. Springer, 2021.

Tropp, J. A., et al. An introduction to matrix concentration inequalities. Foundations and Trends® in Machine Learning , 8(1-2):1-230, 2015.

A Comments About Algorithm Implementation

In this section, we describe the algorithm for Huber loss minimization.

In particular, we solve

From (5), it can be shown that

The algorithm can be designed from above equation. Suppose that the algorithm starts from c 0 . The update rule is

(23) is run iteratively until the norm of update ∥ c k +1 -c k ∥ between two iterations is less than ξ .

We provide a brief analysis on the computational complexity as follows.

Worst case. [82] has shown that the Weiszfeld's algorithm for calculating geometric median needs O (1 /ξ ) steps to achieve precision ξ . The proof can also be used to our algorithm (23). Moreover, from (23), each step requires O ( nd ) time, in which d is the dimensionality of y i , thus the overall time complexity is O ( nd/ξ ) .

Common case. From the analysis in Section 5 and 6, for bounded support, with high probability, Z ( D ) ≤ T holds for balanced users, and Z i ( D ) ≤ T i holds for imbalanced users. In this case, we can just calculate the result within one step:

Therefore the time complexity is O ( nd ) .

B Proof of Lemma 1

Now the users are balanced. (4) and (5) becomes

and

In this section, we prove a strengthened version of Lemma 1 for future usage. Define the modulus of continuity as

Definition 7. (Modulus of continuity) Define

From the definition, ω ( D , 0) is just the local sensitivity LS ( D ) . Then we show the following lemma.

Lemma 5. If Z ( D ) < (1 -2 k/n ) T , then

Let k = 0 , then Lemma 5 reduces to Lemma 1. The remainder of this section shows the proof of Lemma 5.

̸

According to (27), ω ( D , k ) is the supremum change after replacing all items belonging to k users. Denote D ′ as a dataset such that d H ( D , D ′ ) ≤ k . Let I be the set of users such that D i = D ′ i . Throughout this section, we denote y i = y i ( D ) and y ′ i = y i ( D ′ ) for simplicity. From (25),

Let ∇ ϕ be the gradient of ϕ with respect to the first argument. Define

then and

in which the last step comes from (25).

Then we show the following lemma.

Lemma 6. If Z ( D ) ≤ T , then ˆ µ 0 ( D ) = ¯ y , in which ¯ y = (1 /n ) ∑ n i =1 y i .

Proof. The proof has two steps.

(1) ¯ y is a minimizer of ∑ n i =1 ϕ ( s , y i ) . From (26),

in which the second step holds because ∥ ¯ y -y i ∥ ≤ Z ≤ T . Moreover, ∑ n i =1 ϕ ( s , y i ) is convex, thus ¯ y is a minimizer.

(2) ¯ y is the unique minimizer. The Hessian

Therefore ∑ n i =1 ϕ ( s , y i ) is strong convex around ¯ y , and thus ¯ y is unique.

From (30),

Moreover, for s satisfying ∥ s -¯ y ∥ ≤ T -Z ,

in which the last step holds because

Let L be a path connecting ˆ µ 0 ( D ) and ˆ µ 0 ( D ′ ) . Then

Note that from (31) and (35),

From the condition Z ( D ) < (1 -2 k/n ) T in Lemma 1, ( n -k )( T -Z ( D )) > k ( T + Z ( D )) . Hence, (38) and (39) yields

C Proof of Lemma 2

Denote D ′ as a dataset adjacent to D at user-level. Then it remains to bound ∥ ˆ µ 0 ( D ) -ˆ µ 0 ( D ′ ) ∥ . Recall that in the statement of Lemma 2, we have required that there exists a dataset D ∗ such that Z ( D ∗ ) < (1 -2( k +1) /n ) T , and d H ( D , D ∗ ) ≤ k . Denote I as the set of users such that D and D ∗ have different values, while D and D ′ differ at user u . Now we discuss two cases.

Case 1. u / ∈ I . In other words, D and D ′ differ at a user that is the same between D and D ∗ . In this case, D ′ has k +1 users that are different from D ∗ . Without loss of generality, suppose that I = { 1 , . . . , k } , while u = k +1 . D ∗ , D and D ′ can be written as follows:

For convenience of expression, denote y i = y i ( D ) and y ′ i = y ( D ′ i ) .

In this section, define g 1 ( s ) as follows:

then and

Therefore

(a) comes from (33). By taking gradient over s , we have ∥∇ ϕ ( s , y ) ∥ ≤ min { T, ∥ s -y ∥} . (b) comes from Lemma 6, which states that ˆ µ 0 ( D ) = ¯ y .

For all s satisfying ∥ s -ˆ µ 0 ( D ) ∥ ≤ T -Z ( D ∗ ) -ω ( D ∗ , k ) ,

in which the last step holds because

Then

in which L is the line connecting ˆ µ 0 ( D ) and ˆ µ 0 ( D ′ ) .

From (45) and (47),

From the condition Z ( D ∗ ) < (1 -2( k +1) /n ) T in Lemma 2,

Hence

From (50) and (53), the second element in the minimum bracket in (50) will not take effect. Hence,

(54) will be useful in the analysis of the second case. From (50) and (53), we can also get

in which the last step uses the bound of ω ( D ∗ , k ) in Lemma 1. Now we have bounded ∥ ˆ µ 0 ( D ) -ˆ µ 0 ( D ′ ) ∥ for the first case.

Case 2: u ∈ I . In other words, D and D ′ differ in a user that is different between D and D ∗ . Without loss of generality, suppose that I = { 1 , . . . , k } , and u = k . D ∗ , D and D ′ can be written as follows:

In order to bound ∥ ˆ µ 0 ( D ) -ˆ µ 0 ( D ′ ) ∥ , we construct a temporary dataset D temp as follows:

Define

Then g 2 (ˆ µ 0 ( D ′ )) = 0 , and

From (33),

Now we bound ∥ ˆ µ 0 ( D ) -ˆ µ 0 ( D ′ ) ∥ . Corresponding to (54), we get

and

Denote L as the line connecting ˆ µ 0 ( D ) and ˆ µ 0 ( D ′ ) . For all s ∈ L , ∥ s -ˆ µ 0 ( D temp ) ∥ ≤ T -Z ( D ∗ ) -ω ( D ∗ , k -1) . Corresponding to (48), for all s ∈ L ,

Therefore

From (62) and (66),

Combine case 1 and 2, we get

The last step comes from the requirement that Z ( D ∗ ) < (1 -2( k +1) /n ) T . The proof is complete.

D Proof of Theorem 1

Two requirements on smooth sensitivity are shown in Definition 4, i.e.

For any D , S ( D ) ≥ LS ( D ) ;

For any neighboring D and D ′ , S ( D ) ≤ e β S ( D ′ ) .

It has been shown in [48] that based on (1) and (2), the final result ˆ µ is ( ϵ, δ ) -DP. In the remainder of this section, we show that both requirements (1) and (2) are satisfied.

For (1) , from Lemma 1 and 2, LS ( D ) ≤ G ( D , 0) if the corresponding conditions are satisfied. If these conditions are not satisfied, since ∥ Clip(ˆ µ 0 ( D ) , R c ) ∥ ≤ R c always hold, the sensitivity can always be bounded by 2 R c . Hence LS ( D ) ≤ G ( D , 0) holds for all D . From (11), G ( D , 0) ≤ S ( D ) . Therefore the requirement (1) is satisfied.

For (2) , it suffices to show that G ( D , k ) ≤ G ( D ′ , k + 1) . From Lemma 2, if ∆( D ′ ) exists, and k +1 ≤ n/ 4 -1 -∆( D ′ ) , then

From the definition of ∆ in (9), since D and D ′ are adjacent, ∆( D ) ≤ ∆( D ′ ) + 1 holds. Therefore, the condition k +1 ≤ n/ 4 -1 -∆( D ′ ) yields k ≤ n/ 4 -1 -∆( D ) . Use Lemma 2 again, we get

If the conditions in Lemma 2 are not satisfied, i.e. ∆( D ′ ) does not exists or k +1 > n/ 4 -1 -∆( D ′ ) , then G ( D ′ , k +1) = 2 R c , thus G ( D , k ) ≤ G ( D ′ , k +1) holds.

Now we have shown that no matter whether the conditions in Lemma 2 holds, we always have G ( D , k ) ≤ G ( D ′ , k +1) . From (11), it can be easily shown that S ( D ) ≤ e β S ( D ′ ) .

E Proof of Theorem 2

In this section, we analyze the practical performance of the estimator for a random dataset.

Notation. Throughout this section, for convenience, we just denote Z = Z ( D ) , Y i = y i ( D ) and ¯ Y = ¯ y ( D ) for simplicity. We use capital letters here since D is random, thus Y i and ¯ Y are random variables.

From Lemma 11 in Section K, for i = 1 , . . . , n ,

Recall that Z = max i ∥ ∥ Y i -¯ Y ∥ ∥ . If ∥ Y i -µ ∥ ≤ t/ 2 for all i , then ∥ ∥ ¯ Y -µ ∥ ∥ ≤ t/ 2 also holds, thus Z ≤ t . Therefore

with C T > 16 √ 2 / 3 . Then with probability at least 1 -1 / ( mn 2 ) , Z ≤ T/ 2 holds, thus

Note that in the statement of Theorem 2, it is required that n > (4 /β ) ln( nR c /T ) . This yields 2 R c e -nβ/ 4 < 2 T/n , and e -βk 2 T/ ( n -k ) decreases with k . Therefore

Define

Then

Recall that

It remains to bound the estimation error. The mean squared error of mean estimation can be decomposed in the following way:

(a) holds because E [ W | D ] = 0 for any D , thus ˆ µ 0 ( D ) -µ and W are uncorrelated. Now we bound these four terms separately.

Bound of I 1 . From Lemma 6, for Z ≤ R ln( nd ) / √ m , Z < T holds, thus ˆ µ 0 ( D ) = ¯ Y . For convenience, denote Y as an i.i.d copy of Y i , i = 1 , . . . , n and X as an i.i.d copy of X ij , i = 1 , . . . , n , j = 1 , . . . , m . Then

Bound of I 2 .

(a) holds because W ∼ N (0 , ( λ /α ) I ) . (b) comes from (77). (c) comes from (3).

Bound of I . From (74),

Bound of I 4 .

2 2 3

Now all four terms in (78) have been bounded. Therefore

The proof is complete.

F Proof of Theorem 3

In this section, following Appendix E, we still denote Y i = y i ( D ) and ¯ Y = ¯ y ( D ) for simplicity. Define

in which ν = √ d/ ( nϵ ) . From the statement of Theorem 3, T > 4 r 0 . From Lemma 13,

Then n out follows a Binomial distribution with parameter ( n, p ) with p ≤ 1 /n . E [ n out ] ≤ 1 . This indicates that with the increase of n , the number of outliers is still bounded by O (1) with high probability.

Lemma 7. ∆( D , k ) ≤ n out for k < n/ 4 -n out -1 .

Proof. From the dataset D , we construct D ∗ as follows. Let D ∗ = { D ∗ 1 , . . . , D ∗ n } such that D ∗ i = D i if ∥ Y i -µ ∥ ≤ r 0 , and D ∗ i is an arbitrary set with mean value µ . In other words, denote Y ∗ i as the mean value in D ∗ i , then

Then d H ( D , D ∗ ) = n out . Let ¯ Y ∗ = (1 /n ) ∑ n i =1 Y ∗ i . Note that

G ( D , k ) can be bounded using Definition 5. Since R c = R , for k ≥ n/ 4 -n out -1 , G ( D , k ) ≤ 2 R . Therefore, for sufficiently large n , if n out < n/ 8 , then

(a) holds since Theorem 3 requires that n > 8(1 + (1 /β ) ln( n/ 2 T )) , thus the e -β ( n/ 4 -n out -1) ≤ 2 T/n .

Define

Then the mean squared error of ˆ µ can be bounded by

From Chernoff inequality,

Thus

which decays faster than any polynomial. Therefore, I 3 and I 4 can be neglected in asymptotic analysis. Now we bound I 1 and I 2 .

Bound of I 1 . Note that

Since Z ( D ∗ ) < T , ˆ µ 0 ( D ∗ ) = ¯ Y ∗ , then

in which the last step uses Lemma 14. From Lemma 1,

The expectation can be bounded by

Therefore

Bound of I 2 .

in which the last step uses (3) and (89).

Combine all terms, the final bound on the mean squared error is

in which the second step uses T > 4 r 0 (from (13) and (84)) and ν = √ d/ ( nϵ ) . The proof is complete.

G Comment on Two-stage Approach

In this section,we provide a brief comment on the two-stage approach WME in [34].

1) For heavy-tailed distributions. We follow the steps of Appendix D.4 in [34].

[34] defined a ( τ, γ ) concentration, which requires that there exists a point c , with probability 1 -γ , ∥ Y i -c ∥ ≤ τ for all i , in which Y i is the i -th user-wise average. We use capital letter here to denote that it is a random variable.

From Appendix D.4 in [34], γ ∼ 1 / ( mn 2 ϵ 2 ) is used. From Lemma 13, let ν = 1 / ( mn 3 ϵ 2 ) , we have

with probability at least 1 -ν , in which Y is i.i.d with Y i , i = 1 , . . . , n . The value of τ can be obtained by taking union bound for all i :

Follow other parts of Appendix D.4 in [34], we can get

Moreover, [34] has already shown the tightness of the mean squared error for the bounded case. For heavy-tailed distributions, following Appendix D.4 in [34], it can also be shown that the bound (102) is tight.

2) For imbalanced users. For simplicity, we focus on the one dimensional problem. With the two-stage approach, for users with different m i , the final estimate will be

in which π [ a,b ] means clipping on [ a, b ] . The sensitivity is determined by the user with maximum m i . Let m max = max i m i . Then the sensitivity if m max ( b -a ) /n .

From [34], now suppose that m i are the same for all i except one that is significantly larger. Then τ ∼ R √ n ln n/N , b -a = 4 τ , the sensitivity scales as ( Rm max /N ) √ n ln n/N . Denote γ 0 = nm max /N as the ratio between maximum m i and average m i . Then the mean squared error induced by privacy mechanism is

H Proof of Lemma 3

Following the proof of Theorem 2 in Appendix E, denote y i = y i ( D ) , y ′ i = y i ( D ′ ) and ¯ y = (1 /n ) ∑ n y i . The proof begins with the following lemma.

Lemma 8. If T i > Z i ( D ) for all i , then ˆ µ 0 ( D ) = ¯ y i =1 .

Proof. We just need to prove that ¯ y is a minimizer of ∑ n i =1 w i ϕ i ( s , y i ) . Since Z i < T i for all i ,

Now we derive the sensitivity. Without loss of generality, we assume that first k users are replaced by others. Define

then ′

Recall that Z = max i ∥ ¯ y -y i ∥ . From Lemma 8, if Z i < T i for all i , then

Then thus

From (107),

For all s satisfying ∥ s -¯ y ∥ ≤ min i ( T i -Z i ) , ∥ s -y i ∥ ≤ T i always holds, thus

Under condition h ( D , 1) ≤ min i ( T i -Z i ( D )) ,

Recall that we have assumed D ′ = { D ′ 1 , . . . , D ′ k , D k +1 , . . . , D n } , i.e. first k users are replaced by others. Actually, the replaced users can be arbitrarily selected from n users. Therefore, a simple generalization yields:

The proof is complete.

I Proof of Lemma 4

Similar to the proof of Lemma 2 in Appendix C, we analyze two cases.

For the first case, u / ∈ I , in which u and I have the same definition as in Appendix 5, without loss of generality, suppose D and D ∗ differ in the first k users, while D and D ′ differ in the ( k +1) -th user. Then

thus

For the second case, following steps in Appendix B,

Taking maximum, we have

J Proof of Theorem 5

Similar to the proof of Theorem 2, denote Z i = Z i ( D ) , Y i = y i ( D ) and ¯ Y = (1 /n ) ∑ n i =1 Y i . Denote

then from the statement in Theorem 5, w i = m i ∧ m c /N c . From the statement of Theorem 5, recall that m c = γN/n .

The proof starts with the following lemma.

Lemma 9. With probability 1 -( n +1) / ( Nn 2 ) , for all i ,

From now on, denote a i as the right hand side of (121).

Recall that Z i = ∥ ∥ ¯ Y -Y i ∥ ∥ . Therefore

The proof of Lemma 9 is complete.

Since Z i ≤ a i < T i for all i , from Lemma 8, ˆ µ 0 ( D ) = ¯ Y . Moreover, we show that ∆( D ) = 0 , in which ∆ is defined in (17). This needs the following lemma.

Lemma 10. For k ≤ n/ (8 γ ) , if Z i ≤ a i for all i , in which a i is the right hand side of (121) , then

Proof. Recall T i in (19), let then T i = A/ √ m i ∧ m c , thus

Now we provide an upper bound of h ( D , k ) :

(a) comes from (16), and that Z ≤ a for all i . (b) uses the condition k < n/ (8 γ ) .

i i Combine (127) and (128), (125) holds. The proof of Lemma 10 is complete.

Now it remains to bound of G ( D , k ) . From Lemma 10, if Z i ≤ a i for all i , then ∆( D ) = 0 , since h ( D ∗ , k 0 ) < min i ( T i -Z i ) . Then for all k ≤ k 0 -1 ,

(a) comes from Definition 6. Note that ∆( D ) = 0 . For k ≤ k 0 -1 , G ( D , k ) is h ( D , 1) if h ( D , 1) ≤ min i ( T i -Z i ( D )) holds, or 2 max i w i T i / ( ∑ n -k -1 i =1 w i ) . It can be shown that the former one is less than the latter, thus (a) holds. For (b), from Assumption 3,

(c) holds because γ ( k +1) /n ≤ γk 0 /n ≤ 1 / 8 .

From (129), the smooth sensitivity can be bounded by

Recall that k 0 = ⌊ n/ (8 γ ) ⌋ . In Theorem 5, it is required that n > 8 γ (1 + (1 / 2 β ) ln( Nn )) , thus k 0 ≥ ln( Nn ) / (2 β ) , and e -βk 0 = 1 / √ Nn . Therefore, the second term in (131) does not dominate. This result indicates that as long as Z i ≤ a i for all i ,

Now we bound the mean squared error. Denote E as the event such that Z i ≤ a i for all i . Then

Bound of I 1 .

Bound of I 2 .

Bound of I 3 .

Bound of I 4 .

I 3 and I 4 converges to zero faster than any polynomial. Therefore

K Common Lemmas

Lemma 11. (Concentration inequality of bounded random vector) Given a random vector X supported at B d ( 0 , R ) , and E [ X ] = µ . X 1 , . . . , X m are m i.i.d copies of X . Denote ¯ X as the sample mean, i.e. ¯ X = (1 /m ) ∑ m j =1 X j . Then

Proof. We use the following lemma.

Lemma 12. ( [88], Lemma 1.6.2) Let U 1 , . . . , U m be independent centered random vectors with dimension d . Assume that each one is uniformly bounded, i.e. for j = 1 , . . . , m ,

and with probability 1 ,

Let Z = ∑ m j =1 U j 2 , and define

Then for all t > 0 ,

Now we prove Lemma 11 based on Lemma 12. Let U j = X j -µ . Since ∥ X j ∥ ≤ R , ∥ µ ∥ ≤ R holds, ∥ X j -µ ∥ ≤ 2 R always holds, and γ ( Z ) = 4 mR 2 . Hence

Hence

Lemma 13. (Concentrated inequality of unbounded random vector) Given a random vector X , E [ X ] = µ , and E [ ∥ X -µ ∥ p ] ≤ M p . X 1 , . . . , X m are m i.i.d copies of X . Denote ¯ X as the sample mean. Then with probability at least 1 -ν ,

Proof. We still use Lemma 12. Pick r > 0 , whose exact value will be determined later. Define

and

Then and

Let Z = ∑ m j =1 U j . Then from (142),

Note that ∥ U j ∥ < r always holds. Therefore, from Lemma 12,

and

Now we have bounded the first term in (150). For the second term in (150),

Therefore, from (150), (153) and (154),

To make the right hand side of (155) to be no more than ν , we let each term to be no more than ν/ 3 . Note that now r has not be determined. Therefore, we let the third term equals ν/ 3 first, thus

Then we let

With (156) and (157), all three terms in the right hand side of (155) will be no more than ν/ 3 . Therefore, with probability at least 1 -ν ,

The proof is complete.

Lemma14. (Bias caused by outliers) Given a random vector X , E [ X ] = µ , and E [ ∥ X -µ ∥ p ] ≤ M p . X 1 , . . . , X m are m i.i.d copies of X . Denote ¯ X as the sample mean, and

for some constant C p that depends only on p .

Proof. From (159),

Thus

Note that E [ ¯ X ] = E [ X ] = µ . Hence

in which then

Therefore

in which the last step uses Lemma 13, and

From Lemma 13, P ( ∥ ∥ ¯ X -µ ∥ ∥ > r k ) ≤ 2 -k ν , thus

From (166) and (160),

thus

for some constant C p that depends only on p . The proof is complete.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main contribution (i.e. proposing a new Huber loss minimization approach which is more suitable to realistic cases, and providing theoretical analysis) has been made clear in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: It is explained at the end of conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Proofs are shown in the appendix, and intuition is provided in the paper.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Experiment details are explained in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Codes are provided.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( ❤/a116/a116♣/a115✿✴✴♥✐♣/a115✳❝❝✴ ♣✉❜❧✐❝✴❣✉✐❞❡/a115✴❈♦❞❡❙✉❜♠✐/a115/a115✐♦♥/a80♦❧✐❝② ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( ❤/a116/a116♣/a115✿ ✴✴♥✐♣/a115✳❝❝✴♣✉❜❧✐❝✴❣✉✐❞❡/a115✴❈♦❞❡❙✉❜♠✐/a115/a115✐♦♥/a80♦❧✐❝② ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Details are provided in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide error bars in figures. Since experiments are repeated 1000 times, the error is relatively low, thus the error bar may be too small to be visible for some cases.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: The experiments need little computational resources.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics ❤/a116/a116♣/a115✿✴✴♥❡✉/a114✐♣/a115✳❝❝✴♣✉❜❧✐❝✴❊/a116❤✐❝/a115●✉✐❞❡❧✐♥❡/a115 ?

Answer: [Yes]

Justification: Our paper does not violate code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper is foundational and theoretical research and not tied to particular applications.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper has no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: This paper uses IPUMS dataset. We cite them in the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, ♣❛♣❡/a114/a115✇✐/a116❤❝♦❞❡✳❝♦♠✴❞❛/a116❛/a115❡/a116/a115 has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper proposes a user-level differentially private mechanism utilizing Huber loss minimization for mean estimation. This approach is robust to heavy-tailed distributions and addresses data imbalance across different users.

STRENGTHS:
- The paper is well-written and easy to follow overall.
- The differentially private version of Huber loss minimization for mean estimation is an interesting contribution.

WEAKNESSES:
- It is unclear whether the convergence results (Theorem 5) are generally applicable to generic $w_i $'s. Using $m_i \land m_c$ seems somewhat unclear. What if the server (in federated learning) wants to compute the mean with $w_i = \frac{m_i}{\sum m_j}$?
- It would be beneficial to provide a detailed analytical comparison with WME, extending beyond Section G.
- An intuitive explanation of $\gamma$ is required. Does $\gamma$ measure imbalance? If so, how?
- The inequality in (10) appears to have a typo.
- $\lambda$ in Theorem 4 needs a definition.
- The figures are too small, making it difficult to check the results. What does the gray line in Figure 2(b) represent?

QUESTIONS:
See Weaknesses.

LIMITATIONS:
Limitation section is adequately provided
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a user-level differential private mean estimation based on minimizing weighted huber loss. The authors conduct theoretical and empirical assessments, showing that the proposed method is more robust to user-wise sample imbalance as well as heavy-tail distributions compared to the Winsorized mean estimator proposed in [0]\n\nSTRENGTHS:\n* The construction of smooth sensitivity and its analysis in both balanced and imbalanced user settings seem novel. \n* The proposed method is thoroughly analyzed, providing both error upper bounds and empirical evaluations.\n\nWEAKNESSES:\n* Some claims in this paper might need more elaboration:\n    > Line 57-58: “To the best of our knowledge, our method is the first attempt to unify robustness and DP at user-level”\n\n    The definition of robustness should be clarified in the paper. Is it referring to robustness against heavy-tailed data, arbitrary outliers, or specific types of attacks? Therefore, it would be beneficial for the authors to explicitly define both robustness and outliers.\n\n    > Line 93-94: “To the best of our knowledge, Huber loss minimization has not been applied to DP”\n    \n    Please see section 3.2.1 in [2]. Additionally, as a side note, gradient clipping for generalized linear loss is well-known to be connected to Huber loss [3].\n\n* As the author also pointed out (line 347), the proposed algorithm requires the sample size of each local user as input to determine the Huber loss parameter $T$, which is a strong assumption.\n\nQUESTIONS:\n* Could the author compare the results in the balanced user case with existing literature, such as [1] and [4]? For example, a result similar to Theorem 2 has been derived in Theorem 4.1 of [1].\n* What is the purpose of tuning $T_i$ in Section 7.2 when Theorem 5 already provides an optimal choice for $T_i$? Does this parameter tuning require an additional privacy budget?\n\nLIMITATIONS:\nThe authors address the limitation of this work in section 8.\n\n\n\n[0] Levy, Daniel, et al. \"Learning with user-level privacy.\" Advances in Neural Information Processing Systems 34 (2021): 12466-12479.\n\n[1] Narayanan, Shyam, Vahab Mirrokni, and Hossein Esfandiari. \"Tight and robust private mean estimation with few users.\" International Conference on Machine Learning. PMLR, 2022.\n\n[2] Avella-Medina, Marco, Casey Bradshaw, and Po-Ling Loh. \"Differentially private inference via noisy optimization.\" The Annals of Statistics 51.5 (2023): 2067-2092.\n\n[3] Song, Shuang, et al. \"Evading the curse of dimensionality in unconstrained private glms.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2021.\n\n[4] Liu, Daogao, and Hilal Asi. \"User-level differentially private stochastic convex optimization: Efficient algorithms with optimal rates.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2024.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nOverall, the method proposed by the authors is interesting and effectively addresses the issue of privacy protection for users with imbalanced data. Compared to existing methods, the authors' approach is more robust and is supported by mathematical proofs.\n\nSTRENGTHS:\n1. The proposed method demonstrates significant innovation and holds substantial practical value. \n\n2. The writing is clear, the arguments are well-structured, and the mathematical foundations are solid.\n\nWEAKNESSES:\nI have a generally positive view of this paper, though I have some questions and concerns that I hope the authors can address thoroughly.\n\nQUESTIONS:\n1. The authors' discussion lacks comprehensiveness. For instance, the statement \"The most effective approach is the two-stage scheme, which finds a small interval first and then gets a refined estimate by clipping samples into the interval\" is not fully substantiated. It would be beneficial to explore whether there are any end-to-end methods or approaches that integrate both stages. If such methods exist, a comparison should be provided. If they do not, an explanation should be offered as to why this research direction has not been pursued by others.\n\n2. The authors should clearly explain why they chose the Huber loss, highlighting its advantages. Furthermore, they should clarify whether their proposed method involves any deep innovation beyond the simple use of the Huber loss, or if it merely employs the Huber loss without additional enhancements.\n\n3. The authors' work bears similarities to \"Private Mean Estimation with Person-Level Differential Privacy.\" The authors should discuss the distinctions between their work and this paper.\n\n4. The authors should clarify whether their method has practical applications and specify the scale of data it can handle. Additionally, a complexity analysis of the method should be conducted.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}