Title: On Theoretical Limits of Learning with Label Differential Privacy
Venue: NeurIPS 2024
Accepted: no
Keywords: label differential privacy
Abstract: Label differential privacy (DP) is designed for learning problems with private labels and public features. Although various methods have been proposed for learning under label DP, the theoretical limits remain unknown. The main challenge is to take infimum over all possible learners with arbitrary model complexity. In this paper, we investigate the fundamental limits of learning with label DP under both central and local models. To overcome the challenge above, we derive new lower bounds on testing errors that are adaptive to the model complexity. Our analyses indicate that $\epsilon$-local label DP only enlarges the sample complexity with respect to $\epsilon$, without affecting the convergence rate over the sample size $N$, except the case with heavy-tailed label. Under the central model, the performance loss due to the privacy mechanism is further weakened, such that the additional sample complexity becomes negligible. Overall, our analysis validates the promise of learning under the label DP from a theoretical perspective and shows that the learning performance can be significantly improved by weakening the DP definition to only labels.

1

Abstract

Label differential privacy (DP) is designed for learning problems with private labels and public features. Although various methods have been proposed for learning under label DP, the theoretical limits remain unknown. The main challenge is to take infimum over all possible learners with arbitrary model complexity. In this paper, we investigate the fundamental limits of learning with label DP under both central and local models. To overcome the challenge above, we derive new lower bounds on testing errors that are adaptive to the model complexity. Our analyses indicate that ϵ -local label DP only enlarges the sample complexity with respect to ϵ , without affecting the convergence rate over the sample size N , except the case with heavy-tailed label. Under the central model, the performance loss due to the privacy mechanism is further weakened, such that the additional sample complexity becomes negligible. Overall, our analysis validates the promise of learning under the label DP from a theoretical perspective and shows that the learning performance can be significantly improved by weakening the DP definition to only labels.

1 Introduction

Many modern machine learning tasks require sensitive training samples that need to be protected from leakage [1]. As a standard approach for privacy protection, differential privacy (DP) [2] has been extensively studied [3-9]. However, the learning performances under original DP definition are usually far from satisfactory [10-13]. Therefore, researchers attempt to design weakened DP requirements, under which the performances can be significantly improved, while still securing sensitive information. Under such background, label DP has emerged in recent years [14], which regards features as public, while only labels are sensitive and need to be protected. Such setting is realistic in many applications, such as computational advertising [15], recommendation systems [16] and medical diagnosis [17]. These tasks usually use some basic demographic information as features, which can be far less sensitive.

Despite various approaches for learning with label DP [14, 18-21], the fundamental limits are still unknown.

An interesting question is:

much accuracy improvement is possible?

By weakening the DP definitions to only labels, how

From an information-theoretic perspective [22], the underlying limits of statistical problems are characterized by the minimax lower bound, which takes

the supremum over all possible distributions from a general class, and infimum over all learners. 30

Deriving minimax lower bounds for learning under the label DP is challenging in two aspects. Firstly, 31

under label DP, each sample has both public (i.e. the feature) and private (i.e. the label) components. 32

Directly applying the methods for original DP [23-27] treats all components as private, and thus does 33

not yield tight results. Secondly, the classical packing method [47] is only suitable for fixed model 34

structures with fixed dimensionality. However, to establish lower bounds, one needs to take infimum 35

over all possible learners with arbitrary model complexity. 36

vector has bounded support, then the minimax rate of classification and regression are

O

(

N

-

2

β

β

+

)

and

2

d

, respectively.

O

(

)

Table 1: Minimax rate of convergence under label differential privacy. d is the dimension of features.

In this paper, we investigate the theoretical limits of classification and regression problems under label 37 DP. Our analysis involves both central and local models. For each problem, we derive the information38 theoretic minimax lower bound of the risk function over a wide class of distributions satisfying the 39 β -Hölder smoothness and the γ -Tsybakov margin assumption [28] (see Assumption 1 for details). 40 The general idea is to convert the problem to multiple hypothesis testing. To overcome the challenges 41 above, we provide a bound of Kullback-Leibler divergence over joint distributions of private and 42 public random variables, which is tighter than the bound between fully private variables. Moreover, 43 under the central model, instead of using the packing method, we develop a new lower bound on the 44 minimum testing error for each pair of hypotheses based on the group privacy property [4], which 45 is suitable for arbitrary model complexity. After deriving minimax lower bounds, we also propose 46 algorithms with matching upper bounds to validate the tightness of our results. 47

48

49

50

51

The results are shown in Table 1, in which the third row refers to the bounds under the original local DP definition, while the fourth row lists the non-private baselines. To the best of our knowledge, minimax rates under central DP have not been established, and are thus not listed here. The main findings are summarized as follows.

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

Under ϵ -local label DP, for classification and regression with bounded label noise, the sample complexity is larger by a factor of O (1 /ϵ 2 ) . However, the convergence rate remains unaffected, which is in clear contrast with the original DP, under which the convergence rate is slower.

Under ϵ -local label DP constraint, for regression with heavy-tailed label noise, the convergence rate of risk over N becomes slower, indicating that heavy-tailed labels increase the difficulty of privacy protection.

Under ϵ -central label DP constraint, the performance loss caused by the privacy mechanism becomes further weakened. The risk only increases by a term that decays faster than the non-private rate, indicating that the additional sample complexity caused by the privacy mechanism becomes negligible with large N .

In general, our analysis provides a theoretical perspective of understanding label DP. The result shows that by weakening the DP definition to protecting labels only, the learning performances can be significantly improved.

2 Related Work

Label DP. Under the local model, labels are randomized before training. The simplest method is randomized response [30]. An important improvement is proposed in [14], called RRWithPrior, which incorporates prior distribution. [19] proposes ALIBI, which further improves randomized response by generating soft labels through Bayesian inference. There are also several methods for regression under label DP [18,31]. Under central label DP, [20] proposes a clustering approach. [19] proposes private aggregation of teacher ensembles (PATE), which is then further improved in [21].

Minimax analysis for public data. Minimax theory provides a rigorous framework for the best possible performance of an algorithm given some assumptions. Classical methods include Le Cam [32], Fano [33] and Assouad [34]. Using these methods, minimax lower bounds have been widely established for both classification and regression problems [28, 29, 35-41]. If the feature

-

β

(

2

γ

β

+1)

+

d

N

)

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

Minimax analysis for private data. Under the local model, [42] finds the relation between label DP and stochastic query. [23] and [24] develop the variants of Le Cam, Fano, and Assouad's method under local DP. Lower bounds are then established for various statistical problems, such as mean estimation [43-46], classification [26] and regression [27]. Under central model, for pure DP, the standard approach is the packing method [47], which is then used in hypothesis testing [48], mean estimation [49,50], and learning of distributions [51-53]. There are also several works on approximate DP, such as [54,55].

This work studies the theoretical limits of label DP, under which each sample is a mixture of public feature and private labels, thus existing methods can not be directly applied here. Under the central model, the minimax analysis becomes more challenging, since the packing method is only suitable for fixed model structures (i.e. the dimensionality of model output is fixed), while we need to find the minimum possible error over all possible learners with arbitrary output dimensions. As a result, the lower bounds of general classification and regression problems have not been established even under the original DP definition. To overcome such challenge, we develop a new approach to bound the error of hypothesis testing (see Lemma 1 in Appendix D).

3 Preliminaries

In this section, we show some necessary definitions, background information, and notations.

3.1 Label DP

To begin with, we review the definition of DP. Suppose the dataset consists of N samples ( x i , y i ) , i = 1 , . . . , N , in which x i ∈ X is the feature vector, while y i ∈ Y ⊂ R d is the label.

Definition 1. (Differential Privacy (DP) [2]) Let ϵ ≥ 0 . A randomized function A : ( X , Y ) N → Θ is ϵ -DP if for any two adjacent datasets D,D ′ ∈ ( X , Y ) N and any S ⊆ Θ ,

in which D and D ′ are adjacent if they differ only on a single sample, including both the feature vector and the label.

In machine learning tasks, the output of A is the model parameters, while the input is the training dataset. Definition 1 requires that both features and labels are privatized. Consider that in some applications, the features may be much less sensitive, the notion of label DP is defined as follows.

Definition 2. (Central label DP) A randomized function A is ϵ -label DP if for any two datasets D and D ′ that differ on the label of only one training sample and any S ⊆ Θ , (1) holds.

Compared with Definition 1, Definition 2 only requires the output to be insensitive to the replacement of a label. Therefore label DP is a weaker requirement. Correspondingly, the local label DP is defined as follows.

Definition 3. (Local label DP) A randomized function M : ( X , Y ) →Z is ϵ -local label DP if

Definition 3 requires that each label is privatized locally before running any machine learning algorithms. It is straightforward to show that local label DP ensures central label DP. To be more precise, we have the following proposition.

Proposition 1. Let z i = M ( x i , y i ) for i = 1 , . . . , N . If A is a function of ( x i , z i ) , i = 1 , . . . , N , then A is ϵ -label DP.

3.2 Risk of Classification and Regression

In supervised learning problems, given N samples ( X i , Y i ) , i = 1 , . . . , N drawn from a common distribution, the task is to learn a function g : X → Y . For a loss function l : Y × Y → R , the goal is to minimize the risk function , which is defined as the expectation of loss function between the predicted value and the ground truth:

The minimum risk among all function g is called Bayes risk, i.e. R ∗ = min g E [ l ( g ( X , Y ))] . In 122 practice, the sample distribution is unknown, and we need to learn g from samples. Therefore, the 123 risk of any practical classifiers is larger than Bayes risk. The gap R -R ∗ is called excess risk, and we 124 hope that R -R ∗ to be as small as possible. Now we discuss classification and regression problems 125 separately. 126

127

128

129

130

1) Classification. For classification problems, the size of Y is finite. For convenience, we denote Y = [ K ] , in which [ K ] := { 1 , . . . , K } . In this paper, we use 0 -1 loss, i.e. l ( ˆ Y , Y ) = 1 ( ˆ Y = Y ) , then R = P ( ˆ Y = Y ) . Define K functions η 1 , . . . , η K as the conditional class probabilities:

̸

Under this setting, the Bayes optimal classifier and the corresponding Bayes risk is

̸

2) Regression. Now we consider the case with Y having infinite size. We use ℓ 2 loss in this paper, i.e. 131 l ( ˆ Y , Y ) = ( ˆ Y -Y ) 2 . Then the Bayes risk is 132

Then the following proposition gives a bound of the excess risk for classification and regression 133 problems. 134

135

Proposition 2. For any classifier c : X → [ K ] , the excess risk of classification is bounded by

For any regression estimate ˆ η : X → Y , the excess risk of regression is bounded by 136

The proof of Proposition 2 is shown in Appendix A. Finally, we state some basic assumptions that will be used throughout this paper.

Assumption 1. There exists some constants L , β , C T , γ , c , D and θ ∈ (0 , 1] such that

(a) For all j ∈ [ K ] and any x , x ′ , | η j ( x ) -η j ( x ′ ) | ≤ L ∥ x -x ′ ∥ β ;

(b) For any t > 0 , P (0 < η ∗ ( X ) -η s ( X ) < t ) ≤ C T t γ , in which η s ( x ) is the second largest one among { η 1 ( x ) , . . . , η K ( x ) } ;

(c) The feature vector X has a probability density function (pdf) f which is bounded from below, i.e. f ( x ) ≥ c ;

(d) For all r < D , V r ( x ) ≥ θv d r d , in which V r ( x ) is the volume (Lebesgue measure) of B ( x , r ) ∩X , v d is the volume of a unit ball.

Assumption 1 (a) requires that all η j are Hölder continuous. This condition is common in literatures about nonparametric statistics [28]. (b) is generalized from the Tsybakov noise assumption for binary classification, which is commonly used in many existing works in the field of both nonparametric classification [29, 37, 40, 41] and differential privacy [26, 27]. If K = 2 , then η ∗ and η s refer to the larger and smaller class conditional probability, respectively. An intuitive understanding of (b) is that in the majority of the support, the maximum value among { η 1 ( x ) , . . . , η K ( x ) } should have some gap to the second largest one. With sufficiently large sample size and model complexity, assumption (b) ensures that for test samples within the majority of the support X , the algorithm is highly likely to correctly identify the class with the maximum conditional probability. Therefore, in (b), we only care about η ∗ ( x ) and η s ( x ) , while other classes with small conditional probabilities can be ignored. (c) is usually called "strong density assumption" in existing works [39,40], which is quite strong. It is possible to relax this assumption so that the theoretical analysis becomes suitable for general cases. However, we do not focus on such generalization in this paper. Assumption (d) prevents the corner of the support X from being too sharp. In the remainder of this section, denote F cls as the set of all pairs ( f, η ) satisfying Assumption 1.

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

̸

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

4 Classification

In this section, we derive the upper and lower bounds of learning under central and local label DP, respectively.

4.1 Local Label DP

1) Lower bound. The following theorem shows the minimax lower bound, which characterizes the theoretical limit.

Theorem 1. Denote M ϵ as the set of all privacy mechanisms satisfying ϵ -local label DP (Definition 3). Then

Proof. (Outline) It suffices to derive (10) with K = 2 . We convert the problem into multiple binary hypothesis testing problems. In particular, we divide the support into G bins. For some of them, we construct two opposite hypotheses such that they are statistically not distinguishable. Our proof uses some techniques in local DP [24] and some classical minimax theory [28]. The detailed proof is shown in Appendix B.

In Theorem 1, (10) takes supremum over all joint distributions of ( X , Y ) , and infimum over all classifiers and privacy mechanisms satisfying ϵ -local label DP.

2) Upper bound. We then show that the bound (10) is achievable. Let the privacy mechanism M ( x , y ) outputs a K dimensional vector, with each component being either 0 or 1 , such that

̸

and P ( M ( x , y )( j ) = 0) = 1 -P ( M ( x , y )( j ) = 1) , in which M ( x , y )( j ) is the j -th component of M ( x , y ) . For N random training samples ( X i , Y i ) , let Z i = M ( X i , Y i ) , and correspondingly, Z i ( j ) is the j -th component of Z i .

Divide the support X into G bins, named B 1 , . . . , B G , such that the length of each bin is h . B 1 , . . . , B G are disjoint, and these bins form a covering of X , i.e. X ⊂ ∪ G l =1 B l . Then calculate

The classification within the l -th bin is

such that the the prediction given x is c ( x ) = c l for all x ∈ B l . The next theorem shows the privacy guarantee, as well as the bound of the excess risk.

Theorem 2. The privacy mechanism M is ϵ -local label DP. Moreover, under Assumption 1, with h ∼ ( N ( ϵ 2 ∧ 1) / ln K ) -1 2 β + d , the excess risk of the classifier described above can be upper bounded as follows:

Proof. (Outline) For privacy guarantee, we need to show that (11) is ϵ -local label DP: 191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

According to Definition 3, M is ϵ -local label DP. For the performance guarantee (14), according to Proposition 2, we need to bound η ∗ ( x ) -E [ η c ( x ) ( x )] for each x . If η ∗ ( x ) -η s ( x ) is large, then with high probability, c ( x ) = c ∗ ( x ) , and then η ∗ ( x ) = η c ( x ) ( x ) . Thus we mainly consider the case with small η ∗ ( x ) -η s ( x ) . The details of proof are shown in Appendix C.

The lower bound (10) and the upper bound (14) match up to a logarithm factor, indicating that the results are tight. Now we comment on the results.

Remark 1. 1) Comparison with non-private bound. The classical minimax lower bound for nonprivate classification problem is N -β ( γ +1) 2 β + d . Therefore, the lower bound (10) reaches the non-private bound with ϵ ≳ 1 . With small ϵ , N training samples with privatized labels roughly equals Nϵ 2 non-privatized samples in terms of performance.

Comparison with local DP that protects both features and labels. In this case, the optimal excess risk is ( Nϵ 2 ) -β ( γ +1) / (2 β +2 d ) ∨ N -β ( γ +1) / (2 β + d ) , which is worse than the right hand side of (10) . Such result indicates that compared with classical DP, label DP incurs significantly weaker performance loss.

Comparison with other baseline methods. If we use the randomized response method instead of the privacy mechanism (11) , then the performance decreases sharply with the number of classes K . Several methods have been proposed to improve the randomized response method, such as RRWithPrior [14] and ALIBI [19]. However, these methods are not guaranteed in theory.

4.2 Central Label DP

1) Lower bound. The following theorem shows the minimax lower bound under the central label DP. Theorem 3. Denote A ϵ as the set of all learning algorithms satisfying ϵ -label DP (Definition 2). Then

Proof. (Outline) Lower bounds under central DP are usually constructed by packing method [47], which works for fixed output dimensions. However, to achieve a desirable bias and variance tradeoff, the model complexity needs to increase with N . In our proof, we still divide the support into G bins and construct two hypotheses for each bin, but we develop a new tool (see Lemma 1) to give a lower bound of the minimum error of hypothesis testing. We then use the group privacy property [4] to get the overall lower bound. The details can be found in Appendix D.

2) Upper bound. Now we show that (16) is achievable. Similar to the local label DP problem, now divide the support into G bins, such that the length of each bin is h . Now the classification within the l -th bin follows a exponential mechanism [56]:

in which n lj = ∑ N i =1 1 ( X i ∈ B l , Y i = j ) . Then let c ( x ) = c l for x ∈ B l . The excess risk is bounded in the next theorem.

Theorem 4. The privacy mechanism (17) is ϵ -label DP. Moreover, under Assumption 1, if h scales as h ∼ (ln K/ϵN ) 1 β + d +(ln K/N ) 1 2 β + d , then the excess risk can be bounded as follows:

Proof. (Outline) The privacy guarantee of the exponential mechanism has been analyzed in [4]. 227 Following these existing analyses, it can be shown that (17) is ϵ -label DP. It remains to show (18). 228 Note that if η ∗ ( x ) -η s ( x ) is large, then the difference between the largest and the second largest 229 one from { n lj | j = 1 , . . . , K } will also be large. From (17), the following inequality holds with high 230 probability: c l = arg max j n lj = arg max j η j ( x ) = c ∗ ( x ) , which means that the classifier makes 231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

optimal prediction. Hence we mainly consider the case with small η ∗ ( x ) -η s ( x ) . The details of the proof can be found in Appendix E.

The upper and lower bounds match up to logarithmic factors. In (18), the first term is just the non-private convergence rate, while the second term ( ϵN ) -β ( γ +1) β + d can be regarded as the additional risk caused by the privacy mechanism. It decays faster with N compared with the first term, thus the additional performance loss caused by the privacy mechanism becomes negligible as N increases. This result is crucially different from the local model, under which the privacy mechanism always induces higher sample complexity by a factor of O (1 / ( ϵ 2 ∧ 1)) .

5 Regression with Bounded Noise

Now we analyze the theoretical limits of regression problems under local and central label DP. Throughout this section, we assume that the label is restricted within a bounded interval.

Assumption 1 remains the same here. In the remainder of this section, denote F reg 1 as the set of ( f, η ) that satisfies Assumption 1 and 2.

5.1 Local Label DP

1) Lower bound. Theorem 5 shows the minimax lower bound.

Theorem 5. Denote M ϵ as the set of all privacy mechanisms satisfying ϵ -label DP. Then

The proof of Theorem 5 is similar to that of Theorem 1, except for some details in hypotheses construction and the final bound of excess risk. The details are shown in Appendix F.

2) Upper bound. The privacy mechanism is Z = Y + W , in which W ∼ Lap(2 T/ϵ ) . Then the privacy mechanism satisfies ϵ -label DP. In this case, the real regression function η ( x ) can be estimated using the nearest neighbor approach. Let

in which N k ( x ) is the set of k nearest neighbors of x among X 1 , . . . , X N .

Theorem 6. The method described above is ϵ -local label DP . Moreover, with k ∼ N 2 β d +2 β ( ϵ ∧ 1) -2 d d +2 β , then under Assumption 1 and 2,

Proof. (Outline) Since | Y | < T , W ∼ Lap(2 T/ϵ ) , it is obvious that Z = Y + W is ϵ -local label DP. For the performance (21), the bias can be bounded by the k nearest neighbor distances based on Assumption 1(a). The variance of ˆ η ( x ) scales inversely with k . An appropriate k can be selected to achieve a good tradeoff between bias and variance. The details are shown in Appendix G.

From standard minimax analysis on regression problems, the non-private convergence rate is N -2 β/ ( d +2 β ) . From Theorem 5 and 6, the privatization process makes sample complexity larger by a O (1 /ϵ 2 ) factor.

5.2 Central Label DP

1) Lower bound. The following theorem shows the minimax lower bound.

Theorem 7. Let A ϵ be the set of all algorithms satisfying ϵ -central DP. Then

2) Upper bound. For each bin B l , let n l = ∑ N i =1 1 ( X i ∈ B l ) be the number of samples in B l . If 267 n l > 0 , then 268

in which W l ∼ Lap(2 / ( n l ϵ )) . If n l = 0 , i.e. no sample falls in B l , then just let ˆ η l = 0 . For all x ∈ B l , let ˆ η ( x ) = ˆ η l . The excess risk can be bounded with the following theorem.

Theorem 8. (23) is ϵ -label DP. Moreover, under Assumption 1 and 2, if h scales as h ∼ N -1 2 β + d + ( ϵN ) -1 d + β , then the excess risk is bounded by

The upper and lower bounds match, indicating that the results are tight. Again, the second term in (24) converges faster than the first one with respect to N , the performance loss caused by privacy constraints becomes negligible as N increases.

6 Regression with Heavy-tailed Noise

In this section, we consider the case such that the noise has tails. We make the following assumption. Assumption 3. For all x ∈ X , E [ | Y | p | X = x ] ≤ M p for some p ≥ 2 .

Instead of requiring | Y | < T for some T , now we only assume that the p -th order moment is bounded. For non-private cases, given fixed noise variance, the tail does not affect the mean squared error of regression. As a result, as long as p ≥ 2 , the convergence rate of regression risk is the same as the case with bounded noise. However, the label DP requires the output to be insensitive to the worst case replacement of labels, which can be harder if the noise has tails. To achieve ϵ -DP, the clipping radius decreases with ϵ , thus the noise strength needs to grow faster than O (1 /ϵ ) . As a result, the convergence rate becomes slower than the non-private case. In the remainder of this section, denote F reg 2 as the set of ( f, η ) that satisfies Assumption 1 and 3.

6.1 Local Label DP

1) Lower bound. In earlier sections about classification and regression with bounded noise, the impact of privacy mechanisms is only a polynomial factor on ϵ , while the convergence rate of excess risk with respect to N is not changed. However, this rule no longer holds when the noise has heavy tails.

Theorem 9. Denote M ϵ as the set of all privacy mechanisms satisfying ϵ -label DP. Then for small ϵ ,

2) Upper bound. Since now the noise has unbounded distribution, without preprocessing, the sensitivity is unbounded, thus simply adding noise to Y can no longer protect the privacy. Therefore, a solution is to clip Y into [ -T, T ] , and add noise proportional to T/ϵ to achieve ϵ -local label DP. Such truncation will inevitably introduce some bias. To achieve a tradeoff between clipping bias and sensitivity, the value of T needs to be tuned carefully. Based on such intuition, the method is precisely stated as follows. Let Z i = Y Ti + W i , in which Y Ti is the truncation of Y i , i.e. Y Ti = ( Y i ∧ T ) ∨ ( -T ) , and W ∼ Lap(2 T/ϵ ) . The result is shown in the next theorem.

Theorem 10. The method above is ϵ -local label DP . Moreover, with k ∼ ( Nϵ 2 ) 2 pβ 2 pβ + d ( p -1) ∨ N 2 β 2 β + d , and T ∼ ( kϵ 2 ) 1 2 p , the risk is bounded by

Proof. (Outline) It can be shown that the clipping bias scales as T 2(1 -p ) . To meet the ϵ -label DP, an additional error that scales as T/ϵ is needed. By averaging over k nearest neighbors, the variance caused by noise W scales with T 2 / ( kϵ 2 ) . From standard analysis on nearest neighbor methods [29], the non-private mean squared error scales as 1 /k +( k/N ) 2 β/d . Put all these terms together, Theorem 10 can be proved. Details can be found in Appendix K.

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

With the limit of p →∞ , the problem reduces to the case with bounded noise, and the growth rate of k and the convergence rate of risk are the same as those in Theorem 6. For finite p , 2 β ( p -1) / (2 pβ + d ( p -1)) < 2 β/ (2 β + d ) , thus the convergence rate becomes slower due to the privacy mechanism.

6.2 Central Label DP

1) Lower bound. The minimax lower bound is shown in Theorem 11.

Theorem 11. The minimax lower bound is

2) Upper bound. Now we derive the upper bound. To restrict the sensitivity, instead of estimating with (23) directly, now we calculate an average of clipped label values:

in which W l ∼ Lap(2 T/ ( n l ϵ )) . Then for all x ∈ B l , let ˆ η ( x ) = ˆ η l . The following theorem bounds the excess risk.

Theorem 12. (28) is ϵ -label DP. Moreover, under Assumption 1 and 3, if h and T scales as h ∼ N -1 2 β + d +( ϵN ) -1 pβ + d ( p -1) , and T ∼ ( ϵNh d ) 1 /p , then the excess risk can be bounded by

The proof of Theorem 11 and 12 follow that of Theorem 7 and 8. The details are shown in Appendix L and M respectively. With p = 2 , the right hand side of (29) becomes ( ϵ ∧ 1) -2 β 2 β + d , indicating that the privacy constraint blows up the sample complexity by a constant factor. With larger p , the second term in (29) becomes negligible compared with the first one.

The theoretical analyses in this section are summarized as follows. In general, with fixed noise variance, if the label noise is heavy-tailed, while the non-private convergence rates remain unaffected, the additional risk caused by privacy mechanisms becomes significantly higher, indicating the difficulty of privacy protection for heavy-tailed distributions.

7 Conclusion

In this paper, we have derived the minimax lower bounds of learning under label DP for both central and local models. Furthermore, we propose methods whose upper bounds match these lower bounds. The results indicate the theoretical limits of learning under the label DP. From these results, it is discovered that under local label DP constraints, the sample complexity blows up by a factor of at least O (1 /ϵ 2 ) . Under central label DP requirements, the additional error caused by privacy mechanisms is significantly smaller. Finally, it is shown that for regression problem with heavy-tailed label distribution, the additional risk induced by privacy requirement becomes inevitably higher.

Limitations: The limitations of our work include the following aspects. Some assumptions can be weakened. For example, current analysis assumes that feature distributions have bounded supports, which may be extended to the unbounded case. One can let the bin splitting and nearest neighbor method be adaptive in the tails of features, such as [41]. Moreover, the bounds derived in this paper require that samples increase exponentially with dimensionality. However, in practice, the performance of learning under the label DP can be quite well even in high dimensions. The discrepancy can be explained by the fact that the minimax lower bound considers the worst-case distribution over a wide range of distributions. However, in most realistic cases, the distributions satisfy significantly better properties. A better modeling is to assume that these samples lie on a low dimensional manifold [57,58]. In this case, it is possible to achieve a much better convergence rate. Finally, it is not sure whether approximate DP (i.e. ( ϵ, δ ) -DP) can improve the convergence rates.

345

346

347

348

349

350

351

352

353

References

Rao, B., J. Zhang, D. Wu, et al. Privacy inference attack and defense in centralized and federated learning: A comprehensive survey. IEEE Transactions on Artificial Intelligence , 2024.

Dwork, C., F. McSherry, K. Nissim, et al. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3 , pages 265-284. Springer, 2006.

Abadi, M., A. Chu, I. Goodfellow, et al. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security , pages 308-318. 2016.

Dwork, C., A. Roth, et al. The algorithmic foundations of differential privacy. Foundations and 354 Trends® in Theoretical Computer Science , 9(3-4):211-407, 2014. 355

356

357

358

Bassily, R., A. Smith, A. Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 2014 IEEE 55th Annual Symposium on Foundations of Computer Science , pages 464-473. IEEE, 2014.

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

Bassily, R., V. Feldman, K. Talwar, et al. Private stochastic convex optimization with optimal rates. Advances in Neural Information Processing Systems , 32, 2019.

Wang, D., H. Xiao, S. Devadas, et al. On differentially private stochastic convex optimization with heavy-tailed data. In International Conference on Machine Learning , pages 10081-10091. PMLR, 2020.

Asi, H., V. Feldman, T. Koren, et al. Private stochastic convex optimization: Optimal rates in l1 geometry. In International Conference on Machine Learning , pages 393-403. PMLR, 2021.

Das, R., S. Kale, Z. Xu, et al. Beyond uniform lipschitz condition in differentially private optimization. In International Conference on Machine Learning , pages 7066-7101. PMLR, 2023.

Tramer, F., D. Boneh. Differentially private learning needs better features (or much more data). In International Conference on Learning Representations . 2021.

Bu, Z., J. Mao, S. Xu. Scalable and efficient training of large convolutional neural networks with differential privacy. Advances in Neural Information Processing Systems , 35:38305-38318, 2022.

De, S., L. Berrada, J. Hayes, et al. Unlocking high-accuracy differentially private image classification through scale. arXiv preprint arXiv:2204.13650 , 2022.

Wei, J., E. Bao, X. Xiao, et al. Dpis: An enhanced mechanism for differentially private sgd with importance sampling. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security , pages 2885-2899. 2022.

Ghazi, B., N. Golowich, R. Kumar, et al. Deep learning with label differential privacy. Advances in Neural Information Processing Systems , 34:27131-27145, 2021.

McMahan, H. B., G. Holt, D. Sculley, et al. Ad click prediction: a view from the trenches. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 1222-1230. 2013.

McSherry, F., I. Mironov. Differentially private recommender systems: Building privacy into the netflix prize contenders. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 627-636. 2009.

Bussone, A., B. Kasadha, S. Stumpf, et al. Trust, identity, privacy, and security considerations for designing a peer data sharing platform between people living with hiv. Proceedings of the ACM on Human-Computer Interaction , 4(CSCW2):1-27, 2020.

Ghazi, B., P. Kamath, R. Kumar, et al. Regression with label differential privacy. In The 390 Eleventh International Conference on Learning Representations . 2022. 391

Malek Esmaeili, M., I. Mironov, K. Prasad, et al. Antipodes of label differential privacy: Pate 392 and alibi. Advances in Neural Information Processing Systems , 34:6934-6945, 2021. 393

Esfandiari, H., V. Mirrokni, U. Syed, et al. Label differential privacy via clustering. In 394 International Conference on Artificial Intelligence and Statistics , pages 7055-7075. PMLR, 395 2022. 396

Tang, X., M. Nasr, S. Mahloujifar, et al. Machine learning with differentially private labels: 397 Mechanisms and frameworks. Proceedings on Privacy Enhancing Technologies , 2022. 398

Cover, T. M. Elements of information theory . John Wiley & Sons, 1999. 399

Duchi, J. C., M. I. Jordan, M. J. Wainwright. Local privacy and statistical minimax rates. In 400 2013 IEEE 54th Annual Symposium on Foundations of Computer Science , pages 429-438. 401 IEEE, 2013. 402

-. Minimax optimal procedures for locally private estimation. Journal of the American 403 Statistical Association , 113(521):182-201, 2018. 404

Gopi, S., G. Kamath, J. Kulkarni, et al. Locally private hypothesis selection. In Conference on 405 Learning Theory , pages 1785-1816. PMLR, 2020. 406

Berrett, T., C. Butucea. Classification under local differential privacy. arXiv preprint 407 arXiv:1912.04629 , 2019. 408

Berrett, T. B., L. Györfi, H. Walk. Strongly universally consistent nonparametric regression and 409 classification with privatised data. Electronic Journal of Statistics , 15:2430-2453, 2021. 410

Tsybakov, A. B. Introduction to Nonparametric Estimation . 2009. 411

Audibert, J.-Y., A. B. Tsybakov. Fast learning rates for plug-in classifiers. Annals of Statistics , 412 2007. 413

Warner, S. L. Randomized response: A survey technique for eliminating evasive answer bias. 414 Journal of the American Statistical Association , 60(309):63-69, 1965. 415

Badanidiyuru Varadaraja, A., B. Ghazi, P. Kamath, et al. Optimal unbiased randomizers for 416 regression with label differential privacy. Advances in Neural Information Processing Systems , 417 36, 2023. 418

LeCam, L. Convergence of estimates under dimensionality restrictions. The Annals of Statistics , 419 pages 38-53, 1973. 420

Verdú, S., et al. Generalizing the fano inequality. IEEE Transactions on Information Theory , 421 40(4):1247-1251, 1994. 422

Assouad, P. Deux remarques sur l'estimation. Comptes rendus des séances de l'Académie des 423 sciences. Série 1, Mathématique , 296(23):1021-1024, 1983. 424

Yang, Y. Minimax nonparametric classification. i. rates of convergence. IEEE Transactions on 425 Information Theory , 45(7):2271-2284, 1999. 426

-. Minimax nonparametric classification. ii. model selection for adaptation. IEEE Transactions 427 on Information Theory , 45(7):2285-2292, 1999. 428

Chaudhuri, K., S. Dasgupta. Rates of convergence for nearest neighbor classification. Advances 429 in Neural Information Processing Systems , 27, 2014. 430

Yang, Y., S. T. Tokdar. Minimax-optimal nonparametric regression in high dimensions. The 431 Annals of Statistics , pages 652-674, 2015. 432

Döring, M., L. Györfi, H. Walk. Rate of convergence of k -nearest-neighbor classification rule. 433 Journal of Machine Learning Research , 18(227):1-16, 2018. 434

Gadat, S., T. Klein, C. Marteau. Classification in general finite dimensional spaces with the 435 k-nearest neighbor rule. Annals of Statistics , 2016. 436

Zhao, P., L. Lai. Minimax rate optimal adaptive nearest neighbor classification and regression. 437 IEEE Transactions on Information Theory , 67(5):3155-3182, 2021. 438

Kasiviswanathan, S. P., H. K. Lee, K. Nissim, et al. What can we learn privately? SIAM Journal 439 on Computing , 40(3):793-826, 2011. 440

Li, M., T. B. Berrett, Y . Yu. On robustness and local differential privacy. The Annals of Statistics , 441 51(2):717-737, 2023. 442

Feldman, V., T. Koren, K. Talwar. Private stochastic convex optimization: optimal rates in linear 443 time. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing , 444 pages 439-449. 2020. 445

Duchi, J., R. Rogers. Lower bounds for locally private estimation via communication complexity. 446

In Conference on Learning Theory , pages 1161-1191. PMLR, 2019. 447

Huang, Z., Y. Liang, K. Yi. Instance-optimal mean estimation under differential privacy. 448 Advances in Neural Information Processing Systems , 34:25993-26004, 2021. 449

Hardt, M., K. Talwar. On the geometry of differential privacy. In Proceedings of the forty-second 450 ACM symposium on Theory of computing , pages 705-714. 2010. 451

Bun, M., G. Kamath, T. Steinke, et al. Private hypothesis selection. Advances in Neural 452 Information Processing Systems , 32, 2019. 453

Narayanan, S. Better and simpler lower bounds for differentially private statistical estimation. 454 arXiv preprint arXiv:2310.06289 , 2023. 455

Kamath, G., V. Singhal, J. Ullman. Private mean estimation of heavy-tailed distributions. In 456 Conference on Learning Theory , pages 2204-2235. PMLR, 2020. 457

Kamath, G., J. Li, V. Singhal, et al. Privately learning high-dimensional distributions. In 458 Conference on Learning Theory , pages 1853-1902. PMLR, 2019. 459

460

461

462

Alabi, D., P. K. Kothari, P. Tankala, et al. Privately estimating a gaussian: Efficient, robust, and optimal. In Proceedings of the 55th Annual ACM Symposium on Theory of Computing , pages 483-496. 2023.

Arbas, J., H. Ashtiani, C. Liaw. Polynomial time and private learning of unbounded gaussian 463 mixture models. In International Conference on Machine Learning , pages 1018-1040. 2023. 464

465

466

467

Bun, M., J. Ullman, S. Vadhan. Fingerprinting codes and the price of approximate differential privacy. In Proceedings of the forty-sixth annual ACM symposium on Theory of computing , pages 1-10. 2014.

468

469

470

Kamath, G., A. Mouzakis, V. Singhal. New lower bounds for private estimation and a generalized fingerprinting lemma. Advances in neural information processing systems , 35:24405-24418, 2022.

McSherry, F., K. Talwar. Mechanism design via differential privacy. In 48th Annual IEEE 471 Symposium on Foundations of Computer Science (FOCS'07) , pages 94-103. IEEE, 2007. 472

Kpotufe, S. k-nn regression adapts to local intrinsic dimension. Advances in neural information 473 processing systems , 24, 2011. 474

Carter, K. M., R. Raich, A. O. Hero III. On local intrinsic dimension estimation and its 475 applications. IEEE Transactions on Signal Processing , 58(2):650-663, 2009. 476

A Proof of Proposition 2 477

From (5) and (6), the Bayes risk is 478

̸

The risk of classifier c is 479

From (31) and (6), 480

The proof is complete. 481

482

483

484

485

486

487

488

489

B Proof of Theorem 1

In this section, we prove the minimax lower bound of multi-class classification. The problem with K classes with K > 2 is inherently harder than that with K = 2 . Therefore, we just need to prove the lower bound for binary classification, in which Y = { 1 , 2 } . Let

Since η 1 ( x ) + η 2 ( x ) = 1 always holds, we have

Therefore, η ( x ) captures the conditional distribution of Y given x .

Find G disjoint cubes B 1 , . . . , B G ⊂ X , such that the length of each cube is h . Denote c 1 , . . . , c G as the centers of these cubes. Let ϕ ( u ) be some function supported at [ -1 / 2 , 1 / 2] d , such that

Let f ( x ) = c over x ∈ X . For v ∈ V := {-1 , 1 } m , let 490

It can be proved that if for some constant C M , 491

then for any η = η v , η 1 and η 2 satisfies Assumption 1(b). Denote 492

Then the excess risk is bounded by 493

̸

If ˆ v k = v k , then from (38), 494

̸

̸

̸

̸

Therefore 495

Hence 496

̸

in which ρ H denotes the Hamming distance. Then 497

Define 498

in which P ( X,Z ) 1: N | v denotes the distribution of ( X 1 , Z 1 ) , . . . , ( X N , Z N ) with η = η v . D KL 499 denotes the Kullback-Leibler divergence. Then from [28], Theorem 2.12(iv), 500

̸

It remains to bound δ . Without loss of generality, suppose v 1 = v ′ 1 , and v i = v ′ i for i = 1 . Then 501

̸

In (a), P X,Z | v denotes the distribution of a single sample with privatized label ( X,Z ) , with η = η v . 502 In (b), P denotes the conditional distribution of Z given X = x , with η = η v . (c) uses [24], 503

Z | X = x , v Theorem 1. In (d), ∥ ϕ ∥ 2 2 = ∫ ϕ 2 ( u ) d u , which is a constant. Moreover, 504

For (a), note that Z is generated from Y . From data processing inequality, (a) holds. For (b), without 505 loss of generality, suppose that v 1 = 1 , thus η v ( x ) ≥ 0 in B 1 . Then ln(1 + η v ( x )) ≤ η v ( x ) . From 506 (35) and (36), | η v ( x ) | ≤ 1 / 2 . Therefore, -ln(1 -η v ( x )) ≤ 2 η v ( x ) . Therefore (b) holds. 507

̸

From (46) and (47), 508

Let 509

Hence 511

The proof is complete. 512

C Proof of Theorem 2 513

Denote 514

and for Z = M ( X , Y ) , let 515

as the number of training samples whose feature vectors fall in B l , and 516

Recall (12) that defines S lj . From Hoeffding's inequality, 517

in which X 1: N denotes X 1 , . . . , X N . 518

Define 519

and 520

Then δ ≲ 1 . From (45), with m ∼ h γβ -d , 510

Now we bound P ( v ∗ l -v lc l > t ) , in which c l is defined in (13). c l can be viewed as the prediction at 521 the l -th bin. We would like to show that the even if the prediction is wrong, the value (i.e. conditional 522 probability) of the predicted class is close to the ground truth. v ∗ l -v lc l > t only if ∃ j , v ∗ l -v lj > t , 523 and S lj > S lc ∗ l . Therefore either S lj -n l v lj > t/ 2 or S lc ∗ l -n l v ∗ l > t/ 2 holds. Hence 524

Define 525

Then 526

Define 532

then 533

Take integration over cube B l , we get 534

In (a), we use the inequality 527

Now we bound the excess risk. 528

We need to bound ∫ B l ( η ∗ ( x ) -E [ η c ( x ) ( x )] ) f ( x ) d x for each l . From Assumption 1(a), for any 529 x , x ′ ∈ B l , the distance is bounded by ∥ x -x ′ ∥ ≤ √ dL . Thus 530

in which L d is defined as L d := L d . From (63) and (53), 531

in which p ( B l ) = P ( X ∈ B l ) is the probability mass of B l . Moreover, define 535

in which the ˜ η s is the second largest value of ˜ η j among j = 1 , . . . , K , which follows the definition 537 of η s . 538

If ∆ l > 0 , then c ∗ ( x ) is the same over B l . Then either v ∗ l -v lc l = 0 or v ∗ l -v lc l ≥ ∆ l holds. Hence 539

Take expectation over X 1: N , we get 540

Define 541

and 542

From Assumption 1(c), p ( B l ) ≥ cNh d . Therefore, from (67) and (71) 543

It remains to bound ∑ G l =1 min { a l , b l } . Note that for all x ∈ B l , η ∗ ( x ) -η s ( x ) ≤ ∆ l + 2 L d h β . 544 Thus 545

and 536

Let 546

and 547

Then 548

549

It is obvious that there exists a finite constant C ′ < ∞ that depends on γ , such that 550

Combine (74), (79) and (82), 552

To minimize the overall excess risk, let 553

Compare to the simple random response method, the bin splitting avoids the polynomial decrease 555 over K . 556

Therefore 551

then 554

D Proof of Theorem 3 557

We still divide the support as the local label DP setting, except that the value of h is different, which 558 will be specified later in this section. Note that (42) still holds here. Let V takes values from 559 {-1 , 1 } m randomly with equal probability, and V k is the k -th element. Then η V ( x ) is a random 560 function. The corresponding random output of hypothesis testing is denoted as ˆ V k , which is calculated 561 by (38). Then 562

̸

̸

in which the last step holds since ˆ V k for different k are calculated independently. 563

̸

It remains to give a lower bound of P ( ˆ V k = V k ) . Denote n k as the number of samples falling in B k , 564 ¯ Y k as the average label values in B k : 565

in which the last step comes from (36). Then 567

in which X 1: N means X 1 , . . . , X N . We then show the following lemma.

Lemma 1. If 0 ≤ t ≤ ln 2 / ( ϵn k ) , and n k t is an integer, then

568

569

Proof. Construct D ′ by changing the label values of l = n k t items from these n k samples falling in 570 B k , from -1 to 1 . Then the average label values in B k is denoted as ¯ Y ′ k after such replacement. ˆ V k 571 also becomes ˆ V ′ k . Then from the ϵ -label DP requirement, 572

Moreover, define 566

̸

in which (a) uses the group privacy property. The Hamming distance between D and D ′ is l , thus the 573 ratio of probability between D and D ′ is within [ e -lϵ , e lϵ ] . (b) holds because the algorithm does not 574 change after changing D to D ′ . Similarly, 575

576

577

578

579

580

Then (91) can be shown by adding up (92) and (93).

Now we use Lemma 1 to bound the excess risk. With sufficiently large n k , ˆ Y k will be close to Gaussian distribution with mean a k . To be more rigorous, by Berry-Esseen theorem [ ? ], for some absolute constant C E ,

We first analyze cubes with 581

Similarly,

Therefore 589

Under condition (96), the right hand side of (94) and (95) are at least 1 / 4 . Therefore 582

̸

From (86), 583

From (35), (89) and (87), a k ≤ h β . Therefore 584

Recall that each cube has probability mass ch d . Select h such that 585

From Chernoff inequality, 16 C 2 E < n k < ln 2 / ( ϵh β ) holds with high probability. (100) yields 586

Recall the bound of m in (37). Let m ∼ h γβ -d , then (99) becomes 587

Moreover, the standard lower bound for classification [28] is 588

E Proof of Theorem 4 590

Denote 591

592

Therefore 594

Hence 595

Define 596

Thus 599

600

For all j such that n ∗ l -n lj > t , 593

then 597

From Hoeffding's inequality, 598

By Hölder continuity assumption (Assumption 1(a)), for x ∈ B l , 601

in which L d = L √ d , L is the constant in Assumption 1(a). Thus 602

Now take integration over B l . 603

in which p ( B l ) = P ( X ∈ B l ) = ∫ B l f ( x ) d x . (117) is the central label DP counterpart of (67). The 604 remainder of the proof follows arguments of the local label DP. We omit detailed steps. The result is 605

The proof is complete. 608

609

F Proof of Theorem 5

Find G cubes in the support and the length of each cube is h . Let ϕ ( u ) be the same as the classification 610 case shown in appendix B. For v ∈ V := {-1 , 1 } G , let 611

))

Let P

(

Y

= 1

|

x

) = (1 +

η

v

(

x

/

2

, P

(

Y

=

-

1

|

x

) = (1

-

η

v

(

612

The overall volume of the support is bounded. Thus, we have 613

x

))

, then

η

(

x

) =

E

[

Y

|

x

] =

η

v

(

x

)

.

Let 606

then 607

for some constant C G . 614

Denote 615

then the excess risk is bounded by 616

̸

If ˆ v k = v k , from (123), 617

̸

Therefore, if ˆ v k = v k , then 618

Therefore 619

then δ ≲ 1 , and 621

Thus 622

G Proof of Theorem 6 623

According to Assumption 2, | Y | < T with probability 1 , thus Var[ Y | x ] ≤ T 2 for any x . ALaplacian 624 distribution with parameter λ has variance 2 λ 2 , thus 625

Now we analyze the bias first. 627

Hence 626

̸

Similar to the classification problem analyzed in Appendix B, let 620

Thus 628

for some constant C 1 . 629

It remains to bound the variance. 630

For the first term in (135), 631

For the second term in (135), 632

Therefore (135) becomes 633

Combine the analysis of bias and variance, 634

Therefore the overall risk is bounded by 635

The optimal growth rate of k over N is 636

Then the convergence rate of the overall risk becomes 637

̸

H Proof of Theorem 7 638

From (127), 639

Follow the analysis of lower bounds of classification in Appendix D, let h scales as (101), then 640 P ( ˆ V k = V k ) ≳ 1 . Moreover, G ∼ h -d . Hence 641

Moreover, note that the non-private lower bound of regression is 642

̸

Combine (144) and (145), 643

I Proof of Theorem 8 644

1) Analysis of bias. Note that 645

Therefore, for all x ∈ B l , 646

Therefore for all x ∈ B l , 647

2) Analysis of variance. If n l > 0 , 648

Therefore 649

Similarly, 650

The mean squared error can then be bounded by the bounds of bias and variance. 651

Let 652

Then 653

J Proof of Theorem 9 654

Now we prove the minimax lower bound of nonparametric regression under label DP constraint. We 655 focus on the case in which ϵ is small. 656

Similar to the steps of the proof of Theorem 5 in Appendix F, we find B cubes in the support. The 657 definition of η v , ˆ v k are also the same as (121) and (123). Compared with the case with bounded 658 noise, now Y can take values in R . 659

For given x , let 660

Let 665

Here we still define 666

In Appendix F about the case with bounded noise, T is a fixed constant. However, here T is not fixed 661 and will change over N . It is straightforward to show that the distribution of Y in (156) satisfies 662 Assumption 3: 663

Moreover, by taking expectation over Y , it can be shown that η v is still the regression function: 664

̸

K Proof of Theorem 10 670

Define 671

Then 672

Therefore 673

Now we bound I 1 , I 2 and I 3 separately. 674

Bound of I 1 . We show the following lemma (which will also be used later). 675 Lemma 2.

Proof. Firstly, we decompose η T ( x ) and η ( x ) : 676

677

The first term is the same between (168) and (169). Therefore we only need to compare the second 678 and the third term. 679

Without loss of generality, suppose that v 1 = v ′ for i = 1 . Then 667

Let 668

then δ ≲ 1 . Hence 669

Therefore 680

Similarly, 681

and 690

Let 692

then 693

A Combination of these two inequalities yields the (167). 682

Bound of I 2 . Follow the steps in (134), 684

Bound of I 3 . We decompose Var[ˆ η ( x )] as following: 685

For the first term in (175), from Assumption 3, E [ | Y | p | x ] ≤ M p . Since p ≥ 2 , we have E [ Y 2 | x ] = 686 M 2 p p . Therefore 687

Recall (20), we have 688

For the second term in (175), (137) still holds, thus 689

Plug (173), (174) and (179) into (166), and take expectations, we get 691

L Proof of Theorem 11 694

Let Y be distributed as (156). Recall Lemma 1 for the problem of classification and regression with 695 bounded noise. 696

697

Now we show the corresponding lemma for regression with unbounded noise.

Lemma 3. If 0 ≤ t ≤ T ln 2 / ( ϵn k ) , and n k t/T is an integer, then 698

699

700

701

Here we briefly explain the condition n k t is an integer. Recall the definition of ¯ Y k in (88). Now since Y take values in {-T, 0 , T } , n k ¯ Y k /T must be an integer. Therefore, in Lemma 3, we only need to consider the case such that n k t/T is an integer.

Proof. The proof follows the proof of Lemma 1 closely. We provide the proof here for completeness. 702 Construct D ′ by changing the label values of l = n k t/T items from these n k samples falling in B k , 703 from -T to T . Then the average label values in B k is denoted as ¯ Y ′ k after such replacement. ˆ V k also 704 becomes ˆ V ′ k . Then from the ϵ -label DP requirement, 705

in which (a) uses the group privacy property. The Hamming distance between D and D ′ is l , thus the 706 ratio of probability between D and D ′ is within [ e -lϵ , e lϵ ] . (b) holds because the algorithm does not 707 change after changing D to D ′ . Similarly, 708

709

Then (183) can be shown by adding up (184) and (185).

We then follow the proof of Theorem 3 in Appendix D. (101) becomes 710

In (156), note that P ( Y = T ) ≥ 0 and P ( Y = -T ) ≥ 0 . Therefore M p /T p ≥ η v ( x ) /T . This 711 requires h β T p -1 ≤ M p . Let T ∼ h -β p -1 , then 712

Combine with standard minimax rate, the lower bound of regression with unbounded noise is 714

i.e. 713

M Proof of Theorem 12 715

1) Analysis of bias. Note that Lemma 2 still holds here. Moreover, recall (149). Therefore 716

2) Analysis of variance. Similar to (151), it can be shown that 717

Moreover, the noise variance can be bounded by 718

The mean squared error is then bounded by 719

Let T ∼ ( ϵNh d ) 1 /p , then 720

To minimize (194), let 721

then 722

723

724

725

726

727

728

729

730

731

732

733

734

735

736

737

738

739

740

741

742

743

744

745

746

747

748

749

750

751

752

753

754

755

756

757

758

759

760

761

762

763

764

765

766

767

768

769

770

771

772

773

774

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main contribution (i.e. proposing a new Huber loss minimization approach which is more suitable to realistic cases, and providing theoretical analysis) has been made clear in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: It is explained at the end of conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Proofs are shown in the appendix, and intuition is provided in the paper.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: This is a theoretical paper without experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: This is a theoretical paper without experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( ❤/a116/a116♣/a115✿✴✴♥✐♣/a115✳❝❝✴ ♣✉❜❧✐❝✴❣✉✐❞❡/a115✴❈♦❞❡❙✉❜♠✐/a115/a115✐♦♥/a80♦❧✐❝② ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( ❤/a116/a116♣/a115✿ ✴✴♥✐♣/a115✳❝❝✴♣✉❜❧✐❝✴❣✉✐❞❡/a115✴❈♦❞❡❙✉❜♠✐/a115/a115✐♦♥/a80♦❧✐❝② ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: This is a theoretical paper without experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: No experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: No experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics ❤/a116/a116♣/a115✿✴✴♥❡✉/a114✐♣/a115✳❝❝✴♣✉❜❧✐❝✴❊/a116❤✐❝/a115●✉✐❞❡❧✐♥❡/a115 ?

Answer: [Yes]

Justification: Our paper does not violate code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper is foundational and theoretical research and not tied to particular applications.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper has no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, ♣❛♣❡/a114/a115✇✐/a116❤❝♦❞❡✳❝♦♠✴❞❛/a116❛/a115❡/a116/a115 has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

982

983

984

985

986

987

988

989

990

991

992

993

994

995

996

997

998

999

1000

1001

1002

1003

1004

1005

1006

1007

1008

1009

1010

1011

1012

1013

1014

1015

1016

1017

1018

1019

1020

1021

1022

1023

1024

1025

1026

1027

1028

1029

1030

1031

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the theoretical boundaries of learning with Label Differential Privacy (Label-DP) in both central and local models.\nLabel-DP is a weakening of standard differential privacy, where only the privacy of the \"label\" of each example is to be protected (an example is a pair (feature vector, label)).\n\nThe key contributions of the paper are to establish min-max optimal rates for excess error in the settings of:\n* (multi-class) classification,\n* regression with bounded labels,\n* regression with unbounded labels (but under a bounded moment condition).\n\nThe min-max rates are over the class of data distributions that satisfy $\\beta$-Holder smoothness, admits a lower bound on probability density that is bounded away from zero, assumes that there are no “sharp corners” in the input space, and a $\\gamma$-margin assumption (in case of classification), or bounded label range or bounded label moments (in case of regression).\n\nThese min-max rates are then compared against the previously known min-max rates for learning under “full” local-DP (that protects both features and labels), as well as non-private learning.\n\nThe key takeaways are:\n* Local-DP vs Non-Private:\n  * For classification and regression with bounded labels, the sample complexity under Local-DP increases by a factor of $1/\\varepsilon^2$, but has the same rate in terms of desired excess error. This is unlike “Full Local-DP”, where the sample complexity is larger even in terms of the desired excess error.\n  * For regression with unbounded labels, the dependence of sample complexity on desired excess error is worse than the non-private setting.\n* Central-DP vs Non-Private:\n  * The excess error is the sum of the non-private excess error and an additional term that decays faster in the number of samples, so the additional sample complexity due to privacy is negligible for very small excess error.\n\nSTRENGTHS:\nThe paper provides a comprehensive study of the min-max rates for learning under label differential privacy, in both local and central models of DP, and for both classification and regression. This complements prior literature on min-max rates for learning (non-privately) and for learning under (full) differential privacy. The rates highlight the precise cost of _label_ differential privacy and the sample complexity benefits over full differential privacy.\n\nWEAKNESSES:\nWhile there are many results in the paper, I think the proof techniques in both lower and upper bounds use mostly standard tools (This is not necessarily a weakness!).\n\nThe paper writing could be improved at several places though. Some comments are listed below under \"Questions\".\n\nQUESTIONS:\nI don't have any significant questions.\n\nSome suggestions about writing are listed below:\n* Table 1: I understood that the last two rows of Table 1 are prior work, and the first two rows are the new contributions in this paper. But it would be good to include appropriate citations prominently, e.g. in the caption for Table 1.\n* Line 122: $\\mathbb{E}[l(g(X, Y))]$ should be $\\mathbb{E}[l(g(X), Y)]$.\n* Line 135 (Eq 8): I think $\\eta^*(x) = \\max_k \\eta_k(x)$, but it is not formally defined anywhere?\n* Line 137: Proof of Proposition 2 in Appendix A does not cover the regression setting. (It is quite standard, but might be good to include the proof for completeness.)\n* Line 139: Is Assumption 1 only about the classification setting? If so, maybe it makes sense to move it to Section 4?\n* Line 145: Assumption 1 (d): It is not explained until this point that $\\mathcal{X} \\subseteq \\mathbb{R}^d$. I think this should be defined upfront if this is assumed throughout the paper.\n* Line 169: If I understand correctly, the $\\inf_{\\hat{Y}}$ refers to $\\hat{Y}$ that is a function of the outputs of the mechanism on all inputs $(M(x_i, y_i))_i$. This notation is implicit, but would be good to spell out.\n* Line 202: Is there a reference for the bounds for “full” local DP, that protects both features and labels ?\n* Line 244: How is Assumption 1 relevant for regression? What is $\\eta_j$ ? Only points (c) and (d) make sense. Maybe (a) also makes sense for $\\eta$ instead of $\\eta_j$ ?\n* Line 297: $Y_{Ti}$ notation is awkward to read. Might want to make it something like $Y^T_i$?\n\nLIMITATIONS:\nI do not see any potential negative societal impact of this work, as it is primarily theoretical.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the minimax rates for classification and regression under (pure) label differential privacy in both the local and central models. They prove that rates of convergence for classification and regression with bounded label noise in the local label DP model are comparable to those for the non-private tasks, except for the expected $1/\\varepsilon^2$ dependence. This represents an improvement over rates for standard DP in both settings, where there is a worse dependence on the dimension of the covariates. They also prove, however, in the case of regression with unbounded label noise, the convergence rate improvements over “full” DP aren’t as meaningful.\n\nSTRENGTHS:\nThis work makes notable progress in our theoretical understanding of the costs of label DP relative to non-private and full DP algorithms for the same learning task.\n\nWEAKNESSES:\nThe presentation could be improved in several places. Admittedly, this is written from a statistical perspective that is different from the one I am most familiar with, so some of the perceived presentation issues may just be a matter of convention, but the following changes might make this work more understandable to the general NeurIPS community:\n\n\nAbstract:\n\nThe main challenge and the techniques to overcome them as stated in the abstract aren’t clear to me as a reader at this point. It’s not yet stated that the subject of interest is minimax rates, and so there’s no context for the statement “take infimum over all possible learners” and why that would present a challenge. Generally, I did not have a good idea of what the contribution of this work was from the abstract.\n\nIntroduction:\n\n“the learning performances” -> “the learning performance” \n\n“the label DP” -> “label DP”\n\nIn Table 1, attribution for the full DP rates in the local DP setting as well as the rates in the non-private setting should be given in the table. Also, I think there’s an issue with the parentheses in the local label DP rates for regression with bounded label noise.\n\nSection 2:\n\nIn the “Minimax analysis for private data” paragraph, KNLRS11 is attributed with finding the relation between label DP and stochastic queries. This is not accurate, this work characterizes local DP learning by the statistical query model.\n\nSection 3:\n\n“We hope that $R - R^*$ to be as small as possible” -> “we seek to minimize this risk” or something similar\n\n“the Bayes optimal classifier and the corresponding Bayes risk is” -> “the Bayes optimal classifier and the corresponding Bayes risk are”\n\nIn Proposition 2, f(x) is used before it is defined.\n\nSection 4:\n\nI didn’t find the proof outline for Theorem 1 or Theorem 3 to be informative at all. It would be good to add more specifics if possible.\n\n“Let the privacy mechanism M(x,y) outputs” -> “Let the privacy mechanism M(x,y) output”\n\nQUESTIONS:\nDo you have a sense of how the rates would change for approximate label DP?\n\nLIMITATIONS:\nYes, the authors address the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problems of classification and regression under the constraint of local/central pure label DP. The authors derive upper and lower bounds on the excess risk (compared to the non-private Bayes classifier/regression) for these problems, under somewhat standard assumptions on the 'ground truth' randomized label function $\\eta$. For regression, both the case where the labels are bounded and have bounded moments are considered. For the lower bounds, the authors develop extensions of techniques from minimax estimation to label DP. For upper bounds, authors propose some algorithms combining 'binning' different examples with a privacy mechanism chosen according to the problem setting. The upper/lower bounds are matching in each setting up to logarithmic factors. For local label DP, the authors show the minimax excess risk with $N$ samples matches the non-private bounds using $N \\min\\{\\epsilon^2, 1\\}$ samples. In other words, with $\\epsilon = \\Omega(1)$ the minimax risk asymptotically matches the non-private risk, and otherwise there is an inherent separation. For central label DP, the minimax bound is one that approaches the non-private bound as $N \\rightarrow \\infty$ for any fixed $\\epsilon$, showing a qualitative difference. For local \"full\" DP, i.e. the features are also private, even for $\\epsilon = \\Omega(1)$ and large $N$ one cannot achieve the non-private rate.\n\nSTRENGTHS:\n* Derives optimal (up to log factors) upper and lower bounds for several different variants of classification/regression under label DP.\n* To derive these bounds, introduces some new technical tools for minimax analysis of DP algorithms that might be useful in future work.\n* Label DP is a variant of DP that is seeing attention in practice, and classification/regression are fundamental problems, so the results in the paper can have a practical impact easily.\n* The authors do a good job making clear the comparison between the results in different settings. e.g. Table 1 is a very concise summary that allows one to draw all the essential comparisons between the different settings, and there are discussions like Remark 1 that give qualitative interpretations of the quantitative results, and also discuss other baselines to compare to.\n\nWEAKNESSES:\nThe main issue is with the presentation. Specifically, the presentation does a great job explaining what the final results are and helping the reader contextualizing them, but at some points the techniques used to obtain the result are discussed at a very high level in the main body and why they work remains obscure even after reading the proof outlines in the main body multiple times.  There are some cases where the authors do a good job concisely describing a proof, e.g. Theorem 6's proof outline is very concise but it still gives a good idea what the proof looks like, even if they would have to check the appendix for details. But for others, like Theorems 1/2/3, the proof outline is not very informative. See Questions for more details.\n\nI understand the authors are constrained by space requirements, but I think the allocation of space in the main body can be better thought out. For example, I think it might be better to try to give the reader a very good understanding of classification and/or bounded label regression (e.g., Lemma 1 from the Appendix could be brought to the main body without its proof, and the authors could explain how it is used), and omit all but the top-level points on bounded label moment regression, rather than giving a sparse understanding of all three.\n\nQUESTIONS:\nThese are some examples of points I think are not clear from the main body which a reader interested in these problems might be able to understand better if the presentation were improved.\n\n* What techniques from local DP / minimax theory does Theorem 1 use / how would it vary from e.g. the proof for the corresponding non-private lower bound? Saying you use techniques from certain areas/papers without specifying what they are is not particularly informative to a reader. (even giving a high-level summary of the technique would be ok)\n* How do you decide the number of bins in your upper bound for Theorem 2? Presumably by optimizing some tradeoff b/t the number of examples per bin and having narrower bins where $\\eta$ cannot vary too much within a bin, but I'm not confident about this from reading the main body.\n* $\\mathcal{X}$ is a vector space, what does \"a bin of length $h$\" mean for bins defined over a vector space? I believe it means something like each bin has radius at most $h$ in some appropriate norm, but not sure.\n* In Theorem 3, the proof outline's first and second halves seem disconnected. The authors mention the model complexity needs to increase with $N$, how does the second half of the proof outline address this necessity?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-29"}

comments: SUMMARY:
This paper investigates the minimax risks of classification and regression (with both bounded and heavy-tailed noise) under label differential privacy (DP) in both central and local models.

STRENGTHS:
The paper provides a comprehensive analysis by considering both upper and lower bounds for the minimax risks.
It explores both central and local DP models and different settings, covering a broad spectrum of scenarios.

WEAKNESSES:
The writing quality needs improvement to meet publication standards. Several sections are challenging to understand. Specific issues include: 
(1) Around line 178, the output of the mechanism for classification is unclear. Why is it not a one-hot vector, or at least why is the L1 norm not equal to 1?
(2) Some notations are overused. For example, "c" refers to the lower bound of the density function in Assumption 1 and also denotes the classifier in line 186 and subsequent proofs.
(3) The description of the algorithm before Theorem 2 is vague and lacks clarity.
(4) The proofs in the appendix are hard to follow without explanations or discussions. For instance, how is $\phi$ defined in Equation (35), and what purpose does it serve? Why does the construction satisfy the assumptions? There seem to be some typos or missing elements in Equations (39) and (40).

QUESTIONS:
The paper discusses non-private baselines. Are there citations provided for these baselines?

LIMITATIONS:
See weaknesses and questions.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-29