Title: Local Differential Privacy for Mixtures of Experts
Venue: NeurIPS 2024
Accepted: no
Keywords: Mixtures of experts

Local differential privacy

PAC-Bayes

Generalization bounds
Abstract: We introduce a new approach to the mixture of experts model that consists in imposing local differential privacy on the gating mechanism. This is theoretically justified by statistical learning theory. Notably, we provide generalization bounds specifically tailored for mixtures of experts, leveraging the one-out-of-$n$ gating mechanism rather than the more common $n$-out-of-$n$ mechanism.  Moreover, through experiments, we show that our approach improves the generalization ability of mixtures of experts.

Abstract

We introduce a new approach to the mixture of experts model that consists in imposing local differential privacy on the gating mechanism. This is theoretically justified by statistical learning theory. Notably, we provide generalization bounds specifically tailored for mixtures of experts, leveraging the one-out-ofn gating mechanism rather than the more common n -out-ofn mechanism. Moreover, through experiments, we show that our approach improves the generalization ability of mixtures of experts.

1 Introduction 8

Mixtures of experts, initially introduced by Jacobs et al. [1991], have found widespread use in 9 modeling sequential data, including applications in classification, regression, pattern recognition and 10 feature selection tasks (Städler et al. [2010] and Khalili and Lin [2013]). One of the fundamental 11 motivations behind mixtures of experts is their ability to break down complex problems into more 12 manageable sub-problems, potentially simplifying the overall task. The structure of these models is 13 well suited to capturing unobservable heterogeneity in the data generation process, dealing with this 14 problem by splitting the data into homogeneous subsets (with the gating network) and associating 15 each subset with an expert. This intuitive architecture has led to significant interest in mixture of 16 experts models, resulting in a wealth of research (Yuksel et al. [2012]), ranging from simple mixtures 17 of experts ( Jacobs et al. [1991], Jordan and Jacobs [1993]) to sparsely gated models (Shazeer et al. 18 [2017]). Moreover, this architecture has inspired the development of various other models, such as 19 switch transformers (Fedus et al. [2022]). However, despite the considerable attention mixtures of 20 experts have received, advancements in their theoretical analysis have been relatively limited. Azran 21 and Meir [2004] proved data-dependent risk bounds for mixtures of experts (with the n -out-ofn 22 gating mechanism) using Rademacher complexity, but they exhibit a dependence on the complexity 23 of the class of gating networks and the sum of the complexities of the expert classes, which reflects 24 the complex structure of mixtures of experts but unfortunately leads to potentially large bounds. We 25 are not aware of other work proving generalization bounds specifically tailored to mixtures of experts. 26

27

28

29

30

31

32

33

34

To make theoretical progress, we utilize a well-known privacy-preserving technique called Local Differential Privacy (LDP). It was initially introduced by Dwork [2006] and has since been widely used to preserve privacy for individual data points as in Kasiviswanathan et al. [2010]. This is achieved by introducing stochasticity in algorithm outputs to control their dependence on specific inputs. This stochasticity is generally quantified by a positive real number ϵ . In this case, we write ϵ -LDP instead of just LDP. The parameter ϵ quantifies the level of privacy protection in the local differential privacy mechanism. A smaller value indicates stronger privacy protection, which requires the addition of more noise.

In this work, we exploit this noise for regularization in our models by imposing the ϵ -LDP condition 35

on their gating networks. This method allows us to leverage the numerous benefits of the most 36

37

complex architectures, such as neural networks, without compromising theoretical guarantees on risk.

By relying on LDP, we offer tight theoretical guarantees on the risk of mixtures of experts models, 38

provided with the one-out-ofn gating mechanism. Unlike the very few existing guarantees, these 39 bounds depend only logarithmically on the number of experts we have, and the complexity of the 40 gating network only appears in our bounds through the parameter ϵ of the LDP condition. 41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

2 Preliminaries

Let X be the instance space, Y the label space, and Y ′ the output space (which can be different from Y ). As is usual in supervised learning, we assume that data ( x, y ) ∈ X × Y are generated independently from an unknown probability distribution D . We consider a training set of m examples S = (( x 1 , y 1 ) , . . . , ( x m , y m )) ∼ D m and a bounded loss function ℓ : Y ′ ×Y → [0 , 1] .

2.1 Mixtures of experts

We consider classes H i of experts h i : X → Y ′ for i = 1 , . . . , n . Let G be a set of gating functions g : X → [0 , 1] n such that, given any x ∈ X , we have that ∑ n i =1 g i ( x ) = 1 , where g i ( x ) is the i -th component of g ( x ) . This means that each gating function defines a probability distribution on [ n ] = { 1 , . . . , n } for each x ∈ X , where g i ( x ) is the probability of i .

In this work, a mixture of experts consists of n experts, h = ( h 1 , . . . , h n ) ∈ H 1 × · · · × H n , a gating function g ∈ G and a gating mechanism that combines the outputs of the experts and the output of the gating function to produce the final output. Our models use the stochastic one-out-ofn gating mechanism, as described in Jacobs et al. [1991]. It is defined as follows: to make a prediction with ( g , h ) ∈ G × ∏ n i =1 H i given an instance x , draw i ∼ g ( x ) and output h i ( x ) . This stochastic predictor has risk and empirical risk defined by, respectively,

The preference for the one-out-ofn gating mechanism over the n -out-ofn mechanism in mixtures of experts is justified by its ability to induce sparsity and noise, enhancing computational efficiency and robustness to overfitting. This sparsity also offers scalability benefits, particularly in large-scale applications, where activating all experts for each input can lead to increased computational and memory requirements as explained in Shazeer et al. [2017] and Jacobs et al. [1991]. Moreover, the one-out-ofn mechanism is more amenable to certain kinds of theoretical analysis, including ours.

2.2 Local Differential Privacy

Definition 2.1. Let I be a finite set, consider a mechanism that produces an output i ∈ I , given an input x ∈ X , with probability P ( i | x ) , and let ϵ be a nonnegative real number. Then, the mechanism satisfies the ϵ -Local Differential Privacy ( ϵ -LDP) property if and only if

Unless stated otherwise, we assume that each g ∈ G satisfies ϵ -LDP, for some fixed nonnegative real number ϵ . Since we can interpret g as a random mechanism that, given x ∈ X , selects i ∈ [ n ] with probability g i ( x ) , the condition of ϵ -LDP amounts to the following:

Since ϵ -LDP is an important condition for all of our theoretical results, we provide a practical way of obtaining gating functions satisfying ϵ -LDP from an arbitrary set F of bounded functions, in the form of the following theorem.

Theorem 2.2. Let b > 0 and β ≥ 0 be real numbers, and suppose that F is a set of functions f : X → [ -b, b ] n . Let G be the set of functions g : X → [0 , 1] n defined by

Then, each g ∈ G satisfies 4 βb -LDP. 73

Proof. The proof is obtained by performing simple calculations, bounding the ratio g i ( x ) /g i ( x ′ ) , for 74 all x, x ′ ∈ X and all i ∈ [ n ] . The detailed proof is given in Appendix A. 75

3 PAC-Bayesian bounds for mixtures of experts 76

To apply the PAC-Bayes theory, we need to add a level of stochasticity to our predictors: instead of training experts h i , we train probability measures Q i on each expert set H i . For convenience, we write Q = Q 1 ⊗··· ⊗ Q n . Now, putting everything together, a mixture of experts ( g , Q ) makes predictions as follows: given x ∈ X , draw i ∼ g ( x ) , then draw h ∼ Q i , and finally output h ( x ) . Such a predictor has risk and empirical risk defined by, respectively,

Notice that, though probability distributions have replaced the individual experts, there is no need to 77 define a probability distribution on the gating functions to get a PAC-Bayesian bound. Training a 78 single gating function will do, and, remarkably, Lemma 3.1 below shows that it can be obtained from 79 a very complicated function, such as a neural network, provided we impose ϵ -LDP (for example, with 80 Theorem 2.2). 81

Finally, let us recall the notion of Kullback-Leibler (KL) divergence . Given probability distributions 82 Q i and P i on H i , it is defined by 83

where dQ i /dP i is a Radon-Nikodym derivative.

Lemma 3.1. We consider mixtures of experts as defined in section 2.1 and provided with the oneout-ofn routing mechanism. Let ∆ : R 2 → R be a convex function that is decreasing in its first argument and increasing in its second argument, and let ϵ be a nonnegative real number. Then, for any g ∈ G that satisfies the ϵ -LDP property, for any Q = Q 1 ⊗··· ⊗ Q n on H 1 ×··· × H n , and for any x ′ ∈ X :

Proof. Since the gating function satisfies ϵ -LDP, we have that e -ϵ g i ( x ′ ) ≤ g i ( x ) ≤ e ϵ g i ( x ′ ) for all x, x ′ ∈ X and all i ∈ [ n ] . It follows that e ϵ R S ( g , Q ) ≥ E i ∼ g ( x ′ ) R S ( Q i ) and e -ϵ R ( g , Q ) ≤ E i ∼ g ( x ′ ) R ( Q i ) . Given that ∆ is decreasing in its first argument and increasing in its second argument, we find that

Since ∆ is a convex function, we can apply Jensen's inequality to the expression on the right-hand side, yielding the desired result.

Different choices of function ∆ will allow us to obtain different PAC-Bayes bounds:

Let ∆( u, v ) = v -u . This is compatible with typical PAC-Bayes bounds on the difference between the true and empirical risks.

Given λ > 1 / 2 , let ∆ be defined by ∆( u, v ) = v -2 λ 2 λ -1 u . This choice is compatible with a Catoni-type bound, as we will see below.

Let ∆ be defined by ∆( u, v ) = kl( u ∥ v ) = u ln u v + (1 -u ) ln 1 -u 1 -v . This choice is compatible with a Langford-Seeger-type bound. However, note that the function ∆ defined here does not quite obey the hypotheses of lemma 3.1. Indeed, it is only defined for ( u, v ) ∈ [0 , 1] 2 , and only has the right monotonicity properties on the set { ( u, v ) ∈ [0 , 1] 2 | u ≤ v } . We can remedy those defects through small adjustments to the proof.

We prove a generalization bound of Catoni-type as an illustration of the machinery just described.

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

Theorem 3.2 (Theorem 2 in McAllester [2013]) . Let δ ∈ (0 , 1) and λ > 1 / 2 . Fix i ∈ [ n ] , and let P i 108 be a probability measure on H i (chosen without seeing the training data). Then, with probability at 109 least 1 -δ over the draws of S , for all probability measures Q i on H i , we have that 110

Theorem 3.3. Let δ ∈ (0 , 1) , ϵ ≥ 0 , and λ > 1 / 2 . For each i ∈ [ n ] , let P i be a probability measure on H i (chosen without seeing the training data). Then, with probability at least 1 -δ over the draws of S , for all probability measures Q = Q 1 ⊗··· ⊗ Q n on H , all g ∈ G that satisfy ϵ -LDP, and all x ′ ∈ X , we have that

Proof. By n applications of Theorem 3.2, we have that, for each i ∈ [ n ] , with probability at least 1 -δ/n , for all Q i ,

We can make all these inequalities (for each i ∈ [ n ] ) hold simultaneously with a union bound. Now, applying Lemma 3.1 with ∆( u, v ) = v -2 λ 2 λ -1 u , we find that, with probability at least 1 -δ , for all Q , all g ∈ G and all x ′ ∈ X , we have that

We also give a bound of Langford-Seeger type, since they are generally recognized as among the tightest PAC-Bayes bounds available, and to prove the flexibility of our approach.

Theorem 3.4. Let δ ∈ (0 , 1) , ϵ ≥ 0 , and m ≥ 8 . For each i ∈ [ n ] , let P i be a probability measure on H i (chosen without seeing the training data). Then, with probability at least 1 -δ over the draws of S , for all probability measures Q = Q 1 ⊗··· ⊗ Q n on H , all g ∈ G that satisfy ϵ -LDP, and all x ′ ∈ X , we have that, either R ( g , Q ) < e 2 ϵ R S ( g , Q ) , or

Proof. The proof, which is similar to that of Theorem 3.3, is available in Appendix A.

3.1 Comparison with other bounds

Very few generalizations bound tailored specifically to mixtures of experts appear in the literature, and those we could find do not apply to mixtures of experts with the one-out-ofn gating mechanism. We can, however, compare our bounds to those obtained by naively applying generic PAC-Bayes generalization bounds to mixtures of experts. In this case, we need to consider classifiers of the form ( Q G , Q ) , where Q G is a probability measure on G , and Q = Q 1 ⊗··· ⊗ Q n is a probability measure on H 1 ×··· × H n as before. Then, note that

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

This means that a generic PAC-Bayes bound applied to mixtures of experts will depend on the sum of 134 the KL divergences corresponding to the gating functions and each of the experts. Obviously, this 135 sum could be very large. By imposing ϵ -LDP to the gating functions as in our approach, we can 136 eliminate the stochasticity associated to the gating functions, and rid our bounds of the (potentially 137 very large) KL( Q G ∥ P G ) term. Instead, it is ϵ -LDP which controls our gating functions to ensure 138 generalization. Furthermore, our bounds replace the sum of the KL divergences of the experts by 139 a g ( x ′ ) -weighted average, which means we can have many more experts with almost no penalty 140 from the theoretical point of view. Indeed, our bounds only depend on the number n of experts 141 logarithmically, through the use of the union bound. 142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

4 Rademacher bounds for mixtures of experts

Let us start with a slight modification of Lemma 3.1.

Lemma 4.1. We consider mixtures of experts as defined in section 2.1 and provided with the oneout-ofn routing mechanism. Let ∆ : R 2 → R be a convex function that is decreasing in its first argument and increasing in its second argument, and let ϵ be a nonnegative real number. Then, for any g ∈ G that satisfies the ϵ -LDP property, for any h ∈ H , and for any x ′ ∈ X :

Proof. The proof is similar to that of Lemma 3.1 and is provided in Appendix A.

Let us now recall the following definition.

Definition 4.2 (Rademacher complexity) . Given a space H of predictors, a loss function ℓ , and a data generating distribution D , the Rademacher complexity R ( ℓ ◦ H ) is defined by

where σ = ( σ 1 , . . . , σ m ) is distributed uniformly on {-1 , 1 } m .

Our main theorem will make use of the following well-known risk bound.

Theorem 4.3 (Basic Rademacher risk bound) . Given a [0 , 1] -valued loss function ℓ , with probability at least 1 -δ , for all h ∈ H , we have that

Theorem 4.4. Let δ ∈ (0 , 1) and ϵ ≥ 0 . Given a [0 , 1] -valued loss function ℓ , then, with probability at least 1 -δ over the draws of S , for all h ∈ H 1 ×··· × H n , for all g ∈ G that satisfy ϵ -LDP, and all x ′ ∈ X , we have that

Proof. By n applications of Theorem 4.3, we have that, for each i ∈ [ n ] , with probability at least 1 -δ/n , for all h i ∈ H i ,

We can make all these inequalities (for each i ∈ [ n ] ) hold simultaneously with a union bound. Now, 163 applying Lemma 4.1 with ∆( u, v ) = v -u , we find that, with probability at least 1 -δ , for all 164 h ∈ H 1 ×··· × H n , all g ∈ G and all x ′ ∈ X , we have that 165

Note, that the risk bound of Theorem 4.4 depends only on the average Rademacher complexity of 166 the classes of experts instead of the sum of their Rademacher complexities. Note also that, as in the 167 previous section, the complexity of G does not affect the risk bound. Finally, the risk bound does not 168 hold uniformly for all values of ϵ . However, by the union bound, the theorem holds for any fixed set 169 { ϵ 1 , . . . , ϵ k } if we replace δ by δ/k . Consequently, this suggests a learning algorithm that minimizes 170 R S ( g , h ) for ϵ ∈ { ϵ 1 , . . . , ϵ k } . 171

Also note that Lemma 4.1 allows us to obtain risk bounds for mixtures of experts as long as we 172 have bounds on ∆ ( R S ( h i ) , R ( h i ) ) which hold with high probability, whether they are based on 173 Rademacher complexity, margins, VC dimension, or algorithmic stability. 174

4.1 The need to use adaptive experts 175

176

177

178

179

180

Following these theoretical results, we may be tempted to use a gating network satisfying ϵ -LDP to accomplish a learning task all by itself using non-adaptive experts, that is, experts h i each taking a constant value, no matter the input: h i ( x ) = i for all x ∈ X . In that case, each Rademacher complexity R ( ℓ ◦H i ) is zero and we can show that Theorem 4.4 can become vacuous under reasonable circumstances.

181

182

183

Consider, for example, the binary classification case with the 0 -1 loss. In that case, we have two experts h +1 and h -1 such that h +1 ( x ) = +1 and h -1 ( x ) = -1 for all x ∈ X , and a gating network g = ( g +1 , g -1 ) . Then, the following holds:

̸

̸

Under the assumption that the classes are balanced, meaning that the (marginal) probability of a 184 positive label is equal to the (marginal) probability of a negative label, we have the following: 185

186

187

188

189

190

191

192

193

194

195

196

and 197

198

199

It follows that, in the limit m →∞ , the risk bound of Theorem 4.4 for any g has a value of at least e ϵ / 2 ≥ 1 / 2 . Consequently, the risk bound becomes large or even vacuous in this regime, highlighting the importance of having adaptive experts of finite complexity that can drive the empirical risk to zero when they are selected by the gating network.

5 Experiments and results

In what follows, we consider mixtures of n linear experts in binary classification tasks. Let X = R d for some positive integer d . Let S be a training set of m examples. Each expert, denoted by h i , where i ranges from 1 to n , is characterized by a weight vector w i . Given an input x ∈ X , the output of the expert h i is given by h i ( x ) = w i · x . We use the probit loss function ℓ = Φ , which can be seen as a smooth surrogate to the 0 -1 loss function, when it is used with an argument of the form y w i · x ∥ x ∥ . In this case, R ( g , Q ) and R S ( g , Q ) are given by:

where Φ( x ) = 1 √ 2 π ∫ + ∞ x e -t 2 / 2 dt provides the probability that a standard normal random variable is greater than a given value x .

To illustrate the regularizing effect of the LDP condition, we carried out several experiments, on 200 different datasets, by minimizing the empirical risk as defined in Equation 1. For all experiments, our 201 models consist of mixtures of n = 100 linear experts and a gating network. The gating network is a 202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

neural network having 2 hidden layers. It is parameterized by weights W 1 ∈ R 64 × d , where d is the dimension of input vectors, W 2 ∈ R 64 × 64 , and W 3 ∈ R n × 64 , and biases b 1 ∈ R 64 , b 2 ∈ R 64 and b 3 ∈ R n . Given an input x ∈ R d , the output of the gating network g ( x ) = ( g 1 ( x ) , . . . , g n ( x )) is computed as follows: first, we compute f 0 ( x ) = tanh( W 2 ReLU( W 1 x + b 1 ) + b 2 ) . Then, when we want the ϵ -LDP condition to be satisfied, we ensure that the outputs are between -ϵ/ 4 and ϵ/ 4 :

Note that tanh is the hyperbolic tangent activation function, ReLU the Rectified Linear Unit function, ∥ W 3 ∥ F the Frobenius norm of the matrix W 3 , and ∥ f 0 ( x ) ∥ the euclidean norm of the vector f 0 ( x ) . Indeed, if we let W i 3 denote the i -th row of W 3 , then the i -th component of W 3 f 0 ( x ) is

by the Cauchy-Schwarz inequality and the definition of the Frobenius norm. The reason we use the Frobenius norm instead of directly using ∥ W i 3 ∥ is to preserve the proportions between the components of W 3 f 0 ( x ) when setting up ϵ -LDP.

The final output of the gating network is given by

In our experiments, we ran the Stochastic Gradient Descent algorithm 10 times with a learning rate fixed to 0 . 1 . In each experiment, we trained the model for 1000 epochs, except for the MNIST dataset, where the training duration was shortened to 300 epochs due to dataset size. We allocated approximately 75% of the data to the training set and the remaining 25% to the test set. At the outset of each experiment, the weights of our neural networks were reinitialized to ensure a fresh starting point. After each training run, we computed both the training and test loss values to evaluate the model's performance. We first ran the training without imposing any constraints on the gating network, except for the architecture. Then, we ran several experiments with a gating mechanism satisfying ϵ -LDP, with ϵ ∈ { 0 . 5 , 2 , 4 , 5 , 10 } . A summary of the results is shown in Table 1. One can observe that regularization with ϵ -LDP improves results in practice, and this regularization is even more evident when the models employing a gating network not satisfying LDP overfit heavily, as in the Breast Cancer and Heart experiments. The regularization effect is slightly less pronounced on MNIST, where the overfitting is not as severe as with the previous datasets. We can also observe the importance of choosing the right hyperparameter ϵ . Indeed, if the value is too small, the output of the gating network becomes insufficiently dependent on the input x . In this case, the experts have to do all the work, and the gating network does not allow them to specialize in well-defined subsets of the instance space. This makes our model closer to a weighted sum of linear classifiers and significantly reduces its performance. Conversely, if ϵ is overly large, our model tends towards a situation where the LDP condition does not hold, making it prone to overfitting.

Note that our experiments are executed on GPUs in order to parallelize computations and take advantage of the sparsity of our model, but they can also be performed without GPUs. The duration of experiments can range from a few minutes for small datasets such as Breast Cancer to around 3 hours for large datasets like MNIST.

6 Conclusion

In this work, we introduce a new way to regularize mixtures of experts. We provide both theoretical and algorithmic contributions in this regard. Our approach offers a significant advantage in that it allows us to harness the remarkable performances of neural networks by using them as gating networks, without being constrained by their architecture or their complexity from the theoretical point of view. By imposing LDP, we obtain nonvacuous bounds on the mixture of experts' risk. Our bounds can become significantly tighter than those presented in section 3.1 and those presented in

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

Azran and Meir [2004], especially in cases where the empirical risk is close to zero and ϵ < ln n . However, as the empirical risk is multiplied by e ϵ , the bounds can become loose when ϵ is large and the empirical risk is significant.

Even though the ϵ -LDP condition is easy to set up, a challenge arises in striking a balance between the parameter ϵ and the KL divergence or the Rademacher complexity of our experts. Our method introduces an extra hyperparameter ϵ to optimize but does not provide theoretical guidance on configuring it. This forces us to navigate a trade-off between the value of ϵ , which measures the extent to which the output of the gating network can depend on a given x ∈ X , and the complexity of our experts, which reflects how well our model captures the data distribution. Finding the right balance requires empirical testing and careful consideration and can open up new avenues of study in the future.

References

Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computation , 3(1):79-87, 1991. doi: 10.1162/neco.1991.3.1.79.

Nicolas Städler, Peter Bühlmann, and Sara van de Geer. l1-penalization for mixture regression models (with discussion). TEST , 19(2):209-285, 2010. doi: 10.1007/s11749-010-0197-z.

Abbas Khalili and Shili Lin. Regularization in finite mixture of regression models with diverging number of parameters. Biometrics , 69, 04 2013. doi: 10.1111/biom.12020.

Seniha Yuksel, Joseph Wilson, and Paul Gader. Twenty years of mixture of experts, 08 2012.

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

M.I. Jordan and R.A. Jacobs. Hierarchical mixtures of experts and the em algorithm. In Proceedings of 1993 International Conference on Neural Networks (IJCNN-93-Nagoya, Japan) , volume 2, pages 1339-1344 vol.2, 1993. doi: 10.1109/IJCNN.1993.716791.

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer, 2017. URL https://arxiv.org/abs/1701.06538 .

William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2022.

Arik Azran and Ron Meir. Data dependent risk bounds for hierarchical mixture of experts classifiers. In John Shawe-Taylor and Yoram Singer, editors, Learning Theory , pages 427-441, Berlin, Heidelberg, 2004. Springer Berlin Heidelberg. ISBN 978-3-540-27819-1.

Cynthia Dwork. Differential privacy. In Michele Bugliesi, Bart Preneel, Vladimiro Sassone, and Ingo Wegener, editors, Automata, Languages and Programming , pages 1-12, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg. ISBN 978-3-540-35908-1.

Shiva Prasad Kasiviswanathan, Homin K. Lee, Kobbi Nissim, Sofya Raskhodnikova, and Adam Smith. What can we learn privately?, 2010.

David McAllester. A PAC-bayesian tutorial with a dropout bound, 2013.

Matjaz Zwitter and Milan Soklic. Breast Cancer. UCI Machine Learning Repository, 1988. DOI: https://doi.org/10.24432/C51P4M.

Andras Janosi, William Steinbrunn, Matthias Pfisterer, and Robert Detrano. Heart Disease. UCI Machine Learning Repository, 1988. DOI: https://doi.org/10.24432/C52P4X.

Li Deng. The MNIST database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine , 29(6):141-142, 2012.

Michael D. Perlman. Jensen's inequality for a convex vector-valued function on an infinitedimensional space. Journal of Multivariate Analysis , 4(1):52-65, 1974. ISSN 0047-259X. doi: https://doi.org/10.1016/0047-259X(74)90005-0. URL https://www.sciencedirect.com/ science/article/pii/0047259X74900050 .

Andreas Maurer. A note on the PAC bayesian theorem, 2004.

A Proofs and auxiliary results 292

Proof of theorem 2.2. Given x ∈ X , let Z ( x ) = ∑ n i =1 exp( βf i ( x ) + c i ) , for convenience. 293 For all x, x ′ ∈ X and all i ∈ [ n ] , we have that 294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

Theorem A.1 (Jensen's inequality, proposition 1.1 in Perlman [1974]) . Let Ω be a probability space, let A be a convex subset of R k , let X : Ω → A be an integrable vector-valued random variable, and let ϕ : A → R be a convex function. Then, E X ∈ A , and ϕ ( E X ) ≤ E ϕ ( X ) (in particular, the right-hand side of this inequality exists, though it may be infinite).

Theorem A.2 (Theorem 5 in Maurer [2004]) . Let δ ∈ (0 , 1) and m ≥ 8 . Fix i ∈ [ n ] , and let P i be a probability measure on H i (chosen without seeing the training data). Then, with probability at least 1 -δ over the draws of S , for all probability measures Q i on H i , we have that

Proof of theorem 3.4. As remarked earlier, the function ( u, v ) - → kl( u ∥ v ) : [0 , 1] 2 → R does not exactly satisfy the hypotheses of lemma 3.1, but it is convex. Moreover, on { ( u, v ) ∈ [0 , 1] 2 | u ≤ v } , it is decreasing in its first argument and increasing in its second argument. Also note that, assuming that R ( g , Q ) ≥ e 2 ϵ R S ( g , Q ) , then we also have the following inequalities:

It follows that and therefore

by Jensen's inequality. Now, by theorem A.2, for a fixed i , with probability at least 1 -δ/n , we have that √

We can make the above inequality hold for all i ∈ [ n ] simultaneously with the union bound. Then, with probability at least 1 -δ , for all ( g , Q ) , given that R ( g , Q ) ≥ e 2 ϵ R S ( g , Q ) , we have that √

Proof of Lemma 4.1. Since the gating function satisfies ϵ -LDP, we have that e -ϵ g i ( x ′ ) ≤ g i ( x ) ≤ e ϵ g i ( x ′ ) for all x, x ′ ∈ X and all i ∈ [ n ] . It follows that e ϵ R S ( g , h ) ≥ E i ∼ g ( x ′ ) R S ( h i ) and e -ϵ R ( g , h ) ≤ E i ∼ g ( x ′ ) R ( h i ) . Given that ∆ is decreasing in its first argument and increasing in its second argument, we find that

Since ∆ is a convex function, we can apply Jensen's inequality to the expression on the right-hand side, yielding the desired result.

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our claims are supported by theorems, which we prove, and by experiments.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of the work in the conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All theoretical results are proved, either in the main paper or in the appendix. Moreover, all assumptions necessary for our theorems to hold are mentioned in their statement.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All datasets used are freely available, and we describe the architecture of the model and the hyperparameters of our experiments in detail. We also provide the code used to run our experiments as supplemental material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: All datasets used in our experiments are freely available. The code used to run our experiments is provided as supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Experimental details are provided in section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: In experiments, we had multiple runs and reported averages along with standard deviations.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide information on the type of compute workers (GPU) and time of execution in section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: In this work, we provide theoretical results that do not have any direct negative impact on society and do not pose any obvious ethical problem. We have ensured to cite our sources and adhere to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our paper has a theoretical orientation and no clear negative impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our models are trained on freely available datasets as a way to prove the efficacy of our approach, but they pose no risk as such.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We use openly accessible datasets and ensure to cite them properly when necessary.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The code is provided as supplemental material and the details are given in the README file.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve research with human subjects nor crowdsourcing.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

628

629

630

631

632

633

634

635

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper introduces a novel mixture of experts model that applies local differential privacy to the gating mechanism. Their methods leverages the one-out-of-n gating mechanism and provides specific generalization bounds.

STRENGTHS:
1. The overall insight of the paper is clear and strong.
2. Improve the tightness of bounds on the risk for mixtures of experts models.
3. Reduce complexity by relying on fewer parameters.

WEAKNESSES:
1. The section 3 talks about PAC-Bayesian bounds for mixtures of experts, but it lacks insight into why PAC-Bayesian bounds are applied instead of other bounds. More explanation is needed here, similar to the explanation needed in section 4 regarding Rademacher bounds.

2. In the experiment section, it is unclear why the chosen dataset is used for the experiments and why only 5 epsilon values were selected.

3. Even though there are very few existing guarantees, the experiment should include other methods as baselines and compare the results.
4. For the experiment section, only consider mixtures of n linear experts in binary classification tasks seems easy. Need to add other classification tasks.
5. There is no description about the datasets used in experiments.

QUESTIONS:
1. For section 3, why there is no reference for PAC-Bayes theory when you first mention it?
2. Why there is no reference about Kullback-Leibler divergence?

LIMITATIONS:
1. Lack many reference as I motioned in questions part. 
2. The paper lacks a smooth flow, making it difficult to follow. Specifically, there is no clear insight or reasoning provided to explain why the existing mechanisms or bounds were chosen, as highlighted in weakness 1.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel approach to regularize mixtures of experts by imposing local differential privacy (LDP) on the gating mechanism. The authors provide theoretical justifications and derive PAC-Bayesian and Rademacher bounds tailored to this approach. Experiments conducted on various datasets demonstrate that using LDP as a regularizer improves the generalization ability of the models, especially in cases prone to overfitting. The method offers a balance between leveraging neural networks for gating and maintaining robust theoretical guarantees, making it a valuable contribution to the field of machine learning.\n\nSTRENGTHS:\nThis paper demonstrates originality by integrating local differential privacy (LDP) into the mixture of experts model, addressing privacy concerns while improving model generalization. The theoretical contributions, including PAC-Bayesian and Rademacher bounds, are rigorously derived and tailored to the new approach. The clarity of exposition makes complex concepts accessible, and the experiments validate the practical benefits of the method. The significance lies in enhancing the robustness and scalability of mixture of experts models, making them more applicable to real-world scenarios prone to overfitting.\n\nWEAKNESSES:\nThe primary concern regarding this paper lies in its significance. There have been previous works that incorporated differential privacy (DP) into the construction of mixture of experts models with privacy considerations. This paper, however, utilizes local differential privacy (LDP) to analyze the theoretical aspects of mixture of experts models. The introduction of LDP significantly alters the generalization behavior of these models because both LDP and DP are methods that inherently enhance algorithm robustness, thereby affecting generalization. If the main goal of the paper is to enhance privacy, it is imperative to compare this approach with existing DP-based methods and highlight what specific aspects LDP protects that traditional DP cannot. Without this comparison, the added value of using LDP over existing DP methods remains unclear. On the other hand, if the focus is on analyzing the generalization of mixture of experts models, the paper must justify the rationale behind incorporating LDP for this analysis, as LDP is not inherently required for mixture of experts models. The paper needs to elaborate on why LDP is a suitable and necessary tool for this analysis and how it fundamentally impacts the generalization properties of the models in a meaningful way. Additionally, while the theoretical contributions are substantial, the practical implications need to be demonstrated more robustly through experiments. Comparing the results directly with models using traditional DP methods would strengthen the paper by showing the practical improvements and specific scenarios where LDP outperforms DP. Furthermore, using a broader range of datasets could better illustrate the claimed benefits in robustness and scalability. By addressing these concerns, the paper can more convincingly argue the necessity and advantages of using LDP in mixture of experts models, thereby enhancing its significance in the field.\n\nQUESTIONS:\nPlease refer to weakness.\n\nLIMITATIONS:\nNo.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides generalization bounds for a particular type of mixture of experts (MoE) networks. They focus on MoE architectures where an input $x$ first goes through a gating function $g$, and then gets routed to a single (one out of n) expert $i \\in [n]$ according to the $g(x) \\in [0,1]^n$ distribution. The final output of the MoE network is the output of the expert $h_i(x)$.\n\nThe authors observe that when the gating function $g$ has certain regularization properties, which correspond to local differential privacy (LDP), then the resulting network has better generalization bounds than what appears in the existing MoE litterature. The authors provide such bounds. \n\nFinally, the authors evaluate LDP-regularized routing on binary classification tasks with mixtures of linear models. The results show that LDP regularization outperforms an un-regularized baseline.\n\nSTRENGTHS:\nMixture of experts models are still understudied, especially from a theoretical standpoint, so I appreciate the new analysis provided by this paper. The connection with differential privacy is creative and seems fruitful, although I have some reservations about it (see below). \n\nThe bounds do improve on existing generic bounds for this type of MoE models. The paper is well-written and easy to follow, and the experimental code is available.\n\nWEAKNESSES:\nFirst, it is worth emphasizing that the MoE networks in this paper do not satisfy local differential privacy themselves. The gating network is not even *trained* with differential privacy. This paper only uses local differential privacy as a regularization condition on the *gating network only* at *inference* time, which does not provide meaningful privacy guarantees. That is not immediately clear from the title of the paper. To be fair, this work does not try to achieve any privacy goals, and is entirely focused on generalization bounds. But if privacy is not needed, then it is not clear why DP is the right tool for the job. The paper directly uses LDP (it could have been called something like \"exponentially regularized routing\") but does not motivate this choice. Are there other forms of regularization that could achieve similar or better bounds? While there are some known connections between robustness, differential privacy, and generalization, they are not mentioned here. In the context of MoEs, some large models already regularize their gating functions (e.g., the Switch Transformer adds some \"jitter\" noise to the routing logits). \n\nNext, the paper motivates the study of MoE models by mentioning recent progress with LLMs such as the Switch Transformer, which uses multiple layers of experts (deep MoE) and combines the output of different experts. All the modern LLM MoE models I am aware of are such deep MoEs. Meanwhile, the paper focuses on simple shallow MoE models with a single gating network followed a single layer of experts, which limits the potential impact of the paper in my opinion. While theoretical bounds may be of interest even on shallow MoE models, I would appreciate at least some discussion about whether the authors' approach can generalize to deeper models. \n\nFinally, the experiments have some limitations, which mostly stem from the two previous concerns. \n* The authors only evaluate a single, rather simplistic (3-layer MLP gating network followed by linear experts), MoE architecture. More concerningly, they use a fixed number of experts (n = 100), thereby missing an opportunity to evaluate their claim that \"we can have many more experts with almost no penalty from the theoretical point of view\".\n* The only baseline is \"No LDP\", which I think is a quite weak baseline. It is not entirely surprising that adding some regularization, in the form of LDP routing, improves generalization compared to a completely un-regularized baseline. How about other forms of regularization, such as dropout, clipping, or jitter noise (which already exists in the context of MoEs)?\n* Another baseline would be a non-MoE model, with a comparable number of parameters, e.g., even a simple, dense, multi-layer perceptron. Showing that shallow MoEs outperform dense models would alleviate concerns about the practical relevance of this work.\n\nMinor comments:\n* Table 1 might be more readable as a graph.\n* It is quite surprising to see MNIST being qualified as a \"large\" dataset, for which a 4-layer network takes 3 hours to train on a GPU, in 2024.\n* Also, it is unclear why MNIST has to be broken down into 3 binary classification tasks.\n\nQUESTIONS:\nHave you considered other forms of regularization (including the noisy routing techniques that are already used in practice by MoE)? Is there any justification why local differential privacy would be the type of regularization that gives the most desirable generalization bounds?\n\nLIMITATIONS:\nThe authors adequately addressed the limitation they identified (difficulty of tuning epsilon), even though this is not the main limitation of this work in my opinion.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider the mixtures of experts models, in particular the one-out-of-n gating mechanism for ease of theoretical analysis, and show that applying a soft-max, which is also the exponential mechanism, on the gating mechanism gives LDP and can improve generalization. The privacy techniques are largely the same as previous work, PATE, but specifically applied to mixtures of experts. The authors then provide theoretical analysis showing generalization bounds for this approach.\n\n\nUnfortunately, I’m not familiar enough with the mixture of experts literature to evaluate the novelty of applying the soft-max and the corresponding theoretical guarantees. I am rather surprised though that the soft-max has never been applied and am still somewhat confused upon what the previous techniques were in the one-out-of-n mixtures of experts.\n\nSTRENGTHS:\nThe authors show that choosing an expert through the soft-max provides better generalization. The privacy guarantee is then essentially proportional to the regularization factor (\\beta) for the soft-max application. Further they give theoretical generalization bounds for this approach.\n\nWEAKNESSES:\nUnless I am mistaken (please correct me if I’m wrong) the authors are not providing privacy guarantees for the model itself, but only one inference call to the model. In particular, if feature vector x is input to the model, then the expert is chosen randomly according to the soft-max / exponential mechanism, which is \\epsilon-LDP. If inference was then run again, suppose even on the same feature vector, then the random draw from experts would occur again. This is known as composition in the privacy literature, and the privacy guarantees\twould now be 2*\\epsilon. \n\nProviding privacy guarantees on only one inference call to a model is both not very useful nor interesting due to the composition properties.\n\nQUESTIONS:\nSee weaknesses section.\n\nLIMITATIONS:\nSee weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

comments: SUMMARY:
The authors  take the first step (I though so at first) to MoE under LDP theoretically.  I read again and found that the author seems to have raised the utility lower bound of existing studies. Few experiments could be found. Perhaps, I am not an expert in MoE, but it really leave a  hard time.

STRENGTHS:
Important poblem.

WEAKNESSES:
1. Perhaps I am not an expert in MoE, and I cannot tell from the author's introduction that there are any challenges. 
2. The experimental results and application scenarios are not clear.

QUESTIONS:
1. According to my rough understanding, the proposed approach seems to have some similarity with ensemble learning So what is the difference between the proposed approach and existing  ensemble learning under LDP?
2. MoE seems to be widely used in deep learning, so what are the differences between the proposed method and DP-SGD or LDP-SGD? Is there LDP-SGD-MoE?

LIMITATIONS:
see weakness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-20