Losing Less: A Loss for Differentially Private Deep Learning
Authors: Ali Shahin Shamsabadi
Nicolas Papernot
Venue: ICLR 2022 Submitted (2022)
Keywords: Differentially Private Deep Learning
DP-SGD

Abstract: Differentially Private Stochastic Gradient Descent, DP-SGD, is the canonical approach to training deep neural networks with guarantees of Differential Privacy (DP). However, the modifications DP-SGD introduces to vanilla gradient descent negatively impact the accuracy of deep neural networks. In this paper, we are the first to observe that some of this performance can be recovered when training with a loss tailored to DP-SGD; we challenge cross-entropy as the de facto loss for deep learning with DP. Specifically, we introduce a loss combining three terms: the summed squared error, the focal loss, and a regularization penalty. The first term encourages learning with faster convergence. The second term emphasizes hard-to-learn examples in the later stages of training. Both are beneficial because the privacy cost of learning increases with every  step of DP-SGD. The third term helps control the sensitivity of learning, decreasing the bias introduced by gradient clipping in DP-SGD. Using our loss function, we achieve new state-of-the-art tradeoffs between privacy and accuracy on MNIST, FashionMNIST, and CIFAR10. Most importantly, we improve the accuracy of DP-SGD on CIFAR10 by $4\%$ for a DP guarantee of $\varepsilon=3$. 

Under review as a conference paper at ICLR 2022
LOSING LESS: A LOSS FOR DIFFERENTIALLY PRIVATE DEEP LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Differentially Private Stochastic Gradient Descent, DP-SGD, is the canonical ap- proach to training deep neural networks with guarantees of Differential Privacy (DP). However, the modifications DP-SGD introduces to vanilla gradient descent negatively impact the accuracy of deep neural networks. In this paper, we are the first to observe that some of this performance can be recovered when train- ing with loss tailored to DP-SGD; we challenge cross-entropy as the de facto loss for deep learning with DP. Specifically, we introduce a loss combining three terms: the summed squared error, the focal loss, and a regularization penalty. The first term encourages learning with faster convergence. The second term empha- sizes hard-to-learn examples in the later stages of training. Both are beneficial because the privacy cost of learning increases with every step of DP-SGD, The third term helps control the sensitivity of learning, decreasing the bias introduced achieve state-
by gradient clipping in DP-SGD. Using our loss function, of-the-art tradeoffs between privacy and accuracy on MNIST, FashionMNIST. and CIFARI0. Most importantly, improve the accuracy of DP-SGD on CIFARIO by 4% for DP guarantee of é = 3.
INTRODUCTION
Releasing machine learning (ML) models may risk the privacy of training data as ML models unin- tentionally leak information about the training data; this is due to limitations of learning, including overfitting (Song & Shmatikov 2019) and data memorisation (Fredrikson et al. 2015 Carlini al.) 2019). The framework of Differential Privacy (DP) by Dwork et al. (2014) is the gold standard for formalising privacy guarantees. When training ML models, a DP training algorithm provides guar- antees bounding the leakage of private data that may occur during training. This requires that one bound the influence of individual training examples on the output of the algorithm (e.g. parameters gradients of the model). In addition these strong theoretical guarantees, DP models are empiri- cally resistant to various attacks - membership inference attacks (Shokri et al.) [017 Rahman et al. 2018), training data extraction (Carlini et al. 2019) and data poisoning attack (M et 2019).
Differentially Private Stochastic Gradient Descent, DP-SGD, trains a ML model under the frame- work of DP by (1) clipping per-example gradients fixed norm to bound their sensitivity and (2) adding noise to these clipped gradients before they are applied to update the model parame- ters (Abadi et al. 2016). When done separately and the level of a minibatch, clipping (Menon et al., 2019 Zhang et al. 2019) or noising (Foret et al. 202T) effectively regularize learning because they respectively control the dynamics of iterates and smoothen the loss landscape. However, gradi- ent clipping and noising are detrimental to model performance in DP-SGD because they are applied in combination and the granularity of individual training examples. In addition to this, the aceu- racy of DP-SGD is negatively affected by the limited number of iterations that can be computed: each iteration of SPD increases the privacy budget expended by learning.
Recent works have explored bounded activation functions (Papernot et al. 2021), publicly available feature extractors and data (Tramer & Boneh 2021). randomised smoothing (Wang et al. (021)) and gradient embedding Yu et al.) 1021 as a means to improve the tradeoffss between privacy and accuracy of DPDSGD. However, training deep neural networks with strong DP guarantees in DP- SGD still comes significant cost in accuracy for datasets like CIFAR10.

---

Under review as a conference paper at ICLR 2022
In this paper, show that one key aspect of formulating the optimization problem solved in learning is left out of studies seeking to improve deep learning with DP: the loss function. All existing implementations of deep learning with DP we aware of optimise cross-entropy (which performs well in the non-privacy-preserve
setting with SGD). However, we first observe that optimising cross-entropy with DP-SGD leads exploding model weights, layer pre-activations and logit values. This is true even if the activation functions themselves are bounded (Papernot et al. (0211. This phenomenon makes difficult to control the sensitivity of the learning algorithm a minimal impact to its correctness, Indeed, clipping and noising large gradients in DP-SGD introduces an information loss and additional biases. More precisely, the direction of a minibatch of clipped per-example gradients in DP-SGD is not necessarily aligned with the direction of the original gradients in SGD. Furthermore, we also note that the slow training convergence of cross-entropy (Allen-Zhu et al.) 2019) negatively impacts the performance of DP-SGD for which a limited number of iterations are required to achieve strong privacy guarantees.
In this paper, tackle these two limitations of the cross-entropy loss, we propose to tailor the loss function to the specifics of DP-SGD. We design a novel loss function that takes into account sensi- tivity, training convergence and the order in which examples are learned with the overarching goal of improving the tradeoffs between privacy and accuracy of DP-SGD. We achieve these objectives with novel loss function that combines the Sum Squared Error (SSE) between the model logits and ground-truth labels, the focal loss (Lin et al. (017). and a regularisation penalty on pre-activations.
The first two penalties impose curriculum structure in the training improve privacy-accuracy tradeoff's of DP-SGD by exploiting the generalisation and fast convergence speed (Wang et al. 2019 Guo et al. 2018) of curriculum learning (Bengio et al. 2009). We start training with SSE which has Taster convergence (Allen-Zhu et al.] 2019) and smaller gradients than cross-entropy. Then, we gradually shift toward emphasising hard-to tard-to-learn examples using the focal loss: this down-weights gradients assigned to examples that are already learned correctly to focus more on hard-to-leamy and misclassified examples. Finally, our regularisation penalty prevents the weights from exploding to further reduce the magnitude of per-example gradients prior to clipping. Implementation-wise -ise, our new loss function can be easily integrated within existing implementations of deep learning with DP; it requires single line change edit the loss function being optimized.
In summary, our main contributions are as follows:
We are the first to investigate the loss function in the context of deep learning with DP. We analyse the effect of the loss function on the sensitivity of DP models by providing extensive analysis of the norm of activations, weights and gradients in Section
We propose a novel loss function tailored to deep learning with DP. In Section and Sec- tion 4 we analytically and experimentally connect the superior performance of our pro- posed loss function to the gradient norm and direction, smoothness of the loss surface, and convergence of DP-SGD, Our proposed loss function better controls the sensitivity of the learning algorithm by better preconditioning gradients to being clipped and noised.
We show in Section 4 that our proposed loss function is compatible with other DP-SGD improvement strategies, and more importantly allows us to establish new state-of-the-an art tradeoffs between privacy and accuracy several key benchmarks for deep learning with DP, including CIFAR 10. Finally, we perform an ablation study to analyse the effect of each component of our proposed loss function on DP learning.
2 DIFFERENTIAL PRIVACY, DP-SGD AND -F APPROACHES
Trained machine learning models memorise and leak information about their training data (hhkri al. 2017: Rahman et 2018 Song & Shmatikov 2019, Fredrikson al., 2015). Differential privacy (Dwork et al. 20144) is the established gold standard to reason about the privacy guarantees of learning algorithms. An algorithm is differentially private if its outputs are statistically indistin- guishable on neighbouring datasets. More formally, randomised learning algorithm A is (c, (), differentially private if the following holds for any two neighbouring datasets D and D' that differ only in one record and all S E RA)ge(A):
Pr(A(D) S] < Pf((A(DD E S)
2

---

Under review as a conference paper at ICLR 2022
The privacy budget upper bounds the privacy leakage in the worst possible case: the smaller the E. the tighter the upper bounds (or stronger privacy guarantee). Note that the factor d is very small (generally it is chosen to be in the order of the inverse of the dataset size).
In order to train ML models with DP. we can add randomness to the learning algorithm in three ways: output perturbation (Wu et al.) 2017; Zhang et al. 2017). objective perturbation (Chaudhuri et al. 2011 Iyengar et al. 2019) or gradient perturbation (Bassily et al.) 2014 Abadi et al.] 2016).
Among these approaches, differentially private stochastic gradient, DP-SGD of Abadi et al. (0166)
has established itself as the de facto strategy because of its versatility. DP-SGD perturbs gradients computed each step of stochastic gradient descent using the Gaussian mechanism to achieve competitive tradeoffs between privacy and accuracy (Yu et al. 2020; Tramer & Boneh 2021 Next, we describe the process of training a classifier using DP-SGD on private dataset.
Let X' = fx:li = [isj==1,..... N] be the training set containing N private examples X; and V = i = 1.... .V be the ground-truth label set where each y; a D-dimension one-hot en- coded vector. We consider D-class classifier containing M layers parameterised with weights W = -W""|m I,.,,
In each DP-SGD iteration, similarly to the non-private SGD one, the classifier receives each training example x, from a minibatch containing data points that are sampled randomly from X. The activation of each intermediate layer of size dm, a." f(h,") R1xdm, is computed by applying non-linear function, f((), on the pre-activation h, = WT The pre-activation is linear combination of the weight of that layer, W and the activation of the previous layer, a." a!"=!. The last layer outputs the logit values associated with each class without any activation function as a. M as
W M
h M-1
RIxD, The cross-entropy loss, LeE, for each training example x; is computed
D CCE = --vEaa log(p where P(c.d) = Σd
(F.a)
(2)
d=1
The probability that x; belongs to d-th class, P(i.d) E [0, is computed by applying a Softmax function on the logit value of its corresponding class a.
The DP-SGD optimiser computes per-example gradient of LEE with respect to W, opposed to per minibatch gradient in the SGD optimiser, to bound the influence each individual training example on the output of the learning algorithm (i.e. weight gradients). The gradients, G for each training example x, are computed from the last layer and back prok-propagated until the first layer as:
G. = ((p: y:) WM) T a GM--2 = f'( - y.) W W aMM a M-3
G = (p: - y:)
M-2
(3)
One can continue this gradient computation until the first layer. All the per-example gradients are concatenated as G; [G!:..: G. where: denotes the concatenation operation. As there is no priori bound on G, the l2 norm, || 1l2, of each G, are artificially clipped by C' to bound the influence each x; on the final gradients, G:
1 7 G. G.NN(0, where G; + G min(1,
C
G
(4)
GG
Finally, the classifier weights are updated through the average of the clipped and noisy (scaled by the noise multiplier σ) per-example gradients of minibatch.
The accuracy of classifiers trained with SP-SGD is lower than that of classifiers trained by non- private SGD. Recently, researchers investigated the choice of f(-) and data representation for im- proving the accuracy of the classifier trained by DP-SGD. Papernot et al. (2021) demonstrated that exploiting a bounded family of activation functions instead of the more commonly employed un- bounded ReLU activation for the choice of f()) can decrease the bias introduced by DP-SGD and improve the tradeoffs between privacy and accuracy of DP-SGD. Their approach though does not fully bridge the gap between non non-private and private learning for datasets like CIFAR10. [Tramer representation of data outputted by public SeatterNet feature extractor as opposed to using the raw pixels X,. This however requires access to public data in addition to the private dataset.
& Boneh (2021) proposed to train the classifier
3

---

Under review as a conference paper at ICLR 2022
3 PROPOSED APPROACH
We propose to design a new loss function tailored to DP-SGD. Our goal to converge faster with smaller gradient norms, control better the sensitivity of the learning algorithm, increase tolerance to noise, and effectively learn both easy and hard examples. Indeed, these address several crucial differences between DP-SGD and its non-private counterpart SGD:
Due per-example gradient clipping (recall Equation Z.. information contained in gradi- ents whose magnitude is too large is discarded (see Figure [c). This cannot be compen- sated by tuning the training algorithm, e.g., by increasing the model's learning rate. This is because clipping is done at the granularity of individual training examples. Together with the noise injected (recall Equation D. this biases learning and implies that DP-SGD takes a different optimisation path compared to SGD.
As shown in Equation 3 the model weight contribute to the computation of all gradi- ents. In practice, this leads to model weights exploding in DP-SGD. Therefore, one way to decrease the magnitude of gradients to prevent the model weights from exploding, especially the weight of the last layer which contributes in the gradient of all the layers.
The number of training iterations are limited in DP-SGD because each iteration increases the risk of privacy leakage. Hence, faster convergence is beneficial to DP-SGD as well as ensuring both easy and hard examples are attended to during the limited training run.
Next, we describe our choice of loss function. We analyse how our new loss function improves the tradeoff's between privacy and accuracy of DP-SGD, We uncover several effects: our loss limits the information loss of gradient clipping, speeds up convergence, recovers generalisation boost comparable the one achieved by batch normalisation, and improves learning's 's tolerance noise.
3.1 OUR LOSS
We propose to learn the easy examples at the beginning of the training using the per-example Sum Squared Error (SSE) between the logit values and one-hot vector labels:
h y:ll3, where ||h. y.11² Eunda )(
LSSE
(5)
d=1
Later in training, we exploit the focal loss CFoca to learn hard-to-learn examples. C modifies the cross-entropy loss to reduce the loss of easy. cels-classified examples, letting the model focus more on hard, misclassified examples. Frocal multiplies factor (1 P((.))) based on the probability of the ground-truth class P(t.t) to the cross-entropy of each training example x; as:
Lroal -(1 P( Σ v(s.d) log(P(..)));
(6)
d=1
where the tunable focusing parameter 7 adjusts the doighweggh rate: the higher the y, the higher the wein-teight rate of easy, well-classifedd examples. Note that the focal loss is equivalent to cross-entropy loss, when 7 = O.
To further decrease the gradient magnitudes and prevent the explosion of intermediate weights, we impose a regularisation penalty on the intermediate pre-activations as:
M-1
CRSE
1
m=1
Finally, our proposed loss function L combines LFocal, LSSF, and LReg as:
L = Crocall + (1 L + (1 a:)
LReg
(8)
B
Sigmoid(e. ez) (current epoch, 6c. and threshold epoch, ex) to enable curriculum learning where easy examples are learned in the early training iterations using LSsE which eases learning of hard examples in the later training iterations using Crocal. We perform a hyper- parpmetr search to set the best values for a, 3 and n
where we set hyper-parameter
Note that all the terms except those corresponding weights are hounded in Equation 3
4

---

Under review as a conference paper at ICLR 2022
12
10a
Counts (log)
10
12 mnom
12 norm
10a
10
10a
10
10
10
1 10 20
30
10
20
30
20 30
40
2 4 .6 8
Training epochs
Training epochs
L norm
Angle
(a) Logit values
(b) Weights
(c) Gradient
(d) Path divergence
Figure I: lg norm of the logit values (a), l2 norm of the last layer weights (b), histogram of per- sample gradient magnitude in the first epoch (c), and divergence histogram of optimisation path in aggregated clipped gradients from the optimisation path unclipped ones (d) when we train the classifier CIFARIO 1O using cross-entropy loss our proposed loss function. Minimising cross entropy loss using DP-SGD leads to logit exploding (a) and weight exploding (b). However, our proposed loss function prevents logit exploding (a) and weight exploding (b). Therefore, per-sample gradient magnitudes (c) obtained based on our proposed loss function is smaller than per-sample gradient magnitudes obtained based on cross entropy loss. In addition this, the per-sample gradi- ents of our proposed loss function is much more condensed. Finally, direction of the gradients (d) of our proposed loss function is more aligned with the unclipped gradients.
3.2 AN ANALYSIS OF OUR LOSS
Our loss limits information loss from clipping. Figure [ shows that under the supervision of L, we are able to control the sensitivity of the learning algorithm. In particular, LReg the intermediate pre-activations and LSSE on the logit values both prevent logit values and weights from exploding, which decreases the magnitude and variance of per-example gradients (see Equation 3). Smaller and more condensed gradients decrease the negative impact of gradient clipping on the trajectory of the gradient descent shown in Figure d.
Our loss yields faster convergence. LSSE improves the tradcoffs between privacy and accuracy of DP-SGD with strong privacy budgets LSSE converges exponentially O(e where / is the steps, to loss, while cross-entropy loss only converges at O as shown in Allen-Zhu et al. (2019). Our loss achieves an effect similar to batch normalisation. The magnitude of gradients com- puted when optimising LSSE with DP-SGD is smaller than for gradients computed when optimising cross-entropy, as LSsE defined on logit values hM. This prevents logit values and consequently WM from exploding (per-example logit values are computed based on the weight of the last layer). Controlling WMM can limit the changes of weights in other layers as their gradients are a function of W and per layer gradient function is locally lipschitz. Preventing logits from exploding may also help recover the generalisation boost of batch normalisation (Dauphin & Cubuk (202T). Indeed, batch normalisation cannot be used in differentially private learning because the technique shares information across different training examples contained in single minibatch. This violates the pri- vacy analysis of DP-SGD. Therefore, our proposed LSSE penalty may also be beneficial to learning with DP-SGD because it achieves similar effect to batch normalisation (Dauphin & Cubuk 2021 Our loss improves tolerance to noise. Wang et al. (202T) showed that smoother loss functions improve the generalisation bounds and accuracy of DP learning by making the loss surface more tolerant to noise. The most common smoothness notion is based on the Lipschitz constant of the gradient of a function.
Definition (Smoothness (Nesterov) 2003)) A loss function is B-smooth if
| VL l|2 B||l w
where V. is the gradient the loss function with respect to the weights and / is the smoothness constant. Smaller 3 indicates smoother function.
We now derive a lemma illustrating how our loss also favours smoothness of the loss being optimised by DGD.
Lemma When singular values are bounded by Tmax- we have for all w and w' in a simply con- nected set that L V Lew Fmax || w/ ||2-
5

---

Under review as a conference paper at ICLR 2022
Proof.
Let Hu be the hessian of the loss function with respect to the weights w.
=11/ IIc/" Hwrt((--w") wo) (w - -w' w''dt)ll2 - L.' H (ww+t(u-wrr) (w- w') 1l21
VV - VV 16wrl12-114
=
h-" mmax | l2 Omarl!u w
(9)
where max is the maximum of the first singular value along the line from w to w/.
Therefore, the existence of local bounds on the singular values results in local bounds on the smooth- ness. We empirically observe Figure 2 that the singular values our loss function are smaller than cross-entropy loss function. This is particularly true the beginning of training. This, thus, suggests that our proposed loss function is smoother and more noise-tolerant than cross-entropy.
4 VALIDATION
15
We validate the benefit of deploying our proposed loss function in improving how DP-SGD tradeoff's privacy and accuracy. To do so, we show our loss further im- proves the state-of-the-ard results of the two approaches for deep learning with DP described in Section 2 CE-T-CNN (Papernot et al. 202I): using the cross- entropy loss function together with an end-to-end CNN classifier equipped with Tanh activation functions; 2)
Omix
10
0
0
100 200
300
400
Training iterations
Figure 2: Analysing the smoothness of the landscape of our proposed loss func- tion and cross-entropy on MNIST. Our loss function obtains smaller mmx (the maximum of the first singular value), making it more smooth than cross en- tropy.
CE-T-PFE. CNN (Tramèr & Boneh, (02T): using the cross-entropy loss function, this time combining the Scat- terNet public feature extractor (PFE) with private CNN classifier whose internal activations are also Tanh. To en- sure fair evaluation, PaSH and CIFAR10, and the same classifiers (described in Appendix (A) as these two baseline ap- proaches. In addition to this, we fix the DP-SGD configuration across all approaches to the best hyperparameter values reported in the hyperparameter search done by Tramer Boneh (202T) (see Appendix B). Note that our results would only be improved by additional hyperparameter search on the SP- configuration. For the hyperparameters introduced by our own loss function, we use search on the values as described in detail in Appendix B We note that none of the prior work we compare against captures the privacy cost of the hyperparameter search itself when reporting DP guarantees achieved. The cost of this search should however be moderate, analysed by Liu & Talwar (0019).
the same datasets, MNIST,
4.1 IMPROVING THE TRADEOFFS BETWEEN PRIVACY AND ACCURACY
Figure 3 and compare the privacy-accuracy tradeoff's achieved by our proposed loss function and prior approaches for DP training. Coloured regions represent the range of model accuracy with respect the privacy budget needed achieve this accuracy. Experiments are repeated 5 times; the minimum, median and maximum of the 5 experiments are each highlighted by line.
When learning without privacy, end-to-end classifiers achieve a test accuracy on CIFAR1O, Fash- ionMNIST and MNIST of 76.6%, 89.4% and 99.0%, respectively. Instead, when learning with DP-SGD, the maximum test accuracy (across runs) of our first baseline CE-T-CNN for & < are 59.8%, 86.9%, 98.2% on CIFARIO, FashionMNIST and MNIST, respectively. This illustrates how CIFARIO is the most challenging dataset for DP-SGD training among the datasets we considered: there is about 20% gap in accuracy between the privacy-preservigg and non-privacy-preservvvi tings. Our proposed loss function reduces this gap significantly, even more so for CIFAR 10 than simpler datasets like FaShionMNIST and MNIST. For example, training an end-to-end classifier using our loss function on CIFARIO achieves 63.2% test accuracy, instead of 59.8% with the cross- entropy loss. While our second baseline CE-T-PFE CNN sees stronger accuracy than the first
set-

---

Under review as a conference paper at ICLR 2022
CIFARIO
FashionMNIST
MNIST
65
90
100
Test accuracy
55
80
90
45
70
80
35
25
60
70
1
1.5
2
2.5 3
1
1.5 2
2.5
3 1
1.5
2
2.5
3
Privacy budget e
Privacy budget E
Privacy budget
Figure 3: Deployment of our loss function in DP-SGD training of an end-to-end classifier for runs.
Comparison between the test accuracy of CE CE-T-CNN (Papernot et al. 2021 and O T CNN, as a function of privacy budget e.
ours
CIFARIO
NaSH
MNIST
70
100
Test accuracy
60
85
95
80
50
90
40
70
30
65
85
1.5 2
2.5
3
1
1.5 2
2.5
3
1.5
2
2.5
Privacy budget
Privacy hudget
Privacy budget
Figure 4: Deployment of our loss function in DP-SGD training of public feature extractor followed by classifier for 5 runs. Comparison between the test accuracy of CE-T-PFE_CNT (Tramèr & Boneh 2021 and O T--T--P-- PFE_CNN, ours as a function of privacy budget
baseline CE-T-CNN because the learner additionally has access to public data, our proposed loss function also improves its ability to tradeoff privacy and accuracy across the entire regime.
The improvement of our proposed loss function to privacy-accuracy tradeoffs is strongest in the initial epochs. These epochs correspond to stronger privacy guarantees (i.e., smaller values of e) because the privacy guarantee of each individual step of DP-SGD needs to be composed across the entire training run; hence each step further increases the privacy budget expended. For example, our proposed O T- CNN trains an end-to-end classifier with 55.4% test accuracy for s < 1.5, compared 45.9% only with our first baseline CNNN. In another example, C-T-PFEE.COO test accuracy of our second baseline 28-T-PFE_CHE from 64.5% to 66.8% for E < 2.
improves the
4.2 ANALYSIS OF GRADIENTS COMPUTED BY DP-SGD FOR O-T-CNN
We first analyse the impact the loss function on the model weights, pre-activations, and logit val- ues. Figure 5 (first row) shows the impact our loss function has model weights. We visualise the average l2 norm of model weights at each epoch. Across the three datasets, optimising CE-T-CNN with DP-SGD leads to model weight explosion. This is not the case when learning with our pro- posed loss. In particular, note how O-T-CNN prevents last layer weights from exploding, which important because they contribute to the gradients for all layers. Next, Figure [5 (second row) shows the impact of our loss function on pre-activation and logit norms. Although CE-T-CNN uses bounded tanh activation functions to prevent activations from exploding, the pre-activations and logits still explode because model weights are unbounded and explode themselves (as visual- ized above). Instead, replacing the cross-entropy loss function with our loss function decreases this phenomenon for both pre-activations and logits. This is explained by our pre-activation regulariser penalty but also the fact that the sum-squared-erme penalty is defined over logits.
Reducing the bias of gradient clipping. We provide the histogram of per-example gradient l2 norms in Figure 6 For better visualisation, clip the x-axis between [0, 100) and use log scale for the y-axis. The gradient l2 norms of O-T-CNN are smaller than their counterpart for CE-T-CNN. This is consequence of our observations above: our proposed loss function prevents several terms including in the gradient computation from exploding. To further demonstrate the importance of better preconditioning gradients the clipping performed by DP-SGI,, we visualize the divergence
7

---

Under review as a conference paper at ICLR 2022
Second layer (C)
Last layer (C)
Second layer (F)
Last layer (F)
15
12
12 norm
12
10
8
9
1
3
1
10
20
30
1
10
20
30
1
10
20
30
40
10
20
30
40
120
12
80
20
12 norm
90
9
16
12
6
40
60
3
20
8
30
0
0
10
20
30
10
20
30
1
10
20
30
40
10 20 30 40
Training epochs
Training epochs
Training epochs
Training epochs
Figure 5: l2 norm of weights (first row) and pre-activations (second row) in CE-T-CNN and O-T-CNN using CIFAR10 (C) and FashionMNIST (F). See Appendix MNIST and other layers.
C for results of
Figure 6: Histogram of l2 norm per-example gradient CE-TCCCTTTO CNN and O O-T CNN on CIFARIO. 10. Histogram for other epochs and other datasets are presented in Appendix E
CIFARIO
FaShionMNIST
MNIST
80
20
20
Counts
60
15
40
10
20
5
0
0
-4-22
0
2
4
6
.8
-.4 0
2
4
6
.8
-.4-2 0
2 .4
6
88
Cosine
Cosine
Cosine
Figure 7: Cosine similarity between aggregated of clipped per-example gradients and aggregated of unclipped per-example gradients first epoch of CE-T-CNN and O-T T-NN
between the path taken by aggregated clipped per-example gradients and aggregated unclipped per- example gradients in Figure 7 For each minibatch, we compute the cosine similarity between the average of per-example clipped gradients and the average of per-example unclipped gradients. The direction of clipped gradients in O-T-CNN is more aligned with the direction of their unclipped counterparts than is the for the baseline approach. This suggests that our proposed loss function decreases the information loss and bias introduced by DP-SGD.
Learning hard vs. easy examples. Finally, we visualise the confusion matrices of per-class train- ing accuracy for non-private learning with SGD, private learning with the CE-T-CNN baseline, and private learning with our approach O-T-CNN in Figure [. Qualitatively speaking, the per-class training accuracy of O-T-CNN is closer to the non-private one than the case for the baseline CE-T-CNN. During the initial epochs of learning, O-T-CNN converges faster especially on easy- to-learn examples thanks to the sum-squared-error loss function. For example, the training accuracy of O-T-CNN for "car" and "truck" are 51% and 48%, while CE-T-CNN only achieves 34% and 37%, respectively, on these classes. In addition to this, O-T-CNN deals better with hard-to-lean ex- amples (e.g. "cat", "dear" "dog" classes) in later epochs thanks the focal loss. For example, the training accuracy of "cat" class is 49% at epoch 20 of O-T-CNN but is only 35% for CE-T-CNN.
4.3 ABLATION STUDY
Next, we present an ablation study to tease out the contribution of each of the three components of our proposed loss when it comes to improving privacy-accuracy tradeoffs in deep learning with
8

---

Under review as a conference paper at ICLR 2022
Figure 8: Confusion matrices that show per-class training accuracy of O-T-CNN (first column), CE-T-CNN (second columns) and non-private SGD approach (last columns). The first and second row show confusion matrices of 10 at epoch 2 and 20, respectively. See Appendix D for the results of other epochs as well as FashionMNIST and MNIST datasets.
CIFAR10
FashionMMNIT
MNIST
65
90
100
Test accuracy
55
80
90
70
35
25
60
70
1
1.5
2
2.5
1.5
2
2.5
3
1.5
2
2.5
3
70
90
100
Test accuracy
60
80
95
50
90
40
70
30
85
1
1.5 2
2.5
1
1.5 2
2.5 3
1.5
2
2.5
Privacy budget E
Privacy budget E
Privacy budget
Figure 9: Ablation study of DP-SGD training of an end-to-end classifier (first row) and public feature extractor followed by classifier (second row) for 5 runs. Loss functions under study: CE, SSE, SSE+CE, SSE+Focal and SSE+Focal-RRggllaissr.
DP, namely sum-squared-erros focal loss and regularisation penalty. To do that we consider four combinations of these penalties: only using sum-squared-erroor, combination of sum-squared-error and cross-entropy, combination of sum-squared-error and focal loss, and finally our proposed loss function that combines all sarec-sum-squared-eee other combinations because they are not meaningful. To ensure fair evaluation, we perform hyperparameter search for the combination of uun-squrred-error and cross-entropy as well as the combination of sum-squared-error and focal loss, similarly to the search performed for our pro- posed loss function (see Appendix
focal loss and regulariser. We do not consider
B). The privacy-accuracy tradeoffs of these four combinations shown in Figure 9 While the sum-squared-error improves the crivacy-accurace tradeoff in initial epochs, the focal loss improves it for later epochs. Finally, imposing a regularisation penalty on pre-activations further improves these aaccuryy tradeoffs. Our proposed loss thus achieves the best possible performance across each these settings and none its penalties can omitted.
5 CONCLUSION
In this paper, we showed that designing a loss function tailored to the specificities DP-SGD, namely per-example gradient clipping and limited number of iterations, significantly improves tradeoffs between privacy and accuracy. We reduced the bias introduced by gradient clipping using pre-activation regularisation term. In addition to this, improved the convergence speed by imposing a curriculum structure in learning with the sum-squared-erre and focal loss. As future work, will evaluate loss function for SGD in other domains such text.

---

Under review as a conference paper at ICLR 2022
6 REPRODUCIBILITY STATEMENT
We submit our code as the supplementary material. Our code is based on the public Opacus library in Py PyTorch. We use the publicly available datasets. For the theoretical smoothness result, we provide clear explanations of lemma and complete proof.
7 ETHICS STATEMENT
Our work improves the tradeoffs between privacy and aceuracy of training machine learning models in Privacy-rrsservig setting. Therefore, we expect the broader impact of our work to be generally positive especially in privacy-sensitive applications such health-care language modeling where we work with sensitive datasets in practice.
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings the ACM SIGSAC Conference Computer and Communications Security (CCS), Vienna, Austria, October 2016.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. convergence theory for deep learning via over- parameterization. In Proceedings of the International Conference Machine Learning (ICML), Long Beach, California, USA, June 2019.
Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In Proceedings of the IEEE Symposium on Foundations of Computer Science (FOCS), Washington, USA, October 2014.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the International Conference on Machine Leurning (ICML), Montreal Quebec, Canada, June 2009.
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Eval- uating and testing unintended memorization in neural networks. In Proceedings of the USENIX Security Symposium, Santa Clara, California, USA, August 2019.
Kamalika Chaudhuri, Claire Monteleoni, and Anand D Sarwate. Differentially private empirical risk minimization. Journal of Machine Learning Research (JMLR), 12(3):1069-109, 2011.
Yann Dauphin and Ekin Dogus Cubuk. Deconstructngg the regularization of BatchNorm. In Pro- ceedings the International Conference on Learning Representations (ICLR), Virtual, May 2021.
Cynthia Dwork, Aaron Roth, al. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3-4):21 1-407, 2014.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimiza- tion for efficiently improving generalization. In Proceedings of the International Conference on Learning Representations (ICLR), Virtual, May 2021.
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confi- dence information and basic countermeasures. In Proceedings the ACM SIGSAC Conference on Computer and Communications Security (CCS), Denver Colorado, USA, October 2015.
Sheng Guo, Weilin Huang, Haozhi Zhang, Chenfan Zhuang, Dengke Dong, Matthew R Scott, and Dinglong Huang. Curriculummet: Weakly supervised learning from large-scale web images. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, September 2018.
Roger Iyengar, Joseph P Near, Dawn Song. Om Thakkar, Abhradeep Thakurta, and Lun Wang. To- wards practical differentially private convex optimization. In Proceedings of the IEEE Symposium Security and Privacy (SP), San Francisco, California, USA, May 2019.
10

---

Under review as a conference paper at ICLR 2022
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy, October 2017.
Jingcheng Liu and Kunal Talwar. Private selection from private candidates. In Proceedings of the ACM SIGACT Symposium on Theory of Computing, Phoenix, Arizona, USA, June 2019.
Yuzhe Ma, Xiaojin Zhu, and Justin Hsu. Data poisoning against differeentiallyppiie pprivate learners: At- tacks and defenses. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI, Macao, China, August 2019.
Aditya Krishna Menon, Ankit Singh Rawat, Sashank J Reddi, and Sanjiv Kumar. Can gradient clipping mitigate label noise? In Proceedings of the International Conference on Learning Rep- resentations (ICLR), New Orleans, Los Angeles, USA, May 2019.
Yurii Nesterov. Introductory lectures convex optimization: A basic course, volume 87, Springer Science & Business Media, 2003.
Nicolas Papernot, Abhradeep Thakurta, Shuang Song. Steve Chien, and Ülfar Frlingsson. Tempered Sigmoid activations for deep learning with differential privacy. In Proceedings of the Association for the Advancement of Artificial Intelligence (AAAI), Virtual, February 2021.
Md Atiqur Rahman, Tanzila Rahman, Robert Laganière, Noman Mohammed, and Yang Wang. Membership inference attack against differentially private deep learning model. Transactions on Data Privacy, 11(1):61-79, 2018.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference at- tacks against machine learning models. In Proceedings of the IEEE Symposium on Security and Privacy (SP), SAN JOSE, California, USA, May 2017.
Congzheng Song and Vitaly Shmatikov. Auditing data provenance in text-generation models. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery Data Mining (KDD), Anchorage, Alaska, USA, August 2019.
Florian Tramer and Dan Boneh. Differentially private learning needs better features (or much more data). Proceedings of the International Conference on Leurning Representations (ICLR), Vir- tual, May 2021.
Wenxiao Wang, Tianhao Wang, Lun Wang, Nanqing Luo, Pan Zhou, Dawn Song, and Ruoxi Jia. DPlis: Boosting utility of differentially private deep learning via randomized smoothing. Privacy Enhancing Technologies Symposium (PETS), (021(4):163-16) 2021.
Yiru Wang, Weihao Gan, Jie Yang, Wei Wu, and Junjie Yan. Dynamic curriculum learning for imbalanced data classification. In Proceedings the Vision (ICCV), Scoul, Korea, October 2019.
the IEEE International Conference on Computer
Xi Wu, Fengan Li, Arun Kumar, Kamalika Chaudhuri, Somesh Jha, and Jeffrey Naughton. Bolt-on differential privacy for scalable stochastic gradient descent-based analytics. In Proceedings of the ACM International Conference on Management of Data, Chicago Illinois, USA, May 2017.
Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, and Tie- Yan Liu. Gradient perturbation is underrated for differentially private convex optimization. In Proceedings of the International Joint Confer- ence on Artificial Intelligence (IJCAI). Yokohama, Japan, July 2020.
Da Yu, Huishuai Zhang, Wei Chen, and Tie-Yan Liu. Do not let privacy overbill utility: Gradient embedding perturbation for private learning. In Proceedings of the International Conference on Learning Representations (ICLR), Virtual, May 2021.
Jiaqi Zhang, Kai Zheng, Wenlong Mou, and Liwei Wang. Efficient private ERM for smooth ob- jectives. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), Melbourne, Australia, August 2017.
Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. In International Conference on Learning Rep- resentations (ICLR), New Orleans, Los Angeles, USA, May 2019.
11

---

Under review as a conference paper at ICLR 2022
Table 1: The architecture of end-to-end CNNs for Table 2: The architecture of end-to-end CNNs for
Fashion-/MNIST (Papernot et al. 2021).
CIFARIO (Papernol et al.) (0221)
Layer
Parameters
Layer
Parameters
Conv
filters of 8x8, stride 2, padding 2
Conv
32 filters of 3x3, stride 1, padding
MP
2x2, stride 1
MP
2x2, stride 2
Conv
32 filters of 4x4, stride 2, padding 0
Conv x2
filters of 3x3, stride 1, padding
MP
2x2, stride I
MP
2x2, stride 2
FC
32 units
Conv x2
128 filters of 3x3, stride 1, padding
FC
10 units
MP
2x2, stride 2
FC
128 units
FC
10 units
The architecture of CNN trained on of public ScatterNet feature extractor for Fashion- top of public ScatterNet feature extractor for CI- /MNIST (Tammrr& & Boneh, (02T).
Table 3: The architecture CNNs trained on top Table 4:
FAR10 (Tramèr & Boneh, (02T).
Layer
Parameters
Layer
Parameters
Conv
16 filters of 3x3, stride 2, padding 1
Conv x2
64 filters of 3x3, stride 1, padding 1
MP
2x2, stride 1
MP
2x2, stride
Conv
32 filters of 3x3, stride 1, padding 1
Conv x2
64 filters of 3x3, stride 1, padding
MP
2x2, stride
MP
2x2, stride 2
FC
32 units
FC
10 units
FC
10 units
A CLASSIFIERS
Table and Table 2 show the architecture of end-to-end CNNs for MNIST, FashionMNIST and CIFARTO, I0, suggested by Papernot et al. (202T). Table Tahle the architecture CNNs trained on ScatterNet features for MNIST, FashionMNIST and CIFARIO, suggested by Tramer & Bonch 2021
B HYPERPARAMETERS
Table reports hyperparameter values of the DP-SGD configuration across all approaches, sug- gested by Tramèr & Bonch (202I). repoot the hyperparameter values our proposed loss function, obtained through hyperparameter search values reported in Table 7 However, evaluate the performance of our loss function for each hyperparameter value using all three datasets in Figure 10 We can observe that the impact of hyperparameter values on the performance of our loss function is negligible. Therefore, the improvement of our loss function is not sensitive to the hyperparameter values.
Table 5: Training hyperparameter values obtained through line search Tramer & Bonch 2002 ).
Dataset Learning rate Momentum Batch size Epochs Noise scale Clipping norm
End-to-end CNN
CIFAR10
0.9
1024
30
1.54
0.1
Fashion-MNISS
0.9
2048
40
2.15
0.1
MNIST
0.5
0.9
512
40
1.23
0.1
CNN trained on ScatterNet features
CIFAR1O
0.9
8192
60
5.67
0.1
FaShion-MNIST
4
0.9
2048
40
2.15
0.1
MNIST
0.9
1024
25
1.35
0.1
12

---

Under review as a conference paper at ICLR 2022
Table 6: The value of the hyperparameter of our loss function.
CIFARIO
FashionMINIST
MNIST
O T CNNN
6t 7. 11, 71
=
e, 11=0,7 0. 7
6, 0.9-1.7=9
O-T-PFE.CNN
6 = 1-=1-11---
6 e 2, 51,
et = 1
Table 7: Hyperparameters for proposed loss function.
Parameter
Value
C
Start=0, End=10, Step-size=1
B
Start=1, End=200, Step-size=10
C 12 NORM OF PRE-ACTIVATION AND WEIGHTS
Figure 11 and Figure 12 show the l2 norm of weights and pre-activation of models trained by DP- SGD approaches on CIFARIO, FasHionMNIST and MNIST.
D CONFUSION MATRICES OF PER-CLASS ACCURACY
Figure visualises the confusion matrices of per-class training accuracy for non-private learning
with SGD, private learning with the CE-T-CNN baseline, and private learning with our approach O-T T CNN using CIFARIO, FashionMNIST and MNIST.
E PER-EXAMPLE GRADIENTS
Figure 14 provides the histogram of per-example gradient l2 norms CE-T TE-T-CNM and O-T T-CNN on CIFAR10, FasHionMNIST and MNIST.
F COMPARING SSE WITH SAE AND HUBER
In this section, we analyse the performance of SSE, Sum Absolute Error (SAE) and Huber-Loss. in DP-SGD training.
The SAE is defined as:
D Σ d=1
LSAE = [h:" M5 - yil- where [h
yıl
10 - B(rd))
(10)
The d-th element of Huber loss defined as:
[daber(d) = 1 ( V(idd) [(h M((.))) 1/3
(( Mir.)) <B
(11)
otherwise
Figure 15 shows the tradeoffs between the privacy and accuracy of DP-SGD training using SSE, SAE and Huber loss. The performance of SSE is better than SAE. The performance of Huber loss increases as we increase 3, activating SSE part and deactivating SAE part of Huber loss in Equation II
13

---

Under review as a conference paper at ICLR 2022
CIFARIO
MasH
MNIST
53
Figure 10: Test accuracy of ONN-NNN function of hyperparameter values of our loss function.
First layer
Second layer
Third layer
Last layer
CIFARIO
3.8
22 norm
6
3.6
5
4
3.4
3
3.2
2
10
20
30
1
10
20
30
1
10
20
30
10
20
30
15
20
12
FMNIST
12 norm
15
10
10
5
5
30 40
20
30
10
20
10
30
40
12
MNIST
l2 norm
6
10
10
5
10 20 30 40
10 20 30 40
1 10 20 30 40
1 10 20 30 40
Training epochs
Training epochs
Training epochs
Training epochs
Figure 11: le norm of model weights in CE T---T-OA CNN FashionMNIST and MNIST.
and O CNCN
using CIFARIO,
First layer
Second layer
Third layer
Last layer
CIFARIO
12 norm
100
100
60
80
60
40
5
60
40
20
30
20
30
30
10
20
60
20
FMNIST
l2 nom
60
40
40
15
10
10
20
20
5
10
30 40
10
30 40
30
40
20 30
la norm
40
40
15
15
MNIST
30
10
10
20
20
5
10
30
40
10
30
40
10
30
40
20 30
Training epochs
Training epochs
Training epochs
Training epochs
Figure 12: Pre-activation l2 norm per layer of CE-T-CNN FashionMNIST and MNIST.
and O-T-CNN on CIFAR10,
14

---

Under review as a conference paper at ICLR 2022
a) Initial epoch 2 in CIFARIO
b) Late epoch 20 in CIFAR10
c) Initial epoch 2 in FaShionMNIST
d) Late epoch 20 in FashionMNIST
c) Initial epoch 2 in MNIST
f) Late epoch 20 in MNIST
Figure 13: Confusion matrices that show per-class training accuracy of O-T-CNN (first column), CE-T-CNN (second columns) and non-private SGD approach (last columns) using CIFAR10, Fash- ionMNIST and MNIST.
15

---

Under review as a conference paper at ICLR 2022
Dranspedillpee (LARE)
Figure 14: Histogram of l2 norm of per-example gradient in CE-T-CNN and O-T-CNN on CIFAR1O
(top row), FashionMNIST (middle row) and MNIST (bottom row).
CIFARIO
FashionMNIST
MNIST
95 1
80
50
70
www M
90
Test accuracy
40
60
85
80
50
30
40
75
70
30
20
65
20
60
10
10
1.3
2.5
1.5
2
Figure 15:
DP-SGD training of an end-to-end classifier using SSE
SAF
and Huber loss
(5
3
3
.4
.5
3
8
.9
and B 1
16


Reviews:
[{"review_id": "luWkwr9PtVK", "reviewer": "Reviewer_VgSd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "There are several weak points of this paper.\n\nThe paper uses the theoretical evidence to argue that summed squared error is good for fast convergence. We have to admit that the convergence result in Allen-Zhu et al. 2019 has strict conditions: NTK regime, ReLU and square loss, which may not hold in the practical experiments. Even if these results hold, the fastness of exponential convergence is more evident for the final stage of convergence. However, here it is used to argue for the initial stage's fastness, which is not persuasive. In practice, the convergence rate is more related with the choices of the optimizers SGD or Adam, different learning rates. rather than different losses. The paper does not have good comparison with these choices.\n\nThe regularization penalty is quite similar to the function of the weight decay. How does it compare with normal weight decay? It is supposed that DP-SGD naïve application with weight decay can achieve better results than 59% on CIFAR10.\n\nMost importantly, CIFAR10 may not be a good dataset for benchmarking DP algorithms. For one obvious reason, it is not a sensitive dataset from every aspects. It is a too small dataset, with each class 5000 samples so that the DP algorithm may not perform well as expected unless using pretrained model. Benchmarking DP algorithms on CIFAR10 may lead to over-optimized models and/or losses that are not be able to generalize to other real privacy-sensitive scenarios, i.e. language models.\n", "summary_of_the_paper": "The paper proposes a tailored loss for DP-SGD, which includes summed squared error, the focal loss, and a regularization penalty. The summed squared error is for fast convergence at initial stage. The focal loss is used for identify hard samples. The regularization penalty is used for reducing the gradient/weight norm and avoid explosion. For each component of the tailored loss, the paper has empirical/theoretical evidence to argue the necessity. It is a good try to improve the performance of DP-SGD from amending the loss.", "summary_of_the_review": "Good intuitions about DP algorithm but the results and arguments are not convincing. Overall, the review would like to give a weak reject to this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bMA1SdtZXW", "reviewer": "Reviewer_WSQr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: \n\n1.\tImproving the utility of private deep learning from a loss design perspective is novel to me. This may be also valuable to the community. \n2.\tThe model obtained by optimizing the proposed loss achieves the SOTA result.\n\nWeakness: \n\nThe main disadvantage of this article is that it does not explain clearly the motivation for designing the loss and why the proposed loss can work. Although the author tried to analyze the mechanism of loss in section 3.2, these analyses were not convincing enough and did not clearly explain why the loss can work. Specifically,\n1. The author indicated \"Our loss limits information loss from clipping\" through Fig. 1, but did not explain why the proposed loss can prevent logit values and weights from exploding.\n2. The author claimed \"Our loss yields faster convergence\", but did not provide proof.\n3. The author mentioned \"Preventing logits from exploding may also help recover the generalization boost of batch normalization\", which can only be used as a guess based on observations. A rigorous theoretical analysis may be required to strengthen the claim.\n4. It seems that Figure 2 doesn't serve as an indication that the proposed loss function works because it enhances smoothness, because according to Fig 2, the smoothness of the network trained by CE loss is getting better and better, while the smoothness of the proposed loss training model is It's getting worse.\n\nAnother question: Are the parameters robust to different hyper-parameters settings in the loss?\n", "summary_of_the_paper": "This paper attempts to improve the accuracy of the DP-SGD training from the perspective of loss function design. The authors propose a loss composed of SSE loss, focal loss, and L2 regularization penalty. Experiments are conducted to demonstrate the effectiveness of the \nproposed loss. \n", "summary_of_the_review": "None", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Jtrgr5gyP-", "reviewer": "Reviewer_jXWf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think this paper is clear and easy-to-follow. The underlying philosophy that maybe DP learning should not use the same loss function as regular non-DP learning is interesting. This empirical paper has analyzed a set of computer vision datasets and the improvement does exists. I appreciate the authors efforts to do ablation study and to give details of network architectures etc. in the appendix.\n\nHowever, my main concern is the message that this paper is trying to convey. First of all, the experiments are not comprehensive: only computer vision tasks are included and only on toy datasets. I believe including recommendation system and NLP datasets will make the statement of using their new loss much more convincing. Secondly, the empirical improvement is not significant. The seemly most significant improvement (as is the one presented in the abstract) is 4% on CIFAR10. Given that non-private model can easily get over 90% on CIFAR10, I believe the gap is not really closed by introducing the new loss. Lastly, the lack of theory may make people wonder, does this improvement really hold for general DP training? At most we can say on specific datasets and specific models, using the new loss is beneficial. Also all three components of the new loss already exist in regular training, so the current approach seems a simple combination without sufficient justification. ", "summary_of_the_paper": "This paper proposes a new loss that consists of three parts, namely MSE+focal loss+regularization loss. The new loss is supposed to benefit DP deep learning in comparison to the cross-entropy loss. Accuracy improvement has been observed on MNSIT, FashionMNIST and CIFAR10. ", "summary_of_the_review": "The paper is clear and easy-to-follow, combining three existing losses to form a new one that empirically improves DP accuracy on some vision datasets. However, the experiments should include non-vision tasks and provide (at least discuss) the insight from a theoretical viewpoint. Also the improvement is not significant enough for ICLR venue.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0XcZcKE5BRK", "reviewer": "Reviewer_UK9m", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall I think this is an interesting paper that explores the impact of loss function on the performance of models trained by DP-SGD. Also, the experiments are quite comprehensive uncovering the importance of choosing loss functions. Yet, there are still some aspects that could be improved.\n\n1. Though the loss can improve upon cross-entropy, it introduces three extra hyperparameters to tune. It seems the paper only reported the best performance after hyperparameter searches. However, the sensitivity of performance with respect to these hyperparameters is not reported or discussed. It would be better if such results can be added.\n\n2. The impact of loss function on model performance is an important and interesting topic. The loss function proposed by the paper consists of three components, all from existing literature. I wonder have the authors tried other components (e.g. using hinge loss of Huber loss to replace the sum of squared errors or using other penalties)? \n\n-----------------------after rebuttal--------------------------------------\n\nMy concerns are mostly addressed. As pointed out by other reviewers, some intuitive arguments for motivating the losses are not well-supported and misleading (e.g. using Allen-Zhu et al. 2019 to argue faster convergence in practice mentioned by Reviewer VgSd). However, I believe this paper should be viewed as an empirical paper and I am satisfied with the authors' efforts in exploring the impact of loss function on the performance of DP-SGD. Thus, I would like to keep my score.\n", "summary_of_the_paper": "The paper proposes a new loss function to improve the performance of neural network models trained by DP-SGD. The new loss function is a weighted average of the sum of squared error, the focal loss, and a penalty on the squared norm of the pre-activation output of different layers. The new loss achieves state-of-the-art accuracy on the CIFAR-10, FashionMNIST, and MNIST datasets. It is also shown that the new loss can reduce the bias of gradient clipping and encourage learning on hard examples.", "summary_of_the_review": "Overall I think the paper is interesting and could be further improved if more results are added.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers all seemed to agree that the investigation of other losses is an interesting direction of study, and acknowledged there was some empirical performance improvement for standard computer vision tasks. However, they felt the justification of the specific form of loss was a bit shaky and heuristic, and were furthermore unconvinced by results exclusively for image classification (one reviewer was unmoved by the magnitude of improvement). This was a borderline decision, but we hope the authors refine and resubmit their work as this is an interesting but underexplored direction within DPML. 

As one recent related work which investigates the effect of other architecture differences in the DP setting, the authors may be interested in https://arxiv.org/abs/2110.08557.
Decision: Reject