Title: MAKING THE SHOE FIT: ARCHITECTURES, INITIALI-ZATIONS, AND TUNING FOR LEARNING WITH PRIVACY

Abstract: Because learning sometimes involves sensitive data, standard machine-learning algorithms have been extended to offer strong privacy guarantees for training data. However, in practice, this has been mostly an afterthought, with privacypreserving models obtained by re-running training with a different optimizer, but using the same model architecture that performed well in a non-privacy-preserving setting. This approach leads to less than ideal privacy/utility tradeoffs, as we show here. Instead, we propose that model architectures and initializations are chosen and hyperparameter tuning is performed, ab initio, explicitly for privacypreserving training. Using this paradigm, we achieve new state-of-the-art accuracy on MNIST, FashionMNIST, and CIFAR10 without any modification of the fundamental learning procedures or differential-privacy analysis.

1. INTRODUCTION:

Machine learning (ML) can be usefully applied to the analysis of sensitive data, e.g., in the domain of healthcare (Kononenko, 2001) . However, ML models may unintentionally reveal sensitive aspects of their training data, e.g., due to overfitting (Shokri et al., 2017; Song & Shmatikov, 2019) . To counter this, ML techniques that offer strong privacy guarantees have been developed. Notably, the differentially private stochastic gradient descent, or DP-SGD, of Abadi et al. (2016) is an easyto-use, generally-applicable modification of stochastic gradient descent. In addition to its rigorous privacy guarantees, it has been empirically shown to stop the leaking of secrets (Carlini et al., 2019) .

To strictly bound the impact of any training example, DP-SGD makes two changes to every gradient step: first, each example's gradient contribution is limited to a fixed bound (in practice, by clipping all per-example gradients to a maximum 2 norm); second, random (Gaussian) noise of the scale of the clipping norm is added to each batch's combined gradient, before it is backpropagated to update model parameters. Together, these changes create a new, artificial noise floor at each step of gradient descent, such that the unique signal of any individual example is below this new noise floor; this allows differential privacy to be guaranteed for all training examples (Dwork & Roth, 2014) .

Training using DP-SGD is eminently practical and in addition to privacy offers advantages such as strong generalization and the promise of reusable holdouts (Google, 2019; Dwork et al., 2015) . Unfortunately, its advantages have not been without cost: empirically, the test accuracy of differentially private ML is consistently lower than that of non-private learning (e.g., see Papernot et al. (2018) ). Such accuracy loss may sometimes be inevitable: for example, the task may involve heavy-tailed distributions and adding noise will definitely hinder visibility of examples in the tails (Feldman, 2019; Bagdasaryan & Shmatikov, 2019) . However, this does not explain the accuracy loss of differentially private learning on standard benchmark tasks that are known to be relatively simple: MNIST (Yann et al., 1998) , FashionMNIST (Xiao et al., 2017) , CIFAR10 (Krizhevsky et al., 2009) , etc. This paper presents several new results for privacy-preserving learning that improve the state-ofthe-art in terms of both privacy and accuracy. Significantly, these new results stem from a single, simple observation: differentially-private learning with DP-SGD is different enough that all aspects of learning-model architecture, parameter initialization, and optimization strategy, as well as hyperparameter tuning-must be reconsidered. To achieve the best privacy/accuracy tradeoffs, we must tune our learning strategies to the specifics of privacy-preserving learning; i.e., we must "learn to learn" with privacy. Conversely, we concretely demonstrate how the architecture, initialization, and optimization strategy that gives the best accuracy for non-private learning can be a poor fit for learning with privacy. Instead, by revisiting our choices, we can reduce the information loss induced by clipping, limit the impact of added noise, and improve the utility of each gradient step when learning with privacy. Our contributions facilitate DP-SGD learning as follows:

• We show how simple architecture changes, such as the use of tanh instead of ReLU activations, can improve a model's private-learning suitability and achievable privacy/accuracy tradeoffs, by eliminating the negative effects of clipping and noising large gradients. • We explain how high-capacity models can be disadvantageous, as well as the advantages of models with a final, fully-connected layer that can be independently fine tuned, and how both help address the curse of dimensionality and high-dimensional noise.

• We demonstrate the importance of finding good initializations, and show how this can be done with privacy using either transfer learning or weight scaling (Raghu et al., 2019) . • We show that better tradeoffs and increased wall-clock learning speeds can be achieved by tuning hyperparameters and choosing optimizers directly for DP-SGD learning.

By applying the above, we advance the state of the art for MNIST, FashionMNIST, and CIFAR10, significantly improving upon the privacy/accuracy tradoffs from prior work. On MNIST, we achieve 98.1% test accuracy for a privacy guarantee of (ε, δ) = (2.93, 10 -5 ), whereas the previous stateof-the-art reported in the TensorFlow Privacy library (Google, 2019) was 96.6%. On CIFAR10, we achieve 72% test accuracy at (ε, δ) = (2.1, 10 -5 ) in a setup for which to the best of our knowledge the previous state-of-the-art was achieved by Abadi et al. (2016) at 67% accuracy.

2 TRAINING-DATA MEMORIZATION, DIFFERENTIAL PRIVACY, AND DP-SGD Machine-learning models will easily memorize whatever sensitive, personal, or private data that was used in their training, and models may in practice disclose this data-as demonstrated by the attacks of Shokri et al. (2017) , Song & Shmatikov (2019), and Carlini et al. (2019) .

For reasoning about the privacy guarantees of algorithms such as training by stochastic gradient descent, differential privacy has become the established gold standard (Dwork & Roth, 2014) . Informally, an algorithm can be differentially private if it will always produce effectively the same output (in a mathematically precise sense), when applied to two input datasets that differ by only one record. Formally, a learning algorithm A that trains models from the set S is (ε, δ)-differentiallyprivate, if the following holds for all training datasets d and d that differ by exactly one record:

P r[A(d) ∈ S] ≤ e ε P r[A(d ) ∈ S] + δ (1)

Here, ε gives the formal privacy guarantee, by placing a strong upper bound on any privacy loss, even in the worst possible case. A lower ε indicates a stronger privacy guarantee or a tighter upper bound (Erlingsson et al., 2019) . The factor δ allows for some probability that the property may not hold (in practice, this δ is required to be very small, e.g., in inverse proportion to the dataset size).

A very attractive property of differential-privacy guarantees is that they hold true for all attackerswhatever they are probing and whatever their prior knowledge-and that they remain true under various forms of composition. In particular, the output of a differentially-private algorithm can be arbitrarily post processed, without any weakening of the guarantees. Also, if sensitive training data contains multiple examples from the same person (or, more generally, the same sensitive group), ε-differentially-private training on this data will result in model with a kε-differential-privacy guarantee for each person, as long as at most k training-data records are present per person. Abadi et al. (2016) introduced DP-SGD as a method for training deep neural networks with differential-privacy guarantees that was able to achieve better privacy and utility than previous efforts (Chaudhuri et al., 2011; Song et al., 2013; Bassily et al., 2014) . DP-SGD bounds the sensitivity of the learning process to each individual training example by computing per-example gradients {g i } i∈0..n-1 with respect to the loss, for the n model parameters {θ i } i∈0..n-1 , and clipping each per-example gradient to a maximum fixed 2 norm C. Subsequently, to the average of these perexample gradients, DP-SGD adds (Gaussian) noise that whose standard deviation σ is proportional to this sensitivity. In this work, we use the canonical implementation of DP-SGD and its associated analysis that has been made available through the TensorFlow Privacy library (Google, 2019). Figure 1 : Test accuracy as a function of the number of filters k in the convolutional architecture of Table 1 ; when training with vanilla SGD and DPSGD. Each point corresponds to multiple training runs on MNIST (left) or FashionMNIST (right). For both datasets, adding filters always improves non-private learning, whereas after an early point they are not beneficial to learning with privacy.

3. MODEL ARCHITECTURES BETTER SUITED TO LEARNING WITH PRIVACY:

We show here that learning with differential privacy imposes additional constraints that need to be taken into account when designing neural network architectures. They help us control the sensitivity of learning to training examples before the clipping operation is performed in DP-SGD, thus reducing the potential negative impact of clipping on the estimated gradient direction.

3.1. MODEL CAPACITY:

The success of neural networks is in part explained by their ability to scale to complex tasks through an increase in model capacity. ResNets are an illustrative recent examples (He et al., 2016) . Here, we explain how additional capacity may not be beneficial when learning with privacy. One of the major challenges in training models with differential privacy is the curse of dimensionality (Bassily et al., 2014) . The accuracy of privately trained models typically degrades with the increase in the number of dimensions. Unfortunately, strong lower bounds suggest that this dependence on dimensionality is necessary (Bassily et al., 2014) . Consider the convolutional architecture described to the right. With all other architectural details being fixed, we can control the model's capacity by varying the number of filters k in its two convolutional layers. While the relationship between generalization performance and the number of parameters is not always monotonic (Neyshabur et al., 2017) , we leave as future work a study of how different measures of capacity can inform the design of model architectures for private learning. We report the model's accuracy when trained with SGD and DP-SGD in Figure 1 , both on MNIST (left) and FashionMNIST (right). The test accuracy of models trained without privacy monotonically increases with the number of filters in their convolutional layers. Instead, we observe an inflection point at about 15 filters for which models trained with privacy achieve their highest test accuracy. Afterwards, the model's generalization suffers as more filters are added.

There are two competing explanations of this behavior, both compatible with the lower bound stated in Bassily et al. (2014) . First, recall that DP-SGD performs a clipping operation on each per-example gradient before the average gradients is used to update model parameters; i.e., each gradient is subject to the following transformation

g i ← g i • min   1, C n-1 i=0 g 2 i   (2)

where g i is the gradient corresponding to model parameter i. For a fixed clipping norm C (corresponding to a certain, fixed privacy guarantee), the quantity parameters we have, the more likely DP-SGD is to clip the gradient (or signal) at each parameter. This can explain the presence of an inflection point in Figure 1 , after which learning with privacy becomes increasingly difficult as capacity is increased. Second, as the number of parameters (i.e., g i 's) increases, the norm of the noise vector that DP-SGD must add to the gradient average to ensure privacy also increases. This noise norm increases as #parameters, and introduces another source of accuracy degradation with an increased number of parameters.

Our observations may seem to contradict some of the findings in Abadi et al. (2016) . However, their limited experimental setup could offer few general lessons. First, they reduced data dimensionality using PCA to have inputs of only 60 dimensions; second, they explored only a model architectures using a single layer perceptron with between 200 and 2, 000 units. Instead, our experiments involve a realistic setting where the full input is passed to a convolutional neural network with a total of 3 hidden layers and over 26,000 parameters.

3.2. ACTIVATION FUNCTIONS:

When training a model with differential privacy, gradients computed during SGD are clipped (recall Equation 2) to control the sensitivity of learning to training examples. If these gradients take large values, some of the signal will be discarded as gradients are being clipped. One way to reduce the magnitude (or at least control it), is to prevent the model's activations from exploding. However, a common choice of activation function in modern deep neural networks is the ReLU and, unlike other activations functions, ReLUs are unbounded.

Here, we thus test the hypothesis that replacing ReLUs with a bounded activation function prevents activations from exploding and thus keeps the magnitude of gradients to a more reasonable value. This in turn implies that the clipping operation applied by DP-SGD will discard less signal from gradient updates-eventually resulting in higher performance at test time.

On MNIST and FashionMNIST, we train two models based off the architecture of Table 1 : the first model uses ReLU whereas the second model uses tanh1 as the activation for its hidden layers, with other architectural elements kept identical. In our experiments, we later fine-tuned those architectural aspects (i.e., model capacity, choice of optimizer, etc.) separately for each activation function, to avoid favoring any one choice. In all cases, tanh was an improvement, as summarized in our conclusions (Section 6).

Figure 2 visualizes the privacy-utility Pareto curve (Avent et al., 2019) of the two models trained with DP-SGD. Rather than plotting the test accuracy as a function of the number of steps, we plot it as a function of the privacy loss ε (but the privacy loss is a monotonically increasing function of the number of steps). On MNIST, the test accuracy of the tanh model is 98.0% compared to 96.6% for the ReLU model with an identical privacy loss of ε = 2.93. For comparison, baseline tanh and ReLU models trained without privacy both achieve a test accuracy of 99.0%. Similarly, on FashionMNIST, the tanh model trained with DP-SGD achieves 85.5% test accuracy compared to 81.9% with ReLUs. The baselines on FashionMNIST are 89.3% for tanh and 89.4% with ReLUs.

To explain why a simple change of activation functions has a large impact on the model's accuracy, we conjecture that the bounded nature of the tanh prevents activations from exploding during training. We thus monitored the 2 norm of the first layer's activations for our MNIST model while it is being trained in three scenarios: (a) without privacy using vanilla SGD and ReLU activations, (b) with ReLU activations and DP-SGD, and (c) with tanh activations and DP-SGD. The evolution of activation norms on test data is visualized in Figure 3 . As conjectured, the activations of our ReLU network explode by a factor of 3 when training with privacy when compared to without privacy. Switching to tanh activations brings down the norms of activations back to levels comparable with the activations of our non-private ReLU network.

4. INITIALIZATIONS FOR LEARNING WITH DIFFERENTIAL PRIVACY:

Because each gradient step expends some privacy budget, good initialization of learning is important; here, we consider transfer learning (Pratt et al., 1991) and weight scaling (Raghu et al., 2019) .

4.1. INITIALIZING FROM A PRE-TRAINED MODEL USING TRANSFER LEARNING:

Transfer learning can improve the initialization used when learning with privacy, and allow better privacy/accuracy tradoffs to be achieved. 2 For example, to reach reasonable accuracy (> 80%) on CIFAR10, a convolutional neural network may necessarily include many convolutional layers comprising several hundred-thousand parameters. However, since convolutional layers for similar image-processing tasks are known to learn similar representations-at least in early layers-it may be possible to transfer most of these parameters from a public model, either as initializations or as frozen parameters, and subsequently train with DP-SGD. For CIFAR10, the natural choice for such transfer is a CIFAR100 model, and this has been previously explored by Abadi et al. (2016) . -100 ) . From each of these trained models, transfer learning was used to initialize a model to be trained on CIFAR10. In the subsequent CIFAR10 training, all but the last layer was frozen, which simplifies the learning task to that of logistic regression (but also reduces utility, with the best non-private accuracy reduced to 75% on CIFAR10).

Table 3 shows CIFAR10 privacy and accuracy resulting from fine-tuning of different transferlearning models with DP-SGD. As shown in Table 4 , the results improve on those of Abadi et al. (2016) , even though they performed non-linear fine-tuning of two neural-network layers, and their underlying model was able to achieve higher non-private accuracy (86%).

Type Epoch 10:

Epoch 50 Epoch 100 Epoch 200 Epoch 400 Min-rand-10 44.8% ± 4.6 49.6% ± 3.9 51.0% ± 3.9 52.8% ± 3.3 53.7% ± 3.5 (81.0% ± 4.0) 50% = best 54.1% = best 55.7% = best 56.9% = best 57.6% = best Half-rand-50 39.4% ± 2.9 51.4% ± 0.8 54.7% ± 1.5 56.8% ± 1.3 59.0% ± 0.9 (62.1% ± 1.4) 44.3% = best 52.6% = best 56.6% = best 58.3% = best 60.2% = best Max-100 57.0% ± 1.0 66.2% ± 0.6 68.4% ± 0.6 69.7% ± 0.6 71.0% ± 0.5 (54.9% ± 0.7) 59.1% = best 67.2% = best 69.5% = best 70.6% = best 72.1% = best Table 3 : Accuracy of learning with privacy (average/best of 10 runs) compared to a non-private baseline of 75%. A CIFAR10 model is trained from a CIFAR100-transfer-learning initialization, with all-but-the-last layer frozen during training. The DP-SGD ε upper bounds at δ = 10 -5 are ε 10 = 0.32, ε 50 = 0.73, ε 100 = 1.04, ε 200 = 1.48, ε 400 = 2.12 for the subscript-indicated epochs. The source model CIFAR100 accuracy (first column), is uncorellated to the CIFAR10 accuracy. In addition, the results show the benefits of model architectures whose final layer can be fine-tuned using logistic regression training, or other forms of convex optimization. Such training can be made possible by including a final fully-connected layer into a network; in additional experiments (not detailed here), the inclusion of such a layer did not harm the training of the original, source model from which transfer learning was done. Furthermore, the number of parameters in this layer did not seem to matter much: privacy/accuracy tradeoffs remained the same, even when the layer was grown by an order of magnitude, which is consistent with what is known about differentially-private convex optimization (Jain & Thakurta, 2014) . Initialization by transfer learning is only applicable when a suitable public model exists whose weights can facilitate learning with privacy on sensitive data. But, such model may not exist, and DP-SGD learning may possibly benefit from other, non-standard means of initialization. We consider the Mean Var weightscaling approach of Raghu et al. (2019) and initialize DP-SGD learning with Gaussian random parameter distributions whose layer-wise mean and variance are extracted from a seed model trained on the same sensitive input data. The weight-scaling approach does not directly transfer the parameters of an existing model; instead, just the layer-wise mean and variance are extracted, and those statistics are used to configure the Gaussian random distributions from which a second model with the same architecture is initialized.

4.2. INITIALIZATION BY WEIGHT SCALING:

In faster to an accuracy that is within 1 point of its final accuracy, however DP-SGD increases more steadily towards the end of training, thus both achieve comparable results. Given one of the datasets, the privacy budget ε for both models is identical at each epoch.

privacy is very low.) The idea is that the mean and variance pairs can be obtained quickly at a modest privacy budget, but the faster convergence of the Mean Var initialized model both reduces the overall privacy budget needed for training, and mitigates the increased wall-clock time of DP-SGD.

We experiment with a relatively deep CIFAR10 convolutional model (see Appendix A), since Raghu et al. found the benefits of Mean Var initialization most pronounced for large models. We first trained a model using random initialization, and then did weight scaling by transferring that model's statistics to a new model. In this proof-of-concept, both models were trained with the same noise variance (σ = 0.5), but one could reserve a larger portion of the privacy budget for the new model.

We should note that we did not directly transfer the weight statistics between corresponding layers in the original and new models. Rather, we used the weight statistics of each of original model's early layers of the original model for two of the layers in the new model. This gives superior performance to a one-to-one transfer; we conjecture that this is because early layers have higher variance.

Figure 4 shows the results of this experiment for some early training epochs. Each run that used standard He random initialization (He et al., 2015) gave near identical results, achieving 37% accuracy at epoch 33. The Mean Var initialization runs showed substantially higher variance, with the best models having 7% better accuracy at epoch 33. These results are intriguing, and reminiscent of the lottery ticket hypothesis (Frankle & Carbin, 2019) ; they suggest a strategy of training a collection of Mean Var models and keeping those that show early promise.

5. TUNING OPTIMIZERS FOR PRIVATE LEARNING:

Architectural choices presented in Section 3 control how sensitive learning is to training examples. This helps us to learn with privacy-because it eliminates the negative effects of clipping and noising large gradients. We now turn our attention to the training algorithm itself. We find that it is important to tailor algorithm and hyperparameter choices to the specificities of private learning: a batch size or learning rate that yields good results without privacy may not perform well with privacy.

5.1. ADAPTIVE OPTIMIZERS PROVIDE MARGINAL GAINS WHEN LEARNING WITH PRIVACY:

We first explore the choice of optimizer, and in particular whether adaptive optimizers that leverage the history of iterates help convergence when learning privately. We compare learning curves for DP-SGD and the differentially private counterpart of Adam (Kingma & Ba, 2014), a canonical adaptive optimizer. A qualitative analysis of Figure 5 leads to the same conclusion for all datasets (MNIST, FashionMNIST, and CIFAR10). While DP-Adam may converge faster initially, its convergence rate eventually slows down sufficiently for DP-SGD to achieve comparable (if not higher) accuracy.

To explain the ineffectiveness of adaptive optimizers, we hypothesize that the iterates they accumulate during training are affected negatively by noise introduced to preserve privacy. Indeed, while there is enough signal from the training data included in any given batch sampled early in training, later in training most training examples have a loss of zero and do not contribute to the gradients being noised. Carrying this noise from one gradient descent step to the next to adapt learning rates therefore inadequately slows down training. To verify this, we track the estimate of the first moment in Adam on MNIST. The mean absolute value of its components converges when learning without privacy (from 0.5 after the first epoch to about 0.8 for epochs 45 through 60). Instead, it increases steadily throughout training with privacy (from 0.5 at the first epoch to above 1. after 60 epochs). Thus, choosing an adaptive optimizer (e.g., DP-Adam) is not necessary if one is interested in achieving maximal accuracy: given a fixed privacy budget, fine-tuning the learning rate is more important as we confirm in Section 5.2. Note that this resonates well with recent results questioning the generalization capabilities of adaptive optimizers (Wilson et al., 2017) .

5.2. CHOOSING A (LARGE) BATCH SIZE AND LEARNING RATE:

Having observed that few training examples contribute signal after the first epochs, it is natural to ask whether increasing the size of batches could improve the noise-to-signal ratio in DP learning.

To ensure a fair comparison, we fix the privacy budget ε and deduce the number of epochs we can train the model for given a desired batch size. For instance, in Table 5 , we compare models trained for 7 epochs on batches of 1, 024 examples to models trained for 40 epochs on batches of 256 examples. In both cases, the total privacy budget for training these models is ε = 2.7. We run a hyperparameter search to fine-tune the choice of learning rate for both DP-SGD and DP-Adam. We then compare the test accuracy achieved with small and large batch sizes. 

85.1%:

Table 5 : Impact of batch size on trade-off between accuracy and privacy. The privacy budget is fixed to ε = 2.7 for all rows. A hyperparameter search is then conducted to find the best learning rate to train the model with or without differential privacy on FashionMNIST.

Hyperparameters should be tuned for DP-SGD, not SGD. We confirm that DP-Adam does not improve over DP-SGD. Yet, this experiment shows how training for a small number of epochs at a large batch size can do comparably well to a large number of epochs at a small batch size: the wall-clock time gain is important (about 5×) and the cost in performance is moderate-half a percentage point. This suggests that earlier theoretical analysis (Talwar et al., 2014) also holds in the non-convex setting. Furthermore, note how learning rates vary across the non-DP and DP settings.

6. CONCLUSIONS:

Rather than first train a non-private model and later attempt to make it private, we bypass nonprivate training altogether and directly incorporate specificities of privacy-preserving learning in the selection of architectures, initializations, and tuning strategies. This improves substantially upon the state-of-the-art privacy/accuracy trade-offs on three benchmarks, as summarized below. Up to now, we evaluated each component (e.g., change of activation function, optimizer, etc.) individually to demonstrate its influence on private learning. Instead, here this summary table compares each approach after all hyperparameters explored in the paper have been jointly fined tuned. In particular, note how even in their own individually-best setting, tanh continues to consistently outperform 

:

We obtained results similar to the tanh with a sigmoid and a learning rate increased by a factor of

to 8. This is explained by the fact that the tanh is a rescaled sigmoid φ: tanh(x) = 2φ(x) -1.

A different, formal take on how public models and data can facilitate learning with privacy is studied in(Bassily et al., 2018;Feldman et al., 2018).

Figures and Tables:

Figure 2: Test accuracy as a function of the privacy loss when training a pair of models with DP-SGD. The only difference between the two models is the activation function for their hidden layer: ReLU or tanh. All other elements of the architecture (number, type, and dimension of layers) and the training algorithm (optimizer, learning rate, number of microbatches, clipping norm, and noise multiplier) are identical. Results are averaged over 10 runs for each curve.

Figure 2 norm of the first conv activations.

Figure 4: Colored lines show DP-SGD accuracy for three "seed" random initializations of a CIFAR-10 model. Colored bands show accuracy range of 30 DP-SGD models using Mean Var initialization based on per-layer parameter statistics in the corresponding seed model. In all models, the privacy ε at each epoch is identical; however, Mean Var initialization substantially improves the privacy/accuracy tradeoff.

Figure 5: Learning curves for DP-SGD and DP-Adam. Early on in training, DP-Adam convergesfaster to an accuracy that is within 1 point of its final accuracy, however DP-SGD increases more steadily towards the end of training, thus both achieve comparable results. Given one of the datasets, the privacy budget ε for both models is identical at each epoch.

MNIST and FashionMNIST model architecture (33,000 parameters for k = 31).

Table content: <table><tr><td>Layer</td><td>Parameters</td></tr><tr><td>Convolution</td><td>k filters of 8x8, strides 2</td></tr><tr><td>Max-Pooling</td><td>2x2</td></tr><tr><td>Convolution</td><td>k filters of 4x4, strides 2</td></tr><tr><td>Max-Pooling</td><td>2x2</td></tr><tr><td>Fully connected</td><td>32 units</td></tr><tr><td>Softmax</td><td>10 units</td></tr></table>



Table content: <table><tr><td>Taking the Abadi et al. (2016) transfer learn-ing results for CIFAR10 as a baseline, we per-form new experiments using much of the same setup and the model architecture of Table 2. As it is relatively simple, this model is a good candidate for differentially-private learning (al-though it reaches only 84.2% accuracy on CI-FAR10 when all its parameters are trained non-privately, whereas state-of-the-art models can have over 10% higher accuracy).</td><td>: CIFAR10 convolutional model architec-ture (in total, 2,395,434 parameters). Conv × 2 32 filters of 3x3, strides 1 Max-Pooling 2x2 Conv × 2 64 filters of 3x3, strides 1 Max-Pooling 2x2 Conv × 2 128 filters of 3x3, strides 1 Fully connected 1024 units Softmax 10 units</td></tr><tr><td colspan="2">We performed new transfer-learning experiments based on training this model on CIFAR100 data in</td></tr><tr><td colspan="2">three different ways: trained on a total of 5000 examples from 10 classes picked at random (Min-</td></tr><tr><td colspan="2">rand-10 ); trained on 25,000 examples from a random half of the CIFAR100 classes, grouped into</td></tr><tr><td colspan="2">10 new, evenly-sized meta classes (Half-rand-50 ); trained on all examples and all 100 separate</td></tr><tr><td>classes (Max</td><td/></tr></table>



Table content: <table><tr><td colspan="2">: CIFAR10 privacy</td></tr><tr><td colspan="2">and accuracy tradeoffs.</td></tr><tr><td colspan="2">This paper Abadi et al.</td></tr><tr><td>(ε, acc.)</td><td>(ε, acc.)</td></tr><tr><td>(0.3, 59%)</td><td>-</td></tr><tr><td>(1.0, 70%)</td><td>-</td></tr><tr><td colspan="2">(2.1, 72%) (2.0, 67%)</td></tr><tr><td>-</td><td>(4.0, 70%)</td></tr><tr><td>-</td><td>(8.0, 73%)</td></tr></table>

Non-privateDifferentially-private Optimizer Batch Epochs Time Learning Rate Test Acc. Learning Rate Test Acc.

Table content: <table><tr><td>SGD</td><td>256 1024</td><td>40 7</td><td>240s 42s</td><td>1.07 • 10 -1 3.68 • 10 -1</td><td>90.3% 86.3%</td><td>3.32 • 10 -1 4.46</td><td>86.1% 85.1%</td></tr><tr><td>Adam</td><td>256 1024</td><td>40 7</td><td>240s 42s</td><td>1.06 • 10 -3 4.32 • 10 -3</td><td>90.5% 88.7%</td><td>1.32 • 10 -3 7.08 • 10 -3</td><td>86.0%</td></tr></table>

ReLU with for example 98.1% test accuracy (instead of 96.6% for ReLU) on MNIST.

Table content: <table><tr><td>Dataset</td><td>Technique</td><td>Acc.</td><td>ε</td><td>δ</td><td>Assumptions</td></tr><tr><td/><td>SGD w/ tanh (not private)</td><td>99.0%</td><td>∞</td><td>0</td><td>-</td></tr><tr><td>MNIST</td><td>DP-SGD w/ ReLU</td><td colspan="3">96.6% 2.93 10 -5</td><td>-</td></tr><tr><td/><td>DP-SGD w/ tanh (ours)</td><td colspan="3">98.1% 2.93 10 -5</td><td>-</td></tr><tr><td/><td>SGD w/ ReLU (not private)</td><td>89.4%</td><td>∞</td><td>0</td><td>-</td></tr><tr><td>FashionMNIST</td><td>DP-SGD w/ ReLU</td><td>81.9%</td><td colspan="2">2.7 10 -5</td><td>-</td></tr><tr><td/><td>DP-SGD w/ tanh (ours)</td><td colspan="3">86.1% 2.7 10 -5</td><td>-</td></tr><tr><td/><td>Transfer + SGD (not private)</td><td>75%</td><td>∞</td><td>0</td><td>-</td></tr><tr><td>CIFAR10</td><td>Transfer + DP-SGD (Abadi et al.)</td><td>67%</td><td>2</td><td>10 -5</td><td>Public Data</td></tr><tr><td/><td>Transfer + DP-SGD (ours)</td><td>72%</td><td colspan="2">2.1 10 -5</td><td>Public Data</td></tr></table>

Peer review (iclr2020):
This paper presents experimental evidence that learning with privacy requires approaches that are not identical to those used when learning without privacy. These approaches include re-considering different model choices (i.e., its structure and activation functions), its initialization, and its optimization procedure. With these changes, they show that it is possible to obtain state-of-the-art results for some canonical learning tasks. Strengths: This paper questions nearly every component in the training pipeline, including choices about the model structure, initialization strategies, and optimization procedures. For each component, they show that judiciously choosing the components (which go against the standard choices in the non-private learning setting) enables training higher-utility models than in previous works without sacrificing privacy. Moreover, in addition to the experimental evidence alone, most of the components considered in the paper were accompanied by reasonable justification/hypotheses for why the choices enable such improvements. This paper helps push differentially private learning to a more practically-useful realm. First, the suggested changes here are easy for a practitioner to understand and easy to implement. With only these simple changes, the concrete results then show that it is possible to achieve utility close to the analogous non-private model while still maintaining reasonable utility (epsilon less than 3 with delta of 10^-5). Weaknesses: My major concern with this paper lies in the experimental methodology. Specifically, most experiments are based on varying a single component while leaving all other components the same. While this is certainly the scientifically-valid way to demonstrate the component’s influence on the entire system given the other fixed components, it doesn’t convincingly demonstrate that the component has this influence across all (or at least most) reasonable configurations of the other components. This can be made concrete using many experiments in the paper, but let’s take the activation functions experiment of 3.2 as an example. Here, it is shown that after fixing the privacy guarantee, model structure, training procedure, and hyperparameters -- the tanh activation performs better than the ReLU activation. However, suppose instead that we fix all of these components except the hyperparameters; it may then be the case that the ReLU activation is capable of outperforming the tanh activation when its hyperparameters are chosen carefully. In other words, to validly compare the two activations and reach a convincing conclusion, they should be compared against each other in their own individually-best settings (e.g., the results induced by the optimal hyperparameters for ReLU versus the results induced by the optimal hyperparameters for tanh). This is similar to the problem addressed in Avent et al.’s “Automatic Discovery of Privacy–Utility Pareto Fronts” paper (https://arxiv.org/abs/1905.10862). The specific technical details on some experiments were either difficult to find or were lacking. Given that this is fundamentally an experimental paper, having these details clearly listed somewhere for reference is important, even if relegated to an Appendix. Although this applies more broadly to most of the experiments, we can use Section 3 as an example again: the details on the experiment in 3.1 were found in the caption of Figure 1, whereas I would have expected them either in the main body or clearly listed in their own table; the details on the experiment in 3.2 specify that everything is identical between the tests of the two activation functions, however it is never specified exactly what is being altered (and by how much) to vary the epsilon value. 4.2 Initialization by Weight Scaling proposes that judiciously scaling initial weights can improve model privacy/utility. This scaling is done by “transfer from one differentially private model to another”, where “DP-SGD can be applied to train a model with high utility, but less than ideal privacy” and then extracting the relevant information from there in order to initialize a new differentially private model that will be trained with strong privacy guarantees. It is claimed that “this extraction can be done in a differentially-private manner, e.g., as in Papernot et al. (2018), although the privacy risk of summary statistics that drive random initialization should be vanishing”. It is unclear to me how this extraction of summary statistics should be done in such a way that doesn’t consume a significant portion of the privacy budget. If there is such a way, it should be clearly stated and its effect on the privacy budget should be explicitly incorporated into this paper’s results. Minor: The statement that “Such accuracy loss may sometimes be inevitable” on page 1 should include a reference; e.g., Feldman’s “Does Learning Require Memorization? A Short Tale about a Long Tail” paper (https://arxiv.org/abs/1906.05271). Overall, this work provides good practical guidance to practitioners and researchers who wish to do differentially private machine learning. However, given the lack of theoretical novelty, the experimental methodology needs to be improved in order to significantly strengthen the results (assuming they continue to hold). ---------------------------------------------------- Update: Due to the authors* writing clarifications and experimental additions, in conjunction with the concrete and realistically-applicable insights from the paper, I*ve modified my rating to a Weak Accept.