Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper presents experimental evidence that learning with privacy requires approaches that are not identical to those used when learning without privacy. These approaches include re-considering different model choices (i.e., its structure and activation functions), its initialization, and its optimization procedure. With these changes, they show that it is possible to obtain state-of-the-art results for some canonical learning tasks.

Strengths:
This paper questions nearly every component in the training pipeline, including choices about the model structure, initialization strategies, and optimization procedures. For each component, they show that judiciously choosing the components (which go against the standard choices in the non-private learning setting) enables training higher-utility models than in previous works without sacrificing privacy. Moreover, in addition to the experimental evidence alone, most of the components considered in the paper were accompanied by reasonable justification/hypotheses for why the choices enable such improvements.
This paper helps push differentially private learning to a more practically-useful realm. First, the suggested changes here are easy for a practitioner to understand and easy to implement. With only these simple changes, the concrete results then show that it is possible to achieve utility close to the analogous non-private model while still maintaining reasonable utility (\epsilon less than 3 with \delta of 10^-5).


Weaknesses:
My major concern with this paper lies in the experimental methodology. Specifically, most experiments are based on varying a single component while leaving all other components the same. While this is certainly the scientifically-valid way to demonstrate the component’s influence on the entire system given the other fixed components, it doesn’t convincingly demonstrate that the component has this influence across all (or at least most) reasonable configurations of the other components.
This can be made concrete using many experiments in the paper, but let’s take the activation functions experiment of 3.2 as an example. Here, it is shown that after fixing the privacy guarantee, model structure, training procedure, and hyperparameters -- the tanh activation performs better than the ReLU activation. However, suppose instead that we fix all of these components except the hyperparameters; it may then be the case that the ReLU activation is capable of outperforming the tanh activation when its hyperparameters are chosen carefully. In other words, to validly compare the two activations and reach a convincing conclusion, they should be compared against each other in their own individually-best settings (e.g., the results induced by the optimal hyperparameters for ReLU versus the results induced by the optimal hyperparameters for tanh).
This is similar to the problem addressed in Avent et al.’s “Automatic Discovery of Privacy–Utility Pareto Fronts” paper (https://arxiv.org/abs/1905.10862). 
The specific technical details on some experiments were either difficult to find or were lacking. Given that this is fundamentally an experimental paper, having these details clearly listed somewhere for reference is important, even if relegated to an Appendix. Although this applies more broadly to most of the experiments, we can use Section 3 as an example again: the details on the experiment in 3.1 were found in the caption of Figure 1, whereas I would have expected them either in the main body or clearly listed in their own table; the details on the experiment in 3.2 specify that everything is identical between the tests of the two activation functions, however it is never specified exactly what is being altered (and by how much) to vary the \epsilon value.
4.2 Initialization by Weight Scaling proposes that judiciously scaling initial weights can improve model privacy/utility. This scaling is done by “transfer from one differentially private model to another”, where “DP-SGD can be applied to train a model with high utility, but less than ideal privacy” and then extracting the relevant information from there in order to initialize a new differentially private model that will be trained with strong privacy guarantees. It is claimed that “this extraction can be done in a differentially-private manner, e.g., as in Papernot et al. (2018), although the privacy
risk of summary statistics that drive random initialization should be vanishing”. It is unclear to me how this extraction of summary statistics should be done in such a way that doesn’t consume a significant portion of the privacy budget. If there is such a way, it should be clearly stated and its effect on the privacy budget should be explicitly incorporated into this paper’s results.
Minor: The statement that “Such accuracy loss may sometimes be inevitable” on page 1 should include a reference; e.g., Feldman’s “Does Learning Require Memorization? A Short Tale about a Long Tail” paper (https://arxiv.org/abs/1906.05271).


Overall, this work provides good practical guidance to practitioners and researchers who wish to do differentially private machine learning. However, given the lack of theoretical novelty, the experimental methodology needs to be improved in order to significantly strengthen the results (assuming they continue to hold).


----------------------------------------------------

Update: Due to the authors' writing clarifications and experimental additions, in conjunction with the concrete and realistically-applicable insights from the paper, I've modified my rating to a Weak Accept.
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: Overall, this work empirically evaluates different techniques used in privacy learning and suggest useful methods to stabilize or improve performance.

Detail comments:

Strength:
Despite the progress of privacy-preserving learning in theory, there are few works providing learning details for better training. Especially, considering the instability in perturbation-based private algorithms, e.g., most DP ones, the work could be valuable in the sense of practice.

Weakness:
As far as empirical research, the compared techniques are too few. What if we use those less popular techniques, for example, RMSprop optimization method?

The model capacity of neural networks, especially deep networks, has some non-trivial relation to the number of filters or the number parameters. It is important to quantify such relation. A good reference might be [A]. Briefly, the generalization performance may not be monotonic against the number of parameters.

The baselines are not enough. Of course, Abadi et al.’s work is outstanding in handling the privacy learning of deep networks. It has been further developed by the following researchers. For example, [B] and [C]. Does the conclusion still hold for these algorithms?

[A] Neyshabur, B., Bhojanapalli, S., Mcallester, D., & Srebro, N. (2017). Exploring Generalization in Deep Learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, & R. Garnett (Eds.), Advances in Neural Information Processing Systems 30 (pp. 5947–5956). 
[B] Yu, L., Liu, L., Pu, C., Gursoy, M. E., & Truex, S. (2019). Differentially Private Model Publishing for Deep Learning. Proceedings of 40th IEEE Symposium on Security and Privacy. 
[C] Phan, N., Vu, M. N., Liu, Y., Jin, R., Dou, D., Wu, X., & Thai, M. T. (2019). Heterogeneous Gaussian Mechanism: Preserving Differential Privacy in Deep Learning with Provable Robustness. Proceedings of the Twenty-Eighth International Joint Conference on Artificial 
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper methodically analyses the settings and choices used when training neural networks (specifically CNNs) via the DP-SGD algorithm and suggests changes to the standard procedures that empirically lead to higher accuracies despite the added noise. The main statement of the paper is quite simple: optimize hyperparameters for the model that you're training (DP-SGD) rather than the model it is inspired by. Yet, the findings an recommendations may be useful for practitioners.

Nevertheless, to be more practically relevant the paper needs some modifications:

The example models used in demonstrations are quite small (3 hidden layers 26,000 parameters, when, for example, a standard segmentation CNN model U-net can typically have 26,000,000, AlexNet has about 60,000,000 and so on). The results would be much more convincing if these or other models widely used in practice were used as running examples.

In Figure 1 the multitude of point on the plot makes it unclear whether they represent result variability per number of filters or simply reflect variability as the number of filters grows. If it is the latter, it seems appropriate to perform a cross validation analysis and report standard deviations. Especially in the MNIST plot the values for SGD and DP-SGD are so close that they may in fact be statistically indistinguishable. Hard to tell by looking at a point estimate. The same request holds for Figure 2, where the difference may be immaterial, but as the figure currently stands it is unclear.

Section 3.2 reports some numbers for test accuracy but the uncertainty of these numbers with respect to the test set changes (cross validation) is not reported and the numbers are quite close to each other. Furthermore, the dataset is not described and it is unclear what was the size of the training and the test sets.

Conclusions relative Adam vs SGD seem to repeat what's already known or been discussed about these methods outside of the DP topic. May be worth highlighting that when one knows how to set learning rates for SGD (may be via learning rate scheduler, not discussed in the paper but practically relevant) then SGD may be as good or slightly better than Adam. However note, adaptive optimizers are often preferred for their ease of use as no tweaking and searching for an optimal learning rate is required. Would not this problem be detrimental for SGD optimization affecting the privacy budget?

Please add wall-clock time column to Table 5 to support the statement about 4 times gain.

I think it's more accurate to change " This confirms that earlier theoreticalanalysis (Talwar et al., 2014) also holds in the non-convex setting." to "This suggests that earlier theoreticalanalysis (Talwar et al., 2014) also holds in the non-convex setting."


review_title: Official Blind Review #1

Meta-review: This paper presents experimental evidence that learning with privacy requires optimization of the model settings (architectures and initializations) that are not identical to those used when learning without privacy. While acknowledging potential usefulness of this work for practitioners, the reviewers expressed several important concerns such as (1) lack of SOTA baseline comparisons, (2) lack of clarity of the empirical evaluation protocols, (3) large models (that are widely used in practice) have not been studied in the paper, (4) low technical novelty. The authors have successfully addressed some of the concerns regarding (1) and (2). However (3) and (4) make it difficult to assess the benefits of the proposed approach for the community and were viewed by AC as critical issues. We hope the detailed reviews are useful for improving and revising the paper. 
Decision: Reject