Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents an efficient implementation of per-sample gradient clipping on convolutional layers, i.e., mixed ghost clipping for DP training.  The experiments on two CIFAR datasets with large CNN models (i.e., ResNet, VGG, and Vision Transformers) demonstrate that mixed ghost clipping reduces the time and space complexities compared to baselines, and achieves SOTA performances under DP.

strengths_and_weaknesses: Strengths
The proposed method is efﬁcient and scalable for practical use.
Extensive experiments on CIFAR10 and CIFAR100 with large CNN models demonstrate that mixed ghost clipping can reduce the time and space complexities while achieving good utility under DP.
Weakness
The novelty of this work seems to be just applying the same techniques of [32] in the context of the convolution layer. Specifically, the gradient norm calculation in Equation 2.7 is a direct modification of Equation 3 in [32]. Moreover, the layerwise decision of whether to use the ghost norm is based on the memory complexity comparison between $2T_{(l)}^2$ and $p_{(l)} d_{(l)}$, which actually is also discussed in [32] right below Equation 3. Therefore, I think the technical contributions might not be strong enough.
I find Equation 2.1 in preliminaries confusing at first glance. Strictly, following standard DP-SGD, the $\tilde{g}$ is the averaged clipped gradients over one batch, rather than the sum of clipped gradients. The choice of batch size $B$ will significantly affect the utility under DP. However, in Equation 2.1, B is neglected.  It would be better if the authors could mention why B is not considered here.
questions: See Weakness. 
limitations: See weakness.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper introduces mixed ghost clipping for convolutional layers in neural networks (tests are made on CNNs and Vision Transformers). The paper introduces ghost clipping for convolutional layers, and since the advantage depends on the layer parameters, mixed ghost clipping is proposed where ghost clipping is applied only for some of the layers, and is combined with FastGradClip. In regular clipping, we first compute per-sample gradients and then compute the norms. Ghost clipping wants to use ghost norm instead (which takes advantage of activations) to reverse the process and obtain the per-sample gradient norm. The mixed ghost clipping is designed with saving space complexity in mind. This is done through adapting the method for dimension parameters which are different for different layers.



strengths_and_weaknesses: The problem of training NN models, and in particular DP NN models with large batch data is important and long-standing, and the ability to increase the batch for the DP training is the main strength of this paper. That is, the method allows to increase the maximum batch size and throughput for the fixed memory budget. The paper also claims improve the training speed. The paper presents detailed analysis of its mixed ghost clipping, that is when exactly ghost clipping is beneficial.  The paper summarizes and compares the space and time complexity for ghost clipping and per-sample gradient instantiation. Moreover, it reads well and the concepts are well-described.
The bulk of derivation comes from the previous work, and the implementation of convolutional layers also follows matrix multiplication, therefore, correct me if I’m wrong, I’m not sure if this contribution is so significant.  I consider the main contribution of the work, the analysis which method (ghost clipping vs. FastGradClip) to select for a given layer.

questions: From what I understand we get the per-sample activations through the forward pass and the per-weight gradient through the backward pass. The per-sample gradient is obtained by taking inverse matrix multiplication of the per-weight gradient with the activation. Do I understand it correctly? 
Could you please write a few lines of code where you extract dL/ds_i for a single data sample? What about the code release?
Also, in Sec. 4, you’re saying that the speed-priority version has empirically insignificant difference, while later you’re praising the increases in training speed (e.g. <2x). So you mean the extra back-propagation has no real impact? Is this all related to being able to employ a larger batch size, or is there anything else?

limitations: Yes.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: DP-SGD requires each samples gradient in a batch to be clipped to a maximium norm. The simplest way to do this is compute each samples gradient independently and clip. This comes at the expense of both memory and speed -- computing a large number of examples gradients in parallel will eat up available memory, and this in turn limits the size of batches that can be used in DP-SGD without gradient accumulation (which itself is slow). This work extends the idea of finding efficient per-example gradients for linear layers by noticing that per-sample gradients can be found by the product of a single layer input with the partial of the loss with respect to a single output, and these quantities are available with standard hooks in auto-diff libraries.  
strengths_and_weaknesses: Strengths:

Strong experimental results that show the new technique ("mixed ghost-clipping") outperforms other methods in terms of memory and speed. This results in the ability to train large models with larger batch sizes than was previously possible.

Weaknesses:

As someone unfamiliar with this line of work, it is unclear how significant the contributions are over [32] and [38]. I would have appreciated a more in-depth discussion of the comparison between these works.

Some over-claims that are unnecessary: "we are the first to train convolutional ViTs under DP". I think [33] also trained DP models with ViTs.


[33] H. Mehta, A. Thakurta, A. Kurakin, and A. Cutkosky. Large scale transfer learning for differentially private image classification. arXiv preprint arXiv:2205.02973, 2022.
questions: 
The experimental results are impressive and convincing, particularly Fig 3, which shows mixed-ghost-clipping can fit substantially more samples per-batch than other methods. 

Because the technique requires second order methods, how large does a network need to be for the gains in memory + speed to vanish, and this method to perform worse than e.g. Opacus?


Line 89: "In all above-mentioned methods and [38], the per-sample gradients are instantiated,...". Can you please expand on this, as it doesn't look like per-sample gradients need to be instantiated from a look at [38]. Isn't this another extension of [18], that benefits from exactly the opposite -- that per-sample gradients don't need to be individually instantiated? I would really appreciate a detailed discussion between this work and [38].
limitations: I foresee no limitations due to negative societal impacts.
review_title: null

Meta-review: metareview: The paper resulted in reasonably positive reviews, and the rebuttal phase cleared most of the reviewer concerns. I will request the authors to incorporate the reviewer suggestions, in particular a more detailed comparison to the paper [32].
confidence: Certain
Decision: Accept