Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper implements existing DP optimizers with 1.24x training speed than the most efficient competitor. Previous methods accelerate the DP training speed by using two rounds of back-propagation or sacrificing memory. This paper uses one back-propagation and never instantiates per-sample gradients so that it reduces computational costs. 
strength_and_weaknesses: Strength

1. Table 2 provides a clear and comprehensive comparison of the existing methods.

2. The paper provides a complete comparison of pseudo-codes for their algorithm and the competitors.

Weaknesses

W1. The paper combines some tricks from existing works to save computational costs of deep learning with DP. However, when comparing their code with SOTA, it seems that the proposed algorithm only replaces one summation step with matrix operations, which lacks novelty.

W2. The paper repeatedly emphasizes that it can reduce the time and memory complexity of DP deep learning. However, the actual reduction is only a constant ("1.24× training speed in practice").  Moreover, according to the experiments in Figures 2 and 5, when the input dimension is large, the proposed algorithm only shows minor improvement in time or memory compared to the state of the art.

W3. Figure 3 is difficult to read.

W4. The paper is not self-contained in the sense that it refers to a number of tables in the main text, without mentioning that those tables are only available in the supplementary material.
clarity,_quality,_novelty_and_reproducibility: The organization of the paper has some problems. A few examples:

1. Table 1 is shown in Section 1.1 but the first explanation of it lies in Section 2.3.

2. Figure 3 is referred to before Figure 2, but it appears much later than Figure 2.
summary_of_the_review: The novelty of the paper seems low, and the propose solution only achieves minor improvements against the state of the art.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new trick to improve the runtime of DP training with GhostClip. One large cost in DP training is the computation of the per example gradient, which must be clipped before being averaged, so that no example can have an outsized influence on the model. The authors enhance a previously known trick to reduce this cost.

GhostClip, proposed in 2015, allows the computation of the norms of the per example gradients efficiently (i.e. without storing the per-example gradient). Then GhostClip uses a second backpropagation pass to compute the gradients of the weighted loss --- the weighted loss is the sum of the loss for each example weighted by the norm of its gradient, thus achieving clipping.

In the current paper, the authors propose to store the per example gradients of the outputs in the first pass, while computing the norms of the per example gradients. These are used to recompute the gradient of the weighted loss, without needing to use a second backprop pass. Since only the per example gradients of the outputs are stored, and not per example gradients of the parameters, this memory is substantially less than the memory used in standard DP implementations.

The paper then points out that the GhostClip trick works well only when the number of features in a layer is small (each layer's input is of size batchsize * features * channels). For layers where the number of features is large (e.g. the initial layers of a ResNet for ImageNet), the authors do not use GhostClip, instead keeping per example gradients of the parameters. Thus they are able to reduce the huge memory requirement of GhostClip for these cases.
strength_and_weaknesses: The paper explains the implementation very well, comparing nicely against various previous algorithms. It also describe the limitations and the resulting hybrid algorithm well.

The authors mention two tricks, but it was not easy to distinguish these.  Q: What is the purpose of Line 8 in Algorithm 2? The output is not used anywhere else in the code. This is mentioned as one of the two important tricks in the paper, and adds considerably to the time and memory requirements of GhostClip.

The main novelty is in the implementation of the tricks --- the trick itself is straightforward --- instead of recomputing the gradients, the authors store them from the previous pass.

In equation 2, you multiply 2 vectors --- should this be a dot product?
clarity,_quality,_novelty_and_reproducibility: The paper is very clearly written and comprehensive. There are not a lot of experimental results in the paper, but they don't seem needed.

summary_of_the_review: The paper seems to me to be very clear and useful. My only concern is with the question above, hopefully the authors can address it in the discussion.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: the paper proposes a (pytorch-based)  implementation for summing clipped per-sample gradients in DP-SGD and demonstrates empirical gains in terms of compute time. 
strength_and_weaknesses: strength: the authors show empirical gains for common setups of DP training in terms of wall-clock time compared to previous approaches. 

weakness: the paper has several weaknesses, some of which are technical and the others are related to presentation. 

technical weaknesses: 

while the idea for modifying FastGradClip may seem natural, it's worthwhile to note that the current presentation is heavily based on a particular framework -- PyTorch. for instance, Algorithm 1 explicitly states the procedure quoting PyTorch forward and backward hooks. i believe the high-level idea is applicable in other frameworks, but the current algorithmic sol'n is limiting and isn't designed based on shared primitives of automatic-differentiation framework. 

in the main text and appendix C, the authors argue that one major difference between BK and GhostClip is that BK doesn't compute non-private gradients. note that GhostClip need not compute non-private gradients at all, since this quantity is not used anywhere. in fact, computing the non-private gradient is likely an artifact of a particular implementation and is not a general drawback of the approach (of course, neither is this a drawback / limitation of FastGradClip). by not considering this point, the authors likely have "inflated" their computational advantages. 

the authors make several factually incorrect claims. for instance on page 4, authors say "For instance, Li et al. (2021) shows that, when training GPT2-large (774M parameters), Opacus Yousefpour et al. (2021) and JAX Subramani et al. (2021) cannot fit even one single sample into a 16GB GPU." 
  - note the original paper by Li et al. does not present profile results with GPUs which have 16GB video RAM. the selected GPU is TITAN RTX which has 24GB video RAM.
clarity,_quality,_novelty_and_reproducibility: quality: the paper proposes an interesting modification for attaining speed ups, but there are several factually incorrect claims. 
clarity: the paper could improve on the writing and presentation. for instance, authors don't distinguish between \cite and \citep. 
novelty: the minor trick proposed by the paper is new, to the best of my knowledge. 
reproducible: results are likely reproducible to the best of my knowledge. authors agree to release code, which facilitates reproducibility. 
summary_of_the_review: the paper proposes a (pytorch-based)  implementation for summing clipped per-sample gradients in DP-SGD and demonstrates empirical gains in terms of compute time. the trick is new to the best of my knowledge, but the paper has several technical issues.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work presents two "tricks" to speed up the computation of clipped gradients in differentially private training. These are useful engineering insights that give meaningful, even if relatively modest, speed-ups. At the same time the overall novelty and significance of these results is not sufficiently high for acceptance.
justification_for_why_not_higher_score: see review
Decision: Reject