Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacy
Authors: Zhiqi Bu
Jialin Mao
Shiyun Xu
Venue: NeurIPS 2022 Accept (2022)
Keywords: deep learning
differential privacy
complexity
convolutional neural network
vision transformer

Abstract: Large convolutional neural networks (CNN) can be difficult to train in the differentially private (DP) regime, since the optimization algorithms require a computationally expensive operation, known as the per-sample gradient clipping. We propose an efficient and scalable implementation of this clipping on convolutional layers, termed as the mixed ghost clipping, that significantly eases the private training in terms of both time and space complexities, without affecting the accuracy. The improvement in efficiency is rigorously studied through the first complexity analysis for the mixed ghost clipping and existing DP training algorithms.

Extensive experiments on vision classification tasks, with large ResNet, VGG, and Vision Transformers (ViT), demonstrate that DP training with mixed ghost clipping adds $1\sim 10\%$ memory overhead and $<2\times$ slowdown to the standard non-private training. Specifically, when training VGG19 on CIFAR10, the mixed ghost clipping is $3\times$ faster than state-of-the-art Opacus library with $18\times$ larger maximum batch size. To emphasize the significance of efficient DP training on convolutional layers, we achieve 96.7\% accuracy on CIFAR10 and 83.0\% on CIFAR100 at $\epsilon=1$ using BEiT, while the previous best results are 94.8\% and 67.4\%, respectively. We open-source a privacy engine (\url{https://github.com/woodyx218/private_vision}) that implements DP training of CNN (including convolutional ViT) with a few lines of code.

Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacy
Zhiqi Bu*
Jialin Mao'
Shiyun Xu
zbu@ sas. upenn. edu
shiyunxuu sas. upenn edu Department of Applied Mathematics and Computational Science
jmao0sas upenn. edu
University of Pennsylvania
Abstract
Large convolutional neural networks (CNN) can difficult to train the differen- tially private (DP) regime, since the optimization algorithms require computation- ally expensive operation, known as the per-sample gradient clipping. We propose efficient and scalable implementation of this clipping convolutional layers, termed as the mixed ghost clipping, that significantly eases the private training in terms of both time and space complexities, without affecting the accuracy. The im- provement in efficiency is rigorously studied through the first complexity analysis for the mixed ghost clipping and existing DP training algorithms.
Extensive experiments on vision classification tasks, with large ResNet, VGG, and Vision Transformers (ViT), demonstrate that DP training with mixed ghost clipping adds 1 10% memory overhead and < 2x slowdown to the standard non- non-private training. Specifically, when training VGG19 on CIFARIO. 10, the mixed ghost clipping is 3x faster than statc-of-ther ofrttr-artt Opacus library with 18x larger maximum batch size. To emphasize the significance of efficient DP training convolutional layers, we achieve 96.7% accuracy on CIFARIO and 83.0% on CIFARIO0 at 1 using BEiT, while the previous best results 94.8% and 67.4%, respectively. We open- source a privacy engine (nttps: / / g thub. com/ wwtp//w11/ddwvvveea//eeo vision) that implements DP training of CNN (including convolutional VIT) with a few lines of code.
Introduction
Deep convolutional neural networks (CNN) 16 30 are the backbone in vision-related tasks, including
image classification (28), object detection 377, image generation [19], video recognition (39), and
audio classification
22 A closer look at the dominating success of deep CNNs reveals its basis on
two factors.
The first factor is the strong capacity of the convolutional neural networks, usually characterized by the enormous model size. Recent state-of tiite-offthh-arr the-art progresses usually result from very large models, with millions to billions of trainable parameters. For example, ImageNet [TO] accuracy grows when larger VGGs 400 (increasing 11 to 19 layers= to 74% accuracy) ResNets [0] (increasing 18 to 152 layers=>709 to 78% accuracy) are used [42]. Consequently, the eager to use larger models for better accuracy naturally draws people's 's attention to the scalability and efficiency of training.
The second factor the availability of big data, which oftentimes contain private and sensitive information. The usage of such data demands rigorous protection against potential privacy attacks. In fact, one standard approach to guarantee the protection by differentially private (DP) [3] [4] training of the models. Since .1 CNNs have achieved promising results under strong DP guarantee: CIFARIO achieves 92.4% accuracy in [45) and ImageNet achieves 81.1% % accuracy in [9].
"Equal contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Unifying the two driving factors of CNNs leads to the DP training of large CNNs. However, the following challenges are hindering our application of large and private CNNs in practice,
Time and space efficiency in DP training. DP training can be extremely inefficient in memory and speed. For example, straightforward implementation in Tensorflow Privacy library shows that DP training can be 1000 x slower than the non-DP training, even on small RNN [5]; other standard DP libraries, Opacus [49] and JAX (29 41), which trade off memory for speed, could not fit a single datapoint into GPU on GPT2-large has been reported in
Challenge 1:
32 addtionally, 3 ~ 9x slowdown of DP training
29
using JAX.
The computational bottleneck comes from the per-sample gradient clipping at each iteration (see (2.1 a necessary step in DP deep learning. Le., denoting the loss as Σ; Ss, we need to clip the per-sample gradient (a J: individually. This computational issue is even more severe when we apply large batch size, which is necessary to achieve high accuracy of DP neural networks. In [3 29 331, is shown that the optimal batch size for DP training significantly larger than for regular training. For instance. DP ResNet18 achieves best performance on ImageNet when batch size is 64 1024 [29]: and DP 55 and ViT-Large use a batch size 2 in [33.. As a result, an efficient implementation of per-sample gradient clipping is much-needed to fully leverage the benefit of large batch training.
Challenge 2: Do large DP vision models necessarily harm accuracy? An upsetting observation in DP vision models that, over certain relatively small model size, larger DP CNNs seem to underperform smaller ones. This is observed in models that are either pre-trained or trained from scratch 25 As an example of the pre-trained cases, previously state-of-the-art CIFARIO is obtained from a small DP linear model [45, and the fine-tuned DP ResNet50 underperforms DP ResNet18 on ImageNet [29]. On the contrary, the empirical evidence in DP language models shows that larger models can consistently achieve better accuracy (32). Interestingly, we empirically demonstrate that this trend can possibly hold in vision models as well.
1.1 Contributions
In this work, we propose new algorithms to efficiently train large-scale CNNs with DP optimizers. To be specific, our contributions are as follows.
1. We propose a novel implementation, termed as the mixed ghost clipping, of the par-sample gradient clipping for ID~3DD convolutional layers. The mixed ghost clipping is the first method that can implement per-sample gradient clipping without per-sample gradients of the convolutional layers. It works with any DP optimizer and any clipping function almost as memory efficiently as in standard training, thus significantly outperforming existing implementation like Opacus [49].
2. In some tasks, mixed ghost clipping also claims supremacy on speed using a fixed batch size. The speed can be further boosted (say 1.7x faster than the fastest alternative DP algorithms and only 2x slower than the non-private training) when the memory saved by our method used to fit the largest possible batch size.
3. We provide the first complexity analysis of mixed ghost clipping in comparison other training algorithms. This analysis clearly indicates the necessity of our layerwise decision principle, without which the existing methods suffer from high memory burden.
4. Leveraging our algorithms, we can efficiently train large DP models, such as VGG, ResNet, Wide-ResNee DenseNet, and Vision Transformer (ViT). Using DP ViTs at ImageNet scale, we are the first to train convolutional ViTs under DP and achieve dominating SOTA on CIFARION 10/100 datasets, thus bringing new insights that larger vision models can consistently achieve better accuracy under DP.
1.2 Previous arts
The straightforward yet highly inefficient way of per-sample gradient clipping is to use batch size and compute gradients with respect to each individual loss. Recently, more advanced methods have significantly boosted the efficiency by avoiding such a naive approach. The most widely applied method is implemented in the Opacus library [49], which is fast but memory costly par-sample

---

gradients 9: aw are instantiated to compute the weighted gradient Σ, C in (2.1 A more efficient method, FastGradClip [1 is use a second proppagtttt with weighted loss >, C.
D2,
to indirectly derive the weighted gradient.
In all above-mentioned methods and [38 the per-sample gradients are instantiated, whereas this can be much inefficient and not necessary according to the "ghost clipping' technique, as to be detailed in Section] 3 In other words, ghost clipping proves that the claim 'DP optimizers require access the per-sample gradients" is wrong. Note that ghost clipping is firstly proposed by [8] for linear layers, and then extended by [2] to sequential data and embedding layers for language models. However, the ghost clipping has not been extended to convolutional layers, due to the complication of the convolution operation and the high dimension of data (text data mostly 2D, yet image data are 3D and videos are 4D). We give more details about the difference between this work and [32] in Appendix F In fact, we will show that even the ghost clipping alone is not satisfactory for CNNs: e.g. it cannot fit even single datapoint into the memory on VGGs and ImageNet dataset. Therefore, we propose the mixed ghost clipping, that narrows the efficiency gap between DP training and the regular training.
2 Preliminaries
2.1 Differential privacy
Differential privacy (DP) has become the standard approach to provide privacy guarantee for modern machine learning models. The privacy level is characterized through pair of privacy quantities (e, 6). where smaller (e, 6) means stronger protection.
Definition 2.1 (14).. A randomized algorithm M is (c, 6)DDP if for any neighboring datasets S, S that differ by one arbitrary sample, and for any event E, it holds that
P[M(S) E EJ < P [M (S') E E
In deep learning where the number of parameters are large, the Gaussian mechanism 14 Theorem A.1] is generally applied to achieve DP at each training iteration, i.c. we use regular optimizers on the following privatized gradient:
g = O(1gsll: R) gi + oR N Cg.
+....aR--oo - CoR-
(2.1)
where C' is any function whose output upper bounded by R// 11/1 and R is known as the clipping norm. To name a few examples of C, we have the Abadi's 's clipping min (R/lll::)) 1) in [1] and the global clipping I| Z) R/Z for any constant 2 in 6 Here the noise multiplier that affects the privacy loss (c, but R only affects the convergence, not the privacy.
In words, DP training switches from updating with E, 9i updating with the private gradient g: SGD with private gradient is known as DP-SGD; Adam with private gradient known as DP-Adam.
Algorithmically speaking, the Gaussian mechanism can be decomposed into two parts: the per-sample gradient clipping and the Gaussian noise addition. From the viewpoint of computational complexity, the per-sample gradient clipping is the bottleneck, while the noise addition costs negligible overhead.
In this work, our focus is the implementation of per-sample gradient clipping (2.I). We emphasize that our implementation is only on the algorithmic level, not affecting the mathematics and thus not the performance of DP optimizers. That is, our mixed ghost clipping provides exactly the same accuracy results as Opacus, FastGradClip, etc.
2.2 Per-sample gradient for free during standard pack-propagation
In DP training, the per-sample gradient is key quantity which can be derived for free from the standard back- pecc-prppggttin.. We briefly introduce the back-propagation on linear layers, following the analysis from [8 32), as prepare our new clipping implementation for convolutional layers.
Note that the convolutional layers can be viewed as equivalent to the linear layers in Section 2.3
'The method in (88 also extends the outer product trick (similar to Opacus, see 2.4 in 18 to convolution layer, but does not the ghost clipping trick.
3

---

Let the input of a hidden layer be E RBxxd (a.k.a. post-activation). Here can be in high dimension: for sequential data such as text, a E RxTxx where T is the sequence length; for image data, E TR Bx xd where (H, W) is the dimension of image and d is number of channels; for 3D objects or video data, E XX Dxd where D is the depth or time length, respectively.
Denote the weight of linear layer W E Rdxp, its bias as b E IP and its output (a.k.a. pre- activation) as s E I Rxx where B is the batch size and p is the output dimension.
In the 1-th layer of a neural network with L layers in total, we denote its weight, bias, input and output as W (1); b(t), a()) 8()) respectively, and the activation function as cb Consider
a(++1), = () c(a ((C) (1) b()))
(2.2)
Clearly the i-th sample's hiddeffeature feature a(),, at layer is freely extractable during the forward pass. Let C = Ei=1 C be the total loss and L; be the per-sample loss with respect to the i-th sample. During a standard back poopapropag,tion, the following partial product is maintained,
Oc Oc da(L)s d8(L-1),
daq+1).f
OC
Wa+n a (s()a) (2.3)
da(L), dS(L-1), 0a(1-1).3
Üs
O8(+1)s
so as to compute the standard gradient
Oc Σ;
Dc in 2.4) Here o is the Hadamard product aW.,
JW (1) Oc
and is the matrix product. Therefore, Usay, is also available for free from 2.3 and extractable by
Pytorch hooks, which allows compute the per-sample gradient by
ac: ac,
0s OC Os
OC, ac, Os(1a Oc T Os ab Os()), 1. (2.4)
ag),
aw Ösq), dw
2.3 Equivalence between convolutional and linear layer In convolutional layer the forward pass is
a(++1), = ((s(),)) = ((F(U((())))) W() + bbn))
(2.5)
in which F is the folding operation and U is the unfolding operation. To be clear, we consider 2D convolution that a(ts)) E I2 x Win Xd() is the input of hidden feature. (Hi, Win) the input dimension, dey is the number of input channels. Then U unfolds the hidden feature from dimension (Hi,, Win, duy) to (Howr Wout, dig ki k. where kH, kw are the kernel sizes and (Homt W Won) is the output dimension. After the matrix multiplication with W (I) E Rdurkakw Xpa), the intermediate output (1), is folded by F' from dimension (How Wouts P(t)) to (Homt Wons P(t))-
To present concisely, we ignore the layer index l and write the per-sample gradient of weight for the convolutional layer, in analogy to the linear layer in (24).
Oc: OW
Oc
(
U(a:)) = F-1
U(a)..
(2.6)
aF-1(s:)
as:
Here F-1 is the inverse operation of F and simply flattens all dimensions except the last one: from (Hout, W out P())) to (How Wout out, P.t)). P. From (2.6). we derive the per-sample gradient norm for the convolutional layers from the same formula as in (32] Appendix F].
ac, 2
(OC) 0s:
OL
= vec(U(a))U(a)) T 'eec
F-!
F-1
(2.7)
aw
0s:
Pro
3 Ghost clipping for Convolutional Layers
Leveraging our derivation in (.7). we propose the ghost clipping to compute the clipped gradient without ever generating the per-sample gradient aw The entire procedure is comprised of the ghost norm computation and the second back-propagation, demonstrated in Figure I
See detailed explanation in Appendix for the U, F operation and the dimension formulae in convolution.

---

3.1 Ghost norm: computing gradient norm without the gradient
The per-sample gradient norm is required to compute the per-sample C₁ in While it natural
to instantiate the per- sample gradients and then compute their norms 38 31 49 9 [3], this is not always optimal nor necessary. Instead, we can leverage (2.7), the ghost norm, to compute the per-sample gradient norm and avoid the possibly expensive per-sample gradient. Put differently, when
T Ho Wau is small, the multiplication ((a))U(a,)) T plus F-1
F-1 (ac) (85:
F-1
[ac (9,,
cheap, but
U ((a)) is expensive. We demonstrate the ghost clipping's supremacy over complexity empirically in Table 4 and theoretically in Table 2
the multiplication 1-11
BC
bat propeationn
foroard pana
beak prpaaalies
back prosupttion
fowaadd paim adtvedion ag
adiviiz
partll product
para poscat
(*)))
pansanplegra (ai) (lN))
poe gasl (P (1ee) (Dc)
god Hore
peu roripl
surijal
Isl - 1
ciipiing futer -uis (c)
C) sito Sa
vil d.
- Σ, 444
E,
wnighted Σn Cjk
praspation
Σ, Cat
Σ Cip
photwwggt Σ Cup - ellAi1 I)
prlattee gran Σ, Ggg +
griwliww post Σ, Cup + e1tW/ 1)
Figure 1: Per-sample gradient clipping for convolutional layers. Left: Opacus = Back-Bcckkoppee Back-propagation + Gradient instantiation + Weighted gradient. Middle: FastGradClip Back-propagation Gradient instantiation + Second paeo-propagation. Right: Ghost clipping = Back- back-propagation + Ghost norm + Second poo-prropagttin. See Section 4.1 for their complexity analysis.
3.2 Second back-propagation weighted loss leads to weight gradient
We conduct a second back propaapopantioon with the weighted loss E, CiLi to derive the weighted gradient Σ, Cig. in 2.I). which costs extra time. In contrast, Opacus [49] and JAX 9 41 generate and store the per-sample gradient g, for all i [B]. Thus the weighted gradient is directly computable from gi as the memory is traded off for faster computation. However, in some cases like Table 7 on ImageNet and TableIoon on CIFAR100, we can use larger batch size compensate the slowdown of the second back-propagation.
4 Mixed Ghost Clipping: To be a ghost or not, that is the question
While the ghost norm offers the direct computation of gradient norm at the cost of an indirect computation of the weighted gradient, we will show that ghost clipping alone may not be sufficient for efficient DP training, as we demonstrate in Table Figure 33] and Table 77] In Table we give the first fine-grained analysis of the space and time complexity for DP trainnn algorithms. Our analysis gives the precise condition when the per-sample gradient instantiation (adopted Opacus [49] is more or less efficient than our ghost norm method. To take the advantage of both methods, we propose the mixed ghost clipping method in Algorithm I which applies the ghost clipping or non-ghost clipping by layerwise decision.
5

---

Algorithm Mixed Ghost Clipping (single iteration)
Parameters: number of layers L, gradient clipping norm R.
for 1.2..... do
(T (1) < <pidda then
Wan-ghost = True
D Forward pass
Compute ((41), = (F((U(a)))) bbn))
per-splle losses Lt
for lorl=Ia L, ac do
Compute (a
dc (
W (++1) ('(())))) Mixed ghost norm in first back-propagation
( (1))-ghhsst_noo _oom True then
8C, I. =
vec
OC (8)))))) OF (F )
ac
else
Ac. OW
ac
(a(r))) -
BC;
1P
(W (1)
Compute per-sample gradient norm TW
11
E1
JC. Pro Lo (2)
Compute weighted loss Lweighted
E. C(I ol
Il
R
Second back-propagation with Leeighted to generate E, C2 OW
AC,
We highlight that the key reason supporting the success of mixed ghost clipping method is its layerwise adaptivity to the dimension parameters, (p()) dqy= Twy kH, kw), which vary largely across different layers (see Figure 2). The variance results from the fact that images are non-sequential data, and that the convolution and pooling can change the size (T" = How out) of hidden features drastically.
In the next two sections, we will analyze rigorously the time and memory complexities of the regular training and the DP training, using ghost non-ghost clippings.
Remark 4.1. In Algorithm[]] we present the mixed ghost clipping that prioritizes the space complexity by (4.1). We also derive and implement a speed-priority version by comparing the time complexity of ghost norm and gradient instantiation in Table [ However, the efficiency difference is empirically insignificant and implied by Table I
4.1 Complexity analysis
We now break each clipping method into operation modules and analyze their complexities. A similar but coarse analysis from [22 only claims, on sequential layers, O(B2²) space complexity with ghost clipping and O (Bpd) without ghost clipping. The time complexity and/or convolutional layers not analyzed until this work.
Complexity
Back Back- propagation
Ghost norm
Grad instantiation Weighted grad
Time
2BTD(2p+1)
2B(T + 1)pD
2Bp
Space 2Tp+2BTDD+
B22 +1
B(pD 1)
Table 1: Complexities of operation modules in per-sample gradient clipping methods, contributed by
a single 2D convolutional layer.
Here B is the batch size, D dk where d is the number of input channels, i is the kernel sizes, is the number of output channels, and T Hou Wou- We leave the detailed complexity we give the complexities of different clipping computation in Appendix C Leveraging Table 1.
algorithms in Table 2
4.2 Layerwise decision in mixed clipping
From the space complexity in Table 2 derive the layerwise decision that seleets the more memory efficient of FastGradClip (gradient instantiation) and ghost clipping (ghost norm):
Choose ghost norm over per-sample gradient instantiation if 2T pD = pdk (4.1)

---

Complexity
Time
Space
Opacus 49
6B7pD
B(ph) + Tp 2TD)*
CastGradClip B1
8B1pD
B(pD Tp 2I
Ghost clipping (ours)
8BTpD + 2BT D)
B(2 + +Tp 2TD)
Mixed ghost clipping (ours)
SEE CAPTION
((min(2T
+ 2TD)
Table 2: Complexity of different implementations of per sample gradient clipping by a single 2D convolutional layer. Only highest order terms are listed. indicates that Opacus stores the per-sample gradients of all layers, thus a per-layer space complexity does not accurately characterize its memory burden, since other methods only store the intermediate variables one layer at time. The mixed ghost clipping's ss time complexity is between FastGradClip and ghost clipping, depending on which of (27 .DD) is smaller.
Ghost norm
Non-ghost norm
Space
2T 2H out
W
complexity
conv
5.0 x 10
1.7 X 103
conv2
3.0 x 10a
7.3 x 104
conv3
2.0 x 10"
2.9 x 10
5400 S1M0
conv4
2.0 x 10
5.8 10
conv5
1.2 x 10"
1.1 X 106
conv6
1.2 X 10
2.3 x 106
conv7
7.6 X 10*
2.3 x 10
conv8
7.6 x 10*
2.3 x 10
fc9
2
1.0 10
fc10
2
1.6 107
004
fcll
2
4.1 x 10
198
Total complexity Mixed ghost norm
5.34 x 10"
1.33 x 10
3.40 x 10a
Figure 2: VGG-11 architecture on ImageNet (224 x 224).
Table 3:
Layerwise decision of mixed ghost clipping on VGG-11. Green background indicates being selected.
Therefore, our mixed ghost clipping is a mixup of FastGradClip and the ghost clipping (c.f. Figure T We note that the decision by the mixed ghost clipping (4.1) depends on different dimensions: the ghost clipping depends on the size of hidden features (height and width W) which in turn depends on kernel size, stride, dilation and padding (see Appendix B for introduction of convolution), while only the non-ghost clipping depends on the number channels. ResNet and VGG, the hidden feature size decreases as layer depth increases, due to the shrinkage from the convolution and pooling operation; on the opposite, the number of channels increases in deeper layers.
Remark 4.2 (Ghost clipping favors bottom layers). As consequence of decreasing hidden feature size and increasing number of channels, there exists a depth threshold beyond which the ghost clipping preferred in bottom layers, where the save in complexity is substantial. In Figure and Table 3 as the layer of VGG Il goes deeper, the hidden feature size shrinks from 224 + 112
and
the number of channels increases from 3 + 64 + 512.
5 Performance
We compare our ghost clipping and mixed ghost clipping methods state-of-the-art clipping algo- rithms, namely Opacus [49] and FastGradClip [3I], which are implemented in Pytorch. We are aware of but will not compare to implementations of these two algorithms in JAX (4) e.g. [29 41 9. as to only focus on the algorithms rather than the operation framework. All experiments run on one Tesla V 100 GPU (16GB RAM).
We highlight that switching from the regular training to DP training only needs a few lines of code using privacy engine (see Appendix E). For CNNs, use models from https: /github

---

com/
on CIFARIO (32 x 32) 27 and models from Torchvision
42 on ImageNet (224 224) [[O]. For ViTs, regardless datasets, we resize images to 224 224 and use models from PyTorch Image Models (TIMM) [46]
5.1 Time and memory efficiency (fixed batch size)
We first measure the time and space complexities when the physical batch size is fixed. Here we define the physical batch size (or the virtual batch size) as the number of samples actually fed into the memory, which is different from the logical batch size. For example, if we train with batch size 1000 but can only feed 40 samples to GPU at one time, we back-pra prockkate 25 times before updating the weights for time. This technique is known as the gradient accumulation and is widely applied in large batch training, which particularly benefits the accuracy DP training 32
Datasel
Model & # Params
Package
Time (scc) Epoch
Active Memory (GB)
Opacus
12
1.37
FastGradClip
11
0.94
CNN 45 35 0.551M
Ghost (ours)
11
2.47
Mixed (ours)
0.79
Non-DP
5
0.66
Opacus
OOM / OOM / OOM
OOM / OOM OOM
FastGradClip
807807 OOM
6.327 7.3277.657 7//77765000
CIFARIO
ResNet 18 /4//50 11M / 21M / 23.5M
Ghost (ours)
/9/98/ / 158
4.00 / /.90 /9.62
Mixed (ours)
66/ 119
4.31/4.1399.0
Non-DP
14/24 49
Opacus
OOM OOM OOM
OOM/ OOM OOM
FastGradClip
18 25 /3
5 5.61
VGG 11 13 16 9M / /9.4// / 14.7M
Ghost (ours)
14/ 29
2.58
3.30 3.41
Mixed (ours)
13/18/23
2.58 23882.7677 2.84
Non-DP
57678
2.5472.73722 2.81
Table 4: Time and memory of selected models on CIFARIO (32 x 32), with physical batch size 256.
Additional models are in Table Out of memory (OOM) means the total memory exceeds 16GB.
From Table 4 and the extended Table 6 we see clear advantage of mixed ghost clipping: our clipping only incurs I 1% memory overhead than the regular training, and is the fastest DP training algorithm. In contrast on ResNet18. Opacus uses 5x memory, and FastGradClip uses 2 memory. Even the ghost clipping uses 1.2x memory of regular training, while being slower than both Opacus and FastGradClip.
Similar phenomenon is observed on ImageNet in Table at physical batch size 25, while the mixed ghost clipping works efficiently, we observe that the ghost clipping and Opacus incur heavy memory burden that leads to OOM error on all VGGs and wide ResNets. In fact, the ghost clipping fails in memory all models except the small AlexNet 26
5.2 Maximum batch size and throughput
Importantly, the speed efficiency in Table 4 can be further boosted, if we use up the saved memory to increase the batch size. To stress test the maximum physical batch size and the throughput of each clipping method, we train ResNet 200. VGG [40, MobileNet 23,, ResNeXt [47) AllexNett Wide- ResNet[111 DenseNet[24] and ViTs on CIFARIO and ImageNet, as summarized partially in Figure 3 and in Table respectively. For example, on VGG19 and CIFAR10, the mixed ghost clipping has maximum batch size 18x bigger (thus 3x faster) than Opacus, 3x bigger (thus 1.7x faster) than FastGradClip. and 2x bigger (thus 1.3x faster) than the ghost clipping. Similarly, on Wide- Rede-R and ImageNet, the mixed ghost clipping has maximum batch size 5x bigger than Opacus, 11x bigger than the ghost clipping, and < 0.3% more memory costly than the non-private training.

---

Maximum batch size (16GB RAM)
Non-private
Non-private
Mixed ghost clipping (ours)
Mixed ghast clipping (ours)
2500
Ghost clipping (ours)
Maximum throughput
2520
Ghost clipping (ours)
FastGradClip
GradClp
Opacus
2030
Opacus
1,000
1.000
500
500
Be-Neti0
BeddetMM
Reddett0
ResNe:131
BenNer152
Re-hie11
Re-NleMM
Beddlet0
Re<NMe1101
NexNei112
Figure 3: Memory (left) and speed (right) comparison of DP clipping algorithms on CIFARIO.
5.3 Vision transformers with convolution on ImageNet scale
In addition training large-scale CNNs such ResNet1 52, we apply our mixed ghost clipping to train ViTs, which substantially outperform existing SOTA on CIFARIO and CIFAR 100. Notice that ViTs are pretrained on ImageNet scale, by which we resize CIFAR images (from 32 x 32 pixels
to 224 224 pixels).
is worth mentioning that VIT 12 originally proposed as a substitute of CNN. Hence and many variants do not contain convolutional layers. Here we specifically consider the convolutional ViTs, including Scalable 48), X Visformer Cross Vit [), N5sT[33., Ca) DeiT(33) BETT 3, 21 and 155 Performance of these ViTs on CIFARIO and CIFAR100 are listed Appendix Difor single-epoch DP training and several ViTs already beat previous SOTA, even though we do not apply additional techniques as in (e.g. learning rate schedule or random
33
data augmentation).
CIFAR-10
CIFAR-100
1
94.3%
Yu et al. 50
(ImageNetlk)
2
94.8% 8%
Tramer et al. 45
(ImageNet1k)
2
92.7%
1
94.8%
67.4%
De et al.
9
2
95.4%
74.7%
96.1 %
79.2%
(ImageNetlk)
96.6% 8
81.8%
1
95.5%
71.9%
Our Cross ViT base (104M params)
2
96.1% %
74.3%
4
96.2%
76.7%
(ImageNetlk)
96.5% 8
77.8%
1
96.7%
83.0%
Our BEiT large (303M params)
2
97.1 %
86.2%
4
97.2%
87.7%
(ImageNet21k)
8
97.4%
88.4%
Table 5: CIFAR-I0 and CIFAR-ION (resized 224 224) test accuracy when fine-tuning with
DP-Adam. We train Cross ViT base for 5 epochs, learning rate 0.002. We train BEiT large for epochs and learning rate 0.001. Here batch size 1000 and clipping norm 0. '()' indicates pretrained datasets.
By training multiple epochs with best performing ViTs Table g and Table we achieve new SOTA under DP in Ta with substantial improvement especially for strong privacy guarantee (e.g. < 2). Our DP training is at most 2x slower and 10% more memory expensive than the non-private training, even BEiT large, thus significantly improving the 9x slowdown reported in [9].

---

DP time (soe)
DP training memory cost (GB)
1.
1503
1303
Mare
502
10
500
1500 traiing apach
1000
Standard mamory
Figure 4: Memory (left) and speed (right) comparison DP and non-DP training on CIFAR 100 (resized to 224 224, ImageNet scale) with convolutional ViTs. Note that CIFAR10 10 has almost identical pattern.
Discussion
We have shown that DP training can be efficient for large CNNs and VITs with convolutional layers. For example, in comparison to non- non-private training, we reduce the training time to 2x and the memory overhead to < 10% for all vision models examined (up to 303.4 million parameters), including BEiT that achieves SOTA accuracy CIFARI00 (+15.6% absolutely e = 1). We have observed that for many tasks and large CNNs and ViTs, the memory overhead DP training can be
as low as less than 1 %..
We emphasize that our DP training only improves the efficiency, not affecting the accuracy, and therefore is generally applicable, e.g. with SOTA data augmentations in (9] With efficient training
algorithms, we look forward to applying DP CNNs to generation tasks [9 seq-to-seq learning [17], text classification [52] reinforcement learning 34 and multi-modal learning. Further reducing time complexity and prioritizing speed in DP training another future direction.
In particular, our layerwise decision principle in (4.1) highlights the advantages of ghost clipping when 7 HW is small. This advocates the of large kernel sizes in DP learning, as they shrink the hidden feature aggressively, and have been shown to be highly accurate on non-private tasks
20
36
References
[1] M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang. Deep learning with differential privacy. Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308-318, 2016.
[2] A. Ali, H. Touvron, M. Caron, P. Bojanowski, M. Douze, A. Joulin, I. Laptev, N. Neverova, G. Synnaeve, J. Verbeek, et Xcit: Cross-covariance image transformers. Advances in neural information processing systems, 34, 2021.
[3] H. Bao, L. Dong, S. Piao, and F. Wei. Beit: Bert pre-training of image transformers. In International Conference on Learning Representations, 2021.
[4] Bradbury, R. Frostig, P. Hawkins, M. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne and Q. Zhang. JAX: composable transformations of Python+NumPy programs, 2018
[5] Z. Bu, S. Gopi, J. Kulkarni, Y. T. Lee, H. Shen, and U. Tantipongpipat. Fast and memory efficient differentially private-sgd via projections. Advances in Neural Information Processing Systems, 34, 2021.
[6] Bu, H. Wang, Q. Long, and J. Su. On the convergence and calibration of deep learning with differential privacy. arXiv preprint arXiv:2106.078.. 2021.
10

---

[7] C.-F. R. Chen, Q. Fan, and R. Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 357-366, 2021.
[8] Z. Chen, L. Xie. J. Niu, X. Liu, L. Wei, and Q. Tian. Visformer: The vision-friendly transformer. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 589-598, 2021.
[9] S. De, L. Berrada, Hayes, S. L.. Smith, and B. Balle. Unlocking high-accuracy differentially private image classification through scale. arXiv preprint arXiv:2204. 136500 2022.
[10] J. Deng, W. Dong, R. Socher, Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages leee, 2009.
[11] X. Ding, X. Zhang, J. Han, and G. Ding. Scaling your kernels to 31x31: Revisiting large kernel design in cnns. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 1163-11975, 2022.
[12] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2020.
[13] C. Dwork, F. McSherry, K. Nissim, and A. Smith. Calibrating noise sensitivity in private data analysis. In Theory of cryptography conference, pages 265-284. Springer, 2006.
[14] C. Dwork, A. Roth, et al. The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sci., 9(3-4):211 407, 2014.
[15] S. d' Ascoli, H. Touvron, M. L. Leavitt, A. Morcos, G. Biroli, and L. Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. In International Conference on Machine Learning, pages 2286-2296. PMLR, 2021.
[16] K. Fukushima and S. Miyake. Neocognitron: A self-organizing neural network model for mechanism of visual pattern recognition. In Competition and cooperation in neural nets, pages 267-285, Springer, 1982.
[17] Gehring, M. Auli, D. Grangier, Yarats, and Y. N. Dauphin. Convolutional sequence to sequence learning. In International Conference on Machine Learning, pages 1243-1252. PMLR, 2017.
[18] I. Goodfellow. Efficient per-example gradient computations. arXiv preprint 150.0715,, 2015.
[19] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[20] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[21] B. Heo, S. Yun, D. Han, S. Chun, J. Choe, and S. J. Oh. Rethinking spatial dimensions of vision transformers, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11936-11945, 2021.
[22] S. Hershey, S. Chaudhuri, D. P. Ellis, J. F. Gemmeke, Jansen, C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold, et al. Cun architectures for large-scale audio classification. In 2017 ieee international conference on acoustics, speech and signal processing (icassp), pages 131-135 IEEE, 2017.
[23] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint 1rXhv:1704.04861,
2017. 2017.

---

[24] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition. pages 4700-4708, 2017.
[25] H. Klause, A. Ziller, Rueckert, K. Hammernik, and G. Kaissis. Differentially private training of residual networks with scale normalisation. arXiv preprint arXiv:2203.00324 2022.
[26] A. Krizhevsky, One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv:1404.3997, 2014.
[27] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[28] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[29] A. Kurakin, S. Chien, S. Song, R. Geambasu, A. Terzis, and A. Thakurta. Toward training imagenet scale with differential privacy. arXiv preprint arXiv:2201. 12328, 2022.
[30] Y. LeCun, L.. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied document recognition. Proceedings of the IEEE, 86(1 (6(11):2278-233) 1998.
[31] Lee and D. Kifer. Scaling up differentiallyy private deep learning with fast per-example gradient clipping. arXiv preprint arXiv:2009.03106 2020.
[32] X. Li. Tramer, P. Liang, and T. Hashimoto. Large language models can be strong differentially private learners. In International Conference on Learning Representations, 2021.
[33] H. Mehta, A. Thakurta, Kurakin, and A. Cutkosky. Large scale transfer learning for differentially private image classification. arXiv preprint arXiv:2205.02977 2022.
[34] V. Mnih, K. Kavukcuoglu, Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 13/2.56022 2013.
[35] N. Papernot, A. Thakurta, S. Song, S. Chien, and U. Erlingsson. Tempered sigmoid activations for deep learning with differential privacy. Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9312 9312-9321, 2021.
[36] C. Peng. X. Zhang, G. Yu, G. Luo, and J. Sun. Large kernel matters-improve semantic segmentation by global convolutional network. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4353-4361, 2017.
[37] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779-788, 2016.
[38] G. Rochette, A. Manoel, and E. W. Tramel. Efficient exr-example gradient computations in convolutional neural networks. arXiv preprint arXiv:191..06015 2019.
[39] K. Simonyan and A. Zisserman. Two-stream convolutional networks for action recognition in videos. Advances in neural information processing systems, 27, 2014.
[40] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Rep- resentations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[41] P. Subramani, N. Vadivelu, and G. Kamath. Enabling fast differentially private sgd via just-in- time compilation and vectorization. Advances Neural Information Processing Systems, 34, 2021.
[42] Torchvision. Pytorch vision description. vision / and httppp///potorccc.oror/bb//////p vision. reanet/
12

---

[43] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357. PMIR, 2021.
[44] H. Touvron, M. Cord, A. Sablayrolles, G. Synnaeve, and H. Jégou. Going deeper with image transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 32 32-42, 2021.
[45] F. Tramer and D. Boneh. Differentially private learning needs better features (or much more data). In International Conference on Learning Reprexentations, 2020.
[46] R. Wightman. Pytorch image models.
wytorch-image imagemmmdeeleeeee-e models 2019.
[47] S. Xie, Girshick, Dollár, Z. Tu, and He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1492-1500, 2017.
[48] R. Yang, H. Ma. J. Wu, Y. Tang, X. Xiao, M. Zheng, and X. Li. Scalablevit: Rethinking the context-oriented generalization of vision transformer. arXiv preprint arXiv:2203.10799 2022.
[49] A. Yousefpour, Shilov, A. Sablayrolles, D. Testuggine, K. Prasad, M. Malek, Nguyen, S. Ghosh, A. Bharadwaj, J. Zhao, et al. Opacus: User-friendly differential privacy library in pytorch. In NeurlPS 2021 Workshop Privacy in Machine Learning, 2021.
[50] D. Yu, H. Zhang, W. Chen, and T.-Y Y. Liu. Do let privacy overbill utility: Gradient embedding perturbation for private learning. In International Conference on Learning Representations, 2020.
[51] Zagoruyko and N. Komodakis. Wide residual networks. British Machine Vision Conference 2016. British Machine Vision Association, 2016.
[52] X. Zhang. J. Zhao, and Y. LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.
[53] Z. Zhang, H. Zhang, L. Zhao, T. Chen, S. Arik, and T. Pfister. Nested hierarchical transformer: Towards accurate, data-efficient and interpretable visual understanding. 2022.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [N/A]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] No theoretical results.
(b) Did you include complete proofs of all theoretical results? [N/A] No theoretical results.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material as URL)? [Yes] Experiment details are included the appendix. We will release a URL for the experiments.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See appendix.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We state the experiment section that we use single Tesla V100 GPU.
4. If you using existing assets (e.g., code, data, models) or curating//relasing
assets.... (a) If your work uses existing assets, did you cite the creators? [Yes] We cite clearly the codebases such as Torch Vision, Pytorch Image Models, Opacus, etc. We also cite the datasets properly.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [N/A] All datasets are publically available.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] All datasets are publically available.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "vG6sgzx-9K7", "reviewer": "Reviewer_b311", "summary": "The paper introduces mixed ghost clipping for convolutional layers in neural networks (tests are made on CNNs and Vision Transformers). The paper introduces ghost clipping for convolutional layers, and since the advantage depends on the layer parameters, mixed ghost clipping is proposed where ghost clipping is applied only for some of the layers, and is combined with FastGradClip. In regular clipping, we first compute per-sample gradients and then compute the norms. Ghost clipping wants to use ghost norm instead (which takes advantage of activations) to reverse the process and obtain the per-sample gradient norm. The mixed ghost clipping is designed with saving space complexity in mind. This is done through adapting the method for dimension parameters which are different for different layers.\n\n\n", "questions": "From what I understand we get the per-sample activations through the forward pass and the per-weight gradient through the backward pass. The per-sample gradient is obtained by taking inverse matrix multiplication of the per-weight gradient with the activation. Do I understand it correctly? \nCould you please write a few lines of code where you extract dL/ds_i for a single data sample? What about the code release?\nAlso, in Sec. 4, you’re saying that the speed-priority version has empirically insignificant difference, while later you’re praising the increases in training speed (e.g. <2x). So you mean the extra back-propagation has no real impact? Is this all related to being able to employ a larger batch size, or is there anything else?\n", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The problem of training NN models, and in particular DP NN models with large batch data is important and long-standing, and the ability to increase the batch for the DP training is the main strength of this paper. That is, the method allows to increase the maximum batch size and throughput for the fixed memory budget. The paper also claims improve the training speed. The paper presents detailed analysis of its mixed ghost clipping, that is when exactly ghost clipping is beneficial.  The paper summarizes and compares the space and time complexity for ghost clipping and per-sample gradient instantiation. Moreover, it reads well and the concepts are well-described.\nThe bulk of derivation comes from the previous work, and the implementation of convolutional layers also follows matrix multiplication, therefore, correct me if I’m wrong, I’m not sure if this contribution is so significant.  I consider the main contribution of the work, the analysis which method (ghost clipping vs. FastGradClip) to select for a given layer.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ajdT6Wh9uWD", "reviewer": "Reviewer_5r59", "summary": "DP-SGD requires each samples gradient in a batch to be clipped to a maximium norm. The simplest way to do this is compute each samples gradient independently and clip. This comes at the expense of both memory and speed -- computing a large number of examples gradients in parallel will eat up available memory, and this in turn limits the size of batches that can be used in DP-SGD without gradient accumulation (which itself is slow). This work extends the idea of finding efficient per-example gradients for linear layers by noticing that per-sample gradients can be found by the product of a single layer input with the partial of the loss with respect to a single output, and these quantities are available with standard hooks in auto-diff libraries.  ", "questions": "\nThe experimental results are impressive and convincing, particularly Fig 3, which shows mixed-ghost-clipping can fit substantially more samples per-batch than other methods. \n\nBecause the technique requires second order methods, how large does a network need to be for the gains in memory + speed to vanish, and this method to perform worse than e.g. Opacus?\n\n\nLine 89: \"In all above-mentioned methods and [38], the per-sample gradients are instantiated,...\". Can you please expand on this, as it doesn't look like per-sample gradients need to be instantiated from a look at [38]. Isn't this another extension of [18], that benefits from exactly the opposite -- that per-sample gradients don't need to be individually instantiated? I would really appreciate a detailed discussion between this work and [38].", "limitations": "I foresee no limitations due to negative societal impacts.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nStrong experimental results that show the new technique (\"mixed ghost-clipping\") outperforms other methods in terms of memory and speed. This results in the ability to train large models with larger batch sizes than was previously possible.\n\nWeaknesses:\n\nAs someone unfamiliar with this line of work, it is unclear how significant the contributions are over [32] and [38]. I would have appreciated a more in-depth discussion of the comparison between these works.\n\nSome over-claims that are unnecessary: \"we are the first to train convolutional ViTs under DP\". I think [33] also trained DP models with ViTs.\n\n\n[33] H. Mehta, A. Thakurta, A. Kurakin, and A. Cutkosky. Large scale transfer learning for differentially private image classification. arXiv preprint arXiv:2205.02973, 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "CGOIxr024R8", "reviewer": "Reviewer_HXMu", "summary": "This paper presents an efficient implementation of per-sample gradient clipping on convolutional layers, i.e., mixed ghost clipping for DP training.  The experiments on two CIFAR datasets with large CNN models (i.e., ResNet, VGG, and Vision Transformers) demonstrate that mixed ghost clipping reduces the time and space complexities compared to baselines, and achieves SOTA performances under DP.\n", "questions": "See Weakness. ", "limitations": "See weakness. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\nThe proposed method is efﬁcient and scalable for practical use.\nExtensive experiments on CIFAR10 and CIFAR100 with large CNN models demonstrate that mixed ghost clipping can reduce the time and space complexities while achieving good utility under DP.\nWeakness\nThe novelty of this work seems to be just applying the same techniques of [32] in the context of the convolution layer. Specifically, the gradient norm calculation in Equation 2.7 is a direct modification of Equation 3 in [32]. Moreover, the layerwise decision of whether to use the ghost norm is based on the memory complexity comparison between $2T_{(l)}^2$ and $p_{(l)} d_{(l)}$, which actually is also discussed in [32] right below Equation 3. Therefore, I think the technical contributions might not be strong enough.\nI find Equation 2.1 in preliminaries confusing at first glance. Strictly, following standard DP-SGD, the $\\tilde{g}$ is the averaged clipped gradients over one batch, rather than the sum of clipped gradients. The choice of batch size $B$ will significantly affect the utility under DP. However, in Equation 2.1, B is neglected.  It would be better if the authors could mention why B is not considered here.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept