Differentially Private Model Compression
Authors: Fatemehsadat Mireshghallah
Arturs Backurs
Huseyin A Inan
Lukas Wutschitz
Janardhan Kulkarni
Venue: NeurIPS 2022 Accept (2022)
Keywords: Differentially Private Training
DP Model Compression
DP Language Models
NLP Tasks

Abstract: Recent papers have shown that large pre-trained language models (LLMs) such as BERT, GPT-2 can be fine-tuned on private data to achieve performance comparable to non-private models for many downstream Natural Language Processing (NLP) tasks while simultaneously guaranteeing differential privacy. The inference cost of these models -- which consist of hundreds of millions of parameters -- however, can be prohibitively large.  Hence, often in practice, LLMs are compressed before they are deployed in specific applications. In this paper, we initiate the study of differentially private model compression and propose frameworks for achieving 50% sparsity levels while maintaining nearly full performance. We demonstrate these ideas on standard GLUE benchmarks using BERT models, setting benchmarks for future research on this topic.

Differentially Private Model Compression
Fatemehsadat Mireshghallah! Arturs Backurs', Huseyin A. Inan',
Lukas Wutschitz', Janardhan Kulkarni University of California San Diego, Microsoft Research, Microsoft fmi resh@enng ucsd.com com
satturs.backurs, huseyin, inan, lukas wutschitz, jakul (wmiiicooott.ommmmmm
Abstract
Recent papers have shown that large pre-trained language models (LLMs) such as BERT, GPT-2 can be fine-tuned on private data to achieve performance comparable to non-private models for many downstream Natural Language Processing (NLP) tasks while simultaneously guaranteeing differential privacy. The inference cost these models which consist of hundreds of millions of parameters however, can be prohibitively large. Hence, often in practice, LLMs are compressed before they are deployed in specific applications. In this paper, we initiate the study of dif- ferentially private model compression and propose frameworks for achieving 50% sparsity levels while maintaining nearly full performance. We demonstrate these ideas standard GLUE benchmarks using BERT models, setting benchmarks for future research on this topic.
Introduction
Pre-trained Teacher
Fine-tuned Teacher
Student
Non-private training
Private training
Private compression
Public corpus
Private data
The 3-pronged modern deep learning pipeline: Pre-train on public data, fine-tune on private data, and compress the model to meet the memory and latency requirements of specific applications.
Figure 1:
Since the advent of Transformer-based [67] large language models (LLMs) such BERT [13, 37], GPT families [50, 5], there has been paradigm shift in the way deep learning models are trained for natural language processing (NLP) applications. LLMs are first pre-trained extremely large and diverse publicly available datasets. The weights are then fine-tuned for specific task of interest using much smaller private dataset, which may consist of sensitive information about the users. Finally, before deploying, the models are compressed (also referred to as sparsified or distilled) to reduce the parameter count. The reason for this final step is that L.LMs such BERT and GPT-2 consist of hundreds of millions of parameters, and hence inference time and memory footprints the large to be used in many applications. As concrete example, consider a language model that is deployed for sentence completion task in text editors. In this scenario, clearly, the inference (response) time of the model needs be in orders of milliseconds be useful. Besides the
models
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

practical considerations, it is also commonly observed that large deep learning models (even the ones that are not pre-trained) consist of many redundant parameters in the network that can be removed while retaining the full performance [32, 27, 26]. Hence most modern deep learning pipelines use this three-pren pronged approach of pre-train, mune and compress as illustrated in Figure 1.
While transformer- maadderrer-- models swayed deep learning research towards making models larger to achieve better performance, several recent works have shown that over-parameterized large models also increase the risk of leaking sensitive information about their training datasets [53, 8, 9]. Over the past few years, training deep learning models guaranteeing differential privacy (DP) [17], a strong notion of data privacy, has emerged as the defacto method to mitigate such information leakage. Exciting recent works have also shown that large pre-trained models when fine-tuned via DPSGD are as good as non- non-private models for variety of NLP and image applications [75, 34, 12, 42, 44].
The main purpose of this paper is to understand how private training impacts the modern deep learning pipeline of pre-train, fine-tune, and compress. The main observation is that most widely used model compression algorithms such as Knowledge Distillation (KD) and Pruning, use private datasets to produce compressed models, hence if our goal is to deploy a differentially private compressed model, we should consider their impact on the training process. This leads the question:
What algorithms should one use to produce compressed private models and how do they impact private fine-tuning via DPSGD?
The goal of this paper is to investigate this question and propose frameworks for private model compression in the context of NLP applications. Although we investigate model compression techniques the fine-tuning stage using pre-trained models, we would like emphasize that our frameworks for private model compression are not tied to this setting. They are equally applicable to training deep learning models from scratch and to other application domains such as image classification tasks.
1.1 Our Contributions
We give framework for doing model compression using Knowledge Distillation algorithm guaranteeing differential privacy, which we call DPKD. We show that DPKD alone is not enough transfer the knowledge from large models to compressed models. This loss in the accuracy of compressed models can be mitigated by better initialization of compressed models from the weights of the large models, which itself form of knowledge transfer. We propose several zero-shot, fully private methods for initialization compressed models using weights of the large models. Our empirical evaluation of these ideas on standard GLUE benchmarks using BERT models show that DPKD approach to the model compression loses an accuracy of 5% compared to the larger models if the compressed model has half the size of the full BERT model.
To overcome the limitations of DPKD algorithm for model compression, consider framework for evolving the larger models to compressed models via private adaptation of Iterative Magnitude Pruning (DPIMP). We show that on standard GLUE benchmarks using BERT models, DPIMP framework produces compressed models whose performance is comparable to larger models at 50% unstructured sparsity levels.
As byproduct of DPIMP approach for model compression, our work also shows that pre-trained BERT models have sparse subnetworks that can be found via DPSGD that have almost matching performances of the private full model, similar to the Lottery Ticket Hypothesis for BERT models in non-private settings [20, 11].
To the best of our knowledge, no prior work have studied model compression techniques of LLMs in private settings. A problem broadly related to model compression is ensemble learning, where the goal transfer knowledge from an ensemble of teacher models a single student model [14]. This problem was studied in the private setting by [48, 49], who proposed the Private Aggregation of Teacher Ensembles (PATE) framework. private data and the student model is trained by noisy aggregation of teachers' answers. Two recent works combine PATE framework with KD algorithm for doing noisy aggregation of teachers' answers for mobile analytics and text generation problems [38, 63]. The PATE framework and ensemble learning techniques can be applied for fine-tuning (or training) deep learning models. Unfortunately, however, as previous works have shown, the performance of deep learning models trained via PATE
PATE an ensemble of teacher models trained on disjoint
2

---

are inferior to that of DPSGD for complex datasets [76]. As the performance of a fine- the-ttund large model on a sensitive dataset an upper bound on the performance of a compressed model, and we do not know how to fine-tune large models using PATE. to match the performance of DPSGD, we do not consider PATE framework for model compression.
2 Preliminaries
Recall the formal definition of differential privacy.
Definition 2.1 (Differential Privacy (DP) [17, 16]). A randomized algorithm A difffereree private if for any two neighboring datasets D and D', which differ in exactly the data pertaining to single user, and for all sets 8 of possible outputs: Pr[A(D) S] S e Pr[A(D') E +]+
We train all our models via DPSGD as the optimizer. We briefly describe the algorithm.
2.1 Training via DPSGD
To train a deep learning model with privacy, the most widely used algorithm is the DP stochastic gradient descent (DPSGD) [55, 4, 52]. DPSGD augments the standard SGD algorithm with per-example gradient clipping and Gaussian noise addition steps. These two steps serve to limit and mask the contribution of single example. At high level, the privacy analysis of DPSGD proceeds by first showing that each iteration of DPSGD differentially private for some (c, 5), then applying amplification by subsampling and composition across all the iterations. To get the tightest privacy parameters, however, one needs more sophisticated arguments such as the Moments Accountant method [1] numerical composition algorithms [23].
Problem Statement
Input our problem privacy parameters e > 0, 6 O, large model MA with the initial model parameters Ox (0). a private dataset D corresponding to a downstream task we want to solve, and compression factor y. Let |Maa denote the parameter count of MA. Our goal to produce compressed model M satisfying two constraints:
(i) M I I 7 [Mal and (ii) the final weights of model MB (denoted by (t) should be (e, 8)-differentially private with respect to dataset D. A compression algorithm can make use of MA in an arbitrary way as long the final weights of model MB (8 (t)) are differentially private with respect dataset D.
We measure the quality of compression algorithms by comparing the accuracy obtained by MB satisfying (c, 5)-DP on downstream task D to the accuracy obtained by MA satisfying (c. 5)-DP on downstream task D. This allows us to quantify how much performance one loses in private training due to model compression. Note that we not comparing against the performance of non-private models. We would like to find compression algorithms where differentially private Mg has nearly the same performance as differentially private MA.
Compressed Models via Knowledge Distillation
One of the most widely used algorithms for compressing models is knowledge distillation (KD) [28, 6, 51]. In this section, we propose a framework for implementing knowledge distillation with DP constraints and evaluate its effectiveness on standard GLUE benchmarks. We begin by briefly describing how the KD algorithm is applied for compressing models; we refer the readers to [24, 43] for more details. Adopting the naming convention from the literature, for the rest of the paper, we call large pre-trained models as teacher models and compressed smaller models as student models.
4.1 Non-Private Knowledge Distillation
Let T be teacher network with the class probabilities Pr = softmax(a7) (a.k.a. soft labels) where aT is the output of last layer before the softmax operation. Similarly, let S be student network with parameters Ws and class probabilities Ps = softmax (as). The main idea behind KD algorithm is to train S to mimic the output distribution of the teacher Pr and the true labels. The intuition is that Pf
3

---

captures the knowledge learnt by the teacher, in particular probabilities assigned by the teacher to labels that are different from the true label. Hinton al. [28] suggested use softmax-tem -emperrrtee where probability for class i of the teacher is given by
exp(2;/T)
Ps
exp(ey/TT
with logits 2; where T controls the smoothness of the output distribution. Setting higher value for the temperature parameter I' produces a softer probability distribution over classes. The same relaxation applied the output of the student network. The student trained to minimize the weighted combination of the distillation loss and the supervised training loss:
(kD(Ws) :- H(ymee Ps) + A HHTP Ps Ps)
(1)
where H. refers to the cross-entropy and is a hyperparameter.
4.2 Differentially Private Knowledge Distillation (DPKD)
A natural way generalize KD algorithm for private distillation of student model is to train the student with DPSGD to minimize the loss function given by Equation 1. However, such an algorithm fails to produce student models satisfying DP because of the term H H(P:: Ps) in Equation 1. Note that Py is a function of the entire dataset, hence clipping and adding noise alone is not enough to argue that DPSGD produces private student. A natural solution to overcome this hurdle is to first train the teacher models with DPSGD and then apply KD. We propose our DPKD framework in Algorithm 1. In this Algorithm. if the initialization of student model weights does not incur any privacy cost (e.g. random initialization or initialization using parameters from a publicly trained model), 62 would be O. Having said that, there could be student initialization strategies that are functions of the dataset D, in which case, we need to account for the privacy loss using non con-zero c2.
Algorithm 1 Differentially Private Knowledge Distillation (DPKD)
Input:
Teacher model T, student model S. private data D, privacy budget (c, 8)
Output: Student model S satisfying (e, 8)DP
1: Find an allocation (EI, , 61). (e2, 62) and (e3, 63) from the privacy budget (e, 8)
2: Train T on D with DPSGD using privacy budget of (EI, 81)
3: Initialize S (possibly privately with privacy budget 02))
4: Train S on D to minimize Eq. (I) with DPSGD using privacy budget of (e3, 83)
5: return
As the model parameters produced by DPSGD satisfy privacy guarantees, using the post-processing property of DP [18], one can show the following theorem. We omit the proof.
Theorem 4.1. The output of DPKD algorithum is differentially private with privacy parameters obtained by the adaptive composition of privacy parameters in steps 2, and 4 of Algorithm 1.
In this work we use numerical composition of privacy mechanisms as given in [23]. Our framework for DPKD raises several interesting algorithmic and hyperparameter tuning questions. The most interesting one is whether one can have DPKD algorithm where the privacy budget is not wasted on training the teacher. While this a nice theoretical question, we show in our experiments that DPKD framework is competitive with respect to teachers that are not trained with DP.
4.3 Empirical Evaluation of DPKD Algorithm
In this section, we perform experiments evaluate the effectiveness of DPKD algorithm for com- pressing models.
Teacher and Student Architectures. Our teacher models are pre-trained BERT models, which consists of 12 transformer blocks. The architecture of compressed models consist of 6 transformer blocks, which we refer to as -- --BERT.
We use Huggingface's bert -et---aasuuuseeee

---

Table 1: Comparison between the performance of 6-layer 2 -BERT student models with random initialization against full 12-layer BERT teacher models. The first row indicates the performance of fine-tuning the full teacher model. All our models have the same privacy budget 4.
Model
Initialization
Teacher
Training
MNLI
QQP
QNLI
SST-2
Avg
BERT
Pretrained
Finetune
77.8
84.7
87.8
90.5
85.2
BERT BERT
Random
Finetune
55.1
74.0
59,4
69.7
64.5
Random
BERT
DPKD
53.9
73.1
59.2
65.4
62.9
Tasks and datasets. Following prior work [77, 34, 75]. we experiment with the following set of 4 tasks from the GLUE benchmark [68]] MNLI (Multi-Genre Natural Language Inference Corpus), QQP (Quora Question Pairs), QNLI (Stanford Question Answering Dataset) and SST-2 (Stanford Sentiment Treebank).
Training and privacy parameters. We perform experiments with two sets of privacy budgets: (i) 4 with 1 and (ii) c with 6 = 1 where N is the number of samples in the given dataset. We describe the software and hardware specifications in Appendix A.1 and hyperparameter settings in Appendix A.2.
Table 2: Comparison between the performance of 6-layer -ERT student models with random initialization against full 12-layer BERT teacher models. The first row indicates the performance of fine-tuning the full teacher model. All our models have the same privacy budget c 1.
Model
Initialization
Teacher
Training
MNLI
QQP
QNLI
SST-2
Avg
BERT
Pretrained
Finetune
74.8
82.1
85.6
86.8
82.3
BERT
Random
Finetune
49.6
72.6
57.8
51
57.7
-BERT
Random
BERT
DPKD
46.4
70.4
52.9
55.4
We start with random initialization the student models see if DPKD algorithm can be effective in transferring the knowledge from the teacher. We compare the performance of our student model trained with DPKD algorithm with the performance of directly fine-tuned student via DPSGD and the performance of full teacher model fine-tuned with DPSGD. Our results are summarized in Tables and 2. The main takeaway from this experiment
There a large gap in the performance of students trained using DPKD algorithm compared to the teacher when students randomly initialized. In fact, directly fine-tuning the student model using DPSGD achieves better performance compared to DPKD algorithm.
We conclude that DPKD alone is not enough to transfer the knowledge from teacher models to compressed student models.
4.4 Better Student Models via Zero-shot Initializations
In our desire to train better-performing student models, we explore different initialization strategies. We note that initialization also form of knowledge transfer. Here we consider natural zero-shot initialization strategies:
Zero-shot (PT): Here we initialize the student model using the weights of the pre-trained teacher. particular, we simply initialize the layers of the student model with layers of BERT teacher model. We follow [52] for the choice of layers.
Zero-shot (FT): Here we initialize the student model using weights of the privately fine-tuned teacher model. As our DPKD requires the teacher to be private well, one initialize the student model using the weights of the private teacher without incurring any additional privacy cost.
We compare zero-shot initialization strategies against a fully pre-trained student model given by Huggingface's compressed BERT model DistiBBERT [52]. DiStiBBERT is trained using KD algorithm
5

---

Table 3: Comparison between the performance of 6-layer 2 -BERT student models against full 12-layer BERT teacher models and pre-trained DistilBERT, under various initialization strategies. For every student initialization method, we compare fine-tuning using DPKD algorithm vs full fine-tuning via DPSGD. All our models have the same privacy budget 4.
Model
Initialization
Teacher
Training
MNLI
QQP
QNLI
SST-2
Avg
BERT
Pretrained
Finetune
77.8
84.7
87.8
90.5
85.2
BERTT
Zero-shot (PT)
Finetune
71.7
82.4
83.2
82.7
80.0
BERT
Zero-shot (PT)
BERT
DPKD
72.8
82.6
83.0
82.7
80.3
BERT
Zero-shot (FT)
Finetune
71.3
81.8
83.4
82.2
79.7
-BERT
Zero-shot (FT)
BERT
DPKID
72.3
82.1
82.9
82.6
80.0
DistiBBERT
Pretrained
Finetune DPKD
73.0
84.3
82.8
87.7
81.9
DistilBERT
Pretrained
BERT
72.9
83.7
83.0
86.6
81.5
Table 4: Comparison between the performance of 6-layer ] -BERT student models against full 12-layer BERT teacher models and pre-trained DistilBERT, under various initialization strategies. For every student initialization method, we compare fin-ttniig using DPKD algorithm vs full fine-tuning
via DPSGD. All our models have the same privacy budget
Model
Initialization
Teacher
Training
MNLI
QQP
QNLI
SST-2
Avg
BERT
Pretrained
Finetune
74.8
82.1
85.6
86.8
82.3
-BERT
Zero-shot (PT)
Finetune DPKD
66.9
78.3
81.0
79.6
76.4
BERT
7ero-shot (PT)
BERT
67.5
78.4
80.1
78.5
76.1
BERT
Zero-shot (FT)
Finetune DPKD
66.9
77.6
80.1
79.2
75.9
BERT
Zero-shot (FT)
BERT
68.3
77.0
80.3
80.0
76.4
DistiBBER
Pretrained
Finetune 68.4
82.0
81.0
86.0
79.3
DistilBERT
Pretrained
BERT
DPKD
68.1
80.5
80.2
85.1
78.5
with the full BERT model on public data at the pre-training stage. We note that DistiBBERT has transformer blocks and hence has the same architecture as -ERT.. We emphasize that pre-training student model is computationally expensive, and the main aim of this paper to develop algorithms where one can transfer knowledge from the teacher students models without resorting to pre- training the student from scratch. However, DistilBERT serves as the gold standard to compare our zero-shot initialization strategies and also sets the benchmark for comparing ideas in Section 5.
Our results are summarized Tables and For every student initialization method, we also compare fine-tuning using DPKD algorithm vs simply fine-tuning the student via DPSGD. The main takeaways from these experiments are:
Results
Zero-shot initialization strategies give large performance improvements to student models and come 2-3% of the performance achieved by pre-trained DistilBERT: Somewhat surprisingly, there not much difference between our two zero-shot initialization strategies.
Both pre-trained DistiBBERT and student models with zero-shot initialization strategies fall short of matching the performance of teacher models.
Finally, broadly speaking, DPKD algorithm does not give a significant performance boost the student models compared to directly fine-tuning them.
4.5 Better Models are Better Teachers?
Given the results the previous section, one may wonder if better teachers can help improving the performance of the students. In this regard, consider two notions of better teacher: (1) A

---

Table 5: Comparison between the performance of 6-layer -BERTT student models under different teacher models in distillation against full 12-layer BERT teacher models and pre-trained DistilIBERT. All our models have the same privacy budget 4.
Model
Initialization
Teacher
Training
MNLI
QQP
QNLI
SST-2
Avg
BERT
Pretrained
Finetune
77.8
84.7
87.8
90.5
85.2
Student
Pretrained
Finctunc
73.0
84.3
82.8
87.7
81.9
-BERT
Zero-shot (PT)
BERT
DPKD
72.8
82.6
83.0
82.7
80.3
BERT
Zero-shot (PT)
BERTLARGE BERT w/out DP
DPKD
72.4
81.1
83.1
81.5
79.5
BERT
Zero-shot (PT)
DPKD
74.2
82.9
84.5
83.0
81.1
Table 6: Comparison between the performance of 6-layer 1- -BERT student models under different teacher models distillation against full 12-layer BERT teacher models and pre-trained EER.. All our models have the same privacy budget 1.
Model
Initialization
Teacher
Training
MNLI
QQP
QNLI
SST-2
Avg
BERT
Pretrained
Finetune
74.8
82.1
85.6
86.8
82.3
Student
Pretrained
Finetune
68.4
82.0
81.0
86.0
79.3
BERT
Zero-shot (PT)
BERT
DPKD
67.5
78.4
80.1
78.5
76.1
BERT
Zero-shot (PT)
BERTLARGE BERT w/out DP
DPKD
67.6
78.0
80.1
78.0
75.9
-BERT
Zero-shot (PT)
DPKD
70.6
79.0
81.5
79.8
77.7
larger teacher model in Step 2 of Algorithm (2) teacher model that not DP-trained. We note that the final student model in the second case not DP; however, these experiments allow to quantify how much performance gains one could have if one were to come up with a new framework for implementing KD in DP setting without requiring the teacher to be trained with DP.
Our results are summarized in Tables 5 and 6. These experiments show that DPKD does not benefit significantly from having better teacher models. Furthermore, proposed framework of implementing KD in DP framework by training both the teacher and student models via DP is only within 0.8% of the doing KD with fine-tuned teacher without DP.
5 Evolving Teacher to Student Models via Pruning
As the previous section presents, the Knowledge Distillation approach to model compression has two main drawbacks in the private world:
Drop in accuracy: There is considerable drop in the accuracy between the teacher and the student models.
Good initialization of students is crucial: The best performance is obtained by students who already have a good initialization; our experiments, pre-trained DistilBERT mostly achieved the best student performance.
Finding good initialization can be challenging in practice. Often, the student architectures are chosen to suit the hardware and latency requirements of the application for which the model is being deployed, using neural architecture search [19]. Hence, finding a good initialization for every student architecture via pre-training can be expensive and most cases impossible. Our zero-shot initialization strategies alleviate this problem to a certain degree, yet fall short of closing the gap between the teacher and the student performances. Moreover, DPKD requires that (i) the teacher is trained with DPSGD and (ii) the student is distilled via DPSGD. This two-step approach creates additional overheads in terms of training. Given these limitations, it natural ask: Can we evolve the teacher to student model while fine-tuning with DPSGD? In this section, we explore an answer this via structured and unstructured pruning with privacy, which allows to obtain student models that are as good as the teacher models.
7

---

5.1 Model Compression via Pruning
Pruning algorithms are a broad class of model compression techniques where one drops the param- eters from model during or after the training process. Many works have shown that eliminating unnecessary parameters of neural networks via pruning can lead to sparser and compressed models that have shorter inference times without loss in performance [32, 27, 26]. For example, in magnitude pruning, one of the most widely used pruning techniques, we prune a fraction of parameters with the lowest magnitude. However, there are several pruning strategies, and we refer the readers [35] for more details and references.
Pruning can be implemented in both structured and unstructured ways. In structured pruning, all the pruned weights belong to a single building block the model. For example, 6-layer 2 3-BERT can be obtained by pruning 6 layers from the full BERT model, which consists of 12 transformer blocks. On the other hand, in unstructured pruning, pruned weights may be spread across all the layers of the network. In unstructured pruning, it is possible to obtain 50% sparse student model while still having all the 12 layers of BERT. Depending on the hardware architectures, inference latency between models with structured and unstructured sparsity could be quite different. However, in this section, we use sparsity as the main measure of model compression, which is also well accepted in the community [35, 29].
5.2 Iterative Magnitude Pruning (IMP)
Private pruning techniques we study in this section are based on the Iterative Magnitude Pruning (IMP) method, which is specific pruning technique proposed in recent work on Lottery Ticket Hypothesis [20]. The idea behind IMP is rather simple: As we train a deep learning model, after every N iterations we prune an a% of the weights with the lowest magnitude. We repeat this process until we achieve the desired sparsity. Here, both N and a are hyperparameters that need to be tuned. For example, to achieve 50% sparsity, one can perform 5 iterations where after every N iterations additional 10% of the weights with the least magnitudes are dropped. As specified IMP produces unstructured sparsity. However, consider simple modification of the IMP algorithm to produce structured sparsity as well.
Algorithm/ 2 Structured DPIMP
Input: Teacher model T, number of layers to prune L,, hyperparams N and M Output: Private student model S with layers pruned from T
1: Set S :
2: for j = 1 to L do
3: Fine-tune S for N iterations with DPSGD
4: Set Wmin consisting of a% of the remaining model weights with the least magnitude
5: Set W; as the weights of layer i
6: Drop the layer i* from S satisfying i" == arg max; { n }
7: end for
8: Fine-tune S for M more iterations with DPSGD
9: return S
5.3 Structured DPIMP
We first attempt to obtain a student model from the teacher model via a structured IMP technique, using the following modification: During fine-tuning the teacher model with DPSGD, we progressively drop an appropriately chosen transformer block from the teacher model at the end of every N iterations. We repeat this process until we obtain the student model with the required sparsity. The layer to drop is chosen using the following heuristic: Let a > 0 be hyperparameter. At the end of N iterations, fix bottom (by magnitude) a% of all model weights, and denote it by Wmin. For the 5t transformer block, let W; denote the set of model weights belonging to that block. Among all the transformers blocks we find the block i" that has the highest number of weights from the set Wmin; Formally, i* == arg max; (W; n W Wmin). and we prune the transformer layer i". We present this in Algorithm 2. We note that the algorithm satisfies (e, 5D-DP after the pruning steps due to the post-processing property of differential privacy.
8

---

Table Comparing performance of 6-layer 2 -BERT student model produced by structured DPIMP with 12-layer BERT teacher model and pre-trained DisTilIBERT. The results are shown with two privacy budgets e = and = 4.
MNLI
QQP
QNLI
SST-2
Avg
Model
c= 1
6
c=
1
4
c=
c=
4
c=
c
4
<=
BERT
74.8
77.8
82.1
84.7
85.6
87.8
86.8
90.5
82.3
85.2
DistiBERT
68.4
73.0
82.0
84.3
81.0
82.8
86.0
87.7
79.3
81.9
4-BERT
68.7
72.9
80.7
83.1
80.9
82.5
83.3
85.7
78.4
81.0
Empirical Evaluation We evaluate our structured pruning algorithm with the same setup described Appendix A.2. We split the privacy budget equally
in Section 4.3, and provide the hyperparameters' among all the iterations of the algorithm. Our goal is to produce student model which has 2 as many layers as the full BERT model. Table 7 shows the results for this setting where we compare structured DPIMP private fine-tuning of the pre-trained DistilBERT and the full BERT model. The main takeaway from this experiment is:
DP structured pruning algorithm produces student model that has performance comparable to that of DistilBERT. Further, it avoids the pre-training cost associated with DistilBERT.
5.4 Unstructured DPIMP
The structured pruning produces student models that are as good as DistilBERT; our next target is to produce student models that are as good as the full teacher BERT model. Towards that, we explore unstructured pruning techniques, in particular differentially private version of the IMP algorithm. As we fine-tune the deep learning models using DPSGD, after every N iterations we prune increments of a% of the weights with lowest magnitude. The remaining weights are then reset to the original pre-trained initialization. (We do this step to establish connections to Lottery Ticket Hypothesis, see below.) We repeat this process until we achieve the desired sparsity. We note that the initialization of the final student model's non-zero parameters at the end of the pruning step are still non-private, even though weights correspond to pre-trained model weights. This due to fact that pruned parameters are found during the fine-tuning process, which makes use of the private dataset. Therefore. the whole process must be performed with DPSGD to produce a private student model. Finally, perform additional fine-tuning of the pruned student model by using DPSGD for M more iterations. We call the private variation this procedure Unstructured DPIMP. We formally present this process in Algorithm 3.
Algorithm 3 Unstructured DPIMP
Input: Model T and the sparsity level S%, hyperparams a, N and M Output: Private Student model S satisfying the sparsity requirements
Set T :=
2: for == to [SSaa]] do
3: Fine-tune T' for N iterations with DPSGD
4: Prune (ax x i)% of weights with the lowest magnitude from T''
5: Reset the non-zero weights of T' the original T
6: end for
7: Fine-tune T for M iterations with DPSGD
8: return 8=
Empirical Evaluation We refer the student model produced by Algorithm 3 as SparseBERT and indicate the sparsity level in brackets. allocate the privacy budget equally among all iterations of Algorithm 3. Table 8 summarizes our experiments on unstructured pruning via DPIMP. We compare the performance of our student model to pre-trained DistilBERT (whose parameter count is the same our final student model) and the full-sized BERT teacher model. The main takeaways are:
DPIMP produces a student model that has better performance compared to DistilBERT.

---

Table Comparing performance of SparseBERT student model produced by unstructured DPIMP with 12-layer BERT teacher and pre-trained DistilIBERT. The results are shown with two privacy budgets and
MNLI
QQP
QNLI
SST-2
Avg
Model
1
c
4
<=4
1
6 4
4
6
1
BERT
74.8
77.8
82.1
84.7
85.6
87.8
86.8
90.5
82.3
85.2
DistiBBRT
68.4
73.0
82.0
84.3
81.0
82.8
86.0
87.7
79.3
81.9
SparseEERT (50%)
72.9
76.8
81.1
84.0
82.2
85.7
83.7
87.6
80.0
83.5
The average performance of DPIMP is within 2% of the full BERT model. We conclude that unstructured pruning techniques are more effective closing the gap between the teacher and the student models in the private world. Moreover, pruning methods are computationally cheaper as student models do not require pre- pre-training on public data.
DPIMP algorithm as described in Algorithm 3 the end of line 6 finds sparse subnetworks in the pre-trained BERT model that can be fine-tuned to obtain nearly matching teacher performance. This similar Lottery Ticket Hypothesis work for BERT networks [11], although in the private world performance of the student network is not matching that of the teacher.
6 Conclusions and Future Directions
In this paper we initiated the study differentially private model compression via Knowledge Distillation and Pruning, and gave frameworks for implementing both. Our work shows that one can obtain student models whose performance comes within 2% of the full teacher models while having 50% fewer parameters, which can lead to significant reduction in memory and improve inference time. We believe that our work takes the first step in the intersection of private training and model compression techniques, and opens whole in direction with plenty of interesting and important problems. We highlight some of them here.
Better Algorithms For Model Compression: While we gave natural DP adaptations of KD and pruning algorithms, we believe that there are plenty of new techniques to explore.
Better Accounting: In all our results, chose very simple strategies for allocating privacy budget across various steps of training and pruning. Theoretical innovations the form of better accounting can further improve the performance of student models.
Lottery Tickets for DP-training? Both in our experiments on KD and in pruning, we see that models that have "good initialization" lead to dramatic improvements performance. Moreover our DPIMP algorithm finds sparse subnetworks in pretrained BERT models at 50% sparsity that can be fine-tuned to obtain nearly matching teacher performances. This raises an intriguing question akin to Lottery Ticket Hypothesis: Are there good initialization of models where dynamics of DPSGD is similar to SGD? We believe that this is an exciting research direction both from a theoretical point of view and also its implications to practical private training.
References
[1] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM Conference Computer and Communications Security, CCS '6, pages 308 318, New York, NY, USA, 2016. ACM. 2.1, B
[2] Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Manurangsi. Large-scale differentially private BERT. arXiv preprint arXiv:2108.01624 2021. B
[3] Eugene Bagdasaryan, Omid Poursaeed, and Vitaly Shmatikov. Differential Privacy Has Dis- parate Impact on Model Accuracy. Curran Associates Inc., Red Hook, NY, USA, 2019.
[4] Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In Proceedings of the 55th Annual IEEE Symposium
10

---

on Foundations of Computer Science, FOCS '14, pages Washington, DC, USA, 2014.
IEEE Computer Society. 2.1, B
[5] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ila Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv preprint arXiv:2005. /4165, 2020.
[6] Bucilua, R Caruana, and Niculescu-Mizii Model compression, in proceedings of the 12 th acm sigkdd international conference on knowledge discovery and data mining. New York, NY, USA, 2006. 4
[7] Qingqing Cao, Harsh Trivedi, Aruna Balasubramanian, and Niranjan Balasubramanian. De- Former: Decomposing pre-trained transformers for faster question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4487-4497,
2020. B
[8] Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization neural networks. 28th USENIX Security Symposium, USENIX Security 19, pages 267-284. USENIX Association, 2019. 1, B
[9] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song. Ulfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models, 2021.
[10] Kamalika Chaudhuri and Daniel Hsu. Sample complexity bounds for differentially private learning. In Proceedings of the 24th Annual Conference on Learning Theory, COLT '11, pages 155- 186, 2011. B
[11] Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. Advances in neural information processing systems, 33:15834-15846, 2020. 5.4, A.2, B
[12] Soham De, Leonard Berrada, Jamie Hayes, Samuel L Smith, and Borja Balle. Unlock- ing high-accuracy differentially private image classification through scale. arXiv preprini arXiv:2204. 13650, 2022.
[13] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810:0880000005 2018.1
Ming-Wei
[14] Thomas G. Dietterich. Ensemble methods machine learning. In MULTIPLE CLASSIFIER SYSTEMS, LBCS-1857, pages 1-15. Springer, 2000. 1.1
[15] Lifang Ding and Yujuu Yang. SDSK2BERT: Explore the specific depth with specific knowledge to compress BERT. In 2020 IEEE International Conference on Knowledge Graph (ICKG), page 420-425 Aug 2020. B
[16] Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor. Our data, ourselves: Privacy via distributed noise generation. In Proceedings of the 24th Annual International Conference the Theory and Applications Cryptographic Techniques, EUROCRYPT '06, pages 486-503, Berlin, Heidelberg, 2006. Springer. 2.1
[17] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensi- tivity in private data analysis. In Proceedings of the 3rd Conference on Theory of Cryptography, TCC '06, pages 265-284, Berlin, Heidelberg, 2006. Springer. 1, 2.1
[18) Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Founda- tions and Trends@ in Theoretical Computer Science, 9(3-4)2211-44 2014. 4.2
11

---

[19] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey, The Journal of Machine Learning Research, 20x1):1977-2017, 2019.
[20]
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: 03635, 2018. 1.2
[21] Prakhar Ganesh, Yao Chen, Xin Lou, Mohammad Ali Khan, Yin Yang, Hassan Sajjad, Preslav Nakov, Deming Chen, and Marianne Winslett. Compressing large-scale transformer-based models: A case study on bert. Transactions of the Association for Computational Linguistics, 9:1061-1080, 2021.
[22] Antonio Ginart, Laurens van der Maaten, James Zou, and Chuan Guo. Submix: Practical private prediction for large-scale language models. arXiv preprint arXiv:2201.0097) 2022. B
[23] Sivakanth Gopi, Yin Tat Lee, and Lukas Wutschitz. Numerical composition of differential privacy. arXiv preprint wiXiv22066002848 2021. 2.1, 4.2
[24] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey, International Journal of Computer Vision, 17966):189-1819, 2021.
[25] Fu-Ming Guo, Sijia Liu, Finlay Mungall, Xue Lin, and Yanzhi Wang. Reweighted proximal pruning for large-scall scage language representation. arXiv preprint arXiv:990.112486, 2019. B
[26] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015. 5.1
[27] Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems, 5, 1992. 5.1.
[28] Geoffrey Hinton, Oriol Vinyals. Jeff Dean, et al. Distilling the knowledge in neural network. arXiv preprint arXiv: 02531, 2(7), 2015. 4, 4.1
[29] Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22(241):1-124, 2021. 5.1, B
[30] Shlomo Hoory, Amir Feder, Avichai Tendler, Alon Cohen, Sofia Erell, Itay Laish, Hootan Nakhost, Uri Stemmer, Ayelet Benjamini, Avinatan Hassidim, and Yossi Matias. Learning and evaluating differentially private pre-trained language model. In Proceedings the Third Workshop on Privacy in Natural Language Processing, PrivateNLP '21, pages 21-29, 2021. B
[31] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. TinyBERT: Distilling BERT for natural language understanding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages
4163-4174, 2020. B
[32] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989. 1,5.1
[33] Bingbing Li, Zhenglun Kong, Tianyun Zhang, Ji Li, Zhengang Li, Hang Liu, and Caiwen Ding. Efficient transformer-basee based large scale language representations using structured pruning. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3187-3199, 2020. B
friendly block
[34] Xuechen Li, Florian Tramèr, Percy Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners. In Proceedings of the 10th International Conference on Learning Representations, ICLR '22 2022. 1, 4.3, A.2,
[35] Tailin Liang, John Glossner, Lei Wang, Shaobo Shi, and Xiaotong Zhang. Pruning and guantization for deep neural network acceleration: A survey, arXiv preprint a1X1v.2101.09671,
2021. 5.1
[36] Linqing Liu, Huan Wang, Jimmy Lin, Richard Socher, and Caiming Xiong. MKD: A multi-task knowledge distillation approach for pretrained language models. arXiv preprint arXiv:1911.03888 2019. B
12

---

[37] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer. and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint urXiv:/907. /907. 1692, 2019.
[38] Lingjuan Lyu and Chi-Hua Chen. Differentially private knowledge distillation for mobile analyties. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1809-1812, 2020. 1.1
[39] Jimit Majmudar, Christophe Dupuy, Charith Peris, Sami Smaili, Rahul Gupta, and Richard Zemel. Differentially private decoding in large language models. arXiv preprint arXiv:2205. 13621, 2022. B
[40] Yihuan Mao, Yujing Wang, Chufan Wu, Chen Zhang. Yang Wang, Quanlu Zhang, Yaming Yang, Yunhai Tong, and Jing Bai. LadaBERT: Lightweight adaptation of BERT through hybrid model compression. In Proceedings of the 28th International Conference Computational Linguistics, page 3225-3234, 2020. B
[41] H Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. In Proceedings of the 6th International Conference on Learning Representations, ICLR '1, 2018. B
[42] Harsh Mehta, Abhradeep Thakurta, Alexey Kurakin, and Ashok Cutkosky. Large scale transfer learning for differentially private image classification. arXiv preprint arXiv:2203.00 203220033. 2022.
[43] Gaurav Menghani. Efficient deep learning: A survey on making deep learning models smaller, faster, and better. arXiv preprint arXiv:2106.0800 2021. 4
[44] Patemehsadat Mireshghallah, Mohammadkazem Taram, Praneeth Vepakomma, Abhishek Singh, Ramesh Raskar, and Hadi Esmaeilzadeh. Privacy in deep learning: A survey. arXiv preprint arXiv:2004. 12254, 2020.
[45] Subhabrata Mukherjee and Ahmed H. Awadallah. Distilling BERT into simple neural networks with unlabeled transfer data. arXiv preprint arXiv: 2019. B
[46] Subhabrata Mukherjee and Ahmed H. Awadallah. XtremeDistil: Multi-stage stage distillation for massive multilingual models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, page
2020. B
[47] Matan Ben Noach and Yoav Goldberg. Compressing pre-trained language models by matrix decomposition. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 884-889, 2020. B
[48] Nicolas Papernot, Martín Abadi, Ulfar Erlingsson, Ian Goodfellow, and Kunal Talwar. Semi- supervised knowledge transfer for deep learning from private training data. In Proceedings of the 5th International Conference on Learning Representations, ICLR '17, 2017. 1.1
[49] Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Úlfar Erlingsson. Scalable private learning with PATE. In Proceedings of the 6th International Conference on Learning Representations, ICLR 18, 2018. 1.1
[50] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8):9, 2019.
[51] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv: /4f.:::4/266500 2014.
[52] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 108, 2019. 2.1, 4.4, B
[53] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEFE symposium on security and privacy (SP), pages 3-18. IEEE, 2017.
13

---

[54] Kaitao Song, Hao Sun, Xu Tan, Tao Qin. Jianfeng Lu, Hongzhi Liu, and Tie- Yan Liu. Light- PAFF: A two-stage distillation framework for pre-training and fine-tuning. arXiv preprint arXiv:2004. /2817, 2020. B
[55] Shuang Song, Kamalika Chaudhuri, and Anand D Sarwate. Stochastic gradient descent with differentially private updates. In Proceedings of the 2013 IEEE Global Conference on Signal and Information Processing, GlobalSIP '13, pages 445, Washington, DC, USA, 2013.
IEEE Computer Society. 2.1
[56] Pierre Stock, Angela Fan, Benjamin Graham, Edouard Grave, Rémi Gribonval, Herve Jegou, and Armand Joulin. Training with quantization noise for extreme model compression. In International Conference on Learning Representations, 2020. B
[57] Lichao Sun and Lingjuan Lyu. Federated model distillation with noise-free differential privacy. arXiv preprint a0Xiv:2009.05537, 2020. B
[58] Siqi Sun, Yu Cheng. Zhe Gan, and Jingjing Liu. Patient knowledge distillation for BERT model compression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4314-4323, 2019. B
[59] Thierry Tambe, Coleman Hooper, Lillia Pentecost, Tianyu Jia, En-Yu Yang. Marco Donato, Victor Sanh, Paul Whatmough, Alexander M. Rush, David Brooks, and Gu-Yeon Wei. Edgebert: Sentence-level energy optimizations for latency-aware multi-task nlp inference. In MICRO-54; 54th Annual IEEE/ACM International Symposium on Microarchitecture, MICRO '21, page 830-844, New York, NY, USA, 2021. Association for Computing Machinery.
[60] Thierry Tambe, En-Yu Yang, Zishen Wan, Yuntian Deng, Vijay Janapa Reddi, Alexander Rush, David Brooks, and Gu- Yeon Wei. Algorithn-hardware co-design of adaptive floating-point encodings for resilient deep learning inference. In 2020 57th ACM/IEEE Design Automation Conference (DAC), pages 1-6. IEEE, 2020. B
[61] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2820-2828, 2019. B
[62] Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. Distilling task- specific knowledge from BERT into simple neural networks. arXiv preprint 1r03. 12136, 2019. B
[63] Zhiliang Tian, Yingxiu Zhao, Ziyue Huang, Yu-Xiang Wang, Nevin Zhang, and He He. Seqpate: Differentially private text generation via knowledge distillation. 2021. 1.1
[64] Florian Tramèr and Dan Bonch. Differentially private learning needs better features (or much more data). In Proceedings of the 9th International Conference on Learning Representations, ICLR 22,, 2021. B
[65] Iulia Turc, Ming-War Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better: The impact of student initialization knowledge distillation. arXiv preprint arXiv:1908.0890 13, 2019. B
[66] Archit Uniyal, Rakshit Naidu, Sasikanth Kotti, Sahib Singh, Patrik Joslin Kenfack, Fatemehsadat Mireshghallah, and Andrew Trask. DP-SGD PATE: Which has less disparate impact on model accuracy? arXiv preprint arXiv:2106. 12576, 2021. B
[67] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 3 International Conference on Neural Information Processing Systems, pages 6000-6010, 2017.
[68] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In International Conference Learning Representations, 2018. 4.3
14

---

[69] Ji Wang, Weidong Bao, Lichao Sun, Xiaomin Zhu, Bokai Cao, and Philip S. Yu. Private model compression via knowledge distillation. Proceedings of the Thirty-Thire AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances Artificial Intelligence. AAAI Press, 2019. B
[70] Moshe Wasserblat, Oren Pereg, and Peter Izsak. Exploring the boundaries of low-resource BERT distillation. In Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing, page 35-40, 2020. B
[71] Xing Wu, Yibing Liu, Xiangyang Zhou, and Dianhai Yu. Distilling knowledge from pre-trained language models via text smoothing. arXiv preprint arXiv:2005. 008,0 2020.
[72] Jin Xu, Xu Tan, Renqian Luo, Kaitao Song, Jian Li, Tao Qin, and Tie- Yan Liu. Nas-bert: Task- agnostic and adaptive-size bert compression with neural architecture search. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery Data Mining, KDD '21, page 1933-1943, New York, NY, USA, 2021. Association for Computing Machinery. B
[73] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R. Salakhutdinov, and Quoe V. Le. XLNet: Generalized autoregressive pretraining for language understanding. Advances in Neural Information Processing Systems, 32:5753-5763, 2019. B
[74] Yichun Yin, Cheng Chen, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. Autotinybert: Automatic hyper-paramete optimization for efficient pre-trained language models. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5146-5157, 2021. B
[75] Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang. Differentially private fine-tuning of language models. In International Conference on Learning Representations, ICLR '22, 2022. 1, 1,4.3, A.2,
[76] Da Yu, Huishuai Zhang, Wei Chen, and Tie- Yan Liu. Do not let privacy overbill utility: Gradient embedding perturbation for private learning. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net,
2021. 1.1
[77] Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, and Tie- Yan Liu. Large scale private learning via low-rank reparametrization. In Proceedings of the 38th International Conference on Machine Learning, ICML '21. JMLR. Inc., 2021. 4.3
[78] Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8BERT: BERT. In Proceedings the 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing, 2019. B
Quantized Sbit
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes], see Sections 1.1 and 6
(c) Did you discuss any potential negative societal impacts of your work? [N/A], we are studying and evaluating differentially private model compression.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A], See Section 4. The paper is empirical, we only recite existing theoretical work that we build our experiments on.
15

---

(b) Did you include complete proofs of all theoretical results? [N/A] We are not introduc- ing new theoretic notions.
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URLY? [Yes], we have included the dataset names (publicly available) and all of our hyperparameters and package versions. See Sections 4.3, A.I, 1, and A.2.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they chosen)? [Yes], See Sections 4.3, A.I, and A.2.
(c) Did you report error bars (e-g.. with respect the random seed after running experi- ments multiple times)? [No].
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes], see Section A.1.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you'l re using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "opxcBLLLhC3", "reviewer": "Reviewer_yzKR", "summary": "- The authors focused on compressing LLM in a differentially private way. Previous works mostly focused on differentially private fine-tuning but not the compression parts.\n- The proposed method achieves very similar performance with 50% sparse model compression and guaranteeing differential privacy.\n- The authors showed the limitation of straightforward ways of applying DPSGD to Knowledge Distillation and proposed the improved version of it to reduce the accuracy losses.", "questions": "1. Compression rate\n- In the paper, the authors only show the experimental results with 50% sparsity.\n- However, in many applications, we need much higher compression rates to increase the latency and reduce the costs.\n- It would be great if the authors can provide more results with a diverse range of compression rates.\n\n2. Epsilon value\n- Based on the differential privacy definition (Section 2), epsilon determines the maximum differences between two probabilities with and without a single sample.\n- If we select the epsilon value as 4 (which is < 4.25), the difference ratio is exp(4) = 55.\n- In other words, the maximum probability differences would be significantly large.\n- In this paper, most results are based on the epsilon value < 4.25. But I think this would be a too loose threshold. \n- It would be good if the authors provide tighter privacy thresholds and the corresponding performances.\n\n3. DistBERT\n- Even though DistBERT is computationally expensive, we only need one time for training the DistBERT (because it is KD on public data). \n- Then, we can use the DistBERT for various downstream tasks.\n- In that case, I think the computational complexity of the DistBERT does not matter a lot.\n- At the end, the objective is to achieve the differentially private compressed language models and the objectives seem the same between DistBERT and the proposed method.\n- In that point of view, the advantages of the proposed method in comparison to DistBERT are limited.\n\n4. Structured DPIMP vs Unstructured DPIMP\n- It would be good to show the quantitative inference latency comparison between these two methods.\n- Also, directly comparing between unstructured DPIMP and DistBERT is not the fair way.\n- Actually, based on Table 4, DistBERT seems consistently better than Structured DPIMP. ", "limitations": "- As the authors said in the introduction, the proposed method is not limited to the NLP.\n- In that case, it would be good to show how well those compression methods (with DP) can be applicable to other domains in the appendix.\n- Also, the authors do not consider the PATE framework for model compression. However, in some applications, PATE works better than DP-SGD. To show the generalization of the proposed method, it would be better to discuss and consider PATE as well.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- The authors tackled an important but under-explored problem: DP for model compression in LLM.\n- The paper is well-written and easy to follow.\n\nWeakness:\n- The experiments are limited. It would be good if the authors provide more diverse experiments with varying compression rates.\n- Epsilon value is set to be too high. With epsilon < 4.25, I am not sure whether the achieved model is private enough.\n- The advantages from DistBERT is not convincing. Comparing with unstructured DPIMP is not fair.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "jUYHYJk5N9L", "reviewer": "Reviewer_D2Eo", "summary": "This paper focus on a important scenario to consider differential privacy and model compression together. It considers KD and (structured and unstructed) pruning. The paper is generally well-written.", "questions": "In the abstract, the authors state \"we initiate the study of differentially private model compression and propose frameworks for achieving 50% sparsity levels while **maintaining nearly full performance**\". From Tab. 2/3/4/5, it seems it does not achieve as well as the authors claim.  A relatively better result is achieved by unstructured DPIMP, which has 83.5 vs. 85.2 (raw BERT) in AVG (there is still a tiny gap); however, unstructured pruning with 50% sparsity usually has a limited speed-up during inferencing. The above statement seems slightly overclaimed.\n\nThe paper seems to report relatively weak results in GLUE benchmark, not only for the proposed method but also for existing work. By checking other papers (e.g. DistilBERT https://arxiv.org/pdf/1910.01108.pdf or even a stronger one  TinyBERT  https://arxiv.org/abs/1909.10351), we could observe that the reported results for BERT and DistilBERT are lower than original papers. It is unclear whether the results are evaluated in DEV or TEST set. Could you please provide more details? Frankly speaking, with half the size of BERT-base parameters, such achieved performance might not be competitive with some models without pretraining (which might be much faster).", "limitations": "none", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- it is interesting to consider DP and compression together.\n- consider both KD and (structured and unstructed)  pruning.\n- the paper is well-written\n\nWeaknesses:\n- some statements are slightly over-claimed\n- the methods seems not that novel. For example, DPKD is like a pipline to bring many existed methods together.\n- Sec. 3.4 and 3.5 do not introduce anything beneficial, but something that has nearly idential results -- strategies to make better students/teachers.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QePr2azemVc", "reviewer": "Reviewer_BeSb", "summary": "A commonly used technique in practical NLP systems is that of model distillation (or compression or sparsification). This technique converts a Large Language Model (LLM) into a more lightweight/small model so as to make the prediction faster and memory efficient.\n\nOn the other hand, Differentially Private (DP) training has recently started to become a standard notion of safety against various adversarial model attacks. \n\nThis work lies at the intersection of the above two concepts, in that it introduces methods to obtain compressed student models, from large teacher models, that have differential privacy guarantees. To that end, the paper investigates the following approaches:\n\n1) Obtaining student models via a distillation loss function. Here DP is introduced possibly in the fine-tuning steps of teacher and students models. \n\n2) Transforming teacher models into student models via Iterative Magnitude Pruning (IMP). Here DP is introduced in the fine-tuning steps.", "questions": "1) A main motivation for introducing DPKD is mentioned in lines 118-121. While that explanation with regards to the KD loss function makes intuitive sense, is it possible to formalize it properly? The text in those lines is used to justify the introduction of DPKD but it is quite hand-wavy and doesn't cite any previous work for support as well.\n\n2) In Algorithm 1, line 3, what does it mean to initialize the student model with a privacy budget? How can just an initialization step have a privacy guarantee? This is also not clear from the experiments as there only the teacher and student model training steps involve DP.\n\n3) What is the motivation for choosing privacy parameters, epsilon and delta, as shown in line 144? Calibration of these quantities is a problem in current literature as we don't know which values are good, so the current choice seems arbitrary. \n\n4) As mentioned in the weaknesses, the DP guarantees for DPIMP methods are missing. Can we provide epsilon, delta values for the models resulting from those algorithms?\n\n5) Related to the PATE works, there is also very recent work on introducing DP to LLMs at the prediction stage (see recent works such as https://arxiv.org/abs/2201.00971 and https://arxiv.org/abs/2205.13621). It will be helpful to add a few lines regarding these to improve the quality of presentation. \n\nMinor typos:\n\n1) Phrase \"for initialization compressed models\" in line 54.\n\n2) \"show\" should be replaced with \"shows\" in line 55.\n\n3) Phrase \"opens a whole in direction\" in line 295.", "limitations": "Yes.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1) The setup and motivation is practically quite relevant. This is because both distillation and differential privacy are critical in large-scale NLP systems.\n\n2) It studies two distinct compression methods: one based on the distillation loss function and the other based on weight pruning. \n\nWeaknesses:\n\n1) The DP guarantees for pruning based methods are entirely missing. For instance, the output of algorithm 2 is said to be a \"private student model\" but one doesn't know what the epsilon-privacy guarantee is. This makes the algorithm practically useless since the user doesn't have a calibration of privacy gains. \n\n2) The experimental results are limited only to BERT and 1/2-BERT. It would be interesting to see if their results generalize to other architectures. \n\n3) The technical contribution to introduce DP to model compression is a fairly straightforward application of existing tools and doesn't involve significant novelty.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "30wS-tc1Ic6", "reviewer": "Reviewer_pc64", "summary": "This paper explores the links between differential privacy and model compression. Authors propose a framework for training NLP models with high utility by leveraging DP-SGD and knowledge distillation.\nThis is an experimental paper.", "questions": "A number of questions arose during my evaluation of this work. I would like the authors to either point me to the section where I can find the necessary details or to provide an answer outside of the manuscript space:\n- What dataset does a teacher control? There was a discussion about PATE needing a disjoint dataset, which may be unrealistic, but I did not find any mention of what are the data assumptions on the teacher model in this work? Moreover, could the authors elaborate why is a disjoint setup considered unrealistic in this setting?\n- I am not entirely sure I understand section 2.1: what is the meaning of the 'sophisticate argument(s)'? It strikes me as if the authors included a list of words associated with DP without really explaining them or even linking them together.\n- It was unclear to me why in section 3.2 DP-SGD was insufficient? I suspect this links to my qn on data assumptions for the teacher mode and I am unsure whether a teacher model is required to be private or not, so would you kindly clarify?\n- In the experiments an epsilon of < 4.25 was chosen: what is the motivation behind it? Why this specific value and how does it relate to prior literature? If I overlooked it, then please point me to a section/reference.", "limitations": "In addition to the limitations outlined by authors, my comments are largely explained above and include: Similar prior work, constraints on the data of the teacher model etc.\nWhile overall, the work shows some interesting results and is relatively easy to read, I cannot recommend acceptance until points above are addressed by the authors (particularly wrt novelty and comparison with prior works).", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The work is rather easy to follow, with clear motivations of why this technique should be explored, how the scientific community can benefit from private compressed models etc. The method that authors propose seems sound and can prove useful for a broad scientific community. The way the manuscript is structured really aids its interpretability: the motivation is followed by a naive implementation, followed by the comments on why such method does not perform well, followed by another iteration with limitations addressed etc. This in my view makes the paper much friendlier to larger audience.\nAuthors provide a number of evaluations of their technique in comparison to its non-private counterpart. What I find really commendable is the fact that authors report the results which were not entirely positive and demonstrated their understanding of the limitations of their method. The overall results of compressed models being just slightly behind the non-private counterparts look rather promising.\n\nHowever, this work contains a number of shortcomings that I think that authors need to address. \nFirstly, authors reject the idea of using PATE in their evaluation as DP-SGD has better performance. I am not entirely sure A) why is this a valid metric to reject the work that is the closest DP student-teacher implementation to this manuscript and B) I am not even sure that this statement is necessarily true [4,5]. It seems to me that PATE was avoided by the authors without a strong reason behind it, making me question some parts of the evaluation of the proposed method. \nSecondly, I am not convinced that a combination of DP and KD is a particularly novel method. In addition to the aforementioned PATE, I also found a number of works such as [1,2,3] (in fact [1] even shares the name of the framework with this study), which seem to have addressed this issue before in detail (some of which were peer-reviewed previously). So what exactly is the novel contribution of this work which was not previously proposed? Because on a high level, while the compression seems like a fairly useful trait (and a promising result), combining DP-SGD with KD is fairly trivial and has been done before.\nOne other comment I have is the argument on the 'better models': for DP training a 'bigger' model does not necessarily outperform a smaller one, so I am not sure such argument holds [6]. \nMinor: inconsistencies with DP-SGD (the original version) and DPSGD.\n\n[1] Lyu, Lingjuan, and Chi-Hua Chen. \"Differentially private knowledge distillation for mobile analytics.\" Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2020.\n[2] Sun, Lichao, and Lingjuan Lyu. \"Federated model distillation with noise-free differential privacy.\" arXiv preprint arXiv:2009.05537 (2020).\n[3 ]Wang, Ji, et al. \"Private model compression via knowledge distillation.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. No. 01. 2019.\n[4] Papernot, Nicolas, et al. \"Scalable private learning with pate.\" arXiv preprint arXiv:1802.08908 (2018).\n[5] Uniyal, Archit, et al. \"DP-SGD vs PATE: Which Has Less Disparate Impact on Model Accuracy?.\" arXiv preprint arXiv:2106.12576 (2021).\n[6] Klause, Helena, et al. \"Differentially private training of residual networks with scale normalisation.\" arXiv preprint arXiv:2203.00324 (2022).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept