Selective Pre-training for Private Fine-tuning
Authors: Da Yu
Sivakanth Gopi
Janardhan Kulkarni
Zinan Lin
Saurabh Naik
Tomasz Lukasz Religa
Jian Yin
Huishuai Zhang
Venue: Accepted by TMLR ()

Abstract: Text prediction models, when used in applications like email clients or word processors, must protect user data privacy and adhere to model size constraints. These constraints are crucial to meet memory and inference time requirements, as well as to reduce inference costs. Building small, fast, and private domain-specific language models is a thriving area of research. In this work, we show that a careful pre-training on a subset of the public dataset that is guided by the private dataset is crucial to train small language models with differential privacy. On standard benchmarks, small models trained with our new framework achieve state-of-the-art performance. In addition to performance improvements, our results demonstrate that smaller models, through careful pre-training and private fine-tuning, can match the performance of much larger models that do not have access to private data. This underscores the potential of private learning for model compression and enhanced efficiency.

Published in Transactions on Machine Learning Research (06/2024)
Selective Pre-training for Private Fine-tuning
Da Yu
yude:3@maillssyu..dd
Sun. Yot-sen University
Sivakanth Gopi
sigopi@@micrrooft..cmm
Microsoft Research
Janardhan Kulkarni
jubul@@merrosofl.com
Microsoft Research
Zinan Lin Microsoft Research
zinandin@mmierroof..ccom
Saurabh Naik
snakk@microsoft..com
Microsoft
Tomasz Lukasz Religa Microsoft
toreli@microsoft..om com
Jian Yin Sun Yat-sen University
Huishuai Zhang
shanghoishcuai@@kk..ck
Peking University
Reviewed on OpenReview: https: // openreview. net/ forum? id= P3g
Abstract
Text prediction models, when used in applications like email clients or word processors, must protect user data privacy and adhere to model size constraints. These constraints are crucial to meet memory and inference time requirements, as well as to reduce inference costs. Building small, fast, and privale domain-specific language models is a thriving area. of research. In this work, we show that careful pre-training on a subset of the public dataset that is guided by the private dataset is crucial to train small language models with differential privacy. On standard benchmarks, small models trained with our new framework achieve state-of-the-art performance. In addition to performance improvements, our results demonstrate that smaller models, through careful pre-training and private fine-tuning, can match the performance of much larger models that do not have access to private data. This underscores the potential of private learning for model compression and enhanced efficiency.
1 Introduction
Many papers have shown that deep learning models are vulnerable to attacks aimed extracting information from the training data (Shokri al. 2017 Hayes et al. 2019 Carlini et al.) 2021 Zhang al. 2021
Matsumoto et a.l, 2023). ). A provable path for mitigating such Dwork privacy attacks is to train the models with differential privacy DDork et al.
Choquette- Choo et al. 2021 Carlini et al. 2023
(2006). mathematically rigorous notion for quantifying the privacy leakage of a machine learning model. Over the past few years, there has been a rapid progress in our understanding of deep learning with DP, both in terms of computational
efficiency (He et al. 2023 Bu et al. 2021 Loe & Kifer 2021 Subramani et al. 2021 Anil et al. 2022) and
privacy-utility trade-off [De et al. 2022
Zhou al.] 2021 Zhu et al.] 2020 Golatkar et al. 2022 Sander
et al. 2022
Bu al.] 2022a Panda al. 2022 Tuo al. 2021
Kairouz al. 2021 Kurakin al. 2023

---

Published in Transactions on Machine Learning Research (06/2024)
Final DP Model
4. Fine-tuning with (sx, ds)-DP
Large-scale Public Data
Private Data
Pre-trained Model
1. Train Domain
Classifier with (ex, 6+)-DP
3. Public Pre-training
DP Domain Classifier
Subset of Public Data
Small Model
Rank Select
Figure 1: The proposed framework for training a small and domain-specifie model with differential privacy (DP). More details on the process training the domain classifier and the selection of public data can be found in Figure We use the method Abadi al. (2016) for training models with DP.
One of the most important findings is that pre-training is crucial for maximizing performance (Li et al. 2022b Ganesh et al., 2023).
Most of the DP literature mentioned above focus on settings where inference time is not a bottleneck and can deploy models of any size. In such case, existing evidence is that larger models pre-trained vast
amounts of public data perform better when combined with private fine-tuning (Li et al., 2022c Yu et al. 2022; Mchta et al. 2022). However, there are plenty of applications where the size of the model is restricted by the inference time, e-g,, language model of an email client or a face identification model rumning in a security system. In such applications, if the inference time is not good then the quality of predictions becomes irrelevant. Further, note also that both these applications the training data quite sensitive, and the models should protect the privacy of users. Building small, fast, and private domain specific language models is also thriving area in industry with several start-ups (MosiacML. ScaleAT. There is also economic motivation smaller models offer cheaper inference costs.
this work, we introduce selective pre-training as a means to improve DP fine-tuning for small language models. Figure [ presents an overview of the proposed framework. Specifically, our approach selects a tailored subset public data, guided by the private data, for model pre-training. This selection process begins with the training of DP domain classifier, using the private data positive training samples. The domain classifier is subsequently utilized to rank all public samples, relaining only the top-ranked samples. We conduct extensive experiments to demonstrate the superiority of selective pre-training over standard pre-training. The representative results are presented in Figure 2
Our main motivation stems from the observation that for fixed size model, there is optimal size of pre-training data after which enlarging the pre-training dataset does not further improve downstream performance. This phenomenon is observed in our Figures 2 and 0 and aligns with the previously established scaling laws for pre-training language models (Kaplan et al, 2020 Hoffmann et al. 2022). For example, the results in Figure 2 suggest that for a transformer model with only 21 million paramoters, pre-training on a random 15% of OpenWeb/Txx (Gokaslan & Cohen 2019) is no different than pre-training on the full OpenWebText. Therefore, in such case, selecting subset of public data that better aligns with the domain of private data is effective way to fully utilize the constrained capacity of small models.
Our contributions are summarized as follows.
1. We propose selective pre-training novel approach for training small, domain-specific language models with DP. We design and implement the first simple and effective DP algorithm for selecting subset of public data that better aligns with the domain of the private data.
We empirically validate the proposed framework using the Enron email dataset (Cohen 2015 and the GLUE benchmark (Wang al.) 2018). The results from the Enron email dataset. (Section 4.1
2

---

Published in Transactions on Machine Learning Research (06/2024)
54
Pre-train Only (GPT-XL)
52
Sel. Pre-train (15% WebText) DP Fine-tune
39
Pre-train (15% WebText) DP Fine-tune
50
Pre-train (100% WebText) + DP Fine-tune
38
Next-token Accuracy (In %
46
37
Perplexity
44
42
35
38
36
34
30
28
82 124
45
124
355
Number of Parameters (in Millions)
Number of Parameters (in Millions)
Figure 2: A representative result from our findings. We plot perplexity and top-1 next word accuracy of GPT models on the test set of the Enron email dataset (Cohen 2015). Overall privacy budget is (s = 7.3, = 1 x 10- The dashed line shows the zero-shot performance of GPT2-XL with 1.5 billion parameters. The figure shows that our framework yields clear improvements in both perplexity and next-token prediction accuracy, which can significantly improve the overall model behavior.
demonstrate the effectiveness of our framework in real-world scenarios, while the results from the GLUE benchmark (Section 4.2 show its superiority compared to existing baselines.
3. In addition to state-of-the-arl DP small models, our experimental results indicate that fine-tuning with DP benefits more significantly from selective pre-training than non-private fine-tuning, as presented in Figures and This underscores the distinctive value selective pre-training within the DP community.
Real-world Impact Our framework was recently used in training industry grade differentially private text prediction language model that now serves many NLP applications. As text prediction models (on email dlients/serven word processors, etc.) serve billions of queries per hour, the inference cost savings due to the decrease model size are significant. Further, due to better inference time, online performance metries, such as the number of predictions accepted by the users, also improve.
1.1 Preliminaries
We begin with the formal definition of differential privacy.
Definition (c, Diffferrnnti Privacy (DP) (Dwork et al. 2006) ). A randomized algorithm A (c,5)- differentially privale if for any two neighboring dalasels D and D', which differ in exucily one dalapoind, and for every subset S of possible outputs: Pr(A(D) E ES) S Pr(A(D')
Private Deep Learning: In the context of deep learning, DP guarantees that the trained model weights are private with respect to training dataset, and hence be released publicly. To train deep learning model with privacy, the most popular method is to first release the gradients of an optimizer with differential privacy and then update the model with privatized gradients (Song et al. 2013 Bassily et al. 2014 Abadi
et al. 2016). We follow the approach in Abadi et al. (2016) to make gradients differentially private. Abadi
et al. 20066) augment each minibatch of gradients with per-example gradient clipping and Gaussian noise addition steps. The clipping step ensures that no one user's sample significantly changes the weights of the model and the noise added guarantees that the contribution of single example is masked.
2 Problem Statement and Our Algorithmic Framework
Input to our problem is private dataset Dpriv corresponding a downstream task T, a model M of size P, privacy parameters 8 > 0, and public dataset. D Dpub- Our goal is to train M public and
3

---

Published in Transactions on Machine Learning Research (06/2024)
private datasets with the aim of maximizing the downstream performance on the task T. The entire process should be (e, 8)-differentially private with respect to Prriv- The constraint on model size is important to compare various algorithms in our setting. In applications, the constraints on model size arise naturally as a consequence of memory and/or inference time requirements.
Our framework for solving the problem consists of the following steps.
Privacy Preserving Data Selection: Given Dpriv, invoke a privacy preserving data selection algorithm Aselect to find Puub Ppuh- The privacy budget for this step is (11,
2. Non-Private Pre-training: Pre-train the model M on D'ub with standard pre-training algorithm. This step does not consume any privacy budget.
3. Private Fine-tuning: Fine-tune M on Dpriv with a differentially private algorithm Annetune- The privacy budget for this step is (E2,62).
The non- con-private pre-training step can be viewed as a post-processing function to Aselect and thus no privacy budget is consumed. The advanced composition theorem of DP (see (Steinke) 2022 for example) guarantees that our framework is (c, DDD.. In our experiments, we use the Privacy Random Variable (PRV) Accountant (Gopi et al. 2021 Ghazi et al. 2022 Koskela et al. 2020). The PRV accountant gives lighter bounds on privacy parameters & and ó than the moments accountant in Abaai et al.] (2016). The rest of the paper is devoted to describing the first step of our framework, followed by experiments to verify the effectiveness of our methods on different datasels.
3 Privacy Preserving Data Selection
We describe our approach to implementing a privacy-preserving data selection algorithm. We provide a specific implementation of our framework and demonstrate its effectiveness, however, our approach is general and can be combined with other private data selection algorithms.
3.1 Our Implementation of Data Selection
Our framework is loosely inspired by the data cleaning framework used in GPT3 and PaLM models Brown (2000); Chowdhery et al. (002)), although motivations are a bit different. The classifiers in Brown et al. Chowdhery et al. [2022) are trained to filter out noisy documents from datasets. In fact, the source datasets in our paper, L.e., Text and Wikipedia, are considered positive examples Brown et al. (2020). Our classifier is trained to recognize examples that are similar samples in the target data. We initialize the classifier with a pre-trained LM and fine-tune it with differential privacy to predict whether a sentence is sampled from the distribution of the target data. We use the classifier to predict all sentences in the source data and rank them according to confidence scores. Although deep neural networks could be overconfident and need calibration in some applications (Guo et al., 2017 Zhang et al.] 2022), not calibrating the outputs does not affect our algorithm because calibration does not change the relative ranking among sentences. We select the top sentences until we reach the target number of pre-training tokens Figure 3 shows an overview of our implementation.
et al.
(2020)
We create training set to teach the classifier to recognize target data distribution. Sentences in the target dataset are labelled positive. Random samples from the source data are labelled as negative? It has been widely observed that a larger training set helps private learning (Bassily et al. 2014 Tramer & Bonch 2021 h.
Therefore we set the number of negative examples as five times larger than the number of positive examples. The privacy cost of training the classifier is accounted in the overall privacy cost.
We run experiments with the Enron Email (Cohen, 2015 as the larget and the OpenWeb Text dataset (Gokaslan & Cohen 2019) as the source. The classifier is initialized with a 124M GPT model pre-trained on OpenWebText. With single Tesla A100 GPU, it takes approximately one hour for finc-tuning the domain
'In preliminary experiments, we also explored random sampling where the sampling weights are scaled linearly or quadratically with the confidence scores from the domain clussifier. We found no statistical difference when compared to using only the top-ranked samples.
2hhi way of assigning uegative labels may introducc few noisy labels, since some public samples may closely resemble those from the private dataset. However, given the vast. size and diversity of the public dataset, the percentage noisy labels would be low, evidenced by the high test Fi-score of the trained domain classifiers.
4

---

Published in Transactions on Machine Learning Research (06/2024)
Train Domain Classifier with
Rank and
Label as Negative
Sample from Public Data
Training Data for Domain Classifier
Large-scale Public Data
DP Domain Classifier
Label as Positive
Off-the Off-the-shelf Pre-traines LM
Confidence Scores
Private Dala
Fine-tuning with DP
Keep Top Ones
DP Domain Classifier
Selected Public Data
Sorted Public Data
Figure 3: The process of training the domain classifier and the selection of large-scale public data.
classifier. With eight Tesla V100 GPUs, it takes less than two hours for computing the confidence scores for all sequences in OpenWebTexx. The privacy guarantee is (0.7, 10- if we only consider the privacy cost of this step. More implementation details are in Section 4.1 The trained classifier achieves an Fi-score of 98.5%. The classifier achieves an F1-score of 92.7% if it is not initialized with a pre-trained LMM
40 group ais changes thanks list Services
whem power
schedule Company enron work database,
pleass Reeting PP
crading wng week cal colifornia
corning quest tonssions
schedules
Line
tean munberrcody today? EgyeR cany gamegnunte
media
worataerr
city - home
country! Li d people
2a wonen
dreetdent story Osone thing life lot
cup
cinformat case So
Sarty wor place
eon
thingSiine times
day noney wayl
year
project peopl 0 Lifeyears
listent tie weeebustnesssaane sexarple
anp tim mi day worl page year systelmctigg
support t mebustnes
company, yag
2 se numbec place
data 8 telp world d gare
Jee something fi honetreen
Khep code
infomaationn energy date number file 1i,, day deal g5fse "foor....
fonily
service 2P systell com post
week
polce trump Syghe healh
State data
inf Or tonn class things
(a) Enron Email
(b) OpenWebText
(c) Selected OpenWebText
Figure 4: The 100 most frequent nouns in Enron email, OpenWeb or selected subset of OpenWebText (10%). A larger font size indicates that the word appears more frequently. Green words are the 100 most frequent nouns in Enron Email. OpenWebText and selected OpenWebText have 28 and 39 words, respectively, that are among the 100 most frequent nouns in Enron Email.
We use the trained classifier to select 10% of OpenWebTeext. We plot the word clouds of Enron email, OpenWebText, and the selected subset of OpenWebText (Figure 4. to visually illustrate the dataset selected by algorithm. The word clouds only show the nouns to exclude common prepositions and verbs. There are 28 nouns which appear in both the top 100 nouns of the Enron email dataset and the top 100 nouns of OpenWebTexx. The number of overlaps increases to 39 when comparing Enron email with the selected subset of OpenWebText, suggesting the trained domain classifier is an effective tool for data selection. Appendix B.1 we also present the results using GLUE (Wang et al, 2018 tasks as the targets and the pro-training corpus of BERT [Devlin et al. 2019 as the source.
4 Experimental Evaluation
We evaluate our full framework (Section 2 language generation and understanding tasks, comparing datasots that most previous works used (Li et al. 2022c Yu et al. 2022). The goal here is to empirically verify the main claim made in the introduction: our framework can be used as an effective tool for model compression, and beats the existing baselines in the literature (Mireshghallah et al.) 2022). We note that Mireshghallah al. (2022) did experiments on GLUE benchmark only, and we compare against them in the next section. We begin with the language modeling on the email dataset, which was the motivating example from the real world application.
5

---

Published in Transactions on Machine Learning Research (06/2024)
4.1 Implementing the Framework on the Enron Email Dataset
Our first larget lask is causal language modeling on the Enron email dataset. The dataset contains approximately 0.5 million (M) emails written by employees of the Enron Corporation and is publicly available for research use. We choose this dataset because its distribution closely resembles some private datasets in the real world for which it is hard to find off- aff-the-shelf pre-training data.
4.1.1 Experiment Setup
We briefty describe important parameters of our experimental sctup. More details are in Appendix C
Target and Source Data We divide the text into sequences of length 256 and treat each sequence as a datapoint, which constitutes the gramularity of our privacy guarantees. Most of the emails in the dataset are shorter than hundred words. In real world applications, it is important to carefully bound the maximum contribution of single email/user to training point. There are ~70K sequences in total. We use 80% of them for training and evenly split the rest 20% for validation and testing, The source data is OpenWebTText [Gokaslan & Cohen 2019) which contains -4 billion tokens. The sequence size for OpenWebText is 512, following the choice in Radford et al. (2018).
Models Models in this section are from the GPT family (Radford et al. 2019). We change the number of layers, hidden size, and intermediate size of the fully connected block to get five different model sizes (21M. 45M, 82M, 124M, and 355M). Details of the models and pre-training hyperparameters are in Appendix All models are pre-trained with nodes with 8x Nvidia Tesla V100 GPUs.
Data Selection We use the algorithm in Section 3 to select 2M sequences from the source data for pre- training. We train the domain classifier for 3 epochs. The baselines include 1) pre-training with 2M random sequences and 2) pre-training with all of OpenWebText.
Privacy Budget and Hyperparameters The overall privacy budget is (7.3, 1 x 10 10-D), similar to previons works on this topic (Li ct al.
2022c Yu et al.] 2022). To roduce the privacy cost of hyperparameter Mohapatra et al. 2022), we follow the findings in previous work to set most of the hyperparameters and only tume the learning rate to adapt to models of different sizes. The hyperparameters for private learning are listed in Table in Appendix C
luning (Liu & Talwar 2019
Papernot & Steinke 2022
4.1.2 Selective Pre-training Provides Clear Gains, Model Efficiency
Figure 2 shows the perplexity and next-word prediction accuracy of different models on the test split of the Enron email dataset. We also present the next-word accuracy and its standard deviation across random seeds in Table D in Appendix B.2 as a complementary to Figure 2 It is clear from the figure that our framework improves performance compared to existing techniques.
More significantly, we see that smaller models can match the performance of much larger models; for example, the 82M model using selective pre-training matches the 124M model using normal pre-training. This shows that the proposed framework can be used to improve the efficiency-utiile trade-off of private learning. We also include the zero-shot performance of the off- off-the-shelf GPT2-XL model (1.5 billion parameters) in Figure 2 The zero-shot performance of GPT2-XL, is worse than the models that have access to private data and are of much smaller size. These findings highlight the importance of private data, which can be looscly treated high quality data, as well as the importance of privacy-enhancing technologies that facilitate the trustworthy use of such data. Figure 10 in Appendix B.2 also presents the results under different privacy budgets (e ranging from 2.3 to 10.9). We observe consistent gains when the selective pre-training framework is used.
4.1.3 Selective Pre-training is More Important for Private Learning
We also fine-tune the models without differential privacy to whether selective pre-training improves downstream performance in non-private learning. The results are in Figure When using 15% of OpenWeb- Text, selective pre-training still improves the performance of all models though the improvement is smaller compared to the private world. When using 100% of OpenWebText, the benefits of selective pre-training gradually diminish as the model size increascs. This suggests that selective pro-training more important for private learning compared to the case in non-private learning. Our hyperparameters for non-private fine-tuning be found in Appendix C
6

---

Published in Transactions on Machine Learning Research (06/2024)
26
Sel. Pre-train (15% Web Text) Fine-tune
Pre-train
52
Pre-train (100% Weblext) Fine-tune
22
Perploxity
20
Next-token Accuracy
49
18
47
46
16
44
10
41
355
45
82
124
Number of Parameters (in Millions)
Number of Won-embbddiing Parameters (in Millions)
Figure 5: Perplexity and top-1 next word accuracy of GPT models the test set of the Enron email dataset. The models are trained without DP. Selective pre-training still improves over standard pre-training, however, the improvements are smaller compared to private learning.
4.2 Experiments on GLUE
We conduct experiments the GLUE benchmark (Wang et al. 2018 common benchmark for fine-tuiigg
language models with DP Yu et al. 2021: Li et al. 2022c Bu et al. 2022b). Our results show that
selective pre-training also improves DP fine-tuning for language understanding tasks, beating the baselines in Mireshghallah et al. (2022).
4.2.1 Experiment Setup
Target and Source Data Our target tasks in this section are MNLI and SST-2, which have respectively the largest and smallest number of examples among the four tasks studied in previous work (Yu al. 2021
Li et al. 2022c; Bu al., 2022b Mircshghallah et al. 2022). The numbers of training examples (N)
MNLI and SST-2 are 393K and 67K. The source data for GLUE tasks is the pre-training corpus of BERT
Devlin et al. 2019); It consists of subset of Wikipedia and the entire Bookeorpus. The source dataset has
approximately 3.5 billion tokens.
Model Sizes We use models from the BERT family (Devlin et al. 2019 We consider four different model sizes (5M, 10M, 25M, and 44M). Details of the models are in Appendix Following previous work (Xia et al. 2022), we do not include embedding matrices when computing the number of parameters of BERT models. For text classification tasks, the BERT embedding layer during inference simply a lookup table.
Data Selection For MNLI and SST-2, experiment with selecting varying numbers of tokens from the source dala. The larget numbers of pre-training lokens are 20M, 40M, 200M, 400M, S00M, 1200M, and 2000M. More complete implementation details on data selection are in Appendix C
Baselines The baselines include pre-training on randomly selected source data and pre-training on all source data. There are two additional baselines for the 44M model. The first directly fine-tuning DistillBERT (Sanh al.) 2019 with differential privacy. DistillBERT is distilled from BERT-base on the source data. The second the best result. in Mireshghallah
al. (2022) compress DP fine-tuned BERT-base model using differentially private distillation pruning. The architecture of the compressed models in Mireshhhallah el al. (2022) and Sanh et al. (2019) are of the same architecture as the 44M model. Although our framework is compatible with the techniques in Mireshghallah et al. (2022) and Sanh et al. 22019) we include the two additional basclines to demonstrate that the proposed framework alone is a competitive approach for model compression in private learning.
al. (2022). Mireshghallah
Private Learning We adopt the setup in Mireshghallah (2022). The privacy budget (4, (4. The hyperparameters for private learning also documented in Appendix C
7

---

Published in Transactions on Machine Learning Research (06/2024)
SST-2 (SM)
SST-2(MMM
32
Selective
Random Full (3500M)
Accuracy (in 96)
Accuracy (in %6
34
84
92
DistiliB
Compression
Selective
78
78
Random
Full (3500M)
20M
43M
200M
400M
200M
1200M
20M
A0M
200M
400M
800M
1200M
Number of Pre-training Tokens
Number of Pre-training Tokens
MNU (5M)
MNLI (44M)
75
75
Accuracy (in 96)
74
Accuracy (in %
74
72
72
70
70
80
60M
200M
400M
800M
1200M
2000M
40M
200M
400M
800M
1200M
2000M
Number of Pre-training Tokens
Number of Pre-training Tokens
Figure 6: Results of pre-training with various numbers of tokens. The first column shows the results of 5M models and the second column shows the results of 44M models. Selective pre-training outperforms baseline algorithms in most of the cases.
4.2.2 Selective Pre-training Outperforms Baselines, Improves Model Efficiency
Figure G shows the test accuracy on MNLI and SST-2 after privately fine-tuning models pre-trained with varying numbers of tokens. Our first finding is that, for most of the settings, selective pre-training outperforms all the algorithuss examined. On SST-2, selective pre-training achieves accuracy that is 4.6% and 3.7% higher than the accuracy of full pre-training for the 5M and 44M models, respectively. On MNLI, the accuracy improvements are 1.4% and 1.8%, respectively. Our second finding is that, for model of fixed size, increasing the number of pre-training tokens does not necessarily lead to better dounstream accuracy. This suggests that there may be an optimal number of pre-training tokens for a given model size (Sorscher et al. 2022 Hoffmann et al.
2022 further emphasizing the need to choose a task-specific subset from a large source data.
Figure shows the test accuracy models of different sizes. When trained with differential privacy, the 25M model with selective pre-training achieves comparable or better performance than the 44M baseline models, the Enron email dataset. The accuracy gains SST-2 are greater than those achieved MNLI, likely because MNLI data distribution is relatively closer to Wikipedia corpus; see Appendix B.1 for the word clouds comparison.
aligning with our observations
5 Related Work
Here discuss closely related prior work. Additional related work is discussed in Appendix
8

---

Published in Transactions on Machine Learning Research (06/2024)
SST-2
MNLI
82
Selective (400M)
Random (400M)
Full (3500M)
Accuracy (in %6)
Accuracy (in %6)
Seloçtive (400M)
68
Randōm-NP (400M)
84
Selective (200M)
Random (200M)
Full (3500M)
Selective-NP (200M)
80
Random-NP (200M)
Number of Parameters (in Millions)
Number of Parameters (in Millions)
Figure 7: Results of pre-training with different model sizes. The numbers in the brackets are the numbers of tokens used for pre-training. 'NP' denotes that the models are fine-tuned without DP. Selective pre-training consistently improves performance across all settings. The improvements for models trained with DP are larger.
The main scope of this work is to train better small DP language models. This scope aligns closely with the goals of model compression. Mireshghallah et al. (2022) study the implementation of classic model compression techniques, knowledge distillation (Hinton al. 2015) and model pruning (Han et al., 2015). under DP constraints. While their DP variants achieve promising results the GLUF benchmark (Wang) ct al. 2018 they find that the DP constraint reduces the effectiveness of those classic compression techniques compared to the non-private case. In this work, the results in Figure f of Section 4.2 suggest that our
framework improves upon the results in Mireshghallah et al. (2022) on GLUE, indicating that selective pre-training can serve as an effective new method for training better small DP models.
Hoffmann et al. (2022) find that the optimal number of pre-training Lokens scales linearly with the number of model parameters, as shown in their Figure and Figure A3. In this work, we also observe that for a fixed-size model, there is an optimal size pre-training data, after which further increasing the dataset does not improve downstream performance (see Figure 6 in Section 4.2). However, Hoffmann et al. (2022) focus solely on the quantity of pre-training data, not its quality. In our study, we demonstrate that both the quality, measured by the distributional gap between private and public data, and quantity play important role in downstream performance. More importantly, Figure 5 and Figure 7 show that DP fine-tuning benefits more from high-quality pre-training compared to non-private fine-tuning, highlighting the unique value of selective pro-training for the
learning community.
Hou et al. 2023) and Gu et al. (2023) study how to privately select an optimal public dataset from an explicitly given list of public datasets. For instance, suppose the private dataset is CIFAR-10, and available public datasets are MNIST, CIFAR100, and ImageNet. The goal to design a private algorithm find which of the three public datasets is better suited for private learning on CIFAR-10. In this paper, we explore how to select subset of single public dataset on sample-by-sample basis. Our algorithm does not require any explicit division of public data and runs efficiently billions of tokens, making it well-suited for finding the right pre-training data for language models. More importantly, our emphasis is not just on model accuracy, but on how pre-training impacts securacy-vs-model size trade-offs.
6
Conclusion and Limitations
6.1 Conclusion
This work introduces selective pre-training, a new approach for pre-training language models that are better suited for fine-tuning with DP on private data. The proposed framework pre-trains the models on selected subset of public data that is better aligned with the domain of the private data. Additionally, the selection of
9

---

Published in Transactions on Machine Learning Research (06/2024)
public data is designed to satisfy DP with respect to the private dataset. Experiments the Enron email dataset (Cohen 2015) and GLUE benchmark (Wang et al. 2018) demonstrate that selective pre-training improves the fine-tuning of lightweight language models by clear margins.
6.2 Limitations
We provide DP guarantees only for private datasets, not for public ones. In building applications, it is important to consider the privacy risks associated with public data (Tramer et al. (022)). ). One potential
solution is enforce differential privacy even during the pre-training stage (Kurakin al. 2022 Amil et al, 2022
Our methods pre-train models from scratch, thereby incurring an additional computational cost associated with pre-training compared to prior work that can utilize off-tho-shelf pro-trained models (Mireshgaalaah et al. 2022). However, in real-world applications such as email clients or text editors, the backbone models are queried millions or even billions of times every day. Therefore, the cost of training, being one-time expense, is a negligible fraction of the accumulated inference cost.
Broader Impact
Language models, trained with DP or not, are widely used for enhancing the typing experience for users [McMahan et al., 2018 Microsoft 2020 Xu et al. (023). In this work, we introduce a new pre-training method designed to improve the DP fine-tuning process of language models. We hope our work facilitate the adoption of wrivacy-preservinn
techniques, providing strong privacy for users while simultaneously reducing deployment costs and further enhancing user experience. However, as previous rescarch indicates, implementing a DP training pipeline can be complex (Tramer et al. 2022). Therefore, we recommend rigorous implementation and thorough auditing for any real-world deployment of the proposed framework.
References
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of ACM Conference on Compuler and Communica- tions Security, 2016.
Noga Alon, Racf Bassily, and Shay Moran. Limits of private learning with access to public data, Advances in neural information processing systems, 2019.
Ehsan Amid, Arun Ganesh, Rajiv Mathews, Swaroop Ramaswamy, Shuang Song, Thomas Steinke, Vinith M Suriyakumar, Om Thakkar, and Abhradeep Thakurta. Public data-assisted mirror descent for private model training, In International Conference on Machine Learning, 2022.
Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Mamurangsi. Large-scale differentially private bert. Findings of the Association for Compulational Linguisties: FMNLP 2022, 2022.
Raef Bassily, Adam Smith, and Abhradeep Thakurta. Differentially private empirical risk minimization: Efficiont algorithms and tight error bounds. Annual Symposium on Foundations of Computer Science, 2014.
Raef Bassily, Albert Cheu, Shay Moran, Aleksandar Nikolov, Jonathan Ullman, and Steven Wu. Private query relcase assisted by public data. In International Conference on Machine Learning, 2020a.
Raef Bassily, Shay Moran, and Auupama Nandi. Learning from mixtures of private and public populations. Advances in Neural Information Processing Systems, 2020b.
Raef Bassily, Mehryar Mohri, and Ananda Theertha Suresh. Principled approaches for private adaptation from public source. In International Conference on Artificial Intelligence and Stotistics, 2023.
Alex Bie, Gautam Kamath, and Vikrant Singhal. Private estimation with public data. Advances in Neural Information Processing Systems, 2022.
10

---

Published in Transactions on Machine Learning Research (06/2024)
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Πya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Newral Information Processing Systems, 2020.
Zhiqi Bu, Sivakanth Gopi, Janardhan Kulkarni, Yin Tat Lee, Hanwen Shen, and Uthaipon Tantipongpipat. Fast and memory efficient differentially privato-sgd via jl projections. Advances in Neural Information Processing Systems, 2021.
Zhiqi Bu, Jialin Mao, and Shiyun Xu. Scalable and efficient training of large convolutional neural networks with differential privacy. Advances in Neural Information Processing Systems, 2022a.
Zhiqi Bu, Yu-Xiang Wang, Sheng Zha, and George Karypis. Differentially private bias-term only finc-tuming of foundation models. arXiv preprint or 2xXi:2210.00086 2022b.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Berbert-Voss: Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulaar Erlingsson, al. Extracting training data from large language models. USENIX Security Symposium, 2021.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. International Conference on Learning Represen- tations, 2023.
Christopher A Choquette-Chon Florian Tramer, Nicholas Carlini, and Nicolas Papernot. Label-only membership inference attacks. In International Conference on Machine Learning, 2021.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Tsard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Morcira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. arXiv preprinl ar XrXiv:2244.020 2022.
William Cohen. Enron email dataset, 2015. URL https: htops://www.... cs. cmu. edw.eeurun.....
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovee, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Abaha, Ethiopia, April 26-30, 2020. OpenReview.nec, 2020. URL http://///pppoeeeeeeetetftrrrrrr
Soham De, Leonard Berrada, Jamie Hayes, Samuel L Smith, and Borja Balle. Unlocking high-accuracy differentially private image classification through scale. arXiv preprint arXiv:2204.166i 2022.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. Conference of the North American Chapter of the Association for Computational Linguistics, 2019.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptogrophy conference, 2006.
11

---

Published in Transactions on Machine Learning Research (06/2024)
Arun Ganesh, Mahdi Haghifam, Milad Nasr, Sewoong Oh, Thomas Steinke, Om Thakkar, Abhradeep Thakurta, and Lun Wang. Why is public pretraining necessary for private model training? arXiv preprinl arXiv:2302.09483 2023.
Badih Ghazi, Pritish Kamath, Ravi Kumar, and Pasin Manurangsi. Faster privacy accounting via evolving discrctization. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szeposvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 2022.
Aaron Gokaslan and Vanya Cohen.
Openwebtext corpus.
http://ssklion007.githuu io/
OpenWebTexttCorpus 2019.
Adilya Golatkar, Alessandro Achille, Yu-Xiang Wang, Aaron Roth, Michael Kearns, and Stelano Soatto. Mixed differential privacy in computer vision. IEEE/CVF Conference on Computer Vision and Pottern Recognition, 2022.
Sivakanth Gopi, Yin Tat Lee, and Lukas Wutschitz. Numerical composition of differential privacy. Advances in Neural Information Processing Systems, 2021.
Xin Gu, Gautam Kamath, and Zhiwei Steven Wu. Choosing public datasets for private machine learning via gradient subspace distance. aixiv preprint or arXim:2303001555 2023.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, 2017.
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't stop pretraining: Adapt language models to domains and tasks. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Mecting of the Association for Computational, 2020.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in newral information processing systems, 28, 2015.
Jamie Hayes, Luca Melis, George Danezis, and Emiliano De Cristofaro. LOGAN: membership inference attacks against generative models. Proceedings on Privacy Enhancing Technologies, 2019.
Jiyan He, Xuechen Li, Da Yu, Huishuai Zhang, Janardhan Kulkarni, Yin Tat Lee, Arturs Backurs, Nenghai Yu, and Jiang Bian. Exploring the limits of differentially private deep learning with group-wise clipping. International Conference on Learning Representations, 2023.
Danny Hernandez, Tom B. Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El Showk, Nelson Elhage, Zac HatfieldDoodds, Tom Henighan, Tristan Hume, Scott Johnston, Benjamin Mann, Chris Olah, Catherine Olsson, Dario Amodei, Nicholas Joseph, Jared Kaplan, and Sam McCandlish. Scaling laws and interpretabilie of learning from repeated data. CoRR, abs/2205. 10487, 2022. doi: 11.48500/ariiv 2205. 10487. URL https://do1 org/oorrr// 10// 48544/5550/00 /a1/50//rxiv....
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. ar preprint arxi A.Xiv:150..00 003.02251, 2015.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. An empirical analysis of compute-optimal large language model training. Advances in Neural Information Processing Systems, 2022.
Charlie Hou, Hongyuan Zhan, Akshat Shrivastava, Sid Wang, Sasha Livshits, Giulia Fanti, and Daniel Lazar. Privately customizing prefinetuning to better match user data in federated learning, ICLR Workshop Pitfalls of Limited Data and Computation for Trustworthy ML, 2023.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. International Conference on Learning Representations, 2022.
12

---

Published in Transactions on Machine Learning Research (06/2024)
Saachi Jain, Hadi Salman, Alaa Khaddaj, Eric Wong, Sung Min Park, and Aleksander Madry. A data-based perspective on transfer learning. IREE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Peter Kairouz, Mónica Ribero, Keith Rush, and Abhradeep Thakurta. (nearly) dimension independent private ERM with adagrad rates via publicly estimated subspaces. In Proceedings of the 34th Annual Conference on Learning Theory, 2021.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprinl wwXiv:2001.08362 2020.
Gavin Kerrigan, Dylan Slack, and Jens Tuyls. Differentially private language models benefit from public pre-training. In Proceedings of the Second Workshop on Privacy in NIP. Association for Computational Linguistics, 2020.
Antti Koskela, Joonas Jälkö, and Antti Honkela. Computing tight differential privacy guarantees using FFT. In Silvia Chiappa and Roberto Calandra (eds.), International Conference on Artificial Intelligence and Slatislics, 2020.
Alexey Kurakin, Shuang Song, Steve Chien, Roxana Geambasu, Andreas Terzis, and Abhradeep Thakurta. Toward training at imagenet scale with differential privacy. arXiv preprinl arXiv:2201. 12328, 2022.
Alexey Kurakin, Natalia Ponomareva, Umar Syed, Liam MacDermed, and Andreas Terzis. Harnessing large-language models to generate private synthetic text. arXiv preprint (rXi:22366001666 2023.
Jaewoo Lee and Daniel Kifer. Scaling up differentially private deep learning with fast per-example gradient clipping. Privacy Enhancing Technologies, 2021.
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burchh, and Nicholas Carlini. Deduplicating training data makes language models better. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Mecting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 8424-8445. Association for Computational Linguistics, 2022. doi: 10. 24118533/1/2022.acch https://doi.org org/10. 1tt..001111/////2.222111111
URL
577
Tian Li, Manzil Zaheer, Sashank Reddi, and Virginia Smith. Private adaptive optimization with side information. In International Conference on Machine Learning, 2022a.
Xuechen Li, Daogao Liu, Tatsunori B Hashimoto, Huseyin A Inan, Janardhan Kulkarni, Yin-Tat Lee, and Abhradeep Guha Thakurta. When does differentially private learning not suffer in high dimensions? Advances in Newral Information Processing Systems, 2022b.
Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners. International Conference on Learning Representations, 2022c.
Chong Liu, Yuqing Zhu, Kamalika Chaudhuri, and Yu-Xiang Wang. Revisiting -agnssag private learning: Faster rates and active learning. The Journal of Machine Learning Research, 2021a.
Jingcheng Liu and Kunal Talwar. Private selection from private candidates. In Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing, 2019.
Terrance Liu, Giuseppe Vietri, Thomas Steinke, Jonathan Ullman, and Steven Wu. Leveraging public data for practical private query release. In International Conference on Machine Learning, 2021b.
Zelun Luo, Daniel J. Wu, Ehsan Adeli, and Li Fei-Fei. Scalable differential privacy with sparse nel- work finetuning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 5059-5068. Computer Vision Foundation / IEEE, 2021. doi: 10. 1109/CVPP44437.222 222264447.2021005000 URL https::/openaceee Scalable Scalable_ Differettal Pariiarrrenallllliacccca Privacy With SparseeMetwwrrk Biccck__Fiiieeee Finetuning, CVPR, 20a___aae1_________1111 papprrrp........ html
thecvf htwpo//.coo/ooottt/////////////a Network
13

---

Published in Transactions on Machine Learning Research (06/2024)
Tomoya Matsumoto, Takayuki Miura, and Naoto Yanai. Membership inference attacks against diffusion models. arXiv preprint arXiv:2302.0000 032662 2023.
H Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. In International Conference on Learning Representations, 2018.
Harsh Mehta, Abhradeep Thakurta, Alexey Kurakin, and Ashok Cutkosky. Large scale transfer learning for diffcrentially private image classification. arXiv preprint arXiv:2205.02977 2022.
Microsoft. Assistive ai makes replying casier, 2020. htt.:p/:www..ccccaaaa.cocoaaa//s http:///a...attaaee.aa..ca.aaaaa
Sören Mindermann, Jan Markus Brauner, Muhammed Razzak, Mrinank Sharma, Andreas Kirsch, Winnic Xu, Benedikt Höltgen, Aidan N. Gomez, Adrien Morisot, Sebastian Farquhar, and Yarin Gal. Prioritized training on points that are learnable, worth learning, and not yet learnt. CoRR, abs/2206.07133 2022. doi: 10.48550/arrivv..n 220..0072........7. URL https://doi. org/oog//// 48550/ arrvv...vv.. 2206.07137
Fatemehsadat Mireshghallah, Arturs Backurs, Huseyin Inan, Lukas Wutschitz, and Janardhan Kulkarni. Differentially private model compression. Advonces in Neural Information Processing Systems, 2022.
Shubhankar Mohapatra, Sajin Sasy, Xi He, Gautam Kamath, and Om Thakkar. The role of adaptive optimizers for honest private hyperparameler selection. In Proceedings of the AAAI conference arlificial intelligence, 2022.
MosiacML. https:/www.mossieee
Ashwinee Panda, Xinyu Tang, Vikash Sehwag, Saeed Mahloujifar, and Prateek Mittal. Dp-raft: A differentially private recipe for accelerated fine-tuning. arXiv preprint arXiw2212.04440 2022.
Nicolas Papernot and Thomas Steinke. Hyperparameter tuning with renyi differential privacy. International Conference on Learning Representations, 2022.
Nicolas Papernot, Martín shokri, Ulfar Erlingsson, lan Goodfellow, and Kunal Talwar. Semi-supervised knowledge transfer for deep learning from private training data. International Conference on Learning Representations, 2017.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Llya Sutskever. Improving language understanding by generative pre-training, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Tlya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 2019.
Tom Sander, Pierre Stock, and Alexandre Sablayrolles. Tan without a burn: Scaling laws of dp-sgd. arXie preprint arX:x:2210.08400 03,00, 2022.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. NeurIPS Workshop on Energy Efficient Machine Learning and Cognitive Computing, 2019.
ScaleAI. https:/ /scale.ccc.....m
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatíkov. Membership inference attacks against. machine learning models. In IEEE Symposium on Security and Privacy, 2017.
Shuang Song, Kamalika Chaudhuri, and Anand D Sarwate. Stochastic gradient descent with differentially private updates. In Global Conference on Signal and Information Processing, 2013.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 2022.
14

---

Published in Transactions on Machine Learning Research (06/2024)
Thomas Steinke. Composition of differential privacy & privacy amplification by subsampling. arXiv preprint (rXiu:2210000999 2022.
Pranav Subramani, Nicholas Vadivelu, and Gautam Kamath. Enabling fast differentially private sgd via just-in-lime compilation and vectorization. Advances in Newral Information Processing Systems, 2021.
Florian Tramer and Dan Boneh. Differentially private learning needs better features (or much more data). International Conference on Learning Representations, 2021.
Florian Tramer, Gautam Kamath, and Nicholas Carlini, Considerations for differentially private learning with large-scale public pretraining. CoRR, abs/22 /bs/2222.06470,, 2022. doi: 10.48550/ 10.44550/aariv.2222 att.:a..iv.222.....22 URL
https://6oi.org/11 1treg//0..4500 48.10..55//ar.... ariiv. 06470
Florian Tramer, Andreas Terzis, Thomas Steinke, Shuang Song, Matthew Jagielski, and Nicholas Carlini. Debugging differential privacy: A case study for privacy auditing. arXiv preprint ar02 1221122222222.22.222222222 2022.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. EMNLP Workshop BlackborNLP: Analyzing and Interpreting Neural Networks for NLP, 2018.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. International Conference on Learning Representations, 2021.
Qian Wang, Zixi Li, Qin Zou, Lingchen Zhao, and Song Wang. Deep domain adaptation with differential privacy. IEEE Transactions on Information Forensics and Securily, 2020.
Mengzhou Xia, Zexuan Zhong, and Danqi Chen. Structured pruming learns compact and accurate models. Annual Meeting of the Association for Computational Linguistics, 2022.
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language models via importance resampling. arXiv preprint arXim:2302.0110 2023.
Zheng Xu, Yanxiang Zhang, Galen Andrew, Christopher A Choquette- Chooe, Peter Kairouz, H Brendan McMahan, Jesse Rosenstock, and Yuanbo Zhang. Federated learning of gboard language models with differential privacy. arXiv preprint arXiv:2305. 18465, 2023.
Jiaxi Yang and Xiang Cheng. Public data assisted differential private deep learning. In International Joint Conference on Neural Networks, 2022.
Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, and Tie- Liu. Large scale private learning via low-rank reparametrization. In International Conference on Machine Learning, 2021.
Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, et al. Differentially private fine-tuning of language models. International Conference on Learning Represenlations, 2022.
Chiyuan Zhang, Daphne Ippolito, Katherine Lec, Matthew Jagiclski, Florian Tramèr, and Nicholas Carlini.
Counterfactual memorization in neural language models. Xiv preprint arXic:2112. 111.2221333 12938, 2021.
Hanlin Zhang, Xuechen Li, Prithviraj Sen, Salim Roukos, and Tatsunori Hashimoto. A closer look at the calibration of differentially private learners. arXiv preprint arXiv:2210.088 088488, 2022.
L Zhang and X Gao. Transfer adaptation learning: A decade survey. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Maobo Zheng, Xiaojian Zhang, Xuebin Ma, et al. Unsupervised domain adaptation with differentially private gradient projection. International Journal of Intelligent Systems, 2023.
Yingxue Zhou, Zhiwei Steven Wu, and Arindam Banerjee. Bypassing the ambient dimension: Private sgd with gradient subspace identification. International Conference on Learning Representations, 2021.
Yuqing Zhu, Xiang Yu, Manmohan Chandraker, and Yu-Xiang Wang. Private-kun: Practical differential privacy for computer vision. Conference on Computer Vision and Pattern Recognition, 2020.
15

---

Published in Transactions on Machine Learning Research (06/2024)
A
Additional Related Work
For general
literature on private deep learning and fine-tuning we refer the readers to Abadi et al. 2016 He
itt al. 2023
Kerrigan et al.
2020:
Li et al.
2022c
Bu et al.
2021 Lee & Kifer 2021
Subramani et al. 2021
Anil et al.
2022 Yu et al.
2022 De et al.
2022
Mehta et al. 2022 Yu et al., 2021 Zhu et al., 2020) Sander
et al.
2022
Bu et al. 2022a; Panda et al.
2022). and references there in. To the best of our knowledge, no
prior work in DP literature has studied selective pre-training from scratch and its impact the transfer learning abilities of model. Our work is at the intersection of several related topics, and we give brief overview of how our work fits into the broader literature on the topic.
Domain Adaptation and Reducing Distribution Disparity Between Public and Private Data
Public data has been widely used to improve private data analysis (Papernot et al. 2017 Alon et al. 2019
Bassily et al.) 2020b a Kairouz et al. 2021 Liu et al., 202 Zhou et al., 2021 Liu et al.) 2021b Amid
ct al. 2022
Yang & Cheng 2022 Li al.
2022a Bie ct al. (022). To address the distribution shift between
private and public data, a recent line of research explores domain adaption (Wang al. 2021 Zhang &
Gao
2022 in the context of private learning (Wang al.) 2020 Zheng et al.) 2023 Bassily et al.) 2023)
However, these works are not applicable to our setting due to many reasons, but in particular that we are interested in how pre-training dataset affects the model size. Much of the above literature considers simply the performance of the final model. In the absence of the model size restrictions, for NLP applications, it is well established He et al. (2023) that pro-training on large corpus of text using a large model offers better utility-vs-privacy trade ofls.
Non-Private Data Selection Automatic data selection and cleaning, along with how the pre-training data impacts the downstream task performance are important problems in deep learning. See Xie al. (2023);
Gururangan et al. (2020); Brown et al. (2020); Chowdhery et al. 2002 Jain et al. (003): Hernandez et al.
(2022] Mindermann el al. [2022] Lee el al. 2022 : Coleman et al. (2020) and references there in. Yet, the
literature is scarce on the impact of selective pre- pre-training on the model, except the recent concurrent work of Xic et al. (2023). Our work explores these questions in the context of private lcarning, with an emphasis on how the quality of data affects performance and model size, As a pilot study on designing
data selection algorithms, we use simple classification-bet based approaches that are easy to privatize and provide a clear illustration of the main messages of the paper. Exxploring more sophisticated approaches Xie et al. 2023) for private data selection is interesting future direction.
B
More Experiments
B.1 Results of Data Selection for GLUE Tasks
We plot the word clouds of SST-2/MNLI and (selected) source data to further demonstrate that the distribution of selected data is closer to the distribution of target data. The source data for SST-2 and MNLI is a subsct of Wikipedia and the entire Bookcorpus.
The domain classifiers of SST-2 and MNLI are trained the same way as illustrated in Section We select. 400M tokens for SST-2 and MNLI, separately. The word clouds of the most frequent 100 nouns are in Figure and g We exclude common prepositions and verbs in the word clouds. On SST-2, our selection algorithm improves the number of overlaps between the source data and the largel data from 25 to 40. On MNLI, our algorithm improves the number of overlaps from 44 to 51. The results explain our findings in Section 4.2 that selective pre-training yields larger performance improvements on SST-2 than on MNLI.
Table 1: Next word prediction accuracy (in %) of GPT models on the Enron email dataset. The overall privacy budget is ( 10-7).
Parameters
21M
45M
82M
124M
355M
Random
32.840.02
35.010.01
36.510.01
37.210.02
38.4
Top
33.840.03
36.140.02
37.540.04
38.440.02
39.44..01
16

---

Published in Transactions on Machine Learning Research (06/2024)
intellggncs scoealyy year directionyears very scnething movig let cee seriseentert
mmime: work Kerk pi
plpe par tgouermment head
univer sity
Bonin Family Family. World season
tines number product radee home.
1n sters years
Feorrd sud cinea work
novies blot
porformances
spepple e Life house film States
garea district wey hove
gases
county a 281
icentury gastem
gorany
role datesdirase Eetrrcton grouu nan
Hor plagge life any video state bandeeadd
Bilife à
acters
york bootball bebbers s state
war
teleyision Bly Muscc par ght
Garres script conedy palepicture e P9L3, femg
people s.llil www perforancess
love
player y music
year day place
ser les aes
gr groug
miston
League
eyes, school songs me hand
ganes characted
john city
director mis san 26 dialogue action timg
waysaarh party wor ld John
serieres S
york
Kind
rilms and an 4 f1 Maothng wam selr rics
Ci ty company gn. number
characters 2 death epi version
No book
dr crasain toueeatcherss
huror
time
look
church schoo ta order
gamerenber women year 1a
fambyy
history
45 sonething album children en
(a) S
(b) Source Data
(c) Selected Source Data
Figure 8: The 100 most frequent nouns in SST-2, the source data, and a selected subset of source data. The source data is Wikipedia and Bookcorpus. Green words are the 100 most frequent nouns in SST-2. The source data and the selected subset have 25 and 40 words, respectively, that are among the 100 most frequent nouns in SST-2.
kids gbildren house use office thing Snumber years? city? wil lot data
asheah
country yry work state sys
bus iness time g
garea district sey home spopple e fe house film 5ta clo
timme: work di county univers sity ty
uh par Egovermentt head
incone film wopulatipiicooooo. apnpowee
order
data
Lalld tee
series 20 people Viiaoe Time musag
Paservis
hand inf nat Lon game Useg
nowr
get think Id rule
help aoople-me e e
bragn nanagesent Sane servi cics
something home
PP25gans century informa tinneter dimeddd life exenal am
Way manccott today progr ranty anyshing
plmppeop
thingsvice
nothing yeah
players music hand r ld John
church $ schoo Lea
gameresber women year S Mn
june ci Cyt company
waysarch party
york gason year day place
football menbers $ states
y
or der
series
family yy
wars
gr group
Keaguu
icentury
children en numbber statee
vis isizetean day line, university Sguseolds..ng
city company district familees home year
Eyears season part t
go way area work with oup ceety
a state e age age
feman
(a) MNLI
(b) Source Data
(c) Selected Source Data
Figure 9: The 100 most frequent nouns in MNLI, the source data, and a selected subset of source data. The source data is Wikipedia and Bookcorpus. Green words are the 100 most frequent nouns in MNLI. The source data and the selected subset have 44 and 51 words, respectively, that are among the 100 most frequent nouns in MNT.I.
B.2 More Experiments on the Enron Email Dataset
Table D shows the top-1 next word prediction accuracy on the test split of the Enron email dataset as well as the standard deviation over five random seeds. With selective pre-training, 82M model achieves an accuracy of 37.5% which is 0.3% higher than the accuracy of a 124M model that is not carefully pre-trained.
We also test selective pre-training under different privacy budgets. Figure 10 presents perplexity and next-word prediction accuracy of 21M and 355M GPT models under a wide range of e (ranging from 2.3 to 10.9). We fix the privacy parameter d 1 10-7 1/1ON. We found that selective pre-training leads to similar improvements across all the choices of e.
C
Implementation Details
Param21M
45M
82M
124M
355M
Param5M
10M
25M
44M
L
4
6
12
24
6
6
d
312
576
768
768
1024
d
312
384
576
768
dFFN 1248
2304
3072
3072
4096
dFFN 1200
1200
2304
3072
Table 2: Architecture hyperparameters the models Table 3: Architecture hyperparameters of the models for the Enron email dataset.
for GLUF tasks.
This section expands on the implementation details that are omitted from the main text due to space constraints.
Details of the Models Let L, d, and dEEN be the number of layers, hidden size, and intermediate size of the fully connected block, respectively. We change I., d, dEEN to get different model sizes. Other architecture
17

---

Published in Transactions on Machine Learning Research (06/2024)
80
Peoleit
50
Pre-training (21M)
Selective training (21M)
45
Pre-training (355M)
Selective Pre pre-training (355M)
35
7.3
10.9
Privary Parameter
Privacy Parameter
Figure 10: Perplexity and top-1 next-word accuracy on the Enron email dataset. We consider a wide range of (ranging from 2.3 to 10.9). The numbers brackets are the number model parameters. The privacy parameter 5 is 10-7 Seleetive pro-training yiclds consistent gains across all evaluated.
hyperparameters are the same as those in Devlin et al. 2019 and Radford et al. (0019 Table and show
the model details for the Euron email datasel and GLUE lasks, respectively.
Data Selection for Enron Email The text in OpenWebTText is also divided into sequences of length 256. To construct the training set of the domain classifier, randomly sample 5N sequences from the source data as negative samples, and use all N sequences in the larget dataset as positive samples. As a result, the training set of the domain classifier is 6 times larger than the target data. This significantly roduces the privacy cost of training the domain classifier because the probability of a target example being sampled becomes times smaller. We initialize the domain classifier with an 82M GPT model pre-trained on OpenWehText and fine-tune with DP-Adam on the constructed training set.
Data Selection for GLUE Tasks Because the positive examples in SST-2 and MNLI natural sentences instead of sequences of fixed length, sample natural sentences in the source data negative examples for training the domain classifier. The domain classifier is initialized with BERT-base. In MNLI, a single example contains two natural sentences, i.e., a premise and a hypothesis. In this case, only one of the two sentences is chosen randomly as a positive example. The number of negative examples is also 5N.
The pre-training sequences in Devlin et al. (2019) are of fixed length. Each sequence may consist of several natural sentences. To get the ranking score of sequence, we first break a fixed-length sequence into natural sentences and use the domain classifier to predict those sentences. The maximum confidence of the sentences is used as the ranking score for the sequence.
Table 4: Hyperparamcters for private fino-tuning. We use N to denote the size of the target datasct.
Pro-training Method
Standard
Selective
Noise multiplier (Enron)
1.00
1.03
Noise multiplier (SST-2)
1.36
1.38
Noise multiplier (MNLI)
1.44
1.46
Train steps (domain classifier)
N/A
100
Train steps (target task)
[150, 500, 1000]
Clipping norm
Learning rate
[1e-4, 5e-4, 1e-3, 3e-3]
Weight decay
0
Batchsize
[0.03N]
Privacy budget
(7.3, x 10-7) for Enron; (4, 1/10N) for GLUE
18

---

Published in Transactions on Machine Learning Research (06/2024)
Hyperparameters For Pre-training The pro-training process uses common hyperparameters in the literature. For pre-training models from the BERT family, we follow the hyperparameters in Devlin et al. (2019). The hyperparameters for pre-training models from the GPT family are as follows. We use a dropout probability of 0.1 and a weight decay 0.01. The 31 and 5g of Adam are 0.9 and 0.999, respectively. All models are pre-trained from scratch for 100K iterations with batch size of 128. The initial learning rale is 10-4 and follows a linear decay schedule.
Hyperparameters For Private Fine-tuning We follow the findings in previous work set most of the hyperparameters (Li et al., 2022c Mireshghallah et al. 2022). We additionally tune the learning rate to adapt to the various model sizes studied. Table summarizes the hyperparameters for private learning. We use the parameter-effiieen fine-tuning algorithm LoRA (Hu et al., 2022) to improve the efficiency of the DP fine-tuning of GPT models Yu et al. (2022); Kurakin et al. (2023). We do not use LoRA for the DP fine-tuning of BERT models to get a fair comparison to Mireshghallah et al. (2022). For a given set of hyperparameters, we use the PRV accountant to get the noise multiplier of DP-Adam. we use selective pre- pre-training, then the noise multiplier is slightly larger because we need to account for the privacy cost of training domain classifier. We repeat each private fine-tuning experiment and 3 times with different. random seeds for the Enron email dataset and GLUE, respectively.
19


Reviews:
[{"review_id": "GVwsDKrSB6", "reviewer": "Reviewer_n8f8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- Simple and effective method that not only has great potential practical value but also adds to our understanding of scaling laws.\n\n- Solid experimental results that well support the claims, showing the trade-off between both privacy-utility and efficiency-utility\n\nWeaknesses:\n\n- The data selection algorithm is quite simple and all the randomly used source data are labeled as negative samples. Although this work is an excellent start in the differential private data-selection topic, the proposed algorithm is certainly not the optimal way to select source training samples since potential positive data may be labeled as negative. \n\n- On page 5, the author mentioned A100 GPU, but in the later section, it is claimed that all the models are trained on V100 GPU.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this work, the authors show that pre-training on a subset of the public dataset, selected under the guidance of a private dataset, can improve the performance of the small, domain-specific model in the differential privacy fine-tuning scenario, superior to the model pre-trained on the whole public dataset.\n\nA public data selection algorithm is proposed to select the subset of the public dataset under the guidance of the private dataset. Solid experiments show that pre-training a small model on the selected public dataset can improve the performance of the DP-SGD fine-tuning with the private dataset.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See comments above.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YOPv8BVghm", "reviewer": "Reviewer_88tk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strenghts:\n\n- A very well written paper, carefully designed replicable experiments that clearly indicate that the approach has its benefits.\n\n- Simple and useful idea.\n\nWeaknesses:\n\n- The contribution is almost totally based on the experimental results, no theoretical/technical results to support the observations and conclusions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a method to select public samples for the pre-training LLMs for private in-domain-data-assisted private training of LLMs. This is done such that part of the privacy budget is spend on training a binary classifier to detect whether the sample was in the private domain or not (randomly selected public samples are labelled negative, private samples positive for training the classifier). Then, pre-training samples are sought from the public data such that the highest-confidence samples (scores obtained using the classifier) are included for pre-training. Several interesting conclusions can be drawn from the experimental results, such as that this selective pre-training benefits the private models more than it does non-private models and that with selective pre-training much smaller models can be used. Also, the optimal number of pre-training samples to choose seems to be a function of the model size, smaller models have smaller optimal number of pre-training samples.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The scaling laws are shortly mentioned in the beginning and also very shortly in the end in the section on open questions: \"Our work also touches upon scaling laws..\". It feels a bit vague the way it is written, could you elaborate on that? How could those laws (e.g., number of pre-training samples as a function of the model size) look like and what would possibly be the reasoning behind those laws?\n\nI have the following 'open question' in mind, perhaps you could address this: Considering different fine-tuning / pertaining strategies for LLMs, what do you think would be an optimal approach in case you need to retrain the model? I suppose it would help to reselect source samples and retrain the DP selecion model as well in this method? Could you comment which adaptors / fine-tuning strategies might be the best in case you want to retrain?\nRelated to this: the following paper considers different approaches for fine-tuning LLMs (perhaps should be cited?) :\n\nKurakin, A., Ponomareva, N., Syed, U., MacDermed, L., & Terzis, A. (2023). Harnessing large-language models to generate private synthetic text. arXiv preprint arXiv:2306.01684.\n\nAlso, what do you think how would the choice of the fine-tuning / adaptation strategy (see e.g. the paper by Kurakin et al. above) affect the 'scaling laws' i.e. whether more or less pre-trainining samples would be beneficial? Or do you think it would simply be a function of the number of trainable parameters?\n\nDid you try different strategies for choosing the pre-training samples using the privately trained selection model? You seem to simply rank the samples and take the ones with highest scores. Perhaps some randomized approach could be an alternative?\n\nTypo: end of page 9, \"recently\" twice in a sentence.", "broader_impact_concerns": "Considering the method has been implemented for an industry scale differentially private text prediction LM, I think it would be suitable to have a broader impact section in the paper. What are the ethical implications, the pros and cons of having a DP model in the email server and so on.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "se98K922wf", "reviewer": "Reviewer_9H2K", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\nThe paper tries to use data selection techniques to improve downstream task performance under private scenario. The paper makes an effort to provide the experiment of pretraining from scratch, which can take time.\n\n# Weaknesses\n\nFirstly, I think there are several major issues of the figure in the paper; figure 1 is quite rough, which lack the meanings of terms such as \"dp data selection\" and \"private finetuning.\" It's crucial for the authors to clarify these terms within the figure to enhance its clarity and utility; For figure 2, why \"pretrain-only\" is a flat line, for different size of the pretrained model, they are also expected to have different perplexity; Regarding Figure 3, there appears to be a significant flaw in the implication that increasing the quantity of data alone enhances language model performance. This overlooks the quality aspect; simply adding low-quality data may not improve model performance. \n\nAdditionally, I fail to grasp the connection between privacy-preserving learning and high-quality training data, which the author states in the paragraph.\nDespite the authors' assertion that their method serves as a tool for model compression, I don't observe any evident model compression techniques in the paper. As I understand it, the approach seems to involve taking a subset of data and integrating it into the pretraining stage only.\nAdditionally, the claim of being state-of-the-art lacks substantiation, as there's no direct comparison in the experimental figures. Moreover, the cited paper was published in 2022, prompting questions about the results of other data selection methods introduced since then.\n\nA major concern is the lack of a clear main contribution in the paper. The absence of a conclusion section further exacerbates this issue. Even after reviewing the abstract and introduction, I still lack a comprehensive understanding of the paper's primary objective.\nOverall, I believe the paper falls below the journal's threshold due to its lack of novelty and shortcomings in both writing and experimental design. Careful revision is required to address these issues effectively.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper's key contribution is a strategy for pre-training small language models with differential privacy. The strategy involves using the private dataset to guide the selection of a subset of the public dataset, which is then used for pre-training. The authors argue that this approach is essential for achieving good results when training these models with differential privacy.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weakness", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: A majority of reviewers agree that this is a nice addition to the literature on private language models and that the work provides a interesting perspective on scaling laws. One reviewer was not in favor of acceptance, raising concerns regarding novelty (not part of TMLR's criteria) and shortcomings in writing and experimental design (not shared by the other two reviewers).
Decision: Accept (poster)