Scaling Laws for Differentially Private Language Models
Authors: Ryan McKenna
Yangsibo Huang
Amer Sinha
Borja Balle
Zachary Charles
Christopher A. Choquette-Choo
Badih Ghazi
Georgios Kaissis
Ravi Kumar
Ruibo Liu
Da Yu
Chiyuan Zhang
Venue: ICML 2025 poster (2025)
Keywords: scaling laws
differential privacy
Primary area: social_aspects->privacy
TL;DR: We aim to understand and quantify the compute-privacy-utility trade-offs in language models training

Abstract: Scaling laws have emerged as important components of large language model (LLM) training as they can predict performance gains through scale, and provide guidance on important hyper-parameter choices that would otherwise be expensive. LLMs also rely on large, high-quality training datasets, like those sourced from (sometimes sensitive) user data. Training models on this sensitive user data requires careful privacy protections like differential privacy (DP). However, the dynamics of DP training are significantly different, and consequently their scaling laws are not yet fully understood. In this work, we establish scaling laws that accurately model the intricacies of DP LLM training, providing a complete picture of the compute-privacy-utility and the optimal training configurations in many settings.

Scaling Laws for Differentially Private Language Models
Ryan McKenna'
Yangsibo Huang'
Amer Sinha Georgias Kaissis? Ravi Kumar' Chiyuan Zhang'
Borja Balle? Zachary Charles'
Christopher A. Choquette-Choo Badih Ghazi"
Ruibo Liu? Da Yu'
Abstract
The scale of data driving LLM progress also creates critical privacy challenge. ofate-of-the-art models train on massive, diverse datasets (Dubey et al., 2024; Gemma Team et al., 2024a) that are also distributed (Carlini et al., 2024) making it difficult to exclude inadvertently shared personal infor- mation. Paradoxically, user data, a key privacy concern, is also erucial for advancing LLM capabilities. User interac- tions provide invaluable feedback for generating realistic synthetic data (Xie et al.. 2024; Kurakin et al., 2023) and aligning models with human values (Stiennon et al., 2020), reflecting real-world use cases better than web-scraped text. However, direct training on sensitive user data is risky due to memorization and regurgitation (Carlini al., 2021; 2023; Ippolito al., 2023; Lukas et al., 2023; Biderman al., 2023; Prashanth et al., 2025). This tension- need for user data versus protecting user privacy is addressed by differential privacy (DP) (Dwork et al., 2006).
Scaling laws have emerged as important compo- nents of large language model (LLM) training as they can predict performance gains through scale, and provide guidance on important hyper- parameter choices that would otherwise be ex- pensive. LLMs also rely on large, high-quality training datasets, like those sourced from (some- times sensitive) user data. Training models on this sensitive user data requires careful privacy protec- tions like differential privacy (DP). However, the dynamies of DP training are significantly differ- ent, and consequently their scaling laws are not yet fully understood. In this work, we establish scaling laws that accurately model the intricacies of DP LLM training, providing a complete picture of the compute- compute-privacy-uue privacy-utility trade-offs and the optimal training configurations in many settings.
While DP offers a principled solution to the tension between data utility and privacy in L.LM training, applying prac- tice, especially large-scale models, presents significant challenges. DP mechanisms like DP-SGD (Abadi al., 2016) and its variants introduce computational overhead, implementation complexity (Subramani et al., 2021), and utility degradation (Bassily et al., 2014). While it is well- known that DP-SGD benefits substantially from training with very large batch sizes (Anil et al., 2022; De et al., 2022: Ponomareva et al., 2023), little work has been done un- derstand the conditions under which this holds in compute- constrained settings, i.c., when an increase in batch size must be coupled with decrease in model size or number of iterations. In part due to this reliance on large batch sizes, the largest models trained with DP today have hundreds of millions, rather than billions, of parameters (Anil et al., 2022: Li al., 2022: Berrada et al., 2023; Ghalebikesabi et al., 2023; Charles et al., 2024; Sander al., 2023).
1. Introduction
Large language models (LLMs) revolutionizing how interact with technology, powering everything from instant translations and concise summaries to complex reasoning and creative content generation (Achiam et al., 2023; Gem- ini Team, 2023). Training increasingly large models on ever larger datasets is a key success factor for these LLMss, with frontier models being trained for millions of GPU- hours (Dubey et al., 2024) and trillions of tokens (Abdin et al., 2024; Gemma Team et al., 2024a;b). Scaling laws for neural language models are crucial because they provide a framework for understanding and predicting the perfor- mance gains achievable with increased computational re- sources, and importantly, guide the optimal allocation of that compute budget between model size and dataset size (Kaplan et al., 2020; Hoffmann al., 2022).
To train large models with DP, it is crucial to spend both the compute budget and the privacy budget judiciously. In this work, we pave the way towards training at the billion- parameter scale by initiating a study on the scaling laws of DP training. To that end, we extend traditional scaling laws to consider a compute-privace priauut--riivcy-uuiiity trade-off, accounting for intricacies and additional variables introduced by DP
'Google Research Google DeepMind. Correspondence to: Ryan McKenna <mekennar @ google.com>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Scaling Laws for Differentially Private Language Models
training. Through a rigorous set of experiments, we empiri- cally model this trade-off, and provide thorough analysis of these experimental results to answer number of scaling law-style questions, finding (among other things) that:
Algorithm 1 (Informal) Generalized DP-SGD,
Appendix B.1 discusses the informalities.
Input: Dataset D, noise- batch ratio o, (expected) batch size B, iterations T
Output: Model parameters 0.
The compute budget allocation predicted by non-private scaling laws is far from optimal under DP, even for huge privacy budgets, confirming the need for our study.
Initialize model parameters On E IB-
for t t= 1 to T do
Select (possibly random) size RB minibatch B,CD
However, we can accurately predict the optimal break- down of the compute budget into model size, batch size, and iterations for virtually any privacy budget and dataset size. These compute-efficien training configurations save 5x to x00x compute compared to baseline configurations, while retaining comparable privacy and utility.
g म ExEB, clip(Ve((r-iix))
9 g + oV 1)M
0₂ OptimizerUppdate(0l-..
return Or
DP-SGD. DP-SGD is widely used algorithm to train neural networks with DP. It attains provable DP guarantees through limiting the contribution (sensitivity) of each exam- ple by clipping its gradient to some L 2--norm (wlog, 1). and then adding isotropic Gaussian noise to the averaged clipped gradients; see Algorithm 1 for pseudo-code. Our algorithm is a slight generalization of the original DP-SGD (Abadi et al., 2016): to enable adaptive optimizers, which are often crucial for training transformer models, the subroutine Opti- mizerUpdate can be any first-order optimizer. Throughout this work, we set OptimizerUpdafe to be Adam (Kingma & Ba, 2015), which we denote DP-Adam. Algorithm I satisfies a formal DP guarantee that can readily be computed as a function of o, B, N, and T using a suitable privacy accoun- macconntin library provides functions that can
The optimal model size is typically at least an order of magnitude smaller with DP than without. This pro- vides insight into the challenges of training large billion- parameter or larger language models with DP.
In the DP setting, increasing the compute budget can sometimes yield little to no reduction in the loss unless accompanied by a corresponding increase in the privacy budget or dataset size.
2. Preliminaries and Problem Setup
Our dataset ID consists of text sequences, where each indi-
Key
Definition
Privacy budget
vidual contributes single se- quence = (II,.... ss) of S tokens, and each token is
N
Data budget
tant. The efficiently and tightly compute the minimum value of as a function of e, 6, N. and B (Google DP Team, 2022).
C
Compute budget
B
Batch size
T
Iterations
drawn from predefined vo- cabulary V. We let N denote
S
Sequence length
Batch Ratio. Note that we parameterize Algorithm I in terms of the noise-batch ratio o, which is the standard deviation of noise added to the mean minibatch gradient, instead of the usual noise multiplier which is typically added to the summed minibatch gradient. While the noise multi- plier typically governs the privacy properties of the mecha- nism, the noise-batch ratio is better proxy for the down- stream learning performance. Specifically, there are two sources of variance in the stochastic gradient estimate ğ: (1) the minibatch estimate of the true population gradient and (2) the Gaussian noise added to ensure DP. Prior work has shown that the latter dominates the variance in most practical regimes (Ponomareva et al., 2023).
M
Model parameters
the total number of individu-
Noise-batch ratio
als contributing to the dataset.
Masked Language Modeling, In this work we focus on the masked language modeling task (Devlin et al., 2019), where each sequence has a chosen fraction Pmask of tokens masked out, i.e., replaced with a special masking token [MASK], uniformly at random. The goal is to predict the original token for each masked token using the entire context (bidirectionally). Let X represent the original sequence of tokens but masked using the above procedure and M the ids of the masked tokens in x. For given parameter vector 0 E RM, the language model defines conditional probability pa(a; x) for each; j E .M, and the goal is to find 0 to maximize the likelihood of all masked training tokens.
2.1. Compute-Optimal DP Training
We are interested in empirically modeling how the compute- privacy-utility trade-off changes as function of the problem parameters. We follow ideas used to model the compute- utility trade-off in the non-private setting (Kaplan et al., 2020; Hoffmann et al., 2022), but extend them to study the private setting by additionally considering the privacy budget and data budget. The key concepts are:
Differential Privacy. A randomized mechanism A satis- fies (e, d)- (Dwork et al., 2006) if, for any two datasets D. D' that differ by a single individual, all subsets O of possible outputs of A and c > 0,0 8 < 1:
Pr[A(D) E 0 Se <Pr[A(D)) E 0] + 5
Compute Budget (C) refers to the total floating point
2

---

Scaling Laws for Differentially Private Language Models
operations (FLOPs) required to train the model. We use the standard approximation of Kaplan et al. (2020): 6. M-B.S B-S-T to measure this, which is proportional to the model size (M) and the total number of training tokens (B.S.T). Note that unlike the non-private scaling laws, we use B to represent the number of examples in a batch (not tokens) because this quantity is what matters for privacy calculations. This approximation provides a platform- independent estimate of compute requirements, and is justified further in Appendix B.3.
compute. Further, is important consider that collapsing the privacy and data budgets to a single quantity is unlikely to provide generalizable insights.
3. Private Scaling Law Methodology
In this section, we detail our methodology for estimating the validation cross-entropy loss from model size, noise-batch ratio, and training iterations, which in turn lets estimate the utility under a fixed compute, privacy, and data budget.
Privacy Budget (e) refers to the value of e at fixed 6 in (c,ö)-DP. We fix 8 = 10-8 = (1/N) unless otherwise mentioned, which is a common choice in the literature (Abadi et al., 2016).
3.1. Decoupling Noise Calibration
A key part of our methodology is to directly analyze the impact of the noise-batch ratio for a fixed but reasonably large physical butch size, rather than indirectly through changes to the privacy budget or batch size. Via post-hoc accounting, we will predict what could happen at different hypothetical batch sizes, an approach that is justified by the fact that typically the noise-batch ratio is the primary source of noise in the minibatch gradients, outweighing the noise due minibatch sampling (Ponomareva et al., 2023).
Data Budget (N) refers to the number of individuals in the training dataset, [D), which can be different from the number of examples processed by DP-SGD under multiple passes. Note that our analysis and insights also hold in the more general setting where individuals can contribute multiple examples, although the data budget must still be interpreted as the number of individuals rather than the number of examples (see Appendix B.2).
This decoupling enables for better understanding of the underlying trade-offs. Without this approach, the non- linearities in DP accounting (detailed in Section 4.5) make it difficult to assess these. We note that a naive methodology that tries to directly model the scaling law as function of privacy budget (without going through the noise-batch ratio) would either provide less insight (by not generalizing across data budgets), or require much more compute.
The privacy and data budgets are absent in most non-private scaling laws because they often assume that an infinite stream of data is available and no privacy protections are needed. In the private setting, model training is often con- strained by both fixed data budget (i.e., a limited set of examples) and a fixed privacy budget (i.e., e in DP). Both of these impact model training; thus, it is crucial to determine the optimal compute usage given the constraints on privacy and data, by fitting a scaling law accounting for this.
After decoupling, the function we want to fit requires three inputs: the model size M, the number of iterations T, and the batsh ratio'. We require the function to be well- defined for a broad range of possible inputs that could be encountered in practical settings. We also need it to cover extreme points that may not be likely to be useful in practice, but may provide additional scientific insight. The methodol- ogy described below attempts to balance this need with the goal of using compute responsibly.
2.2. Private Scaling Law Challenges
Additional Scaling Factors. As mentioned above, our pri- vate scaling laws account for the additional data and privacy considerations not present in the non-private scaling law studies. These add complexity because DP adds noise be- yond what is introduced through the stochasticity of training. Without DP, training with a batch size of B for T iterations is roughly equivalent to training with a batch size of for B-T iterations, as long as is below the so-called "critical batch size" (McCandlish et al., 2018: Shallue et al., 2019; Zhang et al., 2025). However, this relationship does not hold in DP settings, and further, DP training requires larger batch sizes to mitigate the impact of the added noise (Anil et al., 2022; De et al., 2022).
3.2. Detailed Experimental Setup
Models and Datasets. We train BERT models ranging in scale from Tiny (4M parameters) to Mega (778M parame- ters), summarized in Table 1. We focus on the default BERT dataset, which includes approximately 3.3B words (Zhu et al., 2015; Devlin et al., 2019) before tokenization. Each example is truncated or padded necessary to a sequence of fixed length S = 512.²
Compute Requirements. Even without DP, exhaustive hyperparameter tuning is infeasible for large models. DP training introduces further complexity with additional hy- perparameters and the need to adapt standard defaults (e.g.. learning rate) to new regimes, necessitating careful protocol design to achieve near-optimal selection within reasonable
'The learning rate is hyperparameter that is optimized over and not modeled directly. "Future work could fruitfully consider other sequence lengths, as they are likely to showcase interesting trade-offs.
3

---

Scaling Laws for Differentially Private Language Models
number of iterations, 18 unique noise-batch ratios, and three learning rates. While one can directly query this data to answer a variety of interesting questions, we ultimately need to know what might happen in-between (and possibly outside of) the grid points we specifically evaluated. For that, we need to fit a function to the data, for which we follow a parametrretti approach. See Appendix E for studies with fully parametric fits.
Table 1. Models used in this study, taken from Devlin et al. (2019).
Model
Layers
Heads
Dims
Params (M)
BertTiny
2
2
128
4.5M
BertMini
4
4
256
11,4M
BertSmall
4
4
512
29M
BertMedium
8
8
512
41M
BertBase
12
12
768
109M
BertLarge
24
16
1024
335M
Bert
24
24
1536
778M
Data Cleaning and Smoothing. First, we note that loss should monotonically increase with increased noise-batch ratio, and monotonically decrease with increased iterations (unless training diverges), and we want our fitted function to capture this property. In practice, this invariant only holds approximately due to inherent variance in the training process. To clean the data, we apply the following post- processing steps:
Optimizer. We use DP-Adam throughout. We use 1000 steps of learning rate warm-up, followed by exponential learning rate decay, decreasing the learning rate by a factor of 10x over a horizon of 128K iterations. We use per- example clipping with an l2 clip norm of 1.0 across all experiments. We employ the normalized variant of clipping proposed by De al. (2022), help decouple learning rate tuning from clip norm. We verified that this setting effec- tively clips most per-example gradients, as recommended in prior work (Li et al., 2022: De et al., 2022).
1. For each model size and noise-batch ratio, we apply a rolling average over the 10 previous measurements to calculate a smoothed loss value. This corresponds to an average over 10- 10-1000-100 1024 total examples, but does not perfectly preserve the expected invariant.
Learning Rates. We tune the learning rate with per- example gradient clipping but no noise, finding that the opti- mal learning rate is consistently 2-7 across all model scales. With noise, consider three learning rates: 2-7,2-*, 2 This methodological choice was based on early ablations that showed that when adding noise the optimal learning rate does decrease, but gradually so; see Appendix C.7.
2. For each model size and woise-batch ratio we apply iso- tonic regression to ensure the 1280 loss values are mono- tonically decreasing with respect to the number of iter- ations. For each model size and number of iterations, we apply isotonic regression again to ensure the 18 loss values are monotonically increasing with respect to the noise-batel ratio. We do not enforce any monotonicity with respect to model size.
Batch Sizes. We use a fixed physical batch size of 1024 across all experiments. Via post-hoc accounting, we will analyze what could happen at different hypothetical batch sizes, under the assumption that cross-entropy primarily depends on the privacy budget and batch size through the noise-batch ratio. We may expect this choice underesti- mates the benefit of larger batch sizes, a question we study empirically in Appendix C.3.
We use isotonic regression to enforce desired monotonic- ity properties, rather than simpler alternatives like taking the cumulative min across each dimension. The latter ap- proach suffers from statistical phenomenon known as the minimum selection bias, where one outlier sample can com- promise the validity of the measurements. We visualize our smoothing process in Appendix C.9.
Noise-Batch Ratio. We consider 18 values of noise-batch
ratio: {2-* A 6, 6.... 23), plus a baseline value of corresponding to non- privvate training.
Metrics. Every 100 training iterations, we record the av- erage training loss over the previous 100 iterations (or 102, 400 training examples). Using training loss instead of evaluation loss is standard practice scaling laws work, and is justified by the fact that we are training for less than single physical epoch, so training loss is unbiased esti- mate of evaluation loss.
Training Step Extrapolation. Next, we extrapolate our smoothed data with respect to the number of iterations, by fitting a parametric form to the training curve and predicting where the loss would have gone if training continued beyond 128K iterations. We use a simple parametric form inspired by Hoffmann et al. (2022), namely I, F: A. We fit this function using scipy. optimize. wwtriiieeeeuuu__e_eee curve_ fit, which uses the Gevenberg-Marquaade algorithm to solve a nonlinear least squares problem (Nocedal & Wright, 1999). We indepen- dently fit a function for each model size and natse-batch ratio on data from iterations 16K to 128K.
We provide details on the compute platforms and training throughput in Appendix C.5.
3.3. Semi-Parametro Modeling
After training the models described above, we obtain grid of measurements over 6 unique model sizes, 1280 unique
Scaling Law Fitting. After data cleaning, our goal is to fit a function L(M,T,0) o) that estimates the loss under a M M-parameter model training for T iter-
4

---

Scaling Laws for Differentially Private Language Models
8.550106
Pitaayy Budger
Budget
Prtwwcy huadeet
100020
1.59e106
Model Stoe
4 11w6c7
64
16
Batch Siow
150003
1e103
64
Berettonn
64
2.5e077
70020
1.10ic77
1002
1000
10"
10
10
10
12x
12
10
10
12t
100
10th
10
122
10
10
10
12
127
10th
12
124
12"
1017
10
Compute Budget
Compute Budger
Compune Budget
(a) Model Size
(b) Batch Size
(c) Iterations
Figure/ 1. Optimal model size, batch size, and iterations for varying privacy and compute budgets, with fixed data budget of 10 Lines show minimum values for each haper-parameter that achieve within 1% of optimal cross-entropy for constant-comuute training. Shaded regions indicate the full range of near mear-optimal settings.
Mozil M4 7iii
us a final estimate of the cross-entropy of these training configurations. In addition, we can also specify directly the training configurations instead of the compute budget for the purposes of conducting specific ablations or comparisons.
Compune Budget
Consterkampput coviige
Nokus cxlitrtion
Batch Stee [B]
Pitte fundton
Privacy Budget (4)
Prodicted
4. Experimental Findings of Scaling Laws
Data
(N)
Nolne-Batch Ratis 2el 5.2 522
4.1. Optimal Compute Budget Allocation
We first determine how to best utilize our compute bud- get in different situations. Specifically, for a given com- pute/privacy/datt budget, we aim to understand how to opti- mally allocate our compute budget among the model size, batch size, and number of iterations. Additionally, we seek to understand how the optimal allocation changes per bud- get. While this question can be answered for virtually any setting of the budgets with the data we collected, we visu- alize a few relevant slices of the data in Figure 1. More comprehensive results can be found in Appendix C.8. From this visualization, we make the following observations:
Figure 2. Workflow for estimating cross-entropy of different train- ing configurations under given compute, privacy, and data budgets.
ations with a noise-batch ratio of σ. We fit this function using linear interpolation, and specifically scipy. Seppy.interre interpolate. RegularrGridInteroll..e Since M, T. and ō are all naturally varied in log- space, we apply interpolation to the function F such that Floo M, log d) == L(M, T, σ) instead. This func- tion is well-defined for any T and any M, within the range of experimental settings considered; that is, M E [4.5M, 784M], ō E [0.523, 0.5%). Because use interpola- tion, our fitted function matches the smoothed data exactly at the evaluation points, and approximates it in between them. In Appendix E we also fit parametric form for this function as well, finding that it is largely consistent with the non-parametric fit.
in Python.
For small compute budgets, the optimal allocation of com- pute budget does not exhibit strong dependence e. However, there is small but consistent trend that with larger privacy budgets. one should train a larger model with a smaller batch size and for more iterations than one would train with a smaller privacy budget. This find- ing is somewhat surprising, since as the privacy budget gets larger, the point at which increasing batch size leads to diminishing returns in terms of noise-batch ratio in- creases roughly according to RJ / (Ponomareva et al., 2023).
3.4. Using the Fitted Functions
We are now able to answer DP scaling laws questions. Fig- ure 2 summarizes our approach. We begin with inputs: the compute budget, privacy budget, and data budget. Second, we proceed by enumerating an exhaustive set of constant- compute training configurations; i.c., combinations of model size, batch size, and iterations that require the given com- pute budget. Using privacy accounting and noise calibration functions from the dp_ ac_cccunniiggg library, we compute the batsh-batch ratio as a function of the privacy budget, data budget, iterations, and (expected) batch size. Finally, we query our fitted function with this noise-batch ratio, along with the given model size and number of iterations, giving
There are many settings of model size, batch size, and number of iterations that achieve near-optimal loss, as indicated by the large shaded regions. This suggests some amount of robustness for compute-optimal training hyper- parameters. All else being equal, training smaller models on more tokens should generally be preferred due to their inference-time efficiency advantages.
Optimal model sizes are much smaller than predicted by non-private scaling laws. For instance, at 1022 FLOPss
5

---

Scaling Laws for Differentially Private Language Models
12-0s-
Budget
10
-10*
10
Cross Entropy
Takena Mocel Bos
100000-
100
10a
10c
(Chichllia)
SPM
Data Budge
10
114M
10
1000
10
10a
100
10P
10'
10
10"
10
16
1pm
16
1641
16
10th
1d
19l
16ht
16th
10
10-
Compute Budget (FLOPs)
Compute Budget (FLOPs)
Compute Budget
(a) Privacy Budget: E =
(b) Privacy Budget: =
(c) Token-to-Model Ratio
Figure (a-b) Best cross-entropy loss achieved for varying compute budgets, four data budgets, and two different privacy budgets. Each figure is annotated with the optimal model size at the inflection point for two of the curves. (c) Number of training tokens S- B-B.7 divided by number of model parameters for the compute-optimal training configuration, fixing the data budget to N 10".
~ 10a parameters are compute-optimal, compared to ~ 10 non-privately.
The token-to-model ratio increases with compute budget, especially for smaller privacy budgets. As the privacy budget increases, the slope decreases, and for a sufficiently large privacy budget becomes nearly flat as predicted by the prior work. However, the privacy budget required to exhibit behavior similar to prior work is extremely large. Note that privacy budget of 1000 provides no meaningful formal membership inference protection." Nonetheless, the noise added still has a significant impact on training: its behavior Figure 3c is more similar to a privacy budget of than non-private training (e oo). For moderate privacy budgets in the range of [1, 10), good token-to-model ratio is typically between 1000 and 100, 000, although for sufficiently large compute budgets, it can go beyond this point. This connects back to an carlier observation that even with infinite compute, there is eventually no benefit to increasing the model size when using a modest privacy budget. These ratios roughly cor- respond to training models 10x to 50x smaller than pre- dicted by Hoffmann al. (2022).
4.2. Benefits of Increased Compute
We now aim to understand and measure how much bene- fit increased compute budgets can provide and under what circumstances. In Figure 3a, we look at how the optimal achievable cross-entropy depends on the compute budget for different settings of data/privacy budget. Our main ob- servations are:
Increasing the compute budget can be a very effective strategy for reducing cross-entropy under a fixed pri- vacy/data budget up to a limit, but there is an inflection point where increasing the compute budget beyond this point provides little to no benefit. The "critical compute budget" where this inflection point occurs increases with both privacy budget and data budget. For example, with a data budget of 10 and a privacy budget of 1, the best cross-entropy is achieved with a compute budget Z 1020 and corresponds to a model with 114M parameters. This a qualitatively different behavior than non-private seal- ing laws, where increasing the compute budget continues provide benefits even at the extreme scales.
4.4. Comparison Against Baselines
We now measure the improvement our compute-optimal training configurations provide over natural baselines. In the DP training literature, is common to fix the training configuration (model, iterations, batch size), and vary the privacy budget. To that end, we consider 3 baseline training configurations: Bert trained for 7500 steps with a batch size of 1295, BertMedium trained for 5000 steps with a batch size of 15879 and BertTiny trained for 2500 steps with batch size of 283, 061. In all three, we fix the data budget to N 10?. Each of these training configurations requires 19 FLOPs. The first configuration is close to what would be predicted by non-private scaling laws (Hoffmann et al., 2022), while the last might be selected by an expert in DP
More comprehensive analysis of the saturating compute budget for representative set of data and privacy budgets can be found in Appendix C.I.
4.3. Token-to-Model Ratio
We now aim to understand more about compute-optimal training configurations, specifically the ratio of the number of training tokens (as measured by S-B-T) to model size and privacy budget. other words, we study a form of sample complexity. In the absence of DP, a constant token-to-model ratio of 20 is the recommended best practice (Hoffmann et al., 2022). As see in Figure 3c, the behavior under DP is not as simple:
'However, values even larger than this have been shown to he effective against reconstruction attacks in prior works (Balle al., 2022: Kaissis et al., 2023; Ziller et al., 2024).
6

---

Scaling Laws for Differentially Private Language Models
Compute Budget (Batch Size)
Compute Budget (Batch Size)
10a
10
Privacy Budget (Epsilon)
10a
10"
10
10
1/8
10a
1/64
1/8
64
10
10
107
16
1/640
10"
10
10"
Privacy Budget (Epsilon)
Data Budget (Users)
Data Budget (Users)
(a) Data Budget: N 2
(b) Privacy Budget: e
(c) Batch Size: B 65536
Figure 4. Marginal henefits of increasing the privacy hudget (c), compute budget (B), and data budget (N) on the noise-hatch ratio.
who recognizes the importance of large batch sizes. The results are shown in Figure 5, from which we find:
4.5. Synergy between Privacy//atta/Coppl Budgets
While many of the trade-offs that explore in this work are data-dependent and require significant empirical investi- gation, many generalizable scaling insights can be derived purely by exploring privacy accounting. In this section we detail some of these, which corroborate many of our ex- perimental evidence above and require very little compute. These insights are domain-agnostic, and therefore likely to generalize other machine learning settings beyond lan- guage models, while also helping us understand and explain some of the experimental observations presented earlier.
For most privacy budgets, the training configuration pre- dicted by non-private scaling laws (BertLarge) yields very low utility. While utility improves for sufficiently large privacy budgets, this suggests that private scaling laws are fundamentally distinet from non-private ones.
The optimal training configuration changes with the pri- vacy budget, and naively using fixed training config- uration across all privacy budgets, as is common in the literature, leaves significant utility on the table.
Compute-optimal training can either give better utility, or save compute/privacy budget under fixed utility. Training a compute-optimal model with 2 x 10 FLOPs yields sim- ilar utility as the best baseline models with 5x the FLOPs for the reasonable range of privacy budgets. This is just one instructive example. The savings other settings may change depending factors like data budget, compute budget, and quality of the baseline training configurations (e.g., the compute savings over BertLarge exceeds 100x, although this is not shown).
We analyze how the noise-batch ratio behaves as function of privacy budget (as measured by c), compute budget (as measured by B), and data budget (as measured by N). We fix 16000 training steps here, but our findings hold for any fixed number of steps". We compute the noise- batch ratio for different settings by using the dp_ accountinnn library (Google DP Team, 2022). Although the function that computes the noise-batch ratio is generally well-understood in the sense that we know how to compute it tightly given the privacy and training parameters, its precise behavior as function of the privacy budget, compute budget, and data budget is not common knowledge. Indeed, due lack of clear and simple guidance on how to configure DP-SGD, it is not uncommon use compare against sub-optimal configurations of DP-SGD.
BertLarge
Compute Optimal
BertMedium
1 1-10 FLOPs
Cross Entropy
BerlTiny
2 100 FLOPs
In Figure 4 we plot three vector fields. Along each axis we vary the privacy budget, compute budget, and data bud- get. The direction and magnitude of the vectors indicate how much doubling each of these budgets reduces the noise- batch ratio. Each budget is varied different powers of 2. The length of the x and y components of the vector is determined by ratio of noise-batch ratio mi-
3-
2-
10
16x
Privacy Budget (Epsilon)
logarithmic scale
Figure 5. Comparison of compute-optimal training configuration some natural baselines function of the privacy budget. All models are trained with a compute budget of 19 FLOPs and data budget of N 10" respectively.
*While compute budget could also be varied through T,, the effect of changing T is data-dependent and the noise batch ratio not directly comparable across different T.
7

---

Scaling Laws for Differentially Private Language Models
nus one. For example, a vector of length 1 along the privacy budget axis means doubling the privacy budget reduces the noise-batch ratio by factor of two.
these privacy guarantees are limited to downstream data, leaving the pre-training process exposed. Given that L.MMs are pre-trained on extensive Internet data, which is often sourced without explicit user consent (Gold & Latonero, 2017), this raises ethical and privacy concerns (Tramèr et al., 2024). Safeguarding privacy during pre-training remains significant challenge. This study seeks to provide new insights to advance guage models.
As there are three budgets that together determine the noise- batch ratio and they interact in nuanced ways, we show three plots in Figure 4, where we vary two of the budgets at a time while fixing the third. These plots together provide fairly complete picture of the behavior of the noise-batch ratio. Our main observations are enumerated below:
pre-training of lan-
DP Training of Vision Models. Training DP models from scratch for vision tasks is an active area of research (Yu et al., 2021; De et al., 2022; Bu et al., 2022; Kurakin et al., 2022; Sander et al., 2024). The most related work is that of Sander et al. (2023), who investigate the scaling behavior of DP training on vision tasks by varying key hyperparame- ters. They demonstrate that, under fixed privacy budget, carefully tuning batch size, training steps, and learning rate is critical for better accuracy. However, they do not account for a bounded compute budget, a crucial factor in scaling law studies for language models (Hoffmann et al., 2022). Additionally, it remains unclear how their findings translate to language modeling tasks. In this work, we extend scal- ing law analyses to language models, incorporating both standard optimization hyperparameters and bounded com- pute budgets to align more closely with recent LLM scaling research.
In Figure 4a we see that varying the privacy budget or compute budget alone (while fixing the other) leads to diminishing returns. Increasing the privacy and compute budgets in tandem leads to consistent and predictable reductions in the noise-batch ratio.
In Figure 4b we see similar trend when varying data and compute budgets. At small compute budgets, increasing the data budget provides limited benefit, and vere-versa. Increasing them simultaneously leads to consistent and predictable improvements in the noise-batch ratio.
In Figure 4c we see that while increasing data and pri- vacy budgets can be helpful, for a fixed compute budget, increasing either provides diminishing and eventually neg- ligible benefits.
These observations provide guidance on how effectively configure DP-SGD and corroborate our scaling laws above.
5. Related Work
6. Conclusion and Future Directions
Scaling Laws of Language Models. Recent research has explored the scaling laws govemning the performance of lan- guage models as they increase in size. Kaplan et al. (2020) found power-law relationship between model size, dataset size, and compute budget, with performance downstream tasks following predictable scaling curves. Hoffmann et al. (2022) extended this to open-ended language models, ob- serving smooth scaling over 7 orders of magnitude. Chowd- hery et al. (2022) trained PalM, a 540B parameter model that continued the trends. These results suggest language models may continue improving as they scale, although Gan- guli et al. (2022) note scaling alone may not be sufficient for open-ended intelligence. In the context of training language models with DP, where gradient clipping and noise addition (Abadi et al., 2016) alter training dynamics, the scaling laws have remained largely unexplored until this work.
This work establishes principled methodology for under- standing the compute-prine privacy- trade-off of language models trained under DP, and it represents an important step towards training larger, more capable models efficiently on sensitive user data. This endeavor will require collecting increasingly larger datasets over larger groups of individu- als, while simultaneously scaling up compute. For example, to train a billion parameter model optimally with DP, one could collect data from one billion individuals, using a gen- erous privacy budget of R 10, and train on large compute clusters for x 2023 FLOPs. This is in stark contrast to non-private laws, e.g., Anil et al. (2023) suggest a much larger # 20B parameter model could be trained with z 2B examples.
This work raises several new questions worth exploring in future work, including how do the scaling laws change when (1) doing fine-tuning instead of pre-training, (2) using better underlying mechanisms, and (3) when allowed to vary the sequence length. These questions (along with several others) are discussed in greater detail in Appendix A.
Applying DP in Fine-tuning or Prompting. Recent stud- ies demonstrate that ttniit (Bu et al., 2023; Wang et al., 2024; Du et al., 2023; Thaker et al., 2023; Zhang
et al., 2024b; Tobaben et al., 2023; Wu et al., 2024a; Zhang et al., 2024a; Chua et al., 2024a) or prompting (Duan et al., 2023b;; Wu et al., 2024b; Tang et al., 2024; Hong et al., 2024; Amin et al., 2024) LLMS can achieve strong perfor- mance while ensuring downstream data privacy. However,
8

---

Scaling Laws for Differentially Private Language Models
Impact Statement
Bassily, R., Smith, A., and Thakurta, A. Private empirical risk minimization: Efficient algorithms and tight error bounds. In FOCS, pp. 464-473, 2014.
This paper presents work whose goal is to advance the field of machine learning, specifically in the area of differentially private (DP) language models. It establishes DP scaling laws that shed light on the trade-off's between compute, pri- vacy, and utility, and can lead to more efficient and effective methods for training LLMs on user data while satisfying DP, a gold standard for bounding the privacy loss. The scaling laws presented can help researchers and practitioners choose model sizes, batch sizes, and training iterations based on available compute, data, and privacy budgets. By develop- ing methods to make DP training more feasible, the paper contributes to the responsible development and deployment of Al technologies. We point out that, when applying DP in practice, the privacy unit has to be chosen carefully; in particular, a user-level guarantee may be needed. More- over, while a valuable tool, DP may not be sufficient when training on user data; additional mitigations may need to be simultaneously applied depending on the application.
Berrada, L.,, De, S., Shen, H., Hayes, J.. Stanforth, R., Stutz, D., Kohli, P., Smith, S. L.., and Balle, B. Unlock- ing accuracy and faimess in differentially private image classification. arXiv:2308. 10888, 2023.
Biderman, S., Prashanth, U. S., Sutawika, I.., Schoelkopf, H., Anthony, Q.. Purohit, S., and Raff, E. Emergent and predictable memorization in large language models. In NeurlPS, 2023.
Bu, Z.., Mao, J., and Xu, S. Scalable and efficient training of large convolutional neural networks with differential privacy. In NeurlPS, 2022.
Bu, Z.,, Wang, Y., Zha, S., and Karypis, G. Differentially private optimization on large model at small cost. In ICML, pp. 3192-3218, 2023.
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert- Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. Extracting training data from large language models. In USENIX Security, 2021.
References
Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L. Deep learning with differential privacy. In CCS, pp. 018-318,-00 2016.
Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramèr, F., and Zhang, C. Quantifying memorization across neural language models. In ICLR, 2023.
Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan, A. A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., et al. Phi-3 technical report: A highly capable lan- guage model locally on your phone. 0rXiv:2404.14219 2024.
Carlini, N., Jagielski, M., Choquette-Choo, C. A., Paleka, D., Pearce, W., Anderson, H., Terzis, A., Thomas, K., and Tramèr, F. Poisoning web-scale training datasets is practical. In S & P. pp. 407-425, 2024.
Achiam, J. Adler, S... Agarwal, S., Ahmad, L., Akkaya, I, Aleman, F. L.,, Almeida, D., Altenschmidt, J., Alt- man, S., Anadkat, S., et al. GPT-4 technical report. NxXiv:3303.87774 2023.
Charles, Z., Ganesh, A., McKenna, R., McMahan, H. B., Mitchell, N., Pillutla, K., and Rush, K. Fine-tuning large language models with user-level differential privacy. mwXXi:2407.0077 2024.
Amin, K., Bie, A., Kong, W., Kurakin, A., Ponomareva, N., Syed. U. Terzis, A., and Vassilvitskii, S. Private prediction for large-scale synthetic text generation. In EMNLP (Findings), 2024.
Chen, X., Liang, C., Huang, D., Real, E.,, Wang, K., Liu, Y., Pham, H., Dong, X., Luong, T., Hsieh, C.-J., Lu, Y., and Le, Q. V. Symbolic discovery of optimization algorithms. In NeurIPS, 2023.
Anil, R., Ghazi, B., Gupta, V., Kumar, R., and Manurangsi, P. scale differentially private BERT. In EMNLP (Findings), pp. 6481-6491, 2022.
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B., Pope, R., Bradbury, J., Austin, J., Isard, M., Gur-Ari, G., Yin, P., Duke, T., Levskaya, A., Ghemawat, S., Dev, S., Michalewski, H., Garcia, X., Misra, V., Robinson, K., Fe- dus, L.. Zhou, D.,, Ippolito, D., Luan, D., Lim, H., Zoph, B., Spiridonov, A., Sepassi, R., Dohan, D., Agrawal, S., Omernick, M., Dai, A. M., Pillai, T. S., Pellat, M., Lewkowycz, A., Moreira, E.,, Child, R., Polozov, O., Lee,
Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. Palm 2 technical report. arXiv:2305. 10403, 2023.
Balle, B., Barthe, G., and Gaboardi, M. Privacy amplifica- tion by subsampling: Tight analyses via couplings and divergences. In NIPS, 2018.
Balle, B., Cherubin, G., and Hayes, J. Reconstructing train- ing data with informed adversaries. In S & P. pp. 1138- 1156, 2022.
9

---

Scaling Laws for Differentially Private Language Models
K., Zhou, Z.,, Wang, X., Saeta, B., Diaz, M., Firat, O., Catasta, M., Wei, J., Meier-Hellstemm K., Eck, D., Dean, J. Petrov, S., and Fiedel, N. PaLM: Scaling language modeling with pathways, 2022.
Gemini Team. Gemini: a family of highly capable multi- modal models. arXiv:2312. 1/ 2023.
Gemma Team, Mesnard, T., Hardin, C., Dadashi, R., Bhu- patiraju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini research and technology. arXiv:2403.08299 2024a.
Chua, L., Ghazi, B., Huang. Y., Kamath, P., Kumar, R., Liu, D., Manurangsi, P., Sinha, A., and Zhang, C. Mind the privacy unit! user-level differential privacy for language model fine-tuning, In CoLM, 2024a.
Gemma Team, Riviere, M., Pathak, S., Sessa, G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., al. Gemma 2: Improving open language models at practical size. arXiv:2408.00116 2024b.
Chua, L., Ghazi, B., Kamath, P., Kumar, R., Manurangsi, P., Sinha, A., and Zhang, C. Scalable DP-SGD: Shuftling vs. Poisson subsampling. In NeurlPS, 2024b.
Ghalebikesabi, S., Berrada, L.,, Gowal, S., Ktena, I., Stan- forth, R., Hayes, J., De, S., Smith, S. L., Wiles, O., and Balle, B. Differentially private diffusion models generate useful synthetic images. arXiv:2302.13800 2023.
De, S., Berrada, L.., Hayes, J., Smith, S. L., and Balle, B. Unlocking high-aceuracy differentially private image classification through scale. arXiv:2204. 13650, 2022.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for lan- guage understanding. In NAACI-HIIT, pp. 4171 2019.
Gold, and Latonero, M. Robots welcome: Ethical and legal considerations for web crawling and scraping. Wash. JI. Tech. & Arts, 2017.
Google DP Team. Google's differential privacy libraries., 2022. https://gi http::///gghubbcoooooo wtt..///oooo//o/fe/eeee////////e privacy.
Du, M., Yue, X., Chow, S. S., Wang, T., Huang, C., and Sun, H. fP-forward: Fine-tuning and inference on language models with differential privacy in forward pass, In CCS, pp. 2665-2679, 2023.
Hoffmann. J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. I.., Hendricks, A., Welbl, J., Clark, A., al. Training compute-optimat large language models. urXiv:2203.55556 2022.
Duan, H., Dziedzic, A., Papernot, N., and Boenisch, F. Flocks of stochastic parrots: Differentially private prompt learning for large language models. In NeurIPS, 2023a.
Duan, H., Dziedzic, A., Yaghini, M., Papernot, N., and Boenisch, F. On the privacy risk of in-context learning. In ACI,, 2023b.
Hong. J., Wang, J. T., Zhang, C., Li, Z., Li, B., and Wang, Z. DP-OPT: Make large language model your privacy- preserving prompt engineer. In ICLR, 2024.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Llama 3 herd of models. arXiv:2407.21783, 2024.
Huber, J. Robust estimation of a location parameter. In Breakthroughs statistics: Methodology and distribu- tion, pp. 492-518. Springer, 1992.
Ippolito, D., Tramèr, F., Nasr, M., Zhang, C., Jagielski, M.. Lee, K., Choquette-Choo, C. A., and Carlini, N. Prevent- ing verbatim memorization in language models gives false sense of privacy. In INLG-SIGDIAL, 2023.
Dwork, C., McSherry F., Nissim, K., and Smith, A. Cal- ibrating noise to sensitivity in private data analysis. In TCC, pp. 265-284, 2006.
Gadre, S. Y., Smyrnis, G., Shankar, V., Gururangan, S.,
Kaissis, G., Hayes, J., Ziller, A., and Rueckert, D. Bounding data reconstruction attacks with the hypothesis testing interpretation of differential privacy. arXiv:2307.03992, 2023.
Wortsman, M., Shao, R., Mercat, J., Fang, A., Li, J., Keh, S., et al. Language models scale reliably with over- training and on downstream tasks. In ICLR, 2025.
Ganguli, D., Hernandez, D., Lovitt, L., Askell, A.. Bai, Y., Chen, A., Conerly, T., Dassarma, N., Drain, D., El- hage, N., El Showk, S., Fort, S., Seffield-Doddtt Z., Henighan, T., Johnston, S., Jones, A., Joseph, N., Kernian, J., Kravec, S., Mann, B., Nanda, N., Ndousse, K., Olsson, C., Amodei, D., Brown, T., Kaplan, J., McCandlish, S., Olah, C., Amodei, D., and Clark, J. Predictability and surprise in large generative models. In FAccT, 2022.
Kaissis, G., Kolek, S., Balle, B., Hayes, J., and Rueckert, D. Beyond the calibration point: Mechanism comparison in differential privacy. In ICMI., pp.
2024.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv:200.08300 2020.
10

---

Scaling Laws for Differentially Private Language Models
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In ICLR, 2015.
Sander, T., Stock, P., and Sablayrolles, A. TAN without burn: Scaling laws of DP-SGD. In ICML,, pp. 29937- 29949, 2023.
Kurakin, A., Song, S., Chien, S., Geambasu, R., Terzis, A., and Thakurta, A. Toward training at ImageNet scale with differential privacy. arXiv:2201. 122,2 2022.
Sander, T., Yu, Y., Sanjabi, M., Durmus, A., Ma, Y., Chaud- huri, K., and Guo, C. Differentially private representation learning via image captioning. In ICML, 2024.
Kurakin, A., Ponomareva, N., Syed, U., MacDermed, L., and Terzis, A. Harnessing large-language models to gen- erate private synthetic text. arXiv:2306.01 01684, 2023.
Shallue, C. J., Lee, J., Antognini, J.,
J., Frostig, R., and Dahl, G. E. Measuring the effects of data parallelism on neural network training. JMLR, 2019.
Li, X., Tramèr, F., Liang, P., and Hashimoto, T. Large lan- guage models can be strong differentially private learners. In ICLR, 2022.
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. In NeurlPS, 2020.
2022.
Liu, P. J., Novak, R., Lee, J., Wortsman, M., Xiao, L., Ev- erett, K., Alemi, A. A., Kurzeja, M., Marcenac, P., Gur, I., Kornblith, S., Xu, K., Elsayed, G., Fischer, I., Pennington, J., Adlam, B., and Dickstein, J.-S. NonoDO: minimal transformer decoder-only language model implementa- tion in JAX., 2024. URL http://gi wtw.b.cooo/c.... google- deepmind/nanodo.
Subramani, P., Vadivelu, N., and Kamath, G. Enabling fast differentially private SGD via just-in-time compilation and vectorization. NeurIPS, pp. 26409-26421, 2021.
Tang. X., Shin, R., Inan, H. A., Manoel, A., Mireshghallah, F., Lin, Z., Gopi, S., Kulkarni, J., and Sim, R. Privacy- preserving in-context learning with differentially private few-shot generation. ICLR, 2024.
Loshchilov, I. and Hutter, F. Decoupled weight decay regularization, 2019. URL https://arxiv.org// 1711.05101.
Thaker, P., Setlur, A., Wu, Z. S., and Smith, V. Leverag- ing public representations for private transfer learning. arXiv:2312. 2023.
Lukas, N., Salem, A., Sim, R., Tople, S., Wutschitz, I.., and Zanella- Zanella-Béguelin, Analyzing leakage of personally identifiable information in language models. In & 2023.
Tobaben, M., Shysheya, A., Bronskill, J., Paverd, A., Tople, S., Zanella-Beguelin S., Turner, R. E., and Honkela, A. On the efficacy of differentially private few-shot image classification. TMLR, 2023.
McCandlish, S., Kaplan, J., Amodei, D., and Team, O. D. An empirical model of large-batch training. arXiv: 1812.0616, 2018
Tramèr, Kamath, G., and Carlini, N. Posi- tion:considerations for differentially private learning with large-scale public pretraining. In ICML, 2024.
Nocedal, J. Updating quasi-Newton matrices with limited storage. Mathematics of Computation, 35(151):733-72, 1980.
Nocedal, J. and Wright, S. J. Numerical Optimization. Springer, 1999.
Wang, B., Zhang, Y., Cao, Y., Li, B., McMahan, H., Oh, S., Xu, Z., and Zaheer, M. Can public large language models help private cross-device federated learning? In NAACL (Findings), pp. 934-949, 2024.
Ponomareva, N., Hazimeh, H., Kurakin, A., Xu, 7.,, Denison, C., McMahan, H. B., Vassilvitskii, S., Chien, S., and Thakurta, A. G. How to DP-fy ML: A practical guide to machine learning with differential privacy. JAIR, 2023.
Wu, F., Inan, H. A., Backurs, A., Chandrasekaran, V., Kulka- rni, J., and Sim, R. Privately aligning language models with reinforcement learning. ICI.R, 2024a.
Prashanth, U. Deng, A., O'Brien, K., SV. Khan, M. A., Borkar, J., Choquette-Choo. C. A., Fuehne, J. R., Bider- man, S., Ke, T., et al. Recite, reconstruct, recollect: Mem- orization in LMS as multifaceted phenomenon, In ICLR, 2025.
Wu, T., Panda, A., Wang, J. T., and Mittal, P. Privacy- preserving in-context learning for large language models. ICLR, 2024b.
Xie, C.,, Lin, Z., Backurs, A., Gopi. S., Yu, D., Inan, H. A., Nori, H., Jiang, H., Zhang, H., Lee, T., et al. Differen- tially private synthetic data via foundation model APIs Text. In ICMI., 2024.
Rush, J. K., Charles, Z., Garrett, Z., Augenstein, S., and Mitchell, N. E. DrJAX: Scalable and differentiable mapre- duce primitives JAX. WANT@ ICML,, 2024.

---

Scaling Laws for Differentially Private Language Models
Xu, Y., Lee, H., Chen, D., Hechtman, B., Huang, Y., Joshi, R., Krikun, M., Lepikhin, D., Ly, A., Maggioni, M., et al. GSPMD: general and scalable parallelization for
ML computation graphs. arXiv:2105.04660 2021.
Yeom, S., Giacomelli, I., Fredrikson, M., and Jha, Privacy risk in machine learning: Analyzing the connection overfitting, In CSF, pp. 268-282, 2018.
You, Y., Li, J., Reddi, S., Hseu, J., Kumar, S., Bhojanapalli, S., Song, X., Demmel, J., Keutzer, K., and Hsich, C.- Large batch optimization for deep learning: Training BERT in 76 minutes, 2020. URL https://arxiv.org abs/1904 009622...
Yu, D., Zhang, H., Chen, W., Yin, J., and Liu, T.-Y. Large scale private learning via low-rank reparametrization. In ICML, 2021.
Yu, D., Naik, S., Backurs, A., Gopi, S., Inan, H. A., Kamath, G., Kulkarni, Lee, Y. T.,, Manoel, A., Wutschitz. L., Yekhanin, S., and Zhang, H. Differentially private fine- tuning of language models. In ICLR, 2022.
Zhang, H., Morwani, D., Vyas, N., Wu, J., Zou, D., Ghai, U., Foster, D., and Kakade, S. How does critical batch size scale in pre-training? In ICLR, 2025.
Zhang. L., Li, B., Thekumparampil, K. K., Oh, S., and He, N. DPZero: Private fine-tuning of language models without backpropagation. In JCML, 2024a.
Zhang, X., Bu, Z., Wu, Z. S., and Hong, M. Differentially private SGD without clipping bias: An error-feedback approach. In ICLR, 2024b.
Zhu, Y., Kiros, R., Zemel, R. S., Salakhutdinov, R., Ur- tasun, R., Torralba, A., and Fidler, S. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In ICCV, pp. 19-27, 2015.
Ziller, A., Mueller, T. T., Stieger, S., Feiner, L. F., Brandt, J., Braren, R., Rueckert, D., and Kaissis, G. Reconciling privacy and accuracy in for medical imaging. Nature Machine Intelligence, 6(7):764-774) 2024.
12

---

Scaling Laws for Differentially Private Language Models
A. Limitations and Open Questions
While our methodology revealed a number of interesting findings about the behavior of scaling laws under DP, there are some limitations of our approach and questions that remain unanswered that we enumerate below.
Fixed Physical Batch Size. Our methodology relies crucially on the assumption that the Gaussian noise introduced to preserve privacy far outweighs the randomness introduced from minibatch sampling, and thus it would be sufficient vary the noise-batch ratio while keeping the physical batch size fixed to a large constant value of 1024. Appendix C.3 reveals that this assumption may not be fully true, and that the physical batch size has a more nuanced effect that we cannot fully explain.
Robustness to Other Training Setups. Our methodology focuses single class of BERT models, with a fixed dataset and DP mechanism, which allowed us to do deeper experimentation other relevant variables. Our general methodology holds for different models, datasets, and mechanisms, but the exact quantitative findings may differ under different training setups. As the field continues to make advancements on training transformers with DP, it would be interesting and informative to rerun our experiments with better base mechanisms.
Pre-training vs. Fine-tuning. As an important first step, focused on the pre- pre-training regime in this work, where start with completely random model which we train from scratch. Fine-tuning pre-trained model with DP is often a preferable approach in practice to get the best privacy/utility trade-offs (Yu et al.. 2022: Li et al., 2022). There are number of challenges to overcome to quantify the scaling laws in this regime, but it remains an interesting question for future work.
Sequence Length. Our experiments focus on fixed sequence length of 512 tokens, which was the default value in the experiment branched. However, the sequence length is yet another important knob that can be tuned alongside the batch size, model size, and number of iterations in language modeling tasks. There are likely interesting trade-offs to explore here: with smaller sequence lengths, less context is available to predict the next / missing tokens, but the saved computation can be used to increase the batch size, model size, or number of iterations. Whether the trade-off is worth it likely depends on the exact setting as well as the distributional properties of the training data.
Over-Training and Inference-Time Compute. While this work focuses the FLOPS required to pre-train model to a given loss threshold, in practice language models are often over-trained in order to account for inference-time costs (Gadre et al., 2025). If a model is going to be deployed, it may make sense to over-train a smaller model (which is cheaper to serve) than to train a larger model for compute-optimal FLOPs budget. While we do not study over-training in our work, we note that such a study is particularly fruitful in the case of DP training; the privacy costs already often favor smaller models (when compared to non-private scaling laws). Investigating this confluence would likely yield valuable insights into DP scaling laws.
Larger Model Sizes. The accuracy of any given scaling law is predicated to some degree on the range of model sizes trained on. For example, Hoffmann et al. (2022) train model of up to 16B parameters. Due to the necessity of using very large batch sizes, training models of such scale requires a significant amount of compute. We leave the task of training on model of larger scale to future work, along with analysis of how much this affects the derived scaling law.
Efficient Implementations of Per-Example Gradient Clipping. When considering to use significant compute budget to train a large language model with DP, it is important that that model training code is carefully optimized to minimize the overheads of DP training. Using efficient vectorized per-example clipping implementations in JAX have been shown to work perform well with a reasonable overhead compared to non-private training (Subramani et al., 2021), although this focused on single-machine training scenarios, and more careful study is needed in this area when doing multi-machine training. especially when moving beyond pure data-parall paralpaaallel,sm, which we focused on in this paper.
The Choice of Optimizer Our analysis relies on current optimization techniques, which may not be optimal for privacy- preserving training. Several potential optimizer improvements could affect our findings. A uniformly better optimizer would likely preserve the observed scaling relationships while the actual optimal operating points might shift. In previous scaling law studies we do see the better optimizer can somehow smooth out the discontinuities in scaling behavior (Chen et al., 2023; Loshchilov & Hutter, 2019) or even enable new scaling regimes sometimes (e.g., LAMB (You al., 2020) for large
13

---

Scaling Laws for Differentially Private Language Models
Algorithm 2 Generalized DP-SGD.
Input: Dataset D, noise-batee ratio o, (expected) batch size B, iterations T
Output: Model parameters 6.
Initialize model parameters 0g R-M
for t = 1 to T do
Select a (possibly random) size B minibatch B,CD
9 ExeR, clip(V((0-13x)))
9 = g + (0, 1)M
8, = OptimizerUppaate(0..:11
return 0r
batch size pre-training shows a very different scaling behavior). The optimizers specifically designed for privacy- preeerini training might recommend new set of parameters to enable better absolute performance.
B. Additional Details
B.1. Notes on Generalized DP-SGD
Minibatch Selection We were vague in our description of the minibatch selection step. In most descriptions of DP-SGD, the minibatch is formed by Poisson subsampling with a fixed probability. Sampling with or without replacement, as well as deterministic batching are also possible (Balle et al., 2018). In our paper, we calibrated noise under both the Poisson sampling assumption and the deterministic batching strategy, picking the lower noise multiplier. When doing Poisson sampling, we use the sampling probability B/N and noise multiplier B-ō.
Known Quantities If doing Poisson sampling, typically are operating under the add/remove adjacency definition. Under this definition, N is considered sensitive quantity that we do not have access to directly, hence we cannot technically define the sampling probability as B/N without violating DP. We also rely on N later on, discussing its importance as it is interpreted as the data budget. If necessary, one can approximate N quite accurately with DP since is a simple count.
Alternatively, one can simply use the "zero-out" adjacency notion (Chua et al., 2024b), where N known but Poisson sampling still enjoys the same privacy analysis.
Clipping Function We omit clipping norm parameter in the definition of "clip". This can be any function that maps an arbitrary real-valued vector to one with fn-nrmm at most one. One standard choice is to clip the norm to C, and then divide by C' (De et al., 2022).
B.2. Unit of Privacy and Multiple Participations
In traditional scaling laws work, it is common to assume access to endless stream of data that does not require privacy protections. Therefore, every training example is only seen once, which simplifies the analysis of the scaling laws. In our case, we trained our models for 128K iterations with physical batch size of 1024, which is slightly less than a single pass over our entire dataset, satisfying the typical assumption. However, in our data analysis, we estimate what would happen with significantly larger batch sizes than we ran with, and some cases this would involve multiple passes over the actual private dataset, something we did not account for directly in our analysis. Therefore, the actually setting that is best represented by our experimental methodology is not actually example-level DP, but rather user-level DP, There, may assume that we have finite number of users N (which should now interpret the data budget), but we have an endless stream of data for each user. This circumvents the main concern, while allowing for users to participate multiple times during training which is typically very useful under DP. Alternatively, one can still consider the example-level DP setting, where each base example has multiple augmentations (e.g., rewritten text sequences that are semantically similar) that we can train on. All of our findings should hold, and be more reliable this setting based on our methodology.
B.3. FLOPs estimation under DP
As discussed, we approximate the compute cost C as 6- C-MBB-S. B-S-T based on the non-private scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022) except that B represents the number of examples (not tokens) a batch, as this determines the privacy budget. This cost model is useful because we can directly compare to the non-private scaling laws. Further, this cost model is also accurate because the extra overhead of DP-SGD compared Adam can be directly amortized: compiler-based
14

---

Scaling Laws for Differentially Private Language Models
systems like GSPMD (Xu al., 2021) and parallel machine learning libraries (Rush et al., 2024) let parallelize the per-example gradient computations without linear (in B) increase in memory usage. The total clipping costs are only small linear cost (comprising of only element-wise operations and no matrix multiplications) in M, T, and B (and are independent of sequence length S); the total noising costs are independent of B and are linear in only M and T. Thus, the overall compute in DP-SGD is dominated by the non-private approximation above.
C. Additional Experiments
C.1. Saturating Compute Budget
Building on our findings above, it is natural to ask where the saturation point occurs for different privacy budget and data budgets. This can be helpful to determine how much compute is needed to get the most utility under fixed data and privacy budget, as well as how to spend that compute optimally. These results are shown in Table 2.
With a higher data and privacy budget, we benefit substantially from larger compute budgets.
With DP, the compute-optimal training configurations requires training significantly smaller models over significantly more tokens than without DP. For these training configurations, the ratio of training tokens to model parameters varies in different settings, but all settings it is significantly larger than it would be without DP, where prior work found 20x to be good rule of thumb (Hoffmann et al., 2022).
Table Saturating computc budgets, well as optimal training configurations for those compute budgets across representative of data and privacy budgets.
Data
Privacy Budget
Compute Budget
Cross Eutropy
Model
Iterations
Batch Size
Token Moodel
Budget
Size
Ratio
1.0
1.3 x 1016
7.28
1.6 x 10"
188 10a
1. 1017
6.65
46 109
10a
10a
10a
2.0 x 16'"
5.60
1.7 x 10"
2.7 10a
1.4x 10"
1.1 10"
64
7.5 x 16t5
4.63
2.0 x 107
6.3 10a
1.9 10
3.2 x 10"
1.0 10
2.8 x 10t7
5.89
4.6 10"
2.5 x 102
8.2 x 10
2.3 10"
8.8 x
4.62
1.9 x 107
6.5 10
2.3
4.1 10d
3.3 1619
3.61
107
10
3.2 1020
2.82
4.9 107
1.2 10
1.9
10
x.0 10"
3.8 1619
3.73
1.7 107
9.6 10a
7.8 104
2.3 10a
3.8 x 1020
2.81
4.9 x 107
1.1 10
2.2 10
2.6 104
2.0 10a1
2.15
7.0 102
1.2 10
74 10
6.7 x 10
1022
3.3 x 10
10
2.26
1.3 10a
2.2
4.4 x 10a2
1.66
3.3 10
4.9 10a
8.8 10a
6.7 10a
16
1.0 x 102h
1.32
3.3 10
9.3 10*
1.0 10"
1.5 x 10P
64
1.0 x 10TM
1.23
3.3 10"
1.1 102
8.8 10
1.5 x 10"
1.0 10"
8.5 x 10
1.36
3.3 x 10a
9.4 10
8.8 10
1.3 x 10
1.0 x 1628
1.23
3.3 10a
1.1 10
888
1.22
10
10
10
10P
1.2 1023
1.20
3.3 x 10
1.1 x 10
1.1 X 10"
1.8 10F
C.2. Full Experiment Grid
In Figure 6, we plot the cross-entropy loss for different privacy budgets, data budgets, and compute budgets under varying numbers of iterations, model sizes, and batch sizes. Much can be learned from these plots, including:
The optimal number of iterations typically falls around 7 ~ 10K, and the optimal batch size often falls in the range 10 - 100K, although neither these is universally true and as expected it depends on the values of the privacy, data, and compute budgets. Batch size seems to be the most important parameter, indicated by the steep slope of those lines.
C.3. Physical Batch Size Ablation
Central to our methodology is an assumption that for a fixed noise- batch ratio, the training curves should be similar for different physical batch sizes. In this section, we conduct ablations to test this hypothesis, and quantify the impact of varying physical batch size under fixed noise-batch batch ratio. We consider values for noise-batch ratio: 0.520, 0.515 and 0.510 and
15

---

Scaling Laws for Differentially Private Language Models
Privacy Budget
Cross Entropy
4
4
64
10a
104
10 Iterations
10c
10
10
10
10
10a
10
10
10
Model Size
Batch Size
Data Budget
Cross Entropy
10c
107
10
4-
10
10a
10+
10
10"
10
10
10
10
10a
10a
10
10c
Iterations
Model Size
Batch Size
Compute Budget
Cross Entropy
1017
10
-1021
4-
1023
104
10
10a
10'
10
10a
10"
10
Iterations
Model Size
Batch Size
Figure 6. Cross-entropy of best models trained in each setting. From top to bottom, vary the Privacy Budget, Data Budget, and Compute Budget, keeping the other two budgets fixed to default values (bolded). From right, we vary number of Iterations, the Model Size, and the Batch Size, and treat the other two nuisance parameters which we minimize over.
3.
Physicl Banch Size
128
128
512
512
8192
0192
3.4
42-
54
120
3.2
512
8192
107
Reeattons
forations
Iterattonns
(a) noise-batch ratio 0.
(b) noise-batch ratio 0.5
(c) noise-batch ratio 0.5
Figure 7. Cross-entropy loss of BertTiny averaged over 3 trials for different physical batch sizes and noise-batch ratio values.
physical batch sizes of 128, 512, 2048, and 8192. For this ablation we focus on the BertTiny model, which we train for 128K iterations. We average the losses across three random trials.
The results of this experiment are shown in Figure 7. Our primary findings are:
At the smallest noise-batch ratio in Figure 7a, results expected. Specifically, larger batch sizes lead better model performance, but there diminishing returns. Physical Batch Sizes of 2048 and 8192 have nearly identical training curves.
At the medium and larger noise-batch ratio values shown in Figures 7b and 7c, we observe surprising phenomenon: smaller physical batch sizes lead to models with lower loss. The effect is most prominent in Figure 7c. We do not have a good explanation for this behavior, but we did additional experiments to rule out some plausible explanations in Appendix C.4. Large physical batch sizes (B = 2048 and B 8192) still have very similar learning curves.
While the results of this experiment did not fully match expectations, similar behavior was observed prior work (Sander
16

---

Scaling Laws for Differentially Private Language Models
al., 2023) (Figure 4b). Moreover, for sufficiently large batch sizes the training eurves are very similar across all noise-batch ratio values tested. Thus, believe that the physical batch size of 1024 that (although not perfect) indicator of what would happen with much larger batch sizes that would be needed to get favorable privacy/utilityy trade-offs in real-world settings. Understanding when and why thsi behavior manifests is a very interesting direction for future work.
in main experiments is reasonable
C.4. Physical Batch Size Ablation Extended
In Appendix C.3 we observed surprising phenomenon where for some values of noise-batch ratio, smaller physical batch sizes perform better than larger physical batch sizes. This is in contrast to our initial hypothesis, and our experimental results for very small values of noise-batch ratio that larger physical batch sizes should be on par with or better than smaller physical batch sizes for the same noise-batch ratio.
While we do not have great explanation for the observed phenomenon, have ruled several possible explanations, which we discuss below:
Learning Rate Tuning. While our main experiment used fixed learning rate of 0.5 across all values of noise-batch ratio, we ran further experiments for noise-batch ratio of 0.5 with four different learning rates (0.55, 0.57, 0.5%, 0.5%). and report the best cross-entropy across all learning rates a per-iteration basis. Even with learning rate tuning, the conclusion is the same: smaller physical batch sizes achieve lower loss than larger ones (see Figure 8).
5.25
Physical Batch Size
5-
128
4.75
512
Cross Entropy
2048
4.5
8192
4.25
3.75
3.5
10a
10
10
Iterations
Figure 8. Smaller physical batch sizes achieve lower loss than larger ones.
2. Differences in Train Eval Loss. Our main experiment measures the training loss, but since the loss is computed before incorporating the gradient into the model, and because we train for less than one pass over the entire dataset, this is an unbiased estimate of the evaluation loss. is natural ask which models have lower final loss on the training set (after incorporating those examples into the model). To test whether lower physical batch sizes somehow generalize better, whether they also do better on the training loss, we measured the loss of the final trained model on 1M examples from the training set. We focus on the noise-batch ratio of 0.5 in this test. The table below shows that smaller physical batch sizes also have better performance on the already-seen training examples, ruling out this explanation (see Table 3).
Training Set
Batch Size
Cross Cross-Entropy
Accuracy
128
3.586
43.59%
512
3.971
37.27%
2048
4.01
37.55%
8192
4.057
36.73%
Table 3. Loss over the entire training set also better for lower physical batch sizes.
3. Model Size. The main experiment uses BertTiny, which is a relative small model. is natural to ask whether the same behavior would be observed for larger model like BertBase. The figure below shows that the same phenomenon happens for BertLarge, but only for the largest noise-batch ratio. The other two values of noise-batch ratio do not
17

---

Scaling Laws for Differentially Private Language Models
exhibit this behavior, although the middle noise-batch ratio, the trend line suggests there may be crossover point beyond the limits of the axis. Thus, increasing model size seem to influence and mitigate this behavior, but not eliminate it completely. See Figure 9.
275
475
Plyyical
52
128
Gos32 Entopp
275
375
75
Prys Barch
39
Poystal Batch
825
512
512
16
10
67
Iterations
Perations
Iterations
(a) noise-hatch ratio 0.520
(b) noise-batch ratio 0.5
(c) noise-batch ratio 0.5
Figure 9. Cross-entropy loss of BertLarge averaged over trials for different physical batch sizes and noise-batch ratio values.
4. Training Pipelines. It is natural to question whether this behavior is explained by some bug in the training pipeline. We carefully reviewed the implementation and did not find any bugs that could explain this behavior, and also did additional experiments on a totally separate training pipeline based on NanoDO (Liu et al., 2024), where we observed the same qualitative behavior when training 30M parameter decoder-only transformer model with DP-Adam for 32K iterations. The figures below show the smoothed cross-entropy averaged over 3 random trials.
3.0517578125e-05
0.0009765625
batch_ size
batch_sia size
batch_ _izch
4.8
32
32
6.8
32
128
128
128
Cross Entropy
4.0
512
4.6
512
512
2048
6.6
4.4
6.4
3.6
6.2
3.4
4.0
6.0
102
10a
10a
10a
10a
104
102
10a
10a
Iterations
Iterations
Iterations
Figure 10. Loss NanoDO (Liu al., 2024).
C.5. Training Throughput
By looking at intermediates, using single physical batch size, and separating the accounting from the experimentation we greatly reduce the number of experiments to run. However, the of experiments we outline above still very compute-intensive. We utilize TPUv3 pods to run all experiments, and configured the models to pure data parallelism, using more cores for larger models so that each experiment finishes within four to ten hours. BertTiny was trained on 16 TPUv3 cores, while BertLarge was trained on 128. Table 4 provides the training throughputs for all models in our experiments.
18

---

Scaling Laws for Differentially Private Language Models
Model
Params
Steps/sec
Per Core Batch Size
Records / Sec
BertTiny
4.52M
8.959
64
573
BertMini
11.4M
5.494
64
352
BertSmall
29.0M
6.602
32
211
BertMedium
41.6M
4.196
32
134
BertBase
110M
3.621
16
54
BertLarge
335M
2,225
8
17.8
BertMega
729M
1.536
4
6.1
Table 4. Training throughput for various BERT models
C.6. Reproducing non-private scaling laws results
We now confirm that the experimental data collected matches the expected behavior of Hoffmann et al. (2022), specifically that in the absence of noise, the optimal model size and tokens should grow in roughly equal proportion with increasing compute budget. This is true despite our several methodological differences, including: (1) doing per-example gradient clipping, (2) using a different optimizer and not retraining for each number of iterations, (3) using a large physical batch size, etc. The exact Token / Model ratio predicted here is larger than prior work, but that is well explained by the fact that a batch size 1024 examples is well beyond the critical batch size of compute- training (McCandlish et al., 2018).
4.0
Slope: 0.49
Slope: 0.51
Cross Entropy
3.5
Model Size
Iterations
3.0
10
107
2.5
20
1.5t
1.5
10
17
T8
T0
T6
10
10
10
10
1017
1018
1019
Compute Budget
Compute Budget
Compute Budget
Figure 11. Compute-optimal cross-entropy, model size, number of iterations when running DP-Adam with
C.7. Optimal Learning Rates
We now look at the training curves for different learning rates and different noise-batch ratio values. These results generally match expectations and demonstrate that the learning rates chose were selected from the correct regime.
19

---

Scaling Laws for Differentially Private Language Models
Learring Bats
Learning
6.5
0.5*
7.5
0.5
Cross Entoopy
0.57
0.5
0.57
5,5
675
2-5
4.5
0.5
0.5
127
16
10'
10"
12
10
lerations
7.3-
0.5
0.5
Cropp Entropy
0,57
6.3
0.5'
67
12
1P
10
10
10
Banations
(a) noise-batch ratio 0.5
(b) noise-batch ratio 0.5
(c) noise-batch ratio 0.5
Figure 12. Training curves for BertTiny (top) and MertMmdium (bottom) with varying learning rates different noise-batch ratio values.
C.8. Optimal Compute Budget Allocation
In this section, we extend the results from Section 4.1, including results for more settings of the data budget, ranging from N 10th N 10 The full results shown in Figure 13. Our findings qualitatively similar to the ones identified in the main text across different data budgets, but the precise constants may differ.
C.9. Smoothing and Extrapolation
In Figure 14 we visualize how our semi-parametric smoothing approach works. Since each raw measurement is average cross-entropy 1024-100 100 examples, it is naturally noisy estimate of the "true" cross-entropy. Our smoothing strategy ensures the appropriate monotonicity properties are enforced, while matching the overall trend as closely as possible.
D. Caveats on Privacy Calibration
Throughout the work, we have assumed that hyperparameter choices for model training are made against a fixed privacy budget. In particular, we assume the common scenario which the model trainer fixes (c, 6-bbudget and then utilizes privacy calibration algorithm to choose DP-SGD hyperparameter combinations (sampling probability, training iterations and noise scale) which satisfy this privacy budget. Note that in the main manuscript, express this choice in terms of the noise-bated ratio a and the number of iterations T. but this is merely matter of notation. As also noted in the preceding subsection, the choice of sampling probability (and thus the resulting batch size) play important role in determining the final model's s cross-entropy. As described in the recent work of Kaissis et al. (2024), calibrating against fixed (c. 6) (0)-buuger while varying DP-SGD hyperparameters must be done with care: brief, one cannot assume that DP-SGD with different hyperparameters offers the same privacy guarantees despite having the same nominal (e, buddeet.. This is due to the fact that the privacy guarantees of DP-SGD
only be adequately expressed through a privacy profile, that is, collection of (c, (()) tuples. In simple terms, two DP-SGD algorithms share an buudge,, that is, offer the same privacy guarantees for specific 8 while offering (sometimes drastically) different privacy guarantees at different value of 6. As also described in the aforementioned work, varying the sampling rate (and thus batch size) has drastic impact on this difference in privacy guarantees. The authors the aforementioned work thus recommend reporting the excess vulnerability that DP-SGD algorithms incur with respect to each other when they replace one another in a workflow. We refer to the aforementioned work for technical details. Here, we demonstrate that meaningful differences can indeed arise between models calibrated satisfy the same (e, bbuddet.
Exemplarily, fixed privacy budget of (c, (8, 10 10-8) for specific fixed compute budgets and model sizes while varying the batch size (and adjusting the noise to maintain the privacy budget). We then computed the scaling-law predicted cross-entropy and the vulnerability of the models against membership inference attack (MIA) adversaries measured in
20

---

Scaling Laws for Differentially Private Language Models
835en16
Miwcay Budger
Pitwayy Sudget
1.59n10
100003
64
Batch Sttw
Beraliens
2.5e277
10000
Budgot
100020
1.1ex277
10020
16
15ei00
10
12
10
Compute Budget
10
12x
127
10
10
102
15th
1010
16
Compute Budget
16th
1ith
1d
1d
1030
32"4
1517
1d
Compute Budget
10M
10P
1d
1d
8360+10
Budgee
1o+07
1e405
16
Batth 50w
64
Reraloon
64
Mode
2
1.Me107
4.6+00
167
1
Budget
197
16
1014
1d0
16
Compute Budget
10
1059
100
1dit
100
3244
130
1d
Compute Budget
10
128
120
1dit
1da
3.35erc0
Pfrray Budget
Budget
1c+07
100010
1.09H00
4 4lex47
2.wn27
64
16
Burth Sow
120000
Seratione
10020
10002
1.16x427
0e+00
10%
10
1
1000
1201
1dtt
100
1020
Compute Budget
Compute Budget
Compune Budget
Intaayy Badget
1e403
Frivacy Budget
Pitwacy Budget
16
Batch Size
Herstons
Model
$3e+27
10020
1.10+077
4.erče
10
10
10
100
12"
10
1007
10
12"
10
10
12
10a
Compute Budget
Compute Budget
Compute Budget
(a) Model Sizes
(b) Batch Size
(c) Iterations
Figure 13. Compute optimal model-sizes, batch sizes, and iterations for varying privacy budgets and compute budgets, and data budgets. Each row of plots corresponds to a different budget
105, 10 100 and 10" respectively, Each corresponds to the minimum value of that hyper-parameter that achieves within 1 % of the optimal cross-entropy across all constant-compute training configurations. The shaded region corresponds to full range of possible values for that hyper-parameter that are optimal within 1%.
terms of MIA advantage (Yeom et al., 2018). We note that MIA advantage is proxy metric for other attacks such as reconstruction attacks and related to the A --divergence which quantifies vulnerability described in Kaissis et al. (2024). Figure 15 demonstrates the phenomenon.
Note that in all three cases, it is possible to achieve virtually the same cross-entropy (blue, left vertical axis) while controlling the MIA advantage by judiciously choosing the batch size. Conversely, it is also possible to incur unduly high vulnerability without substantial decrease (or sometimes even an increase) in cross rros-entrropy through poor choice of batch size. As an auxiliary finding, we note that the relationship between cross-entropy and batch size follows the trend observed in De et al. (2022). In brief, there is a Pureto optimal batch size beyond which both the cross-entropy and the excess vulnerability can only become worse (larger). We stress that the models shown here all satisfy the same nominal (c, -b-udger but exhibit (substantial) differences vulnerability against least a subset of adversaries which may pass unnoticed if only reporting single (c. DDP guarantee. We thus recommend practitioners to monitor changes in excess vulnerability that may arise due to hyperparameter tuning and report them alongside the (e, -buugtt to which DP-SGD has been calibrated.
21

---

Scaling Laws for Differentially Private Language Models
3.5
Raw Data
Raw Data
Smoothed + Extrapolated
3.4
Smoothed
Cross Entropy
Cross Entropy
3.3
3.2
3.5
10a Iterations
10
10"
10
10
Noise Batch Ratio
(a) noisc-batch ratio 0.5
(b) T = 32000
Figure 14, Demonstration our semi-parametric smoothing on BertTiny.
02
0.422
5.75
5--
02
0.375
3.25
5.00
0.31
020
0.155
4.75
0.22
Cooos
0.30
4.50
0.36
0.300
0.273
1.000 10
0.12
0.352
302
104
12t
137
104
101
Soich Size
Batcs 5ime
Suxc Suz
(a)
(b)
(c)
Figure 15. Varying the batch size (horizontal axis, log-scale) has drastic effect on excess vulnerability (measured as MIA advantage, red, right vertical axis) for models with fixed compute budget and size and a fixed privacy budget of (c, = (8, 10_ (a): Compute budget: 6- 6-10"7, model size: 4000 000. (b) Compute budget: 6.3- 33 model size: 200 000 000. (c) Compute budget: 2.5-10 model size: 200 000 000. The baaling-law-predieee cross-entropy is plotted on the left vertical axis in blue.
E. Parametric Scaling Laws
Previous work on (non-private) LLM scaling laws use fully parametric form predict the cross-entropy loss based on several key factors. For example, the "Chinchilla" scaling law (Hoffmann et al., 2022) be parameterized as follows:
A
B
[ Tiokens)
E
+
n
purams
n tokens
B
In this section, we explore similar methodology to fit fully parametric form of scaling law in the setting of private training. Following the notation of this paper, we define parametric form based on the following key factors: the model size M, the number examples N and the noise-batch ratio o. Note our notations are slightly different from Hoffmann al. (2022), and we use number of examples instead of number of tokens as a more relevant quantity in private training.
We consider several variations of parametric forms. The first is naive extension of the Chinchilla scaling law, by adding additional term involving the noise-batch ratio:
A
(M,,
+C7
(I)
M
N3
We did not put T in the denominator because the loss increases with the noise-batch ratio. Following Hoffmann al. (2022), we estimate the coefficients (E, A, B, C, {, by minimizing the Huber loss (Huber, 1992) between the predicted and the observed loss using the L-BFGS algorithm (Nocedal, 1980), and we try multiple different initializations and choose the best fit. We restrict the curve fitting data to only the subsets of data points with more than 100, 000 training iterations, notsh-batch ratio larger than 5 x 10-7, and ignore points with very high cross-entropy loss (> 8).
Figure 16 shows the optimal fit. We observe that the prediction is generally accurate for low loss value ranges. However, the prediction starts to diverge at high loss value ranges, corresponding to runs with high noish-batch ratio. This partly due to
22

---

Scaling Laws for Differentially Private Language Models
15
Ossenved
Otsersed
Figure 16. Parametric private scaling law L1 from Equation (1). Optimal fit with a 0.71, 8 12.87, y = 0.19. The two pannels show the same plot of observed cross-entropy loss against the predicted loss from the scaling law, except the data points colored differerently, according to the model size and noise-batch ratio, respectively.
10P
Ch00 100
Count Ernopy LOG0
Mode Batch Ratio
RUSS
Figure 17. Relation between the noise-batch ratio and the cross-entropy loss. (left) The data plotted in log-log scale. (right) The data transformed according to simple rule Equation (2).
plotted linear scale, where noise-batch
the fact that the noise-batch ratio does not impact the loss a log-lineat fashion, as shown on the left panel Figure 17.
Therefore, the parametric form of Equation (I) cannot capture the relation accurately. Instead, we observe S-shaped curves in the log-log plot. To account for this, we apply simple transform to the noise-batch ratio
log + )
T - sigmoid
1.6
The right panel of Figure 17 shows an approximately linear relation after this transformation. Furthermore, we observe that the relation between the noise-batch ratio and the loss changes with the model sizes.
After incorporating those observations, consider alternative variant of private scaling law parameterization:
A
B
Co
N.) 2
Ma
N
Ma2
The optimal fit according to this parameterization shown in Figure 18. We observe that the predicted loss matches with
the observed loss better than the previous parameterization in Figure 16.
19
Procrne Look
-15
15
Figure 18, Parametric private scaling law of La from Equation (3), Optimal fit with a = 0.47, 0.12, 0.95, 02 = -0.07. The two pannels show the same plot of observed cross cooss-y loss against the predicted loss from scaling law, except the data points are colored differerently, according to the model size and noise-batch ratio, respectively.
23

---

Scaling Laws for Differentially Private Language Models
Ritee baiil kit fuend
10"
Note
Hole Deice fatt 7.5m00
Hode fatt 3.100
Hote
Hole Deice fatt 2.w00
Hoie Falls 1.002
New Kort Kutk 1-4-12
332 304
30"
334
309
Figure 19. Optimal model sizes under according to the parametric private scaling law in Equation (3).
In the Chinchilla parameterization of scaling law for non non-private LLMs, the optimal model size under a certain compute budget (approximately represented by Puokem))s)Puueeem))) can be directly solved and takes a power-law form (Hoffmann et al., 2022, Equation (4)). our case, the parameterization is more complicated, for given compute budget and noise-batch ratio, we use scipy. scipy.optimize optimize. wwn.mmiiiiiiiizeee.......ee.. www.imiieesccaalleeeeeeeeeeeeeee
find the optimal model size that minimizes f.2. The results are plotted in Figure 19. We observe that the slope is lower for curves with larger noise-batch ratio, indicating the challenges to scale model sizes under heavy DP noises. As the noise decreases, the curves shift up and the slopes increase, approaching towards the non-private Chinchilla scaling law shown in dashed line.
While fully parametric scaling law can be easier to interpret and understand, as noted above, there is not simple log-linear relation between the loss and the noise-batch ratio. Our sigmoid based transformation (and the coupling with the model size) improved the tightness of the fitting. But the transformation not designed in a very principled way. As a result, we opt to use the semi-parametre fitting in Section in the main analysis of our results. We also leave the exploration of other alternative parametric fitting such fitting a ö-depending delta term top of non-private scaling law for future work.
24


Reviews:
[{"review_id": "CM7nZyIk5e", "reviewer": "Reviewer_qUEy", "summary": "This paper explores the scaling laws applicable to the training of masked language models under differential privacy (DP) constraints. The authors establish that traditional scaling laws, which do not account for privacy considerations, are suboptimal when applied in DP settings. Key findings: optimal model size with DP is generally much smaller compared to non-private models.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the careful review and feedback, and for recognizing the various strengths of this work.  Below we respond some of the questions / criticisms:\n\n**[Masked Language Modeling vs. Other Tasks]** The reviewer is correct to point out that our focus on BERT models raises questions about generality of the results with respect to other model architectures, like decoder-only models used in many modern GenAI applications.  We acknowledged this limitation in Appendix A.  We further provide evidence in Section 3.7 that despite the differences in training setups with prior (non-private) scaling laws work, we are able to reproduce the main finding of Hoffman et al., i.e., that data and model size should be increased in equal proportion with increasing compute.  This provides evidence that our results should translate to other settings.  \n\nOur choice to study BERT models in this research was based on a few considerations:\n* Established pre-determined model sizes and configurations (https://github.com/google-research/bert) \n* Maturity of DP training support in distributed ML libraries (at the time this research started): We decided to use PaxML (https://github.com/google/paxml/) to conduct this research, since it supports large-scale training of transformer models coordinated across many accelerators, and supports per-example gradient clipping routines which are necessary to use DP-SGD that can run in distributed settings.  This open source library does not feature an easily-forkable implementation of a standard decoder-only model and thus would require a large effort to re-engineer purely from the available primitives.\n\nWe believe that future research in this space should probably study (now) open-source decoder-only models like NanoGPT (https://github.com/karpathy/nanoGPT).  We believe that the findings we showed are still useful and informative in their current form, however.  \n\n**[Pretraining vs. Finetuning]** The reviewer is correct to point out that with DP, finetuning has some advantages over pretraining, and would be interesting to study.  We acknowledged this limitation in Appendix A, and agree with the reviewer that this would be an interesting direction for future work. \n\n**[Evaluation task diversity]** This is a valid criticism, and something we will be sure to add to our Limitations section in revision, and highlight this as an important consideration for future work.  \n\n**[Other weaknesses]** We have updated our references in the uploaded revision to include a more expansive set of citations. If the reviewer is aware of any relevant citations that were missed, we are happy to consider and add these as well. We have fixed the other minor issues identified during review, including the references on line 15 and the notational errors.  Thanks for the feedback!", "claims_and_evidence": "The paper’s claims are supported by clear and methodical evidence, with experiments spanning model sizes, noise levels, and compute budgets. Limitations are transparently addressed, and the findings provide actionable insights for DP training. \n\nI think it should be clearly stated that these findings are specific to MLM earlier than sec2, and the authors should make educated guesses about the transferability of the findings.", "methods_and_evaluation_criteria": "Using BERT models (with varying sizes) and masked language modeling is appropriate, as BERT is a standard architecture for studying scaling laws. DP-Adam with per-example gradient clipping follows established DP-SGD practices. However, I’m concerned that results may not generalize to decoder-only models or tasks like autoregressive language modeling, which are common in modern LLMs. The fixed sequence length (512 tokens) and focus on pre-training (not fine-tuning) also limit direct applicability to real-world deployment scenarios.  \n\nEvaluation metrics like cross-entropy loss and compute savings are standard and meaningful, but the lack of diverse tasks/datasets and the reliance on only (ϵ,δ)-DP constrain generalizability.", "theoretical_claims": "The claim that non-private scaling laws are suboptimal under DP is supported by empirical evidence shown in figures. Optimal model sizes are smaller under DP, it is empirically proven.", "experimental_designs_or_analyses": "The experimental design is methodologically generally rigorous in all the experiments for the tested settings (BERT models, masked LM, ≤778M parameters). However, critical assumptions—fixed batch sizes, reliance on (ϵ,δ)-DP, and limited architectural scope—constrain broader validity.", "relation_to_broader_scientific_literature": "The contribution is quite useful as it starts providing a guidance on scaling laws in this field. However, the impact is limited to the chosen setup and there are doubts on generalizability.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths\n* The paper is rich with practical insights, eg line 91 right, sec3.1, annex B2, annex D.\n* The computational exploration is particularly broad and thorough, which gives the claims a solid grounding. \n\nWeaknesses\n* The caveat that the main findings are limited to BERT should be more prominent: neither title nor abstract address this. \n* Citations in sec1 and sec 2 seem biased towards work by authors affiliated with Google without apparent reason. This should be corrected. \n* Line 015 right: both citations are to blog posts; is there no better source?", "other_comments_or_suggestions": "* Algo 1 line 64: should that not be $=\\bar{g}+...$ instead of $=g+$?\n* left col line 84 which typically added. line 86 is\n* line 121 left: different than\n* line 718 is linear\nThe following recent refrence is not essential but might be useful in sec1 or sec5 as it gives a reasoned overview: Miranda+ 2024 https://arxiv.org/abs/2408.05212", "questions_for_authors": "How do these findings transfer to causal language modeling?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KtZi8xoNx9", "reviewer": "Reviewer_1Cfq", "summary": "The paper formulates the problem of identifying scaling laws for differentially private training of Bert models, as identifying the optimal training configurations (model size, batch size, noise-batch ratio, and iterations) given fixed data, compute, and privacy budget. Here clipping thresholds and stepsize are fixed as constants across all configurations. Their approach contains three steps.\n\n1. For a fixed but reasonably large physical batch size, fit a function $L(M, T, \\bar{\\sigma})$ that estimates the training loss of an M-parameter model after T iterations with a noise-batch ratio of $\\bar{\\sigma}$, by repeating the training for different configurations and smoothening and interpolating the training results.\n2. For other batch size $B$, assume that as long as $M, T, \\bar{\\sigma}$ are fixed, their training loss are equal.\n3. For each fixed data, compute, and privacy budget, enumerate all possible training configurations (batch size, noise-batch ratio, and iterations), and identify the configuration that minimizes the estimated training loss $L(M, T, \\bar{\\sigma})$.\n\nThey then draw some insights from the behavior of the fitted scaling laws.\n1. There is a small but consistent trend that with larger privacy budgets, one should train a larger model with a smaller batch size and for more iterations than one would train with a smaller privacy budget\n2. Optimal model sizes under privacy are much smaller than predicted by non-private scaling laws. \n3. Given a fixed privacy/data budget, there is an inflection point where increasing the compute budget provides little to no benefit.\n4. The ratio of the number of training tokens to model size increases with computing budget, especially for smaller privacy budgets. This matches the flat token-to-model ratio as predicted by the prior work for non-private training (Hoffmann et al. (2022)).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the careful review and feedback, and we are glad you liked the paper.  Below we respond to the main questions / critiques:\n\n**[Missing details]** We will be sure to update our discussion around Figure 1 and Section 4.5 to clarify what is shown.  \n* Regarding Figure 1, we do not use a constant noise-batch-ratio here; instead it is varied through the privacy budget, batch size, iterations, and data budget (as shown in the diagram in Figure 2).\n* In Section 4.5, we fix M and T and only consider how the noise batch ratio (sigma / B) changes with respect to the Privacy, Data, and Compute budget.\n\n**[Fixed batch size assumption]** Our modeling assumption that the training loss primarily depends on the batch size through the noise batch ratio (Figure 2) is a limitation we acknowledge in Appendix A.  However, this modeling assumption was necessary to make this work practically feasible, as adding yet another independent variable to consider greatly increases the number of experiments and accelerator hours needed to conduct this research.  \n\nAs the reviewer points out, our ablations in Appendix C.3 and C.4 do study the effect of this variable in isolation, although we believe it will be interesting to revisit this in future work to better understand the phenomenon we observed and under what conditions it manifests. \n\n**[Fixed clipping + learning rate]** We lean on findings from prior work to establish a reasonable default DP training setup.  In particular, [1] advocates for using a small clipping threshold where all (or nearly all) gradients are clipped.  We used a clipping threshold of 1, and checked that this resulted in > 90% of gradients being clipped.  We also used gradient normalization as [1] also suggests, since it decouples the clipping parameter from the learning rate. While more careful tuning of this parameter could help somewhat, our primary goal in this work was to focus on variables that relate directly to the compute budget.\n\nRegarding learning rate, we actually used three learning rates, which were chosen based on comprehensive ablations discussed in Appendix C.7 and Figure 12.  \n\n[1] De et al., Unlocking High-Accuracy Differentially Private Image Classification through Scale", "claims_and_evidence": "In general, the paper is well-written with clear claims and supporting details. However, there are a few places that are harder to interpret due to missing details. \n1. In Figure 1, is the noise-batch-ratio fixed across all settings? If so, why is it reasonable to fix noise-batch-ratio?\n2. Section 4.5, how is the noise-batch ratio computed here? Is it corresponding to the optimal configurations predicted by $L(M, T, \\bar{\\sigma})$ given fixed $M$ and $T$? If so, why is it reasonable to fix $M$ and $T$?\n> We analyze how the noise-batch ratio behaves as a function of privacy budget (as measured by ε), compute budget (as measured by B), and data budget (as measured by N).", "methods_and_evaluation_criteria": "The method is interesting, and the only limitation that I see is the assumption that the training loss curve under different batch sizes is similar, as long as $M, T, \\bar{\\sigma}$ are fixed. The authors also discussed this in Appendix C.3. and show that this assumption may underestimate the benefit of smaller batch sizes when the noise-batch ratio is large.\n\nAnother minor point that lacks clarify, is why it is reasonable to fix the learning rate and clipping threshold across all settings, rather than, e.g., using a larger clipping threshold for larger models.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The experiment results are comprehensive and interesting. Other than a few minor clarity issues as discussed in Claims And Evidence, I only have one doubt regarding what the authors call \"improvements in the noise-batch ratio\" in Section 4.5 and \"diminishing returns in terms of noise-batch ratio\" in Section 4.1. I'm a bit confused about whether we are interested in the increase/decrease of noise-batch-ratio as worsening/improvement of training, rather than the direct eval loss. Maybe it is reasonable under fixed choices of other factors, e.g., training iterations, batch size, and model size, it would be good if the authors could clarify these points.", "relation_to_broader_scientific_literature": "The paper offers a nice study of scaling laws for differentially private training and discusses many connections/differences compared to the literature on scaling laws for standard training.", "essential_references_not_discussed": "Related works are discussed thoroughly. However, some sentences missed citations when referring to prior works. E.g., Line 327 says \"becomes nearly flat as predicted by the prior work\" without any references.", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "See Claims And Evidence and Experimental Designs Or Analyses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dpCLuWCyQ9", "reviewer": "Reviewer_U13Y", "summary": "This paper formulates scaling laws that accurately reflect the complexities of training Differentially Private (DP) Large Language Models (LLMs).", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the feedback, and thank you for sharing ideas for specific ways to improve the paper. We are glad that you found the paper well-written and thorough, but we respectfully disagree with some of your comments, and respond to the main critiques below.\n\n**[Motivation]** There are several complementary reasons why we believe DP scaling laws for pretraining are well-motivated and of interest to the privacy community. \n\n* *Importance of pretraining privacy*: For current frontier models, and pretraining specifically, there is increasing need for data privacy, e.g., due to risks of memorization in large models. This makes DP pretraining research relevant as pretraining data is often scraped from the web (and can contain sensitive personal information [1]). The need for privacy will further increase if companies start training on sensitive user data as they run out of public datasets to train on.\n\n* *Addressing utility challenges*: Utility degradation due to DP actually serves to motivate our work. With enough data and compute, and well-tuned mechanisms, the performance degradation due to DP can be mitigated to a large degree [2]. We believe scaling laws will help in this mitigation.\n\n* *Broader Context*: Privacy-preserving machine learning falls squarely within the [ICML call for papers](https://icml.cc/Conferences/2025/CallForPapers) under the Trustworthy Machine Learning category, and the best paper from ICML 2024 highlighted DP pre-training as an important future direction [1]. Several large organizations have invested in this space already [2,3,4,5,6]. \n\n[1] Tramer et al., Position: Considerations for Differentially Private Learning with Large-Scale Public Pretraining\n\n[2] De et al., Unlocking High-Accuracy Differentially Private Image Classification through Scale \n\n[3] Xu et al., Federated Learning of Gboard Language Models with Differential Privacy\n\n[4] Anil et al., Large-Scale Differentially Private BERT\n\n[5] Pelikan et al., Federated Learning With Differential Privacy for End-to-End Speech Recognition\n\n[6] https://www.microsoft.com/en-us/research/group/privacy-preserving-machine-learning-innovation/\n\n**[Encoder-decoder Models vs. Decoder-only models]**\nOur choice to study BERT models in this research was motivated by previous research on pre-training with differential privacy [3]. We agree that our focus on BERT models raises questions about generality to other model architectures. We acknowledged this limitation in Appendix A. We further provide evidence in Section 3.7 that despite the differences in training setups with prior (non-private) scaling laws work, we are able to reproduce the main finding of Hoffman et al., i.e., that data and model size should be increased in equal proportion with increasing compute. This provides some evidence that our results should translate to other settings. This observation is also consistent with [7] who found qualitatively similar scaling between encoder-decoder and decoder-only architectures. See also our response to Reviewer qUEy.\n\n[7] Yao et al., Towards Neural Scaling Laws for Time Series Foundation Models\n\n**[BertMega]** BertMega is the only non-standard model config we used, to also explore a larger BERT model than the ready-made configs listed at https://github.com/google-research/bert. We will note this explicitly in the revision.\n\n**[1B-2B Models]** This is a good idea, and something we discuss in the limitations section (Appendix A). Scaling to larger models introduces significant engineering challenges, most notably handling model parallelism. For such models, weights—and thus their corresponding gradients—are typically sharded across devices. The gradient clipping step in DP-SGD requires storing and synchronizing per-example gradients across layers and devices for clipping, which necessitates substantial changes to existing implementations. While some work has explored this in the context of LLM fine-tuning [8], these challenges remain for training LLMs from scratch. We will be sure to highlight this as an additional challenge to scale DP training to the billion-parameter scale (in addition to DP scaling laws) in the revision.\n\n[8]: He et al., Exploring the Limits of Differentially Private Deep Learning with Group-wise Clipping\n\n**[Evals on downstream tasks]** Thank you for raising this point. We note that performance on downstream tasks are generally too noisy for scaling law studies themselves, as a result, we follow the previous work on LLM scaling laws and focus on the crossentropy loss. We agree that beyond the scaling law studies, it would be valuable to show some downstream evaluations on large (maybe 1B-2B models) DP pre-trained models. We will highlight this as an important direction for future work in revision.\n\n**[Other suggestions]** We will incorporate your additional comments and suggestions to improve the presentation of our figures and tables, and clarify the relevant parts of the text in revision. Thanks for the suggestions!", "claims_and_evidence": "Yes, the claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "It is not clear whether the proposed methods make sense for the problem or application at hand.", "theoretical_claims": "This paper does not include any proof.", "experimental_designs_or_analyses": "Yes, I checked the soundness/validity of any experimental designs or analyses.", "relation_to_broader_scientific_literature": "I believe the key contribution of the paper lies in conducting numerous experiments.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1. The writing of this paper is excellent, and it is easy for me to follow this paper.\n2. The authors conducted thorough experiments to gain insights into DP LLM training.\n\nWeaknesses:\n1. The motivation of this paper is not clear. I believe that OpenAI, Anthropic, XAI, Google, and DeepSeek are unlikely to use differential privacy for pre-training, as it may negatively impact model performance. Therefore, I am not sure whether we need such a scaling law in reality.\n2. More and more people use decoder-only models such as Qwen or Llama. I am unclear why the authors chose to use Masked Language Modeling (BERT) in this paper.\n3. Could you show me other papers that use BertMega model? From Huggingface, I did not find the config of the BertMega model.\n4. Scaling laws are used to predict the behavior of large models. Therefore, I think the authors need to train a 1B-2B to evaluate the correctness of their scaling laws.\n5. Furthermore, could you show me the results of your pre-trained models over downstream tasks?", "other_comments_or_suggestions": "1. As for Figure 1, could you polish the y-axis? For example, Figure 1(b) sometimes uses scientific notation, and sometimes does not use it.\n2. As for Figure 1(b), I guess that you try to vary the batch size, but the legend shows that you try to vary the privacy budget.\n3. As for Figure 3(b), it is not clear to me that the compute budget increases when you fix the data budget and model size. My understanding is that FLOPs = 6ND, where N is the model size and D is the number of tokens. \n4. As for Table 2, we usually use 35M to represent the model size instead of $3.5 \\times 10^{7}$. We also use 512 for batch size and 1800 for iterations.", "questions_for_authors": "See strengths and weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper provides a careful empirical study of how differential privacy affects language model scaling laws. The authors demonstrate how scaling behaviors differ significantly from non-private regimes. Reviewers broadly commended the thoroughness of the experimental design and appreciated how the results could help practitioners allocate compute more effectively under privacy constraints, despite some reservations regarding real-world adoption of DP pre-training and a preference for decoder-only architectures in modern LLM use cases.

Nevertheless, the paper is clair and rigorous, and its relevance to DP gained strong support. The authors also addressed concerns in their rebuttal. Overall, this is a nice contribution.
Decision: Accept (poster)