Sparsity-Preserving Differentially Private Training of Large Embedding Models
Authors: Badih Ghazi
Yangsibo Huang
Pritish Kamath
Ravi Kumar
Pasin Manurangsi
Amer Sinha
Chiyuan Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: Differential Privacy
Recommendation Systems
Embedding Models
Efficient Machine Learning
TL;DR: We present two novel approaches to preserve gradient sparsity in DP-trained large embedding models while maintaining their utility.

Abstract: As the use of large embedding models in recommendation systems and language applications increases, concerns over user data privacy have also risen.  DP-SGD, a training algorithm that combines differential privacy with stochastic gradient descent, has been the workhorse in protecting user privacy without compromising model accuracy by much. However, applying DP-SGD naively to embedding models can destroy gradient sparsity, leading to reduced training efficiency. To address this issue, we present two new algorithms, DP-FEST and DP-AdaFEST, that preserve gradient sparsity during the private training of large embedding models.  Our algorithms achieve substantial reductions ($10^6 \times$) in gradient size, while maintaining comparable levels of accuracy, on benchmark real-world datasets.

Sparsity-Preserving Differentially Private Training of Large Embedding Models
Badih Ghazi
Yangsibo Huang"
Pritish Kamath Google Research Mountain View, CA pri writishk@google.com com
Google Research Mountain View, CA
Princeton University Princeton, NJ y yangeibo@prinnetoor edu
wwwhhhhhz@@gmail..... com
Ravi Kumar
Pasin Manurangsi
Amer Sinha Google Research Mountain View, CA
Chiyuan Zhang
Google Research
Google Research Mountain View, CA
Google Research Mountain View, CA
Mountain View, CA
ravi. www0kmmmll...mmmmmmmmmmmmm...m.l
pasindgoogle.com
amersinhaQgoogle.cm
chiyuan@google.com
Abstract
As the of large embedding models in recommendation systems and language applications increases, concerns over user data privacy have also risen. DP-SGD, training algorithm that combines differential privacy with stochastic gradient descent, has been the workhorse in protecting user privacy without compromis- ing model accuracy by much. However, applying DP-SGD naively to embedding models can destroy gradient sparsity, leading to reduced training efficiency. To address this issue, we present two new algorithms, DP-FEST and DP- AdaFETT that preserve gradient sparsity during private training of large embedding mod- els. Our algorithms achieve substantial reductions (10%x) in gradient size, while maintaining comparable levels of accuracy, on benchmark real real-world datasets.
Introduction
Large embedding models have emerged as a fundamental tool for various applications recommenda- tion systems [CKH+16, 16, WFFW17, GTY+ 17, 77S+18, NMS 19, WSC+21] and natural language processing [MSC+ 13, PSM14, DCLTT19.. Those models map categorical or string-valued input at- tributes with large vocabularies to fixed-length vector representations using embedding layers, which enable integrating non-numerical data into deep learning models. These models are widely deployed personalized recommendation systems and achieve state-of-the-art performance language tasks such as language modeling, sentiment analysis, and question-answering
In the meantime, the of large embedding models raises significant concerns about privacy violations, as often entails the processing sensitive individual information such as personally identifiable information, browsing habits, and dialogues [CTW*21 21, HSC22, GHZ GHZ+ Various techniques have been proposed to enable private data analysis. Among those, differential privacy (DP) [DMNS06, DKM+06] is widely adopted notion that could formally bound the privacy leakage of individual user information while still allowing for the analysis population-level patterns. For training deep neural networks with DP guarantees, the most widely used algorithm is DP-SGD +ACG+ 16]. It works by clipping the per-example gradient contribution. and noising the average gradient updates during each iteration of stochastic gradient descent (SGD), DP-SGD has
*This work was partially done while the author was interning at Google Research.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

demonstrated its effectiveness in protecting user privacy while maintaining model utility in variety of applications [PTS * LTLH21, DFAP21, TB21, KCS+ 22, INB*22. DBH 22, DGK 23].
However, applying DP-SCD to large embedding models presents unique technical challenges. These models typically contain non-numerical feature fields like user/product IDs and categories, well as words/tokens that are transformed into dense vectors through an embedding layer. Due to the large vocabulary sizes of those features, the process requires embedding tables with substantial number of parameters. In contrast to the number of parameters, the gradient updates are usually extremely sparse because each mini-batch of examples only activates a tiny fraction of embedding rows. This sparsity is heavily leveraged for industrial applications [GGDS19, WWL 22] that efficiently handle the training of large-scale embeddings. For example, Google TPUs [AF22], a family of accelerators designed specifically for training large-scale deep networks, have dedicated hardware such as the SparseCore [JKI.+23] to handle large embeddings with sparse updates. is shown [Sna22] that this leads to significantly improved training throughput compared to training on GPUs, which did not have specialized optimization for sparse embedding lookups the time. On the other hand, DP-SCD completely destroys the gradient sparsity as it requires adding independent Gaussian noise to all the coordinates. This creates a road block for private training of large embedding models as the training efficiency would be significantly reduced compared to non-private training.
Our contributions. In this paper, we present new algorithms that target the issue of diminished gradient sparsity encountered when employing DP-SGD with large embedding models due to the addition of dense noise; this issue represents a significant practical obstacle for leveraging. during DP training, the hardware accelerators commonly used in non-private training of large embedding models. While there are previous studies of the abstract problems of sparse vectors in DP, we study the practical problem of privately training large embedding models, and through systematic evaluations, demonstrate that significant gradient size reduction is achievable in DP training of real-world large-scale embedding models (e.g., recommender systems, language models with 30M embedding parameters) with minimal loss in utility. Our contributions can be summarized as follows:
b We propose two new algorithms preserve gradient sparsity in DP-training of large embedding models while maintaining their utility (Section 3). The first algorithm, Filtering- Enabled Sparse Training (DP-FEST), selectively adds noise to pre-selected embedding rows during training. To improve the adaptivity to varying training dynamics across mini-batches and to accommodate online training, our second algorithm, Adaptive adaptively computes a gradient contribution map for each mini-batch, and only injects noise into the gradients of top contributors (in a manner that complies with the DP requirements).
Entering-EEablled Sparse Training IDP-AdaFEST
We demonstrate the effectiveness of our algorithms on four benchmark datasets for online advertis- ing and natural language understanding. Our results indicate that these algorithms can meet wider range of application-speea specific demands for utility and efficiency trade-offs at a more granular level (Section 4.2). Notably, they achieve substantially sparser gradient, with a reduction in gradient size of over 10 x compared to the dense gradient produced by vanilla DP-SGD, while maintaining comparable levels of accuracy.
We further showcase the potential applicability and effectiveness of our algorithms to the setting of online learning with streaming data, a common setup in practice, which highlights the versatility and broad impact of our algorithms (Section 4.3).
2 Preliminaries
2.1 Embedding Layers and Gradient Sparsity
Models with large embedding layers widely used in application scenarios that need to process high- dimensional sparse inputs such as words/tokens in language models [MSC 13, PSM14, DCLTI9] and categorical features associated with users or items in recommendation systems [CKH 16, WFFW17, G Z 18, NMS 19, WSC+21].
Let be the number of possible values of an input feature (aka vocabulary size in text models); the feature values are also feature buckets (string-valued categorical features are commonly preprocessed via hashmap bucketization). An input x with the ith feature value can be represented as a one-hot vector e; E RRC that on the ith coordinate and everywhere else. Let W E RRexd be the parameters of the embedding layer (i.c., the embedding table), where the output dimension d is the

---

Spanity of Embbading Gradions
1.000
Feature Embedding Location Lookup
Embedding for York
Ansterdax
0.900
Bei-ing
Location:
Calio
0.885
Catagrical Feature-3
Cateprical Fastume-2
New York
0.950
Catepencal Fasture-3
Categrical Fasture44
Key York
0.075
Canagoiial Faature3
osto
All Categorical Featurea
102
Batch Size
(a) An illustration of embedding lookup.
(b) Embedding gradient sparsity in Ads model.
Figure I: Illustration embedding lookup (a) and gradient sparsity the Criteo pCTR model (b). We analyze the gradient sparsity, averaged over 50 update steps, of the top five categorical features (out of a total of 26) with the highest number of buckets, as well as the sparsity of all categorical features. For more information about this dataset and model, see Section 1.1.1.
embedding size. The output of the embedding layer is computed Due to efficiency considerations, in practice, embedding layers are rarely implemented via matrix multiplication (matmul) on one-hot encoded inputs.
W x, which is a linear map.
For a one-hot input x = e;, the embedding output is the ith row of the embedding table: z = [] Figure la illustrates the embedding table lookup operation. Let 0C/0z E gd be the partial derivative of the loss with respect to the embedding output. The gradient of the embedding table is VW x @ 0C/Oz, where & is the outer product. For a one-hot input x = C₁. the result of this outer product a sparse matrix whose ith row equals 0L/Oz. For mini-batch SGD training. the number of non-zero rows in the batch averaged gradient of the embedding table is upper bounded by the batch size, which is typically orders of magnitude smaller than c. Consequently, the gradient sparsity (the fraction of zero gradient coordinates) for large embedding models is very high (Figure 1b).
Due to this structured sparsity, in practice, both forward and backward computations of an embedding layer are implemented efficiently with gathers and scatters', without expensive matmul. This difference from vanilla linear layers is crucial in real-world applications as the embedding tables are usually very large, with the vocabulary size e ranging from tens of thousands (language models) to millions (recommendation systems). Moreover, in some large recommendation system models, there are hundreds of different categorical features, each with a different embedding table. Therefore, maintaining the gradient sparsity is critical for any training process.
For notational simplicity, the description above focuses on single-variate features, which only activate one feature value at a time. In practice, multi-variate features activating multiple values are also used. In this case, it is common for the embedding layer to output the average or sum vectors corresponding to each activated value. We also note that in recommendation systems, each categorical feature generally has a separate embedding table; But for text models, all the words/tokens in an input document share the same embedding table.
2.2 Differentially Private Stochastic Gradient Descent
Differential privacy (DP) [DMNS06, DKM 06] is mathematical framework for ensuring the privacy of individuals in datasets. It can provide strong guarantee of privacy by allowing data to be analyzed without revealing sensitive information about any individual in the dataset. Formally, a randomized algorithm A satisfies (e, DPP if for any two neighboring datasets D and 'D' (i.e., datasets such that one can be obtained from the other by adding/removing one example), and any subset of outputs, holds for privacy parameters Eso and å E [0, 1) that
Pr(A(D) E S] S e PP[[ADD)) S
DP Stochastic Gradient Descent (DP-SGD) [ACG+ 16] is a recipe for training a deep learning model with DP by modifying the mini-batch stochastic optimization process through the of per-example gradient clipping and Gaussian noise injection. When training an ML model f parameterized by with the per-example loss function e(., .)² dataset D, each optimization step involves randomly
GGather/scatter refers to memory addressing technique that enables simultaneous collection (gathering) from or storage (scattering) of data to multiple arbitrary indices. Refer to this wiki page for more details.
The specific loss depends the particular task and model (e.g. cross-entropy loss for classification).
3

---

sampling mini-batch B₁. Given B₁, DP-SCD starts by computing the per-example gradient for each (24. B,, where z; is the feature vector and yi is the corresponding label, as follows:
(b) Vo, (x:). v6).
It then clips the gradient mnrmr to a maximum ---nrrm of C as:
B (Ti, vole := (a Mc)
max |1rr(09
Finally, it produces the private gradient gi by injecting Gaussian noise into the sum of the clipped per-example gradients
8 B (E (c,, w.lc + N (0,
where WN 0²) is a Gaussian distribution with mcan O and covariance o²C²I, and the noise multiplier is computed from (e, 6) by inverse privacy accounting (c.g.. [ACG 16]).
Although DP-SGD has been demonstrated to be an effective algorithm for training ML models with DP, (1) requires adding noise to all the coordinates of the gradient, which would completely destroy any existing sparsity structure the original gradients. This densification of sparse gradients is especially problematic for large industrial embedding models, where the sparsity is heavily leveraged to improve efficiency, e.g., using dedicated APIs (GGDS19, AF22, WWL+ Sna22].
3 Sparsity DP-SGD
In this section, we introduce two algorithms for preserving the gradient sparsity of DP-SGD when training large embedding models. Our main intuition is that in real-world datasets, some buckets of categorical features may be much more frequent, and therefore contain more significant or relevant information, than others. However, vanilla DP SGD adds noise to the model's parameter gradients, even to those with minimal impact on the model's 's accuracy, which is not only wasteful but also can reduce the training efficiency by disrupting gradient sparsity (as discussed earlier in Section 2.2).
3.1 Enttering-Enabler Sparse Training (DP-FEST)
A simple solution to this problem is frequency filtering, where we pre-select the top-k most informative (i.e., frequent) buckets of each categorical feature before training, and add Gaussian noise only to the gradients of these selected buckets during training: this is akin to training a smaller embedding model using a subset of the buckets. Frequency filtering significantly reduces the noise added the gradients by restricting the noise to only the most impactful subset of features.
There are two possible ways of selecting the top-k buckets. First, if there is prior information on bucket frequency available publicly (e.g., token frequency in the pre- pre-training set of the model for language tasks), then can be used select the top-k buckets. If such prior knowledge is unavailable, we can run DP top-k selection on the training dataset by adapting the algorithm of [DR21]; see Appendix B.1 for details.
3.2 Adaptive Filtering-Enabled Sparse Training (DP-AdaFEST)
The limitation of frequency filtering in DP-FEST is its inability to adapt to varying activation patterns across different mini-batches during training. This issue becomes particularly problematic when dealing with streaming data, such as those used in recommendation systems. In such scenarios, the application of frequency filtering poses a challenge as it impedes training until a substantial amount of data is gathered to accurately estimate the frequency information.
To address this issue, we propose DP- AdaFEST, an adaptive method for selecting the most informative buckets using mini-bated information, as outlined in Figure 2 and Algorithm 1. For each mini-batch, DP-AdaFEST first computes the per-example gradient and gradient contribution map, which is a binary vector indicating which categorical feature buckets have been activated for each example in the mini-batch (Line 5). These per-example gradient contributions are then aggregated (with per-example clipping applied) across the mini-bated in order to construct the batch-wist gradient contribution,

---

Step Compute contribution an statistics for buckets in the feature vocabulary.
Step Clip the contribution and Gaussian noise.
Bastect
Bidee
Radie
Bodee
dester
Bastee.
Bastee,
Bostent
Bostet fonter
doudet
Boste
Bustet
Battet
Step 3: Keep only the buckets IN that have noisy contribution that exceeds z for gradient update (highlighted in purple).
Tun wdr
men
Bostec
Boder
Bocket
Mediedd
deder
Besie
Botee
Bosket
Beske.11
Bocker of
Bostet
Bosket
Bodet,17
Bosket
Bedrr
Illustration of the process of AdaSESST on synthetic categorical feature which has 20 buckets. We compute the number of examples contributing to each bucket, adjust the value based on per-example total contributions (including those to other fcatures), add Gaussian noise, and retain only those huckets with a noisy contribution exceeding the threshold for (noisy) gradient update.
Figure 2:
Algorithm 1: DP- Adaptive Filtering Applied Sparse Training.
Input :carning rate 17, batch size B, number training steps T, dataset clipping norms C1, Cz. threshold parameter r, noise multipliers dri 02.
Output: :utpputDPPtrraiiiii DP-trained model parameters 0.
Initialize model parameters / randomly;
for to 1 to 1' do
Sample mini-batch B, of size from D: for i= to B do
L Compute per-example gradient and gradient contribution map vs where 0,15) = [o. * 0]: Compute the private contribution map V = Eies, lvile + +C1 -N (0, of Ia):
for to do L [ :] - O, for all j such that Vi[ii < <::
G = Eies, lolcc + C2 N/ (0, ob diag ((1mbb2r)) & Ila)):
10
Update the model parameters: 0 < 0 n)Gr
and Gaussian noise with scale σ1 is added to the resulting vector to ensure DP (Line 6).² The noisy batch-wise gradient contribution is then thresholded using a parameter 7 to exclude non-significant gradient entries to which only few examples in the mini-batch contribute (Line 8); This thresholding mechanism helps focus on the most relevant and informative features while reducing the impact of noisy insignificant contributions. Finally, the algorithm updates the model parameters by adding gradient noise with scale d2 only to the "surviving" gradient entries (Lines 9 and 10). By dynamically adapting feature selection based on bucket contributions in each mini-batch during training, our algorithm achieves adaptive feature selection and maintains the desired DP guarantees. Note that we apply the standard DP-SGD with noise multiplier 22 to update parameters in non-embedding layers.
3.3 Privacy Accounting
The privacy accounting of DP-AEaFEST can be done in a similar way to that of DP-SGD. The main observation is that the privacy cost of Algorithm 1 is dictated by the Gaussian noise mechanism steps in Lines 6 and 9. In particular, the privacy cost of a single iteration is equivalent to that of the composition of two Gaussian mechanisms with noise scale dj and 02 respectively, which in turn is equivalent to the privacy cost of single Gaussian mechanism of noise scale (o 0. -)1/2 Thus, overall, the privacy cost of the entire algorithm is no more than that of DC- with noise scale a, and identical other parameters of batch size and number of steps. Finally, we adopt the standard approach of analyzing the Poisson subsampled Gaussian mechanism IACG+ 16], using numerical algorithms based privacy loss distributions [KJH20, GLW21, GKKM22, D 22]. In particular, we use the open-source implementation available in Google's 's DP library [Goo20]. We provide more details in Appendix C for completeness.
*When implemented naively, this step has a memory consumption that scales with the number of embedding rows. In Appendix B.2, we discuss an alternative method for sampling the surviving coordinates of the mask in memory-effiiiee manner.
5

---

An important caveat to the privacy analysis is that we follow common practice of analyzing all the algorithms where the mini-batches are Poisson subsampled, wherein each example is included independently with a certain probability, even though, the actual implementation involves selecting mini-batches of fixed sizes after shuffling. See Section 4.3 of [PHK 233 for more context, But since this caveat applies to all algorithms similarly, we consider their relative comparison to be fair.
3.4 Bias- Variance Trade-offs
Apart from the computational advantage, understand the question of when DP-AdaFEST would lead to more accurate models as compared to DP-SGD, we look through the lens of bias-variance trade-off's in stochastic convex optimization. Consider conver objective L(-) over TR for which we have a gradient oracle, that given 0, returns a stochastic estimate g(0) of VL))). We say that the gradient oracle has bias and variance o² if g(0) = VC()) + ((0) such that E [C() 2 S a and E ((() - IE ç 10) S. o² holds for all 0. When optimizing over convex set K C R projected gradient descent with step size 7) defined as iteratively performing 0(+1
(0, ng()))) where (-) is the projection onto K. We recall the following guarantee on the expected excess loss obtained using standard analysis of projected stochastic gradient descent (see e.g. [Haz22]).
Lemma 3.1. For L-Lipschitz loss function, and gradient oracle with bias a and variance o², achieves
projected gradient descent over set K. with diameter R. with step size rj V((+a))+00yT
E [ ((<(+E.11) 0.) c(0)) - R VT
aR.
Suppose the loss is T.-Lipschitz. Ignoring the potential bias introduced due clipping, DP-SGD of noise scale a uses a gradient oracle with zero bias (a 0) and variance Do On the other hand, consider a hypothetical setting where Dd-AST truncates 7 fraction of the gradient due to masking in Line 8, resulting in the final gradient that is supported on l coordinates (h < D). In this case, the bias introduced is LL, but the variance introduced is only ho Plugging into the above excess risk bound, we would expect AdaFSTT to achieve a smaller excess loss when
v (T + 7)2 ho² + TLVT < VT2 Do
This implies that smaller truncated fraction within DP-AdaFEST could potentially yield superior utility compared to vanilla DP-SGD, Even though the discussion above is for the average iterate, and in the convex setting, our experiments corroborate this observation in training deep learning models.
4 Experiments
In this section we evaluate the performance of our sparsity-preserving training algorithms and compare them against vanilla DP-SGD both the recommendation and language understanding tasks (Section 4.1.. To understand adaptivity, we evaluate our algorithms on a time-series version of the data (Section 4.3); the non-time-series evaluations
in Section 4.2. We further demonstrate the applicability of our methods on language models in Section 4.4. The impact of hyper-parameten on the trade-off between utility and embedding gradient size is discussed in Section 4.5.
4.1 Setup
4.1.1 Datasets and Models
Click-through rate prediction tasks. We evaluate our algorithms on the widely-used Criteo predicted tlick-through rate (pCTR) dataser', which includes over four billion ad impressions over 24 days. Each impression is represented by 13 numerical features and 26 categorical features. The objective is to prediet how likely a user clicks
ad based these features.
The pCTR model we experiment with a neural network that uses embedding layers for categorical features and log transformations for numerical features, followed by several fully connected layers. We use binary cross- cross-entropy loss as the training objective and report the AUC as the evaluation metrie. For further details about the training setup, please refer to Appendix D.
We evaluate two Criteo variants with different purposes:
'https: pecttps://atie //ailab. criteo. www.mm/wwwwoddoriiioiooooooooeee 1tb beab-click-aa click-logs-daaee

---

Bec gooteeeeeee C makclii
Kagge,
55T-, 1HED
ONLL C
OOP,
Reistinn andt (1ME))
(ANBI)
5/00
Juee
Utillt (i4(-))
Figure 3: comparison of the hest gradient size reduction achieved by DP -d-AdaESST, DP-FEST, and DP-SGD with exponential selection [ZMH21] compared to DP-SCD at different thresholds for utility difference. A higher curve indicates better utility/efficiency trade-off. UP-AAaFEST consistently outperforms DP-FEST.
DF-AUFFEST
DP-FEST
DP-AM
Rest gradent sire retution
1.5
X10
Criteo-Kaggle, = 1.0
reloction
2.6
X30F
Criteo-Kaggle :
Best godeent size retuztin
1.5
X10
Criteo-Kagale 8.0
1.0
1.0
0.5
0.5
0.8
010
Bist
0.0
-0.005
0.0.0
01015
0.010
-0.010
-0 0005
0,000
0,005
0,010
-0.010
-0.00
0.0000
0.005
0,010
Unilit gain(-) [[los-))) compured DF-SGD
Unilty gain(+)] (onon()))) companed to DR-SCD
Utilyy
companed DP-SGD
Figure 4: A comparison between DP- AdaFEST, DP- DP-FEST. and the combined algorithm (P-AAdFEST+)) ) the Criteo-Kaggle dataset for different values e. The combined algorithm significantly outperforms either alone. Figure demonstrates similar findings on the Seiteo-time-seiie dataset.
Criteo- Kageol a subset of examples released by Kaggle (~45 million examples in total), is widely used in previous studies of click-through rate modeling. We use this to benchmark the performance of our algorithms. Note that timestamps are absent in Criteo-Kaggle
Criteo-time-serr also commonly known as "Criteo-ITB" is the entire Criteo dataset of over billion examples and has the auxiliary information indicating which day the data was collected. To simulate real-world online training scenario, we train the model on the first 18 days of data and evaluate it on subsequent days (i.e., days 19-24); more details on the training process are in Section 4.3.
Language understanding tasks. For language understanding tasks, employ language models from the BERT family [DCLTI9], specifically the RoBERTa model [LOG 19]. This model has a vocabulary size of 50, 265 subword tokens and has been pre-trained on public web data.
We fine-tune the RoBERTa model for downstream classification tasks from the GLUE bench- mark [WSM+ + 191. including SST-2 [SPW* 13], QNLI [RZLL16], and QQP [IYW* 17]. Follow- ing [YNB + 22], we adopt Low-Rank Adaptation (LoRA) [HSW [2]] to introduce trainable rank decomposition matrices into each transformer block of the language model. This approach sig- nificantly reduces the number of parameters required for downstream tasks while also improving the privacy-utility trade-off (see Appendix D.I). A notable difference in our approach compared to [YNB 22] is that we also train the word embedding layers in DP fine-tuning to enable bucket selection. This leads to significant accuracy improvements in the model, shown in Table 6.
4.1.2 Baseline and Algorithms
We evaluate the performance vanilla DP-SGD, and the three sparsity-preserving algorithmic variants, DP-SGD with exponential selection [ZMH21], DP-FEST and DP-AdaFEST, on both the tlick-through rate prediction and language understanding tasks. We use a fixed batch size of 2, 048 for the former and 1, 022 for the latter. For all tasks, we set the privacy parameter 1/N, where N is the number of training examples for the respective task.
4.2 Evaluation on Non-Time-Series Data
We begin by considering datasets with no timestamp Criteo-Kaggle, SST-2, QNLI, and QQP.
The decision to prioritize gradient sparsity (a key factor impacting efficiency) or utility in DP training usually depends on the task and available computational resources. However, can be challenging for vanilla DPD to cater to different needs of gradient sparsity or utility since it lacks mechanisms for trading off between these objectives. In contrast, DP-SGD with exponential selection [ZMH21],
'http://labs. criteo. htttpp//aaas.crttt.ooomo//////// witpogggggeddill-y-eeaee www.aayyddvrrriiiiii wwwwlhhgeeeaaeeeeeoee dataset
7

---

PFFFETEEST
DP-FEST (>l days]
DE-FEST (day 1)
DP-FEST (stroaming bosed)
Bot gradien side relection
Socaming poriod I
(est grident site moductoon Dest groclent sbee redlction
Straming pariod -
Beet gradient wize mubection Best gradiere reduction
Streaming poriod 2
10000
10000
13000
3000
(000
5000
-0.022 0.000
-0.001
-0.02 0.000 0.022
-0.001
0.000 0,02
Unitay gain(+)/ (oloud))))) compared
DF-SGD
Utiley gaint (/]loss))))) companed DF-SGD
Ubilay juint (osa))) companed DP-SCD
Bot gradient siee rubuction
Strsaming perio
Streaiing period
Streaming perldd
5000
5900
5000
-0.004
-0,002
0,000
0,002
[1004
-0.004
-0.022 0.000
0,002
0.004
-0.004
-0.022
0.000
0,002
0.004
USitl gaint )/lost() compared DF SGD
Utily gann [.0oos( comparcd DP -GD
Ubitty gain) 1//loss() comparod to DP SCD
time-series Figure 5: A comparison between DP- AdaFEST and DP-FEST for time-series data, using the Criteo- semoo-ttm--seee series dataset with different streaming periods T and & = 1.0. DP-AdaFEST consistently achieves a higher gradient reduction than DP-FEST at the same level of utility.
Ad-AdaFSST and DP-FEST can all offer diverse options for balancing utility and efficiency via the cuursity-contoolling parameters (see Figure 8 in Appendix D.2), while our proposals DP DSAASTESST and DP-FEST offer much better privacy-utility loss.
While both DP-FEST and AdaASTT offer ways to balance efficiency and utility in DP training, DP-AdaFEST stands out as the more versatile and customizable algorithm. With three adjustable hyper-parameters, 3P-AdaFEST offers more diverse results compared to DP-FEST's s single knob, k which controls the number of preserved top buckets. The effectiveness of DP AdaFEST is evident in Figure 3, where it achieves significantly higher gradient size reduction than DP-FEST while maintaining the same level of utility. Specifically, on the Criglo-Kaggle dataset, Ad-ASTT reduces the gradient computation cost of vanilla DP-SGD by more than 5 x 10a times while maintaining comparable AUC (of AUC loss less than 0.005). This reduction translates into more efficient and cost-effective training process (Appendix D.2.1 shows wall-clock time improvements). Although the best reduction for language tasks is usually smaller since the vocabulary is already relatively condensed, the adoption of pparsity-preserving DP-SCD effectively obviates the dense gradient computation, Furthermore, in line with the bias-variance trade-off presented in Section 3.4, we note that DP- AdaFEST occasionally exhibits superior utility compared to DP-SGD when the reduction in gradient size is minimal (i.e., y in Equation (2) is small). Conversely, when incorporating sparsity, DPD with the exponential mechanism faces challenges in maintaining utility. In all configurations, it fails to achieve a tolerable level of utility loss.
We further explore the potential of integrating DP- AdaFEST with DP-FEST to enhance performance, which involves pre-selecting subset of buckets using DP-FEST and subsequently training on this subset using DP- AdaFEST. Figure 4 shows that the combined algorithm (DP- HBP-AdaFEST+ outperforms either alone in utilitylefficiency trade-off, with the best gradient size reduction being further improved to 10 x. This can be attributed to the complementary strengths of the two algorithms: DP DP-AdTESST offers a more flexible approach to feature selection at the batch level, while DP-FEST provides a simple yet effeective means for feature selection according to the global frequency information. Besides, the combined algorithm expands the range of choices for balancing gradient sparsity and utility through the combination of Hlper-parameter from both DP-FEST and -d-AATESST.
4.3 Evaluation on Time-Series Data
Time-series data are notoriously challenging due non-stationarity. In this section, investigate if our algorithms can adapt to distribution shifts, characteristic of time-series data.
We conduct experiments using the Critco-time-srries dataset, which comprises real-world user-click data collected over 24 days (18 days for training and the rest for evaluation). To simulate the online data streaming scenario, we introduce the concept of streaming period, time interval during which the model is updated as new data is received. Experiments indicate a notable disparity between DP training (using vanilla DP-SGD) and non-DP training, with the former demonstrating increased susceptibility to distribution shifts (Table 5 in Appendix D.2).
Figure 5 presents the utility and efficiency trade-off of DP-ASAFEST and DP-FEST for time-series data. To explore the efficacy of DP-FEST, we investigate different sources of vocabulary frequency, including the information from the first day, all days, and treammig-basee (i.c., running sum updated
8

---

per streaming period). The experimental results indicate that using streaming based frequency infor- mation for DP-FEST almost as effective as using all-day frequency information and significantly better than using information only from the first day. Additionally, DP- Ad-ASTTTT consistently outper- forms DP-FEST, achieving more than twice the gradient reduction at the same level of utility. These findings further demonstrate the importance of adapting to the dynamic nature of time-series data at more fine-grained level, with DP-AdaFEST able to adapt at the batch level and DP-FEST only able to adapt at the streaming period level.
Dcs gradient sae edduction
We further examine the advantages of combin- ing DP-AdATEST and DP-FEST to enhance the util- itylefficiency trade-off on the Criteo-time-sere dataset. We use streaming period of 1 and streaming-hased frequency information for DP-FEST. Figure 6 demonstrates consistent superiority of the combined approach over individual methods, which aligns with the findings presented in Section 4.3.
10P
DI-FEST
DP Ad--EST--
10
201
0,000
0,002
0.004
Utilty gain( Hooos)) compared to DPSGD
Figure 6: The combined approach (DP- AdaFEST+) outperforms either alone on Criteo- Criteo-time-see sime-serie.
4.4 Applicability Language Models
We further demonstrate the applicability of our method to language models by comparing it with LoRA [HSW+22) 22] and reporting its performance with multilingual models.
DP-AdaFEST a better choice than LoRA for embedding layers. LoRA was introduced as a method to efficiently adapt matrices of dimensions approximation, where the initial use case considers the attention layers where = d. The rank-r approximation helps in reducing the memory requirements by factor of m d/(m + d) r min(m, d)/r. However, the applicability of LoRA to embedding layers is limited for several reasons:
d in language models by utilizing rank-r
Limited improvement for unbalanced and d: Embedding layers typically involve large vocabulary size (n often in the millions) and a smaller embedding dimension (d typically in the hundreds). In this context, LoRA's potential benefits are limited.
Inability to harness APIs: For private training the embedding layer, DP- AdaFEST allows for efficient embedding lookup operations (row fetching) using custom APIs; In contrast, LoRA still relies on computationally costly matrix multiplication and cannot effectively utilize these APIs.
Limited to fine-ture fine-tuning use cases: LoRA is designed for adapting pre-trained models, while our DP-AdaFEST is versatile, functioning seamlessly in both pre-training and fine-tuning scenarios.
To more effectively demonstrate the first argument above, Table 1 compares the best embedding gra- dient size reductions achieved by APAAST and LoRA against DPD for SST-2 with on the RoBERTa model. We vary LoRA's rank r from (4,8, 16, 32, 64, 128). DP DP-ASAFEST consis- tently outperforms LoRA in gradient size reduction similar utility levels.
Utility loss
Gradient size reduction
compared to DP-SCD Ad-ASTTEST LoRA
1.0,
0.001
14.59
5,91
0.005
53.90
23.64x
0.01
53.90
47.28x
Table 1: Gradient size reduction by LoRA and Ad-AaTFSTT for RoBERTa's 's word embeddings.
Increased gradient size reduction by DP-AdaFEST for larger vocabularies. Our evaluation in Section 4.2 primarily centered around the RoBERTa model, which has a vocabulary of around 50, 000 entries. However, it is important to note that many tigh-vocabulary models exist, with vocabulary sizes 5 to 20x larger than RoBERTa. For these models, DP- AdaFETT could offer even more pronounced benefits.
Utility loss compared to DP-SGD
Gradient size reduction
RoBERTa ( (5kk))))) XLM-R ([V) (:20k)
0.001
14.59x
17.21 x
0.005
53.90x
64.75
0.01
53.90x
152.94x
Table 2: DP-AdaPEST offers more pronounced gradient size reduction for models with larger vocabulary (i.e.. higher (V) ). RoBERTa results are achieved SST-2 with e 1.0, and the XLM-R [CKG+20] results are achieved on the Cross-Lingual Natural Language Inference (XNLI) dataset [CRL*1 18] with g 1.0.

---

Embodding gradient size ox/ox Embedding gradient
Uplity v.s. dr /it/
Utility v.s.
x10'
Embesding grdden SEC
Embasdiing gee prodieet size
0.705
0.705
Eal. AUC
0.700
0.700
2
18300
hi 50
1y25gm 18.
0.750
500
0.765
0000
10-1
/9/9
(a) Criteo- Criteo-Kaggle,
1.0
01/02
Embedding gradient size v.s. 0x/oz
Embedding gradient size x100
Utility d1/02
Utility
x100
Enlaadiing gration size
Fvl. Accunry
0.50
Accuracy
0.50
Embredding grodient
0.55
0.55
aege 5.0
2000
7//- 100
(14000
33.3
10F
10
10'
10P
10P
107
10P 10
10? 107
(b) SST-2, E 1.0
on utility and embedding gradient size, including the ratio of noise added the contribution map the one added the sparse gradient /02, and the thresholding value The joint impact of two hyper- parameters on utility and embedding gradient size can be found in Figure in Appendix D.2.
Figure 7: Effect of
4.5 Effect of Hyper-Parameter:
Finally, we study the effect of hyper-parameters the utility and gradient size trade-off in DP-AdaFEST. Specifically, we explore the effects of the ratio noise added to the contribution map and the noise added to the sparse gradient (i.e., 01 01/02), and the thresholding value T.
Effeet of σ /02- We find that larger ratio of 01/02 leads to a higher accuracy (Figure 7). Indeed, the contribution map can tolerate higher levels of noise, leading to a more accurate representation of the sparse gradient. For both Triteo-Kaggle and SST-2 datasets, a noise ratio of 5 or 10 delivers the best utility. On the other hand, a larger / results in a higher gradient density because the contribution map can tolerate higher noise levels, resulting in more zero-contrin cero-contribution buckets bypassing the threshold and increasing false negatives. Therefore, choosing appropriate 01/02 is critical to achieving optimal model accuracy and gradient density.
Effect of r. The choice of the thresholding value, т. is also crucial for determining the balance between gradient sparsity and model accuracy in DP-AdaFEST. Figure 7 reveals that increasing the value of usually leads to decrease in the gradient size, which results in sparser gradient. We also find that when T is small, increasing does not significantly impact the model's 's performance. However, setting excessively high value of (e.g.. > 500 for a batch size of 1024) can cause sharp drop in the model's accuracy, especially when the noise ratio 01/02 is small; In such cases, important contributions may be zeroed out, resulting in a loss of information in the gradient. Again, it is critical to choose 7 to best balance gradient sparsity and model accuracy in AdAAATFSST.
Conclusions
In this work present new algorithms- DP-FEST and DP-AdaFEST- for preserving gradient sparsity in DP training, particularly in applications involving large embedding models. Our algorithms achieve significant reductions in gradient size while maintaining accuracy on real-world benchmark datasets. We also provide recommendations on hyper-paramete settings.
Limitations and future work. Despite the advancements made in our proposed algorithms, there are still limitations and areas for future work. One such area is the exploration of customized hardware acceleration for sparse DP training, which has the potential significantly improve the training efficiency of our models. By leveraging specialized hardware, we can further optimize the computational performance and speed up the training process. Additionally, combining our algorithms with other srivacy-preserving techniques (e.g., Federated Learning) could lead to new insights and applications.
10

---

Acknowledgment
Yangsibo Huang supported by Princeton University's Wallace Memorial Fellowship.
References
[ACG*16] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal
Talwar, and Li Zhang. Deep learning with differential privacy. In CCS, 2016
[AF22] Gagik
Amirkhanyan and Bruce recommenders using cloud TPUs, 2022.
Fontaine.
Building large
scale cloud. gooude. com/blog/ wttpa///evveellepppprrraaeeeeeee scaree-scaleeeeooo scale recommenders using cl cluud-tpus.
https://
[CGSS21] Edith Cohen, Ofir Geri, Tamas Sarlos, and Uri Stemmer. Differentially private weighted sampling. In AISTATS, pages 2404-24144 2021.
[CKG 20] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. In ACL., 2020.
[CKH* 16] Heng-Tze Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. Wide & deep learning for recommender systems. /st Workshop on Deep Learning for Recommender Systems, 2016.
[CRL+18] Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: evaluating cross-lingual sentence representations. EMNLP, 2018.
[CTW*21] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom B Brown, Dawn Song. Ulfar Erlingsson, et al. Extracting training data from large language models. In USENIX Security, 2021.
[CWG22) Ricardo Silva Carvalho, Ke Wang, and Lovedeep Singh Gondara. Incorporating item frequency for differentially private set union. In AAAI, 2022.
22] Soham De, Leonard Berrada, Jamie Hayes, Samuel L Smith, and Borja Balle. Unlocking high-accuracy differentially private image classification through scale. arXiv preprint arXiv:2204116650, 2022.
[DCLT19] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre- training of deep bidirectional transformers for language understanding. In NAACL-IILT, 2019.
Weig
[DFAP21] Friedrich Dörmann, Osvald Frisk, Lars Nørvang Andersen, and Christian Fischer Pedersen. Not all noise accounted equally: How differentially private learning benefits from large sampling rates. In International Workshop on Machine Learning for Signal Processing, 2021.
IDGK*221 Vadym Doroshenko, Badih Ghazi, Pritish Kamath, Ravi Kumar, and Pasin Manurangsi. Connect the dots: Tighter discrete approximations of privacy loss distributions. PoPETS, 2022(4):552-570) 2022.
[DGK*23] Carson Denison, Badih Ghazi, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, Kr- ishna Giri Narra, Amer Sinha, Avinash Varadarajan, and Chiyuan Zhang. Private ad modeling with DP-SGD. In AdKDD, 2023.
[DKM+06] Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor. Our data, ourselves: Privacy via distributed noise generation. EUROCRYPT, pages 486-503, 2006.

---

[DMNS06] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. TCC, 2006.
[DR21] David Durfee and Ryan Rogers. One-shot DP top-k mechanisms. DifferentialPri- vacy.org, 08 2021. https::/diifferentiaa..
org/ oho-shhtttop--/
[DRS19] Jinshuo Dong, Aaron Roth, and Weijie Su. Gaussian differential privacy. CoRR, abs/1905.02383, 2019.
[DVGM22] Damien Desfontaines, James Voss, Bryant Gipson, and Chinmoy Mandayam. Differen- tially private partition selection. PoPETS, 1:339-352, 2022.
[GGDS19] Alec Gunny, Chirayu Garg, Levs Dolgovs, and Akshay Subramaniam. Accelerating wide & deep recommender inference on GPUs, 2019. httpp::///eeeeeeee //developereeeeeeeeee nvidia. com http/////acllleattnnngggg.o
recomeeedreemmmdeer inf.renceeon--e-e-e-- gpus.
[GGK*20] 20] Sivakanth Gopi, Pankaj Gulhane, Janardhan Kulkarni, Judy Hanwen Shen, Milad Shokouhi, and Sergey Yekhanin. Differentially private set union. In JCML, 2020.
[2HZ*22] Samyak Gupta, Yangsibo Huang, Zexuan Zhong, Tianyu Gao, Kai Li, and Danqi Chen. Recovering private text in federated learning of language models. In NeurIPS, 2022.
[GKKM22] Badih Ghazi, Pritish Kamath, Ravi Kumar, and Pasin Manurangsi. Faster privacy accounting via evolving discretization. In ICML, pages 7470-7483, 2022.
[GLW21] Sivakanth Gopi, Yin Tat Lee, and Lukas Wutschitz. Numerical composition of differen- tial privacy. In NeurIPS, pages 11631-11642. 2021.
[Goo20] Google's DP Library. DP Accounting Library. https://github. com/s/ogle// differential-prise wwwplllpriviyyyaaeee wwarree/maiinaiee http://pttoooopcoooooieee 2020.
[GTY+17] Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. DeepFM: factorization-maccinn based neural network for CTR prediction. arXiv preprint wvXiv:1703.04247, 2017.
[Haz22] Elad Hazan. Introduction to Online Convex Optimization. MIT Press, 2022.
[HGJ+19] 19] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter- efficient transfer learning for NLP. In ICML, 2019.
[HSC22] Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. Are large pre-trained language models leaking your personal information? In E.MN.P-Findiig 2022.
[HSW*22] 22] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zht -hu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022.
[IYW+17] Srinivasan Iyer, Mark Yatskar, John Wieting, Sheng-Fu Chang, and Nazneen Guo. First Quora Dataset Release: Question Pairs. Quora, 2017. attps://data. quora. com/ First-Quore Aatora-Dataset-Ront
Caastion-Pairse
UKL+ Norm Jouppi, George Kurian, Sheng Li, Peter Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, Suvinay Subramanian, Andy Swing, Brian Towles, et al. TPU v4: an optically reconfigurable supercomputer for machine learning with hardware support for embeddings. In ISCA, 2023.
IKCS* 22] Alexey Kurakin, Steve Chien, Shuang Song, Roxana Geambasu, Andreas Terzis, and Abhradeep Thakurta. Toward training at ImageNet scale with differential privacy. arXiv preprint axXiv:220..12328 2022.
[KJH20] Antti Koskela, Joonas Jälkö, and Antti Honkela. Computing tight differential privacy guarantees using FFT. In AISTATS, pages 2560-2569, 2020.
12

---

[KKMN09]
Aleksandra Korolova, Krishnaram Kenthapadi, Nina Mishra, and Alexandros Ntoulas. Releasing search queries and clicks privately, In WWW. 2009.
[LOG 19] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: a robustly optimized BERT pretraining approach. arXiv preprint MexXiv:9907, 19007.11692 1692, 2019.
[LTLH21] Xuechen LI, Florian Tramer, Perey Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners. In ICLR, 2021.
[MCCD13] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint 0rXiv:1301.378.1 2013,
[MHH* 22] Dheevatsa Mudigere, Yuchen Hao, Jianyu Huang, Zhihao Jia, Andrew Tulloch, Srinivas Sridharan, Xing Liu, Mustafa Ozdal, Jade Nie, Jongsoo Park, et al. Software-hardwaar co-design for fast and scalable training of deep learning recommendation models. In ISCA, 2022.
[MNW+22] Mingyuan Ma, Sen Na, Hongyu Wang, Congzhou Chen, and Jin Xu. The graph-based behavior-aware recommendation for interactive news. Applied Intelligence, 2022.
[MSC*13] 13] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations words and phrases and their compositionality. NeurlPS, 2013.
[NCS 22] Lin Ning, Steve Chien, Shuang Song, Mei Chen, Yunqi Xue, and Devora Berlowitz. Eana: Reducing privacy risk large-scale recommendation models. In RecSys, pages 399-407, 2022.
INMS 19] Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole-Jean Wu, Alisson G Azzolini, et al. Deep learning recommendation model for personalization and recom- mendation systems. arXiv preprint arXiv: 1901000966000,,. 2019.
[PHK Natalia Ponomareva, Hussein Hazimeh, Alex Kurakin, Zheng Xu, Carson Denison, H. Brendan McMahan, Sergei Vassilvitskii, Steve Chien, and Abhradeep Guha Thakurta. How to DP-fy ML: A practical guide to machine learning with differential privacy. J. Artif. Intell. Res., 77:1 1 17,1111-220,, 2023.
[PSM14] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, 2014.
[PTS+21] Nicolas Papernot, Abhradeep Thakurta, Shuang Song, Steve Chien, and Úlfar Brlings- son. Tempered sigmoid activations for deep learning with differential privacy. In AAAI, 2021.
[RZLL16] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv: (606605250)) 2016.
[SDH23] Marika Swanberg, Damien Desfontaines, and Samuel Haney. DP-SIPS: simpler, more scalable mechanism for differentially private partition selection. PoPETS, 2023(4):257- 268, 2023.
[Sna22] Snap Inc. Training large-scale recommendation models with TPUs, 2022. https: //eng. snap. cww/tro/////ee wwwm/traininggnonn models withelssssithhht -witthttuus.
[SPW* 13] Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositional- ity over a sentiment treebank. In EMNLP, 2013.
[TB21] Florian Tramer and Dan Boneh. Differentially private learning needs better features (or much more data). In ICLR, 2021.
[WFFW17] Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. Deep & cross network for ad click predictions. In ADKDD, 2017.
13

---

[WSC*21] Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin. Lichan Hong, and Ed Chi. DCN V2: Improved deep & cross network and practical lessons for web-scale learning to rank systems. In The Web Conference, 2021.
[WSM+ 19] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019.
[WWL*22] Zehuan Wang, Yingean Wei, Minseok Lee, Matthias Langer, Fan Yu, Jie Liu, Shijie Liu, Daniel G Abel, Xu Guo, Jianbing Dong, et al. Merlin hugeCTR: GPU-accelerated recommender system training and inference. In RecSys, 2022.
[YNB*22] Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, et al. Differentially private fine-tuning of language models. In ICLR, 2022.
[ZDW22] Yuqing Zhu, Jinshuo Dong, and Yu-Xiang Wang. Optimal accounting of differential privacy via characteristic function. In AISTATS, pages 4782-4817, 2022.
[ZHH2]] Huanyu Zhang, Ilya Mironov, and Meisam Hejazinia. Wide network learning with differential privacy. arXiv preprint arXiv:2103.01294 2021.
[0X Weijie Zhao, Deping Xie, Ronglai Jia, Yulei Qian, Ruiquan Ding, Mingming Sun, and Ping Li. Distributed hierarchical GPU parameter server for massive scale deep learning ads systems. MISys, 2020.
IZZS* 18] Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. Deep interest network for click-through rate prediction. In SIGKDD, 2018.
14

---

A Related Work
Embedding is fundamental ML technique used in modern deep learning represent nom-numeli features as dense numerical vectors in a key-value store format. In particular, embedding layers play crucial role in recommendation systems [CKH* 16, WFFW17, GTY+ 17, ZZS* 18, NMS* 19, WSC+ 21], with models relying heavily on them for effective predictions; they usually have a small number of parameters in their dense layers but their embedding layers can contain billions of entries and multiple billions of parameters. However, these embedding layers differ significantly from those used in other deep learning tasks, such as natural language processing [MSC+13, 13, PSM14, DCLT19), where models such as BERT [DCLT19] use fewer entries in their embedding layers but compensate with several hundred million parameters in their dense fced-forward and attention layers.
The exponential growth of data is leading to embedding tables reaching sizes spanning multiple gigabytes terabytes, which presents unique training challenges. For instance, fitting Google's 1.2TB model [M],+ Baidu's 10TB model [ZX]+20], and Meta's 12 TB model [MHH 22] onto single compute node or accelerator requires significant effort. Therefore, optimizing the forward (sparse lookup) and backward (gradient backpropagation) passes for embedding layers is critical to unlock the high compute throughput of embedding models, particularly for recommendation systems. Recent research has focused on designing optimized embedding layers [AF22, JKL. 23] and developing efficient sparse optimizers [GGDS19, WWL. 22]. Furthermore, data parallelism techmiques, such as distributing embedding tables across multiple compute nodes, have been explored in order improve performance [Sna22].
However, optimizing embedding models trained with DP is an unexplored area. A recent work [DGK 23] demonstrates the effectiveness of vanilla DCDSGD in the context of recommen- dation tasks; however, this study does not address new challenges such as the need for efficient sparse gradient computation and adaptive noise addition to comply with privacy guarantees. An- other attempt INCS+ 22] seeks to introduce sparsity in DP-trained large embedding models through post-processing DP-SGD gradients. However, this method falls short of providing the necessary DP guarantees. The closest effort to ours is the DP-SGD with exponential selection algorithm [ZMH21], which dynamically selects embedding buckets update based on their gradient norms. However, their evaluation is limited to small-scale Word2 [MCCD13] models, with results mainly presented for a fixed large 30. Furthermore, our experimental results in Section 4 demonstrate that their approach leads to significant loss in utility when applied to large-scale recommendation tasks.
Additionally, prior research has also made efforts improve the efficiency of DP-trained language models. For instance, [LTLH21] observes that embedding layers can significantly contribute to memory consumption during DP training of large language models. They therefore introduce ghost clipping to reduce the memory impact of embedding layers by avoiding per-example gradient generation. While our primary focus is on improving model efficiency, our methods also result in reduced memory usage as a secondary benefit due to gradient size reduction. Besides, [YNB [Y] proposes using parameter-efficient fine-tuning techniques in DP training but does not explore effective training of large embedding layers under DP: They freeze the embedding layers during DP fine- tuning and explore DP's compatibility with parfmeter-efficient fine-tuning methods, such as LoRA [HSW* 22] and Adapter [HGJ+ 19], for attention layers. It is worth noting that, as discussed in Section 4.4, when LoRA is applied to embedding layers, the gradient size reduction achieved is inferior to the reductions introduced by our proposed methods.
In addition, it is worth mentioning that partition selection is also technique for feature selection; its primary goal is to confidentially identify elements, along with their respective frequencies, that occur at least a specified number of times [KKMN09, DVGM22, GGK+20, CGSS21, CWG22, SDH23]. However, in our current work, we introduce a new algorithm as it is more compatible with DP-SGD.
B Methodology
B.1 DP Top-k Selection
Assuming there are unique buckets in L, our goal to return the top-k frequent buckets from in private way. In our scenario, each user's data can modify the count of a single bucket from given feature by at most 1, ensuring that a user contributes to at most one bucket. Therefore, the lo -ensiitivity of the counting function for each feature is 1. To achieve DP top-k selection,
15

---

utilize the algorithm introduced in [DR21] (see Algorithm 2). The algorithm begins by counting the
frequency h E N" of each bucket the list. Next, it injects Gumbel noise into h and subsequently returns the top-k buckets based on the noisy frequency.
Algorithm 2: One-shot DP Top-k Selection [DR21].
Input
The number of unique buckets c, list of bucket occurrences of size L E [d], privacy budget e, the number of selected buckets k.
Output: :To-k frequent buckets selected, with DP, from L. Compute the bucket frequency in L;
2 for to do
L h[][i]] + [[] +1;
Add Gumbel noise to the bucket frequency, h + h + X, where X, ~ Gumbel(1/e);
r <- max+ (h: E [e]}. where arg max; returns the indices of the largest is elements in vector:
Return
In experiments, when selecting the top-k buckets from all p features, distribute the privacy budget e and selection budget k equally among the p features. We then perform DP Top-int (k/p) selection, allocating a budget of s/p for each feature. Finally, we concatenate the outputs from different features to obtain the final set of selected buckets. We set e = 0.01 in our experiments and deduct the value from the total privacy budget accordingly the budget for DP training.
B.2 Memory Efficient Filtering
Implementing the computation of the contribution map and thresholding (Lines 6 and 8 in Algorithm 1) can naively take (cc) time and space, since v; s are c-dimensional. This can be prohibitively expensive when is much larger than the size of the gradients. Here we sketch different approach for computing the contribution map in time and space cost that is linear in the size of the gradient.
Let V/ Eies, lusle, For all j E [c]. we have that Pr[Vi[] 2 r] v
( Volj of
where
(t) == 11 /Z 2 t) is the survival function of the Gaussian distribution.
With this notation, more efficient algorithm is as follows: For each such that V.ly * 0, we sample the bits Vi[i] accordingly. Let c' be the number of coordinates such that V.lj and for simplicity, we assume w.l.o.g. that these coordinates are contiguous. For sampling the other coordinates of V,, we rely on the following generic approach for sampling long binary vector, with each bit being with probability = (T/ Namely, we observe that the difference between the positions of two consecutive 1s is distributed as a geometric random variable Gcom with probability mass function p(1 p)^-1. Thus, we sample these indices directly by sampling the differences iteratively from Geom(p), until the sum of these exceeds c'. This step runs in time that is linear in the number of false positives, which in expectation is c'p. which is proportional the number of non zeros in Gt-
C Privacy Analysis
C.1 Dominating Pairs for Privacy Analysis
The ef -hockey stick divergence between P and Q is given 2-- sups P(S) Q(S). Observe that a mechanism M satisfics (e, 6)DP for all adjacent datascts D D' holds that De (D) (D') 6.
Definition C.1 (Definition 7 in [ZDW22]). A pair (P.Q) of random variables over Q dominates pair (P', Q') of random variables over D! (denoted (P', Q') N (P, Q)) if for all E RR it holds that De (P')Q() < D. (PIIQ).
Remark. The notion of "dominates" is symmetric, namely if (P', Q') 1 (P,Q) then (Q', 1 (Q, P). This follows because D,- (Q] (1 e + e To provide an example, we recall the Gaussian mechanism Maayar which takes as input vectors Ti,...,Ta Fn . E ddd with ||æı|la S and returns Σ, 24 NO o² where a denotes the noise multiplier.
(P-l()
16

---

Lemma C.2 (Gaussian mechanism), For any adjacent datasets D D', it holds that M(D") (N(1,o").
We use two key behaviors of dominating pairs, namely under composition and Poisson subsampling.
Composition. For any mechanism M,, and mechanism M2, let M (M1, M2) denote the com- posed mechanism, namely, one that outputs (y1, U2) sampled as (D) and y2 ~ M2 D1)- Lemma C.3 (Theorem 10 in (ZDW221). For any adjacent datasets D and D', if (D) (D')) 3 (P1, Q1) and (M2(D;y), M2(D':y)) (P2, Q2) for all values of y, then M(D') 3 (P1 x P2, Q1 x Q2), where P1 x P2 refers to the product distribution with marginals Pl and P2-
Poisson Subsampling. For any y E (0, 1]. let Spoj denote the mechanism that takes as input a dataset of arbitrary size, and returns dataset by including each datapoint with probability i.i.d. at random.
Lemma C.4 (Theorem in [ZDW22]). Let M be a mechanism and (P; Q) be a pair of distributions, such that for any adjacent datasets D ~ D) with D containing one more datapoint than D', it holds that (D), M(D)) 3 (P.Q). Then for M = M Spoit it holds that (M M(D)) 1 (1-7))0+pPP0)
Finally, we recall the basic composition theorem of DP.
Lemma C.5. M satisfies (s, D)-DP and M' satisfies (E', D')PP,, then the composed mechanism M') satisfies (s + E', DDP
C.2 Privacy Analysis of DP-SGD
We recall the privacy analysis of DP-SGD using privacy random variables, following [ACG (0GG+1611 16), under Poisson sub-sampling of mini batchs.. Observe that the computation of noisy gradient in each step of DP-SCD corresponds to an invocation of the Gaussian mechanism with noise multiplier of a. Hence, by combining Lemmas C.2 to C.4, we have that for any adjacent datasets D and D' with D containing one more datapoint than D', it holds for M being DP-SCD that (M(D), (M(D),M(D') 3 (PxT, Q*T) where P (1 W>, o²) + "W(,,, Q NO0 o²)), and T denotes the number of steps.
Thus, suffices to choose a such that max 2ee (pxT =Q*T) 2er (Q ((*T)) s privacy parameters (s,
for desired
C.3 Privacy Analysis of DP-FEST
DP-FEST is simply the composition of two mechanisms: the first one being Algorithm (that satisfies (E1, O)-D and DP-SGD with noise multiplier a. Using Lemma C.5, and the above analysis of DGDS suffices choose such that max D Der-r [px*IQxX)) Der-ni (Q** (XxT| })}
for desired
final privacy parameters of (e, 8).
C.4 Privacy Analysis of DP-AdaFEST
DP- AdaFEST can be analyzed essentially identically DP-SGD. We observe that each step of DP- Ad-AdFFEST involves computing the private contribution map, and then the noisy gradient sub- ject the private contribution map. The private contribution map is computed via the Gaussian mechanism with noise multiplier and the noisy gradient is also computed via the Gaussian mecha- nism with noise multiplier of o2. From [DRS19] (Corollary 3.3). it known that the composition of two Gaussian mechanisms with noise multipliers of and 02 respectively is equivalent to Gaussian mechanism with noise multiplier a = (a, + a 2)-2 i.e., for Mo = (MGause: M holds for any adjacent datasets D D' that (Mo(D). MD)D")= = O1,, o²)). Hence, by combin- ing with Lemmas C.3 and C.4, we have that for any adjacent datasets D and D' with D containing one more datapoint than D', it holds for M being DP- AdaFEST that (M (D), MMD)) = (PXT, QxT)
NN(0, o²)), and T denotes the number of steps. Thus, suffices choose a such that max Z.e (PxT [Q 2er (Q < for desired privacy parameters 6.
where P (1 (0, o (1 o²)
17

---

C.5 Numerical Privacy Accounting using Privacy Loss Distributions
Thus, order choose the noise parameters for all the above algorithms, the core primitive needed is to compute the smallest such that max ee (PXT] (I**) Gec ( < for desired privacy parameters e, 6, where P = - -)) + (N(1 Q
Several works have proposed numerical algorithms based on privacy loss distributions [KJH20, GLW21, GKKM22, +GK+222 for privacy accounting and have in particular studied the specific pair (PxT, Q*T) above. In particular, we use the open-source implementation available in Google's DP library [Goo20] for this privacy accounting.
D Experimental Details and Other Results
D.1 Experimental Details
In this section, we provide experimental details for recommendation tasks and NLU tasks.
D.1.1 Click-Through Rate (CTR) Prediction Tasks
The pCTR model's architecture. The pCTR model we comprises total of six layers. The first layer utilizes an embedding layer to map each categorical feature to a dense feature vector. Appendix D.1.1 gives the vocabulary size of each categorical feature. We determine the embedding dimension using a heuristic rule (2V.225),, where represents the number of unique tokens in each categorical feature. The resulting dense features are combined with the ong-transformed integer features to create the first layer representation. We then apply four fully connected layers to this representation, each with output dimension of 598 and ReLU activation function. Finally, use another fully connected layer to compute the scalar prediction as the final output.
Feature
Vocabulary size
Feature
Vocabulary size
categorical-featte featurr--ffeeeeeeeee 14
1, 4.2
categorical Sicegoriaalffeature22 feature-27
27
categorical-feat featual--ettt-115 15
577
categorical nategorica--fettrr-222
1, 550
categorical-fcatuee
82, 741
categorical- 2eature-29
44, 44, 262
categorical-featuee 17
18, 940
categorical-featin feature-30
10
305
categorical-feature-1 feature-31
5, 485
23
categorical- 32
2, 161
categorical-featue feature-20
1, 172
categorical- feature-33
seggorical-feature
633
pangoorical-eeaturee
56, 473
categorical seralllfcttrr-222 22
3
categorical feature-35
17
categorical feattrfetteee-ee 23
9, 090
categorical fcature-36
15
seggoricll-fctture
5,918
categorical feature-37
27, 27,360 360
categorical-feature
64,300
categorical- feature-38
104
categorical-feature-2
3,207
integorical-fcature--
12, 934
Table 3: Vocabulary size categorical features in the Criteo dataset.
sereee-prramttrs. For DP-SGD we fine-tune the clipping norm and report the best accuracy achieved. When evaluating DP-FEST, we adjust the hyper-parameter which represents the number of preserved top buckets, with values ranging from 100 to 300, 000. Regarding DP-AdAFEST, we tune the following pyper-parameters:
The ratio of noise added to the contribution map to the one added to the sparse gradient, 0 1//2. with options of 0.1 to 10.
The thresholding value 7 E (0.5, 1.0, 5.0, 10.0, 20.0, 50.0, 100.0) The clipping norm for gradient contribution C₁ E {1.0, 5.0, 10.0).
D.1.2 NLU Tasks
Datasets. Our evaluation of NLU tasks uses three downstream classification tasks from the GLUE benchmark [WSM+19]
18

---

The SST-2 dataset [SPW*1 13] is sentiment analysis dataset. The task is to predict whether movie review is positive or negative.
The QNLI dataset [RZLL16] is a guestion-answering dataset. Each example consists of question and corresponding sentence, and the task is to predict whether the sentence contains the answer to the question or not.
The QQP dataset [[YW+ 17] is a collection of question pairs that are labeled as either duplicate or duplicate. The task is to predict whether given pair of questions are duplicates or not.
Hlpee-paraamteers For DP-SGD, we fine-tune the clipping norm and report the best accuracy achieved. When evaluating DP-FEST, we adjust the hyper-paramettr k; which represents the number of preserved top buckets, with values ranging from 1, 000 to 50, 000. Regarding DP AP--ddaFET, we tune the following prper-parameters:
The ratio of noise added to the contribution map to the one added to the sparse gradient, 0 0//02. with options of 0.1 to 10.
The thresholding value T E {50, 100, 200, 500, 1, 0,00.
The clipping norm for gradient contribution C₁ E (50.0, 100.0, 500.0).
D.2
More Experiment Results
D.2.1 Wall-Clock Time Reduction of Sparsity- DP-SGD Compared with DP-SGD
We conduct simulation experiments utilizing tf keras. layers. Embedding", the sparse embed- ding API provided by Tensorflow that supports sparse lookup and sparse gradient computation, to assess the efficiency improvement of sparsity-preserving DP training compared vanilla DP-SGD. Our simulations create an embedding layer with a fixed embedding dimension of 64 and batch size of 024. We investigate the impact of changing the vocabulary size [V] on the wall-clock time improvement. Table 4 reports the training time for 100 steps for DP-SGD and DP-AdaFEST, along with the wall-clock time reduction factor. We observe significant wall-clock time improvement in sparse updates, especially for larger vocabulary sizes: For instance, when the vocabulary size is 10M, we are able to achieve 150x wall-clock time improvement over vanilla DP-SGD; For evaluation in Section 4, the Criteo models have a vocabulary of 1.7M, where our approach translates to wall-clock time improvement ranging from 20 to 40x.
Vocabulary Size
DP-SGD
Ours
Wall Clol-Clock Time Reduction Factor
1 x 10
0,985
0,324
3.045
2 10a
1.676
0.312
5.376
X 108
6.607
0.318
20.746
2 X 10a
13.09
0.323
40.515
x 10a
31.326
0.332
94.340
1 x 107
62.473
0.353
176.760
Table Wall-clock time improvement by our methods over DP-SGD under different vocabulary sizes.
This efficiency improvement comes from two sources:
The cost of a sparse update (adding a sparse gradient tensor to the dense weight tensor) is smaller than that of dense update, and this gap increases with the tensor size. For example, in the Tensorflow optimizers, ocatter_add () is used when the gradient tensor is sparse to take advantage of this.
The computation overhead of generating a dense tensor of Gaussian noises at each step becomes significant in large embedding layers and the cost also scales with the embedding layer size.
D.2.2 Recommendation Tasks
DP-SGD offers customizable options for balaneing efficiency and utility. The scatter plot in Figure 8 showcases how both AP-ASAFEST and DP-FEST present range of
19

---

possibilities for effectively balancing utility and efficiency by leveraging controlin parame- ters. We further note that both approaches yield an improved trade-off between privacy and utility compared to the previously suggested DP-SCD with exponential selection [7MH21]].
DP-ASAFEST
DP-FEST
expomential (7MH21)
DP-SGD
Critzo Kaggle, 1.0
Criteo- Kaggle, 3.0
Criteo- Critea-Kaggle.
8.0
0.78
0.78
0.78
Fal. AlIC
Eral: AUC
ab
0.74
0.74
0.74
Embedding gradient sie
Embedding gradient size
Embeciiing gradient size
Figure Comparison of the utility/efficiency trade-off between DP-SGD, DP-SGD w/ exponential selec- tion [ZMH22]] DP-FEST, DP-AdaPEST
Criteo-Kaggle dataset. DP-FEST, AdaATT DP- SGD w/ exponential selection can offer more customized approach for meeting varying requirements of gradient sparsity and utility than DP-SGD, while DP-Ades AdaFEST achieves the best tradc-off.
Effect of the streaming period pCTR model's utility. In Table 5, we present the evaluation
AUC of Criteo-time-series under different streaming periods, with varying values, for both vanilla DP-SGD and non-private training. Our results indicate that, for DP training, the AUC values increase as the streaming period increases, regardless of the value of & (i.e., e = 1.0, e = 3.0, and æ = 8.0). Conversely, non-private training results in similar AUC values across different streaming periods, which suggests that the model is less susceptible to distribution shifts in the data. This observation implies that DP training is more vulnerable to such shifts, thus explaining the effectiveness of DP AdaFFSTES, which adapts to distribution shifts.
Streaming period
1.0
3.0
E 8.0
Non- Mon-privatte (s 00)
0.7313
0.7323
0.7345
0.7846
0.7313
0.7324
0.7346
0.7848
4
0.7314
0.7326
0.7352
0.7847
8
0.7316
0.7328
0.7352
0.7820
16
0.7318
0.7331
0.7356
0.7848
18
0.7320
0.7332
0.7358
0.7848
Table 5: Evaluation AUC of Critco-timed time-series model under vanilla DP- DGD and non non-privan private training with different streaming periods. The streaming period affects DP-SGD models but non-private models.
Joint impact of hyper-parameeee in DP-AdAFSST In Figure 9, we present the interplay between the noise ratio 01/02 and on the utility and embedding gradient size for the Etiteo-Kaggl dataset. Our findings indicate that the optimal selection of hyper-parameters is critical for achieving desirable trade-off between high utility and small gradient size. Specifically, we observe that darker colors on both heatmaps correspond to optimal hyper- parameter choices. Such choices are generally clustered the middle-lower regions of the heatmaps, where we see a prevalence of high values of a / and medium values of r. These results suggest that the choice of hyper- pprrmmparaeeters should be approached with great care and attention detail, can have a significant impact on the performance of DP models.
20

---

Embending Gradient Size
Eval. AUC
700
353
0758
Ea-37
37--0 4--dd) t.uet3 de Sa-
11
03
0,700
8.753
0.73
46-20
37--00
Liu+22
5.5
30
02
793
702
0.700
0.76
0.7
3700
0.700
0.78
07733
te de 8-le-0s
de 1o- Se140
Sm-
01/os
3. 3a
7.4e- 14-06
te-00
44-100
t. 4a 1.5
90
279
0.796
0.708
6:704
0.722
0.911
0.791
0.705
0.733
50
965
0.00
3.6 176-27 28w+87
20180 74 3.2
4.3%
15s
100
9:99
6:33
8:744
0.78
3. E--07 34eet2
5--00
285
1.99
790
0.08
8:33
0.700
0.00
0.53
0. 30--77 3.we88 27 38--0
1:0 46:24 1.5-00
25.0
58
2215
2800
Figure 9: Heatmap illustrating the joint impact of the noise ratio di 01/02 and the utility and embedding gradient size for the Critco Kagooo dataset 1.0).
D.2.3 NLU Tasks
Training word embedding layers in NLU improves accuracy. As shown Table 6, allowing gradient updates for word embedding layers in the NLU tasks improves the accuracy of DP training.
SST-2
QNLI
Non-private
94.31%
92.38%
DP-SGD, 1.0
91.85%
83.49%
DP-SGD, 1.0 (embedding frozen)
91.43%
83.11%
DP-SGD, 3.0
92.43%
84.32%
DP-SGD, 3.0 (embedding frozen)
91.51%
84.02 %
DP-SCD, 8.0
92.87%
86.34%
DP-SGD, 8.0 (embedding frozen)
92.66%
86.28%
Table Accuracy comparison of non-private training and DP-SGD with and without frozen word embeddings four language understanding datasets with varying levels of privacy budget (E).
21


Reviews:
[{"review_id": "T4kwfmPHxH", "reviewer": "Reviewer_upr1", "summary": "This paper focuses on the concept of gradient sparsity in large embedding models during training, with a particular focus on privacy-preserving methods. The commonly used DP-SGD approach adds noise to all embedding gradients, even those that may not appear in the current batch, in order to ensure privacy during model updates. The authors propose algorithms aimed at preserving gradient sparsity while achieving privacy in the training of these models. By doing so, they aim to improve the efficiency and effectiveness of private training for large embedding models.", "questions": "1) The authors use the RoBERTa model for downstream classification tasks from the GLUE benchmark. The model has 50k vocab size. Prior work that the authors cite shows that DP-SGD provides strong performance on large batch sizes (approx 1k-10k) and considering that each sample has (approx 128-512) tokens, do we expect to see the gradient sparsity phenomenon in DP-SGD training of large language models with these hyperparams? The paper shows embedding gradient sparsity in Ads model but I do not see any discussion around this regarding language models.\n\n2) For large language models, transformer layers are in general the dominating part of the model. Could the authors provide examples for how/when this sparsification approach of the embedding layer be effective and help concretely in what sense?\n\n3) Actually DP-SGD hyperparameters for language model experiments are not clearly provided. I only see the sentence that says \"we fine-tune the clipping norm and report the best accuracy\". On the other hand, it's known again from prior work that when set sufficiently small, clipping norm does not play major role in the performance of the model but batch size and learning rate are the critical hyperparameters.\n\nMinor: Many Appendix B references seems to have been pointed to Appendix C.\n\nI can see the contributions of the paper but perhaps rather in a more limited case where the embedding layer is the critical part of the model in terms of size and time bottleneck. It just seems to me that (large) language models may not really be the representative of this scenario.", "limitations": "The authors adequately addressed the limitations according to this reviewer (apart from the reviewer's questions above).", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written and the presentation is clear, which is helpful to follow and understand the paper. The problem is clearly described, DP-SGD adding noise to all embedding gradients destroys the sparsity and the authors propose approaches for private training while keeping the sparsity for effectiveness. The experiments (in certain settings, see below) empirically support the claims of the paper.", "weaknesses": "I think the main weakness of the paper is that the approach is limited in the sense that it is helpful in more specific scenarios where the embedding layer is dominating the network. CTR prediction task might be a good example for this where the sparsity in embedding layer matters but for the language models, the reviewer is not so sure about the effectiveness of this approach. More details are in the questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We genuinely appreciate the valuable feedback provided by the reviewer and have addressed them in a point-by-point manner below. We are more than willing to engage in further discussions with the reviewers should any follow-up questions arise.\n\n### **Q1. Lack of discussions of language models**\n> Prior work that the authors cite shows that DP-SGD provides strong performance on large batch sizes (approx 1k-10k) and considering that each sample has (approx 128-512) tokens, do we expect to see the gradient sparsity phenomenon in DP-SGD training of large language models with these hyperparams? The paper shows embedding gradient sparsity in Ads model but I do not see any discussion around this regarding language models.\n\n**A**:  Great question! The following table reports the number of unique tokens in the batch and the corresponding gradient sparsity for the RoBERTa model on the SST-2, QNLI, and QQP datasets. The table also includes the average sequence length for each dataset. The vocabulary size is 50,265.\n\n| **Batch size** | **SST-2 (avg. seq length: 14.4)** |  | **QQP (avg. seq length: 29.1)** |  | **QNLI (avg. seq length: 36.7)** |  |\n|---|---|---|---|---|---|---|\n|  | # unique tokens | Gradient sparsity  | # unique tokens | Gradient sparsity  | # unique tokens | Gradient sparsity  |\n| 16 | 150.6 | 0.003 | 209.2 | 0.004 | 342.0 | 0.007 |\n| 64 | 486.4 | 0.001 | 667.2 | 0.013 | 1142.2 | 0.023 |\n| 256 | 1,363.0 | 0.027 | 1,948.2 | 0.039 | 3,528.0 | 0.070 |\n| 1,024 | 3,637.2 | 0.072 | 5,062.8 | 0.101 | 9,054.0 | 0.180 |\n| 4,096 | 7,637.6 | 0.152 | 11,306.2 | 0.225 | 18,556.2 | 0.369 |\n| 16,384 | 11,982.8 | 0.238 | 21,605.6 | 0.429 | 30,204.6 | 0.600 |\n| 65,536 | 14,105.0 | 0.281 | 31,494.0 | 0.657 | 37,403.0 | 0.744 |\n\nFrom this table, even for large batch sizes (1,024-16,384), the number of unique tokens is still much smaller (~2-14x) than the overall vocabulary size.  Intuitively, this is because a significant portion of the vocabulary comprises uncommon tokens, which experience infrequent updates during training. Thus our algorithm can be applied even with large batch sizes.  \n\nWe also note that while very large batch sizes (batch size of >1M) seem essential to achieve strong performance for private pre-training [1], this is not the case for private fine-tuning [2] (batch size of ~2k).  \n\n**References**:\n\n[1] Anil R, Ghazi B, Gupta V, Kumar R, Manurangsi P. Large-scale differentially private BERT. EMNLP 2022.\n\n[2] He J, Li X, Yu D, Zhang H, Kulkarni J, Lee YT, Backurs A, Yu N, Bian J. Exploring the limits of differentially private deep learning with group-wise clipping. ICLR 2023.\n\n### **Q2. Applicability of the proposed algorithm on LLMs**\n> For large language models, transformer layers are in general the dominating part of the model. Could the authors provide examples for how/when this sparsification approach of the embedding layer be effective and help concretely in what sense?\n\n**A**: We thank the reviewer for raising this point. Our evaluation primarily centered around the RoBERTa model (vocab size: 50k) because it is a standard backbone. However, it is important to note that many high-vocabulary models exist, with vocabulary sizes 5-20x larger than RoBERTa (shown below). For these models, our proposed sparsification method, DP-AdaFEST, could offer even more pronounced benefits.\n\n| **Model** | **Vocabulary size** |\n|---|---|\n| XLM-R [1] | 250k |\n| VoCAP [2] | 250k ~ 500k |\n| XLM-V [3] | 250k ~ 1M |\n\nFor example, the following are the results on the XLM-R model for the Cross-Lingual Natural Language Inference (XNLI) task when $\\epsilon$=1.0. As shown, DP-AdaFEST is able to achieve **a gradient size reduction of >150x** at a minimal utility loss of 0.01.\n\n| **Utility loss compared to DP-SGD** | **Best gradient size reduction by DP-AdaFEST** |\n|---|---|\n| 0.001 | 19.84x |\n| 0.005 | 73.42x |\n| 0.01 | 162.13x |\n\nWe would like to further clarify that our proposed algorithm is mainly designed to preserve gradient sparsity of embedding layers in a black box fashion. However, users have the flexibility to apply any desired algorithm to enhance model sparsity for other layers (e.g., by employing LoRA for the transformer blocks). We believe this modular approach allows users to tailor the sparsity optimization according to their specific requirements for different layers within the model.\n\n**References**:\n\n[1] Conneau A, Khandelwal K, Goyal N, Chaudhary V, Wenzek G, Guzmán F, Grave E, Ott M, Zettlemoyer L, Stoyanov V. Unsupervised cross-lingual representation learning at scale. ACL 2020.\n\n[2] Zheng B, Dong L, Huang S, Singhal S, Che W, Liu T, Song X, Wei F. Allocating large vocabulary capacity for cross-lingual language model pre-training. EMNLP 2021.\n\n[3] Liang D, Gonen H, Mao Y, Hou R, Goyal N, Ghazvininejad M, Zettlemoyer L, Khabsa M. Xlm-v: Overcoming the vocabulary bottleneck in multilingual masked language models. Arxiv preprint.\n\n### **Q3. Hyperparameters for NLU tasks**\n> DP-SGD hyperparameters for language model experiments are not clearly provided… it's known again from prior work that when set sufficiently small, clipping norm does not play major role in the performance of the model but batch size and learning rate are the critical hyperparameters.\n\n**A**: Please note that Appendix C.1 of our submission contains the hyperparameter choices for NLU tasks. Specifically, to constrain the search space, we fix the batch size to 1024, and vary the learning rate in {5e-4, 1e-3, 2e-3, 5e-3}, and vary the clipping norm in {0.1, 1.0, 2.0, 5.0, 10.0}. For convenience, the table below contains the optimal selection of hyperparameters for various NLU tasks when $\\epsilon$=1. \n\n|  | **SST-2** | **QQP** | **QNLI** |\n|---|---|---|---|\n| Learning rate | 1e-3 | 5e-3 | 5e-3 |\n| Clipping threshold | 10.0 | 10.0 | 10.0 |\n\n### **Typo: Many Appendix B references seem to have been pointed to Appendix C.**\n\n**A**: We appreciate the reviewer’s careful reading, and we will fix these issues in the revision."}, {"review_id": "LZFIK6TFq5", "reviewer": "Reviewer_ah8R", "summary": "This paper considers an interesting and less-studied aspect of DP-SGD: Applying naively to embedding models can destroy gradient sparsity, reducing training efficiency. \nTo address this issue, the paper proposes two algorithms DP-FEST and DP-AdaFEST that apply DP while maintaining gradient sparsity during the training of large embedding models.", "questions": "1. Is there any gap between the privacy guarantee of SP-SGD and that of DP-AdaFEST? If so, can the authors quantify it? \n", "limitations": "Yes!", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The authors did a great job motivating the problem in the introduction. The question addressed in this work is of great importance and a reasonable answer to it can potentially have significant consequences, as evidenced by the experiments. ", "weaknesses": "The paper lacks any rigorous analysis of the proposed algorithms. More precisely, there is no privacy analysis of DP-FEST and DP-AdaFEST.\nInstead, the authors only mentioned (line 169-170):  \"In particular, the privacy cost of a single iteration is equivalent to that of the composition of two Gaussian mechanisms with noise scale $\\sigma_1$\nand $\\sigma_2$ respectively, which in turn is equivalent to the privacy cost of a single Gaussian mechanism of noise scale $\\sigma = (\\sigma_1^{-2}+\\sigma_2^{-2})^{-1/2}$.\" Why not using this result to determine a formal privacy analysis of the algorithm? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We genuinely appreciate the valuable feedback provided by the reviewer and have addressed them below. We are more than willing to engage in further discussions with the reviewers should any follow-up questions arise.\n\n### **Q1. Lack of rigorous privacy analysis**\n> The paper lacks any rigorous analysis of the proposed algorithms. More precisely, there is no privacy analysis of DP-FEST and DP-AdaFEST. Instead, the authors only mentioned (line 169-170): \"In particular, the privacy cost of a single iteration is equivalent to that of the composition of two Gaussian mechanisms with noise scale $\\sigma_1$ and $\\sigma_2$ respectively, which in turn is equivalent to the privacy cost of a single Gaussian mechanism of noise scale $\\sigma = (\\sigma_1^{-2} + \\sigma_2^{-2})^{-1/2}$.\" Why not using this result to determine a formal privacy analysis of the algorithm? Is there any gap between the privacy guarantee of SP-SGD and that of DP-AdaFEST? If so, can the authors quantify it?\n\n**A**:  Please note that a detailed privacy analysis was not included since we felt it is standard.  Indeed, the analysis is near-identical to that of DP-SGD and involves repeated application of the sub-sampled Gaussian mechanism with scale $\\sigma$. In DP-AdaFEST, we have a repeated application of a sub-sampled mechanism where the inner mechanism involves application of two Gaussian mechanisms with noise scales $\\sigma_1$ and $\\sigma_2$. It is well-known (e.g., [1, Corollary 3.3]) that such an inner mechanism is privacy-wise equivalent to an application of a single Gaussian mechanism with noise scale $\\sigma = (\\sigma_1^{-2} + \\sigma_2^{-2})^{-1/2}$, and thus, **DP-AdaFEST is privacy-wise equivalent to a sub-sampled Gaussian mechanism with scale $\\sigma$**.\n\nThe privacy analysis for DP-FEST is also straightforward, since it involves the composition of the first frequency filtering step and the second training step which is privacy-wise equivalent to a sub-sampled Gaussian mechanism.\n\nWe thank the reviewer again for raising this point and we will add these details in the revision.\n\n**References**:\n\n[1] Jinshuo Dong, Aaron Roth, Weijie J. Su. Gaussian Differential Privacy.\n"}, {"review_id": "JN1qnViLX3", "reviewer": "Reviewer_KMWx", "summary": "This work aims at improving the performance of DP-SGD on models with a large embedding layer. The main idea is zeroing out the insignificant coordinates such that the amount of added Gaussian noise is reduced. The proposed methods achieve substantial gradient size reduction with marginal performance loss compared with the vanilla DP-SGD.", "questions": "1. What does \"best\" mean in the term \"best gradient size reduction\"?\n2. According to my knowledge, if the sparsity pattern is changing consistently, it is hard to take advantage of this property and accelerate the computation. Please correct me if this is wrong for specific hardware, e.g. TPU.\n3. Please state the accuracy of the baseline in the main text. This is crucial for the evaluation of the conducted experiments.", "limitations": "N/A", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "1. Differential privacy is an important topic in the study of privacy and security. Making DP more practically useful is a main challenge in the current research.\n2. This work clearly clarify the background and its methodology, the paper is easy to follow.", "weaknesses": "1. The proposed method is similar to the sparse technologies in DP. There is a long line of works studying sparse technology and DP selection, many of them have also studied large embedding layer in the language models. This work is not properly placed in the contemporary literature. \n2. Applying LoRA to the embedding layer can also reduce the gradient size and obtain similar advantages, it would be interesting to compare the proposed methods to such low-rank methods. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We genuinely appreciate the valuable feedback provided by the reviewer and have addressed them in a point-by-point manner below. We are more than willing to engage in further discussions with the reviewers should any follow-up questions arise.\n\n### **Q1. Missing placement of contribution?**\n> The proposed method is similar to the sparse technologies in DP. There is a long line of works studying sparse technology and DP selection, many of them have also studied large embedding layer in the language models. This work is not properly placed in the contemporary literature.\n\n**A**: Yes, there has been extensive work on DP selection. In contrast, to the best of our knowledge, there is no prior work that studies the *sparse DP training of large embedding layers in recommender systems and language models*. If the reviewer has pointers to any such references, please let us know and we are happy to incorporate them.\n\n### **Q2. Applicability of LoRA to embedding layers**\n> Applying LoRA to the embedding layer can also reduce the gradient size and obtain similar advantages, it would be interesting to compare the proposed methods to such low-rank methods.\n\n**A**: We appreciate the reviewer’s suggestion. However, applying LoRA to the embedding layers is not a common practice due to the following reasons:\n1. LoRA was introduced as a method to efficiently adapt matrices of dimensions $n \\times d$ in language models by utilizing a rank-$r$ approximation (the initial use case considers the attention layers where $n=d$). The rank-$r$ approximation helps in reducing the memory requirements by a factor of $m \\times d/(m+d)*r < min(m,d)/r$. However, in the case of embedding layers, where $n$ represents the vocabulary size and $d$ denotes the embedding dimensionality, a notable disparity exists: the vocabulary size $n$ is typically very large (>1M in the evaluated recommendation task and ~50,000 in the evaluated NLU task), while the embedding dimensionality $d$ is in the hundreds. Consequently, **the potential for improvements using LoRA in this context is limited**.\n2. For private training of the embedding layer, using DP-AdaFEST we could still benefit from the efficient embedding lookup (i.e., row fetching operations) via customized APIs. However, LoRA would **not be able to leverage these APIs** as it requires relatively expensive matrix multiplication.\n3. While **LoRA needs to adapt a pre-trained model**, our algorithm works in both pre-training and fine-tuning, as demonstrated in the recommender system and language model experiments, respectively.\n\nTo more effectively demonstrate the arguments above, the table below compares the best embedding gradient size reductions achieved by AdaFEST and LoRA against DP-SGD for SST-2 with $\\epsilon=1.0$. We vary LoRA's rank $r$ from {4, 8, 16, 32, 64, 128}. AdaFEST consistently outperforms LoRA in gradient size reduction at similar utility levels. Additionally, AdaFEST allows effective utilization of reduced gradients through customized APIs, while the implications of LoRA's reduction remain unclear.\n\n| **Utility loss compared to DP-SGD** | **Best gradient size reduction, AdaFEST** | **Best gradient size reduction, LoRA** |\n|---|---|---|\n| 0.001 | 17.41x | 5.91x |\n| 0.005 | 62.14x | 23.64x |\n| 0.01 | 62.14x | 47.28x |\n\n### **Q3. Speedup for changing sparsity patterns?**\n> According to my knowledge, if the sparsity pattern is changing consistently, it is hard to take advantage of this property and accelerate the computation. Please correct me if this is wrong for specific hardware, e.g. TPU.\n\n**A**: Leveraging sparsity efficiently often requires specialized algorithms and hardware capable of handling sparse data. Fortunately, while the sparsity exploited by our algorithm is dynamic, it follows a consistent structure that only a small number of **rows** of the gradient of the embedding layer are non-zero. This (dynamic) structure can be efficiently exploited by modern accelerators that provide dedicated implementation for embedding lookups.\n\nFor example, Google Cloud TPUs can efficiently exploit such dynamic sparsity patterns. Appendix C.2 demonstrates significant wallclock time improvement using TPUEmbedding (please refer to footnote 6 in our Appendix C.2.1 for the link), the sparse embedding API provided by Google Cloud TPUs. This showcases TPUs' proficiency in handling sparse operations with compressed representations and efficient memory access patterns, proving their ability to make use of the dynamic sparsity patterns here.\n\n### **Q4. What does \"best\" mean in the term \"best gradient size reduction\"?**\n\n**A**:  Our experiments explore various combinations of hyperparameters (Section 4.4 has the details) and we reported the maximum gradient size reduction achieved at different utility requirements. We will clarify this in the revision.  \n\n### **Q5. Please state the accuracy of the baseline in the main text**\n\n**A**:  We appreciate the great suggestion. We will report the accuracy in the main text in the revision."}, {"review_id": "XBm5ltP096", "reviewer": "Reviewer_AuCL", "summary": "This paper targets the learning of embedding models with DP-SGD.\nThe main idea is to utilize the sparsity of gradients to the embeddings.\nIf a mask indicating the sparsity of gradients is given, one can simply adapt DP-SGD by adding noises  only to the masked dimensions.\nThey propose two variants, DP-FEST and DP-AdaFEST, where in DP-FEST the mask is computed beforehand and in DP-AdaFEST the mask will be estimated with differential privacy by thresholding noisy version of the original gradients.\nThey evaluate the proposed methods on recommendation and language understanding tasks.", "questions": "See Weakness for some other comments.\n1. It is a good practice to include the performances of non-private baselines for empirical experiments. While in many cases there can still be sizable gaps between DP methods and non-private baselines, it can help readers to understand intuitively how large the gaps remain to be.\n\n2. While quite a few work improving DP-SGD are already mentioned in introduction, there are indeed some improvements of DP-SGD that are overlooked ([1-3]). While it is fine to focus on vanilla DP-SGD for experiments since it is still popular, I suggest to discuss these work as part of the background.\n\n**reference:**\n\n[1] Wang, Wenxiao, Tianhao Wang, Lun Wang, Nanqing Luo, Pan Zhou, Dawn Song, and Ruoxi Jia. \"DPlis: Boosting Utility of Differentially Private Deep Learning via Randomized Smoothing.\" Proceedings on Privacy Enhancing Technologies 4 (2021): 163-183.\n\n[2] Shamsabadi, Ali Shahin, and Nicolas Papernot. \"Losing less: A loss for differentially private deep learning.\" (2021).\n\n[3] Park, Jinseong, Hoki Kim, Yujin Choi, and Jaewook Lee. \"Differentially Private Sharpness-Aware Training.\" arXiv preprint arXiv:2306.05651 (2023).", "limitations": "The authors addressed the limitations fairly.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The problem investigated, i.e. learning embedding models through DP-SGD, is meaningful and the proposed method is very intuitive.", "weaknesses": "1. There seems to be a mismatch of scopes between the title and the rest of the paper. The sparsity of gradients with DP-SGD is only possible with the proposed scheme when there is naturally sparsity introduced by design, such as when each sample involves only a few rows in the entire embedding matrix (i.e. in the vocabulary) and the vocabulary contains many, many embeddings. It is important to have a title accurately reflecting the scope of the work. I would suggest, for example, Sparsity-Preserving Differentially Private Training of Embedding Vocabularies.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We genuinely appreciate the valuable feedback provided by the reviewer and have addressed them in a point-by-point manner below. We are more than willing to engage in further discussions with the reviewers should any follow-up questions arise.\n\n### **Q1. A better title**\n> There seems to be a mismatch of scopes between the title and the rest of the paper. The sparsity of gradients with DP-SGD is only possible with the proposed scheme when there is naturally sparsity introduced by design, such as when each sample involves only a few rows in the entire embedding matrix (i.e. in the vocabulary) and the vocabulary contains many, many embeddings. It is important to have a title accurately reflecting the scope of the work. I would suggest, for example, Sparsity-Preserving Differentially Private Training of Embedding Vocabularies.\n\n**A**: We appreciate the reviewer’s comment.  In the revision, we will change the title to \"Sparsity-Preserving Differentially Private Training of Large Embedding Models\", to more accurately reflect the scope of our work.\n\n### **Q2. Performance of the non-private baseline**\n> It is a good practice to include the performances of non-private baselines for empirical experiments. While in many cases there can still be sizable gaps between DP methods and non-private baselines, it can help readers to understand intuitively how large the gaps remain to be.\n\n**A**:  Tables 2 & 3 in the Appendix of our submission contain the non-private baseline results for Criteo and NLU tasks.  If the reviewer thinks it’d be helpful, we could consider adding in the revision the following table which contains the non-private baseline numbers for all datasets.\n\n| Task                                     | Non-private baseline |\n|------------------------------------------|----------------------|\n|     **Recommendation task** (metric: AUC)    |                      |\n| Criteo-Kaggle                            | 0.8063               |\n| Criteo-time-series (streaming period=1)  | 0.7846               |\n| Criteo-time-series (streaming period=2)  | 0.7848               |\n| Criteo-time-series (streaming period=4)  | 0.7847               |\n| Criteo-time-series (streaming period=8)  | 0.7820               |\n| Criteo-time-series (streaming period=16) | 0.7848               |\n| Criteo-time-series (streaming period=18) | 0.7848               |\n|  **NLU task** (metric: accuracy)  |                      |\n| SST-2                                    | 94.31%               |\n| QNLI                                     | 92.38%               |\n| QQP                                      | 91.67%               |\n\n### **Q3. New references for improvements of vanilla DP-SGD**\n\n**A**: We thank the reviewer for sharing the new references that improve vanilla DP-SGD; we will add them in the revision.\n"}]

Author rebuttal:
We express our gratitude to the AC and all reviewers for their time and valuable feedback. We appreciate the acknowledgment that "the question addressed in this work is of great importance" and that "the proposed method is very intuitive/effective."

Below, we provide a summary of the comments and our corresponding responses for clarity. For more details, please refer to the individual responses to each reviewer. We are more than willing to engage in further discussions with the reviewers should any follow-up questions arise.

## **1. Our contribution and comparison with LoRA** 
- **Novelty** (Reviewer [KMWx](https://openreview.net/forum?id=sqTcCXkG4P&noteId=JN1qnViLX3), Q1): We kindly clarified that to the best of our knowledge, no prior research has addressed sparse DP training of extensive embedding layers in recommender systems and language models. We welcome any references the reviewer might have on this topic for potential inclusion.
- **Comparison with LoRA** (Reviewer [KMWx](https://openreview.net/forum?id=sqTcCXkG4P&noteId=JN1qnViLX3), Q2): We detailed the limitations of applying LoRA to embedding models, including 1) limited room for enhancement, 2) challenges in harnessing its gradient size reduction, and 3) constraints on fine-tuning (unlike our pre-training adaptable methods). Furthermore, our empirical results show that LoRA underperforms in gradient size reduction compared to our proposed approaches.

## **2. Applicability of our methods**
- **For large batch sizes in language models** (Reviewer [upr1](https://openreview.net/forum?id=sqTcCXkG4P&noteId=T4kwfmPHxH), Q1): Our demonstrated sparsity pattern shows that even with substantial batch sizes (1,024-16,384), the unique token count remains much lower (~2-14x) than the total vocabulary size. This affirms the applicability of our algorithm to larger batch sizes.
- **For larger vocabulary language models** (Reviewer [upr1](https://openreview.net/forum?id=sqTcCXkG4P&noteId=T4kwfmPHxH), Q2): We showcased our method's capability to achieve greater gradient size reduction in language models with expanded vocabularies (e.g., 250k compared to 50k in our submission).
- **Compatibility with sparsity optimization in other layers** (Reviewer [upr1](https://openreview.net/forum?id=sqTcCXkG4P&noteId=T4kwfmPHxH), Q2): We clarified that our algorithm primarily preserves gradient sparsity in embedding layers in a black box manner. However, users retain flexibility to apply desired methods for enhancing model sparsity in other layers (e.g., using LoRA for transformer blocks). 

## **3. Privacy Analysis** (Reviewer [ah8R](https://openreview.net/forum?id=sqTcCXkG4P&noteId=LZFIK6TFq5), Q1)
We clarified that our proposal is privacy-wise equivalent to a sub-sampled Gaussian mechanism, and provided more details for the privacy analysis.

## **4. Clarification questions**
We’ve also 
- Clarified that the dynamically changing gradient sparsity patterns in our methods can be effectively leveraged by customized APIs such as  TPUEmbedding (Reviewer [KMWx](https://openreview.net/forum?id=sqTcCXkG4P&noteId=JN1qnViLX3), Q3);
- Explained our evaluation metric (Reviewer [KMWx](https://openreview.net/forum?id=sqTcCXkG4P&noteId=JN1qnViLX3), Q4).

## **5. Planned changes to the manuscript**
We will also make the following changes in the revision as suggested:
- (Reviewer [AuCL](https://openreview.net/forum?id=sqTcCXkG4P&noteId=XBm5ltP096), Q1) Change the title to "Sparsity-Preserving Differentially Private Training of Large Embedding Models", to more accurately reflect the scope of our work;
- (Reviewer [ah8R](https://openreview.net/forum?id=sqTcCXkG4P&noteId=LZFIK6TFq5), Q1) Provided a detailed privacy analysis for the proposed method;
- (Reviewers [KMWx](https://openreview.net/forum?id=sqTcCXkG4P&noteId=JN1qnViLX3) Q5 & [AuCL](https://openreview.net/forum?id=sqTcCXkG4P&noteId=XBm5ltP096), Q2) Report non-private baseline numbers for all datasets;
- (Reviewer [upr1](https://openreview.net/forum?id=sqTcCXkG4P&noteId=T4kwfmPHxH), Q3) Provide more details for the hyper-parameters for NLU tasks;
- (Reviewer [AuCL](https://openreview.net/forum?id=sqTcCXkG4P&noteId=XBm5ltP096), Q3) Add new references for improvements of vanilla DP-SGD;
- (Reviewer [upr1](https://openreview.net/forum?id=sqTcCXkG4P&noteId=T4kwfmPHxH)) Fix typos.


Meta-review: The reviewers were thoughtful enough to participate in a lively discussion. The remaining negative reviewer clearly outlined their main gripes with the paper (listed below, verbatim). The other reviewers, who were borderline, didn't raise much disagreement with these critiques, but varied as to whether these should preclude acceptance or not. I used my own judgement, and believe that while these drawbacks are non-trivial, they are all addressable for a final camera ready version, without requiring additional careful checking by reviewers. They are mostly related to the writing, and toning down overclaiming. While NeurIPS does not have a formal "revision" option, given the public nature of these meta-reviews, I trust the authors will appropriately address them in the final version (which I look forward to reading in depth myself). 

On the positive side, I and the reviewers agree that maintaining sparsity in embedding layers is an interesting, innovative, and important direction. It hasn't received much attention yet, and this paper highlights the problem and provides a nice solution. Given the merits of this paper, and the drawbacks being largely addressible, I judge that this paper should be accepted.

---

1. A Discrepancy between claimed and actual contributions
During the rebuttal, the authors specified the contribution of this work as: "...we specifically target the issue of diminished gradient sparsity encountered when employing DP-SGD with large embedding models due to the addition of dense noise." In the paper, they claim that: "To the best of our knowledge, this study is the first to address the technical challenges of applying DP-SGD to large embedding models." These two expressions are essentially different: the former suggests a specific solution, while the latter asserts a pioneering effort to address a general problem. The actual contribution of this work falls short of what is claimed in the paper.

2. Major revision
The authors overlooked that many previous works have taken the same perspective on increasing optimization efficiency. Moreover, the inherent sparsity of the embedding layer has been observed and utilized in earlier works, e.g., [1]. Additionally, the authors made a mistake in one of the main experiments, presenting the benefit of preserving sparsity (c.f. https://openreview.net/forum?id=sqTcCXkG4P&noteId=BmDe9HygLq). The authors have promised to add necessary discussions and update experimental results.

3. Experiments
Although preserving sparsity in the embedding layer can improve optimization efficiency, its effect might be marginal when the embedding layer is not dominant in the network architecture. The authors have only presented results relating to gradient reduction and efficiency improvement in the embedding layer, making the practical benefits unclear. They have promised to increase transparency regarding the final profit.
Decision: Accept (poster)