SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section we evaluate the performance of our sparsity-preserving training algorithms and compare them against vanilla DP-SGD on both the recommendation and language understanding tasks (Section 4.1). To understand adaptivity, we evaluate our algorithms on a time-series version of the data (Section 4.3); the non-time-series evaluations are in Section 4.2. We further demonstrate the applicability of our methods on language models in Section 4.4. The impact of hyper-parameters on the trade-off between utility and embedding gradient size is discussed in Section 4.5.
4.1.1 Datasets and Models
Click-through rate prediction tasks. We evaluate our algorithms on the widely-used Criteo predicted click-through rate (pCTR) dataset<sup>4</sup>, which includes over four billion ad impressions over 24 days. Each impression is represented by 13 numerical features and 26 categorical features. The objective is to predict how likely a user clicks on an ad based on these features.
The pCTR model we experiment with is a neural network that uses embedding layers for categorical features and log transformations for numerical features, followed by several fully connected layers. We use binary cross-entropy loss as the training objective and report the AUC as the evaluation metric. For further details about the training setup, please refer to Appendix D.1.
We evaluate two Criteo variants with different purposes:
4https://ailab.criteo.com/download-criteo-1tb-click-logs-dataset
Figure 3: A comparison of the best gradient size reduction achieved by DP-AdaFEST, DP-FEST, and DP-SGD with exponential selection [ZMH21] compared to DP-SGD at different thresholds for utility difference. A higher curve indicates a better utility/efficiency trade-off. DP-AdaFEST consistently outperforms DP-FEST.
Figure 4: A comparison between DP-AdaFEST, DP-FEST, and the combined algorithm (DP-AdaFEST+) on the Criteo-Kaggle dataset for different values of  $\varepsilon$ . The combined algorithm significantly outperforms either alone. Figure 6 demonstrates similar findings on the Criteo-time-series dataset.
$\triangleright$  Criteo-Kaggle<sup>5</sup>, a subset of examples released by Kaggle ( $\sim$ 45 million examples in total), is widely used in previous studies of click-through rate modeling. We use this to benchmark the performance of our algorithms. Note that timestamps are absent in Criteo-Kaggle.
$\triangleright$  Criteo-time-series, also commonly known as "Criteo-1TB", is the entire Criteo dataset of over 4 billion examples and has the auxiliary information indicating which day the data was collected. To simulate a real-world online training scenario, we train the model on the first 18 days of data and evaluate it on subsequent days (i.e., days 19–24); more details on the training process are in Section 4.3.
Language understanding tasks. For language understanding tasks, we employ language models from the BERT family [DCLT19], specifically the RoBERTa model  $\left[\mathrm{LOG}^{+}19\right]$ . This model has a vocabulary size of 50, 265 subword tokens and has been pre-trained on public web data.
We fine-tune the RoBERTa model for downstream classification tasks from the GLUE benchmark  $\left[\mathrm{WSM}^{+}19\right]$ , including SST-2  $\left[\mathrm{SPW}^{+}13\right]$ , QNLI [RZLL16], and QQP  $\left[\mathrm{IYW}^{+}17\right]$ . Following  $\left[\mathrm{YNB}^{+}22\right]$ , we adopt Low-Rank Adaptation (LoRA)  $\left[\mathrm{HSW}^{+}22\right]$  to introduce trainable rank decomposition matrices into each transformer block of the language model. This approach significantly reduces the number of parameters required for downstream tasks while also improving the privacy-utility trade-off (see Appendix D.1). A notable difference in our approach compared to  $\left[\mathrm{YNB}^{+}22\right]$  is that we also train the word embedding layers in DP fine-tuning to enable bucket selection. This leads to significant accuracy improvements in the model, as shown in Table 6.
4.1.2 Baseline and Algorithms
We evaluate the performance of vanilla DP-SGD, and the three sparsity-preserving algorithmic variants, DP-SGD with exponential selection [ZMH21], DP-FEST and DP-AdaFEST, on both the click-through rate prediction and language understanding tasks. We use a fixed batch size of 2,048 for the former and 1,024 for the latter. For all tasks, we set the privacy parameter  $\delta$  to  $1 / N$ , where  $N$  is the number of training examples for the respective task.
4.2 Evaluation on Non-Time-Series Data
We begin by considering datasets with no timestamp—Criteo-Kaggle, SST-2, QNLI, and QQP.
The decision to prioritize gradient sparsity (a key factor impacting efficiency) or utility in DP training usually depends on the task and available computational resources. However, it can be challenging for vanilla DP-SGD to cater to different needs of gradient sparsity or utility since it lacks mechanisms for trading off between these objectives. In contrast, DP-SGD with exponential selection [ZMH21],
<sup>5</sup>http://labs.criteo.com/2014/02/kaggle-display-advertising-challenge-dataset
Figure 5: A comparison between DP-AdaFEST and DP-FEST for time-series data, using the Criteo-time-series dataset with different streaming periods  $T$  and  $\varepsilon = 1.0$ . DP-AdaFEST consistently achieves a higher gradient reduction than DP-FEST at the same level of utility.
DP-AdaFEST and DP-FEST can all offer diverse options for balancing utility and efficiency via the sparsity-controlling parameters (see Figure 8 in Appendix D.2), while our proposals DP-AdaFEST and DP-FEST offer much better privacy-utility loss.
While both DP-FEST and DP-AdaFEST offer ways to balance efficiency and utility in DP training, DP-AdaFEST stands out as the more versatile and customizable algorithm. With three adjustable hyper-parameters, DP-AdaFEST offers more diverse results compared to DP-FEST's single knob,  $k$ , which controls the number of preserved top buckets. The effectiveness of DP-AdaFEST is evident in Figure 3, where it achieves significantly higher gradient size reduction than DP-FEST while maintaining the same level of utility. Specifically, on the Criteo-Kaggle dataset, DP-AdaFEST reduces the gradient computation cost of vanilla DP-SGD by more than  $5 \times 10^{5}$  times while maintaining a comparable AUC (of AUC loss less than 0.005). This reduction translates into a more efficient and cost-effective training process (Appendix D.2.1 shows wall-clock time improvements). Although the best reduction for language tasks is usually smaller since the vocabulary is already relatively condensed, the adoption of sparsity-preserving DP-SGD effectively obviates the dense gradient computation. Furthermore, in line with the bias-variance trade-off presented in Section 3.4, we note that DP-AdaFEST occasionally exhibits superior utility compared to DP-SGD when the reduction in gradient size is minimal (i.e.,  $\gamma$  in Equation (2) is small). Conversely, when incorporating sparsity, DP-SGD with the exponential mechanism faces challenges in maintaining utility. In all configurations, it fails to achieve a tolerable level of utility loss.
We further explore the potential of integrating DP-AdaFEST with DP-FEST to enhance performance, which involves pre-selecting a subset of buckets using DP-FEST and subsequently training on this subset using DP-AdaFEST. Figure 4 shows that the combined algorithm (DP-AdaFEST+) outperforms either alone in utility/efficiency trade-off, with the best gradient size reduction being further improved to  $>10^{6}\times$ . This can be attributed to the complementary strengths of the two algorithms: DP-AdaFEST offers a more flexible approach to feature selection at the batch level, while DP-FEST provides a simple yet effective means for feature selection according to the global frequency information. Besides, the combined algorithm expands the range of choices for balancing gradient sparsity and utility through the combination of hyper-parameters from both DP-FEST and DP-AdaFEST.
4.3 Evaluation on Time-Series Data
Time-series data are notoriously challenging due to non-stationarity. In this section, we investigate if our algorithms can adapt to distribution shifts, characteristic of time-series data.
We conduct experiments using the Criteo-time-series dataset, which comprises real-world user-click data collected over 24 days (18 days for training and the rest for evaluation). To simulate the online data streaming scenario, we introduce the concept of a streaming period, a time interval during which the model is updated as new data is received. Experiments indicate a notable disparity between DP training (using vanilla DP-SGD) and non-DP training, with the former demonstrating increased susceptibility to distribution shifts (Table 5 in Appendix D.2).
Figure 5 presents the utility and efficiency trade-off of DP-AdaFEST and DP-FEST for time-series data. To explore the efficacy of DP-FEST, we investigate different sources of vocabulary frequency, including the information from the first day, all days, and streaming-based (i.e., running sum updated
per streaming period). The experimental results indicate that using streaming-based frequency information for DP-FEST is almost as effective as using all-day frequency information and significantly better than using information only from the first day. Additionally, DP-AdaFEST consistently outperforms DP-FEST, achieving more than twice the gradient reduction at the same level of utility. These findings further demonstrate the importance of adapting to the dynamic nature of time-series data at a more fine-grained level, with DP-AdaFEST able to adapt at the batch level and DP-FEST only able to adapt at the streaming period level.
We further examine the advantages of combining DP-AdaFEST and DP-FEST to enhance the utility/efficiency trade-off on the Criteo-time-series dataset. We use a streaming period of 1 and streaming-based frequency information for DP-FEST. Figure 6 demonstrates a consistent superiority of the combined approach over individual methods, which aligns with the findings presented in Section 4.3.
Figure 6: The combined approach (DP-AdaFEST+) outperforms either alone on Criteo-time-series.
4.4 Applicability to Language Models
We further demonstrate the applicability of our method to language models by comparing it with LoRA  $\left[\mathrm{HSW}^{+}22\right]$  and reporting its performance with multilingual models.
DP-AdaFEST is a better choice than LoRA for embedding layers. LoRA was introduced as a method to efficiently adapt matrices of dimensions  $n \times d$  in language models by utilizing a rank- $r$  approximation, where the initial use case considers the attention layers where  $n = d$ . The rank- $r$  approximation helps in reducing the memory requirements by a factor of  $m \times d / (m + d) * r < \min(m, d) / r$ . However, the applicability of LoRA to embedding layers is limited for several reasons:
$\triangleright$  Limited improvement for unbalanced  $n$  and  $d$ : Embedding layers typically involve a large vocabulary size ( $n$  often in the millions) and a smaller embedding dimension ( $d$  typically in the hundreds). In this context, LoRA's potential benefits are limited.
$\triangleright$  Inability to harness APIs: For private training of the embedding layer, DP-AdaFEST allows for efficient embedding lookup operations (row fetching) using custom APIs; In contrast, LoRA still relies on computationally costly matrix multiplication and cannot effectively utilize these APIs.
$\triangleright$  Limited to fine-tuning use cases: LoRA is designed for adapting pre-trained models, while our DP-AdaFEST is versatile, functioning seamlessly in both pre-training and fine-tuning scenarios.
To more effectively demonstrate the first argument above, Table 1 compares the best embedding gradient size reductions achieved by DP-AdaFEST and LoRA against DP-SGD for SST-2 with  $\varepsilon = 1.0$  on the RoBERTa model. We vary LoRA's rank  $r$  from  $\{4,8,16,32,64,128\}$ . DP-AdaFEST consistently outperforms LoRA in gradient size reduction at similar utility levels.
Utility loss compared to DP-SGD	Gradient size reduction
DP-AdaFEST	LoRA
0.001	14.59×	5.91×
0.005	53.90×	23.64×
0.01	53.90×	47.28×
Increased gradient size reduction by DP-AdaFEST for larger vocabularies. Our evaluation in Section 4.2 primarily centered around the RoBERTa model, which has a vocabulary of around 50,000 entries. However, it is important to note that many high-vocabulary models exist, with vocabulary sizes 5 to  $20 \times$  larger than RoBERTa. For these models, DP-AdaFEST could offer even more pronounced benefits.
Table 1: Gradient size reduction by LoRA and DP-AdaFEST for RoBERTa's word embeddings.
Utility loss compared to DP-SGD	Gradient size reduction
RoBERTa (|V|:50k)	XLM-R (|V|:250k)
0.001	14.59×	17.21×
0.005	53.90×	64.75×
0.01	53.90×	152.94×
Table 2: DP-AdaFEST offers more pronounced gradient size reduction for models with larger vocabulary (i.e., higher  $|V|$ ). RoBERTa results are achieved on SST-2 with  $\varepsilon = 1.0$ , and the XLM-R [CKG+20] results are achieved on the Cross-Linguinal Natural Language Inference (XNLI) dataset [CRL+18] with  $\varepsilon = 1.0$ .
Figure 7: Effect of hyper-parameters on utility and embedding gradient size, including the ratio of noise added to the contribution map to the one added to the sparse gradient  $\sigma_1 / \sigma_2$ , and the thresholding value  $\tau$ . The joint impact of two hyper-parameters on utility and embedding gradient size can be found in Figure 9 in Appendix D.2.
(b) SST-2,  $\varepsilon = 1.0$
4.5 Effect of Hyper-Parameters
Finally, we study the effect of hyper-parameters on the utility and gradient size trade-off in DP-AdaFEST. Specifically, we explore the effects of the ratio of noise added to the contribution map and the noise added to the sparse gradient (i.e.,  $\sigma_1 / \sigma_2$ ), and the thresholding value  $\tau$ .
Effect of  $\sigma_{1} / \sigma_{2}$ . We find that a larger ratio of  $\sigma_{1} / \sigma_{2}$  leads to a higher accuracy (Figure 7). Indeed, the contribution map can tolerate higher levels of noise, leading to a more accurate representation of the sparse gradient. For both Criteo-Kaggle and SST-2 datasets, a noise ratio of 5 or 10 delivers the best utility. On the other hand, a larger  $\sigma_{1} / \sigma_{2}$  results in a higher gradient density because the contribution map can tolerate higher noise levels, resulting in more zero-contribution buckets bypassing the threshold and increasing false negatives. Therefore, choosing an appropriate  $\sigma_{1} / \sigma_{2}$  is critical to achieving optimal model accuracy and gradient density.
Effect of  $\tau$ . The choice of the thresholding value,  $\tau$ , is also crucial for determining the balance between gradient sparsity and model accuracy in DP-AdaFEST. Figure 7 reveals that increasing the value of  $\tau$  usually leads to a decrease in the gradient size, which results in a sparser gradient. We also find that when  $\tau$  is small, increasing it does not significantly impact the model's performance. However, setting an excessively high value of  $\tau$  (e.g.,  $\tau > 500$  for a batch size of 1024) can cause a sharp drop in the model's accuracy, especially when the noise ratio  $\sigma_1 / \sigma_2$  is small; In such cases, important contributions may be zeroed out, resulting in a loss of information in the gradient. Again, it is critical to choose  $\tau$  to best balance gradient sparsity and model accuracy in DP-AdaFEST.
In this work we present new algorithms—DP-FEST and DP-AdaFEST—for preserving gradient sparsity in DP training, particularly in applications involving large embedding models. Our algorithms achieve significant reductions in gradient size while maintaining accuracy on real-world benchmark datasets. We also provide recommendations on hyper-parameter settings.
Limitations and future work. Despite the advancements made in our proposed algorithms, there are still limitations and areas for future work. One such area is the exploration of customized hardware acceleration for sparse DP training, which has the potential to significantly improve the training efficiency of our models. By leveraging specialized hardware, we can further optimize the computational performance and speed up the training process. Additionally, combining our algorithms with other privacy-preserving techniques (e.g., Federated Learning) could lead to new insights and applications.
Yangsibo Huang is supported by Princeton University's Wallace Memorial Fellowship.
[ACG+16] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In CCS, 2016.
[AF22] Gagik Amirkhanyan and Bruce Fontaine. Building large scale recommenders using cloud TPUs, 2022. https:// cloud.google.com/blog/topics/developers-practitioners/ building-large-scale-recommenders-using-cloud,tpus.
[CGSS21] Edith Cohen, Ofir Geri, Tamas Sarlos, and Uri Stemmer. Differentially private weighted sampling. In AISTATS, pages 2404-2412, 2021.
$\left[\mathrm{CKG}^{+}20\right]$  Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. In ACL, 2020.
$\left[\mathrm{CKH}^{+}16\right]$  Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. Wide & deep learning for recommender systems. In 1st Workshop on Deep Learning for Recommender Systems, 2016.
$\left[\mathrm{CRL}^{+}18\right]$  Alexis Conneau, Rudy Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: evaluating cross-lingual sentence representations. In EMNLP, 2018.
$\left[\mathrm{CTW}^{+21}\right]$  Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In USENIX Security, 2021.
[CGW22] Ricardo Silva Carvalho, Ke Wang, and Lovedeep Singh Gondara. Incorporating item frequency for differentially private set union. In AAAI, 2022.
$\left[\mathrm{DBH}^{+}22\right]$  Soham De, Leonard Berrada, Jamie Hayes, Samuel L Smith, and Borja Balle. Unlocking high-accuracy differentially private image classification through scale. arXiv preprint arXiv:2204.13650, 2022.
[DCLT19] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pretraining of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
[DFAP21] Friedrich Dörmann, Osvald Frisk, Lars Nørvang Andersen, and Christian Fischer Pedersen. Not all noise is accounted equally: How differentially private learning benefits from large sampling rates. In International Workshop on Machine Learning for Signal Processing, 2021.
[DGK+22] Vadym Doroshenko, Badih Ghazi, Pritish Kamath, Ravi Kumar, and Pasin Manurangsi. Connect the dots: Tighter discrete approximations of privacy loss distributions. PoPETS, 2022(4):552-570, 2022.
[DGK+23] Carson Denison, Badih Ghazi, British Kamath, Ravi Kumar, Pasin Manurangsi, Krishna Giri Narra, Amer Sinha, Avinash Varadarajan, and Chiyuan Zhang. Private ad modeling with DP-SGD. In AdKDD, 2023.
$\left[\mathrm{DKM}^{+}06\right]$  Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor. Our data, ourselves: Privacy via distributed noise generation. In EUROCRYPT, pages 486-503, 2006.
[WSC+21] Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi. DCN V2: Improved deep & cross network and practical lessons for web-scale learning to rank systems. In The Web Conference, 2021.
[WSM+19] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019.
$\left[\mathrm{WWL}^{+}22\right]$  Zehuan Wang, Yingcan Wei, Minseok Lee, Matthias Langer, Fan Yu, Jie Liu, Shijie Liu, Daniel G Abel, Xu Guo, Jianbing Dong, et al. Merlin hugeCTR: GPU-accelerated recommender system training and inference. In RecSys, 2022.
$\left[\mathrm{YNB}^{+}22\right]$  Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, et al. Differentially private fine-tuning of language models. In ICLR, 2022.
[ZDW22] Yuqing Zhu, Jinshuo Dong, and Yu-Xiang Wang. Optimal accounting of differential privacy via characteristic function. In AISTATS, pages 4782-4817, 2022.
[ZMH21] Huanyu Zhang, Ilya Mironov, and Meisam Hejazinia. Wide network learning with differential privacy. arXiv preprint arXiv:2103.01294, 2021.
$\left[\mathrm{ZXJ}^{+}20\right]$  Weijie Zhao, Deping Xie, Ronglai Jia, Yulei Qian, Ruiquan Ding, Mingming Sun, and Ping Li. Distributed hierarchical GPU parameter server for massive scale deep learning ads systems.  $MLSys$ , 2020.
$\left[\mathrm{ZZS}^{+}18\right]$  Guorui Zhou, Xiaogiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. Deep interest network for click-through rate prediction. In SIGKDD, 2018.
Embedding is a fundamental ML technique used in modern deep learning to represent non-numerical features as dense numerical vectors in a key-value store format. In particular, embedding layers play a crucial role in recommendation systems [CKH+16, WFFW17, GTY+17, ZZS+18, NMS+19, WSC+21], with models relying heavily on them for effective predictions; they usually have a small number of parameters in their dense layers but their embedding layers can contain billions of entries and multiple billions of parameters. However, these embedding layers differ significantly from those used in other deep learning tasks, such as natural language processing [MSC+13, PSM14, DCLT19], where models such as BERT [DCLT19] use fewer entries in their embedding layers but compensate with several hundred million parameters in their dense feed-forward and attention layers.
The exponential growth of data is leading to embedding tables reaching sizes spanning multiple gigabytes to terabytes, which presents unique training challenges. For instance, fitting Google's 1.2TB model  $\mathrm{[MNW^{+}22]}$ , Baidu's 10TB model  $\mathrm{[ZXJ^{+}20]}$ , and Meta's 12 TB model  $\mathrm{[MHH^{+}22]}$  onto a single compute node or accelerator requires significant effort. Therefore, optimizing the forward (sparse lookup) and backward (gradient backpropagation) passes for embedding layers is critical to unlock the high compute throughput of embedding models, particularly for recommendation systems. Recent research has focused on designing optimized embedding layers [AF22, JKL $^{+}23$ ] and developing efficient sparse optimizers [GGDS19,  $\mathrm{WWL^{+}22}$ ]. Furthermore, data parallelism techniques, such as distributing embedding tables across multiple compute nodes, have been explored in order to improve performance [Sna22].
However, optimizing embedding models trained with DP is an unexplored area. A recent work  $\left[\mathrm{DGK}^{+}23\right]$  demonstrates the effectiveness of vanilla DP-SGD in the context of recommendation tasks; however, this study does not address new challenges such as the need for efficient sparse gradient computation and adaptive noise addition to comply with privacy guarantees. Another attempt  $\left[\mathrm{NCS}^{+}22\right]$  seeks to introduce sparsity in DP-trained large embedding models through post-processing DP-SGD gradients. However, this method falls short of providing the necessary DP guarantees. The closest effort to ours is the DP-SGD with exponential selection algorithm [ZMH21], which dynamically selects embedding buckets to update based on their gradient norms. However, their evaluation is limited to small-scale Word2Vec [MCCD13] models, with results mainly presented for a fixed large  $\varepsilon = 30$ . Furthermore, our experimental results in Section 4 demonstrate that their approach leads to a significant loss in utility when applied to large-scale recommendation tasks.