SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Additionally, prior research has also made efforts to improve the efficiency of DP-trained language models. For instance, [LTLH21] observes that embedding layers can significantly contribute to memory consumption during DP training of large language models. They therefore introduce ghost clipping to reduce the memory impact of embedding layers by avoiding per-example gradient generation. While our primary focus is on improving model efficiency, our methods also result in reduced memory usage as a secondary benefit due to gradient size reduction. Besides,  $\left[\mathrm{YNB}^{+}22\right]$  proposes using parameter-efficient fine-tuning techniques in DP training but does not explore effective training of large embedding layers under DP: They freeze the embedding layers during DP fine-tuning and explore DP's compatibility with parameter-efficient fine-tuning methods, such as LoRA  $\left[\mathrm{HSW}^{+}22\right]$  and Adapter  $\left[\mathrm{HGJ}^{+}19\right]$ , for attention layers. It is worth noting that, as discussed in Section 4.4, when LoRA is applied to embedding layers, the gradient size reduction achieved is inferior to the reductions introduced by our proposed methods.
In addition, it is worth mentioning that partition selection is also a technique for feature selection; its primary goal is to confidentially identify elements, along with their respective frequencies, that occur at least a specified number of times [KKMN09, DVGM22,  $\mathrm{GGK}^{+}20$ , CGSS21, CWG22, SDH23]. However, in our current work, we introduce a new algorithm as it is more compatible with DP-SGD.
B.1 DP Top- $k$  Selection
Assuming there are  $c$  unique buckets in  $L$ , our goal is to return the top- $k$  frequent buckets from  $L$  in a private way. In our scenario, each user's data can modify the count of a single bucket from a given feature by at most 1, ensuring that a user contributes to at most one bucket. Therefore, the  $\ell_{\infty}$ -sensitivity of the counting function for each feature is 1. To achieve DP top- $k$  selection, we
utilize the algorithm introduced in [DR21] (see Algorithm 2). The algorithm begins by counting the frequency  $h \in \mathbb{N}^c$  of each bucket in the list. Next, it injects Gumbel noise into  $h$  and subsequently returns the top- $k$  buckets based on the noisy frequency.
Algorithm 2: One-shot DP Top-  $k$  Selection [DR21].
Input : The number of unique buckets  $c$ , a list of bucket occurrences of size  $l$ ,  $L \in [c]^l$ , privacy budget  $\varepsilon$ , the number of selected buckets  $k$ .
Output: Top-  $k$  frequent buckets selected, with DP, from  $L$ .
1 Compute the bucket frequency in  $L$ ;
2 for  $i = 1$  to  $l$  do
3  $\left\lfloor h[L[i]]\rightleftharpoons h[L[i]] + 1;$
4 Add Gumbel noise to the bucket frequency,  $\hat{h}\gets h + X$  , where  $X_{i}\sim \mathrm{Gumbel}(1 / \varepsilon)$
5  $r\gets \arg \max_k\{\hat{h}_i:i\in [c]\}$  , where arg maxk returns the indices of the largest k elements in a vector;
6 Return  $r$
In our experiments, when selecting the top- $k$  buckets from all  $p$  features, we distribute the privacy budget  $\varepsilon$  and selection budget  $k$  equally among the  $p$  features. We then perform DP Top-int $(k/p)$  selection, allocating a budget of  $\varepsilon/p$  for each feature. Finally, we concatenate the outputs from different features to obtain the final set of selected buckets. We set  $\varepsilon = 0.01$  in our experiments and deduct the value from the total privacy budget accordingly as the budget for DP training.
B.2 Memory Efficient Filtering
Implementing the computation of the contribution map and thresholding (Lines 6 and 8 in Algorithm 1) can naively take  $O(c)$  time and space, since  $v_{i}$ 's are  $c$ -dimensional. This can be prohibitively expensive when  $c$  is much larger than the size of the gradients. Here we sketch a different approach for computing the contribution map in time and space cost that is linear in the size of the gradient.
Let  $\hat{V}_t = \sum_{i\in \mathcal{B}_t}[v_i]_{C_1}$ . For all  $j\in [c]$ , we have that  $\operatorname*{Pr}[V_t[j]\geq \tau ] = \Psi \left(\frac{\tau - \hat{V}_t[j]}{\sigma_1^2\cdot C_1^2}\right)$ , where  $\Psi (t)\coloneqq \operatorname *{Pr}_{Z\sim \mathcal{N}(0,1)}[Z\geq t]$  is the survival function of the Gaussian distribution.
With this notation, a more efficient algorithm is as follows: For each  $j$  such that  $\hat{V}_t[j] \neq 0$ , we sample the bits  $V_t[j]$  accordingly. Let  $c'$  be the number of coordinates  $j$  such that  $\hat{V}_t[j] = 0$  and for simplicity, we assume w.l.o.g. that these coordinates are contiguous. For sampling the other coordinates of  $V_t$ , we rely on the following generic approach for sampling a long binary vector, with each bit being 1 with probability  $p = \Psi(\tau / \sigma_1^2 C_1^2)$ . Namely, we observe that the difference between the positions of two consecutive 1s is distributed as a geometric random variable  $\operatorname{Geom}(p)$ , with probability mass function  $p(1 - p)^{k-1}$ . Thus, we sample these indices directly by sampling the differences iteratively from  $\operatorname{Geom}(p)$ , until the sum of these exceeds  $c'$ . This step runs in time that is linear in the number of false positives, which in expectation is  $c'p$ , which is proportional to the number of non-zeros in  $G_t$ .
C Privacy Analysis
C.1 Dominating Pairs for Privacy Analysis
The  $e^{\varepsilon}$ -hockey stick divergence between  $P$  and  $Q$  is given as  $\mathcal{D}_{e^{\varepsilon}}(P||Q) = \sup_{S} P(S) - e^{\varepsilon} Q(S)$ . Observe that a mechanism  $M$  satisfies  $(\varepsilon, \delta)$ -DP if for all adjacent datasets  $D \sim D'$  it holds that  $\mathcal{D}_{e^{\varepsilon}}(M(D)||M(D')) \leq \delta$ .
Definition C.1 (Definition 7 in [ZDW22]). A pair  $(P, Q)$  of random variables over  $\Omega$  dominates a pair  $(P', Q')$  of random variables over  $\Omega'$  (denoted  $(P', Q') \preceq (P, Q)$ ) if for all  $\varepsilon \in \mathbb{R}$ , it holds that  $\mathcal{D}_{e^{\varepsilon}}(P' || Q') \leq \mathcal{D}_{e^{\varepsilon}}(P || Q)$ .
Remark. The notion of "dominates" is symmetric, namely if  $(P', Q') \preceq (P, Q)$  then  $(Q', P') \preceq (Q, P)$ . This follows because  $\mathcal{D}_{e^{\varepsilon}}(Q||P) = (1 - e^{\varepsilon}) + e^{\varepsilon} \dot{D}_{e^{-\varepsilon}}(P||Q)$ .
To provide an example, we recall the Gaussian mechanism  $M_{\mathrm{Gauss}}^{\sigma}$ , which takes as input vectors  $x_1, \ldots, x_n \in \mathbb{R}^d$  with  $\| x_i \|_2 \leq 1$  and returns  $\sum_i x_i + \mathcal{N}(\mathbf{0}, \sigma^2 I_d)$ , where  $\sigma$  denotes the noise multiplier.
Lemma C.2 (Gaussian mechanism). For any adjacent datasets  $D \sim D'$ , it holds that  $(M(D), M(D')) \preceq (\mathcal{N}(1, \sigma^2), \mathcal{N}(0, \sigma^2))$ .
We use two key behaviors of dominating pairs, namely under composition and Poisson subsampling.
Composition. For any mechanism  $M_1$ , and mechanism  $M_2$ , let  $M = (M_1, M_2)$  denote the composed mechanism, namely, one that outputs  $(y_1, y_2)$  sampled as  $y_1 \sim M_1(D)$  and  $y_2 \sim M_2(D; y_1)$ .
Lemma C.3 (Theorem 10 in [ZDW22]). For any adjacent datasets  $D$  and  $D'$ , if  $(M_1(D), M_1(D')) \preceq (P_1, Q_1)$  and  $(M_2(D; y), M_2(D'; y)) \preceq (P_2, Q_2)$  for all values of  $y$ , then  $(M(D), M(D')) \preceq (P_1 \times P_2, Q_1 \times Q_2)$ , where  $P_1 \times P_2$  refers to the product distribution with marginals  $P_1$  and  $P_2$ .
Poisson Subsampling. For any  $\gamma \in (0,1]$ , let  $S_{\mathrm{Poi}}^{\gamma}$  denote the mechanism that takes as input a dataset of arbitrary size, and returns a dataset by including each datapoint with probability  $\gamma$  i.i.d. at random.
Lemma C.4 (Theorem 11 in [ZDW22]). Let  $M$  be a mechanism and  $(P, Q)$  be a pair of distributions, such that for any adjacent datasets  $D \sim D'$  with  $D$  containing one more datapoint than  $D'$ , it holds that  $(M(D), M(D')) \preceq (P, Q)$ . Then for  $\hat{M} = M \circ S_{\mathrm{Poi}}^{\gamma}$ , it holds that  $(\hat{M}(D), \hat{M}(D')) \preceq ((1 - \gamma)Q + \gamma P, Q)$ .
Finally, we recall the basic composition theorem of DP.
Lemma C.5. If  $M$  satisfies  $(\varepsilon, \delta)$ -DP and  $M'$  satisfies  $(\varepsilon', \delta')$ -DP, then the composed mechanism  $(M, M')$  satisfies  $(\varepsilon + \varepsilon', \delta + \delta')$ -DP.
C.2 Privacy Analysis of DP-SGD
We recall the privacy analysis of DP-SGD using privacy random variables, following  $\left[\mathrm{ACG}^{+}16\right]$ , under Poisson sub-sampling of mini-batches. Observe that the computation of noisy gradient in each step of DP-SGD corresponds to an invocation of the Gaussian mechanism with noise multiplier of  $\sigma$ . Hence, by combining Lemmas C.2 to C.4, we have that for any adjacent datasets  $D$  and  $D'$  with  $D$  containing one more datapoint than  $D'$ , it holds for  $M$  being DP-SGD that  $(M(D), M(D')) \preceq (P^{\times T}, Q^{\times T})$  where  $P = (1 - \gamma)\mathcal{N}(0, \sigma^2) + \gamma \mathcal{N}(1, \sigma^2)$ ,  $Q = \mathcal{N}(0, \sigma^2)$ , and  $T$  denotes the number of steps.
Thus, it suffices to choose  $\sigma$  such that  $\max \{\mathcal{D}_{e^{\varepsilon}}(P^{\times T}||Q^{\times T}),\mathcal{D}_{e^{\varepsilon}}(Q^{\times T}||P^{\times T})\} \leq \delta$  for desired privacy parameters  $(\varepsilon ,\delta)$
C.3 Privacy Analysis of DP-FEST
DP-FEST is simply the composition of two mechanisms: the first one being Algorithm 2 (that satisfies  $(\varepsilon_{1},0)$ -DP and DP-SGD with noise multiplier  $\sigma$ . Using Lemma C.5, and the above analysis of DP-SGD, it suffices to choose  $\sigma$  such that  $\max \{D_{e^{\varepsilon -\varepsilon_1}}(P^{\times T}||Q^{\times T}),D_{e^{\varepsilon -\varepsilon_1}}(Q^{\times T}||P^{\times T})\} \leq \delta$  for desired final privacy parameters of  $(\varepsilon ,\delta)$ .
C.4 Privacy Analysis of DP-AdaFEST
DP-AdaFEST can be analyzed essentially identically to DP-SGD. We observe that each step of DP-AdaFEST involves computing the private contribution map, and then the noisy gradient subject to the private contribution map. The private contribution map is computed via the Gaussian mechanism with noise multiplier  $\sigma_{1}$  and the noisy gradient is also computed via the Gaussian mechanism with noise multiplier of  $\sigma_{2}$ . From [DRS19] (Corollary 3.3), it is known that the composition of two Gaussian mechanisms with noise multipliers of  $\sigma_{1}$  and  $\sigma_{2}$  respectively is equivalent to a Gaussian mechanism with noise multiplier  $\sigma = (\sigma_{1}^{-2} + \sigma_{2}^{-2})^{-2}$ , i.e., for  $M_0 = (M_{\mathrm{Gauss}}^{\sigma_1}, M_{\mathrm{Gauss}}^{\sigma_2})$ , it holds for any adjacent datasets  $D \sim D'$  that  $(M_0(D), M_0(D')) \preceq (\mathcal{N}(1, \sigma^2), \mathcal{N}(0, \sigma^2))$ . Hence, by combining with Lemmas C.3 and C.4, we have that for any adjacent datasets  $D$  and  $D'$  with  $D$  containing one more datapoint than  $D'$ , it holds for  $M$  being DP-AdaFEST that  $(M(D), M(D')) \preceq (P^{\times T}, Q^{\times T})$  where  $P = (1 - \gamma)\mathcal{N}(0, \sigma^2) + \gamma \mathcal{N}(1, \sigma^2)$ ,  $Q = \mathcal{N}(0, \sigma^2)$ , and  $T$  denotes the number of steps.
Thus, it suffices to choose  $\sigma$  such that  $\max \{\mathcal{D}_{e^{\varepsilon}}(P^{\times T}||Q^{\times T}),\mathcal{D}_{e^{\varepsilon}}(Q^{\times T}||P^{\times T})\} \leq \delta$  for desired privacy parameters  $\varepsilon ,\delta$
C.5 Numerical Privacy Accounting using Privacy Loss Distributions
Thus, in order to choose the noise parameters for all the above algorithms, the core primitive needed is to compute the smallest  $\sigma$  such that  $\max \{\mathcal{D}_{e^{\varepsilon}}(P^{\times T}||Q^{\times T}),\mathcal{D}_{e^{\varepsilon}}(Q^{\times T}||P^{\times T})\} \leq \delta$  for desired privacy parameters  $\varepsilon ,\delta$ , where  $P = (1 - \gamma)\mathcal{N}(0,\sigma^2) + \gamma \mathcal{N}(1,\sigma^2),Q = \mathcal{N}(0,\sigma^2))$
Several works have proposed numerical algorithms based on privacy loss distributions [KJH20, GLW21, GKKM22,  $\mathrm{DGK}^{+}22]$  for privacy accounting and have in particular studied the specific pair  $(P^{\times T},Q^{\times T})$  above. In particular, we use the open-source implementation available in Google's DP library [Goo20] for this privacy accounting.
D Experimental Details and Other Results
In this section, we provide experimental details for recommendation tasks and NLU tasks.
D.1.1 Click-Through Rate (CTR) Prediction Tasks
The pCTR model's architecture. The pCTR model we use comprises a total of six layers. The first layer utilizes an embedding layer to map each categorical feature to a dense feature vector. Appendix D.1.1 gives the vocabulary size of each categorical feature. We determine the embedding dimension using a heuristic rule of  $\mathrm{int}(2V^{0.25})$ , where  $V$  represents the number of unique tokens in each categorical feature. The resulting dense features are combined with the log-transformed integer features to create the first layer representation. We then apply four fully connected layers to this representation, each with an output dimension of 598 and a ReLU activation function. Finally, we use another fully connected layer to compute the scalar prediction as the final output.
Feature name	Vocabulary size
categorical-feature-14	1,472
categorical-feature-15	577
categorical-feature-16	82,741
categorical-feature-17	18,940
categorical-feature-18	305
categorical-feature-19	23
categorical-feature-20	1,172
categorical-feature-21	633
categorical-feature-22	3
categorical-feature-23	9,090
categorical-feature-24	5,918
categorical-feature-25	64,300
categorical-feature-26	3,207
Feature name	Vocabulary size
categorical-feature-27	27
categorical-feature-28	1,550
categorical-feature-29	44,262
categorical-feature-30	10
categorical-feature-31	5,485
categorical-feature-32	2,161
categorical-feature-33	3
categorical-feature-34	56,473
categorical-feature-35	17
categorical-feature-36	15
categorical-feature-37	27,360
categorical-feature-38	104
categorical-feature-39	12,934
Table 3: Vocabulary size of 26 categorical features in the Criteo dataset.
Hyper-parameters. For DP-SGD, we fine-tune the clipping norm and report the best accuracy achieved. When evaluating DP-FEST, we adjust the hyper-parameter  $k$ , which represents the number of preserved top buckets, with values ranging from 100 to 300,000. Regarding DP-AdaFEST, we tune the following hyper-parameters:
$\triangleright$  The ratio of noise added to the contribution map to the one added to the sparse gradient,  $\sigma_1 / \sigma_2$ , with options of 0.1 to 10.
$\triangleright$  The thresholding value  $\tau \in \{0.5, 1.0, 5.0, 10.0, 20.0, 50.0, 100.0\}$ .
$\triangleright$  The clipping norm for gradient contribution  $C_1 \in \{1.0, 5.0, 10.0\}$ .
D.1.2 NLU Tasks
Datasets. Our evaluation of NLU tasks uses three downstream classification tasks from the GLUE benchmark  $\left[\mathrm{WSM}^{+}19\right]$ :
$\triangleright$  The SST-2 dataset  $[\mathrm{SPW}^{+}13]$  is a sentiment analysis dataset. The task is to predict whether a movie review is positive or negative.
$\triangleright$  The QNLI dataset [RZLL16] is a question-answering dataset. Each example consists of a question and a corresponding sentence, and the task is to predict whether the sentence contains the answer to the question or not.
$\triangleright$  The QQP dataset  $[ \mathrm{IYW} + 17 ]$  is a collection of question pairs that are labeled as either duplicate or not duplicate. The task is to predict whether a given pair of questions are duplicates or not.
Hyper-parameters. For DP-SGD, we fine-tune the clipping norm and report the best accuracy achieved. When evaluating DP-FEST, we adjust the hyper-parameter  $k$ , which represents the number of preserved top buckets, with values ranging from 1,000 to 50,000. Regarding DP-AdaFEST, we tune the following hyper-parameters:
$\triangleright$  The ratio of noise added to the contribution map to the one added to the sparse gradient,  $\sigma_1 / \sigma_2$ , with options of 0.1 to 10.
$\triangleright$  The thresholding value  $\tau \in \{50,100,200,500,1,000\}$
$\triangleright$  The clipping norm for gradient contribution  $C_1 \in \{50.0, 100.0, 500.0\}$ .
D.2 More Experiment Results
D.2.1 Wall-Clock Time Reduction of Sparsity-Preserving DP-SGD Compared with DP-SGD
We conduct simulation experiments utilizing tf.keras.layers.Embedding<sup>6</sup>, the sparse embedding API provided by Tensorflow that supports sparse lookup and sparse gradient computation, to assess the efficiency improvement of sparsity-preserving DP training compared to vanilla DP-SGD. Our simulations create an embedding layer with a fixed embedding dimension of 64 and batch size of 1,024. We investigate the impact of changing the vocabulary size  $|V|$  on the wall-clock time improvement. Table 4 reports the training time for 100 steps for DP-SGD and DP-AdaFEST, along with the wall-clock time reduction factor. We observe significant wall-clock time improvement in sparse updates, especially for larger vocabulary sizes: For instance, when the vocabulary size is 10M, we are able to achieve  $>150\times$  wall-clock time improvement over vanilla DP-SGD; For evaluation in Section 4, the Criteo models have a vocabulary of 1.7M, where our approach translates to a wall-clock time improvement ranging from 20 to  $40\times$ .
Vocabulary Size	DP-SGD	Ours	Wall-Clock Time Reduction Factor
1 × 105	0.985	0.324	3.045
2 × 105	1.676	0.312	5.376
1 × 106	6.607	0.318	20.746
2 × 106	13.09	0.323	40.515
5 × 106	31.326	0.332	94.340
1 × 107	62.473	0.353	176.760
Table 4: Wall-clock time improvement by our methods over DP-SGD under different vocabulary sizes.
This efficiency improvement comes from two sources:
$\triangleright$  The cost of a sparse update (adding a sparse gradient tensor to the dense weight tensor) is smaller than that of a dense update, and this gap increases with the tensor size. For example, in the Tensorflow optimizers, scatter_add() is used when the gradient tensor is sparse to take advantage of this.
The computation overhead of generating a dense tensor of Gaussian noises at each step becomes significant in large embedding layers and the cost also scales with the embedding layer size.
D.2.2 Recommendation Tasks
Sparsity-preserving DP-SGD offers customizable options for balancing efficiency and utility.
The scatter plot in Figure 8 showcases how both DP-AdaFEST and DP-FEST present a range of
$^{6}$ https://www.tensorflow.org/api_docs/python/tf/keras/layers/Embedding
possibilities for effectively balancing utility and efficiency by leveraging sparsity-controlling parameters. We further note that both approaches yield an improved trade-off between privacy and utility compared to the previously suggested DP-SGD with exponential selection [ZMH21].
Figure 8: Comparison of the utility/efficiency trade-off between DP-SGD, DP-SGD w/ exponential selection [ZMH21], DP-FEST, and DP-AdaFEST on the Criteo-Kaggle dataset. DP-FEST, DP-AdaFEST, and DP-SGD w/ exponential selection can offer a more customized approach for meeting varying requirements of gradient sparsity and utility than DP-SGD, while DP-AdaFEST achieves the best trade-off.
Effect of the streaming period on pCTR model's utility. In Table 5, we present the evaluation AUC of Criteo-time-series under different streaming periods, with varying  $\varepsilon$  values, for both vanilla DP-SGD and non-private training. Our results indicate that, for DP training, the AUC values increase as the streaming period increases, regardless of the value of  $\varepsilon$  (i.e.,  $\varepsilon = 1.0$ ,  $\varepsilon = 3.0$ , and  $\varepsilon = 8.0$ ). Conversely, non-private training results in similar AUC values across different streaming periods, which suggests that the model is less susceptible to distribution shifts in the data. This observation implies that DP training is more vulnerable to such shifts, thus explaining the effectiveness of DP-AdaFEST, which adapts to distribution shifts.
Streaming period	ε = 1.0	ε = 3.0	ε = 8.0	Non-private (ε = ∞)
1	0.7313	0.7323	0.7345	0.7846
2	0.7313	0.7324	0.7346	0.7848
4	0.7314	0.7326	0.7352	0.7847
8	0.7316	0.7328	0.7352	0.7820
16	0.7318	0.7331	0.7356	0.7848
18	0.7320	0.7332	0.7358	0.7848
Table 5: Evaluation AUC of Criteo-time-series model under vanilla DP-SGD and non-private training with different streaming periods. The streaming period affects DP-SGD models but not non-private models.
Joint impact of hyper-parameters in DP-AdaFEST. In Figure 9, we present the interplay between the noise ratio  $\sigma_1 / \sigma_2$  and  $\tau$  on the utility and embedding gradient size for the Criteo-Kaggle dataset. Our findings indicate that the optimal selection of hyper-parameters is critical for achieving a desirable trade-off between high utility and small gradient size. Specifically, we observe that darker colors on both heatmaps correspond to optimal hyper-parameter choices. Such choices are generally clustered in the middle-lower regions of the heatmaps, where we see a prevalence of high values of  $\sigma_1 / \sigma_2$  and medium values of  $\tau$ . These results suggest that the choice of hyper-parameters should be approached with great care and attention to detail, as it can have a significant impact on the performance of DP models.
Figure 9: Heatmap illustrating the joint impact of the noise ratio  $\sigma_1 / \sigma_2$  and  $\tau$  on the utility and embedding gradient size for the Criteo-Kaggle dataset ( $\varepsilon = 1.0$ ).
D.2.3 NLU Tasks
Training word embedding layers in NLU improves accuracy. As shown in Table 6, allowing gradient updates for word embedding layers in the NLU tasks improves the accuracy of DP training.
SST-2	QNLI
Non-private	94.31%	92.38%
DP-SGD, ε = 1.0	91.85%	83.49%
DP-SGD, ε = 1.0 (embedding frozen)	91.43%	83.11%
DP-SGD, ε = 3.0	92.43%	84.32%
DP-SGD, ε = 3.0 (embedding frozen)	91.51%	84.02 %
DP-SGD, ε = 8.0	92.87%	86.34%
DP-SGD, ε = 8.0 (embedding frozen)	92.66%	86.28%
Table 6: Accuracy comparison of non-private training and DP-SGD with and without frozen word embeddings on four language understanding datasets with varying levels of privacy budget  $(\varepsilon)$ .