SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Edward Raff*
Booz Allen Hamilton raff_edward@bah.com
Amol Khanna*
Booz Allen Hamilton khanna_amol@bah.com
Fred Lu
Booz Allen Hamilton lu_fred@bah.com
To the best of our knowledge, there are no methods today for training differentially private regression models on sparse input data. To remedy this, we adapt the Frank-Wolfe algorithm for  $L_{1}$  penalized linear regression to be aware of sparse inputs and to use them effectively. In doing so, we reduce the training time of the algorithm from  $\mathcal{O}(TDS + TNS)$  to  $\mathcal{O}(NS + T\sqrt{D}\log D + TS^2)$ , where  $T$  is the number of iterations and a sparsity rate  $S$  of a dataset with  $N$  rows and  $D$  features. Our results demonstrate that this procedure can reduce runtime by a factor of up to 2,  $200\times$ , depending on the value of the privacy parameter  $\epsilon$  and the sparsity of the dataset.
Differential Privacy (DP) is currently the most effective tool for machine learning practitioners and researchers to ensure the privacy of the individual data used in model construction. Given parameters  $\epsilon$  and  $\delta$ , on any two datasets  $\mathcal{D}$  and  $\mathcal{D}'$  differing on one example, an (approximately) differentially private randomized algorithm  $\mathcal{A}$  satisfies  $\operatorname*{Pr}[\mathcal{A}(\mathcal{D}) \in O] \leq \exp\{\epsilon\} \operatorname*{Pr}[\mathcal{A}(\mathcal{D}') \in O] + \delta$  for any  $O \subseteq \operatorname{image}(\mathcal{A})$  [1]. Note that lower values of  $\epsilon$  and  $\delta$  correspond to stronger privacy.
While DP has had many successes in industry and government, DP-based machine learning methods have made little progress for sparse high-dimensional problems [2, 3, 4, 5]. We believe that this issue arises because, to the best of our knowledge, given a dataset with  $D$  features and a training algorithm with  $T$  iterations, all current iterative DP regression algorithms require at least  $\mathcal{O}(TD)$  training complexity, as shown in Table 1. This makes it impractical to use these algorithms on any dataset with a large number of features. Our solution to this problem is to take already existing algorithms, and remove all redundant computations with mathematically equivalent steps. This ensures that, by construction, we retain all proofs of correctness — but end with a faster version of the same method.
Table 1: A summary of prior methods for solving  $L_{1}$  regularized Logistic Regression, which do not take advantage of sparsity in the input data.
Method	Complexity
Frank-Wolfe Methods [6, 7, 8, 9]	O(TND)
ADMM [10]	O(TNDM)
Iterative Gradient Hard Thresholding Methods [9, 11, 12]	O(TND)
Coordinate Descent [13]	O(TND)
Mirror Descent [8]	O(TNDM)
*Co-first authors for equal contribution.
We are interested in creating a differentially private machine learning algorithm which scales to sparse datasets with high values of  $D$ , so we look toward the LASSO regularized logistic regression model [14]. Specifically, given a dataset  $\{\mathbf{x}_1,\dots ,\mathbf{x}_N\} \in \mathbb{R}^D$  which can be represented as a design matrix  $X\in \mathbb{R}^{N\times D}$ ,  $\{y_{1},\ldots ,y_{N}\} \in \{0,1\}$ , and a maximum  $L_{1}$  norm  $\lambda$ , we wish to solve
$$
\widehat {\mathbf {w}} = \underset {\mathbf {w} \in \mathbb {R} ^ {D}: \| \mathbf {w} \| _ {1} \leq \lambda} {\arg \min } \frac {1}{N} \sum_ {i = 1} ^ {N} \mathcal {L} (\mathbf {w} \cdot \mathbf {x} _ {i}) \tag {1}
$$
where  $\mathcal{L}(\cdot)$  is the loss function. In this paper, we will use the logistic loss to avoid exploiting any closed-form updates/solutions in the linear case, but our results are still applicable to linear regression. To do so with DP, we use the Frank-Wolfe algorithm as it is well studied for  $L_{1}$  constrained optimization and regularly used in DP literature [15, 16]. Frank-Wolfe is also desirable because when properly initialized, its solutions will have at most  $T$  nonzero coefficients for  $T$  training iterations. Though DP noise can reduce accuracy and cause increased density of solutions through suboptimal updates, when considering problems with more than 1 million features, we are unlikely to perform 1 million iterations, so a benefit is still obtained [17].
We develop the first sparse-dataset friendly Frank-Wolfe algorithm to remediate the problem of no sparse algorithms for high-dimensional DP regression. Because a sparse-efficient Frank-Wolfe algorithm does not exist today even for non-private problems, our work proceeds in three contributions:
1. We analyze the numerical updates of the Frank-Wolfe algorithm to separate the task into (a) "queue maintenance" for determining the next coordinate to update and (b) sparse updates of the solution and intermediate variables. If the average column sparsity of  $X$  is  $S_{c} < D$  and the average row sparsity of  $X$  is  $S_{r} < N$ , we show for the first time that Frank-Wolfe can update its solution in  $S_{r}S_{c}$  work per iteration.
2. We show that in the non-private case, the queue to select the next coordinate can be maintained in  $\mathcal{O}(\| \mathbf{w}\| _0\log D)$  time, but is cache-unfriendly.
3. Finally, we develop a new Big-Step Little-Step Sampler for DP Frank-Wolfe that can be maintained and sampled from in  $\mathcal{O}(\sqrt{D}\log D)$  time.
We test our algorithm on high-dimensional problems with up to 8.4 million datapoints and 20 million features on a single machine and find speedups ranging from  $10 \times$  to  $2$ ,  $200 \times$  that of the standard DP Frank-Wolfe method. Critically, our approach is mathematically equivalent, and so retains all prior proofs of correctness.
The remainder of our work is organized as follows. In section 2 we will review related work in more detail and discuss the lack of sparse dataset algorithms for Frank-Wolfe and DP. Next, we develop the nonprivate and DP variants of our sparse-friendly Frank-Wolfe in section 3. In section 4 we demonstrate the empirical correctness of our algorithm, a speedup in runtime for the DP case, and new state-of-the-art accuracy in high-dimensional logistic regression. Finally, we conclude in section 5.
As best as we can find, no works have specifically considered a sparse-dataset efficient Frank-Wolfe algorithm. Additionally, we find that work studying  $L_{1}$ -regularized DP regression has not reached any high-dimensional problems. Our work is the first to show that Frank-Wolfe iterations can be done with complexity sub-linear in  $D$  for sparse datasets. It also produces a sparse weight vector.
For DP regression, the addition of noise to each variable has limited exploration of sparse datasets. Most works study only one regression problem with less than 100 dense variables or are entirely theoretical with no empirical results [11, 10, 6, 18, 19]. To the best of our knowledge, the largest scale attempt for high-dimensional logistic regression with DP is by Iyengar et al., who introduce an improved objective perturbation method for maintaining DP with convex optimization [16]. This required using L-BFGS, which is  $\mathcal{O}(D)$  complexity for sparse data and produces completely dense solution vectors w [20]. In addition, Wang & Gu attempted to train an algorithm on the RCV1 dataset but with worse results and similar big- $\mathcal{O}$  complexity to Iyengar et al. [11]. While Jain & Thakurta claim to tackle the URL dataset with 20M variables, their solution does so by sub-sampling just  $0.29\%$  of the data for training and  $0.13\%$  of the data for validation, making the results suspect and
non-scalable [21]. Lastly, a larger survey by Jayaraman & Evans shows no prior work considering the sparsity of DP solutions and all other works tackling datasets with less than 5000 features [22]. In contrast, our work does no sub-sampling and directly takes advantage of dataset sparsity in training high-dimensional problems. Our use of the Frank-Wolfe algorithm also means our solution is sparse, with no more than  $T$  non-zero coefficients for  $T$  iterations.
In addition to no DP regression algorithm handling sparse datasets, we cannot find literature that improves upon the  $\mathcal{O}(D)$  dependency of Frank-Wolfe on sparse data. Many prior works have noted this dependence as a limit to its scalability, and column sub-sampling approaches are one method that has been used to mitigate that cost [23, 15, 24]. Others have looked at distributed Map-Reduce implementations [25] or adding momentum terms [26] as a means of scaling the Frank-Wolfe algorithm. However, our method is the first to address dataset sparsity directly within the Frank-Wolfe algorithm, and can generally be applied to these prior works with additional derivations for new steps. Other methods that apply to standard regression, use the  $L_{2}$  penalty [27, 28] would require modification to use our approach. Similarly, pruning methods [29] may require adaption to account for the privacy impact.
Of particular note is [30] which tackles sub-linear scaling in the number of rows  $N$  when  $N > D$ , but is primarily theoretical. Their work is the first to introduce the idea of "queue maintenance" that we similarly leverage in this work. Despite a conceptually similar goal of using a priority queue to accelerate the algorithm, [30] relies on maximum inner product search, where our queues are of a fundamentally different structure.
To the best of our knowledge, the COPT library is the only Frank-Wolfe implementation that makes any effort to support sparse datasets but is still  $\mathcal{O}(D)$  iteration complexity, so we base our comparison against its approach [31]. No DP regression library we are aware of supports sparse datasets [32].
Throughout this section, sparsity refers to an algorithm's awareness and efficiency of handling input data which contains mostly zero values. We will first review the only current method for using Frank-Wolfe with sparse data and establish its inefficiency. We will then detail how to produce a generic framework for a sparse dataset efficient Frank-Wolfe algorithm using a queuing structure to select the next coordinate update. Having established a sparse friendly framework, we will show how to obtain  $\mathcal{O}(NS_c + T\| \mathbf{w}^*\|_0\log D + TS_rS_c)$  complexity for the non-private Frank-Wolfe algorithm by using a Fibonacci heap, where  $\mathbf{w}^*$  is the solution at convergence. Then we replace the Fibonacci heap with a sampling structure to create a DP Frank-Wolfe algorithm with  $\mathcal{O}(NS_c + T\sqrt{D}\log D + TS_rS_c)$  complexity.
3.1 Frank-Wolfe Iterations in sub- $\mathcal{O}(D)$  Time
The only work we could find of any sparse-input aware Frank-Wolfe implementation is the COPT library, which contains two simple optimizations: it pre-computes a re-used dense vector (i.e., all values are non-zero) and it uses a sparse matrix format for computing the vector-matrix product  $X \mathbf{w}$  [31]. The details are abstracted into Algorithm 1, where each line has a comment on the algorithmic complexity of each step. Note to make the algorithm DP, we have added a  $+\mathrm{Lap}\left(\frac{\lambda L\sqrt{8T\log(1 / \delta)}}{N\epsilon}\right)$  to draw noise from a zero-mean Laplacian distribution with the specified scale, where  $\lambda$  is the constraint parameter and  $L$  is the  $L_{1}$ -Lipschitz constant of the loss function  $\mathcal{L}(\cdot)$ . If a non-private Frank-Wolfe implementation is desired, this value can be ignored.
In this and other pseudo-codes, we explicitly write out all intermediate computations as they are impor
Algorithm 1 Standard Sparse-Aware Frank-Wolfe
1:	w0←0
2:	y←XTy	O(NSc)
3:	for t=1 to T-1 do
4:	v_t←Xw_t	O(NSc)
5:	q_t←∇L(v_t)	O(N)
6:	z_t←XTq_t	O(NSc)
7:	α_t←z_t- y	O(D)
8:	j	←
arg min_j |α(t)(j)+ Lap(λL√8T log 1/δ/Nε) |
O(D)
9:	dt=-wt	O(D)
10:	dt(j)←dt(j)-λ·sign(α(t)(j))	O(1)
11:	gt=-<α_t, dt>	O(D)
12:	ηt=2/t+2	O(1)
13:	wt+1=w_t+η_t dt	O(D)
14:	end for
15:	Output wT
tant for enabling sparse updates.  $\bar{\mathbf{y}}$  is an intermediate
variable for the labels in the gradient of a linear problem that is pre-computed once and reused.  $\bar{\mathbf{z}}$  is a temporary variable.  $\bar{\mathbf{q}}$  and  $\alpha$  are the gradients with respect to each row and column respectively.  $\bar{\mathbf{v}}$  is the dot product of each row with the weight vector, and  $\mathbf{d}$  is the update direction. The iteration subscript  $t$  will be dropped when unnecessary for clarity. The superscript  $(j)$  denotes updating the  $j$ 'th coordinate of a vector and leaving others unaltered.
While lines 2, 4, and 6 of the algorithm exploit the sparsity of the data, this only reduces the complexity to  $\mathcal{O}(NS_c)$  plus an additional dense  $\mathcal{O}(D)$  work for lines 7 through 13 and  $\mathcal{O}(N)$  work for line 5. This results in a final complexity of  $\mathcal{O}(TNS_C + TD)$ . For high dimensional problems, especially when  $N \ll D$ , this is problematic in scaling up a Frank-Wolfe based solver.
To derive a Frank-Wolfe algorithm that is more efficient on sparse datasets, we will assume there is an abstract priority queuing structure  $Q$  that returns the next coordinate to update  $j$  for each iteration. We will detail how to design a  $Q$  to use this algorithm in non-private and DP cases in the following two sections.
Sparse  $\mathbf{w}_t$  Updates
In line 13 of Algorithm 1, if we ignore the change to coordinate  $j$  of  $\mathbf{d}_t$ , we can write  $\mathbf{w}_{t + 1} = \mathbf{w}_t - \eta_t\mathbf{w}_t$  which can be re-written as  $\mathbf{w}_{t + 1} = (1 - \eta_t)\mathbf{w}_t$ . If we represent the weight  $\mathbf{w}_{t + 1} = \mathbf{w}\cdot w_m$  with a co-associated multiplicative scalar  $w_{m}$ , we can alter  $w_{m}\gets w_{m}\cdot (1 - \eta_{t})$  to have the same effect as altering all  $D$  variables implicitly. Then the  $j^{\mathrm{th}}$  coordinate can be updated individually, allowing line 13 of Algorithm 1 to run in  $\mathcal{O}(1)$  time. We will use the same trick to represent  $\bar{\mathbf{v}}_t = \bar{\mathbf{v}}\cdot w_m$  as it has the same multiplicative scale.
Sparse  $\alpha$  and  $\bar{\mathbf{v}}$  Updates
The dot product scores  $\bar{\mathbf{v}}_t$  and column gradients  $\alpha_{t}$  are intrinsically connected in their sparsity patterns. When the  $j^{\mathrm{th}}$  value of  $\mathbf{w}_t$  is altered, multiple values of  $\bar{\mathbf{v}}_t$  change, which propagates changes to the gradients  $\alpha_{t}$ . However, the elements  $\{i\}$  of  $\bar{\mathbf{v}}_t$  that change are only those where rows  $\{i\}$  in  $X$  use feature  $j$ . Let  $\tilde{d}_j$  represent a perturbation to the  $j$ 'th update direction. Each row  $i$  that uses the  $j$ 'th feature will then be alter the variable  $\bar{v}^{(i)}$  by  $-\eta_t\tilde{d} X[i,j]$ . This lets us handle line 10 sparsely.
Each row  $\{i\}$  that changes in  $\bar{\mathbf{v}}_t$  propagates to the values in  $\bar{\mathbf{q}}$ . We can represent the change in gradient value between iterations as  $\gamma$ , which can then be used to sparsely update the  $\alpha$  values by noting that  $X^{\top}\bar{\mathbf{q}}$  would change only by the non-zero columns of  $X[i,:]$ . So we can compute the update  $\alpha$  by  $\gamma \cdot X[i,:]$ . By updating  $\alpha$  directly, we do not need to account for the contribution of  $\bar{\mathbf{y}}$  after the first iteration.
Sparse  $g_{t}$  Updates
The final variable of interest is the Frank-Wolfe convergence gap  $g_{t} = -\alpha_{t}^{\top}\mathbf{d}_{t}$ . Instead of recomputing this every iteration, we can keep a base value  $\tilde{g}$  that is altered based on both of the prior two insights.
When  $w_{m}$  is updated, we re-scale  $\tilde{g}$  by  $(1 - \eta_t)$  and add  $\eta_t\tilde{d}\alpha^{(j)}$ , the change in the dot product
Algorithm 2 Fast Sparse-Aware Frank-Wolfe for Linear Models
1: w←0
2:wm←1
3: g←0
4: y←XTy O(NSc)
5: scale←LNε/2λ√8Tlog(1/δ)
6: Q←Priority Queue or Sampling Algorithm
7: for  $t = 1$  to  $T - 1$  do
8: if  $t = 1$  then
9:  $\bar{\mathbf{v}}\gets Xw$ $\mathcal{O}(NS_c)$
10:  $\bar{\mathbf{q}}\gets \nabla \mathcal{L}(\bar{\mathbf{v}})$   O(N)
11:  $\bar{\mathbf{z}}\gets X^{\top}\bar{\mathbf{q}}$  0(NSc)
12:  $\alpha \gets \bar{\mathbf{z}} -\bar{\mathbf{y}}$  0(D)
13:  $Q.\mathbf{add}(j,|\alpha^{(j)}|\cdot scale)\forall j\in$ $\{1,\ldots ,D\}$  0(D)
14: end if
15:  $j\gets Q.\mathrm{getNext()}$  Select coordinate to update
16:  $\tilde{d} \gets -\lambda \cdot \operatorname{sign}\left(\alpha^{(j)}\right)$  O(1)
17:  $g_{t}\gets \tilde{g} -\tilde{d}\cdot \alpha^{(j)}$  0(1)
18:  $\eta_{t} = \frac{2}{t + 2}$  O(1)
19:  $w_{m}\gets w_{m}(1 - \eta_{t})$
20:  $w^{(j)}\gets w^{(j)}\eta_{t}\tilde{d} /w_{m}$
21:  $\tilde{g} \gets \tilde{g}(1 - \eta_t) + \eta_t \tilde{d}\alpha^{(j)}$
22: for all rows  $i$  of  $X$  with feature  $j$  do  $\mathcal{O}(S_r)$
23:  $\bar{v}^{(i)}\gets \bar{v}^{(i)} + \eta_t\tilde{d}*X[i,j] / w_m\mathcal{O}(1)$
24:  $\gamma \gets \nabla \mathcal{L}(w_m\cdot \bar{v}^{(i)}) - \bar{q}^{(j)}$  O(1)
25:  $\bar{q}^{(j)}\gets \bar{q}^{(j)} + \gamma$  01
26:  $\pmb{\alpha} \gets \pmb{\alpha} + \gamma \cdot X[i, \underline{i}]$ $\mathcal{O}(S_c)$
27:  $\tilde{g}\gets \tilde{g} +\gamma \cdot X[i,:]^{\mathrm{T}}\mathbf{w}\cdot w_{m}$ $\mathcal{O}(S_c)$
28: end for
29:  $Q.\mathrm{update}(k,|\alpha^{(k)}|\cdot \mathrm{scale})\forall k$  gradients updated
30: end for
31: Output w
$-\pmb{\alpha}_{t}^{\top}\mathbf{d}_{t}$  caused by just the  $j^{\mathrm{th}}$  coordinate update. After the sparse  $\alpha$  updates,  $\tilde{g}$  is again updated by  $\gamma X[i,:]^{\top}\mathbf{w}\cdot w_{m}$ .
Fast Frank-Wolfe Framework
The final procedure that forms the foundation for our results is given in Algorithm 2. The scale variable holds the noise parameter required for DP and can be ignored for non-private training. Lines 6, 13, 15, and 30 require an abstract priority queue that is populated with values proportional each feature's gradient. This mechanism is different in the non-private and private cases, and we will detail them in the following two sections.
The first iteration of Algorithm 2 performs the same calculations as Algorithm 1, but for all subsequent iterations, values will be updated in a sparse fashion.
Lines 16-21 update the multiplicative  $w_{m}$  variables and perform the single coordinate updates to  $\mathbf{w}$  and  $\tilde{g}$ , all taking  $\mathcal{O}(1)$  time to complete. Lines 22-29 handle the updates for  $\alpha$  and  $\bar{\mathbf{v}}$ , which requires looping over the rows that use feature  $j$ , which we expect to have  $\mathcal{O}(S_r)$  complexity. Within the loop, we use one row of the matrix to perform a sparse update which we expect to have  $\mathcal{O}(S_c)$  complexity. The gradients  $\alpha^{(k)}$  that get updated by this loop are updated in the priority queue  $Q^2$  in  $\mathcal{O}(1)$  time per update, so they do not alter the final complexity. The final complexities of our algorithm are now dependent on the complexity of  $Q.\mathrm{getNext}()$ , which will differ in the non-private and private cases.
3.2 Algorithmically Efficient Non-Private Frank-Wolfe
We first analyze the complexity of the non-private Frank-Wolfe algorithm, though our ultimate goal is to make the private case more efficient. The algorithm we detail in the non-private case will be of superior big- $\mathcal{O}$  complexity and perform significantly less FLOPs than the standard Frank-Wolfe implementation but will not be faster in practice due to constant factor overheads which we will explain.
The primary insight in building an algorithmically faster non-private algorithm is to use a Fibonacci Heap, which allows for  $\mathcal{O}(\log D)$  removal complexity and amortized  $\mathcal{O}(1)$  insertion and decreaseKey operations. We use the negative magnitude as the key in the min-heap. Our insight is that we can decrease a key  $j$  whenever  $|\alpha^{(j)}|$  increases, and ignore cases where  $|\alpha^{(j)}|$  decreases. This means the negative priority is an upper bound on true gradient magnitude. This is favorable because the vast majority of updates to the queue are intrinsically of a magnitude too small to be selected (hence why the solution is sparse) and so even with an inflated magnitude are never top of the queue.
These stale gradients will cause some items to reach the top of the queue incorrectly, which is easy to resolve as detailed in Algorithm 3. The current item  $c$  is popped off the queue, and compared against the current best coordinate  $j$ . This loop continues until the top of the queue has a smaller priority than the current gradient magnitude  $\alpha^{(j)}$ . Because the stale
magnitude can only be larger than the true gradient, once we satisfy this condition it must be the case that no item in the queue can have a higher priority. Thus, the procedure is correct.
The number of items we expect to have to consider must be proportional to the number of non-zero coefficients in the weight vector. This gives a final complexity of getNext  $\mathcal{O}(\| \mathbf{w}^*\| _0)$  for the number of non-zeros in the final solution, multiplied by the  $\mathcal{O}(\log (D))$  cost per pop () call, giving a final complexity of  $\mathcal{O}(NS_c + T\| \mathbf{w}^*\| _0\log D + TS_rS_c)$ .
Algorithm 3 Fibonacci Heap Frank-Wolfe Queue Maintenance
1: function GETNEXT()
2:  $j \gets -1$  Assume  $|\alpha^{(-1)}|$  returns  $-\infty$ 
3: repeat
4:  $c \gets Q.\text{pop}()$ $\mathcal{O}(\log D)$ 
5: if  $|\alpha^{(c)}| > |\alpha^{(j)}|$  then
6:  $j \gets c$ 
7: end if
8: until  $|\alpha^{(j)}| > |Q.\text{peekPriority}|$ 
9: Re-insert removed items  $c$  ... using priorities  $|\alpha^{(c\dots)}|$ 
10: Output  $j$ 
11: end function
12: function UPDATE(i, v)
13:  $v_{cur} \gets$  Current priority of item  $i$ 
14: if  $-v_{cur} > -v$  then  $\text{Min-Heap}$ 
15:  $Q.\text{decreaseKey}(i, v)$ $\mathcal{O}(1)$ 
16: end if
17: end function
There are multiple ways to implement this, but we find a naive re-iteration over the loop to update based on the final values the fastest due to reduced memory overheads.
While this is of superior algorithmic complexity compared to the standard Frank-Wolfe implementation, it has been long known that Fibonacci heaps have high constant-factor overheads that prevent their practical use [33, 34]. We still find the result useful in being the first to demonstrate a faster iteration speed, as well as a deterministic case to verify the correctness of our approach. For this reason we will use Algorithm 3 to show that our method converges at the same rate and with fewer FLOPs compared to the standard Frank-Wolfe implementation, as it does not suffer from the randomness required for differential privacy<sup>3</sup>. Our concern about this inefficiency is limited, as many faster algorithms exist for non-private LASSO regression that are orders of magnitude faster than using Frank-Wolfe [35, 36, 37], so other tools can suffice. However, no tools for high-dimensional and sparse DP LASSO regression exist except for the Frank-Wolfe method, which we make more efficient in the next section.
3.3 Algorithmically Efficient Differentially Private Frank-Wolfe
For our DP Frank Wolfe algorithm, we convert from the Laplacian mechanism originally used by Talwar et al. to the Exponential Mechanism [38]. Rather than adding noise to each gradient and selecting the maximum, in the exponential mechanism each coordinate  $j$  is given a weight  $\propto \exp \left(\frac{\epsilon u(j)}{2\Delta u}\right)$  where  $u(j)$  is the score of the  $j^{\mathrm{th}}$  item and  $\Delta u$  is the sensitivity [39].
This poses two challenges:
1. We need to select a weighted random sample from  $D$  options in sub- $\mathcal{O}(D)$  time to pick the coordinate  $j$  to maintain the  $DP$  of the exponential mechanism.
2. We need to do this while avoiding the numeric instability of raising a gradient to an exponential, which will overflow for relatively small gradient magnitudes.