SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Observe this participation schema is easy for devices to enforce: each device remembers the last step  $i$  in which it participated, and when it again becomes eligible, it checks in to the server, and participates in training as long as the current step is at least  $i + b$ ; it does not need to check in during a narrow (and unknown to the device) time window for a specific step.
We now turn to computing sensitivity under  $b$ -min-sep-participation. For  $(k, b)$ -participation,  $|\Pi_{(k, b)}| = b$ , a fact Choquette-Choo et al. [15, Eq. 3] critically exploited when computing sensitivity via brute force computation of a maximum over the elements in  $\Pi$ . With  $b$ -min-sep-participation, we have  $|\Pi_b| = \mathcal{O}(\exp(n))$ , and hence any brute force approach which requires checking some value for all  $\pi \in \Pi_b$  will be impractical. Following the formalism of [15, Section 2], a participation schema  $\Pi$  (plus a specification of model dimension  $d$ ) yields an expression for the sensitivity of the function  $\mathbf{x} \mapsto \mathbf{C}\mathbf{x}$  assuming that the contributions of any given user to the data structure  $\mathbf{x}$  are restricted to the rows in  $\mathbf{x}$  indexed by some  $\pi \in \Pi$ . By Prop. E.1 of App. E.2, independent of model dimension  $d$ , we show sensitivity for any schema  $\Pi$  may be bounded by
$$
\operatorname {s e n s} _ {\Pi} (\mathbf {C}) ^ {2} \leq \max  _ {\pi \in \Pi} \sum_ {i, j \in \pi} | \mathbf {X} _ {[ i, j ]} |. \tag {5}
$$
Eq. (5) highlights several subtleties in computing sensitivity. First, is the challenge presented by the exponentially large number of patterns in  $\Pi_b$ ? Second is the question of tightness of the inequality in Eq. (5): how much are we losing by effectively ignoring any cancellation in the matrix  $\mathbf{X}$ ?
Banded matrices Fortunately, banded matrices render Eq. (5) both exactly computable and tight (independent of dimension  $d$ ), showing that  $\Pi_b$  satisfies the requirements of (b) and (c) above. We say a (general) matrix  $\mathbf{X}$  is  $\hat{b}$ -banded if for all  $i,j\in [n],|i - j|\geq \hat{b}$  implies  $\mathbf{X}_{[i,j]} = 0$ . While this is off-by-one from the bandwidth ( $\mathbf{X}$  has bandwidth  $b - 1$ ), our definition will be useful as it will be natural to match  $\hat{b}$ -banded matrices with  $b$ -min-separation. Further, for  $\hat{b}$ -banded lower-triangular matrices (which will play a central role),  $\hat{b}$  intuitively gives the number of bands in the matrix.
For non-banded matrices, the right-hand side of Eq. (5) remains efficiently computable (but not easily expressible in a mathematical program), enabling us to provide nontrivial privacy guarantees for matrices which are not  $b$  banded under  $b$ -min-sep, showing that  $\Pi_b$  satisfies (a) as well. The key subroutine is Alg. 3, which gives an efficient dynamic program for solving linear optimization over  $\Pi_b$ . Define  $\mathbf{u}(\pi) \in \{0,1\}^n$  by  $\mathbf{u}(\pi)_i = 1$  if  $i \in \pi$  and 0 otherwise. Then, Alg. 3 solves
$$
\min  _ {\pi \in \Pi_ {b}} \left\langle \mathbf {v}, \mathbf {u} (\pi) \right\rangle .
$$
This is the key subroutine in Alg. 4 and Alg. 5. Proofs for Thm. 2 and Thm. 3 are deferred to App. E.2.
Theorem 2. Let  $\mathbf{C} \in \mathbb{R}^{n \times n}$  be a lower-triangular matrix, and  $\Pi_b$  the  $b$ -min-sep-participation schema. Further, suppose  $k'$  upper-bounds the actual maximum number of participations that occurred in a data stream  $\mathbf{x}$  (at worst, we may take  $k' \leq \lceil \frac{n}{b} \rceil$ ): Then: (1) If  $\kappa$  is an upper-bound on the column norms of  $\mathbf{C}$ , that is  $\forall j \in [n]$ ,  $\| \mathbf{C}_{[j]} \| \leq \kappa$ , and  $\mathbf{C}$  is  $b$ -banded, then the sensitivity is bounded by  $\kappa \sqrt{k'}$ . (2) If  $\mathbf{C}$  is  $b$ -banded, Alg. 5 invoked with Gram matrix  $\mathbf{X} = \mathbf{C}^\top \mathbf{C}$  and  $b, k'$  as in the setup, exactly computes  $\mathrm{sen}(C)$  under schema  $\Pi_b$  in polynomial time for any dimension  $d$ .
Theorem 3. For an arbitrary (non-banded)  $\mathbf{C}$ , let  $\mathbf{X} = \mathbf{C}^{\top}\mathbf{C}$  and  $b, k'$  as in Thm. 2. Then Alg. 4 of App. E upper-bounds  $\mathrm{sen}(C)$  under schema  $\Pi_b$  in polynomial time for any dimension  $d$ .
3While  $b$ -min-sep-participation is required for the cross-device FL setting,  $(k,b)$ -participation is preferred in centralized settings where it can be enforced and will also satisfy our BANDMF amplification guarantees leading to the empirical results we show in Fig. 1.
4 Optimizing Banded Matrices
To enjoy the benefits of banded matrices within the framework of MF-DP-FTRL, we need to design an algorithm that can efficiently optimize over the space of  $\hat{b}$ -banded  $\mathbf{C}$  matrices. To solve this problem, we will work in the domain of  $\mathbf{X} = \mathbf{C}^{\top}\mathbf{C}$ , and utilize the following fact:
Proposition 4.1. Let  $\mathbf{X} \in \mathbb{R}^{n \times n}$  be a  $\hat{b}$ -banded symmetric positive definite matrix. Then there exists a lower triangular  $\hat{b}$ -banded matrix  $\mathbf{C} \in \mathbb{R}^{n \times n}$  such that  $\mathbf{X} = \mathbf{C}^{\top} \mathbf{C}$ .
Utilizing Prop. 4.1, we can modify Problem 1 by introducing the constraint  $\mathbf{X}_{[i,j]} = 0$  if  $|i - j|\geq \hat{b}$ . This additional linear constraint preserves convexity of the optimization problem, and makes the sensitivity calculation tractable as well. However, it is still not immediately obvious how to solve the optimization problem, since we need to run the dynamic program defined in Alg. 5 of App. E to compute sensitivity. For this reason, we impose the additional constraint that  $\mathrm{diag}(\mathbf{X}) = \mathbf{1}$ . This constraint, together with bandedness, ensures that the squared sensitivity is equal to  $k$  for all  $\mathbf{X}$  by Thm. 2. The final optimization problem we seek to solve is stated below:
Problem 2. The matrix factorization optimization problem for banded matrices is to solve
$$
\underset {\mathbf {X} \in \mathbf {S} _ {+} ^ {n}} {\text {m i n i m i z e}} \operatorname {t r} \left[ \mathbf {A} ^ {\top} \mathbf {A} \mathbf {X} ^ {- 1} \right] \quad \text {s u b j e c t t o} \quad \operatorname {d i a g} (\mathbf {X}) = \mathbf {1} \text {a n d} \mathbf {X} _ {[ i, j ]} = 0 \text {i f} | i - j | \geq \hat {b}, \tag {6}
$$
and then find  $\mathbf{C}$  so that  $\mathbf{C}^{\top}\mathbf{C} = \mathbf{X}$  via Prop. 4.1.
We would like to remark on the similarity between Problem 2 and the single-participation version of Problem 1. The two problems are identical modulo the bandedness constraint, which is an equality constraint on individual entries of  $\mathbf{X}$ . Therefore, existing primal-optimization based solvers [43] for the single-participation matrix mechanism can be extended to optimize over this new space of matrices with little modification. Specifically, the only modification necessary is to initialize to an appropriately banded feasible  $\mathbf{X}$  matrix, like  $\mathbf{X} = \mathbf{I}$ , and to post-process the gradient w.r.t  $\mathbf{X}$  by setting  $\frac{\partial L}{\partial \mathbf{X}_{[i,j]}} = 0$  if  $|i - j| \geq \hat{b}$  in each step. Since the equality constraints exactly specify individual entries of  $\mathbf{X}$ , Problem 2 can be solved as an unconstrained optimization problem (over the remaining entries in  $\mathbf{X}$ ), using any number of off-the-shelf unconstrained optimization algorithms. As recommended by McKenna et al. [43], we use the LBFGS algorithm [12] to solve this problem.
Remarks on the diag  $(\mathbf{X}) = 1$  constraint The constraint on diag  $(\mathbf{X})$  serves multiple purposes. First,  $\mathrm{diag}(\mathbf{X}) = \mathbf{1}$  implies that  $\| \mathbf{C}_{[i,i]}\|_2 = 1$  for all  $i$ , i.e., that  $\mathbf{C}$  has equal column norms. This ensures that BANDMF reduces to DP-SGD when  $\hat{b} = 1$ , which is desirable. Second  $\mathrm{diag}(\mathbf{X}) = \mathbf{1}$  simplifies the optimization problem greatly, as the sensitivity computation for both  $(k,b)$ -participation and  $b$ -min-sep are trivial and tight under this constraint (Thm. 2 Claim (1)). Third, imposing this constraint does not drastically change the search landscape, or cost much in terms of RMSE; see Table 2 for a comparison of matrices with and without this constraint, and Fig. 8 for a visualization. Fourth, this constraint allows us to solve a single optimization problem that is simultaneously tailored for  $(k,b)$ -participation and  $b$ -min-sep-participation. In Appendices B and C, we formulate an optimization problem without the  $\mathrm{diag}(\mathbf{X}) = \mathbf{1}$  constraint, discuss how we solve it, and compare matrices generated with and without this constraint empirically.
5 Amplification for Banded Matrix Mechanisms
In the centralized setting where we can control the participation patterns of individual examples, the privacy guarantees of BANDMF can be amplified. We focus on amplification by sampling with fixed batch size in this section, but give a more general statement in App. F.
Existing privacy analysis of MF-DP-FTRL is based on the reduction to the batch release of the entire  $\mathbf{Cx} + \mathbf{z}$  as a single Gaussian mechanism event [15, 17] so standard amplification techniques don't directly apply. Instead, for each participation by an example, we consider the set of rows in  $\mathbf{Cx}$  affected by this participation as a Gaussian mechanism (see the groups of rows in Fig. 4). Then as
Note that the constraint that  $\mathbf{X}$  is positive definite is typically handled implicitly by taking sufficiently small step sizes to avoid violating that constraint [43, 62].
Algorithm 2 Sampling scheme
$\overline{D_1,\ldots,D_b}\gets$  arbitrary partition of  $D$    
for  $i = 1,2,\dots ,n$  do  $j = i$  (mod  $b$  )  $(b$  if  $i / b$  is integer).  $S_{i}\gets$  random size  $B$  subset of  $D_{j}$  Compute  $\mathbf{x}_i$  on  $S_{i}$    
Release  $\mathbf{Cx} + \mathbf{z}$
Figure 3: An example of our sampling scheme that gives privacy amplification for BANDMF.
Figure 4: Visualization of how we can decompose BANDMF into independent queries when using Alg. 2. Larger view in Fig. 10 of App. F.
long as the sets of rows corresponding to different participations do not interact, which is ensured by the bandedness of  $\mathbf{C}$ , we can apply amplification to them separately.
Observe from Fig. 4 that the structure of BANDMF guarantees that the set of rows of  $\mathbf{Cx}$  which depend on each of  $\mathbf{x}_j, \mathbf{x}_{\hat{b} + j}, \mathbf{x}_{2\hat{b} + j}, \ldots$  are disjoint sets. Thus, we use the following sampling scheme for determining which examples participate in each step which is made formal as Alg. 2. Let  $D_1, D_2, \ldots, D_{\hat{b}}$  be an arbitrary partition of  $D$  into  $\hat{b}$  indexed subsets of size  $\breve{m} := \lfloor m / \hat{b} \rfloor$  (for simplicity, we discard extra examples so all  $D_j$  have size exactly  $\breve{m}$ ). In steps  $j, \hat{b} + j, 2\hat{b} + j, \ldots$ , we will only use examples in  $D_j$ . Hence, participation follows  $(k, b)$ -participation for  $b = \hat{b}$ , because it is optimal for  $(k, b)$ -participation to have the number of bands  $\hat{b}$  equal the min-separation  $b$ , in the remainder of this section and the associated appendices we simply write  $b$  instead of  $\hat{b}$ . Within each of these steps, we sample a size  $B$  subset of  $D_j$  uniformly at random to use in computing  $\mathbf{x}$ .
Roughly speaking, Thm. 4 below shows that if we use Alg. 2, BANDMF satisfies any standard privacy guarantees satisfied by DP-SGD run for  $k$  rounds, where in each round we sample  $B$  examples from a dataset of size  $m$ . In other words, it is equivalent to running DP-SGD for  $1 / b$  times as many rounds, but with the sampling probability multiplied by  $b$ .
Theorem 4. Suppose  $\mathbf{C}$  is  $b$ -banded and lower triangular, and the examples participating in each step are chosen according to Alg. 2. Then BANDMF satisfies any standard DP guarantee satisfied by performing  $k$  sensitivity- $\kappa$  queries on a dataset of size  $m$  using the Gaussian mechanism, where each query is run on a random subset of examples of size  $B$ .  $\kappa$  is the maximum column norm of  $\mathbf{C}$ .
The key idea behind Thm. 4 is the following: assume we have two datasets  $D, D'$  that differ in an example in  $D_{1}$  (WLOG), i.e., the differing example can only participate in steps  $1, b + 1, \ldots, (k - 1)b + 1$ . Then by the banded structure of  $\mathbf{C}$  and the standard technique of reducing adaptive queries to non-adaptive queries (see e.g. Claim D.1 in [17]), the first  $b$  rows of  $\mathbf{Cx}$  (i.e., all the rows where examples in step 1 influence the output) can be viewed as a query on the examples in  $D_{1}$  that were included in step 1, the next  $b$  rows can be viewed as an adaptively chosen query on the examples included in steps  $b + 1$ , and so on. See Fig. 4 for a visualization.
Generalization to other privacy amplification techniques Thm. 5 in App. F.3 provides a strong a generalization of Thm. 4. It shows that shuffling, another common amplification technique often used for DP-SGD, can also be applied to our BANDMF. We also provide an explicit algorithm for accounting for amplification via sampling in terms of the dp_accounting library [18], along with examples of concrete privacy parameters derived from these corollaries.
Optimizing the number of bands Thm. 4 shows that different numbers of bands (with a corresponding sampling scheme) give different privacy amplification guarantees. This implies given a particular privacy (or RMSE) target, one should optimize the number of bands to get the best RMSE (or privacy) possible. This can be done efficiently. Generally, for large values of  $\epsilon$  larger numbers of bands perform better, and as  $\epsilon \to 0$ , eventually amplification dominates so  $\hat{b} = 1$  becomes optimal. Details are in App. F.4.
and in our case,  $k = 1$  always.
${}^{6}$  Standard DP includes  $\epsilon$  -DP,  $\left( {\epsilon ,\delta }\right)$  -DP,Renyi DP,zCDP,and Gaussian DP.
(a) DP-SGD vs. MULTI-EPOCH MF
(b) DP-SGD vs Ampl. BANDMF
Figure 5: Comparison between DP-SGD, MULTI-EPOCH MF, and BANDMF in terms of RMSE on the prefix-sum queries, for  $n = 1024$  iterations,  $\epsilon \in [\frac{1}{32}, 16]$ ,  $\delta = 10^{-6}$ , and epochs  $\in [1, 1024]$ . Color indicates the ratio in RMSE between two mechanisms. Additional details in App. G.
(c) RMSE vs Epochs at  $\epsilon = 1$
Our experiments on example-level DP for image classification (of CIFAR10) and user-level DP for next word prediction (NWP) (of Stack Overflow NWP) focus on comparing our BANDMF with the existing state-of-the-art MULTI-EPOCH MF [15] and DP-SGD [1]. We finish by showing that BANDMF improves over state-of-the-art [35] for a production mobile keyboard next word prediction model. In all cases, noise  $\sigma$  is calibrated using privacy loss distributions [38] to achieve the stated privacy guarantees for zero-out adjacency [35, 48], as implemented in [18]. Following the common convention, amplified results are based on privacy accounting for Poisson sampling, though shuffling was used in non-production training. We find BANDMF can outperform both mechanisms across a wide range of privacy budgets to as low as  $\epsilon \approx 0.5$ . Past this, it is no worse than either.
RMSE We begin by comparing DP-SGD with MULTI-EPOCH MF and BANDMF in terms of their RMSE (Sec. 2) on the prefix workload. This is one measure for how much noise these mechanisms add during training, and is a reasonable proxy for learning performance. Observe in Fig. 5(a) that there are regimes where DP-SGD outperforms MULTI-EPOCH MF and vice versa in terms of RMSE. When then number of epochs equals  $n$ , DP-SGD reduces to full gradient descent (GD) (and there is no amplification benefit), and the optimal MF mechanism is close to the identity matrix (that is, GD), and so the algorithms become almost identical (MF has a very small advantage, as the identity matrix is not quite optimal for RMSE). However, as shown in Fig. 5(b), we see that BANDMF is always at least as good as DP-SGD in terms of RMSE. The improvement is most potent in the lower number of epochs and higher  $\epsilon$  regime, which is standard for large model training. In Fig. 5(c), we see that for fixed  $n$  as the number of epochs increases, all mechanisms enjoy improved RMSE, and BANDMF in fact reduces to DP-SGD in that regime (though BANDMF may be outperformed in RMSE by MULTI-EPOCH MF due to our imposed optimization constraint of constant diagonals in X).
Centralized training with amplification Our full experimental setup is described in App. H, and closely follows prior work [15]. We train for 20 epochs on CIFAR-10, and tune all mechanisms to achieve their best performance for each  $\epsilon$ , using 12 repeated runs. Fig. 1(a) shows that BANDMF with amplification and an optimized number of bands  $\hat{b}$  can obtain utility benefits over both prior mechanisms. We find that for  $\epsilon \in [2,5]$ , BANDMF achieves a consistent  $\approx 1$  percentage point boost in performance over MULTI-EPOCH MF. Below  $\epsilon \approx 2$ , where DP-SGD previously dominated, we find that BANDMF obtains a benefit around 3 percentage points. These two findings show that BANDMF is able to balance and leverage the benefits of both amplification and correlated noise effectively. As the budget  $\epsilon$  gets smaller, we find that BANDMF is equivalent to DP-SGD.
We next consider the now-standard StackOverflow next-word-prediction (NWP) task with user-level differential privacy, again following [15] (full details in App. I), in particular 2052 steps and 6 epochs, with  $B = 1000$ . The previous state-of-the-art for centralized training at  $\epsilon \geq 2$  corresponds to their MULTI-EPOCH MF. We again tune  $\hat{b}$  under amplification for optimal RMSE, selecting  $\hat{b} = 9, 18, 32, 64$  for  $\epsilon = 1, 2, 4, 8$  respectively. At  $\epsilon = 16$  we find MULTI-EPOCH MF is optimal. Fig. 1(b) shows substantial improvements for combining amplification with MF for  $\epsilon \in [1, 8]$ ; Table 5 of App. I gives the hyperparameters and accuracy values for this figure.
<sup>7</sup>All StackOverflow experiments use the  $\hat{b} = 2052$  matrices optimized with equal-column-norms via Eq. (6) for MULTI-EPOCH MF, which we observed to be minimally different in terms of RMSE and accuracy.
(a) Simulated cross-device FL, SO NWP
(b) Production mobile keyboard training
Figure 6: BOTH: Amplification is infeasible as outlined in Sec. 6 and App. K. LEFT, (A): Crossdevice FL results under  $b = 342$ -min-sep-participation. BANDMF, and MULTI-EPOCH MF (with the application to FL made possible by Thm. 3) outperform all prior work. RIGHT, (B): Evaluation accuracy of a language model trained in a real-world FL system. BANDMF achieves higher utility and a stronger  $(4.35, 10^{-10})$ -DP compared to the  $(6.69, 10^{-10})$ -DP achieved by ONLINE TREEAGG [35].
Cross-device federated learning We consider SO NWP again, but now assuming each user's data corresponds to the data on one device. We assume 2052 rounds and 6 epochs as above. Amplification is generally not possible in the cross-device setting, and so the prior state-of-the-art was (1) SINGLE-EPOCH MF of Denisov et al. [17] for single-epoch training (using  $B = 167$  rather than  $B = 1000$ ), and (2) OPTIMAL TREEAGG, which essentially takes the binary-tree matrix  $\mathbf{C}_{\mathcal{T}}^{\dagger}$ , and instead of using the less-efficient "online" estimator of Honaker [30], uses the pseudo-inverse  $\mathbf{C}_{\mathcal{T}}^{\dagger}$  for noise generation (see Sec. 2). The  $b$ -min-sep-participation sensitivity of  $\mathbf{C}_{\mathcal{T}}$  can still be calculated using the dynamic program of Kairouz et al. [35], while the use of  $\mathbf{C}_{\mathcal{T}}^{\dagger}$  requires the machinery of Denisov et al. [17]; see e.g. the OPTDECODERHONAKER results of Choquette-Choo et al. [15, Fig. 6]. Our Thm. 3 further enables an upper-bound on the  $b$ -min-sep-participation sensitivity of the MULTI-EPOCH MF matrices of Choquette-Choo et al. [15]; this incurs a penalty of about  $15\%$  (see Table 2 in App. C) compared to  $(k,b)$ -sensitivity. Fig. 6(a) shows that our BANDMF and MULTI-EPOCH MF again outperform prior baselines (though the previously untested multi-epoch OPTIMAL TREEAGG performs quite well); Table 4 gives the hyperparameters and accuracy values for this figure.
Application in production cross-device FL We fine-tune a Spanish next word prediction model, pretrained on the multilingual C4 dataset [49, 60], with on-device user data using FL. Our setup follows [59], and is described in full in App. K. We compared to an existing implementation of the ONLINE TREEAGG algorithm of Kairouz et al. [35] (not the optimal version using  $\mathbf{C}_{\mathcal{T}}^{\dagger}$  in simulation). Both algorithms ran for  $n = 2000$  training rounds. The BANDMF matrix was optimized for  $\hat{b} = 400$  bands; however, the production system only allows approximate control of the separation between participations, and post-hoc we could only bound  $b$  by 390 rounds for ONLINE TREEAGG and 385 for BANDMF, necessitating the use of Thm. 3 for the analysis of BANDMF as  $b < \hat{b}$ .
We used the same clients/round goal of 6500 for both, and tuned noise multipliers to achieve comparable RMSE, hence tuning for a stronger privacy guarantee rather than improved accuracy. Fig. 6(b) shows our results, and we see BANDMF actually achieves a slight improvement in accuracy, possibly due to learning-rate cooldown (which was only implemented for BANDMF). Our primary result is then that we are able to improve the privacy guarantee from  $\rho = 0.52$ -zCDP for ONLINE TREEAGG to  $\rho = 0.24$ -zCDP for BANDMF, or  $(\epsilon = 6.69, \delta = 10^{-10})$ -DP to  $(\epsilon = 4.35, \delta = 10^{-10})$ -DP. Details of the privacy guarantee following the best practices of Ponomareva et al. [48] are in App. K.1.
7 Discussion and Limitations
In this paper, we proposed the BANDMF mechanism, which extends MF-DP-FTRL and enjoys the benefits of privacy amplification. This allows it to solely operate above the previous Pareto frontier defined by both amplified DP-SGD and MF-DP-FTRL in centralized training scenarios. Moreover, BANDMF is well-suited to federated training scenarios, and improves state-of-the-art there as well. Additionally, the computational overhead of BANDMF is less than MF-DP-FTRL by a factor of  $\frac{b}{n}$ . It still has a  $b \times$  time and space overhead compared to DP-SGD, which can be prohibitive for very large models with billions of parameters. This is an interesting and important future research direction.
The authors thank the early feedback and discussion from Natalia Ponomareva, and the support of FL production training from Yanxiang Zhang and Yuanbo Zhang.