SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mechanism	clients per round B	ε	noise mult. σ	server lr ηs	Eval Accuracy (%, Smoothed)	Test Accuracy (%)
DP-SGD	1000	1	4.22468	0.0244	16.82	16.69
Single-epoch MF	167	1	4.22468	0.1197	20.29	20.44
Optimal TreeAgg	1000	1	4.22468	0.2035	21.15	21.25
Multi-epoch MF	1000	1	4.76079	0.2035	21.96	21.92
Band MF (Ours)	1000	1	4.22468	0.2035	22.12	22.05
DP-SGD	1000	2	2.23048	0.0414	18.70	18.42
Single-epoch MF	167	2	2.23048	0.2035	21.66	21.70
Optimal TreeAgg	1000	2	2.23048	0.3460	22.52	22.59
Multi-epoch MF	1000	2	2.51352	0.3460	23.15	23.04
Band MF (Ours)	1000	2	2.23048	0.5882	23.31	23.19
DP-SGD	1000	4	1.19352	0.0704	20.07	19.81
Single-epoch MF	167	4	1.19352	0.3460	22.94	22.90
Optimal TreeAgg	1000	4	1.19352	0.5882	23.66	23.62
Multi-epoch MF	1000	4	1.34498	0.5882	24.19	24.02
Band MF (Ours)	1000	4	1.19352	1.0000	24.35	24.16
DP-SGD	1000	8	0.65294	0.1197	21.26	21.08
Single-epoch MF	167	8	0.65293	0.5882	24.03	23.88
Optimal TreeAgg	1000	8	0.65294	1.0000	24.54	24.45
Multi-epoch MF	1000	8	0.73579	1.0000	24.95	-
Band MF (Ours)	1000	8	0.65294	1.0000	25.06	24.88
DP-SGD	1000	16	0.36861	0.3460	22.51	22.26
Single-epoch MF	167	16	0.36861	1.0000	24.80	24.62
Optimal TreeAgg	1000	16	0.36861	1.7000	25.15	25.14
Multi-epoch MF	1000	16	0.41539	1.7000	25.50	25.33
Band MF (Ours)	1000	16	0.36861	1.7000	25.59	25.41
Table 4: Parameters and metrics for simulated cross-device FL, Fig. 6[a]. The noise multipliers  $\sigma$  are calibrated to achieve the given  $\epsilon$  guarantees at  $\delta = 10^{-6}$  under  $b = 342$ -min-separation. The matrices are scaled to have sensitivity 1 under  $(k = 6, b = 342)$ , see Table 2, and so a larger noise multiplier  $\sigma$  is necessary for the MULTI-EPOCH MF matrices. Test-set accuracy for MULTI-EPOCH MF at  $\epsilon = 8$  was unavailable. All BANDMF matrices use  $\hat{b} = 342$ . Note that  $\sigma$  is reported for reproducibility purposes (it is the parameter passed to Alg. 1), but it does not directly capture the noise added in either gradients  $\hat{\mathbf{x}}_i$  or their cumulative sums as the noise  $\mathbf{z}$  is linearly transformed by  $\mathbf{C}^{-1}$  which varies across the mechanisms above.
Mechanism	clients per round B	ε	noise mult. σ	server lr ηs	Eval Accuracy (%, Smoothed)	Test Accuracy (%)
DP-SGD, w/ ampl.	1000	1	0.37313	0.3460	22.50	22.22
Multi-epoch MF, no ampl.	1000	1	4.22468	0.2035	22.11	22.10
(Band) MF, w/ ampl. (Ours)	1000	1	0.79118	0.3460	23.11	22.83
DP-SGD, w/ ampl.	1000	2	0.30481	0.3460	22.89	22.62
Multi-epoch MF, no ampl.	1000	2	2.23048	0.3460	23.36	23.24
(Band) MF, w/ ampl. (Ours)	1000	2	0.64708	0.5882	24.01	23.71
DP-SGD, w/ ampl.	1000	4	0.25136	0.3460	23.27	22.94
Multi-epoch MF, no ampl.	1000	4	1.19352	0.5882	24.36	24.16
(Band) MF, w/ ampl. (Ours)	1000	4	0.52224	1.0000	24.67	24.42
DP-SGD, w/ ampl.	1000	8	0.20567	0.5882	23.59	23.30
Multi-epoch MF, no ampl.	1000	8	0.65294	1.0000	25.08	24.88
(Band) MF, w/ ampl. (Ours)	1000	8	0.43490	1.7000	25.26	24.99
DP-SGD, w/ ampl.	1000	16	0.16876	0.5882	23.96	23.61
Multi-epoch MF, no ampl.	1000	16	0.36861	1.7000	25.59	25.43
(Band) MF, w/ ampl. (Ours)	1000	16	0.36861	1.7000	25.59	25.43
Table 5: Parameters and metrics for centralized StackOverflow, Fig. 1(b). The noise multipliers are calibrated to achieve the given  $\epsilon$  guarantees at  $\delta = 10^{-6}$  under  $(k = 6, b = 342)$ -participation, assuming Poisson sampling for DP-SGD and BANDMF. For BANDMF, we tune  $\hat{b}$  under amplification for optimal RMSE, selecting  $\hat{b} = 9, 18, 32, 64, 2052$  for  $\epsilon = 1, 2, 4, 8, 16$  respectively. For  $\epsilon = 16$ , we have  $n = \hat{b}$ , and hence BANDMF is identical to MULTI-EPOCH MF optimized with Eq. (6). Values of  $\sigma$  are provided for reproducibility only, see comments on  $\sigma$  from Table 4.
ε	server lr ηs	Eval Accuracy (%, Smoothed)
0.0084	0.0143	0.0244	0.0414	0.0704	0.1197	0.2035	0.3460	0.5882	1.0000	1.7000	2.8900	4.9130	8.3521
Mechanism
1.0	DP-SGD	14.31	15.98	16.82	16.53	15.46	4.67	-	-	-	-	-	-	-	-
Single-epoch MF	-	-	-	-	20.16	20.29	19.68	-	-	-	-	-	-	-
Optimal TreeAgg	-	-	-	-	-	21.08	21.15	20.34	-	-	-	-	-	-
Multi-epoch MF	-	-	-	-	-	21.56	21.96	21.60	-	-	-	-	-	-
Band MF, b=342 (Ours)	-	-	-	-	-	21.70	22.12	21.96	-	-	-	-	-	-
2.0	DP-SGD	-	16.20	17.88	18.70	18.22	17.75	15.52	-	-	-	-	-	-	-
Single-epoch MF	-	-	-	-	-	21.46	21.66	21.26	-	-	-	-	-	-
Optimal TreeAgg	-	-	-	-	-	-	22.40	22.52	21.87	-	-	-	-	-
Multi-epoch MF	-	-	-	-	-	-	22.80	23.15	22.96	-	-	-	-	-
Band MF, b=342 (Ours)	-	-	-	-	-	-	-	23.27	23.31	22.57	-	-	-	-
4.0	DP-SGD	-	-	18.08	19.45	20.07	19.97	19.48	18.08	-	-	-	-	-	-
Single-epoch MF	-	-	-	-	-	-	22.66	22.94	22.60	-	-	-	-	-
Optimal TreeAgg	-	-	-	-	-	-	-	23.57	23.66	23.27	-	-	-	-
Multi-epoch MF	-	-	-	-	-	-	-	23.87	24.19	24.01	-	-	-	-
Band MF, b=342 (Ours)	-	-	-	-	-	-	-	-	24.26	24.35	23.74	-	-	-
8.0	DP-SGD	-	-	-	-	20.61	21.26	21.24	20.89	20.00	-	-	-	-	-
Single-epoch MF	-	-	-	-	-	-	-	23.73	24.03	23.71	-	-	-	-
Optimal TreeAgg	-	-	-	-	-	-	-	-	24.52	24.54	24.15	-	-	-
Multi-epoch MF	-	-	-	-	-	-	-	-	24.72	24.95	24.77	24.17	-	-
Band MF, b=342 (Ours)	-	-	-	-	-	-	-	-	24.76	25.06	24.92	-	-	-
16.0	DP-SGD	-	-	-	-	-	-	22.39	22.51	22.17	-	-	-	-	-
Single-epoch MF	-	-	-	-	-	-	-	-	24.66	24.80	24.50	-	-	-
Optimal TreeAgg	-	-	-	-	-	-	-	-	24.89	25.15	25.15	-	-	-
Multi-epoch MF	-	-	-	-	-	-	-	-	-	25.38	25.50	25.34	-	-
Band MF, b=342 (Ours)	-	-	-	-	-	-	-	-	-	25.38	25.59	25.47	-	-
inf	DP-SGD	-	-	-	-	-	-	-	-	-	-	25.96	26.23	26.24	8.03
Table 6: Federated learning rate tuning for StackOverflow NWP. Validation accuracy smoothed over the final 400 rounds of training, used to select the best server learning rates for the comparison of test-set accuracy presented in Fig. 6[a].
ε	server lr ηs	Eval Accuracy (%, Smoothed)
Mechanism	0.1197	0.2035	0.3460	0.5882	1.0000	1.7000	2.8900	4.9130
1.0	DP-SGD	-	22.39	22.50	22.03	-	-	-	-
Multi-epoch MF	21.75	22.11	21.95	-	-	-	-	-
Band MF, b=9 (Ours)	-	22.83	23.11	23.03	-	-	-	-
2.0	DP-SGD	-	22.70	22.89	22.66	-	-	-	-
Multi-epoch MF	-	22.89	23.36	23.26	-	-	-	-
Band MF, b=18 (Ours)	-	-	23.80	24.01	23.77	-	-	-
4.0	DP-SGD	-	22.88	23.27	23.20	-	-	-	-
Multi-epoch MF	-	-	23.96	24.36	24.22	23.71	-	-
Band MF, b=32 (Ours)	-	-	-	24.52	24.67	24.43	-	-
8.0	DP-SGD	-	-	23.48	23.59	23.28	-	-	-
Multi-epoch MF	-	-	-	24.79	25.08	24.98	24.55	-
Band MF, b=64 (Ours)	-	-	-	-	25.15	25.26	24.79	-
16.0	DP-SGD	-	-	23.85	23.96	23.72	-	-	-
Multi-epoch MF	-	-	-	-	25.42	25.59	25.50	24.92
Band MF, b=342 (Ours)	-	-	-	-	25.37	25.55	25.45	24.90
Band MF, b=64 (Ours)	-	-	-	-	25.38	25.54	25.40	-
Table 7: Centralized learning rate tuning for StackOverflow NWP.. Validation accuracy smoothed over the final 400 rounds of training, used to select the best server learning rates for the comparison of test-set accuracy presented in Fig. 1(b). DP-SGD and BANDMF use amplification.
Algorithm 8 Banded Matrix Multiplication  
Input:  $\hat{b}$ -Banded lower triangular matrix  $\mathbf{C} \in \mathbb{R}^{n \times n}$ , vector  $\mathbf{x} \in \mathbb{R}^n$   
Output:  $\mathbf{Cx}$   
for  $i = 1, \dots, n$  do  
 $\mathbf{y}_i = \sum_{j=i-\hat{b}+1}^i \mathbf{C}_{[i,j]}\mathbf{x}_j$   
return  $\mathbf{y}$
Algorithm 9 Banded Inverse Multiplication  
Input:  $\hat{b}$ -Banded lower triangular matrix  $\mathbf{C} \in \mathbb{R}^{n \times n}$ , vector  $\mathbf{y} \in \mathbb{R}^n$   
Output:  $\mathbf{C}^{-1}\mathbf{y}$   
for  $i = 1, \dots, n$  do  
 $\mathbf{x}_i = (\mathbf{y}_i - \sum_{j=i-\hat{b}+1}^{i-1} \mathbf{C}_{[i,j]} \mathbf{x}_j) / \mathbf{C}_{[i,i]}$   
return  $\mathbf{x}$
Figure 15: Algorithms for matrix-vector and inverse matrix-vector multiplication by a banded matrix. To simplify the presentation, we use the convention that out-of-bounds indexing into a matrix or vector returns 0.
J Efficient Multiplication and Inverse of Banded Matrices
Algorithms 8 and 9 (Fig. 15) give algorithms for lower triangular banded matrix-vector multiplication and inverse banded matrix-vector multiplication. Note that both algorithms are compatible with the streaming nature of gradients. As soon as the next input  $\mathbf{x}_i$  is received, the algorithm can immediately output  $\mathbf{y}_i$ . Both algorithms require storing a state of size  $\hat{b}$ , and run in  $O(n \cdot \hat{b})$  time. While the algorithms are described with respect to computing matrix-vector products, they can also be used to compute matrix-matrix products where the right-hand-side is a  $n \times d$  matrix by multiplying by each column independently. In this setting, these algorithms require  $O(\hat{b} \cdot d)$  space and  $O(n \cdot \hat{b} \cdot d)$  time. Both algorithms have appeared previously in the literature on Monte Carlo methods, which have a similar problem at their core to that of noise generation for MF; see e.g. [56, Section 2].
K Application to a Real-World Cross-Device FL System
We train a one-layer LSTM language model of  $\sim 2.4$  million parameters in a practical cross-device FL system following [58]. The model is used for predicting the next word of Spanish in a mobile virtual keyboard. We pretrain the model on public multilingual C4 dataset [49, 60], and then fine-tune with on-device user data in FL. In a common practical FL system, clients have to satisfy criteria like being charged, idle and connected to unmetered network to participate in a round [10, 27, 31, 47], hence only a subset of clients can be reached and there is a strong diurnal pattern of client participation [61, 63]. It is very challenging to hold a fixed set of clients for evaluation, or develop random sampling for privacy amplification. Though the current implementation of client participation control is feasible through the client timer, the tuning of separation  $b$  in practice can be challenging. Therefore, the training of Fig. 6 (b) can achieve smaller separation  $b$  than in [58].
K.1 Reporting privacy guarantees
We follow the guidelines outlined in [48, Sec. 5.3] to report privacy guarantees.
1. DP setting. This is a central DP guarantee where the service provider is trusted to correctly implement the mechanism.
2. Instantiating the DP Definition
(a) Data accesses covered: The DP guarantee applies to all well-behaved clients  $^{11}$  in a single training run. We do not account for hyperparameter tuning, or the selection of the final model checkpoint using evaluation metrics or A/B testing in our guarantees. Public multilingual C4 data [49, 60] is used for pre-training.
(b) Final mechanism output: Only the final model checkpoint is released for use in production, however the mechanism's output is technically the full sequence of privatized gradients, and so the guarantee also applies at this level, and hence all intermediate models are protected (including those sent to devices participating in federated learning).
11Clients that faithfully follow the algorithm including participation limits. Due to the design of the algorithm, a mis-behaved client does not adversely affect the DP guarantee of any well-behaved clients.
(c) Unit of privacy. Device-level DP is considered, i.e., the notion of adjacency is with respect to arbitrary training datasets on each client device, and the device might have an arbitrarily large local dataset containing arbitrary training examples. For user's with a single device, this corresponds directly to user-level DP; for devices shared with multiple users, this provides a stronger notion of DP than user-level; for a user with multiple devices that happen to both participate in training the model, the notion is weaker, but group privacy can be used to obtain a user-level guarantee.
(d) Adjacency definition for "neighbouring" datasets: We use the zero-out definition [35]. This is a special form of the add-or-remove definition, where neighboring data sets differ by addition/removal of a single client. In the absence of a client at any training step, we assume that the client's model update gets replaced with the all zeros vector. This assumption enforces a subtle modification to the traditional definition of the add/remove notion of DP which allows neighboring data sets to have the same number of records.
3. Privacy accounting details
(a) Type of accounting used: Both  $\rho$ -zCDP [11] accounting, and PLD accounting [18] for  $(\epsilon, \delta)$ -DP are used.
(b) Accounting assumptions: Each client only participates limited times during the training, and there are at least a min-separation of  $b$  rounds between two consecutive participation of a client. This is enforced by a timer on clients in the cross-device FL system.
(c) The formal DP statement: The privacy guarantees are  $\rho = 0.52$ -zCDP and  $(\epsilon = 6.69, \delta = 10^{-10})$ -DP for ONLINE TREEAGG, while BANDMF achieves  $\rho = 0.24$ -zCDP and  $(\epsilon = 4.35, \delta = 10^{-10})$ -DP.
(d) Transparency and verifiability: We are going to open source our code based on TensorFlow Federated and Tensorflow Privacy. Key portions of the cross-device FL system will also open sourced.