SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[52] Adam Smith, Abhradeep Thakurta, and Jalaj Upadhyay. Is interaction necessary for distributed private learning? In 2017 IEEE Symposium on Security and Privacy (SP), pages 58-77. IEEE, 2017.
[53] Shuang Song, Kamalika Chaudhuri, and Anand D Sarwate. Stochastic gradient descent with differentially private updates. In 2013 IEEE Global Conference on Signal and Information Processing, pages 245-248. IEEE, 2013.
[54] Shuang Song, Om Thakkar, and Abhradeep Thakurta. Characterizing private clipped gradient descent on convex generalized linear problems. arXiv preprint arXiv:2006.06783, 2020.
[55] Thomas Steinke. Composition of differential privacy & privacy amplification by subsampling, 2022.
[56] Maxime Vono, Nicolas Dobigeon, and Pierre Chainais. High-dimensional gaussian sampling: a review and a unifying approach based on a stochastic proximal point algorithm, 2021.
[57] Xi Wu, Fengan Li, Arun Kumar, Kamalika Chaudhuri, Somesh Jha, and Jeffrey F. Naughton. Bolt-on differential privacy for scalable stochastic gradient descent-based analytics. In Semih Salihoglu, Wenchao Zhou, Rada Chirkova, Jun Yang, and Dan Suciu, editors, Proceedings of the 2017 ACM International Conference on Management of Data, SIGMOD, 2017.
[58] Zheng Xu, Yanxiang Zhang, Galen Andrew, Christopher Choquette, Peter Kairouz, Brendan McMahan, Jesse Rosenstock, and Yuanbo Zhang. Federated learning of gboard language models with differential privacy, 2023.
[59] Zheng Xu, Yanxiang Zhang, Galen Andrew, Christopher A Choquette-Choo, Peter Kairouz, H Brendan McMahan, Jesse Rosenstock, and Yuanbo Zhang. Federated learning of gboard language models with differential privacy. arXiv preprint arXiv:2305.18465, 2023.
[60] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020.
[61] Timothy Yang, Galen Andrew, Hubert Eichner, Haicheng Sun, Wei Li, Nicholas Kong, Daniel Ramage, and Françoise Beaufays. Applied federated learning: Improving google keyboard query suggestions. arXiv preprint arXiv:1812.02903, 2018.
[62] Ganzhao Yuan, Yin Yang, Zhenjie Zhang, and Zhifeng Hao. Convex optimization for linear query processing under approximate differential privacy. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 2005–2014, 2016.
[63] Chen Zhu, Zheng Xu, Mingqing Chen, Jakub Konečný, Andrew Hard, and Tom Goldstein. Diurnal or nocturnal? federated learning of multi-branch networks from periodically shifting distributions. In International Conference on Learning Representations, 2022.
A Notation summary
n	Number of steps of the streaming linear query (SGD steps or FL rounds)
m	Total number of records (examples or users) in the database/dataset
b	Minimum separation between participations; b = 1 allows participation in every step
ˆb-banded matrix	A (general) matrix X isˆb-banded if for all i, j ∈ [n], |i-j| ≥ˆb implies X[i,j] = 0. We use b to refer to min-separation, and for example write b-banded when we set the number of bands equal to the min-separation; we useˆb-banded for a number of bands possibly different than b.
k	The maximum number of times any user might participate in training
d	Dimension of per-step user contributions (e.g., model size)
xi ∈ R or Rd	Sum of per-example gradients (or per-user model updates) on step i
x ∈ Rn×d	Stream of inputs xi, equiv. matrix with rows xi (so xi = x[i,:])
ζ	Clipping norm that limits the size of per-example contributions to xi
π ⊆ [n]	Participation pattern, the set of steps that an example participates in
Π	Participation schema, set of sets of steps (set of all π) an example could participate in
D	= {x -ˆx | (x,ˆx) ∈ N}, the set of deltas between neighboring input streams x,ˆx.
D	Corners of D when assumed to be a polytope, D = conv(D).
(k,b)-participation	participation schema Π with at most k participations, separated by exactly b steps
b-min-sep-participation	Relaxation of of (k,b)-participation where participations have separation at least b
A ∈ Rn×n	Lower-triangular linear query matrix to be factorized as A = BC
M†	Moore-Penrose pseudoinverse of matrix M
MT	Transpose of M
M[i,j]	The (i,j)th entry of matrix A
M[i,:] and M(:,j]	Theith row and jth column of M (numpy-style indexing)
conv(S)	Convex hull of the set S
[n]	={1,...,n}
||X||F	The Frobenius norm of a matrix X
Table 1: Summary of notation
A.1 Intuition behind MF-DP-FTRL
Fig. 7 compares DP-SGD and MF-DP-FTRL. To gain an intuition for why MF-DP-FTRL can perform better than DP-SGD, observe that vanilla SGD has iterates  $\theta_{t} = \theta_{0} - \eta \sum_{i=1}^{t} \hat{\mathbf{x}}_{i}$ , and hence when the noisy gradients  $\hat{\mathbf{x}}_{i}$  are added, the  $\mathbf{C}_{[i,j]}^{-1} \mathbf{z}_{[i,j]}$  terms in MF-DP-FTRL serve to cancel out some of the noise introduced on previous rounds. The noise cancellation reduces the total error in all the prefix sums of gradients  $\sum_{i=1}^{t} \hat{\mathbf{x}}_{i}$  for  $t \in [n]$ , but also worsens the privacy guarantee of the mechanism, i.e. increases its sensitivity. The privacy worsens as e.g., an adversary trying to learn  $\mathbf{x}_1$  via  $\hat{\mathbf{x}}_1$  can partially learn the value of  $\mathbf{z}_1$  from  $\hat{\mathbf{x}}_2$ , whereas in DP-SGD  $\hat{\mathbf{x}}_1$  and  $\hat{\mathbf{x}}_2$  are uncorrelated. Hence there is a tradeoff between the total error and sensitivity (see next paragraph): DP-SGD sets  $\mathbf{C}_{[i,j]}^{-1} = 0$
Figure 7: MF-DP-FTRL (Alg. 1) enables noise cancelling across steps, where DP-SGD does not. The entries  $\mathbf{C}_{i,j}^{-1}$  are mostly negative (in  $[0, -1)$ ) in matrices  $\mathbf{C}^{-1}$  we consider (see Fig. 8). Thus, the red terms show that MF-DP-FTRL "cancels out" noise added on earlier iterations. For simplicity, we assume  $\mathbf{C}^{-1}$  has 1s on the main diagonal and entries  $\mathbf{C}_{i,j}^{-1}$  otherwise, with  $\mathbf{z}_i \coloneqq \mathbf{z}_{[i,:]}$  the rows of  $\mathbf{z}$ .
below the main diagonal, effectively minimizing sensitivity (assuming a fixed normalization of the main diagonal), but with a large total error due to no noise cancellation. On the other hand, MF-DP-FTRL can arrive at a better compromise between mechanism sensitivity and the total error. This is formalized in the optimization problem of Sec. 4.
Without a sampling assumption, the implied DP adversary knows which examples participated in batch  $S_{i}$ , and for DP-SGD with uncorrelated noise, knows they only need to "attack"  $\hat{\mathbf{x}}_i$ . However, with MF-DP-FTRL, the information from  $S_{i}$  can potentially be masked with a larger amount of initial noise in  $\hat{\mathbf{x}}_i$ , which is then canceled out over subsequent rounds. "Spreading out" the release of information about batch  $S_{i}$  over a larger number of iterations in this way can intuitively provide better privacy, while still allowing for accurate partial sums of gradients (and hence SGD iterates). This is, in a loose sense, similar to the way SGD with sampling "hibes" information about a particular example at randomly chosen iterations.
B Dropping the diag  $(\mathbf{X}) = 1$  constraint
As discussed in Sec. 4, BANDMF by default imposes an equal column norm constraint on the generated factorization. In the optimization problem, this is accomplished by imposing the constraint  $\operatorname{diag}(\mathbf{X}) = \mathbf{1}$ . In this section we show how we can solve the optimization problem without this constraint for  $(k,b)$ -participation. This optimization problem is formulated to minimize total squared error with respect to  $(k,b)$ -participation, although in principle the optimized matrices could be used in the  $b$ -min-sep-participation setting with some degradation in solution quality. Prop. B.1 provides an expression for efficiently computing the sensitivity of a  $b$ -banded matrix.
Proposition B.1. Let  $\mathbf{X} \in \mathbf{S}_{+}^{n}$  be a  $b$ -banded matrix, and let  $\Pi$  denote the  $(k,b)$ -participation schema. Then
$$
\operatorname {s e n s} _ {\Pi} (\mathbf {X}) = \max  _ {i = 1, \dots , b} \sum_ {j = 0} ^ {k - 1} \operatorname {d i a g} \left(\mathbf {X}\right) _ {i + j b}.
$$
To integrate this expression into the optimization problem, we can replace the diag  $(\mathbf{X}) = \mathbf{1}$  constraint with  $b$  linear constraints on diag  $(\mathbf{X})$ . This modification does not affect the convexity of the problem, although it does slightly complicate the algorithms needed to solve it. To handle this new problem, our approach is to replace the gradient with respect to  $\mathbf{X}$  at each iteration, with a projected gradient, which is obtained by setting  $v_{i} = \sum_{j=0}^{k-1} \mathrm{diag}(\Delta \mathbf{X})_{i+jb}$  for all  $i = 1, \ldots, b$ , and setting  $\mathrm{diag}(\Delta \mathbf{X})_{i+jb} = \mathrm{diag}(\Delta \mathbf{X})_{i+jb} - v_{i}/k$ . This ensures that sensitivity does not change between iterations of the numerical optimization procedure.
For the reasons mentioned in Sec. 4, by default we impose the simpler constraint  $\mathrm{diag}(\mathbf{X}) = \mathbf{1}$ . In App. C, we provide some numerical comparisons between these two approaches. Specifically, the rows of Table 2 with Equal column norms? (F)also correspond to the approach described here; observe this results in slightly improved RMSE under  $(k,b)$ -participation compared to the corresponding rows with Equal column norms? (T)true.
Mechanism	Matrix	Bands b	Equal column norms? (Ours)	Sensitivity	Error
k=1 [17]	(k=6, b=342) [15]	b≥342-min-sep (Ours)	(A) RMSE under (k=6, b=342) [15]	(B) RMSE under b≥342-min-sep (Ours)
OPTIMAL TREEAGG [30, 35]	-	F	0.32	1.00	1.00	1.53	1.53
DP-SGD [1]	1	T	0.41	1.00	1.00	9.63	9.63
MF (b=128) (Ours)	128	F	0.52	1.00	1.04	1.23	1.29
MF (b=128) (Ours)	128	T	0.41	1.00	1.00	1.27	1.27
MF (b=342) (Ours)	342	F	0.52	1.00	1.04	1.04	1.08
MF (b=342) (Ours)	342	T	0.41	1.00	1.00	1.05	1.05
MF [15]	-	F	0.50	1.00	≤1.15	1.00	1.15
MF [15]	-	T	0.41	1.00	≤1.13	1.01	1.14
Table 2: A comparison of matrix mechanisms for  $n = 2052$  under different participation patterns. Banded matrices are near-optimal under  $(k,b)$ -participation and best under  $b$ -min-sepparticipation. Each error is computed under the indicated measure of sensitivity. Thus, the error in column (B) can be obtained by multiplying the error in column (A) by the corresponding entry under  $b \geq 342$  sensitivity.
C Empirical evaluation of banded matrices
Table 2 compares the matrix mechanisms studied under different participation patterns but normalized to have sensitivity  $\mathrm{sens}(\mathbf{C}) = 1$  under  $(k = 6, b = 342)$ -participation. The sensitivity under single participation  $k = 1$  is lowest as expected. With column normalization, sensitivity is also 1 under  $b \geq 342$ -min-sep-participation. We make the following observations:
- For the MF mechanisms, column normalization hurts RMSE for  $(k,b)$ -participation compared to the approach of App. B (as it is an additional constraint), but actually improves RMSE under  $b$ -min-sep-participation.
- We conjecture that the  $(k, b)$ -participation optimized matrices (MF without column normalization, App. B) are optimal for the prefix-sum workload<sup>8</sup>; With this in mind, we see there is at most a small increase in RMSE for switching to the more challenging  $b$ -min-sepparticipation schema  $(1.00 \to 1.05)$ . If (as we further conjecture) the optimal matrices for prefix-sum in fact are  $k$ -banded, the gap is even smaller (at most  $1.04 \to 1.05$ ). Hence, at least for the prefix-sum workload  $\mathbf{A}$ , there is limited room for improvement in developing optimization procedures that directly optimize over the larger feasible set offered by  $b$ -min-sep-participation.
- Using fewer than  $b$  bands does degrade performance on the RMSE metric, with DP-SGD being the extreme case, yielding prefix sum estimates almost  $10 \times$  worse than the MF mechanisms.
- The results of Denisov et al. [17] imply that the binary-tree  $\mathbf{C}$  matrix can in fact be used in the online setting, with the Moore-Penrose pseudo-inverse giving the optimal decoder for RMSE [15], corresponding to the 'full' estimator of Honaker [30]. We include this in the table as a baseline, and see that it is in general outperformed by our MF mechanisms by about  $1.5 \times$  in RMSE.
This conjecture is not trivially true, as we still enforce a non-negativity or orthogonality constraint; see Choquette-Choo et al. [15, Appendix I.3]. Hence the conjecture is that these constraints are already satisfied by the optimal matrix for this workload.
D Example structures of MF
Figs. 8 and 9 show the structure of some of the key matrix factorization approaches considered in this work. One can immediately see the impact of the  $(k,b)$ -participation schema in the optimal matrices, in particular for the non-banded MULTI-EPOCH MF matrices (the two top-right matrices), where  $\mathbf{C}$  contains diagonals of negative entries separated by  $b$  steps. In the bottom two rows, we see that requiring equal column norms ("EN-" for equal norms) has a relatively minor impact on the structure of the matrices.
Figure 8: Factorizations for  $n = 64$  of the prefix-sum workload (A taken to be the lower-triangular matrix of 1s). For each factorization  $\mathbf{A} = \mathbf{BC}$ , we show  $\mathbf{C}$  and its inverse  $\mathbf{C}^{-1}$ , as the inverse is the matrix used in noise generation. Single-epoch is the approach of Denisov et al. [17], SGD is simply the identity matrix  $\mathbf{I}$  (shown for completeness), and  $(k = 8)$ -epoch MF and  $(k = 4)$ -epoch are the MULTI-EPOCH MF approach of Choquette-Choo et al. [15] for 8 and 4 epochs, respectively. For our banded matrices (3rd and 4th rows), we fix 4 epochs  $(b = 16)$ , and show  $\hat{b} = 8$  and  $\hat{b} = 16$  bands, with column normalization ("EN-") and without.
Figure 9: The transpose of binary-tree encoder matrix  $\mathbf{C}_{\mathcal{T}}$ , and its pseudoinverse  $\mathbf{C}_{\mathcal{T}}^{\dagger}$ , which corresponds to the "full" or optimal decoder of Honaker [30]. This is the matrix used in OPTIMAL TREEAGG in Fig. 6[a].
E Algorithms and Analysis for Sec. 3
E.1 Algorithms
Algorithm 3 (VECSENS): Maximum of  $\langle \mathbf{v},\mathbf{u}\rangle$  where  $\mathbf{u}$  is a vector in the  $\ell_{\infty}$  unit ball satisfying  $\Pi_b$ .
Inputs: min-separation  $b$  , vector  $\mathbf{v}$  , max participations  $k$    
Initialize  $F\in \mathbb{R}^{n\times k}$    
for  $m = 1,\dots ,k$  do for  $i = n,\ldots ,1$  do  $\triangleright$  We use the convention that  $F[s,t] = 0$  if  $s,t$  are out-of-bounds.  $F[i,m] = \max \Bigl (\mathbf{v}_i + F[i + b,m - 1],F[i + 1,m]\Bigr)$    
return  $F[1,k]$
Algorithm 4 Efficient sensitivity upper bound for  $b$ -min-sep-participation
Inputs: min-separation  $b$  , matrix  $\mathbf{X}$  , max participations  $k$  Initialize  $F\in \mathbb{R}^{n\times k}$ $\mathbf{v}\in \mathbb{R}^n$    
for  $j = 1,\dots ,n$  do  $\mathbf{v}_i = \mathrm{VECSENS}(b,|\mathbf{X}_{[i,:]}|,k)$    
return  $\sqrt{\mathrm{VECSENS}(b,\mathbf{v},k)}$
Algorithm 5 Efficient sensitivity calculation for  $b$ -min-sep-participation, assuming  $\mathbf{X}$  is  $b$ -banded.
Inputs: min-separation  $b$ ,  $b$ -banded matrix  $\mathbf{X}$ , max participations  $k$ .  
return  $\sqrt{\mathrm{VECSENS}(b, \mathrm{diag}(\mathbf{X}), k)}$
E.2 Analysis
Proposition E.1. The sensitivity of  $\mathbf{C}$  for a given participation schema  $\Pi$  may be expressed as:
$$
\operatorname {s e n s} _ {\Pi} \left(\mathbf {C}\right) ^ {2} = \max  _ {\pi \in \Pi} \sup  _ {\mathbf {u} \in \mathfrak {D}} \operatorname {t r} \left(\left[ \mathbf {P} _ {\pi} \mathbf {C} ^ {\top} \mathbf {C P} _ {\pi} \right] \left[ \mathbf {u u} ^ {\top} \right]\right), \tag {7}
$$
where  $\mathbf{P}_{\pi}$  represent the axis-aligned projection onto the set of rows indexed by  $\pi$ ; that is,  $\mathbf{P}_{\pi}[i,i] = 1$  for  $i \in \pi$ , and 0 otherwise. Assuming that  $\mathfrak{D}$  represents a set of matrices with rows bounded by  $\ell_2$  norm 1, this can be upper bounded by:
$$
\max _ {\pi \in \Pi} \sum_ {i, j \in \pi} | \mathbf {X} _ {[ i, j ]} |.
$$
where  $\mathbf{X} = \mathbf{C}^{\top}\mathbf{C}$ . This upper bound is tight when  $\mathbf{P}_{\pi}\mathbf{C}^{\top}\mathbf{C}\mathbf{P}_{\pi} \geq 0 \forall \pi \in \Pi$ , and is independent of the dimension  $d$  of the rows of  $\mathbf{u}$ .
Proof. Recall that  $\Pi$  determines the rows of  $\mathbf{u}$  which may be nonzero in the definition Eq. (2). Take some  $\mathbf{u} \in \mathfrak{D}$ , an element of  $\mathbb{R}^{n \times d}$ , which therefore has nonzero rows only at some set of indices  $\pi \in \Pi$ . Note, clearly  $\mathbf{u} = \mathbf{P}_{\pi} \mathbf{u}$ ,  $\mathbf{P}_{\pi}^{\top} = \mathbf{P}_{\pi}$ , and  $\mathbf{P}_{\pi} = \mathbf{P}_{\pi} \mathbf{P}_{\pi}$ .
$$
\begin{array}{l} \left\| \mathbf {C} \mathbf {u} \right\| _ {F} ^ {2} = \operatorname {t r} \left(\left[ \mathbf {C P} _ {\pi} \mathbf {u} \right] ^ {\top} \mathbf {C P} _ {\pi} \mathbf {u}\right) = \operatorname {t r} \left(\mathbf {u} ^ {\top} \mathbf {P} _ {\pi} ^ {\top} \mathbf {C} ^ {\top} \mathbf {C P} _ {\pi} \mathbf {u}\right) \\ = \operatorname {t r} \left(\mathbf {P} _ {\pi} \mathbf {P} _ {\pi} \mathbf {C} ^ {\top} \mathbf {C P} _ {\pi} \mathbf {P} _ {\pi} \mathbf {u u} ^ {\top}\right) = \operatorname {t r} \left(\left[ \mathbf {P} _ {\pi} \mathbf {C} ^ {\top} \mathbf {C P} _ {\pi} \right] \left[ \mathbf {P} _ {\pi} \mathbf {u u} ^ {\top} \mathbf {P} _ {\pi} \right]\right) \tag {8} \\ = \operatorname {t r} \left(\left[ \mathbf {P} _ {\pi} \mathbf {C} ^ {\top} \mathbf {C P} _ {\pi} \right] \left[ \mathbf {u u} ^ {\top} \right]\right). \\ \end{array}
$$
This implies the statement Eq. (7) by the definition of sensitivity and neighboring in our setting.
Now, let  $\mathbf{X}_{\pi} \coloneqq \mathbf{P}_{\pi} \mathbf{C}^{\top} \mathbf{C} \mathbf{P}_{\pi}$  be the matrix formed by zeroing out the rows and columns not indexed by  $\pi$  from  $\mathbf{X}$ . Assume that every  $\mathbf{u} \in \mathfrak{D}$  has row norms bounded by 1. Expanding the trace in Eq. (7), writing  $x_{ij}$  for the elements of  $\mathbf{X}_{\pi}$  and  $\mathbf{u}_{[j,:]}$  for the  $j^{th}$  row of  $\mathbf{u}$ , we have
$$
\operatorname {t r} \left(\mathbf {X} _ {\pi} \mathbf {u} \mathbf {u} ^ {\top}\right) = \sum_ {i = 1} ^ {k} \sum_ {j = 1} ^ {k} x _ {i j} \langle \mathbf {u} _ {[ i,: ]}, \mathbf {u} _ {[ j,: ]} \rangle \leq \sum_ {i = 1} ^ {k} \sum_ {j = 1} ^ {k} | x _ {i j} |
$$
which yields the claimed bound. When  $\mathbf{X}_{\pi}$  is elementwise nonnegative, taking  $\mathbf{u}_{[i,:]} = \mathbf{u}_{[j,:]}$  for any unit vector shows the claimed tightness in this case.
Remark. This statement can be viewed as a partial extension of [15, Theorem G.1]. It does not imply every case handled there, but also implies results which cannot be derived from that Theorem.
Proof of Thm. 2. Conclusion (1) is implied by (2), noting that the conditions on  $\mathbf{C}$  imply that Alg. 5 will return a value at most  $\kappa \sqrt{k'}$  in this setting.
For (2), let  $c \in \mathbb{R}^n$  with entries  $c_i = \| \mathbf{C}_{[i,i]} \|^2$  for  $i \in \{0, \dots, n-1\}$ . We have
$$
\operatorname {s e n s} _ {\Pi} ^ {1} (\mathbf {C}) = \max  _ {\pi \in \Pi_ {b}} \| \mathbf {C u} (\pi) \| = \max  _ {\pi \in \Pi_ {b}} \| \sum_ {i \in \pi} \mathbf {C} _ {[:, i ]} \| = \max  _ {\pi \in \Pi_ {b}} \sqrt {\sum_ {i \in \pi} c _ {i}} \tag {9}
$$
where  $\mathbf{u}(\pi) \in \{0,1\}^n$  is given by  $\mathbf{u}(\pi)_i = 1$  if  $i \in \pi$  and 0 otherwise. The last equality follows from the orthogonality condition on sufficiently separated columns of  $\mathbf{C}$  trivially implied by bandedness. It is straightforward to verify the dynamic program of Alg. 3 constructs a feasible  $\pi$  which attains the maximum.
Proof of Thm. 3. Via Prop. E.1, the result follows from showing that Alg. 4 outputs a value at least as large as  $\sum_{(i,j)\in \pi}|\mathbf{X}_{ij}|$  for any  $\pi \in \Pi_b$ . So let  $\hat{\pi}$  be an element of  $\Pi_b$ . Note that VECSENS is monotonically increasing in values of the vector  $\mathbf{v}$  if  $\mathbf{v}$  is nonnegative, and therefore Alg. 4 is monotonically increasing in absolute values of  $\mathbf{X}$ . Therefore we will have our conclusion (3) if we can show that, for  $\mathbf{X}_{\hat{\pi}}$  the matrix formed by zeroing out all rows and columns of  $\mathbf{X}$  not indexed by  $\hat{\pi}$ , Alg. 4 returns the value  $\sum_{(i,j)\in \pi}|\mathbf{X}_{ij}|$ . Yet this is straightforward by the characterization of VECSENS as an oracle for computing the maximum of  $\langle \mathbf{v},\mathbf{u}\rangle$ , where  $\mathbf{u}$  is a vector in the  $\ell_{\infty}$  unit ball.
Proof of Prop. 4.1. The proof will be constructive. Let  $\mathbf{J}$  be the  $n\times n$  exchange matrix defined as
$$
\mathbf {J} = \left[ \begin{array}{c c c c} & & & 1 \\ & & 1 & \\ & \ddots & & \\ 1 & & & \end{array} \right]
$$
Let  $\mathbf{Y} = \mathbf{J}\mathbf{X}\mathbf{J}$  and note that  $\mathbf{Y}$  is symmetric and positive definite. Let  $\mathbf{H} = \mathrm{Cholesky}(\mathbf{Y})^{\top}$  and note that (1)  $\mathbf{H}^{\top}\mathbf{H} = \mathbf{Y}$  by definition of Cholesky decomposition, (2)  $\mathbf{H}$  is upper triangular, and (3)  $\mathbf{H}$  is  $\hat{b}$ -banded by Du Croz et al. [19].
We will show that for  $\mathbf{C} = \mathbf{JHJ}$ , we have (1)  $\mathbf{X} = \mathbf{C}^{\top}\mathbf{C}$ , (2)  $\mathbf{C}$  is lower triangular, and (3)  $\mathbf{C}$  is  $\hat{b}$ -banded.
For Claim (1) observe that:
$$
\begin{array}{l} \mathbf {C} ^ {\top} \mathbf {C} = (\mathbf {J H J}) ^ {\top} (\mathbf {J H J}) \\ = \mathbf {J} ^ {\top} \mathbf {H} ^ {\top} \mathbf {J} ^ {\top} \mathbf {J} \mathbf {H} \mathbf {J} \\ = \mathbf {J} \left(\mathbf {H} ^ {\top} \mathbf {H}\right) \mathbf {J} \\ = \mathrm {J Y J} \\ = J J X J J \\ = \mathbf {X} \\ \end{array}
$$
For Claim (2) and (3), note that left-multiplying by  $\mathbf{J}$  reverses the rows and right-multiplying by  $\mathbf{J}$  reverses the columns, and therefore  $\mathbf{C}_{[i,j]} = \mathbf{H}_{n - i + 1,n - j + 1}$ .
For Claim (2), we need to show  $\mathbf{C}_{[i,j]} = 0$  if  $i < j$ . If  $i < j$  then  $n - i + 1 > n - j + 1$ , and since  $\mathbf{H}$  is upper triangular, we know  $\mathbf{H}_{[n - i + 1,n - j + 1]} = 0$ , as desired.
For Claim (3), we need to show that  $\mathbf{C}_{[i,j]} = 0if|i - j|\geq \hat{b}$ . Observe that if  $|(n - i + 1) - (n - j + 1)| = |i - j|$  and therefore since  $\mathbf{H}$  is  $\hat{b}$ -banded, so is  $\mathbf{C}$ .
F Additional Analysis for Sec. 5
In this section we prove our general amplification statement Theorem 5, of which Theorem 4 is a corollary. Recall that we use  $b$  instead of  $\hat{b}$  in this appendix since our sampling scheme enforces  $(k,b)$ -participation. Throughout this section, we slightly abuse notation by letting  $i \pmod{b} = b$  instead of 0 if  $i / b$  is integer.
F.1 Algorithms for Sampling
We first give the general sampling scheme (Alg. 6) as well as sequence of queries (Alg. 7) that provides an upper bound on the privacy guarantees of DP-MF using this sampling scheme.
Algorithm 6 Sampling scheme for banded DP-MF
Inputs: Dataset  $D$  , sampling distribution  $\mathcal{S}$  over  $(2^{[\tilde{m} ]})^k$  noise standard deviation  $\sigma$ $D_{1},\ldots ,D_{b}\gets$  arbitrary partition of  $D$  such that  $\forall j:|D_j| = \tilde{m}$    
Let  $D_{j} = \{d_{j,1},d_{j,2},\dots ,d_{j,\tilde{m}}\}$  for each  $j$    
for  $j = 1,2,\ldots ,b$  do Sample  $k$  sets to index  $D_{j}$  as  $(S_{j},S_{b + j},\dots ,S_{(k - 1)b + j})\sim S$  , with  $S_{j}\subseteq [\breve{m} ]$    
for  $i = 1,2,\dots ,n$  do Let  $j = i$  (mod  $b$  ); compute  $\mathbf{x}_i$  by querying  $\{d_{j,\ell}:\ell \in S_i\}$    
Let  $\mathbf{x} = [\mathbf{x}_1,\dots ,\mathbf{x}_n]^{\top}\in \mathbb{R}^{n\times d}$  , release  $\mathbf{Cx} + \mathbf{z}$  with each entry of  $\mathbf{z}_{[i,j]}\sim \mathcal{N}(0,\sigma^2)$  D If C is lower-triangular, results can also be released in streaming fashion