SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Christopher A. Choquette-Choo
cchoquette@google.com
Arun Ganesh
arunganesh@google.com
Ryan McKenna
mckennar@google.com
H. Brendan McMahan
mcmahan@google.com
Keith Rush
krush@google.com
Abhradeep Thakurta
athakurta@google.com
Zheng Xu
xuzheng@google.com
Matrix factorization (MF) mechanisms for differential privacy (DP) have substantially improved the state-of-the-art in privacy-utility-computation tradeoffs for ML applications in a variety of scenarios, but in both the centralized and federated settings there remain instances where either MF cannot be easily applied, or other algorithms provide better tradeoffs (typically, as  $\epsilon$  becomes small). In this work, we show how MF can subsume prior state-of-the-art algorithms in both federated and centralized training settings, across all privacy budgets. The key technique throughout is the construction of MF mechanisms with banded matrices (lower-triangular matrices with at most  $\hat{b}$  nonzero bands including the main diagonal). For cross-device federated learning (FL), this enables multiple-participations with a relaxed device participation schema compatible with practical FL infrastructure (as demonstrated by a production deployment). In the centralized setting, we prove that banded matrices enjoy the same privacy amplification results as the ubiquitous DP-SGD algorithm, but can provide strictly better performance in most scenarios—this lets us always at least match DP-SGD, and often outperform it.
We consider machine learning (ML) with DP in the centralized (datacenter) setting and the crossdevice FL setting, extending and improving matrix factorization (MF) mechanisms to advance the state-of-the-art in both. Given bounded-sensitivity batch gradients  $\mathbf{x}_i\in \mathbb{R}^d$  for  $i\in [n]$  steps, the MF-DP-FTRL algorithm uses a noise generation matrix  $\mathbf{C}^{-1}\in \mathbb{R}^{n\times n}$  to return DP gradient estimates  $\hat{\mathbf{x}}_i = \mathbf{x}_i + [\mathbf{C}^{-1}\mathbf{z}]_{[i,:]}$  where  $\mathbf{z}\in \mathbb{R}^{n\times d}$  has IID entries  $\mathcal{N}(0,\sigma^2)$  for suitable  $\sigma >0$ . The noise correlation induced by  $\mathbf{C}^{-1}$  is key to the success of MF-DP-FTRL. Alg. 1 and Sec. 2 provide details and intuition, and Table 1 in App. A summarizes notation and symbols.
<sup>1</sup>We use matrix factorization for the algorithmic design of DP mechanisms, and then use these mechanisms in general-purpose gradient-based private optimization algorithms. This use of matrix factorization is unrelated to machine learning settings such as collaborative filtering or recommender systems where the ML problem itself involves matrix factorization [37, 41, 51].
(a) Centralized CIFAR-10
(b) Centralized StackOverflow
Figure 1: In the centralized setting, our BANDMF mechanism consistently performs at least as well as the best prior methods. LEFT, (A): At  $\epsilon \geq 0.5$ , our BANDMF mechanism offers consistent utility benefits of around 1 - 4 percentage points over either DP-SGD [1] or MULTI-EPOCH MF [15]. RIGHT, (B): BANDMF (bands  $\hat{b} = 9, 18, 32$ , and 64 for  $\epsilon = 1 - 8$  respectively) significantly outperform both (unamplified) MULTI-EPOCH MF and amplified DP-SGD.
In datacenter applications, precise control of the sampling/shuffling of training data is possible, and so DP-SGD with privacy amplification [1] is one of the most popular ways to train machine learning models with formal privacy guarantees. However, Choquette-Choo et al. [15] recently demonstrated that a multi-epoch extension of the MF-DP-FTRL algorithm can outperform amplified DP-SGD in some settings, depending on the privacy and computational budget (typically larger budgets above  $\epsilon \approx 2$  and a small number of training epochs). This leaves the state-of-the-art for centralized DP training in the unsatisfactory state where one must try both algorithms to be assured of the best performance.
In cross-device federated learning, devices choose when they are available to participate in training, and so precise sampling and shuffling is generally not possible (see Sec. 3 for more details). Motivated by these limitations which make amplified DP-SGD infeasible, Kairouz et al. [35] developed the (tree-aggregation-based) DP-FTRL algorithm. Their DP-FTRL does not rely on (or benefit from) privacy amplification and instead adds carefully correlated noise to the gradients to boost utility. Denisov et al. [17] proposed MF-DP-FTRL, replacing the tree-aggregation scheme of Kairouz et al. [35] with a general matrix-factorization mechanism. By optimizing over this space to find mechanisms with optimal error, substantial performance improvements were possible. However, the work of Denisov et al. [17] applies only to the single participation (single epoch) setting. Hence, for cross-device FL the state-of-the-art also requires considering multiple algorithms: tree-aggregation-based DP-FTRL when devices may participate more than one time, or MF-DP-FTRL when devices participate only once. Importantly, the extension of MF-DP-FTRL to multiple epochs of Choquette-Choo et al. [15] only applies in the centralized setting, as it again requires precise control of the participation pattern.
In this work, we address the limitations of MF-DP-FTRL (MF for short) noted above, and show that it can, in fact, achieve across-the-board state-of-the-art performance in both settings across all  $\epsilon$ . To accomplish this, we define a family of banded MF mechanisms, shown in Fig. 4 (c.f. Fig. 8 of App. D for visualizations of other factorization structures). We summarize our main contributions below.
Contributions for cross-device FL Here, the  $(k,b)$ -participation schema of Choquette-Choo et al. [15] cannot be enforced. We propose a strict generalization,  $b$ -min-sep-participation, which can be practically enforced by FL infrastructure. We show how to efficiently and exactly bound the sensitivity for banded matrices in Thm. 2, allowing formal DP guarantees and the numerical optimization of optimal mechanisms (Sec. 4). These innovations lead to significant privacy-utility benefits in a production deployment (Fig. 6 of Sec. 6). Our work also generalizes the sensitivity calculations of Choquette-Choo et al. [15] to provide a general upper-bound on  $b$ -min-sep-participation sensitivity (Thm. 3), which allows the matrices of Choquette-Choo et al. [15] to be used in the FL setting, as well as removing the need to exactly bound  $b$  before training (see Sec. 6 and App. K).
Contributions for centralized training The existing privacy amplification analysis of DP-SGD does not allow for the correlated noise that is applied in MF-DP-FTRL. Our paper introduces a novel partitioning of the BANDMF iterates into independent queries. This allows us to prove in Thm. 4 of Sec. 5 that banded matrices enjoy the benefits of privacy amplification, and show that DP-SGD is a special case, giving us the best of both algorithms. This enables us to always pareto-dominate DP-SGD, unlike Choquette-Choo et al. [15] which only does so for large enough  $\epsilon$  as observed in Fig. 1. Further, this allows us to improve on both baselines, between  $1 - 4\%$  -points. Informally:
Figure 2: MF-DP-FTRL (Alg. 1) enables noise cancelling across steps, where DP-SGD does not. The entries  $\mathbf{C}_{i,j}^{-1}$  are mostly negative (in  $[0, -1)$ ) in matrices  $\mathbf{C}^{-1}$  we consider (see Fig. 8). Thus, the red terms show that MF-DP-FTRL "cancels out" noise added on earlier iterations. For simplicity, we assume  $\mathbf{C}^{-1}$  has 1s on the main diagonal and entries  $\mathbf{C}_{i,j}^{-1}$  otherwise, with  $\mathbf{z}_i \coloneqq \mathbf{z}_{[i,:]}$  the rows of  $\mathbf{z}$ .
Theorem 1 (Informal version of Theorems 4 and 5). Suppose we partition the dataset into  $b$  equal-size subsets, and in step  $i$  each example in the  $i$  ( $\bmod b$ )-th subset participates with probability  $\frac{Bb}{m}$  where there are  $m$  examples and the batch size is  $B$ . Then, a  $\hat{b}$ -banded  $n$ -iteration matrix mechanism with  $\hat{b} \leq b$  satisfies the same privacy guarantees as answering  $n/b$  queries on a dataset, where each element of the dataset is independently included in each query with probability  $\frac{Bb}{m}$ .
As an example of Thm. 1, consider doing  $n = 2,000$  iterations of DP-SGD on CIFAR-10, which has  $m = 50,000$  examples, using a minibatch of  $B = 500$  examples in each round. This has the same DP guarantees as answering 2,000 queries using a subsampled Gaussian mechanism with sampling probability  $p = 500 / 50,000 = 0.01$ . If we instead use, e.g., BANDMF with  $\hat{b} = b = 10$ , our suggested sampling scheme is the following: Partition CIFAR-10 into 10 subsets of 5,000 examples each,  $D_{1},D_{2},\ldots ,D_{10}$ . In rounds 1, 11, 21... we sample 500 examples from  $D_{1}$ , in rounds 2, 12, 22... we sample 500 examples from  $D_{2}$ , and so on. We sample from each  $D_{i}$  a total of  $2000 / 10 = 200$  times, and each time our sampling probability is  $500 / 5000 = 0.1$  within the subset. So Theorem 1 shows  $\hat{b} = 10$  BANDMF satisfies the same DP guarantees as answering 200 queries with  $p = 0.1$ . As a special case of Theorem 1, DP-SGD is simply MF with a suitable diagonal matrix with  $\hat{b} = 1$ , and thus Thm. 1 recovers the privacy guarantees of DP-SGD with amplification by sampling. Empirically, we show that MF with amplification has privacy-utility tradeoffs that are no worse than DP-SGD for all  $\epsilon$ , and often significantly better as can be seen in Fig. 1.
Finally, we explore the computational tradeoffs of our approach. We find that banded matrices with  $b$ -min-sep-participation are equally efficient to optimize as those under  $(k,b)$ -participation but significantly reduce the memory and time complexity of the per-iteration noise generation from  $\mathcal{O}(n)$  to a constant  $\mathcal{O}(\hat{b})$  (where often total steps  $n \gg d$ ). We will release all code with the final manuscript.
Related work The matrix mechanism (MF mechanism or MF) [40] has a rich history in offline, statistical queries [22, 28, 42, 62], with many applications including to online PCA [21], estimating marginals [22], and top-k selection [17]. Recently, this has been studied under the adaptive streaming setting, where privacy analysis must account for an adversary adaptively defining the inputs at each step [17, 24]. Denisov et al. [17] showed a connection with the DP-FTRL algorithm of Kairouz et al. [35] and with DP ML broadly; they showed that computing optimal MF significantly improves the privacy-utility-computation tradeoffs when making only a single pass (epoch) over the training data. Choquette-Choo et al. [15] showed that MF achieves state-of-the-art results in DP ML by showing how to optimize MF under arbitrary passes over the training data. Henzinger and Upadhyay [29] study the problem of DP-continual observation [13, 20] and the explicit factorization of the workload matrix A that minimizes the completely bounded norm, which is only off from the optimal by an additive constant. The connection between DP empirical risk minimization [3, 4, 5, 7, 8, 9, 14, 16, 23, 25, 32, 36, 39, 45, 52, 53, 54, 57] and DP online regret minimization [2, 4, 5, 26, 33, 35] has been studied for a long time. Asi et al. [5] demonstrated that DP-FTRL style algorithms [15, 17, 35] achieve the best known regret in certain classes of online learning problems (a.k.a. the realizable regime). An important question that still remains open is whether DP-FTRL style algorithms can obtain optimal population risk guarantees under DP [9].
Algorithm 1 MF-DP-FTRL and DP-SGD
Inputs: Initial model  $\pmb{\theta}_{1} \in \mathbb{R}^{d}$ , dataset  $D$  examples, matrix  $\mathbf{C}^{-1} \in \mathbb{R}^{n \times n}$ , noise  $\mathbf{z} \in \mathbb{R}^{n \times d}$  with entries i.i.d.  $\mathcal{N}(0, \sigma^2)$ , clipnorm  $\zeta$ $\triangleright$  DP-SGD is simply the case  $\mathbf{C}^{-1} = \mathbf{I}$ , the  $n \times n$  identity matrix.  
for  $i = 1, 2, \dots, n$  do  
    Select a set  $S_{i} = \{\mathbf{d}_{1}, \dots, \mathbf{d}_{B}\} \subseteq D$ $\triangleright$  Respecting schema II, possibly sampling, Alg. 2  
     $\mathbf{g}_{j} = \mathrm{clip}(\nabla_{\theta} \mathrm{loss}(\mathbf{d}_{j}, \pmb{\theta}_{i}), \zeta)$  for  $j \in [B]$ , where  $\mathrm{clip}(\mathbf{d}, \zeta) = \min(1, \zeta / \| \mathbf{d} \|) \mathbf{d}$ $\mathbf{x}_{i} = \sum_{j=1}^{B} \mathbf{g}_{j}$ $\hat{\mathbf{x}}_{i} = \mathbf{x}_{i} + \zeta [\mathbf{C}^{-1} \mathbf{z}]_{[i,:]}$ $\triangleright$ $\hat{\mathbf{x}}_{i}$  is now a DP estimate of  $\mathbf{x}_{i}$ $\pmb{\theta}_{i+1} = \mathrm{SGDM}(\pmb{\theta}_{i}, \hat{\mathbf{x}}_{i})$ $\triangleright$  Any first-order optimizer can be used in place of SGDM
2 Matrix Factorization, Sensitivity, and Efficient Implementations
Let  $\mathbf{x} \in \mathbb{R}^{n \times d}$  be a stream of model gradients, and let  $\mathbf{A} \in \mathbb{R}^{n \times n}$  be an appropriate linear query workload (prefix-sums, or a matrix encoding of stochastic gradient descent with momentum (SGDM) [17]). Matrix mechanisms use a factorization  $\mathbf{A} = \mathbf{BC}$  to privately estimate the quantity  $\mathbf{A}\mathbf{x}$  as
$$
\widehat {\mathbf {A x}} = \mathbf {B} (\mathbf {C x} + \mathbf {z}), \tag {1}
$$
where  $\mathbf{z}$  is suitably calibrated noise to the sensitivity of the so-called 'query matrix'  $\mathbf{C}$ .
Efficiently implementing MF-DP-FTRL Eq. (1) can be re-arranged as  $\mathbf{A}(\mathbf{x} + \mathbf{C}^{-1}\mathbf{z})$ . The multiplication by the linear operator  $\mathbf{A}$  can now be viewed as post-processing of the noisy mechanism outputs  $\mathbf{x} + \mathbf{C}^{-1}\mathbf{z}$ ; in many cases, this postprocessing has an efficient streaming implementation, e.g., simply passing gradients into a SGDM implementation. Thus, implementing MF-DP-FTRL is essentially equivalent to DP-SGD. The only difference is that the per-iteration gradient  $\mathbf{x}_i$  is protected with noise  $[\hat{\mathbf{C}}^{-1}\mathbf{z}]_{[i,:]}$  rather than  $\mathbf{z}_{[i,:]}$ . Indeed, we see DP-SGD is a special case simply by taking  $\mathbf{C} = \mathbf{C}^{-1} = \mathbf{I}$ . Further, as long as the noise  $\mathbf{C}^{-1}\mathbf{z}$  is computed correctly, the privacy guarantee holds, independent of the choice of  $\mathbf{A}$ . Alg. 1 gives the complete algorithm; with an appropriate choice of the matrix  $\mathbf{C}^{-1}$ , this algorithm captures DP-SGD, tree-aggregation DP-FTRL² [35], as well as MF-DP-FTRL [15, 17]. The multiplication of  $\mathbf{C}^{-1}$  by Gaussian noise  $\mathbf{z} \in \mathbb{R}^{n\times d}$  (which need never be fully materialized at once) is the critical step in the efficient implementation of MF. In App. J, we note this multiplication can be completed online for  $\hat{b}$ -banded matrices (defined formally in Sec. 3) in time and memory  $\mathcal{O}(\hat{b}d)$  per training iteration compared to  $\mathcal{O}(nd)$  for a non-banded matrix.
Multiple participations We adopt the formalisms for multiple-participations of Choquette-Choo et al. [15]. We assume there are  $m$  examples (or users in FL) in the database where  $B$  examples are selected on each step  $i \in [n]$ . These  $B$  chosen examples are said to participate on this step  $i$ . The examples at each step are processed via any adaptive function, e.g., computing a gradient of the current model (which depends on the model parameter values) as in Alg. 1. The per-example output vectors  $\mathbf{g} \in \mathbb{R}^d$  are each bounded to  $\ell_2$  norm at most  $\zeta$  (noting that our notions of sensitivity scale linearly in  $\zeta$ , without loss of generality (WLOG) we take to be 1 in analysis below). These clipped vectors are then summed to yield  $\mathbf{x}_i = \sum_{j=1}^{B} \mathbf{g}_j$ . The MF-DP-FTRL mechanism releases the privatized estimates of  $\mathbf{x}_i$  in a streaming fashion. The multi-epoch setting occurs when  $m < n \cdot B$ , so that every example necessarily participates more than once.
Intuition for (anti-)correlated noise in MF-DP-FTRL Fig. 2 compares DP-SGD and MF-DP-FTRL. To gain an intuition for why MF-DP-FTRL can perform better than DP-SGD, observe that vanilla SGD has iterates  $\theta_{t} = \theta_{0} - \eta \sum_{i=1}^{t} \hat{\mathbf{x}}_{i}$ , and hence when the noisy gradients  $\hat{\mathbf{x}}_{i}$  are added, the  $\mathbf{C}_{[i,j]}^{-1} \mathbf{z}_{[i,:]}$  terms in MF-DP-FTRL serve to cancel out some of the noise introduced on previous rounds. This reduces the total error in the final model (i.e., the prefix sums). However, this worsens the sensitivity of the mechanism to  $>1$  (as it is for DP-SGD). This is because an adversary trying to learn  $\mathbf{x}_{1}$  via  $\hat{\mathbf{x}}_{1}$  can partially learn the value of  $\mathbf{z}_{1}$  from  $\hat{\mathbf{x}}_{2}$ , whereas in DP-SGD  $\hat{\mathbf{x}}_{1}$  and  $\hat{\mathbf{x}}_{2}$  are uncorrelated. This tradeoff is what MF-DP-FTRL aims to minimize. More details on this intuition are in App. A.1.
2In this case we use a suitable pseudo-inverse  $\mathbf{C}^{\dagger}\in \mathbb{R}^{n\times 2n - 1}$  with noise  $\mathbf{z}\in \mathbb{R}^{2n - 1\times n}$
Adjacency and participation schemas DP requires a notion of adjacent datasets. Two data streams  $\mathbf{x}$  and  $\tilde{\mathbf{x}}$  are adjacent if the data associated with any single example is altered, but not when this example participated. Thus, any  $\mathbf{x}_i$  where example  $\mathbf{d}_j$  participated can be changed subject to the constraint  $\| \mathbf{g}_j^{(i)}\| \leq \zeta$ . However, the participation pattern does not change. A participation schema  $\Pi$  gives the set of possible participation patterns  $\pi \in \Pi$ , with each  $\pi \subseteq [n]$  indicating a set of steps in which a single example might participate. Let  $\mathbf{N}$  be the set of all pairs of neighboring streams  $\mathbf{x}$  and  $\mathfrak{D} := \{\mathbf{x} - \tilde{\mathbf{x}} \mid (\mathbf{x},\tilde{\mathbf{x}})\in \mathbf{N}\}$  represent the set of all possible deltas between neighboring  $\mathbf{x}$ ,  $\tilde{\mathbf{x}}$ . We say a  $\mathfrak{D}$  satisfies the participation schema  $\Pi$  if the indices of all nonzero rows in each  $\mathbb{R}^{n\times d}$  matrix  $\mathbf{u} \in \mathfrak{D}$  are a subset of some  $\pi \in \Pi$ . To illustrate this, single-participation is represented as  $\Pi = \{\{1\},\{2\},\ldots\{n\}\}$  and full-batch gradient descent (every-step) as  $\Pi = \{[n]\}$ . Choquette-Choo et al. [15] studied fixed-epoch-order participation, denoted  $(k,b)$ -participation, where each example participates at most  $k$  times, with any adjacent participations exactly  $b$  steps apart: formally,  $\Pi$  is the set of all  $\pi$  such that  $|\pi |\leq k$ , and if  $\pi = \{i_1,\dots,i_k\}$  indexed in increasing order, we have  $\forall j\in \{2,\dots,k\}, i_j - i_{j - 1} = b$ . For example  $(k = 2,b = 3)$ -participation has  $\Pi = \{\{1,4\},\{2,5\},\{3,6\}\}$ . As discussed in Choquette-Choo et al. [15], this setting faithfully captures centralized multi-epoch ML training setups with single and every-step as special cases. We can now define the sensitivity of the matrix factorization mechanism as
$$
\operatorname {s e n s} _ {\Pi} (\mathbf {C}) = \sup  _ {(\mathbf {x}, \tilde {\mathbf {x}}) \in \mathbf {N}} \| \mathbf {C x} - \mathbf {C} \tilde {\mathbf {x}} \| _ {F} = \sup  _ {\mathbf {u} \in \mathfrak {D}} \| \mathbf {C u} \| _ {F}. \tag {2}
$$
Optimizing factorizations Different factorizations  $\mathbf{A} = \mathbf{BC}$  can have very different performance in practice. Thus, in MF applications it is common to optimize over the space of factorizations, where the objective function is the expected total squared error on  $\mathbf{A}$ , given as  $\mathcal{L}(\mathbf{B},\mathbf{C}) = \mathrm{sens}_{\Pi}^{2}(\mathbf{C})\|\mathbf{B}\|_{F}^{2}$ . We define the expected root-mean-squared error (RMSE) as  $\sigma \sqrt{\mathcal{L}(\mathbf{B},\mathbf{C}) / n}$ , where  $\sigma$  is the standard deviation of the Gaussian noise. We take  $\sigma = 1$  when simply comparing mechanisms, or (in Sec. 6), calibrate  $\sigma$  to achieve specific  $(\epsilon, \delta)$ -DP guarantees.
To facilitate optimization, utilizing the fact that the optimal-for-squared-error decoder  $\mathbf{B}$  is  $\mathbf{AC}^{\dagger}$  [17], we note  $\mathcal{L}(\mathbf{B},\mathbf{C}) = \mathcal{L}(\mathbf{AC}^{\dagger},\mathbf{C})$ . The expected total squared error is invariant to scaling  $\mathbf{C}$  by a constant, and hence it is sufficient to optimize under a sensitivity 1 constraint. Further expressing the sensitivity and error in terms of  $\mathbf{X} = \mathbf{C}^{\top}\mathbf{C}$  (note  $\mathbf{X}$  is unrelated to the data  $\mathbf{x}$ ), we have
$$
\mathcal {L} (\mathbf {B}, \mathbf {C}) = \mathcal {L} \left(\mathbf {A C} ^ {\dagger}, \mathbf {C}\right) = \left\| \mathbf {A C} ^ {\dagger} \right\| _ {F} ^ {2} = \operatorname {t r} \left[ \mathbf {A} ^ {\top} \mathbf {A X} ^ {- 1} \right], \tag {3}
$$
assuming  $\mathrm{sens}_{\Pi}(\mathbf{C}) = 1$  and  $\mathbf{A}$  is in the rowspace of  $\mathbf{C}$ . Thus, we arrive at:
Problem 1. The matrix factorization optimization problem is to solve the convex optimization
$$
\underset {\mathbf {X} \in \mathbf {S} _ {+} ^ {n}} {\text {m i n i m i z e}} \quad \operatorname {t r} \left[ \mathbf {A} ^ {\top} \mathbf {A} \mathbf {X} ^ {- 1} \right] \quad \text {s u b j e c t t o} \quad \operatorname {s e n s} _ {\Pi} ^ {2} (\mathbf {X}) \leq 1, \tag {4}
$$
and then find  $\mathbf{C}$  so that  $\mathbf{C}^{\top}\mathbf{C} = \mathbf{X}$ , e.g., via Cholesky decomposition.
3 A Participation Schema for FL and the Sensitivity of Banded Matrices
In cross-device FL, devices locally evaluate eligibility criteria to determine when they might participate in training [10, 34], for example only checking-in to the coordinating server when they are plugged in, on unmetered wifi, and idle. This makes it practically difficult to enforce the  $(k,b)$ -participation of Choquette-Choo et al. [15], where devices are assumed to participate at the same relative position in each epoch: devices are unlikely to meet the eligibility criteria during the narrow windows of both step  $i$  and  $i + b$ . Further, precise sampling cannot provide the same level of privacy amplification as in the centralized setting. Consider if 6500 devices are needed to complete a round [59]. An extreme (but realistic depending on time of day) setting may have only 6500 devices meeting eligibility criteria. Thus, either the protocol proceed without any sampling/amplification or wait until more devices are available; neither are desirable. We avoid amplification in the cross-device setting and instead proceed by addressing the question: Can MF-DP-FTRL be extended to the cross-device federated learning setting with multiple client participations?
With  $(k,b)$ -participation difficult to enforce in practice, our first challenge is to define a new participation schema with several properties: (a) the sensitivity of any matrix mechanism under this query can be bounded; (b) this bound is tight over an expressive class of matrices; (c) this bound
can be efficiently represented as a constraint in a mathematical program so as to be able to find a near-optimal factorization  $\mathbf{A} = \mathbf{BC}$ . In Defn. 1, we propose  $b$ -min-sep-participation, a generalization of  $(k,b)$ -participation which can be practically enforced by cross-device FL systems, thus enabling us to leverage BANDMF in this setting (see Sec. 6). In  $b$ -min-sep-participation, the distance between any two participations is at least  $b$ , rather than exactly  $b$  as in  $(k,b)$ -participation:
Definition 1. The  $b$ -min-sep-participation schema is given by
$$
\Pi_ {b} = \left\{\pi \subseteq [ n ] \mid \{i, j \} \subseteq \pi , i \neq j \Rightarrow | i - j | \geq b \right\}.
$$