Enabling Differentially Private Federated Learning for Speech Recognition: Benchmarks, Adaptive Optimizers, and Gradient Clipping
Authors: Martin Pelikan
Sheikh Shams Azam
Vitaly Feldman
Jan Silovsky
Kunal Talwar
Christopher Brinton
Tatiana Likhomanenko
Venue: NeurIPS 2025 poster ()
Keywords: automatic speech recognition
federated learning
differential privacy
Primary area: applications
TL;DR: We introduce the first practical benchmark for FL with DP in ASR, combining theoretical insights on gradient heterogeneity with empirical results that demonstrate scalability and strong performance under user-level privacy guarantees.

Abstract: While federated learning (FL) and differential privacy (DP) have been extensively studied, their application to automatic speech recognition (ASR) remains largely unexplored due to the challenges in training large transformer models. Specifically, large models further exacerbate issues in FL as they are particularly susceptible to gradient heterogeneity across layers, unlike the relatively uniform gradient behavior observed in shallow models. As a result, prior works struggle to converge with standard optimization techniques, even in the absence of DP mechanisms. To the best of our knowledge, no existing work establishes a competitive, practical recipe for FL with DP in the context of ASR. To address this gap, we establish **the first benchmark for FL with DP** in end-to-end ASR.  Our approach centers on per-layer clipping and layer-wise gradient normalization: theoretical analysis reveals that these techniques together mitigate clipping bias and gradient heterogeneity across layers in deeper models. Consistent with these theoretical insights, our empirical results show that FL with DP is viable under strong privacy guarantees, provided a population of at least several million users. Specifically, we achieve user-level ($7.2$, $10^{-9}$)-DP (resp. ($4.5$, $10^{-9}$)-DP) with only a 1.3\% (resp. 4.6\%) absolute drop in word error rate when extrapolating to high (resp. low) population scales for FL with DP in ASR. Although our experiments focus on ASR, the underlying principles we uncover — particularly those concerning gradient heterogeneity and layer-wise gradient normalization — offer broader guidance for designing scalable, privacy-preserving FL algorithms for large models across domains.  Code of all experiments and benchmarks is available at https://github.com/apple/ml-pfl4asr.

Enabling Differentially Private Federated Learning for Speech Recognition: Benchmarks, Adaptive Optimizers and Gradient Clipping
Martin Pelikan", Sheikh Shams Azam", Vitaly Feldman, Jan "Honza" Silovsky,
Kunal Talwar, Christopher G. Brinton!, Tatiana Likhomanenko
Apple, Purdue University
(mpelikan, azam, witalyff.jiioovskk jsilovsky] Sappllllyyyypppppe.eppllp @apple. com cgbêpurdue edu, [ktalwar, iaaalwar,attann antares @apple. com
Abstract
While federated learning (FL) and differential privacy (DP) have been extensively studied, their application automatic speech recogmition (ASR) remains largely unexplored due to the challenges in training large transformer models. Specifically, large models further exacerbate issues in FL. as they are particularly susceptible gradient heterogeneity across layers, unlike the relatively uniform gradient behavior observed in shallow models. As a result, prior works struggle to converge with standard optimization techniques, even in the absence of DP mechanisms. To the best of our knowledge, no existing work establishes a competitive, practical recipe for FL with DP in the context of ASR. To address this gap, we establish the first benchmark for FL with DP in end-to-end ASR. Our approach centers on per-layer clipping and layer- layer-wise gradient normalization: thcoretical analysis reveals that these techniques together mitigate clipping bias and gradient heterogeneity across layers in deeper models. Consistent with these theoretical insights, empirical results show that FL with DP is viable under strong privacy guarantees, provided a population of at least several million users. Specifically, 10 10- (resp. (4.5, 10 (0-9)-DP) DDP) with only 1.3% (resp. 4.6%) absolute drop in word error rate when extrapolating high (resp. low) population scales for FL with DP in ASR. Although our experiments focus on ASR, the underlying principles we uncover particularly those concerning gradient heterogeneity and layer-wise gradient normalization - offer broader guidance for designing scalable, privacy- preserving FL algorithms for large models across domains. Code of all experiments and benchmarks is available at https: //g github. coomapppe//eee holl-pll4asz 14arr.
achieve user-level (7.2,
Introduction
Federated learning (FL.) allows training models a distributed manner without storing data centrally on server [1]. While FL eliminates privacy risks associated with data aggregation, it remains vulnerable to inference attacks [2, 3, 4, 5, 6]. Stronger user-level privacy guarantees can be achieved by combining FL with differential privacy (DP) [7, 8] and secure aggregation [9, 10], FL introduces several challenges in training including: heterogeneous data distribution [11, 12], sensitivity to cohort size [13], and slower convergence rate due to local training [14]. practical FL with DP with limited privacy budget also limits extensive apart from communication and computation cost [15, 16], thus necessitating robust training strategies.
tuning as it incurs additional privacy overhead
Equul contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Consequently, training end-to-end (E2E) automatic speech recognition (ASR) models using FL is also challenging [17, 18, 19. 20. 211, primarily due to the inherently heterogeneous data [22, 20] across clients but also exacerbated by the depth of the models [23, 24]. Additionally, training large maasforrmer-baeed models [25, 26, 27, 28] that underlie most E2E ASR models require optimization techniques such as learning rate warm-up and decay, gradient clipping, adaptive optimizers, careful initialization, etc. [29, 30, 31j. Moreover, FL alone provides limited privacy even in the context of ASR [32, 21J. This work is, to our knowledge, the first to demonstrate a practical training recipe to enable FL with DP for ASR, along with a strong benchmark and supporting convergence guarantees.
Most prior works on both FL and DP rely on small-scale models, primarily due to (i) communication complexity [33] and (ii) the difficulty in training large-scatt scale models with DP [34, 35, 36]. We argue that: (i) practical model sizes are steadily increasing - including for ASR [37] and (ii) the optimiza- tion of larger over-parametrizer models is often easier [38]². To address this gap in understanding large-scale models in the context of FL and DP and to mitigate the optimization challenges associated with training smaller models, we focus exclusively on a large vanilla transformer model for ASR in this work. Our key contributions can be summarized as follows:
(i) We empirically study the performance of FL with DP on E2E ASR using a large (250M parame- ters) vanilla mncode--based transformer model trained with the Connectionist Temporal Classifi- cation (CTC) loss [40]. Based on the results, we successfully establish the first practical and competitive benchmark and baselines for FL with DP in ASR with realistic (c, 6)DD guarantees.
(ii) We systematically analyze the impact of several key FL factors including data heterogeneity, optimization hyperparamters, and seed models initialization (pre-trained with or without domain shift) convergence and performance of ASR trained under FL and FL with DP.
(iii) We revisit per-layer clipping deemed ineffective by prior works and demonstrate that combining it with layer-wise adaptive gradient normalization the key to achieving strong model performance under FL with DP. Furthermore, we provide rigorous theoretical analysis of the algorithm's 's convergence properties, offering insights into observed empirical behavior.
We show that FL can be used to train competitive models for several datasets, covering English, Ger- man, French languages: FL models are at worst ~ 0.3%-1.4% absolute word error rate (WER) behind the corresponding central models with limited number of central steps. Competitive models obtained even with heterogeneous data, especially when the training starts from a seed model. The seed model can even come from another domain and perform relatively poorly on the target dataset. We also show that FL with user-level DP, which is more preferable to example-level DP, and large models is viable for E2E ASR and promising even for low-resource languages. With per-layer clip- ping, our models achieve (7.2, 10 DDDP (resp. (4.5, 10-9)-DP) with 1.3% (resp. 4.6%) degradation in absolute WER for extrapolations to high (resp. low) population scale for FL with DP in ASR.
2 Federated Learning with Differential Privacy: Background and Notation
Federated Learning (FL) In this paper, we focus on synchronous cross-device FL where only small fraction of users (clients) participate in each step of central (global) aggregation (step), where K is the total number of users (population): every user is sampled i.idd. with probability from all users, and S = gK, termed cohort size, is the expected number of users participating in every central step. Users do not maintain state across central steps. Each user le has its own local data De. where x E R and DD is k-client's data distribution (æ is paired audio and the corresponding ground-truth transcription for ASR task). The objective of FL is minimize the total loss function (())) given the ASR parameters 0 e R² and all user data: minlERR (0) & Ek= Lx)) where WA > O, Σ Wh- = 1, Lx (0) = En,,n [f( 0)] and {xx 0) is a loss function for a sample NR.. In practice, we optimize (0) by sampling a set of users K2 at central step t who receive copy of latest global model ((0) Each client k then performs optimization over the local copy of the global model e(2.0) = (()) using their own data ~ D₁ via the update step 0 (f.f e (t.Arwo)
Te g (t-troo) ) at step 1no.. where gR (0) = B (,, 0) (e.g. obtained by SGD) is an estimator of the (0), and B. = (x: { I; ~ Dr. The clients periodically upload their model updates
2 Distillation from large small model remains the dominant method for training compact models [39].

---

Algorithm 1: Federated learning with differential privacy (marked as red)
Inputs:
Initial model 6 (either randomil initialized or pre-trained on server data), weights wa (0, that 2- we central steps T, central optimizer opt, clients sampling rale = S/ K, local steps Tice local optimizer optice clipping function clip(t,
max (.11)
local clipping Croc. clipping bound DP ppp.
Result: ASR model 6
Initialize central optimizer opt
for 1,2, ...,, do
Sample every client i..d with prububility form subset X of clients from alll clients X (|x1 K) // For practical ixpla ixplarentation ue fix the wize of the nohort . to S throughout the training. for = 1, 1,2, 1, R E X in parallel do Initialize local model { () 4- g(t and local oprimizer optioe
tiee. 1, .,.,, "al do // Ke also uae local spochas inscand of and compute gradient estimate 8k, Sample train mini-batch Se (lol D Dxxt,
Chan this loop has diffurent musber of stapa per cliant.
(f/ooc) " ( (ax("1oc) o o((//1oe" "1))
(t.11oc) +- clip(BE, (f,f CCoo)) and update a local model o (f.1 +- opt (l.))))) Clip gradients Si,
10
11
Compute client's delia A ((.00)
b2
(e.Tlo 0(e-1)
o (lac)
12
Clip client's is delta A (1) clip(
A
(03
C)
13
Add Gaussian noise to client's delta A
+N 0, 1 oe xE
14
Compute central model's pscudo-gradient g Update the central model g( + opt(g!))
(0)
Eia wk: (())
15
(6)
to the server after Tioe local steps given by (1) 0(.00) (ftTTion) Thoe G where
#
G = ->= Too
-1 (t.5 The server then aggregates the updates A (9) E/ Wh. k; and
(t)
00 85
performs the central model step either through conventional federated averaging [41]. or through an adaptive optimizer [42]. The updated central model is broadcasted to another sampled set of users and the process is repeated either for a fixed number of central steps T or until convergence.
FL with Differential Privacy (DP) Since no prior work exists that can efficiently train private FL for ASR, we establish the first competitive baselines for private FL in ASR in the rest of the paper. We start by referring to DP [43, 44, 7], which provides a mathematical formalism of guarantees on the amount of information leart by machine learning models from the user private data:
Definition Differential privacy: A randomized mechanism M D R with domain D (e.g.. possible training datasets) and range R (e.g., all possible trained models) satisfies (s, o)-difeerential privacy for any two adjacent datasets D, D' eD and for any subset of outputs R. C holds that Pr(M(DD E R| < e Pr(M((D)) E R] +
One key DP component is adjacent datasets [7]. In some applications, prior works consider the example-level privacy [45, 8]. For FL where each user has multiple data points, user-level [46] is preferable to example-leve privacy [45, 8]. We thus use the following adjacency relation:
Definition 2. User-adjacent datasets: Let D and D' be two datasets of training examples, where each example is associated with a user. Then, D and D' are adjacent if D' can be formed by adding or removing all of the examples associated with single user from D.
To incorporate user-level DP into FL, the client updates A (t) are: (i) clipped such that their norm is bounded, i.e., || mechanism, such that client updates under FL with DP are given by A k +N (o, IC "5
A k la < C' at every central training step t and then (ii) perturbed via Gaussian
(9)
((t) and a max 7. BOc I We use the moments accountant [8] to
where = ?h.. a
K
achieve tight privacy bounds and restate the main theorem of [46] in our parametrization of noise added to every user's 's model update before averaging, assuming Wk 1/K for simplicity:
Theorem For the DP-mechanism in Algorithm the moments accountant of the sampled Gaussian mechanism correctly computes privacy loss with the noise scale of = ope / and central steps T, where & = (/(g)) and noise opp probability of user selection q, and total number of users the population K are given in Algorithm I.
3

---

Although this work uses the moments accountant and uniform sampling, alternative approaches such as DP-FTRL [47] or device-level sampling [10] can also be applied. These alternatives are expected to yield similar results, potentially at the cost of small constant overhead in the required population sizes. Since we use large transformer ASR models, user-level DP significantly reduces the utility of training ASR models even in the absence of FL. because the noise overpowers the gradients [48, 34]. Our initial experiments confirmed this problem, which we mitigate via per-layer clipping. The FL with DP and corresponding terminology summarized in Algorithm 1.
3 Theoretical Analysis: Adaptive Optimizers and Per-Layer Clipping
LAMB Optimizer. We utilize the layer-wise adaptive optimizer LAMB [49] for updating the global model using pseudo-gradient A (f) (see Appendix E.4 for its definition). Originally proposed for the large batch training, LAMB scales learning rate for each layer using the ratio of weight norms the gradient norms (termed trust ratio), which makes it particularly effective in handling the gradient scale disparities in deep networks. We posit LAMB helpful in large model training using FL since inter-layer gradient heterogeneity is further exacerbated by "divergence accumulation" [23, 24] wherein deeper layers demonstrate higher divergences in contrast the shallow.
Per-Layer Clipping, Per-layer clipping was proposed by [46]. However, the authors did not report a significant improvement in their setting of L.STM models for language. On the contrary, our work shows that for FL. with DP and large transformer models, per-layer clipping mitigates the imbalance of gradients across different layers in the attention blocks. Formally, we change the global clipping of clients' deltas from Algorithm Step 12, to per-layer clipping cliptager (g. C) defined follows:
Definition 3. Per-layer clipping: Let the model gradient be g = 1 82- .... gu), where gn, is the h-th layer gradient with total H layers in the model. Then per-layer clipping with clipping parameter V=
C C= V Em Cj is given as cliptager (g, C*) 82 ..., gH) where gn. = clip(g Cr).
In our experiments we use either C₁ C ("uniform" variant) or C₁ CV Σ d; ("dim" variant vH
de
based on a layer dimension) where d₁ the dimension of the h-th layer and 1, 2, ...... H. so that after per-layer clipping we still guarantee A
(1) 112 S C necessary for Theorem 1 to hold.
Assumptions. Given global model comprising of H layers, the model parameters defined as (o..... ...
is presumed that the loss function for each sample æ is bounded below: m f(x, 0) oc, where DR. V k. Let ||I || denote the 12 no-mom.. Our analysis uses the following standard assumptions [12, 42, 50, 51, 52, 53, 54]:
1. Smoothness of Loss Function Gradient: V((x, 0) is layer-wise L, smooth for vh [49]:
| (<, vz((x,00) L, ||0 0'll, V0,6' RR.. R.W Vk.
(A1)
where V, denotes gradient with respect to parameters 0₁ of layer h.
2. Local Gradient Property: Given user k, By {a:) 2; Dx and local gradient Vi(x, 0), its unbiased estimator g (0) = ga [B,. 0) has bounded variance Vk [12, 50, 54]:
g ( V 2 (0)
S o of 20, VOERD.
(A2)
3. Global Gradient Property: The variance of global (pseudo-) gradient is bounded [52, 42]:
K
[V (0)1 < o² klob? o Abob 2 0, VOERD.
(A.3)
Corollary 1. Assume A1.1, A2.1, A2.2, and A3, Naw L < and k = 11 8(1 n Twn > 0. If the trust ratio in LAMB optimizer is controlled in the Algorithm 1 (global optimizer is LAMB and local optimizer is SGD) and Noaos = LVT
and The θ (IVTTOT) then Algorithm converges

---

to stationary point of the global loss fincttin with the convergence bound characterized as:
K
E lac
(0tt)
(0(
S
1 VT
+
Tiono"
glab
+0
Tioc
T
T
T
t=
optimization
global update noise
local update noise
+
+0
C²
R dn
+
Tho T
H
M
+
Two
T
H
R
C₁
M
Un
h
inter
differential privacy noise
clipping bias
intra- and inter-client update variance
(1)
where Vn iinlen = E,,
Var, "toe (/GG))))
and V iitee Var,
= =E,
E "Loe
G
Refer Appendix E for the proof of Theorem and its Corollary with derived asymptotic bound.
Interpreting the Bounds. Corollary highlights the key contributors to the convergence behavior: (i) the optimization process, (ii) global and local update noises, (iii) DP noise, and (iv) clipping. Specifically, it emphasizes the complex coupling among per-layer clipping (C₂). layer-wise scaling Ra. and intra-client (V, ( and inter-client (V,, ( update variance. Although the analysis presents several of these terms separately. they are often interdependent and may interact in non-trivial ways. The remainder of this section summarizes the key takeaways from the bound in Corollary 1.
Recovering Prior Bounds As a validation, we recover bounds similar to several prior works. For example, by setting of 0 and letting C₁ - 0o, obtain bound similar to adaptive optimizers [42] and vanilla FL [12, 16] modulo constant factors. Similarly, the bound for [55] can be recovered by choosing a constant clipping value C' for all layers and adding an appropriate DP noise. These reductions demonstrate that our result generalizes several known convergence guarantees as special cases. See Appendix E.7 for details on how specific prior bounds are recovered.
Impact of Gradient Heterogeneity across Batches and Clients. The terms V fatra and V in the convergence bound quantify the impact of hett-heterogeneity within and across clients, respectively. Within-client heterogeneity V jutte can be reduced by shuffling data locally on each client. However, this becomes challenging when client data is limited. In such cases, data augmentation as a practical alternative, reducing batch-level variance and improving performance [56]. Similarly, inter-client heterogeneity V can be tackled by incorporating (i) server-side adaptive optimizers that intrinsically reduce gradient heterogeneity across clients [56], (ii) anchored optimization methods such SCAFFOLD [57], FedProx [11], and (iii) adaptive client weighting [12].
serve
Trade-offs Between Clipping Constant and DP noise. While an inverse relationship with the clip- ping C₁ suggests that increasing Cn, would improve convergence [58], the proportional relationship o2 x C2 complicates the dynamics; while increasing C₁ reduces clipping bias, it also requires proportionally more DP noise for the same privacy guarantees. Additionally, the convergence bound indicates linear decay of clipping bias with T. whereas DP noise increases linearly with it. Thus, over long training horizons, the impact of clipping becomes negligible relative to that of DP noise. However, in practical settings with limited central steps T, clipping bias can remain significant
particularly when gradient norm MA, and intra-client ( and inter-client (V,,, inter update variances are large. Unlike [55], we capture this coupling explicitly that underscores the importance of tuning both C₁ and DP noise jointly to optimize privacy-utility trade-off. Consistent with our theoretical bound, Table shows negligible impact of clipping on centralized model training whereas DP noise significantly degrades performance in FL with DP. While local clipping. used for transformer training stability [59], reduces model's sensitivity to global clipping, the model is still affected by DP noise.
Benefits of Per-Layer Intervention. Our convergence bound is decomposed over several layer-wise dynamics including gradient norm Mn. trust ratio R,, clipping constant Cr, and variance terms 1 intra and V
iuser This per-layer decomposition gives a tighter bound when: (i) heterogeneous gradient distribution is observed across layers and transformer blocks as seen in Figure 5 and Figures 17- 19 and (ii) "divergence accumulation" in deep networks in FL. training [60] further amplifies the mismatch across layers. Based these observations, only redistribute the total clipping budget
5

---

C' across the model via per-layer clipping C₁ given by C, = C or C₁ Cdd d.). thus ensuring that overall DP noise remains unchanged. Consequently, the redistribution of clipping budget can be viewed as altering the signal to noise ratio (SNR) at the layer level relative to DP noise. tandem, the per-layer trust ratio RA further modulates both noise scale and clipping bias. Empirically, under similar settings LAMB extracts better performance in FL. with DP when compared to Adam. Advantages of LAMB was also reported by [56] showing that it improves FL when used as a local optimizer. We instead use SGD locally owing to the memory overhead of LAMB that can be prohibitive on resource-constrained devices. Together, these layer-wise treatments should empirically result in improved convergence compared to global clipping for cases with greater gradient heterogeneity or stronger DP noise. This is in fact evident from the following observations:
(i) Per-layer clipping has a more significant impact FL with DP compared to centralized training. This improvement is more pronounced for higher DP noise levels (see Tables & 18).
(ii) Experiments on CV-en show both a higher improvement compared to CV-fr & CV-de (see Table vs. Table 18) and higher gradient diversity across layers (see Fig. 17 vs Fig. 18 & 20).
4 Empirical Analysis
Data We use LibriSpeech (LS) data [61]] train-clean-106 (LS-100), train-clean-360 (LS-360) and train-other-500 (LS-500) as training data. LS-960 is the union of LS- 100, LS-360 and LS-500. LS-860 the union of LS-360 and LS-500. We use standard validation (dev-clean and dev-other) and test (test-clean and test-other) sets. We also use Common Voice (CV), v13.0 (English, German and French) data [62]: the train, validation and test sets are provided in the dataset. In addition, we split the training data using a specific percentage of users to train a seed model only and the rest of users for FL training: e.g., we create CV-en-traiin-00-- 10(-5) by selecting all the data for randomly chosen 10% (5%) of the users from CV-en-train and we denote the remaining data by CV-en-train-99 90(5).. Statisties on speakers are given in Figure I: it shows that CV data are much more heterogeneous than LS as highlighted by [20]. CV data thus enable a more realistic scenario for testing FL and FL with DP. The most realistic scenario for FL uses a small central dataset to train a seed model (e.g. LS-100), and larger dataset from a different distribution for FL (e.g. CV-en-train).
Central Training We use standard feature extraction for audio [25, 27] by comput- ing log-mel filterbanks with 80 coefficients with a 25ms sliding window and 10ms stride length, later normalized to zero mean and unit variance for each input sequence. We employ vanilla encoder-based transformer model trained with the CTC loss [40]. We start our experimentation with the state-of-the-art model on LS-100 from [63] with 255M parameters. We use SpecAug- ment [64] and clip all gradients during training to have a norm of most 1 (see Appendix FF and G.6 for discussion). We found difficult (see Appendix G.3) to switch to FL from central training when post-LayerNorm was used (similar issues were reported by [31]). Following [31] we thus do central training with Layerryrrorm (also used in FL), LARS [65], and relatively high (0.5) learning rate (LR) without any warmup and with step- wise decay to simplify the recipe and have stable training while maintaining the performance.
10'
CV
CV-S
CV-dc
1.5-70
10'
LS-500
Federated Training We simulate FL by considering every speaker and its data as a separate user. In most experiments, SGD [66] with constant LR used as the local optimizer and LAMB [65] is used as the central optimizer. We found this com- bination most robust (see Appendix G.4). The central LR is constant with further exponential decay unless noted otherwise, gradient clipping is set to for each client. Unless noted other- wise, restrict the number of central steps 2k. Although most simulations would further improve after 2k steps, the per-step latency and DP noise addition typically
10'
sunples
Figure 1: Train distribution in LS and CV: per speaker #mminutss (top) and #samples (bottom).
Datasets such as LibriSpeech, Common Voice. VCTK, TED-LIUM offer speaker metadata necessary for creating heterogeneous FL clients. Datasets like People's Speech, GigaSpeech, SPG1Speech lack speaker metadata. We choose LibriSpeech and Common as they offer the greatest speaker diversity.

---

Seed: None, Train: LS-960
Seed: LS-100, Train: LS-860
Sced: CV-en, Train: LS-960
WER (16). test clean
10"
WER (5)), oette
10a.
10'
10
300d
L=3
L=
32
61
central
s00d
L=3
16
L=
32
L=
61
ventral
Figure 2: Impact of the cohort size 5 and seed models FL. models trained on LSS. We use exponential decay for central LR starting at = 1, 000, decay rate 0.6, and transition steps 500 (w/o seed model) or 250 (w/ seed model) with T' 2k total central steps and 10 local epochs. Local (central) LR 0.4 (0.006) (w/o seed model) or 0.2 (0.003) (w/ seed model). See details in Appendix G.2 Table 3.
Nown
New CV
WER eVacet
Scet: (hims TrEe Clirds
Sest Nime, Thac Dtdde
6x
bosk Chieiih Traa CV
Reed CVS 18 Tras CV Seed Tusk Ch-dul
Sost IPA
4x
x
10
00122
ccntral
Figure 3: Impact of the cohort size S and seed models on FL models trained on CV: English (left) and French/German (right). We use exponential decay for central LR starting at = 1, 000 (w/o seed model) or 750 (w/ seed model), decay rate 0.6, and transition steps 500 (w/o seed model) or 750 (w/ seed model) with T = 2k total central steps and 10 local epochs. Local (central) LR is 0.4 (0.006) (w/o seed model) or 0.2 (0.002) (w/ seed model). See details in Appendix G.2, Tables 4 and 10.
Lasielzzn
CVatteeal
New
10
New
LS-19L Trae 15-500
LS-U0) Thare
WER (10
UV-os, Thanc LS-ee)
4 10
3 100
.020
16
1005
1b
ccntral
6001
2. 16
32
Figure 4: Impact of randomizing the distribution of data across users for LS (left, middle) and CV (right) measured by WER. Parameter settings are described in Figure for LS and Figure 3 for CV. While the original training data are non-III (solid), III (dashed) versions of 1.89660, LS-860 and CV-en-train are created by choosing user id uniformly and randomly from the set of user ids for each data point in the corresponding dataset. Detailed numbers are in Appendix G.2, Tables 5 and 6.
limit the number of iterations in practical private FL systems to this range [67, 56). To keep simple and robust training recipes, we do not do extensive configuration on one training setup we apply the same hyper-parameters the rest of experiments.
search. After finding the best
4.1 Impact of Seed Models and Cohort Size
In Figures 2 and 3 we show that initializing FL with seed models instead of randomly significantly decrease word error rate (WER) for both LS and CV (all languages), even with domain shift for the seed model training (e.g, using LS seed model for CV and vice-versa). Using seed model initialization for FL, we can almost close the gap between central and FL trainings within 2k central steps and moderate cohort sizes: 2 64 (2 128) for LS (CV). Larger cohorts consistently improve the outcomes within 2k central steps increasing the cohort size directly increases the amount of seen data. Even without seed models, FL is competitive with central models given large enough cohort size.
Increasing the amount of data for seed model training improves the trained FL models regardless of whether the data come from the same domain or not (e.g. compare CV-en-train-0S seed vs. CV-en-train-10 seed or LS-100 seed vs. LS-960 seed on CV-en-train Figure 3). In fact, the use of seed models trained on considerably more data from another domain can outperform the use of seed models trained on less data from the same domain: the results on CV-en-train with a IS-960 seed model are better than the results with a CV-en-train-D 10 seed model on CV-en-train- 90 (see
7

---

Table 1: Results for FL with DP and model pre-trained on LS-100 (~100h) used as central data and afterwards fine-tuned with FL on CV-en-train (~1.66 hours) used as clients data. We report added noise N(0, 2 per client (wk k)) and CV dev and test WERs (%) for two clipping variants with clipping C: global and per-layer "uniform" ("dim"). The total number of users is K,
the cohort size is S = qk, and the number of central steps is T. We set ò 10-9 following [46] and report e for which (e, 6DD holds for given S and K using the moments accountant of [8]. For practically intractable to run model training (marked *-"), we extrapolate (s, 6 following [46] and, assuming the training dynamic remains unchanged. thus similar WER could be obtained. Central training gives 14. 14.7%/17.35 17.8% WER on dev/test. Extended results are given in Appendix H and in Table 17. & should be below 10 to be practically useful (marked with blue).
scaling S and K where
C
5
K
== S/K
r
Renyi
global clipping
per-Sayer clipping unifurm (dim)
6-10">
order
dev
dav WER
34,753
54.7
61.2
61.2
0.03072
30.0
0.01
1,024
34,753
0.0295
2,006
1.1-10
1.1
(242))
22.3
0,3072
30.0
0.01
10,340
347,530
0.0295
2,006
3.7-10"
1,536
30.0
0.01
51,200
1,337,500
0.0295
2,006
6.5 10
7.0
0:02048
20.0
0.01
1,024
34,753
0.0295
2,006
2.6-10
1.1
23.7 (22.6)
(266)
LU124
0411
$1,200
1,737,650
0.0295
1.3-10
4.0
2,048
20.0
0.01
102,400
3,475,300
0.0295
2,006
4.5-10"
9.0
0.01024
16.0
0.01
1,021
34,753
0.0295
2,006
1.1-10"
1.1
30.7
35.2
21.3 (20.1)
(2307
0.512
10.0
0.01
51,200
1,737.650
0.0295
2,006
7:2-10*
1.5
0.01
3,475,300
0.0295
2,006
1.3-10"
4.0
2,088
10.0
0.01
201,800
6,950,600
0.0295
2,006
4.5-10P
9.0
0,003072
3,0
0.01
1.024
34,753
0.0295
2,006
122110
1.1
27.0
31.1
17,9 (17.)
21.2 (20.4)
0.3072
0.01
102,400
3,475,300
0.0295
2,006
3:7-10"
1.1
0.6144
0.01
6,950,600
0.0295
2,006
2.0
0.6144
3.0
0.01
204,300
69 00,5000000
0,00299
2,034
7,2-10
3.0
0.6144
3.0
0.01
204,300
090,0000000
0.000295
3,390
327-10"
6.0
0,001024
0.01
1,024
34,753
0.0295
2,006
11-100
1.1
22.9
26.7
(662(16.))
(1.5
0.2048
1.0
0.01
204, 800
6,950,000
0.0295
2,006
1.1-10
0,2048
1.0
0.01
204,800
69,506,000
0.00295
2,034
2.7-10
1.1
0,204K
1.0
0.01
204,800
695,0500,000
0,000295
3,390
94-10'
1,3
0.01
1,024
34,753
0.0295
2,000
inf
15.7
18.9
19.1
L.24
34,753
0.0295
2,000
18.9
18.9
more ablations in Appendix G.9, Table 15). The gap between FL models with different seed models decreases as the cohort size increases the latter directly increases seen data in FL. training.
To demonstrate robustness of found tyyee-parameters and observed results in Figure 3 (left), applied the exact same training configuration to train FL models on CV French and German data. We confirm in Figure 3 (right) that the training configuration found on English data is robust: similar trends and results hold for French and German.
4.2 Impact of Data Heterogeneity
Prior works argued that data heterogeneity poses a challenge for FL [11, 12]. Figure 4 shows that distributing data uniformly and randomly across users indeed improves performance for all settings. Since for LS, every client's 's data are of similar duration and we use dynamic batching, this is unlikely to be due to the differences in the amount of data between clients. The impact of using i.i.d. data decreases with increasing cohort size. Figure 4 suggests that algorithms such as FedProx [11], ProxSkip [68], and SCAFFOLD [57] could further improve FL. performance. We evaluated FedProx, which marginally improved FI. performance in some cases (see Appendix G.7, Table 13).
4.3 Federated Learning with Differential Privacy
For FL with DP consider a setting close to the real-world scenario: LS-100 is used as central data to train a seed model (without DP); CV-en-train is considered as clients' data on which the seed model is trained afterwards using FL. In this setting (i) the clients' data are 16 times bigger than the server data and (ii) there is a domain shift in clients' data.
As discussed in Section 2, DP is challenging for larger models due to their size. To make the model training more resistant to noise, we need to increase the cohort size, e.g. in recent work [69] used 1 150k cohort size for FL with DP. We take exactly the same setup as in Figure 3 with the data CV-en- and the seed model trained on 1.5- 100. First we scale the FL training to the cohort size of 1024; to mitigate the resulting increase in the computational cost of the training, we switch from
8

---

0008
Klaape
velnowight
volk
bcladd
x2
V.oiigtt
0109
5.005
01090
0104
111
0.222
47
2.005
30
10 Tranderme
Trandorncr Maakk Tader
Trans Soreeterett Blick Indos
Thealorra Blok hore
Figure 5: Client delta norms computed per layer in the model. We average statistics across all clients and central steps, and plot the mean and standard deviation. The model is trained with ODP -3-10- and global clients' deltas clipping C = 1002 (Algorithm 1). Transformer block consists of attention parameters (wqkv and wf) with LayerNorm (In1), and MLP (wl and w2) with LayerNorm (In2).
10 local epochs to 10 local steps (see Appendix H.2, all other peper-ppraamters stay the same). As discuss in Appendix H.2, expect that more local compute that would be feasible in real deployment, should lead to better results than what we get in our experiments. Increasing the cohort size further closes the gap with the central baseline. Second, we use and vary the clipping C' applied to clients' deltas without adding DP noise yet. Although the average norm of clients' deltas is 0.7 (see Appendix H, Figure 7), they can be clipped with C' low as C 10-8 without any impact on model's 's quality. This is consistent with Corollary 1: the interaction of trust ratio Rg with C₁ re-normalizes the gradients. Further we C 10 prevent numerical precision errors. Finally, we add different levels of noise DP to every client's 's delta before averaging the deltas across clients.
In Table 1, we estimate (c, 6D by the moments accountant of [8] for every level of noise, number of clients K, clients sampling q, clients' deltas clipping C, and number of central training steps T, where W/- =
Using FL with DP, can improve over the poor performing LS-100 seed model due to limited server data and their domain shift: WER is reduced from 61.2% to 31.1% 1.1% with ODP 3 10-6 and (7.2, 10 10-11-0-000 DDDP assuming the training effectiveness (WER) remains the same if, following [46], we extrapolate to 70M clients with the cohort size of 2000 Lowering the DP noise D decreases model's WER, but DP guarantees become impractical even if we scale K and S.
In Figure 5, we analyse the clients' deltas by computing model's per-layer deltas norm. We highlight that the norms are imbalanced across different transformer layers and also across different types of parameters: (i) first transformer layers have a larger deltas norm magnitude; and (ii) delta norms for attention parameters
order of magnitude lower than those for LayerNorms. This observed imbalance motivates the application of per-layer intervention, as formally discussed in Section 3.
To avoid De dominating the atlention layers and slowing down the convergence, following Theorem 2, we apply per-layer clipping (Definition 3) which significantly improves model convergence (see Figure 12 in Appendix): with the same DF 3 10-6 we are able to closely match the model trained without DP noise (o = 0) with only a small WER degradation (from 19.1% to 21.2% WER) while guaranteeing (7.2, 10 -DP assuming the training effecctiveness remains the same if, following [46], we extrapolate to 70M clients with the cohort size of 200k. Morcover, we can now increase DP noise up to DF 10-5 getting 23.7% WER with (4.5, 10 -D- by following [46] and extrapolating only to ~7M clients with the cohort size of ~200 (see Table 1). The latter is a realistic scenario even for mid/low resource languages. We can further reduce WER by ~ 11% for the same (z, 6DP guarantee if we apply per-layer clipping based on the layer dimension (see Table l).
5 Related Works
FL for ASR was first studied by [70] using attention based Seq2Seq LSTM models. The paper showed that FL in ASR suffers from data heterogeneity, a known problem in FL [71, 33]. They proposed gradient weighting to speed up convergence and improve performance. Building on this, [22] used hybrid LSTM models and introduced client adaptive normalization to mitigate data heterogeneityy Similarly, [17] used RNN from [72] and added noise to local gradients to address data heterogeneity. However, these Fl.-trained ASR models significantly underperformed their centralized counterparts.
End-to-End ASR models in FL [19] used a 120M parameters conformer [27] model together with federated dropout to train only subset of parameters each client. This reduced com-
'169, 67] showed it is realistic to (i) have millions of users to participate in FL and (ii) use large cohort size of 150k in FL deployments.

---

munication and improved FL performance relative to central training. However, the setup used 10k-100k central steps and homogeneous data distribution, which impractical in real- weal-world scenar- ios. [20] used Seq2Seq model with CNN encoder and RNN decoder trained with joint CTC-attention objective. They noted that training E2F ASR model from scratch in a realistic FL. setup is "nearly impossible", ", and proposed an additional training step on held-out server data, after model aggregation. They also emphasized switching from LS data to CV due to its more realistic data distribution. Re- cently, [73] trained a ~ ~130M parameter model using weighted client aggregation and word frequency histograms, initialized from a centrally pretrained model. [56] showed FL training with similarly sized conformer models using adaptive optimizers from scratch. We borrow several real-workd settings from prior works: (i) limiting to 2k central steps [56], (ii) training large transformer models from scratch [56], and (iii) using both CV [62] and LS [61] datasets for experiments [19, 56] to evaluate robustness across datasets and languages. Unlike prior work, we also study: (i) FL. with DP for ASR and (ii) impact of domain mismatch between the data used for central pretraining and FI..
Data Leakage in FL for ASR. [21] improves ASR performance using large (~300M parameters) pre-trained self-supervised model (transformer) to initialize FL and observe speaker information leakage via model updates. Audio can further leak sensitive attributes such as gender and health conditions [74]. Given that FL alone does not guarantee user privacy [2, 4] and several recent works [32. 21] have explored privacy attacks targeting FL in ASR, it is very important to enable FL training with DP. To this end, our work addresses this critical gap by enabling FL with DP for ASR.
Adaptive Clipping and Convergence Bounds Adaptive clipping was first proposed in [8], but the authors reported no observable impact on convergence. Recently, [75] proposed adaptive clipping using privately estimated quartile statistics, incurring a negligible privacy budget. They noted a dependence on non-private data and fixed learning rate (LR), which can be prohibitive in practice. [76] later provided a comprehensive convergence analysis in a central setup, showing that LR depends on the clipping constant. [55] is one of the few works providing convergence bound under clipping using FedAvg [1]. However, it cannot be trivially extended to per-layer clipping or adaptive optimizers. Additionally, [77] is contemporary work that proposes adaptive layer-wise clipping for DP-SGD by distributing clipping budget over the layers proportional to the layer-wise gradient statistics gathered on public dataset. While this method
uncover more fine-graine one-grained gradient distribution over layers, it introduces a reliance on representative public dataset. In contrast, our work adopts a different perspective: rather than conditioning clipping on public dataset, redistribute the clipping budget structurally (uniform or dimension-aware) and rely on the LAMB optimizer to dynamically regulate inter-layer heterogeneity. Thus, while [77] depends static, public-data informed sensitivity distribution, our analysis and experiments highlight the importance of dynamic, optimizer-driven adaptivity. To the best of our knowledge, we present the first explicit convergence bound for FL with DP that incorporates per-layer clipping, LAMB optimizer. and DP noise- highlighting the interdependence among trust ratio in LAMB, per-layer clipping constant and DP noise in FL.
Divergence Accumulation Recently, [23, 24] showed that deeper models in FL suffer from "diver- gence accumulation" accumulation of dissimilarities among client models during back pack-propagation.
Conclusion
ASR provides a valuable and realistic benchmark for (private) federated learning (FL), offering large datasets that are naturally partitioned by speakers and exhibit heterogeneity typical in real-world settings. With the exception of language modeling, benchmarks commonly used in works studying FL with DP lack these characteristics, limiting their practicality. In this work, we focused real- world constraints such as the task of adapting a model trained centrally on LibriSpeech to Common Voice data via FL, a benchmark for both FL and FL with DP that captures core FL challenges: domain shift, user-level heterogeneity, and privacy constraints at scale. We demonstrate that with a practical number of central aggregations, is possible train large transformer models that perform competitively in the federated settings both from scratch or when starting from an seed model. We highlight that enabling FL with DP for ASR is non-trivial and requires solutions that manage the interaction between privacy, clipping. and model size. To this end, we revived per-layer clipping and used layer-wise adaptive optimization, thus achieving user-level (7.2, 10-9>-DP (resp. (4.5, 10 1D-D)) with only 1.3% (resp. 4.6%) absolute drop in the WER, when extrapolating to high (resp. low) population scale. These results establish a practical and scalable foundation for privacy-preserving FL training with DP for large models beyond ASR.
10

---

Acknowledgments
We thank Samy Bengio, David Grangier, Filip Granqvist, Navdeep Jaitly and Vojta Jina for essential general discussion on the paper throughout all stages; Pierre Ablin and Dan Busbridge for discussion on scaling laws; Audra McMillan and Congzheng Song for discussion on differential privacy; Shuangfei Zhai for discussion on transformer stability and behavior of gradient norms; Ronan Collobert, Navdeep Jaitly, Audra McMillan and Barry Theobald for the helpful feedback on the initial drafts of the work; Dan Busbridge for detailed feedback and helpful suggestion to improve the paper; Satyen Kale for checking asymptotic of theoretical bounds and helpful feedback on prior theoretical work; Hassan Babaie, Cindy Liu, Rajat Phull, and the wider Apple infrastructure team for assistance with developing scalable, fault tolerant code. Names are in alphabetical order by last name within the group.
References
[1] Jakub Konečný, Brendan McMahan, and Daniel Ramage. Federated optimization: Distributed optimization beyond the datacenter. CoRR, abs//511..000 1.03575, 2015. 1, 10, 35
[2] Franziska Boenisch, Adam Dziedzic, Roei Schuster, Ali Shahin Shamsabadi, Ilia Shumailov, and Nicolas Papernot. When the curious abandon honesty: Federated learning not private. In 2023 IEEE 8th European Symposium on Security and Privacy (EuroS&P) P), pages 175-199. IEEE, 2023. 1, 10
[3] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations, 2023. 1, 21
[4] Sanjay Kariyappa, Chuan Guo. Kiwan Maeng. Wenjie Xiong, G. Edward Suh, Moinuddin K Qureshi, and Hsien-Hsin S. Lee. Cocktail party attack: Breaking aggregation-basee privacy in federated learning using independent component analysis. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages
PMLR, 23-29 Jul 2023. 1,
10
[5] Martin Bertran, Natalia Martinez, Afroditi Papadaki, Qiang Qiu, Miguel Rodrigues, Galen Reeves, and Guillermo Sapiro. Adversarially learned representations for information obfusca- tion and inference. In International Conference on Machine Learning, pages 614-623. PMLR,
2019. 1
[6] Sheikh Shams Azam, Taejin Kim, Seyyedali Hosseinalipour, Carlee Joe-Wong, Saurabh Bagchi, and Christopher Brinton. Can we Generalize and Distribute Private Representation Learning? In International Conference on Artificial Intelligence and Statistics, pages 11320- 11340. PMLR, 2022. 1
[7] Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3 4):211-407, 2014. 1,
[8] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, llya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. Proceedings of the 2016 ACM SIGSAC conference computer and communications security, pages 308-318, 2016. 1, 3, 8, 9, 10, 34, 37, 50, 56, 57, 58
[9] Kallista A. Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H. Brendan McMa- han, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for federated learning on user-held data. CoRR, abs/161 04482, 2016.
[10] Kunal Talwar, Shan Wang, Audra McMillan, Vojta Jina, Vitaly Feldman, Bailey Basile, Áine Cahill, Yi Sheng Chan, Mike Chatzidakis, Junye Chen, Oliver Chick, Mona Chitnis, Suman Ganta, Yusuf Goren, Filip Granqvist, Kristine Guo, Frederic Jacobs, Omid Javidbakht, Albert Liu, Richard Low, Dan Mascenik, Steve Myers, David Park, Wonhee Park, Gianni Parsa,
11

---

Tommy Pauly, Christian Priebe, Rehan Rishi, Guy Rothblum, Michael Scaria, Linmao Song. Congzheng Song, Karl Tarbe, Sebastian Vogt, Luke Winstrom, and Shundong Zhou. Samplable anonymous aggregation for private federated data analysis. CoRR, abs/2307. 15017, 2023. 1, 4
[11] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In I. Dhillon, D. Papailiopoulos. and V. Sze, editors, Proceedings of Machine Learning and Systems, volume 2, pages 429-450, 2020. 1, 8, 35, 47, 48
[12] Jianyu Wang. Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33:7611-7622 2020. 1, 4, 5, 23, 35, 36, 38, 48, 50
[13] Zachary Charles, Zachary Garrett, Zhouyuan Huo, Sergei Shmulyian, and Virginia Smith.
On large-cohort training for federated learning. Advances in neural information processing systems, 34:20461-20075, 2021. 48
[14] Grigory Malinovsky, Kai Yi, and Peter Richtárik. Variance reduced ProxSkip: Algorithm, theory and application to federated learning. Advances in Neural Information Processing Systems, 35: 35:15176-15189 2022.
[15] Hongyi Wang, Scott Sievert, Shengchao Liu, Zachary Charles, Dimitris Papailiopoulos, and Stephen Wright. ATOMO: Communication--ffiiiee Learning via Atomic Sparsification. Advances in neural information processing systems, 31, 2018.
[16] Sheikh Shams Azam, Seyyedali Hosseinalipour, Qiang Qiu, and Christopher Brinton. Re- cycling Model Updates in Federated Learning: Are Gradient Subspaces Low-rank? In International Conference on Learning Representations, 2021. 1, 5, 35, 36, 38
[17] Dhruy Guliani, Françoise Beaufays, and Giovanni Motta. Training speech recognition models with federated learning: A quality/cost framework. In IEEE International Conference on Acousties, Speech and Signal Processing, ICASSP 2021, Toronto, ON, Canada, June 6-11, 2021, pages 3080-3084. IEEE, 2021. 2,9
[18] Wentao Yu, Jan Freiwald, Soeren Tewes, Fabien Huennemeyer, and Dorothea Kolossa. Fed- erated learning in ASR: Not easy you think. In Speech Communication; 14th ITG Conference, pages 1-5, 2021. 2
[19] Dhruv Guliani, Lillian Zhou, Changwan Ryu, Tien-Ju Yang, Harry Zhang, Yonghui Xiao, Françoise Beaufays, and Giovanni Motta. Enabling on-device training of speech recognition models with federated dropout. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2022, Virtual and Singapore, 23-27 May 2022, pages 8757-8761. IEEE, 2022. 2, 2.9,1 10
[20] Yan Gao, Titouan Parcollet, Salah Zaiem, Javier Femaadez--Maquuee Pedro PB de Gusmao, Daniel Beutel, and Nicholas D Lane. End-to-end Speech Recognition from Federated Acoustic Models. In ICASSP, 2022. 2, 6, 10, 22, 23, 41, 46
[21] Tuan Nguyen, Salima Mdhaffar, Natalia Tomashenko, -ean-FFraçços Bonastre, and Yannick Estève. Federated learning for ASR based wav2vec 2.0. In ICASSP 2023 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5,
2023. 2, 10
[22] Xiaodong Cui, Songtao Lu, and Brian Kingsbury. Federated acoustic modeling for automatic speech recognition. ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6748-6752. IEEE, 2021. 2,
[23] Yun-Hin Chan, Rui Zhou, Running Zhao, Zhihan JIANG, and Edith C. H. Ngai. Internal cross-layer gradients for extending homogeneity to heterogeneity in federated learning. In International Conference Learning Representations (ICLR), 2024. 2, 4, 10
[24] Haolin Wang, Xuefeng Liu, Jianwei Niu, Shaojie Tang, and Jiaxing Shen. Unlocking the Potential of Federated Learning for Deeper Models, arXiv preprint arXiv:2306.0270/ 2023.
2, 4, 10
12

---

[25] Gabriel Synnaeve, Qiantong Xu, Jacob Kahn, Tatiana Likhomanenko, Edouard Grave, Vineel Pratap, Anuroop Sriram, Vitaliy Liptchinsky, and Ronan Collobert. End-to-ent ASR: From supervised to semi-supervised learning with modern architectures. In /CML. 2020 Workshop on Self-supervision in Audio and Speech, 2020. 2, 6, 22, 42
[26] Alexci Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:1 13:114499-12666 12440, 2020. 2
[27] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang. Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-augin -augumttee-e transformer for speech recognition. Interspeech 2020, pages 5036-5040, 2020. 2, 6, 9, 22
[28] Sehoon Kim, Amir Gholami, Albert Shaw, Nicholas Lee, Karttikeya Mangalam, Jitendra Malik, Michael W Mahoney, and Kurt Keutzer. Squeezeformer: An efficient transformer for automatic speech recognition. Advances in Neural Information Processing Systems, 13:9361-93733 2022. 2, 22
[29] Yu Zhang, Daniel S Park, Wei Han, James Qin, Anmol Gulati, Joel Shor, Aren Jansen, Yuanzhong Xu, Yanping Huang, Shibo Wang, al. BigSSL: Exploring the frontier of large- scale super-suee learning for automatic speech recognition. IEEE Journal of Selected Topics in Signal Processing. 16(6):1519-5522 2022. 2
[30] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek. Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning, pages 7480-7512. PMLR, 2023. 2, 51
[31] Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Dan Busbridge, Jason Ramapuram, Yizhe Zhang, Jiatao Gu, and Joshua M Susskind. Stabilizing transformer training by preventing attention entropy collapse. In International Conference on Machine Learning, pages 40770-
40803. PMLR, 2023. 2. 6, 44. 51
[32] Natalia Tomashenko, Salima Mdhaffar, Mare Tommasi, Yannick Estève, and Jean-François Bonastre. Privacy attacks for automatic speech recognition acoustic models in federated learning framework. In ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6972 2022. 2, 10, 21
[33] Peter Kairouz and et.al. Advances and open problems federated learning, 2021. 2,9
[34] Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 20/4 IEEE 55th Annual Symposium on Foundations of Computer Science, pages 464 473, 2014. 2,4
[35] Yinchen Shen, Zhiguo Wang, Ruoyu Sun, and Xiaojing Shen. Towards understanding the impact of model size on differential private classification. CoRR, abs/2111.13895, 2021. 2
[36] Florian Tramèr and Dan Bonch. Differentially private learning needs better features (or much more data). CoRR, abs/201 1.1 1660, 2020. 2
[37] Rami Botros, Anmol Gulati, Tara N Sainath, Krzysztof Choromanski, Ruoming Pang, Trevor Strohman, Weiran Wang, and Jiahui Yu. Practical conformer: Optimizing size, speed and flops conformer for on-device and cloud ASR. arXiv preprint arXiv:2304.00171, 2023. 2
[38] Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Conference on Learning Theory, pages 3635-3673, PMLR, 2020.
[39] Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew G Wilson. Does knowledge distillation really work? Advances in Neural Information Processing Systems, 34:6906-6919, 2021. 2, 22
13

---

[40] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369-376, 2006. 2,
[41] Jakub Konečný, H. Brendan McMahan, Felix X. Yu, Peter Richtarik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. In NIPS Workshop on Private Multi-Party Machine Learning, 2016. 3
[42] Sashank J. Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In International Conference on Learning Representations, 2021. 3, 4, 5, 22, 23, 35, 36
[43] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3, pages 265-2265222 284.. Springer, 2006. 3
[44] Cynthia Dwork. A firm foundation for private data analysis. Communications of the ACM, 54(1):86-95, 2011.
[45] Kamalika Chaudhuri, Claire Monteleoni, and Anand D. Sarwate. Differentially private empirical risk minimization. J. Mach. Learn. Res., 12:1069-11001 jul 2011. 3
[46] H Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. In International Conference on Learning Representations, 2018. 3, 4, 8, 9, 22, 24, 34, 37, 38, 41, 50
[47] Peter Kairouz, Brendan McMahan, Shuang Song. Om Thakkar, Abhradeep Thakurta, and Zheng Xu. Practical and private (deep) learning without sampling or shuffling. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning. ICML 2021, volume 139 of Proceedings of Machine Learning Research, pages 5213-5225, PMLR, 2021. 4
[48] Minghin Xu, Congzheng Song, Ye Tian, Neha Agrawal, Filip Granqvist, Rogier van Dalen, Xiao Zhang, Arturo Argueta, Shiyi Han, Yaqiao Deng, Leo Liu, Anmol Walia, and Alex Jin. Training large- vocabulaay neural language models by private federated learning for resource- constrained devices. In ICASSP 2023 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, 2023. 4,22
[49] Yang You, Jing Li, Sashank J. Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training BERT in 76 minutes. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. 4, 23, 25,44 44
[50] Michael P Friedlander and Mark Schmidt. Hybrid Deterministie-stochastic Methods for Data Fitting. SIAM Journal on Scientific Computing, (43)):A180-AAJ: 2012. 4, 23
[51] Seyyedali Hosseinalipour, Sheikh Shams Azam, Christopher G Brinton, Nicolo Michelusi, Vaneet Aggarwal, David Love, and Huaiyu Dai. Multi stage Hybrid Federated Learning over Large-scale Wireless Fog Networks. arXiv preprint arXiv: 2007:00077.9511 2020. 4, 23, 38
[52] Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang, On the Conver- gence FedAvg on Non-IID Data. International Conference on Learning Representations (ICLR), 2019. 4, 23, 38
[53] Sebastian U Stich. Local SGD Converges Fast and Communicates Little. In International Conference on Learning Representations (ICLR), 2019. 4, 23, 36
[54] Sheikh Shams Azam, Seyyedali Hosseinalipour, Qiang Qiu, and Christopher G. Brinton. Recy- cling model updates in federated learning: Are gradient subspaces low-rank? International Conference on Learning Representations (ICLR), 2022. 4, 23
14

---

[55] Xinwei Zhang, Xiangyi Chen, Mingyi Hong, Zhiwei Steven Wu, and Jinfeng Yi. Under- standing clipping for federated learning: Convergence and client-level differential privacy, In International Conference on Machine Learning, ICMI. 2022, 2022. 5, 10, 22, 44, 50
[56] Sheikh Shams Azam, Tatiana Likhomanenko, Martin Pelikan, and Jan Silovsky. Importance of Smoothness Induced by Optimizers in FL4ASR: Towards Understanding Federated Learning for End-to-Ennd ASR. In IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2023, 5, 6, 7, 10, 22, 25, 45
[57] Sai Praneeth Karimireddy, Satyen Kale, Mchryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. SCAFFOLD: Stochastic controlled averaging for federated learning. In Hal Daumé IIII and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning. volume 119 of Proceedings of Machine Learning Research, pages PMLR, 13-18 Jul 2020. 5, 8, 35
[58] Anastasia Koloskova, Hadrien Hendrikx, and Sebastian U Stich. Revisiting Gradient Clipping: Stochastic Bias and Tight Convergence Guarantees. In International Conference on Machine Learning (ICML), pages 17343-17363 PMLR, 2023. 5,37
[59] Jared Kaplan, Sam McCandlish, Tom Henighan, al. Scaling laws for neural language models. arXiv preprint arXiv:2001.088: 2020.
[60] Xiangyi Chen, Steven Z Wu, and Mingyi Hong. Understanding gradient clipping in private sgd: A geometric perspective. Neural Information Processing Systems (NeurIPS), 33:1377-113728 2020. 5, 35, 37
[61] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an ASR corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206-5210. IEEE, 2015. 6. 10, 41
[62] Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. Common voice: A massively-multlinguaal speech corpus. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4218-4222, 2020. 6, 10, 41
[63] Tatiana Likhomanenko, Qiantong Xu, Jacob Kahn, Gabriel Synnaeve, and Ronan Collobert. SlimlPL: Language-MMdee-FFree Iterative Pseudo-Labeling. Interspeech, 2021. 6, 42
[64] Daniel S Park, William Chan, Yu Zhang, Chung -hheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. Specaugment: A simple data augmentation method for automatic speech recognition. Interspeech 2019, pages 2613-2617, 2019. 6, 42, 46, 47
[65] Yang You, Igor Gitman, and Boris Ginsburg. Scaling SGD batch size to 32k for imagenet training. CoRR, 001.1700...8888 2017. 6, 44
[66] Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In Proceedings of the 30th International Conference International Conference on Machine Learning Volume 28, ICML' 13, page I1I-139--I1-1-1 JMLR.org, 2013. 6, 45
[67] Zheng Xu, Yanxiang Zhang, Galen Andrew, Christopher Choquette, Peter Kairouz, Brendan Mcmahan, Jesse Rosenstock, and Yuanbo Zhang. Federated learning of gboard language models with differential privacy. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), pages 629-639, Toronto, Canada, July 2023. Association for Computational Linguistics. 7, 9, 22
[68] Konstantin Mishchenko, Grigory Malinovsky, Sebastian Stich, and Peter Richtarik. ProxSkip: Yes! Local gradient steps provably lead to communication acceleration! Finally! In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 15750-15769. PMLR, 17-23 Jul 2022. 8
15

---

[69]
Mike Chatzidakis, Junye Chen, Oliver Chick, Eric Circlaeays. Sowmya Gopalan, Yusuf Goren, Kristine Guo, Michael Hesse, Omid Javidbakht, Vojta Jina, Kalu Kalu, Anil Katti, Albert Liu, Richard Low, Audra McMillan, Joey Meyer, Steve Myers, Alex Palmer, David Park, Gianni Parsa, Paul Pelzi, Rehan Rishi, Michael Scaria, Chiraag Sumanth, Kunal Talwar, Karl Tarbe, Shan Wang, and Mayank Yadav. Learning iconic scenes with differential privacy. https: // machineles meriiinnerningg apple. cessed: 2023-09-05. 8,
-ariiiallppiiic.... 2023. Ac-
[70] Dimitrios Dimitriadis, Ken'ichi Kumatani, Robert Gmyr, Yashesh Gaur, and Sefik Emre Eskimez. A federated approach in training acoustic models. In Interspeech, pages 981-985, 2020. 9
[71] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint
2018. 9
[72] Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech Recognition with Deep Recurrent Neural Networks. In ICASSP, 2013. 9
[73] Yonghui Xiao, Yuxin Ding, Changwan Ryu, Petr Zadrazil, and Francoise Beaufays. Federated learning of large asr models in the real world. ArXiv, abs/2408. 000s/2408.10444, 2024. 10
[74] Jacob Leon Kröger, Otto Hans-Martin Lutz, and Philip Raschke. Privacy Implications of Voice and Speech Analysis - Information Disclosure by Inference, pages 242-258. Springer International Publishing, Cham, 2020. 10, 21
[75] Galen Andrew, Om Thakkar, H. Brendan McMahan, and Swaroop Ramaswamy. Differentially Private Learning with Adaptive Clipping, 2022. 10
[76] Egor Shulgin and Peter Richtárik, On the Convergence of DP-SGD with Adaptive Clipping. 2024. 10
[77] Toan N Nguyen, Phuong Ha Nguyen, Lam M Nguyen, and Marten Van Dijk. Batch Clipping and Adaptive Layerwise Clipping for Differential Private Stochastic Gradient Descent. arXiv preprint arXiv:2307.11939 2023.
[78] Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. Large language mod- els can be strong differentially private learners. In International Conference on Learning Representations, 2022. 21
[79] Xuechen Li, Daogao Liu, Tatsunori B Hashimoto, Huseyin A Inan, Janardhan Kulkarni, Yin- Tat Lee, and Abhradeep Guha Thakurta. When does differentially private learning not suffer in high dimensions? Advances in Neural Information Processing Systems, 35:28616-28630, 2022. 21
[80] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Leurning Representations, 2022. 22
[81] Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank J Reddi, Sanjiv Kumar, and Suvrit Sra. Why Adam beats SGD for attention models, 2020. 22
[82] Tatiana Likhomanenko, Qiantong Xu, Vineel Pratap. Paden Tomasello, Jacob Kahn, Gilad Avidov, Ronan Collobert, and Gabriel Synnaeve. Rethinking evaluation ASR: Are our models robust enough? Interspeech, 2021. 22, 41
[83] Shubham Toshniwal, Anjuli Kannan, Chung-Cheng Chiu, Yonghui Wu, Tara Sainath, and Karen Livescu. A comparison of techniques for language model integration in encoder- decoder speech recognition. In 2018 IEEE Spoken Language Technology Workshop (SLTT)
pages 369-375, 2018. 22
[84] Anuroop Sriram, Heewoo Jun, Sanjeev Satheesh, and Adam Coates. Cold fusion: Training seq2seq models together with language models. CoRR, abs/1708 706666426, 2017. 22
16

---

[85] Çaglar Gülgehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Ch Chei-Chh Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. On using monolingual corpora in neural machine translation. CoRR,
2015. 22
[86] Felix Stahlberg, James Cross, and Veselin Stoyanov, Simple fusion: Return of the language model. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 204-211, Brussels, Belgium, October 2018. Association for Computational Linguistics. 22
[87] Dan Berrebbi, Ronan Collobert, Navdeep Jaitly, and Tatiana Likhomanenko. More speaking or more speakers? In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, IEEE, 2023. 22, 23
[88] Jeffrey Fong, Siwei Chen, and Kaiqi Chen. Improving Layer- wise Adaptive Rate Methods using Trust Ratio Clipping. arXiv preprint arXiv:2011. 135844, 2020. 25
[89] Belhal Karimi, Ping Li, and Xiaoyun Li. Layer-wise and Dimension wise Locally Adaptive Federated Learning. arXiv preprint
2021. 35
[90] Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. In International Conference on Learning Representations, 2020. 42
[91] Tatiana Likhomanenko et al. CAPE: Encoding relative positions with continuous augmented positional embeddings. NeurIPS, 2021. 42
[92] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume 2 (Short Pupers), pages 464-468, 2018 42
[93] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. 44
[94] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(61):2121-2159, 2011. 45
[95] Susanne R. Borgwaldt, Frauke M. Hellwig, and Annette M.B. de Groot. Word-initial entropy in five languages: Letter to sound, and sound to letter. Written Language and Literacy, 1(2):165-184, 2004. 46
[96] Elisabeth Borleffs, Ben A. M. Maassen, Heikki Lyytinen, and Frans Zwarts. Mcasuring orthographic transparency and morphological-syllaiicc complexity in alphabetic orthographies: narrative review. Reading and writing, 30(8):1617-116333 2017. 46
[97] Johannes C. Ziegler, Arthur M. Jacobs, and Gregory O. Stone. Statistical analysis of the bidirectional inconsistency of spelling and sound in French. Behavior Research Methods, Instruments, & Computers, 28:504-515 1996. 46
[98] Liliane Sprenger-Charoline Alphabetic Writing Systems: English, French, German and Spanish, pages 44 66. 01 2003. 46
Linguistic Processes in Reading and Spelling: The Case of
[99] Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj. Jessica Zhao, Graham Cormode, and Ilya Mironov. Opacus: User-friendly differential privacy library in PyTorch, arXiv preprint arXiv:2109 12098, 2021. 50
[100] Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. Understanding the difficulty of training transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020), 2020. 53
[101] Jiyan He, Xuechen Li, Da Yu, Huishuai Zhang, Janardhan Kulkarni. Yin Tat Lee, Arturs Backurs, Nenghai Yu, and Jiang Bian. Exploring the limits of differentially private deep learning with woise clipping. The Eleventh International Conference on Learning Representations, 2023. 59
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Refer to Section for theoretical results and Appendix D for empirical results.
Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Refer Appendix E for details.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Refer Appendix B, F, G, H, for details. Also code is available at https: / //g ///gttub.bo.. cwm/ coople/ml-pflee
5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
The paper uses two datasets: LibriSpeech (LS), which is publicly available without restrictions, and Common Voice, which requires usage terms before downloading. The code with all instructions for models training is available https: //:::///tttte com hommapppe//eeeep /m1 ml-pfl14aal 14arr. Additionally, the algorithm and experimental sections in the paper provide rigorous details to allow faithful reproduction of the core experiments. Refer Appendix F, G. H. for details.
Justification:
to agree Mozilla's data
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Refer Appendix B, F, G, H, I for details.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Refer Appendix for details.
18

---

9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https:/ http:///wwwwp...cccoacc.ccccaaaa
Answer: [Yes]
Justification: Refer Appendix A for details.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Refer Appendix C for details.
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Not applicable. The paper does not release new datasets or pretrained models with a high risk for misuse. proposes an algorithm for privacy-preserving optimization of existing ASR models using publicly available datasets. As such, the work does not require additional safeguards beyond the usage terms already associated with the datasets.
Justification:
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientifie rigorousness, originality of the research, declaration is not required.
Answer: [NA]
19

---

Appendices
Ethics Statement
21
Reproducibility Statement
21
C Socictal Impact
21
D Discussion
21
D.1 Need for Private Federated Learning
21
D.2 Why Do We Study Larger Models for FL and DP?
................................
21
D.3 Clipping and Adaptive Optimizers
22
D.4 Fusion of ASR Model with Language Model
22
D.5 Conformer vs Transformer
22
D.6 Seed Models
22
D.7
Limitations
23
E
Theoretical Analysis
23
E.1
Assumptions
23
................................
E.2 DP Assumptions
................................
24
E.3
Helpful Lemmas
24
E,4
LAMB
25
E.5
Adaptive Optimizers and Per-Layer Clipping: The Main Proof
25
E.6
Finite-Time Convergence Rates
34
E.7
Recovering Prior Bounds
35
E,8
Adaptive Optimizers and Per-Layer Clipping: Theorem Under Limited Participation
38
F
Empirical Analysis: Data and Central Models Training
41
G Empirical Analysis: Federated Learning without Differential Privacy
42
G.I Hyper-parameters
42
G.2 Detailed Results for English
G.3 Impact of Model Architecture on FL Performance in ASR
43
G.4 Impact of Server Optimizer on FL Performance in ASR
44
G.5 Detailed Results for CV French and German
46
G.6 Impact of SpecAugment
46
G.7 Performance of FedProx FL for ASR
47
G.8 Extending the Number of Central FL Iterations
48
G.9 Impact of Under-Trained Seed Models
48
G.10 Scaling Larger Cohorts
49
Empirical Analysis: Federated Learning with Differential Privacy
50
H.I
Differential Privacy Noise Discussion
50
H.2 Large Cohort Training Implementation
50
H.3 Empirical Analysis
50
H.4 Detailed Results
53
H.5 Per-Layer Clipping Analysis
H.6 Federated Learning with Differential Privacy for French and German
55
20

---

H.7 Per-Layer Clipping for Different Model Sizes
60
I Compute Resources
63
Contributions
63
A Ethics Statement
For all experiments we use publicly available data for research: LibriSpeech (CC BY 4.0) and Common Voice v13.0 (CC BY-SA 3.0). the paper, we aim understand the behavior of large transformer models in federated learning (FL) with differential privacy. This is a step towards developing private FL in the context of speech recognition to provide strong guarantees of user privacy.
B Reproducibility Statement
For all experiments we use publicly available datasets for research: LibriSpeech (CC BY 4.0) and Common Voice v13.0 (CC BY-SA 3.0). Data processing is described in the main body of the paper. We describe all configurations, training details, ablations, and our procedure of selecting hyper-parameters throughout the paper and in Appendix. We also provide important discussions on different aspects of the empirical results as well as detailed plots of various characteristics tracked during training the Appendix. The code open sourced and available https : /tt.ttb....t.///g//gg thub. com/
apple/ml mal-ppll4ll 14asr..
C Societal Impact
This work explores research in the intersection privacy, optimization, federated learning, and speech recognition. Given the widespread adoption of ASR models deployed in production environments ranging from virtual assistants to accessibility applications, enabling privacy-pre -rraccceessrinnggg training of ASR models using differential privacy has the potential to benefit the end users, particularly in sensitive domains such as healthcare and biometrics. This work contributes towards the responsible development of ASR models by overcoming a long-standing obstacle to applying DP to deep architectures. However, the deployment of Fl. with DP does not eliminate all privacy risks. Real-world deployments must ensure additional measures including secure aggregation and careful consideration of population-scale that influence the strength the privacy introduced by DP this work.
D Discussion
D.1 Need for Private Federated Learning
In Section 1 we discussed that FL on its own does not guarantee user privacy. For example, [2] showed that the gradients sent to the server can used to reconstruct the original training images and text. [3] showed that a model can memorize specific pieces of data that can be reconstructed using only the model itself. In the context of ASR, [32] developed two attacks that aim to infer speaker identity from the model updates without access to the actual users' audio data. [74] showed that audio data reveal information about the content but they can also be used to derive other pieces of sensitive information including biometric identity, physical traits, geographical origin, emotions, level of intoxication, age, gender and health.
These and many other works emphasize the necessity of developing private FL with strong guarantees on the user privacy. In this paper, we focus on providing first insights for private FL with DP for ASR.
D.2 Why Do We Study Larger Models for FL and DP?
As discussed Section I, we focus on the model size 250M parameters. Prior works in FL with DP primarily focused on studying models of up 30M parameters, justifying the use of smaller models by communication and training costs associated with the model size and the difficulty of training reasonable models with DP because the impact of noise scales with the model size. However, [78, 79]
21

---

showed that it is possible to (centrally) fine-tune large language models with hundreds of millions of parameters with DP and DP impact does not prevent efficient training if gradients are low rank.
Our main reason to focus our study on larger models for both FI. and DP is the observation that larger models are simpler train in practice. It is hard and open problem efficiently train small models that perform the same or better than models obtained for example by distillation of large models into smaller models [39]. To disentangle the ability train small models efficiently from the problem of matching central training with FL and FL with DP, we study larger models, Our results give a hint that the gap that existed between FL and central models could be related to the absence of proper training recipes for smaller models.
One could argue that current model sizes are huge in the of large language models, and different techniques, like LoRA [80], could be used to reduce training time on clients as well as communication costs. This done for example by [48] who used partial and low-rank model updates train large language models with private FI.. However, we believe that first we need to train competitive baseline models from scratch or from out-of-domain seed models, and understand their behaviour and limits.
D.3 Clipping and Adaptive Optimizers
[55] investigated how clipping fights data heterogeneity in FL. As discussed in Section 2, clipping is also an essential part of DP. To be able to train transformer models, we must use clipping too, and aligned with FL with DP. In Appendix H Figure we show
thus the recipes used for transformers that gradient clipping during local training leads to bounded norms of user deltas where the latter is necessary for DP. Without applying gradient clipping, the gradient norms would be huge already the beginning of the training and even with LARS, pre-LayerNorm and central training we would not be able train a reasonable model. Thus, it is extremely hard disentangle any empirical results for transformers to understand how clipping helps the training for FL with DP.
[42] and [56] showed that adaptive optimizers alleviate the issue of data heterogeneity for FL. At the same time it is hard to train transformer models without adaptive optimizers [81, 55]. This is yet another example of alignment between FL and central training of transformer models; technique that helps alleviate data heterogeneity in FL is a must when training large transformer models even centrally.
D.4 Fusion of ASR Model with Language Model
To further improve WERs, ASR models can be combined with language models during inference.
This can be done in various ways, e.g. using beam-search decoding for CTC models [25, 82], or using shallow fusion [83], cold fusion [84], deep fusion [85], and simple fusion [86] for Seq2Seq or transducer- based models. In this paper. we leave the study on how language model integration affects the final model performance as a future work. In the latter case, language models can also be trained using FL with DP [46, 48, 67].
D.5 Conformer vs Transformer
Purposefully, we do not use the conformer architecture [27] the paper. In prior work by [28], it was shown that, e.g., for CTC models both conformer and transformer architectures give similar results while conformer has fewer parameters. We focus on larger models to understand their behaviour. Moreover, vanilla transformers are still de facto a standard in other domains, while conformers were adopted only in speech recognition. Therefore, focusing on vanilla transformer models will broaden the impact of our findings for speech recognition on the FL and DP communities at large.
D.6 Seed Models
[20] trained seed models to initialize FL using small fraction of speakers (117 speakers, or 2.8%, for French and 99 speakers, or 13.2%, for Italian) and used the rest the data for FL training. Recent work [87] showed that model quality depends on the number of speakers and the diversity of the training data: better to have more speakers with shorter total audio duration than to have fewer speakers with longer total audio duration.
22

---

Based on the recommendation of [87] to have at least Ik speakers in the training data, we randomly sampled 5% (English) or 10% (all languages) of speakers for the in-domain seed model training. This provided more than IK users for training CV seed models for English. While for French the seed model trained from only 685 users and for German the seed model trained from only 712 users, we note that French and German languages are easier to train. Furthermore, for FL. models training on CV (English) we use a seed model trained on LS-100 100 that has only 251 speakers; however, LS-100 has over 100 hours of audio, which is approximately 6.3% of the total audio in CV.
Preliminary experiments showed that the seed model training on subset of 5% speakers with the shortest total audio does not converge: even for English the subset contains less than 2 hours of audio, which is known to be hard for any E2E ASR model training. In contrast, we take a subset of 5% speakers with the longest total audio as in [20], seed model is very well trained as then the dataset has more than 64% of total audio the CV dataset for English language and training on the rest of the data brings little benefit. Thus, we found the subsets with minimum-duration or rikimuum-duration users to not be practical scenarios.
For LS, validation (test) set has 5h of audio with mean of -8mi and standard deviation of 0. mmin for the total duration per speaker. For CV, validation (test) set has ~30h with mean of 155 and standard deviation of 1.5s for the total duration per speaker. Thus validation and test datasets have homogeneous distribution which weights speakers (users) equally for evaluation. For both LS and CV we use original validation and test sets, without any modification. Thus, the disjoint set of speakers in different splits and the disjoint of speakers in seed model and FL training ensure that speakers (clients) are not accounted twice in the privacy budget.
D.7 Limitations
Our theoretical results are derived under some assumptions listed in Section 3. Empirical results are limited to i) LibriSpeech and Common Voice (en, de, fr) read speech data; ii) monolingual models; iii) CTC-based models of size 100M-500M parameters; iv) absence of external language models; v) audio data assumed to be labeled. Future work would include theoretical and empirical analysis to overcome these limitations.
E Theoretical Analysis
E.1
Assumptions
Given a global model comprising of H layers, the model parameters are defined as O (0,.... .......... 0H). is presumed that the loss function for each sample x bounded below: m f(z, 0) > -c,, where x ~ DA, V k. Let || || denote the In-norm. Our analysis uses the following standard assumptions [12, 42, 50, 51, 52, 53, 54]:
1. Smoothness of Gradient of Loss Function: Gradient of loss function is layer-wise Yh [49]:
L -mooth for [,x, 0) (, 0')l < L, 100 l'1l, V0,0' E ER., E RRN, Yk,
(A1.1)
where V, denotes gradient with respect to parameters 0n of layer h. Consequently, the loss function also L-smooth, where L [[,,, 112
(V((z,)) V((x, 0' S ||0 0'll 40,0' E R D.R.RRR,,V Vk.
(A1.2)
2. Local Gradient Characteristics: Given user k, B. { . Dd and local gradient Vi(x. 0) for DR. its estimator g; (0) = g (B,,
k,,00 0) (c.g. obtained by SGD) is an unbiased
estimator and have a bounded variance [12, 50, 54], thus:
IF A4
[ (0) (.).((). and
(A2.1)
E E
(
V (0))
< x² loe
2 0. VOERD, Vk.
(A2.2)
3. Global Pseudo- Gradient Characteristics: The variance of global (pseudo-) gradient is assumed to be bounded [52, 42] such that:
K We -
02 slob o dlb 0, VOERD.
(A3)
23

---

To give a probabilistic interpretation of this assumption we can estimate the global loss gradient V (0) by sampling one user Categorical( ...... and using the following unbiased
estimator (u, 0) = V Lu ( Then,
K
K
V 2u = > Pl==) = Σ we =
CatCooorricl (k.
From the latter we get the variance of the estimator and extend it to the left-hand side of Equa- tion A3. Thus, can interpret this assumption the variance of global (pseudo-) gradient.
E.2 DP Assumptions
To incorporate user-level DP into FL, we consider that every client is sampled i.idd with probability
q (gK' S) and then the client updates (t)
k are: (i) clipped such that their l2 norm is bounded,
iee,
| 2|2 S C at every central training step and then (ii) perturbed via Gaussian mechanism,
where such that final client updates under FL with DP are given by A k +N (, IC IC 2im1 up
(t) Te k (+) and a max C.lly G
For Ik= Wk 1, where we (0,1). we can
extend Theorem I to the weighted loss case by defining sensitivity S max K w q per Lemma from [46]. Having w/s- K.K. we get exactly sensitivity definition S 1/(gK) from Theorem 1.
E.3 Helpful Lemmas
Lemma 1. For any positive variables C, C, E RR, we have
X -
max (C,Y) I
(2)
max (C
C
Proof. We can prove it by analyzing three independent cases:
ii ifC 2 X and C' 2 Y trivially have
1 1
X
= =0
max (C, X)
max (C,Y)
C
C
C
(ii) C X and < Y we have
1 max (C,X) max (C,Y) X y
1
X Y
X Y
S
and
XY
C
(ii) i <C < X (cquivalently the case Y > C > X)
we have
1
1
-
C
X-Y
<
<
max (C, X) max (C.Y)
X
C
XC
C
Thus, we can conclude A C,, YYYRR
X-Y
max (C, X) max (C,Y)
C
Lemma 2. For X E R and constant C > 0,
X2
(X (X-) <
2C
where (X CX4 max (0. C).
24

---

Proof. For X < C, (X C I O, and inequality holds trivially. For X > C we can use the algebraic identity:
x² 2C (x (
which can be rewritten as (X C) -
x²
Lemma 3. For random vector G Rd with bounded norm ||G|| S U and a clipping constant
C O, define the clipped vector Go I& such that Gc G
max Then the squared
C
distance between G and Ge is upper bounded by
U
|G
Gcll 2
(5)
40
Proof. For [|] S C, we have
C
C
| G 2
G G
G
G
0.
(6)
max (C, E
GI)
For |||
C, we can use the algebraic identity:
C
C
|
Gell
G G
G G
max (C, E
III)
E
||
C
E
G
C
||G|
|G]]
E G
E
Lemana
( G
ID)*
|
U
S
(7)
4C2
(E
II])
40
Thus, the trivial case in Equation 6 together with the inequality in Equation 7 results in the final bound.
E,4 LAMB
The per-layer update rule of LAMB is given by:
① (11)
m
h
(1+1)
(0) Alab
|lu (1)
+ 10
(u 20) where A 2 0,
[//
(0)
(8)
Vv
+
m
(4)
B m
(1-1) (t-1) +1 (1- B1) (1) (0) (2 -2)
A (0)
0 S B1, B2
6 IR + is a scaling function which is often defined as an identity in standard LAMB applications [49, 88]. While & constant generally employed for numerical stability, [56] show that & 0.01 leads to best results in FL, likely because it counteracts spurious pseudo-gradients early in the training. Let's define the trust ratio of LAMB:
④ (10((1)
(
E R
and
(1) Ph
A
rh
(10)
(t) Vi
+
E.5 Adaptive Optimizers and Per-Layer Clipping: The Main Proof
Theorem 2. Assume A1.1, A2.1, A222, and A3, Na L < and = [1 [1- 8(1 Tloe T T > 0. If the trust ratio from Eq. 10 in LAMB optimizer is controlled in the Algorithm (global optimizer is
LAMB and local optimizer is SGD) such that r < R2 and P
(t)
(c)
S Pn- 31 and
25

---

in LAMB optimizer. and clients are i...d. sampled with probability q (no sampling). then after T steps of aggregation the performance of FL with DP. per-layer clipping and layer-wise gradient normalization is characterized by the following upper bound:
K
E loc
V (80))
)1)
2 [ (0 - LL (0))
16H Moo
T202 0²
+ +22 T 0²
T
N Mgto T
wlob
1=0
C DP
Rh , + 2n T
Σ
Ma
16L2n
Mue loo
T + 4P%
1
5²
100
0%
h=1
h=1
T
R2 M
Eg
Var 1
(1
hyk ())] Var, (R, (Bi
G
h, [I)]
2
h
t=0
where k Bate ,kk and ,,, H denotes the expectation over sampled mini-
batch wod every local step t 1,..., Tim from the client data: Fg (t.twe) hoo) 10/h1o0) = Bx-
Di. Fg (t.twe) E
Proof. We assume 31 0 and regularization À 0. Then the update rule for LAMB as the global optimizer at the FL server given by Equation 8 can be rewritten:
VA
(0)
B2v (1-1)
+ (1 B2)
(1)
2
0
B2 S
(11)
A (9)
(1)
11
(12)
Vv + E
(1e (1)
0g
1++)
o(0)
u
(1)
vh.
(13)
|
Given definition of the trust ratio in Equation 10, the update rule can be expressed as:
(++1)
(t) Reia Pi
(14)
The aggregated clients updates, or pseudo grando--rrdiet,, A (t) are given by (as q = 1):
K
A h,k + =
wi
(15)
where A
h.k is the accumulated client update (see Algorithm 1) and hi,k ~ N 10,1, Dr Σ the random independent DP-noise added to client updates. For each client we perform T, steps of SGD optimization by i) sampling mini-batch go (fee every local step t... 1,..., Tie from the
client data: #k (f.f)ee) DE. aik (t.f E [o(/ae) k (foee) Be: ii) performing gradient step with
local step-size (learning rate) The having (t.00) (())
gh,k
(o (o(t/iic))
1
Σ
Vln (t,5 (t.l100)))
(16)
BR
(t.Pioa (lo0))
(1. h,k
(fliiio -1)
[t, -1) (ottitioe-1 -1)
Tho Bh.k
(17)
(ee where g are unbiased estimators of clients' gradients. Then for a given per-layer clipping
constant C₁ > 0, the client updates and the corresponding clipping multipliers are defined as:
Tim
G h.k = g(t.0)
g(t.Troo) = Mio
Σ
8i,k
(t.t
(18)
1 0
26

---

A 1
(1) Ob,kk
G(
(4) with o
C
(19)
Cno (+) l)
max
With triangle inequality we can upper bound the norm a random variable G given the- orem assumption that V4, 0) is I-LLipshiiz smooth and thus ( 0) S Mn. (e.g.
Mn. I|V 0o) + Lmaxocel where θ is compact):
100 -1
|| ll S
The
Ilg
(titinc)
S Thoc The Mn-
(20)
100
We next define the auxiliary terms in the context of clipping:
A (4) (+)
b.b 11,b 11.b
(t) (t)
with (() OA
max (C
E,
C₁
G
(21)
los
hi,k
A k.b 0n G
with 0 (t)
(22)
max Cr E.,,.
(1))
(1) h,k
Since gradient of loss function f(x, 0) is L- Lipschitz smooth, get the following for any two points ((t-1) and ((t):
((++1) - -0(1) + L ( - -ocol t(x, (((+1)) < E(x 0(x) + (VE((x, 0(+))
(23)
By taking expectation over the client ls data æ Dx. for every client can write down:
(2(x, (x( + ( (z) 0(++1) ((()) +
L | - (24)
By multiplying with wk. summing all inequalities across clients, and using the update rule from Equation 14, we can get:
+ (V.8 (BLO), ;00)),((++1)) _0 ((+)) L (g(++1) S S =((())
L
g )(t)
(25)
2
n2lob
IL =L (0(t) nglob (v (0(5) (1) (t)
(t)
(t)
(26)
Bounding loss F, [ (B(++))] with Z, term
27

---

Now, let's take the expectation over the mini-batches D ) sampling in the local SGD optimization for both sides of inequality having random variables Pk
(1) and A (8) (for short notation we use E, $100
[:)::
H
E 'ooe
L (oitti))
S LL (010) - n
glob
Σ
E
[(V.R (0%²
O A >
h=1
Nenn
L
E.
||1/ +
(2)
2
A=1
H
L (0(1)
E
Ln
(0) )1{
Natob
Ph
(1)
(0)
100
2
H1
11 wloh
|| V.S.% (0)0)
(t)
+
Ph
2
'oo
h=1
2,
71 2
T
H
glob
E
Ph
(1)
A (0)
2
1oc
h=1
5 2 (010)
n Neooo
V
(1(+) ||²
I Ugion (1 - Meion L)
H
E
|
10/
glab
H
Z
2
$100
2
h=1
h=1
(1)
S 8 (0(1)
Naub
V.LL (8())
1|²
H
V.
(0%²)
(2)
(0)
2
+
2
E
100
-P
h
70
2n
(27)
where (1) uses- -2 (a, = - ||a||2 [ab|2= Ila b11? and (tt) uses the condition Moto L < 1. We can
next bound Z, using Equation 15, the auxiliary terms (t)
h.k and A (t) defined in Equations 21-22,
V.Lhn (o)
Zn E,,.
K
K
W
G
h,k
+ Σ
W/
4:
km
K
(t)
(t)
(t)
WPPP
h
h,k
k=1
(1)
(t)
(t)
WAPi
W&PA
h.k
K
K
(1)
(
h
h,k
w
h
(28)
As reminder, Jensen's inequality for some y: R
gives us:
K
K
K
Σ
wayk
<
Will
yk 2| where Σ
Wk = 1, 0 05 w/ 1.
(29)
k=1
k=
Helpful inequalities
28

---

Using the triangle inequality first and then applying Hölder's inequality, we get for yr: E R
K
K
2
K
2
K
Σ
yk
S
1 11
=
(1l 1) -
K
yxl1
(30)
1=1
Also, if yı and y/2 are independent random variables and E [y1] 0 then:
E
yl
y2
[
E [llyn |1 +
||
y1 1/2 >]
[1ly [|]]
[11y2]
+
1yl [x22
[11n [1]] [l1y2] []
(31)
Let's estimate for any random variable yn the following entity having that ya and P (e) independent variables:
are not
E
p vn
E, lsc
Ph
[y
S
R
de [ynk
E
E
[|
E2
Loc
E²
Loe
i=1
(32)
Bounding term with DP noise in Zn
1 J.
R and random independent DP noise
Having upper bound on the expectation E,
Ph
S
h ~ N (0,1,
(c) 2ist ui as q 1 per theorem condition (thus pj and Zi.k are independent
variables), let's get the upper bound first for:
K
K
da
K
2
E, loo
Σ
WiPP
h
=
E
luo
Ph
h
E
Two
Ph
(0)
wk
2 1o.k
(t)
A=
1=
1=1
dn
E
Ph
[t)]
E
300
K
11,k
(t)
2
R2
du E
K
W
(t)
2
k=
R2
K
R2
dp
2 02
W
dy C2 022 DP
(33)
DP
K
g²
k=
k=1
Bounding Z, with y, Y,, 2, Y 3, Y, terms
(
Given Equations 29, 33, 30 and 31 (we use the fact that DP noise z zero-mean independent variable), we can bound Z, in the following way:
Zp S
E Loe
K
V.L.
(0())
G
b.kk
E W
two
|
Ph
( A
27
Y
K
2
4
W
E
ph
(t)
A
(8)
A
(4)
+4
W
E
Ph
(t)
A
(8)
A b.k
100
1wc
1=1
k=1
R2 d oz
(34)
Bounding Y, term
29

---

Defining
H h.k
T
Σ,
=0
-1
Bh.
(1.t
and G
Too
T toe
hak
100
Y = E
K
W/
L. (0%
-G
Eg. 29
wk E Inc
V. (0)²)
Tlo T loc h.R
H
(6)
122
lor
1==1
k=1
V V. (0(²)
Mi T V L2 (0%²))
+ Ma Tioe V.Lr (0(²)
Mooo Tioe H (t)
Wille 1wo
Eq. 30
K
V (0)²)
7oo Tio V (0)-) |
S
Wk: $1ve
k=1
M Floo V.Lr (0)))
nor T H (1)
+ 2
Wik E
b.nk
k=1
2(1
To0 T Tiac ) 1000
V.Ln (o )|²
2n T
K
Wk B
V.L..
(0(²)
H
(t)
h.k
k=1
x
(35)
where
K
V.S.. (01²)
-H 1"
X
WiLEE ' 1uc
Am
K
Wis B,
|| (0%
-V (0%10) + V.nn (0%²)
- H R.k
k=1
Eq. 30
V.LL (08))
V.nnk (0!!)
112
[| (o)²) H ||
S 2
We IE
W $00
'nn
k=1
A3
K
V.L.IK (0)²)
1
loc -1
V.Lnn hikk (oftituee)
lac
(offifice) H111)
< 20
+
W
slub
The
T loc
1000
1100
Eq. 30
V.L.A (8%
loc
( (titt
20
slub
+
W
E
loc
T
V L
h.k
1ikk
k=1
m0
K
T
T
(((tttin)) h.k
(m )
4
Am
E
100
T
l0c
1
=0
V.
h.kk
T
100 =0
Bn
Fq. 29
K
loc
V. (0) V.L.I (oftifioo)) )
<
20²
+
Wk
E
T
km1
$100
T
Wkkk
loc
E
V.In
(1
-hi
||]
100
loe
1
A2,2
4
V. V.Lin (0/(01)
(o ))(²
<
20
+
Wk
E
V Dhin
+ 40
clob
T
The
1nc
k=1
0
30

---

A1.1
S 20²
2lob
+ 4o= +
4T,2 K
T
lac
Wk
T
E
lac
h,k
100
27
R=
LLo
=0
1uc
Eq 17
20
+ 40
+
4L2
Wk
E
Tloc
Bh.
(t,o)
(36)
T
lac
1=1
s=0
w
where
Line -1
100 -1
2
T
Inc
10c -1
W
E
noor
Bi.k
(t,s)
Eq. 30
1
E
Bh.k
(1,3)
110
$00
Tioe -1
(a,,(,
The
M
Σ
t2
n
MRTS 100
(37)
1100
Substituting it back X, we get:
4L2 K
X < 20 + 2
Σ wayn M
20
+ 40
+ +AL
T2 Mr.
(38)
T
alub
kmm
which we can substitute in Y, thus getting the bound:
K Σ
Y, s 2(1 Tho T )
V.L, (o(0) [1]
+ 2n T
w E
V.Lr (0%²
H m.k
1nc
k=
2111
Thoe T
) tin
V.La (0)) )1²
2n1 T22
20 slob.
+ 40²
+ 1L'
Tha
T22 Mf
100
l00
10c
V.L. (o%) [|]]
2(1 Tho Ti 2 1wo
+ 4n 2 0²
+ 81
T o²
+ 8LL 1/
T M2.
clab
(39)
Bounding Y, term
We next bound Y, using G defined in Equation 18 and its bound defined in Equation 20:
Y, E,
|
(1)
ai
(t)
G
(
P
o
2
Eq. 30
|| hik
|]]
< 2T,
0h Gi hi
(4) (()
(t) ((t) (t)
(0)
+
[r h.k
- Ph
h.k
Lemma and 20
1122 T2 MR
2Cº
+ 2E
Jnc
(g(0 (
(
h.k
Ph
A
(0)
(40)
31

---

Given the theorem's
1 1-p -Pph
S
PA. assumption", the latter term we can bound as:
(
(())
(1)
|| = =B 10e [1(1-18 p/ o oo o G
2
E loe
h.kk
Ph
h.k
m.kk
similar Eq. 32
Py lon
|
10
h.kk
(t)
|2]
Eq.20 20
S
P. Mh an S Ph
Ph
T
(1)]
Moo T Mr.
(41)
Substituting the latest bound back into 22 we finally can write:
mh T 10c 20 h
ME + +2Ph T Mh.
Y2
S
(42)
100 loc
Bounding Y, term
We can next bound Yg as follows:
Y33 E
lac
|
(1)
A
(C) - (0)
k.kk ||
Eq.
R
E
A
h,k
(1)
-0 l|]
h.k
āz // [1
R²
2 (a (t) - )
27
E
E
/
2²
"lce
Ey. 20
S
R
Tio
1
M2 yor
((1) (ahi - h,, )
27
(43)
22
Using 0 and (1)
(0)
defined in Equations 19 and 21, we have the following: 0
2
(0%%
(0)
-(1) -'iik)
C₁
C₁ Cr
max
(Cps
G
(t) h,k
max
IE
Gh.kk
(1)
h.k
(
(t)
G l1)
Lomma
h.k
1oc
(44)
Consequently, Y3 can be bounded as
R2 MFE,
(a/0) -att0
-i,ik) )
27
Y <
T
loc
E
(1/0//2
-B
B soc
|||
(1)
[])²
R
h.k
<
Mac
2
MR
CF
(1 (1/00) 1)
R2
Var
1/2
T2
Mr
1oc
(45)
loc
C
Bounding Y, term
We can finally bound Y, as follows:
Y2
E
Ph
(1)
A
(0) (1)
-
Eq.
5
R
E
A h,k
(2)
17]
foor
122
E.
o G(()
||]
R2
B,
(1/11
2 a1
E2
hikk
h.kk
E2
loc
Eq. 20 R2
<
72
(1()) ah,k
27
(46)
e2
loe
TThis assumption is reasonable given LAMB optimizer bounds its trust ratio.
32

---

Similarly, using (1 0 and o defined in Equations 21 and 22 we have the following:
2
(0) (a%) - Oi.k) )=
C₁
C₁
max
Chp
IE,
G( h,
max
EE,
G(
E
G
Lemma
lac
hikk
lac
(47)
0%
We can thus bound Y 4 as
Y, S
R2
Wine l0c
12
(a/h -a//1)
-/aik)
(t)
27
E
G((
E
G
(0) 2 hikk
[]]
R Vine loe
T2 Mh
100
(48)
C%
Final bound on Z, term
Substituting y, Y2 Y2 and Y, back in Z, and having is ~ Categorical(a)..... .w), we get:
2 S 8(1 - 1/ T 100
BB loc
(0) V.
11]
161 2 72 0²
clob + 32 T- o² + +3222 T Mr
h
21p T M
8Ppn2
72 Mi im
Inn
Ekmn We Var
(1)
R
h,k
+ 4
C2
Mloc
222 Mi
CH
R2
k= Wk
E
(t)
b.kk
E
G ?
R
+ 22 Mae
T M₁
100
E² d C² oN
Inc
C%
VI (0) [1]
=(11- Moo
T
2
16,
2
72 0²
+221 T2 Too2 0²
100 loc
loc
I0e
+ Tinr
72 M²
32L 2 T2 + ch
+ SPX
Inc
+ 4 R
1 T loc
M2
E, Var Var, to0 (/ + Var, (B.
(1)) h.k
[I]]
R
d C
g2
C
g²
(49)
Final bound on loss E
[L (0(++)))]
We can thus rewrite Equation 27 having
= [1 8(1 - noo T ) as
Le (ol++1)) s 1 (550(88 (B(:)
Kn
V.20 ( 11?]
E.
2
+ 8H 11u Thec 722 o²
16H
wor
Tie T2 o²
lac Blab
loc loe
H
1
T + Moro M
Σ MR
16L 16L
T
+
+ 4P2
1oo h=1
loo
C
+ 2
Besor Rr
g2
12
H
12 M
C
Ey Var a
(/G//) [)]
+ Var, (R
||
h.k
[]]
h=1
not. C² POPP
H
R2dda.
(50)
h=1
33

---

Rearranging and taking an average over all aggregation steps t = ....., - B,
and having o such
that
L
S
[.2 ( we finally get:
1oc
T
[ (0 1 (0')
E
V (0(0))1
16H hec T o²
+32H11 2 T o²
T
loc
Naro T
Rlob
t=0
R2j dl
2n
T
Mh 161.211
M
T
+ 4P%
lue
100
Ch
h=1
T2
loc
H
M
T
B2
Var
(11c1 Var, (B
G h.R
ID]
(51)
22T
h
Per Theorem in [8] and Lemma 1 and Theorem from [46] to guarantee (c, -p-rivacy 2? 2
comst" 2 T'g 1/8 while ODP 2 S = 2 max 1
wi/g. Then to get the final bound with (s, privacy guarantee, must select o const max (w,, 1m
For simplicity assumed that 31 0 and regularizer À = 0 in the LAMB optimizer. However, the proof can be extended to the cases with 81 > 0 and À >
Remark:
E.6 Finite-Time Time Convergence Rates
Corollary Assume A1.1, A2.1, A222, and A3, 7gtos L < and = 8(1 The Tic > 0. If the trust ratio from Eq. 10 in LAMB optimizer is controlled in the Algorithm 1 (global optimizer LAMB and local optimizer is SGD) such that r1 S R2 and 1 Ph S Pm B1 0 and A O in the LAMB optimizer, clients are i.i.d. sampled with probability q (no samplingg, and and Tho θ LVTree then Algorithm converges to stationary point of the global loss function with the convergence bound characterized as:
nas θ LVT
TToco22 whobe
K
|1
(01))
E
<
+
+
Ioc
T
1oc
VT
T
T
1=
optimization
global upslate noise
local updlate woise
H
Tior
H
M
Tipe
H
R
R
+ O
c 2
Σ
R dn
+0 o
+
+
V
V
inter
T
T
C
h=1
h=1
differential privacy noise
elipping
inter-elient update variance
(52)
where L
inten
inter E,, VVa,, soc (/G/)))) and I
(0) (.
E, Var, EE,nc
[
Categorical(uh..... Wx) and E t [-] denotes the expectation over sampled mini-batch F (hro every
local step Iine 1, ...,,, T from the client data: w (titinc)
Drr ky x (3,210))
E spfiae)
| (1a BR-
Proof. Using Theorem 2, we have
V (01
2 [ (0 - 5 (0')
E
16H T22 02 + +32H7, 2 To o
Tho Iac
T t=0
Nator T
CRopp
H Σ
+
Σ
RR,d, + 2 T
MR
16L
12 + + 4PR
2
C
hm
h
34

---

1
H
R
M
T
Var B2
(l G (ollll) [)]
+ Var (R
G []]]
4
T2T
100
h=1
1=
(53)
Choosing Nat.. 1/v7 and Thoe
1 loc
T, we get Bator T = VT and n loc! /T = Ticc/ T2
Substituting these the above bound we get
V
(00))
2 2 (0 L (0')
Thoc 0² clab
Thoe o7 + 32/1
E
16H
VT
T Inc
C P
H
Tin T
H
Tior T
+
R2 +
Mh
16L2
+
+ 4P%
g²
C
h=1
4 T Inc
R MA A
T-1
Var loc
(1G///2) + Var, T 'loc
G| [1)]
(4)
+
2 T
c
h=1
t=0
(54)
Above can be rewritten using the big-O O and definition of V/ juice
and VI
inter
E
|
( (00)
50
O
Tioc 02
slob
O
T t=0
100
VT
T
T
+ o
C CeOp
Σ dl
11
+0
Tioc T
H
M
+
T Tho
R
M
n
inera
V
T
C
h=1
h=1
h=
(55)
E.7 Recovering Prior Bounds
Sublinear Convergence. Similar to prior works in FL [42, 16, 60, 12, 89, 57, 11] we highlight that Algorithm 1 follows the best known convergence rate of (1/VT) for non-convex setting. Furthermore, in this section we provide sketch for recovering the approximate bound for other terms as seen in prior work:
Federated Averaging [1, 12]. Similar to analysis in [42] (see Remark about Theorem & 2 in [42]), setting 7 aa... = 1 does not recover the bound in Federated Averaging. However, starting with the final convergence bound of Thcorem
| (80)) )1
2 [ (0 - L (0')
E
16
T 0² Toc klob
+32H,,
T d² Ioc
T
Meron
CRop
H
H
+
RT, d, + 2 T2
MR Σ
16I 2 p T2 + C2
+ +PR
2
loc
h=l
h=
+ 4
2 T
H
R2 R
T-1
E.
Var tior
(/G//0) 1)]
Var, F
|||
G
[]]]
E2T
t=0
(56)
35

---

and substituting Igtan = 11/VT. Tr = =/ vT), of = 0 and Cj - oc, we get,
T-1
Σ
E
V (0())
(01
2 [ (ol0)) L (0*)
+ 16H
1 T lue o
T t=
loe
(1/VT)T
T
T
glob
T2
T2
H
+ 32H
les T
+
les
Σ
M
16L2
T + 4
T2
T2
T
TT2 loc
100
100
hm1
2 LL (0 L )
16Ho²
32Ho²
2
glob
Mj [161
[16L2/T 4P7]
VT
T
/=
(57)
Above can be rewritten using the big-O as:
T-1
|| (0(*) (0r)) s (
L (B 2 (0))
o²
0²
1 T
E
glob
los
(58)
T
VT
T
Similar to Theorem in [12], the above bound convergences at a rate of 0 optimization term and the update noises glob and other works [53, 16].
(1/VT) and O (1/T) for o2..- Similar convergence rates are also seen in
Adaptive Federated Optimization [42]. Starting with the final convergence bound of Theorem 2:
V (8()) )1)
2 [L (o( - L (0))
E Ioc
-16H1
2 T 0²
loc glob. +32H1) 2 T2 0²
T
nolah T
H
+ C2
R2 + 21/2 T
Σ Ma
16 n
1 + + 4P
22
loc
100 C2
h=1
h=1
+ Inc lnc.
1 T
H
R2 M
T
E
Var,
(/G//) )}
+ Var, (E,
[||
h.k ID)
22T
toc
100
1=1
t=0
(59)
and substituting Nata 1/VT, Tloe =1/ 111(11))) (1 ) oor -0 and C₁ 00, we get:
K
T
| (0(0) )1²
2 2L (010)) LL (0+)]
T2
E
<
+ 16/
T
10c
(1/VTYT
T2 T3/2" klub loc
0=0
T
T2 +2
T2
32H
Σ
MR
16I.
4P
7 T T3/2
loe
72 T3/2
72 T3/2
loc
2 Lf (0(0) L (0*)
16Ho²
32Ho²
2
H
glob
lse
M
26
3/2 + 4P2
VT
T3/2
73/2
T3/2
1=1
(60)
Above can be rewritten as using the big-OI 0 as:
|| (81)) (0=)) so ( L (0
L (0*)
0²
o2
K
E
slob
+
+
(61)
T
VT
T3/2
T3/2
T3/2
t=
Similar to Corollary 1 & 2 in [42], the above bound converges at a rate of 0 (1/VT) and 0 (1/73/2)
for optimization term and the global update noise x² slab respectively. However, it follows faster
convergence rate of (1/T3/2) for the local update noise 02 compared to rate of 0 (1/VT)
in [42].
36

---

Understanding Gradient Clipping in Private SGD [60]. bound of Theorem 2:
Starting with the final convergence
2 2 (0 - L (0*)
K
E lnc
VL
(000)
(
16H
2
T o² clab.
+ +32H11 T oz
T
Malor T
0=0
+ C
Σ
R + 21 T
Σ
Mi
16L 16L2')
The
7
+ 1P%
5²
loc
loc
h=1
1=1
+
Tha T
lnc
H
M
A
T
Ex
Var
(1
(G
(1) )]
+ Var E
||| h. ID)
22T
h=1
t=0
(62)
and substituting Nator 1/VT, Thoe = =1/(VTT 100): o2 =0 mmax (4;)) (per Theorem 1
in [8] and Lemma and Theorem from [46] to guarantee (e, 8)-riiaacy x² > const 2 1og
8"
while ODP = 2 S = max/= ww/ Rs = 1, Cs = C and D >m==1 dn. we get,
VH.
T-1
Σ
E
V (000)
)1
2 2 (0 - L (0*))
16H
T 02 T
T
l0e
(1/VTYT
T
glub
t
+ 32H
1 1
0²
+
1
0
(
(w:)) In 1
P2T
Σ dp
72 100
T
luo
$2
6²
hm
7
H
1
H +
+ loc
M
16L2
T
+
+4p%
T2
T
TT2 lac
C
h=1
+ 4
T2
lac
1
H
MR
T-1
E,
Var
Var, (E, 100
[|
[D)]
2
T
C
100
loc
h=1
t=
2 L (0
L (0+)
16Ho²
32110 Ior
DDC'max (wr) (2x))211 In 1
+ 0
VT
T
T
5282
2 H
16 1
H
+
Σ
2
+
+PP
T
T
C
h=1
4
HR
2
T-1
E
Var,,o t00
(1G%221) [)] + Var, (E
|| []]]
G
(0)
(63)
TT
h,k
Using the big-O, above can be rewritten as
| [I]
L (0(1)) L (0))
02
(wx)) (wx))) In
< 0
lnc
+
+ + CºT
(64)
VT
T
T
E²
By setting C' = 11/4, a similar convergence bound can be recovered up to a constant from Theorem 3.1 in [60] by choosing 77 = 1/T mh = //(T)/+Q), C = n (Q being analogous to T,e in our work), and P = 1 (P is analogous q, i.e., client sampling proportion in our work) and Wi = 1/K, though our bound has better rate of convergence for global and local update noise O (1/T) compared to O (1 (1/VT) in [60].
Inverse Relationship to Clipping Constant. While [60] analyzes clipping it does not highlight an inverse relationship with clipping constant C as seen in our work. Similar inverse relationships have also been highlighted in central optimizer analysis [58].
37

---

E,8 Adaptive Optimizers and Per-Layer Clipping: Theorem Under Limited Participation
Estimator with Bounded Sensitivity for FL with DP
It is common in several FL works [12. 51, 52, 16] to use weighted averaging of client updates given by:
jxct
(4)
Wkk
Wke
(e) (A hike + th,kk)
(65)
where Kt is a set of sampled users, Wa Σt= Dul and [D.] represents the cardinality of the data on client As discussed in prior work on moments accountant for DP [46], this estimator does not have a bounded sensitivity, thus ineligible for guaranteed DP privacy. The unbounded sensitivity can be intuitively seen via the case where all the sampled clients X have low number of data points thus
leading to an explosion the term E
Wk. Because this, our analysis uses the unbiased
sampling estimator from [46] which can be expressed as
|K
(8) Ri
(0) k,,k)
1
Σ WA
(1) A
(2) h.6
(66)
where q S/K, users are sampled i..dd with probability q from the population K and Tk Bernoulli(g) with [x60] the fact:
q. It can be seen that the "unbiasedness" of the estimator results from
K
K
E
W
(1) TR
E
w = q.
(67)
k=1
Also, while our algorithm and analysis use this general form of unbiased sampling estimator in Equation 66, our simulation experiments uniform averaging with W/ 1/K.
Theorem 3. Assume A1.1, A2.1, A2.2, and A3, Mo I. <1 and k [1 10(1 Tiw )3 > If the trust ratio from Eq. 10 in LAMB optimizer controlled in the Algorithm (global optimizer is LAMB and local optimizer is SGD) such that 1/ < R2 and || 1
< Pr B1 0 and 0 in LAMB optimizer, and clients are i.i.d. sampled with probability q. then after T steps of aggregation the performance of FL with DP, per-layer clipping and layer-wise gradient normalization is characterized by the following upper bound:
E
V
(1))
2 [ (0( - 2 (0*)
T2 0² 1/ glob + 40 loc
T22 o² loc
T
Maror
C C²o
H
H
p2
Σ
R2 dr + 5n T2 Σ
M 112212 ni T2 + 2qC
q
1=1
h=1
5 Nin Inn
T2
H
R 2
Var 100
(/c%//11] + Var, (E,
+aar,
G
(6)
(68)
2T
where (ateeoricallw),.... and E, H denotes the expectation over sampled mini-
butch g thaa every local step I 1, ...., T from the client data: ot (tihae)
(titrn))
DD, ay
E
(1100) Bg.
Proof. Using the unbiased sampling estimator from Equation 66 we start by bounding Zn- Bounding Z, Under Client Sampling
38

---

Under client sampling we have the aggregated clients updates A
(0)
in Equation 15 defined as:
A (1)
1
Wk W
(2) (A (1) =)
(69)
q
1=1
where << 1 in usual scenario and 7 ~ Bernoulligg). The definition of A in Equation 69 thus affects the bound on Zr, (from Equation 28) as follows:
V (0)))
Zn
E loc
w/
(1)
Σ
wg
h.k
(1)
x=1
K
K
Wk
7kk
Gh.k h.k
(t)
+ Σ
WA 7k (0 (0)
K
K
1
(6) (1)
(1) (1)
(
Wk 7k
Ph
h.k
7 Ph
9
k=1
K
1
(8)
(1)
(4)
(1)
(1)
(1)
Wk
7k
Ph
Wk
Ph
K
K
0
A x
W/
Ph
h,k
Ph
(70)
Bounding Term with DP Noise in Z₁ Under Client Sampling
(1)
R and random independent DP noise
Having upper bound on the expectation
Ph
2n N 0,
DF R wp per theorem condition (thus P/ and A (1) are independent vari- I0
ables), let's 's get the upper bound first for:
E
(6)
Σ W
P
(t)
E 1ic
Ph
(c)
Wk
(1) (t) 7kk
2i.k
du
(1) 2
(1)
(1)
E
Ph
Wk 7k
Bi.k
2
E
Ph
(0)
E
Wk 7k 24.k
(1) (8)
<
R
d
Σ
WE 7
(1)
Z
(0)
lnc
loe 74
1=1
R2
dg
0
9
K
WE(N (1)g2]
R
d, C2 0² 92
(71)
22
DP
5
K
1=
W
Bounding Z, with y, Y,, Y3. Y, and W Terms Under Client Sampling
39

---

Given Equations 29, 33, 30 and 31 (we use the fact that DP noise (0) kR, is zero-mean independent variable), we can bound 2,, in the following way:
K
2 S 5
R Ine
Σ
w/
V. (0%²) - GO
h,k
11
1 K
+ 5
(
Σ
(1) (t)
k
h,k
9 k=1
Wk E
[1 (G
(t)
(t)
Pa
(0)
A
(9)
$10 7k
5
W
Nk Ph
(t) (t)
o
A
(0)
-
1.k
Jue
+ 5
W
E
Tk
Ph
(2)
h.
(0
A
(1)
h.k
11]
+
R2 dg
o
100
1E
(1)
Ti 7
(8)
is independent variable from the gradients and deltas
S 5Y, 5W 5- Σ
wR E
[|(c/)
P (4)
K
+ 5
Wi
E
Ph
(2)
3
A b.k
A
(1)
h.k
Sac
k=
5
9 k=1
1
K
WE
E
100
Ph
(1)
3
A
h
(1)
A
E2
R
d
op..
(72)
Y,
K
W
Σ
Wh
(( h,k
1 (4)
(())
h,*
K
Eq.
||
(8)
7 k=1
W
E
200
7k
h.k
1
Σ
K
E
(1- 27
E
27
1 -
w
E
|, |2]
d2
The
1uc
q
Eq.
S Tho T, 2M77
(73)
Reusing bounds for Y1, Y2, Yo, Ya from the proof of Theorem 2 and having K )2]. we get:
[1 10(1 Thoc T me
2 2 (0l0) L (0+)
V (010)
E
<
20 T2 02 klob
+ 40H,,2 T2 0²
T
loc
11 ner. T
0=0
40

---

C o 2²
DP
Ri n,
5m
T
Σ
Mh
8L
T 100
+
Ph
+ 9
1
loc
2gC
q
h=1
n T
lue
H
R
T
Bx
Var
(|G////) [)] + Var, (E,
||
h. []]
(74)
Another Estimator for Limited Participation
In prior work [46] it was shown that another estimator can be used for weighted averaging of client updates under client sampling:
A
Wks
(4) A
+ Zhikk
(75)
max(qmin-
Wky
i=1
This estimator is not unbiased compared to the one use in Algorithm I and [46] gives the differential privacy guarantees for it too. To obtain the convergence bound for this estimator similar to Theorems 2 and 3 use the fact that (guin 9)
OX Σ
(9
(A + 2/i,ks))
Wko
(76)
Qmin
Having this bound we can repeat the same steps as we did for the unbiased estimator from Algorithm and get similar asymptotic bound as in Theorem 3 but with change of q 9min and different sensitivity bound for DP noise.
Empirical Analysis: Data and Central Models Training
Data We perform all experiments using two datasets of audio-tranee transcription pairs: LibriSpeech [61] and Common Voice v13.0 [62]. These datasets are read speech but differ in other properties. like data diversity, noise conditions, speaker variation, and speaker distribution. We not only present results with English locale in LibriSpeech and Common Voice v13.0 but also complement them with results on French and German locale from Common Voice v13.0. For LibriSpeech data, the original 16kHz sampling rate maintained, while for Common Voice we downsampled every audio to 16kHz sampling rate.
Every split LS and CV has separate set of speakers as well every validation and test sets have entirely different speakers from the train. Validation data are used to tune all hyper-parameter and to select the best models based on the word error rate (WER), while the test sets are used only for final evaluation. Statistics on the number of speakers and the number of minutes per speaker are given in Figure for both LS and CV datasets and their subsets. The statistics show that CV data are much more heterogeneous than L.S as highlighted by [20]. CV data thus enable a more realistic scenario for testing FL. and FL with DP. The most realistic scenario for uses a small central dataset to train seed model (e.g. LS-1 100), and larger dataset from different distribution for FI. (e.g. CV-en-train). All training subsets used in the empirical analysis and their statistics are listed in Table 2.
Token Set [82] showed that for data from different domains, character tokens more suited than word-pieces. Since in this paper we consider settings with data from different domains, the token set used in all our experiments is composed of English characters (a-z), augmented with word boundary token, hyphen and apostrophe, resulting in a total of 29 characters, For French and German, common -on-English characters included as well.
41

---

Table 2: Speaker statistics for LibriSpeech (LS) and Common Voice (CV) train sets and their subsets.
# minutes per speaker
Subset
# hours
speakers
mean
std
min
max
LS-100
100.6
251
24.1
2.7
5.5
25.2
LS-360
363.6
921
23.7
3.2
1.9
25.3
LS-500
496.9
1,166
25.6
5.9
3.0
30.3
LS-860
860.5
2,087
24.7
5.1
1.9
30.3
LS-960
961.1
2,338
24.7
4.9
1.9
30.3
CV-en-train
1593.7
34,753
32.7
0.02
5,049.6
CV-en-train-
149.5
3,475
2.6
17.3
0.03
755.1
CV-en-train-90
1444.2
31,278
2.8
34.0
0.02
5,049.6
CV-en-train-05
79.5
1,737
2.7
15,8
0.03
508.3
CV-en-train995
1514.2
33,016
2.7
33.4
0.02
5,049.6
CV-fr-train
727.9
6,856
6.4
57.2
0.04
3081.2
CV-fr-train-10
47.6
685
4.2
13.6
0.07
235.1
CV-fi-train-90
680.3
6,171
6.6
60.2
0.04
3081.2
CV-de-train
852.8
7,127
7.2
89.2
0.03
6249.9
CV-de-train-10
52.2
712
4.4
11.4
0.04
120.8
CV-de-train-90 800.6
6,415
7.5
94.0
0.03
6249.9
Data preprocessing For CV English, transcriptions are normalized similarly as for LS by (i) lower casing; (ii) removing punctuation while preserving hyphen; and (iii) converting non non-English characters into English ones with un unidecode" package. For CV French and German, we do not remove non- English characters and we retain single quotes.
Model We start our experimentation with the state-of-the-art model on LS-100 from [63]: (i) convolution to perform striding (kernel of 7 with stride of 3); (ii) transformer encoder with 36 layers, nost-LayerNorm, attention heads, an embedding dimension 768, an MI.P dimension of 3072, dropout and layer drop [90] of 0.3; and (iii) a linear layer to map the target vocabulary. The resulting model has 255M trainable parameters. We focus only on a CTC model as it contains only the encoder part, is simpler to train in practice compared to Seq2Seq or Transducer models, and less likely to over-fit to the language model [25].
Positional Embedding To reduce model training time by a factor of approximately 2-3 and to reduce the memory footprint, embedding [92]; both models perform similarly.
CAPE positional embedding [91] instead of relative positional
SpecAugment SpecAugment [64] is activated from the very first step of training. Two frequency masks with frequency mask parameter F 30, ten time masks with maximum time-mask ratio = 0.1 and time mask parameter T = 50 are used; time warping is not used.
Training We train models on 8 GPUs (A100 80GB). and use dynamic batch size of ~ 240s audio per GPU. For central models training, we use LARS optimizer with the learning rate 0.5 (for models fine-tuned from seed models trained on CV. /0 we use 0.2) without warmup period. Training is done for up 300k-600k steps until full convergence with step-wise (by 2x) learning rate decay every 50k steps started at 40k-330k depending on the model.
G Empirical Analysis: Federated Learning without Differential Privacy
G.1 Hyper-parameters
All dropout and layer drop are fixed to 0.3 We train each client with dynamic batch size of total 120s of audio (CV) or 360s of audio (LS). Figures and we use the same LR and LR decay schedule for all seed models regardless of the cohort size or the data used to train seed model. Optimal hyper- parameters (e.g. LR) are likely to depend on the quality of the seed model and cohort size. Thus, the results could likely be further improved by tuning the LR and its decay schedule for
42

---

Table Results (WER %) on LS. All runs use exponential decay for central LR starting at iteration 1,000, decay rate 0.6, and transition steps 500 (w/o seed model) or 250 (w/ seed model). Local learning rate is 0.4 (w/o seed model) or 0.2 (w/ seed model). Central learning rate 0.006 (w/o seed model) or 0.003 (w/ seed model). The number of central steps is T = 2k and the number of local epochs is 10.
None;
LS-860
central
sead
central
centrl
dev-clcan
100.0
6.6
1.8
4.0
3.3
2.7
6.2
3.3
3.1
2,9
2.7
2.7
16.5
4.0
3.6
3.3
2.9
3.1
test-clean
100,0
6.7
5.1
4.2
34
2.8
6.7
34
3.2
3.0
2.9
2.8
15.5
4.3
38
3.5
322
3:2
dev-other
100.0
19.2
6.9
9.6
test-other
100.00
8.8
19.5
10.3
9.4
8.6
7.8
Table 4: Results (WER %) on CV. We use exponential decay for central LR starting at t = 1,000 (w/o seed model) or 750 (w/ seed model), decay rate 0.6, and transition steps 500 (w/o seed model) or 750 (w/ seed model) with T = 2k total central steps and 10 local epochs. Local (central) LR is 0.4 (0.006) (w/o seed model) or 0.2 (0.002) (w/ seed model).
seed
cohort size WER
central
Sced
Data
Eval.
WER
8
16
32
64
128
256
WER
dev 100.0
62.9
51.9
41.3
32.9
27.2
21.3
15.1
None
CV-en
test
100.0
66.7
56.5
46.3
38.0
31.9
25.7
18.2
dev 31.3
26.6
24.3
22.7
21.2
19.8
18.2
15.2
CV-en-U1
CV-en-95
test 36.4
31.6
28.9
27.0
25.4
23.8
22.1
18.3
dev 23.0
20.3
18.9
17.7
16.7
15.7
14.8
14.5
CV-en-10
CV-en-90 test 27.9
24.4
22.8
21.5
20.1
19.1
18.0
17.6
dev
34.7
24.5
22.2
20.1
18.4
16.8
15.6
14.7
LS-100
CV-en
test
61.2
28.8
26.3
23.9
22.0
20.2
18.9
17.8
dev
27.0
19.7
18,1
16.9
15.6
14.5
13.7
14.1
LS-960
CV-en
test
31.5
23.5
21.6
20.2
18.8
17.6
16.6
17.2
Table Impact of randomizing the distribution of data across users for LS measured by WER (%).
Parameter settings are described in Table 3. While the original train data are non-IlI, IID (columns with "IID") versions LS-960 and LS-860 are created by choosing a user id uniformly and randomly from the set of user ids for each data point in the corresponding dataset.
Data
sood: None; trin: 15-060
sead: 1.5-100, trin: 1S-860
sood: CV-en; train: 1.5-000
seod
8-1ID)
16
16-IID
central
soed
8
S-IID)
16
16-IID
central
soed
8
5-IID
16
16-1ID
central
dev-clean
100.0
6.6
5.3)
4.8
4.5
2.7
8.2
3.3
3.3
3.1
3.6
2.7
16.5
4.0
3.9
3.6
3,5
3.1
tost-clean
100.0
6.7
6.0
5.1
4.7
6.7
2.8
3.4
3.3
3.2
3.1
2.8
15.5
4.3
4.1
3.8
3.7
3.2
dev-other
11.2
6.7
7.4
6.9
10.5
9.6
8.8
test-other
10.9
10.3
9.4
each cohort size and seed model separately. Furthermore, we can improve models by longer training exceeding 2k central steps as shown in ablations in Appendix G.8, Table 14.
G.2 Detailed Results for English
Table 3 details the results for LS from Figure 2 and Table 4 details the results for CV from Figure 3. Table 5 details the results for randomized LS dataset (IID) from Figure (left and middle). Table details the results for randomized CV dataset (IID) from Figure 4 (right).
G.3 Impact of Model Architecture on FL Performance in ASR
Table 7 compares several model architectures for the trivial FL scenario with cohort size and 64k central iterations on L5-100. Cohort size of is impractical but it eliminates the impact federated averaging. The learning rates and learning rate decay schedules are tuned for each architecture. During preliminary FL. experiments we have observed that Pre-LayerNorm models often perform better than post-LayerNorm ones. It is of note that without linear central learning rate warmup,
43

---

Table 6: Impact of randomizing the distribution of data across users for CV measured by WER (%). Parameter settings are described in Table 4. While the original train data are non-llD, the IID (columms with "IID") version of CV-en-train created by choosing a user id uniformly and randomly from the set of user ids for each data point in the corresponding dataset.
seed
cohort size WER
central
Sced
Data
Eval.
WER
16
16-1ID
32
32-IID
WER
dev
100.0
51.9
50.2
11.3
40.9
15.1
None
CV-en
test
100.0
56.5
55.0
46.3
45.8
18.2
dev
54.7
22.2
21.1
20.1
19.1
14.7
LS-100 CV-en
test
61.2
26.3
25.0
23.9
22.7
17.8
Table 7: Comparison (WER, %) between pre-LayerNNom and post-LayerNorm architectures in transformer for trivial FL. scenario with cohort size S and central steps 64k on I.5-100. pre-LayerNorm models perform best and their training robust with respect to hyper-parameters such as the learning schedule. Central models are trained according to Appendix F. FL models use exponential learning rate decay, LAMB central and SGD as local optimizers.
Model
Warmup
dev-clean
dev-other
test-clean
test-other
Central pre- NayeLoa
0
5.9
18.9
6.4
19.2
Fl. pre-IayerNoom
0
5.6
17.7
5.9
17.9
Central post-LayerNorm
0
8.1
25.0
8.6
25.6
FL post-LayerNorm
1000
5.9
17.5
6.3
18.0
Table 8: Comparison (WER, %) of various server optimizers LS-960 with and without seed model. For LAMB, the results and parameters are the same as those in Table 3 (note that these are sub-optimal because for simplicity for each configuration regardless of the cohort size and all runs with seed models use the same configuration). For all other optimizers, the central learning rate and the learning rate decay schedule are tuned separately for each combination of cohort size and seed model.
the same learning rate and learning rate decay schedule
Cohort
Central
dev-clean
test-clean
dev-other
test-other
Seed
Data
size
optimizer
LR
T'= 0
T = 2k
T'= 0
= 2k
T
= 2k
T
T 2k
LAMB
0.006
100.0
100.0
100.0
17.2
100.0
17.5
None
LS-960
8
LARS
0.7
100.0
13.7
100.0
14.1
100.0
30.9
100.0
31.6
Adam
0.001
100.0
14.1
100.0
14.6
100.0
30.4
100.0
31.0
LAMB
0.006
100.0
4.8
100.0
5.1
100.0
13.5
100.0
13.7
None
L.
16
LARS
0.7
100.0
10.5
100.0
11.0
100.0
25.9
100.0
25.9
Adam
LAMB
0.003
16.5
4.0
15.5
25.2
10.5
25.9
10.3
CV-en 1.5-960
8
LARS
1.2
16.5
4.2
15.5
25.2
10.6
25.9
10.6
Adam
0.012
16.5
4.3
15.5
4.3
25.2
10.7
25.9
10.5
were unable to train reasonable FL models with post-LayerNome Our experiments showed that FL models with pre-LayerNorm are easier to train, they do not require a central learning rate warmup, and they are generally more robust with respect to hyper-parameter These observations are similar to prior works on transformers central training [55, 31]. That is why we use the pre-LayerNorm configuration for all experiments in the paper. It is interesting that for this trivial FL scenario FL models outperforms centrally trained models. However, when we switch to larger LS-960 dataset, this does not hold anymore.
G.4 Impact of Server Optimizer on FL Performance in ASR
Table 8 compares the LAMB optimizer [49] used as the central optimizer in all FL runs presented so far with Adam [93] and LARS [65] on several configurations for LS-960 dataset. The results on I-S9600 indicate that LAMB performs significantly better than LARS and Adam without seed model, and it performs slightly better than LARS and Adam with seed model. Adam performs slightly better than LARS.
44

---

Table Comparison (WER, %) of various optimizers on CV-en with and wihout seed models. For LAMB, the results and parameters are the same as those in Table 4 (note that these are sub-optimal because for simplicity we use the same learning rate and learning rate decay schedule for each configuration regardless of the cohort size and all runs with seed models use the same configuration). For all other optimizers, the central learning rate and the learning rate decay schedule are tuned separately for each combination of cohort size and seed model.
Seed
Data
Cohurl
Central
dev
lest
size
optimizer
LR
T'
T
2k
T
0
T 2k
LAMB
0.006
100.0
62.9
100.0
06.7
LARS
3.4
100.0
70.4
100.0
73.8
None
CVien
Adam
0.0005
100.0
68.9
100.0
72.2
AdaGrad
0.003
100.0
100.0
86.2
SGD
2.8
100.0
83.8
100.0
86.0
LAMB
0.006
100.0
51.9
100.0
56.5
LARS
2.6
100.0
37.6
100.0
62.0
None
CV-en
16
Adam
0.0005
100.0
57.7
100.0
62.1
AdaGrad
0.002
100.0
82.1
100.0
84.5
SGD
3.0
100.0
100.0
86.6
LAMB
0.002
23.0
27.9
23.5
LARS
0.3
23.0
18.7
27.9
22.6
CV-en-10
CV-en-900
8
Adam
0.004
23.0
18.9
27.9
22.9
AdaGrad
0.016
23.0
19.4
27.9
23.6
SGD
1.6
23.0
20.9
27.9
25.4
LAMB
0.002
23.0
18.3
27.9
22.1
LARS
23.0
18.0
27.9
21.8
CV-en-10 10 CV-en-90
16
Adam
0.006
23.0
18.3
22.1
AdaGrad
0,015
23.0
19.1
27.9
23.2
SGD
1.6
23.0
20.8
27.9
25.2
LAMB
0.002
23.0
17.3
27.9
21.0
CV-en-10
CV-en-90
32
LARS
0.6
23.0
17.3
27.9
20.9
Adam
0.006
23.0
17.5
27.9
21.1
LAMB
0.002
23.0
16.7
27.9
20.1
CV-en-10
CV-en-90
64
LARS
23.0
16.6
20.1
Adam
0.008
23.0
16.4
27.9
20.0
Table 9 compares LAMB with Adam, AdaGrad [94], LARS, and SGD [66] on several configurations for CV-en dataset. The results on CV show that without seed models, LAMB performs significantly better than other optimizers with seed models, LAMB sometimes outperformed slightly by LARS and Adam. SGD, AdaGrad and Adam are outperformed by LAMB and LARS in almost all scenarios.
During hyper- parameter tuning, some adaptive optimizers (e.g., Adam) often became unstable and the training diverged, especially without a well performing seed model. Furthermore, the optimal parameters of these optimizers oftentimes vary significantly between, e.g., the cohort sizes, indicating that they are less robust than LAMB in our setting.
The robustness of LAMB across all scenarios and its stability are the main reasons for choosing LAMB the central optimizer for most of the experiments the paper. However, the results in Table suggest that some of the models could be further improved with more hyper-parameters tuning and choosing the best optimizer for each case. Also, [56] showed that tuning other optimizer parameters, e.g. e in Adam, can significantly improve FL. model training for ASR. However, in this paper we restrict ourselves tuning only the learning rate and learning rate schedule; the remaining parameters were set to their default values from optax library".
We have not completed extensive evaluation of other optimizers for local training to keep it efficient (no state, no additional memory, no extra computations): SGD as a local optimizer robust and efficient in experiments. However, preliminary experiments show that LARS and LAMB well suited candidates for replacing SGD as the local optimizer and will likely outperform SGD.
For completeness, here we provide more details optimizer tuning. For both IS-960 and CV-en- train without a seed model, we tuned the central LR for LAMB between 0.001 and 0.009, and the
45

---

Table 10: Results (WER, %) on CV for English, French and German. Configurations are identical to those in Figure 3 and Table 4 regardless of the language.
seed
cobort size WER
central
Seed
Data
Eval.
WER
16
32
64
128
WER
dev
100.0
62.9
51.9
41.3
32.9
27.2
15.1
None
CV-en
100.0
66.7
56.5
46.3
38.0
31.9
18.2
dev
100.0
34.7
25.4
18.8
15.0
12.6
10.7
None
CV-fr
100.0
38.7
29.1
21.6
17.5
14.8
12.2
dev 100.0
30.1
22.8
16.1
11.7
9.5
7.7
None
CV-de
test
100.0
32.8
25.5
18.3
13.4
10.9
8.8
CV-en-10
CV-en-90
dev
23.0
20.3
18.9
17.7
16.7
15.7
14.5
test
27.9
24.4
22.8
21.5
20.1
19.1
17.6
dev
24.0
15.6
14.3
13.2
12.0
11.2
10.8
CV-fr-10
CV-fr-90
test
27.5
18.1
16.6
15.3
14.0
13.1
12.6
CV-de-10
CV-de-90
dev
18.6
12.8
11.4
10.2
9.1
8.1
8.1
test
21.2
14.7
13.1
11.7
10.5
9.3
9.2
local LR for SGD from 0.2 to 0.6. We have done the same for one selected seed model for each dataset. Additionally, we tried several learning rate schedules, including constant rate, step decay, and exponential decay on several configurations. After the initial experiments, we chose one configuration for each dataset (LS, CV) without seed model and one configuration for each dataset (LS, CV) with seed model, and we ran the remaining experiments with the chosen configurations. The initial tuning was done smaller cohort sizes. For other optimizers discussed in this section, we tuned the key parameters until a locally optimal value was found for central LR for each presented experiment, and we considered 4 variations of the exponential decay rate for each LR value.
G.5 Detailed Results for CV French and German
Table 10 shows the results of FL on CV for French and German languages, and for comparison provides the corresponding results on CV for English. To demonstrate that the settings used for English language were robust, we did not tune any parameters for French and German, and simply used the exact same configuration that was used in the corresponding training on English language.
The results show that even though French and German have considerably smaller datasets, the training is apparently considerably easier and WERs are significantly smaller than for English whether or not seed model is used. This is likely due to the degree of consistency between the orthography and phonology as was discussed for example in [95, 96, 97, 98]; German and French consistency than English. Furthermore, the results for
have stronger aathorraphy-o--honookee French are considerably better than those presented by [20]. As French and German data are smaller, for the same cohort size and central steps we do more epochs over data for French and German than for English CV. Thus, FL training can match the central training with smaller cohort size for both French and German compared to English. It is of note that French and German turn out to be easier also for FL with DP shown in Appendix H.6, Table 18.
G.6 Impact of SpecAugment
In all experiments so far, we used SpecAugment [64] activated from the very first step of training as was also common in most prior works. Table 11 shows the results with and without SpecAugment for several configurations analyzed in this paper on LS data. These results confirm that SpecAugment improves WER in all the cases.
However, Table 12 shows that SpecAugment appears to have a negative impact on the trained models for CV (English), especially for FL training without seed model and small cohort sizes. This is surprising as prior works reported only improved results with SpecAugment for transformer models. These results also reveal another difference between benchmarks on LS and CV.
is possible that the results with SpecAugment on CV would improve if SpecAAgment was turned on later in the training and its parameters were tuned for each scenario separately. Nonetheless, since in
46

---

Table 11: Results (WER, %) on LS with and without SpecAugment [64]. Configurations are identical to those in Figure 2 and Table except the SpecAugment schedule as noted in the table.
Seed
SpecAugment
Cohort
dev-clean
lest-clean
dev-other
test-other
size
T
T 2k
T
0
T
2k
T
T
2k
T
0
T
2k
None
LS-960
V
100.0
6.0
100.0
6.7
100.0
17.2
100.0
17.5
None
LS-960
100.0
100.0
100.0
100.0
19.4
/
100.0
100.0
100.0
13.7
None
LS-960
X
16
100.0
5.4
100.0
5.5
100.0
16.5
100.0
16.5
LS-100
LS-860
6.2
3.3
6.7
3.4
19.1
9.4
19.5
9.0
LS-100
LS-860
x
6.2
3.3
6.7
3.3
19.2
10.2
19.5
9.8
LS-100
LS-860
/
6.2
3.1
3.2
19.1
8.5
19.5
LS-100
LS-860
6.2
3.2
19.1
9.9
19.5
CV-en
LS-960
V
16.6
4.0
15.5
25.2
10.5
25.9
10.3
CV-en
LS-960
X
16.6
3.8
15.5
4.1
25.2
11.5
25.9
11.2
CV-en
LS-960
V
16
16.6
3.6
15.5
3.8
25.2
9.6
25.9
9.4
CV-en
LS-960
16.6
3.5
15.5
3.8
25.2
10.9
25.9
10.6
Table 12: Results (WER, %) on CV with and without SpecAugment [64]. Configurations are identical to those in Figure and Table 4 except the SpecAugment schedule noted in the table.
Seed
Data
SpecAugment
Cohort
dev
test
size
T'
T 2k
T
' 2k
None
100.0
100.0
None
CV-en
8
100.0
52.3
100.0
37.5
None
CV-en
16
100.0
51.9
100.0
56.5
None
CV-en
100.0
100.0
47.9
None
CV-en
32
100.0
41.3
100.0
46.3
None
CVen
32
100.0
33.8
100.0
39.3
CV-en-10
CV-en-90
23.0
27.9
24.4
CV-en-10
CV-en-90
8
23.0
19.9
27.9
21.3
CV-en-10
CV-en-90
16
23.0
18.9
27.9 9
22.8
CV-en-10
CV-en-90
22.4
CV-en-10
CV-en
32
23.0
17.7
27.9
21.5
CV-en-10
CV-en-90
32
23.0
17.1
27.9
21.2
1.5-100
LS-100
CV-en
54.7
23.3
61.2
27.9
LS-100
CV-en
16
54.7
22.2
61.2
26.3
LS 100
CVen
LS-100
CV-en
32
54.7
20.1
61.2
23.9
I.S 100
CVien
32
54.7
19.0
61.2
23.0
I. 960
CV
23.5
LS-960
CV-en
8
27.0
19.5
31.5
23.5
LS-960
CV-en
16
27.0
18.1
31.5
21.6
LS-960
CV-en
LS-960
CV-en
32
27.0
16.9
31.5
20.2
LS 960
CV-nn
32
27.0
16.4
31.5
20.2
most scenarios Augment either improved models or the differences were marginal, for simplicity, we use SpecAugment in all experiments in this paper.
G.7 Performance of FedProx in FL for ASR
[11] proposed FedProx to alleviate the impact of heterogeneous data on FL performance. Since the results presented earlier in Tables 5 and suggested that heterogeneous data pose a challenge for FL also our training, we also evaluate the impact of FedProx on model quality in ASR. For each config- uration, we FedProx with the regularization weight μ {0.00001, 0.0001, 0.001, 0.01, 0.1, 1.0) and chose the best result, as suggested by [11].
Table 13 presents the results of using FedProx in several scenarios LS and CV datasets presented earlier Tables 3 and 4. The results show FedProx improves model performance (WER decreased) in 8 out of 10 training configurations tested, although in most cases the improvement is marginal.
47

---

Table 13: Results (WER, %) of FedProx on selected configurations on LS (top) and CV (English) (bottom) datasets. All parameters except for FedProx f are identical to those in Tables 3 and 4. Parameter l E {0.00001, 0.0001, 0.001, 0.01, 0.1, tuned separately for every case and the best result is provided for each base configuration
Cohuri
dev-clean
lest-clean
dev-other
test-other
Sced
Data
fedprox j
size
T 0
7 = 2k
T=
T = 2k
7
T
2k
T 0
T 2k
None
LS-960
100.0
6.6
100.0
6.7
100.0
17.2
100.0
17.5
None
LS-960
0.1
8
100.0
6.4
100.0
6.7
100.0
17.5
100.0
17.5
None
LS-960
0
16
100.0
4.8
100.0
5.1
100.0
13.5
100.0
13.7
None
LS-960
0.1
16
100.0
4.9
100.0
5.1
100.0
13.4
100.0
13.5
LS-100 100
LS-860
0
6.2
3.3
6.7
3.4
19.1
9.4
19.5
9.0
I.S1000
IS-860
0.0001
6.2
3.3
6.7
3.5
19.1
9.3
19.5
9.0
LS-100
LS-860
16
6.2
3.1
6.7
3.2
19.1
8.5
19.5
8.3
LS-100
LS-860
6.2
3.2
19.1
8.6
Cohort
Central
dev
Seed
Data
fedprox j
size
LR
T
T 2k
T 2k
None
CV-en CV-en
0.006
100.0
62.9
100.0
66.7
None
0.01
0.006
100.0
63.4
100.0
67.4
None
CV-en CV-en
16
0.006
100.0
51.9
100.0
56.5
None
0.0001
16
0.006
100.0
51.0
100.0
55.8
None
CV-en
32
0.006
100.0
41.3
100.0
46.3
None
CV-en
0.0001
0.006
100.0
40.0
100.0
44.9
I.S-100
CV-en
0.002
54.7
24.5
61.2
28.8
CV-en
0.002
54.7
61.2
LS-100
CV-en
0.002
54.7
22.2
61.2
26.3
LS-100
CV-en
le-05
0.002
54.7
22.0
61.2
26.3
LS-100
CV-en
0.002
54.77
20.1
61.2
23.9
LS-100
CV-en
0.1
32
0.002
54.7
20.1
61.2
23.9
In of the remaining cases there is change and only in one case the results with FedProx considerably worse than without
It is surprising how the optimal value of the key FedProx parameter μ changes considerably between the various scenarios. This suggests that would make sense to evaluate adaptive μ as suggested by [11]. We leave the use of adaptive μ and the investigation of how FedProx may improve FL training robustness (e.g. with respect to the number of local epochs or steps) for future work.
We also tried limiting the number of batches processed on each client [12] and normalizing users' deltas sent to the server [13] but neither approach improved the results. See Table in Appendix H.2 for the results on limiting the number of batches (steps) processed for each client.
G.8 Extending the Number of Central FL Iterations
Table 14 shows that even though most FL models were stopped after 2k central steps, letting these models to train longer would further improve performance. However, due to the communication complexity for each central step, best use moderate number of central steps and maximize utility of the training by optimizing the parameters for local, on-device training, cohort sizes, and other key FL parameters.
G.9 Impact of Under-Trainee Seed Models
Table 15 shows that choosing a better seed model improves performance across the board. Further- more, the results presented previously in Table 4 show that using a seed model trained on more data improves FL performance, even if the data used to train seed models are from different domain.
48

---

Table 14: Results (WER, %) on selected FL configurations on CV obtained after T 4k central steps and their comparison to those obtained after T 2k central steps. All parameters are identical to those Table
Cohort
dev
test
Seed
Data
size
T
T
2k
T
1k
T
T
2k
T 1k
None
CV-en
16
100.0
51.9
43.3
100.0
56.5
48,3
None
CV-en
32
100.0
41.3
34.0
100.0
46.3
38.9
None
CV-en
64
100.0
32.9
27.3
100.0
38.0
32.0
CV-en-10
CV-en-90
16
23.0
18.9
17.8
27.9
22.8
21.4
CV-en-10
CV-en-90
32
23.0
17.7
16.9
27.9
21.5
20.4
CV-en-10
CV-en-90
64
23.0
16.7
16.0
27.9
20.1
19.4
LS-100
CV-en
16
54.7
22.2
19.9
61.2
26.3
23.7
LS-100
CV-en
32
54.7
20.1
18.2
61.2
23.9
21.8
LS-100
CV-en
64
54.7
18.4
16.8
61.2
22.0
20.2
Table 15: Impact of mnder-trained seed models on WER of the final model for CV dataset with LS- 100 seed and cohort size 32. The under-trained seed models are obtained from the first steps of the baseline central training used to generate the actual seed model. The parameters for the experiments without seed models and the one with the high quality seed model are the same as Table 4. The parameters for the seeds of lower quality the same those without seed model.
dev
Lest
Seed
T
T
2k
T
T
2k
None
100.0
39.9
100.0
44.7
LS- 100 (30 steps)
98.9
37.7
100.0
42.8
LS-100 (50 steps)
83.2
32.8
87.8
37.8
LS-100 (70 steps)
75.9
33.3
81.1
38.2
LS-100 (full)
20.1
61.2
23.9
Table Results (WER %) on CV for different cohort sizes. We use exponential decay for central LR starting at t = 750, decay rate 0.6, and transition steps 750 with T = 2k total central steps. Local (central) LR is 0.2 (0.002). All models are trained with the same hyper-parameten
only the cohort size is varied. for left half of Table with cohort size from 8 to 256 we use 10 local epochs, while for the right half of the Table we use 10 local steps to scale efficiently on GPU to 256-5120 cohort sizes. Central models are trained either with the batch discussed in Section F or with 3x batch size, shown in brackets (all other hyper-parameters are the same as in Scction F).
Seed
Data
Eval.
seed
cobort size WER
central
WER
16
32
64
128
256
256
512
1024
2048
3072
4096
$120
WER
LS-100
dev
54.7
24.5
22.2
20.1
18.4
16.8
15.6
15.6
16.8
15.7
14.9
14.5
14.3
(12.7)
test
28.88
26.3
23.9
22.0
20:2
18.9
18.9
17.4
17.2
175.6
LS-960
CV.en
dev
27.0
19.7
18.1
16.9
15.6
14.5
13.7
18.0
14.6
13.9
13.6
13.0
12.8
12.7
14.1
lest
31.5
23.5
21.6
20.2
18.8
17.8
16.6
21.5
17.6
16.7
16.3
15.7
15.5
15.4
17.2
G.10
Scaling to Larger Cohorts
We further scale the cohort size to check limitations on the cohort size and scaling of FL: for efficient GPU utilization we switch from 10 local epochs to 10 local steps for cohort sizes of 256 to 5 5120, while keeping other hyper-parameters the same. Results are shown in Table 16: FL scales to larger cohort sizes lowering further WER. There is also observed degradation by switching from local epochs local steps especially for a stronger seed model, likely due to overfitting to the seed model's 's data, which are out-of-domain data with respect to the FL data.
49

---

H Empirical Analysis: Federated Learning with Differential Privacy
H.1 Differential Privacy Noise Discussion
There are different equivalent formulations how the noise can be added to the clients' deltas to introduce DP, which can cause confusion about the noise scale, and how the moments accountant is applied. Having Algorithm step 12 can be defined as follows:
1. Noise is added on the client level: A = 1/9 E kEXt We A + NNO 1C% (essnt)) We use this definition with Celient a
It was also used by [55].
2. Noise is added on the server level after averaging clients' deltas: A 1/4 [= w
N(0, aag)) This is the definition used by [46].
3. Noise is added on the server level after summation but before normalization to the number of clients (used by [8]): A = 1/( (Exex) Kwy A + N(0, o 2.m))
Different variants of noise are connected with each other via 5um = Javg qK = Javy S
and ddient Gavg
E mi w Then can compute that dDp Gavg in this notation from
Algorithm 1.
Throughout the paper we use Wi = K and the moments accountant implementation from opacus [99] which works with sum noise definition. To re-sclle noise added to each client in order to be consistent with opacus, we re-seale it by multiplying by the cohort size S. Thus, we get Theorem where
defined as a sum and, finally, we get the bound on opp via Theorem from [8] which gives
bound on z² 2 const log 1/5 in our notation. In all experiments with FL with DP, we use the same privacy budget for every training step.
H.2
Large Cohort Training Implementation
Our initial FL implementation processed the clients in each cohort sequentially, potentially paralleliz- ing the training for each client using multiple GPUs. For each client, we train a local model for given number of epochs. However, this approach does not scale well training with large cohorts, e.g. 1,024, which were necessary for experiments with FL with DP.
That is why we implemented another version where every client is trained on GPU and train the models for several clients in parallel utilizing all available GPUs (e.g. with 32 GPUs we can process 32 clients in parallel). To do that efficiently with highly imbalanced data like CV where some clients have much more data than others, we restrict the training on every client to a pre-defined number of training steps (batches processed) instead of epochs. Switching from fixed number of epochs to a fixed number of training steps per client was previously reported to improve performance in the presence of data heterogeneity [12].
Since we always use dynamic batching for efficient implementation and the average number of minutes of audio per client in CV is 2.5, FL training with 10 local epochs and total dynamic batch of 2 minutes per client can be approximated with 10 local steps and the same batch size. This configuration used in all FL with DP experiments.
Unlike reported by [12], we did not observe improved performance after switching the number of local steps but instead observed degradation in performance: see Figure 6 for the results on one configuration of CV with LS-100 seed model. However, it is of note that the differences will likely get smaller with larger cohort sizes. From results in Figure 6, we expect that more local compute that would be feasible in a real deployment, should lead to better results than what we get in our experiments for FL with DP.
H.3 Empirical Analysis
For FL training with the large cohort size of 1,024, the client delta norms are already bounded due to the local clipping (see Algorithm I, line 10) done in each step the local training for every client (see Figure 7). Local clipping is necessary part of the training because otherwise the local training
50

---

-Th- Limk ocal numberof (oasches)
Baselline (local rum esodhc-10)
Text 1E55 (56)
25
25
50
75
100
12.5
150
175
200
Local number stops
Figure Comparison of WER for FL training between local number of steps (solid) and local number of epochs (dashed). Training is done CV-en-train with seed model pre-trained centrally LS- LSO1 The cohort size is S = 64, total number central steps is T 2k, and other parameters set the same as in the corresponding configuration in Figure 3.
0.64
0.62
[[A]]]
0.58
500
1000
1500
2000
Figure 7: Client's delta norm averaged per clients throughout FL training with the cohort size of 1, 024 on CV-en-train from a seed model trained on LS-100. We use exponential decay for central LR starting at 750, decay rate 0.6, and transition steps 750 with T 2k total central steps and 10 local steps. Local (central) LR is 0.2 (0.002).
15-
10.0
0.50
10
144
0.5
fecer Blak Endox
Transome Blak
Tran femee Blak Indox
Tranimee Blci Indox
102
132
100
137
130
(top) averaged across training steps and (bottom) showed per layer along the training. The model is trained with LARS optimizer and the learning rate of 0.5. The norms of the per-layer gradients are balanced differently compared to models trained with FL or with FL and DP in Figure 10, e.g., LayerNorm gradients do not dominate over MLP and attention gradients.
Figure 8: Central training from scratch CV-en-train and its per layer gradients
of the transformer model would not converge [31, 30). This is similar to the standard recipe for the central training of transformer models.
As discussed in Section 4.3, we varied the clipping bound C for clients' deltas and did not observe any impact on the final performance even when = 10-8 We also did not observe the difference between training with the full precision (float32) or training with the reduced precision (bloat 6). The LAMB optimizer's & 10-6, thus was leading term in the denominator during optimization when clipping C' < 10
51

---

bal
Ialocade
1.00
5.75
0000
Thanchoree Riece Tides
Transforea Dlack Srdie
Thandtmmr Block Btdee
Thanshemar Blacc Indoo
3:130a
10"
10F
10"
Lecighe
Light niggh
K2occgll
20003
10000 30390
20000
2000
Thanng
Transea
Figure 9: Central training CV-en-train from the LS-100 seed model and its per layer gradients norm: (top) averaged across training steps and (bottom) showed per layer along the training. The model is trained with LARS optimizer and the learning rate of 0.5. The norms of the per-layer gradients are balanced similarly to models trained with FL or with FL. and DP in Figure 10: LayerNorm gradients do dominate over MLP and attention gradients.
0.125
0002
wloaph
0.008
10
0005
0.00
0.006
0,934
Hrtuha
0.001
0.
D4225
0002
0.002
5e
0:00
0.000
Trandorre Mack
Tranfforee Nucic bakes
Trandiirrer Block Indos
Thandorre Bakkk
0.125
100
0.000
0.000
0.077
0.000
00000
0.000
0.025
0.05
Traefarner Bloc Tudee
Tranchmee Slack Irvie
Tranfrone Block Iades
Tranclrrrr Blick Indie
0.123
1s1. Betk
1n2
0.10
0.006
0.08
vindgl:
01075
0.005
DU000
0.004
0.003
0.000
00002
0.005
00000
0.000
10
20 Tranimmrr Black
Trandommrr
0.005
wiggle
0.0125
ooha ghi
bdoak
windgs
9.30
0/00
0.31
0.0090
0.977
00000
0.0000
100
Figure 10: Client delta norms computed per layer in the model. We average the statistics across all clients and central steps, and plot the mean and standard deviation. The model is trained with (first row) global clients' deltas clipping C' = 10-2 and ODP = 0, (second row) global clients' deltas clipping C = 10-6 and oDp = O, (third row) per-layer clients' deltas clipping (Definition 3, "'nifform") ") C 10-3 and oDp = 0, and (fourth row) per-layer clients' deltas clipping (Definition 3, "uniform") C 10-2 and ODP = 3 10 10-6 The rest of the training configuration is the same in Figure 5. A transformer block consists of attention parameters (wqkv and wf). MLP (w1 and w2), LayerNorm applied input of attention (In1) MLP (ln2). The statisties are consistent with the training with global clipping (Algorithm 1) in Figure 5.
52

---

We assume that the seed model is trained centrally without DP (e.g. LS-100) after which FL with DP run on CV-en-train by initializing FL model with the seed model. When we add DP noise to the training alongside with the clipping of clients' deltas, we also did not observe any difference in the training dynamic and final performance (WER) as long as Copp remained constant (e.g.. halving the clipping bound C' and halving the noise DP would produce a nearly identical model). We hypothesise that this is the outcome of (i) above observation that clipping does not affect training; and (ii) using LAMB as central optimizer, which performs LARS per-layer scaling, and scales both the noise as well as the signal in the same way.
As discussed in Section 4.3, we observe clients' deltas imbalance across different layers of the transformer model (see Figure 5). The first layers (1-10 transformer blocks) have higher delta norms than the last layers (20-36 transformer blocks) for LayerNorm in MLP part and attention final linear projection. This is the opposite behaviour than observed in the deep models, e.g. by [100]. Also, LayerNorms in general have an order of magnitude larger clients' deltas norms than those for MLP and attention. We checked if FL is the source of this deltas imbalance by looking into central training. Central training from scratch from the clients' deltas in FL or FL with DP training. However, when we compare central training on CV-en-train from the same LS- 100 seed model, we will see that per layer gradients behave similarly to the clients' deltas in FL or FL with DP training (see Figure 9).
CV-en-train, Figure 8, has per layer gradients that behave differently
The smallest delta norms are still non-zero and are order of 10 for LayerNorm (In2) and 10-6 for attention (wf) which are re-scaled later by LAMB central optimizer to have the same gradient magnitude across layers. This also highlights necessity of using adaptive optimizers on the server side because otherwise part of the network will not be trained at all. A similar behavior to the one from Figure 5 can be observed (i) with or without DP noise; and (ii) with global clipping or per-layer clipping of clients' deltas (see Figure 10).
H.4 Detailed Results
Comparison for both loss and word error rate (WER) for different values of DP noise and global vs "uniform" per-layer clipping is given in Figure 11, and comparison between "uniform" and "dim" per-layer clipping is given in Figure 12. Training dynamic is shown in Figure 13 for global clipping and in Figure 14 for per-layer clipping. For the per-layer clipping setting we can increase DP noise till ODP = 100 10 and get similar performance as with global clipping but DP noise DD = 3- 10 The former is preferable as it has better (c, 5)- guarantees, detailed results of which are shown in Table 17.
H.S Per-Layer Clipping Analysis
To understand which part of the transformer is most affected by DP noise, we train model by adding DP noise only to a particular group of parameters for both global clipping and per-layer "uniform" clipping (see Figure 15): in this case DP guarantees do not hold, however we do this for the sake of analysis. We can see that adding DP noise to the parameters of MLP layers drastically reduces model performance, while adding it to other parameters changes WER of the model only marginally. This holds for both types of clipping we apply on clients' deltas.
As per-layer clipping "dim" performed the best in our experiments (see Table 1), we analyse the effect of DP noise for this configuration in depth in Figure 16. First, the results are consistent with Figure 15 in that MLP layers are the most susceptible parts of the transformer, e.g. even if we add DP noise to all layers except MLP ones, we see only small degradation in model performance (middle plot in Figure 15). Second, if we add DP noise with oDp to all layers but MLP layers get DP noise with DDP/2 /2, we see a significant improvement in the model performance (right plot in Figure 15). The latter suggests that we could redistribute the clipping budget across layers further alleviate the effect of DP noise during training.
Further experiments with per-layer clipping as C, = C EM ands where d, is the dimension of the i-th layer, 1...., H. and a, = 1 for all layers except MLP and a; B for all MLP layers with 9 E [1.5, 2, 3, 10) did not improve results.
and,
We presume that these data either public do not require privacy protection.
53

---

110
300
300
910
-22
0:1 3900
Wosttwettar
Coctil Daps
Cratal Step.
Figure 11: Loss (top) and word error rate (WER) measured on CV-en-dev (middle) and CV-en-test (bottom) sets for different values DP noise oDp (scale is set to 10 10 9). We apply clipping of 10 either globally (left, Algorithm 1) per-layer (right, Definition 3, "uniform") with T = 2k central steps and I =1,024 cohort size. The rest the training configuration the same as in Figure 7. The seed model is trained on LS-100.
G C
M10
3000
1000
2000
1000
1510
2100
30
500
1000
1500
Cearal Stys
Cenrel Stgss
Oontrd Sieg
Figure 12: Loss (left) and word eтror rate (WER) measured on CV-en-dev (middle) and CV-en-test (right) sets for different values of DP noise ODp (scale is per-layer (Definition 3, "uniform" and "dim") with T 2k central steps and S = 11,04 cohort size. The rest of the training configuration the same as in Figure 7. The seed model is trained on LS-100.
10-9). We apply clipping of 10-2
MW
Coand Doge
Crarsl Supt
Corid Sgo
0.017
110123
100000
1000
2010
Figure 13: Training dynamic of models from Figure 11 with different DP noise ODP (scale to 10-"), global clipping of 10-2 and T 2k central steps. The seed model is trained on LS-100: (top, left) client gradients norm during local training (averaged across clients in the cohort); (top, middle) client's is delta norm before clipping; (top, right) client's delta norm after clipping; (bottom, left) server gradients norm before DP noise is added per clients' deltas; (bottom, middle) server gradients norm after DP noise is added per clients' deltas.
54

---

300
M00
3.31
0.50
380
1300
250
1100
1530
380
130
1000
130
350
1.0090
1.9045
10005
-20 1990
Cound Stggs
Figure 14: Training dynamic of models from Figure 12 with different DP noise ODP (scale set to 10 )) per-layer clipping of 10-2 (Definition 3, "dim") and T 2k central steps. The seed model is trained on LS- 10-100: (top, left) client gradients norm during local training (averaged across clients in the cohort); (top, middle) client's delta norm before clipping; (top, right) client's delta norm after clipping; (bottom, left) server gradients norm before DP noise is added per clients' deltas; (bottom, middle) server gradients norm after DP noise is added per clients' deltas.
Wird (ww.loo.....cc. (V)ccvedd
40
93
50
1.00
1.50
210)
250
200
1000
1.50
200
300
Steps
Stops
Figure 15: WER of models trained on CV-en-train and evaluated on CV-en-dev for different values of DP noise ODP (scale is set 10 0). We add either DP noise all parameters in the model (ODP = 10), no DP noise (ØDP 0), or DP noise to the specific group of parameters: to attention (aDP,wgkrr 10), to MI.P (oDP,
10), LayerNorms (øDP,tv 10), to attention final projection (DDP,wf 10). We apply clipping of 10-2 either globally (left, Algorithm or per-layer (right, Definition 3, "uniform") with 7" = 2k central steps and S =1,024 cohort size. The rest of the training configuration is the same as in Figure The seed model is trained on LS-100.
Weed Drer Rear 181, Cy-arder
25-
100
150
300
200
20
100
1400
7000
7:51
20
150
2000
7.50
Contral Strpe
Conttal Strps
Contral
Figure 16: WER of models trained on CV-en-train and evaluated CV-en-dev for different values of DP noise aDp (scale is set to 10-6). We apply per-layer clipping of 10-2 (Definition 3, "dim") with T 2k central steps and S =1,024 cohort size. The rest of the training configuration is the same as in Figure 7. The seed model is trained on LS-100. We add either DP noise to all parameters in the model (ØDP 10), or no DP noise (opp 0). We also add DP noise (left) the specific group of parameters only; (middle) to all parameters except the specific group of parameters; (right) to all parameters but the DP noise with Øpr 5 to the specific group of parameters.
H.6 Federated Learning with Differential Privacy for French and German
We run out of the box experiments for with DP French and German CV data using the same configuration for English (training parameters given in Figure 7). Seed models trained
55

---

Table 17: Extended results of Table for FL with DP and model pre-trained on LS- (~100h) used as central data and afterwards fine-tuned on CV-en-train (~1.6k hours) used as clients data. We report added noise N(0, IC per client and CV dev and test WERs (%) for two clipping variants with clipping bound C: global and per layer "uniform" ("dim"). Total number of users K, expected number of users sampled per central step S = qK, and the number of central steps T given. We set 8 = -0 9 and report for which (s, 6D- holds for given S and K using the moments accountant of [8]. For scaling S and K where it is practically intractable to run model training (marked "-"), we extrapolate (s, 6)DP assuming training dynamic remains unchanged thus similar WER will be obtained. Central training gives 14.7% WER on dev/test. E should be below 10 to be practically useful (marked with blue).
mapl-10 ") C
8
K
8/K
order
global clipping
per-layer clipping
dev WER (96)
test WER (9)
dev WER (8)
test WER (96)
34,753
$4.7
61.2
54.7
61.2
100.0
0.01
1.024
34,755
0,0295
2,006
3.3.10
20.6
33,9
L,024
100.0
0.01
10.240
347,300
0.0295
2,005
1.3:10
4.0
15.12
100.0
0.01
51.200
L,337650
000295
20055
16-11
25
0:05 12
50.0
0.01
1.024
36,753
0.0295
2,005
35:11P
1.1
27.1 (26.4)
(300)
0,512
50.0
0.01
10.240
347.530
0,0295
2,006
7.2.19
1.5
1556
50.0
0.01
51,200
1,737650
0,0095
2,006
3:5:10P
10.0
0,00072
30.0
0.01
1.024
34,753
0.0095
2,005
1.1-10
1.1
25.2 (24:2)
29.3 (282)
0,3072
31.0
0.01
10.340
347,530
0.0295
2,005
3.3.13F
1.1
1,536
30.0
0..0
51.200
1,737680
0.0995
2,005
6.5-10
7.0
0.07048
20.0
0.01
1.034
34,753
0.0055
2,005
2.6-10P
1.1
237 (22.))
27.6 (26.5)
1,024
20.0
51,200
1.737650
0,295
2,005
1.3-11P
4.0
2/048
20.0
102,000
3.4753000
0,025
20005
0.5-10
9.0
0:01024
10.0
1,024
34,753
0,0255
2,005
1.1-10
1.1
30.7
35.2
(237)
0,512
51,200
1,737.50
0,295
2,005
7.3.10
1.5
0,512
17,376.500
2,034
L,024
10.0
0.01
102,400
3,475,300
0,0255
2,005
1.2.19
2.048
10:0
6,9000000
0.0295
2,005
4.5:10F
9.0
2,048
10.0
0.01
204,800
69 09.506000
0,00295
2,005
7,5-001
25.0
0,00512
5.0
0.01
1.024
34,753
0,0295
2,005
4.2-10
1.1
19.2
22.7
0,512
6.0
0.01
102,400
3,075.300
0,0295
2,005
7.2-19'
1.5
1,004
5.0
0.01
204.800
6,950/00
0.0095
2,005
1.3-19'
4.0
10044
5:0
0.01
204.800
69, 506500
0.00255
2,934
2.1-10P
10.0
1,024
5.0
0.01
204,800
995,060000
0,00225
3,290
1.2-13P
15.0
000003072
0.0
0.01
1,024
34,753
0,0295
2,005
1.2.1P
1.1
27.0
31.1
17.
(2244
0, 5072
3.0
0.01
102.
3,475,300
0,0255
2,005
3.3-10"
1.1
0:61444
204,800
6500600
0,095
2,005
4.2-10
2.0
0.6144
2044,000
005060000
0000295
2,054
1.2/13P
0,00705
37-10PP
34,751
2,005
4.5-10P
20.2
0,184352
0.01
102,400
3,475,300
0,0255
2,005
2.2-1.0
0.56864
0.01
204.800
6,051,00
0.095
2,006
277-10
0,36964
0.01
204,800
60,506000
0.00255
2,044
4.5-10
0,36864
1.5
0.01
204,800
80500000000
0.00095
3,300
1.6-10'
2.5
0.001004
1,0
0.01
1.024
36,753
0.0095
2,005
1.E-10P
111
22.9
25.7
(160)
(193)
0,1024
1,0
0.01
102.40
3,0753100
0,0295
2,005
3.2
1.1
0.2048
1.0
0.01
204.800
6,950.00
0.0095
2,005
1.1-10P
1.1
0,2048
1.0
0.01
204,800
69 506.00
0,00295
2,034
2.7-10P
1.1
0.2045
1.0
0.01
204,800
695,060000
3,390
9.4-18'
1.3
0,006114 44
0,625
0.01
1,024
34,753
BL0295
2,005
4.0-11
1.5
21.3
25.0
16.1
19.3
0,06144
0.625
[02.
3,475,300
0,6295
2,005
33-10P
1.5
0,12288
6,950A00
B,0295
2,005
7.9-10
0
1,024
34,753
0.0095
2,000
inf
157
18.9
15.9
19.1
0.01
1,034
34,753
0.0095
2,000
inf
157
18.9
15.9
19.1
0.1
1,034
34.753
0.0095
2,000
inf
157
18.9
157
19.0
1.0
1.024
34,753
0.0295
2,000
inf
157
18.9
157
18.9
on CV-fr-train-10 10 and CV-de-train--0, while CV-fr-train-90 and CV-de-train-90 are used for further FL with DP training. We get similar results as for English, see Table 18. With the same DP noise ODP = 3 10-6, we are able to closely match the model trained without DP noise (ØDr- = 0) with only a small WER degradation: (i) for French from 15.2% to 16.0% WER while guaranteeing 55.5, 9D-P, and (ii) for German from 11.0% to 12.0% WER while guaranteeing (5.4, 10- 10-9)-DP DD-DP; assuming the training effectiveness remains the same if we extrapolate to 50M clients with the cohort size of 250k. Moreover, we can also increase DP noise to DD 10-5 5 getting 17.9% WER with (1.9. 10 --DD for French and 13.9% WER with (1.8, 10 10-9)-DP for German by scaling only to ~ 16M clients with the cohort size of 250k, assuming the training effectiveness remains the same. The latter is a realistic scenario for mid/low resource languages.
For both French and German we observe that per-layer clipping is not effective as for English and we get only marginal improvements over global clipping. We have checked that the seed model quality and the seed model being out-of-domain are the not the sources of this discrepancy in results
56

---

Table 18: Results for FL with DP and a model pre-trained on WV-ff-rrinn-1OCC-dee-taai (~-50h) used as central data and afterwards fine-tuned on (top/bottom) wttff--raann-0//VVdd-e-e-e
(~~700-
800 hours) used as clients data. We report added noise N(, IC (p per client and CV dev and test WERs (%) for two clipping variants with clipping bound C: global and per layer "dim". Total number of users K, expected number of users sampled per central step S gK, and the number of central steps T are given. We set 8 10-9 and report for which (E, 6 holds for given S and K using the moments accountant of [8]. For scaling S and K where it is practically intractable to run model training (marked "-"), we extrapolate (e, 6D-DP assuming training dynamic remains unchanged thus similar WER will be obtained. Central training gives 10.8%/12.6% WER for French and 8. %/992%% WER for German on dev/test. should be below 10 to be practically useful (marked with blue).
(ar(-10-
q S/K
plobol clipping
per-lyyer clipping "dim""
WER WER
WER
6,171
24.0
27.5
24.0
000102
10.0
1,024
6.171
0.1660
2,002
1,1:307
1,3
2.56
10.0
0.01
256,000
1,542750
0.1660
2,002
2.4-30'
3.0
2.56
10.0
0.01
256,000
15.4273000
000066
2,033
1.9:10"
20:0
0UME3072
3.0
0.00
1,024
6.171
0.1660
2,002
1,2.00
1.3
14.1
16.2
13.9
1601
0.768
3.0
0.01
250,000
1.422750
0.3660
2,002
1S-107
3.0
0.768
300
0.01
256,000
1544275500
0.1666
2,013
1.4-10
3.0
0U768
256,000
40.262.500
55530
500
6.171
0.3660
2,000
inf
13.2
15.2
13.2
0
6.415
0
0
18.6
21.2
18.6
21.2
0U01004
10.0
0,01
1.024
6.415
0.1596
2,002
1,1:30
1.3
12.3
139
2,56
10.0
0.01
256.000
1603,750
0.1596
2,002
2.3-10'
3.0
2:50
10.0
0.01
256,000
15057,500
0.01595
2,086
128300
20.0
0.003072
3.0
0.01
1,024
6,415
0.1596
2,002
1,2:10
1.1
10.7
12.1
10.5
12.0
0.768
256,000
1.033750
0.159
2,002
1.7:30
1.5
0.768
256,000
16,057.500
0.01396
2,016
1.4-30'
0.768
3.0
0.01
256,000
48,112,500
0,0033
2,088
5.4-00
5.0
0.01
1,034
6,415
0.1596
2,000
inf
9.7
11.0
11.0
Table 19: Ablation for FL with DP and model pre-trained either on LS-9600CV-en--train 10 used as central data and afterwards fine-tunes on (top/bottom) Wi-ee-trrai/CV-eeeee noise N(0, IC IC² per client and CV dev and test WERs (%) for two clipping variants with clipping bound C: global and per layer "dim". Total number of users K, expected number of users sampled per central step S = gK, and the number of central steps T are given. Central training gives 14.1% 17.2% WER for training from LS-960 seed and 14.5 14.5%/17 17. .6% for training from CV-en-train-10 seed on dev/test. All the remaining parameters are the same as in Table 17.
We report added
Seed
Data
#pp(-10-9) +)
S
global clipping
lay-layer clipping "dim"
dev (%)
test WER (%)
dev WER (%)
WER (96)
LS-960
34,753
27.0
31.5
27.0
31.5
LS 960
CV-en-train
30
0.01
256
34,753
0.0074
2000
22.5
26.1
18.7
22.2
CV-10
0
34,753
23.0
27.9
23.0
27.9
CV-10
CV-en-train-90
0.01
256
31,278
0,0082
2000
25.1
18.7
22.6
between languages: if we change the seed model for English to better out-of-domain LS-960 seed or to a better in-domain CV-en-train-100 seed, we still observe a drastic improvement from per-layer clipping compared global clipping (see Tables 19 and 20, and Figure 17).
First, there is a discrepancy in gradicnts balance across layers for the central model training for English, French and German with CV-*-train-10 10 seed models. The training of the English model has the issue we discussed above that LayerNorms dominate the attention and MLP, which translates to the similar behavior for FL and FL with DP training. However, French and German models do not have the same imbalance issue as English and, moreover, similar behavior holds for the central training. FL and FL with DP for French and German (see Figures 18, 20, 19 and 21). We attribute the later to the properties of the languages as discussed in Appendix G.5.
One factor that we cannot exclude from the above analysis is the user sampling q S/K, which is significantly higher for French and German (16%) than for English (< 1%) due to a smaller number of speakers in the French and German datasets. Further investigation is needed to evaluate larger
57

---

Table 20: Results for FL with DP and model pre-trained on LS-960 (~1100h) used as central data and afterwards fine-tuned on CV-en-train (~1.6k hours) used as clients data. We report added noise N(O, TCooppg per client and CV dev and test WERs (%) for two clipping variants with clipping bound C: global and per layer "dimm. Total number of users K, expected number of users sampled per central step S = qKK and the number of central steps 7 are given. We set 10-B and report for which (e, 8) -DP holds for a given S and K using the moments accountant of [8]. For scaling S and K where it is practically intractable to run model training (marked "-"), we extrapolate (s, 6DD assuming training dynamic remains unchanged thus similar WER will be obtained. Central training gives 14.1 %/17.2%% WER on dev/test. & should be below 10 to be practically useful (marked with blue).
mmp(110 "
q
S/K
global clipping
pes-layer clipping
WBR (1) dev WER (8)
14,753
27,0
27.0
31.5
0.01072
1,004
34,753
0.0205
2006
1.1110
1:1
261
18.7
0,9012
0,0295
2,005
3
1.1
1,536
33.0
0.01
31,200
1,737.650
0.0295
20065
6.5:10
7.0
0.01024
10.0
0.01
1,024
34,755
0.0295
2,006
1.1:10"
1.1
20.5
24.1
16.5
19.7
0,512
10.0
0.01
51,200
1,737650
0.0295
2,006
7.2.30'
1.5
0,513
10.0
0.01
51,200
17.376
0.00295
2,084
1.3-107
3.0
1,024
10.0
0.01
102,400
3,475,300
0.0295
2,005
1.3:107
4.0
2,048
10.0
0.01
204,300
6,950,000
0.0295
2,006
4,5-10"
9.0
2,1448
69 5003060000
0,80295
2,006
7.5-10
0J00RU72
3.0
1,024
0,0295
2,006
1,2-30
21.6
14.9
17.8
307022
3,45,3000
0.0295
2,006
3.7:10"
1.1
3.0
0.01
304,800
6.950,000
0,0295
2,006
4.2:30
2.0
3.0
0.01
204 900
69 306,000
0.80795
2,034
7:2:10
3.0
0.6144
3.0
0.01
204
500
695
0600,000
0.800295
3,390
3.7-30
6.8
0.01
1,024
34,753
0.0295
2,000
inf
13.9
16.7
14.0
16.8
kel
1.30
0.75
0.53
0.35
0.55
Yrandhemen Blick Tadee
Tranchene Rack Indes
Tranheee Mack Tndee
Trandhome Macc Indes
117
101
107
km
10
V22veigh:
valsinight
33000
-0000
43000
30000
400o0
25:00
43000
Tradiin eap
Thilkgg rop
Traning dap
Tiilkkg 1xg
Jellis Islanda
20
splo gk.
30
sfunge
15
130
10
100
13
20
10
20
0.
10
20
30
20
30
10:
10
10
10
12
12"
10
20000
Trandin
Thankig
Figure 17: (first and second rows) Central training on CV-en-train from the LS-960 seed model and (third and fourth rows) Central training on CV-en-train-90 from the CV-en-train-10 seed model and their per layer gradients norm: (first, third rows) averaged across training steps and (second, fourth) showed per layer along the training. The model is trained with LARS optimizer and the learning rate of 0.5/0.2. LayerNorm gradients do dominate over MLP and attention gradients.
58

---

lallkm
Vi Vooigg
Ialacaaa
Valadd
Vilwigg:
7.3
30
2.5
003
Treafoun Dlock
Trandarast Black
The sooree
might
10
3006
40C.0
20000
V300
30000
20000 5000
Figure 18: Central training on CV-fr-train-90 from the CV-fr-train- 10 seed model and its per layer gradients norm: (top) averaged across training steps and (bottom) showed per layer along the training. The model is trained with LARS optimizer and the learning rate of 0.2. The norms of the per- layer gradients balanced similarly to models trained with FL with FL and DP in Figure 19: LayerNorm gradients do not dominate over MLP and attention gradients.
Talacka
Ialacaka
0.50
0.50
0.04
0.07
0.00
0.02
0.00
Trandiocae Blace Balls
Tranftuna Black bei
Traneforee Black Indis
Teat dicmaa Block Jades
Ialata
0.00
balkeer Intacll:
0.37
vioeogt: WCOoggt:
vebrovogg:
Ialaci
0.04
0.00
Exo
0.02
0.54
0.02
Tranttencr
Traalemm
Trandarncr Black
Thanderrc Indis
0.05
3.04
0.00
0.02
0.50
0.02
0.00
0.00
Trandlocuer
Traafeuna Black
TranFforect Black
Thandiored Block Jades
Figure 19: Client delta norms computed per layer in the French model trained CV-fr-train-99 from a seed CV-fr-train-ID 10 model. We average the statistics across all clients and central steps, and plot
the mean and standard deviation. The model is trained with (first row) global clients' deltas clipping
C== 10 and ODP = O, (second row) global clients' deltas clipping C = 10 and oDp 3 10
(third row) per-layer clients' deltas clipping (Definition 3, "dim"" C 10-2 and ODP 3 10-6
The rest of the training configuration is the same as in Figure 5. A transformer block consists of attention parameters (wqky and wf), MLP (wl and w2), LayerNorm applied input of attention (Inl) or MLP (In2).
datasets with larger number of speakers for French and German (as we need large cohort size to alleviate the impact of DP noise), and to probe other languages.
[101] also used per-layer clipping but for NLP domain and observed the difference in the gradient norms of different transformer layers. However, per-layer clipping did not outperform the global clipping for training with DP (there was no FL component) in many settings. We would like to highlight the main differences with our study for ASR domain: our architecture is encoder- based model trained with a sequence loss (CTC), while [101] use decoder-based (causal) model trained with cross-entropy loss; ii) Tables 3 and 4 of [101] show that per-layer clipping significantly improves results for GLUE tasks, thus it is task dependent; ii) [101] fine-tune pre-trained model for
59

---

Sengle
Transiioma Block Dadis
Transtere Black ladee
Trantienna Black Endos
Tranitomua Black lades
wil wicgpa
107
10"1
BOAA Bolacii
eingl
3260
-0000
200M 4300
3,00 Traning
4EO
20.00 4000 Thinnng
Truningaoyy
Thanngaop
Figure 20: Central training on -V-de-train-90 from the CV-de-train- 10 seed model and its per layer gradients norm: (top) averaged across training steps and (bottom) showed per layer along the training. The model is trained with LARS optimizer and the learning rate of 0.2. The norms of the per-layer gradients are balanced similarly models trained with FL or with FL. and DP in Figure 21: LayerNorm gradients do not dominate over MLP and attention gradients.
0.00
0.04
0.00
Traafemm thack
Tranfornce Block
Thanderao Blick Jndes
hichee
0.37
leoght
webrowoge
Inl
8xo.
Traafouna
Tranforror Black
leiglt
0.02
0.50
0.00
0.02
10:
10
30
33
Client delta norms computed per layer in the German model trained on CV-de-train-e from seed CV-de-train-lo model. We average the statistics across all clients and central steps, and plot the mean and standard deviation. The model is trained with (first row) global clients' deltas clipping C 10-2 and øDP O, (second row) global clients' deltas clipping C' = 10-2 and aDP 10-6 (third row) per-layer clients' deltas clipping (Definition 3, "dim") = 10-2 and
Figure 21:
ODP 10-6 The rest of the training configuration is the same as in Figure 5. A transformer
block consists of attention parameters (wqkv and wf), MLP (wl and w2), LayerNorm applied to input of attention (Inl) or MLP (In2).
downstream task with another objective (this can affect the contribution of different parts of the model) while in ASR we keep the same. Moreover, our theoretical results (Theorem 2) show that per-layer clipping can help to improve convergence in case of higher level of heterogeneity.
H.7 Per-Layer Clipping for Different Model Sizes
We further evaluate effectiveness the per-layer clipping for different model sizes. We take baseline model we used before with 36 layers, 768 embedding and 3072 MLP dimension (244M parameters), set layer drop to and consider the following models: narrow with 14M parameters (reduce embedding to 512 and MLP dimension to 2048), wide with 450M parameters (increase
60

---

300
230
233
3.3
0.07
541
021
3.0
Figure 22: Client delta norms computed per layer in the narrow (row 1), shallow (row 2), baseline (row 3), wide (row 4) and deep (row 5) models trained on CV from seed LS-100 model. We average statisties across all clients and central steps, and plot the mean and standard deviation. are trained with global clients' deltas clipping C' 10-2 and ØDP 10 10-6, A transformer block consists of attention parameters (wqkv and wf), MLP (w1 and w2), LayerNorm applied to input of attention (Inl) or MLP (In2).
models
embedding to 1024 and MLP dimension to 4096), shallow with 114M parameters (reduce only number of layers to 16) and deep with 510M parameters (increase depth to 72 layers). All models are trained with the same hyperparameters as the baseline model only change the model architecture as discussed (with layer drop set 0. for models including the baseline). There are few takeaways and observations from the results (all comparisons are provided test set), shown in Table 21:
1. Per-layer clipping consistently outperforms global clipping for different model sizes.
2. For per-layer clipping, model size increases, the model performance in FL with DP degrades more compared to FL. This holds for both increasing model size via width and depth. Degrada- tion for increasing model width is smaller compared model depth. These results are line with our theoretical results.
3. For global clipping, as model size increases, the model performance in FL with DP degrades more compared to FL. This holds for both increasing model size via width and depth. However, for larger model size (wide and deep) we see significant performance improvement we hypothesize that due to the lower gradient imbalance between layer normalization and layers, see Figures 22 and 23 for global and per-layer clipping- Model sizes > 500M we leave for the future exploration and highlight the need to study larger models considering model size limitations and aforementioned results in the current work.
61

---

2323
370
2007
0.0
042
0.0
0.04
132
Figure 23: Client delta norms computed per layer in the narrow (row 1), shallow (row 2). baseline (row 3), wide (row 4) and deep (row 5) models trained on from a seed LS-100 model. We average the statistics across all clients and central steps, and plot the mean and standard deviation. All models are trained with per-layer (Definition "uniform") clients' deltas clipping C 10-2 and ODP 10 10-6, A transformer block consists of attention parameters (wqkv and wf), MLP (wl and w2), LayerNorm applied to input of attention (In1) or MLP (In2).
Table 21: Ablation for FL and FL with DP with model pre-trained on LS-100 used as central data and afterwards fine-tuned CV-en-train. We report added noise N(0, oC per client and CV dev and test WERs (%) for two clipping variants with clipping bound C = 0.01: global and per layer "uniform". Total number of users K 34, 753, expected number of users sampled per central step S = ş = 1024, and the number of central steps T 2000 are given. We also show relative degradation in performance for test if we switch from FL FL+DP for specific configuration.
Model
fmx(-10-66
global el clipping
per-layer clipping "uniform"
dev WER (%)
WER
dev WER (%)
WER
15.2
18.2
narrow
10
27.5
31.7
74.2
19.5
23.2
27.5
huseline
14.7
17.6
10
29.9
34.6
96.6
19.7
23.3
32.4
wide
10
20.8
24.7
48.8
20.0
23.7
42.8
shallow
0
16.3
19.8
30.6
35.1
77.3
20.9
24.8
25.3
baseline
10
29.9
14.7
34.6
17.6
96.6
19.7
23.3
32.4
deep
10
21.7
25.7
49.4
22.4
26.4
53.5
62

---

I Compute Resources
In Table 22 we show the summary of used compute of the main training configurations for benchmarks of FL and FL with DP for transparency and setting proper expectations for the community.
Table 22: Compute for the main expeimments use LAMB as the central optimizer and SGD as the local optimizer.
for FL and FL with DP. For all experiments
Seed
Data
Model
Clien Toul Bunck Size
Color Size
S
Local
Centrll Step.
* GPUs A160 8OGB
Runtine (h)
Toxal GPU (h)
CVesstain
LS-960
FL.
Baii
10 qpadia
2000
53.
106
CV-es-zain
15-950
FL
Geiin
16
spodis
2000
103
206
CN-en-rain
bminn
1pacs
13-960
fimnn
10
1,112
epectts
Zetin
Zetin
1,216
Ls 100
CV tran
FL
Zntin
1,124
10 stops
2000
32
34
1J58
LS 100
CV tran
FL. DP
Zatin
1,124
10 stops
2000
32
35
1.120
LS 100
CV en trin
[L. + DP
Zetin
250
10 stope
2000
16
258
CV.de tnin 10
CV trais
FL.
Zaiin
1,024
10 stope
2000
16
65
1.056
CV de min 10
CV trara 90
[L DP
Zetin
1,024
10 stope
2000
16
67
1.072
stope stope
siqee
1,152
J
Contributions
The overall vision for enabling differentially private federated learning in ASR was conceived by Martin Pelikan, Sheikh Shams Azam, Jan "Honza" Silovsky, and Tatiana Likhomanenko, who identified the gap in current research and defined the problem scope. The work on Differential Privacy was done in consultation with Vitaly Feldman and Kunal Talwar and the theoretical work was done in consultation with Christopher G Brinton and Kunal Talwar. Specific contributions of the authors can be attributed as:
Algorithm Design. The design of algorithm including per-layer clipping and layer-wise gradient normalization was led by Martin Pelikan, Sheikh Shams Azam, Jan "Honza" Silovsky, and Tatiana Tatiana Likhomanenko in consultation with Vitaly Feldman and Kunal Talwar.
Implementation and Experimental Results. The FL with DP training pipeline was developed by Martin Pelikan and Tatiana Likhomanenko. Martin Pelikan carried out the extensive experiments for FL evaluating the effects of data heterogeneity, optimizer settings, and initialization strategies, while Tatiana Likhomanenko carried the extensive experiments for FL with DP evaluating DP and different clipping strategies. All was done in consultation with Sheikh Shams Azam, Jan "Honza" " Silovsky, Vitaly Feldman and Kunal Talwar.
Theoretical Convergence Analysis. The theoretical analysis of per-layer clipping and layer- wise adaptive optimizer was led by Sheikh Shams Azam and Tatiana Likhomanenko. The FL convergence analysis
done in consultation with Christopher G. Brinton and the analysis of DP in the bound was done consultation with Kunal Talwar. Kunal Talwar double checked all derivations in the final proof.
Writing and Paper Preparation The manuscript was jointly written by Martin Pelikan, Sheikh Shams Azam, and Tatiana Likhonamanenko. was edited and reviewed by all other authors.
63


Reviews:
[{"review_id": "sYNHrdbORJ", "reviewer": "Reviewer_UQPF", "summary": "This work proposes a mechanism for end-to-end training of ASR models that centers on Differentially Private Federated Learning (DP-FL). At the core the proposed approaches utilizes layer wise clipping and gradient normalization to take a step towards DP-FL. The approach is backed by theoretical and empirical studies and analysis to demonstrate its strength.", "questions": "Can you give an example of user level D and D' sets that are adjacent? Each user has their own unique speech characteristics and adding/removing all their data can potentially impact the information  contained inside a dataset.\n\nIt has been highlighted that training with larger models can be challenging for DP. Therefore it would be beneficial to have an ablation study that demonstrates the impact of the proposed techniques on models of different sizes. \n\n\nThe rows marked blue in Table 1. They seem to be extrapolated (perhaps I missed something). To increase confidence in the approach it would be great to have at least one blue ($\\epsilon$ below 10) row with non extrapolated numbers. \n\nIt seems that even small change in $\\sigma_{DP}$ can impact performance. What are the author's thoughts on such an impact (and on $\\epsilon$) if the number of steps is increased.\n\n\nThe per layer clipping in the context of DP-FL does not seem specific to ASR. Are there any characteristics that are especially more useful for ASR ?", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This work proposes a well structured approach and provides theoretical and empirical support/evidences for it. A well known approach of gradient clipping is utilized in the given context of DP-FL and demonstrated to be useful. Empirical evaluation and analysis through the domain of end to end ASR shows strengths of this technique. The proposed approach can potentially be applied to other domains apart from ASR and can have significant impact. The manuscript is well written and provides clear and well rounded, good quality proposal and analysis.\n\nHowever, there is room for improvement. In the main results , experiments with practically useful $\\epsilon$ for DP seem to be extrapolated. Ideally, the proposed approach would be combined with tractable numbers for these $\\epsilon$ values. In addition to this, the proposed work can be further enriched through a more thorough ablation study and utilization of a larger variety of ASR datasets. Furthermore, the concept of per layer clipping for DP-FL seems more generic and can have applications for more than just the ASR domain. Therefore, demonstrating its impact on properties specific to ASR would have been beneficial.", "was_revised": true, "final_justification": "This work proposes a well structured approach that is supported by theoretical and empirical analysis. Apart from ASR, the proposed approach can also be applied to other domains further demonstrating the quality and benefits of it. However, there are some weaknesses such as practical usefulness of the demonstrated $\\epsilon$ and along with the variety of datasets.\n\nThe authors provided several clarifications through their rebuttal which can help mitigate some of the weaknesses. Furthermore, they also explained the practical limitations that impact some of the important weaknesses. \n\nThe score is based on these strengths and weaknesses including the impact of practical limitations.", "rebuttal": "Dear Reviewer UQPF,  \n  \nWe sincerely thank you for your thorough and constructive review of our work. We address each of your concerns below.  \n  \n\n> \"The rows marked blue in Table 1. They seem to be extrapolated... To increase confidence in the approach, it would be great to have at least one blue (ε below 10) row with non-extrapolated numbers.\"\n\n  \nWe acknowledge this concern. The extrapolation to a practical ε value at the scale required for blue rows would require simulating millions of users with cohort sizes of \\~200k. However, we cannot run such simulations since there are no public datasets that can support this scale.   \n  \n**Our extrapolation is based on the moments accountant** (Abadi et al. 2016 [8], McMahan et al. 2018 [46]) method, which is a standard practice in FL and DP (for vision, text, etc.) due to the absence of such large-scale public data. The extrapolation estimates epsilon when population and cohort size increase while preserving the privacy noise-to-signal ratio.  \n  \nWe validated the above assumption by simulating different cohort sizes with ~35k users in Common-Voice (Table 16, Appendix):  \n\n|Cohort Size|Dev|Test|\n|---:|---:|---:|\n|1024|13.9|16.7|\n|2048|13.6|16.3|\n|4096|12.8|15.5|\n|5120|12.7|15.4|\n\nThese results demonstrate the consistent performance trend supporting the validity of the extrapolation.   \n\n---\n\n> \"the proposed work can be further enriched through a more thorough ablation study and utilization of a larger variety of ASR datasets\"\n\nOur choice for FL simulations is constrained by the requirements of speaker metadata:  \n\n- Datasets WITH speaker IDs (suitable for FL): LibriSpeech (\\~2k speakers), CommonVoice (\\~35k speakers), VCTK (\\~100 speakers), TED-LIUM (\\~1.2k speakers), etc.\n- Datasets WITHOUT speaker IDs (unsuitable for FL): People's Speech, GigaSpeech, SPGISpeech, etc.\n\nThe requirement for speaker ID to simulate realistic user partitions severely limits options. Among suitable datasets, LibriSpeech and CommonVoice provide the largest speaker diversity (see Table 2, Appendix).  \n\n|Dataset|# speakers|minutes/speaker|domain|\n|---|---|---|---|\n|LibriSpeech|~2.3k|min: 2, max: 30|read-speech|\n|CV english|~35k|min:0.02, max: ~5k|conversational|\n|CV french|~6.9k|min:0.04, max: ~3k|conversational|\n|CV german|~7.1k|min:0.03, max: ~6k|conversational|\n\nThis diversity ensures our methods are tested across the wide spectrum of FL challenges. Our evaluation spans 4 datasets (LS + 3 CV languages), whereas prior FL-ASR works typically evaluate on 1-2 datasets [19, 20, 56].  \n\n---\n\n> \"The per layer clipping in the context of DP-FL does not seem specific to ASR. Are there any characteristics that are especially more useful for ASR?\"\n\nYou raise an excellent point about the generality of our approach. We view this generality as a strength that enhances rather than diminishes our contribution:  \n\n- We demonstrate that large Transformer models benefit from per-layer clipping when trained using FL with DP unlike the observations of McMahan et al. (2018) [46] who report no significant improvement for LSTMs and LMs.\n- We provide insights by analyzing the ASR models and observing that layer-wise gradient heterogeneity is extreme in speech transformers (Figure 5: attention gradients are 10x smaller than LayerNorm). \n\nGiven that our theoretical analysis is not specific to ASR, these insights should help other applications as well.   \n\n\n---\n\n> \"It would be beneficial to have an ablation study that demonstrates the impact of the proposed techniques on models of different sizes.\"\n\nWe agree with the suggestion and provide the following table summarizing a preliminary result for a 114M parameter model size that we were able to train within the rebuttal timeframe. We can observe that (i) the larger 250M parameter model is more affected by uniform clipping, and (ii) it also benefits more from the per-layer clipping.  \n\n|model size|T|uniform-clipping|per-layer clipping|Rel. Improv.|\n|---|---|---:|---:|---:|\n|114M|2k|29.5|21.3|27.8\\%|\n|250M|2k|31.1|20.4|34.4\\%|\n\nThis is in line with our theoretical analysis (Theorem 2), which emphasizes that per-layer clipping benefits models with greater depth H and layer-wise gradient heterogeneity (Ψ$_h$ terms), which is more prominent in deeper models (Chan et al. [23], Wang et al. [24]; discussed in Sec. 3). We will include this table in the camera-ready version.  \n\n---\n\n> \"What are the author's thoughts on such an impact (and on σ) if the number of steps is increased.\"\n\n  \nThis is an insightful observation that touches on fundamental FL+DP tradeoffs:  \n\n- Based on our theoretical analysis, it can be seen that privacy budget ε must be scaled as √T for T steps.\n- Given such a dependence of noise on the number of training steps, we make the design choice of limiting to T=2k steps, thus keeping the noise manageable. All FL deployments face similar constraints, i.e., longer training requires accepting either higher ε or lower utility.\n- We did not explicitly study extended training with higher noise levels since most FL studies aim to reduce the number of training steps (Azam et al. [16]) owing to the practical time limits for real-world federated learning deployments.\n\n---\n\n> \"Can you give an example of user level D and D' sets that are adjacent?\" Each user has their own unique speech characteristics and adding/removing all their data can potentially impact the information contained inside a dataset.\n\n  \nCertainly. In our user-level DP setting, if we consider a population comprising 3 users X (with 50 data samples), Y (with 30 data samples), and Z (with 45 data samples), then adjacent datasets D and D’ can be defined as:  \n\n- Dataset D: Contains data from all users, i.e., X, Y, and Z, and\n- Dataset D': Contains speech data from only 2 users, i.e., X and Z.\n\nThese datasets are adjacent because D' can be formed by removing all utterances from a single user Y. This DP guarantee ensures that an adversary observing the trained model cannot reliably determine whether User Y (with all 45 data samples) participated in training, protecting individual user privacy comprehensively.  \n  \nIn practice, we train on the actual dataset (e.g., dataset D with all users), but add carefully calibrated noise during training such that the resulting model could plausibly have come from training on D' (without User Y). The unique speech characteristics you mention make this particularly challenging - say, User Y's distinctive speech patterns could otherwise be detectable in the model, which is why we need sufficient noise to mask their contribution while still maintaining model utility.  \n\n---\n\n**Summary**  \nWe appreciate your recognition that our work presents a \"well-structured approach\" with \"good quality proposal and analysis.\" We have identified areas where we can improve the manuscript based on your inputs. We will include these changes in the final draft of the paper."}, {"review_id": "E1Bp270bsC", "reviewer": "Reviewer_TNaa", "summary": "This paper proposes to use federated learning with differential privacy for ASR. The authors investigate a Transformer-based ASR model trained with CTC loss under various federated learning settings and evaluate its performance on two datasets.", "questions": "It is unclear why the proposed model is characterized as a 'large model' in the context of ASR. Many recent ASR studies employ models with over 250 M parameters. A large amount of papers on ASR utilize models larger than 250M parameters. Clarifying this with respect to existing literature would strengthen the positioning of your work.", "limitations": "Given that the paper is positioned as a benchmark study, a more comprehensive experimental evaluation is expected, including a wider range of backbone models and datasets. Relying solely on LibriSpeech and CommonVoice is insufficient to robustly support the claims made since more ASR works report their results on more than two datasets. It is recommended that the authors incorporate additional ASR datasets to enhance the generalizability and credibility of the findings.\n\n\nIn Table 2, the reported WERs for both datasets are relatively high, especially for LibriSpeech, where much lower WERs are typically achieved in recent literature. This suboptimal performance appears to stem from the choice of model architecture and the limited training data scale. While the proposed method shows improvements, the overall weak ASR performance may hinder its impact in the field. I recommend the authors include more recent and competitive ASR models in their experiments to better demonstrate the effectiveness of their approach.", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Quality: The general quality is good but can be improved further with better experiment design. Regarding ASR model selection, the use of a transformer with CTC—while effective in earlier work—is relatively outdated given recent advances in end-to-end ASR, including models utilizing both CTC and sequence-to-sequence (S2S) losses, as well as the emergence of speech foundation models and speech language models (LLMs). \n\nClarity: The paper is well-written with a clear structure. The integration of federated learning with ASR presents a promising research direction and offers valuable insights for future work. The in-depth analysis of data heterogeneity and the application of differential privacy in federated learning is particularly insightful and inspiring.\n\nSignificance: In the context of ASR, it is unclear what specific contributions have been made to address speech recognition challenges beyond the integration of existing federated learning and differential privacy techniques. The current approach appears more aligned with an engineering implementation rather than a novel methodological advancement, which may limit its suitability for a NeurIPS publication.\n\nOriginality: The novelty of the work appears limited, as the use of federated learning with differential privacy has been explored in prior research. It remains unclear what specific technical contributions are made from both the federated learning and differential privacy perspectives. Clarifying these aspects would strengthen the paper’s contribution.", "was_revised": true, "final_justification": "My original concerns remain largely unresolved and they are very important for this paper to be accepted. The authors did not revise or meaningfully address core concerns raised in the initial review. Their response did not demonstrate sufficient willingness to improve the paper based on my feedback for model architecture, dataset, novelty and model scale. Due to unresolved major concerns regarding methodology, dataset evaluation, and novelty, I recommend rejecting this paper.", "rebuttal": "Dear Reviewer TNaa,  \n  \nWe thank you for the thoughtful and detailed feedback. We greatly appreciate the points raised and would like to clarify a few aspects of our work, as some key elements of our contributions and the FL+DP setting may have been misunderstood. We address each point below with the hope of resolving these potential concerns.  \n  \n\n> \"the use of a transformer with CTC—while effective in earlier work—is relatively outdated given recent advances in end-to-end ASR, including models utilizing both CTC and sequence-to-sequence (S2S) losses\"\n\nWe offer a different perspective that CTC is still a highly relevant and widely adopted approach in the literature:  \n\n- NVIDIA's FastConformer-CTC achieves competitive results on the OpenASR leaderboard (2024) (see [HuggingFace Open ASR Leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard))\n- Google's USM employs CTC for their 2B parameter universal speech model [Zhang et al., 2023; Wang et al., 2024]\n- Meta's MMS uses CTC for their massively multilingual model covering 1,100+ languages [Pratap et al., 2023]\n\nAdditionally, our choice of ASR model is deliberate since we aimed to solve the ASR problem with on-device capability. Speech LLMs may still be infeasible due to device constraints and are still behind cascaded models (see “Spirit-LM: Interleaved spoken and written language model,” Transactions of the Association for Computational Linguistics, 2025; “Qwen2. 5-omni technical report,” arXiv:2503.20215, 2025; Voicebench: Benchmarking llm-based voice assistants; arXiv:2410.17196, 2024; MMAU: A massive multitask audio understanding and reasoning benchmark, arXiv:2410.19168, 2024). Foundation models of similar size are not comparable on ASR tasks, thus making the current ASR models more suitable for on-device applications.  \n\n---\n\n> \"it is unclear what specific contributions have been made to address speech recognition challenges beyond the integration of existing federated learning and differential privacy techniques\"\n\nBelow, we clarify the aspects of our work that are beyond just the integration of existing methods:  \n\n1. We observe that a naive integration of FL with DP does not converge in ASR. We arrive at a practical recipe to circumvent the convergence problems by using additional interventions: (i) per-layer clipping and (ii) layer-wise normalization. \n2. We demonstrate a clear departure from existing knowledge about per-layer clipping — previous work [McMahan et al., 2018] found no benefit for LSTMs — by providing empirical evidence supporting its effectiveness.\n3. We also theoretically analyze the effect of per-layer clipping together with layer-wise adaptive optimization on the convergence dynamics. Our theoretical analysis goes beyond existing FL literature to understand the impact of heterogeneity, depth, and optimizer dynamics on convergence.\n4. We “establish the first benchmark of FL+DP in large-scale ASR” (also echoed by Reviewer R1Qm) and further study the framework to explore the impact of: (i) different initialization strategies and (ii) in-domain and out-of-domain pre-training initialization.\n\nThus, in addition to the engineering implementation, our work also provides fundamental insights into optimization dynamics, theoretical guarantees, and practical techniques that advance the field. Furthermore, as echoed by Reviewer UQPF, the \"proposed approach can potentially be applied to other domains apart from ASR and can have a significant impact.\"  \n\n---\n\n> \"the reported WERs for both datasets are relatively high, especially for LibriSpeech, where much lower WERs are typically achieved\"\n\n  \nWe believe that this assessment might be originating from the confusion between the model performance in the centralized and federated learning cases. For context, we compare against Whisper small (244M parameters, see Radford et al. 2022 D.2.2 - similar size to our model):  \n\n- For LibriSpeech test-other, Whisper small achieves 7.6% WER. In comparison, our centralized baseline achieves 6.8% WER on LibriSpeech test-other (Table 3, Appendix). SOTA models trained only on LibriSpeech achieve 5-6% WER for CTC models w/o a beam search and LM and are close to 3% with a beam search and LM - our model is ~3.5% when the beam search and LM are integrated. Our FL baseline on LibriSpeech test-other achieves 7.1% WER (only 0.3% degradation from centralized baseline). For context, prior work, Azam et al. 2023 [56], achieves 10.29% WER for FL setting with CTC-AED model.\n- For Common-Voice English, Whisper small achieves 14.5% WER. In comparison, our centralized baseline achieves 18.2% WER (Table 4, Appendix) without a beam search and LM. Our FL baseline on Common Voice achieves 25.7% WER. For context, no prior work in FL-ASR, including Gao et al. 2022 [20], uses Common Voice English. \n- Additionally, Gao et al. 2022 [20] are able to train a model using FL on Common-Voice French and achieve 20.99% WER. In comparison, we achieve 13.1% WER (Table 10, Appendix) on Common-Voice French, while Whisper small is reported as 22.7%.\n\n---\n\n> \"Relying solely on LibriSpeech and CommonVoice is insufficient to robustly support the claims\"\n\nWe understand the concern and wanted to clarify that our choice for FL simulations is constrained by the requirements of speaker metadata:  \n\n- Datasets WITH speaker IDs (suitable for FL): LibriSpeech (\\~2k speakers), CommonVoice (\\~35k speakers), VCTK (\\~100 speakers), TED-LIUM (\\~1.2k speakers), etc.\n- Datasets WITHOUT speaker IDs (unsuitable for FL): People's Speech, GigaSpeech, SPGISpeech, etc.\n\nThe requirement for speaker ID to simulate realistic user partitions severely limits options. Among suitable datasets, LibriSpeech and CommonVoice provide the largest speaker diversity (see Table 2, Appendix).  \n\n|Dataset|# speakers|minutes/speaker|domain|\n|---|---|---|---|\n|LibriSpeech|~2.3k|min: 2, max: 30|read-speech|\n|CV english|~35k|min:0.02, max: ~5k|conversational|\n|CV french|~6.9k|min:0.04, max: ~3k|conversational|\n|CV german|~7.1k|min:0.03, max: ~6k|conversational|\n\nThis diversity ensures our methods are tested across the wide spectrum of FL challenges. Our evaluation spans 4 datasets (LS + 3 CV languages), whereas most prior FL-ASR works typically evaluate on 1-2 datasets [19, 20, 56].  \n\n---\n\n> \"It is unclear why the proposed model is characterized as a 'large model' in the context of ASR\"\n\n  \nThe characterization of \"large\" is **relative to FL literature**, not centralized ASR. Typical FL models in literature are <30M parameters (e.g., FedAvg [1], FedProx [11]). Prior work in FL-ASR has scaled up to ~120M parameters [19] without DP. Thus, by comparison, our model with 250M parameters is more than 2x larger than the prior works in the domain and ~10x larger than conventional FL works. This scaling is particularly significant in the domain of DP because DP noise scales with model dimension. Additionally, successfully training such models in FL+DP was previously considered infeasible (Gao et. al [20]).  \n\n---\n\n**Summary**  \n  \nWe thank the reviewer for their thoughtful feedback and hope the responses above address their concerns. We will incorporate these clarifications in the final draft of the paper. We request the reviewer to reconsider their score in light of these clarifications and would be happy to provide further details if helpful."}, {"review_id": "Sq85OLNfFD", "reviewer": "Reviewer_R1Qm", "summary": "This paper addresses the challenge of applying Federated Learning with Differential Privacy to Automatic Speech Recognition, a task complicated by the heterogeneity of large transformer models. The authors establish the first practical benchmark for FL with DP in end-to-end ASR, demonstrating that per-layer clipping and layer-wise gradient normalization can mitigate gradient heterogeneity and clipping bias. Their empirical results show that FL with DP is viable under strong privacy guarantees, achieving competitive performance with only a small degradation in WER.", "questions": "1. How does the proposed method scale to even larger models?\n2. Could you provide more insights into how different DP budgets affect WER in practical scenarios?\n3. Does the FL+DP approach ever match centralized training performance, or is there always a gap?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\nThis paper establishes the first benchmark of FL+DP in large-scale ASR and makes an important contribution. The proposed layer-by-layer pruning and gradient normalization techniques effectively balance the model performance and privacy protection, and the demonstration process is relatively detailed. This work is highly original and demonstrates that FL and DP perform differently on large models compared to small models.\n\nWeaknesses:\nThe computational overhead of the proposed methods may limit scalability to larger models or real-time applications. \nAdditionally, the relatively high error rates of baseline models on open-source datasets like Librispeech suggest that more recent ASR models should be used as baselines to better validate the effectiveness of FL+DP on larger ASR systems.", "was_revised": true, "final_justification": "The rebuttal provided by the authors has addressed my concerns well, and I have decided to maintain my positive evaluation.", "rebuttal": "Dear Reviewer R1Qm,  \n  \nWe deeply appreciate your positive assessment of our work and your recognition of its importance as \"the first benchmark of FL+DP in large-scale ASR.\" We address each of your points below.  \n  \n\n> \"The computational overhead of the proposed methods may limit scalability to larger models or real-time applications\"\n\n  \nYou raise an important practical concern about computational overhead. We acknowledge this limitation while emphasizing that our work addresses a more fundamental challenge:  \n\n- Prior works such as Gao et. al 2022 [20] pointed out that training FL-ASR was infeasible. Some improvements were made by Azam et al. 2023 [56] to train a 120M Conformer model, but the training of FL with DP for ASR was still infeasible. Our primary contribution is establishing that such training is possible and studying the underlying reasons why this infeasibility existed.\n- We deliberately focused on the feasibility first, with efficiency optimization being a natural next step that merits dedicated research. We have already seen important insights from contemporary research in the domains of (i) parameter-efficient methods, (ii) gradient compression and quantization (iii) selective layer updates that might be helpful in this regard. However, each of these directions deserves substantial research, which we defer to future improvements that can be studied on top of our findings.\n\n---\n\n> \"the relatively high error rates of baseline models on open-source datasets like Librispeech suggest that more recent ASR models should be used as baselines\"\n\nWe believe that this assessment might be originating from the confusion between the model performance in the centralized and federated learning cases. For context, we compare against Whisper small (244M parameters, see Radford et al. 2022 D.2.2 - similar size to our model):  \n\n- For LibriSpeech test-other, Whisper small achieves 7.6% WER. In comparison, our centralized baseline achieves 6.8% WER on LibriSpeech test-other (Table 3, Appendix). SOTA models trained only on LibriSpeech achieve 5-6% WER for CTC models w/o a beam search and LM and are close to 3% with a beam search and LM - our model is ~3.5% when the beam search and LM are integrated. Our FL baseline on LibriSpeech test-other achieves 7.1% WER (only 0.3% degradation from centralized baseline). For context, prior work, Azam et al. 2023 [56], achieves 10.29% WER for FL setting with CTC-AED model.\n- For Common-Voice English, Whisper small achieves 14.5% WER. In comparison, our centralized baseline achieves 18.2% WER (Table 4, Appendix) without a beam search and LM. Our FL baseline on Common Voice achieves 25.7% WER. For context, no prior work in FL-ASR, including Gao et al. 2022 [20], uses Common Voice English. \n- Additionally, Gao et al. 2022 [20] are able to train a model using FL on Common-Voice French and achieve 20.99% WER. In comparison, we achieve 13.1% WER (Table 10, Appendix) on Common-Voice French, while Whisper small is reported as 22.7%.\n\n---\n\n> \"How does the proposed method scale to even larger models?\"\n\nWhile we were not able to train a larger model within the rebuttal timeframe, we hope that the following table comparing against a model with 114M parameters can shed some light on the phenomenon. We can observe that (i) the larger 250M parameter model is more affected by uniform clipping, and (ii) it also benefits more from the per-layer clipping.  \n\n|Model Size|T|Uniform Clipping|Per-Layer Clipping|Rel. Improv.|\n|---|---|---:|---:|---:|\n|114M|2k|29.5|21.3|27.8\\%|\n|250M|2k|31.1|20.4|34.4\\%|\n\nThis is in line with our theoretical analysis (Theorem 2) which emphasizes that per-layer clipping benefits model with greater depth H and layer-wise gradient heterogeneity (Ψ$_h$ terms) which is more prominent in deeper models (Chan et al. [23], Wang et al. [24]; discussed in Sec. 3). This confirms that **larger models benefit more** from our approach. This is intuitive since more gradient heterogeneity across layers can be expected in deeper models, thus benefiting from per-layer intervention, i.e., per-layer clipping and layer-wise adaptive optimization.\n\n---\n\n> \"Could you provide more insights into how different DP budgets affect WER in practical scenarios?\"\n\n  \nWe provide a detailed analysis in Table 17 (Appendix). There exists a **privacy-utility trade-off** between DP noise and WER, wherein larger DP noise leads to worse WER. However, we point to the trend that better privacy-utility trade-offs are realized by per-layer clipping when compared to constant clipping  \n  \nPractically, the target epsilon value often depends on the application, wherein a smaller epsilon ε ≤ 3 (i.e., more privacy) is preferred for training with sensitive data such as healthcare, and a slightly larger epsilon ε = 3-10 might be acceptable in other domains such as object detection, etc.  \n  \nFurthermore, population size dramatically impacts achievable privacy (Table 1). This is in line with prior works such as McMahan et al. 2018 [46].  \n\n---\n\n> \"Does the FL+DP approach ever match centralized training performance, or is there always a gap?\"\n\n  \nThis touches on fundamental aspects of privacy-preserving ML. In general, if the amount of data is similar between the centralized and FL training, then centralized training acts as an upper bound on FL training performance. This is due to the following:  \n\n- FL training observes more heterogeneous training as data cannot be shuffled, similar to centralized training.\n- DP adds random noise to the training, which creates an irreducible gap in performance.\n\n---\n\n**Summary**  \n  \nWe thank you for recognizing this work as making \"an important contribution\" with \"high originality.\" Your questions about practical deployment are exactly the right ones to ask as we move FL+DP from research to production. While computational challenges exist, this work addresses the fundamental challenge of the feasibility of training."}]

Author rebuttal:

Meta-review: Paper studies Differentially Private Federated Learning (DPFL) of automatic speech recognition (ASR) models. Compared to previous DPFL ASR works, authors are able to do DPFL on models owith 100s of millions of parameters and show that per-layer clipping and gradient normalization lead to better results. Authors provide theoretical analysis of their algorithm and a large set of promising empirical results.

Reviewer R1Qm pointed out concerns about scalability of the model. Authors answered that their focus was more on feasibility than scalability. Reviewer TNaa pointed out that these techniques are not novel in DP/FL literature and there exists much larger models for ASR. Authors responded by saying that FL literature usualy studied smaller models of size <30M. Authors also claimed that their results and analysis of per-layer operations departs from existing understanding in literature. However, meta-reviewer notes that references provided for these claims are very old and may not correctly charaterize the literature, e.g. (Ngyuen et al., 2023). Reviewer TNaa was not satisfied with these explanations. Reviwer UQPF had concerns about diversity of datasets and extrapolated results. Authors addressed these concerns with additional experiments. Overall, while the results are promising and extensive, the novelty of results is somewhat limited.

(Ngyuen et al., 2023) Nguyen, T. N., Nguyen, P. H., Nguyen, L. M., & Van Dijk, M. (2023). Batch clipping and adaptive layerwise clipping for differential private stochastic gradient descent. arXiv preprint arXiv:2307.11939.
Decision: Accept (poster)