Enhanced Federated Optimization: Adaptive Unbiased Client Sampling with Reduced Variance
Authors: Dun Zeng
Zenglin Xu
Yu Pan
Xu Luo
Qifan Wang
Xiaoying Tang
Venue: Accepted by TMLR ()

Abstract: Federated Learning (FL) is a distributed learning paradigm to train a global model across multiple devices without collecting local data.  In FL, a server typically selects a subset of clients for each training round to optimize resource usage. Central to this process is the technique of unbiased client sampling, which ensures a representative selection of clients. Current methods primarily utilize a random sampling procedure which, despite its effectiveness, achieves suboptimal efficiency owing to the loose upper bound caused by the sampling variance. In this work, by adopting an independent sampling procedure, we propose a federated optimization framework focused on adaptive unbiased client sampling, improving the convergence rate via an online variance reduction strategy.
In particular, we present the first adaptive client sampler, K-Vib, employing an independent sampling procedure. K-Vib achieves a linear speed-up on the regret bound $\tilde{\mathcal{O}}\big(N^{\frac{1}{3}}T^{\frac{2}{3}}/K^{\frac{4}{3}}\big)$ within a set communication budget $K$. Empirical studies indicate that K-Vib doubles the speed compared to baseline algorithms, demonstrating significant potential in federated optimization.

Published in Transactions on Machine Learning Research (01/2025)
Enhanced Federated Optimization: Adaptive Unbiased Client Sampling with Reduced Variance
Dun Zeng
umngdunn@std.uustt..eduu
University of Electronic Science and Technology of China
Peng Cheng Lab
Zenglin Xu"
zenglinam@@fudan.cda edu..cm
Fudan Unéversity
Shanghai Academy of AI for Science
Yu Pan
iperryuu@gmail.com com
Horbin Institute of Technology, Shenzhen
Xu Luo University of Electronic Science and Technology of China
frunk.luczS@outtook.cco com
Qifan Wang Meta Al
wafer618@gmaiie
Xiaoying Tang
Sungsiaaoyingeuuu..deee edu..
The Chinese University of Hong Kong, Shenzhen
Reviewed on OpenReview:
https: /// openeevee......e..e net/ forum? id= Hb.HHGG9ee
Abstract
Federated Learning (FL) is distributed learning paradigm train a global model across multiple devices without collecting local data. In FL, server typically selects subset of clients for each training round to optimize resource usage. Central to this process is the technique of unbiased client sampling, which ensures a representative selection of clients. Current methods primarily utilize a random sampling procedure which, despite its effectiveness, achieves suboptimal efficiency owing to the loose upper bound caused by the sampling variance. this work, by adopting an independent sampling procodure, we propose federated optimization framework focused adaptive unbiased client sampling, improving the convergence rate via online variance reduction strategy. In particular, present the first adaptive client sampler, K- K-Vib, employing an independent sampling procedure. K-Vib achieves linear speed-up on the regret bound T /K within a set communication budget K. Empirical studies indicate that K-Vib doubles the speed compared to baseline algorithms, demonstrating significant potential in federated optimization.
1
Introduction
This paper studies the prevalent cross-device federated learning (FL) framework, as outlined in Kairouz al. 2021 Zhang et al. (2024), which optimizes E X C Rd to minimize finite-sum objective:
N
N ME, [F(
min f(x) = Σ A.f(()) : mex
(1)
'Corresponding author.

---

Published in Transactions on Machine Learning Research (01/2025)
where N denotes the total number of clients, and À denotes the weights of client objective (A, 2 0. Ei
). The local loss function fi IRd - R is intricately linked to the local data distribution D.. It is defined as fi(x) = EE [2, E.)l, where &, represents a stochastic batch drawn from D.. Federated optimization algorithms, such as FVDAVG (MMMahan et al. 2017), are designed to minimize objectives like Equation (I by alternating between local and global updates in a distributed learning framework. To reduce communicatiin
and computational demands in FL. (Konččný et al. 2016 Wang et al., [021] Yang et al. 2022], various client
sampling strategies have been developed (Chen al.
2020 Cho et al. 2020b Balakrishnan et al.] 2022,
Wang et al., 2023 Malinovsky et al. 2023 Cho et al.
2023 These strategies are crucial as they decrease
the significant variations in data quality and volume across clients (Khan et al. 2021
Thus, efficient client.
sampling is key to enhancing the performance of federated optimization.
Current sampling methodologies in FL are broadly divided into biased (Cho et al.
2020b Balakrishnan et al.
2022 Chen & Vikalo, 2023) and unbiased categories [El Hanchi & Stephens] 2020; Wang et al. 2023). Unbiasod
client sampling holds particular significance as it maintains the consistency of the optimization objoctive Wang
2023 2020). Specifically; unlike biased sampling where client weights À are proportional to sampling probabilities, unbiased methods separate these weights from sampling probabilities. This distinction enables unbiased sampling to be integrated effectively with strategies that address data hetcrogencity (Zeng et al. 2023c Wu et al. 2024 Zeng et al.] 2024a),, promote fairness (Li et al. 2020c a), and enhance robustness (Li jet al. 2021 2020a). Additionally, unbiased sampling aligns with secure aggregation protocols for confidentiality
et al.
in FL Du& & Atallah 2001 Goryezka & Xiong, 2015 Bonawitz et al. 2017). Hence, unbiased client sampling
techniques are indispensable for optimizing federated systems.
Therefore, better understanding of the implications of unbiased sampling in FL could help to design better algorithms. To this end, we summarize a general form of federated optimization algorithms with unbiased client sampling in Algorithm [ Despite differences in methodology. the algorithm covers unbiased
sampling techniques (Wang et al., 2023; Malinovsky et al.) 2023 Cho et al. 2023 Salehi et al.) 2017 Borsos
ct al., 2018 El Hanchi & Stephens| 2020; Zhao et al. 2021b) in the litcrature. In Algorithm unbiased
sampling comprises three primary steps (relerring to lines 3, 12, and 14). First, the Sampling Procedure generates a set of samples St along with their respective probabilities. Second, the Global Estimation step creates global estimates for model updates, aiming to approximate the outcomes as if all participants were involved. Finally, the Adaptive Strategy adjusts the sampling probabilities based on the incoming information, ensuring dynamic adaptation to changing data conditions.
Typically, unbiased sampling methods in FL are founded on a random sampling procedure, which is then refined to improve global estimation and adaptive strategies. However, the exploration of alternative sampling procedures to enhance unbiased sampling has not been thoroughly investigated. Our research shifts focus to the independent sampling procedure, a less conventional approach yet viable for FL. We aim to delincate the distinctions between these methodologies as follows.
Random sampling procedure (RSP) means that the server samples clients from a black box without replacement.
Independent sampling procedure (ISP) means that the server rolls dice for every client independently to decide whelher lo include the client.
Building on the concept of arbitrary sampling (Horváth & Richtárik 2019 Chen et al.) 2020). our study observes that the ISP can enhance the efficiency of estimating full participation outcomes in FI. servers, detailed in Section 3 However, integrating independent sampling into unbiased techniques introduces new constraints, as outlined in Remark 2. Addressing this innovatively in Lemma 5.1 our paper studies the effectiveness of general FL algorithms with adaptive unbiased client sampling, particularly emphasizing the utility and implications of the ISP from an optimization standpoint.
Contributions This paper presents a comprehensive analysis of the non-convex convergence in FedAvg and its variants. We first establish a novel link between the cumulative variance of global estimates and convergence rates by separating global estimation results from beterogeneity-relaeed factors. Thus to reduce the cumulative variance, we introduce K-Vib, a novel adaptive sampler incorporating the TSP. Given the expected number
2

---

Published in Transactions on Machine Learning Research (01/2025)
Algorithm 1 FedAvg with Unbiased Client Sampler
Require: Client set 8 where |S] N, client weights X, times T, local steps Initialize sample distribution p" and model x.)
for time L in [T] do
Server runs sampling procedure to create S* p'
4: Server broadcasts x' to sampled clients i E 5
5: for each client E St in parallel do
6: 2; = x
7:
for local steps r in [R] do Ti (x)
2 = =
end for
10: Client uploads local updates 9 xx:0 R.
11: end for
12: Server builds estimates d' EiEs X /
13:
Server updates x²+1 = x ngd
14:
Server updates pl+1 based on {llgillies
15:
end for
of clients K to join federated learning al each round, K-Vib notably achieves expected regret bound of ÖNNTT8 /Kt), demonstrating wear-linear speed-up over existing bounds Ö(N+T$) (Borsos et al. 2018 and ON) Hanchi & Stephens 2020). Empirically, K- Vib shows accelerated convergence on standard federated tasks compared to baseline algorithms.
2 Preliminaries
We first introduce previous works on batch sampling (Horváth & Richtárik 2019) in stochastic optimization and optimal client sampling (Chen et al.] (020)) FL. We made few modifications to fit our problem setup. Remark 2.1 (Constraints on sampling probability). We define communication budget K as the expected number of sampled clients. And, its value range is from to N. To be consistent, the sampling probability p always sotisfics the constraint p 0, Ei K, [T] in this paper.
Definition 2.1 (Unbiasedness of client sampling S²). For communication round t E [T') the estimator d' is related to sampllng probability p' and the sompling procedure St p² We define a client sampling unbiased if the sampling St and estimates d' satisfy that
X d
N
Est [d']
E
X.g.. t=1
LEES
Besides, the variance of estimator d' can be derived as:
X
V(S*) Eginp'
Σ
g
(2)
P
where E||s*1] K. We omit the terms X, g for notational brevity.
Optimal unbiased client sampling Optimal unbiased client sampling should achieve the lowest variance as defined in Equation 2 It is to estimate the global gradient of full-client participation, iie.iiiii the variance of est imator d', Given a fixed communication budget K, the optimum of the global estimator depends on the collaboration of sampling distribution p' and the corresponding procedure that outputs St.
In detail, different sampling procedures associated with the sampling distribution p build a different probability matrir P N with the elements defined P Prob((i,j) S). Arbitrary sampling (Horváth &
3

---

Published in Transactions on Machine Learning Research (01/2025)
estimates error
300
0.03
Uniform
Undfom
Optimal R52
Ootimil
0.30
Optimal IP
Ostimal ISP
0.02
0.98
Aurts
0.26
0.24
0.01
0.02
200
400
600
8:00
1000
20
40
1.00
20
60
60
80
100
Communication Budgee X
Sampling Index
Sampling Intee:
Asis X
(a) Sampling procedure illustration experiments
(b) Case study
Figure 1: The variance of ISP estimates is lower than RSP. (a) Uniform indicates estimates with uniform probability. (b) The notations RSP(g: and ISP(g;, gj) represent the global estimates constructed through random sampling and independent. sampling, respectively, using sampled vectors gi and g5- Global indicates the full participation results. We can see (SP 9j) closer to the Global.
Richtárik 2019) has shown the generality of denoting arbitrary sampling procedure with a probability matrix for stochastic optimization. Inspired by their findings, focus the optimal sampling procedure for the FL server in Lemma 2.
Lemma 2.1 (Optimal sampling procedure, Horváth & Richtárik 2019 For any communicaatinn round E [T] in FL, random sampling yielding the Pi = Prob(i,j 8*) K(K 1)/N(N - 1), and independent sompling yielding P = Prob(i,j E St) =p'p, they admil
N
N
X gf
V(S*)
Σ 1=1
(1
(3)
N
P
1=1
ISP
RSP
The lemma indicates that the ISP is the optimal sampling procedure that minimizes the upper bound of variance. Then, we have the optimal probability by solving the minimization of the upper bound in respecting probability in Lemma 2.2
Lemma 2.2 (Optimal sampling probability, Chen al. 2020 Generally, can let a, >ullg/ Vi
[N], E [T] for simplicty of notation. Assuming O < a1 & integer for which 0 < K +1 N E.r we have
S ax and 0 < K s N, and I is the largest.
(K + 1- N)
1/1 V1s1,
p
(ISP)
(4)
1
if
to be a solution to the oplimization problem min, Σn probability for the RSP
In contrast, we provide the optimal sampling
p
K
a;
(RSP)
Σ)=1 aj
Therefore, the optimal client sampling in FL uses ISP with probability given in Equation
3 Case Study on Sampling Procedure
We suggest designing sampling probability for the ISP to enhance the power of unbiased client sampling in federated optimization. Lemma 2.1 has proven that optimal ISP induces a tighter upper variance bound than optimal RSP. To further clarify it, we provide empirical Example 3.1 and Example 3.2 to demonstrate three superior properties of ISP.
4

---

Published in Transactions on Machine Learning Research (01/2025)
Example 3.1. We randomly generate 1,000 vectors with the size of 1,000 dimensions. We set the budget K = (100, 200,.... 1000) and run. 100 times the RSP and ISP with their oplimal probability from Lemma 2.2 to estimate differences between the output estimates and full aggregation results. Then, we present the mean error of estimate results and two specific cases in Figure [(a)
Example 3.2. Consider a case N- 3, K 2 with 91 = V22 = (2.7,
it induces weights vector [l|g: 1I?: || [gs] |]] [1,3,6] 6] if omit X. We have optimal sampling probability p² K (0.1, 0.3, 0.6] for RSP and p" [0.25, 0.75, 1] for ISP with all possible estimate results in Figure T(b)
ISP handles budget K better than RSP With a minimum budget of K = 11, the ISP does not assign any client with probability 1, it returns to the optimal sampling probability of RSP by Equation ( If the budget is K > 1, the optimal probability of ISP changes, while the RSP does not. In other words, The ISP handles the dependency on the budget term K more effectively and produces better estimates compared to RSP. As illustrated in Figure [(a] the error in optimal RSP ceases to decrease when K > 600. Interestingly, the RSP estimates perform worse than the uniform sampling baseline with larger budget, as shown in the third plot of Figure ((a) This discrepancy ariscs because the optimal probability in RSP aims to minimize a loose upper bound on variance, as defined in equation Equation (3). Specifically, it only minimizes the norm of an estimate |p::/x;112 -1 g):Kt for each client. Thus, RSP does not account for the precision between its aggregation and true full results.
ISP estimates are asymptotic to full participation results As shown in Figure I(a) ISP builds estimates asymptotically to the full participation results with an increasing communication budget of K, while RSP does not. In Example 3.2 RSP with full participation (K = 3) will build estimates d' = (6.4, 5.9) and non-zero difference |d' -
= 0.6. In contrast, ISP with full participation induces p" = (1, 1, and l|d' -9 0. This reveals that RSP may fail to build valid estimates with more communication budget.
9.1 9
ISP creates expected sampling size The number of sampling results from independent sampling is stochastic with expectation K. It means that if we strictly conduct the ISP, the number of sampling results Prob[[S'] = K) * 1, but E[|S']] = K. Referring to the example, independent sampling may sample 3 clients with probability = 0.25 * 0.75 * = 1= 3/16 and sample only client with p = (1 - 0.25) * (1 - 0.75) = 3/16. Importantly, the perturbation of sampling results is acceptable due to the straggler clients (Gu et al.) 2021 in a large-scale cross-device FL system. Besides, we can easily extend our analyses to the case with straggler discussed in Appendix
E.1 Moreover, for scenarios that require restricting the probability of TSP from generating a larger selected client set, it induces additional constraints on Remark 2.1
We elaborate on this
constraint in Appendix E.2
The above advantages show that designing sampling probability and building global estimates with ISP is more promising. However, computing the optimal sampling via Equation 4 requires a norm of full gradionts, which is unfeasible in practice. Therefore, FL needs better design of its sampling probability for ISP based on limited information. In the remainder of this paper, we investigate the efficiency of ISP in federated optimization. Unless otherwise stated, all sampling probability p and sampling procedures are related to ISP in the remainder of this paper.
4 General Convergence Analyses of FL with Unbiased Client Sampling
In this section, we first provide general convergence analysis of FedAvg covered by Algorithm specifically focusing on the variance of the global estimator. Our analysis aims to identify the impacts of sampling techniques on enhancing federated optimization. To this end, we define important concepts below to clarify the improvement given by an applied unbiased sampling:
Definition 4.1 (Sampling quality). Given communication budget K and arbilrary unbiased client sampling probability pi, we measure the quality (lower is better) of one sampling step St ~ p by its expectation discrepancy to the optimal sampling:
Q(S²) == Estap'
Σ iESt
X pf
X
(6)
P
iES!
5

---

Published in Transactions on Machine Learning Research (01/2025)
where St ~ p' is the ISP, p" is obtained via Equation with full {llgf Iee and [* B||| K.
6 denotes the best results that can be possibly obtained subjected to communiction budget. It still preserves estimate errors to full results. Therefore, define the sampling quality of one sampling by its gap to the optimal estimate results for practical concern.
Remark Note that the second term of Equation
In practical settings, federated learning typically trains modern neural networks, which are non-convex problems in optimization. Therefore, our convergence analyses rely on standard assumptions on the local empirical function f. E [N] in non-convex federated optimization (Chen al. 2020 Jhunjhunwala al.
2022 Chen & Vikalo 2023).
Assumption 4.1 (Smoothness). Each objective f((x:) for all i E [N] L-smooth, inducing that for all Yx, y E Rd, it holds [[V.f(z=) Vf(li)ll S L||x= yll-
Assumption 4.2 (Unbiasedness and bounded local variance). For each E [N] and x E RR, we assume the access to an unbiased stochaslic gradient (a, of client's ' true gradient V V((()) i.e., Re,,Be,DD, [ (VPi(a, ff(()). The function f: have ot-bounded (local) voriance i.e.,E,,,D [[ V (x())) S of.
Assumption 4.3 (Bounded global variance). We assume the weight-aueraged global variance is bounded, i.e., Σ A (z) V f - og for all æ E Rd.
Assumptions 4.1 and 4.2 are standard assumptions in stochastic optimization analyses. Assumption measures the impacts of data heterogeneity in federated optimization. A larger upper bound of o denotes stronger heterogeneity across clients. Now, we provide the non-convex convergence of Algorithm
Theorem 4.1 (FedAvg with arbitrary unbiased client sampling). Under Assumptions 4.1 4.2 4.3. taking
upper bound E [f(=) f(x")] I M and W max A AiEEN given commundcation budget K, there always
ecists leurning rales 1p2g sz allow Algorillm 1
to generule ileration sequence {wr ..... x') "n? such that
2L
M31 T
M3 (3
ML T
min [VV['')) =o EE|T
O
+0
+ +
T
(7)
Sampking Utility
where
T-1 T E Q(S)), 82
T-1
0² 0(= 81
82 - 1=0
X (NW.NW + N W).
K
Notably, o² denotes the helerogeneily impacts stochastic gradients, B2 denotes the benefils of using oplimal sampling, and B1 denotes the benefits of using sub-optimat sampling.
The convergence guarantees quantify the impacts of client sampling quality the convergence performance of FedAvg. If we always use optimal client sampling (Chen al. 2020), the cumulative sampling quality implies B1 = 0. Then, the convergence rate returns to Theorem 18 &Chenn et al. 2020) and covers the state-of-art complexity guarantees in Karimireddy et al. (2020). As discussed previously, acquiring optimal client sampling is typically unfeasible in FL practice. Therefore, we may use sub-optimal client sampling techniques, which induce 31 as a non-zero value.
Interpretation of Theorem 4.1
Sampling utility Given communication budget K, the utility of optimal client sampling is linked to x (NW,NW W], where W max(/>))eeee For example, x' NW indicates the best case that optimal client sampling accurately approximate full results. Otherwise, the optimal client sampling implements sub-optimal approximation. Morcover, given an arbitrary sub-optimal client sampling stratogy, we use Q(S*) to measure the discrepancy between the applied sampling and optimal sampling. Using limited information when optimal client sampling is inapplicable, we expect to minimize the cumulative discrepancies 8, obtain optimization improvement. In all, the bound of the second term in Equation 7 is related the performance of the applied client sampler in FL. Therefore, minimizing the cumulative sampling quality 3 over federated optimization iteration directly accelerates the FL. convergence.

---

Published in Transactions on Machine Learning Research (01/2025)
5 Theories of the K-Vib Sampler
In this section, we introduce the theoretical design of the K-Vib sampler for federated client sampling, The adaptive sampling objective aligns with the online variance reduction (Salchi et al.) 2017 Borsos et al. 2018 El Hanchi & Stephens] 2020) tasks in stochastic optimization. The difference is that we solve the problem in the scenario of FL using ISP, which induces the constraints on sampling probability given in Remark 2.1
5.1 Adaptive Client Sampling as Online Optimization
To enhance federated optimization, we aim to minimize the sampling quality Q(S*) to achieve tighter convergence bound Equation Aud, using ISP variance, the upper bound of Equation can be known as:
l 1
N
1 Pi
Q(S*)
Then, model the client sampling objective online convex optimization problem Salehi et al. 2017
Borsos et al. 2018 El Hanchi & Stephens 2020). Concretely, define
local feodback function xz X |g/||, and cost function (p
Σ
P:
for a online convex optimization task rrepeccting sampling probability p. Therefore, online convex optimization is minimize dynamic regret defined as:
T Σ Q(S) - Regret (T) = T
T
T
T
1
l (p(p) Σ Σ - min le ((())
(8)
What does regret measure? Regret measures the cumulative discrepancy of applied sampling probability and the dynemic optimal Oracle. In Theorem I.1 we decomposed the cumulative sampling quality as an error term. And, the upper bound of cumulative sampling quality is given by the regret. According to Equation (3, the ISP induces tighter regret. Minimizing the regret. Equation (S) can devise sampling probability for ISP to create a tighter convergence rate for applied FL. In this paper, we are to build an efficient sampler that outputs an exemplary sequence of independent sampling distributions {p')T", such that limy-x Regret (T))T 0.
5.2 Analyzing the Best Fixed Probability
In the federated optimization process, the local updates g' change, making it challenging to directly bound the cumulative discrepancy between the sampling probability and the dynamic optimal probability. Consequently. we explore the advantages of employing the best-fixed probability instead. We decompose the Equation S
into:
T
T
T
T
Regrel D (T) = Σ 4(p') min Σ l((p) + min
(())
Σ
min (p))
(2)
Turp
The static regret Regrety(7) denotes the cumulative online loss gap between an applied sequence of probabil- itics and the best-fired probability in hindsight. The second term indicates the cumulative loss gap between the best-fixed probability in hindsight and the optimal probabilities. We are to bound the terms respectively.
Our analyses rely on a mild assumption of the convergence status of the federated optimization that sampling methods are applied (Wagg et al. 2021 Notably, stochastic optimization (Salchi et al. 2017 Duchi et al.
Please distinguish the online cost function fc(-) from local empirical loss of client ft(-) and global loss function f(). While 4() always convex, f()) and fi(-) can non-convex.
7

---

Published in Transactions on Machine Learning Research (01/2025)
Algorithm K-Vib Sampler
Require: N, K, T, 7: and 0.
Initialize client feedback storage w(i) = 0 for all i [N].
for time in [T] do
P w(( + P
by Lemma 5.1
- -1-0) -2)- 00. + No for all i [N]
Draw St ~ p'
ISP
Receive leedbacks Te (i), and update w(i) + (ii (1)/pt for i S
end for
2011
Bovd et al. 2004 and federated optimization algorithms (Reddi et al. 2020; Wang et al. 2020
ct al., 2019 Zeng ct al. 2024b) typically achieve a sub-lincar convergence speed O(1/ v T) at least. Therefore,
we assume feedback function related the convergence behaviors of local objectives f((), [M] using the following notions:
Notions We denote the overall feedback 11, := Σn (i) at t-th round. Then, denote the local convergence results n. (1) == lime-+so (i))), and the overall convergence results IT, : D n. m.(1,, Vi E [N]. Notably, we know Σ 11, 2 II, as FL converges and denote Vi Ein (Tx(1) (1)) YT 2 1,. Besides, we denote the largest feedback with G, i.e., (il & G, [T]. E [N]. Importantly, the G' denotes the largest feedback during the applicd optimization process, instead of assuming bounded gradients.
Then, we assume the convergence federated optimization will induce a decaying speed feedback function (local update norms ||g|| in this paper):
Assumption 5.1 (Convergence of applied federated optimization). As we discussed above the sub-linear convergence speed OT)/VT) can be obtained by general nonconver federated learning algorithms. We assume that (i) n. (i)l S ((1/V)))) and hence implies Vr S O(log(T)). The above assumptions guarantee the regret concerning basic convergence speed of applied FL algorithms, with an additional cost of Ö(vT).
Now, we bound the second term of Equation 9 below:
Theorem 5.1 (Bound of best fixed probability). Under Assumplions 5. sampling a set of clients with an expected size of K, and for any i [M] denote Vi(1) Σt (T. To (i)) Olog(T)). For any T 1, the averaged hindsight gap admits,
T
Vi(i)
Vi(i)
TEFP S
2TT
T
Sketch of proof. This bound can be directly proved to solve the convex optimization problem respectively. Please see Appendix D.1 for details.
Theorem 5.1 indicates a fast convergence of foderated optimization induces a better bound of Vi((), yiclding tighter regret. Therefore, it also covers better optimization problems implying tighter upper bound assumption for (m+( n. (i)l (e.g., strongly convex and convex federated learning problems). As the hindsight. bound vanishes with an appropriate FL solver, objective turns to devise a {Pi,...,Pry Pr] that bounds the static regret s in Equation
5.3
Upper Bound of Static Regret
We utilize the classic follow-the- wew.ettheergglriieeeeeeeddd (FTRL) (Shalev-Shwartz al. 2012 Kalai & Vempala
2005
Hazan
2012
framework to design a stable sampling probability sequence, which is formed time t;
N - (i) +
p' == arg min
(10)
pi
8

---

Published in Transactions on Machine Learning Research (01/2025)
where the regularizer y ensures that the probability does not change too much and prevents assigning vanishing probability to clients. It also ensures a minimum sampling probability Pmin for some clients. Therefore, we have the closed-form solution as shown below:
and 0 < aj - aB S ... S a'y and Lemma 5.1 (Solution to Equation 100). Letting of ( + 0 <<D K S N, we have
1,
if 2 l2,
p
Va
i/ E (1,12):
(11)
Pmiia
if S l
where at = EiE(lid) Va, 28 = K (N - 12) + 11 vi e (4,,)),
Pmin and the 1 S 4 S le S N, which sotisfies that
Pminn Enjsscia ag a Elnetet af
2t
2t
Remark. Compared with vanilla optimal sampling probability in Equation 4 our sampling probability especially guarantees a minimum sampling probability Pwin on the clients with lower feedback. This probability encourages the exploration of the FL system and prevents the case that Besides, the minimum sampling probability Pmin is determined by the 7 and the cumulative feedback from clients during training. For t = 1.... T, if applied sampling probability follows Lemma 5.1 with proper Y we guarantee that R (T'/ O(L/VT), as proved in Appendix D.2
clients never sampled.
However, under practical constraints, the server can only access past sampled clients' feedback. Hence, Equation 111 can not be computed accurately. Inspired by work et al. 2018), we construct an additional estimate of the true feedback for all clients denoted by P and let St ~ pt. Concretely, P mixed by the original estimator p' with static distribution. Let f E 1], we have
p' (1- ( 01- + K
Mixing strategy:
(12)
N
where 5¹ 2 OKK No and hence 5 x1))
S
OR G
OK
Analogous regularizer 7, the mixing strategy guarantees the least probability that any clients be sampled, thereby encouraging exploration. We present the expected regret bound of the sampling with mixed probability and the K-Vih sampler outlined in Algorithm 2 with theoretical guarantee in Theorem 5.2
Theorem 5.2 (Static expected regret with partial feedback). Under Assumptions 5.1 sampling St ~ f³ with G2 N 0 ) with T K > N, we obtain the expected ko
E||S = K for all I,...,T, and letting regret
E [[Regrety (T)) ((N+T$/Kt)
(13)
where Ô hides the logarithmic factors.
Skelch of proof. Denoting (i)) as partial feedback from sampled points, it incurs
21) =i(1) p
Thes and =
Analogous to Equation
S we define modified cost functions and their unbiased estimates:
N
70() and ((G(()))'' 6) (p()
E((p) = Σ =
Relying on the additional estimates, we have the full cumulative feedback in expectation. detail, we provide regret bound Regrety(T) by directly using Lemma 5.1 Appendix D.2 Analogously, we can extend the mixed sampling probability z to derive the expected regret bound [[Regret, given in Appendix D.3 ]
9

---

Published in Transactions on Machine Learning Research (01/2025)
Synthetic, K=10
Synthetic, K=10
Synthetic, K=10
0.30
unafrem
Axai
2.3
600
0.25
Dyramic Rogiet
Vib
400
2.1
0.35
2.0
200
0.10
1.0
0.09
1000
200
300
400
500
100
200
300
4.00
500
100
200
300
400
500
Communication Round
Communication Round
Communication Round
Figure 2: Evaluation on dynamic regre in Equation gradient variance in Equation
and loss.
Synthetic
Synthetic dataset, K=10
Oyranic Rages
50
1000
200
300
6.1
0.5
0.6
43
0:51 Depalrierr
Commeninaaaion fourd
(a) Regret improvement.
(b) Optimality of 0.
(c) Total variation distance
uniform probabilities.
Syrthetic, K=30
Synthetic, K-30
0.250
0:300
2.075
0,050
103
250
100
500
150
200
300
450
500
Communicatton Sound
Commun catton Brund
(d) Sensitivity to regularization 7.
Figure 3: Scnsitivity study on synthetic datasets.
Implications of ayperparameters The K- Vib sampler has two key hyperparameters, 7, 0. The Y inherited from the FTRL framework, guarantees the stability of the designed probability sequence. A larger value limits the exlent to which the sampling probability can change after each feedback update. The mixing strategy parameter 0 extends the FTRL framework into the partial foedback scenarios and is tuned to optimize the regret bound. Intuitively, during the early training stages, K-Vib explores the system information using a near-uniform sampling probability, controlled by these two parameters define the duration of the exploration phase. In the detailed proof in the Appendix, we suggest setting 7 G2 N in Equation (50) and 0K
0 = ( ) in Equation
51 for minimizing expected regret bound.
Enhanced convergence rate of Fed with K-Vib sampler The K-Vib sampler can work with a federated optimization process providing unbiased full result estimates. Comparing with previous regret bound O ÖN (Borsos et al. 2018 and T (El Hanchi & Stephens, 2020), it implements lincar speed up with communication budgel K. This advantage relies on lighter lormulation of variance obtained via the ISP Furthermore, the Assumption 5. holds in FedAvg. Meanwhile, the regret bound of the K- K-Vib sampler is independent of the convergence of Algorithm [ Therefore, the K-Vib sampler can accelerate FedAvg by minimizing the second term in Theorem 4.1 Concretely, the upper bound of dynamic regret in Equation 19 is dominated by static regret from Theorem
5.2 And then, we combine Equation (8) and substitute 3, in Equation (7) which derives that K- K-Vib sampler improves the second term O O(VMB,/T
in Theorem 4.1 to δ M4/2 For computational complexity, the primary cost involves sorting the (TK)9/3
cumulative foodback sequence [ in Algorithm 2 This sorting operation can be performed efficiently with an adaptive sorting algorithm Setivill-Castre Wood 1992), resulting in a time complexity of at most ONN log N). And, we provide sketch of efficient implementation in Appendix F.3
10

---

Published in Transactions on Machine Learning Research (01/2025)
6
Experiments
This section evaluates the convergence benefits of utilizing FL client samplers. Our experiment evaluation
aligns with previous works (Li el al. 2020b; Chen et al. 2020 on syuthetic and Federated EMNIST datasets. And, we additionally evaluate our method on language model and text datasets.
Baselines We compare the K-Vib sampler with the uniform sampling and other adaptive unbiased samplers including Multi-armed Bandit Sampler (Mabs) (Salehi et al. 2017), ; Variance Reducer Bandit (Vrb) (Borsos et al. 2018) and Avare (EI Hanchi & Stephens, 2020). We run experiments with the same random seed and vary the scods across five independent runs. We present the mean performance with the standard deviation (error bars). To ensure a fair comparison, we set the hyperparameters of all samplers to the optimal values prescribed in the original papers, and concrete hyperparameters are detailed in Appendix F
FL and sampler hyperparameters We use vanilla SGD optimizers for client-side optimization. And, we fix 1/3 = 1 on the server and tune yi for different tasks. For K-Vib sampler, we set 0 = ( _N (TR) )+, which aligns
with Theorem 5.2 Then, we guarantee the stability of designed probability via setting 7 x G2 N
8K In practice,
we suggest using the mean value of first-round client feedback as a naive estimate of G. The first-round feedback is typically the largest during FL training based on Assumption 5.1 This hyperparameter tuning experience can be applied in future applications.
6.1 Synthetic Dataset
We evaluate the theoretical results via experiments on Synthetic datasets, where the data are generated from Gaussian distributions (Li al. 2020b) and the model is logistic regression f(z) arg max(W x + b). We generate N 100 clients of cach has a synthetic dataset, where the size of cach dataset follows the power law. We set local learning rate vp- 0.02, local epoch 1, and batch size 64.
In Figure 2 we show the action of all samplers on three metrics. Concretely, the K-Vib implements a lower curve of regret in comparison with baselincs. Honce, it creates a better estimate with lower variance for global model updating. Connecting with Theorem 4.1 FedAvg with K-Vib achieves a faster convergence.
We present Figure 3(a) to prove the linear speed up about communication budget K in Theorem 5.2 In detail, with the increase of budget K, the performance of the K-Vib sampler with regret metric is reduced significantly. Due to page limitation, we provide further illustration examples of other baselines in the same metric in Appendix Figure 7 where we show that the regret bound of basclines methods are not reduced with increasing communication budget K. The results demonstrate our unique improvements in theories.
In Figure 3(b) we show the cumulated dynamic regret with 7 = le 4,T = 500 and different 0 = (0.1, 0.3, 0.5, 0.7, 0.9). The results demonstrate the importance of choosing the proper mixing parame- ter 0 for regret minimization. And, Figure 3(d) reveals the effects of regularization ? in Algorithm 2 The regret slightly changes with different y. The variance reduction curves remain stable, indicating the K-Vib sampler is not sensitive to y. This is because the regularizer y only decides the minimum probability in solution Equation (11). In Figure 3(c] we show how the hyperparameters control the divergence degree of sampling probabilities from uniform probabilities. Large 0 and small allow the TV distance to be large.
6.2 Federated EMNIST Dataset
We evaluate the proposed sampler on the Fodcrated EMNIST (FEMNIST) following Chen et al. 2020 for image classification. The FEMNIST tasks involve three degrees of unbalanced level (Chen et al.) 2020), including FEMNIST v1 (2,231 clients in total, 10% clients hold 82% training images), FEMNIST v2 (1,231 clients in total, 20% client hold 90% training images) and FEMNIST v3 (462 clients in total, 50% client hold 98% training images). We use the same CNN model in (McMahan et al. 2017). We set batch size 20, local epochs 3, 1 = 0.01, and k= 111, 62, 23 as 5% of total clients.
In Figure 4 the variance of data quantity decreased from FEMNIST v1 to FEMNIST v3. We observe that the FedArg with the Vib sampler converges about 3x faster than baseline when achieving 75% accuracy in FEMINIST v1 and 2x faster in FEMINIST v2. At early rounds, the global estimates provided
11

---

Published in Transactions on Machine Learning Research (01/2025)
FEMNIST
FEMNIST
FEMNIST v3
Los5
1.0
Uniite Mazs
Acare KVIo
1.00
1.0
0.50
0.25
0.0
100
200
300
400
500
100
200
500
400
502
0
100
200
500
400
500
Cammunication Round
Communication Round
Communic ation Round
1.0
1.0
1.0
Test Accurac)
0.4
044
0.4
0.2
100
200
500
400
300
100
2003
500
400
500
0
100
200
300
400
550
Communication Round
Communication Round
Communication Round
Figure 4: Federated EMNIST dataset experiments. Training loss and test accuracy of FedAvg with different unbiased samplers.
Number training samples
6:00
Faderated AGNews
600
Federsted AGiNews
600
Federated AGNows
Dersity
Dansky
by
2.00
200
200
600
800
1000
200
400
600
860
1000
200
400
600
8:00
1000
id
Clentic
Federated AGNews vl. K-25
Federated AGNews v2, K-25
Federated AGNews va, K-25
1.25
Unffom Mabs
1.25
1.25
Train LoGs
1.00
0.0
0.50
0.50
025
0.25
Communication
Communication Round
commonicaaion
(a) Experiments on AGNews dataset with Blrt model.
Numbar taaning Sar0ples
6000
Federated CCNews vl
Federated CCNews v2
6000
Federated CCNews v3
Dendity
Denotty
4005
4000
4000
2000
2000
200
KBD
800
1000
220
6:00
1503
200
1000
Oliont
Rederated CCNews v1, K-25
Federated CCNows v2, K-25
Rederated CCNows va, K-25
4.0
Unifoom Habs
35
3.0
2.5
50
100
150
200
250
300
100
1.00
200
250
300
Communication Round
Communication
Communication Round
(b) Experiments on CCNews dataset with Pythia-70M model.
Figure 5: Federated text dataset experiments.
by naive independent sampling are better as demonstrated in Lemma 2.1 it induces faster convergence by Theorem 4.1 Mcanwhile, the K-Vib sampler further enlarges the convergence benefits by solving an online variance reduction task. Hence, maintains a fast convergence speed. For baseline methods, we observe that the Vrb and Mabs do not outperform the uniform sampling in the FEMNIST task due to the large number of
12

---

Published in Transactions on Machine Learning Research (01/2025)
clients and large data quantity variance. In contrast, the Avare sampler fastens the convergence curve after about 150 rounds of exploration in the FEMNIST v1 and v2 tasks. On the FEMNIST v3 task, the Avare sampler shows no clear improvement in the convergence curve, while the V- sampler still implements marginal improvements. Horizontally comparing the results, we observe that the curve discrepancy between K-Vib and baselines is the largest in FEMNIST vl. And, the discrepancy narrows with the decrease of data variance across clients. indicates that the K-Vib sampler works better in the cross-device FL system with a large number of clients and data variance.
6.3 CCNews and AGNews Datasets
We evaluate the efficacy of the K-Vib sampler on two large sizes of models and datasets, including fine-tuning task on AGNews (Zhang al. 2015) and a pre-training task on CCNews (Mackenzie et al.) 2020). AGNews is a text classification task with 119,999 train samples and 4 labels. And, CCNews contains 708,241 articles. For models, fine-tune a pretrained language model DistillBert (Sanh et al.) 2019 on the AGNews task. DistillBert is a language model with 67 Million parameters. And, we train from scratch a GPT2 model called Pythia-70M (Biderman et al. 2023 (70 Million parameters) on CCNews using the next token prediction loss. For both Lasks, we partition the datasets into N = 1, 000 clients with three different. levels of heavy long tails (Charles et al. 2024). Then, we set communication round T = 300 and budget K = 25. We set local learning rate le-d. batch size 16, and epoch 1 for the local SGD optimizer. We report the data distribution and the training loss for comparing the convergence benefits in Figure 5
lext dataset that
Analogous to FEMNIST experiments, observe that K-Vib achieves 2x faster convergence than baseline methods, while baseline methods only implement a marginal improvement compared to uniform sampling. And, the improvement is related to the degree of data variance across clients. Moreover, we surprisingly observed that baselines can be less efficient than uniform sampling in some cases. This is because RSP derives a loose upper bound, which can be less effective when estimating results of larger dimensions. In the Appendix F.2] we show how the K-Vib sampling probabilities deviate from uniform probabilities. shows that K-Vib sampling can accurately adapt to local data distribution based feedback information. In all, our results prove that K-Vib can enhance woal-world FL applications, even large model training.
7 Extension, Limitation & Conclusion
Our theoretical findings can be extended to general applications that estimate global results with partial information. Additionally, our extension of independent sampling can be applied to previous works employing random sampling. Besides, the global estimate variance FT. also comes from the data heterogeneity issues, which may incur unstable local foodback, brcaking the Assumption 5.1 This can be addressed with client clustering Lechniques (Ghosh et al., 2020 Ma el. al.
2022 Zeng et al. 2023a), analogous previously cluster Besides, can replace FedAvg with more stable
sampling works (Fraboni et al. [021 Song et al. 2023).
FedAvg variants (Sun et al. 2024 Zeng et al. 2023c).
In conclusion, our study provides a thorough examination of FL frameworks utilizing unbiased client sampling techniques from an optimization standpoint. Our findings highlight the importance of designing unbiased sampling probabilities for the ISP to enhance the efficiency of FL. Building upon this insight, we further extend the range of adaptive sampling techniques and achieve substantial improvements. We are confident. that our work will contribute to the advancement of client sampling techniques in FT., making them more applicable and beneficial in various practical scenarios.
Acknowledgments
We sincerely thank the oditors and all anonymous reviewers for their time and constructive comments. This work was partially supported by the Major Key Project of PCL (No. 2022ZD0115301, and the National Key Research and Development Program of China (No. 2018AAA0100204).
13

---

Published in Transactions on Machine Learning Research (01/2025)
References
Ravikumar Balakrishnan, Tian Li, Tianyi Zhou, Nageen Himayat, Virginia Smith, and Jeff Bilmes. Diverse client selection for federated learning via submodular maximization. In International Conference on Learning Represenlations, 2022.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397 2430. PMLR, 2023.
Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcodone, H Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for privacy-preserving machine learning. In proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 1175-1191, 2017.
Zalán Borsos, Andreas Krause, and Kfir Y Levy. Online variance reduction for stochastic optimization. In Conference On Learning Theory, pp. 324-357. PMLR, 2018.
Zalán Borsos, Sebastian Curi, Kfir Yehuda Levy, and Andreas Krause, Online variance reduction with mixtures. In International Conference on Machine Learning, pp. 705 714. PMLR, 2019.
Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Conver optimization. Cambridge university press, 2004.
Antonin Chambolle, Matthias J Ehrhardt, Peter Richtárik, and Carola-Bibiane Schonlieb. Stochastic primal- dual hybrid gradient algorithm with arbitrary sampling and imaging applications. STAM Journal on Optimization, 28(4):2783 2808, 2018.
Zachary Charles, Nicole Mitchell, Krishna Pillutla, Michael Reneer, and Zachary Garrett. Towards federated foundation models: Scalable dataset pipelines for group-structured learning. Advances in Neural Information Processing Systems, 36, 2024.
Huancheng Chen and Haris Vikalo. Accelerating non-iid federated learning via heterogeneity-guide client sampling. arXiv preprinl 0rx::23310.001 2023.
Wenlin Chen, Samuel Horvath, and Peter Richtarik. Optimal client sampling for federated learning. arXiv preprinl Xiv:2000 13723, 2020.
Yae Jee Cho, Samarth Gupta, Gauri Joshi, and Osman Yagan. Bandit- communicttion-fficieett client selection strategies for federated learning. In 2020 54th Asilomar Conference on Signals, Systems, and Computers, pp. 1066 1069. IEEE, 2020a.
Yae Jee Cho, Jianyu Wang, and Gauri Joshi. Client selection in federated learning: Convergence analysis and ofwer-of-choice selection strategies. arXiv preprint arXiv:2010.0122 2020b.
Yae Joe Cho, Pranay Sharma, Gauri Joshi, Zheng Xu, Satyen Kale, and Tong Zhang. On the convergence of federated averaging with cyclic client participation. arXiv preprint arXi::230..0811 03109, 2023.
Aaron Defazio, Francis Bach, and Simon Lacoste-Julien. Saga: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in neural information processing systems, 27, 2014.
Canh T Dinh, Nguyen H Tran, Tuan Dung Nguyen, Wei Bao, Albert Y Zomaya, and Bing B Zhou. Federated learning with proximal stochastic variance reduced gradient algorithms. In Proceedings of the 49th International Conference on Parallel Processing, pp. 11, 2020.
Wenliang Du and Mikhail J Atallah. Secure multi-party computation problems and their applications: a review and open problems. In Proceedings of the 2001 workshop on New security paradigms, pp. 13 13-22, 2001.
14

---

Published in Transactions on Machine Learning Research (01/2025)
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7); 2011.
Ayoub El Hanchi and David Stephens. Adaptive importance sampling for finite-sum optimization and sampling with decreasing step-sizes. Advances in Neural Information Processing Systems, 33:15702-15713, 2020.
Vladmir Sstivill-Castry and Derick Wood. survey of adaptive sorting algorithms. ACM Computing Surveys (CSUR), 24(4):441-476, 1992.
Yann Fraboni, Richard Vidal, Lactitia Kameni, and Marco Lorenzi. Clustered sampling: Low-variance and improved representativily for clients selection in federated learning. In International Conference on Machine Learning, pp. 3407-341 3416. PMLR, 2021.
Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. Advances in Neural Information Processing Systems, 33:19586-19597, 2020.
Slawomir Goryezka and Li Xiong. A comprehensive comparison of multiparty secure additions with differential privacy. IEEE transactions dependable and secure computing, 14(5):463-477, 2015.
Xinran Gu, Kaixuan Huang, Jingzhao Zhang, and Longbo Huang. Fast federated learning in the presence of arbitrary device unavailability. Advances in Neural Information Processing Systems, 34:12052 12064, 2021.
Elad Hazan. 10 the convex optimization approach to regret minimization. Optimization for machine learning, pp. 287, 2012.
Samuel Horváth and Peter Richtárik. Nonconvex variance reduced optimization with arbitrary sampling, In International Conference on Machine Learning, pp. 2781-2788, PMLR, 2019.
Divyansh Jhunjhunwala, Pranay Sharma, Aushim Nagarkatti, and Gauri Joshi. Fedvarp: Tackling the variance due to partial client participation in federated learning. Tn Uncertainty Artificial Intelligence, pp. 906 916. PMLR, 2022.
Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. Advances in neural information processing systems, 26, 2013.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trends@ in Machine Learning, 14(1-2):1-2) 2021.
Adam Kalai and Santosh Vempala. Efficient algorithms for online decision problems. Journal of Computer and System Sciences, 71(3):291-300 307, 2005.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha. Suresh. Scaffold: Stochastic controlled averaging for fodcrated learning. In Inter- national conference on machine learning, pp. 5132 5143. PMLR, 2020.
Angelos Katharopoulos and François Flcurct. Not all samples are created equal: Decp learning with importance sampling. In International conference on machine learning, pp. 2525 2534.. PMLR, 2018.
Latif U Khan, Walid Saad, Zhu Han, Ekram Hossain, and Choong Seon Hong. Federated learning for internet of things: Recent advances, taxonomy, and open challenges. IEEE Communications Surveys & Tutorials, 23(3):1759-1798 2021.
Tachyeon Kim, Sangmin Bac, Jin-woo Lee, and Seyoung Yun. Accurate and fast federated learning via combinatorial multi-armed bandits. arXie preprint orXiv:2012.032: 03270, 2020.
Anastasia Koloskova, Nicolas Loizou, Sadra Borciri, Martin Jaggi, and Scbastian Stich. A unified theory of decentralized sgd with changing topology and local updates. In International Conference on Machine Learning, pp. 5381-5393. PMT.R, 2020.
15

---

Published in Transactions on Machine Learning Research (01/2025)
Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arviv preprint 1r110:1610.00 (610.05499 2016.
Tian Li, Ahmad Beirami, Maziar Sanjabi, and Virginia Smith. Tilted empirical risk minimization. ar preprint xxX::2007.00162, 2020a.
Tian Li, Anit Kumar Sahu, Manzil Zahocr, Maziar Sanjabi, Amoct Talwalkar, and Virginia Smith. Fedorated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429 450, 2020b.
Tian Li, Maziar Sanjabi, Ahmad Beirami, and Virginia Smith. Fair resource allocation in federated learning.
In TCLR. OpenReview.net, 2020c.
Tian Li, Shengyuar uan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning, pp. 1357-6368 PMLR, 2021.
Xiang Li, Kaixuan Huang, Wenhao Yang. Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. arXiv preprint ar 1rXi::1977.02188, 2019.
Jie Ma, Guodong Long, Tianyi Zhou, Jing Jiang, and Chengqi Zhang. On the convergence of clustered federated learning. Xrxie preprint arXiv:2202.0611 061877 2022.
Joel Mackenzie, Rodger Benham, Matthias Petri, Johanne R Trippas, J Shane Culpepper, and Alistair Moffat. Co-news-en: A large english news corpus. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pp. 3077-3080 2020.
Grigory Malinovsky, Kai Yi, and Peter Richtárik. Variance reduced proxskip: Algorithm, theory and application to federated learning. arXiv preprint arXiv:2207.44333 2022.
Grigory Malinovsky, Samuel Horváth, Konstantin Burlachenko, and Peter Richtárik. Federated learning with regularized client participation. arXiv preprint 233662230.00362, 2023.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication- efficient learning deep nelworks from decentralized data. In Artificial intelligence und statistics, pp- 1273 1782- PMLR, 2017.
Deanna Needell, Rachel Ward, and Nati Srebro. Stochastic gradient descent, weighted sampling, and the randomized kaczmarz algorithm. Advances in neural information processing systems, 27, 2014.
Zheng Qu and Peter Richtárik. Coordinate descent with arbitrary sampling i: Algorithms and complexity. Optimization Methods and Software, 31(5):829 857, 2016.
Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization. arXiv preprint arXiv:2003.008 00299, 2020.
Peter Richtárik and Martin Takáč. On optimal probabilities in stochastic coordinate descent methods. Optimization Letters, 10:1233-122 1243, 2016a.
Peter Richtárík and Martin Takáč. Parallel coordinate descent methods for big data optimization. Mathemationl Programmiing 156:433 484, 2016b.
Farnood Salehi, Elisa Celis, and Patrick Thiran. Stochastic optimization with bandit sampling. arXiv preprint xrXio:170s 02544, 2017.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1110.01100, 2019.
Shai Shaley-Shwarzz al. Online learning and online convex optimization. Foundations and Trends0 in Machine Learning, 4(2):107-194, 2012.
16

---

Published in Transactions on Machine Learning Research (01/2025)
Guangyuan Shen, Dehong Gao, DuanXiao Song, Xukai Zhou, Shirui Pan, Wei Lou, Fang Zhou, et al. Fast heterogeneous federated learning with hybrid client selection. arXic preprinl or arXiv:2208.051 05135, 2022.
Duanxiao Song, Guangyuan Shen, Dehong Gao, Libin Yang, Xukai Zhou, Shirui Pan, Wei Lou, and Fang Zhou. Fast heterogencous federated learning with hybrid client selection. In Uncertainty in Artificial Intelligence, pp. 2006 2015. PMLR, 2023.
Yan Sun, Li Shen, and Dacheng Tao. Understanding how consistency works in federated learning via stage-wise relaxed initialization. Advances in Neural Information Processing Systems, 36, 2024.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33: 7611 7623, 2020.
Jianyu Wang, Zachary Charles, Zhong Xu, Gauri Joshi, H Brendan McMahan, Maruan Al-Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, et al. A field guide to federated optimization. arXiv preprint arXiv:2107.0691 2021.
Lin Wang, YongXin Guo, Tao Lin, and Xiaoying Tang. Delta: Diverse client sampling for fasting federated learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Zheshun Wu, Zenglin Xu, Dun Zeng, Qifan Wang, and Jie Liu. Advocating for the silent: Enhancing federated generalization for nonparticipating clients. IEEE Tronsactions on Neurol Networks and Learning Systems, 2024.
Xiaohui Xu, Sijing Duan, Jinrui Zhang, Yunzhen Luo, and Deyu Zhang, Optimizing federated learning on device heterogeneity with a sampling strategy. In 2021 TEEE/ACM 29th International Symposium on Quality of Service (IWQOS), pp. 1 10. IEEE, 2021.
Miao Yang, Ximin Wang, Hongbin Zhu, Haifeng Wang, and Hua Qian. Federated learning with class imbalance reduction. In 2021 29th European Signal Processing Conference (EUSIPCO), pp. 2174 2178. IEEE, 2021.
Wenti Yang, Naiyu Wang, Zhitao Guan, Longfei Wu, Xiaojiang Du, and Mohsen Guizani. A practical cross-device lederated learning framework over 5g networks. IEFE Wireless Communications, 29(6): 128 134, 2022.
Dun Zeng, Xiangjing Hu, Shiyu Liu, Yue Yu, Qifan Wang, and Zenglin Xu. Stochastic clustered federated learning. arXio preprint arXiw:2303.008: 2023a.
Dun Zeng, Siqi Liang, Xiangjing Hu, Hui Wang, and Zenglin Xu. Fodlab: A flexible foderated lcarning framework. J. Mach. Learn. Res., 24:100 1, 2023b.
Dun Zeng, Zenglin Xu, Yu Pan, Qifan Wang, and Xiaoying Tang. Tackling hybrid heterogencity on federated optimization via gradient diversity maximization. arXiv preprint #xxx:2310.002000 2023c.
Dun Zeng, Yong Dai, Pengyu Cheng, Longyue Wang, Tianhao Hu, Wanshun Chen, Nan Du, and Zenglin Xu. On diversified preferences of large language model alignment. In Findings of the Association for Compulational Linguistics: EMNLP 2024, pp. 9194 9210, 2024a.
Dun Zeng, Zheshu Wu, Shiyu Liu, Yu Pan, Xiaoying Tang, and Zenglin Xu. Understanding gencralization of federated learning: the trade-off bet ween model slability and optimization. arXiv preprint urXiv:2411.16303, 2024b.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.
Yifei Zhang, Dun Zeng, Jinglong Luo, Xinyu Fu, Guanzhong Chen, Zenglin Xu, and Irwin King. A survey of trustworthy federated learning: Issues, solutions, and challenges. ACM Transactions on Intelligent Systems and Technology, 15(6):1-4) 47, 2024.
17

---

Published in Transactions on Machine Learning Research (01/2025)
Boxin Zhao, Ziqi Liu, Chaochao Chen, Mladen Kolar, Zhiqiang Zhang, and Jun Zhou. Adaptive client sampling in federated learning via online learning with bandit feedback. arxio preprint or Xiv::2 12. 14332, 2021a.
Boxin Zhao, Ziqi Liu, Chaochao Chen, Mladen Kolar, Zhiqiang Zhang, and Jun Zhou. Adaptive client sampling in federated learning via online learning with bandit feodback. arXiv preprint arx.:2112.143322 2021b.
Peilin Zhao and Tong Zhang. Stochastic optimization with importance sampling for regularized loss mini- mization. In international conference machine learning, pp. 1-9. PMLR, 2015.
18

---

Published in Transactions on Machine Learning Research (01/2025)
Appendices
Related Work
20
B Useful Lemmas and Corollaries
20
Auxiliary Lemmas
B.2 Arbitrary Sampling
21
B.3 Proof of Solution to Independent Sampling with Minimal Proba bility
23
C
Convergence Analyses
24
Analysis on Sampling
24
C.2
Non-convex Analyses
25
D Detail Proofs of Online Convex Optimization
29
D.1 Vanising Hindsight Gap: Proof of Thcorem 5.1
29
D.2 Regret of Full Information
30
D.3 Expected Regret of Partial Feedback: Proof of Theorem 5.2
33
E
Further Discussions
35
E.1 A Sketch of Proof for FL with Client Stragglers
35
Hard Capping for ISP
35
E.3
Differences between biased client sampling methods
36
E.4
Thcoretical Comparison with OSMD
36
F
Further Experiments
37
1
Experiment Details
37
2
Additional Experiments
37
3
Efficient Implementation
38
19

---

Published in Transactions on Machine Learning Research (01/2025)
A
Related Work
Our paper contributes to the literature on the importance sampling in stochastic optimization, online convex optimization, and client sampling in FL.
Importance Sampling. Importance sampling is a non-uniform sampling technique widcly used in stochastic
optimization KKtharoppoloss & Fleuret 2018) and coordinate descent (Richtárik & Takáë 2016a). Zhao &
Zhang (015): ; Needell et al. (2014) connects the variance the gradient estimates and the optimal sampling
distribution is proportional to the per-sample gradient norm. The insights of sampling and optimization
quality can be transferred into foderated client sampling, as we summarised in the following two topics.
Online Variance Reduction. Our paper addresses the topic of online convex optimization for reducing variance. Variance reduction techniques are frequently used in conjunction with stochastic optimization algorithms [Defazio et al. 2014 Johnson & Zhang, 2013) to enhance optimization performance. These same variance reduction techniques have also been proposed to quicken federated optimization ion (Dinh et al., 2020: Malinovsky et al. 2022 On the other hand, online learning (Shalev-Shward et al. 2012) typically employs an exploration-cxplottaiiee
paradigm to develop dccision-making strategics that maximize profits. Although some studies have considered client sampling as a multi-armett bandit problem, they have only
provided limited theoretical results (Kim al., 2020 Cho et al. 2020a Yang et al.) 2021). In intriguing combination, certain studies (Salehi et al. 2017
Borsos et al. 2018 2019 have formulated data sampling in
stochastic optimization as an online learning problem. These methods were also applied to client sampling
in FL by treating each client as a data sample in their original problem (Zhao et al. 2021a E1 Hanchi &
Stephens 2020
Client Sampling in FL. Client sampling methods in FL fall under two categories: biased and unbiased methods. Unbiased sampling methods ensure objective consistency in FL by yielding the same expected value of results global aggregation with the participation of all clients. In contrast, biased sampling methods converge to arbitrary sub-optimal outcomes based the specific sampling strategies utilized. Additional discussion about biased and unbiased sampling methods is provided in Appendix E.3] Recent research has focused on exploring various client sampling strategies for both biased and unbiased methods. For instance, biased sampling methods involve sampling clients with probabilities proportional to their local dataset size (McMahan et al., (017)), selecting clients with large update norm with higher probability (Chen) et al. 2020), choosing clients with higher losses [Cho al. 2020b), and building a submodular maximization to approximate the full gradients (Balakrishnan al.) 2022). Mcanwhile, several studies (Chen et al. 2020 Cho et al.) 2020b) have proposed theoretically optimal sampling methods for FL utilizing the unbiased sampling framework, which requires all clients to upload local information before conducting sampling action. Moreover, cluster-based sampling (Fraboni et al. 2021 Xu et al. 2021 Shen et al. 2022) relies on additional clustering operations where the knowledge of utilizing client clustering can be transferred into other client sampling techniques.
B
Useful Lemmas and Corollaries
B.1
Auxiliary Lemmas
Lemma B.1 (Lemma 13, Borsos et al. 2018). For any sequence of numbers C1...., er E [0, 1] the following holds:
33/2 S 44
where Ciat -E- Cr.
Lemma B.2. For an arbitrary set of n vectors a; E Rd
a;
< n
|a ||
(14)
i=
20

---

Published in Transactions on Machine Learning Research (01/2025)
Lemma B.3. For random variables 21, 21,...; 2n, have
E || +
zn |1*]
nk
|| .... + 1| 112
(15)
Lemma B.4. For independent, mean 0 random variables 21, 21..... 2n, we have
E | +.... zne
E2
|| 1|
+ 1|
(16)
Lemma B.5 (Tuning the stepsize Koloskova et al. 2020 ) For any parameters ro 20,6 there ezists constant stepsize n I y such that
0,e
0,
bro T
Fo 3 dro
V
+ by + ET/ S
+ 2/3
nf
Lemma B.6 (Upper bound of local drift, Reddi et al. 2020 Let Assumption 4.2 4.3 hold. For all client E [M] with step size m I 8LR arbitrary local iteration steps r E [R, the local drift can be bounded as follows,
E E| 2¹11 < 5Rm (of + 6Ro 6R [V[('))
B.2 Arbitrary Sampling
In this section, we summarize the arbitrary sampling techniques and present key lemmas used in this paper. The arbitrary sampling is mainly used either for generating mini-batches of samples stochastic algorithms (Chambolle et al. [2018] Richtárik & Takáë, 2016a) or for coordinate descent optimization (Qu & Richtárik 2016). In contrast, we explain the background in the context of federated optimization.
In detail, let S denote sampling, which is random set-valued mapping with values in 2|x), where [= NY. An arbitrary sampling S is generated by assigning probabilities to all 2 subsets of [N], which associates a probability matria P E NNNN defined by
Ps = Prob((i.j)
Thus, the probability vector = (pI.... PN) R is composed of the diagonal entries of P, and ==
Prob(i E 5). Furthermore, we say that S is proper if p; > 0 for all Thus, it incurs that
N
K = [[| Traco
ΣEe Pi-
The definition of sampling can be naively transferred to the context of federated client sampling. We refer to K as the expected number of sampled clients per round FL. The following lemma plays a key role in our problem formulation and analysis.
Lemma B.7 (Generalization of Lemma Horváth & Richtárik (011]). Let a1 a2......... aN be vectors in Rd and let a = Σm A,a; be their weighted average. Let S be a proper sampling. Assume that there is u E R such that.
P pp' piss(ppvi,puu. PNUN).
(17)
Then, we have
N
Esup
Σ
A
a
U2 la.
(18)
p
ieS
where the expectation is taken over sampling S. Whenever Equation
17 holds, must be the case that
2 1 1
Moreover, The random sampling admits 0, makes Equationn (18) hold equality.
N-K The independent sampling admits v; 1 P; and N
21

---

Published in Transactions on Machine Learning Research (01/2025)
Proof. Let 4is 1 if E S and kes = 0 otherwise. Similarly, let Lijes if E S and I.jes = 0 otherwise. Note that [B[iess] P: and B[[laess] = Pij. Then, we compute the mean of estimates á Eies a.a:
A.a ps
N
A Ties
A E IEs)
E[a] = E
= E
aa = ā.
pi
Ps
iES
Let A [G ......., CN] RdxN where ;= A.a, and let e be the vector of all ones R.N. We now write the variance of à in form that will be convenient to establish bound:
p.
Ellla
E[a]]
|| |B[a]]|
B[|
Σ
Aa;
11-1- |a||
pi
iES
: X LiJES
-|la|1
pi
Pj
(19)
: Xوaj
Pij
Σ
>;>;aT aj
pi
Pj
Σ
(Pij - p.P;)) C: Si
hd
((P -p pp ) ATe
Since by assumption we have P pp
Diag(po o v), we can further bound
e ((P -IP-p1 ppp) AATA)ess e
(Diag(poo) o
Σ p.v. ||::11
(20)
To obtain Equation 18). it remains to combine Equation 20 with Equation 19 Since P - pp" T is positive serni-definite (Richtárik Takád 2016b we can bound pp NDiag pp Diag(p v), where
v (11-p.).
Overall, arbitrary sampling that associates with probability matrix P will determine the value of v. As result, we summarize independent sampling and random sampling follows,
Consider now the independent sampling.
Pi (1 Pt)
0 Pa (1 - P2)
0
pp
Diig (pr Un)
Pr. (1 p..)
where V; - P:- Therefore, independent sampling always minimizes Equation 18 as equality.
making it hold
Consider the random sampling,
K
R2
K(K-1)
K(K
N
N
NIN
N
-)
K(K
NON
NVN
P P-pp-
K(K-1) N(N-1)
K(K-)
K
N(N-I)
22

---

Published in Transactions on Machine Learning Research (01/2025)
N-K for As shown in Horváth & Richtárik 2019 the standard random sampling admits v; N-1
Equation
18
Conclusion. Given probabilities that defines all samplings S satislying p; Prob(i E S), it turns out that the independent sampling (i.e., Pay = Prob(i,j E S) Prob(i E S)Prob(j S) = P.P,) minimizes the upper bound in Equation 188. Therefore, depending on the sampling distribution and method, we can rewrite the Equation
18 as follows:
V(S) = Eswpll
Σ
A.a;
alp?]
E P.)
N
X a2.1
N
K
N
X a
(21)
P:
P:
P
iES
i=1
Independent sempling procedure
Random sampling procedure
B.3 Proof of Solution Independent Sampling with Minimal Probability
In this section, we present lemmas and their proofs for our theoretical analyses. Our methodology of independent sampling especially guarantees minimum probability of clients in comparison with Lemma 2.2 Our proof involves general constraint, which covers Lemma 2.2 Then, we provide several Corollaries B.1 B.2 for our analysis in the next section.
Lemma B.8. Let 0 < 01 S a2 S ... S. aN and 0 < K < N. We consider the following optimization objective
with a restricted probability space = {p E IR N Pmin I P: S 1, E P: = K, Vi E [N]} where Pmnin I K/N,
N
a
minimizepEs
1(p)
p
im1
(22)
subject to
Σ ps K
i=1
Pmán S pi 1, i= 1,2,...,
Proof. We formulate the Lagrangian:
C(p.y.a,, aN, aN, ..... BN)
Σ
Σ P. K)
Σ a. p.) Σ Bi(p: 1). (23)
p
The constraints are linear and KKT conditions hold, Hence, we have,
if Vu S a;
a?
pi V
if V Pmin < as < vv.
(24)
1 as + Bi
Pminn
af, S vD Pmin.
Then, we analyze the value of y. Letting 1 = [ila, S Vy Pminn H 12 = l + HV V Pmii < <a < vull
N kilvos a:} and using EN p. K implies,
Σp= Σ D
Σ P + Σ pt =1=1 l Pmin +
Σ hasch
+ N 12 K.
1=1
ish
12
Arrange the formula, get
En Eicic a;
(25)
K- N + lg Pmin
23

---

Published in Transactions on Machine Learning Research (01/2025)
Moreover, can plug the results into the objective to get the optimal result:
N
a
a
a
Σ
E
Σ
+
P
Pi
P
p.
i=1
ish
licca
12N N21
Eish, ag
+ vo(
Σ
a;) + Σ a
(26)
Pmnin
liich
82N-11
Eish, ot
(1i,<eel, a,)²
Σ
af
Pmin
K- (12 11 Pmmin)
12N-12
where the 1 2 < N, which satisfies that Vi E (1,,2),
Enciks a;
Enerch
pmin
a, < K N + la
+12 1 Pmin
Punin
short, we note that if let Pmin = 0,4 = O, the Lemma 2.2 is proved as a special case of Equation 26
Besides, we provide further Corollary B.1 and B.2 as preliminaries for further analysis.
Corollary B.I. With ON s Eis ag and pmiin O, we have I = 0,12 N for Equation 26 and induce
(Ei=1 a,,² K
arg Sgmmin
Corollary B.2. With K aN < E
N
ag and Peise > O, we have 12 N and l is the largest inleger that Penin- The optimal value of Equation (26) becomes
di, salisfies 0 < (K- l Pmiin
<
Eish a
Vol Σ a:) ISiSN
Eq. 26 def. in line 2
Pi
Pmnn
Σish a²
+ y(K (ipmman)
Eq. 25 replacing
Σ a
Pinia
USSN
< 11Pmmn + y(K - (1pmin)
Eg. a4
VV Prnin
(EN a:)²
K(EN,, a²))
K
(K 1.pinn) )2
(K N Pmin) ]2
K(EN a:)²
< (K Npmin)) )
Convergence Analyses
C.1 Analysis on Sampling
We start our convergence analysis with a clarification of the concepts of optimal independent sampling. Considering an Oracle always outputs the optimal probabilities p", we define
8 =
E
Σ IES
X.g
N
X.
27
=
E
N
P 1
11
P
1=1
24

---

Published in Transactions on Machine Learning Research (01/2025)
where we have || equation to obtain
|| P. Then, plug the optimal probability in Equation 4
into the above
N
2
6
P
|1g
= E
Σ
||
- Σ
|
p
K - (N - 1)
Using the fact that Kllg5v1
S
Di=1
N
|
we have
8 E
|
Σ |9 11
2
K Σ Σ
1
EE
1
2
(
27
N K NK
E
|
To clarify the improvement of utilizing the sampling procedure, we provide two baseline analyses respecting independent sampling and random sampling. For an uniform independent sampling S U(p; = N we have
8
E
Σ iES!
gt
N
g
E
K
N
K K
Σ
(27)
Here, we provide the definition of optimal factor to quantify the improvement of using well-designed sampling probability.
Definition C.1 (The optimal factor). Given an iteration sequence of global model {æ ( x'}, under the constraints of communication budget K and local updotes statues (g) bE t [T], we define the ëmprovement factor of applying optimal client sampling St. ~ p² comparing uniform sumpling U as:
E
Eies:
d ot
X
a
E
Eieu+ d
and optimal p is computed via Equalion
with (oiheNN-- Moreover, we can know
2
de gf - Eim1 A Eies-
p.
a
du
E
Σ iEs'
Ai
o
p,
(28)
KE
Emm
N
((
E
(1
$1.
NKE
Σ Ei=1
|0
NE
Σ i==
||
C.2 Non-convex Analyses
Our non-convex convergence analysis follows the standard framework the optimization literature (Roddi et al. 2020: Chen et al. 2020). In particular, we decompose the sampling quality from the conventional framework in Equation 31 as this paper focuses the efficiency of sub-optimal client sampling in federated
25

---

Published in Transactions on Machine Learning Research (01/2025)
optimization. Referring to the discussion in the main paper, can compare the efficiency of the applied sub-optimal samplers in federated optimization and optimal client sampling (Chen et. al. 2020
We start the convergence analysis by recalling the updating rule during round as:
A.g! = x² nnd where gi = xx x..iR 171
R V (2, 1.
x 1+1 x²
VES* p.
Without loss of generality, we rewrite the global descent rule as:
Σ X
++1 x²
94 :- x²
R
p
iES'
where y= Ripgg o Σ r= VF(
Therefore, we know [g( E X.g Moreover, we denote
W (ax(A,)ieMM)
Descent lemma. Using the smoothness of f and taking expectations conditioned on x' and over the sampling St, we have
E [f(x(+]]] [ - njd'' < E[f(=')) (())() Rs: [d' + L
2 " [||c5]]]]
L 7T [1
< < T/li|
V5(x'') - gg (d')
L
(29)
$ f(x²)) - -"- "(((''))
+ バタ B fz') - Es (d ||
+ ク 1 [|
L n?
$ f(x²) "
2 [ff(='))1
バ/ E [IV)(x') - Es |1
E
[[|
d
2
R2
where the last inequality follows b) S 1 |a||2 + |bb1 Vu, b Rd.
Bounding Ti. We first investigate the expectation gap between global first-order gradient and utilized global estimates,
N Σ X
E [[V('') Eg [d'] =E E
(
M
R
1 Σ
E
A;
Vfi(x')
R
1
E
V1((2')
V2 1) + vi
-
R r=1
N
R
N
< 2E
X R
Vi(((²)) - vf((o!--)
+ +2
V
f((2 4r
- -VF((a
R
26

---

Published in Transactions on Machine Learning Research (01/2025)
Then, using the L-smoothness of f and unbiasedness of local stochastic gradients, we know
1
+2 of
fl')((') Es Es-(d')) [d'1]]]]]] s 2E
Σ
Vf((a')
R
NV
212
B
xx x
1-
1
of
R
i=1
N
o
< 21,
A (5Rmh of
+
30R
|
E
D. using Lemma
B.6
R
1=1
< 10RL + 60 D
L2
X. / + +
+0//+++0)+++ + oं
<
6
60R (10RT2 + - 200
Then, using 16RT. S m S SRT. we have
15
+ +102 (u+ 5nß re 1 )o? + +RRo²) Bn
[lVV((x)) -Es [d]
B|
16
(30)
15
67L2 (o0)) Roff/Re + o R2n
Bounding Tz. Now, we need to bound estimates:
A
N
N
E 21 <
E
X
+ Σ
p
1E.S
1=1
Aggi
X.g
X
N
27
N
(31)
< E
+ E
Σ
+ +
Σ
p
p
P
NES!
SES
lES.
i=1
((S+)
(A)
Here, the Q(S*) indicates the discrepancy between applied sampling and optimal sampling. The term (A) indicates the intrinsic gap for the optimal sampling to approach its targets and the quality of the targets for optimization.
27

---

Published in Transactions on Machine Learning Research (01/2025)
We can bound the term (A) follows:
(A) = E
Σ
+ E
X
Pi
1E8
i=1
i=1
N
N
K
1
E
g 11
E
g
D using Eq.
27 and Eq.
28
K
i=1
N
K
A
E
|
N
A
TE
|
K
i=1
N-
K
N Σ
+
F
| 1
K
i=1
a
(N K)
N
+ N
A.E |ooi| W Σ
K
==1
a (N- - K) +N) W (5 30RR Σ A,
E.E
V fi((*)))3
P using Lemma B.6
K
(N - K) + N Wni
using Assumption
K
Letting X" (N-KK K + N) W, we obtain
= + 30R2x''1 B (f('')]
Therefore, we have
T2 E [|dd]] [2]] <
/R + + +3xx 1 (f('')]
(32)
Putting together. Reorganizing the descent lemma, we obtain
|f 2(f(x') [[(x(+1]]
Ln T2 R2m
+ +7 +
1
Then, taking full expectation on both sides and substituting corresponding terms in Equation Equation 30 and Equation 332 to finish the descent lemma, we have
29 with
B| < 2(f(x'' E [[])
7
+ El V f= 21 + 60L (90f R o R²1
16
Ln
+
Q+5((S*) +
- 5x
R R + 30LX 'E [f('')||
n#
Using 17 = Rym,, and
1 16AL S m S
ERI we have
1
1 (1 - 40004 ( < 2(f(x²) E [E((at+))]) + 2561. 5x
16
600.2
(0? / R + ?n?
(33)
113
2(f(x') E [[(()+)))
1
Q(5*) + +5x
n 7+
256L
28

---

Published in Transactions on Machine Learning Research (01/2025)
where the last incquality uses o² = O oz)) and sets fg 2 Then, we have the descent lemma
s2 without loss of generality.
1
(1 48)
[ff(" 2(f(x') - E [f(x(+)]])
1 Q(5*) 5x 5x"LO² + 0212
+
(34)
256L
Then, taking averaging of both sides of Equation
34
over from time to T 1, we have
480Lx'
m) E
< 2(E [f(x") f(x"))
T-
1
T-1
vff="))
Σ
Q(S))
Σ
- (4-5Lo))++0
T
16
T'
256L
T
2=
tm0
Supposing upper bound E[f(x0) f(x²²)
s M and defining
T-1 B1
Q(S),
Q(*),
B2
T
I
Σ
x
min
(1
480Lx'n) m1)
0
T 1=
rE
16
=0
use Lemma B.5 with 17 S
to tune the stepsize n and obtain
(2/55 + +LL0 80)
$
2M
3
16LM
P min E (f(*') < E
+ +2(500)t
+
T
T
T
which concludes the proof.
Detail Proofs of Online Convex Optimization
D.1 Vanising Hindsight Gap: Proof of Theorem 5.1
We prove this theorem by directly solving a convex optimization problem, w.r.t the sampling probability. We first arrange the term TBFP in Equation as follows,
T
T min (p) = min
T
T
N
21)
min 1=1
(p)
Σ
Σ
min
(35)
P
P
t=1
==
[=1
1=1
Here, we recall our mild Assumption 5.1
(.) == lim Te (1), 11, Σ (i), vi [N].
Then, denoting Vi(1)= ETs 2 we bound the cumulative variance over time T per client
E [N).
n. (1) (T) (T))))²
ST 21) + 2n.
(T) - (.) + Σ (Te( - (i))²
(36)
ST. 20 + 2m. (i) T T. Vi(1) + Vi(1)
-T' (1) + Vi((i)
T
29

---

Published in Transactions on Machine Learning Research (01/2025)
Using the Lemma 2.2 and non-negativity of feedback we have,
(((1)
(Ei= mT(
min
(37)
Pi
K
We obtain the upper bound of the first term in Equation
35
T
N
N
[ (i)
T
min
min
p:
Ps
K
1=1
N
Vi
(38)
(1)
T
T
N
(i
N
Vi
2
II:
+ 21I.
K
T
where we use Lemma 2.2 in the second line, and Equation Then, we bound the second term Equation
36
in the third line.
1
II
(1)
(i())
S
(i))
T
T
(39)
K
(i)
min
P
P
where the first incquality uses the average assumption, the third incquality uses Jensen's incquality, and the last inequality uses Equation (37
Overall, we combine the results in Equation
38
and Equation
33
and conclude the proof:
T
Vi(
N
Vi(i)
min
((())
Σ min (p))
S
21I
P
K
T
T
t=1
D.2 Regret of Full Information
In this section, we prove the preliminary theorem for Theorem 5.2 We first investigate the upper bound of (T) in an idcal scenario called full feedback, where the server preserves feedback information of all clients, i.e., (- (i)) Vi E [N],4 E [T]. Technically, we prove this bound follows the framework of FTRL in online convex optimization, w.r.t sampling probability. Then, extend it into practical settings in the next section and derive the expected regret bound.
Theorem D. (Static regret with full information). Under Assumptions 5.1 sampling a batch of clients with an expected size of K, and setting 7 = G2, the FTRL scheme in Equation
10 gields the following regret,
T T Eu(p' min (((p) N
(22G z
2V6NG
N Eviir
22NG2
(1) +
(40)
K
z
where note the cunulative feedback L ((VT) following Assumption
5.1
Proof. We consider a restricted probability space A = {p E R. lp. 2 Puinn ΣN P. K. E [N] where Pmin < K/N. Then, we decompose the regret,
T
Regrets(T)) =
Σ
4('') min PEA
(((
min DEA [=1
4(())
min
4(p)
(41)
(4)
(B)
30

---

Remark. According to the above inequality, we note that the stability of online convex optimization is highly related to the changing probability. We can have trivial upper bound T (&(''(()) ( (p'+1)) D =i( (1/Pmin - 1), which indicates that the stability is restricted by Pmin- Solving the sampling Published in Transactions on Machine Learning Research (01/2025)
We separately bound the above terms in this section. The bound of (A) is related to the stability of the online decision sequence by playing FTRL, which is given in Lemma D.1 The minimal results of directing calculation bound term (B).
Bounding (A). Without loss of generality, we introduce the stability the online decision sequence from FTRL. to cost function l as shown the following lemma(Kalai & Vempala, 2005) (similar proof can also be found in (Hazan 2012; Shalev-Shwartz et al. 2012)).
Lemma D.1. Let K. be conver set and R K + R be regularizer. Given sequence of functions {4 {4eeet defined over K, then setting p' = arg pERN
E (- + R(p) ensures,
Σ 4(p')
Σ (p) t=1
Σ (4(0') - ((p (R(p) R(p')), Vip')),Vee VpEK. t=1
We note that R(p) E 9/p; in our work. Furthermore, R(p) is non-negative and bounded by N Numin with P A. Thus, the above lemmn incurs,
T Σ h(p)) S Σ (4(p')) ()) +
Ny
E 4(l'
(42)
Pmin
Bounded Below
To simply the following proof, we assume that 0 < (t) S ... ... ( 'T] to satisfies Lemma B.8 without the loss of gencrality. The stability relies on the evolution cumulative feedback (f) and hence relies on the index in solution 4 according to Lemma 2.2 Following the Lemma B.8 we have
1.
if 12
P.
+
if (H, 12) ih
(43)
21
Prain+
where 24 = K-N Pmin S K and ct = Eiegg (1) +7 Ein V (T) + 7 factor Then, we investigate the first term in the above inequality,
is the normalization
T
T
N
Σ (ell')) (((p
7i(1)
P
P 1+1
probability requires sorting cumulative foodbacks (f), the combinations of client-index and pt are dynamic. Hence, directly bounding the above equation generally can be difficult. obtain a tighter bound for FTRL, we investigate the possibility.
Lemma D.2. Assuming that pt < t for all i [N], [T - 1], the upper bound of
1 pi
is given
p.
by:
C1
c++1
0 E
<
( ( +
(44)
P
P
min(2t, 2041)
(1)
Proof. For all & [T], we have cumulative foedbacks Tia-1 (i), E [N] on the server. The server is able compute results Equation [1 the cases below:
As we are interested in the upper bound, we assume p p .+1 and discuss
31

---

Published in Transactions on Machine Learning Research (01/2025)
Caase 1: letting (p), P.
+ = (Puin 2t+1
va,,(0+7), ) c++1
we have
c++1
P
p. pt
Pmin
2+41 V (T. (1) +
C
c++1
V (1) +1)
2411 / (1) (1) +
Ct
C++1
min(2e, 21-1)
(1) +7
V +
10+7
where the second incquality uses Equation
(24
indicating Puiin
2
Ct
Case 2: letting (p', Dol.p/11 +1) = (zr 1
[x (1)
Equation 644b naturally holds.
0++1
Case 3: letting (P..P. 1) (2e 1i 10)42
+0+
1), can know that 2141
by Equation 24
62
6441
and prove the conclusion analogous case 1.
Case 4: analogous to the case 1 and 3, letting (p/, +1;1) (Pmin: 1), Equation
44
naturally holds.
Summarizing all cases conclude the proof.
Using Lemma D.2, we are ready to bound the stability of the online decision sequence:
T
T
N
Σ (l((')) - le(p
Ti(1)
Ct
c++1
BHIV (1) +
1=1
1=1
21
+7
T
N
DCc S C+11
min (22, 2141)
(Tite- +v
T
min (Zt, 2411 (i) 77 (V++ 1 +7
x2(() δe
72(1)
T N 1
(1)1
IVI+ 15 2
2
min (21, 2111 V (1) +7 (TT.-- (0) +7)
where the third line uses definition a S à = EN
+
Letting 7 G2, we have that (1) < ( +7 and ( S (1) and conclude the bound,
+% We define 2 = min(21]7
T E - -c(0")))((p(+1)) 5
T N
T()1
2
(T)E
1=1
1=1
i=1
=G :т
(z/0//))
Lemma B.1
2
(TT())))))))
(45)
22NG
N
(1) G
DDefinition of öz
N Σ
22NG
(V +G)
32

---

Published in Transactions on Machine Learning Research (01/2025)
Finally, we get the final bound of (A) by plugging Equation follows,
45
into Equation
42 and summarizing
T
22NG
(vitr(0) (G)(0)+))+++)
NG
(pp')
(((p) -
2
Pmin
Bounding (B). Using Corollaries B.1 and
B.2
we bound the term (B) as follows,
min PEA
(p))
min
(())
P
1=1
K(> N
Ci=1
T (iz((i))
(K
N Punin )2
K
(46)
K
K
Npmmnn2
N
6 Pinin
(i)
K2
In the last line, we use the fact that
(1-2)² 1 I 6z for [0, 1/2]. Hence, we scale the coefficient
K
1
6.N Pmin K2
KK
Npmin)
K K K(11- N Pminn KKY²
where we let Pmin <K/2NN).
Summary. Setting = G2, and combining the bound in Equation
42
and Equation 46
we have,
Regrets (T)
Ed(p')
4pp')
-mn Σ (p))
1=1
(47)
22NG N
(Vrim()+))
+
NG2
6.N
Σ
N
V
+ Pmin
K2
The Pinin is only relevant for the theoretical analysis. Hence, the choice of it is arbitrary, and we can set it to Pmin min { K/(2)), GK/(V6IN V=)r(()) which turns the upper bound the minimal value. Hence, yield the final bound of FTRL in the end,
22NG
2/ENG K
N
22NG (1)
Σ El('')
min (e(p) -
(48)
t=1
0
D.3 Expected Regret of Partial Feedback: Proof of Theorem 5.2
In this section, we exlend the full feedback solution to the partial feedback scenario, where the server only has access to the feedback information from the past sampled clients. Technically, we use a mixing strategy transfer the deterministic regret into expected regret. Then, we bound the expected regret by playing the FTRL framework in the expectation space. Finally, we can minimize the expected regret bound by setting mixing parameter 0.
33

---

Published in Transactions on Machine Learning Research (01/2025)
Using the property of unbiasedness, we have
min
E
)
{(p)
1=1
T
T
min E
L (Ø'))
Σ
t(p)
(49)
1=1
1=1
T
E
Σ E((0"))
+
min I
(pp')
Σ
(p)
t=1
1=1
t=1
(4)
(B)
Bounding (A). We recall that f 2 4K for all Le (7], E [N] due to the mixing. Therefore, p 2 K/N
implics p 2 K/N. Thus, we have
1
P
p
0
N K
=
0
5 0
P
p
P.P
P.P.
P
Morcover, if pf S K/N, the above inequality still holds. We extend the (A) as follows,
T
(A)
E
(
Σ
kpp'
1=1
-=1
E
#
1
D
T N ΣΣ
20(1)
K
1=1 1=1
IG N2
T.
K
where we use [[+{(1]]= = x)(1) S G
Bounding (B). We note that p' is the decision sequence playing FTRL with the mixed cost. functions. Thus, Replacing G with G
we combine the mixing bound of feedback (i.e., #1 < we get
GPN
and Theorem D.1
N OK
Ok
22NåG ēVOK
2V6NiG VOk3
22
Σ
min P
(p)
EE
" (i)
(50)
50K
Using Jensen's incquality, we have E EM ( S EN Finally, we can get the upper bound of the regret in partial-bandit leedback,
Summary.
V
B (1)
E
EN
1 (i).
T
OG
22G
2V6G
N Σ
2202
2 min E
l((p')
Σ ((p) <
T
E
2 (i)
K
5VONK
VONK
ZOK
i=1
(51)
O
22
2/GN G2
22G
< K
T +
VT +
žVOK
V0K3
50K
where the last line uses the bound Σ (1) (1) < NGVT. Now, we can optimize the upper bound of regret in terms of O. Notably, 0 is independent. T and we set (7xx } to get the minimized bound. Additionally,
34

---

Published in Transactions on Machine Learning Research (01/2025)
pursuing expected regret, which is (s (T) in the original definition in Equation the unbiasedness of the mixed estimation and modified costs, we can obtain the final bound:
Using
T
T
min N E.Reret (5) (T) = (pp))
(p))
T
T
T
E
((('')
min t=1
(p)) + E
min
(p)
min
4(p)
ON /K +
E min
(p) min
(p))
s O(N
where the last inequality uses Lemmaa 5.1 and the conclusion in Theorem 8 [Borsos et al, 2018 It proves the second term induces an additional log term to the final bound.
Remark. Baseline works have additional averaging coefficient N² in their final bound. This is because they consider the weights À = 1/N in stochastic optimization, while we include the À for clients' weights in federated optimization. To align with them, we omit the coefficient of N and report the final bound for E (s) (Z'),, as 2² can be absorbed by excluding the À from client feedback function rT-).
E
Further Discussions
E.1 A Sketch of Proof for FL with Client Stragglers
We note the possibility that some clients are unavailable to participants due to local failure or being busy in each round. To exlend our analysis the case, we assume there is known distribution of client availability A such that a subset A' A of clients are available at the f-th communication round. Let q: Prob(i A*) denote the probability that client i is available round t. Based on the setting, we update the definition of estimation d':
Xg
d'
q.p
iES'CAL
where 5t S A indicates that we can only sample from available set. Then, we apply the estimation variance and obtain the following target:
Regret (T))
min
q:P:
i=1 q:P:
Analogous to our analysis in Appendix we could obtain similar bound of the above regret that considers the availability.
E.2 Hard Capping for ISP
We know that ISP might have more or less than clients selected, which implies the sampling results St can be unexpectedly large. Concretely, this section studies a setting which RSP with hard budget K₃, is compared soft budget K, ensure that Prob[[S*] > Kx) S We intend analyze the constraints on sampling probability p w.r.t K,, K₁, and c. This objective can be formally described as:
KA Σ
Prob[[S* > K2)) = -rrob[[S & K.)) = 1
Σ
I
P II (I(-))))))) JēC
Sc.
(52)
k=
Ccckk
LEG
where 5 N
P: Ks, is the possible size of St and Cl is the combinations of all possible selected clients with 5t k: If we consider simple case where P K,/N, Vi (N), [S*| follows standard Binomial
35

---

In other words, adding this Published in Transactions on Machine Learning Research (01/2025)
distribution B(N, s/N). And, the above inequality can be solved by checking the cumulative distribution function (CDF).
For general cases with different p., Vi E [N], we can use Normal approximation to solve the inequality with large N. Using Hoeffding's s inequality to estimate the probability of large deviations:
Prob[|S"| - Ks 2 T) se 2- 2///////// / pi()-
Letting K₁ K, + T, we know
Prob[[S 2 Kn) 2 pi(l-Di) Sc.
Taking the logarithm and rearranging the second inequality, we get
N ( - - K.)//Inne.
Inc,
as a new constraint for ISP sampling probability to satisfy Equation 52
constraint on Equation 22 may result in a satisfied sampling probability.
E.3 Differences between biased client sampling methods
This section discusses the main differences between unbiased client sampling and biased client sampling methods, The proposed K-b sampler is an unbiased sampler for the first-order gradient of objective Recent biased client sampling methods include Power-of-Chne ChhCccc (POC) (Cho al., 2020b) and DivFL (Balakrishnan) et al. 2022). Concretely, POC requires all clients to upload local empirical loss as prior knowledge and selects clients with the largest empirical loss. DivFL. builds submodular based on the latest gradient from clients and selects clients to approximate all client information. Therefore, these client sampling strategies build biased gradient estimation that may deviate from fixed global goal.
FL with biased client sampling methods, such as POC and DivFL, can be considered dynamic re-weighting algorithms adjusting p;. Analogous to the Equation (T), the basic objective of FL with biased client sampling methods can be defined as follows (Li et al., 2020b Balakrishnan et al. 2022
Cho al. 2020b
N
min f(x) #EX
Σ p.f(()) = Σ D. (x),)
(53)
where p is the probability simplex, and p; is the probability of client i being sampled. The gradient estimation is defined as g" K Eies Ii accordingly. The targets of biased FL client sampling are determined by the sampling probability as replacement of X in the original FedAvg objective [ Typically, the value of p is usually dynamic and implicit.
E.4
Theoretical Comparison with OSMD
The K-Vib sampler proposed in this paper is orthogonal with the recent work OSMD sampler b021b in theoretical contribution. We justify our points below:
Zhao et al.
a) According to Equations (6) and (7) in OSMD, it proposes an online mirror descent procedure that optimizes the additional estimates to replace the mixing strategy in Vrb Borsos ct al. 20188) The approach can be also utilized an alternative method in Equation [12]
b) The improvement of the K- Vib sampler is obtained from the modification of the sampling procedure, In contrast, the OSMD still follows the conventional RSP, as we discussed Lomma 2.1 Hence, our thcoretical findings of applying the ISP in adaptive client sampling can be transferred to OSMD as well.
In short, the theoretical improvement of our work independent of the OSMD sampler. And, our insights about utilizing the ISP can be used to improve the OSMD sampler. Meanwhile, the OSMD also suggests future work for the K-Vib sampler in oplimizing the additional estimates procedure instead of mixing.
2e refer the latest version https: http::://arrsw..rg/patt///// http://pf/22222.
36

---

Published in Transactions on Machine Learning Research (01/2025)
Dutaset vs (3, 22,000 Sralting Imageal
Dutiest 12 129 teaiiing imegeal
Dataaet vi 127. trakking Images)
Sywthek cotnect
The womooe streeds
cians tali Azb marig inaes
20% die hol sus ronning Ioages
20% daet sols stading Inages
1000
1000
1000
500
500
500
201
18C
burring
(a) Federated EMNIST datasct
(b) Syuthetic dataset
Figure 6: Distribution of used datasets across clients.
Mabs
Vrb
Dynamic Regret
5000
Km10
2500
4000
K-30
2000
K-30
4000
K- 30
Km50
K=50
K-70
1500
K-
3000
K-
2000
2000
1000
100
200
300
400
500
100
200
300
400
500
0
100
200
300
400
500
Communication Round
Communication Round
Communication Round
Figure 7: Regret of baseline algorithms with different K
F
Further Experiments
The experiment implementations are supported by FedLab framework (Zeng al. 2023b We provide additional experimental details, experiments, and discussion in this section.
F.1 Experiment Details
Hyperparameters Setting. All samplers have an implicit value G related to the hyperparameters. We set G = 0.01 for the Synthotic dataset task and G 0.1 for FEMNIST tasks. We set 17 = 0.4 (stability
hyperparameter) for Mabs (Salehi et al. 2017) as suggested by the original paper. Vrb Borsos al. (0018) also utilize mixing strategy
(N/T)$ and regularization y G2 * N/0. For the case that N T
FEMNIST tasks, we
0.3 following the official source code 3 For Avare El Hanchi & Stephens (2020);
set Pmin 5N C 2 Pinin
and 5 1 for constant-stepsize
suggested in Appendix D of original paper.
For the K-Vib sampler, we study in Figure 3(d)
(7x
N 1/3 and 7 G2 K8
We also fix y and 0 0.3 for our sensitivity
Data distribution across clients. We present the data distribution across clients in Figure 6 The data samples are extremely unbalanced and tailed, matching the real-world applications. It also shows the significance of variance reduction techniques.
F.2 Additional Experiments
Baselines with budget K. Our theoretical results in Theorem 5.2 and empirical results in Figure 3(a) reveal key improvement of our work, that is, the linear speed up in online convex optimization. In contrast, we provide additional experiments with the different budget K in Figure 7 Baseline methods do not preserve the improvement property respecting large budget K in adaptive client sampling for variance reduction. Moreover, with the increasing communication budget K, the optimal sampling value is decreasing. As a result, the regret of baselines increases Figure indicating the discrepancy to the optimal is enlarged.
Total variation distance over communication rounds. In Figure we provide the TV between probabilities of K-Vib and uniform probabilities. It shows that the trends of TV distances dependent data distribution across clients. The TV distances change more with large degree of heterogeneity across
37

---

Published in Transactions on Machine Learning Research (01/2025)
Synthetic, K=30
Synthetic, K=50
Synthetic, K=70
Gradient Variance
0.30
Unifom
0.30
0.30
0.25
0.20
0.20
0.20
0.15
0.10
0.10
0.10
0.05
0.05
0.00
0.00
0.00
100
Communication Round
Communication Round
Communication Round
Figure 8: Gradient variance with different
TV distance on AGNews
TV distance on CCNews
v1
12
v1
10
V3
v3
TV Distance
8-
2
TV
0-
0
50
1000
150
200
250
300
0
50
100
150
200
250
300
Communication Round
Communication Round
Figure 9: Evolution of TV distance between K-Vib probabilities and Uniform probabilities.
clients. This indicates that K- Vib can accurately be adaptive data distributions across clients. Moreover, we note that the trends of TV curves are different in AGNews and CCNews, as their loss types (classification loss and next-loken-predichee
loss) and model architectures (Bert and GPT) are different.
F.3 Efficient Implementation
In experiments, we do not find heavy computation time increase compared to baselines as our experiments only involve thousands of clients. To guarantee practical usage for large-scale systems, we present efficient implementation details of Vibib.
We can maintain sorted list of cumulative local weights (w(1), For each communication round, the server receives feedback values as list [T( Vj St. Then, the server will traverse the feedback list. For each element in the list, the server conducts two main steps below:
such that w(1 S w()), Wij e
[N] in Algorithm 2
Step 1: For each j E St, server computes estimates wij) w()) + m2()/p/, Then, the server uses binary-search find the index such that w(k) s a(j) w(k + 1) in the cumulative local weights.
Step 2: Then, server update w(j) = a(j) and move the position of w(j) behind w(k) to update the weights sequence.
This implementation implements a time complexity of O(T K log N), where T is the communication round, K is the communication budget, and N is the number of clients. For each communication round t e [7], the server updates K times of the list with each time cost O(log N) to conduct one binary search.
38


Reviews:
[{"review_id": "yDtPbCBXTX", "reviewer": "Reviewer_RuQ8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strenghts\n- I am reading a revised version, with various improvements. As such, the paper was relatively straightforward to follow through it's introduction\n- As far as I can tell, relevant previous works are included\n- For a theory paper, strong experimental validation with good results that follow the theory\n- in particular I want to commend the authors for adding _interpretations_ to their prose (partially as a revision it appears, but nonethelss commendable)\n\n# Weaknesses\n\n## Nits\n\n- on page 4, last paragraph,\n\n>Interestingly, the ISP estimates perform worse than the uniform sampling baseline with a larger budget, as shown in the third plot of Figure 1(a). This discrepancy arises because the optimal probability in ISP aims to minimize a loose upper bound on variance, as defined in equation equation 3. This is because the optimal probability of RSP is minimizing a loose upper bound of variance equation 3\n\nappears to refer to _RSP_ not ISP in the first two instance, and the sentence is doubled\n\n- Interpretation of Theorem 4.1 paragram\n\n>If we always use optimal client sampling (Chen et al., 2020), the cumulative sampling quality β1 = 0.\n\nis probably meant to read\n\n>If we always use optimal client sampling (Chen et al., 2020),  the cumulative sampling quality implies β1 = 0.\n\nor something like it?\n\n## More serious cirtique\n\n- changing the definition of budget from strict K to probabilitic K implies that that per iteratoin, we might have much more or much less than K clients selected, unless I am mistaken? This means that the algorithm is mainly applicable if we are operating at a scale in which we are softly budget constrained, and it might be worth studying a setting in which RSP with hard $K_h$ is compared with ISP with a soft $K_s$ chosen to ensure that $P(|S_t| >K_h) \\leq  \\eps$  with $\\eps \\to 0. Alternatatively it could be commented on in the prose or the impact of being capped by a hard constraint would need to be analyzed (I would assume it would simply introuce introduce a bias term that needs to be controlled ?)\n\n-  given the different training distributions on the real/synthetic dataset, I feel like a rebalancing/importance sampling strategy with fixed-but-nonuniform sampling probabilities would represent a stronger baseline, unless it is established that we assume we do not know the distribution across clients (I might have missed this but I think it wasn't assumed?)\n\n## Not a critique but a curiosity\n\nBoth the $\\gamma$ and the mixing incentivise the algorithm to hew close to the uniform probablity while allowing divergence. I would be interested in seing the deviatoi from the unfirom distribution (e.g. in wasserstein or TV ) across the optimization procedure, as it would shed some light in what particular types of adaptivity occur. The real world experiments indicate that when local client variance dominates the heterogeneity, the sampling", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents an analysis of independent sampling procedures  (ISP) vs random samplin procedures (RSP) for the selection of participating clients in federated learning. The former (ISP) being bernoulli samples with per-client probabilities and the latter being choose-k without replacement. The paper studies the contribution of the SP to the variance of the gradient estimate, focusing on unbiased client sampling, and shows that optimal independent sampling strictly outperforms RSP in terms of the sampling quality, accelerating the convergence rate by an almost linear factor. This is done by seperating the variance due to client heterogeneity from variance due to the estimation. Experiments are presented on synthetic, F-EMNIST and two NLP datasets with what used to be LLMs (pythia on CCnews/AGNews).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The point about bias via hard capping is critical and would need to be addressed, the rest are nice to haves.", "broader_impact_concerns": "I think no BIS is required.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SwsyehW4iU", "reviewer": "Reviewer_pLCp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper studies client sampling for FL, which is an important aspect of modern machine learning.\n- The paper presents both theoretical analysis and empirical evaluations, which well demonstrate the usefulness of the proposed sampling algorithm. The theoretical parts seem overall correct and sound; the experiments are comprehensive and cover both synthetic and real world image and text data sets, touching on even large scale language models and data sets.\n\nWeaknesses:\n- Regarding the claim of superiority of K-Vib, I think more evidence may be necessary. First, it is unclear to me that the two referenced baselines are fair comparison, as they are not explicit handling the budget dependency term based on my understanding; it is unclear if there is a true improvement. On the empirical front, it is also not clear to me if the improvement is significant (e.g., Figure 2, many parts of Figure 5).\n- Many of the visualizations seem hard to parse in my opinion. For instance, Figure 1(a) may be converted to histograms (rather than scatter plots) for better readability; Figure 4(a) seems slightly convoluted - I don't think the bars provide useful information, maybe just stating the data distribution suffices. In addition, there are occasional typos/minor errors (e.g., citation styles) that require further polishing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies client sampling procedures for federated learning (FL) and introduces K-Vib, an adaptive unbiased client sampler, which utilizes an independent sampling procedure (ISP) instead of traditional random sampling (RSP). In FL, multiple devices (clients) collaborate to train a global model without sharing their local data. Instead of sending data to a central server, clients perform local computations and send model updates to the server, which aggregates them into a global model. Sampling procedures are needed to determine how the server acquires updates from the clients.\n\nThe key contribution is the reduction in variance during client sampling, which improves convergence rates in federated optimization. The authors argue that K-Vib achieves an improved regret bound compared to baselines, showing near-linear speedups in communication efficiency. Empirical studies validate K-Vib’s superior performance across various FL tasks, particularly in cross-device settings with heterogeneous client data distributions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical:\n- In order to better understand the significance of the contribution, I hope to see additional discussion on the two theoretical baselines used for comparison and on the empirical improvements. A better quantification of the improvement (e.g., comparing ISP vs RSP) can be helpful.\n- Regarding Section 3 on cases study on sampling procedures, I am not sure if I am convinced with these toy examples, since after all we should be concerned with cases where $N$ and $K$ are both large. The examples on small universes may not provide insights that are robust when scaled up. More discussion on this is useful.\n- I would love to see more discussion on the hyperparameters, $\\theta$ in particular.\n\nMinor:\n- Many proofs in the appendix are not easily readable. It may help to break up long chains of equations and provide an overview at the beginning of long proofs.\n- End of Section 1, paragraph \"Contributions\": The budget $K$ has not been defined at this point.\n- Legend of Figure 1(b) seems erroneous: Where is ISP? Also, caption to the Figure is confusing (Is \"Global estimates on the X-Y plane\" a run-away statement?).\n- Theorem 5.2, proof: Should be \"Proof sketch\" or \"Sketch of proof\".", "broader_impact_concerns": "I don't see additional concerns that need to be addressed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "f9GNzvR4V5", "reviewer": "Reviewer_Bvvp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Given that I am not an expert in this field, I highlight the following:\n\n__Strengths:__ \n1) _Novel Client Sampling Approach:_ The proposal of K-Vib introduces a new adaptive unbiased client sampling framework that is conceptually innovative. The independent sampling procedure and variance reduction strategy represent a meaningful improvement over conventional random sampling methods.\n\n2) _Empirical Validation:_ The empirical studies convincingly demonstrate that K-Vib outperforms baseline algorithms, doubling the speed of training in the experiments. This practical evidence strengthens the validity of the theoretical claims and shows the potential impact of K-Vib on real-world FL systems.\n\n__Weaknesses:__\n\n1) _Limited Benchmarking:_ While the experimental results highlight the superiority of K-Vib over certain baselines, the set of comparison methods is relatively narrow. It would be useful to see K-Vib benchmarked against a wider range of advanced FL techniques.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The article addresses a significant challenge in Federated Learning (FL), particularly the client selection process, which directly affects the convergence rate and efficiency of FL models. The authors propose a new client sampling method, K-Vib, based on an independent, adaptive sampling procedure that reduces the variance in client selection, leading to improved convergence. The theoretical contribution is supported by empirical results showing that K-Vib significantly improves the speed of training compared to existing baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "_Broaden Experimental Comparisons:_ Expanding the set of baseline algorithms in the experiments will provide a more comprehensive evaluation of K-Vib’s relative performance.", "broader_impact_concerns": "See above", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an adaptive sampling strategy for client selection in federated learning, guided by an online variance reduction technique.

Specifically, the paper:
- advocates for the arbitrary sampling framework (Horváth & Richtárik, 2019; Chen et al., 2020) for client sampling in federated learning,
- presents a convergence upper bound for FedAvg with arbitrary client sampling, highlighting the impact of the sampling quality measure on convergence,
- extends the online learning VRB (Borsos et al., 2018) to multi-client sampling (K-ViB) and establishes a regret bound for this procedure,
- evaluates the K-ViB sampler through numerical experiments.

This manuscript has been revised based on the feedback from reviewers across two rounds of review. The presentation has been substantially improved and now includes enhanced discussions and interpretations of the results, improved numerical experiments, and strengthened theoretical findings. All reviewers recommend acceptance of the manuscript.
Decision: Accept (poster)