SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Objective inconsistency in FL. Several works also encounter difficulties from the objective inconsistency caused by partial client participation [31, 7, 2]. [31, 7] use the local-global gap  $f^{*} - \frac{1}{m}\sum_{i = 1}^{m}F_{i}^{*}$  to measure the distance between the global optimum and the average of all local personal optima, where the local-global gap results from objective inconsistency at the final optimal point. In fact, objective inconsistency occurs in each training round, not only at the final optimal point. [2] also encounter objective inconsistency caused by partial client participation. However, they use  $\left|\frac{1}{n}\sum_{i = 1}^{n}\nabla F_{i}(x_{t}) - \nabla f(x_{t})\right| \leq \epsilon$  as an assumption to describe such update inconsistency caused by objective inconsistency without any analysis on it. To date, the objective inconsistency caused by partial client participation has not been fully analyzed, even though it is prevalent in FL, even in homogeneous local updates. Our work provides a fundamental convergence analysis on the influence of the objective inconsistency of partial client participation.
Client selection in FL. In general, sampling methods in federated learning (FL) can be classified as biased or unbiased. Unbiased sampling guarantees that the expected value of client aggregation is equal to that of global deterministic aggregation when all clients participate. Conversely, biased sampling may result in suboptimal convergence. A prominent example of unbiased sampling in FL is multinomial sampling (MD), which samples clients based on their data ratio [59, 12]. Additionally, importance sampling (IS), an unbiased sampling method, has been utilized in FL to reduce convergence variance. For instance, [4] use update norm as an indicator of importance to sample clients, [49] sample clients based on data variability, and [40] use test accuracy as an estimation of importance. Meanwhile, various biased sampling strategies have been proposed to speed up training, such as selecting clients with higher loss [7], as many clients as possible under a threshold [45], clients
with larger updates [48], and greedily sampling based on gradient diversity [2]. However, these biased sampling methods can exacerbate the negative effects of objective inconsistency and only converge to a neighboring optimal solution. Another line of research focuses on reinforcement learning for client sampling, treating each client as an agent and aiming to find the optimal action [69, 62, 6, 53, 67]. There are also works that consider online FL, in which the client selection must consider the client's connection ability [44, 17, 26, 71, 46, 8]. Recently, cluster-based client selection has gained some attention in FL [12, 64, 42, 52, 37, 50, 25, 41, 61]. Though clustering adds additional computation and memory overhead, [12, 52] show that it is helpful for sampling diverse clients and reducing variance. Although some studies employ adaptive cluster-based IS to address the issue of slow convergence due to small gradient groups [52, 11], these approaches differ from our method as they still require an additional clustering operation. The proposed DELTA  $^3$  in Algorithm 1 can be viewed as a muted version of the diverse client clustering algorithm, while promising to be unbiased.
Importance sampling. Importance sampling is a statistical method that allows for the estimation of certain quantities by sampling from a distribution that is different from the distribution of interest. It has been applied in a wide range of areas, including Monte Carlo integration [10, 70, 1], Bayesian inference [22, 23], and machine learning [54, 19].
In a recent parallel work, [49] demonstrated mean square convergence of strongly convex federated learning under the assumption of a bounded distance between the global optimal model and the local optimal models.[4] analyzed the convergence of strongly convex and nonconvex federated learning by studying the improvement factor, which is the ratio of the participation variance using importance sampling and the participation variance using uniform sampling. This algorithm dynamically selects clients without any constraints on the number of clients, potentially violating the principle of partial user participation. It is worth noting that both of these sampling methods are based on the gradient norm, ignoring the effect of the direction of the gradient. Other works have focused on the use of importance sampling in the context of online federated learning, where the client selection must consider the client's connection ability. For example, [69] proposed an adaptive client selection method based on reinforcement learning, which takes into account the communication cost and the accuracy of the local model when selecting clients to participate in training. [62] also employed reinforcement learning for adaptive client selection, treating each client as an agent and aiming to find the optimal action that maximizes the accuracy of the global model.[6] introduced a bandit-based federated learning algorithm that uses importance sampling to select the most informative clients in a single communication round. [53] considered the problem of federated learning with imperfect feedback, where the global model is updated based on noisy and biased local gradients, and proposed an importance sampling method to adjust for the bias and reduce the variance of convergence.
B Toy Example and Experiments for Illustrating Our Observation
B.1 Toy example
Figure 5 is a separate illustrated version of each sampling algorithm provided in Figure 3.
We consider a regression problem involving three clients, each with a unique square function:  $F_{1}(x,y) = x^{2} + y^{2}; F_{2}(x,y) = 4(x - \frac{1}{2})^{2} + \frac{1}{2}y^{2}; F_{3}(x,y) = 3x^{2} + \frac{3}{2}(y - 2)^{2}$ . Suppose  $(x_{t},y_{t}) = (1,1)$  at current round  $t$ , the gradients of three clients are  $\nabla F_{1} = (2,2)$ ,  $\nabla F_{2} = (4,1)$ , and  $\nabla F_{3} = (6, -3)$ . Suppose only two clients are selected to participate in training. The closer the selected user's update is to the global model, the better.
For ideal global model,  $\nabla F_{global} = \frac{1}{3}\sum_{i=1}^{3}\nabla F_i = (4,0)$ , which is the average over all clients.
For FedIS,  $\nabla F_{FedIS} = \frac{1}{2} (\nabla F_2 + \nabla F_3) = (5, - 1)$ : It tends to select Client 2 and 3 who have large gradient norms, as  $\| \nabla F_3\| >\| \nabla F_2\| >\| \nabla F_1\|$ .
For DELTA,  $\nabla F_{DELTA} = \frac{1}{2} (\nabla F_1 + \nabla F_3) = (4, -\frac{1}{2})$ : It tends to select Client 1 and 3 who have the largest gradient diversity than that of other clients pair, where the gradient diversity can be formulated by  $div_i = \| \nabla F_i(x_t,y_t) - \nabla F_{global}(x_t,y_t)\|$  [55, 32].
3With a slight abuse of the name, we use DELTA for the rest of the paper to denote either the sampling probability or the federated learning algorithm with sampling probability DELTA, as does FedIS.
For FedAvg,  $\nabla F_{FedAvg} = \frac{1}{2} (\nabla F_1 + \nabla F_2) = (3,\frac{3}{2})$ : It assigns each client with equal sampling probability. Compared to FedIS and DELTA, FedAvg is more likely to select Client 1 and 2. To facilitate the comparison, FedAvg is assumed to select Client 1 and 2 here.
From Figure 3, we can observe that the gradient produced by DELTA is closest to that of the ideal global model. Specifically, using  $L2$  norm as the distance function  $\mathcal{D}$ , we have  $\mathcal{D}(\nabla F_{DELTA}, \nabla F_{global}) < \mathcal{D}(\nabla F_{FedIS}, \nabla F_{global}) < \mathcal{D}(\nabla F_{FedAvg}, \nabla F_{global})$ . This illustrates the selection of more diverse clients better approaches the ideal global model, thereby making it more efficient.
(a) Overview of different methods.
(b) FedAvg.
(c) FedIS.
(d) DELTA.
Figure 5: Overview of objective inconsistency. The intuition of objective inconsistency in FL is caused by client sampling. When Client 1 & 2, are selected to participate the training, then the model  $x^{t+1}$  becomes  $x_{FedAvg}^{t+1}$  instead of  $x_{global}^{t+1}$ , resulting in objective inconsistency. Different sampling strategies can cause different surrogate objectives, thus causing different biases. From Fig 5(a) we can see DELTA achieves minimal bias among the three unbiased sampling methods.
B.2 Experiments for illustrating our observation.
Experiment setting. For the experiments to illustrate our observation in the introduction, we apply a logistic regression model on the non-iid MNIST dataset. 10 clients are selected from 200 clients to participate in training in each round. We set 2 cluster centers for cluster-based IS. And we set the mini batch-size to 32, the learning rate to 0.01, and the local update time to 5 for all methods. We run 500 communication rounds for each algorithm. We report the average of each round's selected clients' gradient norm and the minimum of each round's selected clients' gradient norm.
Performance of gradient norm. We report the gradient norm performance of cluster-based IS and IS to show that cluster-based IS selects clients with small gradients. As we mentioned in the introduction, the cluster-based IS always selects some clients from the cluster with small gradients, which will slow the convergence in some cases. We provide the average gradient norm comparison between IS and cluster-based IS in Figure 6(a). In addition, we also provide the minimal gradient norm comparison between IS and cluster-based IS in Figure 6(b).
Performance of removing small gradient clusters. We report on a comparison of the accuracy and loss performance between vanilla cluster-based IS and the removal of cluster-based IS with small gradient clusters. Specifically, we consider a setting with two cluster centers. After 250 rounds, we replace the clients in the cluster containing the smaller gradient with the clients in the cluster containing the larger gradient while maintaining the same total number of participating clients. The experimental results are shown in Figure 7. We can observe that vanilla cluster-based IS performs
(a) Average gradient norm comparison
(b) Minimal gradient norm comparison
Figure 6: The gradient norm comparison. Both results indicate that cluster-based IS selects clients with small gradients after about half of the training rounds compared to IS.
worse than cluster-based IS without small gradients, indicating that small gradients are a contributing factor to poor performance.
(a) Accuracy performance comparison
Figure 7: An illustration that cluster-based IS sampling from the cluster with small gradients will slow convergence. When the small gradient-norm cluster's clients are replaced by the clients from the large gradient-norm cluster, we see the performance improvement of cluster-based IS.
(b) Loss performance comparison
C Techniques
Here, we present some technical lemmas that are useful in the theoretical proof. We substitute  $\frac{1}{m}$  for  $\frac{n_i}{N}$  to simplify the writing in all subsequent proofs.  $\frac{n_i}{N}$  is the data ratio of client  $i$ . All of our proofs can be easily extended from  $f(x_{t}) = \frac{1}{m}\sum_{i = 1}^{m}F_{i}(x_{t})$  to  $f(x_{t}) = \sum_{i = 1}^{m}\frac{n_{i}}{N} F_{i}(x_{t})$ .
Lemma C.1. (Unbiased Sampling). Importance sampling is unbiased sampling.  $\mathbb{E}\left(\frac{1}{n}\sum_{i\in S_t}\frac{1}{mp_i}\nabla F_i(x_t)\right) = \frac{1}{m}\sum_{i = 1}^{m}\nabla F_i(x_t)$ , no matter whether the sampling is with replacement or without replacement.
Lemma C.1 proves that the importance sampling is an unbiased sampling strategy, either in sampling with replacement or sampling without replacement.
Proof. For with replacement:
$$
\begin{array}{l} \mathbb {E} \left(\frac {1}{n} \sum_ {i \in S _ {t}} \frac {1}{m p _ {i} ^ {t}} \nabla F _ {i} (x _ {t})\right) = \frac {1}{n} \sum_ {i \in S _ {t}} \mathbb {E} \left(\frac {1}{m p _ {i} ^ {t}} \nabla F _ {i} (x _ {t})\right) = \frac {1}{n} \sum_ {i \in S _ {t}} \mathbb {E} \left(\mathbb {E} \left(\frac {1}{m p _ {i} ^ {t}} \nabla F _ {i} (x _ {t}) \mid S\right)\right) \\ = \frac {1}{n} \sum_ {i \in S _ {t}} \mathbb {E} \left(\sum_ {l = 1} ^ {m} p _ {l} ^ {t} \frac {1}{m p _ {l} ^ {t}} \nabla F _ {l} \left(x _ {t}\right)\right) = \frac {1}{n} \sum_ {i \in S _ {t}} \nabla f \left(x _ {t}\right) = \nabla f \left(x _ {t}\right), \tag {19} \\ \end{array}
$$
For without replacement:
$$
\begin{array}{l} \mathbb {E} \left(\frac {1}{n} \sum_ {i \in S _ {t}} \frac {1}{m p _ {i}} \nabla F _ {i} (x _ {t})\right) = \frac {1}{n} \sum_ {l = 1} ^ {m} \mathbb {E} \left(\mathbb {I} _ {m} \frac {1}{m p _ {l} ^ {t}} \nabla F _ {l} (x _ {t})\right) = \frac {1}{n} \sum_ {l = 1} ^ {m} \mathbb {E} (\mathbb {I} _ {m}) \times \mathbb {E} (\frac {1}{m p _ {l} ^ {t}} \nabla F _ {l} (x _ {t})) \\ = \frac {1}{n} \mathbb {E} \left(\sum_ {l = 1} ^ {m} \mathbb {I} _ {m}\right) \times \mathbb {E} \left(\frac {1}{m p _ {l} ^ {t}} \nabla F _ {l} \left(x _ {t}\right)\right) = \frac {1}{n} n \times \sum_ {l = 1} ^ {m} p _ {l} ^ {t} \frac {1}{m p _ {l} ^ {t}} \nabla F _ {l} \left(x _ {t}\right) \\ = \frac {1}{n} \sum_ {l = 1} ^ {m} n p _ {l} ^ {t} \times \frac {1}{m p _ {l} ^ {t}} \nabla F _ {l} \left(x _ {t}\right) = \frac {1}{m} \sum_ {l = 1} ^ {m} \nabla F _ {l} \left(x _ {t}\right) = \nabla f \left(x _ {t}\right), \tag {20} \\ \end{array}
$$
where  $\mathbb{I}_m\triangleq \left\{ \begin{array}{ll}1 & ifx_l\in S_t,\\ 0 & \mathrm{otherwise}. \end{array} \right.$
In the expectation, there are three sources of stochasticity. They are client sampling, local SGD, and the filtration of  $x_{t}$ . Therefore, the expectation is taken over all of these sources of randomness. Here,  $S$  represents the sources of stochasticity other than client sampling. More precisely,  $S$  represents the filtration of the stochastic process  $x_{j}, j = 1,2,3\ldots$  at time  $t$  and the stochasticity of local SGD.
Lemma C.2 (update gap bound).
$$
\chi^ {2} = \mathbb {E} \| \frac {1}{n} \sum_ {i \in S _ {t}} \frac {1}{m p _ {i} ^ {t}} \nabla F _ {i} (x _ {t}) - \nabla f (x _ {t}) \| ^ {2} = \mathbb {E} \| \nabla \tilde {f} (x _ {t}) \| ^ {2} - \| \nabla f (x _ {t}) \| ^ {2} \leq \mathbb {E} \| \nabla \tilde {f} (x _ {t}) \| ^ {2}. \tag {21}
$$
where the first equation follows from  $\mathbb{E}[x - \mathbb{E}(x)]^2 = \mathbb{E}[x^2] - [\mathbb{E}(x)]^2$  and Lemma C.1.
For ease of understanding, we give a detailed derivation of the Lemma C.2.
$$
\begin{array}{l} \mathbb {E} \left(\| \nabla \tilde {f} (x _ {t}) - \nabla f (x _ {t}) \| ^ {2} \mid S\right) = \mathbb {E} \left(\| \nabla \tilde {f} (x _ {t}) \| ^ {2} \mid S\right) - 2 \mathbb {E} \left(\| \nabla \tilde {f} (x _ {t}) \| \| \nabla f (x _ {t}) \| \mid S\right) \\ + \mathbb {E} \left(\left\| \nabla f \left(x _ {t}\right) \right\| ^ {2} \mid S\right), \tag {22} \\ \end{array}
$$
where  $\mathbb{E}(x\mid S)$  means the expectation on  $x$  over the sampling space. We have  $\mathbb{E}\left(\| \nabla \tilde{f} (x_t)\mid S\right) = \nabla f(x_t)$  and  $\mathbb{E}\left(\| \nabla f(x_t)\| ^2\mid S\right) = \| \nabla f(x_t)\| ^2 (\| \nabla f(x)\|$  is a constant for stochasticity  $S$  and the expectation over a constant is the constant itself.)
$$
\mathbb {E} \left(\| \nabla \tilde {f} (x _ {t}) - \nabla f (x _ {t}) \| ^ {2} \mid S\right) = \mathbb {E} \left(\| \nabla \tilde {f} (x _ {t}) \| ^ {2} \mid S\right) - \| \nabla f (x _ {t}) \| ^ {2} \leq \mathbb {E} \left(\| \nabla \tilde {f} (x _ {t}) \| ^ {2} \mid S\right). \tag {23}
$$
We can further take the expectation on both sides of the inequality according to our needs, without changing the relationship.
The following lemma follows from Lemma 4 of [47], but with a looser condition Assumption 3, instead of  $\sigma_G^2$  bound. With some effort, we can derive the following lemma:
Lemma C.3 (Local updates bound.). For any step-size satisfying  $\eta_L \leq \frac{1}{8LK}$ , we can have the following results:
$$
\mathbb {E} \| x _ {i, k} ^ {t} - x _ {t} \| ^ {2} \leq 5 K \left(\eta_ {L} ^ {2} \sigma_ {L} ^ {2} + 4 K \eta_ {L} ^ {2} \sigma_ {G} ^ {2}\right) + 2 0 K ^ {2} \left(A ^ {2} + 1\right) \eta_ {L} ^ {2} \| \nabla f (x _ {t}) \| ^ {2}. \tag {24}
$$
$$
\begin{array}{l} \mathbb {E} _ {t} \left\| x _ {t, k} ^ {i} - x _ {t} \right\| ^ {2} \\ = \mathbb {E} _ {t} \| x _ {t, k - 1} ^ {i} - x _ {t} - \eta_ {L} g _ {t, k - 1} ^ {t} \| ^ {2} \\ = \mathbb {E} _ {t} \| x _ {t, k - 1} ^ {i} - x _ {t} - \eta_ {L} \left(g _ {t, k - 1} ^ {t} - \nabla F _ {i} \left(x _ {t, k - 1} ^ {i}\right) + \nabla F _ {i} \left(x _ {t, k - 1} ^ {i}\right) - \nabla F _ {i} \left(x _ {t}\right) + \nabla F _ {i} \left(x _ {t}\right)\right) \| ^ {2} \\ \leq \left(1 + \frac {1}{2 K - 1}\right) \mathbb {E} _ {t} \| x _ {t, k - 1} ^ {i} - x _ {t} \| ^ {2} + \mathbb {E} _ {t} \| \eta_ {L} \left(g _ {t, k - 1} ^ {t} - \nabla F _ {i} \left(x _ {t, k} ^ {i}\right)\right) \| ^ {2} \\ + 4 K \mathbb {E} _ {t} [ \| \eta_ {L} (\nabla F _ {i} (x _ {t, K - 1} ^ {i}) - \nabla F _ {i} (x _ {t})) \| ^ {2} ] + 4 K \eta_ {L} ^ {2} \mathbb {E} _ {t} \| \nabla F _ {i} (x _ {t}) \| ^ {2} \\ \leq \left(1 + \frac {1}{2 K - 1}\right) \mathbb {E} _ {t} \left\| x _ {t, k - 1} ^ {i} - x _ {t} \right\| ^ {2} + \eta_ {L} ^ {2} \sigma_ {L} ^ {2} + 4 K \eta_ {L} ^ {2} L ^ {2} \mathbb {E} _ {t} \left\| x _ {t, k - 1} ^ {i} - x _ {t} \right\| ^ {2} \\ + 4 K \eta_ {L} ^ {2} \sigma_ {G} ^ {2} + 4 K \eta_ {L} ^ {2} (A ^ {2} + 1) \| \nabla f (x _ {t}) \| ^ {2} \\ \leq \left(1 + \frac {1}{K - 1}\right) \mathbb {E} \| x _ {t, k - 1} ^ {i} - x _ {t} \| ^ {2} + \eta_ {L} ^ {2} \sigma_ {L} ^ {2} + 4 K \eta_ {L} ^ {2} \sigma_ {G} ^ {2} + 4 K \left(A ^ {2} + 1\right) \| \eta_ {L} \nabla f (x _ {t}) \| ^ {2}. \tag {25} \\ \end{array}
$$
Unrolling the recursion, we obtain:
$$
\begin{array}{l} \mathbb {E} _ {t} \| x _ {t, k} ^ {i} - x _ {t} \| ^ {2} \leq \sum_ {p = 0} ^ {k - 1} (1 + \frac {1}{K - 1}) ^ {p} \left[ \eta_ {L} ^ {2} \sigma_ {L} ^ {2} + 4 K \eta_ {L} ^ {2} \sigma_ {G} ^ {2} + 4 K (A ^ {2} + 1) \| \eta_ {L} \nabla f (x _ {t}) \| ^ {2} \right] \\ \leq (K - 1) \left[ \left(1 + \frac {1}{K - 1}\right) ^ {K} - 1 \right] \left[ \eta_ {L} ^ {2} \sigma_ {L} ^ {2} + 4 K \eta_ {L} ^ {2} \sigma_ {G} ^ {2} + 4 K \left(A ^ {2} + 1\right) \| \eta_ {L} \nabla f (x _ {t}) \| ^ {2} \right] \\ \leq 5 K \left(\eta_ {L} ^ {2} \sigma_ {L} ^ {2} + 4 K \eta_ {L} ^ {2} \sigma_ {G} ^ {2}\right) + 2 0 K ^ {2} \left(A ^ {2} + 1\right) \eta_ {L} ^ {2} \| \nabla f (x _ {t}) \| ^ {2}. \tag {26} \\ \end{array}
$$
In the following Proposition, we will demonstrate that the convergence rate in this paper with the relaxed version of Assumption 3 remains unchanged.
Proposition C.4 (convergence under relaxed Assumption 3 [24]). The relaxed version of Assumption 3 in this paper is:
$$
\mathbb {E} \| \nabla F _ {i} (x) \| ^ {2} \leq 2 B (f (x) - f ^ {i n f}) + (A ^ {2} + 1) \| \nabla f (x) \| ^ {2} + \sigma_ {G} ^ {2}. \tag {27}
$$
Since we have  $f(x) - f^{inf} \leq f^0 - f^{inf} \leq F$ , where  $F$  is a positive constant. This implies that we can substitute  $\sigma_g$  with  $2BF + \sigma_G$  in all analyses without altering the outcomes (one can directly conclude this from using the above bound in Lemma C.3). In the final convergence rate, it is straightforward to see that the convergence rate remains unchanged, yet the constant term  $\sigma_g$  becomes  $2BF + \sigma_G$ .
Thus, we can assert that we have furnished the analysis under the relaxed assumption condition.
D Convergence of FedIS, Proof of Theorem 3.1
The complete version of FedIS algorithm is shown below:
We first restate the convergence theorem (Theorem 3.1) more formally, then prove the result for the nonconvex case.
Theorem D.1. Under Assumptions 1-3 and the sampling strategy FedIS, the expected gradient norm will converge to a stationary point of the global objective. More specifically, if the number of communication rounds  $T$  is predetermined and the learning rate  $\eta$  and  $\eta_L$  are constant, then the expected gradient norm will be bounded as follows:
$$
\underset {t \in [ T ]} {\min } \mathbb {E} \| \nabla f (x _ {t}) \| ^ {2} \leq \frac {F}{c \eta \eta_ {L} K T} + \Phi , \tag {28}
$$
where  $F = f(x_0) - f(x_*)$ ,  $M^2 = \sigma_L^2 + 4K\sigma_G^2$ , and the expectation is over the local datasets samples among workers.
Algorithm 2 FedIS and FedPracIS : Federated learning with importance sampling
Require: initial weights  $x_0$  , global learning rate  $\eta$  , local learning rate  $\eta_{l}$  , number of training rounds  $T$    
Ensure: trained weights  $x_{T}$    
1: for round  $t = 1,\dots ,T$  do   
2: Select clients by using IS (5) or Practical IS (15).   
3: for each worker  $i\in S_t$  ,in parallel do   
4:  $x_{t,0}^{i} = x_{t}$    
5: for  $k = 0,\dots ,K - 1$  do   
6: compute  $g_{t,k}^{i} = \nabla F_{i}(x_{t,k}^{i},\xi_{t,k}^{i})$  Local update:  $x_{t,k + 1}^{i} = x_{t,k}^{i} - \eta_{L}g_{t,k}^{i}$    
8: Let  $\Delta_t^i = x_{t,K}^i -x_{t,0}^i = -\eta_L\sum_{k = 0}^{K - 1}g_t^i$    
9: Send gradient to server   
10: At Server:   
11: Receive  $\Delta_t^i,i\in S_t$    
12: let  $\Delta_t = \frac{1}{|S_t|}\sum_{i\in S_t}\frac{n_i}{np_i^t}\Delta_t^i$    
13: Server update:  $x_{t + 1} = x_{t} + \eta \Delta_{t}$    
14: Broadcast  $x_{t + 1}$  to clients
Let  $\eta_L < \min(1/(8LK), C)$ , where  $C$  is obtained from the condition that  $\frac{1}{2} - 10L^2 K^2 (A^2 + 1) \eta_L^2 - \frac{L^2 \eta_K(A^2 + 1)}{2n} \eta_L > 0$ , and  $\eta \leq 1 / (\eta_L L)$ , it then holds that:
$$
\Phi = \frac {1}{c} \left[ \frac {5 \eta_ {L} ^ {2} L ^ {2} K}{2 m} \sum_ {i = 1} ^ {m} \left(\sigma_ {L} ^ {2} + 4 K \sigma_ {G} ^ {2}\right) + \frac {\eta \eta_ {L} L}{2 m} \sigma_ {L} ^ {2} + \frac {L \eta \eta_ {L}}{2 n K} V \left(\frac {1}{m p _ {i} ^ {t}} \hat {g} _ {i} ^ {t}\right) \right]. \tag {29}
$$
where  $c$  is a constant that satisfies  $\frac{1}{2} - 10L^2 K^2 (A^2 + 1)\eta_L^2 - \frac{L^2\eta K(A^2 + 1)}{2n}\eta_L > c > 0$ , and  $V\left(\frac{1}{mp_i^t}\hat{g}_i^t\right) = E\left\| \frac{1}{mp_i^t}\hat{g}_i^t - \frac{1}{m}\sum_{i = 1}^{m}\hat{g}_i^t\right\|^2$ .
Corollary D.2. Suppose  $\eta_L$  and  $\eta$  are such that the conditions mentioned above are satisfied,  $\eta_L = \mathcal{O}\left(\frac{1}{\sqrt{T}KL}\right)$  and  $\eta = \mathcal{O}\left(\sqrt{Kn}\right)$ , and let the sampling probability be FedIS (75). Then for sufficiently large  $T$ , the iterates of Theorem 3.1 satisfy:
$$
\min  _ {t \in [ T ]} \mathbb {E} \| \nabla f (x _ {t}) \| ^ {2} = \mathcal {O} \left(\frac {\sigma_ {L} ^ {2}}{\sqrt {n K T}} + \frac {K \sigma_ {G} ^ {2}}{\sqrt {n K T}} + \frac {\sigma_ {L} ^ {2} + 4 K \sigma_ {G} ^ {2}}{K T}\right). \tag {30}
$$
$$
\begin{array}{l} \mathbb {E} _ {t} [ f (x _ {t + 1}) ] \stackrel {(a 1)} {\leq} f (x _ {t}) + \langle \nabla f (x _ {t}), \mathbb {E} _ {t} [ x _ {t + 1} - x _ {t} ] \rangle + \frac {L}{2} \mathbb {E} _ {t} [ \| x _ {t + 1} - x _ {t} \| ^ {2} ] \\ = f (x _ {t}) + \langle \nabla f (x _ {t}), \mathbb {E} _ {t} [ \eta \Delta_ {t} + \eta \eta_ {L} K \nabla f (x _ {t}) - \eta \eta_ {L} K \nabla f (x _ {t}) ] \rangle + \frac {L}{2} \eta^ {2} \mathbb {E} _ {t} [ \| \Delta_ {t} \| ^ {2} ] \\ = f \left(x _ {t}\right) - \eta \eta_ {L} K \| \nabla f \left(x _ {t}\right) \| ^ {2} + \eta \underbrace {\left\langle \nabla f \left(x _ {t}\right) , \mathbb {E} _ {t} \left[ \Delta_ {t} + \eta_ {L} K \nabla f \left(x _ {t}\right) \right] \right\rangle} _ {A _ {1}} + \frac {L}{2} \eta^ {2} \underbrace {\mathbb {E} _ {t} \| \Delta_ {t} \| ^ {2}} _ {A _ {2}}, \tag {31} \\ \end{array}
$$
where (a1) follows from the Lipschitz continuous condition. The expectation is conditioned on everything prior to the current step  $k$  of round  $t$ . Specifically, it is taken over the sampling of clients, the sampling of local data, and the current round's model  $x_{t}$ .
Firstly we consider  $A_{1}$ :