Title: Exact and Linear Convergence for Federated Learning under Arbitrary Client Participation is Attainable
Venue: NeurIPS 2025
Accepted: yes
Keywords: Federated Learning

Arbitrary Client Participation

Graph

Stochastic Matrix

Decentralized Optimization
Abstract: This work tackles the fundamental challenges in Federated Learning (FL) posed by arbitrary client participation and data heterogeneity, prevalent characteristics in practical FL settings.  It is well-established that popular FedAvg-style algorithms struggle with exact convergence and can suffer from slow convergence rates since a decaying learning rate is required to mitigate these scenarios. To address these issues, we introduce the concept of stochastic matrix and the corresponding time-varying graphs as a novel modeling tool to accurately capture the dynamics of arbitrary client participation and the local update procedure. Leveraging this approach, we offer a fresh perspective on designing FL algorithms, provide a rigorous quantitative analysis of the limitations inherent in the FedAvg algorithm, and present FOCUS, Federated Optimization with Exact Convergence via Push-pull Strategy, a provably convergent algorithm designed to effectively overcome the previously mentioned two challenges. More specifically, we provide a rigorous proof demonstrating that FOCUS achieves exact convergence with a linear rate regardless of the arbitrary client participation, establishing it as the first work to demonstrate this significant result.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses a fundamental problem in Federated Learning (FL): achieving exact convergence under arbitrary client participation and data heterogeneity without resorting to diminishing learning rates. The authors propose a new algorithm, FOCUS, inspired by decentralized optimization techniques, especially push-pull gradient tracking. The paper makes strong theoretical contributions; in particular, it models client participation using stochastic matrices and establishes sound convergence guarantees, with no bounded heterogeneity or participation bias assumptions. I believe this paper provides new insights into federated learning algorithms and has the potential to inspire further research in the field.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n\nTo me, the idea in the paper is innovative and departs meaningfully from prior work. I believe the key contribution is a novel viewpoint to understand federated learning algorithms. Previous work often states that optimization methods in FL are special cases with decentralized algorithms with the communication network being the fully-connected graph, or equivalently, $W = (1/N) \\mathbf{1} \\mathbf{1}^T$. This paper departs from the mainstream and introduces a novel analytical framework grounded in the push-pull strategy in decentralized optimization.\n\n**Areas to improve**\n\n1. Is it possible to compare FOCUS with SCAFFOLD? I understand that they are just different. But I wonder whether the authors have any intuition behind the distinction of update rules.\n\n2. Do the authors have an intuition why pushing gradient $\\boldsymbol{y}$ requires column style rather than row style?  In other words, why does the update of $y_{r+1}$ requires the **sum** of $y_{\\tau, i}^{(r)}$ rather than the **average**?\n\nQUESTIONS:\nThis part collects some minor concerns on the paper, mainly in presentation and writing.\n\n1. In line 6, the article states that \"we introduce the concept of stochastic matrix ...\". This wording may create the impression that the concept of stochastic matrix is being introduced by this work.\n\n2. In line 52, the authors claim that FOCUS exhibits linear convergence for both **convex** and non-convex scenarios. I suggest to replace _convex_ with _strongly convex_.\n\n3. In lines 54-55, the authors claim that SG-FOCUS demonstrates **faster** convergence and **higher** accuracy. It is recommended to clarify the other side of the comparison. It is weird that a stochastic-gradient variant converges faster than its deterministic counterpart.\n\n4. In (2c), the authors may need to double check whether $x_{\\tau+1,i}^{(r)}$ should be $x_{\\tau,i}^{(r)}$.\n\n5. In lines 74-75, the authors state that \"Yet, this approach incurs increased communication costs, doubling them\ndue to the transmission of a control variate with the same dimensionality as the model parameters.\" However, [1, Sec. 4.2] shows that SCAFFOLD can be implemented with a single communicated variable. So this claim should be revisited and clarified.\n\n6. In Assumption 1, \"which occurs with **a** unknown probability\" should be \"which occurs with **an** unknown probability\".\n\n7. In the paragraph in lines 152-158, the authors use both $\\mathbb 0$ and $\\mathbf 0$. Are they the same?\n\n8. In line 184, \"equivalency\" should be \"equivalence\".\n\n[1] X. Huang, P. Li, X. Li. Stochastic controlled averaging for federated learning with communication compression. ICLR 2024.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper propose FOCUS via push-pull strategy to address the arbitrary participation problem in federated learning.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\n1. This paper finds the relation between distributed optimization over directed graphs and federated learning with arbitrary participation.\n2. This paper proposes FOCUS via push-pull strategy, which shows faster convergence than FedAvg.\n3. The theory seems solid and rigorous.\n4. The paper is well-written, well-organized.\n\n**Weaknesses**:\n\n**The motivation for constructing the matrices**. The constructions of the assign matrix ($R$), average matrices ($A$) and other matrices ($C$ and $W$) are interesting. However, the motivation for constructing these matrices **in this paper** is not strong or convincing. In my opinion, all the content can be presented clearly even without these constructions: (i) The model of arbitrary client participation relies on Assumption 1, instead of these constructions. (ii) The convergence analysis of FedAvg has been established in Wang and Ji [2022] without the constructed matrices. (iii) FOCUS can also be interpreted without these constructions (see Section 4.2.2). (iv) After checking the proofs of Theorem 1, I think the proofs can also be written without these constructions.\n\nUsing the matrices can increase the workload of reading this paper. For example, when checking the proofs, I need to revisit the definitions of some matrices (e.g., Appendices B, C.1). Some definitions of them are not straightforward, e.g., $\\nabla \\boldsymbol{f}(\\boldsymbol{x}\\_k) = [\\nabla f_0(x\\_{k,0}),\\nabla f_1(x\\_{k,1}),\\ldots,\\nabla f_N(x\\_{k,N})]^T$ and $\\nabla \\boldsymbol{f}(1\\bar x\\_k) = [\\nabla f_0(\\bar x\\_k),\\nabla f_1(\\bar x\\_k),\\ldots,\\nabla f_N(\\bar x\\_k)]^T$ in Appendix B.\n\nOne example where the use of matrix notation is necessary: The construction of the mixing matrix in decentralized federated learning (or decentralized SGD) is necessary, as the assumptions relies on the mixing matrix. See Koloskova et al. [2020].\n\nQUESTIONS:\n**Questions**:\n\n1. It is unclear whether $x$ (or $y$) is a row or column vector. See $x$ in Equation (1) and $\\boldsymbol{x}$ in Line 152; see $\\boldsymbol{x}$ between Lines 444 and 445 and Line 446.\n\n2. Lines 549-550: Why we say $\\boldsymbol{y}_k$ has one more inner iteration?\n\n3. Equation (67) and (68): Why is only $C_{k,all}$ defined ($R_{k,all}$ is not defined)?\n\n4. Line 607:\n\n   > the client's $y_{r\\tau+1,i} = 0$ for any clients.\n\n   Is this right?\n\n5. Line 610:\n\n   > the first equation means the difference between the model before pulling ...\n\n   Does \"the model\" represents \"the clients' model\"?\n\n6. Line 667:\n\n   > (102) holds when $\\eta \\leq \\frac{2q_\\min}{3N \\left( \\frac{8L^2}{\\mu}+\\frac{\\mu}{2}\\right)}$.\n\n   Does this cause an additional constraint $\\eta \\lesssim \\frac{\\mu q_min}{NL^2}$ for the strongly convex case in Theorem 1?\n\n**Suggestions**:\n\n1. Figures: It would be better to use the PDF format for the figures.\n\n2. Theorem 1. It seems that the convergence of FOCUS is affected by $p_\\min$. The convergence of FOCUS can be slow when $p_\\min$ is very small (e.g., $p_\\min = \\frac{1}{R}$). I think this limitation can be added to Section 5.3.\n\n3. Missing references. The following paper gave a similar conclusion to Wang and Ji [2023]. \n\n   Federated Learning Under Heterogeneous and Correlated Client Availability, IEEE/ACM Transaction on Networking, 2024.\n\n4. Table 1:\n\n   > There is no convergence proof of SCAFFOLD under arbitary client participation scenario.\n\n   The following paper can be relevant.\n\n   Federated Learning under Periodic Client Participation and Heterogeneous Data A New Communication-Efficient Algorithm and Analysis, NeurIPS, 2024.\n\n5. Line 74, Section 2:\n\n   > Yet, this approach incurs increased communication costs, doubling them due to the transmission of a control variate with the same dimensionality as the model parameters.\n\n   This sentence can be misleading. See Appendix A.1 in the following paper for the efficient implementation of SCAFFOLD.\n\n   TCT Convexifying Federated Learning using Bootstrapped Neural Tangent Kernels, NeurIPS, 2022.\n\n6. Section 3.1: Could you kindly give detailed explanation (or proofs) for the conclusions in the special cases. For example, explain why $q_i \\approx p_i/(\\sum_j p_j)$ when $p_i$ are close to each other in Case 2, and explain why $p_i = 1-(1-q_i)^m$ when it is sampled with replacement in Case 3. I think this would make the cases easier to understand for the readers.\n\n7. Section 3.1: Could you kindly give some practical FL scenarios (or give some simple and illustrative examples) for these cases, just like Appendix B.2 in the following paper?\n\n   A Unified Analysis of Federated Learning with Arbitrary Client Participation, NeurIPS, 2022.\n\n8. Lines 272-273.\n\n   > To see that, leveraging the tracking property of the variable $y_k$ and special construction of matrix $C_k$, we can establish that the server's $y_{r+1} = \\sum_{i=1}^N \\nabla f_i(x_{k+1},i)$.\n\n   Could you kindly add the proof for this?\n\n9. Lines 275-276:\n\n   > FOCUS effectively transforms arbitrary participation probabilities into an arbitrary delay in gradient updates.\n\n   This interpretation is interesting. This sounds like MIFA in the following paper. Could you kindly add some comparison with respect to the convergence rates.\n\n   Fast Federated Learning in the Presence of Arbitrary Device Unavailability, NeurIPS, 2021.\n\n10. References. There are some papers which are cited as preprint, but have been published. For example, Wang and Ji [2023] has been published in ICLR, Xiang et al. [2024] has been published in NeurIPS.\n\n11. Figure 9: Please add the results of FedAvg.\n\nIf the problems (weaknesses, suggestions, questions) I raised above are wrong, please free free to point them out.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of partial participation in federated learning, in the particular context where clients participate with (fixed) arbitrary probabilities.\nFirst, an interpretation of the properties of FedAvg-like algorithm as a decentralized algorithm with time-varying graphs is proposed, allowing to express the updates through a sequence of stochastic matrix products.\nBased on this interpretation, a novel method, coined FOCUS, is proposed, which is built on a primal-dual approach, and recovers gradient-tracking-type ideas.\nThe proposed method is shown to achieve good empirical performance in comparison with existing methods.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The interpertation of FedAvg with partial participation as a decentralized algorithm with time-varying graph is interesting, and allows for representation of algorithms as a product of stochastic matrices, which is often a fruitful approach. This is directly applied to the development of a novel FL method that outperforms many of the existing ones, which is very interesting.\n2. The propsed method is the first to achieve linear convergence under arbitrary participation, with without relying on tricks like almost-surely bounded time between successive participation of clients.\n3. Theoretical convergence rates are provided, together with numerical evaluation that shows the superiority of the proposed methods on some problems.\n\n\n**Weaknesses**\n1. The convergence rates rely on conditions on the step size like $\\eta \\le \\mu^2 / L$, which are way worse than the typical $\\eta \\le 1/L$. Moreover, they require the step size to scale with the smallest participation rate $q_i$, which is unknwon and may prevent the algorithm from converging.\n2. Convergence rates are somewhat misleading, as they seem to claim that the convergence is faster as the number of clients grow, but at the same time the step size is small than $1/N$. Contrarily to what is written in Theorem 1, there is thus no acceleration with the number of clients $N$ in the convergence rate (which is to be expected in FL).\n3. Overall, it is not clear how the convergence speed is affected by the probability of communication, although one could expect this to play a major role in the algorithm's properties.\n4. The method is only applied to deterministic gradients, although it borrows analysis techniques that are typical from stochastic approximation (using stochastic matrices).\n\n\nTypos:\n- In Equation (3), the left-hand term should not be the square of the norm, but rather the norm.\n- In Equation (18), it seems it should be $\\Leftarrow$ since it involves communication.\n\nQUESTIONS:\n1. What would the convergence rate look like if removing the artifical dependence in $N$ in the rate? Is it possible to fully state communication and sample complexity results, with full dependency on the constants and on the probability of communication?\n2. In particular, $q_i$ seem to appear in Theorem 1 only through its minimal value $\\min_i q_i$. Is it possible to obtain something better, like on the average of the $q_i$'s, or is it in some sens optimal?\n3. Although the matrix interpretation of the algorithm helps in the design of the new method FOCUS, it seems that it results in unusual conditions on the step size ($\\eta \\le \\mu^2 / L$). Is it because the matrix method is limited? Or because this is not the best matrix decomposition possible? Could it be improved or is there another more profound reason for these conditions to show up?\n\nLIMITATIONS:\nYes limitations have been properly discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the arbitrary client participation problem in federated learning, where clients may join or leave at will during FL training. To address this, the authors propose a new analysis framework from the time-varying graph perspective. The theoretical analysis shows that exact and linear convergence for federated learning is achievable.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. The paper is well written. I enjoy reading the paper.\n2. The proposed approach is novel and outperforms the state-of-the-art.\n3. I appreciate the beauty of applying insights from stochastic matrix theory to the analysis of the federated learning algorithm.\n\n### Weaknesses\n1. FOCUS combines insights from a fully decentralized algorithm and a federated learning algorithm with a server. In particular, it is partially inspired by the gradient tracking algorithm. It is a pity that the authors did not provide much intuition to help the readers understand.\n2.  In the experiment section, only one case of arbitrary participation is provided.\n3. To deal with client participation, we also have algorithms based on variance reduction techniques, including MIFA [R1] and FedVARP [R2]. \n\n**References:**\n[R1] Gu, X., Huang, K., Zhang, J., & Huang, L. (2021). Fast federated learning in the presence of arbitrary device unavailability. Advances in Neural Information Processing Systems, 34, 12052-12064.\n\n[R2] Jhunjhunwala, D., Sharma, P., Nagarkatti, A., & Joshi, G. (2022, August). Fedvarp: Tackling the variance due to partial client participation in federated learning. In Uncertainty in Artificial Intelligence (pp. 906-916). PMLR.\n\nQUESTIONS:\n1. Can the authors elaborate more on the intuition behind exact convergence gradient tracking?\n2. Can the authors provide more experiments\n * under more arbitrary client participation settings?\n * on MIFA and FedVARP algorithms?\n3. Are there any connections with the push-sum algorithm in the fully decentralized learning literature, for example, in [R3]?\n\n[R3] Olshevsky, A., Paschalidis, I. C., & Spiridonoff, A. (2018, June). Fully asynchronous push-sum with growing intercommunication intervals. In 2018 Annual American Control Conference (ACC) (pp. 591-596). IEEE.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}