Title: Layer-wise Update Aggregation with Recycling for Communication-Efficient Federated Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: federated learning

update recycling

communication-efficient
Abstract: Expensive communication cost is a common performance bottleneck in Federated Learning (FL), which makes it less appealing in real-world applications. Many communication-efficient FL methods focus on discarding a part of model updates mostly based on gradient magnitude. In this study, we find that recycling previous updates, rather than simply dropping them, more effectively reduces the communication cost while maintaining FL performance. We propose *FedLUAR*, a Layer-wise Update Aggregation with Recycling  scheme for communication-efficient FL. We first define a useful metric that quantifies the extent to which the aggregated gradients influences the model parameter values in each layer. *FedLUAR* selects a few layers based on the metric and recycles their previous updates on the server side. Our extensive empirical study demonstrates that the update recycling scheme significantly reduces the communication cost while maintaining model accuracy. For example, our method achieves nearly the same AG News accuracy as FedAvg, while reducing the communication cost to just 17%.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes FedLUAR, a novel federated learning method that recycles previous model updates on selected layers based on a gradient-to-weight ratio metric. The approach is motivated by the need to reduce communication costs in federated learning without sacrificing accuracy, and it challenges conventional methods that drop updates. Extensive theoretical analysis and empirical evaluations on benchmark datasets demonstrate that the layer-wise update recycling mechanism can significantly reduce communication overhead while maintaining or even accelerating model convergence.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper introduces an innovative update recycling strategy that selectively reuses past layer updates instead of dropping them, offering a fresh perspective on reducing communication costs in federated learning. The method leverages a newly defined metric based on the gradient-to-weight ratio to prioritize layers effectively. \n\n2. The proposed FedLUAR framework achieves significant communication efficiency while preserving model accuracy as demonstrated by experiments on multiple datasets, including image and text classification tasks. The empirical results show that the method reduces communication cost. The experiments are comprehensive and include comparisons with several state-of-the-art communication-efficient federated learning methods.\n\n3. The paper provides rigorous theoretical analysis including convergence proofs and noise bounding under realistic assumptions. The theoretical section clearly lays out the assumptions, lemmas, and theorems necessary for understanding how the update recycling mechanism maintains the stability of model updates. The proofs are detailed enough to give confidence in the method’s practical applicability in non-convex settings.\n\n4. Extensive ablation studies are performed to validate the effectiveness of the proposed layer prioritization metric and the recycling scheme. The ablation experiments compare different layer selection strategies and reinforce the idea that the gradient-to-weight ratio is a strong indicator for identifying less critical layers. The experiments also systematically explore the impact of key hyperparameters on both communication cost and model performance.\n\n\nWeaknesses:\n\n1. The method introduces additional memory overhead on the server as previous updates for certain layers must be stored, which may become a bottleneck for models with very deep or large architectures. The paper does mention the extra memory requirement but does not quantify its impact extensively. The discussion leaves open questions on scalability when server resources are very limited.\n\n2. The approach relies on the tuning of a key hyperparameter (δ) that determines the number of layers to recycle, and optimal settings appear to be dataset dependent. The ablation results indicate that performance can drop if recycling is applied excessively, suggesting sensitivity to this parameter. There is limited guidance on how to best select δ in various federated learning scenarios in real-world deployments.\n\n3. Although the experiments are extensive, the evaluation is limited to a handful of benchmark datasets and standard models, leaving some uncertainty about how the approach will perform in more diverse or larger-scale federated learning environments. The paper does not include scenarios with extremely heterogeneous data distributions that might reveal additional challenges. Information on performance in more varied real-world applications is thus lacking.\n\n4. Some highly related works about compression in FL are not discussed. [1,2] proposed compressing the communicated parameters or the model delta. [2,3] proposed using SVD to compress model parameters. [5] can be regarded as a layer-wise model compression methods.\n\n\n[1] Z. Tang, S. Shi, B. Li and X. Chu, \"GossipFL: A Decentralized Federated Learning Framework With Sparsified and Adaptive Communication,\" In IEEE Transactions on Parallel and Distributed Systems. 2022\n\n[2] Z. Tang, J. Huang, R. Yan, Y. Wang, Z. Tang, S. Shi, A.C. Zhou, X. Chu. Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning. In Proceedings of the 53rd International Conference on Parallel Processing, 2024.\n\n[3] Chai, Di, et al. \"Practical lossless federated singular vector decomposition over billion-scale data.\" Proceedings of the 28th ACM SIGKDD, 2022.\n\n[4] Haolin Wang, Xuefeng Liu, Jianwei Niu, Shaojie Tang. SVDFed: Enabling Communication-Efficient Federated Learning via Singular-Value-Decomposition. In IEEE INFOCOM 2023.\n\n[5] H Wang, M Yurochkin, Y Sun, D Papailiopoulos, Y Khazaeni. Federated learning with matched averaging. In ICLR 2020.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nLimitations are mentioned in the text as the extra memory overhead.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes FedLUAR, a Layer-wise Update Aggregation with Recycling scheme for communication-efficient federated learning. FedLUAR selects a subset of layers based on a proposed metric that quantifies how aggregated gradients influence model parameter values in each layer, and recycles previous updates on the server side rather than discarding them.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: \n- The concept of recycling previous updates rather than simply dropping them presents an interesting approach to improving communication efficiency in federated learning.\n\nWeakness:\n- The paper lacks clear theoretical justification for the Gradient-Weight Ratio Analysis. Is there an optimal proof or empirical motivation for this metric, or does it serve primarily as a heuristic function? The theoretical grounding needs strengthening.\n- As acknowledged in the limitations section, FedLUAR requires storing weights from all users in previous communication rounds, leading to significant memory overhead. This limitation may prevent the method from scaling to scenarios with large numbers of users training large models.\n- The main contributions of this work should be more explicitly stated and clearly distinguished from existing methods. \n- Beyond YOGA, the paper should include comparisons with other layer-wise model aggregation methods for federated learning with centralized servers. A comprehensive comparison would better position this work within the existing literature.\n\nQUESTIONS:\nAs stated in weakness.\nFurther experiments suggestions:\n- Experiments examining how varying degrees of data heterogeneity affect the proposed method's performance would strengthen the evaluation.\n- Evaluation with different numbers of participating users to understand scalability limitations and performance trade-offs.\n- Investigation of the method's benefits in few-shot learning scenarios, if applicable to the problem domain.\n- Experiments on larger datasets and modern foundation models to demonstrate practical applicability.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a communication-efficient approach to distributed training by selectively reusing updates for certain layers, avoiding aggregation in every round. Layer selection is guided by the gradient-to-weight ratio, in contrast to prior work that considers only gradient magnitude or weight norms. Experiments reveal that layers with large gradients may also have large weights, making their updates less impactful. Theoretical analysis bounds the noise introduced by partial aggregation, while empirical results demonstrate improvements in model performance, communication efficiency, and layer criticality metrics.\n\nSTRENGTHS AND WEAKNESSES:\nThe observation on balancing weight norms with gradient norms is insightful and, according to empirical results, contributes to improved distributed training.\n\nHowever, the algorithm description lacks clarity regarding where computations occur: criticality scores are computed on the server, yet clients require them to determine which parameters to share (line 181).\n\nWhile the theoretical analysis provides bounds on the noise under a diminishing learning rate, it does not offer justification for the critical layer selection strategy.\n\nEmpirical evaluation is not taking as benchmark method in [1].\n\n[1] Kamp M, Adilova L, Sicking J, Hüger F, Schlicht P, Wirtz T, Wrobel S. Efficient decentralized deep learning by dynamic model averaging. InMachine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2018, Dublin, Ireland, September 10–14, 2018, Proceedings, Part I 18 2019 (pp. 393-409). Springer International Publishing.\n\nQUESTIONS:\n1 -    On which side is the computation of layer importance performed? When only a subset of parameters is shared, does the selection happen on the client side before sending, or is it determined by the server?\n\n2 -    What does \\kappa refer to in Remark 1 (line 222)? It appears without a clear definition.\n\n3 -    Remark 2 (line 226) states that all terms on the right-hand side of inequality (7) vanish with a diminishing learning rate, except part of the second term. However, the first term seems to contain the learning rate in its discriminant - could you clarify?\n\n4 -    The evaluation and the last paragraph of Section 3.3 discuss data heterogeneity, although the method itself is not specifically designed for it. Why is heterogeneity relevant in this context?\n\nLIMITATIONS:\nN\\A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces FedLUAR, a novel communication-efficient Federated Learning (FL) method designed to mitigate the high communication costs associated with training large models. The central idea is to reduce the volume of data transmitted from clients to the server by selectively \"recycling\" historical updates for certain model layers. The method proposes a layer prioritization metric, $s_{t,l}$, which is the ratio of a layer's aggregated gradient magnitude to its weight magnitude. This metric identifies layers where an update is likely to have a less substantial impact. In each communication round, FedLUAR uses a weighted random sampling scheme to select a subset of these low-priority layers. For the selected layers, clients do not transmit their newly computed updates; instead, the server re-applies the aggregated update from the previous communication round. For all other layers, training proceeds as in standard FedAvg. The authors provide a theoretical convergence analysis and conduct extensive experiments on CIFAR-10/100, FEMNIST, and AG News, demonstrating that FedLUAR can achieve comparable accuracy to FedAvg while reducing communication costs by over 80%. Furthermore, they show that the recycling mechanism is compatible with other advanced FL optimizers like FedProx and MOON.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- Originality and Novelty: The core concept of recycling historical updates is a significant and novel contribution. While the field has extensively explored compression techniques like quantization, sparsification, and structured updates, the idea of reusing a previous update as a proxy for the current one is a fresh perspective. The ablation study, which demonstrates that recycling is substantially better than simply skipping the updates for low-priority layers, provides strong evidence for the value of this approach. This clearly distinguishes the work from prior art.\n\n- Quality of the Proposed Metric: The paper proposes a nuanced layer-prioritization metric ($s_{t,l}$) that goes beyond simple gradient norms. By considering the update's magnitude relative to the layer's weights, it offers a more stable and potentially more accurate way to gauge an update's impact. This is a valuable insight, as it challenges the common assumption that larger gradients always correspond to more important updates. The empirical results appear to validate the effectiveness of this metric.\n\n- Technical Soundness and Empirical Rigor: The claims are well-supported by both theoretical analysis and comprehensive experiments. The convergence analysis, which shows that the noise introduced by recycling is bounded under standard assumptions, provides a solid theoretical foundation. Empirically, the method is tested across multiple datasets, models, and degrees of data heterogeneity, consistently demonstrating significant communication savings with minimal to no loss in final accuracy.\n\n- Significance and Practicality: The method's \"plug-and-play\" nature is a major strength. By demonstrating compatibility with established FL optimizers like FedProx and MOON, the authors position FedLUAR not as a competing algorithm but as a general-purpose enhancement. This significantly increases its potential for adoption by researchers and practitioners looking to make existing FL pipelines more efficient. The reported communication savings are substantial and practically meaningful.\n\nWeaknesses:\n\n- Unaddressed Server-Side Overheads: The paper claims the method requires \"slightly more memory\" on the server to store historical updates. This claim lacks quantification. For very large models (e.g., LLMs with billions of parameters), storing even a single round of global updates could be a significant memory burden, potentially becoming a new bottleneck. This aspect of practical feasibility needs a more thorough analysis.\n\n- Hyperparameter Sensitivity: The performance of FedLUAR hinges on the hyperparameter $\\delta$ (the number of recycled layers), which must be manually tuned. The paper does not provide clear guidance or an adaptive strategy for setting $\\delta$, which could make it difficult to apply in practice. An effective trade-off between communication and accuracy depends on finding a near-optimal $\\delta$, which may require a costly hyperparameter search for each new task.\n\nLimited Evaluation in Dynamic Scenarios: The theoretical analysis assumes a standard FL setting. However, the core assumption that a stale update is a good proxy might break down in highly non-stationary environments where data distributions shift rapidly across rounds. In such cases, recycling could introduce significant error and slow down convergence. The practical impact in these more challenging (and realistic) scenarios is not sufficiently explored.\n\n- Insufficient Comparison with Contemporary Layer-wise Methods: While the comparison to FedAvg is essential, the paper's positioning would be much clearer with a direct comparison to other recent layer-wise FL techniques. For instance, methods like FedLAMA [3, 4] also use layer-wise strategies (adaptive aggregation frequency) to improve efficiency. Understanding how FedLUAR's recycling mechanism compares to or complements such approaches is critical for assessing its state-of-the-art standing.\n\nQUESTIONS:\n1.  Adaptive $\\delta$: The number of recycled layers, $\\delta$, is a crucial hyperparameter. Have you considered an adaptive schedule for $\\delta$ that might, for example, decrease the number of recycled layers as training progresses and the model fine-tunes? A dynamic approach could potentially yield a better and more robust trade-off between communication and accuracy.\n\n2.  Impact of Data Heterogeneity: The paper notes that convergence is guaranteed under non-IID conditions. Could you provide a deeper analysis or intuition on the interplay between the degree of data heterogeneity and the effectiveness of FedLUAR? Specifically, as non-IIDness increases, client updates diverge more. Does this make the *previous global update* a poorer proxy for the *current required update*, potentially slowing convergence more than in the IID case?\n\n3.  Scalability to Large-Scale Models: Regarding the server-side memory required to store historical updates, could you provide a more formal analysis of its scaling behaviour? For a model with $L$ layers and parameters $\\Theta$, the overhead appears to be $O(\\sum_{l \\in \\text{recycled}} |\\theta_l|)$. How would this fare with modern foundational models?\n\n4.  Orthogonality with Update Compression: FedLUAR is shown to be compatible with different FL optimizers. Is the mechanism also orthogonal to other communication efficiency techniques like quantization or sparsification? For instance, could one apply 8-bit quantization to the non-recycled layer updates while recycling the others? An analysis of whether these benefits are additive would significantly strengthen the paper's positioning as a general-purpose tool.\n\nLIMITATIONS:\nThe authors acknowledge some technical limitations related to the method's overhead, their treatment of broader limitations, particularly concerning privacy and fairness, is absent. Their discussion of negative societal impacts is also notably shallow, simply stating that none are anticipated and focusing exclusively on positive environmental benefits, which falls short of the comprehensive self-assessment expected for responsible AI research.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-01"}