Title: Reshape-then-Factorize: Communication-Efficient FL via Model-Agnostic Projection Optimization
Venue: NeurIPS 2025
Accepted: no
Keywords: Federated Learning

Low-Rank Adaptation

Communication Efficiency

Subspace Optimization
Abstract: Federated learning (FL) enables collaborative model training across distributed clients without sharing sensitive data. However, communication overhead remains a significant bottleneck, particularly for large-scale models. Low-rank decomposition techniques address this by approximating each layer’s weights or gradients with a product of low-rank matrices, thereby reducing the communication cost in FL. While effective, these methods are constrained by the layer's architecture and shapes, limiting their flexibility and performance.
We propose *Model-Agnostic Projection Optimization* (MAPO), a novel method that reshapes and factorizes the full model gradient into a *fixed reconstruction matrix* and a *trainable projection vector*, avoiding layer-wise decomposition and architecture constraints. MAPO directly optimizes the projection in a randomly sampled subspace, with all clients generating the reconstruction matrix via a shared random seed, incurring no additional communication overhead for synchronization.
By decoupling the gradient from architectural constraints through reshaping and enabling communication-free exploration of dynamic subspaces via seed sharing, MAPO provides a more flexible and efficient low-rank representation.
Empirical results demonstrate the effectiveness of MAPO in various FL settings.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission introduces a new method (MAPO) for reducing the communication overhead of exchanging full model updates in federated learning (FL). The method is based on reshaping and factorizing the full model gradient into a fixed recostruction matrix and trainable projection vector, which makes it independent on per-layer dimensions and architecture.\nThe method is theoretically analyzed and experiments complement the claims on efficiency.\n\nSTRENGTHS AND WEAKNESSES:\n## Strenghts\n* **The paper in general well-written:** besides being a little imprecise on piece of the theoretical results (see questions), in general the paper is well-written, discusses in sufficient detail the background and related section and is complemented by useful illustrations and tables (e.g. Table 1)\n* **The proposed approach is theoretically grounded:** the properties of MAPO are well introduced in section 3.1, which serves as a guide through the theoretical results. There are possibly some typos in the naming of references (e.g. assumptions and definitions are referred as theorems, I believe it is a cleverref config. issue), but overall the organization is clear.\n* **Good empirical results:** the provided results support the claims on communication efficiency and accuracy.\n\n## Weaknesses\n* **Unclear aspects of Theorem 4.3:** for details please see the questions\n* **Unclear postioning of the method regarding FL settings:** This is in part also tackled in the questions. Is the method being proposed for cross-silo or cross-device FL settings?\n* **Most of the datasets are academic and small-scale:** I believe the results would be much stronger if larger-scale datasets had been used, like Google Landmarks and INaturalist. A previous paper [1] has introduced federated versions compatible with cross-silo/cross-device settings that authors could use to provide stronger benchmarking.\n* **Missing comparison with [2]:** given the similarity in the end objective with this paper, I believe it is necessary to compare the proposed approach with [2], both in the methodological differences (e.g. when one would choose one method or the other in practical cases) and in the experimental results.\n\n\n[1] Hsu et al., Federated Visual Classification with Real-World Data Distribution, ECCV 2021  \n[2] Azam et. al, Recycling Model Updates in Federated Learning: Are Gradient Subspaces Low-Rank?, ICLR 2022\n\nQUESTIONS:\n1. **About Assumption 3.1:** it is not clear if it is an assumption or if the property of gaussian matrices has been proved. In fact, reading line 155 it seems to be a proven result more than an assumption.\n2. **Does the algorithm require full participation?** Similarly, under what conditions do the provided theoretical guarantees hold? From algorithm 1, line 5, it seems that the algorithm requires full client participation, while in theorem 4.3 it is not clear how the number of selected clients affects the rate, e.g. it is not clear the dependency of the noise from the participating clients.\n3. **Does Theorem 4.3 assume iid conditions?** Usually FL algortihms need to assume that the variance between clients gradients is bounded (Bounded Gradient Dissimilarity, BGD), which is used to quantify the dependency on heterogeneity. It seems that this assumption is not mentioned in the paper, as assumption 4.2 only bounds the stochastic variance due to mini-batch sampling inside each client. Please clarify if the assumption is implicitly used (i.e. we assume there is zero dissimilarity) or what is the reason the theorem does not depend on it.\n4. **Does Theorem 4.3 take into account local steps?** Algorithm 1 line 10 indicates that local epochs are used, but it seems that local steps do not appear in the theorem (e.g. do not influence the choice of the learning rate, and do not appear in the dependency on stochastic noise). Also, why $\\eta_t$ remains explicitly reported in the final bound? I would expect to only see the dependency on its upper bound.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper focuses on communication-efficient federated learning and aims to improve the flexibility of low-rank decomposition methods. It proposes reshaping and factorizing the gradient via a universal vector. The proposed method is evaluated on several datasets and demonstrates improved performance.

STRENGTHS AND WEAKNESSES:
Strengths
- The topic is relevant and important in federated learning.
- The motivation for improving low-rank decomposition is well demonstrated.
- The method is novel, and the experiments show good results.

Weaknesses
- The practical implications of the convergence analysis are unclear and could be further elaborated.
- The comparison methods for quantization are somewhat outdated; it would be better to include more recent quantization works in the comparison.
- The advantages of using fresh A are obvious, but it would be better to discuss the potential reasons further.

QUESTIONS:
- Are there any other factors that could affect the reinitialization of A? Is sharing the seed sufficient?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-02

comments: SUMMARY:
MAPO is a model agnostic method of compressing gradients for efficient communication between client and server. The merits of MAPO are demonstrated through theoretical analysis and practical application.

STRENGTHS AND WEAKNESSES:
**Strengths**
* The authors do a good job of comparing MAPO to CEFL methods and highlighting its advantages over existing methods
* Model agnostic 
* Strong set of baselines that demonstrate several important use cases

**Weaknesses**
* No explicit evaluation of the ability of the low rank projection to capture meaningful gradient directions. This evaluation would lend strength to the claim that MAPO's performance is due to dynamic subspace exploration 
* may overlook important minority features for clients with long tailed distributions

QUESTIONS:
* as tasks become more complex/ have higher dimensional data, it seems likely that the minimum value of k required to get a good approximation would increase. Have you tried experiments with high dimensional data? (medical images?), is the k required to train on this data prohibitive?
* re-defining A could help with subspace exploration but it could also make updates unstable with gradients that change drastically between updates, particularly early in training. Have you observed this behaviour? Are there steps taken to mitigate it?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo reduce the communication overhead in federated learning, the authors proposed a model-agnostic projection optimization method based on low-rank decomposition technology, which reshapes and factorizes the gradient of the complete model into a fixed reconstruction matrix and a trainable projection vector, while reducing the communication overhead and solving the limitations of low-rank decomposition technology. Finally, experiments have proved the effectiveness of MAPO in a variety of federated learning scenarios.\n\nSTRENGTHS AND WEAKNESSES:\n(1)In terms of quality, the submitted paper is reasonable in its technical application, and the designed technical solutions and their performance in specific scenarios are fully demonstrated through rigorous convergence analysis, theoretical proof and systematic experimental verification.The research method of the paper is appropriate, the experimental design is perfect, and the whole paper is a well-structured and detailed academic paper. It is worth affirming that the author not only clearly explains the academic contribution of this study, but also objectively analyzes the limitations of the current solution in the discussion section, which reflects the rigor of the solution.\n(2)In terms of clarity, the content of the submitted paper is clearly stated, the article structure is well-organized, and the layout is standardized and reasonable, which can effectively guide readers to understand the innovative points and core contributions of the research.\n(3)In terms of significance, this paper has important practical application value for federated learning, and provides a foundation for subsequent research on low-overhead, high-performance, and deployable federated learning. At the same time, the paper is superior to traditional efficient communication solutions in terms of communication overhead and flexibility. In addition, the author verifies the applicability of the method in practical environments through overhead and performance analysis under multiple datasets and heterogeneous data.\n(4)In terms of originality, this paper starts with the balanced optimization of communication overhead, model performance, and computational flexibility. Compared with previous studies, it more comprehensively considers the coordinated optimization of computational flexibility and model performance on the basis of reducing communication overhead, and cites relevant literature to support its innovation. The optimized federated learning communication strategy proposed in the study takes into account flexibility and high performance while reducing overhead, and verifies its applicability through heterogeneous data environments, providing new optimization ideas for the development of the field.\n\nQUESTIONS:\n(1). There are some symbolic errors in the text and figures, such as \"△W\" in \"Multi-Layer Gradient Decomposition\" in Figure 3, and some parameters in Table 4 lacking units. The reviewer believes that the description and symbols in the text and figures, tables, and algorithms need to be consistent. If the author can analyze this problem in depth and make a comprehensive check and correction, it will make the research more comprehensive and further enhance the authority of the article.\n(2). In the paper, MAPO requires each client to use the gradient descent algorithm to update the projection, and the server and client update the model based on the aggregated projection. The reviewer is curious that the author did not mention the threat model of the scheme in the paper. If the client maliciously tampers with the corresponding projection vector, will it cause the final model to be unable to be reconstructed? What impact will it have if the server broadcasts the wrong projection vector?  The reviewer hopes that the author can explain it.\n(3). The labels of Figures 5, 6, and 8 in the article cover up some curves, affecting the readers' observation of the experimental results. The reviewer hopes that the author can adjust the above figures to improve the readability of the article.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-04"}