Title: Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism
Venue: NeurIPS 2025
Accepted: yes
Keywords: Decentralized training

Distributed training

LLMs

Pipeline Parallel

Compression
Abstract: Scaling models has led to significant advancements in deep learning, but training these models in decentralized settings remains challenging due to communication bottlenecks. While existing compression techniques are effective in data-parallel, they do not extend to model parallelism. Unlike data-parallel training, where weight gradients are exchanged, model-parallel requires compressing activations and activation gradients as they propagate through layers, accumulating compression errors. We propose a novel  compression algorithm that  compresses both forward and backward passes, enabling up to 99% compression with no convergence degradation with negligible memory/compute overhead. By leveraging a recursive structure in transformer networks, we predefine a low-dimensional subspace to confine the activations and gradients, allowing full reconstruction in subsequent layers. Our method achieves up to 100x improvement in communication efficiency and enables training billion-parameter-scale models over low-end GPUs connected via consumer-grade internet speeds as low as 80Mbps, matching the convergence of centralized datacenter systems with 100Gbps connections with model parallel.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a communication compression algorithm designed to overcome bottlenecks in decentralized, model-parallel training of large-scale transformer models. The core idea is to leverage the natural rank collapse observed in transformer projection matrices by explicitly constraining these weights to a predefined low-dimensional subspace shared across devices. This constraint enables the nearly lossless compression and reconstruction of activations and gradients during both forward and backward passes, thereby avoiding the error accumulation that typically plagues compression techniques in MP settings. The authors empirically demonstrate that their method achieves up to 100x communication efficiency, enabling billion-parameter models to be trained over slow, consumer-grade internet connections while matching the convergence performance of centralized systems with high-speed interconnects.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper tackles the critical and highly relevant problem of communication efficiency in model-parallel training. By proposing a nearly-lossless compression method, this work has significant practical implications.\n2. The central insight is novel and well-motivated. The authors convincingly demonstrate—both theoretically and empirically—that key transformer weight matrices can be constrained to a shared low-dimensional subspace throughout training without degrading model convergence. \n3. The empirical results are a standout feature. Most notably, experiments show that a 2B-parameter model trained with the proposed compression over a 80Mbps connection achieves convergence on par with a baseline on a 100Gbps link.\n4. The methodology is well-grounded in theory, built upon the observation of rank collapse in projection matrices. The authors provide a solid theoretical framework, including proofs that justify the method's lossless nature and formal guarantees that the constrained network converges.\n\n\n\nWeaknesses:\n\n1. The paper lacks a clear, explicit description of the mechanism used to enforce the subspace constraint on the projection matrices. While it is stated that the rows of these matrices lie within the common subspace, the paper does not specify the detailed implementation. \n2. A crucial ablation study on the hyperparameter k (the subspace dimension) is notably absent. The experiments use a fixed k=40, but an analysis of the trade-off between compression ratio and model performance is essential for understanding the method's robustness and practical application.\n3. While a mechanism to update the shared subspace using the Grassmann manifold is proposed, it lacks a thorough sensitivity analysis. An ablation study on the necessity and frequency of this update is needed to understand its practical contribution.\n4. The paper's narrative flow could be improved. The separation of the methodology (Section 4) from its theoretical justification (Section 5) can be disruptive; integrating the theory more directly into the method's description could make the core ideas clearer.\n\nQUESTIONS:\n1. The proposed method focuses on compressing the projection matrices by constraining their row spaces. Is this approach orthogonal to, and potentially compatible with, other compression techniques? For example, could Subspace Networks be combined with methods that induce low-rank structure in self-attention matrices  or with other techniques used in DDP to achieve even greater communication savings?\n2. Regarding Statement 5.4, which guarantees convergence to a first-order stationary point for the constrained network: How does this constrained optimum relate to the stationary points of the original, unconstrained problem? Could the subspace constraint potentially prevent the model from reaching a better (e.g., lower loss) stationary point that exists outside the predefined subspace? \n3. In Figure 4, the 16-layer model appears to reach a similar training loss as the 8-layer model in much fewer training time. This is a notable result, as the 16-layer model is significantly larger and presumably requires more computation per step. Could the authors provide insight into why the larger model does not incur a proportionally longer training time to reach convergence in this setting?\n\nMinor suggestions:\n\n1. I would recommend increasing the font size for axis labels and legends in several figures, particularly Figures 1 and 4. The current size makes the details difficult to parse.\n2. There is a minor formatting inconsistency in the plot titles within Figure 4 ('8-layers' vs. '16 layers').\n3. In Figure 2, the convergence curves for the proposed compressed model and the centralized baseline are highly overlapping. It makes the plots difficult to visually distinguish.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on distributed training using model parallelism in bandwidth-constrained environments (e.g., Internet/WAN).\nObserving that activations and gradients mostly reside in a low-dimensional subspace, the authors project both (forward and backward) onto a low-rank basis for nearly lossless compression. As a result, their method achieves up to 99 % compression without degrading convergence. The improvement delivers a comparable time-to-accuracy under 80 Mbps links to that measured in datacenter-grade networks (>16 Gbps).\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The paper provides a thorough theoretical background based on mathematical formulations, establishing a clear link between the proposed algorithm to its experiment.\n- The paper shows impressive compression, 99\\% reduction without degrading convergence, through nearly lossless subspace projection.\n- The algorithm is simple (though not trivial) and effective. It appears generally applicable to all kinds of LLMs including transformer blocks, and is highly suitable for geo-distributed deployments aimed at cost-effective LLM serving or training.\n\n**Weakness**\n- **Novelty**: The motivation is quite conventional. Although the algorithm is theoretically concrete and effective, both the motivation and the way it is addressed lack novelty. As mentioned in Section 4.1, the low-rank property is a well-known phenomenon, and a line of prior work already compresses parameters (weights, activations, or gradients) using low-rank projection.\n- **Lack of evaluation**: Consistent with the point above, numerous works with similar motivations and solutions exist, yet the evaluation does not cover them thoroughly. While the experiments demonstrate the algorithm’s own merits, the paper doesn't include comparisons with other MP or DP methods. Similarly, there are insufficient comparisons with alternative compression techniques, including other low-rank projection approaches.\n\nQUESTIONS:\n- Although Section 6.5 addresses robustness, a broader evaluation would be helpful. Is rank collapse consistently observed across different models? Even if so, might the optimal rank dimension vary with model type, architecture, size, or training scheme (pre-training vs. fine-tuning)?\n- If the rank does vary, how can it be detected? For instance, the $k$ parameter seems almost randomly chosen, are there principled ways to adapt it to each model?\n- How do the training curves change as a function of $k$?\n- What does the loss curve look like if training continues far longer than in Figures 2–5 (until full convergence)? I’m wondering that the compression does not affect the final convergence.\n- How does the loss curve compare with other recent works that compress activations and gradients for MP? What advantages does this method offer over those approaches? What are the key differences from other low-rank projection techniques?\n- What is the overhead of the subspace-projection and full-reconstruction steps?\n\nTypo\n\nL311: 13x -> 13$\\times$\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel compression algorithm for model parallelism in decentralized training. By constraining projection matrices and activations to a low-dimensional subspace learned via Grassmann manifold updates, the method enables exact reconstruction of activations and gradients across layers. This yields up to 100× communication reduction with negligible compute/memory overhead, allowing billion-parameter transformer models to converge on 80 Mbps consumer-grade links as fast as on 100 Gbps datacenter networks\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The paper provides a novel and effective method for the decentralized training, and the method is validated through thorough experiments and analyses. \n- The writing provides a clear, concise motivation for decentralized model parallelism and its communication challenges.\n- The linkage between empirical observations (rank collapse) and theoretical analysis (Statements 5.2–5.3) is well articulated.\n- The mathematical treatment of the algorithm is beautiful.\n\n**Weaknesses**\n- More ablations would improve the quality of paper. \n  - The experiments involve training across different locations, but there’s no study on how the number of machines affects training speed or results.\n  - The paper doesn’t explore how the initial subspace  setup of $U_k$, especially the choice of $k$. affects training. Analyses on how the subspaces drift in the experiments (as mentioned in section 4.5) is helpful.  \n- The proposed method’s main advantage is that it enables training over low-bandwidth connections. What isn’t clear is exactly at what bandwidth threshold it starts to outperform other approaches—answering this would require comparing multiple methods across a range of bandwidths.\n\nQUESTIONS:\n-  The paper says the method has little extra computation or memory cost. It would provide more clarity if the measurements of the computation and memory is clearly stated in the start of section 6\n- The comparison to a centralized method is too brief. As the performance of the proposed method is very similar to the centralized method, can you provide more background introduction of the centralized method and explanation to understand this?\n- The words in the plot is too small.\n- In the figures in Appendix I, why the decentralized with no compressed is not visible when the x-axis is training steps? Would the loss be similar as the proposed method regarding the training steps?\n- The paper uses the Frobenius norm to measure distance between subspaces. What is the intuition behind the use of F norm?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

comments: SUMMARY:
The work proposes exploiting the observation that the weight gradients of projection matrices in a transformer block reside in a low rank space after training. The authors explicitly constrain the projection matrices to span a low rank space by matrix decomposition. By compressing the activations and weight gradients, the communication overhead is significantly reduced, allowing for model parallelism schemes where network bandwidth is limited.

STRENGTHS AND WEAKNESSES:
Strengths:
- The proposed method seems to be theoretically sound and well motivated.
- The theoretical insights are empirically verified, notably in 6.4 (fig 5) where the decentralized compressed model achieves similar convergence to the centralized model.
- The provided experiments, especially with cross-regional compute instances, are compelling . 

Weaknesses:
- The baseline methods are not representative of other communication efficient training schemes. The work compares to lossy compression schemes but not to other pipeline parallelism methods (e.g., Swarm parallelism).

QUESTIONS:
My primary request is to include baseline methods that cover pipeline parallelism or existing model parallelism strategies (see weakness). The work is technically sound and doesn’t sacrifice model convergence or performance. However, it is hard to contextualize the results within the broader literature without these points of comparison. With the inclusion of additional baselines, I am willing to raise my score.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-03