Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism
Authors: Sameera Ramasinghe
Thalaiyasingam Ajanthan
Gil Avraham
Yan Zuo
Alexander Long
Venue: NeurIPS 2025 poster ()
Keywords: Decentralized training
Distributed training
LLMs
Pipeline Parallel
Compression
Primary area: deep_learning
TL;DR: We propose a method to perform compression in pipeline parallel setting.

Abstract: Scaling models has led to significant advancements in deep learning, but training these models in decentralized settings remains challenging due to communication bottlenecks. While existing compression techniques are effective in data-parallel, they do not extend to model parallelism. Unlike data-parallel training, where weight gradients are exchanged, model-parallel requires compressing activations and activation gradients as they propagate through layers, accumulating compression errors. We propose a novel  compression algorithm that  compresses both forward and backward passes, enabling up to 99% compression with no convergence degradation with negligible memory/compute overhead. By leveraging a recursive structure in transformer networks, we predefine a low-dimensional subspace to confine the activations and gradients, allowing full reconstruction in subsequent layers. Our method achieves up to 100x improvement in communication efficiency and enables training billion-parameter-scale models over low-end GPUs connected via consumer-grade internet speeds as low as 80Mbps, matching the convergence of centralized datacenter systems with 100Gbps connections with model parallel.

Subspace Networks: Scaling Decentralized Training with Communication-Efficie Model Parallelism
Sameera Ramasinghe
Ajanthan Thalaiyasingam
Gil Avraham
Yan Zuo
Alexander Long
Pluralis Research
Abstract
Scaling models has led to significant advancements deep learning, but training these models in decentralized settings remains challenging due to communication bottlenecks. While existing compression techniques are effective in data- mata-parallel, they do not extend to model parallelism. Unlike data-paralled training, where weight gradients are exchanged, model model-parallel requires compressing activations and activation gradients they propagate through layers, accumulating compression errors. We propose a novel compression algorithm that compresses both forward and backward passes, enabling up to 99% compression with convergence degradation with negligible memory/compute overhead. By leveraging a recursive structure in transformer networks, we predefine low-dimensional subspace confine the activations and gradients, allowing full reconstruction in subsequent layers. Our method achieves up 100x improvement in communication efficiency and enables training billion- billion-parameee sarameter-scale models over low-end GPUs connected via consumer-grad grade internet speeds low as 80Mbps, matching the convergence of centralized datacenter systems with 100Gbps connections with model parallel.
Introduction
Scaling models and datasets has been pivotal in driving deep learning advancements, with model sizes expanding from millions of parameters (24] billions [21 [4] and trillions (38). These larger
models exceed the memory capacity of single device, requiring distributed training approaches to manage computation across multiple devices.
common solution distributed data parallelism (DDP) 27 or its more advanced variant, fully sharded data parallelism (FSDP) [22] which distributes data across nodes while replicating the model on each device. This enables larger batch sizes and higher throughput, but constrains the model size to the memory of single device. Model parallelism (MP) addresses this limitation by distributing parameters across devices 24 18] enabling the training of models that surpass single-node memory constraints. MP includes tensor parallelism, which splits individual layers, and pipeline parallelism, which distributes layers across devices; the latter is the focus of this work. Modern large-scale training combines DDP and MP to achieve scalability. Despite these strategies, all approaches require the transfer of large amounts of data between devices (34), limiting training to High-performance computing clusters with fast interconnects. These infrastructures are costly and accessible only to resource-rich organizations [46, 36), creating disparities that restrict broader research and risk centralizing innovation.
Decentralized training provides an alternative by leveraging consumer-grade devices, enabling individuals with small devices participate in large-scale model training, reducing reliance major corporations. This approach democratizes access to large-scale model training by leveraging
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

underutilizze GPUs in personal computers and volunteer networks [58 40). However, decentralized training faces significant challenges due to limited bandwidth and high latency in heterogeneous networks [58] necessitating communication-effiien compression algorithms to minimize data transfer while preserving training performance.
Most existing communication gradients are computed independently on each node (for each model replica) and then compressed before peer-to-peer communication. To this end, techniques such as sparsification [54] 52 30),
techniques focus on DDP 128
35,, where weight
and low-rank approximations [6]] exploit the redundancy in weight gradients to reduce communication. However. in MP, information must be passed between layers, requiring the communication of activations and activation gradients. Unlike weight gradients, activations lack inherent redundancy and approximation errors accumulate across layers, leading to degraded convergence [ [9. These challenges prevent the straightforward application of DDP compression techniques to MP, and hence to date, MP decentralized training remains infeasible, resulting in massive slowdowns over centralized training.
quantization [53
To bridge this gap, we propose a novel compression algorithm tailored for MP. We show that as training progresses, weight matrices exhibit rank collapse, converging low-rank subspaces. Thus, by explicitly constraining specific weight matrices to such low-rank subspaces and leveraging recursive structure inherent in transformer networks, we demonstrate that transformer layer activations- despite their high rank- can be decomposed into a dynamic low-rank component and a static high-rank component. This decomposition enables efficient compression of information passed between layers during both the forward and backward passes, ensuring lossless reconstruction in subsequent layers.
We validate the practical effectiveness of our approach through extensive evaluations billion- parameter-scale models. Our compression method enables the distribution of large-scale models across consumer-grade GPUs with internet-grade connections (80Mbps) while matching the con- vergence performance of centralized setups with 100 Gbps interconnects. We achieve up to 100x improvement in communication efficiency without any degradation in convergence. Further, we successfully train an 8B 3B-parameter LLaMA model [14] with layers split across four different geo- graphical regions, connected via the internet, and achieve convergence on par with baselines utilizing datacenter -raae connections. By addressing critical limitations in decentralized training, our method intend to remove significant barriers to scaling large models in resource-constralle environments, democratizing access to large-scale deep learning.
Related works
Decentralized training involves a group of autonomous devices (i.e., no central orchestrator) collaborating to train a large-scale model by leveraging MP/DDP methods. These devices, often geographically distributed and heterogeneous, are connected via networks with varying delays and bandwidth constraints. Key advancements in this area encompass both theoretical insights (8 23 [2] as well as practical approaches 42 [T]. Despite the progress, current efforts predominantly focus on DDP 28 23 22 11], which hinders scaling models beyond the memory capacity of local nodes. SWARM parallelism 40 and Tasklets (58), treat the problem scheduling challenge, with the former addressing the stochasticity inherent in decentralized cluster settings. However, all methods to date still face significant scalability challenges due to communication bottlenecks inherent in MP. Our method overcomes this limitation by introducing an effective communication compression technique for MP (specifically pipeline parallel), mitigating major barrier in scaling decentralized training.
Communication compression accelerates distributed training over handwidth-limine networks by reducing data transfers. Key strategies include sparsification, which transmits only significant parameter updates [54
52
30] quantization, which lowers communication by reducing parameter 55]; and low-rank projection, which compresses gradients via
precision [10
53 2
19
45
projection onto lower- diver-dimessioal subspaces [6]] 47]. While successful in data- mata-parallel settings (DDP), these techniques face difficulties in model-ppralle (MP) setups, including error accumulation across layers and unstructured activations 39 causing degraded convergence. Recent work by (50) proposed low-rank MP communication, but required significant architectural changes, preventing training from scratch. In contrast, our approach involves minimal initialization changes without architectural modifications, enabling efficient MP training from scratch with improved scalability.

---

3 Transformer block
We provide a brief exposition of the transformer block and proceed describe the proposed pression method. Let the imput to the 1t layer be X' Rxxnxd, where b. and d are the batch size, sequence length. and embedding dimension, respectively. Given the weight matrices of each V.A [Rdxd with dæ d/ H where H is the number of at- tention heads, the following computations
attention head h as W Q WK, " W
performed for each attention head: X2,b W Q. x = W K.h XV,, = X'W V.A- The rest of the computations are as follows:
X XQ. Kib vdu
head, fsallmax
XVvn
(1)
X = [X 11 Xbead
X min = X W + +
(2)
X = fietu (atu(Xinn (Ximn W!)
xl+1_ Xhitttnn W + X
where W Rdxd w E Rddd and W E Rddxd We omit the layer norms for brevity, which does not affect any of derivations. dg is usually integer multiple of d. We will refer to W and W as projection matrices from here onward.
Pı
4 Subspace networks
4.1 Rank collapse of projection matrices
We leverage the observation that weight gradi- ents of projection matrices inherently reside in a low-dimensional subspace, as widely reported
1800
across various architectures [26] 47 51 59 60
W
1700
7 566 16 (see Appendix
I for empirical val- idation). Together with AdamW's decoupled
W2
Stable Rank
weight decay-which suppresses negligible gra- dient components this drives projection matri- ces to converge toward a subspace spanned by dominant gradients, resulting effectively low- rank structures. Formal treatment is provided in
1500
1400
1300
Statements 5.2 5.3 rems.
and their associated theo-
1100
2000
4000
6000
8000
10000
Training steps
To validate this phenomenon, train an 8-layer, 2B-parameter model on WikiText (33 and track the stable rank of its projection layers during training (hidden dimension of 4096 and a con- text length of 2048. Stable rank is computed as E, / (ax)) where denotes the jth sin- gular value. As shown in Fig. [ the stable rank of projection matrices sharply declines, consis- tent with our theory. This structure, combined with the recursive nature of transformers, allows us to design a nearly-lossless low-rank compres- sion scheme. While prior works observe similar rank collapse in self-attention matrices 12 we focus specifically on projection matrices as the basis for compression. We also validate this using official checkpoints of fully-tru arainet large scale models (Appendix
Figure 1: Rank collapse in projection matrices. Consistent with Statements 5.2 and 5.3 (and Theo-
Appendix), we empirically observe natural collapse in the projection matrices (of non-compressed models). Shown is an 8-layer, 2B- 13-parrmettr model, with the stable (effective) ranks of the projection matrices for the 4th (middle) and 7h (penultimate) layers plotted over training steps.
rems E.2 G.I

---

4.2 Investigating the activation structure
We utilize the natural rank collapse we discussed thus far for compression. Observing the transformer we identify that a recursive structure emerges for layer outputs due to the skip
block in Eq. | aadd [
connections:
x
hiddeen W + X concal W Pi + X
(3)
which can be expanded as:
X
X
hidden
W + X
P2
concat p:
W
X
(4)
1=1
Here,
X PE + TE,
where PE, TE E Rxxxxd represent the positional and token embeddings, respectively. Let Pr..p.....,. .P.] be the sequence of positional indices, and (2,,2,.... 4n] be the corresponding token indices. We denote embedding matrices P E Rn for positional embeddings and T E Roxd for token embeddings, where the vocabulary length. Thus the tokens and positional indices are embedded via a lookup: PE = P P(p and TE = :].
Observing Eq. note that X PE TE contributes common additive term all layer outputs. Therefore, we consider the rank of the residual activations when PE and TE are subtracted:
1+1 X 1+1
PE
TE
Σ 1=1
(Xiitteen hidden W + X X W
(6)
Recall that Row(AB) C. Row(B): for any matrices A, B where Row(.) denotes the space. Thus, it is clear that if the rows of the projection weights (W p2s W pi up to layer / span common low-dimensional subspace, then the rows of X is also restricted to the same subspace, since vector spaces are closed under addition.
4.3 Compressing the forward pass
Recall that our analysis so far indicates that the projection matrices naturally confine themselves to smaller subspace as training progresses. Consequently, the residual activations /+1 are also restricted smaller subspace, if the union of those subspaces is low-dimensional. Based on this insight, it is intriguing to explore the feasibility of explicitly forcing the rows of projection matrices to vary within common low- -iw-dimensional subspace S, throughout training, facilitate activation compression. discussed Section 4.2 this forces the rows of activation outputs (X'+1) to span the same subspace S. Surprisingly, we find that even with extreme low-dimensional S, the networks can achieve almost the same convergence rates as in the unaltered ones. This allows us to significantly reduce the communication between blocks, which show next.
Let Row( w! ).) Row(R Ro S Further, Let U E Rdck be a matrix with orthonormal columns and Col(U,) S. Then, the following holds:
X - X U
In other words, X remains unaltered by the projection, since it is already in S. The above formulation introduces a property that can be leveraged for compression during the forward pass. Specifically, the dimensionality of x+ UJ E Rxxnxk is substantially smaller than that of X e R since k < d. If each node in a distributed system is initialized with shared copy of U this matrix does not need be transmitted repeatedly. Instead, X be transmitted to the next node, and the original XI+1 is reconstructed as:
X
- + PE+11 TE.
This approach ensures exact recovery of x without approximation.

---

4.3.1 Decomposition of high-rank components
For practical utilization of above approach, we still need to subtract PE and TE from X to compute X (and add them back in the next layer). While PE is deterministic and can be computed locally within each node, TE varies depending on the batch, making it impossible to do so.
One potential solution is restricting the embedding table 'T also to S. However, we observed that this degrades network performance due severely limiting the representation capacity the token embeddings, Instead, we propose modeling TE as a composition of a fixed high-rank component and a trainable low-rank component:
TE T Txxe (4 Ts(Lmm: :|
where we obtain trainable low rank embedding table Ts T training, T Tixed is transmitted to all nodes and stored. During the forward pass, we compress the activations as:
U.U At the beginning of
X x+l
= =(x1+1 + TTs(fims UR
compressed
=(X PE T Tixed (tmn::))us
which are generally high-rank-arr subtracted from X so the remaining (X!+1 PE - Tixed :l) is already S and is low-rank. Further, since Row(Ts :)) f Co)(U,), Reconstruction at the next node is then performed as:
Note that in Eq. both PE and Tixed (f whhhc
implicitly captured in the compressed activations X compressed
X X recovered
U + PE+ Tixed [f.m x
compaessed
ensuring a lossless recovery of X while not compromising its high ranked-ness.
natural question arises: would explicitly restricting the projection matrices fixed S, instead of allowing this property to emerge organically, adversely affect convergence? Note that this is form of constraint optimization and there are well known convergence guarantees. However, in Sec. [ provide convergence proof (to at least first-order stationary point) for completeness for the above partial projection case. Furthermore, we conduct extensive experiments across a variety of settings to empirically validate the convergence. Further, since the fixed embeddings are ephemeral, they have negligible effect on the peak GPU memory (Appendix K
4.4 Compression in backpropagation
In the previous section, we showed that constraining the rows of projection matrices to a shared low-dimensional subspace S, coupled with decomposing the embedding table into low-rank and high-rank components, facilitates compression of activations in the forward pass. This same constraint naturally facilitates lossless gradient compression in the backward pass. Specifically, let V (X(+1) denote the gradient of X with respect the loss, that needs to be propagated to the previous layer. This gradient can be compressed as:
V (x!+1)) compressed
(r(('++)),, E bxxnk
(9)
and subsequently fully recovered in the previous layer as:
L VX(X++) recovered =(V (X++)) compressed
U
(10)
Remarkably, this formulation ensures that the gradient flow to the computational graph prior X remains lossless, with no approximation error. Intuitively, after backpropagation through the parame- ter matrix W pa. the gradient has the form (x(X++))W) WT
pp Because Row(W, p.) S Col(U,) S.
the projection V vx(X+++ (x1+1 U, U. p2
WT does not alter the resulting gradient flow. Full derivation
provided in Appendix
5

---

4.5 Subspace updates using Grassmann manifold
We observe that restricting the column spaces of projection layers to a fixed subspace, even at high- ratios, is able to maintain surprisingly adequate convergence. To further improve the convergence, we allow the subspaces to slowly drift. To align the subspace with the gradient directions, we minimize the norm of the gradient components that lie outside the subspace, as measured at the last Transformer E IR fxxxxx denote the activation gradients at the last compressed transformer layer. The leftover gradient, which lies outside the subspace, is X. = V ( fimal (1 UxU
layer. Let V (X(ma)
where I - U U is the projection operator onto S-, We accumulate X over K iterations to obtain the metric Grassmann = R > km |Xfinal 1 where || || is the Frobenius norm. We aim to minimize LGnssmann over all possible Uk.
+ K
straightforward way to minimize Lonassman is by performing SVD on it and updating the subspace using the left singular vectors. However, abrupt changes to the subspace can disrupt convergence. Thus, we perform smooth updates by taking steps on the Grassmann manifold. The Grassmann manifold ((k, n) is the set of all k-dimensional subspaces of R.. A point on G(k, n.) is represented by an orthonormal matrix U E RR where the columns of U form a basis for the subspace. Thus, defining S as point on the Grassmann manifold enables taking smooth steps on the manifold. To minimize LGmssmann, we employ gradient descent on G(k, n). First, the Euclidean gradient of LGrassaann with respect UR. denoted V CGasomnn (U&), projected onto the tangent space obtain the Riemannian gradient:
(V Langeent
CGramaan (UR) U² U Crnaammm (Ux)-
(11)
Then, we perform a gradient descent step U = U₁ (V UR Jungent where 1j is the step size. Then, to map Unew back to the manifold, we apply retraction by orthonormalizing the columns of U using QR decomposition U R QR(U (UU))), ), where QR(.) denotes the QR decomposition. In practice, perform this subspace update U very infrequently (per every 500 iterations), and transmit to all the layers.
Theoretical insights
This section provides several theoretical insights for the proposed method. We structure our analysis into four key statements (and put the corresponding formal theorems in the Appendix). The first statement indicates that if there is a lossy compression between the layers, as the model depth increases, the approximation error of compression can grow exponentially. The second and third statements demonstrate that, even in an uncompressed network, if the weight gradients are confined particular subspace, the weight matrices also naturally converge to a oow-dimensional subspace with AdamW. The fourth observation establishes that explicitly enforcing a subset of weights onto dineeeeniional subspace does not harm convergence.
Statement 5.1
the compression activations and activation gradients between layers in a model-paralled setting introduces approximation errors, these errors can accumulate exponentially with increasing depth, provided the weight and activation norms are sufficiently large. Refer to Theorem D.I for formal proof.
The above result suggests that extending compression techniques from DDP (which are lossy) to MP (which requires compressing information passed between adjacent layers) leads the accumulation of approximation errors. This occurs because the compression at one layer directly impacts downstream layers, a phenomenon not present DDP, Additionally, the lack of exploitable structure in activations and activation gradients 6 33 typically results in larger approximation errors compared to the gradients of weights. This limitation makes such compression methods unsuitable for MP in large- scale models.

---

Statement 5.2
the gradients of a particular weight matrix network are constrained to a fixed subspace, then under AdamW, the weight updates asymptotically converge to that subspace over sufficiently large number of training steps. For a formal proof, see Theorem E.2
This provides a critical insight: if the gradients of an unconstrained network predominantly lie within a specific oow-dimensional subspace S-a property we empirically validate (see Appendix [ then the A optimizer asymptotically restricts updates outside of S. While this behavior is straight- forward for vanilla stochastic gradient descent (SGD), it non-trivial for Adam W due to its adaptive learning rate mechanism.
Statement 5.3
If Statement 5.2 holds, then the corresponding weight matrices asymptotically converge to the same subspace, irrespective of their initialization. For formal proof, see Theorem G.1
Intuitively, this result indicates that the decoupled weight decay mechanism in Adam systematically suppresses components of the weight matrix that receive negligible gradient updates. Consequently, the learned weights converge to a low-dimensional subspace defined by the gradient updates.
Statement 5.4
network in which a subset of weights is constrained to low-dimensional subspace converges first-order stationary point with a convergence rate of O(1/T). For formal proof, see Proposition H.l
This result is straightforward extension of the the convergence rate guarantces for constrained optimization using proximal gradient descent on non-convex functions. It shows that even when subset of parameters is restricted to lower-dimensional subspace, the standard convergence rate of O(1/T) in terms of stationarity remains intact.
Decantraaaed 120 oxprend (Dunc) BEMEge
Coc ancarnaaed 130 roeprent (htan) d8silge Contenlled 1000kps
Eecestiladd 130 copprean (OuSH 000rggg
Centrnleet 10002p
Centrnlaet 1000pp
Tranksg Lice (hard
Thinig 15005
Treake 15A
Figure Convergence in low-bandwidth settings. From left to right: WebTee, WikiText, and BookCorpus. In each plot, the training curves are presented against wall-clock time for an 8-layer (2B) model. Decentralized models utilize 80Mbps connections while the centralized model has datacenter- grade 100Gbps links. Our compressed model achieves on-par convergence to the centralized model, even under a 80Mbps bandwidth budget. In contrast, the non-compressed decentralized model with 80Mbps links suffers from significantly slower convergence due to the communication bottleneck.
Experiments
6.1 Experimental Setup
We evaluate decoder-only models (based on Llama 3 [14) across four large-scale datasets: WikiText (WT) (33. BookCorpus (BC) [63] WpenWeeText (OWT) [15, and C4 [37]. For WT, we use the standard splits; for BC and OWT, we randomly select 10% of training data as validation; for C4, due to computational constraints, we report training loss only. The base model has a context length of 1024, embedding dimension 4096, 24 heads, and 8 layers (~2B parameters); larger models (up to parameters) are noted explicitly in ablation sections. We use a base learning rate v) = 3e-4 (with warmup and lincar decay), weight decay 0.01, and batch size 32, unless otherwise specified. We
7

---

use GPipe 18 via torch. distributed. ing integrating our compression into all but the final transformer layer.
We initialize UR with isotropic Gaussian noise and set k = 40, achieving 100x compression. Bandwidth simulations sample from (B, 0.28) per pass, defining 'centralized' 100Gbps or 16Gbps setups, with all others as 'decentralized". Experiments (except the 8B Llama run on L4 GPUs with anternet-based decentralized connections) use A 10g GPUs (24GB VRAM) with one layer per GPU. Our method's 's effectiveness increases with faster accelerators, as slower GPUs allow more cmmputatio-coommniitt overlap.
Zafercnce
Tuaning
20
50
Strlps
40
10
s0Klape
100ihtx
Bundoralt
Figure 3: Throughput gain. Compressed mod- els significantly improve throughput under band- width constraints for inference (left) and training (right). Results shown for 8-layer (2B) models.
Figure 4: Performance vs. depth. Our compres- sion matches surpasses centralized baselines as depth increases (left: 8 layers; right: 16 lay- ers). With two layers per GPU, computational load rises, slightly narrowing the gap between centralized and decentralized models.
Square-Cube Law. Square-cube law [40] states that in distributed training, computation scales cubically with model size per node, while communication grows only quadratically. This partially offsets communication bottlenecks with computational overhead. Thus, c-times slower communica- tion does not lead to a c-times slower convergence. Hence, by improving communication efficiency by 100x, we achieve convergence speeds comparable to 100Gbps setups, even with 80Mbps links.
11
No Comperssion compression Oc1 DO)
TE TEST0OE
sauee Cang Shist BrHipe
10
Quantization (X16)
SVD (c100)
K (x1000
3-
5000
10000 12500 15000 27500 Training Stare
Figure 5: Training convergence and throughput on an 8B LLaMA model. All runs use 64 L4 GPUs distributed over 8 instances. Centralized instances reside within one region (min bandwidth 16 Gbps), while decentralized instances span 4 regions (min bandwidth 60 Mbps), highlighting pipeline parallel bottlenecks due to reduced inter- node bandwidth.
Figure 6: Comparison against lossy compres- sion methods. Top-k, low-rank (SVD), and quan- tization fail to converge at 100x compression, with quantization additionally limited by numer- ical precision. Our method matches the conver- gence rate of the uncompressed baseline.
6.2 Convergence in low-bandwidth settings
Our method enables training models over extremely low-dandwidth connections. We trained networks on both 80 Mbps and datacenter- grade 100 Gbps connections. Fig. 2 iillustrate the train curves of an 8-layer (2B) model against wall-clock time. As expected, training over 80 Mbps links in the decentralized setting significantly degrades convergence. In contrast, with our compression. the decentralized model achieves on par convergence to the network trained over 100 Gbps connections. To demonstrate the generalization, the perplexity scores of each model over validation sets is shown in Table. D. As evident, the decentralized model with our compression even surpasses the performance of the centralized model for the same training time. To further validate test-time performance,
8

---

train models to convergence using the compute-optimal 1:20 model-to- todel-to-token ratio from the Chinchilla scaling law [77. reaching compute-optimality at 12B training tokens with superior performance (Appendix
6.3 Throughput gain
Our compression also significantly ac- celerates inference. As inference re- quires less computation than train- ing, bandwidth becomes the dominant bottleneck; hence, our compression yields substantial gains. Fig. 3 illus- trates gains in both training and in- ference: inference, achieve al- most a 100x speedup at 80 Mbps. Al- though this advantage diminishes at higher bandwidths (e.g., 100 Gbps), still observe about 3 improve- ment. This indicates that even cen- tralized systems benefit from re- duced inference latency using our approach, which can translate into considerable cost savings with the recent trend of inference time scaling of large language models (44 A similar trend holds for training throughput as well.
Table I: Perplexity scores. Models trained for 12 hours on OpenWebText (OWT), BookCorpus (BC), and WikiText (WT), Bandwidth (B/W) and tokens per second (TPS) are re- ported. Our method outperforms even the centralized model, achieving significantly higher TPS compared to the non- compressed decentralized baseline.
Model
B/W
OWTT
BCL
WT|
TPS
Dec. Dec. Comp.
80Mbps
925.19
108.83 08.85
601.84
36.12
80Mbps
46.75
17.63
23.01
592.41
Cen.
100Gbps
47.22
18.35
23.08
602.57
6.4 Scaling across globally distributed GPUs
To further explore the scalability, we trained LLaMA 8B parameter variant 14 with 2048 context length, using TorchTTitan [29] on the C4 dataset across 64 L4 GPUs distributed across 8 instances. We use a pipeline parallel setup with 32 stages running layers are distributed one layer per stage. We evaluated two environment configurations: Centralized and Decentralized. In the Centralized setting all instances were located in the same cloud region and the bandwidth spans between 16Gbps-27Gbps. For the Decentralized case the 8 instances were distributed across 4 distinct regions (North America, Europe, and Asia). Additionally, no two consecutive stages were placed in the same region for the decentralized setup, hence, bandwidth spans from 60Mbps-350Mbps As shown in Fig. 5 our compression method in the decentralized configuration matches the wall-clock time (even slightly improving) and TPS with the centralized setting. In contrast, the decentralized setting w/o compression was 13x slower.
FSDP dimensions, where the 32 transformer
6.5 On other communication efficient distributed training methods
Our method is tailored for model parallel compression and therefore is orthogonal and complementary to data parallel communication-redde of them. To illustrate this, present experiment: we applied top-k and low-rank compression (DowerSGD [47)) to the weight gradients (replicating a DP scenario) while simultaneously using our scheme to compress activations and activation gradients. Under an 80Mbps link between devices, we achieved a substantial increase in TPS with no loss of convergence (see Table
techniques. This enables layering our compression on top
6.6 Ablations
We conduct ablations on the C4 dataset eval- uate the robustness of our method. Fig. 4 com- pares performance across model depths. If our compression were lossy, deeper models would accumulate errors, degrading performance rela- tive to the centralized baseline [6] (see also The- orem D.I). However, our results show that even as depth increases from 8 (2B) to 16 (3.5B) lay- ers, convergence remains on par with centralized baselines. Further, our large-scale experiment
Table 2: Our compression can be overlayed on DDP compression methods.
Method
TPS
Perplexity
Top-K (10%)
32
52.98
Top-K (10%) Ours
599
52.45
PowerSGD
29
31.18
PowerSGD Ours
532
30.26

---

(Fig. confirms that 32-layer models scale ef-
fectively, demonstrating decentralized training of large models with MP for the first time. Fig. also highlights that in the 16-layer model, assigning two layers per GPU (A100) 40GB VRAM) increases per-GPU computation, reducing bandwidth bottlenecks and narrowing the gap between decentralized and centralized models. This validates the square-cube law [40]. showing how computation-to. communication balance impacts decentralized training. However, note that decentralized training primarily targets low-end GPUs rather than high-end hardware. Ablations over other design choices and the negligible memory overhead are discussed Appendix K
6.7 Comparison against lossy compressions
As per Statement 5.1 standard compression methods used in DDP do not effectively extend to MP. We train an 8-layer model on the WikiText, comparing our compression method with TopK. quantization, and low-rank projection. As shown in Fig. O with an aggressive compression rate of 100, such compression schemes fail to converge. contrast, our method achieves convergence on with the non-compressed model. Note that for quantization, the best compression rate we can achieve is 16x for 16bit precision.
7 Conclusion
We propose a novel compression technique that, for the first time, enables aggressive compression in model parallel. By leveraging structured subspace constraints, we achieve up to 100x communication efficiency while preserving convergence. Our compression enhances both inference and training efficiency, reducing latency even in centralized settings. Extensive experiments across varying model depths, bandwidth conditions, and large-scale deployments validate the effectiveness and scalability of our approach. Notably, we demonstrate its real-world applicability by successfully training an 8B- SB-parameter Ilama model on low-end GPUs distributed across multiple global regions, connected solely via -rrdde (60 Mbps) links, while achieving convergence comparable to a centralized setup. Our results open practical pathway towards decentralized training of large-scale models using model parallelism.
Limitations
While provide foundational theoretical analysis, deeper insights into method would be beneficial. Notably, our compressed models sometimes outperform uncompressed baselines at equal training iterations, potentially due to implicit regularization effects. A rigorous exploration of this phenomenon remains open. Additionally, extending our analysis to advanced optimizers (e.g.. AdamW) and deriving tighter convergence bounds would strengthen the theoretical grounding; leave these directions for future work.
References
[1] Emmanuel Abbe, Samy Bengio, Enric Boix-Adsera, Etai Littwin, and Joshua Susskind. Transformers learn through gradual rank increase. Advances in Neural Information Processing Systems, 36, 2024.
[2] Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. Qsgd: Communication- efficient sgd gradient quantization encoding. Advances neural information processing systems, 30, 2017.
[3] Lukas Balles and Philipp Hennig. Dissecting adam: The sign, magnitude and variance of stochastic gradients. International Conference on Machine Learning, pages 404-113. PMLR, 2018.
[4] Jeremy Bersstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. signsgd: Compressed optimisation for non-convex problems. In International Conference on Machine Learning, pages 560-569. PMLR, 2018.
[5] Zhenni Bi, Kai Han, Chuanjian Liu, Yehui Tang, and Yunhe Wang. of Scaling test-time compute for enhancing Ilm reasoning. arXiv preprint arXiv:2412. 290778 2024.
[6] Song Bian, Dacheng Li, Hongyi Wang, Eric Xing, and Shivaram Venkataraman. Does compressing activations help model parallel training? Proceedings of Machine Learning and Systems, 2024.
10

---

[7] Romain Cosson. Ali Jadbabaie, Anuran Makur, Amirhossein Reisizadeh, and Devavrat Shah. Low-rank gradient descent. IEEE Open Journal of Control Systems, 2023.
[8] DeepSeck AT Team. DeepSeek LLMss, 2023.
[9] Alexandre Défossez, Léon Bottou, Francis Bach, and Nicolas Usunier. simple convergence proof of adam and adagrad. arXiv preprint arx:v:2003.00000 02395. 2020.
[10] Tim Dettmers, Mike Lewis, Sam Shleifer. and Luke Zeulemoyer. 8-bit optimizers via block-wise quantiza- tion. arXiv preprint arXiv:2110.02861, 2021.
[11] Michael Diskin, Alexey Bukhtiyarov, Max Ryabinin, Lucile Saulnier, Anton Sinitsin, Dmitry Popey, Dmitry V Pyrkin, Maxim Kashirin, Alexander Borzunov, Albert Villanova del Moral, al. Distributed deep learning in open collaborations. Advances in Neural Information Processing Systems, 34:7879-7897. 2021.
[12] Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. International Conference on Machine Learning, pages 2793-2803. PMLR, 2021.
[13] Arthur Douillard, Qixuan Feng. Andrei A Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Marc' Aurelio Ranzato, Arthur Szlam, and Jiajun Shen. Diloco: Distributed low-communication training of language models. arXiv preprint arXiv:231 1.02311.00100. 2023.
[14] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd models. arXiv preprint arXiv:2407.21788 2024.
[15] Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanic Tellex. Openwebtext corpus. Skylion007 www.ub.tttubbbb.... httpppeneeeeextCrrpuu..e 2019.
http://
[16] Guy Gur-Ari, Daniel A Roberts, and Ethan Dyer. Gradient descent happens in tiny subspace. arXiv preprint 112.::811.04785 2018.
[17] Jordan Hoffmann, Sehastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welhl, Aidan Clark, et al. Training compute-optimal large language models.
[18] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quae V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019.
[19] Sai Praneeth Karimireddy, Quentin Rebjock, Sebastian Stich, and Martin Jaggi. Error feedback fixes signsgd and other gradient compression schemes. In International Conference on Machine Learning, pages 3252-3261. PMLR, 2019.
[20] Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint 14Xiv: 2014.
[21] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learnin. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 28-28, 2020, Proceedings, Part V 16. pages 491-507. Springer, 2020.
[22] Anastasia Koloskova, Nicolas Loizou, Sadra Boreiri, Martin Jaggi, and Sebastian Stich. A unified theory of decentralized sgd with changing topology and local updates. In International Conference on Machine Learning, pages 5381-5393 PMLR, 2020.
[23] Anastasia Koloskova, Sebastian Stich, and Martin Jaggi. Decentralized stochastic optimization and gossip algorithms with compressed communication. In International Conference on Machine Learning, pages 3478 3487. PMLR, 2019.
[24] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[25] Haochuan Li, Alexander Rakhlin, and Ali Jadbabaie, Convergence of adam under relaxed assumptions. Advances in Newral Information Processing Systems, 36, 2024.
[26] Hongkang Li, Meng Wang, Shuai Zhang, Sijia Liu, and Pin-Yt Chen. Learning on transformers is provable low-rank and sparse: A one-layer analysis. In 2024 IEEE 13rd Sensor Array and Multichannel Signal Processing Workshop (SAM), pages 1-5. IEEE, 2024.

---

[27] Shen Li. Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, al. Pytorch distributed: Experiences accelerating data parallel training, arXiv preprint Xiv:20006.15704 2020.
[28] Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsich, Wei Zhang, and Ji Liu. Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent. Advances in neural information processing systems, 30, 2017.
[29] Wanchao Liang, Tianyu Liu, Less Wright, Will Constable, Andrew Gu, Chien-Chin Huang, Iris Zhang, Wei Feng, Howard Huang, Junjic Wang, et al. Torchtitan: One-stop pytorch native solution for production ready Ilm pre-training. arXiv preprint 1rXiv:2410.06511, 2024.
[30] Yujun Lin, Song Han, Huizi Mao, Yu Wang, and William Dally. Deep gradient compression: Reducing the communication bandwidth for distributed training. arXiv preprint arXiv:1712.01887, 2017.
[31] Mingrui Liu, Wei Zhang, Francesco Orabona, and Tianbao Yang. AdamO A stochastic method with adaptive variance reduction. arXiv preprint arXwv20011.11985, 2020.
[32] Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv: 2017.
[33] Stephen Mcrity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
[34] Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-Im. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-15, 2021.
[35] Bowen Peng, Jeffrey Quesnelle, and Diederik P Kingma. Decoupled momentum optimization. arXiv preprint urXiv:2411.19870 2024.
[36] How Much Longer Can Computing Power and Drive Artificial Intelligence Progress. Ai and compute.
[37] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints, 2019.
[38] Xiaozhe Ren, Pingyi Zhou, Xinfan Meng. Xinjing Huang, Yadao Wang, Weichao Wang. Pengfei Li, Xiaoda Zhang, Alexander Podolskiy, Grigory Arshinov, et al. Pangu-( (Sigma): Towards trillion parameter language model with sparse heterogeneous computing. arXiv preprint Xrv: 10845, 2023.
[39] Mikhail Rudakov, Aleksandr Nikolaevich Beznosikov, Ya A Kholodov, and Alexander Vladimirovich Gasnikov. Activations and gradients compression for model- training. In Doklady Mathematics, volume 108, pages S272 $281. Springer, 2023.
[40] Max Ryabinin, Tim Dettmers, Michael Diskin, and Alexander Borzunov. Swarm parallelism: Training large models can he surprisingly communication-effciene In International Conference on Machine Learning, pages 29416-29440. PMLR, 2023.
[41] Max Ryabinin, Eduard Gorbunov, Vsevolod Plokhotnyuk, and Gennady Pekhimenko. Moshpit sgd: Communication--fficiees decentralized training on heterogeneous unreliable devices. Advances Neural Information Processing Systems, 34:18195-18211 2021.
[42] Max Ryabinin and Anton Gusev. Towards crowdsourced training of large neural networks using de- centralized mixture-of-experts. Advances in Neural Information Processing Systems, 33:3659-3672, 2020.
[43] Sunny Sanyal, Ravid Ahwartz-Ziv, Alexandros G Dimakis, and Sujay Sanghavi. Inheritune: Training smaller yet more attentive language models, 2024. URL https://urxiv. htg/ubs/2404.086644
[44] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling 1lm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024.
[45] Hanlin Tang. Shaoduo Gan, Ammar Ahmad Awan, Samyam Rajbhandari, Conglong Li. Xiangru Lian, Ji Liu, Ce Zhang, and Yuxiong He. 1-bit adam: Communication efficient large-scale training with adam's convergence speed. In International Conference on Machine Learning, pages
PMLR, 2021.
12

---

[46] Verónica G Vergara Larrea, Wayne Joubert, Michael J Brim, Reuben D Budiardja, Don Maxwell, Matt Ezell, Christopher Zimmer, Swen Boehm, Wael Elwasif, Sarp Oral, al. Scaling the summit: deploying the world's fastest supercomputer. In High Performance Computing: ISC High Performance 2019 International Workshops, Frankfurt, Germany, June 16-20, 2019, Revised Selected Papers 34, pages 330-351. Springer, 2019.
[47] Thijs Vogels, Sai Praneeth Karimireddy, and Martin Jaggi. Powersgd: Practical low-rank gradient compression for distributed optimization. Advances in Neural Information Processing Systems, 32, 2019.
[48] James Vuckovic. Kalman gradient descent: Adaptive variance reduction in stochastic optimization. arXiv preprint 1rx1v:1810.12273, 2018.
[49] Chong Wang. Xi Chen, Alexander Smola, and Eric P Xing. Variance reduction for stochastic gradient optimization. Advances in neural information processing systems, 26, 2013.
[50] Hongyi Wang. Saurabh Agarwal, and Dimitris Papailiopoulos. Pufferfish: Communication models at no extra cost. Proceedings of Machine Learning and Systems, 3:365 386. 2021.
[51] Hongyi Wang. Scott Sievert, Shengchao Liu, Zachary Charles, Dimitris Papailiopoulos, and Stephen Wright. Atomo: Conmunication-efficiie learning via atomic sparsification. Advances in neural information processing systems, 31, 2018
[52] Jialei Wang. Mladen Kolar, Nathan Srebro, and Tong Zhang. Efficient distributed learning with sparsity. In International conference machine learning, pages 3636-3645. PMLR, 2017.
[53] Jue Wang, Yucheng Lu, Binhang Yuan, Beidi Chen, Percy Liang. Christopher De Sa, Christopher Re, and Ce Zhang. Cocktailsgd: Fine-tuning foundation models over 500mbps networks. In International Conference on Machine Learning, pages 36058-36076. PMLR, 2023.
[54] Jianqiao Wangni, Jialei Wang. Ji Liu, and Tong Zhang. Gradient sparsification for communication--fficcnn distributed optimization. Advances in Neural Information Processing Systems, 31, 2018.
[55] Jiaxiang Wu, Weidong Huang. Junzhou Huang, and Tong Zhang. Error compensated quantized sgd and its applications to large-scale distributed optimization. International conference on machine learning. pages 53325-533,, PMLR, 2018.
[56] Greg Yang, James B Simon, and Jeremy Bernstein A spectral condition for feature learning. arXiv preprint arXiv:2310.18813 2023.
[57] Xingyi Yang. Stochastic gradient variance reduction by solving filtering problem. arXiv preprint 1iXiv20012.12418 2020.
[58] Binhang Yuan. Yongjun He, Jared Davis, Tianyi Zhang. Tri Dao. Beidi Chen, Percy S Liang, Christopher Re, and Ce Zhang. Decentralized training of foundation models in heterogeneous environments. Advances in Neural Information Processing Systems, 25:25464-25477, 2022.
[59] Qi Zhang. Yi Zhou, and Shaofeng Zou, Convergence guarantees for rmsprop and adam in generalized- smooth non convex optimization with affine noise variance. arXiv preprint arXiv:2404.01438 2024.
[60] Jiawei Zhao, Florian Schäfer, and Anima Anandkumar. Zero initialization: Initializing neural networks with only zeros and ones. arXiv preprint arXiv:21 10. 12661, 2021.
[61] Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficiee 1lm training by gradient low-rank projection. arXiv preprint arXi::2403.03077, 2024.
[62] Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.111 11277, 2023.
[63] Yukun Zhu, Ryan Kiros, Rich Zemel. Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books, In The JEEE International Conference on Computer Vision (ICCV), December 2015.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification: see Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
Provided with supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeee 7/nips.com / ssiniioiicyp for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Experiments too expensice and therefore it is computationally expensive to report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative етror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics /ttpp:::///euriss... http://rrpps.cccpbblt////ttttaaa http://cccc///tttccccceeeeeeeeee
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate
17

---

to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
18

---

Justification:
No assets provided.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No human participants were used in the study.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: No study participants were used in the study.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMS if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality the research, declaration is not required.
Answer: [No]
Guidelines:
19

---

The answer NA means that the core method development in this research does not involve LLMs as any important. original, non non-standard components.
Please refer to our LLM policy (https://neurip:: /ttp://pp.ccc//offfeeeee//////// for what should or should not be described.
20


Reviews:
[{"review_id": "LEkFTkbijQ", "reviewer": "Reviewer_3Fcv", "summary": "The work proposes exploiting the observation that the weight gradients of projection matrices in a transformer block reside in a low rank space after training. The authors explicitly constrain the projection matrices to span a low rank space by matrix decomposition. By compressing the activations and weight gradients, the communication overhead is significantly reduced, allowing for model parallelism schemes where network bandwidth is limited.", "questions": "My primary request is to include baseline methods that cover pipeline parallelism or existing model parallelism strategies (see weakness). The work is technically sound and doesn’t sacrifice model convergence or performance. However, it is hard to contextualize the results within the broader literature without these points of comparison. With the inclusion of additional baselines, I am willing to raise my score.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- The proposed method seems to be theoretically sound and well motivated.\n- The theoretical insights are empirically verified, notably in 6.4 (fig 5) where the decentralized compressed model achieves similar convergence to the centralized model.\n- The provided experiments, especially with cross-regional compute instances, are compelling . \n\nWeaknesses:\n- The baseline methods are not representative of other communication efficient training schemes. The work compares to lossy compression schemes but not to other pipeline parallelism methods (e.g., Swarm parallelism).", "was_revised": true, "final_justification": "My main concern revolved around the baseline to compare to, most notably to pipeline parallelism, which I felt the authors adequately addressed in their rebuttal. To ask the authors for another MP activation-compression algorithm or PP algorithm when it does not exist yet, would be out of scope. I think the work stands alone regardless and the baselines in the paper and response are sufficient. As a result, I will be raising my score from a 4 to a 5.", "rebuttal": "Thank you for the encouraging comments and constructive feedback. We address each of the reviewer’s specific concerns below.\n\n# My primary request is to include baseline methods that cover pipeline parallelism or existing model parallelism strategies (swarm parallalism)\n\nWe thank the reviewer for the suggestion. Swarm implementation relies on standard activation and gradient quantization, and multiple node integration for computation pipes, for compression and increased throughput. Our technique is therefore orthogonal and can be layered on top of Swarm to further reduce bandwidth. To demonstrate this complementarity and address the reviewer’s concern, we have conducted an additional experiment that applies our compression within a Swarm setting; we train a 1B parameter model in swarm, with and without our compression. Results are summarized below and we will add the loss curve in the revised supplementary. As illustrated in, with our compression, swarm achieves a remarkable >11x throughput gain.\n\n| Method                    | Loss (after 12 hours) | TPS ↑   |\n|---------------------------|------------------------|--------|\n| Swarm  | 6.05                   | 1,502  |\n| Swarm + our compression  | 4.28                   | 17,072 |\n\nPlease note that activation and activation‑gradient compression during model‑parallel pre‑training has long been viewed as harmful to convergence, making it difficult to find suitable baselines. The recent survey paper [1], which proposes no new algorithm, evaluates 8‑bit / 4‑bit quantization and top‑k sparsification for MP. We already include these baselines, along with a vanilla low‑rank SVD variant, in Fig. 6; our method consistently outperforms them.\n\n[2] introduces an activation‑delta compression scheme for fine‑tuning, but it is impractical for LLM pre‑training for two reasons. (1) **Memory overhead**: the method must cache full activations for every training sample, which is infeasible at LLM scale. (2) Multi‑epoch dependence: the accumulated error is only removed after many epochs, so the approach fails to converge in the few‑epoch regimes typical of large‑scale training, where workloads rarely exceed a single pass over the data.\nA different research thread ([3,4,5]) focuses on memory efficiency by compressing activations within each module and decompressing them later to compute weight gradients. Because they leave the inter‑device traffic unchanged, these methods do not address communication efficiency and are therefore not comparable to ours.\n\nUpon further inspection, we did find a very recent workshop paper [6] that applies column masking to compress activations, but it achieves at most a 10 × reduction. In contrast, our technique delivers up to 100 × compression. We ran an experiment to validate the efficacy of our method using a 1B model on BookCorpus. The results are shown below after 5k iterations. Note  that their method’s convergence is  notably inferior to ours at 100x compression rate.\n\n| Method                | Perplexity \t↓ |\n|-----------------------|------------|\n| [6] (100× compression) | 147.15     |\n| Ours (100× compression)| 17.63      |\n\n**If the reviewer is aware of any other MP activation‑compression algorithms, we would greatly appreciate the citations. We will do our best to add ablation results for those methods during the discussion period and include them in the supplementary material.**\n\n[1] - Rudakov et al. - Activations and Gradients Compression for Model-Parallel Training\n\n[2] - Wang et al. AQ- SGD\n\n[3] - Shamshoum et al. - CompAct\n\n[4] - Chen et al. - ActNN\n\n[5] - Evans et al.  AC-GC \n\n[6] - Ramasinghe et al. Beyond Top-K"}, {"review_id": "dpRsfhtp7v", "reviewer": "Reviewer_ppVQ", "summary": "The paper introduces a novel compression algorithm for model parallelism in decentralized training. By constraining projection matrices and activations to a low-dimensional subspace learned via Grassmann manifold updates, the method enables exact reconstruction of activations and gradients across layers. This yields up to 100× communication reduction with negligible compute/memory overhead, allowing billion-parameter transformer models to converge on 80 Mbps consumer-grade links as fast as on 100 Gbps datacenter networks", "questions": "-  The paper says the method has little extra computation or memory cost. It would provide more clarity if the measurements of the computation and memory is clearly stated in the start of section 6\n- The comparison to a centralized method is too brief. As the performance of the proposed method is very similar to the centralized method, can you provide more background introduction of the centralized method and explanation to understand this?\n- The words in the plot is too small.\n- In the figures in Appendix I, why the decentralized with no compressed is not visible when the x-axis is training steps? Would the loss be similar as the proposed method regarding the training steps?\n- The paper uses the Frobenius norm to measure distance between subspaces. What is the intuition behind the use of F norm?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n- The paper provides a novel and effective method for the decentralized training, and the method is validated through thorough experiments and analyses. \n- The writing provides a clear, concise motivation for decentralized model parallelism and its communication challenges.\n- The linkage between empirical observations (rank collapse) and theoretical analysis (Statements 5.2–5.3) is well articulated.\n- The mathematical treatment of the algorithm is beautiful.\n\n**Weaknesses**\n- More ablations would improve the quality of paper. \n  - The experiments involve training across different locations, but there’s no study on how the number of machines affects training speed or results.\n  - The paper doesn’t explore how the initial subspace  setup of $U_k$, especially the choice of $k$. affects training. Analyses on how the subspaces drift in the experiments (as mentioned in section 4.5) is helpful.  \n- The proposed method’s main advantage is that it enables training over low-bandwidth connections. What isn’t clear is exactly at what bandwidth threshold it starts to outperform other approaches—answering this would require comparing multiple methods across a range of bandwidths.", "was_revised": true, "final_justification": "The paper presents solid theoretical analysis and comprehensive empirical experiments. During the rebuttal stage, the authors provided extensive ablation studies that were missing in the original submission. Given these improvements, I maintain my recommendation to accept the paper.", "rebuttal": "We thank the reviewer for the positive comments and intriguing questions. Please find our answers below.\n\n#  It would provide more clarity if the measurements of the computation and memory is clearly stated in the start of section 6\n\nThank you for the helpful suggestion. We have included detailed memory and computation analysis in Supplementary Sections K and L (see Tables 3 and 4) and show that ours incur extremely minimal overhead. As per the reviewer’s recommendation, we will also add a brief summary of these measurements at the beginning of Section 6 to improve clarity.\n\n# Can you provide more background introduction of the centralized method and explanation \n\nWe appreciate the reviewer’s concern. In our setting, the centralized model refers to the same architecture as the decentralized version, but deployed on a high-performance GPU cluster, where each transformer block (residing on a separate GPU) is connected to the others via high-bandwidth, low-latency links. This setup allows near-seamless communication between layers.\nIn contrast, our decentralized model places each transformer block on a node connected via low-bandwidth links, such as internet-grade connections. This introduces significant communication constraints, particularly for activations and gradients, which our method is specifically designed to address.\n\nThe strong alignment in performance between these two regimes highlights the effectiveness of our compression scheme in bridging the performance gap caused by bandwidth limitations. We will expand this explanation in the revised version to clarify the distinction and significance of the centralized baseline.\n\n# The words in the plot is too small.\n\nThank you for pointing this out. We will fix this.\n\n# Why the decentralized with no compressed is not visible when the x-axis is training steps? Would the loss be similar as the proposed method regarding the training steps?\n\nThe **decentralized uncompressed model** exhibits identical training dynamics to the **centralized model** when plotted against training steps, as the architecture and optimization procedure are the same. The only difference lies in the communication bandwidth between GPUs. As a result, its loss curve completely overlaps with that of the centralized model in figures where the x-axis is training steps, which is why it is not separately visible.\n\nHowever, it is important to note that, as shown, the tokens-per-second (TPS) of the decentralized uncompressed model is significantly lower due to the high communication cost over low-bandwidth links. This results in **much slower convergence** in wall-clock time, which is where our proposed method provides a clear advantage by enabling efficient training under bandwidth constraints.\n\n# What is the intuition behind the use of F norm?\n\nThis is indeed a great question. Below we summarize the rationale behind this choice.\n\n1. **Relation to principal angles**  \n   For two $k$‑dimensional sub‑spaces $\\mathcal U,\\mathcal V $ with orthonormal bases $U$ and $V$,\n   \n$$d_{F}(\\mathcal U,\\mathcal V)\n       =(|UU^{\\top}-VV^{\\top})|_{F}\n       =\\sqrt{2\\sum \\sin^{2}\\theta_i},$$\n\n   where $\\theta_i$ are the principal angles.  Therefore, minimising $d_F$ therefore minimises the **average squared mis‑alignment** across all $k$ directions.\n\n2. **Rotational invariance**  \n   Because the metric relies only on the projection operators $UU^{\\\\top}$ and $VV^{\\\\top}$, any orthonormal basis that spans the same sub‑space yields **exactly the same value**.  This ensures the distance is defined for sub‑spaces themselves, not for a particular choice of basis.\n\n3. **Average‑case discrepancy**  (instead of worst‑case)\n   While the spectral norm captures only the largest principal angle (worst‑case deviation), the Frobenius norm **sums over all angles**, giving an overall or average measure of how much the two sub‑spaces differ. This is the relevant notion of closeness when we care about preserving information in every direction, not just the single worst one.\n\n# The experiments involve training across different locations, but there’s no study on how the number of machines affects training speed or results.\n\nThank you for raising this point. To address it fully, could you please clarify what configuration you have in mind? Specifically, are you referring to increasing the number of nodes in a data‑parallel (DDP) setup? In that case, the results would be the same since that would be akin to training with a larger batch size. If the reviewer has a different setup in mind, we would like to  respectfully ask for more information so we can run the appropriate ablations and report their effects on throughput and convergence. \n\n# How the initial subspace setup of $U_k$ affects training. \n\nWe initialise $U_k$​ by drawing an isotropic Gaussian matrix and taking its top‑k principal components via SVD. After orthonormalising $U_k$​, the specific directions and the initial standard deviation no longer influence convergence.\n\n# How does $k$ affect training.\n\nIn our experiments, under specific hyperparameters, we achieved compression ratios as high as 200× without much performance degradation. Nonetheless, we used a more conservative 100× ratio ($k=40$) by default for stable performance across setups. This ratio also meets our practical goal: training in a pipeline-parallel setup even when nodes are connected via internet-grade networks. To address the reviewer’s request, we now include an ablation study varying k (and therefore the compression ratio). The results are shown below for a 1B model over BookCorpus. We thank the reviewer for the suggestion.\n\n| Method                   | Perplexity | Compression|\n|--------------------------|------------|------------|\n| Centralized              | 18.35      | - |\n| Decentralized (K = 40)   | 17.63      | 100x |\n| Decentralized (K = 20)   | 18.31      | 200x |\n| Decentralized (K = 10)   | 20.28      | 400x |\n| Decentralized (K = 5)    | 34.81       | 800x |\n\nWe agree that exploring the true upper limits and deriving more intuitive heuristics (potentially tied to model architecture or task requirements) is an exciting direction for future research. \n\n# Analyses on how the subspaces drift in the experiments  is helpful.\n\nWe agree that reporting subspace drift would be valuable, and we will add these statistics in the revision.\n\n# What isn’t clear is exactly at what bandwidth threshold it starts to outperform other approaches—answering this would require comparing multiple methods across a range of bandwidths\n\nThank you for this important question. As requested, we provide TPS comparing our method with alternative approaches across a range of bandwidths.\n\nWe observe that top‑k sparsification allows up to a 10× compression and 8‑bit quantization enables around 4× compression before significant degradation in convergence. Note that these findings are consistent with those reported in [1]. Accordingly, we fix those compression ratios for the baselines. We exclude SVD‑based low‑rank method as it is extremely slow.\n\nOur method achieves up to 100× compression without compromising convergence. This allows it to outperform other methods even when the available bandwidth drops to 80Mbps. The following results (to be included in the supplementary) show that our approach demonstrates a consistent TPS compared to  other methods across the range of bandwidths, making it highly practical for pipeline‑parallel training over internet-grade connections.\n\nWe appreciate the reviewer’s suggestion and will include the table and performance plots in the supplementary.\n\n| Method       | TPS @ 100 Gbps | TPS @ 1 Gbps | TPS @ 800 Mbps | TPS @ 500 Mbps | TPS @ 80 Mbps |\n|--------------|----------------|--------------|----------------|----------------|----------------|\n| Baseline     | 602            | 378          | 333            | 222            | 36             |\n| Top-K        | 612            | 532          | 521            | 451            | 129            |\n| Quantization | 608            | 499          | 487            | 415            | 83             |\n| Ours         | 624            | 612          | 610            | 603            | 592            |\n\n[1] - Rudakov et al. - Activations and Gradients Compression for Model-Parallel Training"}, {"review_id": "SYP61TUbfY", "reviewer": "Reviewer_HSht", "summary": "The paper focuses on distributed training using model parallelism in bandwidth-constrained environments (e.g., Internet/WAN).\nObserving that activations and gradients mostly reside in a low-dimensional subspace, the authors project both (forward and backward) onto a low-rank basis for nearly lossless compression. As a result, their method achieves up to 99 % compression without degrading convergence. The improvement delivers a comparable time-to-accuracy under 80 Mbps links to that measured in datacenter-grade networks (>16 Gbps).", "questions": "- Although Section 6.5 addresses robustness, a broader evaluation would be helpful. Is rank collapse consistently observed across different models? Even if so, might the optimal rank dimension vary with model type, architecture, size, or training scheme (pre-training vs. fine-tuning)?\n- If the rank does vary, how can it be detected? For instance, the $k$ parameter seems almost randomly chosen, are there principled ways to adapt it to each model?\n- How do the training curves change as a function of $k$?\n- What does the loss curve look like if training continues far longer than in Figures 2–5 (until full convergence)? I’m wondering that the compression does not affect the final convergence.\n- How does the loss curve compare with other recent works that compress activations and gradients for MP? What advantages does this method offer over those approaches? What are the key differences from other low-rank projection techniques?\n- What is the overhead of the subspace-projection and full-reconstruction steps?\n\nTypo\n\nL311: 13x -> 13$\\times$", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- The paper provides a thorough theoretical background based on mathematical formulations, establishing a clear link between the proposed algorithm to its experiment.\n- The paper shows impressive compression, 99\\% reduction without degrading convergence, through nearly lossless subspace projection.\n- The algorithm is simple (though not trivial) and effective. It appears generally applicable to all kinds of LLMs including transformer blocks, and is highly suitable for geo-distributed deployments aimed at cost-effective LLM serving or training.\n\n**Weakness**\n- **Novelty**: The motivation is quite conventional. Although the algorithm is theoretically concrete and effective, both the motivation and the way it is addressed lack novelty. As mentioned in Section 4.1, the low-rank property is a well-known phenomenon, and a line of prior work already compresses parameters (weights, activations, or gradients) using low-rank projection.\n- **Lack of evaluation**: Consistent with the point above, numerous works with similar motivations and solutions exist, yet the evaluation does not cover them thoroughly. While the experiments demonstrate the algorithm’s own merits, the paper doesn't include comparisons with other MP or DP methods. Similarly, there are insufficient comparisons with alternative compression techniques, including other low-rank projection approaches.", "was_revised": true, "final_justification": "I have raised my score from 4 to 5 (“Accept”). The paper presents a simple yet powerful approach, firmly grounded in a solid theoretical foundation, that I believe will have substantial impact.", "rebuttal": "We thank the reviewer for the insightful comments and feedback. We address the reviewer's concerns below.\n\n# Novelty,  the low-rank property is a well-known phenomenon, prior work already uses low-rank projection.\n\nWe thank the reviewer for their feedback and would like to clarify key points that distinguishes our contribution.\n\n**1. Low‑rank structure of  activations is *not* a well known property**\n\nWhile the community has long recognized that **weight matrices and their gradients often exhibit low effective rank** (enabling techniques such as LoRA, PowerSGD) **the same assumption does not hold for activations or activation gradients**. In transformers, activations are inflated in rank because PE and TE are re‑added at every layer. This resulted in an obscure view in earlier studies, leading them to report  accuracy drops when activations were compressed beyond ≈10%, concluding that aggressive activation compression is infeasible [1,2]. Another contributing factor to this is the exponential error accumulation (Theorem D.1). \n\n**2. Our key insight: remove embedding components, reveal a hidden low‑rank subspace**\n\n**We show that once PE and TE contributions are removed, the residual activations lie in the span of the layer’s output‑projection matrices**, which is a provably low‑rank subspace. \n\nConcretely: 1) PE can be subtracted inside each block because they are deterministic. 2) TE are batch‑dependent and high‑rank. We decompose them into a fixed high‑rank component and a learnable low‑rank component that can be compressed efficiently.\n\n**This decomposition exposes a low‑rank structure in activations and their gradients that, to our knowledge, has not been identified or exploited in prior work.**\n\n**3. Practical impact: efficient pipeline‑parallel training**\n\nBy isolating the compressible subspace, we achieve activation and activation‑gradient compression well beyond the previous rates,  without measurable loss in accuracy, enabling end‑to‑end pipeline‑parallel training with dramatically lower bandwidth. Previous methods focused on weight/gradient compression could not unlock this regime. **Also the network provably converges to a feasible solution with constrained optimization**\n\nWe hope this clarifies that our contribution goes beyond applying a “standard” low‑rank trick: it uncovers and leverages a previously overlooked low‑rank structure in activations.\n\n#  Comparisons with other MP or DP methods. \n\n**1. Comparison against other DP methods**\n\nBecause our algorithm is tailored to MP training, it is not directly comparable to DP approaches. DP assumes each full model replica lives on a high‑capacity cluster, which is less restrictive than splitting the computational graph across the internet, which is the setting we target. Existing decentralized or communication‑efficient DP methods address slow bandwidth between clusters, not between individual model layers, making that problem comparatively easier.\n\n**Our method is therefore orthogonal and complementary to DP communication‑reduction techniques and can be layered on top of them.** To illustrate this, we present an additional experiment below: we applied top‑k and low‑rank compression to the weight gradients (replicating a DP scenario) while simultaneously using our scheme to compress activations and activation gradients. Under an 80 Mbps link between devices, we achieved a substantial increase in TPS with no loss of convergence.\n\n| Method                                                | TPS | Perplexity @ 1k iters ↓ |\n|-|-|-|\n| Top‑K (weight gradients)                               | 32  | 52.98                    |\n| Top‑K (weight gradients)  + Ours (activations and activation gradients)    | 599 | 52.45                    |\n| PowerSGD (weight gradients)                            | 29  | 31.18                    |\n| PowerSGD (weight gradients)  + Ours (activations and activation gradients) | 532 | 30.26                    |\n\n**2. Comparison against other MP methods**\n\nAs noted in our previous response, activation compression in model‑parallel (MP) pre‑training has generally been considered harmful to convergence, so we are not aware of any published method that successfully compresses both activations and activation gradients during pre‑training.\n\nThe recent study [2] (which does not propose a new algorithm) benchmarks 8‑bit/4‑bit quantization and top‑K sparsification. We already include these baselines, and a vanilla low‑rank SVD variant, in Fig. 6, where our method significantly outperforms them. We would also like to add a note on why our method outperforms low-rank SVD methods.\n\n*Why our algorithm outperforms low‑rank SVD?*\n\n1. Embedding removal & constrained factorization of the projection matrices expose a strictly lower‑rank subspace; naïve SVD sees the full high‑rank tensor.\n\n\n2. No per‑batch SVD: we replace the O(d³) decomposition with a single matrix multiplication derived from our constrained optimization, yielding substantial runtime savings. \n\n3. Convergence guarantees as our method can be formulated as a projected gradient descent problem.\n\nFurther, [3] proposes compressing activation deltas during fine‑tuning, but this technique is infeasible for large‑language‑model (LLM) pre‑training: 1) **Memory overhead.** It must store the full activations for every training example, which is infeasible at LLM scale. 2) **Requirement for multi epoch training.** Error elimination requires many epochs, and the method fails to converge in the few‑epoch regimes typical of large‑scale training, where workloads often cover less than a single pass over the dataset. A different research thread ([4,5,6]) focuses on memory efficiency by compressing activations within each module and decompressing them later to compute weight gradients. Because they leave the inter‑device traffic unchanged, these methods do not address communication efficiency and are therefore not comparable to ours.\n\n\nWe did identify a very recent workshop paper [7] that uses column masking to compress activations. However, it achieves only up to 10× compression, whereas our technique reaches 100×. For completeness we have added a comparison: their convergence degrades sharply at an aggressive compression rate, while ours remains stable. We will add this to supplementary.\n\n| Method                | Perplexity \t↓ |\n|-----------------------|------------|\n| [7] (100× compression) | 147.15     |\n| Ours (100× compression)| 17.63      |\n\n**If the reviewer is aware of additional MP activation‑compression algorithms, we would be grateful for citations. We will try our best to provide ablation tables for such methods in the discussion period and include in the supplementary material.**\n\nWe further conducted an experiment with  Swarm parallelism [8]. Swarm’s  primary goals are fault tolerance and dynamic volunteer participation rather than communication efficiency. Swarm implementation relies on standard activation and gradient quantization for compression. Our technique is therefore is orthogonal to swarm too and can be layered on top of Swarm to further reduce bandwidth. To demonstrate this, we have conducted an additional experiment that applies our compression within a Swarm setting; the results are provided below.\n\n| Method                    | Loss (after 12 hours) | TPS ↑   |\n|---------------------------|------------------------|--------|\n| Swarm  | 6.05                   | 1,502  |\n| Swarm + our compression  | 4.28                   | 17,072 |\n\n#  Is rank collapse consistently observed across different models? Even if so, might the optimal rank dimension vary with model type, architecture, size, or training scheme?\n\nYes. As shown in Fig. 16, we analyzed the rank distributions of the projection matrices in multiple frontier open‑source models (DeepSeek, Llama, Qwen, and OLMo, spanning 4 B to 70 B parameters) and found them all to be highly rank‑deficient. \n\nIn our experiments, we were able to achieve compression ratios of up to 200× under certain hyperparameter settings, with minimal impact on performance. However, we adopt a more conservative 100× compression (using $k=40$) as the default to ensure stable performance across all of our configurations. Note that we use the gain consistent performance with this value across all our ablations (different datasets, model sizes etc.). This setting also aligns with our practical objective: enabling pipeline-parallel training over internet-grade connections. In response to the reviewer’s suggestion, we now include an ablation study varying $k$ (and thus the compression ratio). Results are reported below for a 1B model trained on BookCorpus. We sincerely thank the reviewer for this valuable feedback.\n\n| Method                   | Perplexity | Compression|\n|-|-|--|\n| Centralized              | 18.35      | - |\n| Decentralized (K = 40)   | 17.63      | 100x |\n| Decentralized (K = 20)   | 18.31      | 200x |\n| Decentralized (K = 10)   | 20.28      | 400x |\n| Decentralized (K = 5)    | 34.81       | 800x |\n\n# What does the loss curve look like if training continues until full convergence?\n\nWe have already trained models to full convergence using the compute-optimal 1:20 model-to-token ratio from the Chinchilla scaling law [9] in Appendix F. We achieve even better convergence than centralized models.\n\n# What is the overhead of the subspace-projection?\n\nWe have already provided these measurements in Appendix L.1. The overhead is approximately 1%, indicating a negligible computational burden.\n\n\n[1] - Bian et al. - Does Compressing activations help model parallel training?\n\n[2] - Rudakov et al. - Activations and Gradients Compression for Model-Parallel Training\n\n[3] - AQ-SGD\n\n[4] - Shamshoum et al. - CompAct\n\n[5] - Chen et al. - ActNN\n\n[6] - Evans et al. AC-GC\n\n[7] - Ramasinghe et al. - Beyond Top-K\n\n[8]- Ryabinin et al. - Swarm Parallalism.\n\n[9] - Chinchilla"}, {"review_id": "mev8yA1ouE", "reviewer": "Reviewer_VZDz", "summary": "This paper presents a communication compression algorithm designed to overcome bottlenecks in decentralized, model-parallel training of large-scale transformer models. The core idea is to leverage the natural rank collapse observed in transformer projection matrices by explicitly constraining these weights to a predefined low-dimensional subspace shared across devices. This constraint enables the nearly lossless compression and reconstruction of activations and gradients during both forward and backward passes, thereby avoiding the error accumulation that typically plagues compression techniques in MP settings. The authors empirically demonstrate that their method achieves up to 100x communication efficiency, enabling billion-parameter models to be trained over slow, consumer-grade internet connections while matching the convergence performance of centralized systems with high-speed interconnects.", "questions": "1. The proposed method focuses on compressing the projection matrices by constraining their row spaces. Is this approach orthogonal to, and potentially compatible with, other compression techniques? For example, could Subspace Networks be combined with methods that induce low-rank structure in self-attention matrices  or with other techniques used in DDP to achieve even greater communication savings?\n2. Regarding Statement 5.4, which guarantees convergence to a first-order stationary point for the constrained network: How does this constrained optimum relate to the stationary points of the original, unconstrained problem? Could the subspace constraint potentially prevent the model from reaching a better (e.g., lower loss) stationary point that exists outside the predefined subspace? \n3. In Figure 4, the 16-layer model appears to reach a similar training loss as the 8-layer model in much fewer training time. This is a notable result, as the 16-layer model is significantly larger and presumably requires more computation per step. Could the authors provide insight into why the larger model does not incur a proportionally longer training time to reach convergence in this setting?\n\nMinor suggestions:\n\n1. I would recommend increasing the font size for axis labels and legends in several figures, particularly Figures 1 and 4. The current size makes the details difficult to parse.\n2. There is a minor formatting inconsistency in the plot titles within Figure 4 ('8-layers' vs. '16 layers').\n3. In Figure 2, the convergence curves for the proposed compressed model and the centralized baseline are highly overlapping. It makes the plots difficult to visually distinguish.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper tackles the critical and highly relevant problem of communication efficiency in model-parallel training. By proposing a nearly-lossless compression method, this work has significant practical implications.\n2. The central insight is novel and well-motivated. The authors convincingly demonstrate—both theoretically and empirically—that key transformer weight matrices can be constrained to a shared low-dimensional subspace throughout training without degrading model convergence. \n3. The empirical results are a standout feature. Most notably, experiments show that a 2B-parameter model trained with the proposed compression over a 80Mbps connection achieves convergence on par with a baseline on a 100Gbps link.\n4. The methodology is well-grounded in theory, built upon the observation of rank collapse in projection matrices. The authors provide a solid theoretical framework, including proofs that justify the method's lossless nature and formal guarantees that the constrained network converges.\n\n\n\nWeaknesses:\n\n1. The paper lacks a clear, explicit description of the mechanism used to enforce the subspace constraint on the projection matrices. While it is stated that the rows of these matrices lie within the common subspace, the paper does not specify the detailed implementation. \n2. A crucial ablation study on the hyperparameter k (the subspace dimension) is notably absent. The experiments use a fixed k=40, but an analysis of the trade-off between compression ratio and model performance is essential for understanding the method's robustness and practical application.\n3. While a mechanism to update the shared subspace using the Grassmann manifold is proposed, it lacks a thorough sensitivity analysis. An ablation study on the necessity and frequency of this update is needed to understand its practical contribution.\n4. The paper's narrative flow could be improved. The separation of the methodology (Section 4) from its theoretical justification (Section 5) can be disruptive; integrating the theory more directly into the method's description could make the core ideas clearer.", "was_revised": true, "final_justification": "I think this is a technically good paper. The design are well-motivated with solid theorectical foundation and are verified through experiments.  The main problem with this manuscript lies in the presentation. The logical completeness of the writing can be further enhanced. However, the minor flaws do not overshadow the overall excellence, so I have decided to increase my score.", "rebuttal": "We are grateful for the constructive feedback and insightful comments. Please find our answers below.\n\n# The paper lacks a clear, explicit description of the mechanism used to enforce the subspace constraint on the projection matrices.\n\nWe thank the reviewer for pointing this out. In practice we enforce the constraint with an inexpensive matrix multiplication:\n\n$W_{projected}  = WUU_k$.\n\nWe will clearly mention this in the revision.\n\n# Ablation with $k$\n\nIn our experiments, we observed that compression ratios as high as 200× can be achieved under certain hyperparameter configurations with only minimal performance degradation. However, we chose to use a more conservative 100× compression (with $k = 40$) as the default and gain consistent performance across a variety of setups (different datasets, model sizes etc.), showcasing its robustness. This choice also supports our practical goal of enabling pipeline-parallel training over internet-grade bandwidth. In response to the reviewer’s suggestion, we now include an ablation study that varies $k$ (and thus the compression ratio). The results, shown below, are for a 1B model trained on BookCorpus. We thank the reviewer for this insightful suggestion.\n\n| Method                   | Perplexity | Compression|\n|-|-|--|\n| Centralized              | 18.35      | - |\n| Decentralized (K = 40)   | 17.63      | 100x |\n| Decentralized (K = 20)   | 18.31      | 200x |\n| Decentralized (K = 10)   | 20.28      | 400x |\n| Decentralized (K = 5)    | 34.81       | 800x |\n\n\n# While a mechanism to update the shared subspace using the Grassmann manifold is proposed, it lacks a thorough sensitivity analysis.\n\nWe thank the reviewer for pointing this out. Our observation is that the grassman updates matter towards mid/end training (we have provided an ablation in Fig.14), and very infrequent updates (every 500th iteration) are enough. We would add a more thorough ablation in the supplementary. \n\n# The paper's narrative flow could be improved. The separation of the methodology (Section 4) from its theoretical justification (Section 5) can be disruptive; integrating the theory more directly into the method's description could make the core ideas clearer.\n\nWe appreciate the reviewer’s input. We intentionally separated Section 4 (method) from Section 5 (theory) so that readers can first grasp the overall procedure without being slowed by mathematical details. In the camera‑ready, we will improve the flow by adding forward pointers in Section 4 (e.g., “See Sec. 5.2 for the formal proof of Step 3”) and brief intuitive summaries inside Section 5. This integrates the theoretical justification more tightly with the algorithm while still keeping the main text accessible to readers who prefer a math‑light first pass.\n\n# Could Subspace Networks be combined with methods that induce low-rank structure in self-attention matrices or with other techniques used in DDP to achieve even greater communication savings?\n\nThis is a very insightful question. Our subspace compression approach is indeed orthogonal to, and compatible with, other compression techniques, including those used in data-parallel (DDP). To demonstrate this, we conducted an experiment using a hybrid setup that combines DDP and pipeline-parallel PP training. In this configuration:\n\n1. We apply low-rank compression and top-K compression to weight gradients to mimic typical DDP compression strategies.\n\n2. Simultaneously, we apply subspace compression across the pipeline-parallel segments.\n\nWe train a 1B model in this setup with 80Mbps links. This combined setup yields strong end-to-end communication savings while preserving convergence as shown in below results.\n\n| Method                                                | TPS | Perplexity @ 1k iters ↓ |\n|-|-|-|\n| Top‑K (weight gradients)                               | 32  | 52.98                    |\n| Top‑K (weight gradients)  + Ours (activations and activation gradients)    | 599 | 52.45                    |\n| PowerSGD (weight gradients)                            | 29  | 31.18                    |\n| PowerSGD (weight gradients)  + Ours (activations and activation gradients) | 532 | 30.26                    |\n\n We also include results with Swarm parallelism [1], where our method is layered on top of standard Swarm compression (e.g., quantization).\n\n| Method                    | Loss (after 12 hours) | TPS ↑   |\n|---------------------------|------------------------|--------|\n| Swarm  | 6.05                   | 1,502  |\n| Swarm + our compression  | 4.28                   | 17,072 |\n\nThese results highlight that Subspace Networks can be composed with other compression schemes (e.g., low-rank, quantized gradient exchange) to enable efficient training across heterogeneous environments with varying bandwidth constraints.\n\nWe thank the reviewer for this excellent suggestion and will expand on these findings in the supplementary material.\n\n# How does this constrained optimum relate to the stationary points of the original, unconstrained problem? Could the subspace constraint potentially prevent the model from reaching a better (e.g., lower loss) stationary point that exists outside the predefined subspace?\n\nThis is an intriguing question. Our central argument for the overlap, and comparable performance, between the constrained and unconstrained optima is that the final solutions for $W_{p1}$  and $W_{p2}$ already reside in the low‑rank (constrained) sub‑space. We support this claim theoretically (Theorem E.2, G.1) and  empirically: as shown in Fig. 16, we analyzed the rank distributions of the projection matrices in multiple frontier open‑source models (DeepSeek, Llama, Qwen, and OLMo, spanning 4 B to 70 B parameters) and found them all to be highly rank‑deficient.\n\nBecause the true solution lies within the constrained space, projected gradient descent naturally converges to a first‑order stationary point that contains a feasible low-rank solution. While it is conceivable that alternative training strategies could discover higher‑rank optima with lower loss, our experiments and every set of publicly available weights we have examined indicate that state‑of‑the‑art models consistently converge to solutions inside the proposed low‑rank manifold.\n\n\n# In Figure 4, the 16-layer model appears to reach a similar training loss as the 8-layer model in much fewer training time.\n\nThis is an intriguing question and we thank the reviewer  for noticing this. Although the 16‑layer model requires more computation, (which is visible by lower TPS in Fig. 4), it also reduces the loss more quickly per training iteration. Similar behaviour can be seen in the publicly available OLMo training curves. We invite the reviewer to inspect the public WandB logs for OLMo, which exhibit the same trend. (We omit direct links here to comply with NeurIPS rules.)  Because the larger model needs fewer iterations to reach a given loss, its wall‑clock time to a similar loss can be shorter despite the higher per‑step compute cost.\n\n# Minor suggestions:\n\nThank you for pointing these out. We will address all of the above. \n\n[1] - Ryabinin et al. -  Swarm Parallalism"}]

Author rebuttal:

Meta-review: This paper introduces a novel technique for compressing communication in model-parallel training by projecting activations and gradients onto a low-dimensional subspace. The method achieves a remarkable 99% lossless compression ratio, enabling highly efficient decentralized training over low-bandwidth wide-area networks while matching the performance of datacenter-grade infrastructure.

The approach is both novel and effective. The connection between the empirical observation of rank collapse and the theoretical foundation is well articulated, and the experimental results—particularly the cross-regional training simulations—are compelling. The authors have thoroughly addressed all reviewers' concerns through extensive additional experiments and clarifications. As a result, all reviewers converged to strong positive ratings. Given the strength of the methodology and author responses, I recommend acceptance.
Decision: Accept (poster)