Mixtures of Subspaces for Bandwidth Efficient Context Parallel Training
Authors: Sameera Ramasinghe
Thalaiyasingam Ajanthan
Hadi Mohaghegh Dolatabadi
Gil Avraham
Violetta Shevchenko
Yan Zuo
Chamin P Hewa Koneputugodage
Alexander Long
Venue: NeurIPS 2025 poster ()
Keywords: Decentralized training
LLMs
Compressio
Distributed training
Primary area: deep_learning
TL;DR: We propose compression algorithm for large scale context parallel training

Abstract: Pretraining language models with extended context windows enhances their ability to leverage rich information during generation. Existing methods split input sequences into chunks, broadcast them across multiple devices, and compute attention block by block which incurs significant communication overhead. While feasible in high-speed clusters, these methods are impractical for decentralized training over low-bandwidth connections. We propose a compression method for communication-efficient context parallelism in decentralized settings, achieving a remarkable compression rate of over 95% with negligible overhead and no loss in convergence. Our key insight is to exploit the intrinsic low-rank structure of activation outputs by dynamically constraining them to learned mixtures of subspaces via efficient reparameterizations. We demonstrate scaling billion-parameter decentralized models to context lengths exceeding 100K tokens on networks as slow as 300Mbps, matching the wall-clock convergence speed of centralized models on 100Gbps interconnects.

Mixtures of Subspaces for Bandwidth Efficient Context Parallel Training
Sameera Ramasinghe
Ajanthan Thalaiyasingam
Hadi Mohaghegh Dolatabadi
Gil Avraham
Violetta Shevchenko
Yan Zuo
Chamin Hewa Koneputugodage
Alexander Long
Pluralis Research
Abstract
Pretraining language models with extended context windows enhances their abil- ity leverage rich information during generation. Existing methods split input sequences into chunks, broadcast them across multiple devices, and compute at- tention block by block which incurs significant communication overhead. While feasible in high-speed clusters, these methods are impractical for decentralized training over low-bandwidth connections. We propose a compression method for coimunicatioo-effccine context parallelism in decentralized settings, achieving a remarkable compression rate of over 95% with negligible overhead and no loss convergence. Our key insight is to exploit the intrinsic low-rank structure of activa- tion outputs by dynamically constraining them to learned mixtures subspaces via efficient reparameterizations. We demonstrate scaling billion-pant parameter decen- tralized models to context lengths exceeding 100K tokens on networks as slow as 300Mbps, matching the wall-clock convergence speed of centralized models on 100Gbps interconnects.
Introduction
Rapid scaling of large language models (LLMs) has made distributed training a necessity 23 20 38). As both model size and context length continue grow, efficient training increasingly depends on parallelization across multiple devices. Traditional distributed training paradigms assume high- bandwidth, low-latency interconnects, typically available in centralized data centers. In contrast to such centralized settings, the emerging paradigm of decentralized training [53] 39 [2 21] enables collaborative and democratized machine learning by distributing computation across heterogeneous, geographically dispersed nodes over the Internet, without requiring specialized networking hardware or centralized orchestration.
However, decentralized training presents a core technical challenge: limited communication band- width. When nodes are connected via commodity networks, communication quickly becomes bottleneck. Most prior work, has addressed this issue in the context of distributed data parallelism (DDP), where each node maintains a full model replica and synchronizes gradients during training. A variety sandwidth-efficiee techniques, such as gradient quantization [5]] 31 46), sparsification
49 44 45
and delayed synchronization [40 10] 27 9 have been proposed to reduce overhead in
this setting. Pipeline parallelism (PP) [19 where model layers are partitioned across devices, has
also been explored limited extent 36 50
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

A significantly more challenging and, to our knowledge, entirely unexplored setting is context parallelism (CP) in decentralized environments. CP has become critical for pretraining frontier LLMs, as it enables efficient training with extremely long sequences, often exceeding 100K tokens (e.g.- LLAMA [T] 130K, LLAMA 4: 256K, DEEPSEFK [6] 128K, QWEN [52]: 128K), thereby enhancing the model's 's ability to capture long-ran rrage dependencies. In context- context-paralle training, each node processes local chunk the input and broadcasts its attention activations to all other nodes in every layer and at every step. This imposes substantial communication demands, attention mechanisms require global access to all key and value activations. While centralized systems handle this using high-ban banddidhhh interconnects, all-to-all communication becomes prohibitively expensive in decentralized environments with low-bandwidth links.
In this work, we propose a method to drastically reduce the communication required for context- parallel attention without sacrificing model quality, enabling decentralized systems connected via standard internet-grade links to match the convergence performance of centralized systems with aatacenter-grade bandwidth. Our approach leverages the observation that attention activations (queries, keys, and values) often reside on low-dimensional manifolds. We exploit this structure by factorizing the attention weights so that outputs lie within dynamic mixtures of low-dimensional sub- spaces. To ensure convergence, we optimize the factored weights Riemannian product manifold and introduce an efficient reparameterization scheme that significantly reduces computational and communication overhead. Additionally, we provide theoretical guarantees on the expressivity and convergence of our method, offering principled justification for each design choice.
Crucially, this approach introduces only minor architectural changes with negligible training overhead, and these components can be removed after training, yielding a standard transformer architecture compatible with existing inference infrastructure and downstream deployment frameworks. We employ our method up to billion-parameter scale models under various settings and demonstrate that our method achieves over 95% communication compression without harming performance, enabling training with long context windows across devices connected via commodity internet (300Mbps), while matching the wall-clock convergence of centralized systems with high-speed interconnects (100Gbps).
2 Background and motivation
2.1 Context-paralle training and the communication bottleneck
We begin with brief exposition on CP training and refer the reader 11 for extended read. Transformer attention requires each query to interact with all key-value pairs, resulting a com- putational complexity that grows quadratically with context window. This becomes particularly prohibitive for long sequences, which makes parallelization strategies essential. In context-parallel settings, the input sequence X RRa where n is the context length and is the model dimension, is partitioned across m devices along the context dimension:
X
X
X T
X, BERRNX
Σ
n==
Each device i computes local queries, keys, and values per head. For clarity suppress the head index; all quantities are understood to be per attention head unless otherwise stated:
Q+ X.W, E Rd K, X,W, Rd I=WW E Rd
Computing attention locally requires global access to keys and values:
Kg K KJJT Raxd
VE [v]
V
Raxd
Typically, CP performs (some form of) all- all-gather, where each device broadcasts its local K,, V2 to form Kg, Vg. incurring communication cost O(nd) per device where d & n. Recently proposed Ring Attention 22 pipelines this communication in ring topology, incrementally exchanging local key-value blocks and computing partial attentions at each stage. Above methods fundamentally rely on the costly communication of large K, V matrices.

---

2.2 Attention Outputs Exhibit Low-Rank Structure
Our compression scheme inspired by the observation that the attention outputs of pretrained transformers lie on low-dimensional manifold. To support this,
we analyze publicly available checkpoints of large-scale pretrained LLMs and ex- amine their attention activations. Fig. presents an illustration of LLAMA 70B.
Specifically, we measure the stable rank of the query (Q), key (K), and value (V) activations across each attention layer. The stable rank of a matrix A E Rnxd is defined as:
0.000
(A) = ||ll where || denotes the Frobenius
|
0.000
norm and ||Al the spectral norm. Unlike the conven- tional matrix rank which is highly sensitive to small perturbations and numerical noise- the stable rank offers a robust, continuous measure of effective dimensional- ity. This makes it particularly suitable for characterizing learned neural representations, where numerous singular values are typically small yet non-zero due to noise or over-parameetrrzzaton.
0400
Lagar
Figure I: Attention outputs of LlaMa- 70B. Shown is the empirical rank of the Q. K, and V activations, normalised by their maximum possible rank, for ev- ery layer of the official LLAMA 70B checkpoint. All three projections extremely low-rank: Q and sit at roughly 0.1% of full rank, while V is slightly larger at ~ 0.5%.
As depicted in Fig. T the stable ranks of attention activa- tions remain low across all layers. Interestingly, Q and generally exhibit slightly lower ranks than V. indicating a higher degree of compressibility This observation underpins our approach, leveraging low-rank factorization for efficient compression. Further evidence of this phe- nomenon in other architectures provided in Appendix Next, we formalize this idea.
3 Method
We now present proposed method for efficient context-parallel transformer training. First, formalize how the empirically observed low-rank structure in attention activations enables effective compression. Next, we explain why using fixed subspace for compression can be overly restrictive, motivating our joint learning strategy that adaptively optimizes both the projection subspace and the attention weights ($3.I). We then introduce computationally efficient reparameterization approach that maintains optimality guarantees while significantly reducing overhead ($3.2). Finally, we describe how to reduce communication costs by dynamically compressing attention activations through per-chunk rotations and demonstrate how the model can seamlessly revert to a standard transformer architecture at inference time ($3.3) 3.5
In $ 2.2. we saw that the Q, K, V activations of large pretrained transformers exhibit a pronounced low-rank structure (Fig. D.. This finding implies that is feasible transmit only the low-dimensional components of these activations between devices, thereby achieving near-lossless compression in practice. Formally, let the columns of an orthonormal matrix U E Rdxr, with r << d, span the dominant subspace of the activations. Rather than communicating the full local activation matrix Z (WW E Rd,xd, where 7 E {K,V V} denotes key/value activations, W E (Wr, W.) and X is the input to the attention layer, we transmit only its compressed representation: Zeomp = WU E Xr The original activations can then be reconstructed at the receiving node Z Zcomp This compression method preserves all information within the subspace spanned by U, and is lossless when activations lie entirely in this subspace. Equivalently, this projection can be folded onto the attention weights and be interpreted as factorizing them into low-rank representation: W B(UUT), B E Iddxd
Sub-optimality of a fixed subspace. The formulation above implicitly assumes that an a priori choice of U is sufficiently expressive for every layer and every chunk in every optimization stage. It
We only need to compress K and V since Q can remain local.
3

---

is straightforward to see where this assumption can break down. Even if there is an optimal low-rank attention weight matrix, restricting weights to the form W BUU limits the search to the column space of U. If this space does not contain the true optimum, the model may converge to a suboptimal solution. In short, fixing U can prevent the model from reaching the best possible performance.
3.1 Joint Optimization over a Product Manifold
To address the limitations of a fixed U, we propose jointly optimizing factorization W = BUUT. Specifically, we simultaneously learn both the subspace representation U and the matrix B on the product manifold: M = dddd x St(n,, r). Here, B E Rddd is optimized in standard Euclidean space, whereas resides on the Stiefel manifold St(n, r). where updates can be naturally performed via Riemannian gradient descent D The following result establishes that this joint optimization achieves linear convergence under gradient descent.
Convergence. Let (WW, d) be smooth loss and consider the factorization W BUUT for attention weights where B E Rddd U St(d, and & R2 denotes all other parame- ters. Minimizing the reparameterized objective @B U, (BBUT,, 9) over the product manifold M : Rdxd x St(d, r) x IRP with Riemannian gradient descent, and under mild assumptions, yields Q-linear (geometric) convergence to first-order stationary point. For the formal result and proof, see Lemma (Appendix).
Note that since ||'|| = 1, the factorized objective remains Lipschitz smooth, and the convergence result naturally follows from the standard gradient descent theory both Euclidean and Riemannian manifolds. We include full proof in Appendix [ for completeness, explicitly treating the product manifold structure and assuming a Lojaskewe (PL. condition.
3.2 Reducing Computational Cost via Reparameterization of U
Direct optimisation of U on the Stiefel manifold St(n,, r) via Riemannian gradient descent provides strong theoretical guarantees but is costly: after every Fuclidean update, U must be re-orthonormalised (the standard "retraction" on to the manifold), which is performed with QR or SVD factorisation restore U U L,.. To mitigate this, we use an efficient reparameterization of U using fixed orthonormal basis U and learnable rotation R(0) O(d):
U(6) R(O)U,
where ((d) denotes the orthogonal manifold consisting of all d x d orthonormal matrices. If the mapping 0 + R()) is sufficiently expressive, rotations R(0) can fully parameterize (d), preserving the representational power of the manifold while significantly reducing computational overhead.
Preservation of geometry and stationary points. Reparameterizing U as U(0)= U(O) U moves the orthonormal constraint onto an unconstrained Euclidean variable 0, eliminating expensive QR/SVD steps and letting run ordinary SGD/Adam in 8-space. A natural concern is that this change of variables might distort the loss landscape and hinder optimization: however, we show that this is not the case. The chain rule shows V O 0, 0) = (B) g ((). f), so V is exactly the pull-back of the original Riemannian gradient. Thus, the first-order critical points remain unchanged. The following statement formalizes this result.
Equivalence of stationery points. Under the reparameterization U = R(0) U, minimizing & possesses exactly the same local minima and strict saddle points minimizing &. For the formal result and proof, see Theorem (Appendix).
Thus, we can effectively represent and optimize the projection subspace implicitly through rotations without compromising the quality or optimality of solutions.
"The Stiefel manifold St(d, r) is defined as set of all x matrices with orthonormal columns, formally given by rt(d,r) (U E dx U lr).

---

3.3 Reducing the Communication Cost
The reparameterization U()) R(0) U allows us to locally cache the fixed orthonormal frame U at each node, and transmit only the parameters 8. However, to fully parameterize rotations in the orthogonal group (dd), one typically requires }d(dd - 1) parameters, iee,, 0 E TR 1R/d2 We show next that in practice performing a dense search over all possible rotations unnecessary. Specifically, we can obtain a trade-off between the search space and the communication efficiency by controlling the dimensionality of 0.
To achieve more compact representation for the communication cost reduction, we select small set
of fixed skew-symmetric matrices {Ai...., } C o(d), A?" =- AA, (where (dd) denotes the Lie algebra of the orthogonal group) and define the corresponding k-dimensional Lie subgroup [16] 12
R& R(() = exp 0(1).) 10c R
where O(1) is the 1th element of 0. Because the exponential map is a local diffeomorphism around 0 = O. the set (0)) = R(0) U forms a k-dimensional submanifold of St(d, r) for sufficiently small ||@|]. Choosing k < dd(d 1) thus provides favorable trade-off between communication cost and representational flexibility. Importantly, our earlier result on the absence of spurious minima remains valid provided an optimal frame U. lies within (or sufficiently close to) the reachable manifold
{RU R E R as the mapping @ + (6) remains locally surjective onto this manifold.
3.4
Dynamic mixtures of subspaces via per-chunk rotations
depicted that the rotation dimension & controls and communication efficiency. With well-chosen a priori U, it becomes feasible to use small k; restricting the optimization to a local neighborhood around U.
3.3
trade-off between representational flexibility
We generate this prior through short, uncompressed warm-up phase, in which the model is trained for a small number of iterations (< 500) using a reduced context length to avoid communication bottlenecks. After this phase, each node computes the top r principal components of its local attention weights and stores them as a fixed subspace basis T St(d,r) r). Empirical evidence from prior work on
stabilization (e.g.. [3] [8] ) suggests that dominant activation subspaces emerge early in training, supporting this strategy.
Per-sample adaptation. Using a single global rotation for all inputs may underfit heterogeneous data. To retain expressivity without increasing k. we introduce a lightweight mechanism to predict unique rotation parameter for each sequence chunk. Recall that, in context- -prralel training, each node i processes distinct chunk X; E RRd from the input sequence. For an attention output chunk Z, = W X,, we employ a small linear prediction head: & : IR + IR* = & (Zmg.s). where Zagg,a is the average attention output of the chunk, generating chunk-specife rotation parameters. Given set of preshared skew-symmetric generators {A1) o(d) cached locally each node, construct the rotation as: R(O.) exp ((1)A) E RR (dd). Locally, keys and values are compressed as Zeomp,t = Z, R(O,) U Raxr (Z(ompp.) .). 02) then broadcasted, and the receiving nodes reconstructs the keys/values as: Z, (it.comp) U R(O,)T Note that peak memory is dominated by the attention computation, scaling as O(n?), making the linear head's overhead O(dk) negligible- an observation we also demonstrate empirically. The overall procedure is summarized in Algorithm
Bandwidth cost. In method, each node transmits nr floats (activations) in Ž and k additional scalars in 0. Typically, we have k < nr < nd, ensuring low communication overhead. Remarkably, we found that even using& k- 1- a single rotation angle that defines a plane is sufficient to preserve training stability and anput-adaptive flexibility, achieving bandwidth efficiency comparable to that of a fixed global rotation.
In implementation, we set S _V 1) dxd to be fixed and define the skew-symmetric generator A := [ (S ST,, A = AA, 2 For 0 E R we set the rotation R(0) = eXp(AA) E O(d), so A fixes the rotation plane while 0 sets its magnitude.
5

---

Algorithm Compression-awarre context parallel attention (per node, per head)
Require: Inpul E R Altention weight RR Warm started basis E R learnable linear head a - R."
syne syne
interval current step
1: Compute local keys and values: Z t- XW
2: Zug MaanTokon|
3: 8 v(2ne)
Compute rotation param local Construct data dependent subspace
4: R(0) C
5: Compress: Zexmp t- Z0
6: Breadcast (Zoop. 0) to all other nodes
7: Receive (somp.3); )
other nodes
8: all received (2 (omp).)) 8,) do
9: U3 15(,))
10:
7: (amp 1
b. Decompress
11:
end for
12: Aggregate global 2g E [, V4J, v/ from all nodes
Campute blockwise allention: A +- Softmax /vaiv
14:
if mod
then
15: W f- AllReeuceAvggW)
> Sparse syne of attention weights
16: end if
Second-order approximation.
Since is skew-symmetric, its spectral norm satisfies ||A||2 =
0. For sufficiently small [0] < < 1, the rotation matrix R(0) admits a second-order Taylor
approximation:
x10 04 A².
(1)
This approximation provides two key advantages. 1) Computational cost: scaling as O(d)), in contrast to the exact matrix exponential computation (c.g., by Padé Schur decomposition), which scales as Odd)). 2) Near identity bias: induces a beneficial near-identity bias, effectively acting approximately unbiased estimator of identity I when 0 is small and centered around zero (enforced via clipping). In this regime, higher-order terms vanish in expectation, yielding E[R(0)] = I. This property allows rotations to remain close to the initial warm-start subspace U, facilitating controlled local adaptation without significant drift. By fixing A and using scalar 8, achieve a communication complexity of (nr), significantly lower than the naive ((nd).
Attention weights must still be synchronized across devices, but they evolve far more slowly than acti- vations 6 [. We therefore average the corresponding weights only every e steps; in experiments we use 200, which incurs negligible communication overhead.
3.5 Unplugging the Projection Components
Our method augments the transformer architecture with two non-standard components: (i) small linear rotation head predicting the rotation parameters 0, and (ii) low-rank projection matrices U used for compressing activations. Although these components pose minimal overhead during training, strict API compatibility with off-the-shelf transformer models might be necessary for certain downstream applications.
As training proceeds, the learnable weights associated with our auxiliary projection heads collapse onto the data-dependent subspaces they steer. Once the model is close to convergence we can therefore drop these heads entirely, reverting vanilla Transformer without losing the predictive gains accumulated during training. The following result formalizes the collapse mechanism.
Bound on "idle" attention directions with data dependent projectors. Let the sample projector be P(x) (x)) Pick any other projector Q that projects onto arbitrary subspace. Define the average overlap PQ := Eg [[ E [0, 1]. Run stochastic gradient descent with weight decay O. Then, the attention weights that inside the Q-subspace obey lim;-.oo [
< L for an L Lipshchitz bounded loss. Hence, if the data almost never excites those directions (PQ < 1), the corresponding weights shrink away. That is, idle directions are pruned for free. For the formal theorem and proof, see Theorem 3 (Appendix).
Once the weights have collapsed onto their data-aligned subspaces, both the rotation head and its basis matrix U are redundant. We can therefore detach these components and perform a brief,

---

low-learning-rät rate fine-tuning pass to polish the remaining parameters. As Fig. 4 remains smooth across this transition, indicating that no optimisation shock is introduced.
ssows,, the loss curve
At inference time the model is now indistinguishable from standard transformer: it adds zero extra parameters, requires no custom kernels, and is fully compatible with existing deployment pipelines.
4 Related Work
Decentralized training. Decentralized learning dispenses with central coordinator, instead relying collective of autonomous devices that cooperate over mesh-style networks train large-scale models. These devices are typically heterogeneous and geographically dispersed, confronting links of nonuniform latency and bandwidth. The foundational theory convergence and robustness has been established by [24 22 21]. while complementary systems work has demonstrated practical viability on real clusters 41 7. Most prior art, however, is confined to DDP settings [24 22 21 77. limiting
model size to the aggregate memory of an individual node. Note that this is a comparatively well studied domain, and is orthogonal to the unexplored decentralized context parallel setting that explore. A notable work in DDP domain is Power Gossip (48), which replaces synchronous all-to-all communication with gossip-style information exchange among neighboring replicas arranged in mesh. Its key insight is that, when each replica trains independently via local SGD, the pairwise weight differences evolve in a low-rank sub-space, enabling them efficiently compress the weight differences during gossip. Another interesting DDP method is Photon [2).. where its communication savings stem primarily from infrequent gradient exchanges rather than from any explicit compression scheme. Such skip-syne approaches are infeasible in context- -prraalll pipelines, where activations must be transferred between nodes at every forward and backward pass. Nevertheless, these DDP-style techniques are orthogonal to our method and could be combined with it in hybrid setups.
Scheduling-oriented approaches such as SWARM parallelism 39 and Tasklets (53) alleviate straggler effects and network stochasticity, yet they still inherit the communication overhead intrinsic to the decentralized setting. In contrast, we introduce the first communication-compeeee comppeesooooo strategy tailored to CP, removing a critical bandwidth bottleneck that has thus far hindered scaling decentralized
models across larger context windows.
Context-parallel attention. For single single-device device long- unn-sseuence processing, sparse approximations such BigBird [54] cut attention complexity to On)), while IO-aware exact kernels like FlashAtten tion 33 maximize hardware throughput with tiling and on-chip caching. Recent systems research
parallelizes the sequence dimension itself [25] 37 15: Blockwise Parallel Transformers overlap
compute and ring-all-reducc to achieve near-linear speedups on sequences of 32K tokens (28). and RingAttention extends the idea virtually unlimited contexts via pipelined block exchanges (29)
These methods, however, still broadcast full key/value tensors. Our approach instead transmits a compact low-rank representation plus a lightweight rotation, reducing bandwidth while preserving exact attention semantics and thus complementing existing context-pane context-paralle frameworks.
Experiments
Das. OP Goep. (oES)) 3c0M0ps
10
Dics CP 200
10
Dac CP- 30004bgx
130
200
350
430
102 200
300
400
130 2000
350
600
Hooms
Hours
Figure 2: Convergence in bondddhdwidth settings. From left to right: Fineweb. C4, and BookCorpus. The training curves are presented against wall-clock time for an 8-layer (800M) model trained with 132K context window parallelized across GPUs. Decentralized models utilize 300Mbps connections while the centralized model has datacenter-grade 100Gbps links. Our compressed model achieves on-par convergence the centralized model, even under 300Mbps bandwidth budget. In contrast, the non-compressed decentralized model with 300Mbps links suffers from significantly slower convergence.

---

unplugging
10
Early unplugging
Loss
Chipp 300Mbys
20
15000
Thrusphpet
10030
6-
5000
0
2000
4000
6000
8000
10000
23
50
19
1.00
125
1.00
Iterations
Figure 3: Scaling across parallelism strategies. Our compression based CP scheme can be seam- lessly fused with other parallel training strategies. We train 3B- parammeeet model (32 layers) with both pipeline parallel and CP enabled across 32 A100s. Our compressed approach yields substan- tial throughput gains over uncompressed decen- tralized CP and nearly matches the performance of centralized CP.
Figure Unplugging projection components. After sufficient training, the rotation head and projection layers can be removed-ree revertigreverttng the network to a vanilla transformer- without impairing convergence. Training curves with dashed lines marking projection removal points. Late removal preserves convergence (see $.5. while early removal causes temporary disrup- tion followed by surprisingly rapid recovery.
5.1 Experimental Setup
We evaluate decoder-only models on three large-scale corpora Fine (FW)
30 C4 35 and
BookCorpus (BC) [56]. For each dataset, we reserve 10 % of the training split for validation. All model backbones follow LLAMA 3 [II] exact model specifications are given in the corresponding sections. We use sase-learring--aae
10- with linear warm-up and decay, and apply a weight-decay = 0.01. Every transformer layer is compressed except for the final block, where K and V projections are compressed by 98 % and 95 % (overall 96.5%), respectively by choosing w.r.t. d appropriately. We use the GPT2 tokenizer for all models.
5.2 Bandwidth efficiency in decentralized settings
We train 8-layer, 800M-parameter model (embedding-sizz)
2048, attention-heads 8 under two network settings: a centralized 100Gbps fabric and decentralized 300Mbps internet-grade links. Us- ing CP. process sequence length of 132K tokens across eight A100 GPUs connected the respective bandwidths. Fig. shows that vanilla CP over a 300Mbps link is more than 20x slower compared to a centralized 100Gbps mesh With our compres- sion, the same 300Mbps setup con- verges almost fast as the centralized baseline.
Table 1: Design ablations (val. perplexity J). All models are trained for 10K steps with a 132K context. Second-order approximations preserve performance, while overcompress- ing V degrades it.
SETTING
FW
C4
BC
Ours
22.64
23.33
25.27
Ours Fixed Ú
26.57
27.11
30.33
Ours + Rand. rot. R(O)
24.93
25.17
29.58
Ours 2nd-order approx.
22.64
23.33
25.27
Ours No warm start
26.63
26.91
30.15
Ours (K, V - 98%)
24.74
24.99
29.46
Ours (K 99%, 95%)
24.68
24.91
29.22
Validation. Table reportts test-time performance of the trained models. To this end, we train each model up to its compute-optima point, following the Chinchilla scaling law [17]. Specifically, for our 800M-parameter models, we use 20 model-to-token ratio and train for 16B tokens on each dataset. Remarkably. our compressed decentralized model matches, and even slightly outperforms, perplexity of the centralized model at the same number of training iterations, while delivering significantly higher throughput than vanilla (uncompressed) CP over commodity links. Training the uncompressed model to completion over wwwdaahwidtth links is computationally infeasible (estimated at over 150 days), report only its throughput (TPS) in this setting.

---

Table 2: Validation perplexity (1) and throughput (TPS), All models are trained with 132K
context window to the compute-optimal point [17] (16B training tokens). Our method yields a 2 TPS boost while slightly outperforming centralized CP in perplexity, with minimal memory overhead.
MODEL
FW
C4
BC
TPS
MEM (GB)/GPU
Cen. 1000bps
17.18
17.51
17.88
56K
38.4
Dec. CP- 3000bpss
2.7K
38.4
Dec. CP Comp. 300Mbps (ours)
17.06
17.47
17.81
55K
(x20)
38.7 (+0.7%)
Training uncompressed models to convergence at 300Mbps is infeasible (> 150 days); only throughput is reported.
5.3 Ablations
We perform ablations on 800M-parameter models with 132K context across eight A100 GPUs (see Table D Models using learned rotations outperform those with fixed or random projections. The second-order exponential approximation does impact per- formance, confirming its adequacy. Omitting the warm-start initialization of princi- pal directions (U) noticeably degrades results, highlighting the importance of this prior.
Table 3: Effect of warmup steps (val. perplex- ity 1). All models trained for 10K steps with a 132K context. The method not highly sensitive the number of warmup steps.
Our compression based CP scales well and can be seamlessly fusing with other parallel training strategies. We scale the model 32 layers (3B parameters) with both pipeline parallel and CP enabled over 32 A100s (Fig. B and achieve significant throughput gain.
Scaling:
WARMUP-STEPS PERPLEXITY
0
26.63
Reparameterizzation: A key step of our method is reparamcterizing U which bypasses expensive Riemannian operations (QR/SVD pullbacks). As
100
24.44
300
22.66
500
22.64
shown in Table 4 this reparameterization signifi-
1000
22.87
2000
22.64
cantly improves throughput (TPS). More ablations against architecture choices are provided in Ap- pendix C
5000
22.71
Warmup steps: To measure the effect warmup
steps of we conducted an ablation study varying the warm-up duration and evaluated the resulting perplexity on the Fin dataset. The results are shown in Table As demonstrated, even with a reduced warm-up of 300 steps, the model achieves comparable performance, indicating significant degradation. In practice, we default to 500 steps to provide a safe and stable baseline. This study further emphasizes the lightweight and robust nature of our warm-up strategy, especially in contrast to the more elaborate scheduling mechanisms commonly employed in modern LLM pre-training. Note that the perplexity differences are minor and stable, indicating performance is stable after 300 warm-up steps.
5.4 Unplugging Projections and Rotation Heads
As discussed in $. 3.5 practitioners may prefer reverting to a standard transformer after pretraining for compatibility with downstream frameworks. We empirically validate our theoretical prediction that attention weights progressively align with the projection subspace, allowing safe removal of projection layers and rotation heads near the end of training. Fig. 4 shows that removing these components late preserves convergence, while doing so prematurely disrupts training.
5.5 Comparison Against Baselines
As no prior baselines exist for CP compression, we construct two: (i) Sparsification Top- 10% scheme (90% compression), transmitting only the largest-magnitude entries of the K, V chunks, inspired by common DDP compression methods; (ii) Quantization 4-bit quantization (75% compression) of the K, V activations prior transmission, following standard practices in activation

---

compression. As shown in Fig. (left), we outperform these baselines comprehensively (132k context window) even when using more aggressive compression rate of 96.5%.
For completeness, we also compare against long-
context models BigBird [54] and CosFormer [34. which are not designed for CP and can handle at most 32K tokens on A000. For fair compar- ison, we apply our compression to CP across four GPUs, each processing 8K tokens. As shown in Fig. [ (right), both baselines exhibit significantly worse convergence than our method. All experiments are performed on 800M parameter models.
(96 5%6
Quantias Topk (90%) Basskine
SigBled CosPartier
Staselin
5000
10000
5000
1000s
Itenstion
Iterntaois
Figure 5:
Baseline comparisons. Left: Because no method currently compresses pontext-parallel training, we build two base- lines; Top-k sparsification and quantization. Right: We also compare with long-context models BigBird and CosFormer. Both limited to 32K tokens on A A100 GPUs, so all models are evaluated that length. In both panels, our compressed CP curve is nearly in- distinguishable from the uncompressed refer- ence, whereas every baseline falls well short.
6 Conclusion
We propose the first compression method that en- ables context-paralle training of language models in decentralized environments with low-bandwidth interconnects. Our approach supports training with context lengths over 100K tokens isolated GPUs connected via internet-grade links (e.g., 300Mbps), while matching the wall-clock convergence of cen- tralized systems with high-speed (100Gbps) connec- tions. Additionally, our method preserves compatibil- ity with standard transformer architectures by allow- ing the projection layers to be removed after training.
facilitating seamless deployment in downstream frameworks. We provide a theoretical analysis of the key properties of our method and validate its effectiveness through extensive empirical evaluation.
Limitations
Our compression method delivers noar-lossless convergence in context-parallee training, but sev- eral open questions remain. First, alternative reparameterisations beyond simple subspace ro- tations may unlock further accuracy or efficiency gains. Second, the method's surprising ability to locate good minima even as the search space heavily reduced (via very low-dimensional 0) lacks a rigorous explanation; its ties to re- cent work on implicit regularisation and lottery- ticket-style phenomena deserve closer study. De-
Table 4: TPS gain from design choices. Reparam- eterization and second-order approximation yield significant throughput improvements.
SETTING
TPS (t)
Ours
55K
w/o reparam.
37K
w/o 2nd ord. approx.
30K
spite these gaps, this work establishes the first baseline for pontext-parallel compression and we hope it spurs deeper theoretical and empirical exploration.
References
[1] Emmanuel Abbe, Samy Bengio, Enric Boix-Adsera, Etai Littwin, and Joshua Susskind. Transformers
Ieam through gradual rank increase. Advances in Neural Information Processing Systems, 36, 2024.
[2] P-A. Absil, Robert Mahony, and Rodolphe Sepulchre. Optimization Algorithms on Matrix Manifolds. Princeton University Press, 2008.
[3] Alibaba DAMO Academy. Qwen: Efficient and scalable language models. 1xXiv22309.16009 2023.
arXiv preprint
[4] Allen Al. Olmo: Open language models, arXiv preprint arXiv:2402.02300 2024.
[5] Jimmy Ba, Geoffrey Hinton, Volodymyr Mnih, Joel Z. Leibo, and Catalin Ionescu. Using fast weights to attend to the recent past. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
10

---

[6] Chen Chen. Hong Xu, Wei Wang, Baochun Li, Bo Li, Li Chen, and Gong Zhang. Synchronize only the immature parameters: Conmunication-effiieenl federated learning by freezing parameters adaptively. IEEE Transactions on Purallel and Distributed Systems, 2023. Early access.
[7] Michael Diskin, Alexey Bukhtiyarov, Max Ryabinin, Lucile Saulnier, Anton Sinitsin, Dmitry Popey, Dmitry V Pyrkin, Maxim Kashirin, Alexander Borzunov, Alhert Villanova del Moral, et al. Distributed deep learning in open collaborations. Advances in Neural Information Processing Systems, 34:7879-7897, 2021.
[8] Yihe Dong, Sean-Baptiste Cordonnier, and Andreas Loukas, Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International Conference on Machine Learning, pages 2793-2803. PMLR, 2021.
[9] Arthur Douillard, Yanislav Donchev, Keith Rush, Saryen Kale, Zachary Charles, Zachary Garrett, Gabriel Teston, Dave Lacey, Ross Mcllroy, Jiajun Shen, al. Strcaming diloco with overlapping communication: Towards a distributed free lunch. arXiv preprint arXiv:2501. 185 2025.
[10] Arthur Douillard, Qixuan Feng, Andrei A Rusu, Rachita Chhaparia. Yani Donchev, Adhiguna Kuncoro, Mare Aurelio Ranzato, Arthur Szlam, and Jiajun Shen. Diloco: Distributed now-comuunication training of language models. arXiv preprint arXiv:2311.000 1.08105, 2023.
[11] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llaaa 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
[12] Alan Edelman, Tomás A. Arias, and Steven T. Smith. The geometry of algorithms with orthogonality constraints. SIAM Journal on Matrix Analysis and Applications, 20(2):303-353, 1998.
[13] Ruili Feng, Kecheng Zheng, Yukun Huang, Deli Zhao, Michael Jordan, and Zheng-Jun Jun Zha. Rank diminishing in deep neural networks. Advances in Neural Information Processing Systems, 35:33054- 33065, 2022.
[14] Elias Frantar and Dan Alistarh. Gptq: Accurate post-training quantization for generative pretrained transformers. arXiv preprint arXiv:2303.00775, 2023.
[15] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint 2407. 2024.
[16] Brian C. Hall. Lie Groups, Lie Algebras, and Representations: An Elementary Introduction, volume 222 of Graduate Texts in Mathematics. Springer, Cham, 2 edition, 2015.
[17] Jordan Hoffmann, Sehastian Borgeaud, Arthur Mensch, Flena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks. Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models.
[18] Yuxuan Hu, Jing Zhang, Zhe Zhao, Chen Zhao, Xiaodong Chen, Cuiping Li, and Hong Chen. Sp Enhancing structured pruning via pea projection. In ACL (Findings), 2024.
[19] Yanping Huang. Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee. Jiquan Ngiam, Quoe V Lc, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019.
[20] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learnin.. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part V 16. pages 491-507. Springer, 2020.
[21] Anastasia Koloskova, Nicolas Loizou, Sadra Boreiri, Martin Jaggi, and Sebastian Stich. unified theory of decentralized sgd with changing topology and local updates. In International Conference on Machine Learning, pages
PMLR, 2020.
[22] Anastasia Koloskova, Sebastian Stich, and Martin Jaggi Decentralized stochastic optimization and gossip algorithms with compressed communication. In International Conference on Machine Learning, pages 3478-3487. PMLR, 2019.
[23] Alex Krizhevsky, 1lya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
11

---

[24] Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsieh, Wei Zhang, and Ji Liu. Can decentralized algorithms outperform centralized algorithms? case study for decentralized parallel stochastic gradient descent. Advances in neural information processing systems, 30, 2017.
[25] Wanchao Liang, Tianyu Liu, Less Wright, Will Constable, Andrew Gu, Chien-Chin Huang, Iris Zhang, Wei Feng, Howard Huang. Junjie Wang. et al. Torchtitan: One-stop pytorch native solution for production ready IIm pre-training. arXiv preprint crXiv:2410.06511, 2024.
[26] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, al. Deepseek-v3 technical report. arXiv preprint 2xXv:24412.19337 2024.
[27] Bo Liu, Rachita Chhaparia, Arthur Douillard, Satyen Kale, Andrei Rusu, Jiajun Shen, Arthur Szlam, and Marc' Aurelio Ranzato. Asynchronous local-sgd training for language modeling. arXiv preprint arXiv:2401.099 133.,,5............... 2024.
[28] Hao Liu and Picter Abbecl. Blockwise parallel transformers for large context models. Advances in neural information processing systems, 06:8828-8844 2023.
[29] Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context. In International Conference on Learning Representations (ICLR), 2024. arXiv:2310.0188)
[30] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
[31] Roula Nassif, Stefan Vlaski, Marco Carpentiero, Vincenzo Matta, Marc Antonini, and Ali H Sayed. Quan- tization for decentralized learning under subspace constraints. IEEE Transactions on Signal Processing, 71:2320- 23300022333,00 2023.
[32] Jorge Nocedal and Stephen J. Wright. Numerical Optimization. Springer, 2nd edition, 2006.
[33] Mattco Pagliardini, Daniele Paliotta, Martin Jaggi, and François Fleuret. Faster causal attention over large sequences through sparse flash attention. arXiv preprint arXiv:2306.01 160, 2023.
[34] Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong. cosformer: Rethinking softmax in attention. arXiv preprint wrXiv:2202.0879) 2022.
[35] Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-lo-text transformer. arXiv e-prints, 2019.
[36] Sameera Ramasinghe, Thalaiyasingam Ajanthan, Gil Avraham, Yan Zuo, and Alexander Long. Beyond top-k: Structured sparsification for compression in pipeline parallel. In ICLR 2025 Workshop on Modularity for Collaborative, Decentralized, and Continual Deep Learning.
[37] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 3505-3506, 2020.
[38] Xiaozhe Ren, Pingyi Zhou, Xinfan Meng, Xinjing Huang, Yadao Wang, Weichao Wang, Pengfei Li, Xiaoda Zhang, Alexander Podolskiy, Grigory Arshinov, et al. Pangu-() Sigma): Towards trillion parameter language model with sparse heterogeneous computing, arXiv preprint arXiv:2303.1084. 2023.
[39] Max Ryabinin, Tim Dettmers, Michael Diskin, and Alexander Borzunov. Swarm parallelism: Training large models can be surprisingly communication--fficiee. Learning, pages 29416-29440. PMLR, 2023.
International Conference on Machine
[40] Max Ryabinin, Eduard Gorbunov, Vsevolod Plokhotnyuk, and Gennady Pekhimenko. Moshpit sgd: Communication-efficient decentralized training on heterogencous unreliable devices. Advances in Neural Information Processing Systems, 34:18195-18211, 2021.
[41] Max Ryabinin and Anton Gusev, Towards crowdsoureed training of large neural networks using de- centralized cxxturt--ofeeperts. Advances in Neural Information Processing Systems, 33:3659-3672 2020.
[42] Lorenzo Sani, Alex Iacob, Zeyu Cao, Royson Lee, Bill Marino, Yan Gao, Dongqi Cai, Zexi Li, Wanru Zhao, Xinchi Qiu, et al. Photon: Federated IIm pre-training, arXiv preprint 222002241, 2024.
12

---

[43] Shuvam Sanyal, Angela Fan, and Eric Wallace. Inheritune: Leveraging inherent low-rank structure in transformers. arXiv preprint arXiv:4404.03789 2024.
[44] Shaohuai Shi, Qiang Wang, Kaiyong Zhao, Zhenheng Tang, Yuxin Wang. Xiang Huang, and Xiaowen Chu. A distributed synchronous sgd algorithm with global top-k sparsification for low bandwidth networks. In 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS), pages 2238-22222 2247. IEEE, 2019.
[45] Shruti Singh and Shantanu Kumar. Efficient distributed training through gradient compression with sparsification and quantization techniques. arXiv preprint arXiv:2502.076:3 2024.
[46] Hanlin Tang, Ce Zhang, Shaoduo Gan, Tong Zhang, and Liu. Decentralization meets quantization. arXiv preprint 1
803. 00443, 3, 2018.
[47] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Ami Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 13971, 2023.
[48] Thijs Vogels, Sai Praneeth Karimireddy, and Martin Jaggi. Powergossip: Practical low-rank communication compression in decentralized deep learning. arXiv preprint arXiv:2008.0144 2020.
[49] Haozhao Wang, Song Guo, Zhihao Qu, Ruixuan Li, and Ziming Liu. Eroor-compensated sparsification for ciamuunication-efliiiee decentralized training in edge environment. IEEE Transactions on Purullet and Distributed Systems, (3(1):142255 2021.
[50] Hongyi Wang. Saurabh Agarwal, and Dimitris Papailiopoulos. Pufferfish: Communication-eficn models at extra cost. Proceedings Machine Learning and Systems, 3:365-386 2021.
[S1] Jiaxiang Wu, Weidong Huang. Junzhou Huang, and Tong Zhang. Error compensated quantized sgd and its applications to large-scale distributed optimization. In International conference machine learning, pages 5325 5333, PMLR, 2018.
[52] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng. Bowen Yu,, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, et al. Qwen2. technical report. arXiv preprint arXiv:2412.15115 2024.
[53] Binhang Yuan, Yongjun He, Jared Davis, Tianyi Zhang, Tri Dao. Beidi Chen, Percy S Liang, Christopher Re, and Ce Zhang. Decentralized training foundation models heterogeneous environments, Advances in Neural Information Processing Systems, 35:25464 25477, 2022.
[54] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey. Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 13:17283 17297, 2020.
[55] Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Mcmory-efficient Ilm training by gradient low-rank projection. arXiv preprint 2433:22403.33507, 2024.
[56] Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-lik like visual explanations by watching movies and reading books. In The IEF. International Conference Computer Vision (ICCV), December 2015.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification: Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer:
[Yes]
Justification:
Provided in supplementary materials
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeee //nips.com / ssiniiiiccypyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Section and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Experiments too expensive and therefore it is computationally expensive to report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative етror rates).
If error bars reported in tables plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: httpar/:////iie.eeeee/ee http://ss..cc.....ccttccccccceea
Answer: [Yes]
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Implications are discussed Section
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate
17

---

to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
No data or models that are of high rist of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided relcasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwprspipetattteeeeee thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
18

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No human participants were used in the study.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector:
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No study participants were used in the sudy.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMSs if it is an important, original, or non-standard component of the methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality the research, declaration is required.
Answer: [No]
19

---

Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy http:///aaarrpsscc/ooooooorrrr/e for what should or should not be described.
20


Reviews:
[{"review_id": "c0mP4H6wae", "reviewer": "Reviewer_sGpa", "summary": "This paper introduces a compression method enabling efficient context-parallel training of large language models across decentralized devices with limited bandwidth. By dynamically exploiting the low-rank structure of attention activations through learned mixtures of subspaces, the method achieves over 95% compression without sacrificing model convergence or performance. Experiments demonstrate successful training at scale, matching centralized high-bandwidth systems in convergence speed.", "questions": "How does the method proposed relate to PowerGossip (https://arxiv.org/abs/2008.01425) and Photon (https://arxiv.org/abs/2411.02908)?\n\nYou mention an initial warm-up phase is crucial to your method; how sensitive is your approach to variations in this phase (such as reduced length or interrupted warm-up), and have you explored strategies to mitigate dependency on it in dynamic decentralized environments?\n\nCould you clarify how robust your method is when the low-rank assumption of attention activations is violated or weaker than expected? What happens to performance and convergence in such scenarios?", "limitations": "Nothing too strong to push on.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "I really liked how effectively they tackle the major bottleneck in decentralized training, namely communication. Reducing bandwidth usage by over 95% while preserving accuracy is seriously impressive. The method is also flexible, integrating seamlessly into existing transformer architectures, so you can drop their special components after training and just deploy a normal model. Plus, they show practical scaling to billion-parameter models, which is exactly what decentralized setups need.\n\nOn the downside, though, the technique relies heavily on an initial \"warm-up\" phase, which might be tricky in unpredictable real-world setups. Another concern is that the theoretical guarantees lean on assumptions (like low-rank activations) that might not hold perfectly across all tasks or domains. Finally, while compression rates are great, their rotation-based subspace method introduces extra computational complexity, even though they've tried to minimize it.", "was_revised": true, "final_justification": "Below is my original justification part way through the process. After the authors clarified how the additional material discussed during the rebuttal period would appear in the paper, I have become more comfortable with the work. And so have a raised my score.\n----\n\nI have considered the reply from the authors. I think the information it provides provides important additional context and observations for the main paper. At this stage I haven't received word if they would include the rebuttal material into the main paper, and so have asked about it. If they can more clearly describe how the final paper will look, then I might consider increasing my score further. I am already fairly positive. I plan to edit this, if I receive a reply from them.", "rebuttal": "We sincerely thank the reviewer for the positive and encouraging feedback, with remarks such as our work being “seriously impressive,” the method being “flexible,” “integrating seamlessly into existing transformer architectures,” and demonstrating “practical scaling to billion-parameter models.” Below, we address the specific questions raised.\n\n## 1) Initial \"warm-up\" phase, which might be tricky\n\nWe thank the reviewer for the thoughtful question. Below, we note that our warm‑up phase is **agnostic across different hyperparameters** and **can be seamlessly integrated into existing context‑length warm‑up practices.**\n\n- **We use a fixed warm- setting across all settings:**  In our experiments, we used a **fixed warm-up phase of 500 steps with 2k context length across all setups**, regardless of model size (from 800M to 3B parameters), number of layers (8 to 32), or final context length (132K in the main paper and 256K in Fig. 13 of the supplementary). This includes models shown in Fig. 10 (3B), Fig. 12 (2.56B), and others. We also ensured that the warm-up configuration was consistent with matched uncompressed baselines. Based on these results, **we do not observe strong sensitivity of our method to the length of the warm-up phase across varying model scales or configurations.**\n\n- **Context length warm-up is already a standard practice across many frontier models:**\n\n  - We would also like to emphasize that context‑length warm‑up is now standard in many frontier‑model training recipes:\n    - **Llama 3 400B** gradually expands the context window in **six stages**, from **8 K → 128 K** tokens [1].  \n    - **Qwen 2** increases sequence length from **4 K → 256 K** tokens [2].  \n    - **Hugging Face SmolLM 3** steps from **4 K → 32 K → 64 K** tokens [3].  \n    - **Li et al.** demonstrate that sequence‑length warm‑up mitigates gradient variance, allowing GPT‑3 to train with **8 × larger batch sizes** and **40 × higher learning rates** [4].\n\nOur own schedule (training briefly at a shorter context length and computing $\\bar{U}$ before switching to the final length) fits naturally within these prevailing practices. Moreover, our warm‑up is **extremely lightweight** and **less imposing** than above schedules: we use only 500 steps at the shorter length, far fewer than the thousands of iterations reported in the works above, highlighting the practicality of our recipe.\n\nWe would also like to emphasize that modern LLM pre‑training already employs multiple, often intricate warm‑up schedules, covering learning rate, batch size, and context length, whereas our recipe is comparatively simple and far less sensitive to hyperparameter choices.\n\n## 2) How sensitive is your approach to variations in the warm up phase\n\nIn addition to the above-mentioned points, to further address the reviewer’s concern, we conducted an ablation study varying the warm-up duration and evaluated the resulting perplexity on the FineWeb dataset. The results are shown below. As demonstrated, even with a reduced warm-up of 300 steps, the model achieves comparable performance, indicating no significant degradation. In practice, we default to 500 steps to provide a safe and stable baseline. This study further emphasizes the lightweight and robust nature of our warm-up strategy, especially in contrast to the more elaborate scheduling mechanisms commonly employed in modern LLM pre-training.\n\nPlease note that perplexity is computed as the exponential of validation loss, so differences in decimal points are minor and stable, indicating performance is stable after 300 warm-up steps.\n\n| Warm-up Steps | Perplexity |\n|---------------|------------|\n| -             | 26.63      |\n| 100           | 24.44      |\n| 300           | 22.66      |\n| 500           | 22.64      |\n| 1000          | 22.87      |\n| 2000          | 22.64      |\n| 5000          | 22.71      |\n\nWe will include these results in the supplementary. **If the reviewer needs more experiments or clarifications, we would like to kindly ask the reviewer to mention them so we can try our best to provide them during the discussion period.** Thank you for the suggestion.\n\n\n## 3) Relationship to Power Gossip and Photon\n\nThank you for pointing us to these interesting works. We note that both papers operate in a data-parallel (DDP) setting, where each node holds a full model replica and exchanges weights or weight gradients to simulate a larger batch size. Note that this is a comparatively well studied domain, and is orthogonal to the unexplored decentralized context parallel setting that we explore. \n\nPower Gossip, for example, replaces synchronous all-to-all communication with gossip-style information exchange among neighboring replicas arranged in a mesh. Its key insight is that, when each replica trains independently via local SGD, the pairwise weight differences evolve in a low‑rank sub‑space, enabling them to efficiently compress the *weight differences* during gossip. For instance, assume $W_i^t$ is a weight matrix of replica $i$ at time $t$. It's update at time $t+1$ is denoted as,\n\n$$W_i^{t+1}  = W_i^t + \\\\sum M_{ij} \\\\mathcal{C} (W_i^t  - W_j^t ) - \\\\eta G$$\n\nwhere $\\eta$ is the learning rate, $G$ is the local gradient matrix, $\\\\mathcal{C}$ is the low-rank compression, and $M_{ij}$ denotes weighting between the nodes i and j in the connectivity graph. This shows that the weight $(W_i^t  - W_j^t )$ at any particular time $t$ can be approximated with a low-rank matrix. Now, consider the following interesting upperbound.\n\n$$\\text{rank}(W_i^t  - W_j^t ) \\leq \\text{rank}(W_i^t  ) + \\text{rank}(W_j^t )$$\n\nThis means, that if both $W_i^t $ and $W_j^t$ are low rank, their differences are likewise low‑rank: precisely the property Power Gossip exploits! This insight parallels our finding that attention projection matrices collapse to a low‑rank sub‑space during training. Note that however, our sub‑space compression is complementary to Power Gossip‑style techniques: while they compress weight differences in a DDP setting, we compress activations and activation gradients in context‑parallel settings. Together, these methods could further reduce communication in hybrid training setups.\n\nPhoton also targets end‑to‑end LLM training in a DDP setting. Its communication savings stem primarily from infrequent gradient exchanges rather than from any explicit compression scheme. Such skip‑sync approaches are infeasible in context‑parallel pipelines, where activations must be transferred between nodes at every forward and backward pass. Nevertheless, like PowerSGD, these DDP‑style techniques are orthogonal to our method and could be combined with it in hybrid setups.\n\nOur work fills a gap in decentralised distributed training: while communication‑efficient DDP and model‑parallel techniques have been explored, we are, to our knowledge, the **first to focus specifically on communication‑efficient context parallelism.**\n\n## 4) Could you clarify how robust your method is when the low-rank assumption of attention activations is violated or weaker than expected?\n\nThank you for raising this point. We address it in three parts.\n\n**Empirical evidence across diverse frontier models:** We analysed attention activations in several open‑source LLM families (Llama, Qwen, and OLMo) spanning 3 B to 235 B parameters and trained on multilingual corpora (see Figs. 6–8). In every case the singular‑value spectrum is sharply concentrated. This suggests the low‑rank property is not an artefact of a single architecture or task, but a consistent outcome of large‑scale pre‑training. Further, this is a strong empirical proof that our method can produce competitive models.\n\n**Dynamic expressiveness:** Hypothetically, if a future setting required richer attention representations, one possible solution (among many other options) is to add a lightweight router that selects among multiple projection heads with different k values, conditioned on the activation statistics. This would allow the model to adaptively adjust expressiveness, with a trade-off on communication efficiency.  We leave such interesting explorations to future work.\n\n**Orthogonality to expressiveness‑enhancing techniques:** Our approach targets the communication bottleneck; it does not alter the functional form of the attention mechanism. Therefore it can be combined with methods that boost expressiveness, e.g., MoE routers, dynamic sparsity, or low‑rank adapters, should the low‑rank assumption ever become limiting.\n\nIn summary, extensive empirical evidence shows the low‑rank assumption holds across today’s frontier models, and even if future tasks weaken this property, our scheme offers straightforward levers (larger k or adaptive routing) to maintain accuracy while still providing substantial bandwidth savings.\n\n[1] - The Llama 3 Herd of Models\n\n[2] - Qwen2.5-1M: Deploy Your Own Qwen with Context Length up to 1M Tokens\n\n[3] - SmolLM3: smol, multilingual, long-context reasoner\n\n[4] - The Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Models"}, {"review_id": "vyNGB8WV4q", "reviewer": "Reviewer_w6eq", "summary": "The paper attempts to solve the efficiency problem of long-context pretraining under low-bandwidth connections situation. Concretely it studies compression to reduce the communication problem, incurred by ring-style sequence parallelism. It achieves scaling billion-parameter models with more than 100k on slow networks.", "questions": "(1) Is it possible that the author provides experiment result on larger models?  The convergence can be different for larger model (the paper pretrains a 800M model, where the reviewer suggest the scale should be > 7B to draw significant applicable conclusion), especially when the paper method is based on compression.\n(2) Is it possible to show effectiveness on larger cluster? The current experiments are done on 8xA100, where typically models are pretrained at the scale of > 100 H100 GPUs.\n(3) How does the method applies to Ulysses style sequence parallelism, which typically induces less communication?\n(4) (Minor) The figure 3 and figure 4 ratio are different. Could the author align them for a more compact view of the paper?", "limitations": "Yes.", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n-  The problem on ring-style sequence parallelism is significant.\n- The paper is well written and easy to follow.\n\nWeakness:\n- The convergence is done on a small model, compared to the models mentioned in the introduction, e.g. Llama4.  The convergence can behave differently.\n- It does not studies Ulysses style sequence parallelism, where the communication is less a problem, compared to ring-style.", "was_revised": true, "final_justification": "The paper is generally good, but as the original review points out, larger scale experiments are needed to validate the method.", "rebuttal": "We thank the reviewer for identifying the significance of our work and for the constructive feedback. Please find our answers below.\n\n## 1)  Is it possible to show effectiveness on larger cluster with a bigger model than 800M? The current experiments are done on 8xA100, where typically models are pretrained at the scale of > 100 H100 GPUs. \n\nWe appreciate the reviewer’s concern. As noted in the paper (Figures 3, 10 (supp) and 12(supp)), we have already evaluated our method on **2.5B** and **3B** parameter models using 32 A100 GPUs. In both cases, our method maintains convergence while achieving substantial communication savings, **yielding approximately 20× improvement in tokens per second (TPS)** compared to the uncompressed baselines.\n\nWhile we currently lack access to large H100 GPU clusters to train larger models at massive scale, we see no fundamental limitation in the method that would prevent it from scaling. Our experiments already operate at the billion-parameter level, and the design of the algorithm, particularly its low overhead and compatibility with standard training procedures, suggests it would extend naturally to larger models and clusters.\n\nWe hope our results offer a strong foundation for future work on communication-efficient context parallelism at larger scales, and we welcome further exploration of the method in such settings.\n\n## 2) How does the method applies to Ulysses style sequence parallelism?\n\nWe thank the reviewer for pointing out Ulysses. We now clarify why Ulysses is not applicable in decentralized settings. \n \n### Why Ulysses is **not** applicable in decentralized settings?\n\n**Ulysses communication savings *only* stems from modern GPU clusters with intra-node NVSwitch interconnect and inter-node fat tree IB topology**. Specifically, Ulysses's communication savings are entirely dependent  on intra-node NVSwitch crossbars and an inter-node 1 : 1 fat-tree InfiniBand fabric that ensures the communication volume transmitted per link for an all-to-all for aggregate message of size M over P GPUs is M/P.  In decentralised setting, such topologies do not exist and peers communicate over heterogeneous, asymmetric residential links that traverse multiple ISP hops and NATs. Further, in Ulysses, every Transformer block performs a synchronous NCCL _all-to-all_ of QKV and context vectors. In decentralized settings, this is also not possible.\n\nWe would also like to highlight that our method is **not restricted to ring-style collectives and is fully compatible with all-to-all gather and other communication patterns**. Since it only compresses the exchanged K and V vectors, the approach is agnostic to the underlying communication protocol and can be seamlessly integrated with any collective communication strategy.\n\n### Can our method improve *centralized cluster* communication efficiency? \n\nAs we explained above, **Ulysses cannot be applied in a decentralized environment** due to its structural reliance on synchrony and fabric-level assumptions. Because these assumptions are architectural, Ulysses cannot be ported by pure algorithmic tweaks. \n\nInterestingly, we emphasize that the **reverse** is possible. That is, if deployed in a centralized cluster setting, it is possible that our method can be **overlaid on Ulysses** to reduce traffic significantly, making it attractive for centralized, large-scale, long-context models. Below, we compare the theoretical communication cost per Transformer block in this case. For a context length `N` and an embedding dimension `h`,\n\n| Scheme | Data moved per GPU (forward **or** backward) | If we **overlay our 95 % QKV compression** |\n|--------|--------------------------------------------------|-------------------------------------------|\n| **Ulysses**| `3 · N · h` bytes  (Q, K, V) + `1 · N · h` bytes  (context)  → **`4 N h`** | **`0.05 × 3 N h + 1 N h ≈ 1.15 N h`** |\n| **Our Context-Parallel (CP)** | **`≈ 0.1 N h`** bytes  | unchanged |\n\n\n**Summary** Even in a *central* cluster our compression reduces Ulysses traffic by **≈ 3.5 ×**, i.e.,  (`4 N h → 1.15 N h`).  In contrast, Our CP is *already* an order-of-magnitude lower (`0.1 N h`). Below, we demonstrate a toy scenario with some practical numbers. For example considering a config, `N = 4 096`, `h = 4 096` we get, \n\n| Variant | Bytes / GPU / block | \n|---------|---------------------|\n|Ulysses| 100 MB |\n| Ulysses + ours | 26 MB | \n| **Ours** | 2.5 MB | \n\nTherefore, although Ulysses is an elegant solution for data-centre long-context training, but its dependence on  intra-node NVSwitch interconnects and inter-node fat tree IB topology-based fabrics makes it unusable for the decentralised, bandwidth-limited setting our work targets.  On the contrary, **our  compression shrinks Ulysses traffic significantly on *central* clusters and could indeed be combined with Ulysses there**.  Consequently, we do not view Ulysses as a realistic option for decentralised training, whereas our context-parallel method is expressly designed for, and validated in, such resource constrained setting.\n\n## 3) (Minor) The figure 3 and figure 4 ratio are different. Could the author align them for a more compact view of the paper?\n\nWe thank the reviewer for pointing this out and we will correct this in the revised version."}, {"review_id": "lgjs4o0jkB", "reviewer": "Reviewer_2JUi", "summary": "The paper addresses a critical and previously unexplored bottleneck in training large language models (LLMs): the immense communication overhead of Context Parallelism (CP) in decentralized, low-bandwidth environments. Standard CP requires each device to broadcast its local key (K) and value (V) activation matrices to all other devices, which is prohibitively expensive over commodity internet connections. The authors' core contribution is a novel and highly effective compression method for these activations. The method is motivated by the empirical observation that the K and V activation matrices in large transformers have an intrinsically low-rank structure. The proposed solution leverages this by: Projecting the high-dimensional K and V activations onto a low-dimensional subspace, drastically reducing the amount of data to be communicated. Dynamically adapting this subspace for each input chunk via a lightweight, learnable rotation. This provides flexibility without sacrificing compression. Using an efficient reparameterization of these rotations based on Lie groups, which avoids costly matrix operations (like SVD/QR) and makes the method computationally cheap. Introducing a mechanism to \"unplug\" the auxiliary compression components after training, resulting in a standard transformer architecture at inference time with no overhead.", "questions": "1. Could you provide more details on the warm-up stage? How was the duration (<500 iterations) and reduced context length determined? How sensitive is the final performance to these choices?\n2. The success with a single rotation parameter (k=1) is fascinating. Do you have a hypothesis for why this highly constrained adaptation works so well? Have you identified any failure cases or do you foresee scenarios (e.g., highly multi-modal data) where a larger k would be necessary?\n3. Figure 4 shows that late unplugging is effective. Is there a clear, automatable heuristic for determining the optimal point in training to remove the projection components? Does the subsequent \"polishing\" fine-tuning pass require a specific learning rate schedule to avoid disrupting the learned weights?\n4. The paper uses 98% and 95% compression for K and V respectively. Have you explored the trade-off curve between the compression rate (i.e., the rank r) and the final model perplexity? It would be interesting to see the point at which performance begins to degrade meaningfully.\n5. The proposed method compresses activations. Could it be productively combined with other techniques, such as weight quantization or gradient compression (for the infrequent weight syncs), to achieve even greater efficiency in a fully decentralized setting?", "limitations": "see questions.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The paper tackles a highly relevant and timely problem. As context windows in LLMs expand beyond 100K tokens, CP is becoming essential. However, its application has been confined to expensive, centralized data centers. This paper is the first to propose a viable solution for CP in decentralized settings, potentially democratizing the training of long-context LLMs.\n2. The method is built upon a simple yet powerful insight: the low-rank nature of attention activations. This is convincingly demonstrated in Figure 1 with an analysis of LLaMA-70B. This strong empirical grounding makes the subsequent methodological choices feel well-justified and intuitive.\n3. The technical approach is very clever. The authors build a logical progression from a simple low-rank factorization to a more robust solution. The reparameterization of the projection subspace U via a fixed basis Ū and a learnable rotation R(θ) is an elegant way to maintain expressive power while making optimization tractable (avoiding expensive Riemannian gradient steps). The use of a low-dimensional Lie subgroup to parameterize R(θ) is a principled way to control the communication-flexibility trade-off.\n4. The results presented are outstanding and strongly support the paper's claims. Performance Parity (Figure 2): Matching the wall-clock performance of a 100Gbps system on a 300Mbps link is a remarkable achievement and the most compelling result in the paper. Scalability (Figure 3): The method is shown to scale effectively to a 3B parameter model and integrates seamlessly with other parallelism techniques like Pipeline Parallelism. Ablations and Baselines (Table 2, Figure 5): The authors conduct thorough ablation studies that validate their design choices (e.g., the importance of the warm-start, the adequacy of the 2nd-order approximation). The comparisons against sensible baselines like quantization and sparsification clearly demonstrate the superiority of the proposed approach.\n\nWeaknesses\n1. The method relies on a \"short, uncompressed warm-up phase\" to establish the initial subspace basis Ū. This step is crucial, as the ablations show that performance degrades significantly without it (\"No warm start\" in Table 2). However, the details are sparse. The sensitivity to the duration and context length of this phase is not explored, making it seem somewhat heuristic. A more detailed analysis or guideline on setting up this phase would strengthen the paper.\n2. The paper finds that a rotation parameterized by a single scalar (k=1) is sufficient. This is a surprising and powerful result for communication efficiency, but the paper offers little intuition as to why such a constrained, planar rotation is sufficient to adapt the subspace effectively across diverse data chunks. Is this a fundamental property, or could it be a limitation on more complex tasks or architectures?\n3. The experiments are large-scale and computationally expensive, so this is understandable. However, the results are based on single training runs. As acknowledged in the paper's checklist, there are no error bars or confidence intervals to assess the variance of the results (e.g., due to different random seeds for initialization).", "was_revised": true, "final_justification": "questions answered.", "rebuttal": "We appreciate the reviewer’s constructive feedback and their encouraging remarks that our work tackles “a highly relevant and timely problem,” is “strongly grounded empirically,” employs a “clever technical approach,” and delivers “outstanding results.” Below, we address each of the reviewer’s questions in turn.\n\n## 1) Could you provide more details on the warm-up stage? How sensitive is the final performance to these choices?\n\nThank you for the thoughtful question. We clarify that our warm-up strategy is **hyperparameter-agnostic** and **aligns with standard context-length warm-up practices** used in modern LLM training.\n\n  - **Fixed warm-up configuration across all settings:** In all our experiments, we applied a **fixed warm-up phase of 500 steps** with a 2K context length, regardless of model size (ranging from 800M to 3B parameters), number of layers (8 to 32), or final context length (132K in the main paper; 256K in Fig. 13 of the supplementary). This includes all major results reported (e.g., Fig. 10 for 3B, Fig. 12 for 2.56B). Importantly, this warm-up setup is also used in matched uncompressed baselines to ensure a fair comparison. Across these varied settings, we did not observe significant sensitivity to the warm-up duration. The fixed schedule worked consistently well and demonstrates the stability of warm-up phase across scales.\n\n  - **Observed insensitivity to warm-up duration:**- We observed that the model maintains comparable performance even with a shorter warm-up of 300 steps. In practice, we adopt 500 steps as a conservative default.  In addition, we conducted an ablation study varying the warm-up duration to further address the reviewer’s concern. We evaluated the PPL on FW, with results presented below. This ablation highlights the robust nature of our warm-up, particularly when compared to the more complex scheduling strategies (learning rate, batch size, context length) often used in large-scale LLM pre-training. **Please note that perplexity is calculated as the exponential of the validation loss, so variations at the decimal level correspond to very small differences** (mostly the third decimal point in the validation loss).This indicates that performance remains stable after 300 warm-up steps.\n\n| Warm-up Steps | Perplexity |\n|-|-|\n| - | 26.63      |\n| 100 | 24.44      |\n| 300| 22.66      |\n| 500 | 22.64      |\n| 1000 | 22.87      |\n| 2000 | 22.64      |\n| 5000 | 22.71      |\n\n  - **Context-length warm-up is a well-established practice in frontier model training:** Note that context-length warm-up has become a standard component of many large-scale LLM training recipes, and our approach can be seamlessly fit into them. \n    - **Llama 3 400B** progressively increases context length in six stages, from 8K to 128K tokens [1].\n    - **Qwen 2** gradually grows the sequence length from 4K to 256K tokens [2].\n    - **Hugging Face SmolLM 3** follows a multi-stage progression from 4K to 32K to 64K tokens [3].\n    - **Li et al.** show that sequence-length warm-up helps reduce gradient variance, enabling GPT-3 to train with 8× larger batch sizes and 40× higher learning rates [4].\n\nOur own strategy, training briefly at a shorter context length and computing $\\bar{U}$ before transitioning to the full sequence length, aligns well with these practices. Notably, our warm-up phase is much lighter-weight: we use only 500 steps at the reduced length, significantly fewer than the thousands of iterations employed in the examples above. This highlights the practicality and efficiency of our approach.\n\nMoreover, we would like to point out that modern LLM pre-training already incorporates multiple warm-up mechanisms, including those for learning rate, batch size, and context length, each of which has much stronger effects on the final convergence. In comparison, our method is both simpler and less sensitive to hyperparameter tuning. \n\n**If the reviewer would like to see additional experiments to further clarify this point, we would kindly invite them to indicate so in their response. We will make every effort to provide the requested results within the discussion period.**\n\n## 2) The success with (k=1) is fascinating. Do you have a hypothesis for why? \n\nWe agree with the reviewer that the effectiveness of a single rotation parameter (k = 1) is fascinating. While a full theoretical explanation remains an open question, we believe this behavior aligns with a growing body of evidence that the loss landscapes of large models contain wide, flat, and highly connected regions of low loss, resulting in equally good multiple local minima.\n\nIn particular, it is well-established that overparameterized networks often admit many near-optimal local minima that are linearly connected [7,8]. In this regime, optimisation landscape becomes a high‑dimensional web of connected, flat valleys containing multiple solutions with near‑identical performance, and first‑order methods fall into this region. This implies that even low-rank updates (such as a rank-1 rotation) can be sufficient to move within a flat valley of good solutions without increasing loss. Relatedly, recent analyses of permutation-aligned weight spaces [5] show that large models fine-tuned independently remain linearly mode-connected after weight matching. Practical techniques such as SLERP, MergeKit [6], and model soups further exploit this connectivity, achieving successful model merging via low-rank updates or averaging.\n\nFrom this view, the success of k = 1 can be interpreted as navigating a dominant direction in this connected low-loss region. Since many pre-trained representations are already highly linearly separable, a single rotation may suffice to realign critical features.\n\n## 3) Have you identified any failure cases or do you foresee scenarios where a larger k would be necessary?\n\nWe appreciate the reviewer's concern. We did not observe any failure cases in our experiments. However, if a larger k becomes necessary, a router could select among several projection heads with different *k* based on activation statistics, trading bandwidth for expressiveness, an avenue for future work. Note that because our method addresses only the comm bottleneck, it can be combined with expressiveness-oriented methods (MoE routers, dynamic sparsity, low-rank adapters). Should *k = 1* prove limiting, such hybrids can restore capacity while retaining communication savings.\n\n## 4) Is there a automatable heuristic for determining the optimal point in training to remove the projection components?\n\nThank you for the question. In our experiments, we found that removing the projection components around the last 30%-5% range of training consistently results in stable performance across diverse settings. Based on this observation, we recommend using the last 30%-5% range of training as a simple and effective heuristic, and we will make this recommendation explicit in the revised version of the paper.\n\nFurther, as in Fig. 4, even when the projection components are removed earlier in training, the model is able to recover quickly, indicating a degree of robustness to the exact transition point. Nonetheless, the last 30% heuristic offers a practical, safer, and an automatable guideline. We present the below ablation on FW to further illustrate this (total steps 10k). Variations correspond to the third decimal point in the validation loss in most cases.\n\n| Unplugging Step | PPL|\n|-|-|\n| Not unplugged | 22.64 |\n| 5k| 22.69|\n| 7k| 22.64|\n| 8k| 22.71|\n| 9k| 22.71|\n| 9.5k| 22.77|\n|9.9k | 23.12|\n\n## 5) Have you explored the trade-off curve between the compression rate  and the final PPL?\n\nThank you for the thoughtful question. We have already provided an ablation in Table 2 showing the drop in performance when V-compression is increased to 98% and K-compression to 99%. Further, we present a more granular ablation on the FW to show the trade-off between compression rate and final model perplexity.\n\nThe baseline perplexity w/o compression is 22.65.\n\n| K| V| PPL|\n|-|-|-|\n| 98%           | 95%            | 22.64       |\n| 98%           | 96%            | 22.69       |\n| 98%           | 97%            | 22.69       |\n| 98%           | 98%            | 22.74       |\n| 99%           | 95%            | 24.68       |\n| 99%           | 96%            | 24.71       |\n| 99%           | 97%            | 24.77       |\n| 99%           | 98%            | 25.23       |\n| 99%           | 99%            | 26.44       |\n\nWe will include above results in the revised version.\n\n## 6) Could the method be combined with weight quantization or gradient compression?\n\nWe absolutely believe this is indeed possible! Our method is orthogonal to and fully compatible with other DDP-style compression techniques. Since we targets activation and activation-gradient compression, it can be directly combined with weight quantization methods that reduce model size and memory footprint. These methods operate on different components of the training pipeline and can complement each other effectively. Further, our technique applies to context-parallel setups where activations are transmitted across layers. In contrast, gradient compression techniques like Top‑K, PowerSGD, or quantized all-reduce are designed for DDP settings, where gradients are exchanged between full-model replicas. We believe  that our method can be layered on top of DDP-style gradient compression, yielding compounded communication savings.\n\n[1]-The Llama 3 Herd of Models\n\n[2]-Qwen2.5-1M: Deploy Your Own Qwen with Context Length up to 1M Tokens\n\n[3]-SmolLM3: smol, multilingual, long-context reasoner\n\n[4]-The Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Models\n\n[5]- Analysis of Linear Mode Connectivity via Permutation-Based Weight Matching\n\n[6]- Activation-Informed Merging of Large Language Models\n\n[7]-Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs\n\n[8]-Essentially No Barriers in Neural Network Energy Landscape"}, {"review_id": "0aZSt8Jvs8", "reviewer": "Reviewer_WpGw", "summary": "This paper introduces a novel method for bandwidth-efficient context-parallel (CP) training in decentralized environments. The authors propose dynamically compressing key and value activations using learned low-rank projections into mixtures of subspaces. By jointly optimizing both the subspaces and attention weights on a product manifold, and using per-chunk rotations with compact parameterization, the method achieves over 95% communication compression. The compressed activations can be transmitted efficiently even over low-bandwidth (e.g., 300Mbps) links, and decompressed without degrading model quality. Extensive experiments on long-context training (132K tokens) across datasets show that the proposed method achieves convergence comparable to centralized setups, with 20× speedup over uncompressed CP on commodity links.", "questions": "Please the the weakness part.", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 1, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. The use of dynamic mixtures of low-rank subspaces and efficient rotation-based reparameterization is both novel and theoretically grounded.\n2.  The paper includes careful breakdowns of design choices, including the role of warm starts, second-order approximations, and reparameterization.\n\nWeakness:\n1. The motivation is not persuasive. Why do we need decentralized long-context training over low-bandwidth connections? Why a long context is distributed across \"internet-grade\" linked devices if it has inherent long-range dependencies?\n2. Context parallel introduces frequent communication operations. Pipeline parallel might seem to be a better choice for low-bandwidth training with a much smaller number of communication volumes and frequencies.\n3. The surprising effectiveness of extremely low-dimensional rotations is acknowledged but not fully understood. The paper lacks comparison with other attention sparsification methods.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and intriguing questions. Please find our answers below.\n\n## The motivation is not persuasive. Why do we need decentralized long-context training over low-bandwidth connections? Why would a long context be distributed across \"internet-grade\" linked devices if it has inherent long-range dependencies?\n\nThank you for raising this important question. We would like to clarify the motivation behind our setting and why decentralized long-context training is both **necessary** and **increasingly relevant**.\n\n---\n\n### 1. Centralized large-scale training is prohibitively expensive\n\nModern LLMs are trained on massive HPC clusters with high-bandwidth interconnects (e.g., InfiniBand), which are extremely costly to build and operate. These clusters are typically accessible only to a few well-funded organizations. As a result, most researchers cannot afford to run large-scale experiments needed to rigorously evaluate new ideas. This creates a significant **barrier to entry** and limits scientific progress.\n\n---\n\n### 2. Volunteer computing offers a scalable and democratized alternative\n\nA promising alternative is **scaling out** via **volunteer computing**, where thousands of regular PCs contributed by volunteers provide compute. This paradigm has already been successfully applied in domains like language modeling [1], biology [2], high-energy physics [3], and astronomy. The aggregate FLOP capacity of such distributed systems is comparable to that of the largest supercomputers.\n\nHowever, these machines are connected via **internet-grade networks**, which are significantly slower (especially in terms of latency) and more failure-prone than server-grade infrastructure. As a result, existing distributed training methods that require **continuous high-throughput data exchange** are generally **incompatible** with this setup.\n\n---\n\n### 3. Long-context training is essential for competitiveness and scientific impact\n\nIf crowdsourced models are to be **competitive with centralized LLMs**, they must be able to model **long-range dependencies**. Long-context capability is crucial for richer language modeling and better long-form reasoning, as well as other scientific domains.\n\n---\n\n### 4. Our work fills a critical gap\n\nPrevious research has explored communication-efficient **data-parallel** [4, 5] and **model-parallel** [1,6, 7] strategies. However, **no prior work has enabled long-context training in bandwidth-constrained, decentralized environments**.\n\nOur method is the first to support **communication-efficient context parallelism**, making it feasible to train large, long-context models across **volunteer devices connected via the Internet**. We believe this opens the door to more inclusive LLM research and broader scientific applications beyond centralized infrastructure.\n\n---\n\n## Context parallel introduces frequent communication operations. Pipeline parallel might seem to be a better choice for low-bandwidth training with a much smaller number of communication volumes and frequencies.\n\nWe would like to clarify that pipeline parallel allows splitting transformer blocks into devices, thereby enabling training large parameter scale models over a set of GPUs. However, it would not allow models to be trained with long context. For instance, the O(BN^2) memory complexity of the attention alone would consume 4154GB (include calculation) for a single transformer layer, for a model trained with pipeline parallel, which surpasses the most high-end single GPU capacity available even in most advanced data centers. This is why context parallel approaches have been proposed even in centralized settings. This need is more dire in decentralized volunteer settings since the lack of avaiability of high end GPUs.\n\n## Context parallel introduces frequent communication operations. Pipeline parallel might seem to be a better choice for low-bandwidth training with a much smaller number of communication volumes and frequencies.\n\nWe appreciate the reviewer’s point and would like to clarify the distinction between **pipeline parallelism** and **context parallelism** in this setting.\n\nPipeline parallelism enables partitioning transformer blocks across devices, which is useful for **scaling model size** across limited GPU memory. However, **it does not address the challenge of long-context training**. Specifically, attention mechanisms incur **quadratic memory growth** with respect to the context length, $\\mathcal{O}(B N^2)$, where $B$ is the batch size and $N$ is the sequence length.\n\nAs a concrete example, consider a single attention layer in a model with:\n- Batch size $B = 32$\n- Sequence length $N = 132 \\text{K}$\n\nThe memory requirement just for the attention score matrix would be:\n$$\nB \\times N^2 \\times \\text{dtype size} = 8 \\times (132{,}000)^2 \\times 4 \\text{ bytes} \\approx \\mathbf{4.15\\text{TB}}\n$$\n\nThis **far exceeds the memory of even the largest available GPUs**, and such memory pressure applies to *every stage* in pipeline parallelism, as each stage needs to hold the full context during its local computation. Hence, pipeline parallelism alone becomes infeasible for training long-context models.\n\nThis is exactly why **context-parallel approaches** have been proposed and adopted, even in centralized settings [8, 9]. The challenge is even **more pressing in decentralized environments**, where high-end GPUs are scarce and memory constraints are tighter. Our work makes context parallelism feasible under **internet-grade bandwidth**, allowing long-context models to be trained over distributed, volunteer devices.\n\n## The surprising effectiveness of extremely low-dimensional rotations is acknowledged but not fully understood.\n\nAlthough a complete theoretical account is still an open problem, our observations align with a growing body of work indicating that large‑scale models possess wide, flat, and highly connected low‑loss regions in their optimization landscapes. Over‑parameterized networks often admit many near‑optimal local minima that are linearly connected [12, 13]. In this regime, the landscape resembles a web of flat valleys containing multiple solutions of nearly identical training, and frequently test, performance, so first‑order optimizers naturally converge within this region.\n\nWithin such flat valleys, even low‑rank perturbations, for example, a rank‑1 rotation, can move the model to another point of equal loss. Recent analyses of permutation‑aligned weight spaces [10] show that independently fine‑tuned billion‑parameter models remain linearly mode‑connected after weight matching. Practical tools like SLERP, MergeKit [11], and model soups exploit this connectivity, merging models via low‑rank updates or simple averaging with minimal loss increase.\n\nViewed through this lens, the effectiveness of setting  k=1 reflects traversal along a dominant direction inside the connected low‑loss region. Because many pre‑trained representations are already highly linearly separable, a single rotation can be sufficient to realign the features most critical for the downstream task.\n\nWe will add a note of the above in supplementary. Thank you.\n\n## The paper lacks comparison with other attention sparsification methods.\n\nWe already compare with BigBird (see Fig. 5) which is performing sparse attention. If the reviewer can provide more citations, we will try our best to provide comparisons during the discussion period. \n\n[1] - SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient\n\n[2] - Folding@home and genome@home: Using distributed computing to tackle previously intractable problems in computational biology\n\n[3] - Atlas@ home: harnessing volunteer computing for hep \n\n[4] - DiLoCo: Distributed Low-Communication Training of Language Models\n\n[5] - Photon: Federated LLM Pre-Training\n\n[6] - Protocol Models: Scaling Decentralized Training with Communication-Efficient Model Parallelism\n\n[7] - Beyond Top-K: Structured Sparsification for Compression in Pipeline Parallel\n\n[8] - Ring Attention with Blockwise Transformers for Near-Infinite Context\n\n[9] - DeepSpeed-Ulysses\n\n[10] -  Analysis of Linear Mode Connectivity via Permutation-Based Weight Matching\n\n[11] - Activation-Informed Merging of Large Language Models\n\n[12] - Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs\n\n[13] - Essentially No Barriers in Neural Network Energy Landscape"}, {"review_id": "U38h4RzEeF", "reviewer": "Reviewer_Pnue", "summary": "This paper addresses the bandwidth bottleneck of context-parallel (CP) training for very-long-context transformers. CP divides a single long sequence across GPUs; every layer therefore requires an all-gather of full key/value (K,V) activations, which is trivial on datacentre interconnects but prohibitive on commodity networks. Similar to previous work, the authors show empirically that K,V lie on an extremely low-rank manifold (0.1-0.5% stable rank). They propose Mixtures-of-Subspaces (MoS), where each attention weight is factorised so that its outputs are constrained to an r-dimensional subspace. \nA short warm-up run provides an initial orthonormal basis for the low rank subspace $\\bar U$, and thereafter the model learns only a rotation $R(\\theta)$ of that basis together with a low-rank matrix $B$. This re-parameterisation keeps the columns of $U=R(\\theta)\\bar U $ orthonormal without per-step QR/SVD. Projecting $K,V$ onto this subspace reduces communication from $O(nd)$ to $O(nr)$.\n\nOn eight A100 GPUs linked by 300 Mbs Ethernet, MoS reduces CP traffic by ≈20x and matches the wall-clock convergence of an uncompressed baseline on 100 Gb s InfiniBand using an 800 M-parameter, 132 K-token model. Because weight-decay shrinks directions outside the learned subspace, the projection and rotation heads can be “unplugged” after training. Hereby, the final model is a standard transformer.", "questions": "1. Warm-start sensitivity:  How many uncompressed steps are needed to build \\bar U? Does the answer change with dataset choice?\n2. Unplugging: How sensitive is the final PPL to the number of fine-tune steps and weight-decay value used when removing the projector? I see Figure 4 shows an example of this, but how sensitive is it between the late and early “unplugging” steps in this figure?\n3. Training-loss vs validation gap: In Figs 2–3 the “cen-CP 100 Gbps” curve appears to reach lower training loss than MoS, yet Table 1 reports better validation perplexity for MoS. Can you comment on this discrepancy? A zoom-in plot (similar to App.\\ Fig 14) in the main paper would help.\n4. Variance and robustness: All main curves and ablations are single runs. Could you provide some error bars for some of the smaller ablation experiments?\n5. Relation to DeepSeek MLA: DeepSeek’s Mixed-Linear Attention (MLA) also projects K and V to a fixed low dimension, but as I understand it, it keeps the keys/values low-rank all the way through the attention kernel, giving a lasting memory/compute win, whereas MoS re-expands the compressed blocks after the all-gather and therefore targets communication rather than peak RAM.\nCould you clarify this distinction and comment on whether there is any practical benefit (or incompatibility) in combining MoS with MLA", "limitations": "Yes, Section 7 discusses theory gaps and the unexplained success of very low-dimensional rotations.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Quality**\nSolid emprical results: Three data sets, 132 K-token context, and a ~20 × wall-clock speed-up on commodity 300 Mbs links. Interesting ablations: Layer-wise rank study, rotation-size sweep, comparisons to 4-bit and Top-10 % sparsity. Theory: Proof that the re-parameterised optimisation retains convergence guarantees.\nNo variance/error bars. Table 2 and Fig. 5 would be more convincing with ± 1 σ over at least three seeds. (I understand this is harder (costly) for the full runs in Table 1.)\n\n**Clarity**\nClear high-level motivation; algorithms and all hyper-parameters are spelled out; code is included in supplementary.\nFigures 2 and 3 are hard to read: the overlapping “cen-CP 100 Gbps” vs “dec-CP 300 Mbps (comp)” curves look identical but the axes are tiny. A zoom-in would be useful in the main paper. More detailed on explored hparams could also make it easier to reproduce experiments.\n\n**Significance**\nFirst method that makes Context paralism practical on commodity networks, potentially opening > 100 K-token training to a far wider pool of researchers. Orthogonal to existing attention kernels since it does not require architectural changes at inference.\nImpact hinges on the low-rank property holding for other modalities or tasks; scalability beyond the 800 M-parameter (and 3B) range is not demonstrated.\n\n**Originality**\nThe paper is the first to use low-rank structure specifically to minimise all-gather traffic in context-parallel training. This communication-first application of low-rank projections plus the “unplug-after-training” trick that removes the projector at inference is, to my knowledge, new and distinguishes the work from prior memory- or weight-centric compression schemes.", "was_revised": true, "final_justification": "The authors provided a clear rebuttal that addressed my concerns, and my questions are now resolved. The authors showed insensitivity to the warm-start duration, provided a simple, relatively robust “unplugging” heuristic.\nI find the results impressive and the method appealing, and I see it as a practical approach for bandwidth-limited context-parallel training. I’ll keep my score at 5 (accept), noting the setting is fairly specific to GPUs without high-speed interconnects.", "rebuttal": "We appreciate the reviewer’s positive remarks, describing our work as addressing “a highly relevant and timely problem,” providing “solid empirical results,” delivering “a clever technical approach,” and representing “the first practical method for context parallelism.” Below, we respond to each of the reviewer’s specific concerns in turn.\n\n## 1) Warm-start sensitivity: How many uncompressed steps are needed to build \\bar U? Does the answer change with dataset choice?\n\nWe thank the reviewer for the thoughtful question. Below, we clarify that our warm-up strategy is **hyper-parameter agnostic** and fits naturally into the **context-length warm-up practices** widely adopted in modern LLM training.\n\n---\n\n### • Fixed warm-up configuration across all settings  \nWe employ the **same warm-up phase (500 steps with a 2 K context length) in every experiment across all the datasets**, regardless of  \n\n* **Model size:** 800 M → 3 B parameters  \n* **Depth:** 8 → 32 layers \n* **Target context:** 132 K (main paper) and 256 K (Fig. 13, supp.)  \n\nThis covers all major results (e.g., Fig. 3, 10 for 3 B, Fig. 12 for 2.5 B). We apply the identical warm-up to matched uncompressed baselines to ensure a fair comparison. Across these diverse setups we saw **no meaningful sensitivity** to the warm-up duration; the fixed schedule worked consistently well.\n\n---\n\n### • Observed insensitivity to warm-up length  \nTo further address the reviewer’s concern, we varied the warm-up duration and measured perplexity on **FineWeb**. We observed that the model’s performance remains stable even at **300 warm-up steps**, however,  we retain 500 steps as a conservative default. Note that Perplexity is computed as the exponential of validation loss; hence decimal-level changes are minor  (mostly the third decimal point in the loss).\n\n| Warm-up steps | Perplexity |\n|--------------:|-----------:|\n| –             | 26.63 |\n| 100           | 24.44 |\n| 300       | 22.66 |\n| 500 (default) | 22.64 |\n| 1 000         | 22.87 |\n| 2 000         | 22.64 |\n| 5 000         | 22.71 |\n\nThese results confirm the **lightweight and robust** nature of our warm-up, especially relative to the more elaborate schedules (learning-rate, batch-size, context-length) common in large-scale LLM training.\n\n---\n\n### • Context-length warm-up is standard in frontier models  \nOur approach aligns and fits naturally into the established practice of context length warm-up in state-of-the-art models:\n\n* **Llama-3 400B:** six-stage increase from 8 K → 128 K tokens&nbsp;[1]  \n* **Qwen-2:** gradual growth from 4 K → 256 K tokens&nbsp;[2]  \n* **Hugging Face SmolLM-3:** 4 K → 32 K → 64 K tokens&nbsp;[3]  \n* **Li et al. (2022):** sequence-length warm-up lowers gradient variance, enabling GPT-3 to train with **8 ×** larger batches and **40 ×** higher learning rates&nbsp;[4]\n\nOur strategy of brief training at a shorter context, computing $\\bar{U}$, then switching to full length, follows the same principle yet is **far lighter**: only **500 steps** at reduced length vs. thousands of iterations in the examples above.\n\n---\n\nModern LLM pre-training already combines warm-ups for **learning rate**, **batch size**, and **context length**, each with a far bigger impact on convergence. Compared with these, our method is **simpler** and **less sensitive** to tuning, while maintaining strong empirical performance.\n\n## 2) Unplugging: How sensitive is the final PPL to the number of fine-tune steps and weight-decay value used when removing the projector?\n\nThank you for the question. In our experiments, **removing the projection components during the last 30 % – 5 % of training consistently yields stable performance** across model sizes and datasets. Based on this observation, we recommend unplugging in that interval as a simple, effective heuristic. Notably, we see **no meaningful variance in perplexity** within this range, highlighting the robustness of the method.\n\nBelow is an ablation on **FineWeb** (10 k total iterations). Perplexity is reported as $\\exp(\\text{validation loss})$, hence, decimal-level changes reflect very small differences (mostly the third decimal point in the loss). Even with an earlier removal (e.g., 5 k steps), the model recovers surprisingly quickly (as we mention in the paper), so the unplugging point is *not* a dominant factor in final performance.\n\n| Unplugging Step | Perplexity |\n|-----------------|-----------:|\n| **Not unplugged** | 22.64 |\n| 5 k             | 22.69 |\n| 7 k             | 22.64 |\n| 8 k             | 22.71 |\n| 9 k             | 22.71 |\n| 9.5 k           | 22.77 |\n|9.9 k  | 23.12\n\nThese results confirm that unplugging anywhere in the final 30 - 5% of training maintains essentially a similar perplexity, making the procedure both **robust and easy to automate**. We use the same 0.1 weight decay throughout the training process.\n\n\n## 3) Training-loss vs validation gap: In Figs 2–3 the “cen-CP 100 Gbps” curve appears to reach lower training loss than MoS, yet Table 1 reports better validation perplexity for MoS.\n\nThank you for the keen observation. Table 1 reports scores **after each model is trained to its compute-optimal point**. For the **uncompressed centralized baseline**, reaching that point would require roughly **150 days**, which was beyond our experimental budget.  Therefore, in **Figs. 2–3** we instead plotted each run **only until the the curves first align to a reasonable training loss**, to illustrate how quickly each method descends. Because the centralized curve stops earlier, it has not yet entered the long tail of optimization.\n\nIn the **final phase of training** (omitted from the figure), the **decentralized compressed model continues improving and ultimately slightly surpasses the centralized baseline**, which is why MoS achieves a better validation perplexity in Table 1. We will clarify this in the revised manuscript.\n\n## 4) A zoom-in plot (similar to App.\\ Fig 14) in the main paper would help.\n\nWe will add this. Thank you.\n\nCertainly. We reran the **warm-up vs. no-warm-up** experiment with **three independent seeds** and report the mean and standard deviation (σ) of the resulting perplexities on **FineWeb**.  As perplexity is $\\exp(\\text{validation loss})$, the observed variation corresponds to changes mostly only in the third decimal place of the loss.\n\n| Method        | Run #1 | Run #2 | Run #3 | Mean ± σ |\n|---------------|-------:|-------:|-------:|----------|\n| w/ warm-up    | 22.64  | 22.59  | 22.66  | **22.63 ± 0.03** |\n| No warm-up    | 26.63  | 26.69  | 25.57  | **26.30 ± 0.63** |\n\nWe will strive our best to provide similar multi-seed error bars for the remaining ablations in the revised version.\n\n## 5) Relation to DeepSeek MLA\n\nWe thank the reviewer for the insightful question. The reviewer’s understanding is correct: **MLA** projects **Q, K, V** to a low-dimensional latent space and performs attention computation *entirely* in that space. The primary goal is **memory and compute savings**, particularly for **KV caching during inference**. This complements our work in spirit. Both are motivated by the empirical observation that **QKV spaces are low-rank**, however, there are fundamental differences in their design goals and applicability: **MLA** reduces the dimensionality for attention *computation*, but still requires **each GPU to hold the full attention state** for its batch. **MoS** re-expands K and V and compute attention *block-wise* after the all-gather, which avoids increasing peak memory. It also **significantly reduces inter-GPU communication**.\n\nAdapting MLA directly to context-parallel training poses a **critical synchronization issue**: Since **Q is projected locally** on each GPU, the same query token (replicated across GPUs) would produce **inconsistent projections** unless the projection matrix is fully synchronized. This means that **identical Q tokens would produce different attention scores** when matched with the same K,V, impacting the attention scores. Synchronizing the Q projection matrix across GPUs (e.g., via weight sharing or frequent broadcast) introduces **heavy communication overhead**, defeating the goal of bandwidth-efficient training.  In contrast, MoS compresses and transmits K and V, and each receiving GPU knows the shared basis U and the low-rank parameters  $\\theta$. Thus, U and $\\theta$ are known to all GPUs, allowing consistent reconstruction of K and V after all-gather. This ensures that identical Qs yield consistent attention scores, regardless of which GPU computed them, maintaining semantic consistency across devices without extra communication. Therefore, while MLA and MoS are conceptually related in exploiting low-rank structure, their goals differ (memory vs. communication), and MLA is not directly compatible with context parallelism due to projection consistency issues. Nevertheless, MLA reinforces our empirical thesis, that QKV spaces are compressible without loss, and may inspire future designs that combine memory and communication efficiency in a unified framework."}]

Author rebuttal:

Meta-review: The rebuttal was quite successful here, and the motivation and new experimental data convinced most reviewrs. Two reviewers raised their scores to solid accepts. While one reviewer complained about lack of >7B experiments, another reviewer thought this was not grounds for rejection and other reviewers did not raise this concern (if we were to disregard this complaint, the paper would have a score above the suggested borderline range). There was still some concerns from one reviewer about motivation. Overall though the paper was praised for its novel and impactful solution to an important problem.
Decision: Accept (poster)