Alignment-Enhanced Integration of Connectivity and Spectral Sparsity in Dynamic Sparse Training of LLM
Authors: Wenjing Wu
Yingtao Zhang
Jialin Zhao
Carlo Vittorio Cannistraci
Venue: ICLR 2026 Poster (2026)
Keywords: dynamic sparse training
low-rank factorization
spectral sparse training
efficient training
Primary area: foundation or frontier models, including LLMs

Abstract: With the rapid development of large language models (LLMs), identifying efficient strategies for training such large-scale systems has become increasingly critical. Although LLMs have achieved remarkable success across diverse applications, the necessity of maintaining full dense matrices during pre-training has been questioned, giving rise to parameter-efficient sparse pre-training methods which retains parameter-efficiency in both training and inference. These methods can be further divided into connectivity sparse training and spectral sparse training, with dynamic connectivity sparse training and low-rank factorization emerging as representative approaches for the two branches.
However, a unified framework that effectively combines the strengths of both has yet to be established. In this work, we observe that the cancellation effect between the sparse and low-rank branches may limit the expressivity of the model, manifesting as output conflicts when the two components are combined. To address this issue, we first quantify the cancellation effect using the overlap cancellation ratio (OCR) and then propose a novel scheme that integrates dynamic sparse training with low-rank training, introducing a simple yet effective **alignment loss** to mitigate the disagreement between the two branches and promote better collaboration. We validate this scheme by combining a representative dynamic sparse training method, CHTs, with low-rank training, resulting in a new parameter-efficient training approach termed **CHTsL**. The method is evaluated on LLaMA60M and LLaMA130M using the OpenWebText and C4 datasets, where only 10%, 20%, and 30% of the parameters are preserved compared to dense training. Experimental results demonstrate that our proposed scheme effectively alleviates the cancellation effect, especially in the Q and K matrices of the attention layers, and improves training stability and performance compared to the naive combination of sparse and low-rank components. Additionally, the new scheme enables CHTsL to consistently outperform other parameter-efficient sparse training methods under the same parameter budget, achieving performance closest to that of dense training.

Published as a conference paper at ICLR 2026
ALIGNMENT-ENHANCED INTEGRATION OF CONNEC- TIVITY AND SPECTRAL SPARSITY IN DYNAMIC SPARSE TRAINING OF LLM
Wenjing Wut?, Yingtao Zhang!?; Jialin Zhao'2, Carlo Vittorio Cannistraci
Center for Complex Network Intelligence (CCNI), Tsinghua University. Chinal
2 Department of Computer Science and Technology, Tsinghua University, China
3 Department of Biomedical Engineering, Tsinghua University, China
ABSTRACT
With the rapid development of large language models (LLMs), identifying effi- cient strategies for training such large-scale systems has become increasingly crit- ical. Although LLMs have achieved remarkable success across diverse applica- tions, the necessity of maintaining full dense matrices during pre-training has been questioned, giving rise to parameter-effiiien sparse pre-training methods which retains saraameer-efficiecyy in both training and inference. These methods can be further divided into connectivity sparse training and spectral sparse training, with dynamic connectivity sparse training and low-rank factorization emerging rep- resentative approaches for the two branches. However, unified framework that effectively combines the strengths of both has yet to be established. In this work, we observe that the cancellation effect between the sparse and low-rank branches may limit the expressivity of the model, manifesting as output conflicts when the two components are combined. To address this issue, we first quantify the cancel- lation effect using the overlap cancellation ratio (OCR) and then propose novel scheme that integrates dynamic sparse training with low-rank training, introduc- ing a simple yet effective alignment loss to mitigate the disagreement between the two branches and promote better collaboration. We validate this scheme by combining a representative dynamic sparse training method, CHTs, with low- rank training, resulting in a new parameter-efficien training approach termed CHTsL. The method evaluated on LLaMA60M and LLaMA 33MM using the OpenWeb and C4 datasets, where only 10%, 20%, and 30% of the parameters are preserved compared to dense training. Experimental results demonstrate that our proposed scheme effectively alleviates the cancellation effect, especially in the Q and K matrices of the attention layers, and improves training stability and performance compared to the naive combination of sparse and low-rank compo- nents. Additionally, the new scheme enables CHTssL. to consistently outperform other parameter-efficien sparse training methods under the same parameter bud- get, achieving performance closest to that of dense training.
INTRODUCION
Large language models (LLMs) have attracted tremendous attention due to their superior perfor- mance across wide range of tasks. Despite their impressive capabilities, training LLMs from scratch remains extremely memory-inee inntorr-intensiveeee and computation-intensiiee (Samsi et al. 2023). mak- ing challenging scale such models under reasonable resource constraints. This has motivated extensive research on efficient methods that reduce computational and memory costs while retaining competitive performance. One of the most direct strategies is reduce the number of parameters.
Early studies on pruning and low-rank fine-tuning (Hu et al. 2022 Zhang et al. 2023 Renduchin
tala et al. 2023 Sheng et al. 2023 Liu al. 2024 Kopiczko al. 2023 Detimers et al.) 2023)
Corresponding author. Correspondence: Kalokagathos.aggn wwwo@ggmii.cooooooo gmail.com
Reseaacch Center in Tsinghua Laboratory of Brain and Intelligence (THBI), Department of Psychological and Cognitive Seiences.

---

Published as a conference paper at ICLR 2026
have shown that even after removing or compressing large fraction of parameters, models still preserve much of their original representational capacity. These findings suggest that parameter- efficient model manipulation is feasible, and they naturally motivate the extension from pruning or finetuning to sparse pretraining, where models are trained from scratch under constrained parameter budgets while maintaining competitive performance compared with dense training.
We divides current approaches to sparse pre-training can be broadly into two branches: connec- tivity sparse training and spectral sparse training, which refers those methods utilizing low-rank factorization during pretraining.
The former branch focuses on enforcing sparsity in the connectivity of weight matrices, with dy- namic connectivity sparse training emerging as representative technique (Mocanu et al. 2018 Jayakumar al. 2020; Evei al., 2020; Yuan et al. 2021 Zhang et al. 2024] 2025). Dynamic connectivity sparse training maintains sparse connectivity pattern throughout traiiiinn dynam- ically changing the sparse connectivity and updating active weights to approximate the capacity of dense models. Recent works have shown that on multiple tasks, dynamic sparse training can approach or even surpass the performance of dense models with as little as 10% of the trainable parameters, marking significant step forward in efficient training (Zang et al. 2024).
Cechnique(Mocanu
The second branch, spectral sparse training (Zhao et al., 2024a), is typically instantiated through low-rank factorization. Since the low-rank factors are updated during training, the spectral rep- resentation they induce also evolves accordingly, which makes spectral sparse training inherently dynamic. Initially proposed in the context of LLM fine-tuning (Hu et al. 2022), low-rank methods decompose weight matrices into low-dimensional components, training only the low-rank repre- sentations while freezing the full-rank backbone. These approaches drastically reduce the number of trainable parameters and leverage the pre-trained dense model's representational power. More recent attempts have extended low-rank factorization to the pre-training stage (Lialin et al. 2023 Zhao et al., 2024bb Xia et al., [024;] Meng et al. 2024 Zhao et al. 2024a). However, these methods still require the use of full dense matrices during the forward pass, rather than maintaining the spec- tral sparse structure consistently from training to inference. Overcoming this limitation, successors like CoLA (Liu et al. 2025) preserve the low-rank structure throughout both training and inference, further validating the feasibility of spectral sparse training.
While previous attempt SLTrain (Han et al. 2024) explored combining sparse and low-rank com- ponents, the design remains limited in two key aspects. First, the sparse branch in SLTrain is static, serving only as a supplementary term to spectral sparse training rather than leveraging the full poten- tial of dynamic connectivity sparse methods. Second, SLTrain simply performs pure summation of sparse and low-rank outputs, without any mechanism to promote effective interaction.
In this work, we take step in this direction. We observe that naive integration of sparse and low- rank branches often suffers from a cancellation effect, where the two components produce conflict- ing representations that weaken expressivity and hinder convergence. To address this challenge, propose a new scheme that integrates dynamic connectivity sparse training with low-rank training under the guidance of alignment loss, which aligns the two branches and promotes cooperative learn- ing. Specifically, we instantiate our framework by combining the ad advanced dynamic sparse training method CHTs (Zhang et al., (025) with low-rank factorization, resulting in a new parameter-eeffiieet pre-training approach, CHTsL. sL. Extensive experiments on LLaMA-60M and L.LMM--33M (Tou-) vron al.) ()) with WebText and C4 show that CHTsL. consistently outperforms state- of-the-art parameter-efficient sparse training baselines under the same parameter budget. Notably, with only 10%, 20%, or 30% of parameters preserved relative to dense training, CHTsL achieves performance closest to dense models, which would benefit by retaining efficiency in training, infer- ence, and storage.
Our contributions can be summarized as follows:
First integration of connectivity sparse and spectral sparse in dynamic sparse training. We make the first attempt to genuinely integrate connectivity sparse and spectral sparse in dynamic sparse training, with dynamic connectivity and dynamic low-rank representaion. Unlike prior work such as SLTrain, where static connectivity sparsity merely served supplement to spectral spar- sity, our approach fully leverages the complementary strengths of both paradigms.
2

---

Published as a conference paper at ICLR 2026
Alignment-enhancee unified scheme. We identify the cancellation effect key obstacle in com- bining sparse and low-rank branches, where conflicting representations weaken model expressivity. To address this, we introduce the overlapping cancellation ratio (OCR) as quantitative measure, and propose a unified integration scheme that emphasizes interaction and cooperation rather than naive branch summation. By incorporating alignment loss, our framework explicitly mitigates conflicts, enhances collaboration, and alleviates the observed cancellation phenomenon in attention Q and K matrices.
Instantiation with CHTsL and empirical superiority. We instantiate the framework by combining advanced CHTs with low-rank factorization, yielding the proposed method CHTsL. Extensive exper- iments across different datasets, models, and parameter budgets demonstrate that CHTsL achieves consistently strong performance, ranking first among all parameter-effiient methods with the same parameter scale, and approaching dense model performance with significantly fewer parameters.
2 RELATED WORK
The rapid growth of large language models (LLMs) has stimulated extensive research into improving efficiency in pre-training Among various directions, parameter-effiient approaches have emerged as particularly promising, aiming at training models with limited number of parameters without significantly sacrificing performance. Broadly, parameter-efffcient methods in the context of pre- training can be divided into two branches: connectivity sparse training, which reduces parameters by enforcing sparse connectivity patterns, and spectral sparse training, which constrains weight matrices into low-rank subspaces. Dynamic connectivity sparse training and low-rank factorization are the representative approaches for these two paradigms.
2.1 DYNAMIC CONNECTIVITY SPARSE TRAINING
Connectivity sparsity originates from the classical line of pruning (LeCun et al., 1989 Han et al.) 2015 Molchanov al. 2016), where removing parameters from dense models was shown to pre- serve much of the model's 's performance. Inspired by this, researchers began to explore whether spar- sity could be maintained throughout training, rather than applied only post-hoc compression. Among these efforts, methods that promote sparse training through dynamic adjustment of connec- tivity have gained increasing attention, they often outperform static sparse training approaches that prune connections solely at initialization (Prabhu et af., 2018 [ee et al. 2018 Dao et al.) 2022 Stewart al. 2023). The pioneering work Sparse Evolutionary Training (SET) (Mocanu et al., 2018) removes links while introducing random rewiring of sparse connections during training to maintain model plasticity. RigL (Evci et al. 2020) further dynamically regrows connections based on gradient for more effective exploration, though it requires computing gradients of the full weight matrix during the backward pass. MEST (Yuan al., (02I) improves upon this by leveraging both weight and gradient information. CHT (Zhang et al. 2024 and its successor CHT's (Zhang et al.) 2025) enhance dynamic sparse training using the Cannistracci-Hebhie
theory (Muscoloni et al,, 2022 from network science, inspired by brain connectomes, achieving state-of-lie the-art performance on multiple tasks. Collectively, these studies demonstrate that dynamic sparse training can attain competitive or even superior performance compared to dense training, while using only 10% or fewer of the parameters (Zhang et al. 2025).
2.2 LOW-RANK FOR SPECTRAL SPARSE TRAINING
Complementary to connectivity sparsity, spectral sparse training leverages low-rank factorization reduce the dimensionality of weight matrices. This idea was first popularized in the fine-tuning setting, where LoRA (Hu et al. 2022) adapts pretrained models by learning only low-rank updates
rather than full weight matrices. Subsequent works (Hu et al., 2022, Zhang et al. 2023 Renduch- intala et al. 2023 Sheng et al. 2023 Kopiczko et al.) 2023 Detimers et al., 2023 Liu et al. 2024)
further demonstrate the effectiveness of low-rank fine-tuning and inspire the exploration of training from scratch with low-rank factorization. ReLoRA (Lialin al. (023) introduces reparameteriza- tion to improve training efficiency and stability, while GaLore (Zhao et al. 2024b) reduces memory usage by applying low-rank projections in the gradient space during training. However, common limitation of these approaches is that the full dense weight matrix is still required during the for- ward pass, providing parameter efficiency only during training but not during inference. In contrast,
3

---

Published as a conference paper at ICLR 2026
CoLA (Liu al. 2025) explicitly maintains the low-rank representation throughout both training and inference, enabling reduced storage and runtime costs.
As side note, while previously discussed the relevance of pruming mainly in the context of connectivity- -bssed sparse training, in contrast to spectral low-rank training, structured pruning can also be viewed as implicitly inducing low-rank structure in the resulting model. This is because pructured-pruned models remove entire channels or filters, which correspond to removing rows or columns in the unfolded weight matrices, thereby potentially reducing their effective rank. Rep- resentative structured-pruning -aaree works include channel pruning via LASSO (He et al., 2017). network slimming (Liu et al. 2017), and more recently Only- Train-Once (OTO) (Chen al. 202 which explicitly consider structural constraints during training to improve efficiency for subsequent pruning. In this study, we adopt CoLAA (Liu al. 2025) as the baseline under the same restriction of parameter efficiency in both forward and backward passes.
2.3 HYBRID ATTEMPT
Beyond individual paradigms, researchers have also begun explore combining connectivity and spectral sparsity. SLTrain (Han et al. 2024) represents one of the carliest attempts in this direction. augments low-rank factorization with sparse branch, but its design exhibits several limitations. Specifically, the sparse component is static rather than dynamic, serving merely tary term spectral sparsity instead of leveraging genuine connectivity sparse training. Moreover, SLTrain integrates the two branches via a simple summation, without introducing any collaborative mechanism to exploit their potential synergy. As result, while SLTrain marks an important step toward hybrid warameter-efficien pre-training, it remains an immature solution, lcaving room for more principled approaches.
supplemen-
3 ALIGNMENT-ENIANCE INTEGRATION OF CONNECTIVITY SPARSE AND SPECTRAL SPARSE
In this section, we present a unified approach for combining dynamic sparse training (connectivity sparse) with low-rank factorization (spectral sparse) under extreme sparsity. While each method alone can improve parameter-efficiency and memory-efficiency, their naive combination often leads conflicting outputs that limit the model's effective capacity. We address this challenge with three key steps: (i) identifying and quantifying the cancellation effect, (ii) introducing a training frame- work that stabilizes low-rank outputs and encourages cooperation between branches, and (iii) in- stantiating method, CHTsL, that integrates connectivity sparse and spectral sparse for dynamic sparse training based on this framework.
3.1 CANCELLATION EFFECT AND OCR METRIC
When sparse branch and low-rank branch are trained together, a common phenomenon emerges: their outputs sometimes point opposite directions. This cancellation effect means that some portion of the signal from one branch can be neutralized by the other, wasting representational power. other words, even each branch individually carries meaningful information, their naive sum may not fully reflect that information, effectively underutilizing the model's capacity.
To quantify this, we define the Overlap Cancellation Ratio (OCR):
OCR min( 1[S,L, < <0} E, (S.n (, +
(L)
where S and are the outputs of the sparse and low-rank branches, respectively. OCR measures the fraction of overlapping signal that is canceled due opposite directions, with naturally restricted in the range [0, 1). A higher OCR indicates more severe cancellation.
3.2 TRAINING FRAMEWORK: ALIGNMENT LOSS AND ACTIVATION ADJUSTMENT
Alignment Loss for Cooperative Learning. When training using two distinct components, the sparse and low-rank branches can produce conficctig signals. Intuitively. one branch pushes a feature in direction while the other pushes in the opposite direction, the net effect is reduced
4

---

Published as a conference paper at ICLR 2026
expressivity. To address this cancellation effect, introduce alignment loss that encourages the outputs to move in similar directions:
(C)
11 [[F Lalign BN
1 1
Σ
( aliigns
aliggn
where is the batch size and is the number of elements in one sample's 's output at layer I. This loss penalizes discrepancies between the sparse and low-rank outputs, reducing destructive interference and letting each branch focus on complementary aspects of representation. Each layer contributes to the total alignment loss, which is then weighted in the final objective.
Activation Adjustment for Low-rank Stability. Low-rank factorization reduces the number of trainable parameters but sometimes produce unstable outputs, particularly under extreme spar- sity. Inspired by CoLA Liu et al. (2025). we apply mild non-linear activation between the factor- ized matrices:
L = BAA (AA
where (-) is a non-linear function (SiLU (Hendrycks & Gimpel, 2016) in our experiments). Here, the activation primarily serves to stabilize the low-rank outputs, maintaining a reasonable scale and preventing numerical issues during training. Its role is mainly supportive, ensuring the low-rank branch contributes reliably alongside the sparse branch.
Overall Objective. Combining these ideas, the output of each 00 and the total training loss L are respectively:
Output
Alignment Loss
O 5t) + L L Lrack + Aaalign (4)
where balances the contribution of align- ment. This objective ensures that the sparse and low-rank branches are jointly optimized, stabilizing low-rank training and encouraging two branches to cooperate, mitigating can- cellation.
Spectral Sparse
Dynamic Connectivity Sparse
(CHTs)
B
SILU
Connectivity Evolution
3.3 CHTSL:
Forward
INSTANTIATING THE FRAMEWORK
Backward
Based on this training framework, we propose CHTsL, which integrates dynamic connectiv- ity sparse training method CHT's ÇZhang et al.) 2025) with spectral sparse (low-rank) com- ponents. In CHTsL, the sparse branch fol- lows the CHTs update rules, while the low- rank branch incorporates mild activation ad- justment, and the alignment loss applied layer-wise to encourage cooperative outputs. This instantiation demonstrates how our framework naturally combines dynamic connectivity and spectral sparsity, providing practical approach for training extremely sparse models under a unified scheme. Figure Iilluttrasss how CHTsL works.
Figure Workflow of CHTsL. The figure il- lustrates CHTsL an example of alignment- enhanced integration between dynamic connec- tivity sparse training and spectral sparse train- ing. Specifically, the dynamic connectivity sparse branch adopts the CHT's method.
4
EXPERIMENT
4.1 MODELS
Experiments are based Transformer models from the LLaMA family (Touvron et al. 2023a by. with parameter sizes ranging from 60M 130M. All models are trained and evaluated on NVIDIA A100 or A800 GPUs.
4.2 DATASETS
For training and evaluation, adopt two widely used large-scale text corpora:
5

---

Published as a conference paper at ICLR 2026
Teb (Gokaslan & Cohen 2019): A publicly available open-source replication of the WebText dataset used in GPT-2. It is constructed by scraping URLs shared on Reddit with high karma scores, covering broad range of high high-quality web content.
Colossal Clean Crawled Corpus (C4) (Raffel et al. 2020): A large-scale dataset derived from web pages collected through Common Crawl. After extensive cleaning and filtering, it provides high-quality natural language text suitable for large language model pre-training.
4.3 BASELINE METHOSS
To verify the effectiveness of our method, we compare it against several parameter-efficient training baselines with an equivalent number of trainable parameters. Specifically, we consider dynamic connectivity sparse training methods including SET (Mocanu et al., 2018). RigL (Evei et al. 2020). MEST (Yuan et al.) 2021) and CHTs (Zhang et al.
2025); spectral sparse training method CoLA Liu
ct al. (2025); hybrid method SLTrain (Han et al.
2024). We also report the performance of dense
training for comparison.
4.4 DEFINITION OF SPARSITY
Since this work integrates connectivity- -asse sparsity with spectral (low-rank) sparsity, it is neces- sary to establish a consistent definition of sparsity. For both connectivity sparse and spectral sparse (based low-rank factorization of full matrix), we adopt the same definition of sparsity s and corresponding density d, representing the fraction of parameters relative to a full-rank dense matrix, which allows fair comparison across methods by reflecting the total number of trainable parameters:
#params
1
d = 1-
params
For connectivity sparse method, the original sparsity corresponds to the true sparsity of the net- work. For a low-rank factorization of dense matrices of size m n with rank r, the effective density is (m + n)r/ (m n). For method that integrates both connectivity and spectral sparsity, the total sparsity can be computed as
(6) In our experiments, all methods are compared under the same total sparsity to ensure an equivalent number of trainable parameters. For clarity in the Section [3] we report the total sparsity of each method, and we additionally provide the pparsity-coniiguratonn for the integrated methods, which includes the sparsity of the connectivity sparse component, the rank r of the low-rank component, and the proportion deannsctivviy of parameters between two branches in Appendix B
Stotal = 1 - dsonncctivity dspectral-
depectral
4.5 HYPERPARAMETER SETTINGS
Alinmeen--enhaneed training scheme introduces the coefficient À to control the effect of alignment loss. We searched the X in the range [O, 0.1, 0.3 0.5, 0.7, 1] with preliminary experiments. For L.L.MA60M on Open OpenWetText and ILLMA-130M, the appropriate À is 0.5; For LLaMA-60M on C4, the appropriate À is 0.3.
For methods combining sparse and low-rank training (including SLTrain and CHTsL), the sparsity- configuration mentioned in Section [4.4 need to be considered under the same total parameter budgets. We systematically varied the allocation of parameters between the sparse and low-rank branches steps corresponding to total sparsity of 5% and the best results across all sparsity- configurations were reported. The step size for rank adjustment in the low-rank branch was calcu- lated based on the model architecture, resulting in approximate step values of 16 for LLaMA-60M and 24 for -.aMA-110MM of which the concrete calculation process be found in Appendix A
All the other hyperparameters can be found in Appendix B which set to be the same maximally for different methods for fair comparison.
RESULT AND DISCUSSION
In this section, we present the experimental results. We first present the effectiveness of the alignment-enhanced training scheme by comparing it with the naive integration of CHT's and low- rank factorization. And then we compare different efficient training methods under the same param- eter budget to present that CHTsL consistently improves the performance, realizing the performance most close to dense training with limited parameters.

---

Published as a conference paper at ICLR 2026
Table 1: Comparison between different integration strategies. The table consists of two parts: a. The performance of different integration strategies, reported in terms of validation perplexity (PPL.L). The Naive strategy corresponds to simple sum of CHTss and low-rank factorization. The Act strategy applies activation adjustment to the low-rank factorization branch. The Act+Align strat- egy combines activation adjustment with the alignment loss. The coefficient the alignment loss À is reported in Section 4.5 The sparsity configuration is set such that the sparse branch and the
low-rank branch have the same number of trainable parameters( dconnectivity 1). b. The Wilcoxon
depectraal
signed-rank test p-values, which indicate whether the differences in performance between strate- gies statistically significant.
Model
Dataset
Total Sparsity
Naive
Act
Act+Align
0.9
32.64
32.21
31.77
OpenWebText
0.8
33.35
29.42
29.11
0.7
27.89
29.94
27.40
LLaMA-60M
0.9
189.55
39.66
39.29
C4
0.8
36.71
36.54
36.16
0.7
591.42
34.55
34.33
0.9
119.35
24.45
24.07
OpenWebText
0.8
22.11
21.98
21.87
0.7
21.12
20.90
20.65
LLaMA-130M
0.9
30.77
30.30
30.03
C4
0.8
27.83
27.68
27.59
0.7
920.16
26.55
26.19
Wilcoxon signed-rank
against Naive
0.0093
0.00049
p-value
against Act
0.00049
5.1 EFFECTIVENESS OF ALIGNMENT-ENIIACEE INTEGRATION
Performance improvement To verify the effictiveness the alignment- enhanced training scheme, compare CHTsL with the naive integration between CHTss and low-rank factorization. In Table [] we present the result of CHTss plus low-rank with different integration strategy on dif- ferent models and datasets with different total sparsity, under the constraint that sparse component and low-rank component dominates the same number of parameters (rannertisitty 1). The re- depectra!
sults are validated by Wilcoxon signed-rank test for the statistical comparison. It shows that, with p value 0.05, activation adjust of low-rank improves the training stability and the whole align- ment training scheme makes CHTsL significantly better than the naive integration.
Eased Cancellation Effect. Figure 2 presents the OCR defined in Equation 1 comparing the can- depectral
cellation effect between the naive integration and the alignment-enhanced approach for the experi- ment I.LMA-60MM with the OpenWebText dataset under total sparsity of 0.9 with sparsity-
configuration 0.95, 16, deannsctivity
We observe that incorporating the alignment loss significantly reduces the OCR in the Query and Key layers, with performance substantially surpassing that of the naive integration. A plausible explanation is that Q and K, as the core com- ponents of attention, directly determine the attention weights via their dot product, making them highly sensitive inconsistencies between the dynamic sparse branch and the low-rank branch. En- forcing alignment therefore stabilizes the attention maps and mitigates gradient conflicts, whereas feed-forward value projections more tolerant internal variations due residual connec- tions. Consequently, this targeted consistency in Q and enhances the robustness of the attention mechanism, leading to overall performance improvements. More evidence of experiments under other sparsity levels can be found in Appendix
5.2 CHTSL OUTPERFORMS OTHER SPARSE TRAINING METHODS
Table 2 reports the results of CHTsL in comparison with all baseline methods under the total parameter budget. The results demonstrate that CHTsL consistently outperforms all other methods given the same parameter constraint. This provides strong evidence for the potential of integrating connectivity sparse training with spectral sparse training, achieving performance closest to dense training while preserving only 30% or fewer of the training parameters.

---

Published as a conference paper at ICLR 2026
Layer 0,self_attnn. attn.o_proj
Layer 0.self attn.k proj
Layer 0.self attn..v proj
Layer 0.self ottn.o proo
0.4
0.4
0.4
0.4
8
0.2
0.2
0.-
0.2
7500 10000
2500
7500 10000
2500
7500 10000
2300
50.00
10000
Thin Steps
Train Steps
Train Steps
Train Steps
Layer 1.self attnatttttrttt proj
Layer 1.self attn k_ proj
Layer 1.self attn.y proj
Layer 1. self attn.o proj
0.4
8 0.2
8 6.2
8
0.2
0.0
0.0
0.0
0.0
2500 5000 7500 10000
0
2500 5000
7500 10000
2500
5000
7500 10000
2500 50000
7500 10000
Train Steps
Train Steps
Train Steps
Train Steps
Layer 2.self attn.o proj
Layer selt proj
Layer 2.self attn.vv proj
Layer 2 proj
0.4
0.4
0.4
0.4
8
8
0.0
0.0
0.0
0.0
2500
5000
7500 10000
2500
7500 10000
2500
5000
7500 10000
2500
50.00
7500 10000
Train
Train
Train
Layer 3.self attanttttprreee proj
Layer 3. sele attn.kk proj
Layer 3.self attnivv proj
Layer 3 self ottn.o otroo
0.4
0.4
15
0.2
0.2
0.2
0.2
7500 10000
2500
7500 10000
2500
7500 10000
2500
10000
Than Steps
Train Steps
Train Steps
Train Steps
Layer 4.self attnnttttrppt proj
Layer 4. 4.self, attn.k, proj
Layer 4.self attn.yy proj
Layer 4.self, attn.o proj
00
OCN
0.2
0.2
0.2
0.0
0.0
0.0
0.0
2500
5000
7500 10000
2500
5000
7500 10000
2500
5000
7500 10000
2500
50000
7500 10000
Thain Steps
Train Steps
Train Steps
Train Steps
Layer 5.self attn.a proj
Layer salf attn. k. K_proj
Layer S.self altn.v pro]
Layer 5. self pro)
0.4
0.4
0.4
0.4
0.0
0.0
0.0
0.0
2500
5000
7500
10000
2500
5000
7500 10000
2500
5000
7500 10000
2500
50000
7500 10000
Train
Train
Train
Layer 6.self_ aatnnnt_praa
Layer 6.self, attn.kkttnn proj
Layer 6.self 6.self_attn.v aattnat_ttreet __ror
Layer 6. 6.self_alte www..aatt..o...... o_proj
0.4
0.4
0.4
0.2
0.2
0.2
7500 10000
2500
7500 10000
2500
10000
2500
10000
Stegs
Train Steps
Tran Steps
Train Sleps
Layer 7.self. wwttttttttttttt proj
Layer 7. 7.self attn proj
Layer 7.self attn..y __trr.
Layer 7 7eself ottn.o proj
08
0.2
0.2
0.0
0.0
9.0
0.0
0
2500
5000
7500 10000
0
2500
5000
7500 18000
0
2500
5000
7500
16000
2500
5000
7500
10000
Train Steps
Thain Steps
Train Steps
Train Steps
Nove
Act
ActeAlign
Figure 2: The layer-wise OCR plot of LLaMA60M on WebTeet with a total sparsity of 0.9, with sparsity-configuratonn s 0.95, r 16, deonnectivitty dssectrat = 1 1. Each subplot in the figure reports the changes of OCR over training steps. The plot is based on the experiment of the first row of Table [ For space limit, we report here the self-attention layers in the model, where each column refers to Q, K. V, O respectively.
8

---

Published as a conference paper at ICLR 2026
Table 2: Validation perplexity of different methods. Validation perplexity (PPL.)) is reported in this table for different methods on different datasets under the same constraint of total sparsity Stotal- Bold values are the best performance out of all sparse methods.
LLaMA-
LLaMA-130M
Dataset
Method
Stotal =0.9
Stotal=0,
Stolul Stotut=099
Stota|=0.8
Stota(=0.7
Dense
26.56
SET
35.26
30.69
31.77
25.70
23.20
22.03
RigI.
45.34
41.33
39.96
41.25
44.49
70.11
MEST
33.6
29.94
28.26
25.59
22.93
21.63
OpenWebText
CHTs
33.03
29.84
28.12
24.75
22.67
21.48
CoLA
37.58
30.87
28.53
27.07
23.24
21.61
SL.rain
33.90
29.83
27.86
25.33
22.81
21.25
CHTsL.
31.77
29.11
27.40
24.07
21.87
20.65
Dense
33.21
24.55
SET
42.32
37.70
35.62
32.45
29.47
27.75
RigI.
53.39
48.59
47.34
43.57
55.82
64.93
MEST
41.46
37.28
35.40
32.54
29.29
27.59
C4
CHTs
40.62
37.55
35.23
31.00
28.69
27.46
CoLA
46.41
38.58
35.87
33.52
29.26
27.25
SLTrain
41.05
37.00
34.89
31.38
28.28
26.78
CHTsL
39.29
35.95
34.19
30.03
27.59
26.19
Sensitivity to Sparsity Configuration
LLaMA60M TpenWebText
LLaMAGOM C4
31
68552
35.5
22
30
35.0
29
s=0.7
s=10
s=0.9
5=0.85
s=0.75
r-80
r-48
r-32
r-16
r-0
r-96
r-80
r-64
r-48
r-16
Sparsity Configuration (Sparsity s. Rankr)
Sparsity Configuration 1Sparsity Rank
LLaMA130M OpenWebText
LLaMA130M CC4
25
24
27.5
23
27.0
22
26.5
21
5-1.0
sm0.95
S=0.9
sm0.85
5-0.8
sm0.75
sm0.7
s=1.0
sm0.95
5m0.9
5m0.85
5m0.8
5-0
sm0,7
r-144
r-120
r-96
r=72
/m48
r=24
rm0
[-144
r=120
r-96
r=72
1-48
r=24
rm)
Sparsity Configuration (Sparsity Rank
Sparsity configuration iSparsity
Figure 3: Sensitivity analysis of sparsity configurations under a total sparsity of 0.7. The pparsity-coffiuuraton is defined by the sparsity s the connectivity-sparee branch and the rank r in the low-rank branch. Each subplot illustrates the variation of validation perplexity (PPL.) as the rank decreases by step of 5% total sparsity. Outliers with PPL. values exceeding the correspond- ing thresholds are highlighted in red, with their true values explicitly annotated.
5.3 SENSITIVITY TEST FOR SPARSITY CONFIGURATIONS
In Figure 3 we illustrate how validation perplexity (PPL.) varies with different sparsity configu- rations across models and datasets under fixed total sparsity of 0.7. On Open WebText, when the low-rank branch dominates the parameter budget far more than the snneectivity-sppare branch (spar- sity in the connectivity sparse branch exceeds 0.9), performance collapses. This instability may be attributed to the dataset's 's relatively limited complexity. Since WebTWe more homogeneous, the model becomes more sensitive imbalanced sparsity allocation. By contrast, C4, which

---

Published as a conference paper at ICLR 2026
contains substantially more diverse and heterogeneous text, higher proportion of low-rank param- eters proves beneficial. A possible explanation is that the increased variety of linguistic patterns likely requires broader adaptations of the entire weight matrix, making low-rank components more effective capturing such variability.
CONCLUSION
In this work, we present novel framework for parameter-efficien pre-training by systematically integrating connectivity sparse training with spectral sparse in dynamic sparse training. We identify the cancellation effect naive integration as the key challenge, where conflicting representations branches reduce expressivity and hinder convergence. To address this, we introduce the overlapping cancellation ratio to quantify the effect and an alignment loss to encourage cooperative learning- Building this framework, we instantiate CHTsL by combining the advanced dynamic sparse training method CHTs with low-rank factorization. Extensive experiments on LLaMAA60M and LLaMA-130M with Text and C4 demonstrate that CHTsL consistently outperforms isting methods under equivalent parameter budgets. Our work is the first to systematically unify dynamic connectivity and spectral sparse training, moving beyond static connectivity sparsity and naive integration; it identifies and mitigates the cancellation effect, fostering effective collaboration between the sparse and low-rank components; and it provides practical instantiation that validates the benefits of this integration. Overall, this study offers both theoretical insights and practical solutions for efficient sparse pre-training highlighting the potential of combining complementary sparsity paradigms to maximize model expressivity under constrained resources.
ACKNOWLEDGMENTS
This work was supported by: The National Natural Science Foundation of China, Research Fund for International Scientists III (grant number W253 to C.V.C). The Zhou Yahui Chair Profes- sorship award of Tsinghua University (to C.V.C). The National High-Level Talent Program of the Ministry of Science and Technology of China (grant number 20241710001, to C.V.C). The authors thank Yuchi Liu and Mo Yang for their administrative support.
REPRODUCIBILITY STATEMENT
The code for this work is provided in the supplementary material. Detailed hyperparameter settings for each method are presented in Appendix B facilitate reproducibility.
REFERENCES
Tianyi Chen, Bo Ji, Tianyu Ding, Biyi Fang, Guanyi Wang, Zhihui Zhu, Luming Liang, Yixin Shi, Sheng Yi, and Xiao Tu. Only train once: A one-shot neural network training and pruning framework. Advances in Neural Information Processing Systems, 34:19637-19651. 2021.
Tri Dao, Beidi Chen, Nimit Sohoni, Arjun Desai, Michael Poli, Jessica Grogan, Alexander Liu, Aniruddh Rao, Atri Rudra, and Christopher Ré. Monarch: Expressive structured matrices for efficient and accurate training. In International Conference on Machine Learning, pp. 4690- 4721. PMLR, 2022.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36: 00018-1880115,, 2023.
Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International conference machine learning, pp. 2943 2953-2952. PMLR, 2020.
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http://Skyyioon00...tttt 1o/ TeeeteebTextCorppee 2019.
Andi Han, Jiaxiang Li. Wei Huang, Mingyi Hong. Akiko Takeda, Pratik Kumar Jawanpuria, and Bamdev Mishra. Sltrain: a sparse plus low rank approach for parameter and memory efficient pretraining. Advances in Newral Information Processing Systems, 37:118267-118295 2024.
10

---

Published as a conference paper at ICLR 2026
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In Proceedings of the IEEE international conference computer vision, pp. 1389-1397, 2017.
Dan Hendrycks and Kevin Gimpel. nee::66060..11 2016.
Gaussian
error linear units (gelus). arXiv preprint
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Siddhant Jayakumar, Razvan Pascanu, Jack Rae, Simon Osindero, and Erich Elsen. Top-kast: Top-k always sparse training. Advances in Neural Information Processing Systems, 33:20744-2077 207544, 2020.
Dawid Kopiczko, Tijmen Blankevoort, and Yuki M Asano. Vera: Vector-based random matrix adaptation. arXiv preprint arXiv:2310.11455 2023.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based connection sensitivity. arXiv preprint arXiv:1810.02344 2018.
Vladislav Lialin, Namrata Shivagunde, Sherin Muckatira, and Anna Rumshisky. Relora: High-rank training through low-rank updates. arXiv preprint arXiv:2307.05695, 2023.
Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight decigt--ccomppsedd low-rank adaptation. In Forty-first International Conference on Machine Learning, 2024.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In Proceedings of the IEEE international conference on computer vision, pp. 2736-2744, 2017.
Ziyue Liu, Ruijie Zhang, Zhengyang Wang. Zi Yang, Paul Hovland, Bogdan Nicolae, Franck Cap- pello, and Zheng Zhang. Cola: Compute-efficnn pre-training of lims via low-rank activation. arXiv preprint wrXx::2502.10940 2025.
Xiangdi Meng, Damai Dai, Weiyao Luo, Zhe Yang, Shaoxiang Wu, Xiaochen Wang, Peiyi Wang. Qingxiu Dong, Liang Chen, and Zhifang Sui. Periodiclora: Breaking the low-rank bottleneck in lora optimization. arXiv preprint ar 241.24402.16111, 2024.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connec- tivity inspired by network science. Nature communications, 9(1):3383, 2018
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv: 1,06400 2016.
Alessandro Muscoloni, Umberto Michieli, Yingtao Zhang, and Carlo Vittorio Cannistraci. Adaptive network automata modelling of complex networks. 2022.
Ameya Prabhu, Girish Varma, and Anoop Namboodiri. Deep expander networks: Efficient deep networks from graph theory. In Proceedings of the European Conference Computer Vision (ECCV), pp. 20 20-35, 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
Adithya Renduchintala, Tugrul Konuk, and Oleksii Kuchaiev. Tied-lora: Enhancing parameter effi- ciency of lora with weight tying. arXiv preprint arXiv:2311.09578 2023.

---

Published as a conference paper at ICLR 2026
Siddharth Samsi, Dan Zhao, Joseph McDonald, Baolin Li, Adam Michaleas, Michael Jones, William Bergeron, Jeremy Kepner, Devesh Tiwari, and Vijay Gadepally. From words to watts: Benchmarking the energy costs of large language model inference. In 2023 IEEE High Perfor- mance Extreme Computing Conference (HPEC), pp. 1-9. IEFE, 2023.
Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Liammin Zheng, Kurt Keutzer, et al. S-lora: Serving thousands of concurrent lora adapters. arXiv preprint WrXiv:2311.03283 2023.
James Stewart, Umberto Michieli, and Mete Ozay. Data-free model pruning at initialization via expanders. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recog- nition, pp. 4519-4524, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini arXiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.0920 092888, 2023b.
Wenhan Xia, Chengwei Qin, and Elad Hazan. Chain of lora: Efficient fine-tuning of language models via residual learning. arXiv preprint arXiv:2401 000:2440..4451,, 2024.
Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang Li, Zhenglun Kong, Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang He, Qing Jin, et al. Mest: Accurate and fast memory-econome sparse training framework on the edge. Advances Neural Information Processing Systems, 34:20838-20850, 2021.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter- efficient fine-tuning. arXiv preprint arXiv:2303. 10512, 2023.
Yingtao Zhang, Jialin Zhao, Wenjing Wu, and Alessandro Muscoloni. Epitopological learning and cannistraci-hebb network shape intelligence brain-inspired theory for ultra-sparse advantage in deep learning. In The Twelfth International Conference on Learning Representations, 2024.
Yingtao Zhang, Diego Cerretti, Jialin Zhao, Wenjing Wu, Ziheng Liao, Umberto Michieli, and Carlo Vittorio Cannistraci. Brain network science modelling of sparse neural networks enables transformers and Ilms to perform as fully connected. arXiv preprint arXiv:2501. /9/07, 2025.
Jialin Zhao, Yingtao Zhang, Xinghang Li, Huaping Liu, and Carlo Vittorio Cannistraci. Sparse spectral training and inference euclidean and hyperbolic neural networks. arXiv preprint arXiv:2405. 15481, 2024a.
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficien Ilm training by gradient low-rank projection. arXiv preprint arXiv:2403.03507, 2024b.
12

---

Published as a conference paper at ICLR 2026
A SPARSITY CONFIGURATION FOR LLAMA-60M AND LLAMA-130M
The sparsity configuration for methods combining a sparse branch with a low-rank branch is defined by two values: s, the sparsity of the connectivity-spars component, and r, the rank of the low-rank component.
In experiments, for each fixed total sparsity, we varied the sparsity-configuratinn in steps of As = 0.05. That is, whenever the parameter count of one branch was reduced by 5% relative dense training. the parameter count of the other branch was increased accordingly. Since the sparsity of the connectivity-spaase branch is directly tied to the total sparsity, the main challenge determining the corresponding rank adjustment in the low-rank branch, which depends on the structure of the LLaMA model.
All linear layers in LLaMA are replaced by our sparse components. Because LLaMA models of different sizes are built from repeated Transformer blocks with identical architecture, it suffices to analyze single block to establish the relationship between 8 and r. Each block contains seven linear layers, denoted as Q K, V, O. up, down, and gate. Among them, Q. K, V, and O have weight matrices of size h x h, while up, down, and gate have size f. where
the embedding dimension and is the feed-forward dimension. Hence, the step size of the rank Tsep corresponding to As 0.05 determined by:
4(h + h)yssep + 3(h + fsss As 0.05.
4(h h) + 3(h f)
For LLaMA-60M with h = 512 and f = 1376, this yields a rank step size of approximately Tstep R 16. For LLLMA-1300 1330 with h 768 and 2048, the corresponding step size Tstep Rs 24.
13

---

Published as a conference paper at ICLR 2026
B DETAILED HYPERPARAMETER SETTINGS FOR EACH METHOD
For fair comparison, almost all experiments adopt the common hyperparameter settings listed in Table 3 consistent with prior work.
Table Common hyperparameter settings for experiments on LLaMA-60M and LLaMA-130M. The settings align with previous research.
Hyperparameter
LLaMA-60M
LLaMA-130m
Embedding Dimension
512
768
Feed-forward Dimension
1376
2048
Global Batch Size
512
512
Sequence Length
256
256
Training Steps
10000
20000
Warmup Steps
1000
2000
Learning Rate
3e-3
3e-3
Optimizer
Adam
Adam
Layer Number
8
12
Head Number
12
Iterative warmup steps
20
20
Update Interval for DST
100
100
There are several exceptions, particularly for dense training and CoLA. For dense training, due the substantially larger number of parameters, a high learning rate leads to model collapse. Therefore, we adopt learning rate of le-3, following the setup in Zhang et al. (2025). For CoLA, we observed strong sensitivity to the choice of optimizer: using Adam causes training collapse (with perplexity exceeding 100). To stabilize training, we use the Adam V optimizer provided in their official code.
Method-specific hyperparameter settings
follows:
DST methods (SET, RigL, MEST, CHTs):
We follow the hyperparameter configurations reported directly imported
in Zhang et al. 2025). Specifically, results for LLaMA-60M on OpenWebTex.
from Zhang et al. (2025). For experiments not covered in that study, we set r = 0.25 for the BRF initialization of CHT's, it was reported to yield the highest win rate.
we use the same settings as those provided in CoLA: Apart from the hyperparameters in Table B
the official code release.
SLTrain: The coefficient a that controls the contribution of the low-rank branch is set to 32 for LLaMA-60M and 16 for LLaMA-130M, following Han et al. (2024). We also found SLTrain to be highly sensitive to the sparsity-configuratinn (i.e., the allocation of parameters between branches) under total sparsities of [0.9, 0.8, 0.7]. To provide reliable results and fair comparison, searched configurations with a step size of 0.05 sparsity (corresponding to rank steps of 16 for LLaMA-60M and 24 for 1LLMMA-13MM) The best configurations are summarized in Table
CHTsL: We employ CHTs with BRF initialization and set = O. The alignment loss coefficient is set to 0.5 for LLaMA-60M on WebneetText and LLaMA-130M and 0.3 for LLaMA-60M on C4. The carrsit--oofiguraati configurations consistently converge to 1:1 allocation between the two branches on WebTex, and 0.95 on C4. A full summary of the best configurations is provided in Table 5
is tuned with a step size of 0.05. As shown in Section 5.3 the best
14

---

Published as a conference paper at ICLR 2026
Table 4: The best consstty-onfiguuaatinn for SLTrain under different total sparsity. Stotal refers total sparsity, refers sparsity in the connectivity sparse branch, refers to the rank in low- rank branch. The last column reports the proportion of parameters in connectivity sparse branch compared with spectral sparse (low-rank) branch.
Dataset
Model
Stotal
dconnectivity dspectral
0.9
0.95
16
1:1
LLaMA-60M
0.8
0.9
32
1:1
0.7
0.85
48
1:1
Open WehText
0.9
0.95
24
LLaMA-130M
0.8
0.85
24
3:1
0.7
0.85
72
1:1
0.9
0.95
1:1
LLaMA-60M
0.8
0.9
32
1:1
0.7
0.9
64
1:2
C4
0.9
0.95
24
1:1
LLaMA-130M
0.8
0.95
72
1:3
0.7
0.85
72
1:1
for CHTsL under different total sparsity. Stotel refers total sparsity, refers sparsity in the connectivity sparse branch, refers to the rank in low- rank branch. The last column reports the proportion of parameters in connectivity sparse branch compared with spectral sparse (low-rank) branch.
Table 5: The best
Sparsity- Cosisity-Confiuuattionn
Dataset
Model
Stotal
deonectivstty depectral
0.9
0.95
16
1:1
LLaMA-60M
0.8
0.9
32
1:1
0.7
0.85
48
1:1
Open Web
0.9
0.95
24
1:1
LLaMA-130M
0.8
0.9
48
1:1
0.7
0.85
72
1:1
0.9
0.95
16
1:1
LLaMA-60M
0.8
0.95
48
1:3
0.7
0.95
80
1:5
C4
0.9
0.95
24
1:1
LLaMA-130M
0.8
0.95
72
1:3
0.7
0.95
120
1:5
15

---

Published as a conference paper at ICLR 2026
C EASED CANCELLATION EFFECT UNDER ALIGNMENT-ENHANCEE INTEGRATION
In this section, present the OCR curves of different integration schemes across various total spar- sity levels for LLaMA-60M on Open WebText, as a supplement to Section and the OCR curves under total sparsity levels of 0.8 and 0.7, respectively, where the sparsity configu- ration is constrained such that the two branches contain the same number of trainable parameters. These results correspond the second and third rows of Table respectively.
D USAGE OF LLM
In this work, Large Language Model (LLM) is primarily used to assist with tasks such text refine- ment, summarization, and improving the clarity and readability of the manuscript. The LLM helps streamline writing and editing, ensuring that technical content is clearly and accurately presented.
16

---

Published as a conference paper at ICLR 2026
Layer @ proj
Layer 0.self attn.k proj
Layer 0.self, attn.v..vvv... prrr
Layer _ www.antntto..oo p_roo
0.6
0.6
0.4
0.4
8
0.4
0.4
0.2
0.2
0.2
0.2
0.0
2500 5000
19000
2500
10000
5000
7500 10000
7500 10000
Thair Steps
Trai Steps
Train Steps
Thain Steps
Layer 1.self attn.q, proj
Layer 1.self attn.k proj
Layer 1 self attn.v ppro
Layer 1.self_attn.o. wew___ttttttttgee proj
0.6
0.4
0.4
0.4
8 0.2
0.2
0.
0.2
0.0
0.0
0.0
0.0
2500 5000
7500 10000
2500 5000
7500 10000
2500
5000
7500 10000
2500
5000
7500 10000
Train Steps
Train Steps
Train Steps
Train Steps
Layer 2.self attniq proj
Layer 2.self attn.k proj
Layer self attn
Layer 2.self att.__ttttttttme
0.6
0.4
0.4-
8
0.2-
0.
0.25
0.0
0.0
0.0
0.00
2500
5000
7500 10000
2500
5000
7500 10000
2500
5000
7500 10000
5000
7500 10000
Train
Train
Train
Train
Layer 3.self attniq.ppt g. p.oo
Layer 3.self, attnikk proj
Layer 3. selef wwt.atttttee.e v_proj
Layer 3.self. _.self_tttn.o att.n.ttttoo.o proj
0.6
85
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
2500 5000
10000
2500
10000
10000
10000
Tran Stops
Tran Stops
Train Stops
Train Stops
Layer 4.self, attn.q. proj
Layer 4.self attn.k proj
Layer 4. 4.self, attn.v. proj
Layer 4.self www.aattttttttoo proj
0.6
0.4
0.4
08
0.2
0.2
0.2
0.2
0.0
0.0
0.0
2500 5000 7500 10000
2500 5000
7500 10000
25000
5000
7500 10000
2500
5000
7500 10000
Train Steps
Train Steps
Train Steps
Train Steps
Layer S.self. attriq, proj
Layer S.self attn.k proj
self attn.v _itnvv_prjj
Layer S.self attn.o,
0.4
0.4
0.4
0.4
0.2
0.0
0.0
0.0
0.0
2500
5000
7500 10000
2500
5000
7500 10000
2500
5000
7500 10000
2500
5000
7500 10000
Train
Train
Train
Layer 6.self. attnaq_ppt q_proj
Layer 6.self. attinkk proj
Layer selff attn.vv.tee v_proj
Layer 6.self attn.o proj
0.6
0.4
0.4
0.6
0.4
8
0.4
0.2
0.
0.2
0.0
0.0
2500 5000
10000
10000
7500 10000
5000 7500 10000
Train Steps
Thin Stope
Train Stops
Train Steps
Layer 7.self, attn-q. proj
Layer 7.self attn.kttnnnnk. proj
Layer 7. selef attn. fict-v__rrr proj
Layer 7.self wwt.aattttttrtoo proj
0.6
0.4
85
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
2500 5000
7500 10000
2500 5000
7500 10000
2500
5000
7500 10000
2500
5000
7500 10000
Thein Steps
Train Staps
Thain Steps
Train Steps
Nove
Act
ActeAlign
Figure 4: The layer-wise OCR plot of LLaMA60M on WebTeet with a total sparsity of 0.8, with connitttccoiiiuutttionn s 0.9, r 32, deonnectivity dapectral 1 1. Each subplot in the figure reports the changes of OCR over training steps. The plot is based on the experiment of the second row of Table I For space limit, we report here the self-attention layers in the model, where each column refers to Q K, V. O respectively.
17

---

Published as a conference paper at ICLR 2026
Layer _. atto.._proj
Layer 0.self attn.kk proj
Layer 0.self attn..v proj
Layer 0.self atw..oonneeee. p_rro
0.4
0.4
0.6
8
0.2
0.2
0.-
7500 10000
2500
7500 10000
2500
7500 10000
2300
50.00
7500
10000
Than Steps
Train Steps
Train Steps
Train Steps
Layer 1.self attn.ttttrppt proj
Layer 1. 1.self attn k_ proj
Layer 1.self attn.yy proj
Layer 1.self attn.o proj
8 0.2
8 0.2
8
0.4
0.0
0.0
0.0
0.0
2500 5000
7500 10000
0
2500 5000
7500 10000
25000
5000
7500 10000
2500
50000
7500 10000
Train Steps
Train Steps
Train Steps
Train Steps
Layer 2.self attn.o proj
Layer selel attn. proj
Layer 2.self attn.vv proj
Layer 2. ottnnoo proj
0.4
0.4
0.5
8 0.2
8
02
0.2
0.0
0.0
0.0
0.0
2500
5000
7500 10000
2500
5000
7500 10000
2500
5000
7500 10000
2500
50.00
7500 10000
Train
Train
Train
Layer 3.self attr.tt_preee proj
Layer 3. sese attn.kk proj
Layer 3.self attnivv proj
Layer 3. sele attn.o.nnnn o_proj
0.4
0.4
0.4
15
0.22
0.2
0.2
0.2
7500 10000
2500
7500 10000
2500
7500 10000
2500
10000
Than Steps
Train Steps
Train Steps
Train Steps
Layer 4.self_ attnntt_trrtt proj
Layer 4. d.self attn.k.re proj
Layer 4.self attn.yy proj
Layer 4.self, attn.o proj
00
08 0.-
0.2
0.2
0.0
0.0
0.0
0.0
2500
5000 7500 10000
2500
5000
7500 10000
2500
5000
7500 10000
2500
50000
7500 10000
Thain Steps
Train Steps
Train Steps
Train Steps
Layer S.self attn.a proj
Layer salf attn. K_rroj
Layer S.self attn.v pro)
Layer 5. salf attn.o, pro)
0.4
0.4
88
02
0.0
0.0
0.0
0,0
2500
5000
7500 10000
0
2500
5000
7500 10000
2500
5000
7500 10000
2500
50.00
7500 10000
Train
Train
Train
Layer 6.self_ aattnnttttrraa
Layer 6.self, attn.kttttttt proj
Layer 6.self 6.selt_attn.v aattnattttrrer __ro
Layer 6.self_ wtw.faatttooo..o. aatnn.oppor o_proj
0.4
0.4
0.4
0.2
02-
0.2
7500 10000
2500
7500 10000
2500
7500 10000
2500
50.00
10000
Stegs
Train Steps
Tain Steps
Train Sleps
Layer 7.self. attnntt_prpptt proj
Layer 7.self, attn.k k. proj
Layer 7.self attn.vt.tvn..... proj
Layer er 7.self. attn.o. o. proj
0.6
0.5
0.4
0.4
08
0.2
0.2
0.2
0.2
0.0
0.0
9.0
0.0
0
2500 5000
7500 10000
0
2500
5000
7500 18000
0
2500
5000
7500 16000
2500
5000
7500
10000
Train Steps
Thain Steps
Train Steps
Train Steps
Nove
Act
ActeAlign
Figure 5: The layer-wise OCR plot of LLaMA60M on WebTeet with a total sparsity of 0.7, with cannity-configuratiio s = 0.85, r 48, deonnectivityy dsspcttaa = 1 1. Each subplot in the figure reports the changes of OCR over training steps. The plot is based on the experiment of the third row of Table [] For space limit, report here the self-attention layers in the model, where each column refers to Q K. V, O respectively.
18

---

Published as a conference paper at ICLR 2026
E ALIGNMENT SCHEME ON STATIC SPARSE TRAINING WITH LOW-RANK TRAINING
To further validate the effectiveness of the proposed alignment-enhanced integration scheme, ad- ditionally evaluate it in the "static sparse + low-rank" setting by comparing models trained with and without alignment. reports results LLaMA-130M across multiple datasets and total spar- sity levels, under the constraint that the connectivity-spet decervativty
and low-rank components occupy the same number of parameters dyecra 1). Statistical significance is confirmed using the Wilcoxon signed-rank test. With p-value < 0.05, the alignment-enhanced model achieves significantly better performance than the naive integration baseline.
Table 6: Comparison between different integration strategies for "Static Low-rank" Com- bination. The table consists two parts: a. The performance of different integration strate- gies, reported in terms of validation perplexity (PPL.). The Naive strategy corresponds simple sum of static sparse and low-rank factorization. The Act strategy applies activation adjustment to the low-rank factorization branch. The AcI+Align strategy combines activation adjustment with the alignment loss. The coefficient of the alignment loss À is 0.3. The sparsity configuration set such that the sparse branch and the low-rank branch have the same number trainable parameters( degencativity = 1). b. The Wilcoxon signed- rank test p-values, which indicate whether depectral
the differences in performance between strategies are statistically significant.
Model
Dataset Total Sparsity
Naive
Act
Act+Align
0.9
31.52
25.44
25.41
openwebtext
0.8
22.44
22.37
22.36
0.7
21.25
41.51
20.97
LLaMA-130M
0.9
31.49
31.62
31.44
c4
0.8
28.21
28.35
28.12
0.7
26.90
26.52
26.36
Wilcoxon signed-rank
against Naive
0.03125
p-value
against Act
0.03125
19

---

Published as a conference paper at ICLR 2026
F OCR AND GLOBAL COSINE SIMILARITY
To better understand the cancellation phenomenon between the sparse and low-rank branches, we compare the proposed Overlap Cancellation Ratio (OCR) with commonly used global cosine simi- larity. While these metrics are related, they capture fundamentally different aspects of cancellation.
F.1 CONCEPTUAL DIFFERENCE
Global cosine similarity measures only directional alignment between vectors but ignores magni- tude, which is crucial for assessing the severity of cancellation. In contrast, OCR explicitly quanti- fies the fraction of overlapping magnitude that is canceled due to opposite signs.
This distinction be illustrated by the following examples, both with cosine similarity equal to 0:
Example A (strong cancellation): S [10, -10]. L [10, 10). Then S L [20, 0],
indicating half cancellation. OCR = 0.5 (iignoring the small in the denominator).
Example B (minimal cancellation): [100, 0). [0. 100]. Then S [100, 100).
indicating almost cancellation. OCR 0.
These examples demonstrate that while cosine similarity may indicate similar opposite directions, does not capture the magnitude of signal lost. OCR complements cosine similarity by explicitly measuring this magnitude-based cancellation.
F.2 EMPIRICAL OBSERVATION WITH COSINE SIMILARITY
To provide thorough observation, we include here cosine similarity plots corresponding to the training of LLaMA-60M on OpenWebTen under different total sparsities (0.9, 0.8, 0.7) as a sup- plement. The settings are exactly the same as the reported in Section and Section [ where OCR plots are reported. Figures and [ show the evolution of cosine similarity between the outputs of the sparse and low-rank branches during training.
These curves demonstrate that the alignment-enhancee training scheme increases directional align- ment between the two branches, with notably higher cosine similarity observed in the Q and K layers. This observation complements the OCR measure, confirming that the alignment loss not only reduces magnitude-basee based cancellation but also improves global directional alignment.
20

---

Published as a conference paper at ICLR 2026
Layer 0,self attn.q_projj
Layer 0,self aat.atttttrrrrr proo
Layer _.self_attn.v p_roj
attn.
Layer attno proj
0.4
0.4
0.4
8
0.2
0.2
0.2
0.0
05.0
0.0
10000
5000
5000
10000
5000
10000
Steps
Steps
Steps
Steps
Layer 1.self wtt..ttttrrrr proj
Layer 1.self attn.k, proj
Layer sele attn. proj
Layer 1seef attn.o__ttrr.oo prooj
0.6
0.4
0.4
0.1
0.4
0.2
0.0
0.0
0.0
5000
10000
5000
10000
5000
10000
5000
10000
Train Staps
Train Stops
Train Staps
Staps
Layer 2.self www.ttttttt..to.
Layer 2.self attn.kk proj
Layer self
Layer 2.self attn.o proj
0.6
0.1
0.6
0.4
0.2
0.4
0.2
0.2
0.1
0.2
0.0
0.0
0.0
0.0
0
5000
10000
0
5000
10000
5000
10000
5000
10000
Train Slaps
Train Sleps
Truin Simps
Train Steps
Layer 3.solf attnn._prr)
Layer 3.self attn.kkppp... proo
Layer 3.sclf attn. v p_rr
Layer 3.self 3.stlf_attt.ooo___.... wwt.ttt_tttrroooc pproj
0.4
0.6
00
0.4
8
0.4
0.2
0.4
0.2
0.2
0.0
0.0
0.0
5000
10000
5000
10000
5000
10000
5000
10000
Steps
Steps
Steps
Steps
Layer 4.solf 4,self_attn. prar
Layer 4.self att.nkk_ttpn. pro)
Layer 4. 4.self, attn v v_proj
Layer 4.salf. 4.self_attn.o.ooommmm www.tttttt__no prrej
0.6
0.4
0.6
0.4
0
0.4
0.2
0.2
0.2
0.2
0.0
0.0
5000
10000
5000
10000
5000
10000
5000
10000
Train Steps
Train Steps
Train Steps
Train Steps
Layer seleff attn.qt_tnn
Layer 5.self attn.k. proj
Layer 5 self attn.
Layer 5.self attn.o proi
0.4
0.6
0.50
8
0.50
0.4
0.25
0.25
n.2
0.2
0000
0.00
0.0
0.0
5000
10000
5000
10000
5000
10000
5000
10000
Train Steps
Train Steps
Train Steps
Train Steps
Layer attn.q_proj
Layer attn.k proj
Layer 6.self attn. v proj
Layer 6.self attn.o proj
0.75
0.75
0.4
0.6
0.50
0.50
0.4
0.25
0.2
0.2
0.00
0.00
0.0
0.0
5000
10000
5000
10000
5000
10000
5000
10000
Train
Steps
Train
Layer 7.self_ aatt.ntttrraat
Layer 7.self att..k_ttto.o proj
Layer 7.self, attn v. v_pro
Layer 7.self, attn.o 7.self_attn.o p_rro
0.75
0.75
0.6
0.4
0.50
0.50
0.25
0.25
0.2
0.2
0.00
0.c
0.0
5000
5000 Train Steps
10000
5000
10000
Train Steps
Train Steps
Train Steps
Act4Align
Figure 6: The layer-wise cosine similarity plot of LLaMA60M on OpenWebText with total sparsity of 0.9, with pparsity-configuraee Each subplot in the figure reports the changes of OCR over training steps. The plot is based on the experiment of the first row of Table I For space limit, we report here the self-attention layers in the model, where each column refers to Q K, V, O respectively.
= 0.95, r = 16, deornectivity : depectral = 1 : 1.
21

---

Published as a conference paper at ICLR 2026
Layer 0,self attn.q_proj
Layer 0. atttaktttrrrrr proj
Layer 0. _ p_rej
attn.
Layer attno proj
0.4
0.4
0.0
0.2
8
0.2
0.0
10000
5000
10000
5000
10000
Steps
Steps
Steps
Steps
Layer 1.self wwt.tttttp..n proj
Layer 1.self attn.k proj
Layer seself attn. v prej
Layer 1.self wwwntttttttaeeeoeee p_ooj
0.4
8
0.2
8 0.2
0.4
0.1
0.2
0.0
0.0
0.0
5000
10000
5000
10000
5000
10000
5000
10000
Train Staps
Train Stops
Train Staps
Train TraisStaps
Layer 2ele www.ttttttt.ttoo
Layer 2.self attn.kk.pr... proj
Layer self
Layer 2.self attn.o pro
0.4
0.4
0.2
0.25
0.2
0.0
0.00
0.0
-6.25
5000
10000
0
5000
10000
5000 Train Simps
10000
5000
10000
Train Slaps
Train Sleps
Train Steps
Layer 3.solf attnn.__rra
Layer 3.self attn.kk_tnn.. proj
Layer 3.sclf attn. v v_proj
Layer 3.self attn.o 3.self_attntooo___. .rro
0.6
0.6
0.4
0.6
0.4
0.4
0.2
0.4
0.2
0.2
0.0
0.0
0.0
5000
10000
5000
10000
5000
10000
5000
10000
Steps
Steps
Steps
Steps
Layer 4,solf 4,self_attnn.a_ .tttt.g_praj
Layer 4,self attn.kkttt...m. pro)
Layer 4. 4.self, attn v v_proj
Layer 4.salf. www.ttttttooooo 4.stlf_attn.o..ooommm p_rre
0.75
0.75
0.4
0.50
0.50
0.4
0.2
0.25
0.25
0.0
0.0
5000
10000
5000
10000
5000
10000
5000
10000
Train Steps
Train Steps
Train Steps
Train Steps
Layer 3.self_ attnttttttnnn
Layer 5.self attn.k, proj
Layer 5 self attn.
Layer S.self attn.o proj
0.75
0.75
0.4
0.50
8
3 0.2
0.4
0.25
0.25
0.2
0.0
0.00
0.00
0.0
5000
10000
5000
10000
5000
10000
5000
10000
Train Steps
Train Steps
Train Steps
Train Steps
Layer 6.self attn.q_proi
Layer attn.k proj
Layer 6.self attn. v proj
Layer 6.self attn.o proj
0.75
0.75
0.4
0.6
0.50
8
0.50
0.2
0.4
0.25
0.0
0.0
0.00
0.00
-0.2
5000
10000
5000
10000
5000
10000
5000
10000
Train
Steps
Train
Layer 7.self_ attn.ntttrraat
Layer 7.self att..kttttoooo p_rj
Layer 7.self, attn v. ._roj
Layer 7.self. attn.o 7.self_attn.o p_rro
0.4
0.75
0.50
0.50
0.2
0.4
0.25
0.25
0.2
0.00
0.00
0.0
5000
10000
5000
10000
5000
L0000
Train Steps
Train Steps
Train Steps
Train Steps
Act4Align
The layer-wise cosine similarity plot of LLaMA60M on OpenWebText with total sparsity of 0.8, with Each subplot in the figure reports the changes of OCR over training steps. The plot is based on the experiment of the second row of [T] For space limit, we report here the self-attention layers in the model, where each column refers to Q K, V. O respectively.
Figure 7:
s = 0.9. = 32, deonnectivity : dipectral = 1 : 1.
22

---

Published as a conference paper at ICLR 2026
Layer 0,self atton._proo
Layer 0,self att.nntttttrrtnm proj
Layer _.self_attn.v
Layer attno proj
0.4
0.05
8
8
8 0.00
0.2
0.2
0.0
5000
10000
5000
10000
Steps
Steps
Steps
Steps
Layer 1.self wwt.tttttt... proj
Layer 1.self attn.kk proj
Layer 1eself attn. prej
Layer 1seef awwn.otttttrroooe prooj
8 0.2
8
0.4
8
0.2
0.25
0.00
0.0
-0.25
5000
10000
5000
10000
5000
10000
5000
10000
Train Staps
Train Steps
Train Staps
TrainSteps
sele www.ttttttt..too
Layer 2.self kttnnkk.p jrrj
Layer self
Layer 2.self attn. proj
0.4
0.6
0.4
0.4
0.00
0.0
0.0
0.0
-0.25
5000
10000
5000
10000
5000 Truin Simps
10000
5000
10000
Train Slaps
Train Sleps
Train Steps
Layer 3.solf attnn.__rr)
Layer 3.self, attn.kk prro
Layer 3.self attn. v_proj
Layer 3.self attn.o .rro
0.75
0.4
0.50
0.50
0.4
0.25
0.25
0.2
0.2
0.00
0.00
0.0
5000
10000
5000
10000
5000
10000
5000
L0000
Steps
Steps
Steps
Steps
Layer 4,self 4,self_attn.g_ p_rrn.gtptaaaa
Layer 4.self wtt.nkttttpnnnm pro)
Layer 4. 4.self attn v v_rroj
Layer 4.self www.ttt.ttoooo p_pra
0.75
0.75
0.4
0.4
8
0.25
0.25
0.2
0.0
0.0
5000
10000
5000
10000
5000
10000
5000
10000
Train Steps
Train Steps
Train Steps
Train Steps
Layer 5.self attniqt_ttnnn
Layer S.self attn.k, proj
Layer 5 self attn.
Layer 5.self attn.o proj
0.75
0.4
0.5
0.50
8
0.50
0.2
0.4
0.25
0.2
0.00
0.00
0.0
0.0
5000
10000
5000
10000
5000
10000
5000
10000
Train Steps
Train Steps
Train Steps
Train Steps
Layer attn.q_proj
Layer attn.k proj
Layer 6.self attn. v proj
Layer 6.self attn.o proj
0.75
0.75
0.6
0.50
8 0.50
0.4
0.25
0.25
0.2
0.2
0.00
0.00
0.0
5000
10000
5000
10000
5000
10000
5000
10000
Train
Steps
Train
Train
Layer 7.self_ attn.ttttraaaa
Layer 7.self att..k_tttraaao proj
Layer 7.self, attn v y_rroj
Layer 7.self attn. 7.self_attn.e
.rroj
0.4
0.75
0.75
0,50
0.50
6.25
0.25
0.25
0.0
0.00
5000
10000
5000
10000
5000
L0000
Train Steps
Train Steps
Train Steps
Train Steps
Act4Align
The layer-wise cosine similarity plot of LLaMA60M on OpenWebText with total sparsity of 0.7, with caasstt-confiuurration = 0.85, r = 48, deonnectivity : depectral = 1 : 1. Each subplot in the figure reports the changes of OCR over training steps. The plot is based on the experiment of the third row of Table [ For space limit, we report here the self-attention layers in the model, where each column refers to Q K, V. O respectively.
Figure 8:
23

---

Published as a conference paper at ICLR 2026
G FURTHER DISCUSSION: APPLYING ALIGNMENT LOSS ONLY TO Q,K LAYERS
According the OCR plots, when applying the alignment loss all linear layers, the cancellation effect is primarily mitigated in the Q and K layers rather than uniformly across all layers. This observation motivates more efficient approach: applying the alignment loss exclusively to the Q and K layers.
To investigate this, we conducted experiments with CHTsL applying alignment loss only Q and K layers (Align_gk) and only except Q/K layers (Align,others), compared with those obtained by ap- plying alignment loss all layers (Align all), using exactly the same best hyperparameter settings as summarized in Section B The results, presented in Table demonstrate that restricting the align- ment loss Q and K layers yields similar or even improved performance compared to applying to all layers. Align Q/K layers significantly outperforms align other linear layers, with the Wilcoxon signed-rank p-value lower than 0.05.
These findings further validate that OCR accurately captures the cancellation effect and highlight that the alignment loss can be implemented more efficiently, achieving comparable or better results with reduced computational cost. Morcover, this observation provides useful perspective for future work building upon this study, suggesting that targeted alignment may be sufficient to achieve strong performance while saving computation.
Table Validation perplexity of models based on alignment different components. Align qk refers to CHTsL with alignment only to Q, K layers, while Alignaal refers to the original CHTsL with alignment all linear layers. Validation perplexity (PPL.) is reported this table for different methods on different datasets under the same constraint of total sparsity Stotal Bold values are the best performance.
Model
Dataset
Total Sparsity
Align gk
Align other
Align all
0.9
32.012
32.224
31.772
OpenWebText
0.8
29.066
29.353
29.109
0.7
27.279
27.802
27.400
LLaMA-60M
0.9
39.376
39.713
39.291
C4
0.8
35.860
36.281
35.949
0.7
34.194
34.437
34.191
0.9
24,251
24.400
24.071
DeenWetText
0.8
21.878
21.772
21.866
0.7
20.690
26.655
20.648
LLaMA-130M
0.9
30.135
30.454
30.034
C4
0.8
27.567
27.798
27.593
0.7
26.143
38.250
26.190
Average score
29.04
30.76
29.01
Win Rate against
0.42
0.08
0.5
signed-rank
Align.ql Align other
0.00098
0.62207
p-value
against
0.00098
24

---

Published as a conference paper at ICLR 2026
H RESULTS ON MODELS WITH LARGER SIZE
We conducted experiments on the larger LLaMA-3500 and LLaMA-IB on Open Web'ext to further evaluate the effectiveness of CHTsL L at scale. Due limitations in time and computational resources, we selected SLTrain, CHTs and CoLA as the most competitive sparse training baselines. Also, we provide the performance of dense model for reference.
The common hyperparameter settings for all methods are listed in Table 8 All hybrid methods, including CHTsL and SLTrain, use a sparsity configuration of deonnectivity dspectral 1
For CHTsL, the coefficient À which controls the contribution of alignment loss is set to be 0.5.
For SLTrain, the coefficient a controlling the contribution of the low-rank branch is set to 16 for LLaMA-350M and 8 for LLaMA- IB, following Han et al. (2024).
For CHTs under sparsity level 0.7 on LLaMA-1B, we directly imported the result reported Zhang et al. (2025). with learning-rate 3e-3.
For dense model, since the number of trainable parameters is different from the sparse training methods, we applied smaller learning rate following previous literature Zhang et al. (2025). For LLaMA-350M, the adopted learning rate is Ie-3, while for LLaMA-IB, we directly imported the result reported in Zhang et al. (2025) with learning rate 4e-4.
The results are reported Table 9 They show that CHTsl. consistently achieves lower perplexity across different total sparsity levels compared to the baselines on L.aMA-350M, On (LaMA-1BB, CHTsL outperforms all other methods at the 0.9 sparsity level, where the benefits of our approach be- come particularly pronounced, while at the 0.7 sparsity level it remains competitive, though slightly below CoLA. Overall, CHTsL demonstrates robust performance on large-scale models, underscor- ing its scalability and its strong advantage especially under higher sparsity regimes.
Table 8: Common hyperparameter settings for experiments LLaMA-350M and LLaMA-1B.
The settings align with previous research.
Hyperparameter
LLaMA-350M
LLaMA-1B
Embedding Dimension
1024
2468
Feed-forward Dimension
2736
5461
Global Batch Size
512
512
Sequence Length
256
256
Training Steps
60000
100000
Warmup Steps
6000
10000
Learning Rate
3e-3
le-3
Optimizer
Adam
Adam
Layer Number
Head Number
16
32
Iterative warmup steps
Update Interval for DST
100
100
Table 9: Validation perplexity of different methods on L.LMAA-50M,, Validation perplexity (PPL.)) reported in this table for different methods different datasets under the constraint of total sparsity Stotal- Bold values are the best performance out of all sparse methods.
LLaMA-350M
LLaMA-1B
Dataset
Method
$_total=0.9
s_total=0.8
s_total=0.7
$_total=0.9
s_total=0.7
Dense
14.90
14.62
CHTs
19.69
17.82
17.88
17.35
14.53
WebTeet
CoLA
20.92
17.60
16.13
16.03
13.08
SLTrain
18.99
16.88
15.98
16.00
14.58
CHTsL
18.40
16.54
15.86
15.16
13.31
25

---

Published as a conference paper at ICLR 2026
I ABLATION TEST FOR ACTIVATION FUNCTION
As an ablation study, we conducted experiments on the low-rank branch using different activation functions, including ReLU and GeLU, in comparison with SiLU. The hyperparameter settings were kept the same as those reported in Section B
The results, presented in Table 10 show that CHTsL with SiLU activation in the low-rank branch outperforms the alternatives in most cases.
Table 10: Validation perplexity of CHTsL based on different activation function. SiLU is the default one used the main text. Validation perplexity (PPLI) reported in this table for different methods on different datasets under the same constraint of total sparsity Stotel. Bold values are the best performance.
Model
Dataset
Total Sparsity
ReLU
GeLU
SiLU
0.9
32.431
32.081
31.772
OpenWebText
0.8
29.607
29.280
29.109
0.7
28.125
35.625
27.400
LLaMA-60M
0,9
39.930
39.364
39.291
C4
0.8
36.871
36.045
35.949
0.7
35.251
34.157
34.191
0.9
24.530
24.049
24.071
Open WebText
0.8
22.219
21.999
21.866
0.7
20.999
20.789
20.648
LLaMA-130M
0.9
30.639
30.171
30.034
C4
0.8
28.252
27.762
27.593
0.7
27.044
26.192
26.190
26

---

Published as a conference paper at ICLR 2026
J ZERO-SHOT EVALUATION
To further evaluate the generality of CHTsL, we assessed the trained models on downstream datasets from GLUE and SuperGLUE. We compared CHTsL with the strongest sparse training baselines, SLTrain and CHTs, as well as with the dense model. All sparse training methods were evaluated under total sparsity 0.9, with corresponding hyperparameter settings listed in Section Exper- iments were conducted using the 1m-eval package, and accuracy (Acc) is reported.
The results, presented in Table [I] show that CHTsL achieves the highest win rate among the sparse training baselines and also outperforms the dense model. These findings further demonstrate the generality and effectiveness of CHTsL across downstream tasks.
Table 11: Zero-shot results on downstream tasks. CHTsL, SLTrain, and CHTs are evaluated under total sparsity of 0.9. Results reported in terms of accuracy (Acc), with the best rest-performing value in each row highlighted in bold. Note that two or more methods achieve the same accuracy, all corresponding values bolded and counted toward the win rate.
Model
Pretrain
Downstream
CHTsLL
SLTrain
CHTs
Dense
CoLA
0,5292
0.6894
0.6913
0.6913
Copa
0.5300
0.5700
0.5700
0.5200
Hellaswag
0.2653
0.2649
0.2663
0.2619
MNLI
0.3278
0.3310
0.3290
0.3282
OpenWebTeds
MRPC
0.3235
0,3995
0.6838
0.6789
QNLI
0.4955
0.4935
0.4944
0.4946
QQP
0.4126
0.3688
0.3682
0.3682
RTE
0.5235
0.5126
0.4838
0.5018
SST-2
0.5482
0.4908
0,4908
0.4920
LLaMA-60M
CoLA
0.6913
0.6903
0.4276
0.6846
Copa
0.4800
0.5200
0.5400
0.4500
Hellaswag
0.2666
0.2644
0.2714
0.2656
MNLI
0.3255
0.3340
0.3291
0.3281
C4
МRPC
0.6544
0.6495
0.6324
0.6740
QNLI
0.4944
0.4915
0,4926
0.4939
QQP
0.3682
0.3727
0.3692
0.3730
RTE
0.5487
0.4874
0.5271
0.5162
SST-2
0.4908
0.4931
0.4908
0.4908
CoLA
0.6606
0.6568
0.6913
0.6913
Copa
0,5700
0.5600
0.5600
0.6000
Hellaswag
0.2687
0.2656
0.2678
0.2699
MNLI
0.3254
0.3320
0.3275
0.3272
Open WebText
MRPC
0.5662
0.3775
0.6814
0.6740
QNLI
0.4961
0,4950
0,4941
0.4946
QQP
0.4015
0.3762
0.3682
0.3717
RTE
0.5018
0.4477
0.4585
0.4765
SST-2
0,5161
0.4920
0.4908
0.4908
LLaMA-130M
CoLA
0.6002
0.6443
0.6903
0.6903
Copa
0.4700
0.4900
0.5300
0.5700
Hellaswag
0.2713
0.2694
0.2701
0.2770
MNLI
0.3282
0.3274
0.3276
0.3274
C4
MRPC
0.5686
0.3211
0.4412
0.5074
QNLI
0.5041
0.5003
0.4946
0.4946
QQP
0.3720
0.4326
0.3683
0.3682
RTE
0.5199
0.5162
0.4946
0.5018
SST-2
0.4908
0.4920
0.4908
0.4908
Win Rate
0.4167
0.1944
0.25
0.25
27

---

Published as a conference paper at ICLR 2026
K INFERENCE MEMORY AND THROUGHPUT
In this section, we report the inference memory usage and throughput for CHTsL, SLTrain, and the dense baseline. For CHTsL and SLTrain, the sparsity configuration was set deonnectivity depectral = 1 1, under a total sparsity 0.9. Each model was run for 5000 inference steps with dummy inputs of batch size 128 and sequence length 256. We record the maximum memory usage using torch cuda. max www.rmmmm....... Lallllloo www.llaaeeeeeee. ocated ed (in GB) and measure the average throughput (Tokens/sec). Experiments were conducted on single NVIDIA A100-80GB GPU.
The inference memory and throughput of CHTsL are theoretically identical to those of SLTrain, as the two differ only in the training procedure while sharing the same inference-stage architecture. Both methods gain efficiency from:
sparse matrix multiplication in the connectivity-sparse branch
low-rank multiplication the spectral-sparse branch.
For accurate inference benchmarking, we evaluated the trained CHTsL checkpoints using the SLTrain C++ codebase. For CHTsL, two minimal modifications were made: (1) adding the ac- tivation function required by the apectral-sparse branch, and (2) correcting the computation order in the low-rank branch, which in the original SLTrain implementation computed B@A and then B@A@X X and thus introduced redundant operations.
Table 12 shows that CHTsl. achieves both lowest memory usage and highest throughput out of three models. The advantage throughput arises from the corrected low-rank computation order, mentioned above in the second modification. This demonstrates that CHTsL already provides practical efficiency benefits while holding even greater theoretical potential.
Finally, we emphasize that neither CHTsL nor SLTrain can fully realize their theoretical speed-ups due to current software and hardware limitations. Py does not provide efficient kernels for un- structured sparsity, and modern GPUs offer minimal acceleration for unstructured sparse operations. Thus, all unstructured sparse methods are currently operating below their theoretical limits.
Overall, CHTsL outperforms the dense baseline in both memory and speed, and its efficiency ad- vantage is likely to increase further as frameworks and hardware improve support for unstructured sparsity.
Table 12: Inference memory and throughput of different methods. For each model, inference was conducted for 5000 steps, with maximum memory and average throughput reported. Experi- ments are conducted on x NVIDIA A100-80GB, with dummy input of batch size 128 and sequence length 256.
Model
Method
Memory (GB)
Throughput (Tokens/s)
Dense 2.606
773111
LLaMA-60M
SLTrain
2.573
749985
CHTsL
2.573
786004
Dense
3.392
343310
LLaMA-130M
SLTrain
3,278
337149
CHTsL
3.278
386483
28

---

Published as a conference paper at ICLR 2026
L GRADIENT LEVEL CANCELLATION
To further examine the cancellation effect at the gradient level, we analyze the input gradients of the two branches (i.e., the gradients of with respect to Wx and Bf(Ax)). Experiments were conducted on LLaMA-60M with Web under a total sparsity level of 0.9 and a sparsity configuration 1, corresponding the first row of Table
of deconnectviity dspectral
We plot the OCR curves between the imput gradients of the two branches. Figure 9 compares the curves of the Naive combination with those of the alignment-enhaneed integration. The results show that, at the gradient level, certain layers exhibit reduced cancellation effect (e.g., the 5th Q layer). However, the phenomenon is less pronounced than the output level, which expected since the alignment loss directly acts on the outputs rather than on the gradients.
29

---

Published as a conference paper at ICLR 2026
Layer 0,self p_ro
Layer 0.self attn.k proj
Layer 0.self attn..v proj
Layer 0. 0.self attn.o proj
0.6
0.4
0.4
8
0.4
0.2
0.2
0.2
7500
7500 10000
2500
10000
2500
5000
7500 10000
Staps
Train Stops
Train Stops
Train Stops
Layer 1.self attn.q_ppoo proj
Layer 1. 1.self attn k_ proj
Layer 1.self attn.y proj
Layer 1. self attn.o proj
0.4
0.4
00
66 0.2
0.2
0.2
0.0
2500
5000
7500 10000
2500
5000
7500 10000
2500
5000
7500 10000
2500
5000
7500 10000
Trai Steps
Train Stops
Train Staps
Train Stups
Layer 2.self q
self attn K_okk_Prrrr
Layer 2.self attn.y proj
Layer 2. sele ottn.o proj
0.4
0.4
0.4
0.4
OCR
0.0
0.0
0.0
0.0
25.00
5000
7500
10000
0
2500
So00
7500 10000
2500
5000
7500 103co
2500
7500 10000 Train Steps
5000
Trii Simps
Train
Train Staps
Layer 3,self q_tro
Layer 3. 3.self attn.kk proj
Layer 3.self attn.v_proj
Layer 3. self _e www.:attnnoooooo p_proj
0.4
0.4
0.4
85
0.2
0.2
0.2
0.2
5000
10000
7500 10000
2500
10000
7500 10000
Steps
Steps
Steps
Steps
Layer 4.self attn.q_projj
Layer d.self, attn.k, proj
Layer 4.self attnnty.tprn proj
Layer 4.self attn.o proj
05
0.2
0.2
8 0.2
0.2
2500
5000
7500 10000
0
2500
5000
7500 10000
2500
5000
7500 10000
2500
5000
7500 10000
Train Steps
Train Stops
Train Steps
Train Stops
Layer 5.self attn.q_
Layer self attn. k proj
Layer S.self attn.v projj
Layer 5 S.self attn.o proj
0.4
0.4
0.4
0.4
OCR
0.0
0.0
0.0
0.0
0
25.00
5000
7500 10000
0
2500
5000
7500 10000
0
2500
5000
7500 10000
2500
7500 10000 Train Steps
5000
Train Staps
Train Steps
Train Staps
Layer 6.self attn.q_praa
Layer 6.self attn.kk proj
Layer 6.self attn..v proj
Layer 6. s.leff o proj
0.4
0.4
0.2
0.2
0.0
5000
10000
2500
2500 10000
2500
7500 10300
2500
7500 10000
Layer 7.self. attnnttttapa proj
Layer 7.self attn.k K.n.k.Kp proj
Layer 7.self attn.yt.tvvnc... proj
Layer 7 7.self ottn.o_ p_roj
0.4
0.4
0.4
0.2
0.2
0.2
0.2
2580
5000
7500
10000
2500 5000
7500
10000
2500
5000
7500 10000
2500 5000
7500 10000
Train Steps
Train Stops
Train Stops
Stops
Noire
A1
Act4Align
Figure 9: The layer-wise gradient-level OCR plot of LLaMA60M on OpenWebText with a total sparsity of 0.9, with sparsity-coffigraaiin
= 0.95. r = 16, deonnectinity dapectraa = 1 : 1. Each subplot in the figure reports the changes of OCR over training steps. The plot is based on the experiment of the first row of Table I For space limit, we report here the self-attention layers in the model, where each column refers to Q. K, V. O respectively.
30

---

Published as a conference paper at ICLR 2026
M ALIGNMENT ON DIFFERENT COMBINATION OF CONNECTIVITY SPARSITY AND SPECTRAL SPARSITY
In this section, present results demonstrating how alignment works when combining different connectivity-based sparse training methods with low-rank training.
We conducted experiments using connectivity-based sparse training methods including static sparse training, SET, and CHTs, with different initialization strategies such as random and BSW in the work of Zhang al. (2025..
We compare naive integration with alignment-enhanced integration on LLaMA- 10MMM--130M using the OpenWebText dataset, under total sparsity level 0.9 with a sparsity configuration of deonnsetivity dspecral = 1: 1. The coefficient was chosen from 0.3 and 0.5.
Results in Table 13 show that the alignment-enhanneed training scheme consistently improves perfor- mance compared with naive integration. Moreover, for dynamic connectivity- sparse training method CHTs, BSW initialization outperforms random initialization, which is consistent with previ- ous literature and further validates the reliability of our results. These findings further confirms the generality the alignment training scheme on combining connectivity sparse training and spectral sparse training.
Table 13: Validation perplexity under different combination of connectivity sparsity and spec- tral sparsity. Naive refers to simple integration of connectivity sparsity. Act+Align refers to activa- tion and alignment-enhanced integration. Bold value refers the better performance considering different integration strategy.
DST
Initialization
Naive
Act+Align
random
22.44
22.36
Static
BSW
380.88
21.88
random
356.08
22.54
SET
BSW
22.55
22.22
random
22.46
22.20
CHTs
BSW
22.11
21.87
31

---

Published as a conference paper at ICLR 2026
N EASED CANCELLATION EFFECT ON LLAMA-350M
In this section, we present the OCR curves of LLaMA-3300 with and without the alignment training scheme. Due time constraints, we compare only the naive integration approach, which simply sums the outputs of the two branches, with the proposed alignment scheme, which leverages both the activation in the low-rank branch and an explicit alignment loss.
All experiments are conducted under an overall sparsity of 0.9, using the sparsity configuration 0.95, r 32, deonnectivity dspectral
1. The validation perplexity is reported Table [4] where the performance of the naive integration collapses. The OCR curves in Figure TO] during training show that, under the alignment training scheme, the OCR value decreases significantly, especially in the Q, K layers, as also observed in smaller models. In contrast, the OCR of the naive integration is highly unstable, which stems from the training collapse.
Table 14: Validation perplexity on LLaMA-350M under different integration strategy CHT's and low-rank trainin. Naive refers simple integration. Act+Allign refers to activation and alignment- enhanced integration. Bold value refers to the better performance considering different integration strategy.
Model
Dataset
Sparsity
Naive
Act+Align
LLaMA- -50M
Open
0.9
604.48
18,40
32

---

Published as a conference paper at ICLR 2026
Layer a.seil.attn.e att.ntttttppa proj
Layer 0.self attnnttttnnnnk _.teet_attnn.k proj
Layer 0. sele wwt.avtttttttoo v_proj
Layer 0.self att.anttttttrgge proj
0.6
0.4
8
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
20000
4c0on
60000
20000
40000
6:0000
20000
40000
60000
20000
40000
6c000
Train Stops
Train Stops
Trann Steps
Train Stops
Layer Lself attn.o, proj
Layer 1.self. attn.k proj
Layer 1 self attn.v v_proj
Layer 1.self attn.o proj
0.6
0.6
0.4
0.4
0.50
0.2
0.25
0.2
0.0
0.0
0.00
0.0
0
20000
40000
60000
20000
40000
60000
20000
40000
60000
20000
40000
60000
Train Stops
Trai Stats
Trann Stops
Train Staps
Layer 2.self attnn.. proj
Layer 2.self attn.k pro]
Layer 2.5off attn.v proj
Layer 2.salf attn.oo proj
0.75
0.6
0.4
0.4
0.50
0.4
0.2
0.c
0.0
0,00
0.0
20000
40000
60000
20000
40000
20000
40000
60000
20000
40000
60000
Train
Tran
Layer 3.self
3.selt_attn.o attnatttttpee proj
Layer 3.self a proj
Layer 3. self attn. v. proj
Layer 3,self attrn.o_proooo.ooooooo
0.75
0.4
0.2
n.2
0.2
20000
4co0n
60000
20000
40000
60000
200000
40000
800c0
20000
48000
6c000
Train Steps
Tran Staps
Train Steps
Train Steps
Layer 4 Self attrioo proj
Layer 4.self attnkk proj
Layer 4. self. attn.v proj
Layer 4.self attn.o proj
06
0.4
0.2
n.2
0.2
0.0
0.0
0.0
0.0
0
20000
4000n
n0000
20000
40000
60000
200000
40000
60000
20000
40000
6con0
Trann Stops
Train Stope
Train Steps
Train Stops
Layer S.self attn.o, proj
Layer 5.self attnkk proj
Layer self fallvvpprgg proj
Layer 5.self attn.o proj
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.9
0.0
0.0
20000
40000
60000
20000
40000
60000
20000
40000
60000
20000
40000
60000
Train
Train
Train
Layer a.ttlf.attn.q proj
attn.o
Layer 6.self attn.kkttee proj
Layer 6.self attn.v_proj
Layer 6.self attnn.o_pro
0.4
0.4
0.4
85
0.2
0.2
0.2
0.2
20000
40000
60000
20000
40000
60000
250000
40000
50000
20000
40000
60000
Train Steps
Trin Staps
Train Steps
Tlain Steps
Layer 7.self, attn.oo proj
Layer 7.self attn proj
Layer selt attn.v proj
Layer 7.self. attn.oo.ttooooo _roj
0.6
8
0.4
0.4-
0.4
0.2
0.2
0.0
8.0
0.0
0.0
0
20000
40000
60006
0
20000
400000
60000
250000
40000
50000
20000
400000
6c000
Train Steps
Train Staps
Train Steps
Train Steps
Nalve
Act-Align
The layer-wise output-level OCR plot of LLaMA-35OM on OpenWebText with a total sparsity of 0.9, with sparsity s 0.95, r 32, deonnectiviyy dipectrra = 1 1. Fach subplot in the figure reports the changes of OCR over training steps. For space limit, we report here the first 8 self-attention layers in the model, where each column refers Q. K. V, O respectively.
Figure 10:
33


Reviews:
[{"review_id": "u1s1SNAUcK", "reviewer": "Reviewer_ob7g", "summary": "This paper studies the cancellation effect between connectivity-sparse and spectral-sparse branches in dynamic sparse training and introduces an alignment loss. The idea is simple and practical, and experiments show clear gains over baselines.", "questions": "- Is there a correlation between OCR and global cosine similarity? Since OCR measures element-wise sign inconsistency, it may be influenced by local fluctuations rather than true directional cancellation. Such analysis could clarify OCR’s distinct role.\n\n- How does the model's performance differ when the alignment loss is applied only to the Q/K layers compared to applying it across Q/K/V/O or FFN layers?\n\n- Do larger models (e.g., LLaMA-7B) exhibit similar cancellation patterns, and does alignment maintain its effectiveness at scale?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper identifies the \"cancellation effect\" and proposes the OCR metric, which provides a valuable quantitative perspective on hybrid sparse training.\n\n- The proposed method is simple and easy to implement.", "weaknesses": "- The alignment loss is conceptually orthogonal to any combination of dynamic sparsity and low-rank training, yet experiments are limited to the CHTs + low-rank setup. Testing additional combinations would be necessary to confirm its generality.\n\n- OCR captures output-level discrepancies but does not fully demonstrate whether alignment mitigates gradient-level conflicts between branches. A more comprehensive analysis at the gradient level is recommended.\n\n- The paper lacks practical efficiency evaluations such as inference memory and throughput.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QVNn1dE5rz", "reviewer": "Reviewer_QMmU", "summary": "This work proposed CHTSL, an approach that unifies connectivity sparse training and spectral sparse training. They introduces an alignment loss mitigate the disagreement between the two branches and promote better collaboration. Experiments on small-scale LLama validate the efficacy of their approach.", "questions": "See the weakness.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper is written and organized well.\n- The idea of alignment loss is reasonable and inspiring.", "weaknesses": "- Lack of literature review. The paper discussed with pruning works, accounting for connectivity sparse training, differentiating against spectral sparse (low-rank) training. But in my opinion, structured pruning actually results in low-rank for the full model. Since the structured pruned models would have many columns or rows zero-out. The paper needs to discuss with structured-pruning-aware works, such as Only-Train-Once.\n\nOnly Train Once: A One-Shot Neural Network Training And Pruning Framework\n\n- Lack of sufficient numerical experiments. \n\n  - The numerical results are conducted under small-scale LLMs. It would be better to conduct over larger-scale LLMs to show the generality. \n  \n  -  Besides SiLU, it would be better to show over other activations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q1NBPVXBts", "reviewer": "Reviewer_iVGQ", "summary": "The paper explores a combination of LoRA, activation and dynamic sparse connections to improve pre-training efficiency. It attempts to unify CoLA and CHTs approaches through an alignment loss, aiming to maintain performance while reducing parameter usage.", "questions": "None.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of integrating CoLA with dynamic sparse connections is conceptually reasonable and consistent with recent trends in parameter-efficient pre-training. The use of alignment loss to bridge different sparsity paradigms shows an effort toward unified optimization.\n\n2. The experiments show solid performance across different sparsity levels, demonstrating the stability and robustness of the proposed approach.\n\n3. The paper is well-written and clearly structured, allowing readers to easily follow the methodology and experimental design.", "weaknesses": "1. The proposed method mainly unifies CoLA and CHTs via an alignment loss, which appears to be an incremental combination and is somewhat limited in novelty.\n2. The evaluation is somewhat limited. It would be beneficial to include downstream task benchmarks such as HellaSwag and COPA, similar to what was done in CHTS, to better demonstrate generalization capability and practical value.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)