CR-Net: Scaling Parameter-Efficient Training with Cross-Layer Low-Rank Structure
Authors: Boao Kong
Junzhu Liang
Yuxi Liu
Renjia Deng
Kun Yuan
Venue: ICLR 2026 Poster (2026)
Keywords: Parameter-efficient
LLMs pre-training
cross-layer low-rank
low-rank pre-training.
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a low-rank framework for LLMs pre-training named CR-Net which leveraging cross-layer activation residuals to enhance model efficiency while maintaining performance, reducing computational/memory costs.

Abstract: Low-rank architectures have become increasingly important for efficient large language model (LLM) pre-training, providing substantial reductions in both parameter complexity and memory/computational demands. Despite these advantages, current low-rank methods face three critical shortcomings: (1) compromised model performance, (2) considerable computational overhead, and (3) limited activation memory savings. To address these limitations, we propose **C**ross-layer Low-**R**ank residual **Net**work (**CR-Net**), an innovative parameter-efficient framework inspired by our discovery that inter-layer activation residuals possess low-rank properties. CR-Net implements this insight through a dual-path architecture that efficiently reconstructs layer activations by combining previous-layer outputs with their low-rank differences, thereby maintaining high-rank information with minimal parameters. We further develop a specialized activation recomputation strategy tailored for CR-Net that dramatically reduces memory requirements. Extensive pre-training experiments across model scales from 60M to 7B parameters demonstrate that CR-Net consistently outperforms state-of-the-art low-rank frameworks while requiring fewer computational resources and less memory.

Published as a conference paper at ICLR 2026
CR-Ner: SCALING PARAMETER-EFFICHET TRAINING WITH CROSS-LAYER LOW-RANK STRUCTURE
Boao Kong' Junzhu Liang"*, Yuxi Liu RRejja Deng', Kun Yuan 4f
'Peking University, Beijing, China
"Equal contributions
'Corresponding author: kunyuan@pku. edu. cn
ABSTRACT
Low-rank architectures have become increasingly important for efficient large language model (LLM) pre-training, providing substantial reductions in both pa- rameter complexity and memory/computational demands. Despite these advan- tages, current low-rank methods face three critical shortcomings: (I) compromised model performance, (2) considerable computational overhead, and (3) limited activation memory savings. To address these limitations, we propose Cross-layer Low-Rank residual Network (CR-Net), an innovative parameter-efficctt work inspired by our discovery that inter-layer activation residuals possess low-rank properties. CR-Ner implements this insight through dual-path architecture that efficiently reconstructs layer activations by combining previous-layct outputs with their low-rank differences, thereby maintaining high-rank information with mini- mal parameters. We further develop a specialized activation recomputation strategy tailored for CR-Ner that dramatically reduces memory requirements. Extensive pre- training experiments across model scales from 60M to 7B parameters demonstrate that CR-Net consistently outperforms state-of-ther of-tte-art low-rank frameworks while requiring fewer computational resources and less memory.
frame-
INTRODUCTION
Large language models (LLMs) have achieved remarkable success across diverse domains (Brown) et al. 2020; Touvron et al., 2023 Grattatiori et al., [2024] Liu et al. 2024a), with strong empirical evidence demonstrating that scaling both model parameters and training data consistently improves model performance (Kaplan et al. 2020 Hoffmann et al.) 2022, Rae et al., (022). However, as model sizes grow from millions to billions of parameters, the computational requirements for pre-training increase exponentially in both computation power and memory consumption. This necessitates months-long training cycles on large clusters of perfforrrrmmec GPUs, making the process both time-prohibitee and economically challenging. For example, scaling from GPT-3's 3's 2. 2.7B parameter version to its 175B counterpart increases memory requirements from 21GB to 1.4TB (a 66x increase) and computational costs from 55 3,640 petaflop-days (another 66x increase) (Brown et al.; (020). These challenges underscore the critical need for developing efficient architectures that systematically leverage the inherent characteristies of LLM model structures optimize both memory utilization and computational efficiency while maintaining cost-effectivenees.
Low-rank structures in LLM training. The low-rank property has emerged as one of the most prominent structural characteristics in transformer-ben based models, attracting significant research attention due its consistent empirical validation. Existing approaches leveraging this property can be fundamentally classified based on two principal observations:
(01). Low-rank parameter. Representative approaches such as Low-Rank Adaptation (LoRA) and its variants (Hu et al. 2022 Lialin et al.) 2023 Zhang et al. 2023 Liu et al.) 2025 Xia et al.) 2024 Kamalakara et al.) 2022 Miles et al. 2024 Dettmers et al. 2023) utilize learnable low-rank parameter matrices memory efficcet substitutes for full-rank weight updates. This design achieves significant parameter reduction while preserving model capacity and can achieve further memory savings with quantized parameters (Dettmers et al. 2023). Furthermore, this paradigm demonstrates strong compatibility with sparsity-inducing techniques (e.g.. SLTrain (Han et al.) (024) and LOST (Li et al. 2025)) achieve improved model performance under constrained parameter budgets.

---

Published as a conference paper at ICLR 2026
(02). Low-rank gradient. Recent research has explored the intrinsic low-rank properties of LLM gradients (Zhao et al. |024]] Chen
20en et al. 2024al Hao et al., 20224] Huang et al.; Robert et al.) 2024 He et al.] 2024: Liang et al. 2024). By projecting optimizer states into low-dimensional subspaces, these methods achieve significant memory reduction during pre-training. Extensions like RSO (Chen et al., 2025) provide theoretical convergence guarantees under low-rank gradient constraints.
Advanced approaches such as Apollo (Zhu et al. (222)) demonstrate additional benefits by combining channel-wise adaptive learning with low-rank gradients, yielding both improved validation accuracy and optimized memory efficiency throughput the pre-training process.
Limitations in existing literature. Despite their computational and storage efficiency, current approaches utilizing low-rank LLM structures suffer from several critical limitations:
(L1). Suboptimal performance with low-rank parameter training. Low-rank parameterization
reduces memory and computational costs at the expense of suboptimal performance (Biderman et al.
2024). Techniques such as full-rank initialization (Hu et al., (022), update aggregation (Huh et al.
2021: Lialin et al., 2023). and non-linear operators (Liu et al. (2025] can mitigate this issue but may
diminish computational benefits. Furthermore, recent studies show transformer weights typically
exhibit wear-full-rank properties (Aghajanyan et al. 2020 Yu & Wu 2023 Li et al. 2024). with
higher-rank weights being essential for knowledge representation (Meng et al. 2022) and sensitive to
low-rank approximation (Ji et al. 2024), indicating low-rank parameterization limit LLM capacity.
(L2). Computational bottlenecks with low-rank gradient training. While training approaches
utilizing low-rank gradient empirically achieve better performance than low-rank parameterization, the gradient compression process itself introduces computational overhead. Approaches such as GaLore (Zhao et al. (024)) and FIRA (Chen al. (202a) exploits Singular Value Decomposition (SVD) to identify effective low-rank gradient subspace, which would substantially reduce throughput during training. Other methods like GoLore (He al. 2024) and RSO (Chen et al., 2025) propose to use random low-rank gradient subspace, which sometimes leads to suboptimal performance.
(L3). Limitations to save activation memory. Existing approaches reduce memory load across multiple dimensions: parameters, gradients, and optimizer states. Yet a critical yet often overlooked memory burden stems from activation storage - intermediate variables cached during forward propagation required for gradient computation. Empirical studies (Zhao et al. 2024 Han et al. 2024 reveal that activation memory overhead typically ranges from 1x to 4x the model parameter size, exhibiting strong dependence batch size configurations. While low-rank methods such as RSO (Chen et al. 2025) and CoLA-M (Liu et al. 2025) have been proposed to reduce activation memory, remains open question whether activations still possess potential for further compression.
Main results. Our contributions are threefold, forming a principled framework rather than an ad-hoc combination:
(C1). Novel Foundational Principle: We propose a novel principle in LLMs: the difference between activations of adjacent layers exhibits a strong low-rank structure. This phenomenon has been consistently observed across various models and at different training stages. Unlike low-rank properties in gradient updates or parameter modifications-sutt LoRA toRs-this structural property of activations represents an previously unreported characteristic of LLMs. It thereby serves as foundational basis for more efficient pre-training paradigms.
as those exploited by GaLoRE and
(C2). efffcentr-efficieen framework: We propose the Cross-layer low- Rank residual Network (CR-Net), a pre-training framework that inherently utilizes cross-layer low-rank activation differences. By computing each linear layer's activation through a combination of low-rank outputs and preceding layer activations, our design directly applies the observed low-rank structure while avoiding infor- mation loss from repeated low-rank approximations in LoRA-based approaches. This addresses Limitations (L1) and (L2) through reduced parameter complexity and memory/computation costs.
(C3). activation-efficien Re-computation: we develop re-computation strategy for CR-Ner that removes activation storage for most layers during backward propagation. Our analysis shows superior memory efficiency and lower re-computation overhead compared to existing methods like vanilla gradient checkpointing (GCP) and CoLA-M, resolving Limitation (L3)..
(C4). Empirical Validation: Large-scale pre-train pre-training experiments demonstrate that CR-Ner achieves better validation performance than existing low-rank parameter methods while maintaining training
2

---

Published as a conference paper at ICLR 2026
W/
SILU activation
Self-ttttntiion
Self-Attuntion
self-actertion
Residual connection
production
11
Cross-layer Residual
Re-compute activation
Layer
Layer
Layer
Figure 1: Illustration CR-Net hase on ILaMA-2 architecture with transformer layers. Layer normalization and RoPE are omitted for simplicity.
throughput. Integration with our re-computation strategy further reduces memory consumption without compromising model capability.
RELATED WORKS AND PRELIMINARY
2.1
RELATED WORKS
Here we display some prior works of parameter-efficient training for LLMs. More related works can be found in Appendix
Parameter-efficient training for LLMs. The pursuit of reducing memory and computational over- head LLM pre-training has driven significant progress in parameter-efficient training frameworks. Central to these efforts is Low-Rank Adaptation (LoRA) (Hu et al.) 2022), which constrains weight updates to low-rank subspaces through matrix decomposition. This approach fundamentally differs from sparse training methodologies (Houlsby al. 2019 Thangarasa et al. 2023) that selectively update subsets of model parameters. Recent hybrid architectures like SLTrain attempt to syner- gize low-rank and sparse parameterization, while multi-LoRA update strategies (Lialin al. 2023 Xia et al.) 2024) aim to recover full-rank expressiveness through sequential low-rank adjustments. Although introducing nonlinear operations to low-rank activations (Liu al. 2025) theoretically enhances model capacity, such innovations often incur prohibitive computational costs that undermine their practicality for large-scale pre-training scenarios.
2.2 PRELIMINARY
Notations. In this paper, assume that the amount of transformer blocks of the model is L. Denote & and h as the sequence length and hidden dimension, respectively. hn stands for the intermediate dimension of the model. We X₁ R represent the input of layer WP, X and YPP stand for the parameters, inputs and outputs of linear layers at the position P. For example, P E {Q, K, V, O, gate, up, down] in LLaMA-baser model and P {Q. K, V, O, up, down) in GPT-based model. Finally, under Frobenius loss as
(AA) to denote the optimal approximation of A with rank
(A) = arg min ||A Alla, s.t. rank(A) <
A
3

---

Published as a conference paper at ICLR 2026
Transformer layers. In this paper, present our proposed CR-Ner based on LLaMA architecture (Touvron et al.] 2023) Grattafiori et al. 2024) with SwiGLU activation. And we will present the pre-training performance of CR-Ner based various architecture in Section
We assume that the batch size is 1, unless indicated otherwise. We omit layer normalization and rotary position embedding (RoPE) operations for simplicity. Then the multi-head attention can be summarized as follows:
(x19) (x,
Att: softmax
WP + XI: Y
Vh
where ye == X.W, Y =
= x
For W With hR and Widown E Rlxxh as the weight (projection) matrices the feed- forward network (FFN) in the layer I, where du denotes the dimension of projection. Then the FFN operation can be summarized as:
X1+1 (SwiGLU(AAt) W (Att: W(T))) Wrown
Att.,
(2)
where 3 denotes the element-wise production.
CR-Net: CROSS-LAYER RESIDUAL NETWORK FOR LLM PRE-TRAINING
3.1 OBSERVATION: PREVIOUS-LAYER AND LOW-RANK TERM PRESENT A BETTER ACTIVATION APPROXIMATION
Existing studies have demonstrated that the activations in LLMs exhibit intrinsic low-rank structural properties (Cui et al. 2020 Huh et al. 2021: Yu & Wu 2023 Liu et al. 2025) and thus low-rank approximations activation matrices achieve acceptable reconstruction fidelity, making pre-training model with low-rank parameters possible. Furthermore, empirical evidence from (Liu et al. 2024b. [Bandon et al. 2024) highlights inter-layer activation correlations, suggesting that historical activations from preceding layers may encode critical information for current layer computations. Unlike existing observations, we have found a critical structural characteristic that the difference between activations of adjacent layers exhibits a significant low-rank property.
Estimating activation from the historical layer and low-rank difference. For -orl=22 = 3,
L.
consider the following estimation for the activation YP as follows:
Y.P 1, = - + (A,, where Y Y -
(3)
Here, 30 is a tuned scaling coefficient. We aim validate that this hybrid approach achieves superior approximation accuracy compared to direct low-rank approximation y,, LY,")
Emprical evaluation for different activation approximation approaches. To compare the perfor- mance for different methods including Eq. (3) and direct low-rank approximation for the activation estimation, we conducted comparative experiments by fine-tuning the pre-trained LLaMA-3 8B (Grattafiori et al. 2024), GPT-2-small (Radford et al.) 2019), and models using the TinyShake- speare dataset. In both schemes we use the same low-rank dimension 0.25h and we quantify approximation quality via relative error defined follow.
Relative error (YP. YPP) = | - YP
YP
where ? is an estimation of Y.".
Figure 2 iillssrrates the average relative error of two approaches for activation approximation over all the transformer layers. can observed that the previous-layer-basee yields smaller relative errors, indicating that performing low-rank approximation with the history information provides better reconstruction capability compared to direct approximation the original activations when preserving the same rank of singular values. Additional evaluation can be fined in Appendix E.1 E. We also present a theoretical insight for this observation, whose detail is in Appendix [
Such an observation forms the foundational insight of CR-Net, it enables more accurate activation reconstruction with fewer parameters compared to direct low-rank approximation of Y,,, making CR-Net not simple extension of existing low-rank frameworks for pre-training.
4

---

Published as a conference paper at ICLR 2026
tcvettion Aparodiaation Liantd
Edette Activtter Approciation dusing
Figure 2: The average relative error of activation recovery by using low-rank approximation and using over all transformer layers. (Left: LLaMA-3 8B, right: GPT-2 small.)
3.2 MODEL STRUCTURE
We utilize the empirical observation of low-rank structure for cross-layer activation differences to guide the architectural design of CR-Ner with low-rank parameters. As evidenced by (3), the matrix A 6g YI" possesses an intrinsic low-rank property, which motivates the approximation as YP BoYP-1 LR, (A&,
Motivated by the low-rank cross-layer activation difference, replace the full-shape weight WPP Rou to two low-rank learnable parameter matrices AB E Rhuxr and By RRrxhu for / 2,3, 2, .3,,,, I, where r < minn hoa) is a hyper-parameter. Then the activation of layer at location P can be computed by
YP +
(5)
Furthermore, to enhance the scalability of the cross-layer residual operation, we let the scaling factor 3o as learnable parameter 3.. With the learnable scaling factor, it can dynamically adjust the impact the historical activation and low-rank outputs in the current activation, allowing the model balance how much the historical information and incremental information should be carried. Specifically, when 3 is near zero, the model relies heavily on the low-rank residual. Otherwise, the residual becomes a refinement over strong propagated signal. This spectrum allows CR-Ner to smoothly interpolate between shallow, expressive layers and deeper low-rank transitions, all within fixed memory and computation budget. Ablations Section [5.2] also illustrates that the learnable scaling factor can improve the training performance of CR-Net.
Moreover, we still use the full-size parameter matrix WP the first layer. We add normalization term E to avoid SP equals to zero, where E can be set to 10-6 in practical. At this stage, CR-Ner change the standard matrices production to the cross-layer residual operation as follow:
XFWW, sign (PP + 1 1 2,3, T..
1=1,
YP
(6)
With the low-rank parameters and cross-layer residual, we can present the framework of CR-Net, which can be illustrated as Figure [ The full-rank parameter in the first layer as well as the residual for CR-Net avoid the information loss by direct training with low-rank parameters.
CR-Net's Stability Mechanism. CR-Net overcomes the instability of low-rank pretraining by preserving high-rank activations through full-rank first layer and modulated residual connections. learnable scalar 3 dynamically balances the high-rank state and low-rank increments, enabling robust signal reconstruction without collapsing into low- dimeennnsional subspaces. Unlike methods that enforce strict low-rank constraints via projections or decompositions (e.g., QR, SVD), which often cause information loss and numerical issues, CR-Ner uses standard optimization methods without additional overhead. This design ensures stable training dynamics, similar to full-rank approaches, while achieving significant parameter and memory reductions.
3.3
CR-Net WITH ACTIVATION-EFFICII RE-COMPUATION
Although CR-Ner directly reduces memory and computational costs through its parameter-efficient framework, activation values during forward propagation still consume significant GPU memory, particularly under large batch sizes.
5

---

Published as a conference paper at ICLR 2026
Algorithm 1 The back-propagation of CR-Ner with re-computation
Require:
Layer inputs X for / = 2, .. ,, low-rank output Xi A for = 1,2, ... , and
linear layer activations Y for E .A.
for 1 L, 1.... do Compute activations of layer for back-propagation by X1, Xp Af and yp.
if I + 1 and / 1 E A then for P in all linear layers do Reconstruct the previous-layer activation YP by Eq.
end for end if end for
In practice, gradient checkpointing (GCP) (Chen et al. 2016) enhances memory efficiency by storing only subset of activations (checkpoints) during forward propagation, with the remaining activations recomputed during backpropagation. The vanilla GCP implementation requires storing layer-wise inputs while recomputing other activations through full forward passes. However, CR-Nel introduces a critical architectural dependency: the computation of linear layer activations relies on both current low-rank outputs and historical activations from preceding layers. This dependency necessitates full forward passes through all antecedent layers during GCP recomputation, resulting in an ( computation overhead.
To address this, we propose a tailored recomputation strategy for CR-Net. The core technique of the strategy is store subset of linear activations. Specifically, we select a subset of Transformer layers indexed by A with E A and 1 & A typically. During forward propagation, we store:
All layer inputs X₁.
All linear layer activations Y?" for layers E A.
Low-rank outputs XPAA for 1 2,3, .., .. T. to minimize matrix recomputation costs.
During backpropagation (for layers I = ..2, L), we recompute activations using stored inputs and checkpoints. If Y is stored, we can use it to obtain other activations in this transformer layer. Otherwise, it can be recovered via the inverse of the cross-layer residual connection in (6):
YP
sign (3g M1 + (Y Af
(7)
The back- prok-propagation with re-computation of CR-Ner is summarized Algorithm [ It enables full activation recovery using only a stored subset of activation. We also emphasize that such storage can effectively reduce the accumulation of prediction error during the activation recovery process. By setting |A] as L/8 and storing low-rank outputs across multiple layers, such errors remain controllable (see empirical validation in Section 5.2
4 COMPLEXITY ANALYSIS
In this section, we present the analysis for the parameter complexity, memory complexity, and computation complexity of CR-Ner based LLaMA architecture (Touvron et al. 2023 Grattatiori et al. 2024) and Adam optimizer (Kingma & Ba 2014). We also present the communication overhead and HBM memory analysis for CR-Ner under multiple GPU training in Appendix G
4.1 COMPLEXITY ANALYSIS WITHOUT CE-COMPUTATION
Parameter and Memory Complexity. For CR-Net, the parameter complexity of the first layer is identical that of the full-rank model. In subsequent layers, CR-Ner employs two low-rank matrices with hr parameters to replace the full-rank matrices in self-attention, and uses two additional low-rank matrices with hr and ha? parameters respectively for the FFN operators. The overall parameter complexity is summarized as follows:
Ali² + 3h/
(L- 1)(1krr + 3h
Parameters first layer
Parameters the other layers

---

Published as a conference paper at ICLR 2026
Table 1: Computation complexity of different efficient pre-training approaches for one gradient step based on LLaMA architecture. The optimizer are all standard Adam. Lower-order terms are omitted for brevity. The computation complexity of other methods are referred from (Liu ct al. 2025).
Approach
FLOPs
Full-rank
L(24sh² + h + 18shha)
LRe)
L(40sh² + 24s"h + 30shha)
SLTrain
L(24shi + 12s5h 18shbe + -244 18h/ugr)
GaLore
L(24sh² + 12s"h + 18shhg 16hhr 12hhgr)
CoLA
L(48shr 12s2h 18sr(h hg))
CR-Net
24sh 12s2 + 18shda + (L 1)(Ssrrr 12s + 18sr (l hn))
Regarding memory overhead for storing parameters, gradients, and optimizer states with the Adam optimizer, the requirement is approximately 4 times the parameter count. Consequently, the memory complexity is:
16h² 12hhg
(L ()-1) + 12har)
Memory first layer
Memory in the other layers
With low-rank parameters, the parameter complexity is reduced, particularly if S h/2. 2. In practice, setting 0.25h achieves approximately 50% saving in parameters while maintaining comparable pre-training performance to full-rank training, as demonstrated in Section
Computational Complexity. For the first transformer layer of CR-Ner, the forward- and backward- propagation process maintains identical computational complexity to full-rank training. In subsequent layers, CR-Net employs low-rank weights while eliminating the nonlinear activation step applied to low-rank outputs, combined with cross-layer residual connections in contrast to CoLA (Liu al. 2025). Given that both operations introduce lower-order computational complexity terms, conclude that these layers maintain equivalent FLOPs to CoLA. Following existing analysis (Liu
al. 2025). the total computation complexity is (see Appendix B.2 for details):
24sh² 12s 18shhg + (L (48shr + 12s + 18sr(h + ht))
(10)
FLOPs in layer
FL.OPs in the other layers
Table I demonstrates computational requirements per gradient step. As ha =s 8h/3 in LLaMA-based model with SwiGLU activation, CR-Ner achieves complexity reduction over full-rank pre-training when r < 0.5d. Notably, while CR-Nel exhibits marginally higher FLOPs than CoLA with the same due to the full-size first layer, enables superior training performance lower values with reduced computation. See Section 5.1 I for empirical validation.
4.2 COMPLEXITY ANALYSIS WITH RE-COMPUTATION
We evaluate recomputation overhead and activation memory costs for CR-Net based LLaMA- like architecture in C Table 2
compares activation memory and recomputation complexity across frameworks. including practical measurements for LLaMA2-7B (batch size 16). Althrough re-computation strategy introduce additional memory overhead than Vanilla GCP and CoLA-M, CR-Net also achieves 67..4% reduction in total computation overhead compared to Vanilla GCP and a 8.0% reduction compared to CoLA with significant memory saves, validating the effectiveness of our proposed cross-layer framework and re-computtiion strategy.
5 EXPERIMENTS
This section conducts numerical experiments to systematically assess the efficiency of the proposed CR-Ner methodology. Our evaluation framework encompasses pre-training tasks spanning diverse model scales. Furthermore, we empirically validate the performance advantages of our innovative re- computation approach. To substantiate design choices, comprehensive ablation studies are performed focusing two critical components: the determination of optimal rank values for low-rank parame- terization and the adaptive learning dynamics of scaling coefficients 3. within our parameterization scheme. More experimental results be referred in Appendix F
7

---

Published as a conference paper at ICLR 2026
Table 2: Activation memory and re-computation complexity of different activation-efficcent approaches based on LLaaM architecture with batch size and BF16 precision. For CR-Ner, the notation represents the elements of A where = 4. The last two column is the evaluated computation and memory of LLaMA2-7B framework with r 512, batch size B = 16, and sequence length 256. We also add a explanation of the re-computation complexity of CoLA-M in Appendix C.2
Algorithms
Activation Memory
Re-compute
Memory (GB)
FLOPs (x (x10's)
Vamilla GCP
Lsh 2Lsh TLsr (L+ 10b)sh + 1
24Lsh' + 4Ls
51.22 (1, 000xx)
2.119 (1. x00x)
CoLA-M
20.67Lshr 4Ls
24.78 (0. 484x)
0.752 355x)
CR-Ner
20.67 b)shr 4Ls"h
23.35 (0. 456x)
0.692 (8. 326
Full-rank
20. - 2Ls²a
N.A.
70.97 (1 386x)
1.608 (8. 759
Table 3: Comparison of validation perplexity (PPL) (L), parameter complexity (M) (L) and evaluated memory overhead (GB) (4) of parameter, gradients, and optimizer states of different effective training approaches in the pre-training task of LLaMA model with C4-en dataset. The results of compared methods referred from
(Huang ot al. Han al. 2024 Liu al. 2025 2025). For CR-Ner, it is compared with other cecuuntte--fffcccnn methods with aligned parameter complexity (marked as () and compared with other optimizer-effienee methods with aligned memory overhead (marked as f). N.A. means that the corresponding experiment has not been taken.
[Chen [
2025 Zhu et al. [024] Miles al.) [024] Mo al.)
60M
130M
350M
IB
Approach
1.IB tokens
2.2B tokens
6.4B tokens
13.1B tokens
PPL
Para
Mem
PPL
Para
Mem
PPL
Para
Mem
PPL
Para
Mem
Full-rank
34.06
58
0.43
24.36
134
1.00
18.80
368
2.74
15.56
1339
9.98
LoRA
34.99
58
0.37
33.92
134
0.86
25.58
368
1.94
19.21
1339
6.79
ReLoRA
37.04
58
0.37
29.37
134
0.86
29.08
368
1.94
18.33
1339
6.79
VeL.oRA
34.35
58
0.37
25.88
134
0.86
N.A.
N.A.
FLoRA
33.76
58
0.37
25.29
134
0.86
N.A.
N.A.
SLTrain
34.15
0.32
26.04
97
0.72
19.42
194
1.45
16.14
646
4.81
CoLA
34.04
43
0.32
24.48
94
0.70
19.40
185
1.38
15.52
609
4.54
LORO
33.96
43
0.32
24.59
94
0.70
18.84
185
1.38
15.19
609
4.54
CR-Ner
32.76
43
0.32
24.31
90
0.67
18.95
183
1.36
15.22
583
4.35
GaLore
34.88
58
0.36
25.36
134
0.79
18.95
368
1.90
15.64
1339
6.60
RSO
34.55
0.36
25.34
134
0.79
18.87
368
1.90
15.86
1339
6.60
Apollo
31.55
0.36
22.94
134
0.79
16.85
368
1.90
14.20
1339
6.60
CR-Ner
32.76
43
0.32
23.74
106
0.79
17.08
250
1.86
14.05
870
6.48
5.1
PRETRAINING WITH CR-Net
Experiment setup. We evaluate the proposed CR-Net framework by pre-training I.LaM--2 models (Tourro et al.] (023) with size varying from 60M to 13B. Following the existing experimental settings (Zhao et al. 2024). we train the model over C4-en dataset (Raffel et al. 2020), which is primarily intended for pre-training language models and word representations with large scale. We compared CR-Ner with existing parameter-efficient methods including LoRA (Hu al. 2022). ReLoRA (Lialin et al. 2023), VeLoRA (Miles et al.) 2024), FLoRA (Hao et al. 2024), SLTrain (Han al.) 2024). SLTrain (Han al.) 2024), CoLA (Liu et al.) 2025). LORO (o al. 2025) with aligned parameter complexity. We also compared CR-Ner with other optimizee-efffccent algorithms including Gal.ore (Zhao et al. 2024), RSO (Chen et al. 2025), and Apollo (/hu et al. 2024) with aligned memory overhead. We use the same configurations as those reported in (Zhao et al. 2024). and the detailed experiment setting are in Appendix E.2.1 For models with sizes ranging from 60M to IB parameters, do not employ the re-computation strategy to maintain consistency with existing baselines. For the 7B and 13B models, we employ re-computation to reduce memory consumption.
Pre-training results. Table 3 has shown that CR-Net generally outperforms other parameter-efficien with the same of lower training parameters. It can be observed that CR-Ner even achieve a better performance full-rank training while reducing the parameter complexity by 56.5% and the per-step
8

---

Published as a conference paper at ICLR 2026
Table 4: Comparison of validation perplexity (4) and memory (4) of different approaches in LLaMA-2 7B
pre-training tasks. The results of compared methods are referred from (Liu et al.
2025 Zhu et 2024).
Training steps
Memory (GB)
10K
40K
65K
80K
8-bit Adam
72.59
N.A.
18.09
N.A.
15,47
8-bit Gal.ore
65.16
26.87
17.94
N.A.
15.39
Apollo
N.A.
N.A.
17.55
N.A.
14.39
CoLA-M
28.82
22.76
16.21
14.59
13.82
CR-Ner re-computation
27.60
23.11
16.01
14.47
13.72
Training tokens (B)
1.3
5.2
8.5
10.5
Table 5: Comparison of validation perplexity (4) of different approaches in LLaMA-2 13B pre- training tasks.
Table 6: Comparison of validation perplexity (L) for CR-Net with or without low-rank first layer in LLaMA-2 350M pre- pre-training tasks.
Steps
40K
Training tokens
6.4B
8-bit Adam
17.85
CR-Net w.o. low-rank first layer
18.95
CR-Net w. re-computttion
18.12
CR-Net w. low-rank first layer
19.68
computation complexity by 63.2% for LLaMA-2 IB network. When aligning the memory overhead, CR-Ner achieves better validation perplexity than that of optimizer-efffciiet algorithms especially with the model size larger than IB, showing the benefits of CR-Net with large-scale scenarios.
Table 4 illustrates the validation perplexity of CR-Net when training with LLaMA2-7B as well as other competitive approaches. It can be observed that CR-Net outperforms all baselines in the training process with a lower memory overhead when using re-computation.
Furthermore, we report the validation perplexity for the pre-training of LLaMA-2 13B after 40,000
initialization steps. Experimental details are provided in Appendix E.2.3 As demonstrated in Table 5 CR-Ner achieves more than 50% reduction in parameters, while incurring only 2% degradation in validation performance.
Pre-training and inference throughput. Figure 3 illustrates the throughput performance of our method alongside comparable approaches in both pre-training and inference tasks, with detailed experimental configurations provided in Appendix E.2.1 As shown, CR-Net achieves superior throughput compared to other methods in LLaMA-2 1B pre-training and inference scenarios. Notably, when accounting for data aata-parallel communication overhead across GPUs for LLaMA-2 1B pre-training, CR-Ner demonstrates over 6% improvement relative to state-of-the-arr methods that do not incur such communication costs. Furthermore, CR-Net exhibits 8.6% throughput enhancement compared to LORO in LLaMA-2 7B pre-training.
5.2 ABLATIONS
How does rank selection impact pre-training performance? We train the LLaMA-2 350B network with CR-Ner using identical parameters but varying ranks across transformer layers. Experimental details are provided in Appendix E.2.6. Figure shows that networks with higher ranks in middle layers and lower ranks in side layers exhibit superior performance.
Whether does the learnable scaling factor S. benefit the model convergence? We train the LLaMA-2 350M network with the same hyperparameters as that Section (.1] except make the scaling SP. From the perplexity shown in Figure 5 we can obtain that traiinn CR-Ner with learnable 3r can improve the numerical stability as there are spikes in the runs with 3. fixed. This match our observation Section that the best S to achieve the lowest rank varies during the training process. More discussions be found in Appendix F22
Whether other cross-layer residual strategies do well in pre-training with low-rank parameters? To compare different closs-layer residual patterns under the same global low-rank parameterization,

---

Published as a conference paper at ICLR 2026
116109
5350 6999
31449
6012
210
Thoughhy
10h0 190 0M5
080 DN
Figure 3: The average throughput (tokens/s) for each device of different algorithms. (Left: LLaMA-2 IB
pre-training on an Nvidia A100 40G GPU, with results of other comparable methods from (Liu et al. 2025) Middle: LLaMA-2 1B inference on Nvidia A A100 80G GPU. Right: LLaMA-2 7B pro-training Nvidia A100 80G GPU.)
275
32.
210
2
Her 56000 50000
Benntions
10000 .0000 2:203
Rerationa
52.00
Figure 4: The evaluation perplex- ity CR-Net training LLaMA- 350M model with different strate- gies of rank selection.
Figure 5: The Comparison of Figure 6: The evaluation per- evaluation perplexity for CR-Net plexity for CR-Net, Learnable- in training LLaMA-2 350M with ResFormer and DenseFormer in fixed of and learnable of.
the LLaMA-2 IB pre- pre-training task.
we implement with learnable skip from the first layer's activation across all low-rank linear layers, inspired by the first-layer value residual in ResFormer (Zhou et al. (2024) and DenseFormer (Pagliardini et al.) (2024). As Figure 6 illustrates, CR-Ner outperforms the other two low-rank adaptation under the similar low-rank parameter budget.
Ressoore variant that replaces CR-Ner's previous-layer residual term
Whether the full-rank first layer necessary? We trained the LLaMA-2 350M model with either full-rank or low-rank parameters in the first transformer layer. As shown in Table utilizing low- rank first layer introduces 3.5% degradation validation perplexity. This finding demonstrates the critical role of the full-rank first layer in reconstructing high-rank signals from low-rank information.
6 CONCLUSION
We propose CR-Net, parameter-efffcient LLM pretraining framework that computes linear acti- vations via low-rank transformations with cross-layer residuals. This dual-path design preserves high-rank capacity using fewer parameters, achieving better validation performance with reduced computation and memory than conventional low-rank frameworks.
We outline two directions for future work CR-Ner framework. First, system-level implementations integrating sixeed-rrecssion training could alleviate the growing memory overhead encountered when scaling CR-Net to larger models, particularly for activation recomputation strategies. Second, gener- alizing the framework's 's current multi-head attention foundation to alternative attention architectures, including multi-head latent attention (Liu al. 2024a), would broaden its architectural versatility.
ACKNOWLEDGMENTSS
This work is supported by the National Key Research and Development Program of China (No. (024YFA1012902) and National Natural Science Foundation of China (No. 12288101, 92370121, 12301392). This work is also supported by AI for Science Institute, Beijing, China.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning, arXiv preprint arXiv:20/2./3255, 2020.
Joshua Ainslie, James Lee-Thorp,
Thorp, Michiel De Jong, Yury Zemlyanskiy. Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprini arXiv:2305. 13245, 2023.
Albert S Berahas, Liyuan Cao, Krzysztof Choromanski, and Katya Scheinberg. A theoretical and empirical comparison of gradient approximations in derivative optimization. Foundations of Computational Mathematics, 22(2):507-560, 2022.
Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, et al. Lora learns less and forgets less. arXiv preprint arXiv:240..0967, 2024.
William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, and Jonathan Ragan- Kelley. Reducing transformer key-value cache size with cross-layer attention. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv: 160.006144 2016.
Xi Chen, Kaituo Feng, Changsheng Li, Xunhao Lai, Xiangyu Yue, Ye Yuan, and Guoren Wang. Fira: Can we achieve full-rank training of ilms under low-rank constraint? arXiv preprint arXiv:2410.01623, 2024a.
Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, et al. Symbolic discovery of optimization algorithms. Advances in neural information processing systems, 36:4920544452 49233, 2023.
Yiming Chen, Yuan Zhang, Liyuan Cao, Kun Yuan, and Zaiwen Wen. Enhancing zeroth-order fine-tuning for language models with low-rank structures. arXiv preprint arXiv:2410.07698, 2024b.
Yiming Chen, Yuan Zhang, Yin Liu, Kun Yuan, and Zaiwen Wen. A memory efficient randomized sub- space optimization method for training large language models. arXiv preprint 2:Xi:22502.07222, 2025.
Chunfeng Cui, Kaiqi Zhang, Talgat Daulbaev, Julia Gusak, Ivan Oseledets, and Zheng Zhang Active subspace of neural networks: Structural analysis and universal attacks. SIAM Journal on Mathematics of Data Science, 2(4): 122, 2020.
Tri Dao. Flashattention- Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness, Advances in neural information processing systems, 35: 16344-16359, 2022.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36:10088-10115 2023.
John C Duchi, Michael I Jordan, Martin J Wainwright, and Andre Wibisono. Optimal rates for zero-order convex optimization: The power of two function evaluations. IEEE Transactions on Information Theory, (1)
2015.
Jianwei Feng and Dong Huang. Optimal gradient checkpoint search for arbitrary computation graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14433-1142.. 2021.
11

---

Published as a conference paper at ICLR 2026
Tammay Gautam, Youngsuk Park. Hao Zhou, Parameswaran Raman, and Wooseok Ha. Variance- reduced zeroth-order methods for fine-tuning language models. arXiv preprint arXiv:2404.08080, 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Andi Han, Jiaxiang Li, Wei Huang, Mingyi Hong, Akiko Takeda, Pratik Jawanpuria, and Bamdev Mishra. Sltrain: a sparse plus low-rank approach for parameter and memory efficient pretraining. arXiv preprint arXiv:2406.022 022002222222 14, 2024.
Jitai Hao, Yuke Zhu, Tian Wang. Jun Yu, Xin Xin, Bo Zheng, Zhaochun Ren, and Sheng Guo. Omnikv: Dynamic context selection for efficient long-context llms. In The Thirteenth International Conference on Learnnng Representations, 2025.
Yongchang Hao, Yanshuai Cao, and Lili Mou. Flora: Low-rank adapters are secretly gradient compressors. arXiv preprint arXIv:2402.03293, 2024.
Horace He and Shangdi Yu. Transcending uuntim--memon tradeoff's in checkpointing by being fusion aware. Proceedings of Machine Learning and Systems, 03411-427, 2023.
Yutong He, Pengrui Li, Yipeng Hu, Chuyan Chen, and Kun Yuan. Subspace optimization for large language models with convergence guarantees. arXiv preprint arXiv:2410. 12899, 2024.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pp. 2790-2727 2799. PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Aluen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1,2):3 2022.
Weihao Huang, Zhenyu Zhang, Yushun Zhang, Zhi Chi-Quan Luo, Ruoyu Sun, and Zhangyang Wang. Galore-mini: Low rank gradient learning with fewer learning rates. In NeurIPS 2024 Workshop Fine-Tuunin in Modern Machine Learning: Principles and Scalability.
Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoe V Le. Yonghui Wu, al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019.
Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias deep networks. arXiv preprint arXiv:2 2033 10427, 2021.
Yixin Ji, Yang Xiang, Juntao Li, Wei Chen, Zhongyi Liu, Kehai Chen, and Min Zhang. Feature- based low-rank compression of large language models via bayesian optimization. arXiv preprint arXiv:240S.10616. 2024.
Jiachen Jiang, Jinxin Zhou, and Zhihui Zhu. Tracing representation progression: Analyzing and enhancing layer-wise similarity. arXiv preprint arXiv:2406. 14479, 2024.
Siddhartha Rao Kamalakara, Acyr Locatelli, Bharat Venkitesh, Jimmy Ba, Yarin Gal, and Aidan N Gomez. Exploring low rank training of deep neural networks. arXiv preprint arXiv:2209.13569. 2022.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint (=Xiv:2001.08361 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 442266980, 2014.
12

---

Published as a conference paper at ICLR 2026
Guangyan Li, Yongqiang Tang, and Wensheng Zhang. Lorap: Transformer sub-layers deserve differentiated structured compression for large language models. arXiv preprint arXiv:2404.09695, 2024.
Jiaxi Li, Lu Yin, Li Shen, Jinjin Xu, Liwu Xu, Tianjin Huang, Wenwu Wang, Shiwei Liu, and Xilu Wang. Lost: Low-rank and sparse pre-training for large language models. arXiv preprint (:Xiv:2508.02666 2025.
Vladislav Lialin, Namrata Shivagunde, Sherin Muckatira, and Anna Rumshisky. Relora: High-rank training through low-rank updates. arXiv preprint arXin:2307.00569, 2023.
Kaizhao Liang, Bo Liu, Lizhang Chen, and Qiang Liu. Memory-efficient Ilm training with online subspace descent. arXiv preprint arXiv:2408. 12857, 2024.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint 19444 2024a.
Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Reza Haffari, and Bohan Zhuang. Minicache: Kv cache compression in depth dimension for large language models. Advances in Neural Information Processing Systems, 37:139997-140031, 2024b.
Hong Liu, Zhiyuan Li, David Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training. arXiv preprini arXiv:2305. /4342, 2023.
Ziyue Liu, Ruijie Zhang. Zhengyang Wang. Zi Yang, Paul Hovland, Bogdan Nicolae, Franck Cappello, and Zheng Zhang. Cola: Compute-efficient pre-training of Ilms via low-rank activation. arXiv preprint arXiv:2502. 10940, 2025.
Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lec, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. Advances in Neural Information Processing Systems, 36:53038-523 53075, 2023.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems, 05117399-17772 2022.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv: 160907840 2016.
Roy Miles, Pradyumna Reddy, Ismail Elezi, and Jiankang Deng. Velora: Memory efficient training using rank-1 sub-token projections. Advances in Neural Information Processing Systems, 37: 42292-42 42310, 2024.
Zhanfeng Mo, Long-Kai Huang, and Sinno Jialin Pan. Parameter and memory efficient pretraining via low-rank riemannian optimization. In The Thirteenth International Conference on Learning Representations, 2025.
Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. In Proceedings of the 27th ACM symposium operating systems principles, pp. 1-15, 2019.
Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-Im. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-15, 2021.
Tam Nguyen, Tan Nguyen, and Richard Baraniuk. Mitigating over-smoothing in transformers via regularized nonlocal functionals. Advances in Neural Information Processing Systems, 36: 80233-80256, 2023.
13

---

Published as a conference paper at ICLR 2026
Matteo Pagliardini, Amirkeivan Mohtashami, Francois Fleuret, and Martin Jaggi. Denseformer: Enhancing information flow in transformers via depth weighted averaging. Advances in Neural Information Processing Systems, 01111134799166667 2024.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Jack Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring. Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, (1440))116) 2020.
Thomas Robert, Mher Safaryan, Ionut- Vlad Modoranu, and Dan Alistarh. Ldadam: Adaptive optimization from oww-dimesiional gradient statistics. arXiv preprint arXiv:2410.66103 2024.
Max Ryabinin, Tim Dettmers, Michael Diskin, and Alexander Borzunov, Swarm parallelism: Training large models can be surprisingly communication-efficeen In International Conference on Machine PMLR, 2023.
Learning, pp.
Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-3: Fast and accurate attention with asynchrony and low-precision. Advances in Neural Information Processing Systems, 37:68658-68685 2024.
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pp. 4596-4604, PMLR, 2018
Vithursan Thangarasa, Abhay Gupta, William Marshall, Tianda Li, Kevin Leong, Dennis DeCoste, Sean Lie, and Shreyas Saxena. Spdf: Sparse pre-training and dense fine-tuning for large language models. In Uncertainty in Artificial Intelligence, pp. 2134 2134-2146. PMLR, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv: 092888, 2023.
Jue Wang, Binhang Yuan, Luka Rimanic, Yongjun He, Tri Dao, Beidi Chen, Christopher Ré, and Ce Zhang. Fine-tuning language models over slow networks using activation quantization with guarantees. Advances in Neural Information Processing Systems, 35:19215-19234 2022.
Wenhan Xia, Chengwei Qin, and Elad Hazan. Chain of lora: Efficient fine-tuning of language models via residual learning. arXiv preprint arXiv:2401.04151, 2024.
Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint wrxiv:2505.09388 2025.
Hao Yu and Jianxin Wu. Compressing transformers: features are low-rank, but weights are not! In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 1 11007-11015, 2023.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao Adalora: Adaptive budget allocation for eframeter-efficient fine- 2023.
tuning. arXiv preprint arXiv:2303.
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient I1m training by gradient low-rank projection. arXiv preprint arXiv:2403.03507, 2024.
Zhanchao Zhou, Tianyi Wu, Zhiyun Jiang, and Zhenzhong Lan. Value residual learning for alleviating attention concentration in transformers. arXiv preprint arXiv:2410. 00.44/0....7..... 2024.
Hanqing Zhu, Zhenyu Zhang, Wenyan Cong, Xi Liu, Sem Park, Vikas Chandra, Bo Long, David Z Pan, Zhangyang Wang. and Jinwon Lee. Apollo: Sgd-like memory, adamw-level performance. arXiv preprint arXiv:2412.05277 2024.
14

---

Published as a conference paper at ICLR 2026
Appendix
A ADDITIONAL RELATED WORKS
Low-rank training for LLMs. Beyond explicit parameter sparsity, low-rank principles permeate various aspects of optimization mechanics. The Adafactor algorithm (Shazeer & Stern 2018) pioneers this direction by replacing full-rank momentum buffers with factorized recond-momen estimators, establishing a blueprint for memory-efficient optimizers. Subsequent developments like Sophia (Liu et al. 2023) integrate low-rank Hessian approximations with lightweight curvature estimation, achieving both speed and memory advantages. The Lion optimizer (Chen et al. (023) takes a radical approach through sign-based gradient compression, creating natural compatibility with low-rank parameterization. A paradigm shift occurs with Gal.ore (Zhao et al.
2024), which projects entire optimization trajectories into low-rank subspaces via Singular Value Decomposition (SVD). This framework inspires derivative works (Hao et al.) 2024 He et al. 2024 Chen et al. (025)) that replace computationally intensive SVD with stochastic orthogonal projections. To address information loss inherent in low-rank approximations, error-feedback mechanisms in (Robert et al.
2024 Chen et al.
2024a) iteratively compensate for residual gradients. Meanwhile, Apollo (Zhu et.
2024) reimagines
adaptive learning rates by incorporating projected optimizer states as scaling factors, demonstrating how traditional algorithms like Adam (Kingma & Ba 2014) can be retrofitted for low-rank efficiency.
Activation-efficcent training for LLMs. Activation memory optimization operates through dual pathways: algorithmic innovation and system-level engineering. Zero-order optimization methods (Malladi et al. 2023 Gautam et al.) 2024 Chen et al. 20224b circumvent backpropagation by estimat- ing gradients through forward pass perturbations, though their adoption is hindered by fundamental convergence limitations
in optimization theory (Duchi et al. 2015 Berahas et al.)
2022). The randomized subspace optimization (RSO) framework (Chen et al.) 2025) offers a middle ground by performing gradient updates in dimension-reduced spaces, thereby implicitly reducing activation storage requirements. On the systemic front, gradient checkpointing (GCP) techniques
(Chen et al. 2016 Feng & Huang. 202T He & Yu 2023) strategically store partial activations during
forward propagation and recompute missing values during backward passes, achieving linear memory
savings the cost of increased computation. FlashAttention (Dao al.] 2022 Dao 2023 Shah
et al. 2024) revolutionizes attention layer implementations through block-wise computation and dynamic memory management, effectively decoupling peak memory demand from sequence length. These complementary approaches collectively address the "memory wall" challenge in modern LLM training.
Cross-layer structure in transformer. Recent research increasingly highlights distinctive data distribution patterns across transformer layers, particularly large-scale model optimization. For inference acceleration, (Liu et al. 2024b) first identified high inter-layer KV-cache similarity, propos- ing shared caching mechanisms between adjacent layers-an insight further developed by (Brandon) et al. 2024) through cross-layer attention operators for dynamic KV compression. In pre-training contexts, architectural innovations like token-level attention initialization (Zhou et al.) 20244 Pagliar dini et al. 2024 Nguyen et al. 2023) enhance information propagation across layers. However, these innovations predominantly target attention layer optimizations while inadequately addressing two persistent gaps: the interplay of fred-forward network dynamics across layers remains underexplored. and existing frameworks demonstrate limited compatibility with low-rank adaptation paradigms, constraining their applicability to parameter-efficient training scenarios.
B COMPUTATION AND MEMORY ANALYSIS OF CR-Net WITHOUT RE-COMPUTATION
In this section, we present detailed computation and memory analysis of CR-Net.
15

---

Published as a conference paper at ICLR 2026
B.1 TRANSFORMER LAYERS WITH LLAMA ARCHITECTURE
In order to make a Comparison with other efficient methods, the computation and memory analysis based on LLaMA framework (Touvron al. 2023) with L transformer layers. Here, focus on the forward- and backward-propagation of CR-Ner. should be reminded that we omit the layer normalization, rotary position embedding, and in-layer residual for simply.
Forward Propagation. For the input X₁ E IR**,, where denotes the sequence length and ft denotes the hidden dimension. For the first layer, the transformer block's attention mechanism implements series of linear transformations through three fundamental computational stages:
y = YK = X,WI, Yy
=X,
(11)
where WP. WT. WY E Raxh are full-size weight parameters. For the other layers, the weight matrices of linear transformations are replaced by low-rank weights and the cross-layer residual should also be taken:
ye
=Y= X
XAY (12)
where A,, AX A R and Bi BR, By E R are low-rank parameter matrices. Then for 1 1= 1,2, L, the immediate activations can be combined follows:
At (0 Att = softmax
Att
Att: = Anty
(13)
Vh
Then for the first layer, the attention output can obtained by:
y Aut W
(14)
where WP E Rixh denotes the full-size output matrix. For = 2.3. ... the attention output can be obtained by:
= Aut Bi,
(15)
where Al Rixr and Bi- E harr denote the low-rank parameter matrices.
Next, the feed-forward network consists of three linear layers. Among them, the gate layer and up-projection layer are computed in layer as:
ygale -Y YIP WPWT, Xdown SwiGLU(Y YIP
(16)
where W WiP E ighxtn are full-size weight matrices. For the other layer, it can computed as: y.guie = geua y + y, p Baaee X Xdown SwiGLU(Y, Y,P,
y yup + Y B.
(17)
where APth A E gaxr and Bqaae B"P E Rr denote low-rank parameters and O denotes the element-wise production. Finally, the down-projection in the first layer can be computed as:
ydoun x down down
(18)
where Wddwnn ha are full-size weight matrices. For the other layer, it can be computed as:
Ydlown =odoon down + +X. - Adown Bulow
(19)
where Adown Rlaxr and Bdown E R denote low-rank parameters.
Backward Propagation. For matrix M and the loss function É, DM == 3M To compute the gradients of all the parameters in layer I, the back propagation begins with the partial gradient of the loss function & with respect the input of the next layer, i.e., DX1+1-
0t
Then, for the last layer, the gradient of the output holds that DY = DX where X denotes the input of output layers. However, for the other layer, it holds that:
Dy-doa 3down BooonDD-down
down
DXL+1.
(20)
16

---

Published as a conference paper at ICLR 2026
For the first layer, the gradient of inputs and weights for down-projection be computed
D DY (Wrown)T, DW (X "DY
(21)
For the other layers, the gradient inputs and weights for down-projection can be computed as:
DXpowm =DY,own (Afonn B DBp
(XXoow Adown) DYy
(22)
DA = (x Dopoownn
For the last layer, the gradient of activations do not rely on the former layers. Thus the gradient of y and Y. can be obtained as:
DYL DXLL @ SwiGLU(YZ
D = DX @ Y @ SwiGLU"(YZ")
(23)
For the other layers, the corresponding gradient of activations should combine the prok-propagated gradients and the historical gradients. Thus it holds that:
1+1 DY, 3 DY + DX @ SwiGLU(Y,
(24)
Dy B (41 DXx & Y1Th @ SviGLU'(Yy"u))
Then the gradient for parameter of up-projection and gate in the first layer can be computed by:
DYP D.DY, + DD," (WT") + DY (W DW (YT9) DY,", DWT (Y,9) DY,
(25)
For = 2, .. L 1, the gradient parameter of up-projection and gate the layer can be computed by:
DY, B (A, + DY, (A B
DB = (Y, A DYP DA T
(26)
DB (V, Alowa) Dydoon DA (270) T D D
The gradient for parameter of up-projection and gate in the last layer can be computed by:
DYL (A DY (A 1 B )
DB AY) DY DAY = (Y DB
(27)
DB (Y Adowa) DyLowa DA
Then for the first layer, the gradients for parameter of output layer in attention can be computed by:
DA TWP,, DWP (Au)) DY.
(28)
For the other layers, the gradicnts can be computed by:
DAul (AP) DB. = (Att) AP) DYP, DA = DBB.
(29)
Next, the gradient for Atti can be computed by:
DAtti = Aut
(30)
Then for the last layer, the gradient for the activation y YK Vy can obtained by:
Att.
DYY T DAul, DY =
DAILi
softmax'
Yk.
Vh
Vh.
(31)
1
Att VT
Dyk = DAIti 3
softmax'
Yo
Vh
17

---

Published as a conference paper at ICLR 2026
And the gradients of the other layers be obtained by:
DY 3 DY + (At)) DAU
Att:
DY, Pi DY +
DAtt O
softmax
Y
Vh
vh
(32)
Attz 11O.
DY = D DY +
DAtt O
softmax
Vh
Vh
Finally, the gradients of the weights for query, key, and value in the first layer can be obtained as:
DW -xx) DWF = (X)) DW = DYY,
(33)
DX, (W) (WF) + DDY
the gradients of the weights for query, key, and value in the other layers can be obtained as:
DBP =XxAA DBK (XIAR)TDX
DBY = (XIA))
DA -Xi) DA = DBF, DAY = (Xi)
(34)
DX, =DY, + DYY +DYY
B.2 COMPUTATION ANALYSIS WITHOUT RE-COMPUTATION
Here we present the computation analysis of CR-Net with re-computation in detail. The brief analysis shown in Section 4.1 To begin with, we should remind that the matrices production with size n and
need 2mnr FLOPs.
For the first layer, the computation of forward propagation is totally the same as full-rank training. Thus the computation FLOPs can be computed by:
Attention Q, K, V: Three matrices productions with size x and h x h, 6sh FLOPs in total. Attention SDP: One matrices production with size s x h and h x s and one matrices production with size x and x h, 4s2h FLOPS in total.
Attention O: One matrices production with size h and h x h, 2sh FL.PPs in total.
FFN gate and up: Two matrices productions with size 8 x and x hr, 4shhg FLOPs in total.
FFN down: One matrices production with size x bo and ho h, 2shhg FLOPs in total.
Thus, the computation complexity of the first layer
8sh² 2 + 4s h + 6sh/q-
For the other layers, we can omit the cross-layer residual operations as its complexity lower-order term. Thus, the computation complexity of different components are as follow:
Attention Q K, Three matrices productions with size X and h x r, and three matrices
productions with size x and h, 12shr FLOPs in total.
Attention SDP: The same as that of the first layer, 4s h FLOPs in total.
Attention O: One matrices production with size x h and x r, and one matrices production with size and h, 4shr FLOPs in total.
FFN gate and up: Two matrices productions with size x and h x and two matrices productions with size $ X and r X hg, 4sr(h + hg) FLOPs total.
FFN down: One matrices production with size s and r, and one matrices production with size r and r x hn, 2sr(h hn) FLOPs in total.
Thus, the computation complexity of the other layers is
16shr 4 h + 6sr (h ha).
18

---

Published as a conference paper at ICLR 2026
Finally, the total computation complexity in the forward- pwwaard-ropagattonn..o is:
8sh + 4s + 6shha + +L 1)(16shr + 4s + 6sr(h + hg))
(3.5)
FLOPs in the first layer
FLOPs the other layers
Backward propppattooopatttionn For backward propaarddrrppaatioo.n. It can obtained that the toatl computation if we omit the lower-order terms. Thus, the total FLOPs is twice of that in forward-propagutte propagation is: computation complexity in the forward
16sh 8 + 12shhg +(L (3(shrr + 8 + 12sr(h + ha))
(36)
FLOPs the first layer
PLOPs in the other layers
Thus, we can obtain that the total computation of CR-Nel for one gradient step is:
24sh² + 12s²h + 18shhn + (L - 1)(4srr + 12s + 18sr(h h + ha)).
(37)
C COMPUTATION AND MEMORY ANALYSIS WITH RE-COMPUTATION
C.I ACTIVATION MEMORY AND RE-COMPUTATION ANALYSIS OF CR-Nel
In this section, we present the computation and memory analysis of CR-Ner with re-computation strategy which has been shown in Section 3.3
C.1.1 MEMORY ANALYSIS
We first present the analysis of memory overheads of CR-Net with re-computation. In fact, the following variables need to be stored:
Inputs for each layer: Lsh parameters in total.
Low-rank outputs XP for all linear layers except the first transformer layer: 7(I 1)
parameters in total.
Linear outputs X AP BP for 1 E A: 5|A4sh + 2|,4shhr parameters in total.
Thus, the total memory overhead of CR-Ner with re-computation is:
(L + 5|A4)sh + 2|A[sh + 7(L - 1)sr.
(38)
C.1.2 COMPUTATION ANALYSIS
Then we present the analysis of re-computation overheads CR-Net, which can be computed by:
Attention Q. K. V: If E A, no FLOPs need as all the activation with is necessary for the backward- propagation has been stored. I A, the linear combination between low-rank output and the matrix B BK BY and the stored matrices X,A, X,A, XG,A respectively. The shape of the multiplied matrices are x r and r x respectively. Thus the total FLOPPs 6(T. [A()shr.
Attention SDP: All the layers should take one matrices production with size s and s and matrices production with size x and x h. Thus the total FLOPs is 4Lshh.
Attention O: It is the same the discussion for attention Q. K. V. Thus the total FLOPs is 2(L - s
FFN gate and up: It is the same as the discussion for attention Q, K, V. Note that the size of the multiplied matrices are
and rtimeshys respectively, thus the total FLOPs is 4(L- - [A))sher. FFN down: It is the same as the discussion for attention Q, K, V. Note that the size of the multiplied matrices are X r and rtimesh respectively, thus the total FLOPs 2 [Al)shr.
Thus, the total re-computation complexity is:
10( [A))shr +(T (Alsshar 4T.shh.
(39)
19

---

Published as a conference paper at ICLR 2026
C.2 RE-COMPUTATION COMPLEXITY OF CoLA-M
Table the recomputation complexity of activation-effeectvee approaches, including CoLA-M. In fact, the re-computation complexity of CoLA-M can be obtained by the similar analysis process of The total re-computation complexity for one gradient step of CoLA-M is:
CR-Nef in Appendix C.1.2
10Lshr + 4Lshar + 4Ls"h.
(40)
When taking the intermediate dimension hi 8h/3, the total re-computation complexity for one gradient step of CoLA-M is:
20.67 + 4Ls
(41)
In Liu et al. takes half of total flops of linear layers with the assumption. However, for gate and up-projection layer, the production of X/ R (X²") and Afaee E R (A(P) takes 2shr FLOPs while production of X1 Apal E xx (X,"",") A and B # (B,T) takes 2sh FLOPs. For down nown-projection layer, the production of X-lown E Raxhr and Adunn R takes 2s/rr FLOPs while production of X.own Adown [R*** and Bdown R takes 2shr FLOPs. Thus, the computational FLOPs of the FFN are not given by 3shar + 3shr-han of the total FLOPs but rather by 4shgr + 2shr.
2025), the re-computation FLOPs of CoLA-M is obtained as the re-computation only
D THEORETICAL INSIGHT: RESIDUAL SUBTRACTION IMPROVES LOW-RANK APPROXIMATION
In this section, present theoretical insight to explain the phenomenon we observed in Section 3.1 To begin with, we present an assumption of the cosine similarity of the activations for adjacent layers. Assumption 1 (Cosine similarity of adjacent activations). Forl=30 .... let Y," E [Rd the activation of the linear of position P for the 1-th layer. There exists constant (0, 1) such that:
(YP, YL,)F
| F 21-2,
where ) denotes the inner production of matrices induced by Frobenius norm.
Assumption Diuumption[lllustrattts Dillustrates that the activation of Transformer-basee models have high cosine similarity. Such a characteristic have been observed in different kinds of models and different positions (Liu
al. 2024b Hao et al. 2025 Jiang et al., 2024).
Moreover, we present the definition the stable rank.
Definition (Stable rank). For A gdxd, the stable rank of A (represented as sp( is denoted as the ratio between the square of Frobenius norm of and the square t2 f2-norm of A. Specifically, holds that
| TA
(AA)
Remark 1. The stable rank can be also defined as the ratio between the quadratic sum of singular values and the square of the maximum singular value.
Then we can obtain the theorem that the approximation in Eq. 33) is better estimator than low-rank approximation, especially with small rank
Theorem 1. Suppose Assumption 7 holld. Then there exists ro such that the approximation Yi, obtained by Eq. (3) has lower error than the direct low-rank approximation (Y,) by a properly-selecteee if To- Specifically, it holds that:
rP Vro 1|
VI (YP))
Proof. First, we denote 0.(.) as the i-th singular value of matrix, which is listed in descending order. Then, the term AgY? holds that:
| YYp BY, ||Y: 23 (3 (.P.YL,) + 1
(42)
20

---

Published as a conference paper at ICLR 2026
Thus
|| - -1 23 (XP, -32 1
(43)
From Weyl's inequality, the i-th singular value of y? and A, holds that:
- -
1/ A < sB
(44)
Thus, it holds that:
- o3 - 177 -3
<3
(o.))) ( +
(45)
<3 1 (2), + 8 ||
Taking summation over i = 1,2, ............. ,, where 1 S r S d, it holds that:
Σ (di(x))) - -??0x)))) 5B 1
2 Σ
(x,) + Br 1r1x1_1l12)
t=1
1/2
(46)
s 1 11
2VF Σ o?(xP))
+ Br |
1 (2vr || r + Br
(1))
where the second inequality is due to Next, we consider the approximation error for LR, and VP... We can obtain that:
inequality.
YP v.. 1 -1V? (YP)
r Σ |18.111 -
||A Σ Eoi(Ao1?)
s 23 (Y,, x + 8 Y 1p 28VF
||P || + f3r ||
(47)
s 23(1 s) || + 1 | 11 ||x|' ||
83r ||r 1
=B² (1x=-111 +r -1--- - | ||] lT
+r
-
=3² (I1E2 +r -r> - 23
(0 vr) ||
where the second inequality is due to Assumption
I
--P-P- "p 2 < Y - LX,
Taking = (1 - =))pp)))(EE))) Ifr S ro. then
holds if
F
8 = (1 e) / vr) || ||P|IF
|1_ _-- + r yP,11 11
Thus, we finish the proof of this theorem.
E EXPERIMENTAL DETAILS AND ADDITIONAL EXPERIMENTS
E. EXPERIMENTAL DETAILS FOR THE OBSERVATION OF CROSS-LAYER LOW-RANK STRUCTURE
To empirically validate the cross-layer low-rank structure in different models and training periods..
we conducted comparative experiments by fine-tuning the pre-trained LLaMA-3 8B (Grattafiori et al. 2024) and GPT-2-small (Radford et al. 2019) models. And we also pre-training L.LaMA-3 8B
21

---

Published as a conference paper at ICLR 2026
Table 7: Hyperparameter configurations for Qwen-3-based model with MoE architecture.
Hidden
Intermediate
MoE intermediate
KV Heads
Heads
1024
2736
704
16
Layers
Experts
Activated experts
Steps
Training tokens (B)
28
4
60K
7.8
Table 8: Hyperparameters for fine-tuning experiments for the low-rank observation.
Hyperparameters
LLaMA-3 8B
GPT-2 small
Qwen-3 MoE
Optimizer
AdamW
Learning rate
le-5
le-5
2.5e-3
Total batch size
128
1024
256
Sequence length
64
512
256
Warmup iterations
500
500
1000
Evaluate every steps
20
10
500
Update Steps for fine-tuning
2000
2000
N.A.
Update Steps for pre-training
100
N.A.
100
O Here we aim to validate the cross-layer low-rank property in the initial period of pre-training process. Thus we only need train the model with fewer steps.
Approdmation
D00-ant rant
EreVous baya
crevious loyal + curt
Rebativv Browr
Figure 7: The average relative error of activation recovery by using low-rank approximation and using B over all transformer layers. (Left: LLaMA-3 8B, right: Qwen-3-MoE.)
model as well as model built from Qwen-3 MoE model (Yang et al. 2025). The hyperparameter configurations for Qwen-3-based model is as Table We use the TinyShakespeare dataset for both pre-training and fine-tuning.
We subsequently quantified the activation approximation errors through two distinct methodologies: (a) direct low-rank estimation LY,((Y)) and (b) matrix recovery via Equation (3). The rank parameter
0.25h, 0.5h respectively. For Ye,0 defined in Eq. (3), the scaling factor So is selected from the set {0, 0.2, 0.4, 0.6, 0.8, 2, 3, 5} minimize the relative error given by And the other hyperparameters can be referred from Table
Figure [ also presents detailed relative error of activation approximation for each layer in fine- tuning tasks. For I.LaM--base models, the linear layers at position V, up. down suffer from a higher approximation error, calling for higher rank to enhance the model performance. For GPT-based models, the linear layers position and down-projection suffer from higher approximation emror than the other layers.
Figure illustrrate the average relative error of activation recovery by using low-rank approximation and using 3 over all transformer layers in pre-training tasks. Same as the observation in Section (.1] it can be observation uniform advantage for estimating activation by the previous layer and a low-rank approximation the difference between the layers, which illustrates that the cross-layer low-rank structure for activation exists in the initial stage of training.
Furthermore,
22

---

Published as a conference paper at ICLR 2026
Rudiadios ocrers cack
LiaMM (Hotis 6,28)
Rebetivo atroon lagwe Llakd aodi Ouatios BLME)
031
layer
layer
Hip Hilp oown
Ralabur Brooe
Belabbie Erroe
10-1
20-
nig. mip.dwwn.ng
Layer
Figure 8: The relative enror of activation approximation by using (3) for each transformer layers. 'Ratio' stands for the ratio between the low-rank dimension r and the hidden dimension h. (Top: LLaMA-3 8B, bottom: GPT-2 small.)
Table 9: Hyperparameter configurations for LLaMA-2 models of different scales, along with the corresponding number of training steps.
Parameters
Hidden
Intermediate
Heads
Layers
Steps
Training tokens (B)
60M
512
1376
8
10K
1.3B
130M
768
2048
12
20K
2.6B
350M
1024
2736
16
24
60K
7.8B
1B
2048
5461
100K
13.1B
7B
4096
11008
32
32
150K
19.7B
13B
5120
13653
40
200K
26.2B
Table 10: Layer rank selections for pre-training LLaMA models with CR-Ner under the scenario of aligned parameters (marked as ) in Table 3) and aligned memory (marked as i in 3).
Parameters
Rank for aligned parameters
Rank for aligned memory!
60M
Layer 2-4: 96. Layer 5-8: 1 112.
Layer 2-4: 96. Layer 5-8: 112. Layer 2-4: 192. Layer 5-12: 256.
130M
Layer 2-4: 192. Layer 5-12: 224.
350M
Layer 2-16: 224. Layer 17-24:256.
Layer 2-24: 384.
IB
Layer 2-24: 448.
Layer 2-24: 768.
7B
Layer 2-32: 896.
N.A.
13B
Layer 2-40: 1260.
N.A.
E.2
EXPERIMENTAL DETAILS FOR C4 PRE-TRAINING TASKS
E.2.1 BASIC EXPERIMENTAL SET UP
During pre-training across all LLaMA model scales, we implement the standardized configuration framework from (Zao et al. 2024), with key technical specifications comprising a 256-token maximum sequence length and global batch size of 512 samples, translating to 13.1K tokens per batch. The learning rate scheduling integrates two-phase optimization: initial linear warm-up during the first 10% of training iterations, succeeded by cosine decay gradually reducing the learning rate to 10% of its initial magnitude. Complete architectural configurations and training protocol details are systematically documented in Table 9
23

---

Published as a conference paper at ICLR 2026
Table 11: Comparison of validation perplexity (4) of different approaches in LLaMA-2 1B pre- training tasks. The results of compared methods are referred from (Liu et al. 2025
Zhu et al. (024).
Training tokens (B)
3.5
10.7
13.1
LoRA
N.A.
N.A.
19.21
CoLA
N.A.
N.A.
15.52
CR-Net
19.20
15.52
15.22
Table 12: Comparison of validation perplexity (4) of different approaches in LLaMA-2 13B pre- training tasks.
Steps
10K
20K
30K
40K
Full-rank with 8-bit Adam
24.31
20.53
18.84
17.85
CR-Net with re-computation
25.99
21.73
19.32
18.12
As for CR-Net, turn the learning rate over (0.0003, 0.0006, 0.001, 0.005, 0.01, 0.012,012222. 0.015, 0.02) and select the learning rate that achieve the lowest validation perplexity over 10% training steps for the whole training process. For low-rank parameter matrices, the learning rate is multipled by 0.25 for scaling. The selection of ranks of CR-Net when training with aligned parameter and aligned memory are listed in Table TO
Experimental setup for training and inference throughput evaluation. To measure throughput performance, we pre-trained the LLaMA-2 IB model on an A100 40G GPU with a batch size of 16. For CR-Ner, we evaluated both single-GPU configurations and multi-GPU scenarios employing data parallelism across 4 GPUs. For inference throughput measurements, we utilized a A100 80G GPU with microbatch size of 64. Additionally, when assessing throughput for the LLaMA-2 2 7B model, we employed a A100 80G GPU with batch size set to 512. all experiments, we utilized the maximum microbatch size that could be accommodated without triggering out-of- errors.
E.2.2 CR-Net ACCELERATES THE PRE-TRAINING TASKS
According to the evaluation perplexity results in Table CR-Net achieves perplexity of 15.22 when training the LLaMA-2 IB model on 13.1B tokens, representing an approximately 2% improvement over both full-rank training and CoLA. Regarding pre-training acceleration, a comparison of validation perplexity during the training of LLaMA-2 1B is provided Table [1] CR-Ner reaches the same perplexity as LoRA and CoLA with only 29.02% and 81.52% of their training steps, respectively. When considering the per-step computational FLOPs listed in Table 19 CR-Nel achieves a 1.326x acceleration over CoLA and 18.489 acceleration LoRA.
E.2.3 SCALING CR-Net TO LARGER MODEL SIZE
To validate the performance of CR-Net with larger model size, we pre-train a LLaMA-2 13B model under 8-bit Adam and CR-Net separately. We use the re-computation present in Section 3.3 for CR-Net and save series of full activation every layers. We trained 40K steps due the time limitation. The hyperparameter in CR-Ner is set to 1260.
The evaluation perplexity shown as Table T2 illustrates the performance of CR-Net in 13B model, which demonstrates that CR-Nel achieves more than 50% reduction in parameters, while incurring only a 2% degradation validation performance.
E.2.4 PRE-TRAINING CR-Net WITH ACTIVATION QUANTIZATION
To validate the performance CR-Net with activation quantization, we pre-train a LLaMA-2 350M model under activation quantization. We adopt the same quantization scheme as Q-LoRA Dettmers
et al. (023),, which employs NF4 precision and the DoubleQuant strategy with a block size of 16. Table 13 presents the evaluation perplexity of CR-Net with and without activation quantization. compared to full-rank training. The results indicate that CR-Net with 4-bit activation quantization achieves additional activation memory savings while incurring less than a 3% increase in perplexity.
24

---

Published as a conference paper at ICLR 2026
Table 13: Comparison validation perplexity (4) for CR-Net with and without activation quantization
in LLaMA-2 350M pre-training tasks, Training tokens
6.4B
Full-rank
18,80
CR-Ner
18.95
CR-Net with activation quantization
19.32
Table 14: Comparison of validation perplexity ()) and accuracy (1) of fine-tuning tasks with different approaches.
Dataset
Wikitext
ArXiv
perplexity
accuracy
perplexity
accuracy
GaLore
21.98
40.86%
25.17
40.59%
CR-Net
20.66
41.14%
24.48
41.00%
E.2.5 DOWNSTREAM PERFORMANCE
To examine the downstream performance of our proposed framework, we fine-tune LLaMA-2 IB models pre-trained by GaLore and CR-Net for 15.6B tokens. The training set up the same that in the manuscript. The fine-tune datasets consist of Wikitext-2 dataset (Merity et al., (016) and 30K arXiv abstracts (Wang al., 2022). We tume each data for epochs. Table presents the evaluation perplexity and accuracy of these tasks, which illustrates that CR-Ner has better performance than GaLore these downstream tasks.
E.2.6 EXPERIMENTAL SETTING AND ADDITIONAL RESULTS FOR THE ABLATIONS
In this section, we present the experimental setting and additional results for the ablations we presented in Section 5.2
How does rank selection impact pre-training performance? In this ablation, we train LLaMA-2 350M with CR-Ner for 60K iterations. The max learning rate is set 0.01 and all the other configs is the same as that in Appendix [.2.1 For the selection of layer ranks, we present the following three strategies ensure the their parameters are all the same:
S1: 256 in Layer 2-8. 192 in Layer 9-24.
$2: 256 in Layer 10-16. 192 Layer 2-9 and Layer 17-24.
S3: 256 Layer 18-24. 192 in Layer 2-17.
Whether does the learnable scale factor Bf benefit the model convergence? In this ablation, use CR-Net to train LLaMA-2 350M and LLaMA-2 1B model with learnable of and series of fixed 3 for 2,3, ... and P (Q. K, O, gate, up, down). respectively. The fixed 3p varies from 0.1, 0.2, 0.5, 1.0, 2.0 for LLaMA-2 350M and 8 = 1.0 for LLaMA-2 IB. We trained 40000 iterations for LLaMA-2 350M and 80000 iterations for LLaMA-2 1B, respectively. Figure illustrates the evaluation perplexity of each cases, shown the benefit of letting S learnable as the higher validation performance.
Whether other cross-layer residual strategies do well in pre-training with low-rank parameters? To compare CR-Ner with different efficient cross-layer residual strategies for L.MMs pre-training including ResFormer (Zhou al.) (024) and DenscFormer (Pagliardini et al.) 2024). pre-trained LLaMA-2 IB model with CR-Ner, Learnable- RestubbbeeeReFFreer.r.. and DenseFormer using low-rank parame- ters. The rank for transformer layers (except the first layer) is set to 448, while all other configs the same as those in Appendix E.2.1 We stopped training early at the 50,000-th step.
Figure Glillustrates the evaluation perplexity of CR-Ner and other cross-layer residual strategies for the LLaMA-2 IB training task. It can be observed that CR-Net outperforms the ResFormer architecture and Fonmer in pre-training with low-rank parameters.
25

---

Published as a conference paper at ICLR 2026
200
0-001
8.
1.5
Ck-Net with iambbll CR-lee with fioed
277
2.0.
1.5
125
8.0
10.55
15.00
23
30
15
5000 10000 15000 20000
25000
30000 35030
40000
9
10000 20000 30000 42000 50000 60000 70000
80000
iterations
iterations
Figure 9: The Comparison of evaluation perplexity for CR-Net in pre-training tasks with fixed By and learnable B.. (Left: LLaMA-2 350M, right: LLaMA-2 1B.)
Table 15:
Comparison of validation perplexity (4) of different approaches in LLaMA-2 pre-training tasks with longer sequence length and more training tokens.
Model size
130M
350M
1B
Training tokens (B)
2.9
8.0
29.5
GaLore
25.14
18.87
15.03
CR-Net
23.73
18.86
14.79
E.2.7 PRE-TRAINING WITH LONGER SEQUENCE AND MORE TRAINING TOKENS
To align with the basic experimental setup of existing efficient frameworks for LLMs pre-training like LoRA and GaLore, we follow the setup in Zhao et al. (2024) and use a short sequence length, to validate the pre-training performance in long training tokens well pre-trained L-aMA-2 models using the C4-en dataset with maximum sequence length set to 2048. The total training tokens more than 22x the parameter size follow the Chinchilla scaling law
training tokens. We
(Hoffmann et al. 2022). As shown in Table 15 CR-Ner outperforms GaLore in this scenario.
E.3 PRE-TRAINING GQA AND MOE MODELS WITH CR-Ner
In this section, we present additional experiments on pre-training the LLaMA-3 model (Grattafiori et al.) (2024) with a grouped query attention (GQA) (Ainslie et al., 2023) architecture, well as pre-training a Qwen-3-based (Yang et al. 2025) model with a mixture of experts (MoE) architecture, validate the proposed framework across different model structures.
Pre-training LLaMA-3 with GQA architecture. We pre-trained 1 1B-parameter LLaMA-3 model on 15.6B tokens from the C4-en dataset using both GaLore and CR-Net methods. The model hyperparameters are shown in Table 16 The low-rank coefficient rT was set to 512 for GaLore and 448 for CR-Net. The remaining configuration follows the same setup as the LLaMA-2 pre-training
task described in Appendix E.2 The evaluation perplexity results are presented in Table 17 indicating that CR-Net outperforms GaLore in pre-training tasks on the LLaMA-3 model.
Pre-training models with MoE architecture. We pre-trained a Qwen-3- model under the
MoE architecture, containing 1.8B total parameters with 650M activated parameters, on 7.8B tokens from the C4-en dataset. Training was conducted under both full-rank and CR-Net settings. The model hyperparameters are provided in Table 7 The low-rank coefficient rr CR-Net was set to 224. For CR-Net, replaced the parameter matrices within the linear operators of the MoE layers. Each MoE layer was treated as unified operator, and cross-layer residual connections were applied after obtaining the output of the MoE operators. The remaining experimental setup matches that of the LLaMA-2 pre-training task in Appendix E.2 The evaluation perplexity, shown Table I8] demonstrates the competitive performance of CR-Net compared full-rank training.
26

---

Published as a conference paper at ICLR 2026
Table 16: Hyperparameter configurations for LLaMA-3 model.
Parameters
Hidden
Intermediate
KV Heads
Heads
Layers
8B
2048
7168
32
32
Table 17: Comparison of validation perplexity (4) of different approaches in LLaMA-3 pre-training tasks.
40K
80K
100K
110K
Galore CR-Net
19.29
16.89
16.47
16.40
18.29
16.05
15.70
15.65
Training tokens (B)
5.2
10.5
13.1
14.4
F ADDITIONAL DISCUSSIONS
F.1 PER-STEP COMPUTATION ACCELERATION ANALYSIS.
To illustrate the computation efficiency of CR-Ner in practical training tasks, we provide the com- putation complexity of different efficient pre-training approaches for one gradient step (based on LLaMA-2 architecture) Section 5. as Table 19 can be observed that CR-Nei achieves the same training performance full-rank training with 36.8% computation overhead for both LLaMA-2 1B and 7B models.
F.2 ADDITIONAL DISCUSSION FOR THE LEARNABLE SCALING FACTOR SP.
The layer-specific learnable scaling factor oP serves as critical dynamic balancing mechanism between historical information propagation and new low-rank feature generation, rather than conventional hyperparameter. When exceeds appropriate ranges (e.g., of 2.0), low-rank features dominate current activations, whereas values approaching zero excessively prioritize historical information both scenarios are demonstrated in Figure adversely impact model performance.
Notably, S. exhibits multidimensional adaptability requirements: optimal values vary across layers, spatial positions, and temporal phases of training. This intrinsic variation motivates our proposal for learnable 3f parameterization. As evidenced by Figure and dynamic 33f adaptation yields consistent performance improvements across model scales compared to static configurations.
To examine the empirical value distribution for beta terms, we obtain the distribution of all of terms in pre-trained LLaMA-2 350M model with 20,000 steps. As shown in Table 20 the BP terms mostly range from 0.20 to 1.00, where the model may not suffer from the negative impact of extreme betas.
G MULTI-GPU TRAINING: AN ANALYSIS FOR CR-Net WITH PIPELINE PARALLELISM
While the single-GPU throughput results (Figure 3 validate the computational efficiency of CR-Net. multi-GPU training especially with parallelism strategies like Pipeline Parallelism (PP) (Narayanan et al. 2019: 2021 Huang et al. 2019 Ryabinin et al. 2023 int-oduces critical considerations around communication overhead. Cross-layer residual connections, a core component of CR-Net, could theoretically increase data transmission between GPUs, as forward/backward passes require sharing activation residuals across worker nodes. To address this concern, we conduct a quantitative tradeoff's for CR-Net in PP-enadled multi-GPU setups as well as an empirical verification. We also present an analysis of HBM memory with pipeline parallelism.
analysis of computation-commuiice
Basic setup. We conducted a pre-training task of CR-Net with re-computation and full-rank training with vanilla in pipeline parallel deployment A100 80G GPUs for L.MMA-2 models with 13B
27

---

Published as a conference paper at ICLR 2026
Table 18: Comparison of validation perplexity (L) of different approaches in Qwen-3-based pre- training tasks.
10K
20K
30K
40K
Full-rank
24.31
20.53
18.84
17.85
CR-Net
25.99
21.73
19.32
18.12
Training tokens (B)
1.3
2.6
3.9
5.2
Table 19: Computation complexity of different efficient pre-training approaches for one gradient step based on LLaMA architecture for the task in Section 5.1 Lower-order terms are omitted for brevity. The selection of the rank r for CR-Net are based on Table 10 while that of the other algorithms are
based on (Zhao et al., [2024] Liu et al., 2025).
Approach
LLaMA-2 350M
LLaMA-2 IB
LLaMA-2 7B
Full-rank
4.838 x 10" (1. x1)000x)
2.525
10 (1.000x)
1.005 x 10 (1.000
LRe)
8.128 x 10" (1.67
4.226
10 (1.)73x)
1.678 x 10ts (1.670x)
SLTrrin
9.482 x 10 (1. x1)960x)
7.473
10 (2.959x)
4.984 1013 (4.959x)
GaLore
7.934 x 10 (1 640x)
5.824
10 (0.30)x)
3.658 x 10 (3. 639x)
CoLA
1.934 x 10th (0. x00x)
1.005
10 (9839)x)
0.398 x 10 (0. 396x)
CR-Net
1.985 10 (0. 410x)
0.930 10 (0.368x)
0.369 10ah (0. 367x)
Table 20: Distributions of beta values for pre-trained LLaMA-2 350M model.
Beta
[0.00, 0.20)
[0.20, 0.60)
[0.60, 1.00)
[1.00, 1.25)
Frequency 0.05625
0.2625
0.55
0.13125
and 70B parameters. We assume the intermediate dimension of FFN layers is 8/3 times the hidden dimension. The other model hyperparameters are listed in Table[1]
For CR-Net, we set the low-rank coefficient r 0.25/ and save the entire series of activations every 8 layers for re-computation. We ignore higher-order computation overhead in the complexity analysis. Following the analysis in our manuscript, the computation complexity of CR-Net and full-rank training for one gradient step can obtained.
Communication overhead. Then, when using BF16 precision, the communication overhead can be computed by:
Microbatch x Communication dimension x x 3
Volume (GBytes)
(48)
109
where the multiplication by 2 due to the 2-byte requirement for storing each parameter in the BF16 precision, and the multiplication by 3 accounts for the communication overhead incurred during the forward pass, backward pass, and re-computation
Using the peak computation performance of A 100 with BF16 (312 TFLOPS) and PCIe 4.0 bandwidth (64 GB/s), the computation and communication times per gradient step are shown in Table 22 Notably, CR-Ner's computation time savings are overwhelmingh, significant exceeding about 30 times of the additional communication time for 13B and 70B models respectively- far outweighing the minimal communication overhead. Moreover, when using a GPU-connection strategy with higher bandwidth (e.g., NVLink), the communication time can be further reduced, allowing computation time to dominate the training process even more significantly. This results in greater total time savings due to CR-Ner's computational efficiency. Since inference mirrors the forward pass of pre-training these findings directly validate that CR-Net's computation efficiency gains surpass trivial communication costs, confirming its superiority in multi-GPU deployment.
For full-rank training, if split the model the end of transformer layer, then the re-communication may not lead communication overhead here.
28

---

Published as a conference paper at ICLR 2026
Table 21: Hyperparameter configurations for LLaMA-2 models of different scales, along with the corresponding number of devices for a propagation pipeline. 'Microbatch" denotes the size of one microbatch. 'PP size' number of devices for GPU pipeline for the forward- and backward- propagation of one microbatch.
Parameters
Hidden
Heads
Layers
Sequence length
Microbatch
PP size
13B
5120
40
40
4096
16
2
70B
8192
64
80
4096
16
8
Table 22: Computation and communication times for one gradient step. CR-Ner achieves time saving of 13.42s and 127.63s in 13B and 70B training tasks per step, respectively.
Computation FL.OPs (x10'))
Computation time (s) overhead (GB)
Communication
Communication time (s)
Parameters
Methods
Full-rank
7.48
23.97
1.88
0.029
13B
CR-Net
3.19
10.23 (13.74-)
22.5
0.352 (0.323+)
Full-rank
36.67
177.53
21.00
0.328
70B
CR-Net
14.51
46.51 (131.02-)
238.0
3.719 (3.391+)
Empirical verification. To empirically validate communication efficiency, we pre-trained a LLaMA- 2 70B model with CR-Net under pipeline parallelism using 8 NVIDIA A800 GPUs interconnected via NVLink, the C4-en dataset (sequence length=256). Pipeline parallel training was implemented with the torch. distributed. wwweeiiippplleeeeeee library, with each GPÚ handling 10 transformer layers.
The measured communication time for forward and backward step 1.380 seconds, yiclding a bandwidth of 192.5 GB/s. This bandwidth approximately three times higher than the PCLe bandwidth assumed in our theoretical analysis (Table (2). These findings demonstrate that under NVLink connectivity, CR-Net achieves shorter communication times than theoretically projected, further confirming that for large scale models, the communication volume does not substantially impair overall efficiency, thus supporting our theoretical conclusions.
HBM memory. We also clarify that CR-Ner low-rank parameterization inherently reduces overall HBM memory requirements. For the pre-training tasks for LLaMA-2 70B with BFI6 precision, under the configuration outlined in Table 21] CR-Net introduces 11.GGB additional activation memory device versus 1GB in full-rank training. Meanwhile, CR-Ner achieves about 50% reduction in memory for parameters, gradients, and optimizer states, yielding net 29.6GB HBM saving per device. This memory efficiency enables LLM training with fewer devices, fundamentally alleviating both communication overhead and collective HBM demands.
LLMs USAGE.
In this paper, generative LLMs were used solely for writing polishing, such as grammar and wording improvements. All LLM-edited content was manually verified to ensure compliance with ICLR policies, and authors bear full responsibility for the submission.
29


Reviews:
[{"review_id": "zZRFW33UtY", "reviewer": "Reviewer_kvx9", "summary": "The authors empirically discovered that the low rank reconstruction error for the *difference* from the previous layer's activations is less than that for the layer's activations, relatively, suggesting that we should do low rank modeling for this difference instead. The scale of the previous layer's activation $Y_{l-1}^P$ going into the current layer $Y_l^P$ is a learnable parameter, balancing between whether it relies heavily on the low rank residual term or refines from the previous activation. Moreover, the first layer uses full rank.\n\nThe paper proposes activation-efficient re-computation to address the challenges of new dependencies for gradient checkpointing.\n\nExperiments provided for pretraining Llama 2 with varying sizes up to 7B on the C4-en dataset and comparing with methods with similar low rank complexity or full rank with similar optimizer memory overhead.", "questions": "- In the experiment leading to the observation, what is the value of beta_0 and how did you choose it? Do results hold with varying selections of beta_0?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is simple but works well. The flow of the paper is intuitive, starting from an empirical observation to developing the full method.\n- The authors propose beta_0 as a scaling factor and provide a theoretical insight that under the assumption of high cosine similarity between activations, we can select a beta_0 such that the error for eq (3) is less than the regular low rank approximation, which seems reasonable to me.\n- The paper includes complexity analysis", "weaknesses": "- The communication overhead from the residual is high in multi gpu training, although the authors argue that the computation time outweighs that. However, it is unclear if the communication volume would affect real-life large scale distributed training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E8wvE8rjBg", "reviewer": "Reviewer_DNpk", "summary": "This paper proposed a new parameter-efficient pre-training framework for LLMs, termed CR-Net. Specifically, CR-Net is designed to exploit the inherent low-rank structures (indicated by Figure 2) of the difference between activations of adjacent layers. It replaces the full-size weight parameters at each layers with low-rank parameterized matrices and learnable linear combination factors that reconstruct the activation of current layer from that of the previous layer (Equation 6). The authors also proposed a novel activation-recomputation strategy (Algorithm 1) to further reduce the memory footprint of the model in propagation, which is essential for large-scale pretraining.", "questions": "See **Weaknesses**.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors evaluate the CR-Net's performance on the C4 dataset with LlaMA-2 models scaling from 60M to 7B parameters (Table 3 and Table 4), showing that CR-Net achieves a comparable or better perplexity than the full-rank model, with a significant reduction in the number of parameters and the amount of computation. \n\n- The authors also conduct systematic analysis on the FLOPs and parameter complexity of CR-Net and several related baselines (Table 1 and Table 2).", "weaknesses": "In general, the logic flow of this paper is clear, and the experiment results are convincing. I am considering raise my score accordingly if my concerns are properly addressed.\n\n\n**Concern 1. Comparison with the pretraining method that leverages fully low-rank weights [1].**\n\nWhile CR-Net proposed to replace the full-size weight parameters at all but the first layers with low-rank factors, I aware that a pretraining method that leverages fully low-rank weight parameters has been proposed in [1]. What is the most essential benefit of CR-Net over this method, which also leverages fully low-rank weights? Moreover, including this baseline in Table 3 would be helpful for understanding the state of low-rank pretraining methods. \n\n\n> [1]. Mo, Z., Huang, L.-K., & Pan, S. J. (2025). Parameter and memory efficient pretraining via low-rank Riemannian optimization. In The Thirteenth International Conference on Learning Representations (ICLR 2025). \n\n-----\n\n**Concern 2. How does CR-Net overcomes the difficulty of low-rank pretraining?**\n\nAs discussed in Table 2 of [2], pretraining models with fully low-rank weights from scratch is difficult and usually suffers from severe numerical instability. How does CR-Net overcome this difficulty, while it also trains models with a lot of low-rank weights?\n\n> [2]. Zhao, J., Zhang, Z., Chen, B., Wang, Z., Anandkumar, A., & Tian, Y. (2024). Galore: Memory-efficient LLM training by gradient low-rank projection. arXiv preprint arXiv:2403.03507.\n\n-----\n\n**Concern 3. Run-time efficiency of CR-Net.**\n\nAs modern GPUs are optimized for large-scale parallel computation, some low-rank computations with lower FLOPs does not necessarily lead to better run-time efficiency (e.g., token per second). Therefore, I recommend the authors to provide additional runtime efficiency experiments between CR-Net and the full-rank model in inference and training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TuXSZIsv5b", "reviewer": "Reviewer_BJ5z", "summary": "Unlike prior works, which focus on low-rank approximations of activations or the weights directly, CR-Net proposes to use the previous layers activations to enable a lower reconstruction error. I believe this is a very interesting approach which the authors verify empirically and also theoretically to some extent.\n\n\nThe authors found that naively training with this alone is not sufficient and will lead to poor stability. To address this, they introduce a learnable parameter beta for dynamically adjusting the contribution of the previous activation and the current low-rank output.", "questions": "Does the first layers activations have to be full-rank for good performance? It would be interesting to see an ablation on this. It makes sense for this decision to be important, but I think highlighting and showing why this is needed could be nice to see.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The experimental evaluation is very thorough and of sufficient scale to highlight the practical utility of CR-Net. The idea is simple, makes a lot of sense, and is proven to work very well. If the authors are able to provide reproducible code, then extending this to other architectures is a very natural progression of this work.\n\nThe computational cost in terms of FLOPs is quite a complex expression. This is not a problem in itself and the authors provide a discussion on the various regimes for L355:358, which is good. Furthermore, the fact that CR-Net can enable much smaller values of r to compensate for the first layer cost, while also enabling improved training performance, really highlights the benefit of CR-Net.", "weaknesses": "Although I do not have any major concerns, there are a few points which I believe the authors should address.\n\nMissing important related work [1] that uses very low rank projections of the intermediate activations to save memory. However, in contrast to this proposed work, they use random projection (no SVD init) of sub-tokens and an optimal reconstruction during the backwards pass. It seems CR-Net outperforms both [1] and [3], but including their results in table 3 would be good to see. Finally, there is missing a discussion on QLoRA, and finally how does CR-Net perform when the activations are also quantised? \n\n[1] VeLoRA : Memory Efficient Training using Rank-1 Sub-Token Projections. NeurIPS 2024\n\n[2] Flora: Low-rank adapters are secretly gradient compressors. ICML 2024\n\n[3] Qlora: Efficient finetuning of quantized llms. NeurIPS 2023\n\nSec 2.2 Related works would be much better before the preliminary section. This RW section does not use the preliminary notation anyway and it seems to just break up the flow of reading from preliminary notation -> method (section 3).\n\nMinor comments\n\n\"inputs and outputs of linear layers at the position P, where P stands for the detailed position in the transformer layer.\" the second part (after the comma) describing what P is again seems a bit redundant.\n\neqn (2) should the activation not be a swiglu? for consistency with the start of this paragraph, which is presenting CR-Net with the LLaMA architecture.\n\n\nenlarging the text in figure 1. At the moment it is very hard to read without having to zoom in a lot.\n\n\n\"making CR-Net NOT a simple extension of existing low-rank frameworks for pre-training.\" - I understand the intent, but I would encourage the authors to remove the bolding and capitalising of \"NOT\".\n\n\n\"Whether does the learnable scaling factor βPl benefit the model convergence?\" grammar mistake -> \"Does the learnable scaling factor βPl benefit model convergence?\"\n\nL464: \"ovservation\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LGpaPJUCA5", "reviewer": "Reviewer_WfvT", "summary": "Paper presents a method to change the linear layers in LLMs into a low-rank (AB) decomposition plus residual activations from the corresponding matrix in the previous (l-1) layer in the transformer, together with a learnable interpolation factor per each layer. It reduces the training time, the training memory footprint, and sometimes reduces the perplexity.", "questions": "* Figure 1, ROPE -> RoPE\n* Table 4 description mentions that the “10K, …” are the number of training steps.\n* Table 3, the reader is also interested in 7B. I assume in order to fit in memory you can only do it with “re-computation”, so I suggest mentioning that, and to refer the reader to table 4.\n* Theorem, (44), I assume a quantifier “for each i” should be added.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* This is an interesting method which reduces pre-training time significantly. \n* Reduces memory significantly", "weaknesses": "* Table 4 shows improvement over baseline (Adam) for 7B model, but only in table 10 (in the appendices) we discover that the method is less good in the case of 13B. For fairness, I think the two tables should be adjacent so the reader can see them both and get the whole picture easily. So we understand the benefits in reducing training time and memory, but it should be clear that as the model size gets bigger, the quality (perplexity) of CR-Net gets worse than the baseline.\nIn line 1225 it says “The evaluation perplexity shown as Table 10 illustrates the performance of CR-Net in 13B model, which demonstrates that CR-Net exhibits a comparable performance than full-rank training with 8-bit Adam. Thus can be scaled to models with 13B parameters.”\nI think that “slightly worse” is more appropriate than “comparable”, and the assertion “Thus can be scaled” is wrong. There is no evidence for it. When we analyze the trend of what happens in larger models, we see that the CR-Net edge decreases (ppl-wise), so it feels like in larger models it would become worse. Anyhow, without evidence, giving such statement is probably a bit too far-reaching \n\n* Line 196, “Different from…” should be checked for grammar errors.\n* Line 204, “approached”. Do you mean “approaches”?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)