Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks
Authors: Jialin Zhao
Yingtao Zhang
Xinghang Li
Huaping Liu
Carlo Vittorio Cannistraci
Venue: ICML 2025 poster (2025)
Keywords: Parameter-efficient training; Pre-training; Hyperbolic Network
Primary area: deep_learning->algorithms

Abstract: The growing demands on GPU memory posed by the increasing number of neural network parameters call for training approaches that are more memory-efficient. Previous memory reduction training techniques, such as Low-Rank Adaptation (LoRA) and ReLoRA, face challenges, with LoRA being constrained by its low-rank structure, particularly during intensive tasks like pre-training, and ReLoRA suffering from saddle point issues. In this paper, we propose **S**parse **S**pectral **T**raining **(SST)** to optimize memory usage for **pre-training**. SST **updates all singular values** and **selectively updates singular vectors** through a multinomial sampling method weighted by the magnitude of the singular values. Furthermore, SST employs **singular value decomposition to initialize and periodically reinitialize** low-rank parameters, reducing distortion relative to full-rank training compared to other low-rank methods. Through comprehensive testing on both Euclidean and hyperbolic neural networks across various tasks, SST demonstrates its ability to outperform existing memory reduction training methods and is comparable to full-rank training in various cases. On LLaMA-1.3B, with only 18.7\% of the parameters trainable compared to full-rank training (using a rank equivalent to 6\% of the embedding dimension), SST reduces the perplexity gap between other low-rank methods and full-rank training by **97.4\%**. This result highlights SST as an effective parameter-efficient technique for model pre-training.

Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks
Jialin Zhao 12 Yingtao Zhang!?
Xinghang Li² Huaping Liu
Carlo Vittorio Cannistraci 123
Abstract
1. Introduction
The growing demands GPU memory posed by the increasing number of neural network param- eters call for training approaches that are more memory-efficient. Previous memory reduction training techniques, such as Low-Rank Adapta- tion (LoRA) and ReLoRA, face challenges, with LoRA being constrained by its low-rank struc- ture, particularly during intensive tasks like pre- training, and ReLoRA suffering from saddle point issues. In this paper, we propose Sparse Spectral Training (SST) to optimize memory usage for pre-training. SST updates all singular values and selectively updates singular vectors through multinomial sampling method weighted by the magnitude of the singular values. Furthermore, SST employs singular value decomposition initialize and periodically reinitialize low-rank parameters, reducing distortion relative to full- rank training compared to other low-rank meth- ods. Through comprehensive testing on both Fu- clidean and hyperbolic neural networks across various tasks, SST demonstrates its ability to out- perform existing memory reduction training meth- ods and is comparable to full-rank training in var- ious cases. On LLaMA-1.3B, with only 18.7% of the parameters trainable compared to full-rank training (using a rank equivalent to 6% of the em- bedding dimension), SST reduces the perplexity gap between other low-rank methods and full- rank training by 97.4%. This result highlights SST as an effective parameter-efficient technique for model pre-training
The development and scaling up of large language mod- els (Kaplan al., 2020; Brown et al., 2020: Touvron et al., 2023b) pose great challenges the feasibility of training large language models from scratch. Normal training meth- ods that update all parameters of models become extremely expensive due to their extensive GPU memory requirements.
Recent developments in parameter-efficient fine-tuning (PEFT) methods. such as Low-Rank Adaptation (LoRA) (Hu et al., 2022), have sought to mitigate the challenge of fine-tuning memory requirements by introducing trainable low-rank matrices that efficiently reduced the memory footprint. However, limiting the model's parameter updates to a low-rank subspace can severely restrict the ability of a model to capture and represent complex data patterns, leading to suboptimal performance, especially in the pre-training stages. Recent advancements such ReLoRA (Lialin et al., 2024), COLA (Xia et al., 2024), and PLoRA (Meng et al., 2024b) have addressed the limitation of low-rank constraint, by iteratively merging low-rank parameters with frozen parameters. However, they still encounter saddle point issues due to zero gradient of low-rank parameters that occurs after each merging step. This challenge results in slower and less effective convergence compared full-rank models during pre-training
In response to these challenges, we introduce Sparse Spec- tral Training (SST), a new training framework designed to optimize memory consumption while closely approxi- mating the overall learning dynamics and performance of full-rank training. Unlike previous methods (Hu et al., 2022: Lialin et al., 2024; Zhang al., 2023; Ding et al., 2023) that primarily focus updating within low-rank sub- space at each step, SST adopts a more effective approach by updating all singular values at each step. SST also lever- ages the intrinsic spectral properties of the weight matrices, focusing selective updates of singular vectors sampled from a multinomial distribution weighted by the magnitude of the singular values. Additionally, SST uses singular value decomposition to initialize and reinitialize low-rank parameters during training, reducing distortion relative to full-rank training compared to other low-rank methods.
'Center for Complex Network Intelligence (CCNT), Ts- inghua Laboratory Brain and Intelligence (THBI), Depart- ment of Psychological and Cognitive Sciences Depaarmment of Computer Science 'Department of Biomedical Engineering, Ts- inghua University, China. Correspondence to: Jialin Zhao <jialin.zhaad zhao97 @ comaiil.omm.. Carlo Vittorio Cannistraci <kalok- agathos.agen agon @ gwa.l..co.......... com>>
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks
Our comprehensive evaluations cover different tasks, including pre-training large language models OPT model family, ranging from 125m to 1.3b (Zhang et al., 2022), using Transformer (Vaswani et al., 2017) for machine translation tasks and hyperbolic graph neural networks (Chen et al., 2022) on node classification and link prediction tasks. For the OPT and LLaMA model family, SST reduces the perplexity gap between other low-rank methods and full-rank training by 50%-97.4%. In machine translation with Transformers, SST reduces the BLEU gap by an average of 66.7%. Furthermore, we are the first to incorporate parameter-efficient pre-training process in hyperbolic space, demonstrating that SST is a general technique applicable across various data structures and models. On the hyperbolic Transformer, SST even outper- forms full-rank training in most scenarios. For hyperbolic graph neural networks, SST reduces the performance gap by an average of 73.7% in node classification and 82.5% in link prediction. Our code is available at ht tps: //github. www.://iooeeiialllyyeeeeeeeeeeee sparrreeppecrraltttaee
demonstrated by Muscoloni et al. (2017) and Ganea et al. (2018). Further developments by Chen et al. (2022) explore manifold-specific properties to enrich both theoretical under- standing and practical deployment. The use of hyperbolic spaces has been shown to significantly improve data repre- sentation and generalization across various tasks, marking a notable advancement in managing complex, non- Non-Euclidean data structures (Gulcehre et al., 2019; Liu et al., 2019; Tifrra et al., 2019; Yang al., 2024).
3. Low Rank Adaptation
This section introduces the fundamentals and limita- tions of Low-Rank Adaptation (LoRA) (Hu et al., 2022), ReLoRA (Lialin et al., 2024), and GaLore (Zhao et al., 2024). These limitations are addressed by Sparse Spectral
Training (SST) in Section 4.
3.1. LoRA
LoRA (Hu et al., 2022) fine-tunes a pre-trained model by learning an incremental update AW to the pre-trained and frozen pre-trained weight matrix Wo. Here, Wo. AW e R=xx with m S n.. LoRA decomposes AW into the prod- uct of two low-rank matrices, B E mxr and A E Rr**, such that A WW = BA. This decomposition is applied to a linear layer with input x and output as follows:
2. Related Work
Low-Rank Adaptation. Low-rank adaptation has become a key strategy for reducing the computational and memory requirements of training large-scale neural networks. Hu et al. (2022) introduced Low-Rank Adaptation (LoRA), a technique that fine-tunes pre-trained models by integrat- ing low-rank matrices to significantly reduce the number of parameters updated during training. Various enhance- ments to LoRA have since been developed to improve its efficiency and broaden its application (Zhang et al., 2023; Dettmers et al., 2024; Zi et al., 2023; Valipour et al., 2023). Lialin et al. (2024) introduced ReLoRA specifically for the pre-training phase, which requires a full-rank warm-up to achieve performance comparable full-rank training. A similar approach is also found in COLA (Xia et al., 2024) and PeriodicLoRA (Meng et al., 2024b). Additionally, Zhao et al. (2024) introduced Gore, which projects gradients into a low-rank subspace. Meng et al. (2024a) introduced PiSSA, which uses dominant singular vectors of pre-trained weight as initialization of low-rank matrices. These ad- vancements highlight the versatility and ongoing evolution of low-rank adaptation techniques in response to the grow- ing complexity of neural network models.
h (Wo + AW)x (Wo + BA)x
(1)
Given < min (m. n2), LoRA significantly reduces GPU memory usage compared to full-rank fine-tuning.
Limitation of LoRA. Consider W' as the optimal weight matrix which minimizes loss. The deviation from the current weights is A W = W WD. Performing a singular value decomposition on AW* yields A AW = UEVT, where U E RMxm Σ E pmxm VT gmxx
U and V orthonormal bases, U [u u...... Um. V V= [v1, V2,..., Vnn. Σ is diagonal matrix with entries {01, Jm). Then the Eckart- Young Mirugyy theorem (Eckart& & Young, 1936) states:
AAW - AW 2 V o7t1 + o
(2)
where || ||
2 is the Frobenius norm, 2i=1 Σド=1 w
122
with wij being the element at row i and column j of W. Equality holds when B = [Voiu, / Vozu2..... Vo,u, and A [Voivi, vo2vv 20222, ...: v.l This suggests that LoRA can only closely approximate the performance of full-rank training in simple tasks like fine-tuning, where a 2 0, E {r + l,..,, m.. However, in more complex scenarios like pre-training, where o, i E {r + 1,..., m] are non noglleglig,,,e, LoRA may struggle to achieve the same level of performance as full-rank training.
Hyperbolic Neural Networks. Hyperbolic neural net- works an emerging area in deep learning, exploiting the unique properties of hyperbolic space that make it ideal for processing hicrarchical and graph-structured data (Mus- coloni et al., 2017; Cannistraci & Muscoloni, 2022). Innova- tions in this area have adapted fundamental neural network mechanisms to function within hyperbolic geometries, as
2

---

Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks
3.2. ReLoRA*
low-rank updates calculated by Adam are re-projected back to the original weight shape and used update the weights.
ReLoRA (Lialin et al., 2024), COLA (Xia et al., 2024), and PLoRA (Meng al., 2024b) address the limitation of fixed low ranks by iteratively merging the low-rank matrices B and A back into the base weight matrix Wo- Although ReLoRA is designed for pre-training, it includes an initial period of full-rank training (referred to as a "warm start"), which prevents it from being fully end-to-end parameter- efficient. Meanwhile, COLA and PLoRA are primarily intended for fine-tuning. In this paper, we unify these meth- ods into generalized, end-to-end parameter-efficient pre- training paradigm, which we refer to as ReLoRA* and for- malize in Algorithm 1.
Limitations of GaLore. Although GaLore presents valu- able contribution by exploring low-rank gradient projection, it has some limitations. Firstly, Pr is calculated based solely on the SVD of the gradient from a single batch, which can be affected by data sampling noise. Secondly, Gal-or al- ways selects the top-r singular vectors, which, combined with the previous limitation, restricts its effectiveness during pre- training with small In our experiments, we observed that with a small r (less than 1/12 of the dimension, differ- ent from the 1/2 to 1/4 used in the GaLore article), GaLore showed instability, leading to a sudden increase in loss on OPT-350M. Consequently, we chose to include the detailed explanation and comparison with GaLore in Appendix G rather than in the main text.
Algorithm 1 R
input Initial weight W of each layer; total iteration Ti;
iteration interval T2
for 4 0 - do
4. Sparse Spectral Training
Initializing: Initialize and for each layer.
Subtracting: Subtract B and A from W maintain the original model output, W W BA
To address the limitations discussed previously, this section introduces Sparse Spectral Training (SST) and detailed its implementation.
Updating: Update B and A for T2 steps while keeping
W frozen.
Merging: Merge B and A back to W, updating W W + BA.
4.1. Sparse Spectral Layer
end for
Sparse Spectral Training (SST) leverages sparse updates within the spectral domain of neural network weights. SST transforms each linear layer as follows:
For our experimental setup, ReLoRA* follows ReLoRA's initialization- B initialized to zero and A with Kaiming initialization (He et al., 2015). The initial zero setting for B allows the subtraction step to be skipped. Notably, the optimizer states for B and A are reset after each merging step (99% optimizer state is pruned in ReLoRA).
h = Wx UEV [. SVD(W) (3)
where U E R Σ E Rmxm, and V E Rm*X repre- sent the full-rank matrices derived from the singular value decomposition (SVD) of W E Rmxx, assuming m S n. is important to note that unlike other LoRA-based methods, U, Σ, VT in this context are utilized full rank, and the original weight matrix W is removed from networks. For simplicity, in the following discussion, we continue to use W to represent UΣVT.
Limitation of ReLoRA*. * Each iteration of ReLoRA* * learns only small subset of singular values. Additionally, its reliance zero initialization can result in zero gradients of low-rank matrices at each reinitialization, as discussed in Section 4.3. These issues hinder ReLoRA* from achiev- ing the convergence speed and training quality of full-rank training.
The singular value decomposition is performed only during initialization and periodically reinitialized at each round (see Eq. 10), ensuring that the training process remains efficient (see Table 22 for the actual proportion of training time). However, training progresses, U, Σ. and V may gradually deviate from the true singular vectors and singu- lar values of W. In the subsequent section, we introduce improvements designed to mitigate this deviation.
3.3. GaLore
Gradient Low-rank Projection (GaLore) (Zhao et al., 2024) introduces a different approach by projecting the gradient using a low-rank projection matrix, rather than the weight matrix, as done by LoRA and ReLoRA*. The projection matrix P, is obtained by computing the top-r singular vec- tors of the gradient of the weight matrix W, and it is re- calculated every T' steps. This matrix P, is then used to project the gradient of the weight matrix into the low-rank space, allowing the low-rank gradient to update the first and second-order low-rank momentum Adam. Finally, the
4.2. Gradient Update of U, VT with Σ
Update all Σ. The diagonal matrix E, simplified as a vector of dimension m, is updated at every step due to its low memory overhead. This ensures that all singular values are consistently adjusted to refine the model's performance.
3

---

Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks
Sample
M Mawl
Index
W Jott
W x
Periodic merge and reinitialize
Periodic re-SVD
(a) LoRA
(b) PiSSA
(c) ReLoRA
(d) SST
Figure I: Illustration of Sparse Spectral Training (SST) and comparison with LoRA, PiSSA, and ReLoRA*, LoRA learns an additive low-rank update to a pre-trained and frozen weight matrix. ReLoRA* periodically initializes and merges low-rank matrices into the full-rank weight matrix, limiting updates to a low-rank subspace in each iteration. PiSSA initializes low-rank parameters using SVD but always updates the same set of singular vectors. In contrast, SST follows sampling- sarttngg-paat----e swappinng paradigm, where singular vectors are dynamically selected via multinomial sampling, all singular values updated each step, and periodic re-SVD maintains orthogonality. This ensures better exploration and stability during pre-training.
The update rule is as follows:
Enhance gradient of U and VT. Within sparse spectral layer where = UEVTx x (using W to denote UEVT), the gradient for U is detailed below (derivation included in Appendix D):
E+ max - max(E''n7Vis. 0)
(4)
where 1) represents the learning rate, and VL is the gradi- ent backpropagated to E. The max function ensures that Σ values remain non-negative.
OL OL
V.E; VLU, JU,
OW
(8)
OL Σ, OL
VLv. OV. OWT
U.
Selectively update U and VT. To update U and VT, selective updating strategy is employed, where specific parameters are chosen for each iteration based on a multi- nomial sampling method, as depicted in Figure Consider == {1, 2, ..., m) as the set of all indices of singular vectors in U and VT, with the sampling process defined by:
where U., and V. column vectors of U and VT spectively, and E, represents the diagonal elements of Σ. This represents the default gradient calculation for these matrices. We propose an enhanced gradient calculation for U.. and V., as follows:
S SCI, S Multinomial(r, Σ)
(5)
OC VLu, aw V VLV. = OWT U.
Oc
Here, S represents the selected indices for update, with |S] = r, where r is the predetermined number of vectors to be updated each iteration. The update formula for U is:
In the enhanced gradient, the learning of direction (U.; and V.;) is decoupled from the magnitude (E;), allowing singu- lar vectors with lower singular values to retain substantial gradients.
U U = - - CV.
i
(6)
++ -VVvv
Periodic re-SVD. During training, the orthogonality among the vectors of U and VT tends to diminish. Pre- serving the orthogonality of these singular vectors is crucial, as it prevents the learning process from degenerating into low-rank subspace, thus preserving the model's full ex- pressive capabilities. To maintain this orthogonality, is essential to periodically perform singular value decomposi- tion:
where U., means the i-th column vector of U. To maintain the unit norm of each vector during training, and to ensure that magnitude information is encapsulated solely by E, the vectors are normalized post-update as follows:
U U LU.
U V.ul
if S
V VVVL.
v
U E1+1 v(+T] SVD(U''''V (10)
V. -vv..

---

Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks
Fut wiightt grdd
52T activ U grad
miors, A grad
0.20
9.925
0.001
0.20
0.15
0.10
0002
0.000
0.20
0.000
0.003
250
736
1030
1250
1500
1750
2030
253
502
750
1835
3250
1500
1732
2030
500
750
1000
1250
1500
1752
2006
2tpp
Full-rank
(b) SST
ReLoRA*
Figure 2: ReLoRA* * suffers saddle point issue at each restart. This plot depicts the average Frobenius Norm of gradients of: (a) all weight matrices in full-rank training; (b) all sampled U in SST: (c) all A in ReLoRA*, in first 2000 steps. All methods are trained on Transformer with dimension = 64, 8 on IWSLT" 14. Both SST and ReLoRA* set iteration interval to 200. When the average Frobenius Norm of gradients approaches zero, it indicates that a saddle point issue happens. Figure (c) shows that ReLoRA* suffers saddle point issue periodically at the beginning of each iteration. The correlation between SST and full-rank gradient norm along the steps is 0.85, whereas the correlation between ReLoRA * and full-rank is 0.58. This demonstrates that the gradient of SST more closely approximate the gradient curve of full-rank training. compared with ReLoRA*.
Each time we perform this re-SVD, we consider it a new round. Each time we select vectors for updating, as de- scribed Eq. 5, we call it a new iteration. The full method detailed Algorithm 2.
SST initializes and reinitializes its low-rank matrices U and V using the singular vectors of W. In contrast to ReLoRA* which relies on random or zero initialization for its low-rank matrices, SST better captures the direction W's updates, allowing it to more closely approximate full-rank training. As demonstrated in the ablation study (Appendix H), re- placing SVD-based initialization with random initialization leads significant drop in performance, highlighting the critical role of SVD in SST's effectiveness.
4.3. Why SVD Decomposition is Important
This section discusses the advantages of using SVD ini- tialization and periodic re-SVD over zero initialization as employed in ReLoRA* methods.
4.4. SST Balances Exploitation and Exploration
Saddle point issues after each merging in ReLoRA*. The gradient of A and B in ReLoRA* is:
From another perspective, SST combines the strategies of exploitation and exploration in spectral domain. LoRA pri- marily focuses on exploitation by repeatedly adjusting the top-r singular values, as detailed in Section 3. while ne- glecting the remaining spectral vectors. ReLoRA* on the other hand, emphasizes exploration by periodically reini- tializing the matrices B and A after each merging, thereby constantly seeking new directions for learning but ignoring previously established dominant directions.
dc Oc
0A = B OC
OC
AT and
(11)
JB ow
OW
After each merging, B is reinitialized to zero, and the gradi- ent of A is calculated as BC JA = 0 BC PW = O, which causes a slow learning progress at the beginning of each iteration. Additionally, ReLoRA* resetting the momentum of B and A after each merging aggravates this issue, particularly when the merging interval T2 is short.
SST boosts learning efficiency by updating all magnitudes (E) each step and cyclically revisiting previously estab- lished dominant directions. By continuously updating all singular values, SST ensures unbiased sampling of U and VT, enabling thorough exploration of the parameter space. As result, SST balances the exploitation of known critical directions with the exploration of emerging opportunities within the spectrum matrix decomposition.
Compared with ReLoRA*, SST more closely approxi- mates full-rank training. In Figure 2, we compare the average Frobenius Norm of gradients of weight matrices in full-rank training, low rank matrices in SST and ReLoRA*. This plot shows that ReLoRA* suffers saddle point issue periodically the beginning of each iteration. We also calculate the correlation between SST and full-rank gradi- ent norm along the steps is 0.85, whereas the correlation between ReLoRA* * and full-rank is 0.58. The fact that SST's gradient norm is more closely correlated with the full-rank gradient norm than ReLoRA* suggests that SST more closely approximates the gradient of full-rank training.
4.5. Sparsity of SST
We analyze the efficiency of parameter usage.. Specifically, the ratio of trainable parameters in SST at a given rank r,
5

---

Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks
This param- eter ratio is slightly higher than that of LoRA the same rank, TORRA, = +(m++)) ma yet remains lower than LoRA at rank 1, LORA,,+11 (++1((m-)) indicating slightly mn
denoted as ISST. r. is calculated as r(m++))++ yan
For all low-rank methods, replace all linear layers in the baseline models (including query, key, value, output pro- jections and feedforward layers) with their corresponding low-rank implementations (e.g., LoRA, ReLoRA*, or SST), This ensures that all methods operate under comparable parameter efficiency constraints during pre-training, as op- posed to fine-tuning scenarios where only a subset of layers is typically modified. Hyperparameters and implementation details are provided in Appendix E.
increase in trainable parameters.
Vactive
As discussed in Section 3.3, the comparison between SST and GaLore (Zhao et al., 2024) is provided in Appendix G, as Gal.ore is unstable during OPT pre-training with r = 64. We highlight SST's superior performance across all of our experiment settings. Ablation studies are documented in Ap- pendix H, and a detailed analysis of memory consumption and training time can be found in Appendix I. Additionally, an experiment on image classification tasks is included in Appendix J.
Swap to freeze
V
reexe
Swap active
Newly Sampled
Figure 3: Illustration of the memory-efficient implemen- tation for SST. After each sampling step, the sampled vec- tors are swapped with the active vectors from the previous iteration.
5.1. Machine Translation
We employ the vanilla transformer (Vaswani et al., 2017) as the Euclidean transformer and HyboNet (Chen et al., 2022) as the hyperbolic transformer. Our experiments include three widely-used machine translation datasets: IWSLT 14 Engrish-to-German (Cettolo et al., 2014), IWSLT' German-to-Engee English (Cettolo al.. 2017), and Multi30K tnngllishh (Elliott et al., 2016). For IWSLT' 14, the hyperparameters are aligned with those from HyboNet.
4.6. Memory-Efficient Implementation for SST
To achieve similar memory reduction LoRA, SST stores optimizer states for all Σ and only for the vectors sampled in each iteration from U and VT. However, standard im- plementations of Adam optimizer (Kingma & Ba, 2014) in PyTorch (Paszke et al., 2019) do not support sparse opti- mizer states. To address this, we partition and VT into active and frozen segments. Only active segments store the optimizer states, where Unctive Rmxr and VT active R The frozen segments, Ufrezze and V do not store opti- mizer states. Vectors newly sampled from the frozen seg- ments are swapped with unsampled vectors in the active segments (illustrated in Figure 3). This approach enables SST to function as a time-sharing operating system, effec- tively balancing resouree allocation among the vectors in U and VT.
Euclidean Transformer Table presents BLEU scores for IWSLT' 14 across various dimensions and ranks (r). The results confirm that SST consistently outperforms other low- rank methods. On average, SST reduces the BLEU gap (defined as the BLEU score difference from full full-rank train- ing) by 66.7 % for Euclidean Transformers on IWSLT' 14.
Further comparative results on the Multi30K and IWSLT' datasets using the standard dimensions for vanilla Fuclidean transformers are documented in Table 3. Here, SST not only surpasses other low-rank methods but also demonstrates superior performance compared to full-rank training.
5. Experiments
Hyperbolic Transformer In Table 1, some BLEU scores for the hyperbolic transformer are zero, due to the training process encountering NaN losses, whereas SST maintains stability throughout. SST consistently outperforms other low-rank methods across all settings and even exceeds the performance of full-rank training in various configurations.
To validate our Sparse Spectral Training (SST) approach, we conducted experiments on both Euclidean and hyperbolic neural networks, demonstrating the generalization of SST across various neural network architectures and embedding geometries.
We compared SST with full-rank training, LoRA, and ReLoRA# The key distinctions between ReLoRA* * and ReLoRA (Lialin et al., 2024) is that ReLoRA includes a full-rank training as "warm start", which prevents it from being an end-to-end memory-efficient pre-training method.
Previous hyperbolic neural network articles have predomi- nantly focused on low-dimensional configurations (Ganca et al., 2018; Shimizu et al., 2021; Nickel & Kiela, 2017). A key characteristic of hyperbolic space is its exponential growth in volume with distance from a reference point,
6

---

Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks
Table 1: BLEU scores on IWSLT'14 for Euclidean and hyperbolic Transformers. Values in bold indicate the highest performance among low-rank methods. Values marked with an "*" exceed the performance their full-rank counterparts. Some BLEU scores are 7ero because that training resulted NaN losses. Notably, SST consistently outperforms other low-rank methods. Furthermore, the hyperbolic Transformer trained by SST shows improved performance over the full-rank hyperbolic Transformer, particularly the dimension size increases.
Euclidean
Hyperbolic
Dimension
Full
LoRA
ReLoRA*
SST
Full
LoRA
ReLoRA*
SST
18.08
18.12
22.28
17.50
0.0
23.40
64
24.27
25.69
14.05
15.49
20.27
0.0
0.0
23.03
16
23.30
22.92
25.12
23.70
0.0
25,22
128
25.79
20.56
20.61
24.19
24.70
20.81
0.0
25.12
16.37
18.00
22.80
17.58
24.42
24.60
32
23.76
23.02
23.97
24.16*
0.0
25.04
16
22.88
22.01
23.42
23.93
0.0
25.52
256
23.92
19.94
20.32
20.36
22.65
21.58*
24.02*
24.67
16.72
17.85
21.39
18.72
24.08*
24.51
Table 2: Validation perplexity Open Text across various model sizes of OPT and LLaMA along with the number of trainable parameters of each method. Values in bold highlight the highest performance among the low-rank methods.
Model
r/dandel
Training Tokens
Full
LoRA
ReLoRA*
SST
OPT-125M
64/768
19.7B
23.50 (125.2M)
34.23 (50.9M)
35.80 (50.9M)
26.98 (51.0M)
OPT-350M
64/1024
19.7B
21.78 (331.2M)
34,26 (57.5M)
39.21 (57.5M)
27.72 (57.7M)
OPT-1.3B
64/2048
19.7B
15.10 (1.316))
1716 (164.4M)
29.52 (164.4M)
22.31 (164.7M)
LLaMA-130M
64/768
2.6B
20.04 (134.11M)
29.71 (60.38M)
31.33 (60.38M)
23.35 (60.44M)
LLaMA-1.3B
128/2048
13.1B
14.54 (1.339B)
16.50 (250.71M)
17.32 (250.71M)
14.59 (251005M)
Table 3: Comparison of BLEU scores on Multi30k and IWSLT'17 datasets using Euclidean Transformer (dimension = 512), r = 32. Scores highlighted in bold represent the highest performance achieved by low-rank methods.
as the baseline for our language generation experiments. For LLaMA, we follow the experiment setup from (Zhao et al., 2024). All models are pre-trained Open WebText (Gokaslan & Cohen, 2019), open-source reproduction of OpenAl's WebText. We applied rank of 64 for all OPT models and LLaMA-130M, and r 128 for LLaMA- 1.3B.
Full
LoRA
ReL.RAA*
SST
Multi30K
40.7
40.1
41.6
43.4
Table 2 displays the validation perplexity results on the OpenWebTex. dataset across different sizes of all LLMs. The results indicate that SST achieves lower perplexity scores compared to LoRA and ReLoRA significantly re- ducing the perplexity gap defined as the difference be- tween the perplexity of the low-rank method and the full- rank training. Specifically, SST reduces this gap by 67.6% % (OPT-125M), 52.4% 4 (OPT-350M), 50.0% (OPT-1.3B), 65.8% % 1LLMA-130MM and 97.4% (LLaMA-1.3B),
IWSLT'17
31.7
31.9
32.0
32.3
which is significantly more rapid than the polynomial growth seen in Euclidean space (Cho et al., 2019). This pansive nature makes hyperbolic spaces particularly prone to overfitting as dimensionality increases. By imposing con- straints on the parameter search space of hyperbolic neural networks, SST prevents the overfitting typically associated with such high- digesiimensional settings.
Figure 4 presents plot of validation loss against effective steps for various training methods. The effective step metric, defined as the product of the number of training steps and the number of trainable parameters, provides insight into the ef- ficiency of parameter updates. Although parameter-effice efficiet training methods typically exhibit slower convergence com-
5.2. Natural Language Generation
Language modeling. We utilize the OPT (Zhang et al., 2022) and LLaMA (Touvron et al., 2023a) architecture

---

Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks
Table 4: Zero-shot evaluations on the same 16 NLP tasks featured in the OPT article (Zhang et al., 2022). Values in bold indicate the highest performance among low-rank methods. Except for the ReCoRD task, which uses F1 score, all other tasks are evaluated using accuracy, with values presented as percentages. Mean scores in bold represent superior performance among the low-rank methods. Additionally, we include the win percentage (ineluding ties) for each low-rank method compared the full-rank training.
OPT-125M
OPT-350M
OPT-1.3B
Full
LoRA
ReLoRA
SST
Full
LoRA
ReLoRA*
SST
Full
LoRA
ReLoRA
SST
ARC (Challenge)
21.2
22.9
21.1
21.3
22.0
22.3
21.3
21.1
24.6
24.2
22.9
21.5
ARC (Easy)
35.8
34.2
33.9
34.3
35,9
32.3
33.0
35.7
43.2
26.1
35.9
37.8
BoolQ
59.5
54.2
60.8
62.0
53.6
56.2
62.2
57.7
57.7
37.8
61.4
59.5
CB
51.8
48.2
28.6
48.2
44.6
44.6
33.9
41.1
59.0
41.1
37.5
42.9
COPA
67.0
61.0
57.0
66.0
69.0
61.0
59.0
60.0
70.0
51.0
HcllaSwag
27.7
26.5
27.1
26.9
28.4
26.6
26.9
27.5
35.0
26.1
27.2
28.1
MuluRC
55.4
57.2
55.9
57.2
52.0
52.6
56.4
57.0
56.8
42.8
57.7
56.9
Open
24.6
24.6
23.6
26.2
26.4
24.2
23.0
25.2
29.0
27.0
24.8
25.0
PIQA
58.7
57.2
56.3
58.3
56.9
56.9
59.0
64.0
50.3
57.1
59.1
ReCoRD
16.7
17.5
22.6
18.5
19.4
17.6
19.0
23.2
13.7
17.6
23.0
18.1
RTE
50.5
56.7
53.1
53.4
52.0
49.1
54.9
50.2
51.6
52.7
52.0
53.8
StoryCloze
55.8
53.8
53.6
54.5
57.2
53.7
53.0
54.6
61.1
49.7
54.0
56.1
WIC
49.8
51.4
50.0
50.0
50.5
50.0
50.0
50.2
50.3
50.0
50.0
50.0
Winograd
52.0
48.7
50.6
50.6
55.0
51.7
50.2
51.3
55.7
50.9
52.4
55.3
Winogrande
49.1
49.2
50.7
50.1
50.7
50.3
50.8
52.0
51.1
47.9
50.0
49.1
WSC
36.5
38.5
36.5
36.5
36.5
37.5
36.5
36.5
39.4
63.5
36.5
36.5
Mcan
44.5
43.8
42.6
44.6
44.5
42.9
42.9
43.9
47.6
41.2
44.4
44.7
Win Percentage
50.0
43.8
56.3
31.3
31.3
31.3
18.8
25.0
25.0
Zero-shot evaluations. Each pretrained model performs zero-shot evaluations all 16 NLP tasks used in the OPT article (Zhang et al., 2022), including ARC Easy and Chal- lenge (Clark et al., 2018), HellaSwag (Zellers et al., 2019), OpenBookQA (Mihaylov et al.. 2018). PIQA (Bisk et al., 2020), StoryCloze (Mostafazadeh et al., 2016), SuperGLUE (Wang et al., 2019), WinoGrad (Levesque et al., 2012), and WinoGrande (Sakaguchi al., 2019). Evaluations are con- ducted using the LM Evaluation Harness framework (Gao et al., 2023). Except for the ReCoRD task, which uses F1 score, all other tasks are evaluated using accuracy.
JOPT:3 25M0
FullIOFT-1.30) 1OPT-350M! SST (T-12500
COPT(OTT-ST (OPT-150M)
(OPT-125M)
(OPE-33)
SST [OPT-125M1
SST (OPT)1.8))
Validation
0.2 3 1 0.6 066100' 10
0.8 1
100110t
Effective Step (Step Trainable Parameters)
Table presents the zero-shot evaluation results across the 16 NLP tasks. SST achieves a higher average score than other low-rank methods across sizes of the OPT models. On the OPT-125M, the average score for zero-shot evalua- tions of SST 44.6, slightly exceeding the average score of full-rank training, which is 44.5. Additionally, lated the win percentage (including ties) for each low-rank method compared to full-rank training. On the OPT-125M, the win percentage of SST is 56.3%, indicating that SST performed as well as or better than full-rank training on more than half of the zero-shot evaluation tasks.
Figure 4: Comparison performance on effective steps between SST and full-Rank training. Effective steps are calculated the product of the number of trainable parame- ters and the number of steps taken. All methods and model sizes utilize the same number of tokens in each step.
calcu-
pared to full-rank training, the effective step metric illus- trates that SST updates parameters more effectivelyy. At the final effective step for SST on O.T-1.3B, SST achieves validation perplexity of 22.31, whereas full-rank training at the same effective step only reaches a validation perplex- ity of 34.05, demonstrating that SST is more efficient in updating parameters compared to full-rank training.
5.3. Hyperbolic Graph Neural Networks
Hyperbolic Graph Neural Networks (HGNNs) (Chami et al., 2019; Chen al., 2022) capitalize on the expansive and hi- erarchical nature of hyperbolic space efficiently manage
8

---

Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks
Table 5: Node Classification and Link Prediction Results. Model's dimension d = 16. Results are reported as test FI scores for node classification and test precision for link prediction, expressed percentages. Values highlighted bold represent the highest performance among the low-rank methods, while those marked with an "** denote performance that exceeds that of the full-rank variants.
Node Classification
Link Prediction
Method
Airport
Cora
Disease
PubMed
Airport
Cora
Disease
PubMed
Full d = 16
92.88 + 0.5
81.13 = 0.2
91.83 0.4
78.1 = 0.4
95.77 0.08
94,62 +.2
91.49 + 1.5
96.55 # 0.03
LoRA = 1
85.75 = 1.0
45.5 # 0.3
79,66 1.9
69.17 = 2.1
94.01 = 0.2
84.22 0.1
84.29 + 1.5
89.34 = 0.4
SST r
88.61 = 0.5
75.07 = 0.5
89.22 = 1.7
77.47 = 0.3
95.37 = 0.4
91.11 = 0.6
93.63 = 0.7
95.57 0.1
LoRA 2
89.06 + 1.0
64.73 = 0.8
83.84 3 4.3
76.27 = 0.8
94.75 + 0.15
88.8 + 0.5
91.38 + 0.7 94.83 + 0.6*
92.14 0.3
SST r
87.92 + 0.09
77.5 + 0.7
90.64 # 1.7
77.93 + 0.1
95.59 0.2
91.89 # 0.3
95.71 0.1
Table 6: BLEU scores for different sampling mechanisms on IWSLT' 14. Bold indicates the highest performance.
Sampling Mechanism
MULTINOMIAL
UNIFORM
SEQUENTIAL
TOP.R
BLEU
22.28
22.01
22.13
18.28
and analyze graph-structured data. This geometric space is particularly suitable for graphs due to its ability to closely mimic the underlying data structures with minimal distor- tion, offering a substantial improvement over traditional Euclidean methods.
UNIFORM: Uniform random sampling.
SEQUENTIAL: Iterating through all singular vectors without repetition.
TOP R: Selecting the top-r singular vectors with the largest singular values.
We evaluated the effectiveness of SST on HyboNet (Chen et al., 2022) version of HGNN in node classification and link prediction across four distinct datasets: Airport (Chami et al., 2019), Cora (Sen et al., 2008), Disease (Anderson & May, 1991), and PubMed (Namata et al., 2012). Each experiment was conducted with three random seeds.
We also considered a Binomial sampling mechanism; how- ever, it could not guarantee that the number of selected singular vectors would remain consistent with the specified rank, making it unsuitable for direct comparison.
The results, detailed in Table 5, demonstrate SST has strong performance in both node classification and link prediction tasks. With 1, SST reduces the performance gap, by an average of 73.7% % in node classification and 82.5% in link prediction. In the Disease link prediction task, SST outperforms full-rank training at both r = and r 2. Notably, SST's advantage over LoRA is greater at r than at r = 2, likely due to SST's sampling strategy being particularly effective in sparser scenarios.
The results indicate that TOP_R __PR performs the worst, as its search space collapses into restricted low-rank subspace. In contrast, as long as all singular vectors are visited, the other methods deliver comparable performance. Among these, MULTINOMIAL demonstrates a slight advantage.
6. Conclusion and Discussion
In this work, Sparse Spectral Training (SST) has demon- strated its efficacy as parameter--efficient pre-training methodology that surpasses other parameter-effiiien meth- ods, and better approximates the learning dynamics and performance of full-rank training across diverse architec- tures, tasks, and embedding geometries. SST introduces novel approach by updating all singular values and se- lectively adjusting the singular vectors of network weights. Moreover, SST incorporates SVD both for the initializa- tion and periodic reinitialization of low-rank parameters. Future directions for SST include: (I) Investigating faster convergence approaches that avoid optimizer state reset. (2) Extending the application of SST to the embeddings of large language models (L..Ms).
5.4. Impact of Sampling Mechanisms
To evaluate the impact of different sampling mechanisms on the performance of SST, we conducted additional experi- ments using vanilla Transformer with model dimension of 64 and r = 8 on the IWSLT'14 14 dataset. The evalua- tion metric is BLEU, where higher scores indicate better performance. Table 6 summarizes the results:
Descriptions of Sampling Mechanisms:
MULTINOMMAL: The multinomial random sampling method used in SST.

---

Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks
Acknowledgements
2014, URL UR.https://acllatthoolll ogg.o///..../...... iwslt-evallutiinn.
This work was supported by the Zhou Yahui Chair Professor- ship award of Tsinghua University (to CVC), the National High-Level Talent Program of the Ministry of Science and Technology of China (grant number 20241710001, to CVC). We also thank Weize Chen for the helpful discussions on hyperbolic models.
Cettolo, M., Federico, M., Bentivogli, L., Niehues, J., Stüker, S., Sudoh, K., Yoshino, K., and Federmann, C. Overview of the IWSLT 2017 evaluation cam- paign. In Proceedings of the 14th International Con- ference on Spoken Language Translation, pp. 2-14, Tokyo, Japan, December 14-15 2017. International Work- shop on Spoken Language Translation. URL https: //aclanthology wrt. iwsit-1.1.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Chami, I., Ying. Z., Ré, C., and Leskovec, J. Hyperbolic graph convolutional neural networks. Advances in neural information processing systems, 32, 2019.
References
Chen, W., Han, X., Lin, Y., Zhao H., Liu, Z., Li, P., Sun, M., and Zhou, J. Fully hyperbolic neural net- works. In Muresan, S., Nakov, P., and Villavicen- cio, A. (eds.), Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 5672-5686, Dublin, Ire- land, May 2022. Association for Computational Linguis- ties. doi: 10.118653w1/22022.cclllo / //aclantholoyy wwg. act.:222.cclll.. awlalcllnnn.ooe..aaa 389.
Anderson, R. and May, R. Infectious Diseases of Hu- mans: Dynamics and Control. Infectious Diseases of Hu- mans: Dynamics and Control. OUP Oxford, 1991. ISBN 9780198540403. URL https: /books............o....k..... google. com.
URL https:
Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. Piga: Reasoning about physical commonsense in natural language. In Fourth AAAI Conference on Artificial Intelligence, 2020.
Cho, H., DeMeo, B., Peng. J., and Berger, B. Large-margin classification in hyperbolic space. In Chaudhuri, K. and Sugiyama, M. (eds.), Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learn- ing Research, pp. 1832-1840. PMLR, 16-18 Apr 2019. URL https::///// /tstts::/ppeeee proceedo ngs ml m.r.... pww.s/rsssvv cho19a.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D.,
Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL ptttps://prcceeednggg... neurips. cc/ cc/paper_ htapeerrffieessppeee////////// 145700600fc3401111118B0111212 pdf.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/ 803.05557 2018.
Cohen, G., Afshar, S., Tapson, J., and van Schaik, A. Emnist: Extending mnist to handwritten letters. In 2017 Interna- tional Joint Conference on Neural Networks (IJNNN, pp. 2921 2017. doi: 10.100/UCNNN.2017..6666
Cannistraci, C. V. and Muscoloni, A. Geometrical congru- ence, greedy navigability and myopic transfer in complex networks and brain connectomes. Nature Communica- tions, 13(1):7308, 2022.
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized 1lms. Advances in Neural Information Processing Systems, 36, 2024.
Cettolo, M., Niehues, J., Stüker, S., Bentivogli, L.,, and Federico, M. Report on the 11th IWSLT evaluation campaign. In Federico, M., Stüker, S., and Yvon, F. (eds.), Proceedings of the 11th International Work- shop on Spoken Language Translation: Evaluation Cam- paign, pp. 2-17, Lake Tahoe, California, December 4-5
Ding, N., Lv, X., Wang, Q., Chen, Y., Zhou, B., Liu, Z., and Sun, M. Sparse low-rank adaptation of pre- trained language models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https: /oppnrevieww net forum? o/osss id=jxgz7FEqüu..
10

---

Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks
Eckart, C. and Young, G. The approximation of one matrix by another of lower rank. Psychometrika, 133):111211, 1936.
Hu, E. Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L.,, and Chen, W. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https:// openreview, net htt//oormmm/// wtt.ummmiddneeeeeeFF....
Elliott, D., Frank, S., Sima'an an, K., and Specia, L. Multi30k: Multilingual english-german image descriptions. In Proceedings of the 5th Workshop Vision and Lan- guage, pp. 70-74. Association for Computational Linguis- URL http:
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models, 2020.
tics, 2016. doi: 10. 10.11863/11/116-22222 //www. acl web. www.g//atoollggggoooooooooo owy/ 6- 3210.
Evci, U., Gale, T., Menick, J., Castro, P. S., and Elsen, E. Rigging the lottery: Making all tickets winners. In International Conference Machine Learning, pp. 2943- 2952. PMLR, 2020.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Klein, G., Kim, Y., Deng, Y., Senellart, J., and Rush, M. Opennmt: Open-source toolkit for neural ma- chine translation. In Proc. ACL, 2017. doi: 18 P17-4012. URL https: http://///i..og/11111111111. v1/217-4012.
Ganea, O., Becigneul, G., and Hofmann, T. Hyper- bolic neural networks. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc.,
Kopiczko, D. J., Blankevoort, T., and Asano, Y. M. Vera: Vector- random matrix adaptation. In The Twelfth International Conference on Learning Representations.
2018. URL https://prcceeddiiii nee.riiiinnnnnnnn............iis cc/ scc/pape__fie /taeee__files//pppr/////////// (bab2addeff9007000888888 pdf.
42 Paper.
Lecun, Y., Bottou, L., Bengio, Y., and Haffner, Gradient- based learning applied document recognition. Pro- ceedings of the IEEE, 86(11):2278-2324 1998. doi: 10. 10.1109/5.726791.
Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi,
A., Foster, C., Golding, I.., Hsu, J., Le Noac'h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation, 12 2023. URL https:://..... : /ww.///eoodoo///////////o/ org/ records/ 10256836.
Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning. In Moens, M.-F., Huang, X., Specia, L., and Yih, S. W.-t. (eds.), Pro- ceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045-3059, On- line and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18653/112221..coo htt.://www..mmmmmmmmm.aaaaaaaaam 20133///12221..mm/mmmm2222444 URL https:// a sclanthology org/ emnlp mainn. 243.
Gokaslan, A. and Cohen, V. Openwebtext cor- pus. http://Skyl on007 www.007..ithub.ii. to/ OpenWebTextCorpue 2019.
Gugger, S., Debut, L., Wolf, T., Schmid, P., Mueller, Z., Mangrulkar, S., Sun, M., and Bossan, B. Ac- celerate: Training and inference at scale made sim- ple, efficient and adaptable. httpss//githhb... com/ huggingface/ acceleratee 2022.
Levesque, H., Davis, E., and Morgenstern, L. The wino- grad schema challenge. In 13th International Confer- ence on the Principles of Knowledge Representation and Reasoning, KR 2012, Proceedings of the International Conference Knowledge Representation and Reason- ing, pp. 552-561. Institute of Electrical and Electronies Engineers Inc., 2012. ISBN 9781577355601 13th In- ternational Conference on the Principles of Knowledge Representation and Reasoning, KR 2012 Conference date: 10-06-2012 Through 14-06-2012.
Gulcehre, C., Denil, M., Malinowski, M., Razavi, A., Pas- canu, R., Hermann, K. M., Battaglia, P., Bapst, V., Ra- poso, D., Santoro, A., and de Freitas, N. Hyperbolic atten- tion networks. In International Conference on Learning Representations, 2019. URL https: openreview. net/ forum? wtt.rummmi==xxxxix J=r J=
He, K., Zhang, X., Ren, S., and Sun, J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE inter- national conference on computer vision, pp. 1026-1034, 2015.
Lialin, V., Muckatira, S., Shivagunde, N., and Rumshisky, A. ReloRA: High-rank training through low-rank updates. The Twelfth International Conference on Learning Representations, 2024. URL htttps:///ppeeeeeeeeeeeee net: tttp:/oorumiidd....ndd

---

Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks
Liu, Q.. Nickel, M., and Kiela, D. Hyperbolic graph neural networks. Advances in neural information processing systems, 32, 2019.
Nickel, M. and Kiela, D. Poincaré embeddings for learning hierarchical representations. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https:: wwwriiinnnnnneunrreoree.. cc cc//apper_fille/pppeer//////// 5 oataaadffteeeeea111ff00022222222 pdf.
Liu, S.-Y., Wang, C.-Y., Yin, H., Molchanov, P., Wang, Y.-C. F., Cheng, K.-T., and Chen, M.-H. Dora: weight- decomposed low-rank adaptation. In Proceedings of the 41st International Conference Machine Learning. ICML'24. JorR.org, 2024.
Liu, X., Zheng, Y., Du, Z.,, Ding, M., Qian, Y., Yang, Z., and Tang, Gpt understands, too. arXiv:21033 10385, 2021.
Paszke, A., Gross, S., Massa, F,, Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L.. Desmaison, A., Köpf, A., Yang. Z., De Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L.,, Bai, J., and Chintala, S. Pytorch: An imperative style. high-performance deep learning library. CoRR, abs/1912.01173. 2019. URL abs/1912.
Meng, F., Wang, Z., and Zhang, M. PiSSA: Princi- pal singular values and singular vectors adaptation of large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a. URL https: http:p/:///peeeeeiieeeee net/ id=6XBHTEtdP4.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, J. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21 (140):1-67, 2020.
Meng, X., Dai, D., Luo, W., Yang, Z., Wu, S., Wang, X. Wang, P., Dong. Q.. Chen, L., and Sui, Z. Peri- odiclora: Breaking the low-rank bottleneck in lora opti- mization, 2024b. URL https: / /ttp/axxvvv..//// org/ abs/ 1602.
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint arXiv: 10641, 2019.
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP, 2018.
Sen, P., Namata, G., Bilgic, M., Getoor, L.,, Galligher, B., and Eliassi-Rad, T. Collective classification in network data. Al magazine, 29(3):93 93, 2008.
Mocanu, D. C., Mocanu, E., Stone, P., Nguyen, P. H.,, Gibescu, M., and Liotta, A. Scalable training of arti- ficial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 9 (1):1-12, 2018.
Shimizu, R., Mukuta, Y., and Harada, T. Hyperbolie neural networks+++ In International Conference on Learning Representations, 2021. URL ptttps://openreveeee net (atrrummidd:Ee:500000000
Mostafazadeh, N., Chambers, N., He, X., Parikh, D., Batra, D., Vanderwende, L., Kohli, P., and Allen, A corpus and cloze evaluation for deeper understand- ing of commonsense stories. In Knight, K., Nenkova, A., and Rambow, O. (eds.), Proceedings of the 2016 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Lan- guage Technologies, pp. 839-849, San Diego Cali- fornia, June 2016. Association for Computational Lin- guisties. doi: 10. 10.11865//1/N16- 1098. URL https: //aclanthology org
Tifrea, A., Beeigneul, G., and Ganea, O.-E. Poincare glove: Hyperbolic word embeddings. In International Confer- ence Learning Representations, 2019. URL https: / /ooenrevi ew net/ / forum?irroooo d=5ke5
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv: 13971, 2023a.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y.. Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn A., Hosseini, S., Hou, R.. Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A,, Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog,
Muscoloni, A., Thomas, J. M., Ciucci, S., Bianconi, G., and Cannistraci, C. V. Machine learning meets complex networks via coalescent embedding in the hyperbolic space. Nature communications. 8(1): 2017.
Namata, G., London, B., Getoor, L., and Huang, B. Query- driven active surveying for collective classification. 2012.
12

---

Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks
I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, F. M., Subramanian, R., Tan, X. E., Tang. B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A.. Kambadur, M., Narang, S., Rodrigue7, A., Stojnic, R., Edunov, S.,
for parameter-efficient fine-tuning. In The Eleventh International Conference Learning Representations, 2023. URL https: http///pppneeeeeeeeeeeeeeeeeeeee nett nww.:/rumtffrr forum? id=1q62uwRJjiX.
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01060 2022.
and Scialom, T. Llama 2: Open foundation and fine-tuned chat models, 2023b.
Valipour, M., Rezagholizadeh, M., Kobyzev, I., and Gh- odsi, A. Dylora: Parameter-efficien tuning of pre-trained models using dynamic search-free low-rank adaptation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 3274-3287, 2023.
Zhang, Y., Zhao, J., Wu, W., Muscoloni, A., and Can- nistraci, C. V. Epitopological learning and cannistraci- hebb network shape intelligence brain- brain-inspired theory for ultra-sparse advantage in deep learning. In The Twelfth International Conference on Learning Representations, 2024. URL https:/ /ttp//pppnneeeeeeeeeee/eeeoe.eep net.ieewwweeeeeeee.ooe net/forunt forum? id- id-iayEcORsGd.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Zhao, J., Zhang, 7., Chen, B., Wang. Z., Anandkumar, A., and Tian, Y. Galore: Memory-efficient 1lm training by gradient low-rank projection, 2024.
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J. Hill, F., Levy, O., and Bowman, S. Su- perglue: A stickier benchmark for general-purposs language understanding systems. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL https::t//// http:pssss///pceeedeee/m neurips.cc/ peerippe.cc/ppperr///1///e. 4498122aaeeTaa0000064442222284-3 pdf.
Zi, B., Qi, X., Wang. L., Wang, J., Wong, K.-F,, and Zhang, L. Delta-lora: Fine-tuning high-rank parameters with the delta of low-rank matrices, 2023.
Xia, W.. Qin, C., and Hazan, Chain of lora:
Efficient fine- tuning of language models via residual learning, 2024.
Xiao, H., Rasul, K., and Vollgraf, R. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms, 2017. URL hwtphhttt/a/aattaa //trxiv..avvv//aa///a org/ / 1708. 07747.
Yang, M., Feng, A., Xiong, B., Liu, J., King, I. and Ying, R. Hyperbolic fine-tuning for large language models, 2024. URL https::////. //axiiivvvv/ v. org/ abs/ 2410 0 04010.
Yuan, G., Ma, X., Niu. W., Li, Z., Kong, Z., Liu, N., Gong, Y., Zhan, 7.,, He, C., Jin, Q. al. Mest: Accurate and fast memory-economic sparse training framework on the edge. Advances in Neural Information Processing Systems, 34: 20838-20855 2021.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can machine really finish your sen- tence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Zhang, Q. Chen, M., Bukharin, A., He, P., Cheng, Y.,
Chen, W., and Zhao, T. Adaptive budget allocation
13

---

Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks
A. Algorithm of Sparse Spectral Training
Algorithm 2 Sparse Spectral Training (SST)
input Dataset D; total round T1; number of iterations T2; iteration interval T3
Use Kaiming initialization to initialize origin model's weight k k 1,.... n, where n is the number of linear layers. Replace origin model's 's weight with SVD decomposition
10 2/100) (1..0) ((1..0) T ]= =SVD(W
for 1 0, .,.., Ti 1 do
for 12 0,..., T2- do
1 = (1, m be the set of all possible indices of singular vectors
s ((ti.tta
C IR. s(tt.tz)
Multinomial(y, (2((1.223a))
for t3 = 0 do Represent 12 x T3 + t3: Sample mini-batch from D and compute the forward pass by Eq.3 and compute the gradient VC; Update > ((1.++1) = max ((..)) ny V 0)
Update
(1.00) 1/V VU
Villad)
VVV..
(1,4+1) kid
if E Sk
((hitt))
(6.1)
1/V LU,.1
(1.
-1VLvE..
k
kyd
where UR means column vector i of Up end for end for Reinitialize with new SVD decomposition
T (t.+) (..+ v(t++10)) SVD(U (2xTTxxs (t..T2
(( (ta 1)²)
>
end for
B. Related Work of Other Parameter-Efficiet Training Methods
Apart from low-rank adaptations, researchers have developed a variety of parameter-efficien training techniques to optimize resource consumption while preserving learning effectiveness. Prompt tuning is an effective method that integrates tunable prefixes or soft prompts into the input embeddings of models. It enables lightweight task-specite adaptations with minimal impact the model's overall architecture (Lester et al., 2021; Liu al., 2021). Dynamic sparse training (DST), through methods like SET (Mocanu et al., 2018), RIGL (Evei et al., 2020), MEST (Yuan et al., 2021), and CHT (Zhang et al., 2024), employs dynamic prune-and-gre strategy that adjusts network topology during training. This approach optimizes training efficiency and can improve generalization by continuously adapting the network's 's sparse structure. This presents significant shift from static training methods.
C. Experiments on Larger Datasets and Hyperparameter Tuning
To further evaluate the performance of SST, conducted additional experiments using larger datasets and varied hyperpa- rameter settings. Specifically, we pre-trained LLaMA-130M the C4 dataset (Raffel et al., 2020), which is about 25 times larger than OpenWeTText. We also compared the performance of SST, LoRA, and ReL.RRA* under two different learning rates.
Table 7 presents the validation perplexity (PPL) results for LLaMA- 30M on both C4 and DpenWeeText.. The results show that SST consistently outperforms other low-rank methods, achieving lower perplexity across all configurations.
14

---

Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks
Table 7: Validation perplexity on C4 and Open WebText Text for LLaMA- 300 with different learning rates. Bold values indicate the lowest PPL among all low-rank methods.
r/d
(Inwle-3)
Ir=le-3 ReLoRA*
Ir=3es3 ReLoRA*
LoRA
SST
LoRA
SST
C4
LLaMA-130M
64/768
24.91
35.91
37.34
32.13
30.75
133.06
29.79
WeenWe
LLaMA-130M
64/768
20.04
29.71
31.33
25.89
795.24
230.43
23.35
Each method was trained with 2.6 billion tokens. The learning rate 1e-3 for full-rank training aligns with the configuration used in the ReLoRA article. For consistency, we applied the same learning rates (Ir = le-3 and ir = 3e-3) across LoRA, ReLoRA* and SST.
SST consistently achieves lower perplexity than LoRA and ReL.oRA* * the same learning rate. Notably, with ir 3e 3, SST surpasses all other low-rank methods, reducing the perplexity gap by 16.4% C4 and 65.8% OpenWeb These findings highlight SST's effectiveness and robustness larger datasets and varied learning rate configurations.
D. Proof of Gradient of Sparse Spectral Layer
We can express the differential of W as the sum of differentials:
dw dUEV +UDEV UE
(12)
We have chain rule for the gradient of W:
0C
OC ōh
Dc Th
x
(13)
OW
Th OW
dc
dC
dw
OW OC
OC
OL
EV
VT
UE
OW
oW
OW
Oc
VE dU + U dc
V dΣ TUT OC
ow dVT
ow
aw
where: is the Frobenius inner product. So we have the gradient of U, E and VT:
Oc OL
OL
EUT OL OC
UT OL
VΣ,
V
(14)
OU
OW
UVT
OW
0Σ
UW
In vector perspective, for the zth vector, it is:
ac OL aU., aw V.E,
OL OL av.;
OL
U,T dL OW
E,
U.
V
(15)
OWT
OΣ;
where U., means the th column vector and E, is the 7th value of the diagonal matrix Σ.
E. Experiment Details
E.1. Implementation Details for SST
Sampling of U and VT. In our experiments, we employ a more exploratory approach when sampling U and VT;
1 Σ; 1 m Σ, Σラ
(()
(16)
15

---

Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks
where p(i) is the possibility sample index i vector of U and VT. This method modifies the earlier Eq. 5 by combining the multinomial distribution with uniform distribution. This adjustment ensures that vectors associated with lower singular values still have substantial likelihood of being sampled, preventing their probabilities from becoming excessively low and promoting more balanced exploration across the spectral components.
Optimizer state reset and warmup. Before each iteration, Sparse Spectral Training (SST) resets all optimizer states for U, V and Σ. For example, for optimizers like Adam, this involves clearing the first and second moments as well as the timestep. Consequently, a brief warmup period essential at the beginning each iteration to accommodate the reset states. This warmup period is typically 20 steps, guided by the exponential decay rate used in the Adam optimizer.
Hyperbolic SST.
The formula of hyperbolic linear layer (Chen al., 2022) is:
Wxlla W
+ V
Wxll2 v
h fx(X)=
x
(17)
Wx
where v E Ra+1, W E Rmx((+11) and K the curvature. The formula of Hyperbolic SST is:
h VIUEVTIIl v UEVIX
(18)
E.2. Hyperparameters of Machine Translation
14SLT'14 The hyperparameters be found in Table 8. We employ the same codebase and hyperparameters those used in HyboNet (Chen al., 2022), which is derived from OpenNMT-py (Klein al., 2017). For all methods, last checkpoint is utilized for evaluation. Beam search, with beam size of 2, is employed to optimize the evaluation process. Experiments were conducted on one A100 GPU.
For SST, iteration interval (T3) is set to 200. Each iteration begins with a warmup phase lasting 20 steps. The number of iterations per round (T2) is determined by the formula T: d/r, where d represents the embedding dimension and denotes the rank used in SST.
Table 8: Hyperparameters on IWSLT'14 for Euclidean and hyperbolic Transformer.
Hyper-parameter
Euclidean
Hyperbolic
Embedding Dimension
64, 128, 256 256, 512, 1024 10240 tokens
64, 128, 256
Feed-forward Dimension
256, 512, 1024 10240 tokens
Batch Size
Gradient Accumulation Steps
4
Training Steps
40000
40000
Dropout
0.0
0.1
Attention Dropout
0.1
0.1
Max Gradient Norm
0.5
Warmup Steps
6000
6000
Decay Method
noam
noam
Label Smoothing
0.1
0.1
Layer Number
6
6
Head Number
4
4
Learning Rate
5
2
Optimizer
Adam
rAdam
Multi30K and 17 The hyperparameters can found in Table 9. Because of overfitting, model checkpoint with lowest validation loss is utilized for evaluation. larger learning rate (0.0003) is used for low rank parameters (U, VT and
16

---

Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks
Σ for SST, B and A for LoRA and ReLoRA*. Experiments were conducted one A100 GPU.
For SST, interation interval (Iz) is set to 200 for Multi30K and 400 for IWSLT 17. Each iteration begins with warmup phase lasting 20 steps. The number of iterations per round (T₃) is determined by the formula T2 = d/r, where d represents the embedding dimension and denotes the rank used in SST.
Table 9: Hyperparameters Multi30K and IWSLT'17 for vanilla Transformer.
Hyper-parameter
Multi30K IWSLT'17
Embedding Dimension
512
512
Feed-forward Dimension
2048
2048
Batch Size
128 sentences
128 sentences
Gradient Accumulation Steps
1 100000
1
Training Steps
150000
Dropout
0.1 constant
0.1
Decay Method
constant
Layer Number
6
6
Head Number
8
8
Learning Rate
0.0001
0.0001
Weight Decay
1
0.1
Optimizer
AdamW
AdamW
E.3. Hyperparameters of Natural Language Generation
Hyperparameters for OPT. The hyperparameters for OPT are detailed in Table 10. We employ a linear warmup of 2000 steps followed by a stable learning rate, without decay. A larger learning rate (0.001) used for only low rank parameters (U, V and Σ for SST, B and A for LoRA and ReLoRA*, The total training tokens for each experiment 19.7B, roughly epochs of Open WebTet. Distributed training facilitated using the Accelerate (Gugger et al., 2022) library across four A100 GPUs on Linux server.
For SST, interation interval (I3) is set 200. Each iteration begins with warmup phase lasting 20 steps. The number of iterations per round (T2) is determined by the formula Tr d/r, where d represents the embedding dimension and denotes the rank used in SST.
Table 10: Hyperparameters for OPT Models
Hyper-parameter
OPT-125M
OPT-350M
OPT-1.3B
Embedding Dimension
768
512 (project to 1024)
2048
Feed-forward Dimension
3072
4096
8192
Global Batch Size
240
240
240
Sequence Length
2048
2048
2048
Training Steps
40000
40000
40000
Learning Rate
0.0001
0.0001
0.0001
Warmup Steps
2000
2000
2000
Optimizer
AdamW
AdamW
Adam W
Layer Number
12
24
24
Head Number
12
16
32
Hyperparameters for LLaMA. The hyperparameters for L.LaMA are detailed in Table 11. We follow the same experiment setup from (Zhao et al., 2024). We employ a linear warmup of 2000/10000 steps followed by cosine decay. For LLaMA- 130M the learning rates for LoRA, ReLoRA* and SST are selected from (1e-3, 3e-3} based on the lowest PPL observed in Table 7. For LLaMA-1.33 1.3B, the learning rates for LoRA, ReLoRA*, and SST are fixed at 1e-3, The learning rates for full-rank training are set 1e-3 for LLaMA- 130M and 4e-4 for LLaMA-1 11.3B, consistent with the configuration in the
17

---

Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks
Full
42.5
SST
40.0
100%
99%
37.5
98%
Perplexity
97%
35.0
96%
32.5
95%
93%
30.0
90%
27.5
25.0
0.0
0.1
0.2 Pruned Ratio
0.3
0.4
Figure 5: Singular Value Pruning. We conduct singular value pruning on full-rank and SST pretrained OPT-125M model. After performing singular value decomposition on weight matrices, we preserve the top k singular values so that the cumulative sum of preserved singular values ranges from [100%, 99%, 98%, ..., 93%, 90%] of the original cumulative sum. The pruned ratio of singular values plotted along the x-axis.
ReLoRA article.
For SST, interation interval (Z3) is set to 200. Each iteration begins with a warmup phase lasting 20 steps. The number of iterations per round (T2) is determined by the formula T₃ d/r, where d represents the embedding dimension and denotes the rank used in SST.
Table 11: Hyperparameters for LLaMA Models
Hyper-paraamter
LLaMA-130M
LLaMA-1.3B
Embedding Dimension
768
2048
Feed-forward Dimension
2048
5461
Global Batch Size
512
512
Sequence Length
256
256
Training Steps
20000
100000
Learning Rate
0.001
0.0004
Warmup Steps
2000
10000
Optimizer
Adam
Adam
Layer Number
12
24
Head Number
12
32
E.4. Hyperparameters of Hyperbolic Graph Neural Networks
We use HyboNet (Chen al., 2022) as full-rank model, with same hyperparameters those used in HyboNet. Experiments were conducted on one A100 GPU,
For SST, interation interval (Ta) is set to 100. Each iteration begins with warmup phase lasting 100 steps. The number of iterations per round (T2) is determined by the formula T- d/r, where d represents the embedding dimension and denotes the rank used in SST.
We set dropout rate to 0.5 for the LoRA and SST methods during the node classification task on the Cora dataset. This is the only one deviation from the HyboNet configuration.
18

---

Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks
F. Singular Value Pruning
We further conduct an analysis study the potential for using SST model for further compression. The results, as shown in Figure 5, indicate that the SST model retains lower perplexity across wider range of pruning ratios compared the full-rank model. This suggests that the SST method effectively concentrates the informational content of the weights into fewer singular values, making it more suitable for further compression.
This enhanced performance underscores the potential of SST in maintaining essential model characteristics even under significant compression, making it a promising approach for developing lightweight yet powerful language models for inference.
G. Evaluating SST and GaLore: Complementary Approaches to Memory Efficiency
Table 12: The BLEU score on I4SLT'14 for Euclidean Transformer, compared with GaLore. Values highlighted in bold represent the highest performance among the low rank methods, while those marked with an "** denote performance that exceeds that of the full-rank variants.
Dimension
r
Full
GaLore
SST
8
18.08
22.28
64
24.27
4
14.07
20.27
16
23.43
25.12
128
8
25.79
19.71
24.19
4
16.01
22.80
32
24.01
23.97*
16
22.82
23.42
256
23.92
8
20.12
22.65
4
15.94
21.39
Recently, new approach named Gradient Low-Rank Projection (GaLore) (Zhao al., 2024) has been proposed to address the memory challenges associated with pre-training large language models. GaLore, by implementing memory-efficient gradient projection method.
Using the released code of GaLore', we conducted comparative experiments on the IWSLT' 14 dataset with Transformer models, employing the same configurations as other low-rank methods. We set the scale factor a = in these experiments because a = 0.25, which is used in the article, performs much worse than a = 1. As illustrated in Table 12, SST method consistently outperformed GaLore across various model dimensions and ranks, except for d 256, r 32.
In addition, we evaluated validation perplexity the WpenWebText dataset with OPT-1 1 and OPT-350M models. As shown in Table 13, SST outperformed GaLore 0PT- and OPT-350M. Zero-shot evaluations comparing SST with GaLore are presented in Table 14, which also demonstrate SST's superior performance.
Here, we discuss our guess on why SST may have an advantage over Gal.ore Lore on low-rank settings. GaLore utilizes a projection matrix P. E R derived from the singular value decomposition (SVD) of a single step's gradient. Only one step's gradient may introduce noise due to data sampling variability, Conversely, SST employs U and V as projection matrices, which are initialized and reinitialized with the SVD of W. W could be seemed the momentum of gradient of W, less noisy than step's gradient. Furthermore, SST updates all E values, regardless of r, making it more robust decreases.
H. Ablation Study
Impact of Σ updates. We conduct an ablation study to evaluate the impact of various components and configurations within SST on the IWSLT'14 14 using Fuclidean Transformer with a dimension of 128 and rank of 4. The results of this study are summarized Table 15, which highlights the contributions of specific elements the overall performance
hhtpps:// thub. com/ cttbb/ccoo//aeeeeeeeeeee jiawei zzhao/ zzhao/Galoree Gooor
19

---

Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks
Table 13: Validation perplexity, compared with GaLore on Open OpenWebText dataset with OPT-125M and OPT-350M, along with the number of trainable parameters of each method. 7 = 64. Values highlighted bold represent the highest performance among the low rank methods.
Full
Gal.ore
SST
OPT-125M
23.50 (125.2M)
32.17 (45.6M)
26.98 (51.0M)
OPT-350M
21.78 (331.2M)
1994 (43.4M)
27.72 (57.7M)
Table 14: Zero-shot evaluations, compared with GaLore with same tasks Table 4. Mean scores in bold represent superior performance among the low-rank methods. Win percentage (including ties) for each low-rank method is compared to the full-rank training.
OPT-125M
OPT-350M
Full
GaLore
SST
Full
GaLore
SST
ARC (Challenge)
21.2
21.2
22.0
21.1
ARC (Easy)
35.8
33.7
34.3
35.9
25.7
35.7
BoolQ
59.5
61.8
62.0
53.6
37.8
57.7
CB
51.8
37.5
48.2
44.6
41.1
41.1
COPA
67.0
64.0
66.0
69.0
52.0
60.0
HellaSwag
27.7
27.0
26.9
28.4
26.2
27.5
MultiRC
55.4
57.2
57.2
52.0
42.8
57.0
OpenBookQA
24.6
23.6
26.2
26.4
27.8
25.2
PIQA
58.7
57.1
58.3
59.2
50.5
59.0
ReCoRD
15.0
18.5
19.4
17.5
23.2
RTE
50.5
51.6
53.4
$2.0
52.7
50,2
StoryCloze
55.8
53.5
54.5
57.2
49.7
54.6
WIC
49.8
50.0
50.0
50.5
50.0
50.2
Winograd
52.0
50.9
50.6
55.0
50.2
51.3
Winogrande
49.1
51.7
50.1
50.7
49.4
52.0
WSC
36.5
36.5
36.5
36.5
63.5
36.5
Mcan
44.5
43.3
44.6
44.5
41.4
43.9
Win Percentage
43.8
56.3
25.0
31.3
measured in BLEU score.
One variation tested involves changing the update mechanism for Σ. Instead of updating all E, only sampled E are updated, same as update for U and VT, This modification results in lower BLEU score of 22.40, indicating that full updates of Σ contribute positively to the model's performance.
Initialization method. We experiment with configuration similar to the ReLoRA*, where h = (W + UEVT) )xx with U and VT randomly initialized and Σ initialized to zero. After each round, U, VT and Σ are reinitialized. This setup significantly reduces the BLEU score to 16.03, which similar to the performance of LoRA (16.37) and ReLoRA* (18.00). This demonstrates that the most important feature of SST is that instead of randomly initialized, SST uses SVD of W the initialization of U and VT, which is aligned with our analysis in section 4.3.
Impact of iteration interval (Tz). We also conducted additional experiments study the impact varying iteration interval T3 (sampling period). All methods were trained on vanilla Transformer model with hidden dimension of 64 and == 8 on the IWSLT' 14 dataset. In the original setup (Table 1), T3 was set to 200 steps iteration.
As shown in Table 16, both excessively large and small values of Ta result in decreased performance. A large T3s may cause SST degrade to LoRA, while a small Ty leads to frequent resets of the optimizer's momentum, thereby affecting convergence.
Impact of Number of Iterations. We conducted an additional experiment on the IWSLT' 14 dataset using a vanilla Transformer to evaluate the impact the number of iterations per round, with model dimension of 64 and r 8. The
20

---

Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks
Table 15: Ablation Study on IWSLT" 14 dataset with Euclidean Transformer. Dimension is 128 and r is 4.
BLEU
LoRA
16.37
ReLoRA#
18.00
SST Instead of update all Σ, only update sampled Σ
22.40
SST Use formula similar as ReLoRA*: h (W + -UEVV) xx (U and V random initialized, and E zero initialized)
16.03
SST
22.80
Table 16: Impact of iteration interval (Tz) on BLEU scores for IWSLT" 14.
Steps per Iteration T3
800
400
200
100
50
25
10
BLEU Score
21.85
23.64
22.47
22.49
22.60
22.46
22.25
results are summarized in Table 17:
Table 17: Impact of number of iterations per round on BLEU scores for IWSLT' 14.
Number of Iterations per Round
2
4
8
16
32
BLEU Score
22.28
22.21
22.24
22.28
22.30
22.37
The results indicate that different numbers of iterations yield comparable performance. our experiments, this hyperparam- eter not tuned; instead, fixed it to d/r.
Impact of Rank. For all low-rank methods, including LoRA, ReLoRA*, and SST, rank is more of constraint determined by available resources rather than hyperparameter be extensively tuned. Higher ranks generally lead to better performance but at the cost of increased memory consumption. To ensure fairness, the same rank values were used for LoRA, ReLoRA*, and SST in all experiments, as these methods have a similar number of trainable parameters under the same rank.
Additionally, we conducted an experiment on the IWSLT" 14 dataset using a vanilla Transformer with model dimension of 128 to analyze the impact rank on different methods. The results are presented in Table 18:
The evaluation metric is BLEU, where higher scores indicate better performance. The BLEU score for full-rank training 25.79. The results demonstrate that the rank increases, the performance of all methods improves. Notably, SST consistently outperforms other low-rank methods, especially at smaller ranks, highlighting its robustness under resource-constraing settings.
Impact of Training Steps. To investigate whether additional training steps benefit SST, conducted experiment on the IWSLT' 14 dataset using vanilla Transformer with model dimension of 64 and = 4. Table 19 presents the BLEU scores for full-rank training and SST under different training steps (evaluated on the model at the last step):
The results demonstrate that as the number training steps increases, the gap between full-rank training and SST narrows. Even with r 4, SST approaches the performance of full-rank training 640,000 steps. These findings confirm that while SST may require more steps converge at lower ranks, it remains competitive with full-rank training given sufficient steps.
Memory Consumption and Training Time
Memory consumption. As shown Table 20, the memory consumption of SST is comparable to LoRA and much smaller than full-rank models. SST has a similar number of trainable parameters (about 0.2% higher) as LoRA (as stated in Table 2), more frozen parameters (about 45% higher) than LoRA. However, this can be mitigated if we use low precision for the
21

---

Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks
Table 18: Impact of rank on BLEU scores for IWSLT" 14. Dimension is 128.
Rank (r)
2
4
8
16
32
64
LoRA ReLoRA
12.44
14.16
16.37
20.56
23.30
25.12
26.
14.53
15.39
18.00
20.61
22.92
24.15
25.25
SST
17.49
20.69
22.80
24.19
25.12
26.08
26.15
Table 19: BLEU scores under different training steps.
The default training step in Table is 40,000.
Steps
20,000
40,000
80,000
160,000
320,000
640,000
Full 22.95
24.27
24.85
24.72
24.71
25.05
SST
17.23
20.27
21.91
22.86
23.32
23.92
frozen parameters, as in (Dettmers et al., 2024).
Table 21 shows that the memory consumption of SVD decomposition for the largest weight in each model about 3%, which is small compared with the whole model.
Training time. Table 22 shows that the time spent SVD in SST is very low, about 0.5%-0.8% compared with the whole training time. SST has comparable training time as LoRA and full-rank model. The increasement of training time of SST is mainly due to SST's linear function, h = UEV x, which is slower than original h Wx. However, during inference, replacing UΣVT with a single matrix W could obtain same computation efficiency as full-rank models. ReLoRA* has comparable computation time as LoRA.
Performance with Fewer Steps. Despite requiring slightly more time per step, SST achieves superior performance with fewer training steps compared to other low-rank methods. The choice of 20% fewer steps for SST corresponds to the maximum additional training time incurred by SST compared to other low-rank methods, as shown in Table 22. Table 23 compares the perplexity (PPL.) of SST trained with 20% fewer steps to that of other methods trained with full steps.
These results demonstrate that SST maintains significantly lower perplexity even with fewer training steps, highlighting its efficiency. SST effectively balances its computational overhead while achieving superior performance compared to other low-rank methods. This makes SST compelling choice for high-quality pretraining.
J. Experiment on Image Classification
We conduct additional experiments on image classification tasks using MLP-based models. In this section, we provide a comparison of full-rank training. LoRA, ReLoRA* and SST on three datasets: MNIST (Lecun et al., 1998), EMNIST (Cohen et al., 2017), and Fashion_MNISS (Xiao al., 2017).
The architecture of the MLP is 784 512 - 512 512 #class. Each method is trained for total of 100 epochs. Learning rate is set to 0.01 for all methods.
We use a rank of 16 for all low-rank methods, which corresponds to 1/32 of the full-rank dimension. For ReLoRA* o and SST, one epoch per iteration is used. The results are averaged over three random seeds, and all datasets were evaluated based on test accuracy.
As shown in Table 24, SST outperforms both LoRA and ReLoRA* * across all three datasets. SST reduces performance gap between low-rank method and full-rank training by 49% % in average.
K. Memory Efficiency Analysis
To better understand the memory efficiency of SST compared to baseline methods, we provide a detailed joint analysis of GPU memory consumption and performance trade-offs.
22

---

Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks
Table 20: GPU memory consumption on different sizes of OPT models, including optimizer state and gradient. Model weight uses float32. AdamW optimizer state uses float32 (same data type as used in OPT experiments in Table 2).
Full
LoRA/ReLoRA
SST
OPT-125M
1956.05 MB
1118.56 MB
1254.41 MB
OPT-350M
5070.41 MB
2046.41 MB
2573.77 MB
OPT-1.3B
20093.22 MB
7133.24 MB
9345.72 MB
Table 21: GPU memory consumption of SVD decomposition in SST.
Model
Largest Weight Shape
Peak GPU Memory Consumption
OPT-125M
768 3072
41.25 MB (3.3%)
OPT-350M
1024 x 4096
72.00 MB (2.8%)
OPT-1.3B
2048 x 8192
288.01 MB (3.1%)
Memory and Performance Trade-Off. SST's GPU memory consumption is comparable to ReLoRA*, while achieving
significant improvements in perplexity (PPL). A comparison of memory reduction and PPL increase is provided in our analysis (Figure 6).
We define the following metrics for clarity:
Memory Reduction (%) Full memory Low rank memory
100
Full memory
PPL Increase (%) Low rank PPL Full PPL
100
Full PPL
To provide more intuitive understanding of SST's memory efficiency, we introduce new metric called the efficiency ratio, defined as:
Memory Reduction (%)
Efficiency Ratio
PPL Increase (%)
This efficiency ratio quantifies how much memory be reduced the cost of 1% increase in PPL. A higher efficiency ratio indicates a more memory-efficient method.
Results. SST achieves a significantly higher efficiency ratio than ReLoRA* across various pretraining tasks. Figure 7 shows the efficiency ratio improvements of SST compared to ReLoRA*:
167.4% (OpenWebText, LLaMA-130M)
99.7% (C4, LLaMA-130M)
196.1%
(OpenWebText, OPT-125M)
142.3%
(OpenWebText, OPT-350M)
65.9%
(epenWeeText, OPT-1.3B)
4434.3% (OpenWebText, LLaMA-1.33)
Conclusion. These results demonstrate that SST achieves a substantially better trade-off between memory reduction and PPL increase compared to * This highlights SST's effectiveness in optimizing memory efficiency while maintaining strong model performance, making it a practical choice for resource-constrained pretraining tasks.
23

---

Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks
Table 22: Overall training time on different sizes of OPT models with 19.7 billion training tokens, using 4 A A100 GPU. "Time of SVD in SST" is the overall time of singular value decomposition within SST.
Model
Full
LoRA
SST
Time of SVD in SST
OPT-125M
62.5h
64.4h
65.0h
0.3h (0.5%)
OPT-350M
135.8h
153,3h
170.0h
0.8h (0.5%)
OPT-1.3B
303.4h
324.8h
387.2h
3.0h (0.8%)
Table 23: Validation perplexity with SST trained 20% fewer steps compared to full steps for other methods.
Model
Full
LoRA
ReLoRA* *
SST (20% fewer steps)
OPT-125M
23.50
34.23
35.80
28.03
OPT-350M
21.78
34.26
39.21
29.42
OPT-1.3B
15.10
1716
29.52
22.98
LLaMA-130M
20.04
29.71
31.33
24.74
LLaMA-1.3B
14.54
16.50
17.32
15.65
L. Additional Baseline Comparisons: Dora and Vera
In the main text, we focused on comparing SST with ReLoRA and GaLore, as they are specifically designed for pre-training, Other methods, such as Dora (Liu et al., 2024) and Vera (Kopiczko et al.), primarily target fine-tuning scenarios, where the parameter search space is restricted to a low-rank subspace. As discussed Section 3.1, this restriction limits their expressiveness, which may be sufficient for simple fine-tuning tasks but becomes a bottleneck in more challenging tasks like training from scratch, the focus of this work.
To further assess the limitations of Dora and Vera in pre-training tasks, conducted additional experiments comparing them with SST on IWSLT' 14 using Transformer models. The BLEU scores (higher is better) are reported in Table 25.
The results show that while Dora slightly outperforms LoRA, remains inferior to SST. Vera performs poorly across all settings. Dora decomposes the pre-trained weights into magnitude and direction components, but this decomposition does not overcome the fundamental limitation of low-rank subspace. Vera further constrains the trainable parameters to single vector, which significantly reduces memory but fails to capture the complexity required for effective pre-training.
This analysis highlights the importance of flexible, spectral- based methods like SST for pre-training tasks.
24

---

Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks
Table 24: Image classification tasks test accuracy.
Dataset
Full
LoRA
ReLoRA**
SST
MNIST
98.63 = 0.04
97.69 0.10
97.72 + 0.05
98.33 + 0.04
EMNIST
85.32 = 0.24
79.45 # 0.26
84.12 0.12
84.96 + 0.11
F MNNSTT
Fashion
90.44 # 0.06
88.30 0.01
89.08 0.16
89.22 0.06
Table 25: BLEU scores on IWSLT'14 14 for different low-rank methods. SST consistently outperforms other approaches across various model dimensions and ranks.
(d,
Dora
Vera
LoRA
ReLoRA*
SST
(64, 8)
18.19
9.22
18.08
18.12
22.28
(64,
14.38
9.30
14.05
15.49
20.27
(128, 16)
23.46
12.22
23.30
22.92
25.12
(128, 8)
21.09
12.25
20.56
20.61
24.19
(128, 4)
18.12
11.61
16.37
18.00
22.80
Memory Reduction Compared Full-Rank (%)
10
20
30
40
50
60
0
SBo-Web-ext,OPT-12550
Open Text, LLMMA-133M
20
LLMM
PPL Increase compared to Full-Rank (%)
Open/WebText.PPT335000
40
C4 MALLMMA-130PPe C4 Www.eeeetetttt0
OpenWebbeext.pii.22
OpenWeb Let.
60
80
Ope Deen OPT-350M
ReLoRA* SST
SpenWebText.OPT-11
100
Figure Memory reduction vs. PPI. increase. Comparison of SST and R on multiple datasets and models.
25

---

Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks
2.5
ReLoRA+
SST
(Memory Reduction %6 (PPL Increase %)
2.0
196 1%
1.5
167.4%
142.3%
99 4%
1.0
0.5
0.0
OpenWebText.LaMM- LLA.A-110MAA 130M
C4LLLMM-113MM
OpenWeb Opet-/ebeext.OP--3550 OPT-350M
OpenWeb Tpxt Text, OPT-1 Text.o011111 3.38
OpenWebText.0PT12255
4434.3%
OpenWebTexx_LLMMAL.BBB
Figure 7: Efficiency Ratio Improvements. SST achieves significantly higher efficiency ratios compared to ReLoRA across various tasks and model sizes. The LLaMA- 1.3B result is included at the bottom of the plot due to its large value.
26

---

Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks
200
Full
Full
40
175
SST
SST
Singular Value
150
LoRA
Singular Value
30
LoRA
100
20
75
50
10
25
0
100
200
300
400
500 600
700
800
0
100
200
300
400
500
600
700 800
Singular Value Index
Singular Value Index
fc1
fc2
8
Full
Full
SST
SST
LoRA
LoRA
Singular Value
Singular Value
4-
4
2
2
0
100
200
300
400
500 600
700
800
0
100
200
300
400 500 600
700 800
Singular Value Index
Singular Value Index
proj
k proj
Full
100
Full
80
SST
SST
LoRA
80
LoRA
Singular Value
60
Singular Value
60
40
40
20
20
0
0
100 200
300 400 500 600 700 800
100 200 300 400 500 600 700 800 Singular Value Index
Singular Value Index
v proj
out proj
Figure 8: Singular Value Distribution. This visualization depicts the distribution of singular values for the OPT-125M model with full-rank, LoRA, and SST, with 64). The x-axis represents the index singular values, sorted from largest to smallest, while the y-axis shows the magnitude of each value. It highlights how LoRA predominantly captures and overestimates the top-r singular values. in contrast SST, which shows much similar distribution as full-rank training.
27


Reviews:
[{"review_id": "JkJd8i6HAp", "reviewer": "Reviewer_WF2f", "summary": "Training large language models faces challenges due to high GPU memory demands. Existing methods like LoRA and ReLoRA have limitations, such as being constrained by low - rank structure or suffering from saddle point issues. This paper proposes Sparse Spectral Training (SST). It updates all singular values, selectively updates singular vectors via multinomial sampling, and uses SVD for initialization and re - initialization to optimize memory usage during pre - training. SST outperforms existing memory - reduction training methods in various tasks across Euclidean and hyperbolic neural networks. It reduces the performance gap between low - rank methods and full - rank training, and is more memory - efficient, making it a promising technique for model pre - training.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and suggestions. We are grateful for the opportunity to clarify and improve our work.\n\n**Weakness 1:** There are not enough baselines and they are not particularly advanced. Consider introducing Dora, HIRA, Vera, FourierFT (at least Dora and Vera).\n\n**Reply:** Thank you for the suggestion. In the original manuscript, we did not include Dora and Vera as baselines because their parameter search spaces are restricted to the low-rank subspace. As discussed in Section 3.1 \"Limitation of LoRA,\" this constraint limits their expressiveness, while it may suffice for simple tasks like fine-tuning, it becomes a major bottleneck for more complex tasks such as training from scratch, which is the focus of our work. Since ReLoRA and GaLore are explicitly designed for pre-training, we prioritized comparing with them as the most relevant baselines.\n\nTo further demonstrate the limitations of Dora and Vera in pre-training, we conducted additional experiments comparing them to SST:\n\n**BLEU scores (higher is better) on IWSLT’14 with Transformer models:**\n\n| (d, r)    | Dora  | Vera  | LoRA  | ReLoRA* | SST     |\n|-----------|-------|-------|-------|---------|---------|\n| (64, 8)   | 18.19 | 9.22  | 18.08 | 18.12   | **22.28** |\n| (64, 4)   | 14.38 | 9.30  | 14.05 | 15.49   | **20.27** |\n| (128, 16) | 23.46 | 12.22 | 23.30 | 22.92   | **25.12** |\n| (128, 8)  | 21.09 | 12.25 | 20.56 | 20.61   | **24.19** |\n| (128, 4)  | 18.12 | 11.61 | 16.37 | 18.00   | **22.80** |\n\nThese results show that while Dora slightly improves over LoRA, it still falls short of SST. Vera performs poorly across all settings.\n\nDora decomposes the pre-trained weights into magnitude and direction components, but this does not overcome the fundamental limitation of the low-rank subspace. Vera further constrains the trainable parameters to a single vector, which significantly reduces memory but fails to capture the complexity required for effective pre-training.\n\nWe appreciate this valuable suggestion and have added this analysis and experiment to the revised manuscript.\n\n**Weakness 2:** I would like to discuss ... because the baselines are all PEFT methods).\n\n**Reply:** Thank you for raising this important point. We agree that training and fine-tuning are fundamentally different tasks and should be clearly distinguished. In this paper, our focus is specifically on **parameter-efficient pre-training**, rather than parameter-efficient fine-tuning (PEFT) or sparse training.\n\nThere are two main reasons we emphasize pre-training. First, **pre-training is considerably more challenging** than fine-tuning, as it starts from random initialization without any prior knowledge. The model must learn meaningful representations entirely from scratch, making it more sensitive to optimization stability and expressiveness constraints. Methods that work well in fine-tuning often fail to generalize to this more difficult setting.\n\nSecond, **pre-training requires significantly more computational and memory resources**, which has become a major bottleneck. Many academic labs today lack the resources to conduct large-scale pre-training. SST is motivated by the desire to lower the resource barrier, making pre-training more accessible to the broader research community.\n\nWe appreciate the reviewer’s suggestion and have revised our manuscript to explicitly clarify that our work focuses on **parameter-efficient pre-training**. We have also updated the wording from \"Previous memory reduction training techniques\" to \"Previous parameter-efficient training techniques\", since LoRA is a PEFT method and ReLoRA is designed for pre-training. We hope these changes enhance the clarity of our contributions.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "Yes.", "essential_references_not_discussed": "Yes.", "other_strengths_and_weaknesses": "**Strengths:**\n\n1. The paper is well written. Although I'm not that familiar with the hyperbolic concepts, I can roughly take this paper.\n\n2. The proposed SST method is interesting, and the novelty of the method is sufficient, as it is one of the few studies on efficient LLM in hyperbolic space.\n\n**Weaknesses:**\n\n1. There are not enough baselines and they are not particularly advanced. Consider introducing Dora, HIRA, Vera, FourierFT (at least Dora and Vera).\n\n2. I would like to discuss issues related to training and fine-tuning with the authors. The paper does not actually distinguish between the two and describes a number of PEFT methods as \"Previous memory reduction training techniques\", but I personally have doubts about this. At least formally speaking, efficient training (such as pruning technology) and PEFT (such as LoRA) are two machine learning tasks. The former focuses on the efficiency of training and reasoning, while the latter focuses on the number of parameters to be fine-tuned (and people have recently begun to pay attention to issues such as memory and time in PEFT). Therefore, as a paper, if there is no additional explanation, the two tasks should be clearly distinguished. This is for the development of the entire field of efficient AI from a macro perspective, and for the readers to better understand your paper from a micro perspective. In addition, I think training and fine-tuning are intuitively different. The biggest difference is that fine-tuning is based on a certain amount of knowledge, and the common concept is that good fine-tuning will retain and amplify useful knowledge and effectively learn new knowledge. Of course, without comprehensive research, I can only have this intuitive feeling. But in any case, I think the two should be distinguished before the author has a more comprehensive study and discussion. If the authors are willing to change the writing, I would suggest changing the training rhetoric to fine-tuning as much as possible (this seems to be the easiest, because the baselines are all PEFT methods).\n\n3. So on the weakness 2, the author can be open to refute me using some reference facts and logic, cause it is somewhat of an open question.", "other_comments_or_suggestions": "N.A.", "questions_for_authors": "N.A.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Am1hApfLqG", "reviewer": "Reviewer_C46V", "summary": "Sparse Spectral Training (SST) is a memory-efficient pre-training method for large language models. It updates all singular values, selectively updates singular vectors and achieves performance comparable to full-rank training. It first incorporates parameter-efficient pre-training process in hyperbolic space, demonstrating its superior performance and generalizability across various data structures and models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and valuable insights.\n\n**Weakness 1:** Result analysis should provide insights into future work and underlying reasons for the experimental results in the performance tables, not only provide numerical values already shown in the table.\n\n**Reply:** Thank you for this valuable feedback. We agree that the original version of our paper focused too heavily on numerical results and lacked sufficient analysis. In the revised version, we have incorporated more discussion on the underlying causes of the observed performance differences, and we highlight two key examples below:\n\nFirst, in our ablation study on **sampling mechanisms** (Appendix E.2), we examined why multinomial sampling outperforms other strategies such as uniform, sequential, and top-$r$ selection. As shown in Table 17, top-$r$ performs the worst because it repeatedly updates only a fixed subset of singular vectors, restricting the search space to a narrow low-rank subspace. In contrast, stochastic strategies like multinomial sampling encourage exploration across a broader spectral domain, leading to improved generalization and convergence. This observation suggests future work could explore adaptive or learnable sampling strategies to further enhance training dynamics.\n\nSecond, in our extended comparison of **SST and GaLore** (Appendix G), we analyzed why SST consistently outperforms GaLore in low-rank regimes. GaLore relies on a low-rank projection of the gradient derived from a single batch, which can be noisy and unstable due to sampling variability. SST, on the other hand, uses the SVD of the weight matrix $\\Vec{W}$, a more stable and aggregated representation over training steps, and updates all singular values at every iteration. This makes SST more robust, especially at small ranks. These findings also suggest a potential future direction in combining SST’s spectral stability with GaLore’s gradient-based projection approach.\n\nWe have incorporated these analyses into the main text in the revised version, utilizing the additional page allowed for the final version.\n\n\n**Weakness 2:** Detailed analysis is needed for rank choice.\n\n**Reply:** Thank you for raising this important point. We agree that a more in-depth discussion of rank is valuable. In our study, the rank is not treated as a tunable hyperparameter but rather as a practical constraint determined by available GPU memory. While increasing the rank generally improves performance, it also significantly increases memory consumption. To ensure fair comparisons, we used the same rank across LoRA, ReLoRA*, and SST in all experiments.\n\nTo offer further insight, we report BLEU scores on the IWSLT'14 dataset using a Transformer with model dimension 128:\n\n| Rank ($r$)         | 1     | 2     | 4     | 8     | 16    | 32    | 64    |\n|--------------------|-------|-------|-------|-------|-------|--------|--------|\n| **LoRA**           | 12.44 | 14.16 | 16.37 | 20.56 | 23.30 | 25.12 | 26.11 |\n| **ReLoRA**         | 14.53 | 15.39 | 18.00 | 20.61 | 22.92 | 24.15 | 25.25 |\n| **SST**            | 17.49 | 20.69 | 22.80 | 24.19 | 25.12 | 26.08 | 26.15 |\n\nThese results show that **SST consistently outperforms both LoRA and ReLoRA across all rank settings**, particularly in the low-rank regime. This demonstrates SST’s robustness and practical utility for memory-efficient training under constrained compute budgets.\n\n**Weakness 3:** Experiments on Hyperbolic graph neural networks are limited. Comparison between hyperbolic transformer and graph neural networks needs furtheranalysis.\n\n**Reply:** Thank you for this suggestion. We acknowledge that our current experiments on hyperbolic graph neural networks and hyperbolic transformers are limited in scope. However, the results presented already highlight the advantages of SST in this setting.\n\nSST consistently outperforms other low-rank methods across all tested configurations of hyperbolic models (with the exception of node classification on Airport with $r=2$), and in several cases, especially for the hyperbolic Transformer, it even surpasses full-rank training. As shown in Table 1, baseline methods like LoRA and ReLoRA suffer from training instability (e.g., NaN losses), whereas SST maintains stable convergence throughout.\n\nGeometrically, hyperbolic space exhibits exponential volume growth with distance, unlike the polynomial growth of Euclidean space. This makes hyperbolic models highly expressive but also prone to overfitting, particularly in high dimensions. By selectively updating a small subset of singular vectors, SST constrains the parameter search space, helping to regularize the model and prevent overfitting.\n\nWe appreciate the reviewer’s suggestion and are actively running additional experiments to further evaluate SST on other hyperbolic architectures. We will share updated results as soon as they are available.", "claims_and_evidence": "Experiments are conducted for large language models with Euclidean and hyperbolic backbone networks respectively. Performances of SST under the evaluation of perplexity, BLEU score, F1 score, and accuracy are comparable with full-rank training. Besides, memory consumption and training time analysis is also included here.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria make sense for the problem of training large language models and inference efficiently.", "theoretical_claims": "Yes. they are based on well-established mathematical principles such as singular value decomposition and gradient descent.", "experimental_designs_or_analyses": "Result analysis should provide insights into future work and underlying reasons for the experimental results in the performance tables, not only provide numerical values already shown in the table.\n\nDetailed analysis is needed for rank choice.\n\nExperiments on Hyperbolic graph neural networks are limited. Comparison between hyperbolic transformer and graph neural networks needs furtheranalysis.", "relation_to_broader_scientific_literature": "Sparse Spectral Training (SST), are related to the broader scientific literature on parameter-efficient training methods and hyperbolic neural networks.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "No.", "other_comments_or_suggestions": "No.", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D7AL3Nb3j1", "reviewer": "Reviewer_bJFt", "summary": "The authors introduce sparse spectral training (SST), a memory-efficient way to update neural network parameters. Unlike low-rank methods, SST begins by computing the singular-value decomposition of a weight matrix. At each step, SST updates all singular values and a sample of singular vectors (weighted by singular values). Because these updates can violate orthogonality, SST occasionally reinitializes the low rank parameters as $\\mathbf{U, \\Sigma, V^T} \\gets \\operatorname{SVD}(\\mathbf{U\\Sigma V^T})$. On a variety of benchmarks spanning natural language processing, graph data science, and image classification, SST consistently outperforms low-rank methods; for hyperbolic methods, it often even beats the full-rank models.\n\n## Update after rebuttal\nI was satisfied with my discussion with the authors and am continuing to recommend acceptance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and suggestions.\n\n**W1:** There is another work ...\n\n**Reply:** Thank you for pointing out this recent work. We have included it in the related work section of our revised manuscript.\n\n**W2:** I am interested in ...\n\n**Reply:** Low-precision and mixed-precision training are standard techniques to accelerate LLM pre-training while reducing memory usage. In our experiments:\n\n- For LLaMA pre-training, we used bfloat16 precision, which offers a wide dynamic range and is well-suited for stable large-scale training.\n- For OPT pre-training, we used a mixed-precision setup, combining bf16 and TF32 to leverage fast matrix operations on modern hardware while preserving numerical stability.\n\nAlthough low-precision training greatly improves efficiency, it can increase sensitivity to instability. We observed that LoRA and GaLore occasionally experienced training instability, which may be partially due to interactions between low-rank updates and reduced precision.\n\n**W3:**  Comparison to GaLore ...\n\n**Reply:** Thank you for the suggestion. In the revised manuscript, we have moved the relevant results and analysis from the appendix into the main body.\n\n**W4:**  Other Comments Or Suggestions\n\n**Reply:** Due to character limit, couldn't address this part in detail. We have carefully revised the manuscript accordingly. Thank you again for your constructive feedback.\n\n**W5:**  It seems to me ...\n\n**Reply:** The motivation behind the Enhanced Gradient is to decouple the updates of singular vectors and singular values, allowing singular vectors with smaller singular values to still receive meaningful updates. From another perspective, this effectively corresponds to using a larger learning rate for less dominant directions, similar in spirit to adaptive optimization methods like Adam, where updates are rescaled by the inverse of the magnitude. Enhanced Gradient remains faithful to the model’s computation in the sense that when the gradient of W approaches zero, the Enhanced Gradients of U and V also approach zero. This ensures convergence behavior remains aligned with the model’s optimization objective.\n\n**W6:**  What happens with ...\n\n**Reply:** Due to the page limit, we had originally omitted the update formulas for $V^T$ in Equations 6–8, but now added them back.\n\nTo clarify, the indices for updating U and $V^T$ are shared. This is necessary because the gradient update of $U_i$ depends on $V_i$ and vice versa, which promotes faster convergence.\n\n**W7:**  In 4.3 ...\n\n**Reply:** Thank you for the insightful question. We conducted additional experiments where A and B in ReLoRA* are initialized with random values instead of zeros. The results (https://anonymous.4open.science/r/sparse_spectral_training-6A2C/relora_random.png) show clear improvement over the original ReLoRA*, but performance still falls short of SST.\n\nOne possible reason is that SST uses singular vectors of the weight matrix W , which may better align with the update directions. In contrast, randomly initialized vectors in high-dimensional space are typically close to orthogonal and may not capture meaningful structure. We're also testing random-initialized ReLoRA* on LLaMA and will update results as they become available.\n\n\n**W8:**  In L323--325... \n\n**Reply:** Thank you for the question. The key difference from standard fine-tuning setups is that LoRA typically replaces only the query and value projection layers during fine-tuning, while freezing the rest of the model. In contrast, for **pre-training**, all linear layers, including Q, K, V, O projections and FFN, must be trainable. Therefore, to apply low-rank methods like LoRA, ReLoRA*, or SST to pre-training fairly, we replace **all `torch.nn.Linear` layers** of original model with their corresponding low-rank implementations to maintain parameter efficiency across the entire model.  \n\nWe confirm that the **\"Full\" benchmarks are not affected** by this replacement; they use standard full-rank training with all linear layers unchanged.\n\n**W9:** What's going on...\n\n**Reply:** Thank you for pointing this out. Table 5 was originally intended as part of the Appendix but was referenced in the main text, so it wasn’t placed under any specific appendix section. We have revised the manuscript and moved it into the main body for clarity and consistency.\n\n**W10:** Why did you feel...\n\n**Reply:** Thank you for the question. Hyperbolic models often suffer from overfitting and instability in high-dimensional settings due to the exponential growth of space. Prior work has typically restricted them to low-dimensional use cases for this reason. We included hyperbolic models to show that SST, by limiting the parameter search space, mitigates overfitting and improves stability. Compared to both full-rank and other low-rank methods, SST is more robust and consistently delivers better performance in high-dimensional hyperbolic setups.", "claims_and_evidence": "All of the authors' claims are well-supported by the paper.", "methods_and_evaluation_criteria": "I am not an expert in memory-efficient/low-rank training methods and what typical benchmarks for them are, but the experiments make sense to me. I appreciate that the authors cover a wide range of NLP tasks and even evaluate zero-shot performance. Their approach to hyperbolic networks on graphs is basically canonical, though it is surprising that the authors include hyperbolic models in their evaluations at all.", "theoretical_claims": "To the best of my knowledge, the only proof in this paper is in Appendix D (gradient of sparse spectral layer), which I read. It appears correct to me, and I do not feel that the authors make any other theoretical claims which need further justification.", "experimental_designs_or_analyses": "The experiments as described in the main body of the paper appear sound. I read the descriptions of additional experiments in Appendices G, H, and J, and found them to be adequate as well.", "relation_to_broader_scientific_literature": "The authors lump together a number of memory-efficient training approaches into their \"ReLORA*\" generalization, demonstrate intuitively and empirically that all of these approaches suffer from saddle point issues which compromise their efficiency, and propose SST as an alternative. In Appendix G, they also compare to GaLore. They demonstrate a substantial improvement over existing methods, and also expand their", "essential_references_not_discussed": "In general, the references seem reasonably complete. There is another work at the intersection of hyperbolic geometry and LLM fine-tuning [1] that may be worth discussing, though direct comparison to their approach could be difficult.\n\n[1] Yang et al, 2024. Hyperbolic Fine-tuning for Large Language Models. https://arxiv.org/abs/2410.04010", "other_strengths_and_weaknesses": "**Strengths:**\n* The authors craft an excellent narrative, providing elegant justifications for their way of thinking. I particularly liked their explanations for the following topics:\n  * Their explanation of saddle point issues in ReLORA*, combined with the immediate empirical evidence in Figure 2.\n  * In Section 4.4, they give an intuitive exploration-exploitation perspective on low-rank methods that feels quite obvious once it is stated. I have \n  * Why hyperbolic geometry is prone to overfitting, especially in high dimensions, and why SST can remedy this issue\n  * Introducing the effective step metrics to motivate their analysis in Figure 4\n* Strong performance on all benchmarks, including those in the Appendix\n* Extension to non-Euclidean geometry, where their method really shines, outperforming full-rank methods\n\n**Weaknesses:**\n* Some more pragmatic considerations could be included. In particular, I am interested in how this method interacts with quantization and how it scales to larger models.\n* I feel that the comparison to GaLore should be incorporated into the main body of the paper\n* I have concerns about the enhanced gradients (see questions for the authors)", "other_comments_or_suggestions": "* Throughout the introduction, the authors use \\citep for in-text references, where they should be using \\citet:\n  * L084: reference to Hu et al\n  * L092: Reference to Lialin et al\n  * L097: Zhao et al\n  * L098: Meng et al\n* Table 5: It would be better to bold the best method for each specific task, like in the other tables\n* Tables 4 and 5 should probably be reversed in order\n* ReLoRA* should be benchmarked in Table 4", "questions_for_authors": "* It seems to me that the Enhanced Gradients in Section 4.2 are not actually the partial derivatives of the loss with respect to each of these parameters. While this may work in practice, it seems like a misnomer to refer to them as gradients. Can you please either:\n  * Demonstrate to me why these \"enhanced gradients\" are still faithful to the computation performed by the model, or\n  * Perform ablations of these enhanced gradients to demonstrate their empirical usefulness\n* What happens with $\\mathbf{V^T}$ in Section 4.2? I assume it is updated analogously to $\\mathbf{U}$, via Equations 5-7 and using the enhanced gradients in Equation 9. If this is true, please edit the paper to reflect this. Please also be clear about whether the indices for updating $\\mathbf{V^T}$ and $\\mathbf{U}$ are shared or drawn independently. \n* In 4.3, you explain that saddle point issues are due to the initialization $\\mathbf{B}=0$. It occurs to me that initializing $\\mathbf{B}$ differently (i.e. to a nonzero matrix) could help avoid this issue, reducing the need for approaches like SST. What happens if you attempt this more trivial solution, and why is SST still a better option?\n* In L323--325, you say \"For all low-rank methods, all linear layers in the baseline models were replaced by low-rank layers.\" Can you elaborate on what exactly this means? It is not explained in Appendix E, and appears like a non-standard way of evaluating methods like LoRA.\n  * Additionally, please confirm that the \"Full\" benchmarks in Tables 1--4 are not affected by this change.\n* What's going on with the placement of Table 5? Is it supposed to be part of the Appendix, or is it a float from the main body of the paper that has crawled over the page limit?\n* Why did you feel it was important to include hyperbolic models throughout your benchmarks?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces Sparse Spectral Training (SST), a memory-efficient pretraining method that updates all singular values while selectively updating singular vectors. It addresses the limitations of low-rank approaches (e.g., LoRA, ReLoRA) in the context of large-scale pretraining and shows strong empirical results across a diverse set of tasks, including hyperbolic models where it even outperforms full-rank baselines.

The reviewers were overall positive, highlighting the novelty of the method, strong performance, and clear writing. Concerns around missing baselines and clarity on training vs. fine-tuning were adequately addressed in the rebuttal, with new experiments and clarification added to the revision. Given the novelty, practical relevance, and solid evaluation, I agree with the positive consensus and recommend acceptance.
Decision: Accept (poster)