Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks

## Abstract

Abstract The growing computational demands posed by increasingly number of neural network’s parameters necessitate low-memory-consumption training approaches. Previous memory reduction techniques, such as Low-Rank Adaptation (LoRA) and ReLoRA, suffer from the limitation of low rank and saddle point issues, particularly during intensive tasks like pre-training. In this paper, we propose Sparse Spectral Training (SST), an advanced training methodology that updates all singular values and selectively updates singular vectors of network weights, thereby optimizing resource usage while closely approximating full-rank training. SST refines the training process by employing a targeted updating strategy for singular vectors, which is determined by a multinomial sampling method weighted by the significance of the singular values, ensuring both high performance and memory reduction. Through comprehensive testing on both Euclidean and hyperbolic neural networks across various tasks, including natural language generation, machine translation, node classification and link prediction, SST demonstrates its capability to outperform existing memory reduction training methods and is comparable with full-rank training in some cases. On OPT-125M, with rank equating to 8.3% of embedding dimension, SST reduces the perplexity gap to full-rank training by 67.6%, demonstrating a significant reduction of the performance loss with prevalent low-rank methods. This approach offers a strong alternative to traditional training techniques, paving the way for more efficient and scalable neural network training solutions.

## 1 Introduction

The development and scaling up of the size of large language models [, , ] pose great challenges to the feasibility of training large language models from scratch. Normal training methods that update all parameters of models become extremely expensive due to their extensive memory requirements.

Recent developments in parameter-efficient fine-tuning (PEFT) methods, such as Low-Rank Adaptation (LoRA) [], have sought to mitigate the challenge of fine-tuning memory requirements by introducing trainable low-rank matrices that efficiently reduced memory footprint. However, the constraint of the predetermined rank can severely limit the ability of a model to capture and represent complex data patterns, leading to suboptimal performance, especially in the pre-training stages. The recent improvements of ReLoRA [] and Chain of LoRA [] break the limitation of low-dimension search space. However, they will still suffer from saddle point issues. Saddle points are locations where the gradient is zero but are not true minima, potentially leading to slower and less effective convergence compared to full-rank models during pre-training.

In response to these challenges, we introduce Sparse Spectral Training (SST), a new training framework designed to optimize memory consumption while closely approximating the overall learning dynamics and performance of full-rank training. Unlike previous methods [, , , ] that primarily focus on updating only a partial number of parameters, SST adopts a more effective approach by updating all singular values. SST also capitalizes the intrinsic spectral properties of the weight matrices, focusing updates on components that are most influential to the model’s learning process based on their singular values. Additionally, SST proposes to use singular value decomposition to initialize low-rank parameters, minimizing distortion compared to full-rank training.

Our comprehensive evaluations across different tasks including pre-training large language models on OPT model family from 125m to 1.3b [], Transformer [] on machine translation tasks and hyperbolic graph neural networks [, ] on node classification and link prediction tasks. The empirical performance shows that with rank equals to 6.25% of model dimension, SST outperforms full-rank training on machine translation tasks and obtains SOTA performance among prevalent parameter-efficient training methods. Furthermore, we are the first to embed the parameter-efficient training process on hyperbolic space, which proves that SST is a general technique applicable across various data structures and models, effectively enhancing the adaptability and scalability of neural network training in resource-constrained environments.

## 2 Related Work

#### Low-Rank Adaptation

Low-rank adaptation has become a key strategy for reducing the computational and memory requirements of training large-scale neural networks. Hu et al. [] introduced Low-Rank Adaptation (LoRA), a technique that fine-tunes pre-trained models by integrating low-rank matrices to significantly reduce the number of parameters updated during training. Various enhancements to LoRA have since been developed to improve its efficiency and broaden its application [, , , ]. Lialin et al. [] introduced ReLoRA specifically for the pre-training phase, which requires a full-rank warm-up to achieve similar performance with full-rank training. A similar approach is found in COLA []. Additionally, Zhao et al. [] introduced GaLore, which project gradient to low-rank subspace. These advancements highlight the versatility and ongoing evolution of low-rank adaptation techniques in response to the growing complexity of neural network models.

#### Other Parameter-Efficient Training Methods

Apart from low-rank adaptations, researchers have developed a variety of parameter-efficient training techniques to optimize resource consumption while preserving learning effectiveness. Prompt tuning is an effective method that integrates tunable prefixes or soft prompts into the input embeddings of models. It enables lightweight task-specific adaptations with minimal impact on the model’s overall architecture [, ]. Dynamic sparse training (DST), through methods like SET [], RIGL [], MEST [], and CHT [], employs a dynamic prune-and-grow strategy that adjusts network topology during training. This approach optimizes training efficiency and can improve generalization by continuously adapting the network’s sparse structure. This presents a significant shift from static training methods.

#### Hyperbolic Neural Networks

Hyperbolic neural networks are an emerging field in deep learning, exploiting the unique properties of hyperbolic space that make it ideal for processing hierarchical and graph-structured data [, ]. Innovations in this area have adapted fundamental neural network mechanisms to function within hyperbolic geometries, as demonstrated by Muscoloni et al. [] and Ganea et al. []. Further developments by Chen et al. [] explore manifold-specific properties to enrich both theoretical understanding and practical deployment. The employment of hyperbolic spaces has been shown to significantly improve data representation and generalization across various tasks, marking a notable advancement in managing complex, non-Euclidean data structures [, , ].

## 3 Low Rank Adaptation

This section introduces the fundamentals and limitations of Low-Rank Adaptation (LoRA) [] and ReLoRA []. These limitations are addressed by Sparse Spectral Training (SST) in Section 4.

### 3.1 LoRA

LoRA [] fine-tunes a pre-trained model by learning an incremental update $\Delta\vec{W}$ to the pre-trained and frozen weight matrix $\vec{W}_{0}$. Here $\vec{W}_{0},\Delta\vec{W}\in\mathbb{R}^{m\times n}$ with $m\leq n$. It decomposes $\Delta\vec{W}$ into the product of two low-rank matrices, $\vec{B}\in\mathbb{R}^{m\times r}$ and $\vec{A}\in\mathbb{R}^{r\times n}$, such that $\Delta\vec{W}=\vec{B}\vec{A}$. This decomposition is applied in a linear layer $\mathbf{h}$ with input $\mathbf{x}$ as follows:

$$ $\mathbf{h}=(\vec{W}_{0}+\Delta\vec{W})\mathbf{x}=(\vec{W}_{0}+\vec{B}\vec{A}) \mathbf{x}$ (1) $$

Given $r\ll min(m,n)$, LoRA significantly reduces GPU memory usage compared to full-rank fine-tuning.

### 3.2 Limitation of LoRA

Consider $\vec{W}^{*}$ as the optimal weight matrix minimizing loss. The deviation from the current weights is $\Delta\vec{W}^{*}=\vec{W}^{*}-\vec{W}_{0}$. Performing a singular value decomposition on $\Delta\vec{W}^{*}$ yields $\Delta\vec{W}^{*}=\vec{U}\vec{\Sigma}\vec{V}^{\mathrm{T}}$, where $\vec{U}\in\mathbb{R}^{m\times m}$, $\vec{\Sigma}\in\mathbb{R}^{m\times m}$, $\vec{V}^{\mathrm{T}}\in\mathbb{R}^{m\times n}$.

$\vec{U}$ and $\vec{V}^{\mathrm{T}}$ as orthonormal bases, $\vec{U}=[\vec{u}_{1},\vec{u}_{2},...,\vec{u}_{m}]$, $\vec{V}=[\vec{v}_{1},\vec{v}_{2},...,\vec{v}_{m}]$. And $\vec{\Sigma}$ is a diagonal matrix with entries $\{\sigma_{1},\sigma_{2},...,\sigma_{m}\}$. Then the Eckart–Young–Mirsky theorem [] states:

$$ $\|\Delta\vec{W}^{*}-\Delta\vec{W}\|_{\text{F}}\geq\sqrt{\sigma_{r+1}^{2}+ \cdots+\sigma_{m}^{2}}$ (2) $$

where $\|\mathbf{W}\|_{\text{F}}=\sqrt{\sum_{i=1}^{m}\sum_{j=1}^{n}w_{ij}^{2}}$ is the Frobenius norm, with $w_{ij}$ being the element at row $i$ and column $j$ of $\vec{W}$. Equality holds when $\vec{B}=[\sqrt{\sigma_{1}}\vec{u}_{1},\sqrt{\sigma_{2}}\vec{u}_{2},...,\sqrt{
\sigma_{r}}\vec{u}_{r}]$ and $\vec{A}^{\mathrm{T}}=[\sqrt{\sigma_{1}}\vec{v}_{1},\sqrt{\sigma_{2}}\vec{v}_{2
},...,\sqrt{\sigma_{r}}\vec{v}_{r}]$. This suggests that LoRA can approach the performance of full-rank training for simple tasks like fine-tuning, where $\sigma_{i}\approx 0,i\in\{r+1,...,m\}$. However, LoRA may struggle in more complex scenarios like pre-training due to insufficient exploration of the weight space.

### 3.3 ReLoRA*

A straightforward idea to solve the limitation of fixed low ranks is to iteratively merge the low rank matrices $\mathbf{B}$ and $\vec{A}$ back into the base weight matrix $\vec{W}_{0}$. This process, formalized as Algorithm 1, is termed ReLoRA* to differentiate it from ReLoRA.

Figure: Algorithm 1 ReLoRA*

This improvement theoretically permits LoRA to transcend the limitations of a predetermined rank $r$. ReLoRA [] and COLA [] represent specific implementations of this strategy, where they employ LoRA’s initialization techniques—$\vec{B}$ initialized to zero and $\vec{A}$ with a Gaussian distribution []. The initial zero setting for $\vec{B}$ allows the subtracting step to be skipped. ReLoRA* thus serves as an end-to-end memory-efficient methodology, differing from ReLoRA, which incorporates a period of full-rank training initially. Notably, the optimizer states for $\vec{B}$ and $\vec{A}$ are reset after merging step ($99\%$ optimizer state is pruned in ReLoRA).

However, each iteration of ReLoRA* learns only a small subset of singular values. Additionally, its reliance on random initialization can lead to stucking at saddle points, as discussed in Section 4.3. These issues hinder ReLoRA* from achieving the convergence speed and training quality of full-rank training.

## 4 Sparse Spectral Training

To address the limitations discussed previously, this section introduces Sparse Spectral Training (SST) and its detailed implementation.

### 4.1 Preliminaries

Sparse Spectral Training (SST) leverages sparse updates within the spectral domain of neural network weights. By updating singular vectors selectively based on their associated singular values, SST prioritizes the most significant spectral components. This transforms each linear layer as follows:

$$ $\mathbf{h}=\vec{W}\mathbf{x}=\vec{U}\vec{\Sigma}\vec{V}^{\mathrm{T}}\mathbf{x} ,\quad[\vec{U},\vec{\Sigma},\vec{V}^{\mathrm{T}}]=\text{SVD}(\vec{W})$ (3) $$

where $\vec{U}\in\mathbb{R}^{m\times m}$, $\vec{\Sigma}\in\mathbb{R}^{m\times m}$, $\vec{V}^{\mathrm{T}}\in\mathbb{R}^{m\times n}$ represent the full-rank matrices derived from the singular value decomposition (SVD) of $\vec{W}\in\mathbb{R}^{m\times n}$, assuming $m\leq n$. It is important to note that unlike other LoRA-based methods, $\vec{U},\vec{\Sigma},\vec{V}^{\mathrm{T}}$ in this context are utilized at full rank.

Figure: Figure 1: Illustration of the Sparse Spectral Training (SST). For each iteration, all singular values and selected singular vectors are updated based on their significance, determined by a multinomial sampling using singular values as probabilities.
Refer to caption: x1.png

### 4.2 Gradient Update of U → → 𝑈 \vec{U} over→ start_ARG italic_U end_ARG , V → T superscript → 𝑉 T \vec{V}^{\mathrm{T}} over→ start_ARG italic_V end_ARG start_POSTSUPERSCRIPT roman_T end_POSTSUPERSCRIPT with Σ → → Σ \vec{\Sigma} over→ start_ARG roman_Σ end_ARG

#### Update Σ → → Σ \vec{\Sigma} over→ start_ARG roman_Σ end_ARG .

The diagonal matrix $\vec{\Sigma}$, simplified as a vector of dimension $m$, is updated every step due to its low memory overhead. This ensures that all singular values are consistently adjusted to refine the model’s performance. The update is as follows:

$$ $\vec{\Sigma}^{t+1}=\max(\vec{\Sigma}^{t}-\eta\nabla\mathcal{L}_{\vec{\Sigma}},0)$ (4) $$

where $\eta$ represents the learning rate, and $\nabla\mathcal{L}_{\vec{\Sigma}}$ is the gradient backpropagated to $\vec{\Sigma}$. The $\max$ function with zero ensures that $\vec{\Sigma}$ values remain non-negative.

#### Update U → → 𝑈 \vec{U} over→ start_ARG italic_U end_ARG and V → T superscript → 𝑉 T \vec{V}^{\mathrm{T}} over→ start_ARG italic_V end_ARG start_POSTSUPERSCRIPT roman_T end_POSTSUPERSCRIPT .

To update $\vec{U}$ and $\vec{V}^{\mathrm{T}}$, a cyclic updating strategy is employed, where specific parameters are chosen for each iteration based on a multinomial sampling method, as depicted in Figure 1. Consider $I=\{1,2,...,m\}$ as the set of all indices in $\vec{U}$ and $\vec{V}^{\mathrm{T}}$, with the sampling process defined by:

$$ $S\subseteq I,\quad S\sim\text{Multinomial}(r,\vec{\Sigma})$ (5) $$

Here, $S$ represents the selected indices for update, with $|S|=r$, where $r$ is the predetermined number of vectors to be updated in each cycle. The update formulas for $\vec{U}$ and $\vec{V}^{\mathrm{T}}$ are:

$$ $\vec{U}^{t+1}_{\cdot i}=\vec{U}^{t}_{\cdot i}-\eta\nabla\mathcal{L}_{\vec{U}_{ \cdot i}},\quad\vec{V}^{t+1}_{\cdot i}=\vec{V}^{t}_{\cdot i}-\eta\nabla \mathcal{L}_{\vec{V}_{\cdot i}},\quad\text{if }i\in S$ (6) $$

where $\vec{U}_{\cdot i}$ means column $i$ vector of $\vec{U}$. To maintain unit norm of each vector during training, and to ensure that magnitude information is encapsulated solely by $\vec{\Sigma}$, the vectors are normalized post-update as follows:

$$ $\vec{U}^{t+1}_{\cdot i}=\frac{\vec{U}^{t}_{\cdot i}-\eta\nabla\mathcal{L}_{ \vec{U}_{\cdot i}}}{|\vec{U}^{t}_{\cdot i}-\eta\nabla\mathcal{L}_{\vec{U}_{ \cdot i}}|},\quad\vec{V}^{t+1}_{\cdot i}=\frac{\vec{V}^{t}_{\cdot i}-\eta \nabla\mathcal{L}_{\vec{V}_{\cdot i}}}{|\vec{V}^{t}_{\cdot i}-\eta\nabla \mathcal{L}_{\vec{V}_{\cdot i}}|},\quad\text{if }i\in S$ (7) $$

#### Enhanced gradient of U → → 𝑈 \vec{U} over→ start_ARG italic_U end_ARG and V → T superscript → 𝑉 T \vec{V}^{\mathrm{T}} over→ start_ARG italic_V end_ARG start_POSTSUPERSCRIPT roman_T end_POSTSUPERSCRIPT .

Within a sparse spectral layer where $\mathbf{h}=\vec{U}\vec{\Sigma}\vec{V}^{\mathrm{T}}\mathbf{x}$, the matrix $\vec{W}=\vec{U}\vec{\Sigma}\vec{V}^{\mathrm{T}}$ serves as the product of $\vec{U}$, $\vec{\Sigma}$, and $\vec{V}^{\mathrm{T}}$. The gradients for $\vec{U}$ and $\vec{V}^{\mathrm{T}}$ are detailed below (derivation included in Appendix B):

$$ $\nabla\mathcal{L}_{\vec{U}_{\cdot i}}=\frac{\partial\mathcal{L}}{\partial\vec{ U}_{\cdot i}}=\frac{\partial\mathcal{L}}{\partial\vec{W}}\vec{V}_{\cdot i}\vec {\Sigma}_{i},\quad\nabla\mathcal{L}_{{\vec{V}_{\cdot i}}}=\frac{\partial \mathcal{L}}{\partial{\vec{V}_{\cdot i}}}=\vec{\Sigma}_{i}\frac{\partial \mathcal{L}}{\partial\vec{W}^{\mathrm{T}}}{\vec{U}_{\cdot i}}$ (8) $$

where $\vec{U}_{\cdot i}$ and $\vec{V}_{\cdot i}$ are column vectors of $\vec{U}$ and $\vec{V}^{\mathrm{T}}$, respectively, and $\vec{\Sigma}_{i}$ represents the diagonal elements of $\vec{\Sigma}$. This represents the default gradient calculation for these matrices. We propose an enhanced gradient calculation for $\vec{U}_{\cdot i}$ and ${\vec{V}_{\cdot i}}$ as follows:

$$ $\tilde{\nabla}\mathcal{L}_{\vec{U}_{\cdot i}}=\frac{\partial\mathcal{L}}{ \partial\vec{W}}\vec{V}_{\cdot i},\quad\tilde{\nabla}\mathcal{L}_{{\vec{V}_{ \cdot i}}}=\frac{\partial\mathcal{L}}{\partial\vec{W}^{\mathrm{T}}}{\vec{U}_{ \cdot i}}$ (9) $$

###### Theorem 4.1 (Decomposition of $\frac{\partial\mathcal{L}}{\partial\vec{W}}$ ) .

Suupose $\mathbf{W}$ is initialized to zero ($\mathbf{W}_{0}=\mathbf{0}$) and there exists an optimal weight $\mathbf{W}^{*}$ that directs the expectation of $\frac{\partial\mathcal{L}}{\partial\vec{W}}$ from the current $\mathbf{W}$ towards the optimal $\mathbf{W}^{*}$. Under these conditions, the expected gradient can be expressed as:

$$ $\mathbb{E}[\frac{\partial\mathcal{L}}{\partial\vec{W}}]=\vec{U}\vec{D}\vec{V}^ {\mathrm{T}}$ (10) $$

where $\mathbb{E}$ is the expectation operator, $\vec{D}$ is a diagonal matrix and $[\vec{U},\vec{\Sigma},\vec{V}^{\mathrm{T}}]=\text{SVD}(\vec{W})$ are derived from the singular value decomposition (SVD) of $\vec{W}$.

###### Theorem 4.2 (Advantage of Enhanced Gradient over Default Gradient) .

Suppose the gradient of $\vec{W}$ conforms to the decomposition $\frac{\partial\mathcal{L}}{\partial\vec{W}}=\vec{U}\vec{D}\vec{V}^{\mathrm{T}}$, as stated in Theorem 4.1. Then:

$$ $\|3\Delta\mathbf{W}_{\text{full}}-\Delta\mathbf{W}_{\text{enhance}}\|_{F}\leq \|3\Delta\mathbf{W}_{\text{full}}-\Delta\mathbf{W}_{\text{default}}\|_{F}$ (11) $$

where $\Delta\mathbf{W}_{\text{full}}$ is the update of $\vec{W}$ from full-rank training, and $\Delta\mathbf{W}_{\text{enhanced}}$ and $\Delta\mathbf{W}_{\text{default}}$ denotes the update of $\vec{W}$ in SST with enhanced and default gradient respectively.

This theorem indicates that the enhanced gradient more closely approximates full-rank updates per iteration compared to the standard approach. Specifically, the default gradient’s dependence on $\vec{\Sigma}_{i}$ for magnitude could result in smaller updates if the current $\vec{\Sigma}_{i}$ is low, potentially stalling training. By decoupling the update mechanisms for direction and magnitude, the enhanced gradient method mitigates this issue.

Theorem 4.2 indicates that the enhanced gradient more closely approximates full-rank training in each step update, than the default gradient (proof in Appendix D). Specifically, the default gradient’s dependence on $\vec{\Sigma}_{i}$ could result in smaller updates if the current $\vec{\Sigma}_{i}$ is low, potentially stalling training. By decoupling the update mechanisms for direction ($\vec{U}_{\cdot i}$ and ${\vec{V}_{\cdot i}}$) and magnitude ($\vec{\Sigma}_{i}$), the enhanced gradient method mitigates this issue:

$$ $\vec{U}^{t+1}_{\cdot i}=\frac{\vec{U}^{t}_{\cdot i}-\eta\tilde{\nabla}\mathcal {L}_{\vec{U}_{\cdot i}}}{|\vec{U}^{t}_{\cdot i}-\eta\tilde{\nabla}\mathcal{L}_ {\vec{U}_{\cdot i}}|},\quad\vec{V}^{t+1}_{\cdot i}=\frac{\vec{V}^{t}_{\cdot i} -\eta\tilde{\nabla}\mathcal{L}_{\vec{V}_{\cdot i}}}{|\vec{V}^{t}_{\cdot i}- \eta\tilde{\nabla}\mathcal{L}_{\vec{V}_{\cdot i}}|},\quad\text{if }i\in S$ (12) $$

###### Theorem 4.1 (Decomposition of ∂ ℒ ∂ W → ℒ → 𝑊 \frac{\partial\mathcal{L}}{\partial\vec{W}} divide start_ARG ∂ caligraphic_L end_ARG start_ARG ∂ over→ start_ARG italic_W end_ARG end_ARG ) .

###### Theorem 4.2 (Advantage of Enhanced Gradient over Default Gradient) .

#### Periodic Re-SVD.

During the course of training, the orthogonality among the vectors of $\vec{U}$ and $\vec{V}^{\mathrm{T}}$ tends to diminish. Preserving the orthogonality of these singular vectors is crucial as it prevents the learning process from being restricted to a constrained low-rank subspace, thus preserving the model’s full expressive capabilities. To maintain this orthogonality, it is essential to periodically perform singular value decomposition:

$$ $[\vec{U}^{t+1},\vec{\Sigma}^{t+1},{\vec{V}^{t+1}}^{\mathrm{T}}]=\text{SVD}( \vec{U}^{t}\vec{\Sigma}^{t}{\vec{V}^{t}}^{\mathrm{T}})$ (13) $$

Each time we perform this Re-SVD, we consider it a new round. Each time we select vectors for updating, as described in Eq. 5, we call it a new iteration. The full method is detailed in Algorithm 2.

### 4.3 Why SVD Initialization is Important

This section outlines the advantages of using SVD initialization and periodic Re-SVD over zero initialization as employed in LoRA and ReLoRA methods.

#### Saddle Point Issues with Zero Initialization.

Using zero initialization, the gradient updates for matrices $\vec{A}$ and $\vec{B}$ can lead to stagnation at saddle points. The gradient of $\vec{A}$ and $\vec{B}$ in Eq. 1 is:

$$ $\frac{\partial\mathcal{L}}{\partial\mathbf{B}}=\frac{\partial\mathcal{L}}{ \partial\Delta\mathbf{W}}\mathbf{A}^{\mathrm{T}}\quad\text{and}\quad\frac{ \partial\mathcal{L}}{\partial\mathbf{A}}=\mathbf{B}^{\mathrm{T}}\frac{\partial \mathcal{L}}{\partial\Delta\mathbf{W}}$ (14) $$

In LoRA and ReLoRA, where $\vec{B}$ is initialized to zero, the gradient of $\vec{A}$ is calculated as $\frac{\partial\mathcal{L}}{\partial\mathbf{A}}=\mathbf{0}^{\mathrm{T}}\frac{
\partial\mathcal{L}}{\partial\Delta\mathbf{W}}=\mathbf{0}$ at the start of each iteration. Additionally, in ReLoRA*, resetting the momentum of $\vec{B}$ and $\vec{A}$ aggravates this issue, leading to slow learning progress and a tendency to get stuck at saddle points, particularly if the merging interval $T_{2}$ is short.

###### Theorem 4.3 (Zero Distortion with SVD Initialization) .

Suppose the gradient of $\vec{W}$ can be expressed as $\frac{\partial\mathcal{L}}{\partial\vec{W}}=\vec{U}\vec{D}\vec{V}^{\mathrm{T}}$, in accordance with Theorem 4.1. Then:

$$ $\|2\Delta\mathbf{W}_{\text{full}}-\Delta\mathbf{W}_{\text{LoRA}}\|_{F}\geq 0$ (15) $$

where $\Delta\mathbf{W}_{\text{full}}$ and $\Delta\mathbf{W}_{\text{LoRA}}$ represent the updates of $\vec{W}$ in full-rank and LoRA training, respectively. Equality is achieved when $\mathbf{A}=\vec{V}^{\mathrm{T}}$ and $\mathbf{B}=\vec{U}$.

###### Theorem 4.3 (Zero Distortion with SVD Initialization) .

#### Zero Distortion with SVD Initialization.

As demonstrated by Theorem 4.3 (proof in Appendix E), $\mathbf{A}=\vec{V}^{\mathrm{T}}$ and $\mathbf{B}=\vec{U}$ ensure that $\Delta\mathbf{W}_{\text{LoRA}}=2\Delta\mathbf{W}_{\text{full}}$. Consequently, reducing the learning rate in full-rank training by half results in identical updates between LoRA and full-rank training. This illustrates that SVD initialization effectively achieves zero distortion compared to full-rank training.

### 4.4 SST Balances Exploitation and Exploration

From another prospective, SST combines the strategies of exploitation and exploration in spectral domain. In contrast, LoRA primarily focuses on exploitation by repeatedly adjusting the top-$r$ singular values, as detailed in Section 3.2, neglecting the remaining spectral vectors. ReLoRA*, on the other hand, emphasizes exploration by periodically reinitializing the matrices $\mathbf{B}$ and $\vec{A}$ after each merging, thereby constantly seeking new directions for learning but ignoring previously established dominant directions.

SST boosts learning efficiency by updating all magnitudes ($\vec{\Sigma}$) at each step and cyclically revisiting previously dominant directions ($\vec{U}$ and $\vec{V}^{\mathrm{T}}$). By continuously updating all singular values, SST ensures unbiased sampling of $\vec{U}$ and $\vec{V}^{\mathrm{T}}$, enabling a thorough exploration of the parameter space. As a result, SST balances the exploitation of known critical directions with the exploration of emerging opportunities within the spectrum of matrix decomposition.

### 4.5 Memory-Efficient Implementation for SST

To achieve similar memory reduction as LoRA, SST stores optimizer states for all $\vec{\Sigma}$ and only for the vectors sampled in each iteration from $\vec{U}$ and $\vec{V}^{\mathrm{T}}$. However, standard implementations of Adam optimizer [] in PyTorch [] do not support sparse optimizer states. To address this, we partition $\vec{U}$ and $\vec{V}^{\mathrm{T}}$ into active and frozen segments. Only active segments store the optimizer states, where $\vec{U}_{\text{active}}\in\mathbb{R}^{m\times r}$ and $\vec{V}^{\mathrm{T}}_{\text{active}}\in\mathbb{R}^{r\times n}$. The frozen segments, $\vec{U}_{\text{freeze}}$ and $\vec{V}^{\mathrm{T}}_{\text{freeze}}$, do not store optimizer states. Vectors newly sampled from the frozen segments are swapped with unsampled vectors in the active segments. This approach enables SST to function as a time-sharing system, effectively balancing resource allocation among the vectors in $\vec{U}$ and $\vec{V}^{\mathrm{T}}$.

### 4.6 Sparsity of SST

We analyze the efficiency of parameter usage.. Specifically, the ratio of trainable parameters in SST at a given rank $r$, denoted as $\Gamma_{\text{SST},r}$, is calculated as $\frac{r(m+n)+m}{mn}$. This parameter ratio is slightly higher than that of LoRA at the same rank, $\Gamma_{\text{LoRA},r}=\frac{r(m+n)}{mn}$, yet remains lower than LoRA at rank $r+1$, $\Gamma_{\text{LoRA},r+1}=\frac{(r+1)(m+n)}{mn}$, indicating a slightly increase in trainable parameters.

## 5 Experiments

To validate our Sparse Spectral Training (SST) approach, we conducted experiments on both Euclidean and hyperbolic neural networks, demonstrating the generalization of SST across various neural network architectures and embedding geometries.

We compared SST with full-rank training, LoRA, and ReLoRA*. The key distinctions between ReLoRA* and ReLoRA [] is that ReLoRA includes a full-rank training as "warm start", making it not an end-to-end memory-efficient method. Moreover, ReLoRA* resets all optimizer states for low-rank parameters, unlike ReLoRA, which resets 99%.

For our experiments, all linear layers in the baseline models were modified to their low-rank counterparts. Hyperparameters and implementation details are provided in Appendix F.

Further comparisons of SST with the contemporaneous work GaLore [] are elaborated in Appendix H, highlighting SST’s superior performance in low-rank configurations. Ablation studies are documented in Appendix I.

### 5.1 Machine Translation

We employ the vanilla transformer [] as the Euclidean transformer and HyboNet [] as the hyperbolic transformer. Our experiments include three widely-used machine translation datasets: IWSLT’14 English-to-German [], IWSLT’17 German-to-English [], and Multi30K German-to-English []. For IWSLT’14, the hyperparameters are aligned with those from HyboNet.

Table 5 presents BLEU scores for IWSLT’14 across various dimensions and ranks ($r$). The results confirm that SST consistently outperforms other low-rank methods. Notably, some BLEU scores for the hyperbolic transformer are zero, due to the training process encountering NaN losses, whereas SST maintains stability throughout.

Previous hyperbolic neural network articles have predominantly focused on low-dimensional configurations [, , ]. A key characteristic of hyperbolic space is its exponential growth in volume with distance from a reference point, which is significantly more rapid than the polynomial growth seen in Euclidean space []. This expansive nature makes hyperbolic spaces particularly prone to overfitting as dimensionality increases. By imposing constraints on the parameter search space of hyperbolic neural networks, SST prevents the overfitting typically associated with such high-dimensional settings. This spectral sparse constraint enhances the stability and robustness of our models, ensuring consistent performance during training.

Further comparative results on the Multi30K and IWSLT’17 datasets using the standard dimensions for vanilla Euclidean transformers are documented in Table 5.1. Here, SST not only surpasses other low-rank methods but also demonstrates superior performance compared to full-rank training.

### 5.2 Natural Language Generation

We utilize the OPT [] architecture as the baseline for our language generation experiments. All models are pre-trained on OpenWebText [], an open-source reproduction of OpenAI’s WebText. To facilitate fair comparisons across different OPT model sizes, we standardize the total training tokens for all models at 19.7 billion. A consistent rank ($r=64$) is applied for all low-rank methods.

Table 1 displays the validation perplexity results on the OpenWebText dataset across different sizes of OPT models. The results indicate that SST not only achieves lower perplexity scores compared to LoRA and ReLoRA* but also approximates the performance of full-rank training, with significantly fewer trainable parameters.

Figure: Figure 2: Comparison of performance on effective steps between SST and full-Rank training. Effective steps are quantified by multiplying the number of trainable parameters by the number of steps taken. All methods and model sizes utilize the same number of tokens in each step.
Refer to caption: x2.png

Figure 2 illustrates a comparison of effective steps among various training methods. The effective step metric, which considers both the number of trainable parameters and the number of training steps, demonstrates that SST offers a more efficient training approach compared to the full-rank method.

Each pretrained model undergoes zero-shot evaluations on all 16 NLP tasks used in OPT article [], including ARC Easy and Challenge [], HellaSwag [], OpenBookQA [], PIQA [], StoryCloze [], SuperGLUE [], WinoGrad [], and WinoGrande []. Evaluations are conducted using the LM Evaluation Harness framework []. Except for the ReCoRD task, which uses F1 score, all other tasks are evaluated using accuracy.

Table 2 details the zero-shot evaluation results across the 16 NLP tasks. SST consistently performs comparably or better than other low-rank methods and shows competitive performance against the full-rank models.

We further conduct an analysis experiment on inference by doing post-training singular value pruning on SST model (see appendix G).

**Table 1: Validation perplexity on OpenWebText across various OPT model sizesalong with the number of trainable parameters of each method. Rank $r=64$. Values in bold highlight the highest performance among the low-rank methods.**
|  | Full | LoRA | ReLoRA* | SST |
| --- | --- | --- | --- | --- |
| OPT-125M | 23.50 (125.2M) | 34.23 (50.9M) | 35.80 (50.9M) | 26.98 (51.0M) |
| OPT-350M | 21.78 (331.2M) | 34.26 (57.5M) | 39.21 (57.5M) | 27.72 (57.7M) |
| OPT-1.3B | 15.10 (1.316B) | 1716 (164.4M) | 29.52 (164.4M) | 22.31 (164.7M) |

**Table 2: Zero-shot evaluations on the same 16 NLP tasks featured in the OPT article []. Except for the ReCoRD task, which uses F1 score, all other tasks are evaluated using accuracy, with values presented as percentages. Mean scores in bold represent superior performance among the low-rank methods. Additionally, we include the win percentage (counting ties) for each low-rank method compared to the full-rank training.**
|  | OPT-125M | OPT-350M | OPT-1.3B |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Full | LoRA | ReLoRA* | SST | Full | LoRA | ReLoRA* | SST | Full | LoRA | ReLoRA* | SST |
| ARC (Challenge) | 21.2 | 22.9 | 21.1 | 21.3 | 22.0 | 22.3 | 21.3 | 21.1 | 24.6 | 24.2 | 22.9 | 21.5 |
| ARC (Easy) | 35.8 | 34.2 | 33.9 | 34.3 | 35.9 | 32.3 | 33.0 | 35.7 | 43.2 | 26.1 | 35.9 | 37.8 |
| BoolQ | 59.5 | 54.2 | 60.8 | 62.0 | 53.6 | 56.2 | 62.2 | 57.7 | 57.7 | 37.8 | 61.4 | 59.5 |
| CB | 51.8 | 48.2 | 28.6 | 48.2 | 44.6 | 44.6 | 33.9 | 41.1 | 59.0 | 41.1 | 37.5 | 42.9 |
| COPA | 67.0 | 61.0 | 57.0 | 66.0 | 69.0 | 61.0 | 59.0 | 60.0 | 70.0 | 51.0 | 68.0 | 65.0 |
| HellaSwag | 27.7 | 26.5 | 27.1 | 26.9 | 28.4 | 26.6 | 26.9 | 27.5 | 35.0 | 26.1 | 27.2 | 28.1 |
| MultiRC | 55.4 | 57.2 | 55.9 | 57.2 | 52.0 | 52.6 | 56.4 | 57.0 | 56.8 | 42.8 | 57.7 | 56.9 |
| OpenBookQA | 24.6 | 24.6 | 23.6 | 26.2 | 26.4 | 24.2 | 23.0 | 25.2 | 29.0 | 27.0 | 24.8 | 25.0 |
| PIQA | 58.7 | 57.2 | 56.3 | 58.3 | 59.2 | 56.9 | 56.9 | 59.0 | 64.0 | 50.3 | 57.1 | 59.1 |
| ReCoRD | 16.7 | 17.5 | 22.6 | 18.5 | 19.4 | 17.6 | 19.0 | 23.2 | 13.7 | 17.6 | 23.0 | 18.1 |
| RTE | 50.5 | 56.7 | 53.1 | 53.4 | 52.0 | 49.1 | 54.9 | 50.2 | 51.6 | 52.7 | 52.0 | 53.8 |
| StoryCloze | 55.8 | 53.8 | 53.6 | 54.5 | 57.2 | 53.7 | 53.0 | 54.6 | 61.1 | 49.7 | 54.0 | 56.1 |
| WIC | 49.8 | 51.4 | 50.0 | 50.0 | 50.5 | 50.0 | 50.0 | 50.2 | 50.3 | 50.0 | 50.0 | 50.0 |
| Winograd | 52.0 | 48.7 | 50.6 | 50.6 | 55.0 | 51.7 | 50.2 | 51.3 | 55.7 | 50.9 | 52.4 | 55.3 |
| Winogrande | 49.1 | 49.2 | 50.7 | 50.1 | 50.7 | 50.3 | 50.8 | 52.0 | 51.1 | 47.9 | 50.0 | 49.1 |
| WSC | 36.5 | 38.5 | 36.5 | 36.5 | 36.5 | 37.5 | 36.5 | 36.5 | 39.4 | 63.5 | 36.5 | 36.5 |
| Mean | 44.5 | 43.8 | 42.6 | 44.6 | 44.5 | 42.9 | 42.9 | 43.9 | 47.6 | 41.2 | 44.4 | 44.7 |
| Win Percentage | - | 50.0 | 43.8 | 56.3 | - | 31.3 | 31.3 | 31.3 | - | 18.8 | 25.0 | 25.0 |

### 5.3 Hyperbolic Graph Neural Networks

Hyperbolic Graph Neural Networks (HGNNs) [, ] capitalize on the expansive and hierarchical nature of hyperbolic space to efficiently manage and analyze graph-structured data. This geometric space is particularly suitable for graphs due to its ability to closely mimic the underlying data structures with minimal distortion, offering a substantial improvement over traditional Euclidean methods.

We evaluated the effectiveness of SST on HyboNet [] version HGNN in node classification and link prediction across four distinct datasets: Airport [], Cora [], Disease [], and PubMed []. Each experiment was conducted with three random seeds.

The results, detailed in Table 5.3, demonstrate strong performance in both node classification and link prediction tasks. SST not only shows comparable performance to full-rank training (exceeding it in the Disease link prediction task) but also significantly outperforms LoRA at equivalent ranks. Notably, SST’s advantage over LoRA is larger on $r=1$ than $r=2$, likely due to SST’s sampling strategy being particularly effective in sparser scenarios.

## 6 Conclusion and Discussion

In this work, Sparse Spectral Training (SST) has demonstrated its efficacy as a resource-efficient training methodology that closely approximates the performance of full-rank training across diverse architectures, tasks and embedding geometries. SST introduces a noval approach by updating all singular values and selectively adjusting the singular vectors of network weights, optimizing resource utilization while closely mirroring the performance of full-rank training. Moreover, some areas that need further explorations are: (1) Investigating faster convergence approaches that avoid optimizer state reset (2) Extending the application of SST to the embeddings of large language models (LLMs).

## 7 Broader Impacts

This research enhances the memory efficiency of training large language models (LLMs), which contributes positively by reducing the environmental impact and making LLM training accessible to researchers with limited resources. On the downside, the ease of access to powerful LLMs raises concerns about potential misuse [, ]. Careful consideration and management of these factors are essential to maximize the benefits and mitigate risks.

## Appendix A Algorithm of Sparse Spectral Training

Figure: Algorithm 2 Sparse Spectral Training (SST)

## Appendix B Proof of Gradient of Sparse Spectral Layer

We can express the differential of $\vec{W}$ as the sum of differentials:

$$ $\mathrm{d}\vec{W}=\mathrm{d}\vec{U}\,\vec{\Sigma}\vec{V}^{\mathrm{T}}+\vec{U} \,\mathrm{d}\vec{\Sigma}\,\vec{V}^{\mathrm{T}}+\vec{U}\vec{\Sigma}\,\mathrm{d} \vec{V}^{\mathrm{T}}$ (16) $$

We have chain rule for the gradient of $\vec{W}$:

$$ $\frac{\partial\mathcal{L}}{\partial\vec{W}}=\frac{\partial\mathcal{L}}{ \partial\vec{h}}\frac{\partial\vec{h}}{\partial\vec{W}}=\frac{\partial\mathcal {L}}{\partial\vec{h}}\vec{x}^{\mathrm{T}}$ (17) $$

$$ $\displaystyle\mathrm{d}\mathcal{L}$ $\displaystyle=\frac{\partial\mathcal{L}}{\partial\vec{W}}:\mathrm{d}\vec{W}$ $\displaystyle=\frac{\partial\mathcal{L}}{\partial\vec{W}}:\mathrm{d}\vec{U}\, \vec{\Sigma}\vec{V}^{\mathrm{T}}+\frac{\partial\mathcal{L}}{\partial\vec{W}}: \vec{U}\,\mathrm{d}\vec{\Sigma}\,\vec{V}^{\mathrm{T}}+\frac{\partial\mathcal{L }}{\partial\vec{W}}:\vec{U}\vec{\Sigma}\,\mathrm{d}\vec{V}^{\mathrm{T}}$ $\displaystyle=\frac{\partial\mathcal{L}}{\partial\vec{W}}\vec{V}\vec{\Sigma}: \mathrm{d}\vec{U}+\vec{U}^{\mathrm{T}}\frac{\partial\mathcal{L}}{\partial\vec{ W}}\vec{V}:\mathrm{d}\vec{\Sigma}+\vec{\Sigma}\vec{U}^{\mathrm{T}}\frac{ \partial\mathcal{L}}{\partial\vec{W}}:\mathrm{d}\vec{V}^{\mathrm{T}}$ $$

where $:$ is the Frobenius inner product. So we have the gradient of $\vec{U}$, $\vec{\Sigma}$ and $\vec{V}^{\mathrm{T}}$:

$$ $\frac{\partial\mathcal{L}}{\partial\vec{U}}=\frac{\partial\mathcal{L}}{ \partial\vec{W}}\vec{V}\vec{\Sigma},\quad\frac{\partial\mathcal{L}}{\partial \vec{V}^{\mathrm{T}}}=\vec{\Sigma}\vec{U}^{\mathrm{T}}\frac{\partial\mathcal{L }}{\partial\vec{W}},\quad\frac{\partial\mathcal{L}}{\partial\vec{\Sigma}}=\vec {U}^{\mathrm{T}}\frac{\partial\mathcal{L}}{\partial\vec{W}}\vec{V}$ (18) $$

In vector perspective, for the $i^{th}$ vector, it is:

$$ $\frac{\partial\mathcal{L}}{\partial\vec{U}_{\cdot i}}=\frac{\partial\mathcal{L }}{\partial\vec{W}}\vec{V}_{\cdot i}\vec{\Sigma}_{i},\quad\frac{\partial \mathcal{L}}{\partial{\vec{V}_{\cdot i}}}=\vec{\Sigma}_{i}\frac{\partial \mathcal{L}}{\partial\vec{W}^{\mathrm{T}}}{\vec{U}_{\cdot i}},\quad\frac{ \partial\mathcal{L}}{\partial\vec{\Sigma}_{i}}={\vec{U}_{\cdot i}}^{\mathrm{T} }\frac{\partial\mathcal{L}}{\partial\vec{W}}\vec{V}_{\cdot i}$ (19) $$

where $\vec{U}_{\cdot i}$ means the $i^{th}$ column vector of $\vec{U}$, and $\vec{\Sigma}_{i}$ is the $i^{th}$ value of the diagonal matrix $\vec{\Sigma}$.

## Appendix C Proof of Decomposition of Gradient of Weight

See 4.1

###### Proof.

Given that the expectation of $\frac{\partial\mathcal{L}}{\partial\vec{W}}$ pointing from current $\mathbf{W}$ to the optimal $\mathbf{W}^{*}$, we have:

$$ $\mathbb{E}[\frac{\partial\mathcal{L}}{\partial\vec{W}}]\propto-(\vec{W}^{*}- \vec{W})$ (20) $$

SVD decompose $\vec{W}^{*}$ and get $[\vec{U},\vec{\Sigma}^{*},\vec{V}^{\mathrm{T}}]=\text{SVD}(\vec{W}^{*})$.

Assuming that at step $k$’s $\mathbf{W}_{k}$ statisfy $[\vec{U},\vec{\Sigma},\vec{V}^{\mathrm{T}}]=\text{SVD}(\vec{W}_{k})$, then:

$$ $\mathbb{E}[\frac{\partial\mathcal{L}}{\partial\vec{W}}]\propto-\vec{U}(\vec{ \Sigma}^{*}-\vec{\Sigma})\vec{V}^{\mathrm{T}}$ (21) $$

This will make step $(k+1)$’s $\mathbf{W}_{k+1}$ statisfy $[\vec{U},\vec{\Sigma}^{\prime},\vec{V}^{\mathrm{T}}]=\text{SVD}(\vec{W}_{k+1})$

With $\mathbf{W}_{0}=\mathbf{0}$ at the initial step, this establishes the conditions for Eq. 10 to hold throughout the iterative process.

∎

###### Proof.

## Appendix D Proof of Advantage of Enhanced Gradient over Default Gradient

See 4.2

###### Proof.

We compare update of $\vec{W}$ in each step for full-rank training, default gradient of SST and enhanced gradient of SST. For simplicity, we only consider stochastic gradient descent:

For full-rank training, update of $\vec{W}$ in each step is:

$$ $\Delta\vec{W}_{\text{full}}=-\eta\frac{\partial\mathcal{L}}{\partial\vec{W}}$ (22) $$

For default gradient of SST, update of $\vec{W}=\vec{U}\vec{\Sigma}\vec{V}^{\mathrm{T}}$ in each step is:

$$ $\Delta\vec{W}_{\text{default}}=-(\eta\frac{\partial\mathcal{L}}{\partial\vec{U }}\,\vec{\Sigma}\vec{V}^{\mathrm{T}}+\eta\vec{U}\,\mathrm{d}\frac{\partial \mathcal{L}}{\partial\vec{\Sigma}}\,\vec{V}^{\mathrm{T}}+\eta\vec{U}\vec{ \Sigma}\,\frac{\partial\mathcal{L}}{\partial\vec{V}^{\mathrm{T}}})$ (23) $$

According to Eq. 18, the update of $\vec{W}$ can be converted to:

$$ $\Delta\vec{W}_{\text{default}}=-\eta(\frac{\partial\mathcal{L}}{\partial\vec{W }}\,\vec{V}\vec{\Sigma}^{2}\vec{V}^{\mathrm{T}}+\vec{U}\vec{U}^{\mathrm{T}} \frac{\partial\mathcal{L}}{\partial\vec{W}}\vec{V}\vec{V}^{\mathrm{T}}+\vec{U} \vec{\Sigma}^{2}\vec{U}^{\mathrm{T}}\frac{\partial\mathcal{L}}{\partial\vec{W}})$ (24) $$

For enhanced gradient of SST, according to Eq. 9, the update of $\vec{W}$ is:

$$ $\Delta\vec{W}_{\text{enhance}}==-\eta(\frac{\partial\mathcal{L}}{\partial\vec{ W}}\,\vec{V}\vec{\Sigma}\vec{V}^{\mathrm{T}}+\vec{U}\vec{U}^{\mathrm{T}}\frac{ \partial\mathcal{L}}{\partial\vec{W}}\vec{V}\vec{V}^{\mathrm{T}}+\vec{U}\vec{ \Sigma}\vec{U}^{\mathrm{T}}\frac{\partial\mathcal{L}}{\partial\vec{W}})$ (25) $$

We can decompose $\frac{\partial\mathcal{L}}{\partial\vec{W}}=\vec{U}^{\prime}\vec{D}\vec{V^{
\prime}}^{\mathrm{T}}$. This doesn’t need to be singular value decomposition. Just need to guarantee $\vec{U}^{\prime}$ and $\vec{V^{\prime}}^{\mathrm{T}}$ are orthogonal matrices and $\vec{D}$ is diagonal matrix. We assume there exists a $\vec{V^{\prime}}$, making $\vec{V^{\prime}}=\vec{V}$. Because $\vec{W}$ is the accumulation of previous steps’ $\frac{\partial\mathcal{L}}{\partial\vec{W}}$, they are likely to share similar projection matrix, which explains why the assumption is reasonable. Similarly, we assume $\vec{U^{\prime}}=\vec{U}$. Then the update of default gradient of SST could be approximated to:

$$ $\displaystyle\Delta\vec{W}_{\text{default}}$ $\displaystyle=-\eta(\vec{U}\vec{D}\vec{V}^{\mathrm{T}}\vec{V}\vec{\Sigma}^{2} \vec{V}^{\mathrm{T}}+\vec{U}\vec{U}^{\mathrm{T}}\vec{U}\vec{D}\vec{V}^{\mathrm {T}}\vec{V}\vec{V}^{\mathrm{T}}+\vec{U}\vec{\Sigma}^{2}\vec{U}^{\mathrm{T}} \vec{U}\vec{D}\vec{V}^{\mathrm{T}})$ $\displaystyle=-\eta(\vec{U}\vec{D}\vec{\Sigma}^{2}\vec{V}^{\mathrm{T}}+\vec{U} \vec{D}\vec{V}^{\mathrm{T}}+\vec{U}\vec{\Sigma}^{2}\vec{D}\vec{V}^{\mathrm{T}})$ $\displaystyle=-\eta\vec{U}(2\vec{D}\vec{\Sigma}^{2}+\vec{D})\vec{V}^{\mathrm{T}}$ (26) $$

Similarly, the update of full-rank training and the update of enhanced gradient of SST could be approximated to:

$$ $\Delta\vec{W}_{\text{full}}=-\eta\vec{U}\vec{D}\vec{V}^{\mathrm{T}},\quad \Delta\vec{W}_{\text{enhance}}=-\eta\vec{U}(2\vec{D}\vec{\Sigma}+\vec{D})\vec{ V}^{\mathrm{T}}$ (27) $$

As only the direction of update matters, the scale of update can be adjusted by changing learning rate. We measure similarity using the Frobenius norm of the differences between SST updates and 3 times of the full-rank update.

Norm Differences Calculation:

$$ $\displaystyle\text{Error}_{\text{default}}$ $\displaystyle=\|3\Delta\mathbf{W}_{\text{full}}-\Delta\mathbf{W}_{\text{ default}}\|_{F}=\eta\|\mathbf{U}(3\mathbf{D}-\mathbf{D}-2\mathbf{D}\mathbf{ \Sigma}^{2})\mathbf{V}^{\text{T}}\|_{F},$ $\displaystyle\text{Error}_{\text{enhance}}$ $\displaystyle=\|\Delta\mathbf{W}_{\text{full}}-\Delta\mathbf{W}_{\text{enhance }}\|_{F}=\eta\|\mathbf{U}(3\mathbf{D}-\mathbf{D}-2\mathbf{D}\mathbf{\Sigma}) \mathbf{V}^{\text{T}}\|_{F}.$ $$

Using the property that the Frobenius norm is invariant under multiplication by orthogonal matrices:

$$ $\displaystyle\text{Error}_{\text{default}}$ $\displaystyle=2\eta\|\mathbf{D}-\mathbf{D}\mathbf{\Sigma}^{2}\|_{F},$ $\displaystyle\text{Error}_{\text{enhance}}$ $\displaystyle=2\eta\|\mathbf{D}-\mathbf{D}\mathbf{\Sigma}\|_{F}.$ $$

We seek to establish:

$$ $\|\mathbf{D}-\mathbf{D}\mathbf{\Sigma}\|_{F}\leq\|\mathbf{D}-\mathbf{D}\mathbf {\Sigma}^{2}\|_{F}.$ $$

Analysis for $\sigma_{i}\leq 1$ and $\sigma_{i}>1$:
Let $d_{i}$ represent the diagonal elements of $\mathbf{D}$, and $\sigma_{i}$ represent the diagonal elements of $\mathbf{\Sigma}$.

- •
For $0\leq\sigma_{i}\leq 1$: $|d_{i}(1-\sigma_{i})|\leq|d_{i}(1-\sigma_{i}^{2})|$ because $1-\sigma_{i}^{2}\geq 1-\sigma_{i}\geq 0$.
- •
For $\sigma_{i}>1$: $|d_{i}(1-\sigma_{i})|<|d_{i}(1-\sigma_{i}^{2})|$ because $1-\sigma_{i}^{2}<1-\sigma_{i}<0$.

Therefore, for all $\sigma_{i}$, $\|\mathbf{D}-\mathbf{D}\mathbf{\Sigma}\|_{F}\leq\|\mathbf{D}-\mathbf{D}\mathbf
{\Sigma}^{2}\|_{F}$.

This inequality shows that the enhanced gradient of SST is more similar to the full-rank update than the default gradient of SST, providing better approximation to full-rank training.

∎

###### Proof.

## Appendix E Proof of Zero Distortion with SVD Initialization

See 4.3

###### Proof.

$$ $\displaystyle\Delta\mathbf{W}_{\text{LoRA}}$ $\displaystyle=\Delta\vec{B}\vec{A}+\vec{B}\Delta\vec{A}$ $\displaystyle=-(\eta\frac{\partial\mathcal{L}}{\partial\vec{B}}\vec{A}+\eta \vec{B}\frac{\partial\mathcal{L}}{\partial\vec{A}})$ $\displaystyle=-\eta(\frac{\partial\mathcal{L}}{\partial\Delta\mathbf{W}} \mathbf{A}^{\mathrm{T}}\mathbf{A}+\mathbf{B}\mathbf{B}^{\mathrm{T}}\frac{ \partial\mathcal{L}}{\partial\Delta\mathbf{W}})$ $\displaystyle=-\eta(\vec{U}\vec{D}\vec{V}^{\mathrm{T}}\mathbf{A}^{\mathrm{T}} \mathbf{A}+\mathbf{B}\mathbf{B}^{\mathrm{T}}\vec{U}\vec{D}\vec{V}^{\mathrm{T}})$ (28) $$

when $\mathbf{A}=\vec{V}^{\mathrm{T}}$ and $\mathbf{B}=\vec{U}$, then:

$$ $\displaystyle\Delta\mathbf{W}_{\text{LoRA}}$ $\displaystyle=-\eta(\vec{U}\vec{D}\vec{V}^{\mathrm{T}}\mathbf{V}\mathbf{V}^{ \mathrm{T}}+\mathbf{U}\mathbf{U}^{\mathrm{T}}\vec{U}\vec{D}\vec{V}^{\mathrm{T}})$ $\displaystyle=-\eta(\vec{U}\vec{D}\mathbf{V}^{\mathrm{T}}+\mathbf{U}\vec{D} \vec{V}^{\mathrm{T}})$ $\displaystyle=2\Delta\mathbf{W}_{\text{full}}$ (29) $$

∎

###### Proof.

## Appendix F Experiment Details

### F.1 Implementation Details for SST

#### Sampling of U → → 𝑈 \vec{U} over→ start_ARG italic_U end_ARG and V → T superscript → 𝑉 T \vec{V}^{\mathrm{T}} over→ start_ARG italic_V end_ARG start_POSTSUPERSCRIPT roman_T end_POSTSUPERSCRIPT .

In our experiments, we employ a more exploratory approach when sampling $\vec{U}$ and $\vec{V}^{\mathrm{T}}$:

$$ $p(i)=\frac{1}{2}(\frac{1}{m}+\frac{\vec{\Sigma}_{i}}{\sum_{j}\vec{\Sigma}_{j}})$ (30) $$

where $p(i)$ is the possibility to sample index $i$ vector of $\vec{U}$ and $\vec{V}^{\mathrm{T}}$. This method modifies the earlier Eq. 5 by combining the multinomial distribution with a uniform distribution. This adjustment ensures that vectors associated with lower singular values still have a substantial likelihood of being sampled, preventing their probabilities from becoming excessively low and promoting a more balanced exploration across the spectral components.

#### Optimizer state reset and warmup.

Before each iteration, Sparse Spectral Training (SST) resets all optimizer states for $\vec{U}$, $\vec{V}^{\mathrm{T}}$ and $\vec{\Sigma}$. For example, for optimizers like Adam, this involves clearing the first and second moments as well as the timestep. Consequently, a brief warmup period is essential at the beginning of each iteration to accommodate the reset states. This warmup period is typically 20 steps, guided by the exponential decay rate $\beta$ used in the Adam optimizer.

#### Hyperbolic SST.

The formula of hyperbolic linear layer in [] is:

$$ $\vec{h}=f_{\vec{x}}(\vec{M})\vec{x}=\begin{bmatrix}\frac{\sqrt{\|\vec{W}\vec{x }\|_{2}-\frac{1}{K}}}{\vec{v}^{\top}\vec{x}}\vec{v}^{\top}\\ \vec{W}\end{bmatrix}\vec{x}=\begin{bmatrix}\sqrt{\|\vec{W}\vec{x}\|_{2}-\frac{ 1}{K}}\vec{v}^{\top}\\ \vec{W}\vec{x}\end{bmatrix}$ (31) $$

where $\vec{v}\in\mathbb{R}^{n+1}$, $\vec{W}\in\mathbb{R}^{m\times(n+1)}$ and $K$ is the curvature. The formula of Hyperbolic SST is:

$$ $h=\begin{bmatrix}\sqrt{\|\vec{U}\vec{\Sigma}\vec{V}^{\mathrm{T}}\vec{x}\|_{2}- \frac{1}{K}}\vec{v}^{\top}\\ \vec{U}\vec{\Sigma}\vec{V}^{\mathrm{T}}\vec{x}\end{bmatrix}$ (32) $$

### F.2 Hyperparameters of Machine Translation

#### IWSLT’14.

The hyperparameters can be found in Table 3. We employ the same codebase and hyperparameters as those used in HyboNet [], which is derived from OpenNMT-py []. The final model checkpoint is utilized for evaluation. Beam search, with a beam size of 2, is employed to optimize the evaluation process. Experiments were conducted on one A100 GPU.

For SST, number of steps per iteration ($T_{3}$) is set to 200. Each iteration begins with a warmup phase lasting 20 steps. The number of iterations per round ($T_{2}$) is determined by the formula $T_{2}=d/r$, where $d$ represents the embedding dimension and $r$ denotes the rank used in SST.

**Table 3: Hyperparameters on IWSLT’14 for Euclidean and hyperbolic Transformer.**
| Hyper-parameter | Euclidean | Hyperbolic |
| --- | --- | --- |
| Embedding Dimension | 64, 128, 256 | 64, 128, 256 |
| Feed-forward Dimension | 256, 512, 1024 | 256, 512, 1024 |
| Batch Size | 10240 tokens | 10240 tokens |
| Gradient Accumulation Steps | 4 | 4 |
| Training Steps | 40000 | 40000 |
| Dropout | 0.0 | 0.1 |
| Attention Dropout | 0.1 | 0.1 |
| Max Gradient Norm | - | 0.5 |
| Warmup Steps | 6000 | 6000 |
| Decay Method | noam | noam |
| Label Smoothing | 0.1 | 0.1 |
| Layer Number | 6 | 6 |
| Head Number | 4 | 4 |
| Learning Rate | 5 | 2 |
| Optimizer | Adam | rAdam |

#### Multi30K and IWSLT’17.

The hyperparameters can be found in Table 4. Because of overfitting, model checkpoint with lowest validation loss is utilized for evaluation. A larger learning rate (0.0003) is used for only low rank parameters ($\vec{U}$, $\vec{V}^{\mathrm{T}}$ and $\vec{\Sigma}$ for SST, $\vec{B}$ and $\vec{A}$ for LoRA and ReLoRA*. Experiments were conducted on one A100 GPU.

For SST, number of steps per iteration ($T_{3}$) is set to 200 for Multi30K and 400 for IWSLT’17. Each iteration begins with a warmup phase lasting 20 steps. The number of iterations per round ($T_{2}$) is determined by the formula $T_{2}=d/r$, where $d$ represents the embedding dimension and $r$ denotes the rank used in SST.

**Table 4: Hyperparameters on Multi30K and IWSLT’17 for vanilla Transformer.**
| Hyper-parameter | Multi30K | IWSLT’17 |
| --- | --- | --- |
| Embedding Dimension | 512 | 512 |
| Feed-forward Dimension | 2048 | 2048 |
| Batch Size | 128 sentences | 128 sentences |
| Gradient Accumulation Steps | 1 | 1 |
| Training Steps | 100000 | 150000 |
| Dropout | 0.1 | 0.1 |
| Decay Method | constant | constant |
| Layer Number | 6 | 6 |
| Head Number | 8 | 8 |
| Learning Rate | 0.0001 | 0.0001 |
| Weight Decay | 1 | 0.1 |
| Optimizer | AdamW | AdamW |

### F.3 Hyperparameters of Natural Language Generation

The hyperparameters for our experiments are detailed in Table 5. We employ a linear warmup of 2000 steps followed by a stable learning rate, without decay. A larger learning rate (0.001) is used for only low rank parameters ($\vec{U}$, $\vec{V}^{\mathrm{T}}$ and $\vec{\Sigma}$ for SST, $\vec{B}$ and $\vec{A}$ for LoRA and ReLoRA*. The total training tokens for each experiment is 19.7B, roughly 2 epochs of OpenWebText. Distributed training is facilitated using the Accelerate [] library across four A100 GPUs on a Linux server.

For SST, number of steps per iteration ($T_{3}$) is set to 200. Each iteration begins with a warmup phase lasting 20 steps. The number of iterations per round ($T_{2}$) is determined by the formula $T_{2}=d/r$, where $d$ represents the embedding dimension and $r$ denotes the rank used in SST.

**Table 5: Hyperparameters for OPT Models**
| Hyper-parameter | OPT-125M | OPT-350M | OPT-1.3B |
| --- | --- | --- | --- |
| Embedding Dimension | 768 | 512 (project to 1024) | 2048 |
| Feed-forward Dimension | 3072 | 4096 | 8192 |
| Global Batch Size | 240 | 240 | 240 |
| Sequence Length | 2048 | 2048 | 2048 |
| Training Steps | 40000 | 40000 | 40000 |
| Learning Rate | 0.0001 | 0.0001 | 0.0001 |
| Warmup Steps | 2000 | 2000 | 2000 |
| Optimizer | AdamW | AdamW | AdamW |
| Layer Number | 12 | 24 | 24 |
| Head Number | 12 | 16 | 32 |

### F.4 Hyperparameters of Hyperbolic Graph Neural Networks

We use HyboNet [] as full-rank model, with same hyperparameters as those used in HyboNet. Experiments were conducted on one A100 GPU.

For SST, number of steps per iteration ($T_{3}$) is set to 100. Each iteration begins with a warmup phase lasting 100 steps. The number of iterations per round ($T_{2}$) is determined by the formula $T_{2}=d/r$, where $d$ represents the embedding dimension and $r$ denotes the rank used in SST.

We set dropout rate to 0.5 for the LoRA and SST methods during the node classification task on the Cora dataset. This is the only one deviation from the HyboNet configuration.

## Appendix G Singular Value Pruning

Figure: Figure 3: Singular Value Pruning. We conduct singular value pruning on full-rank and SST pretrained OPT-125M model. After performing singular value decomposition on weight matrices, we preserve the top $k$ singular values so that the cumulative sum of preserved singular values ranges from $[100\%,99\%,98\%,...,93\%,90\%]$ of the original cumulative sum. The pruned ratio of singular values is plotted along the x-axis.
Refer to caption: x3.png

We further conduct an analysis study of the potential for using SST model for further compression. The results, as shown in Figure 3, indicate that the SST model retains lower perplexity across a wider range of pruning ratios compared to the full-rank model. This suggests that the SST method effectively concentrates the informational content of the weights into fewer singular values, making it more suitable for further compression.

This enhanced performance underscores the potential of SST in maintaining essential model characteristics even under significant compression, making it a promising approach for developing lightweight yet powerful language models for inference.

## Appendix H Evaluating SST and GaLore: Complementary Approaches to Memory Efficiency

Recently, a new approach named Gradient Low-Rank Projection (GaLore) has been proposed to address the memory challenges associated with training large language models. GaLore, by implementing a memory-efficient gradient projection method, enhances training efficiency without compromising the training dynamics as traditional low-rank adaptation methods, like LoRA, often do.

Using the released code of GaLore(^1^11[https://github.com/jiaweizzhao/GaLore](https://github.com/jiaweizzhao/GaLore)), we conducted comparative experiments on the IWSLT’14 dataset with Transformer models, employing the same configurations as other low-rank methods. We set the scale factor $\alpha=1$ in these experiments because $\alpha=0.25$, which is used in the article, performs much worse than $\alpha=1$. As illustrated in Table H, SST method consistently outperformed GaLore across various model dimensions and ranks, except for $d=256$, $r=32$.

In addition, we evaluated validation perplexity on the OpenWebText dataset with OPT-125M models. We tested GaLore with scale factor $\alpha=0.25$ (used in the article) and $\alpha=1$. As shown in Table 6, SST surpassed GaLore at both settings of $\alpha$.

In GaLore experiments, the rank $r\geq d/4$, whereas in our studies, we use $r\leq d/8$. Here, we discuss our guess on why SST may have an advantage over GaLore on low-rank settings. GaLore utilizes a projection matrix $P_{t}\in\mathbb{R}^{m\times r}$ derived from the singular value decomposition (SVD) of a single step’s gradient. Only one step’s gradient may introduce noise due to data sampling variability, potentially distorting the gradient updates. Conversely, SST employs $\vec{U}$ and $\vec{V}^{\mathrm{T}}$ as projection matrices, which are initialized through the SVD of $\vec{W}$. $\vec{W}$ could be seemed as the momentum of gradient of $\vec{W}$, less noisy than one step’s gradient. Furthermore, SST updates all $\vec{\Sigma}$ values, regardless of $r$, making it more robust as $r$ decreases.

**Table 6: Validation perplexity, compared with GaLore on OpenWebText dataset with OPT-125M, along with the number of trainable parameters of each method. $r=64$. Values highlighted in bold represent the highest performance among the low rank methods.**
|  | Full | LoRA | ReLoRA* | GaLore $\alpha=0.25$ | GaLore $\alpha=1$ | SST |
| --- | --- | --- | --- | --- | --- | --- |
| OPT-125M | 23.50 (125.2M) | 34.23 (50.9M) | 35.80 (50.9M) | 37.08 (45.6M) | 32.17 (45.6M) | 26.98 (51.0M) |

## Appendix I Ablation Study

We conduct an ablation study to evaluate the impact of various components and configurations within SST on the IWSLT’14 using a Euclidean Transformer with a dimension of 128 and rank $r$ of 4. The results of this study are summarized in Table 7, which highlights the contributions of specific elements to the overall performance measured in BLEU score.

The baseline configuration of SST achieves a BLEU score of 22.80. When we modify the SST by removing the enhanced gradient updates for $\vec{U}$ and $\vec{V}^{\mathrm{T}}$, the BLEU score slightly increases to 22.87. This may due to Adam optimizer [] used in training Transformer mitigate the influence of scaling of gradient. In our sgd experiments, enhanced gradient updates for $\vec{U}$ and $\vec{V}^{\mathrm{T}}$ show prominant improvements.

Another variation tested involves changing the update mechanism for $\vec{\Sigma}$. Instead of updating all $\vec{\Sigma}$, only sampled $\vec{\Sigma}$ are updated, same as update for $\vec{U}$ and $\vec{V}^{\mathrm{T}}$. This modification results in a lower BLEU score of 22.40, indicating that full updates of $\vec{\Sigma}$ contribute positively to the model’s performance.

We experiment with a configuration similar to the ReLoRA*, where $\mathbf{h}=(\vec{W}+\vec{U}\vec{\Sigma}\vec{V}^{\mathrm{T}})\mathbf{x}$, with $\vec{U}$ and $\vec{V}^{\mathrm{T}}$ randomly initialized and $\vec{\Sigma}$ initialized to zero. After each round, $\vec{U}$, $\vec{V}^{\mathrm{T}}$ and $\vec{\Sigma}$ are reinitialized. This setup significantly reduces the BLEU score to 16.03, which is similar to the performance of LoRA and ReLoRA*. This demonstrates that the most important feature of SST is that instead of randomly initialized, SST uses previously dominant singular vectors as the initialization of $\vec{U}$ and $\vec{V}^{\mathrm{T}}$, which is aligned with our mathematical derivation in section 4.3.

**Table 7: Ablation Study on IWSLT’14 dataset with Euclidean Transformer. Dimension is 128 and $r$ is 4.**
|  | BLEU |
| --- | --- |
| Without enhanced gradient of $\vec{U}$ and $\vec{V}^{\mathrm{T}}$ | 22.87 |
| Instead of update all $\vec{\Sigma}$, only update sampled $\vec{\Sigma}$ | 22.40 |
| Use formula similar as ReLoRA*: $\mathbf{h}=(\vec{W}+\vec{U}\vec{\Sigma}\vec{V}^{\mathrm{T}})\mathbf{x}$. ($\vec{U}$ and $\vec{V}^{\mathrm{T}}$ random initialized, and $\vec{\Sigma}$ zero initialized) | 16.03 |
| SST | 22.80 |

Figure: (a) fc1
Refer to caption: x4.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: - How would SST perform if compared against models capable of achieving lower perplexity scores (e.g., below 10)? What is the extent to which SST’s performance might vary when applied to higher-performing models with more rigorous baseline comparisons?
- Are there alternative sampling mechanisms that the authors considered, and could these impact the efficiency or stability of SST?
soundness: 2
strengths: Sparse Spectral Training (SST) introduces an interesting approach to memory-efficient training by updating all singular values while selectively focusing on significant singular vectors. This approach shows promise for reducing memory requirements and can be a creative improvement on existing low-rank adaptation methods.
summary: This paper proposes Sparse Spectral Training (SST), a memory-efficient training method designed for large neural networks. SST aims to reduce memory usage while maintaining performance close to full-rank models. The method updates all singular values of network weights and selectively updates singular vectors based on their significance, using a sampling strategy that prioritizes impactful components. SST is evaluated on tasks across Euclidean and hyperbolic neural networks, showing that it often outperforms existing memory reduction techniques and occasionally matches or exceeds full-rank training performance, especially in low-dimensional or resource-constrained settings.
weaknesses: While SST is promising, the current experiments lack rigor, especially as the performance comparisons rely on relatively high perplexity values and low zero-shot accuracies, limiting the conclusions that can be drawn about SST's effectiveness. For a more meaningful evaluation, it would be valuable to benchmark against models that achieve stronger results (e.g., perplexity below 10). Additionally, the selection of baselines could be strengthened to more comprehensively illustrate SST’s relative performance. These changes would help clarify SST’s potential as a competitive memory-efficient training method.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. I would like to ask whether the authors tries to tune the hyper-parameters of baseline methods, such as learning rate, rank value.
soundness: 2
strengths: 1. The paper focused on an important problem about memoey-efficient LLM pre-training. 

2. The proposed memory-efficient LLM pre-training method SST is easy to follow. 

3. This paper provides diverse results on different tasks to verify the performance of proposed method. The results illustrate that the proposed method can achieve a better performance than LoRA and its variants.
summary: This paper propose a novel memory-efficient LLM pre-training method SST, which employs singular value decomposition to initialize and periodically reinitialize lowrank parameters. The experimental results illustrate that the proposed can achieve a better performance than LoRA and its variants.
weaknesses: 1. The motivation and intuition is not very clear. I'm not very clear about the reason why we need to use the proposed method Sparse Spectral Training (SST) for pre-training.  I mean why we need to leverage sparse updates within the spectral domain of neural network
weights. I understand the paper  provided some limitations of LoRA ReLoRA and Galore, but some related work tried to solve these limitations. For example, the limitation of ReLoRA about zero initialization of B can result in zero gradients of A, some related work has solved it, such as Pissa [1]. 

2. The additional training cost from SST could be increased with scaling the model size. For example, the results of Table 16 about overall training time shows the training time will be increased from 303.4h to 387.2h when using SST. 

3. The sensitivity analysis of the proposed method about hyper-parameters, such as the number of iterations and iteration interval. 

4. The experiments for pre-training task mainly focus on the small scale, not sure the results when sclaing to a larger dataset and model. 

[1] PISSA: PRINCIPAL SINGULAR VALUES AND SINGULAR VECTORS ADAPTATION OF LARGE LANGUAGE MODELS

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: Please refer to weaknesses.
soundness: 2
strengths: 1. This paper is the first to propose parameter-efficient pre-training process in hyperbolic space, in which the design of SST is quite novel and effective. 

2. Using SVD to approximate low-rank computation is somehow novel especially with the design of selective updates of singular vector and iterative reinitialization during training. 

3. The experimental results show promising performance of SST.

4. This paper is well organized with clear discussion on the disadvantages of the baseline methods, such as ReLoRA.
summary: The paper introduces sparse spectral training (SST) to optimize memory usage for pre-training. SST updates all singular values and selectively updates singular vectors through a multinomial sampling method weighted by the magnitude of the singular values, and employs singular value decomposition to initialize and periodically reinitialize low rank parameters, therefore reducing distortion. The author of the paper conduct experiments on natural language generation, machine translation, node classification, link prediction, and image classification, and SST demonstrates its ability to outperform existing memory reduction training methods and is comparable to full-rank training in various cases.
weaknesses: 1. The overall framework and algorithm design are similar to ReLoRA, which limits the novelty aspect of the paper.

2. Trainable parameter count does not showing significant advantages over baseline methods, such as ReLoRA, and the GPU memory consumption is much larger compared with other methods. Considering the design is using SVD, this weakness is inevitable. Therefore, it limits the wide adoption of this method in the future.

3. Although the author of the paper discusses why the SVD decomposition approach is important, the discussion fails to convince the reader that this approach is necessary. According to the experimental results, ReLoRA achieves quite comparable results with lower memory consumption.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. How does periodic SVD reinitialization affect scalability as model size increases, especially in larger language models or high-dimensional settings?
2. How sensitive is the polynomial sampling strategy to the selection of singular values? Can the stability and efficiency of specific application scenarios be further improved by adopting different sampling methods?
3. Does periodic SVD reinitialization affect the convergence speed of models that are more constrained by low rank? In this case, is it possible that SST requires more training steps to achieve convergence compared to full-rank training?
4. Can the authors provide more practical guidance to help tune SST's hyperparameters (such as SVD reinitialization frequency) to achieve the best balance of memory efficiency and performance in different tasks?
soundness: 3
strengths: 1. By focusing on selective updates, SST provides an efficient memory management technique that significantly reduces GPU memory usage during training.

2. The proposed method reduces the perplexity and BLEU score gaps on benchmarks, showing competitive or superior performance on a range of neural network architectures and tasks, often approaching the effect of full-rank training.

3. Regular use of SVD helps SST alleviate the saddle point problem of ReLoRA, bringing its training dynamics closer to full-rank training.
summary: By selectively updating singular values ​​and vectors in the weight matrix, this paper proposes Sparse Spectral Training (SST), a memory-efficient technique for training large-scale neural networks. SST reduces memory usage while maintaining comparable performance to full-rank training by using SVD for startup and periodic reinitialization. SST outperforms current low-rank techniques such as LoRA and ReLoRA in machine translation and natural language processing, and shows competitive results when tested on a variety of neural network architectures and tasks
weaknesses: 1. Frequent SVD reinitialization may affect scalability for very large models and incur significant computational overhead.

2. As with other low-rank techniques, stability issues can still occur in very low-rank models even with the sampling mechanism of SST.

3. Users who are not familiar with these techniques may find it challenging to get started, as memory efficiency optimizations (such as dividing active and frozen parts) and selective update methods may make the implementation more complex.

Meta-review: This paper studies a memory-efficient training approach called Sparse Spectral Training (SST). The method selectively updates the singular vectors of weight matrices based on importance scores and demonstrates greater training stability compared to other low-rank update methods. Through experiments on Euclidean and Hyperbolic neural networks, the study shows that the proposed approach outperforms several other memory-reduced training approaches.

Strengths:
- Easy-to-follow methods.
- Promising performance compared to baselines.
- Studies of hyperbolic space.

Weaknesses:
- Limited novelty compared to baselines. 
- Limited improvement compared to baselines.
- Pretraining results are mainly limited to small-scale high-perplexity/low average-score benchmarks.
- Missing reference and limited comparison to prior work, e.g., PISSA.
- Gap between SST and full-rank training.
- Much larger GPU memory consumption.

My main concern is that the authors need to explain why this is a better method for memory consumption and training efficiency in plain words in the method section and in experiments. I don’t quite understand, but why are these results buried in Appendix I and K, with very limited mention in the main paper? Table 22 says training time, but SST does not seem to provide much gain. There is also a limited comparison of these different methods on the BLEU-memory-time Pareto front, with Figure 6 being an exception (with only one baseline).

Decision: Reject