Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks

Authors: Jialin Zhao, Yingtao Zhang, Xinghang Li, Huaping Liu, Carlo Vittorio Cannistraci
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Efficient AI, Parameter-efficient training, Pre-training, Hyperbolic Network

Abstract:
The growing demands on GPU memory posed by the increasing number of neural network parameters call for training approaches that are more memory-efficient. Previous memory reduction training techniques, such as Low-Rank Adaptation (LoRA) and ReLoRA, face challenges, with LoRA being constrained by its low-rank structure, particularly during intensive tasks like pre-training, and ReLoRA suffering from saddle point issues. In this paper, we propose Sparse Spectral Training (SST) to optimize memory usage for pre-training. SST updates all singular values and selectively updates singular vectors through a multinomial sampling method weighted by the magnitude of the singular values. Furthermore, SST employs singular value decomposition to initialize and periodically reinitialize low-rank parameters, reducing distortion relative to full-rank training compared to other low-rank methods. Through comprehensive testing on both Euclidean and hyperbolic neural networks across various tasks, including natural language generation, machine translation, node classification, link prediction, and image classification, SST demonstrates its ability to outperform existing memory reduction training methods and is comparable to full-rank training in various cases. On LLaMA-1.3B, with only 18.7\% of the parameters trainable compared to full-rank training (using a rank equivalent to 6\% of the embedding dimension), SST reduces the perplexity gap between other low-rank methods and full-rank training by 97.4\%. This result highlights SST as an effective parameter-efficient technique for model pre-training, offering a promising new paradigm for achieving scalable and memory-efficient neural network training. Our code is available at https://anonymous.4open.science/r/sparse_spectral_training-6A2C/.

Decision: reject
Reviewer scores: 5, 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1730315688143

thread_type:
official_review

<thread_object>
creator: reviewer_tbso

<content>
summary:
By selectively updating singular values ​​and vectors in the weight matrix, this paper proposes Sparse Spectral Training (SST), a memory-efficient technique for training large-scale neural networks. SST reduces memory usage while maintaining comparable performance to full-rank training by using SVD for startup and periodic reinitialization. SST outperforms current low-rank techniques such as LoRA and ReLoRA in machine translation and natural language processing, and shows competitive results when tested on a variety of neural network architectures and tasks

strengths:
1. By focusing on selective updates, SST provides an efficient memory management technique that significantly reduces GPU memory usage during training.

2. The proposed method reduces the perplexity and BLEU score gaps on benchmarks, showing competitive or superior performance on a range of neural network architectures and tasks, often approaching the effect of full-rank training.

3. Regular use of SVD helps SST alleviate the saddle point problem of ReLoRA, bringing its training dynamics closer to full-rank training.

weaknesses:
1. Frequent SVD reinitialization may affect scalability for very large models and incur significant computational overhead.

2. As with other low-rank techniques, stability issues can still occur in very low-rank models even with the sampling mechanism of SST.

3. Users who are not familiar with these techniques may find it challenging to get started, as memory efficiency optimizations (such as dividing active and frozen parts) and selective update methods may make the implementation more complex.

questions:
1. How does periodic SVD reinitialization affect scalability as model size increases, especially in larger language models or high-dimensional settings?
2. How sensitive is the polynomial sampling strategy to the selection of singular values? Can the stability and efficiency of specific application scenarios be further improved by adopting different sampling methods?
3. Does periodic SVD reinitialization affect the convergence speed of models that are more constrained by low rank? In this case, is it possible that SST requires more training steps to achieve convergence compared to full-rank training?
4. Can the authors provide more practical guidance to help tune SST's hyperparameters (such as SVD reinitialization frequency) to achieve the best balance of memory efficiency and performance in different tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730520661665

thread_type:
official_review

<thread_object>
creator: reviewer_mmze

<content>
summary:
The paper introduces sparse spectral training (SST) to optimize memory usage for pre-training. SST updates all singular values and selectively updates singular vectors through a multinomial sampling method weighted by the magnitude of the singular values, and employs singular value decomposition to initialize and periodically reinitialize low rank parameters, therefore reducing distortion. The author of the paper conduct experiments on natural language generation, machine translation, node classification, link prediction, and image classification, and SST demonstrates its ability to outperform existing memory reduction training methods and is comparable to full-rank training in various cases.

strengths:
1. This paper is the first to propose parameter-efficient pre-training process in hyperbolic space, in which the design of SST is quite novel and effective. 

2. Using SVD to approximate low-rank computation is somehow novel especially with the design of selective updates of singular vector and iterative reinitialization during training. 

3. The experimental results show promising performance of SST.

4. This paper is well organized with clear discussion on the disadvantages of the baseline methods, such as ReLoRA.

weaknesses:
1. The overall framework and algorithm design are similar to ReLoRA, which limits the novelty aspect of the paper.

2. Trainable parameter count does not showing significant advantages over baseline methods, such as ReLoRA, and the GPU memory consumption is much larger compared with other methods. Considering the design is using SVD, this weakness is inevitable. Therefore, it limits the wide adoption of this method in the future.

3. Although the author of the paper discusses why the SVD decomposition approach is important, the discussion fails to convince the reader that this approach is necessary. According to the experimental results, ReLoRA achieves quite comparable results with lower memory consumption.

questions:
Please refer to weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730642381706

thread_type:
official_review

<thread_object>
creator: reviewer_cein

<content>
summary:
This paper propose a novel memory-efficient LLM pre-training method SST, which employs singular value decomposition to initialize and periodically reinitialize lowrank parameters. The experimental results illustrate that the proposed can achieve a better performance than LoRA and its variants.

strengths:
1. The paper focused on an important problem about memoey-efficient LLM pre-training. 

2. The proposed memory-efficient LLM pre-training method SST is easy to follow. 

3. This paper provides diverse results on different tasks to verify the performance of proposed method. The results illustrate that the proposed method can achieve a better performance than LoRA and its variants.

weaknesses:
1. The motivation and intuition is not very clear. I'm not very clear about the reason why we need to use the proposed method Sparse Spectral Training (SST) for pre-training.  I mean why we need to leverage sparse updates within the spectral domain of neural network
weights. I understand the paper  provided some limitations of LoRA ReLoRA and Galore, but some related work tried to solve these limitations. For example, the limitation of ReLoRA about zero initialization of B can result in zero gradients of A, some related work has solved it, such as Pissa [1]. 

2. The additional training cost from SST could be increased with scaling the model size. For example, the results of Table 16 about overall training time shows the training time will be increased from 303.4h to 387.2h when using SST. 

3. The sensitivity analysis of the proposed method about hyper-parameters, such as the number of iterations and iteration interval. 

4. The experiments for pre-training task mainly focus on the small scale, not sure the results when sclaing to a larger dataset and model. 

[1] PISSA: PRINCIPAL SINGULAR VALUES AND SINGULAR VECTORS ADAPTATION OF LARGE LANGUAGE MODELS

questions:
1. I would like to ask whether the authors tries to tune the hyper-parameters of baseline methods, such as learning rate, rank value.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731047788710

thread_type:
official_review

<thread_object>
creator: reviewer_3vn4

<content>
summary:
This paper proposes Sparse Spectral Training (SST), a memory-efficient training method designed for large neural networks. SST aims to reduce memory usage while maintaining performance close to full-rank models. The method updates all singular values of network weights and selectively updates singular vectors based on their significance, using a sampling strategy that prioritizes impactful components. SST is evaluated on tasks across Euclidean and hyperbolic neural networks, showing that it often outperforms existing memory reduction techniques and occasionally matches or exceeds full-rank training performance, especially in low-dimensional or resource-constrained settings.

strengths:
Sparse Spectral Training (SST) introduces an interesting approach to memory-efficient training by updating all singular values while selectively focusing on significant singular vectors. This approach shows promise for reducing memory requirements and can be a creative improvement on existing low-rank adaptation methods.

weaknesses:
While SST is promising, the current experiments lack rigor, especially as the performance comparisons rely on relatively high perplexity values and low zero-shot accuracies, limiting the conclusions that can be drawn about SST's effectiveness. For a more meaningful evaluation, it would be valuable to benchmark against models that achieve stronger results (e.g., perplexity below 10). Additionally, the selection of baselines could be strengthened to more comprehensively illustrate SST’s relative performance. These changes would help clarify SST’s potential as a competitive memory-efficient training method.

questions:
- How would SST perform if compared against models capable of achieving lower perplexity scores (e.g., below 10)? What is the extent to which SST’s performance might vary when applied to higher-performing models with more rigorous baseline comparisons?
- Are there alternative sampling mechanisms that the authors considered, and could these impact the efficiency or stability of SST?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732278393170

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Reply

comment:
We sincerely thank the reviewers for their valuable and constructive feedback on our submission. Your insights have been very important in enhancing the quality of our work, and we greatly appreciate the time and effort you dedicated to reviewing our paper.

- **In response to Reviewer 3vN4's and Reviewer cEiN's comments on scaling to larger models and achieving lower PPL**:  
    We extended our experiments to include LLaMA-1.3B.  

    The final PPL for LLaMA-1.3B are summarized below (bold indicates the lowest PPL among all low-rank methods):

    | Dataset       | Model      | r/d        | Full lr=4e-4 | LoRA lr=1e-3 | ReLoRA* lr=1e-3 | SST lr=1e-3 |
    |---------------|------------|------------|--------------|--------------|-----------------|-------------|
    | OpenWebText   | LLaMA-1.3B | 128/2048   | 14.54        | 16.50        | 17.32           | **14.59**   |

    Each method was trained with 13.1 billion tokens. The learning rate for full-rank training is 4e-4, consistent with the configuration in Table 1 of the ReLoRA article. As recommended in Appendix A of the ReLoRA article, the learning rate for ReLoRA should be twice that of full-rank training. Thus, we used a learning rate of 1e-3 for all low-rank methods. SST outperforms all other low-rank methods, closely matching the PPL of full-rank training and reducing the perplexity gap to full-rank by **97.4%**.

- **In response to Reviewer cEiN's comment on using larger datasets and addressing hyperparameter search**:  
    We pre-trained LLaMA-130M on the C4 dataset, which is 25 times larger than OpenWebText. To address the comment on hyperparameter tuning, we evaluated each method with two different learning rates.

    The final PPL for LLaMA-130M on C4 and OpenWebText are summarized below (bold indicates the lowest PPL among all low-rank methods):

    | Dataset       | Model        | r/d       | Full lr=1e-3 | LoRA lr=1e-3 | ReLoRA* lr=1e-3 | SST lr=1e-3 | LoRA lr=3e-3 | ReLoRA* lr=3e-3 | SST lr=3e-3 |
    |---------------|--------------|-----------|--------------|--------------|-----------------|-------------|--------------|-----------------|-------------|
    | C4            | LLaMA-130M  | 64/768    | 24.91        | 35.91        | 37.34           | 32.13       | 30.75        | 133.06          | **29.79**   |
    | OpenWebText   | LLaMA-130M  | 64/768    | 20.04        | 29.71        | 31.33           | 25.89       | 795.24       | 230.43          | **23.35**   |

    Each method was trained with 2.6 billion tokens. The learning rate of 1e-3 for full-rank training matches the setting used in Table 1 of the ReLoRA article. We ran LoRA, ReLoRA*, and SST with both $lr=1\mathrm{e}{-3}$ and $lr=3\mathrm{e}{-3}$. The results show that SST achieves lower PPL than LoRA and ReLoRA* for the same learning rate, and with $lr=3\mathrm{e}{-3}$, SST surpasses all other low-rank methods, reducing the perplexity gap by **16.4%** on C4 and **65.8%** on OpenWebText.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732280172367

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Reply 2

comment:
Below is a summary of the items we have addressed in our responses to the reviewers, but are currently working on incorporating into the updated manuscript. This ensures that no points have been overlooked by us when revising the manuscript:

- **New Experiments**:
  - Incorporate experiments with LLaMA-130M and 1.3B on OpenWebText and C4 datasets.  
  - Include comparisons of different sampling mechanisms, such as always selecting top-$r$ singular vectors.  
  - Add the sensitivity analysis for the number of iterations.  
  - Add the sensitivity analysis of SST under different rank. 
  - Add the experiment to evaluate SST's performance with additional training steps compared to full-rank training.

- **Manuscript Revisions**:
  - Include the plot comparing efficiency ratios.  
  - Add the comparison of SST with fewer training steps against other low-rank methods trained with full steps.
  - Add PiSSA to the related work section.  
  - Standardize terminology by replacing "the number of steps per iteration" with "iteration interval".
</content>
</thread_object>

</thread>
<thread>
creation date:
1732775430231

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Reply 3

comment:
We are pleased to share that all the revisions we committed to in our responses to the reviewers have been successfully incorporated into the updated manuscript. The revised parts are clearly marked in red for ease of review.

We sincerely thank the reviewers for their invaluable feedback and constructive suggestions, which have greatly enhanced the quality of our work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732861577558

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Request for Assistance Regarding Reviewer Feedback for Submission 6242

comment:
Dear Area Chairs, Senior Area Chairs, and Program Chairs,  

We hope this email finds you well. We are the authors of Submission 6242. We posted our replies to all four reviewers 7 days ago, on November 22nd, but have not received any response. We provided another update to our revised manuscript and requested further feedback yesterday, on November 28th, yet there has still been no reply.  

We noticed that most other submissions have received responses from their reviewers, and we are concerned that our situation might be due to a technical issue, such as:  
1. Reviewers not receiving email notifications about our replies.  
2. Reviewer responses being posted but not visible to us.  

If this is not due to technical reasons, we kindly request your assistance in notifying the reviewers of our replies. We feel it might be inappropriate for us to send repeated notifications directly to the reviewers, and your support in this matter would be greatly appreciated.  

We deeply value the feedback we have received so far, which has significantly helped us improve the quality of our work. Hearing from the reviewers at this stage would allow us to incorporate any additional experiments or provide further clarifications, ensuring the manuscript fully reflects their valuable insights.  

Thank you very much for your time and consideration. We sincerely appreciate your efforts in managing the review process.  

Best regards,  
*The authors of Submission 6242*
</content>
</thread_object>

</thread>
<thread>
creation date:
1733033983838

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Reply 4

comment:
Thank you once again to all reviewers for your thoughtful and constructive feedback, which has been instrumental in guiding our revisions. We are pleased to share that we are currently conducting an additional experiment with **LLaMA-7B** to further evaluate the scalability of SST on larger models.  

As of now, LLaMA-7B has been trained for 40,000 steps for both SST and full-rank training. Unfortunately, the server for training ReLoRA* crashed, but the training has been restarted.   

The perplexity (PPL) for LLaMA-7B on OpenWebText at various training steps is summarized below:

| r/d        | Step    | 5000   | 10000 | 15000 | 20000 | 25000 | 30000 | 35000 | 40000 |
|------------|---------|--------|-------|-------|-------|-------|-------|-------|-------|
| 4096/4096  | Full      | 272.12 | 62.55 | 36.04 | 27.13 | 23.57 | 21.65 | 20.53 | 19.76 |
| 256/4096   | ReLoRA* | 84.41  | 59.04 | 34.50 | 30.23 | 27.69 |  25.31  | 24.11  | 23.54  |
| 256/4096   | SST     | 65.41  | 29.84 | 24.59 | 22.34 | 21.02 | 20.40 | 19.76 | 19.66 |

**(Table updated on December 4th to include results up to 40,000 steps)**

By 40,000 steps, each method has been trained on 5.2 billion tokens. The learning rate for full-rank training is set to 2e-4, which is half the learning rate used for full-rank training in LLaMA-1.3B experiments. Similarly, the learning rate for SST and ReLoRA* is 5e-4, half of the SST and ReLoRA* learning rate used in LLaMA-1.3B. Since the ReLoRA and GaLore articles and codebases do not specify the learning rate for full-rank training of LLaMA-7B, we have followed the trend of reducing learning rates with increasing model size as observed in their articles.  

As of 40,000 steps, SST continues to outperform full-rank training in terms of PPL, demonstrating its effectiveness even at larger scales. We will provide further updates as the training progresses and additional results become available.  

Thank you for your continued support and valuable suggestions, which have been crucial in shaping this work. We look forward to any further feedback you may have.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733311633194

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary for Area Chair and Reviewers of Submission 6242 (1/3)

comment:
Dear Area Chairs and Reviewers,  

We are truly grateful for all the effort Area Chair have made to encourage reviewers to respond. We fully understand the difficulty of organizing the review process this year, especially with submissions increasing by 61%. Your dedication has been essential in maintaining the fair and supportive culture of the AI research community. 

We would also like to thank all the reviewers for their insightful suggestions, as well as the recommendation to test our approach on larger models and datasets. To date, our experiments have spanned a wide range of tasks, including image classification, node classification, link prediction, machine translation, and language modeling. The tested models range from simple architectures to complex ones, including MLPs, Graph Neural Networks (Hyperbolic), Transformers (Euclidean and Hyperbolic), the OPT model family (125M, 350M, 1B), and the LLaMA-1 model family (130M, 1.3B, 7B). Across all these scenarios, SST outperforms SOTA low-rank pre-training methods and achieves results comparable to, or even surpassing, full-rank counterparts, as demonstrated below.

To further assist you and minimize the effort required to review all the discussions, we have provided a brief summary of the key points raised and addressed during the review process:

1. **Strengths**:
    - The problem this paper is focusing on, i.e. memory-efficient LLM pre-training, is **important** and difficult.
      - Reviewer cEiN: "focused on an important problem"
    - Our proposed method, SST, is **novel**.
      - Reviewer 3vN4: "introduces an interesting approach"
      - Reviewer cEiN: “propose a novel memory-efficient LLM pre-training method”
      - Reviewer MMzE: "the design of SST is quite novel"
    - SST adopts a more effective approach by updating **all** singular values at each step. SST **selective** updates of singular vectors sampled from a multinomial distribution weighted by the magnitude of the singular values. Additionally, SST uses **singular value decomposition to initialize and reinitialize** low-rank parameters during training, reducing distortion relative to full-rank training compared to other low-rank methods.
      - Reviewer 3vN4: "updates all singular values of network weights and selectively updates singular vectors based on their significance"
      - Reviewer cEiN: "employs singular value decomposition to initialize and periodically reinitialize lowrank parameters"
      - Reviewer MMzE: "updates all singular values and selectively updates singular vectors"
      - Reviewer TBSo: "focusing on selective updates"
    - On various experiments, SST shows much better performance compared with other low-rank methods. SST reduces performance gap by **97.4%** (LLaMA-1.3B), **>100%** (LLaMA-7B), **65.8%** (LLaMA-125M, OpenWebText), **67.6%** (OPT-125M), **52.4%** (OPT-350M), **50.0%** (OPT-1.3B), **66.7%** (Transformer), **73.7%** (node classification), **82.5%** (link prediction), and **49%** (image classification). SST surpasses full-rank training with *1/8* trainable parameters (LLaMA-7B, until current steps), *1/4* (Transformer), and *1/8* (Hyperbolic Transformer).
      - Reviewer 3vN4: "often outperforms existing memory reduction ... occasionally matches or exceeds full-rank training ..."
      - Reviewer cEiN: "achieve a better performance than LoRA and its variants"
      - Reviewer MMzE: (1) "on natural language generation ... image classification, SST outperform existing memory reduction ... and is comparable to full-rank training in various cases"; (2) "show promising performance of SST"
      - Reviewer TBSo: (1) "outperforms current low-rank"; (2) "reduces the perplexity and BLEU score gaps"; (3) "often approaching the effect of full-rank training"
    - SST addresses and solves issues in prior low-rank methods.
      - Reviewer MMzE: (1) "reducing distortion"; (2) "clear discussion on the disadvantages of the baseline methods, such as ReLoRA"
      - Reviewer TBSo: "alleviate the saddle point problem of ReLoRA, bringing its training dynamics closer to full-rank training"
    - The presentation is clear and accessible.
      - Reviewer cEiN: "easy to follow"

2. **Questions**:
   - Question: Require for larger model (Reviewer 3vN4, cEiN)  
    Reply: We trained full-rank, LoRA, ReLoRA and SST on LLaMA-1.3b (**Global Reply 1**) and LLaMA-7b (**Global Reply 4**). On LLaMA-1.3B, with 1/16 rank, SST achieves a PPL of **14.59** vs **14.54** for full-rank training. On LLaMA-7B, with 1/16 rank, until 40K steps, SST achieves a PPL of **19.66** vs **19.76** for full-rank training. These findings demonstrate SST’s scalability and effectiveness when applied to larger models.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper studies a memory-efficient training approach called Sparse Spectral Training (SST). The method selectively updates the singular vectors of weight matrices based on importance scores and demonstrates greater training stability compared to other low-rank update methods. Through experiments on Euclidean and Hyperbolic neural networks, the study shows that the proposed approach outperforms several other memory-reduced training approaches.

Strengths:
- Easy-to-follow methods.
- Promising performance compared to baselines.
- Studies of hyperbolic space.

Weaknesses:
- Limited novelty compared to baselines. 
- Limited improvement compared to baselines.
- Pretraining results are mainly limited to small-scale high-perplexity/low average-score benchmarks.
- Missing reference and limited comparison to prior work, e.g., PISSA.
- Gap between SST and full-rank training.
- Much larger GPU memory consumption.

My main concern is that the authors need to explain why this is a better method for memory consumption and training efficiency in plain words in the method section and in experiments. I don’t quite understand, but why are these results buried in Appendix I and K, with very limited mention in the main paper? Table 22 says training time, but SST does not seem to provide much gain. There is also a limited comparison of these different methods on the BLEU-memory-time Pareto front, with Figure 6 being an exception (with only one baseline).