TensorGPT: Efficient Compression of Large Language Models based on Tensor-Train Decomposition

Authors: Mingxue Xu, Yao Lei Xu, Danilo Mandic
Primary area: infrastructure, software libraries, hardware, systems, etc.
Keywords: model compression, low-rank factorization, tensor decomposition

Abstract:
The Small Language Models (SLMs, or on-device LMs) is a concept corresponding to the Large Language Model (LLM), which has significantly fewer parameters and is typically deployed on low-end devices, like mobile phones and single-board computers (e.g. Raspberry Pi). Unlike LLMs, which utilize the increasing model size for better generalization, SLMs are expected to adjust the exact deployment environment changes. 
Furthermore, most edge applications have battery life concerns, which have never been considered in the GPU servers for data centres.
Targeting these two issues, this paper focuses on the token embedding compression for adaptivity and low energy requirements in edge applications.
We propose a training-free model compression approach based on the Tensor-Train Decomposition (TTD), 
whereby each pre-trained token embedding vector is converted into a lower-dimensional Matrix Product State (MPS).
We then comprehensively investigate the low-rank structures extracted by this approach, regarding the compression ratio, language task performance, latency and energy consumption on a typical low-end device (i.e. Raspberry Pi). 
Taking the sub-billion parameter versions of GPT-2/Cerebres-GPT and OPT as examples, the model compressed with our approach can achieve a comparable language task performance to the original model with around $2.0\times$ embedding layer compression, while the energy consumption of single query drops by half.

Decision: reject
Reviewer scores: 3, 3, 3, 6

Reviews and discussion:
<thread>
creation date:
1730529817251

thread_type:
official_review

<thread_object>
creator: reviewer_cxwr

<content>
summary:
The paper presents TensorGPT, a novel approach for compressing LLMs through tensor-train decomposition of embedding layers. The key innovation is applying tensor-train decomposition to individual token embeddings without requiring additional training data or computation. The authors evaluate their method on GPT family models (GPT-2 and CerebrasGPT), demonstrating meaningful parameter reduction while maintaining or sometimes improving model performance. The work provides comprehensive evaluations on GPT-family models and demonstrates practical applicability on edge devices.

strengths:
1. Novel training-free compression method specifically targeting embedding layers. Strong practical value for edge device deployment
2. Comprehensive experiments across multiple tasks and model sizes.  Solid theoretical foundation with clear mathematical derivations.
Thorough analysis of compression vs. performance trade-offs

weaknesses:
1. Limited comparison with existing compression methods and baselines. The comparison would be more comprehensive by comparing with more baselines which are train-free and trained. 
2. Evaluation focused mainly on GPT-family models and mainly focus on small models. 
3. It would be a great to combine the proposed embedding compression method with other model compression methods to check compatibility.

questions:
1. Could this approach be extended to show performance of other model architectures beyond the GPT family?
2. Could this approach compatible with other model compression methods? 
3. Have you investigated the effect on model robustness like in the multilingual setting using more diverse tokens?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730655503038

thread_type:
official_review

<thread_object>
creator: reviewer_xafk

<content>
summary:
The paper presents "TensorGPT," a model compression method using Tensor-Train Decomposition (TTD) to reduce the parameter size of Large Language Models (LLMs), particularly in embedding layers, without additional training. This training-free approach preserves model performance while significantly reducing memory requirements, demonstrated on low-end devices like Raspberry Pi.

strengths:
1.	The approach does not require extra training, making it applicable for scenarios with limited resources or when extra data is unavailable.
2.	TensorGPT achieves substantial compression with low memory overhead, retaining performance across language modeling and sentiment analysis tasks.
3.	The paper includes experiments on low-end hardware (Raspberry Pi) and larger models, evaluating compression ratios, latency, and task-specific performance.

weaknesses:
1.	The novelty is limited. Tensor-Train Decomposition is explored in several language model compression works. What are the differences between the proposed method and existing works?
2.	Lack important baselines. Compressing token embedding layers is studied in some papers and the paper does not compare with them. For example, [1] and baselines used in [1] should be set as baselines.
3.	The empirical evaluation is primarily on language modeling and sentiment classification, with potential limitations in representing other complex NLP tasks, such as math reasoning and multi-hop question answering.

[1] Wang, Haoyu, et al. "LightToken: A task and model-agnostic lightweight token embedding framework for pre-trained language models." Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2023.

questions:
refer to weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730661609595

thread_type:
official_review

<thread_object>
creator: reviewer_qct4

<content>
summary:
The authors applied tensor train to an edge device application, and found we can use that to compress some NLP models. It reads to me that the author is simply applying a pre-existing, well-known tensor decomposition method on a problem. And I didn't see any new knowledge developed or presented (I might be wrong, so hopefully the authors could point out clearly what's the real technical contribution other than yet another application).

strengths:
- Working on a problem that's important.

weaknesses:
- The authors claim "- As far as we know, we are the first to compress LLMs with low-rank factorization, specifically", which is not true to me. The architecture is the same for most NLP models and thus pardon me I really don't know what's added. I think all the tensor-train stuff has been developed and applied before. It's not immediately obvious to me what's new conclusions or findings drawn from this paper.

questions:
- Pardon me that I really don't know what's the biggest contribution. Other than your claim of try things on LLM, what's the real technical contribution in your paper not discussed by previous works?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730867638316

thread_type:
official_review

<thread_object>
creator: reviewer_jblc

<content>
summary:
This paper proposes using tensor-train decomposition to compress the token embedding matrix, aiming to reduce model size and accelerate inference, particularly in edge device scenarios. The technique leverages a higher-order tensor approximation method, built upon singular value decomposition (SVD), to efficiently represent embeddings while maintaining performance. This tailored approach is well-suited for resource-constrained environments, promising benefits in storage and computational efficiency.

strengths:
The paper is well-structured and easy to follow, with a clear presentation of the methodology and its applications.

weaknesses:
**Limited Novelty:** The method largely builds upon the existing TT_SVD approach, with Algorithm 1 in this paper replicating methods already established in prior literature [1]. The methodological advancements or generalizations beyond TT_SVD appear minimal based on the methodology section.

**Unconvincing Experiments:** The experiments are conducted on older and relatively small models, such as GPT-2 and models up to only 1.3B parameters. This limits the relevance of the results, as they don’t reflect performance on contemporary large language models (LLMs). Additionally, the evaluation setup lacks modern benchmarking practices; for instance, LLM harness [2] would provide a more standardized evaluation framework.

**Unintuitive Rationale:** The foundational rationale for embedding a matrix using a tensor is unclear. The method requires reshaping the matrix into a higher-order tensor before applying tensor decomposition, but the paper does not provide an intuitive explanation for why this approach is effective or reasonable. Also, the embedding matrix only occupies a small part of the model even for a mid size of model, and a common way to reduce parameter size is through weight tie, which directly reduce the parameter sizes of embedding into half. How does you method performs in this case?

**Overstatement of Contribution:** Some claims appear overstated. For instance, the authors state, "As far as we know, we are the first to compress LLMs with low-rank factorization, specifically for low-end devices." However, the cited reference [3] already demonstrates compression of LLMs using SVD for similar purposes. Furthermore, there are numerous existing works that apply low-rank decomposition for LLM compression, such as [4-6].

**Limited Impact on Overall Model Size:** The embedding matrix occupies only a small portion of the model's parameters, even in mid-sized models. A commonly used technique, weight tying, can directly halve the embedding parameter size, offering a straightforward compression approach. The paper does not address how the proposed method compares to weight tying or performs when weight tying is already applied, raising questions about its practical impact on overall model size reduction.

[1] V. Oseledets. Tensor-train decomposition. SIAM Journal on Scientific Computing, 33(5):2295–2317,
2011. doi: 10.1137/090752286.

[2] Gao, Leo, et al. "A framework for few-shot language model evaluation." Version v0. 0.1. Sept 10 (2021): 8-9.

[3] Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. Language model
compression with weighted low-rank factorization. In International Conference on Learning
Representations, 2022.

[4] Yuan, Zhihang, et al. "Asvd: Activation-aware singular value decomposition for compressing large language models." arXiv preprint arXiv:2312.05821 (2023).

[5] Lin, Chi-Heng, et al. "MoDeGPT: Modular Decomposition for Large Language Model Compression." arXiv preprint arXiv:2408.09632 (2024).

[6] Ashkboos, Saleh, et al. "Slicegpt: Compress large language models by deleting rows and columns." arXiv preprint arXiv:2401.15024 (2024).

questions:
How does your method perform with weight tying?
What is the percentage of overall compression for model sizes larger than 13B?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732503940641

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
## **General Response (1/6)**

We sincerely appreciate the time, effort, and detailed comments from the Reviewers. 

We first respond to the four common issues in the reviews and then respond separately to each Reviewer.\
&nbsp;
&nbsp;
&nbsp;
&nbsp;
### **G0. What is the novelty and contribution of this work?**

This paper focuses on **compressing the Small Language Models (SLMs)** [8,9] **deployed on low-end devices (i.e. Raspberry Pi) in edge applications.** 
The edge applications pose two requirements to our compressing approach, which are not common in LLM applications:
- **Adaptivity**: the approach should dynamically adjust the model to the environmental changes (e.g. tokens registered or deregistered);
- **Low energy**: the computation and memory operations should consider the energy consumption (i.e. for longer battery life). 

Centred on these two issues (detailed discussion is in Section 2 of the updated submission), our approach based on Tensor-Train Decomposition (TTD) is **specifically designed for compressing SLMs**:
1. Adaptivity:  TTD works on embedding vector level, which allows the application to update the vocabulary without operating on the whole compressed embedding matrix;
2. Low energy: As computation operations are "cheaper" than memory operations regarding energy consumption, we chose to "exchange" the memory with computation to save energy during the forwarding passes, with negligible extra latency.

As far as we know, none of the current LLM (at least for those designed for GPUs) compression work has these concerns, though these concerns are critical for low-end devices and edge applications. 
TT format has an expressive and flexible form, which makes it easier for us to analyze and satisfy the two requirements.

Apart from common metrics like compression ratio and language task performance, we also give the estimated **energy costs** (with a similar approach in [18]) of our approach and SVD-based approach. At a quick glance, the following is the estimated inference energy costs with their best language task performance are

&nbsp;
&nbsp;

**Tab.G.0 Inference energy costs of an input text of 100 tokens.** (% percentage of uncompressed model energy costs, the lower, the better)
| Model               | OPT-125M | OPT-350M | DistilGPT2 | GPT-2  | GPT2-M | GPT2-L | CerebrasGPT-111M | CerebrasGPT-256M | CerebrasGPT-590M |
|---------------------|------------:|------------|------------:|--------|--------:|--------|--------------------:|--------------------|--------------------:|
| **SVD**  | 84.44%     | **43.35%**     | 70.28%     | 70.28% | 54.01% | 44.24% | 84.44%             | 81.80%             | 75.34%             |
| **Ours** | **59.21%**     | 52.73%     | **66.84%**     | **59.21%** | **52.74%** | **43.23%** | **61.51%**             | **51.71%**             | **50.45%**             |
| ||  | | |
 
The details of this part can be found in line 137-186, line 281-300, line 515-522 of the updated submission. 

&nbsp;
&nbsp;
&nbsp;
&nbsp;
### **G1. The investigated LLMs are too small.**

We feel there might be some misunderstanding here, especially since our paper focuses on the sub-billion language models (as stated in MobileLLM [8]) running on **low-end devices**, and we had stated this in the abstract and introduction of our original submission. 

We have therefore decided to change the title of our paper to avoid this ambiguity. The new title is **"TensorSLM: Sub-billion Parameter Language Model Compression for Low-end Devices based on Tensor-Train Decomposition"**. We have also rephrased our paper to further emphasise the focus (mainly Section 1,2,3 in the updated submission).
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes TensorGPT, a method for compressing small language models (SLMs) using tensor-train decomposition (TTD) of the token embedding layer. The authors claim this approach is training-free and suitable for deploying SLMs on low-end devices. The method is evaluated on GPT-2, OPT, and CerebrasGPT models with up to 1.3B parameters.

The paper addresses the important issue of SLM compression for edge devices and provides experiments on low-end hardware like Raspberry Pi. The proposed training-free compression method and consideration of energy efficiency are relevant for resource-constrained scenarios.

However, the paper's novelty is limited, as tensor-train decomposition has been used before in language model compression. The lack of comparisons to state-of-the-art baselines and the focus on relatively small models limit the generalizability and impact of the work. Additionally, the initial claims about being first to compress LLMs with low-rank factorization were overstated.

The primary reasons for rejection are the paper's failure to demonstrate significant technical novelty or empirical contributions beyond applying existing techniques to a specific use case. While the work addresses an important problem, it falls short of the level of innovation and impact expected for publication at ICLR.