TensorGPT: Efficient Compression of Large Language Models based on Tensor-Train Decomposition

## Abstract

Abstract High-dimensional token embeddings underpin Large Language Models (LLMs), as they can capture subtle semantic information and significantly enhance the modelling of complex language patterns.
However, this high dimensionality also introduces considerable model parameters and prohibitively high model storage and memory requirements, which is particularly unaffordable for low-end devices.
Targeting no extra training data and insufficient computation cases, we propose a training-free model compression approach based on the Tensor-Train Decomposition (TTD),
whereby each pre-trained token embedding is converted into a lower-dimensional Matrix Product State (MPS).
We then comprehensively investigate the low-rank structures extracted by this approach, in terms of the compression ratio, the language task performance, and latency on a typical low-end device (i.e. Raspberry Pi).
Taking GPT family models (i.e. GPT-2 and CerebrasGPT) as case studies, our approach theoretically results in 46.89 % percent 46.89 46.89\% 46.89 % fewer parameters of the entire model, with a compression ratio 39.38 × 39.38\times 39.38 × - 65.64 × 65.64\times 65.64 × for the embedding layers. With different hyperparameter choices, the model compressed with our approach can achieve a comparable language task performance to the original model with around 2.0 × 2.0\times 2.0 × embedding layer compression. This empirically proves the existence of low-rank structure in GPT family models, and demonstrates that about half of the parameters in the embedding layers are redundant.

## 1 Introduction

Memory and storage efficiency are currently prohibitive to unlocking the full potential of lightweight applications of Large Language Models (LLMs).
Typical LLMs for low-end devices can have less than one billion parameters (Mehta et al., ; Liu et al., ; Laskaridis et al., ), yet they are still a burden for low-end devices.
In the composition of the LLMs parameters, the embedding layers account for a significant portion, especially for sub-billion LLMs (Liu et al., ).
As shown in Fig. 1(a), for the current typical sub-billion LLMs, the smaller the LLMs, the higher the embedding layer portion, which can be up to $48.08\%$.

Low-rank factorization, which means breaking matrices or higher-dimensional tensors into smaller units, has recently been recognized as a promising solution for LLM efficiency, since its successful application in parameter-efficient fine-tuning Hu et al. (). There are already efforts aiming to utilize such a technique in language model compression; some are specialized for embedding layers (Chen et al., ; Hrinchuk et al., ; Lioutas et al., ; Acharya et al., ) while others are not (Chekalina et al., ; Chen et al., ; Hsu et al., ; Dao et al., ; ).
However, all of these require an extra training process, such as fine-tuning, meta-learning (Chen et al., ; ; Hsu et al., ; Edalati et al., ; Lioutas et al., ; Dao et al., ; ) and training from scratch (Hrinchuk et al., ; Chekalina et al., ). There are two limitations to this extra training: 1) extra training involves additional computation and training data, which may be unavailable for low-end devices; 2) training the language model from scratch discards the valuable knowledge stored in the weights of the original models. Given these two limitations, we are wondering:

Without extra training, by how much can the embedding layers be compressed without sacrificing language task performance?

Figure: (a) Embedding layer parameter count ratio for common sub-billion LLMs.
Refer to caption: x1.png

To answer this question, we propose TensorGPT, an approach that induces a high-dimensional tensor structure to capture the information stored in the embedding parameters.
We use the Tensor-Train Decomposition (TTD) , which excels in factorizing high-order tensors, to represent embeddings in a lower-rank Matrix Product State (MPS) format.
Due to the look-up table nature of the embedding layer, we treat each token embedding individually (i.e. each row of the token embedding matrix) rather than taking the token embedding matrix as a whole.
This prevents damaging the individual information of each token, and even applies to the use cases with ever-changing vocabulary.

We provide a comprehensive discussion of our proposed approach, including general perspectives like compression ratio, language task performance, and more technical perspectives like the impacts of different hyperparameter choices in TensorGPT and the effects of the original model size.
We found that no matter how we choose the tensor size, which refers to the dimension of each mode of a high-dimensional tensor, the compressed models will finally achieve a comparable language task performance, with a compression ratio $0.5\times$ - $2.0\times$. Some tensor sizes are particularly easier to find reasonable TT ranks, which decide how Tensor-Train Decomposition deal with each tensor mode, to achieve comparable language task performance. For example, in Fig. 1(b), $3$-order tensors are better at maintaining the task performance compared with the others. Moreover, the larger-scale models have better accuracy-compression trade-offs, as shown in Fig. 1(c).

Taking GPT family models as our case study, the contributions of this work are as follows:

- 1.
As far as we know, we are the first to compress LLMs with low-rank factorization, specifically for low-end devices. We adjust Tensor-Train Decomposition for non-parallel operations of embedding layers, where block-wise approaches (Dao et al., ; ) are incompetent.
- 2.
We test our approach in language modelling and sentiment classification tasks. In both tasks, the compressed models can even outperform the uncompressed models. The larger-scale models generally outperform the uncompressed models in precision and the F1-score in the sentiment classification, and they are more robust in language modelling when the compression ratio increases.
- 3.
We provide an ablation study on the general technical aspects of our approach. We measured the latency of various compressed cases of sub-billion GPT models on the low-end Raspberry Pi 5(^1^11[https://www.raspberrypi.com/products/raspberry-pi-5/](https://www.raspberrypi.com/products/raspberry-pi-5/)), and give a detailed systematic analysis of running TensorGPT on low-end edge devices like Raspberry Pi and higher-end servers.

## 2 Preliminaries

Figure: Figure 2: The execution of TensorGPT in edge computing scenario. The central server (server in public or private cloud services, or higher-end personal computer) compresses the token embedding vectors of the whole vocabulary with TensorGPT (step 1). Then, the compressed vocabulary and other parts of the language model (e.g. the decoder) are downloaded and deployed (step 2) on a low-end device (e.g. Raspberry Pi). During the application runs, a new token might be required according to the actual requirements, and registered by the service on the edge device (step 3). The low-end device then compresses this added token embedding with TensorGPT (step 4), and updates (step 5) the current vocabulary of the language model. The compression process of a single token embedding follows a pipeline of tensorization and Tensor-Train Decomposition.
Refer to caption: x4.png

This section gives the essential concepts related to tensor, tensor operations and Tensor-Train Decomposition. A more complete introduction about tensors can be found in Appx. A.

#### Order-N Tensor.

An order-$N$ real-valued tensor, $\mathcal{A}$, is a high-dimensional matrix (or multi-way array), denoted by $\mathcal{A}\in\mathbb{R}^{I_{1}\times\dots\times I_{N}}$, where $N$ is the order of the tensor (i.e., number of its modes), and $I_{k}$ ($1\leq k\leq N$) is the size (i.e., the dimension) of its $k$-th mode. In this sense, matrices (denoted as $\mathbf{A}\in\mathbb{R}^{I_{1}\times I_{2}}$) can be seen as order-$2$ tensors ($N=2$), vectors (denoted as ${\bf a}\in\mathbb{R}^{I}$) can be seen as order-$1$ tensors ($N=1$), and scalars (denoted as $a\in\mathbb{R}$) are order-0 0 tensors ($N=0$).

#### Tensorization.

A vector ${\bf a}=(a_{1},a_{2},\ldots,a_{I_{1}I_{2}\cdots I_{N}})\in\mathbb{R}^{I_{1}I_{
2}\cdots I_{N}}$, can be tensorized (or “folded”, “reshaped”) into an order-$N$ tensor $\mathcal{A}\in\mathbb{R}^{I_{1}\times I_{2}\times\cdots\times I_{N}}$, so that

$$ $\mathcal{A}[i_{1},i_{2},\dots,i_{N}]=a_{1+\sum_{k=1}^{N}(i_{k}-1)\prod_{p=1}^{ k-1}I_{p}},\quad\quad 1\leq i_{k}\leq I_{k},$ (1) $$

where $\mathcal{A}[i_{1},i_{2},\dots,i_{N}]$ denotes the $(i_{1},i_{2},\dots,i_{N})$-th entry of tensor $\mathcal{A}$.

#### Vectorization.

Given an order-$N$ tensor, $\mathcal{A}\in\mathbb{R}^{I_{1}\times\cdots\times I_{N}}$, its vectorization reshapes the high-dimensional matrix into a vector, $\texttt{vec}\left(\mathcal{A}\right)=\mathbf{\bf{a}}\in\mathbb{R}^{I_{1}\cdots
I
_{N}}$.

#### Tensor Contraction.

The contraction of $\mathcal{A}\in\mathbb{R}^{I_{1}\times\dots\times I_{N}}$ and $\mathcal{B}\in\mathbb{R}^{J_{1}\times\dots\times J_{M}}$, over the $k$th and $p$th modes respectively, where $I_{k}=J_{p}$ is denoted as $\mathcal{A}\times^{p}_{k}\mathcal{B}$ and results in a tensor $\mathcal{C}\in\mathbb{R}^{I_{1}\times\cdots\times I_{k-1}\times I_{k+1}\times
\cdots\times I_{N}\times J_{1}\times\cdots\times J_{p-1}\times J_{p+1}\times
\cdots\times J_{M}}$, with entries

$$ $\displaystyle\mathcal{C}[i_{1},\dots,i_{k-1},i_{k+1},\dots,i_{N},j_{1},\dots,j _{p-1},j_{p+1},\dots,j_{M}]$ (2) $\displaystyle=$ $\displaystyle\sum_{q=1}^{I_{k}}\mathcal{A}[i_{1},\dots,i_{k-1},q,i_{k+1},\dots ,i_{N}]\mathcal{B}[j_{1},\dots,j_{p-1},q,j_{p+1},\dots,j_{M}]$ $$

#### Tensor-Train Decomposition (TTD).

The most common Tensor-Train Decomposition (Oseledets, ) formats a tensor into a Matrix Product State (MPS or TT-MPS) form, which applies the Tensor-Train Singular Value Decomposition (TT-SVD) algorithm (described in Sec. A.1) to an order-$N$ tensor, $\mathcal{X}\in\mathbb{R}^{I_{1}\times I_{2}\times\cdots\times I_{N}}$. This results in $N$ smaller $2$-nd or $3$-rd order tensors, $\mathcal{G}^{(k)}\in\mathbb{R}^{r_{k-1}\times I_{k}\times r_{k}}$ for $k=1,\dots,N$, such that

$$ $\displaystyle\mathcal{X}\approx\mathcal{G}^{(1)}\times_{2}^{1}\mathcal{G}^{(2) }\times_{3}^{1}\mathcal{G}^{(3)}\times_{3}^{1}\cdots\times_{3}^{1}\mathcal{G}^ {(N)}.$ (3) $$

Tensor $\mathcal{G}^{(1)},\ldots,\mathcal{G}^{(N)}$ are referred to as the tensor cores, while the set $\{r_{0},r_{1},\ldots,r_{N}\}$ represents the TT-rank of the TT decomposition ($r_{0}=r_{N}=1$).

## 3 Methodology

Figure: Algorithm 1 TT_SVD(Oseledets, ) for a Single Token Embedding Compression

This section clarifies the technical cornerstones of our approach. A practical pipeline of our approach is depicted in Fig. 2. The whole vocabulary is processed on higher-end servers, while inference and vocabulary update happens on lower-end edge devices.

### 3.1 Individual Embedding Vector Compression

For the compression of the embedding matrix, rather than decomposing the whole embedding weight matrix, we propose to decompose each embedding vector. The lower half of Fig. 2 is a simplified illustration of such a process, with a detailed description in Alg. 1.

#### Tensorization.

Each token embedding ${\bf x}\in\mathbb{R}^{d}$ is reshaped (or folded, tensorized, as in Sec. A.1) into an order-$N$ tensor.
Denote $\texttt{reshape}(\cdot)$ as the reshape function, $\mathcal{X}=\texttt{reshape}({\bf x},\{I_{1},I_{2},\ldots,I_{N}\})$ and $\mathcal{X}\in\mathbb{R}^{I_{1}\times\cdots\times I_{N}}$ such that $d=\prod_{k=1}^{N}I_{k}$. In the example in Fig. 2, the token embedding vector ${\bf x}$ is a $27$-dimensional vector, $d=27$. In this way, vector ${\bf x}$ is reshaped into an order-$3$ ($N=3$) tensor $\mathcal{X}$, with tensor size for each mode $I_{1}=I_{2}=I_{3}=3$.

#### Tensor Decomposition.

Tensor $\mathcal{X}$ is then decomposed and stored in a Matrix Product State (MPS) form as $\mathcal{X}\approx\mathcal{G}^{(1)}\times_{3}^{1}\cdots\times_{3}^{1}\mathcal{
G}^{(N)}$, with hyperparameters as TT ranks $r_{0},r_{1},\ldots,r_{N}$. For the case in Fig. 2, the MPS cores are $\mathcal{G}^{(1)}$, $\mathcal{G}^{(2)}$, $\mathcal{G}^{(3)}$, with TT ranks $r_{0}=r_{1}=r_{2}=r_{3}=1$.
In other words, instead of storing the entire token embedding vector $\textbf{x}\in\mathbb{R}^{d}$, we store the corresponding MPS cores, $\mathcal{G}^{(k)}\in\mathbb{R}^{r_{k-1}\times I_{k}\times r_{k}}$, for $k=1,\ldots,N$. The parameter count of the MPS cores $\{\mathcal{G}^{(k)}\}$ is $\sum^{N}_{k=1}|\mathcal{G}^{(k)}|=\sum^{N}_{k=1}r_{k-1}I_{k}r_{k}$, where $|\cdot|$ represents the parameter count.

A more detailed explanation of individual token embedding compression is given in Alg. 1, and its cornerstone TT_SVD is further described in Alg. 2 (in Sec. A.1), where $\|\cdot\|_{F}$ denotes the Frobenius norm. Although the embedding vector is reshaped into a tensor, the decomposition for each mode of this tensor is still based on the matrix-level SVD (Alg. 1). Then the complexity of TT_SVD can be derived from SVD and its variants, such as truncated SVD (Oseledets, ). Given the vocabulary size $V$, the original parameters of the embedding layers are compressed from $Vd$ to $V\sum_{k=1}^{N}r_{k-1}I_{k}r_{k}$, and the compression ratio can be obtained via $\eta_{\texttt{TTD}}=\frac{d}{\sum_{k=1}^{N}r_{k-1}I_{k}r_{k}}-1$. The computation and memory complexities for all the above processes are summarized in Tab. 1.

**Table 1: Computation and memory complexity during the compression (Sec. 3.1) and inference process (Sec. 3.2) of TensorGPT. $\mathcal{M}_{\texttt{trans}}$ is the transformer module, $V$ denotes the vocabulary size, $d$ is the original token embedding dimension, and $l$ is the token number of the input text. For simplicity, the dimensions for each mode of the tensor and TT rank are represented as $I$ and $r$, respectively, which yields the highest compression ratio when $r=1$ and $I=2$ (the proof is in Appx. B).**
| Memory | Original | Compressed | Encoded Texts | Input to $\mathcal{M}_{\texttt{trans}}$ |
| --- | --- | --- | --- | --- |
| $\mathcal{O}(Vd)$ | $\mathcal{O}(VNIr^{3})$ | $\mathcal{O}(lNIr^{2})$ | $\mathcal{O}(ld)$ |  |
| Computation | TT-SVD | Reconstruction |  |  |
| $\mathcal{O}(NIr^{3})$ | $\mathcal{O}(NIr^{2})$ |  |  |  |

### 3.2 Language Model Inference Process with the Compressed Embeddings

The original inference process with embedding vectors is as follows: when the encoded texts (separated as tokens) are forwarded to the embedding layer, the embedding layer outputs the embedding vectors according to the input tokens; the embedding layer here acts like a look-up table. The embedding vectors are then forwarded to the hidden layers of the transformer, whose size is the same as the dimension of the embedding vectors. Thus, if there is no internal change in the hidden layers, the dimension of the embedding vectors should compile with the dimension of the hidden layers. So, the compressed embeddings should be reconstructed to the original dimension to enable the forwarding process. This inference happens at the application phase shown in the upper right part of Fig. 2.

Thus just before forwarding embedding vectors to the hidden layers, the memory usage increases from $l\sum_{k=1}^{N}r_{k-1}I_{k}r_{k}$ to $ld$. However, given that the vocabulary size $V$ is normally much larger than the input token number $l$, that means $V\gg l$. Thus our approach can still significantly reduce the memory usage if the embedding layer takes a significant part of the whole model parameters. The reconstruction process follows the tensor contraction in Eq. 2, turning the TT cores $\{\mathcal{G}^{(k)}\}$ into a $N$-order tensor $\mathcal{X}$ according to Eq. 3, and then vectorizing $\mathcal{X}$ into a full-size embedding vector according to Sec. 2.

## 4 Empirical Evaluation

### 4.1 Experimental Setup

#### Models, Tasks and Dataset.

The sub-billion models we used in the GPT family are DistilGPT2 (Sanh, ), GPT2, GPT2-M, GPT2-L (Radford et al., ), CerebrasGPT-111M, CerebrasGPT-256M and CerebrasGPT-590M (Dey et al., ). We also tested the models of over a billion parameters for language task performance with GPT2-XL (1.5 billion parameters) and CerebrasGPT-1.3B (1.3 billion parameters). The evaluated language tasks were language modelling and sentiment classification. For language modelling, the considered datasets are WikiText2, WikiText103 (Merity et al., ) and 1BW (Chelba et al., ). For sentiment classification, the considered dataset is IMDB (Maas et al., ).

#### Hardware.

Our main experiments were completed on a GPU workstation with an RTX A6000 48GB GPU and AMD Ryzen Threadripper PRO 5955WX CPU. The GPU resource was mainly used to fine-tune language modelling models for sequence classification, which is the requirement of the sentiment classification task. The inference latency of the low-end devices was measured on a Raspberry Pi 5, with a 64-bit Arm Cortex-A76 CPU and 8GB SDRAM.

### 4.2 Evaluation Metrics

#### Compression Ratio.

Denote $\mathcal{M}$ as a model block set containing a list of model modules like embedding layers and attention layers. With $\mathcal{M}_{0}$ as the original model block set, $\mathcal{M}_{\texttt{cmpr}}$ as the compressed version of $\mathcal{M}_{0}$, and $|\mathcal{M}|$ as the parameter count of $\mathcal{M}$. The compression ratio $\eta$ is defined as

$$ $\eta=\frac{|\mathcal{M}_{0}|-|\mathcal{M}_{\texttt{cmpr}}|}{|\mathcal{M}_{ \texttt{cmpr}}|}.$ (4) $$

Specifically, the embedding compression rate is $\eta_{\texttt{emb}}=\frac{|\mathcal{T}_{0}|-|\mathcal{T}_{\texttt{cmpr}}|}{|
\mathcal{T}_{0}|}$, where $\mathcal{T}$ only contains token embedding layer and position embedding layer.

#### Perplexity and Logarithmic Perplexity.

Perplexity is used as a performance evaluation metric of the language modelling task, which has the following form

$$ $\texttt{PPL}(S,\mathcal{M})=\left(\prod^{|S|}_{i=1}p_{\mathcal{M}}(x_{i}|x_{1} ,x_{2},\ldots,x_{i-1})\right)^{-1}$ (5) $$

where $S$ is an ordered set (token sequence), consisting of a set of tokens $\{x_{t}\}$, $t=1,2,\ldots,|S|$, and $\mathcal{M}$ is the model block that contains all the modules of the language model we evaluate.

Notice that the compression ratio Eq. 4 has a linear form, while perplexity Eq. 5 has an exponential form, so it is hard to combine them as a description of a model compression result, since when compression ratio $\eta$ linearly increases, the perplexity PPL explodes exponentially. To this end, we use the following logarithmic form to describe the language modeling performance

$$ $\ln{\texttt{PPL}(S,\mathcal{M})}=-\sum^{|S|}_{i=1}\ln p_{\mathcal{M}}(x_{i}|x_ {1},x_{2},\ldots,x_{i-1})$ (6) $$

Now, the language modelling performance change before and after compression is given by

$$ $\Delta\ln{\texttt{PPL}(S,\mathcal{M})}=\ln{\texttt{PPL}(S,\mathcal{M_{\texttt{ cmpr}}})}-\ln{\texttt{PPL}(S,\mathcal{M}_{0})}=\sum^{|S|}_{i=1}\ln\frac{p_{ \mathcal{M}_{0}}(x_{i}|x_{1},x_{2},\ldots,x_{i-1})}{p_{\mathcal{M}_{\texttt{ cmpr}}}(x_{i}|x_{1},x_{2},\ldots,x_{i-1})},$ (7) $$

observe that Eq. 7 exhibits linearity, like Eq. 4.

#### Accuracy, Precision, Recall and F1-Score.

We use these four common evaluation metrics for classification to comprehensively analyze the classification performance of the compressed model. To investigate the performance change before and after compression, we use the difference between the metric values after and before the compression.

Figure: (a) Accuracy
Refer to caption: x5.png

### 4.3 Experimental Results

#### Compression Ratio and Language Task Performance.

The language modelling performance and the compression ratio of different tensor orders and models are shown in Fig. 3. There is no general conclusion as to whether higher-order tensors are better than lower-order tensors, but roughly speaking, compression ratio and language modelling performance in Fig. 3(e)-Fig. 3(h) are better than those in Fig. 3(i)-Fig. 3(j). This implies that there may exist a high-dimensional representation in the weight of embedding layers, no more than six-dimensional and most probably around three-dimensional. On the other hand, due to the combination of tensor size and TT ranks exponentially exploding, we could not test all the possible combinations. However, we can still observe that independent of the tensor orders and the models used for the compression, significant language modelling performance loss tends to appear when the compression ratio exceeds $2.0\times$. We further compared our proposed approach with the Tucker decomposition in Fig. 3(k) with the same tensorization strategy in Sec. 3.1, and found our adopted Tensor-Train Decomposition outperforms the Tucker Decomposition in perplexity.

The results of the sentiment classification task also show that the robustness of larger-scale models (Cerebras-590M and Cerebras-1.3B) is better than that of the smaller models (Cerebras-111M and Cerebras-256M), similar to the trend in language modelling tasks mentioned above. The compressed larger-scale models tend to outperform the original model in precision and F1-score, indicating that our compression improves the ability of the larger models to recognize the positive texts. In contrast, the smaller models tends to have worse performance when the compression ratio increases.

A notable observation is that in both language modelling (Fig. 3(e), 3(f), 3(g), 3(h), 3(i) and 3(j)) and sentiment classification (Fig. 3(a), 3(b), 3(c) and 3(d)), the larger models (GPT-2-M, GPT-2-L, GPT-2-XL, CerebrasGPT-590M and CerebrasGPT-1.3B) are more robust to the compression ratio increase, compared with smaller models (DistilGPT2, GPT-2, CerebrasGPT-111M and CerebrasGPT-256M), especially when the compression ratio is less than $1.0\times$.
This is probably because the embedding layers take a smaller proportion of the entire model, and also sheds light that TensorGPT might be used to improve the language task performance for large-scale models.

#### Latency.

While TensorGPT significantly reduces the model parameters and even improves the language tasks performance, in practice it also introduced latencies - compression latency in Sec. 3.1 and inference latency Sec. 3.2.

For the compression latency, we investigated the compression latency on the token level, as shown in Tab. 2. Here, “original” means the uncompressed model, while $\texttt{PPL}_{\alpha}$ means the compressed model with a negligible task performance drop. In our case, “negligible task performance drop” means in the language modelling task, the perplexity is no more than $100.0$. The notation $\varphi_{\text{max}}$ refers to the compressed model with maximum compression ratio.
We observed that for individual token embeddings, there was no significant latency difference between high-end servers and Raspberry Pi, typically no more than 2 milliseconds for each token. Thus, it is acceptable for the Raspberry Pi to compress the individual token embeddings.

For the inference latency of a single text, we chose a typical text length of 50 tokens, as shown in Tab. 3. we used “original”, $\texttt{PPL}_{\alpha}$, $\varphi_{\text{max}}$ same as those in Tab. 2, to represent the uncompressed model, the compressed model with a negligible task performance drop and the model with a maximum compression ratio. A typically induced latency for an input text was no more than $0.3$ seconds, which is acceptable for edge applications.

**Table 2: The latency (ms/token) of tensorization & decomposition token embedding vectors and reconstruction on the high-end and lower-end devices. $\texttt{PPL}_{\alpha}$ means the compressed model with a negligible task performance drop, and the symbol $\varphi_{\text{max}}$ represents the case with a maximum compression ratio. $d_{\text{emb}}$ is the embedding dimension of the token embedding vector, and the tested models are GPT-2 and GPT-2-M. On the CPU level, for single token embedding vector decomposition and reconstruction, both server and edge devices have no significant computation overhead.**
| Device (CPU)<br>(ms/token) | $d_{\text{emb}}$ | tensorization<br>& decomposition | reconstruction |  |  |
| --- | --- | --- | --- | --- | --- |
| $\texttt{PPL}_{\alpha}$ | $\varphi_{\text{max}}$ | $\texttt{PPL}_{\alpha}$ | $\varphi_{\text{max}}$ |  |  |
| Server | 768 | 0.627 | 1.429 | 0.117 | 0.238 |
| 1024 | 0.452 | 1.512 | 0.114 | 0.261 |  |
| Raspberry Pi 5 | 768 | 0.760 | 1.948 | 0.330 | 0.468 |
| 1024 | 0.612 | 2.148 | 0.364 | 0.614 |  |

**Table 3: Parameters, number of floating-point operations (flops) of the compressed and uncompressed sub-billion models, and latency on Raspberry Pi CPU. For flops, the token number of the input texts is $100$, while for latency on Raspberry Pi, the token number is $50$.**
| GPT Models | GPT2 | CerebrasGPT |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | DistilGPT2 | GPT-2 | GPT-2-M | GPT-2-L | 111M | 256M | 590M |  |
| # Params<br>(M) | original | $81.9$ | $124.44$ | $354.82$ | $774.03$ | $111.05$ | $255.98$ | $590.31$ |
|  | $\texttt{PPL}_{\alpha}$ | $67.06$ | $106.36$ | $326.45$ | $734.28$ | $101.78$ | $226.69$ | $543.45$ |
|  | $\varphi_{\text{max}}$ | $43.45$ | $85.99$ | $303.88$ | $710.83$ | $71.87$ | $200.59$ | $511.07$ |
| flops<br>($10^{6}$/text ) | original | $20250$ | $40490$ | $142250$ | $330980$ | $14470$ | $40400$ | $103060$ |
|  | $\texttt{PPL}_{\alpha}$ | $+1.65$ | $+1.88$ | $+3.11$ | $+2.30$ | $+0.38$ | $+1.63$ | $+2.30$ |
|  | $\varphi_{\text{max}}$ | $+0.13$ | $+0.13$ | $+0.20$ | $+0.25$ | $+0.13$ | $+0.12$ | $+0.26$ |
| Latency on<br>Raspberry<br>Pi (s/text) | original | $0.19_{\pm 0.02}$ | $0.50_{\pm 0.19}$ | $1.23_{\pm 0.12}$ | $3.01_{\pm 0.47}$ | $0.47_{\pm 0.21}$ | $0.71_{\pm 0.02}$ | $1.81_{\pm 0.25}$ |
|  | $\texttt{PPL}_{\alpha}$ | $0.36_{\pm 0.19}$ | $0.50_{\pm 0.16}$ | $1.26_{\pm 0.22}$ | $3.01_{\pm 0.29}$ | $0.48_{\pm 0.23}$ | $1.01_{\pm 0.29}$ | $1.89_{\pm 0.28}$ |
|  | $\varphi_{\text{max}}$ | $0.19_{\pm 0.03}$ | $0.71_{\pm 0.38}$ | $1.55_{\pm 0.36}$ | $3.52_{\pm 0.44}$ | $0.72_{\pm 0.42}$ | $0.95_{\pm 0.27}$ | $1.91_{\pm 0.24}$ |

## 5 Related Work

### 5.1 Matrix or Tensor Factorization for Language Model Compression

Low-rank factorization can break the high-dimensional weight matrices into smaller matrices or tensors, so that the overall size of the model can be shrunk. According to the dimensions of the structure that the original weight matrices are broken into, these approaches can be divided into matrix-based and tensor-based.

#### Matrix-based Approaches.

A straightforward way to shrink the model size is to decompose weight matrices via singular value decomposition (SVD) (Acharya et al., ), which can be further improved by the weighted approach considering the model performance afterwards (Hsu et al., ), knowledge distillation (Lioutas et al., ; Mao et al., ) and pruning (Mao et al., ). There are also some block-wise decomposition approaches used in language model compression, like Kronecker Products (Tahaei et al., ; Edalati et al., ) and data-driven block-wise partitioning (Chen et al., ; ).

(Dao et al., ; ) used the block-diagonal matrices to reduce the FLOPs in the linear layers computation, with the bonus of shrinking the model size. However, our paper focuses on reducing the parameters of embedding layers, and there is no monotonous relationship between the FLOPs (computation cost) and parameters (memory usage) (Lin et al., ). Also, their investigated matrix multiplication only occurs in feed-forward layers, thus their approaches do not fit the embedding layer compression. Moreover, block-diagonal matrices are optimised for GPUs for better parallelization. Our aim of minimizing the number of parameters, makes it optimized for lower-end edge devices rather than GPUs. Indeed, on Raspberry Pi 5, the additional forwarding latency due to compressed embeddings (0.330 - 0.364ms /token in Tab. 2) is even faster than that on GPU (measured as 0.463ms /token in our setting), since there is no parallelization during this forwarding process.

#### Tensor-based Approaches.

Despite some efforts to use tensor decomposition to compress the language model size, all come with an extra training process. The works in (Abronin et al., ) use Kronecker decomposition with row-column permutation during the GPT model fine-tuning process, while (Hrinchuk et al., ) and (Chekalina et al., ) propose a tensor-train structured embedding layer and GPT model respectively, yet both train the new-structured model from scratch.

### 5.2 Tensor Network and Tensor Network Structure Search

The works in (Li et al., ; ) propose a local search technique to solve a model-agnostic objective function to optimize model complexity, with (Li et al., ) or without (Li et al., ) a simultaneous tensor shape optimization, and with image tasks (e.g. image compression and completion) as case studies. However, since they do not consider more complex functions like language modelling, which is discussed in our work, this higher complexity might drive the optimization process unsolvable.
Another branch of tensor network search involves a learning-based approach (Yin et al., ), which trains a meta-model to learn the patterns of the low-rank pattern of the model parameters.

## 6 Conclusion and Future Work

In the context of Large Language Models (LLMs) with a parameter count of less than one billion, the embedding layers take significant proportions of the total model parameters. The low-rank approach is a promising technique for parameter reduction, however, the existing low-rank approaches to compressing LLMs all involve an extra training process. Based on the Tensor-Train Decomposition, this work aims to find the low-rank structure without the need for extra training, and investigate the compression ratio, language task performance (i.e. language modelling and sentiment classification), and latency on typical low-end edge device Raspberry Pi. We have found that for the larger models (GPT-2-M, GPT-2-L, GPT-2-XL, CerebrasGPT-590M and CerebrasGPT-1.3B), the language task performance was maintained and even improved while reducing the model size. Also, we have obtained an empirical compression bound on our proposed approach, with a $0.5\times$ to $2.0\times$ compression ratio of the embedding layers, without language task performance loss.
We measured the latency of our approach on a typical low-end device (i.e. Raspberry Pi), with a significant parameter reduction, the on-device compression latency was typically no more than $2$ ms for each token, highly competent for edge applications.

There are both limitations to our work, and more importantly, rather a broad range of future work following our proposed TensorGPT. Firstly, the tensorized embedding layers do not natively compile with the hidden layers, so tensorized hidden layers are required. Also, although tensor operations, like contraction, do not require much memory, they might need more arithmetic operations on the CPU, thus requiring accelerated tensor operations. Last but not least, this work focuses on GPT-series models and reaches the “half parameters redundancy” conclusion. Extensions to other kinds of models, like the LLama family models, are the subject of future work.

## Appendix A Preliminaries

Our notation in this paper is summarized as follows:

**Table 4: Notation in this paper.**
| Symbol | Meaning |
| --- | --- |
| $a$ | Scalar |
| ${\bf x}$ | Vector |
| ${\bf A}$ | Matrix |
| $\mathcal{X}$, $\mathcal{A}$, $\mathcal{B}$ | Tensor |
| $N$ | Tensor order |
| $\mathcal{X}[i_{1},i_{2},\ldots,i_{N}]$ | The $(i_{1},i_{2},\ldots,i_{N})$th entry of the tensor |
| $I,I_{k}$ | Tensor dimension, tensor dimension for the $k$th mode |
| $\mathcal{M}$ | Model module set |
| $|\mathcal{M}|$, $|\mathcal{G}|$, $|S|$ | Parameter count of the model module set $\mathcal{M}$, tensor $\mathcal{G}$ or cardinality of set $S$ |
| $V$ | Vocabulary of the language model |
| $d$ | Token embedding dimension |
| $l$ | Input text length |
| $r$, $r_{k}$ | TT rank, TT rank of the $k$th mode of the tensor |
| $\mathcal{G}^{(k)}$ | TT(MPS) core of the $k$th mode of the tensor |
| $\times^{p}_{k}$ | Tensor contraction for the $p$th (formal tensor) and $k$th (latter tensor) mode |
| $\eta$ | Compression ratio of the entire model |
| $\eta_{\texttt{emb}}$ | Compression ratio of the embedding layer |
| $\varphi$ | Parameter reduction ratio of the whole model. |
| $\varphi_{\texttt{emb}}$ | Parameter reduction ratio of the embedding layer. |

Figure: Algorithm 2 Tensor-Train Singular Value Decomposition (TT-SVD)

### A.1 Tensors and Tensor Operations

This section gives brief mathematical preliminaries of tensor algebra, and basic knowledge in LLMs to facilitate the understanding of our proposed methodology in Sec. 3.

#### Order-N Tensor.

An order-$N$ real-valued tensor is a multi-dimensional array, denoted by a calligraphic font, e.g., $\mathcal{A}\in\mathbb{R}^{I_{1}\times\dots\times I_{N}}$, where $N$ is the order of the tensor (i.e., number of modes), and $I_{n}$ ($1\leq n\leq N$) is the size (i.e., the dimension) of its $n$-th mode. Matrices (denoted by bold capital letters, e.g., $\mathbf{A}\in\mathbb{R}^{I_{1}\times I_{2}}$) can be seen as order-$2$ tensors ($N=2$), vectors (denoted by bold lower-case letters, e.g., $\mathbf{a}\in\mathbb{R}^{I}$) can be seen as order-1 tensors ($N=1$), and scalars (denoted by lower-case letters, e.g., $a\in\mathbb{R}$) are order-0 0 tensors ($N=0$).

#### Tensor Entries.

The $(i_{1},\ldots,i_{N})$-th entry of an order-$N$ tensor is denoted by $a_{i_{1},\cdots,i_{N}}\in\mathbb{R}$, where $i_{n}=1,\ldots,I_{n}$ for $n=1,\ldots,N$. A tensor fiber is a vector of tensor entries obtained by fixing all but one index of the original tensor (e.g., $\mathbf{a}_{:,i_{2},i_{3},\ldots,i_{N}}\in\mathbb{R}^{I_{1}}$). Similarly, a tensor slice is a matrix of tensor entries obtained by fixing all but two indices of the original tensor (e.g., $\mathbf{A}_{:,:,i_{3},i_{4},\ldots,i_{N}}\in\mathbb{R}^{I_{1}\times I_{2}}$).

#### Tensorization.

A vector, $\mathbf{a}\in\mathbb{R}^{I_{1}I_{2}\cdots I_{N}}$, can be tensorized (or folded) into an order-$N$ tensor, $\mathcal{A}\in\mathbb{R}^{I_{1}\times I_{2}\times\cdots\times I_{N}}$, with the relation between their entries defined by

$$ $\mathcal{A}[i_{1},i_{2},\dots,i_{N}]=a_{i}$ (8) $$

for all $1\leq i_{n}\leq I_{n}$, where $i=1+\sum_{n=1}^{N}(i_{n}-1)\prod_{k=1}^{n-1}I_{k}$.

#### Matricization (Mode-n unfolding).

Mode-$n$ matricization of a tensor, $\texttt{mat}\left(\mathcal{A},n\right)=\mathbf{A}_{\{n\}}\in\mathbb{R}^{I_{n}
\times(I_{1}\cdots I_{n-1}I_{n+1}\cdots I_{N})}$, is a procedure of mapping the elements from a multidimensional array to a two-dimensional array (matrix). Conventionally, such procedure is associated with stacking mode-$n$ fibers (modal vectors) as column vectors of the resulting matrix. For instance, the mode-$1$ unfolding of $\mathcal{A}\in\mathbb{R}^{I_{1}\times I_{2}\times\cdots\times I_{N}}$ is represented as $\texttt{mat}\left(\mathcal{A},1\right)=\mathbf{A}_{\{1\}}\in\mathbb{R}^{I_{1}
\times(I_{2}\cdots I_{N})}$, where the subscript, $\{1\}$, denotes the mode of matricization, and is given by

$$ $\mathbf{A}_{(1)}\bigg{[}i_{1},\overline{i_{2}i_{3}\ldots i_{N}}\bigg{]}= \mathcal{A}[i_{1},i_{2},\ldots,i_{N}]$ (9) $$

Note that the overlined subscripts refer to linear indexing (or
Little-Endian), given by:

$$ $\displaystyle\overline{i_{1}i_{2}\dots i_{N}}$ $\displaystyle=1+\sum_{n=1}^{N}\Bigg{[}(i_{n}-1)\prod_{n^{\prime}=1}^{n-1}I_{n^ {\prime}}\Bigg{]}$ (10) $\displaystyle=1+i_{1}+(i_{2}-1)I_{1}+\cdots+(i_{n}-1)I_{1}\ldots I_{N-1}$ $$

#### Tensor contraction.

The contraction of an order-$N$ tensor, $\mathcal{A}\in\mathbb{R}^{I_{1}\times\dots\times I_{N}}$, and an order-$M$ tensor $\mathcal{B}\in\mathbb{R}^{J_{1}\times\dots\times J_{M}}$, over the $n$^th and $m$^th modes respectively, where $I_{n}=J_{m}$, results in $\mathcal{C}\in\mathbb{R}^{I_{1}\times\cdots\times I_{n-1}\times I_{n+1}\times
\cdots\times I_{N}\times J_{1}\times\cdots\times J_{m-1}\times J_{m+1}\times J
_{M}}$, with entries

$$ $\displaystyle c_{i_{1},\dots,i_{n-1},i_{n+1},\dots,i_{N},j_{1},\dots,j_{m-1},j _{m+1},\dots,j_{M}}=\sum_{i_{n}=1}^{I_{n}}a_{i_{1},\dots,i_{n-1},i_{n},i_{n+1} ,\dots,i_{N}}b_{j_{1},\dots,j_{m-1},i_{n},j_{m+1},\dots,j_{M}}$ (11) $$

A $(2,1)$-tensor contraction between two order-$2$ tensors, $\textbf{A}\in\mathbb{R}^{I_{1}\times I_{2}}$ and $\textbf{B}\in\mathbb{R}^{J_{1}\times J_{2}}$, where $I_{2}=J_{1}$, is equivalent to a standard matrix multiplication, $\textbf{C}=\textbf{A}\times_{2}^{1}\textbf{B}=\textbf{A}\textbf{B}\in\mathbb{R
}^{I_{1}\times J_{2}}$. Similarly, a $(2,1)$-tensor contraction between an order-$2$ tensor, $\textbf{A}\in\mathbb{R}^{I_{1}\times I_{2}}$, and an order-$1$ tensor, $\textbf{b}\in\mathbb{R}^{J_{1}}$, where $I_{2}=J_{1}$, is equivalent to a standard matrix-by-vector multiplication, $\textbf{c}=\textbf{A}\times_{2}^{1}\textbf{b}=\textbf{A}\textbf{b}\in\mathbb{R
}^{I_{1}}$.

#### Tensor-Train Singular Value Decomposition (TT-SVD)

is clarified in Alg. 2.

## Appendix B Proof of the Highest Compression Ratio in Tab. 1

Setting the tensor size $\left[I_{1},\ldots,I_{N}\right]$ for the tensor $\mathcal{X}$ to achieve the highest compression rate, we next give the proof of this hyperparameter selection.

Regarding the definition of the compression rate in Sec. 4, and $r_{0}=r_{N}=1$ in Sec. 3.1, the compression rate can be represented as

$$ $\displaystyle\eta$ $\displaystyle=\frac{V\times d}{\sum_{j=1}^{V}\sum_{n=1}^{N}(r_{n-1}\times I_{n }\times r_{n})_{j}}$ (12) $\displaystyle=\frac{V\times d}{I_{1}r_{1}+r_{1}I_{2}r_{2}+\cdots+r_{N-2}I_{N-1 }r_{N-1}+r_{N-1}I_{N}}$ (13) $\displaystyle=\frac{V\times d}{\sum_{k=1}^{\lfloor\frac{N+1}{2}\rfloor}r_{2k-1 }\left(r_{2k-2}I_{2k-1}+I_{2k}r_{2k+1}\right)}$ (14) $$

For the simplest case, assume $I_{1}=\cdots=I_{N}=I$ and $r_{1}=\cdots=r_{N}=r$. Given $d=\prod_{n=1}^{N}I_{n}=I^{N}$, we have $N=\log_{I}{D}$, and

$$ $\displaystyle\eta$ $\displaystyle=\frac{V\times d}{rI\left[2+(N-2)r\right]}$ (15) $\displaystyle=\frac{V\times d}{rI\left[2+(\log_{I}d-2)\right]}.$ (16) $$

In Equation 16, the numerator is a constant, and in the denominator, $R$ is a hyperparameter for the Tensor-Train Decomposition.
Thus the objective function for the highest compression rate $\eta$ is

$$ $\displaystyle\min_{I,N}rI\left[2+(N-2)\right]\quad\quad\textbf{s.t.}\quad$ $\displaystyle N=\log_{I}{d}$ (17) $\displaystyle I,N,r\in\mathbb{Z}^{+}$ (18) $\displaystyle 2\leq I\leq N\leq\lfloor\log_{2}d\rfloor$ (19) $$

Regarding Eq. 17, if eliminate $N$ then we have a function $h=rI\left[2+(\log_{I}{d}-2)\right]$. Regarding $d$ in Eq. 19, the largest token embedding size of recent GPT-3 (Brown, ) is 12,888. Thus, for the GPT series models no later than GPT-3, Eq. 17 should be $2\leq I\leq N\leq 13$. In this range, $h$ is a monotonically increasing function, where the minimum $h$ occurs at $I=2$.

Therefore, for the simplest case, we have the best hyperparameter selection of $I_{1}=I_{2}=\cdots=I_{N}=2$, $N=\lfloor\log_{2}d\rfloor$.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: How does your method perform with weight tying?
What is the percentage of overall compression for model sizes larger than 13B?
soundness: 2
strengths: The paper is well-structured and easy to follow, with a clear presentation of the methodology and its applications.
summary: This paper proposes using tensor-train decomposition to compress the token embedding matrix, aiming to reduce model size and accelerate inference, particularly in edge device scenarios. The technique leverages a higher-order tensor approximation method, built upon singular value decomposition (SVD), to efficiently represent embeddings while maintaining performance. This tailored approach is well-suited for resource-constrained environments, promising benefits in storage and computational efficiency.
weaknesses: **Limited Novelty:** The method largely builds upon the existing TT_SVD approach, with Algorithm 1 in this paper replicating methods already established in prior literature [1]. The methodological advancements or generalizations beyond TT_SVD appear minimal based on the methodology section.

**Unconvincing Experiments:** The experiments are conducted on older and relatively small models, such as GPT-2 and models up to only 1.3B parameters. This limits the relevance of the results, as they don’t reflect performance on contemporary large language models (LLMs). Additionally, the evaluation setup lacks modern benchmarking practices; for instance, LLM harness [2] would provide a more standardized evaluation framework.

**Unintuitive Rationale:** The foundational rationale for embedding a matrix using a tensor is unclear. The method requires reshaping the matrix into a higher-order tensor before applying tensor decomposition, but the paper does not provide an intuitive explanation for why this approach is effective or reasonable. Also, the embedding matrix only occupies a small part of the model even for a mid size of model, and a common way to reduce parameter size is through weight tie, which directly reduce the parameter sizes of embedding into half. How does you method performs in this case?

**Overstatement of Contribution:** Some claims appear overstated. For instance, the authors state, "As far as we know, we are the first to compress LLMs with low-rank factorization, specifically for low-end devices." However, the cited reference [3] already demonstrates compression of LLMs using SVD for similar purposes. Furthermore, there are numerous existing works that apply low-rank decomposition for LLM compression, such as [4-6].

**Limited Impact on Overall Model Size:** The embedding matrix occupies only a small portion of the model's parameters, even in mid-sized models. A commonly used technique, weight tying, can directly halve the embedding parameter size, offering a straightforward compression approach. The paper does not address how the proposed method compares to weight tying or performs when weight tying is already applied, raising questions about its practical impact on overall model size reduction.

[1] V. Oseledets. Tensor-train decomposition. SIAM Journal on Scientific Computing, 33(5):2295–2317,
2011. doi: 10.1137/090752286.

[2] Gao, Leo, et al. "A framework for few-shot language model evaluation." Version v0. 0.1. Sept 10 (2021): 8-9.

[3] Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. Language model
compression with weighted low-rank factorization. In International Conference on Learning
Representations, 2022.

[4] Yuan, Zhihang, et al. "Asvd: Activation-aware singular value decomposition for compressing large language models." arXiv preprint arXiv:2312.05821 (2023).

[5] Lin, Chi-Heng, et al. "MoDeGPT: Modular Decomposition for Large Language Model Compression." arXiv preprint arXiv:2408.09632 (2024).

[6] Ashkboos, Saleh, et al. "Slicegpt: Compress large language models by deleting rows and columns." arXiv preprint arXiv:2401.15024 (2024).

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 2
questions: - Pardon me that I really don't know what's the biggest contribution. Other than your claim of try things on LLM, what's the real technical contribution in your paper not discussed by previous works?
soundness: 2
strengths: - Working on a problem that's important.
summary: The authors applied tensor train to an edge device application, and found we can use that to compress some NLP models. It reads to me that the author is simply applying a pre-existing, well-known tensor decomposition method on a problem. And I didn't see any new knowledge developed or presented (I might be wrong, so hopefully the authors could point out clearly what's the real technical contribution other than yet another application).
weaknesses: - The authors claim "- As far as we know, we are the first to compress LLMs with low-rank factorization, specifically", which is not true to me. The architecture is the same for most NLP models and thus pardon me I really don't know what's added. I think all the tensor-train stuff has been developed and applied before. It's not immediately obvious to me what's new conclusions or findings drawn from this paper.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: refer to weaknesses
soundness: 2
strengths: 1.	The approach does not require extra training, making it applicable for scenarios with limited resources or when extra data is unavailable.
2.	TensorGPT achieves substantial compression with low memory overhead, retaining performance across language modeling and sentiment analysis tasks.
3.	The paper includes experiments on low-end hardware (Raspberry Pi) and larger models, evaluating compression ratios, latency, and task-specific performance.
summary: The paper presents "TensorGPT," a model compression method using Tensor-Train Decomposition (TTD) to reduce the parameter size of Large Language Models (LLMs), particularly in embedding layers, without additional training. This training-free approach preserves model performance while significantly reducing memory requirements, demonstrated on low-end devices like Raspberry Pi.
weaknesses: 1.	The novelty is limited. Tensor-Train Decomposition is explored in several language model compression works. What are the differences between the proposed method and existing works?
2.	Lack important baselines. Compressing token embedding layers is studied in some papers and the paper does not compare with them. For example, [1] and baselines used in [1] should be set as baselines.
3.	The empirical evaluation is primarily on language modeling and sentiment classification, with potential limitations in representing other complex NLP tasks, such as math reasoning and multi-hop question answering.

[1] Wang, Haoyu, et al. "LightToken: A task and model-agnostic lightweight token embedding framework for pre-trained language models." Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2023.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Could this approach be extended to show performance of other model architectures beyond the GPT family?
2. Could this approach compatible with other model compression methods? 
3. Have you investigated the effect on model robustness like in the multilingual setting using more diverse tokens?
soundness: 3
strengths: 1. Novel training-free compression method specifically targeting embedding layers. Strong practical value for edge device deployment
2. Comprehensive experiments across multiple tasks and model sizes.  Solid theoretical foundation with clear mathematical derivations.
Thorough analysis of compression vs. performance trade-offs
summary: The paper presents TensorGPT, a novel approach for compressing LLMs through tensor-train decomposition of embedding layers. The key innovation is applying tensor-train decomposition to individual token embeddings without requiring additional training data or computation. The authors evaluate their method on GPT family models (GPT-2 and CerebrasGPT), demonstrating meaningful parameter reduction while maintaining or sometimes improving model performance. The work provides comprehensive evaluations on GPT-family models and demonstrates practical applicability on edge devices.
weaknesses: 1. Limited comparison with existing compression methods and baselines. The comparison would be more comprehensive by comparing with more baselines which are train-free and trained. 
2. Evaluation focused mainly on GPT-family models and mainly focus on small models. 
3. It would be a great to combine the proposed embedding compression method with other model compression methods to check compatibility.

Meta-review: This paper proposes TensorGPT, a method for compressing small language models (SLMs) using tensor-train decomposition (TTD) of the token embedding layer. The authors claim this approach is training-free and suitable for deploying SLMs on low-end devices. The method is evaluated on GPT-2, OPT, and CerebrasGPT models with up to 1.3B parameters.

The paper addresses the important issue of SLM compression for edge devices and provides experiments on low-end hardware like Raspberry Pi. The proposed training-free compression method and consideration of energy efficiency are relevant for resource-constrained scenarios.

However, the paper's novelty is limited, as tensor-train decomposition has been used before in language model compression. The lack of comparisons to state-of-the-art baselines and the focus on relatively small models limit the generalizability and impact of the work. Additionally, the initial claims about being first to compress LLMs with low-rank factorization were overstated.

The primary reasons for rejection are the paper's failure to demonstrate significant technical novelty or empirical contributions beyond applying existing techniques to a specific use case. While the work addresses an important problem, it falls short of the level of innovation and impact expected for publication at ICLR.

Decision: Reject