Q-Sparse: All Large Language Models can be Fully Sparsely-Activated

## Abstract

Abstract We introduce, Q-Sparse , a simple yet effective approach to training sparsely-activated large language models (LLMs). Q-Sparse enables full sparsity of activations in LLMs which can bring significant efficiency gains in inference. This is achieved by applying top- K 𝐾 K italic_K sparsification to the activations and the straight-through-estimator to the training. We also introduce Block Q-Sparse for batch training and inference. The key results from this work are, (1) Q-Sparse can achieve results comparable to those of baseline LLMs while being much more efficient at inference time;
(2) We present an inference-optimal scaling law for sparsely-activated LLMs; (3) Q-Sparse is effective in different settings, including training-from-scratch, continue-training of off-the-shelf LLMs, and finetuning; (4) Q-Sparse works for both full-precision and 1-bit LLMs (e.g., BitNet b1.58 [ 29 ] ). Particularly, the synergy of BitNet b1.58 and Q-Sparse (can be equipped with MoE) provides the cornerstone and a clear path to revolutionize the efficiency, including cost and energy consumption, of future LLMs. Figure 1 : Q-Sparse achieves a superior inference-optimal scaling law than the dense models. It saves significant compute of matrix multiplication by top- K 𝐾 K italic_K sparsification of the activations.

## 1 Fully Sparsely-Activated LLMs

Large language models (LLMs) have achieved remarkable performance on a wide range of natural language processing (NLP) tasks. However, the deployment of LLMs in real-world applications is challenging due to their high computational cost and memory footprint, especially during the inference stage. To address this challenge, recent works [, , , , ] have focused on improving the efficiency of LLMs with various approaches, including quantization [, , ], pruning [], distillation [], better decoding [], and so on. One promising approach is to use sparsity to reduce the number of activated parameters in LLMs.

Sparsity contributes two factors to the efficiency of LLMs. First, sparsity can reduce the amount of computation of the matrix multiplication as zero elements are not computed. Second, sparsity can reduce the amount of input/output (I/O) that transfers the parameters between the memory and the computation units. The I/O transfer serves as the major bottleneck in the inference stage of LLMs.

One common approach to sparsity in LLMs is to use weight sparsity, which prunes the model weights to save the computation. However, unstructured weight sparsity is difficult to parallelize in GPU devices, while structured weight sparsity has a large impact to the accuracy of the model.

Another approach is to use activation sparsity, which reduces the number of activated elements in the activation tensors. Activation sparsity can be achieved by using the mixture-of-experts (MoE) mechanism [, ], modifying the activation function [, ], or predicting the position to be sparsed []. However, these approaches do not enable full sparsity of activations in LLMs, which can limit the efficiency gains during the inference stage. Moreover, compared to the dense models, the scaling laws for the sparsely-activated LLMs have not been well studied.

To explore the full potential of sparsity in LLMs, we introduce Q-Sparse, a simple yet effective approach to enable full sparsity of activations in LLMs. The major modification on LLMs is in the linear projection (i.e., matrix multiplication). As shown in Figure 1, for each linear projection, it has a top-K sparsification function that selects the top-K activations in the input tensor. For the backprogation, we use the straight through estimator to compute the gradients of the activations. We also introduce a squared ReLU function for the feed-forward layers to further improve the sparsity of the activations. Q-Sparse can be used with both full-precision and quantized LLMs. Furthermore, we introduce, Block Q-Sparse, a block sparsity implementation to make Q-Sparse compatible with batch training and inference.

To study the scaling law of sparsely-activated LLMs, we conduct a series of scaling experiments and derive an inference-optimal scaling law for sparsely-activated LLMs. We summarize the findings from the scaling experiments and the implications of the scaling law as below:

- •
The performance of the sparsely-activated models is better than the dense baselines with the same inference compute budget (i.e., activated parameters or FLOPs).
- •
As the parameters $N$ scales, the performance gap between the sparsely-activated models and the dense baselines decreases.
- •
The performance of the sparsely-activated models with around 40% sparsity ratio can match the performance of the dense baselines with the same model size and training tokens.
- •
Given the same inference budget $N_{a}$, a sparsely-activated full-precision model with a sparsity ratio of 45.58% (or $1.84N_{a}$ parameters) can achieve the best performance. For the 1.58-bit models, the optimal sparsity ratio is 61.25%.

We also conduct experiments to evaluate the effectiveness of Q-Sparse in different settings, including training-from-scratch, continue-training of off-the-shelf LLMs, and finetuning. We show that Q-Sparse can achieve results comparable to those of baseline LLMs with the same training cost while being much more efficient at inference time.

## 2 Q-Sparse

### 2.1 Architecture

The Q-Sparse architecture is based on the Transformer architecture [, ] with modifications to enable sparsity in the activations.

Top-K Sparsity

The Transformer architecture uses *nn.Linear* to perform the projection in both attention and feed-forward layers, which can be written as:

$$ $\mathbf{Y}=\mathbf{X}\cdot\mathbf{W}^{T}$ (1) $$

where $\mathbf{X}\in\mathbb{R}^{N\times D}$ is the input tensor, $\mathbf{W}\in\mathbb{R}^{M\times D}$ is the weight tensor, and $\mathbf{Y}\in\mathbb{R}^{N\times M}$ is the output tensor. The *nn.Linear* operation is equivalent to the matrix multiplication operation.

We introduce a top-K sparsity function on top of the matrix multiplication operation. The top-K sparsity function is defined as:

$$ $\mathbf{Y}=(\mathbf{X}\odot\mathbf{M})\cdot\mathbf{W}^{T}$ (2) $$

$$ $\mathbf{M}=\text{Top}_{k}(\mathbf{|X|})$ (3) $$

where $\mathbf{M}\in\mathbb{R}^{N\times D}$ is the mask tensor that indicates the top-K activations in the input tensor $\mathbf{X}$ in terms of the absolute values, $\odot$ is the element-wise multiplication operation, and $\text{Top}_{k}$ is the function that selects the top-K elements in the tensors.

To reduce the interval around zero, we re-scale the tensor by its $L_{2}$ norm after performing the top-K sparsity function.

Quantized Top-K Sparsity

Recent works [] have shown that quantization can be used to reduce the memory footprint and computational cost of LLMs without the loss of performance. We introduce a quantized version of the top-K sparsity function. The quantized top-K sparsity function is defined as:

$$ $\mathbf{Y}=(\text{Q}(\mathbf{X})\odot\mathbf{M})\cdot\mathbf{W}^{T}$ (4) $$

where $\text{Q}(\cdot)$ is the quantization function that quantizes the input tensor $\mathbf{X}$ to a 8-bit representation:

$$ $\text{Q}(X)=\text{RoundClip}(\frac{127}{\gamma+\epsilon}\mathbf{X},-128,127)$ (5) $$

$$ $\gamma=\max(|\mathbf{X}|)$ (6) $$

$$ $\text{RoundClip}(X,a,b)=\min(\max(\text{round}(X),a),b)$ (7) $$

where $\epsilon$ is a small constant to avoid division by zero, and $\gamma$ is the maximum absolute value in the input tensor $\mathbf{X}$.

Q-Sparse can be used with both full-precision and quantized LLMs. Specifically, the quantized version of Q-Sparse is compatible with 1-bit LLMs, such as BitNet b1.58 []. When using Q-Sparse with 1-bit LLMs, the quantization function is performed on the weight tensor $\mathbf{W}$:

$$ $\mathbf{Y}=(\text{Q}(\mathbf{X})\odot\mathbf{M})\cdot\text{Q}_{w}(\mathbf{W})^ {T}$ (8) $$

where $\text{Q}_{w}(\cdot)$ is the quantization function that quantizes the weight tensor $\mathbf{W}$ to a 1.58-bit representation:

$$ $\text{Q}_{w}(W)=\text{RoundClip}(\frac{\mathbf{W}}{\alpha+\epsilon},-1,1)$ (9) $$

where $\alpha$ is the mean absolute value in the weight tensor $\mathbf{W}$:

$$ $\alpha=\text{mean}(|\mathbf{W}|)$ (10) $$

Squared ReLU

To further improve the sparsity of the activations, we use the squared ReLU function [] for the feed-forward layers. The squared ReLU function is defined as $\text{ReLU}(\mathbf{X})^{2}$.

Following the LLaMA architecture, we use the gated linear unit (GLU) for the feed-forward layers. The squared ReLU function is applied with the GLU function into a ReLU^2GLU function. The ReLU^2GLU function is defined as:

$$ $\text{ReLU}^{2}\text{GLU}(\mathbf{X})=\mathbf{X}\mathbf{W}_{\text{up}}^{T} \odot\text{ReLU}^{2}(\mathbf{X}\mathbf{W}_{\text{gate}}^{T})$ (11) $$

Block Q-Sparse

While the top-k sparsification can be used in the single-sample mode, it is not friendly with the batch mode for the current GPU devices. Recent work [, ] shows that N:M sparsity, where N out of M consecutive elements to be zero, is more hardware friendly and can be used in the batch mode with an optimized GPU kernel.
To leverage this feature of the modern GPU devices, we introduce Block Q-Sparse. The key idea of Block Q-Sparse is to apply the top-K sparsity function on the activations in the block level, and the block size is set to $M$ so that there are always $M-K$ zeros out of $M$ consecutive values. The top-K sparsity function is applied to the activations in each block independently. The block level sparsity can be used to reduce the memory footprint and computational cost of the LLMs in the batch mode.

### 2.2 Training

Most of the existing works [] on training sparsely-activated models use the vanilla back-propagation algorithm to compute the gradient through the sparsity function:

$$ $\frac{\partial\mathbf{Y}}{\partial\mathbf{X}}=\frac{\partial\mathbf{Y}}{ \partial(\mathbf{X}\odot\mathbf{M})}\odot\mathbf{M}$ (12) $$

where $\mathbf{M}$ is the mask tensor that indicates the top-K activations in the input tensor $\mathbf{X}$, and $\odot$ is the element-wise multiplication operation.

The vanilla back-propagation algorithm has a limitation. It zero-outs the gradients of the non-activated elements, which can lead to the vanishing gradient problem, especially when the sparsity ratio is high. In this work, we propose to use the straight-through estimator [] to back-propagate the gradients through the sparsity function. In this way, the gradients are passed through the sparsity function without being zeroed-out. The straight-through estimator is defined as:

$$ $\frac{\partial\mathbf{Y}}{\partial\mathbf{X}}=\frac{\partial\mathbf{Y}}{ \partial(\mathbf{X}\odot\mathbf{M})}$ (13) $$

Figure: Figure 2: The average magnitude of each projection’s gradient of dense baseline, Q-Sparse with and without STE across different layers. The visualization is conducted with 300M model size on a subset of the valid set of C4 []. It shows that the gradient vanishes without STE.
Refer to caption: x4.png

We visualize the average $l2$ norm of each projection’s gradient across different layers for dense model, Q-Sparse with and without STE. We adopt top-K as 50% for Q-Sparse. Without STE, the gradient is much smaller at the bottom layers, while STE can preserve the magnitude of the gradients. As shown in Figure 2, STE estimator significantly eases the issue of gradient vanishing, especially at the bottom of the layers. We present more visualizations for each components in the Appendix A.

### 2.3 Q-Sparse for Continue-Train and Finetuning Settings

Q-Sparse can be used in different settings, including training-from-scratch, continue-training, and finetuning. In the continue-train and finetuning settings, we use the same architecture and training procedure as in the training-from-scratch setting. The only difference is that we initialize the model with the pre-trained weights and continue training with the sparsity function enabled.

For the pre-trained models that do not have the squared ReLU function in the feed-forward layers, we apply the top-K sparsity function after the activated function (e.g., SiLU) in the feed-forward layers. It can improve the sparsity of the activations without changing the model architecture.

## 3 Scaling Laws

Recent work on large language models has shown that the performance of LLMs scales with the model size and the amount of training data. [] argues that the converged performance of a dense Transformer model with $N$ parameters follows a power-law scaling law, which can be written as:

$$ $L(N)\triangleq E+\frac{A}{N^{\alpha}}$ (14) $$

where $L(N)$ is the performance of the model with $N$ parameters, $E$ is the performance of the model with infinite parameters, $A$ is a constant, and $\alpha$ is the scaling exponent. Note that the number of training tokens are fixed in this setting, which is part of the constant $E$.

In this work, we investigate the scaling law of sparsely-activated LLMs. We find that the performance of sparsely-activated LLMs also follows a power-law scaling law, which can be written as:

$$ $L(N,S)\triangleq E+\frac{A(S)}{N^{\alpha}}$ (15) $$

$$ $A(S)=B+C\exp{(\frac{\beta}{1-S})}$ (16) $$

where $L(N,S)$ is the performance of the sparsely-activated model with $N$ parameters and a sparsity ratio of $S$, and $\alpha$ and $\beta$ are the scaling exponents.

In the following part, we will introduce how we derive the scaling law and the corresponding findings.

### 3.1 Scaling Experiments and Findings

To determine the form of the scaling law of sparse-activated LLMs, we begin with a series of scaling experiments. In the experiments, we train a series of language models with Q-Sparse of various scales, ranging from 300M to 7B. The models are trained on the Redpajama dataset []. We use the Sentencepiece tokenizer from LLaMA to preprocess data. Besides Q-Sparse, we also train the dense baselines with the same datasets and settings. More details can be found in the Appendix B.

The observed losses of the sparsely-activated models and the dense baselines are shown in Figure 3. We summarize the findings as below:

- •
The performance of the sparsely-activated models scales with the model size and the sparsity ratio.
- •
Given a fixed sparsity ratio $S$, the performance of the sparsely-activated models follows a power-law scaling law with regards to the model size $N$.
- •
Given a fixed parameters $N$, the performance of the sparsely-activated models follows an exponential-law scaling law with regards to the sparsity ratio $S$.
- •
As the parameters $N$ scales, the performance gap between the sparsely-activated models and the dense baselines decreases.

According to these findings, our main hypothesis is that the performance of the sparsely-activated models follows a combination of a power-law scaling law with regards to the model size $N$ and an exponential-law scaling law with regards to the sparsity ratio $S$.

### 3.2 Power Law in the Model Size N 𝑁 N italic_N

Figure: Figure 3: The scaling curves of the sparsely-activated models regrading to the model size given a fixed sparsity ratio $S$ (Left), and regrading to the sparsity ratio given a fixed model size $N$ (Right).
Refer to caption: x5.png

With a fixed sparsity ratio $S$, the scaling law should follows []’s scaling law, which can be written as:

$$ $L(N,S)\triangleq E+\frac{A(S)}{N^{\alpha(S)}}$ (17) $$

where $\alpha(S)$ is the scaling exponent, and the scaling factor $A(S)$ is a function of the sparsity ratio $S$. Given any model size $N$, the function $L(N,S)$ should follow the Lipschitz continuity with regards to the sparsity ratio $S$. Therefore, the scaling exponent $\alpha(S)$ should be a non-decreasing function. Given any model size $N$, the function $L(N,S)$ is increasing with the sparsity ratio $S$, so $\alpha(S)$ should be a non-increasing function. Above all, the scaling exponent $\alpha(S)$ should be a constant, and the scaling function can be written as:

$$ $L(N,S)\triangleq E+\frac{A(S)}{N^{\alpha}}$ (18) $$

Figure: Figure 4: The inference-optimal scaling curves of the sparsely-activated models with full-precision (Top) and 1.58-bit (Bottom) weight. It shows that a sparisty of 45.58% for full-precision models and 61.25% for 1.58-bit models can achieve the best performance with the same inference compute budget (i.e., activated parameters or FLOPs).
Refer to caption: x7.png

### 3.3 Exponential Law in the Sparsity Ratio S 𝑆 S italic_S

According to the above finding, the performance of the sparsely-activated models follows an exponential-law scaling law with regards to the sparsity ratio $S$. Therefore, the scaling factor $A(S)$ should also follow an exponential law. Besides, given any model size $N$, the scaling function is increasing with the sparsity ratio $S$. Therefore, the scaling factor $A(S)$ should be a non-decreasing function. The scaling factor $A(S)$ can be written as:

$$ $A(S)=B+C\exp{(\frac{\beta}{1-S})}$ (19) $$

where $B$ is the scaling factor for extremely sparse LLMs, $C$ is the scaling factor for dense LLMs, and $\beta$ is the scaling exponent of the scaling factor $A(S)$ with regards to the sparsity ratio $S$.

### 3.4 Fitting the Parameters

We fit the parameters of the scaling law to the observed losses of the sparsely-activated models. We use the L-BFGS algorithm [] to minimize the Huber loss [] between the predicted and observed log loss.

$$ $\min_{E,B,C,\beta,\alpha}\sum_{\text{Runs }i}\text{Huber}_{\delta}\left(\log \hat{L}(N_{i},S_{i})-\log L_{i}\right)$ (20) $$

Following [], $\delta$ is set as $10^{-3}$. We select the best fit from a grid of initialisations around possible local optimas. $E$, $B$, $C$, $\alpha$ and $\beta$ are estimated as 1.86, 0.01, 1.89, 0.10 and 0.05, respectively.

### 3.5 Diminishing Gap between Sparsely-Activated Models and Dense Baselines

Given the above scaling law, we can derive the performance of the sparsely-activated models and the dense baselines with the same model size $N$ and the same sparsity ratio $S$. The performance gap between the sparsely-activated models and the dense baselines decreases as the model size $N$ scales. The performance gap can be written as:

$$ $\displaystyle L(N,S)-L(N,0)$ $\displaystyle=\frac{A(S)}{N^{\alpha(S)}}-\frac{A(0)}{N^{\alpha(0)}}$ (21) $\displaystyle=\frac{A(0)}{N^{\alpha}}(\frac{A(S)}{A(0)}-1)$ (22) $$

Since $\alpha$ is a constant that satisfies $\alpha>0$, the performance gap decreases as the model size $N$ scales. It means that given a large enough model size $N$, the performance of the sparsely-activated models can eventually match the performance of the dense baselines with the same model size.

### 3.6 Inference-Optimal Scaling Law

The scaling law can also be transformed into a form that is dependent on the activated parameters $N_{a}$, which reflects the effective compute (i.e., FLOPs) of the model during inference:

$$ $L(N_{a},S)\triangleq E+A(S)(\frac{1-S}{N_{a}})^{\alpha}$ (23) $$

where $N_{a}$ is the number of activated parameters in the model, which is equal to $N\times(1-S)$. Since $A(S)$ is an increasing function and $(1-S)^{\alpha}$ is
a decreasing function, there exists a sparsity ratio $S^{*}>0$ that minimizes the loss of the sparsely-activated models. This leads to the inference-optimal scaling law of the sparsely-activated models:

$$ $L(N_{a})\triangleq E+A(S^{*})(\frac{1-S^{*}}{N_{a}})^{\alpha}$ (24) $$

It shows that the performance of the sparsely-activated models is better than the dense baselines with the same inference compute budget. We further solve the optimal sparsity ratio $S^{*}$, finding that $S^{*}\approx 45.58\%$. It means that a sparsely-activated model with a sparsity ratio of 45.58% (or $1.84N_{a}$ parameters) can achieve the best performance with the same inference budget $N_{a}$.
We follow the same process to estimate the inference-optimal scaling law for 1.58-bit Q-Sparse models. We find that the optimal sparsity ratio is 61.25% (or $2.58N_{a}$ parameters). Figure 4 shows the inference-optimal scaling curves of the sparsely-activated models with full-precision and 1.58-bit weight. It shows that with the same performance, the sparsely-activated models can achieve a significant reduction in the number of activated parameters or FLOPs during inference.

The inference-optimal scaling law shows that the performance of the sparsely-activated models can be optimized by adjusting the sparsity ratio $S$. It can be used to guide the training of the sparsely-activated models and to optimize the performance of the models during inference.

Figure: (a) 700M model size
Refer to caption: x11.png

Figure: (a) 700M model size
Refer to caption: x13.png

Figure: (a) 300M model size
Refer to caption: x15.png

## 4 Experiments

We conduct experiments to evaluate the effectiveness of Q-Sparse in different settings, including training-from-scratch, continue-training of off-the-shelf LLMs, and finetuning.

### 4.1 Training-from-Scratch

#### Setting

We train a series of language models with Q-Sparse in both full-precision and 1.58 bits. The models are trained with 50B tokens on the Redpajama dataset []. We compare Q-Sparse with the dense baselines with the same datasets and settings.

#### Results

The observed losses of the sparsely-activated models and the dense baselines are shown in Figure 5. It shows that Q-Sparse with 40% sparsity ratio can match the performance of the dense baselines with the same model size and training tokens.

#### BitNet b1.58 + Q-Sparse

We further evaluate the effectiveness of Q-Sparse on 1-bit LLMs. We train a series of BitNet b1.58 models with Q-Sparse of various scales. We plot the training loss curves of both Q-Sparse and the BitNet b1.58 baseline. Figure 6 shows that the performance of the sparsely-activated BitNet b1.58 models is better than the dense baselines with the same inference compute budget. It demonstrates that Q-Sparse is compatible to 1-bit LLMs and their synergy can be used to optimize the performance of the models during inference.

#### Block Q-Sparse

We evaluate the effectiveness of Block Q-Sparse. We compare it with Q-Sparse of the same sparsity ratio.
The sparsity ratio is 50%, and the block size is set to 32 (i.e., N:M=16:32). The experiments are performed with the model sizes of 300M and 700M. The training loss curves of Q-Sparse and Block Q-Sparse are shown in Figure 7. It shows that Block Q-Sparse has a similar convergence to Q-Sparse with the same sparsity. It demonstrates that Block Q-Sparse can match the performance of Q-Sparse when training from scratch.

#### Ablation Study of top-K Sparisty and STE

To evaluate the effect of the top-K sparsity function, we compare the performance of the sparsely-activated models with the top-K sparsity function and the ReLU sparsity function. Moreover, we study the effect of the STE by comparing the models with and without STE. Figure 8 illustrates the results. It shows that either removing STE or replacing with ReLU function significantly hurt the performance. Besides, the sparsity ratio of the models with the ReLU function decreases as the training processes. In constrast, the sparisty ratio remains unchanged with the top-K sparisty function. As shown in Figure 9, we break down the contribution of the sparsity ratio from different components, finding that the decreasing sparisty is mainly from the QKV projection, the gating projection and the up projection of the feed-forward layers. This proves the superior of top-K over ReLU function.

Figure: Figure 8: The training loss curves (Left) and the overall sparsity ratio (Right) of different sparsity functions. All models are trained with 300M size and 50B tokens.
Refer to caption: x17.png

Figure: Figure 9: The sparsity ratio of each model’s component of different sparsity functions.
Refer to caption: x19.png

### 4.2 Continue-Training

#### Setting

We continue-train the Mistral 7B model [] for 40B tokens on the FineWeb-Edu dataset []. We use the Sentencepiece tokenizer from Mistral to preprocess data. We use the batch size of 4M tokens and the learning rate of 5e-5. We use the Adam optimizer with the weight decay of 0.01. More training details can be found in Appendix B.

#### Results

For a fair comparison, we continue-train the Mistral 7B model with the same recipe as the dense baseline. We compare Q-Sparse with the ReLUfication [] and dReLU Sparsification [] methods, which sparsify the model by changing the activation function. Following the origin paper [], we adopt a two-stage training strategy that first replaces the non-ReLU activation and then adds the ReLU functions. For the dReLU Sparsification method, we implement the dReLU sparsification method following the origin paper []. We evaluate these models on a range of language tasks, including ARC-Challenge [], HellaSwag [], Winogrande [], MMLU [] and TruthfulQA []. Results are shown in Table 1. It shows that Q-Sparse achieves comparable performance to the dense baseline while being much more efficient at inference time. Moreover, Q-Sparse outperforms the ReLUfication and dReLU Sparsification methods in terms of the performance and the sparsity ratio.

To break down the sparsity of each component in the model, we present the sparsity ratio of the query, key, value, output, up, down, and gate tensors in Table 2. It shows that Q-Sparse achieves a higher sparsity ratio than the ReLUfication and dReLU Sparsification methods. The sparsity ratio of the query, key, value, output, up, and down tensors is higher than 40%, and the sparsity ratio of the gate tensor is higher than 60%. It demonstrates that Q-Sparse can achieve full sparsity of activations in LLMs.

**Table 1: The results of the continue-training for Q-Sparse and the baselines on the end tasks.**
| Models | Activated | ARC | HS | MMLU | WG | TQA | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Dense Baseline | 7.0B | 61.8 | 81.4 | 59.8 | 77.5 | 42.7 | 64.6 |
| ReLUfication [] | 5.0B | 57.2 | 78.8 | 54.7 | 74.7 | 38.8 | 60.8 |
| dReLU Sparsification [] | 5.4B | 59.2 | 78.0 | 54.0 | 75.8 | 38.3 | 61.0 |
| Q-Sparse (this work) | 2.9B | 59.0 | 79.0 | 55.6 | 74.0 | 41.0 | 61.7 |
| 3.8B | 60.5 | 80.7 | 58.0 | 75.9 | 43.5 | 63.7 |  |

**Table 2: The activated parameters and the sparsity ratio of the continue-training for Q-Sparse and the baselines on the test set of Wikitext2.**
| Models | Activated | QKV | Out | Up | Gate | Down | Overall |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Dense Baseline | 7.0B | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| ReLUfication [] | 5.0B | 12.3 | 0.0 | 10.3 | 10.3 | 79.3 | 28.3 |
| dReLU Sparsification [] | 5.4B | 0.1 | 0.0 | 0.1 | 0.1 | 85.5 | 23.0 |
| Q-Sparse (this work) | 2.9B | 51.4 | 50.0 | 50.0 | 50.0 | 80.0 | 58.2 |
| 3.8B | 42.0 | 40.0 | 40.0 | 40.0 | 60.4 | 45.7 |  |

### 4.3 Supervised Finetuning

**Table 3: The results of the supervised fine-tuning for Q-Sparse and the dense baselines on the end tasks.**
| Models | Activated | ARC | HS | MMLU | WG | TQA | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Qwen1.5-4B | 3.2B | 42.8 | 68.2 | 53.6 | 67.1 | 47.9 | 55.9 |
| Qwen1.5-7B | 6.5B | 47.7 | 74.6 | 61.5 | 71.4 | 50.7 | 61.2 |
| Q-Sparse | 3.6B | 46.3 | 72.6 | 59.1 | 67.5 | 50.3 | 59.2 |
| 4.1B | 47.9 | 73.2 | 59.2 | 69.4 | 51.1 | 60.1 |  |
| Mistral-7B | 7.0B | 62.5 | 82.6 | 61.2 | 77.6 | 50.3 | 66.8 |
| Q-Sparse | 3.8B | 60.5 | 81.5 | 60.0 | 77.1 | 50.5 | 65.9 |
| 4.3B | 61.4 | 81.6 | 60.6 | 77.6 | 50.7 | 66.4 |  |

**Table 4: The results of the supervised fine-tuning for Block Q-Sparse and the dense baselines on the end tasks.**
| Models | Activated | ARC | HS | MMLU | WG | TQA | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Qwen1.5-4B | 3.2B | 42.8 | 68.2 | 53.6 | 67.1 | 47.9 | 55.9 |
| Qwen1.5-7B | 6.5B | 47.7 | 74.6 | 61.5 | 71.4 | 50.7 | 61.2 |
| Block Q-Sparse | 3.6B | 47.0 | 71.1 | 56.7 | 67.6 | 50.5 | 58.6 |
| 4.1B | 47.2 | 73.1 | 59.7 | 69.0 | 49.7 | 59.7 |  |
| Mistral-7B | 7.0B | 62.5 | 82.6 | 61.2 | 77.6 | 50.3 | 66.8 |
| Block Q-Sparse | 3.8B | 59.7 | 80.6 | 58.7 | 75.5 | 50.3 | 65.0 |
| 4.3B | 60.0 | 81.4 | 59.9 | 76.8 | 51.3 | 65.9 |  |

#### Setting

We finetune the base model of Mistral 7B [] and Qwen1.5 7B [] on Open-Orca dataset [] for both the dense baselines and Q-Sparse. The batch size is set as 128. The learning rates are selected from {3e-6, 5e-6, 7e-6}. All models are trained with 1 epoch for a fair comparison. The hyper-parameters are detailed in Appendix B. We conduct the evaluation for these models on a range of language tasks, including ARC-Challenge [], HellaSwag [], Winogrande [], MMLU [] and TruthfulQA [].

#### Results

The results are shown in Table 3. It shows that Q-Sparse with 3.6B activated parameters achieves significant better performance than the Qwen1.5 4B dense model. Moreover, Q-Sparse with around 4B activated parameters achieves comparable performance to the Mistral 7B model and the Qwen1.5 7B model. It demonstrates that Q-Sparse can be used to finetune a dense pretrained model to a much more efficient sparse model with almost no loss at accuracy.

### 4.4 Evaluation of Block Q-Sparse

#### Setting

We finetune the base model of Mistral 7B [] and Qwen1.5 7B [] on Open-Orca dataset [] for Block Q-Sparse. The block size is set as 32, which is recommended by the previous work [] on N:M sparse kernels. The other hyper-parameters are consistent with the experiments shown in Section 4.3.

#### Results

Table 4 summarizes the results for Block Q-Sparse. Similar to the results of Q-Sparse, Block Q-Sparse achieves comparable performance to the dense baselines with much fewer activated parameters. It demonstrates that Block Q-Sparse can be used for a much more efficient sparse model while supporting the batch mode.

## 5 Discussion and Future Work

Scaling BitNet b1.58 + Q-Sparse + YOCO

We have shown promising results of combining 1-bit LLMs (i.e., BitNet b1.58) and fully sparse activations (i.e., Q-Sparse). We are working on scaling up the training in terms of both model size and training tokens. Furthermore, we will incorporate YOCO [] to address the issue of KV cache for LLM inference. The integration of BitNet, Q-Sparse, and YOCO provides a comprehensive approach to optimizing all data types in LLM inference and deployment, which includes systematic optimization of model weights, activations, and KV cache.

Q-Sparse + MoE

Mixture-of-Experts has been the most widely method to achieve sparse activations in LLMs. Q-Sparse is orthogonal and can be seamlessly integrated with MoE.

## Appendix A Visualizations

Figure: (a) Query projection
Refer to caption: x21.png

## Appendix B Hyperparameters

**Table 5: Model configurations for the scaling experiments of both BitNet b1.58 and LLaMA LLM with Q-Sparse.**
| Size | Hidden Size | GLU Size | #Heads | #Layers | Seq Length |
| --- | --- | --- | --- | --- | --- |
| 300M | 1024 | 2730 | 16 | 24 | 2048 |
| 700M | 1536 | 4096 | 24 | 24 | 2048 |
| 1.3B | 2048 | 5460 | 32 | 24 | 2048 |
| 7B | 4096 | 11008 | 32 | 32 | 2048 |

**Table 6: Hyper-parameters for the scaling experiments of both BitNet b1.58 and LLaMA LLM with Q-Sparse.**
| Model | Size | Learning Rate | Weight Decay | Batch Size | Adam $\beta$ |
| --- | --- | --- | --- | --- | --- |
| BitNet b1.58 | 300M | $1.8\times 10^{-3}\rightarrow 1.5\times 10^{-3}$ | $0.1\rightarrow 0$ | 0.5M | (0.9, 0.95) |
| 700M | $1.5\times 10^{-3}\rightarrow 1\times 10^{-3}$ | $0.1\rightarrow 0$ | 0.5M | (0.9, 0.95) |  |
| 1.3B | $1.2\times 10^{-3}\rightarrow 8\times 10^{-4}$ | $0.1\rightarrow 0$ | 0.5M | (0.9, 0.95) |  |
| 7B | $1\times 10^{-3}\rightarrow 6\times 10^{-4}$ | $0.1\rightarrow 0$ | 0.5M | (0.9, 0.95) |  |
| LLaMA LLM | 300M | $6.0\times 10^{-4}$ | 0.1 | 0.5M | (0.9, 0.95) |
| 700M | $2.5\times 10^{-4}$ | 0.1 | 0.5M | (0.9, 0.95) |  |
| 1.3B | $2.0\times 10^{-4}$ | 0.1 | 0.5M | (0.9, 0.95) |  |
| 7B | $1.5\times 10^{-4}$ | 0.1 | 0.5M | (0.9, 0.95) |  |

**Table 7: Hyper-parameters for the continue-training of Mistral 7B with Q-Sparse on Findweb Edu dataset.**
| Hyperparameters | Value |
| --- | --- |
| Training updates | 10K |
| Tokens per sample | 4M |
| Adam $\beta$ | (0.9, 0.95) |
| Learning rate | 5e-5 |
| End learning rate | 1e-6 |
| Learning rate schedule | Polynomial decay |
| Warmup updates | 375 |
| Gradient clipping | 2.0 |
| Dropout | ✗ |
| Attention dropout | ✗ |
| Weight decay | 0.01 |

**Table 8: Hyper-parameters for the supervised fine-tuning of Mistral 7B and Qwen-1.5 7B with Q-Sparse on OpenOrca dataset.**
| Hyperparameters | Value |
| --- | --- |
| Training epoch | 1 |
| Batch Size | 128 |
| Adam $\beta$ | (0.9, 0.95) |
| Learning rate | {3e-6, 5e-6, 7e-6} |
| Learning rate schedule | Cosine decay |
| Warmup ratio | 0.03 |
| Dropout | ✗ |
| Attention dropout | ✗ |
| Weight decay | ✗ |

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Given that MoE models are a special type of sparsely-activated model, what are the connections and differences between the scaling law for MoE models and the scaling law proposed here?
soundness: 3
strengths: 1. The paper is well-written and easy to follow.
2. The extension of the scaling law to sparsely-activated LLMs in Section 3 is well-motivated and innovative.
3. Experimental evaluations are comprehensive, including both dense and quantized LLMs, reflecting realistic deployment settings.
summary: This paper proposes a method to train sparsely-activated large language models (LLMs) that are more efficient at inference. The authors introduce the use of squared ReLU to enhance sparsity and apply STE to address the gradient vanishing problem during the training of these sparsely-activated LLMs. Experiments are conducted on both dense and quantized models.
weaknesses: 1. MoE models are also a type of sparsely-activated model. It would be valuable to include a comparison between MoE models and the proposed sparsely-activated models in terms of accuracy and efficiency.
2. A concern lies in the actual inference speedup of the proposed method, especially during the decoding step in batched inference. In batched settings, the columns activated may vary across cases, and in the worst case, it may still be necessary to load all weights during each decoding step. Since the decoding step can be memory-bound rather than compute-bound at certain batch sizes, this could potentially limit the speedup achieved by the proposed method in practice.
3. While empirical results suggest that STE is effective, it remains unclear why gradients should be maintained for non-activated neurons. A more rigorous theoretical discussion or proof would strengthen this point.
Questions:

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: How about conducting experiments for Llama models (e.g., Llama 3 8B)? The reason why I brining it up is that Llama is more popular than Mistral and Qwen, so I believe that using Llama is more beneficial to researchers and engineers.
soundness: 2
strengths: 1. The sparsely-activated models with around 40% sparsity ratio can perform comparably to the dense baselines with the same model size and training tokens.

2. The performance gap between the sparsely-activated models and the dense baselines decreases as the number of parameters goes up.
summary: The authors introduced Q-Sparse that enables full sparsity of activations for efficient inference. They also presented an inference-optimal scaling law for sparsely-activated LLMs. They conducted experiments on commonsense reasoning tasks and MMLU.
weaknesses: 1. The paper mentioned "Q-Sparse enables full sparsity of activations in LLMs which can bring significant efficiency gains in inference", but there is neither discussion nor experiment about measuring the execution time and/or peak GPU memory usage.

2. The evaluation for math (e.g., GSM8K, MATH) and code (e.g., HumanEval, MBPP) would be required to ensure that the performance of sparsely-activated models can match the performance of the dense baselines with the same model size and training tokens. Given that  Orca’s capability is designed for open-ended generation, only the evaluation for commonsense reasoning tasks and MMLU seems not enough.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. The core principle of sparsity is not clear and is only explained by formulas, lacking a theoretical explanation.
2. The proposed sparsity method is more like a trick novelty since it is simple and applied to different components of LLM.
3. This paper has some findings and is more like an analytical paper, but what guiding significance do the experimental findings in the paper have? This can reflect the contribution of the paper to the LLM field.
4. It is recommended to provide comparative experiments with different models under different architectures on different datasets. The comparison in Table 1 is not sufficient to illustrate the effectiveness of the method.
soundness: 2
strengths: The authors present valuable works on analyzing the impact of sparsity on LLM.
summary: The authors analyzes the impact of sparsity on different components and experimental settings of LLM, and propose Q-Sparse to training sparsely activated large languge models.
weaknesses: This paper lacks of methodological clarity in terms of novelty and the structure of the paper is not clear. Moreover, methodological novelty is limited since the proposed sparsity method may be considered as trick novelty in different components of LLM.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please refer to Weaknesses
soundness: 3
strengths: i) Method. Q-Sparse is simple, yet effective. The proposed top-K sparsification could be implemented in modern compute units (e.g. CPU, GPU)

ii) Performance. Q-Sparse demonstrated superior performance compared to its vanilla counterparts with the same inference compute budget (i.e. FLOPs)

iii) Analysis. The paper provided the scaling law analysis for sparsely activated LLMs, which is potentially valuable to the community.
summary: The paper proposed Q-Sparse to enable full sparsity of activations in LLMs, aiming to improve the efficiency in inference. Different from prior works, Q-Sparse selectively activates the top-K features of each activation tensor. To stabilize the training, Q-Sparse also back-propagates the gradients of the non-activated features using straight-through estimator (STE). It was shown that Q-Sparse could be effectively applied to full-precision and 1-bit LLMs for pre-training, continued-training, and fine-tuning. The paper further presented the inference-optimal scaling law for sparsely activated LLMs, showing that they outperform their vanilla counterparts with the same inference compute budget (i.e. FLOPs).
weaknesses: The reviewer has several concerns about the efficiency evaluation.

i) It was claimed Q-Sparse "can bring significant efficiency gains in inference" (L013), "including cost and energy consumption" (L023). However, in the current paper, these claims were only supported by the metrics like activated parameters or FLOPs, which is not fully convincing. Metrics like FLOPs do not have strong correlations with real-world efficiency like wall-clock speed-up, as argued in [1]. The authors are encouraged to demonstrate the inference efficiency of Q-Sparse by metrics that are directly linked to the inference overhead, e.g. speed, throughput, peak-memory usage, energy consumption, etc, as claimed in the abstract. 


ii) The reviewer is also expecting some clarifications about the efficiency metrics used in the paper:

A) About the activated parameters. The reviewer agrees with the authors that the I/O for weight loading is one of the major bottlenecks for LLM inference (L038-040), as the weight matrices could be significantly larger than the batched activation tensors. The reviewer wonders how was the number of activated parameter calculated in the evaluation? Specifically, if the weight sharing for batched inputs was considered? This is important as for dense LLMs, the weight matrix may only be loaded once and reused for each input in the batch in GEMM. On the other hand, for sparsely activated LLMs, there were high chances that the full weight matrix may still be loaded if the activated channels, i.e. at least 40% for each input tensor, are uniformly distributed within the batch. In this case, the sparsely activated LLMs may show no I/O benefits compared to the dense LLMs.  

B) About the FLOPs. Compared to the dense LLMs, Q-Sparse introduced extra operations during inference, i.e. dynamically computing the top-K activations for each tensor. The reviewer wonders if their computational overheads were also considered in the FLOP metric. The overheads of such operations could not be ignored as they typically have O(NlogK) complexity (i.e. K >= 40%*N) and cannot be parallelized directly (i.e. reduction).


[1] FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. NeurIPS 2022.

Meta-review: While the work introduces an interesting approach for sparse activation in LLMs and explores efficiency gains, it lacks sufficient clarity and rigor in several key areas. The methodology, while claiming innovation, appears incremental, building primarily on existing sparse activation techniques. The paper also fails to provide adequate theoretical justification for critical components like the straight-through estimator (STE) and its impact on gradient flow. Without rigorous analysis, these claims remain speculative, diminishing the paper's contribution. Additionally, the experimental results lack breadth, with insufficient comparisons to established baselines like Mixture of Experts (MoE) models and limited evaluation on complex tasks.

Decision: Reject