The Super Weight in Large Language Models

## Abstract

Abstract Recent works have shown a surprising result: a small fraction of Large Language Model (LLM) parameter outliers are disproportionately important to the quality of the model.
LLMs contain billions of parameters, so these small fractions, such as 0.01%, translate to hundreds of thousands of parameters. In this work, we present an even more surprising finding: Pruning as few as a single parameter can destroy an LLM’s ability to generate text – increasing perplexity by 3 orders of magnitude and reducing zero-shot accuracy to guessing.
We propose a data-free method for identifying such parameters, termed super weights , using a single forward pass through the model.
We additionally find that these super weights induce correspondingly rare and large activation outliers, termed super activations . When preserved with high precision, super activations can improve simple round-to-nearest quantization to become competitive with state-of-the-art methods. For weight quantization, we similarly find that by preserving the super weight and clipping other weight outliers, round-to-nearest quantization can scale to much larger block sizes than previously considered.
To facilitate further research into super weights, we provide an index of super weight coordinates for common, openly available LLMs 1 1 1 Code is available in https://github.com/mengxiayu/LLMSuperWeight . .

## 1 Introduction

Large Language Models (LLMs) have been growing in size and capability at an unprecedented rate, enabling them to capture increasingly complex linguistic patterns across a wide range of tasks. However, with this increase in model scale, new and unexpected behaviors have emerged.
Dettmers et al. () discovered that once LLMs reach a certain scale, a small set of hidden state features contains outliers of exceptionally large magnitude. These outliers account for a small percentage of all activations but are crucial for preserving the compressed model’s quality (Dettmers et al., ; Xiao et al., ; Wei et al., ; Shao et al., ).

However, not all outliers are equally important.
In this paper, we study a tiny yet important set of outliers in LLMs, termed *super weights*. In Llama-7B, pruning the super weight, a single scalar, completely destroys the model’s ability to generate text; the average accuracy of zero-shot downstream tasks effectively plummets to zero. Conversely, pruning the other top 7,000 outliers, including outliers that are larger than the super weight, affects no more than a few percentage points.

Intriguingly, super weights behave similarly across model families and sizes. For one, the super weight is always found in the mlp.down_proj weight, always in an early layer. We also find that the super weight amplifies input activation inliers to ultimately produce the exceptionally large magnitude activation observed by Sun et al. () – we term this the super activation. This super activation persists throughout the model at exactly the same magnitude and position regardless of the prompt, and we find this is uniquely enabled by skip connections. Finally, super weights suppress stopword likelihood. Taken together, pruning the super weight destroys quality by dampening the super activation and shifting almost all logit probability mass to stopwords.

Both super weights and super activations, which we collectively refer to as super outliers, are critical to model quality. Fortunately, there are no more than a handful of scalar super outliers per tensor; in light of this, we revisit round-to-nearest quantization, equipped only with the ability to hold out and restore super outliers. This yields a data-free, hardware-friendly method. For activation quantization, we find this technique competitive with SmoothQuant; for weight quantization, we can scale round-to-nearest to much larger block sizes with higher quality.

Our contributions are summarized as follows.

- 1.
Super Weights: We discover a tiny subset of outliers in LLMs, at most six scalars, that are disproportionately important; pruning these super weights destroys model quality.
- 2.
Identifying Super Weights: We present a data-free way to identify super weights using only a single forward pass and provide an index of super weights for existing, open LLMs.
- 3.
Super Activations: We analyze how super weights influence inference and relate them to the activation outliers observed in prior work.
- 4.
Compression: By preserving super outliers, we show that round-to-nearest quantization increases effectiveness noticeably; preserving super outliers improves compression quality.

Figure: Figure 1: Super Weight Phenemenon. We discover that pruning a single, special scalar, which we call the super weight, can completely destroy a Large Language Model’s ability to generate text. On the left, the original Llama-7B, which contains a super weight, produces a reasonable completion. On the right, after pruning the super weight, Llama-7B generates complete gibberish. As we show below, this qualitative observation has quantitative impact too: zero-shot accuracy drops to guessing and perplexity increases by orders of magnitude.
Refer to caption: x1.png

## 2 Related Work

### 2.1 Outliers in LLMs

LLM outliers are widely observed in existing literature.
Kovaleva et al. () notes weight outliers, which emerge gradually, beginning early in pre-training, and cause abnormal spikes at select dimensions in the output embedding vectors. Disabling those outliers significantly degrades both the training loss and the downstream task performance.
Bondarenko et al. () notes activation outliers, which encourage specific attention patterns, such as attending to the special separator token.
However, Sun et al. () first observes an exceptionally extraordinary outlier; in particular, they discover massive activations in LLMs that persist across layers in a fixed position, which Yang et al. () hypothesizes is caused by gated linear units (GLU) and its variants, such as GEGLU and SwiGLU. To mitigate these massive activations, Sun et al. () proposes a learnable attention bias, and (Son et al., ; Yang et al., ) inserts certain prefixes.
To complement these mitigation studies, our focus is instead to leverage, rather than mitigate, these super activations.

### 2.2 Outlier-aware quantization methods

Quantization is one of the most popular techniques for reducing LLM resource consumption. However, quantizing LLMs is non-trivial, due to outliers that increase the range of values. Existing works typically study two settings for LLM quantization: (1) Weight-only quantization, where only weights are quantized into low-bit integers; (2) Weight-activation quantization, where both activation and weights are quantized.

For weight-only quantization, several common solutions including using smaller block sizes, to limit the number of values any single outlier can impact (Dettmers et al., ; Shao et al., ; Dettmers & Zettlemoyer, ; Frantar et al., ; Dettmers et al., ); scaling sensitive weights, via a grid-searched channel-wise scaling, Lin et al. (); or clipping outliers via learned optimal thresholds (Shao et al., ; Lin et al., ). The most common approach is to extract and store sensitive weight outliers in higher-precision (Dettmers et al., ; Kim et al., ; Dettmers et al., ). However, decomposed, mixed-precision arithmetic for hundreds of thousands of weights is unfriendly for hardware and incurs significant latency penalties. We take a different approach, handling at most a half dozen scalars to maintain hardware friendliness.

For activation quantization, there are an increased number of even more aggressive outlier values, making activation quantization more challenging. To tackle this, previous work rotates (Liu et al., ; Ashkboos et al., ; Chee et al., ),
clips (Wei et al., ) or shifts (Wei et al., ; Shao et al., ) activations to mitigate activation outliers. One effective approach scales activations (Xiao et al., ), migrating the difficulty of quantization from activations to weights with a mathematically equivalent transformation. However, this method – SmoothQuant – requires calibration data to find the optimal hyperparameters. We show a competitive alternative that is alternatively data-free, with a small change to a naive round-to-nearest method.

Recent studies have discovered that activation outliers are associated with weight outliers. The hidden dimensions where activation outliers emerge have a high correlation to sensitive weight channels (Heo et al., ; Lee et al., ). Along these lines, activation magnitudes have been used as an indicator to find salient weight channels to preserve in weight quantization (Lin et al., ). We find the relationship between activations and weights is even more striking: Rather than channel-wise pairs, we find relationships between two individual scalars – up to six weights and one activation.

## 3 Super Weights

Many studies corroborate the importance of weight outliers, showing that a small percentage of the largest magnitude outliers are essential to model quality. This percentage can be as small as 0.01%, but for these billion-parameter models, 0.01% can still include hundreds of thousands of weights. Our investigation reveals a surprising fact about even this group of weight outliers: There exists a single, scalar weight that, despite not being the largest, holds more importance than thousands of other outlier weights combined. We call this single scalar weight a super weight.

In our analysis, we find that the super weight is necessary for quality, having an outsized influence on quality if removed. Without the super weight, LLMs fail to generate text, resulting in qualitatively (Figure 1) and quantitatively (Table 1) gibberish responses. In particular, zero-shot dataset accuracy is reduced to guessing, and perplexity increases by several orders of magnitude (Prune SW). To quantify this influence, we prune all other outlier weights (Prune Non-SW), comparing the impact of a single super weight against 7,000 other outliers. Remarkably, the accuracy drop associated with pruning this single weight is much greater than the effect of all other outliers *combined*.

**Table 1: Super Weight Importance. (Section 3) Prune SW: Pruning the single, scalar-valued super weight significantly impairs quality – reducing accuracy on zero-shot datasets and increasing perplexity by orders of magnitude. Prune Non-SW By contrast, retaining the super weight and instead pruning the other 7,000 largest-magnitude weights marginally affects quality. In other words, a single super weight is more important than even the top 7,000 largest weights combined. (Section 3.2) Prune SW, +SA: Pruning the super weight but restoring the super activation partially recovers quality. Note that quality is still drastically impaired however, so we conclude that super activations only partially explain how super weights operate. This also shows that super weights and super activations both need special handling, to preserve quality.**
| Llama-7B | Arc-c | Arc-e | Hella. | Lamb. | PIQA | SciQ | Wino. | AVG | C4 | Wiki-2 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Original | 41.81 | 75.29 | 56.93 | 73.51 | 78.67 | 94.60 | 70.01 | 70.11 | 7.08 | 5.67 |
| Prune SW | 19.80 | 39.60 | 30.68 | 0.52 | 59.90 | 39.40 | 56.12 | 35.14 | 763.65 | 1211.11 |
| Prune Non-SW | 41.47 | 74.83 | 56.35 | 69.88 | 78.51 | 94.40 | 69.14 | 69.22 | 7.57 | 6.08 |
| Prune SW, +SA | 26.60 | 54.63 | 56.93 | 12.79 | 67.95 | 61.70 | 70.01 | 50.09 | 476.23 | 720.57 |

### 3.1 Identification of Super Weights

##### Super weights create super activations.

Sun et al. () first discover a handful of exceptionally massive activations, which are crucial to model quality. Massive activations persist across many layers, feature constant magnitude, and always exist at the same position, regardless of input.
We find a further intriguing property: The activation’s channel aligns with our super weight’s, and the activation first appears right after our super weight. To confirm whether this is correlation or causation, we prune the super weight and check the massive activation’s magnitude. Per Figure 4, we discover that pruning the super weight drastically reduces the massive activation’s magnitude. This suggests that the massive activations are created by super weights. For consistency, we dub these massive activations “*super activations*”.

With further investigation, we reveal the mechanism of super weights and super activations. Sun et al. () explained super activations as bias terms, but they did not explain how super activations are created and why they are always in the same positions. Through empirical analysis, we find that before down projection, the Hadamard product of the gate and up projection creates a relatively large activation, which aligns with the findings of Yang et al. (). More importantly, the super weights further amplify it and create super activations.

##### Identifying super weight by activation spikes.

Based on the above analysis, we present an efficient way to locate super weights: SWs can be located by detecting the spikes in the down_proj inputs and outputs distributions across the layers. This detection only requires a single input prompt, rather than a set of validation data or use-case examples.

Suppose that we have a down_proj weight matrix $\mathbf{W}\in\mathbb{R}^{D\times H}$, where $D$ is the dimension of the activation feature and $H$ is the intermediate hidden dimension. Let $\mathbf{X}\in\mathbb{R}^{L\times H}$ be the input matrix, where $L$ is the sequence length.
$\mathbf{Y}=\mathbf{X}\mathbf{W}^{T}$, where $\mathbf{Y}_{ij}=\sum_{k=1}^{d}\mathbf{X}_{ik}\mathbf{W}_{jk}$. Suppose $\mathbf{Y}{ij}$ is a super activation. If $\mathbf{X}{ik}$ and $\mathbf{W}{jk}$ are both outliers that are much larger than other values, $\mathbf{Y}{ij}$ will be dominated by their product. That is, $\mathbf{Y}{ij}$ $\approx$ $\mathbf{X}{ik}$ $\mathbf{W}{jk}$. In this case, $j$ and $k$ are determined by $\mathbf{X}{ik}$ and $\mathbf{Y}{ij}$. Therefore, we start by plotting extreme outliers in the input and output activations of mlp.down_proj. Then, we determine the layer and coordinates of the super weight, as illustrated in Figure  4. Once we have detected one super weight, we remove it from the model and repeat the above process, until the magnitudes of large maximum activations are greatly suppressed.

We have identified super weights for commonly available LLMs across different LLM families and model sizes, presented in Table 2. Most of the models we have examined have no more than three super weights.
The model with the most super weights, i.e., Phi-3-mini-4k-instruct, contains six.
We have also examined the instruction-finetuned models, such as Mistral-7B-Instruct-v0.1 and Llama-2-7B-chat. We find that their super weights are located at the same coordinates as the pre-trained models, which suggests that instruct fine-tuning does not change the position of super weights.

**Table 2: Super Weight Directory. The above layer numbers, layer types, and weight types can be directly applied to Huggingface models. For example, for Llama-7B on Huggingface, access the super weight using layers[2].mlp.down_proj.weight[3968, 7003].**
| Model | No. | Type | Weight | Coordinates |
| --- | --- | --- | --- | --- |
| Llama 7B | 2 | mlp | down_proj | [3968, 7003] |
| Llama 13B | 2 | mlp | down_proj | [2231, 2278] |
| 2 | mlp | down_proj | [2231, 6939] |  |
| Llama 30B | 3 | mlp | down_proj | [5633, 12817] |
| 3 | mlp | down_proj | [5633, 17439] |  |
| 10 | mlp | down_proj | [5633, 14386] |  |
| Llama2 7B | 1 | mlp | down_proj | [2533, 7890] |
| Llama2 13B | 3 | mlp | down_proj | [4743, 7678] |
| Mistral-7B<br>v0.1 | 1 | mlp | down_proj | [2070, 7310] |

Figure: Figure 2: How Super Weights behave. I: Super weights are often found in an early layer’s down projection, indicated with a blue-purple box. The super weight immediately creates an incredibly large-magnitude super activation. II: Super activations are propagated through skip connections, indicated with blue-purple lines. III: This has a net effect of suppressing stopword likelihoods in the final logits. Removing the super weight causes stopword likelihood skyrocket, indicated with the blue-purple stacked bars. See Appendix A.3.
Refer to caption: x2.png

Figure: Figure 3: How to identify the Super Weight for Llama-7B. down_proj input features a large maximum-magnitude activation only in Layer 2, where the super activation first appeared. The value’s channel index, e.g., 7003, tells the row of SW. down_proj output likewise features a large maximum-magnitude activation at Layer 2. This value’s channel index, e.g., 3968, gives us the column of the SW.
Refer to caption: x3.png

### 3.2 Mechanisms of super weights

We find that super weights (1) induce super activations, which have lasting effects throughout the entire model, and (2) suppress stopword likelihood (Figure 2).

##### Super weights (partially) operate via super activations.

To assess whether the super weight’s impact on model quality is solely mediated by the super activations or also by activations of other tokens, we conducted an experiment involving the removal of super weights (SW) and restoration of super activations (SA). Note that a super weight should influence the same channel for all tokens.

We conduct an ablation experiment with three conditions: (1) the original model, (2) remove the super weight (Prune SW), i.e., setting the weight scalar as zero, (3) remove the super weight and restore the super activation at the layer where it first appears (Prune SW,+SA). The third condition allows us to isolate the impact of super weights on super activations only.

Results are shown in Table 1. Specifically, when we restore the super activations, the average accuracy recovers to 49.94 from 35.14, indicating that the restoration of super activations salvaged approximately 42% of the quality loss. These findings suggest that while the super activations contribute substantially to the model’s performance, they do not fully account for the super weight’s overall influence on quality.

##### Super weights affect output token probability distributions.

We studied the impact of super weights with respect to the output token probability distribution, averaged over 500 prompts from Lambaba validation set. We find that when super weights are removed, the stopword probabilities are amplified, e.g., with Llama-7B, the probability of “the” is amplified by around 2$\times$, “.” by 5$\times$, and “,” by 10$\times$ (Figure 5, Appendix Figure 11).

To dive deeper on how SW impact the output token distribution, we conduct a case study with a prompt “Summer is hot. Winter is ”. The correct next token should be “cold”, which is a word that has strong semantic meaning. With the original model with SW, it correctly predicts the next token “cold” with a high probability 81.4%. However, when the SW is removed, the model’s top prediction is a stopword “the” with a non-confident low probability of 9.0%. This indicates that the SW is essential for the model to make a correct and confident prediction of meaningful words.

Figure: Figure 5: Super weights suppress stopwords. Above, we consistently observe that removing super weights results in 2-5$\times$ larger stopword probabilities, across a variety of LLMs. At the same time, we observe non-stopwords decrease sharply in probability, reducing by 2-3$\times$ to as little as 0.1% probability. Overall, this results in stopwords dominating the highest likelihood tokens.
Refer to caption: x6.png

##### Sensitivity of super weights.

We aim to illustrate how variations in the magnitude of super weights impact the model’s quality, especially, how does increasing the magnitude affect model quallity. We multiply the super weights by a scaling factor ranging from 0.0 to 3.0. Results in Figure 6 show that amplifying super weights can improve model accuracy, to some extent. See full versions of these plots, for more models and all datasets, in Appendix A.1.

Figure: Figure 6: Amplifying super weight improves quality. Across model sizes, we consistently observe that there exists some scaling where quality is improved. Although the quality improvement is miniscule, a consistent and noticeable trend is surprising, given we’re changing only one scalar out of billions. The purple line is the original model’s zero-shot average accuracy.
Refer to caption: x9.png

## 4 Super-outlier Aware Quantization

Quantization is a powerful technique for compressing models and reducing memory requirements. However, the presence of outliers can significantly degrade quantization quality, for both weight quantization and activation quantization. As we mentioned before, we refer to these problematic outliers, both super weights and super activations, as super outliers.
As we have shown above, these super outliers carry disproportionate importance for model quality, making their preservation during quantization critical. Quantization generally maps continuous values to a finite set of values; we consider one of the simplest forms – namely, asymmetric round-to-nearest quantization:

$$ $Q(\mathbf{X})=\text{Round}\left(\frac{\mathbf{X}-\textsc{min}(\mathbf{X})}{ \Delta}\right),Q^{-1}(\mathbf{\hat{X}})=\Delta\cdot\mathbf{\hat{X}}+\textsc{ min}(\mathbf{X})$ $$

where $\Delta=\frac{\textsc{max}(\mathbf{X})-\textsc{min}(\mathbf{X})}{2^{N-1}-1}$ is the quantization step and $N$ is the number of bits. Note that the maximum value is used to calculate $\Delta$, so super outliers in $X$ drastically increase the step size. With larger step sizes, inliers are rounded to more distant values on average, increasing the quantization error. With increasingly super outliers, inliers are rounded to fewer discrete values, and more quantization bins remain unused. In this way, super outliers cause poor quantization fidelity.

We specifically consider the case where hardware performs arithmetic in half precision, meaning the tensor $X$ is quantized and dequantized before usage; in this setting, we can leverage prior knowledge of super outliers in two ways. First, hold out the super outlier to prevent adverse effects on inlier quantization. Second, restore the super outlier’s value after dequantization, to ensure the super outlier’s effects are preserved. We adopt this insight in two forms below, for weights and activations.

### 4.1 Activation Quantization

We conduct experiments using round-to-nearest quantization, with a small modification – replace the super activation with the median value (Replace), quantize ($Q$) and dequantize ($Q^{-1}$) activations, then restore the super activation in FP16 (Restore). This can be expressed as the following,

$$ $\hat{A}=\textsc{Restore}(Q^{-1}(Q(\textsc{Replace}(A)))$ (1) $$

Since the super activation is a single scalar, the bitrate and kernel complexity are not significantly impacted.

### 4.2 Weight Quantization

Prior art uses (Dettmers et al., ; Lin et al., ) small group sizes of 64 or 128, as Dettmers & Zettlemoyer () finds that small group sizes are required for precise low-bit quantization. However, the small group sizes come with computational and bitrate overhead, requiring other techniques to handle a high number of half precision scales and biases.

To address this challenge, we propose a simple method to improve INT4 quantization with large blocksizes. First, we identify super weights using Section 3.1. Second, to improve inlier fit, we clip (Clip) the outlier weights; in this step, the super weight is clipped as well. Quantize ($Q$) and dequantize ($Q^{-1}$) the clipped weights. Then, to ensure the effect of the super weight is preserved, we restore the half-precision super weight after dequantization (Restore).

$$ $\hat{W}=\textsc{Restore}(Q^{-1}(Q(\textsc{Clip}_{z}(W)))$ (2) $$

As described in the equation above, we parameterize clipping using a z-score. Assuming all weights fit a Gaussian, we consider all values with a z-score beyond a certain threshold $z$ to be an outlier. To tune this hyperparameter $z$, we find the minimum reconstruction error z-score using 500 examples from the Wikitext-2 train set.

## 5 Experiments

**Table 3: Round-to-nearest with super-activation handling is competitive. W8A8 is the baseline 8-bit weight and activation quantization, and the small italicized, parenthesized percentages denote what percentage of SmoothQuant’s quality improvement is retained. We observe that a naive round-to-nearest, while handling a single scalar super activation per tensor, is competitive with SmoothQuant. Note that SmoothQuant uses calibration data to compute scales, whereas our method does not require data.**
| PPL ($\downarrow$) | Llama-7B | Llama-13B | Llama-30B |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Wiki-2 | C4 | Wiki-2 | C4 | Wiki-2 | C4 |
| FP16 | 5.68 | 7.08 | 5.09 | 6.61 | 4.10 | 5.98 |
| Naive W8A8 | 5.83 (0%) | 7.23 (0%) | 5.20 (0%) | 6.71 (0%) | 4.32 (0%) | 6.14 (0%) |
| SmoothQuant | 5.71 (100%) | 7.12 (100%) | 5.13 (100%) | 6.64 (100%) | 4.20 (100%) | 6.06 (100%) |
| Ours | 5.74 (75%) | 7.14 (82%) | 5.15 (71%) | 6.66 (71%) | 4.22 (83%) | 6.08 (75%) |

**Table 4: Handling the super activation improves activation quantization. Perplexity $\downarrow$ on Wikitext-2 and C4 for OLMo models and various quantization methods. We can see that simply restoring the scalar-valued super activation after quantizing and dequantizing successfully improves quantization’s effectiveness at preserving quality. Notably, note that SmoothQuant does not work on OLMo, as its LayerNorms do not have adjustable parameters. See more results in Appendix A.4**
| PPL ($\downarrow$) | OLMo-1B | OLMo-7B | Mistral-7B |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Wiki-2 | C4 | Wiki-2 | C4 | Wiki-2 | C4 |
| FP16 | 10.12 (100%) | 12.31 (100%) | 7.51 (100%) | 9.52 (100%) | 5.25 (100%) | 7.75 (100%) |
| Naive W8A8 | 10.79 (0%) | 12.84 (0%) | 8.70 (0%) | 10.41 (0%) | 5.32 (0%) | 7.83 (0%) |
| Ours | 10.23 (84%) | 12.52 (60%) | 7.80 (76%) | 9.72 (78%) | 5.31 (14%) | 7.81 (25%) |

To comprehensively demonstrate the effects of super weights, we conduct experiments across LLaMA 7B to 30B, (Touvron et al., ), Mistral 7B (Jiang et al., ), and OLMo (Groeneveld et al., ) (^2^22For OLMo, we use their latest Huggingface checkpoints, e.g., allenai/OLMo-7B-0724-hf.) To assess the practical application capabilities of LLMs, we evaluate their accuracy on zero-shot benchmarks, including PIQA (Bisk et al., ), ARC (Clark et al., ), HellaSwag (Zellers et al., ), Lambada (Paperno et al., ), and Winogrande (Sakaguchi et al., ). We use the lm-evaluation-harness (Gao et al., ) library to evaluate the above tasks. We also calculate the perplexity for Wikitext-2 (Merity et al., ) and C4 (Raffel et al., ), following the widely accepted setting from (Frantar et al., ).

### 5.1 Activation Quantization

Following SmoothQuant’s setting, we simulate W8A8 quantization with FP16 arithmetic. Specifically, we perform 8-bit per-tensor quantization for weights, and 8-bit per-token quantization for activations. We quantize the inputs and weights for linear layers (including q, k, v, gate, up, down projections), and BMM (i.e., batched matmul) in attention layers. For SmoothQuant, we use the default $\alpha$ as 0.85.

We compare our method with SmoothQuant in Table 3. For three Llama models on both datasets, we achieve over 70% of SmoothQuant’s improvement over naive quantization. On C4 with Llama-7B and on Wikitext with Llama-30B, we attain above 80% of SmoothQuant’s improvement. Our method demonstrates that a significantly simplified approach to quantization can achieve competitive results compared to more complex methods. Unlike SmoothQuant, which applies scales to every activation channel, our method focuses solely on addressing one critical activation outlier.

We extended our evaluation to include additional LLMs: OLMo (1B and 7B), Mistral-7B, and Llama-2-7B. Results are shown in Table 4 and Appendix Table 7. These models represent a diverse set of architectures and training paradigms, allowing us to assess the generalizability of our quantization method. Since SmoothQuant does not report on this set of models, we compare our results with naive W8A8 quantization. Across all models and datasets, our method consistently outperforms naive W8A8 quantization.
Our method demonstrates remarkable performance on OLMo models.

Notably, OLMo models use non-parametric LayerNorm, making them incompatible with the SmoothQuant method, which relies on LayerNorm weights to apply the per-channel scales. On Mistral-7B, the improvements are smaller. We hypothesize that this is because the LayerNorm of these models may have learned weights that aggressively suppress the super activation, resulting in a more uniform distribution of activation magnitudes.

These results underscore the critical importance of the super activation in maintaining model performance during quantization. By addressing this single activation with minimal computational overhead, our method captures a significant portion of the benefits achieved by more complex quantization schemes. This finding suggests that the super activation plays a disproportionately large role in preserving model quality during the quantization process.

### 5.2 Weight Quantization

Recent advancements in LLM quantization techniques have inadvertently highlighted the importance of super weights. Two notable methods, AWQ (Lin et al., ) and SqueezeLLM (Kim et al., ), demonstrate the significance of preserving these super weights, albeit through different approaches.

#### 5.2.1 Existing Workarounds for the Super Weight

AWQ, recognizing the need to minimize quantization errors for important weights, introduced a per-channel scaling method. This technique automatically searches for optimal scaling factors, effectively amplifying crucial weight channels. Our analysis of Llama-7B revealed that AWQ scales up the super weight by a factor of 12, corroborating our assessment of the super weight’s importance.
Similarly, SqueezeLLM proposes a sparse matrix approach that retains the top 0.05% of outlier values in FP16 precision. Our investigation confirmed that this sparse matrix consistently includes the super weights, further validating their importance. Both AWQ and SqueezeLLM, despite employing different strategies, converge on the same principle: protecting super weights is crucial for effective weight quantization in LLMs.

#### 5.2.2 Scaling up Block Sizes

To evaluate the effectiveness of the proposed super weight-aware quantization method, we compare it with the traditional round-to-near quantization approach. We assess the models on a suite of zero-shot downstream tasks, with results illustrated in Figure 7.

In the traditional round-to-near quantization, we observe a clear trend: as the block size increases, model quality significantly degrades. This decline likely results from the increased quantization error introduced when larger blocks of weights are quantized together, which allows outliers to impact more weights. In contrast, our super weight-aware quantization method demonstrates much greater robustness to larger block sizes. As the block size increases, the degradation in model quality is noticeably smaller compared to the round-to-near method. This robustness stems from our method’s ability to preserve the most critical weight (the super weight) while minimizing the influence of outlier weights on the overall quantization process. By clipping outliers and focusing on inlier weights, our method maintains higher fidelity in representing the model’s parameters.

A key advantage of our method is its ability to support larger block sizes with less loss in model quality. This capability leads to a lower average bitrate and smaller file sizes, which are essential for deploying models in resource-constrained environments, such as mobile devices or edge computing scenarios.

Figure: Figure 7: Restoring super weight improves block scaling. Smaller block sizes are often used to handle outliers implicitly. We note that block sizes can scale slightly more gracefully by just handling the single scalar-valued super weight.
Refer to caption: x12.png

## 6 Conclusion

Our study of Large Language Models has revealed the critical role of super outliers – specifically, the super weight and its induced super activation. Although these super outliers are small in number, identifying and preserving them is essential for model quality; pruning the super weight completely destroys the model’s ability to generate text, and retaining the super weight can significantly improve the quantized model’s quality.

Our findings shed light on how these outliers influence model behavior and provide practical strategies for their detection and management. By sharing a directory of super weights, we furthermore hope to inspire further research into their properties and implications.

## Appendix A Appendix

### A.1 Super Weight Sensitivity

In this section, we show the full set of results on zero-shot downstream datasets. Results are shown in Table 5 for FP16 models. From the table, we can see that some datasets are more sensitive to super weight (SW) amplification. For example, Winogrande and Lambada show consistent improvements across models when amplifying SW, while PIQA shows same or slightly lower accuracy. We also evaluate the 4-bit quantized Llama-7B with amplified SW in Table 6. We witness similar minor improvements when SW is amplified.

**Table 5: Accuracy of zero-shot benchmarks of amplifying super weights in FP16 models. The scaling factor is chosen by the highest average accuracy.**
|  | Llama-7B | Llama-13B | Llama-30B | Mistral-7B |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Original | Amplified | Original | Amplified | Original | Amplified | Original | Amplified |
| ARC-C | 41.81 | 41.89 | 46.42 | 46.76 | 52.82 | 52.47 | 50.25 | 49.74 |
| ARC-E | 75.29 | 75.63 | 77.36 | 77.19 | 80.39 | 80.51 | 80.89 | 81.02 |
| Hella. | 56.93 | 56.76 | 59.96 | 59.82 | 63.34 | 63.21 | 61.23 | 61.39 |
| Lamb. | 73.51 | 74.09 | 76.15 | 76.58 | 77.59 | 77.68 | 75.62 | 75.92 |
| PIQA | 78.67 | 78.67 | 79.11 | 78.94 | 80.96 | 81.28 | 80.73 | 80.47 |
| SciQ | 94.60 | 95.40 | 95.00 | 95.30 | 96.10 | 96.10 | 95.90 | 96.00 |
| Wino. | 70.01 | 71.11 | 72.77 | 72.85 | 75.69 | 76.01 | 73.71 | 74.11 |
| AVG | 70.12 | 70.51 | 72.39 | 72.49 | 75.27 | 75.33 | 74.05 | 74.09 |

**Table 6: Average accuracy of zero-shot benchmarks of amplifying super weights in models with round-to-nearest 4bit weight quantization with blocksizes of 8x8 and 64x64. On quantized models, amplifying super weights also yields a small yet consistent quality improvement.**
|  | RTN-4bit-8x8 | RTN-4bit-64x64 |  |  |
| --- | --- | --- | --- | --- |
|  | Original | Amplified | Original | Amplified |
| Llama-7B | 69.59 | 69.88 | 66.19 | 67.54 |
| Llama-13B | 72.09 | 72.13 | 71.86 | 72.07 |
| Llama-30B | 74.93 | 75.16 | 73.88 | 74.04 |

We visualize the full sensitivity graph starting from zero. We note that the average of all zero-shot datasets is around 30% when datasets are reduced to guessing accuracies.

Figure: Figure 8: Amplifying super weight improves quality. Full results for scaling super weight from 0 to 3.
Refer to caption: x14.png

### A.2 Maximum-magnitude Activation of Down Projection Module

Below, we show more examples of identifying super weights. We visualize the maximum-magnitude activations in the inputs and outputs of down_proj of all the transformer layers. The outlier ”sparks” indicate the row and column index of super weights.

Figure: Figure 9: Maximum-magnitude activation of down_proj across all transformer layers of Mistral-7B.
Refer to caption: x17.png

Figure: Figure 10: Maximum-magnitude activation of down_proj across all transformer layers of OLMo-7B.
Refer to caption: x19.png

### A.3 Logit Distribution with Super Weight Removal

Below, we visualize more of the logit distribution, when super weights are removed from Llama-7B. Despite the more thorough visualization, the conclusions remain the same: stopwords are amplified and non-stopwords see a drastic decrease in likelihood.

Figure: Figure 11: Output token distribution before and after removing the super weight on Llama-7B.
Refer to caption: x21.png

Figure: Figure 12: Output token distribution before and after removing the super weight on Mistral-7B.
Refer to caption: x22.png

Figure: Figure 13: Output token distribution before and after removing the super weight on OLMo-7B.
Refer to caption: x23.png

### A.4 Additional Activation Quantization Results with Super Activations

Below, we include results for Llama-2 7B using our activation quantization. See Table 7.

**Table 7: Perplexity ($\downarrow$) on Wikitext-2 and C4 with Llama-2-7B.**
| PPL ($\downarrow$) | Llama-2-7B |  |
| --- | --- | --- |
|  | Wiki-2 | C4 |
| FP16 | 5.47 (100%) | 6.97 (100%) |
| W8A8 | 5.58 (0%) | 7.09 (0%) |
| Ours | 5.57 (9.1%) | 7.07 (16.7%) |

### A.5 Super Weights and Attention Sinks

Given that super activations are typically observed on the first token of an input sequence, we hypothesized a potential relationship between super weights and the well-documented phenomenon of attention sinks (Xiao et al., ; Son et al., ). To test this hypothesis, we conducted experiments comparing attention weight patterns in the presence and absence of super weights. Contrary to our initial expectations, we find that attention sinks persist even when super weights are removed from the model, while not preserving model quality.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Could the authors provide clarification on the points I raised in the weaknesses section, especially if I may have misunderstood some of the contributions?

2. In Figure 6, do the authors have any insights into the concave behavior of the scaling factor? Are there specific explanations or potential methods for identifying this optimal scaling factor?

3. Regarding the stop word shift in distribution, is it generally accepted that a higher probability of stop words negatively impacts LLM performance?
soundness: 3
strengths: The paper is well-written and effectively illustrates the importance of superweights and superactivations. I appreciate the discussion on the percolation of superactivations across the network and the identification of superweights across layers (Figure 3). Additionally, I find the potential implications of superweight upscaling presented in Figure 6 quite interesting.
summary: This paper focuses on the impact of outlier weights in large language models (LLMs), specifically larger weights, which the authors term superweights and superactivations. First, the authors analyze how much these weights and activations affect LLM performance. They then use this as motivation to discuss quantization methods designed to account for superweights and superactivations. Throughout the paper, the authors also discuss the impact of superweight scaling and provide experimental results showing how their quantization method improves upon standard rounding, especially when using larger block sizes within the network.
weaknesses: While I appreciate the analysis presented in this paper, I am struggling to see the novelty of this work. I may be misunderstanding, but from what I gather, superweights and superactivations have already been discussed in prior analyses of LLMs. Additionally, it seems that methods like AWQ and SqueezeLLM inherently focus on superactivations. Furthermore, compared to other weight quantization techniques, the proposed method does not appear to offer significant improvements.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: The paper mostly focuses on post training model weight/activation analysis and identifies certain handful of importance weights/activations. The authors also say that irrespective of the input prompt the super weights are always the same and they mostly occur in the early layer's down projection with some reasoning via skip connections diagram. 

Though these insights are helpful, but it would be good if authors can follow up with what happens during the training process that such super weights are formed in the first place. Does the training methodology in terms of quantization during training/layernorm, gradient scaling, etc play any role in the forming of these super weights?
soundness: 3
strengths: Novel discovery about the importance of a few handful of neurons: The identification and analysis of super weights and super activations as critical outliers and their positive influence on model's performance is noteworthy and interesting. 

Quantization proposals: Authors went one step further to propose a super weight-aware quantization method to make the best use of these super weights/activations. Data free quantization proposal with on par performance compared to SmoothQuant is also a worthy contribution.
summary: The paper is about the discovery of super weights in LLMs that are disproportionately important, pruning these hurts model quality quite a bit. The authors have provided a way to identify these super weights using a forward pass. Super weights are activations are sensitive to quantization effects and hence authors propose a super weight aware quantization method enabling effective quantization.
weaknesses: Though the discovery is quite interesting, the improvements of proposed methods with existing baselines are quite marginal. In general, such kind of super weights might be a natural phenomenon in any machine learning model. How can one say this is relevant only to LLM's?

The work seems to be very much based on empirical observations (which is not my concern) but more discussions/intuitions/explanations around how/why these super weights are formed will be useful.

---

confidence: 5
contribution: 1
initial_rating: 1
presentation: 1
questions: Please refer to the weaknesses section above.
soundness: 1
strengths: The authors conducted experimental explorations on the so-called "super weights."
summary: This paper investigates the sensitivity of a subset of outliers in LLMs, referring to them as "super weights." The authors conducted experiments to examine the impact of these super weights on model performance.
weaknesses: 1. The necessity of "super weights" is unclear, as outliers are already identified based on the threshold. Increasing the threshold will naturally reduce the number of outliers with very large weights. Given the known importance of outliers in LLMs, emphasizing "super weights" (outliers at a higher threshold) does not appear novel.

2. Figure 1 is misleading. According to the author's definition, "super weights" are a subset of outliers. However, the figure suggests -1.9 is a typical outlier with nearby values being quite small (.1 and .2), implying that zeroing out outliers produces nonsensical text—a widely acknowledged fact. To better demonstrate the significance of super weights, it would be beneficial to explore whether zeroing out all outliers results in poor performance, and similarly, whether zeroing out just a small subset (e.g., 20-30) leads to comparably severe degradation.

3. Table 1 raises critical concerns. First, the criterion for selecting outliers needs specification. Second, the "Prune SW, +SA" setting in Lines 146-152 is confusing, as it suggests pruning super weights while partially restoring super activations enhances quality. However, the authors did not prune activations, leading to confusion about this claim.

4. Table 2 appears redundant and fails to convey meaningful information. Replacing it with visual representations of "super weights" distributions would be more informative, as the current table occupies considerable space without offering clear insights.

5. Figure 2 is difficult to interpret. The depiction of super weights and their impact, such as generating nonsensical text, is not clear. The use of the same color block in both the network and the output is puzzling. Are the model's dynamics linear? How do the output and weights share the same significance? Clarification is needed on whether this figure is based on assumptions or empirical data.

6. In Lines 189-190, the term "super activations" is introduced but lacks clarity on whether it is threshold-based or aligns with corresponding weights, which could be time-consuming. The authors should clarify this terminology.

7. The paper contains several unprofessional notations. For example, "Yij" should be corrected to "Y_{ij}" in Line 204, and similarly, "Xik" and "Wjk" should be "X_{ik}" and "W_{jk}" in Line 205. The inconsistency in notation and dimensions between "d" and "D" in Line 204 suggests a lack of careful writing and review, raising concerns about the overall professionalism of the paper.

8. Lines 198-210, which discuss the identification of super weights, are crucial yet unclear. The selection criteria for super weights remain ambiguous and need a precise mathematical description. Readers should understand the definition of outliers and the criteria for their selection explicitly.

9. The paper lacks consistency in terminology. "Super weights" sometimes refer to both activations and weights, and at other times only to weights, adding confusion. In Line 306, the term "super outliers" is introduced, suggesting that the paper should maintain consistent terminology from the start, including in the title, if both weights and activations are discussed.

After several careful readings, there are numerous additional concerns throughout the paper. The issues are substantial and critical, making it unlikely to meet the standards of ICLR. I recommend a strong reject based on the quality of this paper and will not change my rate.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. For equation 1, the median is used to replace super activation. Is getting the median
time-consuming since GPU is not good at sorting? (Although there are GPU-version
sorting algorithms)
2. The authors mentioned that SmoothQuant does not report on some models this paper
evaluates, they compare our results with naive W8A8 quantization (line 407 - line 409).
Can the authors run SmoothQuant on these methods since it is open-source? The naive
W8A8 is a too-weak baseline.
soundness: 3
strengths: The discovery is interesting and the proposed quantization method is easy to implement, which
can maintain better performance compared to Round to nearest quantization with the same
block size.
summary: This paper reveals that Large Language Models (LLMs) contain a very small subset of weights
(super weights) that are extremely important, where removing them severely degrades model
performance. The researchers developed an efficient, data-free method to identify these super
weights using only a single forward pass. They further investigated how these super weights
influence network behavior by analyzing their relationship with activation outliers. Building on
these insights, they proposed a quantization approach that carefully preserves these super
weights while effectively compressing other weights, resulting in the maintenance of model
quality after compression.
weaknesses: The authors failed to show how the proposed methods can improve the SOTA.
1. Although the method is data-free, its performance does not exceed SOTA methods like
SmoothQuant, given incorporating a small calibration dataset would not increase the
quantization complexity much.
2. The author mentions that this method is hardware-friendly, but no experiments to show
its effectiveness in improving latency, throughput, memory usage, etc.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - Section 3.2, "Prune SW+SA":  The description of the "Prune SW+SA" condition in Section 3.2 is unclear.  Specifically, how does this condition differ from the original model?  I understand that super activations typically precede super weights in the model.  Therefore, I am unsure what modification is being made in "Prune SW+SA" and how it distinguishes itself from the original, unpruned model.  Could you please elaborate on this procedure?
soundness: 2
strengths: - The identification of "super weights" and their connection to super activations represents a novel and potentially significant finding in understanding the inner workings of LLMs.
- Connection of "super weights" to quantization accuracy is quite interesting and has practical implications.
- The paper provides a clear methodology for identifying super weights and evaluating their impact, along with an index of super weight coordinates for common LLMs, facilitating further research.
summary: This paper introduces the concept of "super weights" in Large Language Models (LLMs), identifying a small number of individual weight parameters (as few as one) that have a disproportionately large impact on model performance.  Pruning these super weights drastically reduces the quality of generated text, while pruning thousands of other larger-magnitude outliers has a negligible effect.  The paper proposes a data-free method for identifying super weights based on their connection to "super activations," exceptionally large activation outliers previously observed in LLMs.  Finally, the paper demonstrates that preserving super weights and activations during quantization significantly improves compression quality, achieving competitive results methods like SmoothQuant.
weaknesses: # Major

- Connection to Adversarial Examples: The literature extensively documents how small changes in the input domain can drastically alter output probabilities. Consequently, significantly harming the network by removing weights, as demonstrated, is somewhat expected.  A discussion addressing the connection between super weight removal and adversarial examples would strengthen the paper.

- Magnitude Pruning Baseline: In Table 1, the comparison of super weight pruning with global magnitude pruning may not be the most informative.  A stronger baseline would involve pruning only within the layer where super activations occur. This would better isolate the impact of the super weight itself.

- Quantization Baseline: The "Naive W8A8" quantization baseline should incorporate clipping. The current presentation makes it unclear whether the observed improvements stem from outlier removal or clipping, especially since super weight handling affects only a single layer during quantization, while clipping is applied to every layer.  Furthermore, it should be noted that the clipping threshold is determined using Wikitext-2, which is also included in the evaluation of quantized models.

# Minor

- Terminology:  The term "extreme" might be more descriptive and informative than "super" when referring to these weights.

- Weight Distribution Visualization: Including a histogram visualizing the position of the super weight within the overall weight distribution would enhance understanding of its magnitude relative to other weights.

Meta-review: This paper explores the impact of superweights—defined as weights with larger magnitudes—on the performance of large language models (LLMs). The authors analyze the influence of these superweights on LLMs’ performance and propose specialized quantization methods tailored to superweights. The experimental results show some performance improvements achieved by the method, particularly when employing larger block sizes within the network.

However, three reviewers have raised notable concerns regarding the limited experimental improvements, the lack of clarity in the experiments, and also lack of deep analysis. Furthermore, one reviewer has questioned the definition and conceptual clarity of the superweight concept. As a result, the current version of the paper does not appear ready for publication. We recommend that the authors thoroughly address these issues in alignment with the reviewers' feedback.

Decision: Reject