Demystifying Singular Defects in Large Language Models
Authors: Haoqi Wang
Tong Zhang
Mathieu Salzmann
Venue: ICML 2025 poster (2025)
Keywords: Machine Learning
ICML
Singular Defect
Large Language Models
High-Norm Tokens
LLM Quantization
LLM Signature
Primary area: deep_learning->large_language_models
TL;DR: The high-norm tokens in LLMs are thoroughly analyzed.

Abstract: Large transformer models are known to produce high-norm tokens. In vision transformers (ViTs), such tokens have been mathematically modeled through the singular vectors of the linear approximations of layers. However, in large language models (LLMs), the underlying causes of high-norm tokens remain largely unexplored, and their different properties from those of ViTs require a new analysis framework. In this paper, we provide both theoretical insights and empirical validation across a range of recent models, leading to the following observations: i) The layer-wise singular direction predicts the abrupt explosion of token norms in LLMs. ii) The negative eigenvalues of a layer explain its sudden decay. iii) The computational pathways leading to high-norm tokens differ between initial and noninitial tokens. iv) High-norm tokens are triggered by the right leading singular vector of the matrix approximating the corresponding modules. We showcase two practical applications of these findings: the improvement of quantization schemes and the design of LLM signatures. Our findings not only advance the understanding of singular defects in LLMs but also open new avenues for their application. We expect that this work will stimulate further research into the internal mechanisms of LLMs. Code is released at https://github.com/haoqiwang/singular_defect.

Demystifying Singular Defects in Large Language Models
Haoqi Wang'
Tong Zhang
Mathieu Salzmann
Abstract
tokens during the forward pass, as illustrated in Fig. 1. This behavior has been observed in both large vision transform- ers (Darcet et al., 2024; Wang et al., 2024) and large lan- guage models (LLMs) (Sun et al., 2024).
Large transformer models are known to produce high-norm tokens. In vision transformers (ViTs), such tokens have been mathematically modeled through the singular vectors of the linear approx- imations of layers. However, in large language models (LLMs), the underlying causes of high- norm tokens remain largely unexplored, and their different properties from those of ViTs require a new analysis framework. In this paper, we pro- vide both theoretical insights and empirical val- idation across range of recent models, leading to the following observations: i) The layer-wise singular direction predicts the abrupt explosion of token norms in LLMs. ii) The negative eigenval- ues of layer explain its sudden decay. iii) The computational pathways leading to high-norm to- kens differ between initial and noninitial tokens. iv) High-norm tokens triggered by the right leading singular vector of the matrix approximat- ing the corresponding modules. We showcase two practical applications of these findings: the improvement of quantization schemes and the de- sign of LLM signatures. Our findings not only advance the understanding of singular defects in LLMs but also open new avenues for their appli- cation. We expect that this work will stimulate further research into the internal mechanisms of LLMss. Code is released at https: //g tiuub. com/haoqu http:/haqiiiaaae wang///wa////a/ wiwsssiinnnddereeeeeee defect.
In particular, in vision foundation model DINOv2 (Oquab et al., 2024; Darcet et al., 2024), high-norm tokens appear as defective patch tokens, degrading the feature quality and hindering performance
downstream dense prediction tasks. In pursuit of repairing these defects, Wang et al. (2024) revealed a connection between the direction of the high-norm tokens and the leading left singular vector of the matrix that provides a linear approximation of a transformer layer, and therefore name these tokens as singular defects.
In the context of LLMs, high-norm hidden states have also been observed (Sun et al., 2024) and referred to as mas- sive activations. Specifically, the following properties were empirically identified: (1) The appearance of a massive activation is abrupt, emerging suddenly in one layer and diminishing later in the model after another. (2) Massive activations appear mostly for the initial token and the delim- iter tokens. These properties differ from those made with DINOv2, where the norm of the defective tokens increases gradually layer by layer, and the defective tokens are ran- domly scattered across the feature map, with a tendency to appear in low-semantic regions. Given these different be- haviors of high-norm tokens in LLMs vs. ViTs, it is natural to wonder (i) whether the theory of singular defects can be applied to LLMs; and (ii) how to further explain the new obxervations related to massive activations?
In this paper, we confirm that singular defects can predict the direction of high-norm tokens in LLMs. However, it falls short of explaining the high-norm properties that are unique to LLMs.. To understand the full life cycle of the high-norm tokens, we thus expand the theory and provide the following insights:
1. Introduction
While large foundation models are proving increasingly ef- fective, their intrinsic behavior remains poorly understood. An example of this is the emergence of tokens whose norm is unexpectedly higher than that of the majority of the other
1. (Development) The explosion of the initial' token norm is linked to self-attention, whereas that of the noninitial high-norm tokens is unrelated to self-attention.
"School of Computer and Communication Sciences, EPFL, Switzerland University of Chinese Academy of Sciences, China ? "Swss Data Science Center, Switzerland. Correspondence to: Mathieu Salzmann aamathiuu.salzmann @epfl.ch>.
(Trigger) A norm explosion initiated when the input vector has a projection onto the leading right singular
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).
We define the initial token as the first token in the user input.

---

Demystifying Singular Defects in Large Language Models
(a) LLamm2-7B
(b) Phi3-Medium
(c) MPT-7B
(d) Pythia-160M
2593
1073
(c) Vicunal. 1.5-7B
(f) Falcon2-11B
(g) GPT2-Medium
(h) Qwen2.5-1.5B
Figure 1. High norm tokens in various LLMs. Each subfigure plots the norm of the first few tokens and the token '.' in the sentence 'The quick brown fox jumps over the lazy dog. ' The a-axis the layer id, the y-axis shows different tokens, and the a-axis is the norm. Layer 0 is the input embedding layer, and the others are transformer layers. Additional models are shown in Appendix A.
vector of the linear approximation of the explosion layer's feed-forward network (FFN) module.
via Singular Defects. The singular defect direction, which stabilizes in the late training stages and persists through fine-tuning, serves as robust model signature. This signa- ture enables distinguishing whether an LLM was fine-tuned from another model and thus detect model infringement.
3. (Explosion) Once triggered, the high-norm token in the layer's output aligns with the direction of the layer- wise singular defect direction.
Ultimately, we believe that understanding singular defects will not only stimulate novel applications but also spur new insights into the internal mechanism of LLMs.
4. (Decay) The layer that decays the high-norm tokens has a negative eigenvalue associated with an eigenvector that aligns with the singular defect direction.
2. Related Work
We empirically validate these findings LLMs, including LLaMA2 (Touvron et al., 2023), Phi3 (Ab- din et al., 2024), MPT (Team, 2023), Pythia (Biderman et al., 2023), Vicunal.5 .5 (Platzer & Puschner, 2021), Fal- con2 (Malartic et al., 2024), GPT2 (Radford et al., 2019), Qwen2.5 (Team, 2024), to name a few.
variety of
Large Language Models (LLMs). Transformer-based LLMs (Minaee et al., 2024) have achieved remarkable per- formance in various natural language processing tasks. They are mainly pre-trained with causal language modeling (e.g.. LLaMA2 (Touvron al., 2023)) or masked language mod- cling (e.g.. BERT (Devlin et al., 2019)). BERT-like mod- els employ bidirectional attention mechanism to learn contextual representations, typically adopting an encoder- only architecture. contrast, LLaMA-like models utilize a decoder-only architecture and are trained autoregressively with causal self-attention, modeling the probability distri- bution of the next token in sequence. In our observations, LLaMA-like models exhibit extremely high-norm tokens in their hidden states, whereas BERT-like models do not. In this work, we focus on LLaMA-lik, models and investigate the characteristics of these high-norm tokens.
For the behavior of high-norm tokens during training, our experiments reveal that the direction of high-norm tokens gradually stabilizes in training and remains consistent even after fine-tuning, Moreover, we conjecture that the causal self-attention mechanism is one of the defining factors for the emergence of high-norm tokens.
Finally, we demonstrate that better understanding of the high-norm tokens in LLMs can lead novel applications. High-Norm Aware Quantization Design. Outlier activations induced by high-norm tokens cause significant performance in low-bit quantization. To mitigate this, we propose a high-norm aware quantization strategy that selectively pre- serves precision for these critical layers, improving robust- ness without compromising efficiency. ii) I.I Signature
High-Norm Tokens in ViTs and LLMs. Several recent works have observed the presence of high-norm defective tokens in the feature maps of ViTs (Darcet et al., 2024;
2

---

Demystifying Singular Defects in Large Language Models
Wang al., 2024) and proposed methods to repair them. In particular, Wang et al. (2024) used the leading left singular vector of the matrix representing linear approximation of the transformer layer to predict the direction of the de- fective tokens. In the context of LLMs, Sun et al. (2024) noticed that certain activations in the hidden states have a huge magnitude (i.e., massive activations). They observed that massive activations are consistently present in very few fixed dimensions but did not provide a mathematical expla- nation. To account for the different properties of high-norm tokens in ViTs and L.M,s, we extend the theory of Wang et al. (2024) to LLMs, providing a systematic explanation for their high-norm tokens. Furthermore, while Sun et al. (2024) study massive activations from the perspective of the individual scalar values within a token, we analyze high- norm tokens whole vector and provide a mathematical framework explain high- high-norm tokens in LLMs.
The quick brown
Explosion
Trigger
Docay
Development
10
15
25
Figure 2. The four properties of high-norm tokens. The x-axis is
the layer id, and the y-axis is the norm of the tokens. The results are obtained from a
25 x Ax
23 x Rx
se Ax
F(x + Ax)
Self-Attention
FFN
Figure 3. Transformer layer in LLaMA2. Given an input token ø, we show the approximate oulput each module in red.
pathways leading to norm increases. (2) Trigger: the cause of the norm increase just before the explosion layer. (3) Explosion: the appearance of high-norm tokens in interme- diate layers and their correlation with network parameters. (4) Decay:
Applications of High-Norm Tokens. While recent stud- ies have identified high-norm tokens in LLMs, their practical implications remain largely unexplored. We highlight two key applications: Quantization and Model Signature. High- norm tokens induce activation outliers, posing challenges for tensor-wise quantization methods (Dettmers et al., 2022; Xiao et al., 2023; Grattafiori et al., 2024). Additionally, as more open-source LLMs become available, tracing model lineage is increasingly important. Prior works (Yu & Wang, 2024; Mu et al., 2024) investigate model tracing through internal representations, yet, they can only deal with models in vision. To the best of our knowledge, we are the first to leverage insights from the high-norm token analysis to address both challenges in LLMs.
how high-norm tokens disappear.
3.1. Explosion: Appearance of High-Norm Tokens
The most obvious pattern in Fig. I is that the hidden states of some tokens have extremely high norms in the intermediate layers, and so consistently across different LLMs. Interest- ingly, we find that for each model, all the high-norm tokens share the same direction, regardless of the input text, the layer, and the position of the token in the sentence. We define the average of the high-norm tokens as the empirical high-norm direction.
Taking I.LAMA2-7B as an example, we extract the hid- den states of IK random rows from the WikiText2 dataset (Merity et al., 2017) across all layers and compute the norm of each token in each layer. We collect all hidden states with a norm larger than 500. The average pairwise angle between any two high-norm tokens only 3.1 de- grees, which verifies our claim. Additional statistics for more models are provided in Appendix C, demonstrating that this phenomenon is general across models. Sun et al. (2024) observed that the massive activations appear at fixed feature dimensions, which echos our observation from the perspective of vector direction.
3. Analysis of High-Norm Tokens in LLM
As shown in Fig. 1, most recent LLMs manifest high-norm tokens in intermediate layers, regardless of the model size, training data, variations in model architecture, etc. Echoing the observations in (Sun et al., 2024), we note that the high- norm tokens appear abruptly in certain layer and decay suddenly after another layer, with the initial token consis- tently having high norm in the middle layers, while certain later tokens, such as the delimiter .', also exhibit high norm in some models. These phenomena differ from those in vision transformers, where norms gradually increase with- out sudden drop (see Figure 4 in (Darcet et al., 2024)), and high-norm tokens appear at random spatial locations.
Notably, the behavior that the same high-norm direction ap- pears across layers differs from DINOv2 where the direction of the high-norm tokens varies across layers. Despite the dis- tinctions between high-norm tokens in VITs and LLMs, we next verify whether the theory of singular defects developed for DINOv2 is applicable to LLMs.
This section provides comprehensive understanding of the distinctive high-norm phenomenon in LLMs by expanding the theory of singular defects that was originally developed for ViTs. As illustrated in Fig. 2, we study this phenomenon from four perspectives, covering the full life cycle of the high-norm tokens: (1) Development: the computational
Let us first review the basic concepts introduced in (Wang et al., 2024). The core of the theory of singular defects
3

---

Demystifying Singular Defects in Large Language Models
the linear approximation of transformer layers under the single-token assumption. It assumes that only single token is provided input, so that the interaction across tokens in self-attention can be ignored, making the mathematical analysis tractable. Note that most recent LLMSs are trained with causal self-attention, where a token can only attend to itself and the previous tokens. Thus, the inference of the first token perfectly matches the single-token assumption.
Angle (deigrees)
80-
60
40
20
6.05
2.93
10
20
25
Layer
Figure 4. Acute angle between the predicted layer- wise singular de- fect directions and the empirical high- norm direction for LLaMA2- 7B. The predictions for more LLMs provided in Appendix D.
We use LLaMA2 proximate the transformer layer as linear operator. The structure of the transformer layer is shown in Fig. 3, where the self-attention module and the FFN module are the two residual components. The self eeffaattenion module can be ap- proximated as a matrix-vector multiplication,
example to illustrate how to ap-
80
Angle (degree)
60
Eigenvalue
40
11.33
2.86
20-
Attention(z) (z) Ar A02,
(1)
0
0.50
-20
where Ao is a diagonal matrix representing the weight of the attention RMSNorm, A1 is the weight matrix of the value projection, and A2 is the weight matrix of the output projection. Note that for LLMs that use different self- attention design, the approximation of the self-attention module can be adjusted accordingly. For example, LLaMA3 uses grouped query attention (Ainslie et al., 2023) whose key-value heads are repeated. Then, we can, accordingly, repeat the weight matrix of the value in the approximation.
0
5
10
15 Layer
20
25
30
Figure 5. For Lamma2778, the minimum angles hetween the eigen- vectors of R and the empirical high-norm direction are shown in red, and the corresponding eigenvalues are shown in blue. More examples are shown in Appendix E.
The FFN module can be approximated as
high-norm direction. These two layers correspond to the layer that increases the norm and decreases the norm of the tokens, respectively (see Fig. Ia). Layers 3-30 only cause small perturbations to the high-norm tokens. Thus, the high-norm tokens in the hidden states are created by layer-2, then preserved by the identity path between layer-3 and layer-30, and suppressed by layer-31. This explains why the set of high-activation channels observed (Sun et al., 2024) is fixed: the high-norm tokens are (nearly) unchanged in the intermediate layers. Unsurprisingly, we observe that the angles between the layer-wise predicted directions and the empirical directions are large, they do not modify the high-norm tokens. Note that the accumulated singular defect directions (see (Wang et al., 2024) for a formal definition) may mislead us about the contribution of the intermediate layers the high norms. Therefore, in the remainder of the paper, we will focus on layer-wise singular defect directions.
FFN x Fx = Foz,
(2)
where Fo is a diagonal matrix representing the weight of the feed-forward RMSNNrm, F2 is the weight matrix of the down_proj. and F1 is the least-square linear approxima- tion (see (Wang et al., 2024) for details) of the nonlinear function Fix s silu (x) @ (W3z), in which W1 is the gate-proj and W3 is the up-proj. For LLMs that use alterna- tive FFN designs, e.g., Fix R GELU(W,rr in Pythia, the least-square linear approximation of the FFN module can be adjusted accordingly.
Combining the two modules with the identity paths, the transformer layer can be approximated as
Layer(z) Re La : (1+ R1+,)x, (3)
where the right-hand side decomposes the approximated matrix into an identity path and a residual path R. The layer-wise singular defect direction then defined as the leading left singular vector of the matrix for each layer.
3.2. Decay: Eigenvalues of Decay Layers
The leading left singular vector of the matrix L encodes the output direction having the largest norm for all possible unit-length inputs, and the corresponding singular value is the norm of the output in that direction. However, this interpretation contrasts with the behavior of the decay layer, which greatly reduces the norms of high-norm tokens. To better describe the behavior of such layer, we instead propose to use eigenvalue and eigenvector decomposition.
We compute the layer-wise singular defect directions of LLAMMA-7B and compare them with the empirical high- norm direction. The acute angles between the predicted directions and the empirical one are provided Fig. 4. We observe 1ayer-2 and layer-31 to yield very small angles (6.05 and 2.93 degrees, respectively) with the empirical

---

Demystifying Singular Defects in Large Language Models
Norm after Layer
10
<a>: 3670.0
Output Norm of FFN
leading right singular vector
10a
775. 0 357.
4000
10"
75.5
nnddrrhrrd''
2000
101
100
0
5000
10000
15000
25000
30000
1000 Right Singulaar Vectors of
2000
3000
4000
Token ID
Figure 6. Attention irention-independen exploding path LLaMA2-7B. The y-axis is the norm of cach token at the output of layer-2 after removing all self-attention blocks from the model. The largest five tokens logether with their norms are annotated in the figure. The noninitial high-norm tokens are labeled red. Results of additional models is presented in Appendix F.
Figure 8. Norm of output tokens of FFN at layer-2 of LLaMA2-7B using the right singular vectors of F as input tokens to FFN. More examples are provided in Appendix G.
2
Coefficient
1-
Initial Position
Second Position
Norm after Layer
0-
1000
750
Trained Tokens
-1
500
Random Embeddings (sorted)
5000
10000
15000
20000
25000
30000
Token
250
Figure 9, Coefficient of tokens projected the leading right sin- gular vector of F just before the FFN in layer-2 of LLaMA2-7B. Analysis for additional models is provided in Appendix G.
5000
10000
20000
25000
30000
Token ID
Figure 7. Attention related exploding path of LLaMA2-7B for ini- tial tokens. The y-axis is the norm of each token at the output of 1ayer-2. We also show the norm (sorted) 32,000 random input token embeddings (not learned by the network) after layer-2. See more figures in Appendix F.
3.3. Development: The Two Types of Exploding Paths
We now focus on the development of the high-norm tokens before the "explosion". To this end, we refer to the compu- tation path related to the appearance of high norms between the input token embedding and the explosion layer as the exploding path. We differentiate two types of high-norm tokens: the initial token and the noninitial high-norn to- kens. An example of the noninitial high-norm token is the delimiter token *." in LLaMA2-7B ABB.. We observed that the first occurrence of the intermediate hidden states, regardless of the content before it. This implies that the self self-attention layers, the only places where inter-token interactions occur, play an insignificant role in noninitial high-norm tokens.
Consider the decomposition I. = T R Eq. (3), and let be high-norm token input to decay layer. If after the layer, the token does not have a high norm anymore, then La se O, and therefore Rx x x. This implies that the high-norm token x should be an eigenvector of the residual matrix R with a negative eigenvalue.
token has a high norm in the
To verify this intuition, we compute the minimum angle between the eigenvectors of the residual matrix R and the empirical high-norm norm direction for LLaMA2-7B. In Fig. 5, we plot these angles for different layers, together with the corresponding eigenvalues. At the decay layer-31, the eigen- vector of the residual matrix R forms a small angle with the empirical high-norm direction, with a relatively large negative eigenvalue?. We verify this for other models in Appendix E. A corollary of this observation is that, if the input to the decay layer is not high-norm token, it will not produce a new high-norm token, i.e., the layer specializes in shrinking the token norm in the horm direction.
To verify this argument, we remove the self-attention layers and feed each token available in the tokenizer individually to the model. Without self-attention layers, all tokens in sequence effectively behave as independent single-token within the network. the noninitial high- norm tokens are indeed unrelated self-attentions, then we expect that the noninitial high- morm- tokens retain their high norms after removing self-attention layers from the model. This is veri- fied in Fig. 6, where we plot the norm of the hidden states after the explosion layer for all possible input tokens. Out of the five tokens with the highest norm, four of them belong to noninitial high-norm tokens, including '.', '., '
'
For simplicity, we only consider the real part of eigenvalues and eigenvectors, having observed that the imaginary part of the eigenvalues and eigenvectors of interest is close to zero.
5

---

Demystifying Singular Defects in Large Language Models
norm of the initial tokens before/after FFN is 2.49/932.25,
respectively. Let F be the linear approximation of this FFN module following Eq. (2), and the SVD of F be F = UEVT. The columns of V form an orthonormal basis of the input space. We feed these unit-length base vectors to the FFN and plot their output norms in Fig. 8. Note that the leading right singular vector of F is the only direction that undergoes norm explosion when passed through FFN. We thus refer to the 1 1-dimensional subspace spanned by this vector as the explosion subspace. This lets us conjecture that the high-norm tokens have a large component in the explosion subspace, while the normal tokens do not. This is verified in Fig. 9. For any token at the initial position in the input text (which is thus a high-norm token), the projection of its feature just before the FFN module at layer- onto the explosion subspace has large coefficient (around 1.5). However, when the same token is placed at the second position in the input (where it is not a high-norm token, except for the few noninitial high-norm tokens discussed in Section 3.3), the coefficient is much smaller.
800
600
400
15
20 25
30
Figure 10. Norm of tokens after removing the component on the explosion subspace before layer-2's FFN in LLaMA2-7B. High- norm tokens the intermediate layers disappear. however, the scale of the final outputs becomes abnormal, leading to low-quality generated text.
and "(n'. Among them, the Chinese delimiter token "o" and the special token *<>>' are newly discovered high-norm tokens that were not identified in previous work.
One potential way to avoid high-norm intermediate layers could therefore be to remove the component in the explo- sion subspace just before the FFN the explosion layer. An example of the resulting token norms is shown in Fig. 10, where the high norms in the intermediate layers indeed dis- appear. However, we observe that the network loses its ability to generate high-quality text and instead produces random texts. This shows that high-norm tokens are im- portant for the performance of an existing trained model. This observation echoes the experiments done by Sun et al. (2024), where the authors set the massive activations to zero and find performance degradations.
Given that the high norm of noninitial tokens is unrelated to self-attention, such tokens can also be analyzed under the single-token assumption. Therefore, our methodology for predicting the high-norm directions can be applied to both the initial token and the noninitial high-norm tokens. This sheds light on why the two types of high-norm tokens share the same direction.
For the initial token, we observed the exploding path to be related to self-attention. For example, in Fig. 7, we plot the norm of all the trained tokens after the explosion layer-2 in LLaMA2-7B. Almost all tokens have a high norm. This aligns with the observation that the initial token always has high norm, regardless of the word it encodes. Comparing Fig. 7 with Fig. 6, we can see that the initial high-norm tokens lose their high norms (except for few noninitial high-norm tokens) when the self-attention layers are re- moved from the model. This indicates that self- celf-attention is indispensable for initial tokens to have high norms.
In summary, the life cycle of high-norm tokens can be out- lined as follows. Firstly, the potential high-norm tokens, either the tokens at the initial position of the input sequence, or those noninitial high-norm tokens, develop a sufficiently large component in the explosion subspace. Then, this com- ponent explodes, yielding high-norm token, whose direc- tion can be predicted by the singular defect theory. Finally, the decay layer produces a negative direction to neutralize the high-norm token, which is described by the negative eigenvalue and eigenvector of that layer.
Surprisingly, even a random token embedding that is not learned by the network also has higher norm than a non- initial normal token, whose average norm is 46.88 in the explosion layer-2 of LLaMA2-7B, although the norm of the random token may be lower than that of the learned tokens. The analysis for other models is provided in Appendix F.
4. Discussion
High-Norm Tokens During Training High-norm tokens in ViTs appear only in the later stage of training and only in large model variants (Darcet et al., 2024). However, high- norm tokens in LLMs emerge early in training and across a variety of model sizes, as demonstrated in Fig. 11 using the suite of Pythia models (Biderman al., 2023). After 5K out of a total of 143K iterations, the norms of both the initial token and the token ', have become significantly
3.4. Trigger: The Explosion Subspace
Let us now focus on the explosion layer where the norm of tokens undergoes a sharp increase. Tracking the change in norm within that transformer layer, we observed that the abrupt increase occurs in the FFN module. For example, in the explosion layer of LLLMMA-7B (layer 2), the average
6

---

Demystifying Singular Defects in Large Language Models
124
Ber 143K
254
1264
1004
Jeer 50K
22.0
224
1000
lie 5K.
183
P1MMia-10M
Pythio-1.40
Pythis22888
Pythis-6.98
Figure 11. High norm tokens in the Pythia model suite. Each row shows a different training iteration, with the top row being the final state 143K iteration. Each column different model size, including 410M, 1.4B, 2.8B, and 6.9B.
Angle Diff (degree)
1.5
14.99
ment learning with human feedback (RLHF), languages, context length, model size, or training iterations. The root cause therefore remains an open question. Nevertheless, pirical evidence suggests that the causal self-attention mech- anism may be one of the defining factors for the emergence of high-norm tokens. Firstly, note that all the high-norm examples we presented so far were trained with causal at- tention. Secondly, Mistral (Jiang al., 2023), which uses a sliding window attention mechanism, does not exhibit high- norm tokens among the initial tokens, as shown in Fig. 13, This indicates that modifying the causal self-attention mech- anism affects the emergence pattern of high-norm tokens. Thirdly, we did not observe high-norm tokens in bidirec- tional models, such as BERT, DistilBERT (Sanh et al., 2019), and RoBERTa (Liu, 2019). This implics that masked lan- guage modeling does not induce high-norm tokens as in causal language modeling.
10-
6.36
5-
3.47
2.69
2.08
1.57 0.99
0.75
0,56 0.31 0.31 0.36 0.45
20000
40000
60000
80000
100000
120000 140000
lteration
Figure 12. Direction of norm tokens stabilizes during training. The y-axis is the angle between the empirical high-norm directions of adjacent Pythia- IB checkpoints at 10, 000 interval.
5. Applicationss
(a) Mistral-7B-an
(b) BERT-Large-Cann Cased
In this section, we showcase two applications motivated by the insights of previous sections. In Section 5.1, we improve quantization by specializing it according to the explosion and decay layers. In Section 5.2, we show that the direction of high-norm tokens can be used as a signature of an L.M.
Figure 1.3. (a) Modifying the causal self-attention mechanism af- fects the emergence high norm tokens. (b) Bi-directional models do not show high-norm tokens. More examples are in Appendix B.
higher than that of the other tokens in the hidden states. Fur- thermore, analyzing the change in the high-norm tokens as training progresses, we found that the high-norm direction gradually stabilizes during training, as shown in Fig. 12.
5.1. Improving the Design of Quantization
LLMs are memory and computation intensive because of their huge number of parameters. Quantization common technique to reduce memory footprint and improve infer- ence speed. Quantization in LLM mainly follows two set- tings: W8A8 quantization (Dettmers et al., 2022; Xiao et al., 2023), where both the weights and activations are quantized to 8-bit; and low-bit weight-only quantization (Lin et al., 2024; Frantar et al., 2023). We focus on W8A8 quantization. Converting model from 16-bit Moating-poimt to 8-bit inte- ger reduce the model size by half. Int8 quantization (Ja-
Which Factor Matters for the Emergence of High-Norm? From our extensive experiments with various LLMs, observed that the presence of high-norm tokens is not signif- icantly affected by factors such as the training dataset, posi- tional embedding scheme, FFN design, the use of grouped- query attention, parallel attention-FFF -FN structures (Wang & Komatsuzaki, 2022), supervised (uentuunngg(ST)) reinforce-
7

---

Demystifying Singular Defects in Large Language Models
cob et al., 2018) can be expressed as X X
X
where A
Table 1. Improving tensor-wise quantization by skipping the down_proj matrix F/ of the FFN module in the explosion and decay layers. PPL the perplexity on the validation set WikiTex12-VL.
I) the quantization step size. Note that A is A 2v-1 -1
determined by the maximum absolute value of X. There- fore, outliers in the input tensor will lead to non-negligible underflow and cause large quantization enrors.
Model
Method
Skip F2 in Layers
PPL1
5.47
Outlier channels in quantization are different from high- norm tokens (Sun et al., 2024). High-norm tokens refer to hidden states after each transformer layer, whereas out- lier channels denote the input activations linear layers within a transformer layer. Most current LLMs adopt pre- LayerNorm, where a normalization layer is prepended be- fore the self-attention and FFN modules. Hence, the input activations for the query, key, value projections in attention, and W,, W3 in FFN, are normalized. These inputs are not high-norm tokens, yet they possess outlier channels.
RTN RTN
10.18
LLaMA2-7B
(1)
6.51
Smooth Quant
13.87
SmoothQuant
(2, 31)
6.78
6.14
RTN RTN SmoothQuant
59.38
LLaMA3-8B
(2, 32)
8,80
54.99
SmoothQuant
(2, 32)
9.14
Mamaa-7b--hl
llama2- Damaa--7-code-pyttor
Ulama3-8b
Outliers in activations are more severe than in weights (Xiao et al., 2023). As such, the quantization of activations is more challenging than that of weights. For example, tensor-wise W8A8 quantization, where both weights and activations are quantized to int8, does not work well for LLaMA3 (Grattafiori et al., 2024). To tackle this issue,
Angle with llama2-77 7b
laam22776 -b-ode
llama2- 1amaa-7b-code-instuue
80
60-
40-
20-
0
LLaMA3 resorts to the row-wise quantization, where the quantization step size is computed across rows of acti- vation and weight matrices. However, hardware support for row-wise quantization is limited. For instance, the FBGEMM (FBG) implementation of row-wise quantiza- tion only supports recent GPUs such as Nvidia H100 and AMD MI300X. Furthermore, wise quantization has lower latency compared to row-wise quantization (Xiao et al., 2023). It is therefore desirable to improve the quanti- zation strategy to make cois quantization effective.
0
5
10
15
20
25
30
Layer
Figure 14. The singular defect directions (at explosion layer 2 and decay layer 31 for LLaMA2-7B) remain stable after fine-tuning.
5.2. Signature of an LLM
In Section 4, we have shown that the direction of the high- norm tokens gradually stabilizes during training. By ex- tension, this suggests that the singular defect direction also robust to fine-tuning, since fine-tuning is a continuation of the training process. To verify this claim, in Fig. 14, we compare the pairwise angles of the layer-wise singu- lar defect directions between the base model LLaMA2-7B and its fine-tuned variants I.LaMA2-7B-Chatt LLaMA2- 7B-Code, LLaMA2-7B-Code Python, LLaMA2-7B-Coue Instruct. For the explosion layer 2 and the decay layer 31, the pairwise angles very small, which confirms that the direction of the high-norm tokens is stable w.r.t. fine-tuning. Additionally, we added LLaMA3-8B into the comparison, and the pairwise angles are all nearly 90 degrees, suggesting that the singular defect direction could distinguish different models.
We found that the main obstacle to tensor-wise quantization in LLMs is related to the outlier channels induced by the high-norm tokens. In the explosion layer and the decay layer, the residual modules produce high-norm tokens (Section 3), and thus, in the middle of the computation, the intermediate token norms can be very large. Specifically, inside FFN, the norm of some tokens after ai @ (Waz) is extremely high, hence affecting the subsequent down.proj F2æ.
Our solution is straightforward: we simply allow the down-proj layer F/ of the FFN module in explosion and decay layers to operate at a higher precision. The differ- ence between our strategy and the mixed- mixeed-recisiion strategy in inLM.int0) (Dettmers et al., 2022) is that, i_L,,int8)) decomposes matrix into an 8-bit part and a 16-bit part, while operate the (very few) full problematic matrices in higher precision. We compare the performance of ours with the two tensor-wise quantization in Table 1. The per- plexities of both the standard RTN (Round-To- Nearest) and the recent SmoothQuant (Xiao et al., 2023) are significantly reduced after applying our solution.
An application of this stability is to use the singular defect direction as signature of an LLM. Specifically, we define the signature of a LLM as the empirical high-norm direction. Based on our analysis, the leading left singular vector of ei- ther the explosion layer or the decay layer serves as a reliable approximation of the empirical high-norm direction. This approach offers two key advantages: it is data-independent and eliminates the need for manually selecting a threshold
8

---

Demystifying Singular Defects in Large Language Models
1a2
90
6. Conclusion
Ja2 chat-
la2 code
80
We have studied the behavior of high-norm tokens LLMs from four perspectives, covering their whole life cycle. To this end, we have extended the theory of singular defects from ViTs to LLMs. Specifically, our theory gives mathe- matical characterization of the behavior of high-norm tokens that is unique to LLMs. We have extensively verified our theory on a variety of recent models, and our insights have also motivated two practical upplications, including a high- norm aware quantization scheme and the design of an L.L.M signature that may be used to detect model infringement. We hope that our study will motivate advances in the under- standing of the internal mechanisms of LLMs.
la2 python
la2 inst
70
la3
la3 inst-
60
1a3 guard 10 n 15
la3.1
50
1a3.1 inst
la3.1 guard a
vi1.1
v11.3
30
v11.5-
vi1.5 16k-rr
20
mpt-c
mpt inst
10
mpt chat
mpt story
la2.
la2 chat
la2 code
Ia2 python
la2 inst-
la3
Ja3 inst-
1a3 guard
la3.1-
1a3.1 guard
vil.1
vi1.3-
vil
vi1.5 16k
mpt:
mpt inst
mpt. chat
mpt story
0
Acknowledgements
1a3.
This work was supported in part by the Swiss National Science Foundation via the grant 200020.20 212808.
Figure Model distances based on angles between model signa- tures. la2, la3, 1a3.1. v. vi1.3, vil.5, mpt mean LLaMA2-7B, LLMM3--88, I.L.MA3.1-88 Vicuna-7B, MPT-7B, respectively. We can see that the distances within each model series are very small, as they originate from the same base models.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning, in particular, the understanding of
LLMs. There are many potential societal consequences of our work, none which we feel must be specifically high- lighted here.
define high norms. By defining the distance between two models (with the same hidden dimension) as the acute angle between their signature vectors, we can test whether one model is derived from another by computing their signature distances. In practice, to avoid having to manually locate the explosion/decay layers, we can simply compute the acute angle between the corresponding layer-wise singular defect directions for all layers and take the minimum value as the distance between the two models.
References
FBGEMM.
https://github. com/pytorch/ (BGaaa//treee/maa///hgeeee experimental/gen_.ae Accessed: 2025-01- 30.
Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan,
Fig. 15 presents the pairwise distances among several se- ries of LLMs, including the LLaMA2-7B, LLaMA3/3.1- 8B, Vicuna-7B, and MPT-7B families. These models nat- urally cluster into several groups, with models from the same family appearing close to each other. We can infer that LLaMA3 was trained from scratch without reusing the LLaMA2 checkpoint, while LLaMA3.1 was obtained by continuing training from the LLaMA3 checkpoint. Besides, we confirm that Vicunal.5 is fine-tuned from LLaMA2 in- stead of 1icunal.1/1.3. These findings are aligned with the publicly available training details. In addition, we also test distances between two models (Pythia-410M seedl and seed2) trained with the same data and same architecture, where only the random seeds are different. Their distances are approximately 90 degrees, showcasing the uniqueness of model signatures for independently trained models. Alto- gether, these results indicate that our LLM signature effec- tively captures and differentiates the relationships between models, validating its reliability for tracing model lineage.
A. A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., et al. Phi-3 technical report: A highly capable lan- guage model locally on your phone. arXiv preprint arXiv:2404. 220244444419999 14219, 2024.
Ainslic, J., Lee-Thorp J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., and Sanghai, S. Gqa: Training generalized multi-query transformer models from multi-head check- points. Empirical Methods in Natural Language Process- ing, 2023.
Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O'Brien, K., Hallahan, E.,, Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for ana- lyzing large language models across training and scaling. In International Conference on Machine Learning, pp. PMLR, 2023.
Darcet, T., Oquab, M., Mairal, J., and Bojanowski, P. Vi- sion transformers need registers. In The Twelfth In-

---

Demystifying Singular Defects in Large Language Models
ternational Conference on Learning Representations, 2024. URL httttpp://openreviw..oto forum? forumm 1d-2dm03LLiJ1.
K., Alami, R., and Hacid, H. Falcon2-11b technical re- port, 2024. URL https://: /ttpp:::///axivei//// org/ abs///b////// /tbb//477 14885.
Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35:30318 8033322, 2022.
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017. URL https:// openreview net worum Borummid-Byjj2udde
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for lan- guage understanding. In Burstein, J., Doran, C., and Solorio, T. (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume I (Long and Short Pupers), pp- 4171-41 1111--188 86, Min- neapolis, Minnesota, June 2019. Association for Compu- URL
Minaee, S., Mikolov, T., Nikzad, N., Chenaghlu, M., Socher, R., Amatriain, X., and Gao. J. Large language models: A survey. arXiv preprint 2rXiv:2402.06 196, 2024.
Mu, X., Wang. Y., Zhang, Y., Zhang, J., Wang, H., Xi- ang, Y., and Yu, Y. Model provenance via model dna. Endriss, U., Melo, F. S., Bach, K., Diz, A. B., Alonso-Moral, M., Barro, S., and Heintz, F. (eds.), ECAI 2024 27th European Conference on Artificial Intelligence, 19-24 October 2024, Santiago de Com- postela, Spain Including 13th Conference on Presti- gious Applications of Intelligent Systems (PAIS 2024), volume 392 of Frontiers in Artificial Intelligence and Applications, pp. 1043-1050. IOS Press, 2024. ISBN 978-1-64368-548-9 doi: 10.3233/FAIA240595 URL https: /dtii org/ 10. 3233 /FAIA244A2999
tational Linguistics. doi: 10.18653/1//19-1 https: /ttp://aaaaaatt /aw.aaaannnnoooooaaaaaaaaaaa
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre- trained transformers. The Eleventh International Confer- ence on Learning Reprexentations, 2023.
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The Ilama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Oquab, M., Darcet, T., Moutakanni, T. Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F,, El- Nouby, A., et al. Dinov2: Learning robust visual features without supervision. Transactions Machine Learning Research Journal, pp. 1-31, 2024.
Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A. Adam, H., and Kalenichenko, D. Quantization and training of neural networks for efficient integer- arithmetic-only inference. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2704-2713 2018.
Platzer, M. and Puschner, P. Vieuna: A timing prenng-predictable risc-v vector coprocessor for scalable parallel computa- tion. In 33rd euromicro conference on real-time systems (ECRTS 2021). Schloss Dagssuhl-Leibniz-Zennnnu für Informatik, 2021.
Jiang A. Q. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.- A., Stock, P., Scao, T. L., Lavril, T., Wang. T., Lacroix, T., and Sayed, W. E. Mistral 7b, 2023. URL https: //arxiv.one org/ abs/ 2210.. 06825.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, Language models are unsupervised multitask learners. 2019.
Sanh, V., Debut, L., Chaumond, J., and Wolf, T. Dis- tilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. In 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing @ NeurlPS 2019, 2019. BtLhhtt::// http://axxiv... wttpo//////////////// abs/1910. 01108.
Lin, J.. Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. Awq: Activation-aware weight quantization for on-device 11m compression and acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024.
Sun, M., Chen, X., Kolter, J. Z., and Liu, Z. Massive activa- tions in large language models. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foun- dation Models, 2024.
Liu, Y. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: /907. 1692, 364, 2019.
Malartic, Q. Chowdhury, N. R., Cojocaru, R., Farooq, M., Campesan, G., Djilali, Y. A. D., Narayan, S., Singh, A., Velikanov, M., Boussaha, B. E. A., Yafeai, M., Alobeidli, H., Qadi, L. A., Seddik, M. E. A., Fedyanin,
Team, M. N. Introducing mpt-7b: A new standard for open-source, commercially usable llms, 2023. URL www. mosaicmi www.//oo///oo..oo httm/blog/mppt.. 7b. Accessed: 2023- 05-05.
10

---

Demystifying Singular Defects in Large Language Models
Team, Q. Qwen2.5: A party of foundation models, Septem- ber 2024. URL https:://wwenmm.ggtt
/o/
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint 090292230..09288 2023.
Wang, B. and Komatsuzaki, A. Gpt-j-6b: a bil- lion parameter autoregressive language model (2021). URL https://github. 2022.
Wang, H., Zhang, T., and Salzmann, M. Sinder: Repairing the singular defects of dinov2. European Conference Computer Vision, pp. 20-35. Springer, 2024.
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference Machine Learning, pp. 38087-38099. PMLR, 2023.
Yu, R. and Wang, X. Neural lineage. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern 2024.
Recognition, pp.

---

Demystifying Singular Defects in Large Language Models
Mistral-7B-vDl
(b) Mistral-7B-Instrrce
(c) Mistral- Instruct-v0.2
(d) -B-tttttttttr
274
2.5
(e)
(f) SERT-Base-Uncse
BERT-Base- Case-Chinese
(h) BERT-Large Uncased
(i) RoBERTa-Baa Base
(i) LaBeRRa-Large
(k) DistilBERT BaasiBERT-Baae---e
(I) DistilBERT-Baa
Figure (Continuation Fig. 13). The example token norms in Mistral, BERT, RoBERTa DistilBERT models.
A. More Examples of High Norm Tokens
Fig. 17 shows more examples of high-non nigh-norm tokens in the LLM series. Specifically, it displays plethora of model variations in the LLaMA2, LLaMA3, ILLaMA3.1, LLaMA3.2, Phi3, and Qwen2 series. Without exception, they all exhibit the high-norm phenomenon, especially in the initial token. The patterns of the high-norm look similar if one model is fine-tuned from another, which echos our findings about the stability of high-norm tokens in Section 5.2.
B. More Examples of Token Norms in Mistral and BERT
Fig. 16 shows the norms of the first few tokens and the token '." in the sentence 'The quick brown fox jumps over the lazy dog." for more models in the Mistral, BERT, RoBERTa, and DistilBERT families.
More High-Norm Direction Statisties
Table 2 shows the average pairwise angles between all high-norm tokens more LLMs. The thresholds are determined by inspecting the visualization of the token norms in Fig. 1. The results confirm that all the high-norm tokens in one model have very similar directions across all layers and all tokens.
D. More Examples of Explosion Direction Prediction
Fig. 18 shows the angles between the predicted layer-wise singular defect directions and the empirical high-norm direction for more models. Notice that in some models, such as Phi3-Medde Mhi3-Medium, there are multiple explosion layers and decrease layers, and the angles between the predicted directions and the empirical high-norm directions are close those layers.
12

---

Demystifying Singular Defects in Large Language Models
Table Average pairwise angles hetween all high-norm tokens for each L.M. High tokens are collected all layers, and all tokens from 1000 rows in the WikiText2-vl dataset.
Model
Threshold
Mean Pairwise Angle (degree)
LLaMA2-7B
500
3.12
Phi3-Medium
2500
5.31
MPT-7B
1500
2.81
Pythia-160M
200
8.01
Vicumal.5-7B
400
4.15
Falcon2-118 1BB
3000
4.11
GPT2-Medium
3000
1.49
Qwen2.5-1.5B
8000
1.00
E. More Examples of Eigenvalues in Decay Layer
Fig. 19 shows the minimum angle between the eigenvectors of the linear approximation of the layer residual and the empirical high-norm direction for more models. Their corresponding eigenvalues are also plotted.
F. More Examples of Exploding Path
Fig. 20 shows the of single tokens after the explosion layer when removing all self-attention blocks from the model. The tokens with highest norms candidates for the noninitial high-norm tokens. Fig. 21 shows the norm of the initial token after the explosion layer. All trained tokens are plotted. Besides, number of random input embeddings are also used as initial tokens, and their norms after the explosion layer are plotted.
G. More Examples of Explosion Subspace
Fig. 22 shows that the leading right singular vector of the FFN module the explosion layer ignites the explosion of the token norms for more models. Fig. 23 shows the coefficients of tokens projected to the explosion subspace just before the FFN in the explosion layer of LLMs for more models.
13

---

Demystifying Singular Defects in Large Language Models
180
(a) LLaMA2-7B Chat
(b) LLaMA2-7B B-Code
(c) LLaMA2-7B -LaMee--B-Cooe PodeoodPt-tonnn (d) LLaMA2-7B Code-Instruce
(e) LLLMA2-13B
LLaMA2-70B
LLaMA3-8B
(h) LLaMA3-8B-Gun
LLaMA3-8B BB--trructt
LLaMA3.1-8B
(k) LLaMA3.1 1-B-GuarB
(I) LLaMA3.1 -
(m) LLaMA3.2-IB
(n) LLaMA3.2- IB-nnturuct
(0) LLaMA3.2-3B
(p) LLaMA3.2 2-.33..BBBBBBB BB-BIBIIute
(q) Phi3-Mini
Phi3-Mini-1225
Phi3-Medium-1253
(I) Phi3.5-Mini
(u) Qwen2-7B
(v) Qwen2-7B-B In-ssrruct
(w) Swen2-7B-Mand
(x) Down2--B-Mel 18-Maah-Instruet
Figure 17. (Continuation Fig. 1). High norm tokens in various LLM series. Each subfigure plots the norm of the few tokens and the the lazy dog." Here, the token 'The' is the initial token,
token '." in the sentence 'The quick brown fox jumps and it has a high norm in all models. The z-axis is the layer id, the y-axis shows different tokens, and the z z-axis is the norm. Layer 0 the input embedding layer, and the others transformer layers.
14

---

Demystifying Singular Defects in Large Language Models
Angle (degroo)
80
Angle (degrea)
80
60
40
20
20
6.32
3.24
3.36
3.29
10
15
5
25
Layer
Layer
(a) LLaMA2-20 7B Chat
(b) LLaMA2 7B-Code Code
Angle (dogroe)
80
Angle (degree))
80
40
20
15.93
18.68
20
16,41
18.89
5
10
20
25
30
35
40
15
20
25
30
35
40
Layer
Layer
(c) LLaMA2-13B
(d) LLaMA2- 13BB-113--- 13B-Chat
Angle (degrve)
Angle (degree)
50
60
40
35.25
40
35,08
15.51
20
15.65
%
5
10
15
20
25
30
5
10
15
20
25
30
Layer
Layer
(e) LLLMM3-SB
(I)
Angle (dogroe)
80
Angle (degree)
80
80-
60
20
8.468.60
16.55
15.30
10.58
8.708.76
15.15
10,19
0
10
15
25
30
10
20
25
30
Layer
Layer
(g) Phi3-Mini
(h) Phi3.5-Miin
Angle (dogroo)
80-
Angle (degree)
80
40-
40-
22.68
20-
8.438.82
16,50
15.85
10.81
20
16.94
10.94
12,69
10
15 Layer
30
Layer
Phi3-Mini- 128k
ii) Phi3-M Phi3-Medium
Angle (dogroe)
Angle (degree)
60-
60
40-
40
20-
11.28
10.81
20
10.46
11.32
5
10
15 Layer
20
25
10
15 Layer
20
25
(k) Qwen2-7B
(1) Qwen2-7B-Instruct
Figure 18. (Continuation of Fig. 4). Acute angles between layer- wise singular defect directions and empirical high-norm norm direction for models. We that small angles align with either the explosion the decay layer.
15

---

Demystifying Singular Defects in Large Language Models
Angle (degroo)
80
Angle (degree)
80
60-
Eigenvalue
60
Eigenvalue
40-
12.67
3.22
12.59
5.65
20-
20
20.
20
0.44
-20
1.14
-20
13.51
5
10
25
30
10
13
30
Layer
Layer
(a) LLLMM2277B-CL
(b) LLaMA2-7B Code
Angle (dogroo)
60-
80
441.14
Eigenvalue
Angle (degree)
60
37.42
Eigenvalus
40-
19.22
19.47
20-
20
20
0-
XXXXXX
0
0-
0.50
-20
0.51
10
15
25
15
Layer
Layer
(c) LLaMA2-13B
(d) LLaMA2- 13B-Chat
Angle (dogroc)
Angle (degree)
60-
Eigenvalus
60-
Eigenvalue
40
8.20
12.88
40
9.06
13.01
0
0.02
0.02
-2.51
20
0
5
10
15
20
25
30
10
13
20
25
30
Layer
Layer
(e) LLaMA3-SB
(I) LLaMA3-SB- 8B--Intructt
80
80
Angle (dogrve)
Angle (degree)
80-
40-
22.3
22.9
Eigenvalue
60-
40-
17.9
21.5
26.4
Eigenvalue
.88
20
20
20
11.7
5.9
20
6.0
29.00
5
10
15
20
25
30
10
15
20
25
30
Layer
Layer
(g) Phi3-Mini
(h) Phi3.5-Mini
62.6
Angle (degroe)
Angle (degree)
60.
38
60-
40-
19.9
22.2
25.4
Eigenvalue
40
16.4
19.5
Eigenvalus
7.8
20
XXXX
1.4
20
0.0
4.7
22.5
-00
-40
49.6
5
10
15
20
25
30
5
15
20
30
35
Layer
Layer
Phi3-Mini-1288
Phi3-Medium
Angle (degroo)
80-
Angle (degree)
80
60-
Eigenvalue
60
Eigenvalue
40-
16.0
10.9
13.5
11.1
20
20
20-
20
4.2
8.2
4.6
-20
20
15
Layer
Layer
(k) Qwen2-7B
(1) IB-Itruutt
Figure 19. (Continuation of Fig. 5). For each layer, the minimum angles between the eigenvectors of R and the empirical high-norm direction are shown in red, and the corresponding eigenvalues are shown in blue. Numbers for the explosion and decay layers are annotated.
16

---

Demystifying Singular Defects in Large Language Models
Norn after Layer 2
10
'PP
Norm after Layer"
10
10a
10a
'vn'
"oondssten"
nedertba barrd
10a
10
10
100
10"
5000
10000
15000
20000
25000
30000
5000
10000
15000
20000
25000
30000
Token ID
Token ID
(a) LLaMA2-7B-CHI
(b) LLaMA2-7B-COO
Norm after Layer 4
10
Norm after Layer 4
10
'c/s>"
10a
10a
10a
10a
10
10
10"
5000
10000
15000
20000
25000
30000
5000
10000
15000
20000
25000
30000
Taken ID
Token ID
(c) LLaMA2-13B
(d) LLaMA2-13B-Chr
Norm after Layer? 2
10
10c
Norm after Layer
10a
<<lbegin_of text
10a
'< Dfeeein
10a
10a
10a
10
10
10
20000
40000
60000
80000
100000
120000
20000
40000
60000
80000
100000
120000
Taken ID
Token ID
(c) LLaMA3-8B
(f) LLaMA3-8B ----A-----nnttuttt
Norm after Layer 3
104.
104
Norm after Layer
10a
"Guras" 'HOWND'
10a
'Gund"
10a
10a
101
10a
10
10a
5000
10000
15000
20000
25000
30000
0
5000
10000
15000
20000
25000
30000
Token
Token
g Phi3-Mini
(h) Phi3.5 MMiii
10
10a
<endofftext
Norn after Layer
Norm after Layer
10a
10a
10a
10a
a0
10'
100
10"
5000
10000
15000
20000
25000
30000
5000
10000
15000
20000
25000
30000
Token ID
Token ID
(i) Phi3- 1hi3-Mini-10 128k
(j)
10a
10a
Norm after Layer
Norm after Layer
10a
10a
10a
10a
10
101
10
10"
20000
40000 60000 80000 100000 120000 140000
20000
40000 60000 80000 100000 120000 140000
Taken ID
Token ID
(k) Qwen2-7B
()) Qw 2-B-tt-ct
Figure 20. (Continuation of Fig. 6). Attention-inenn innrriinennddeeeeettt exploding paths. The y-axis the norm of single tokens after the explosion layer when removing all self-attention blocks from the model. The largest few tokens together with their norms are annotated in the figure. Red color means they are noninitial high high-norm tokens.
17

---

Demystifying Singular Defects in Large Language Models
Norm after Layer' ?
1000
Norm after Layer
750
Trained Tokens
Trainod Tokens
500
Random Emboddings (sortod)
Random Embeddings (sortod)
250
1000
3000
10000
15000
20000
25000
30000
3000
10000
15000
20000
25000
30000
Token ID
Token ID
(a) LLaMA2-7B-Char
(b) LLaMA2-7B-Code
Norm after Layer 4
1500-
Norm after Layer
2000
1000
Trained Tokens
Traized Tokens
1000
Random Emboddings (sorted)
Emhheddings (sorted)
10000
15000
20000
30000
3000
10000
15000
20000
23000
Token ID
Token ID
(c) LLaMA2-13B
(d) LLaMA2-13B-Chr
Norm after Layer 2
600
Norm after Layer
400
200
Trained Tokens
Trained
Random Embeddings (sorted)
Random Emboddings (sorted)
200
200
20000
40000
20000
60000
100000
Token ID
Token ID
(c) LLLMA3-8B
(f) LLaMA3-88-Instuua
Norm after Layer I 3
4000
Norm aftor Layor
3000
3000
2000
Random Embediiigg (sorted)
Trained Tokens
2000
Random Embeddings (sorted)
Trained Tokens
1000
5000
10000
15000
20000
25000
30000
5000
10000
15000
20000
25000 30000
Token ID
Token ID
(g) Phi3-Minn
(h) Phi3.5-Mini
Norm after Layor 3
4000
Norm after I Layer 6
3000
Trained Tokens
Trained Tokens
2000
Random Embeddings (sorted)
Rantom Embeddings (sorted)
2000
1000
5000
10000
15000
20000
25000
30000
5000
10000
15000
20000
25000
30000
Token ID
Token ID
(i) Phi3-Mini-1288
(i) Phi3- MMddiMm
15000
15000
Norm after Layor
Norm after Layer
10000
Trained Tokens
Trained Tokens
Random Embeddiags (sorted)
Random Embeddings (sorted)
5000
5000
20000
10000 00000 80000 100000 120000110000
0
20000
40000 60000 30000 100000 120000 140000 Trken ID)
Token ID
(k) Qwen2-7B
()) Qwen2-7B I-Inntructt
Figure 21. (Continuation of Fig. 7). Norm of all trained tokens after the explosion layer when they used the initial token in input sequence. We also plot the norm of a number of random input embeddings (sorted by their output norms) after the explosion layer.
18

---

Demystifying Singular Defects in Large Language Models
Outpul Norm of FTN
Output Norm of FFN
10000
1000
leading right singular vector
leading right singular vector
7500
5000
2000
2500
1000
2000
3000
4000
1000
2000
3000
4000
Right Singular Vectors of F Layer's
Right Singular Vectors of F Layer
(a) LLaMA2-7B-Cha
(b) LLaMA2-7B-Code
Output Norm of FTNN
Output Norm of FPN
3000
leading right singular vector
3000
leading right singular vector
2000
2000
1000
1000
2000
3000
4000
5000
0
1000
2000
3000
4000
5000
Right Singular Vectors Layer 4
Right Singular Vectors in Layer 4
(c) LLMMA2113B
(d) L.LAAM--33B-CC 13h-Chat
Output Norm of FPN
Output Norm of FPN
600
leading right singular vector
600
leading right singular vector
400
400
200
200
2000
3000
4000
1000
3000
Right Singular Vectors of in Layer
Right Singular Vectors of F in Layer 2
(e) LLaMA3-SB
(I) SaLaMA3--B-Insteee
Output Norm of FFN
Output Norm PFN
8000
leading right singular vector
leading right singular
6000
5000-
2000
0
1000 Right Singular Vectors of F in Layer 3
2000
2500
3000
500
1000
1500
2000
2500
Right Singular Vectors of F in Layer 3
(g) Phi3-Mini
(h) Phi3.5-Minn
Output Norm of FTN
Output Norm of PFN
8000
leading right singular ventor
leading right singular vector
6000
20000
4000
10000
2000
500
1000
1500
2000
2500
3000
1000
2000
3000
4000
5000
Right Singular Vectors of F Layer 3
Right Singular Vectors of F in Layer
(i) Phi3-Mini- 128k
(i) thi3-Medium
Output Norm TFN
Output Norm FFN
40000
40000
leading right singular vector
leading right singular vector
30000
30000
20000
20000
10000
500
1000
1500
2000
2500
3000
3500
500
1000
1500
2000
2500
3000
3500
Right Singular Vectors of in Layer
Right Singular Vectors of Layer
(k) Qwen2-7B
(1) Dieen2-BB-Instruue
Figure 22, (Continuation Fig. 8). Norm of output tokens FFN the explosion layer using the right singular vectors F as inputs to FFN. We can see that the leading right singular vector of the FFN module in the explosion layer ignites the explosion of the token norms.
19

---

Demystifying Singular Defects in Large Language Models
Coefficient
Coefficient
Iuitial Position
Initial Position Second Pasition
Second Position
5000
10000
1,5000
20000
25000
30000
5000
10000
15000
20000
25000
30000
Token
Token ID
(a) LLa.A2-7B--CCC
(b) LLaMA2-78-Com
10
10-
Coofficiont
Coefficient
Initial Position
Initial Position Second Position
5
Second Position
5000
10000
15000
20000
25000
30000
5000
10000
15000
20000
25000
30000
Token ID
Token
(c) LMMM22-13
(d) L
Cosfficient
Coefficient
2-
1-
Ioitial Position Second Positiin
Initial Position Second Position
20000
40000
60000
80000
1 100000
120000
20000
40000
60000
80000
100000 120000
Token
Token ID
(c) LL
(f) LLLMA3-SB--nnaa
Coefficient
Confficient
5-
Initial Position
Initial Position
Second Position
Second Position
0-
15000
5000
25000
30000
Token ID
Tokes ID
(g) Phi3 Mini
(h) Phi3.5-Mini
300
Confficient
5
Coefficlent
Initial Position
200
Initial Position
Second Position
Second Position
100
5000
10000
15000
20000
25000
30000
5000
10000
15000
20000
25000
30000
Token ID
Token
(i) Phi3 Phi3-Mini-Il 128k
(i) Phi3 --dduum
10.
Coefficiont
Coefflcient.
Initial Position
Initial Position
Second Position
Second Position
0
20000
40000
60000
80000 100000 120000 140000
20000
40000
60000
80000 100000 120000 140000 Token ID
Token ID
(k) Qwen2-7B
Figure 23. (Continuation of Fig. 9). Coefficient of tokens projected to the explosion subspace just before the FFN in the explosion layer of LLMss The initial tokens have high component in the explosion subspace.
20


Reviews:
[{"review_id": "Fd4mGwUjpq", "reviewer": "Reviewer_3U5B", "summary": "This paper is a direct follow-up to SINDER (Wang et al, 2024). In this paper, the authors use the tool of “singular defects” to analyze the occurrence of high-norm tokens in language models, which was observed in (Sun et al. 2024). They analyze the weights of the model to understand where high-norm tokens come from, and how they are decayed. They find that certain weight matrix properties can be used to predict which layers produce high norm tokens. Finally, the authors demonstrate that high-norm tokens can be used to (1) improve quantization by not quantizing the layers responsible for handling the high-norm tokens and (2) act as a signature between language models, since it does not change much in fine tuning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed and thoughtful comments and we are encouraged by the remark that \"applications section is particularly strong\".\nWe will incorporate the suggestions in the revision.\n\n> How it fits into the broader literature and what exactly new it contributes\n\nWhile Sun et al. spotted high-norm tokens and SINDER provided a basic tool, our work advances the understanding from three aspects:\n\n* **Structural Characterization**. We introduce explosion subspace and decay eigenvalue analysis, providing a structured framework for high-norm token evolution in LLMs.\n* **Causal Insights**. We conjectured a link between high-norm token emergence and the causal self-attention.\n* **Practical Applications**. We demonstrate their impact on quantization and introduce model signatures for lineage tracing.\n\nWe highlight our unique contributions and whether they relate to SINDER and Sun et al. below.\n\n|Type|Source|Our Contribution|Sun et al.|SINDER|Comment\n|:-|:-|:-|:-|:-|:-\n|analysis|Sec. 3.1|The layer-wise singular direction predicts the explosion direction|x|related|SINDER studied ViT. We extend it to LLMs\n||Sec. 3.2|Negative eigenvalues of a layer explain its sudden decay|x|x\n||Sec. 3.3|The explosion pathway of the initial token links to self-attention, whereas that of the noninitial high-norm token is unrelated to self-attention|x|x\n||Sec. 3.4|High-norm tokens are triggered by the explosion subspace|x|x\n|observation|L326|High-norm tokens emerge early in training|x|x\n||L303 Column2, L379 Column2|The high-norm direction stabilizes during training and is robust to fine-tuning|x|x\n||L267-274|A systematic analysis that reveals all noninitial high-norm tokens|partial|x|Sun et al. failed to find tokens `。` and `
` for LLaMA2-7b\n|application|Sec. 5.1|Improvement on quantization schemes|x|x\n||Sec. 5.2|Design of LLM signatures|x|x\n\n> The linear model applies to one token and makes the analysis of noninitial tokens challenging\n\nFig. 6 shows that the noninitial high-norm tokens retain their high norms after removing all self-attentions.\nThey effectively behave as independent single-token within the network (L262-264), which makes their analysis by linear model feasible.\n\n> Sun et al. found that HNs occur in the same locations in the vectors, which is the same as \"high norm tokens share the same direction\"\n\nTheir observation is **not sufficient** to reach our conclusion that \"high norm tokens share the same direction\".\nFirstly, they show no evidence that the relative magnitude of massive locations is consistent.\nFor example, Tab. 1 and 2 of Sun et al. track the scalar magnitude by sorting, which discards the channel locations.\nSecondly, they did not quantify the influence of the non-massive locations (which are in majority) on vector direction.\nAs such, they cannot rigorously infer our conclusion.\n\n> \"Self-attention plays an insignificant role in noninitial HN tokens\" is the same as Sun et al's \"high-norm tokens act as fixed biases\"\n\nWe respectfully disagree.\nOur statement refers to the development of noninitial high-norm tokens, namely, they still exhibit high norm after removing self-attention (Fig. 6).\nContrastingly, self-attention is **indispensable** for initial high-norm tokens: When it is removed, initial tokens lose high norms (Figs. 7 and 6).\nThis underscores a fundamental difference to Sun et al's.\n\n> The findings from removing the explosion subspace are equivalent to Sun et al.'s zeroing out massive activations\n\nWe will add the following clarification in the revision. \"This observation echoes the experiments done by Sun et al., where the authors set the massive activations to zero\".\n\n> How does the quantization procedure differ from that of LLM.int8()?\n\nThe core innovation of LLM.int8() is to decompose a matrix into an 8-bit part and a 16-bit part.\nWe did not use this trick.\nBesides, they rely on row-wise quantization whereas we use tensor-wise quantization (L354-357 Column2).\nFinally, more than 50% of their matrices are affected (Fig. 3 in their paper), whereas ours only affects 2 matrices.\nWe will add LLM.int8() to the related work.\n\n> Why high-norm tokens occur during training in the first place?\n\nWe have ruled out 11 factors (L308-313 Column2) and singled out the most probable reason: The causal self-attention.\nIt is evidenced by the impact of different attention formulations (L318-329).\nWith causal attention, the possible output features for the initial token are finite ($=V$, the vocabulary size, e.g., 32K for LLaMA2).\nThe softmax loss pushes the norm of these features to grow for each training sample, resulting in high-norms.\nBy comparison, the count of possible output at the 2nd position is 1 billion ($=V^2$), which disperses the push toward increasing their norms.\n\n> The \"explanation for why there exists a set of fixed channels with high activations\" is not supported\n\nTo avoid confusion, we will revise it to \"This explains why the set of high-activation channels observed in Sun et al. is *fixed*\".", "claims_and_evidence": "The claims are generally supported by evidence. However, in my view much of the findings are \"re-discoveries\" of findings that were initially observed in Sun et al. rather than new findings.\n\nIn particular, Section 3.1 says that it identifies an \"explanation for why there exist a set of fixed channels with high activations\" - I don't think this is supported by the content of the paper. The fact that there is an explosion layer and a decay layer already was already identifed in Sun et al (2024) (Page 4, Fig 4). The singular defect analysis explains why tokens with high activations in those particular channels do not change, but it doesn't explain why those particular channels contain high activations, or how those occur in the first place. \n\nSimilarly, the finding in Section 3.3 that self attention doesn't affect high norm tokens seems to be a clear corollary of the finding in Sun et al that these tokens act as \"biases\" and are somewhat input independent (Page 6). \n\nFinally, the notion of the \"explosion subspace\" (section 3.4) and removing the relevant component is equivalent to the experiment in Sun et al that zeros out the high-norm channels of these tokens and finds a drop in performance; as expected. this finding exactly matches that of Sun et al. \n\nIn summary, I think a lot of the claims in this paper are certainly backed by evidence; it's just that these aren't new findings and just confirmation of phenomena already observed in prior work, couched in a different theoretical framework.", "methods_and_evaluation_criteria": "This paper is mostly based on analysis and measuring certain properties and phenomena that occur inside a fixed language model. The evaluation of these seems reasonable.\n\nThis paper doesn’t really require strong benchmarking in its experiments except for the applications section. I am not familiar with quantization benchmarks or LLM signature verification, but those experiments seem reasonable to me.", "theoretical_claims": "While the “theory of singular defects” is a main tool applied in this paper, it was developed originally in SINDER and does not really require rigorous proof. Although it relies on somewhat strong assumptions (ie single token, heavy linearization) I think it is a reasonable tool to analyze properties of the weights of a model, especially since it clearly identifies which layers are responsible for producing and decaying high norm tokens.", "experimental_designs_or_analyses": "I think the experiments and analysis are similar to those in SINDER, Darcet et al, and Sun et al; these relate to taking a pre-trained model and probing different parts of it. This seems reasonably sound to me.", "relation_to_broader_scientific_literature": "This paper is a direct extension / follow-up to SINDER. The original papers that described the phenomenon of large-norm tokens were Vision Transformers Need Registers (2023) and Massive Activations in LLMs (2024). These papers explored the existence and probed the function of these high norm tokens in language models. Notably, these papers did not try to find how these high norm tokens came to be during the training process.\n\nA more recent paper (Wang et al, 2024), SINDER, introduced the terminology of “defective” tokens and analyzed the weights of the revelation model. SINDER attempts to ‘repair’ the model by imposing a strong regularization on the singular values of the weight matrices. This paper applies the same technique, but to language models.", "essential_references_not_discussed": "I think this paper is missing a key reference: LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale\nby Dettmers et al. \n\nThis paper identifies the exact phenomeon in the Applications section - that quantization can be improved by acknowledging the existence of large activations, and keeping those in higher precision. As such, the application related to quantization is not new at all.", "other_strengths_and_weaknesses": "Strengths: The paper does do a careful analysis of the properties of these high norm tokens, and adds valuable analysis. I think the applications section is particularly strong, and it shows that acknowledging the existence of these special tokens is crucial for modifying pre-trained weights.\n\nWeaknesses: \nMy first main issue is that many of the empirical findings in this work are not new and were already discovered in Sun et al; the paper should be re-written to reflect that these confirm prior findings and are not observations of new phenomena, as I note in the Claims+Evidence section.\n\nMy second main issue is that the concept of the paper itself seems rather tenuous. It amounts to looking at the weight matrices and finding that certain values in it lead to large norm tokens, which is not actually the question of interest posed in Sun et al (2024) or Darcet et al (2023). That question is: why does this occur during training? Why do the weight matrices have these properties in the first place?. This  more interesting question remains unanswered (as noted in Section 4) and thus limits the value of this analysis towards further understanding the occurrence of high-norm tokens; the analysis in this paper doesn’t really add much beyond what was introduced in the SINDER paper.\n\nThe contribution of this paper instead is showing that you can apply linear algebra tools to analyze the behavior of activations under very strong assumptions (linearizing the attention and FFN blocks). This was already demonstrated in SINDER, and this paper applies it to the FFN layer. I don’t think there is much more theoretical contribution in this work beyond what was originally proposed in SINDER.\n\nFinally, the applications section does not not acknowledge prior work of Dettmers et al (noted in the above essential references section) which already applies mixed precision quantization to handle the existence of particularly high norm features. While this is not exactly the same (high norm channels/tokens vs features), this is essentially the same idea.\n\nAs such, I rate the paper as a weak reject.", "other_comments_or_suggestions": "1. The figures are far too small. These should either be larger or omitted from the paper. Figures 6-9 are also very small; the captions of these figures should not be the same size as the figures themselves.", "questions_for_authors": "The initial token analysis makes sense, since the linearization model only applies to one token. Doesn’t this make the analysis of non-initial tokens very challenging? I’m very confused as to how we can expect to draw any meaningful conclusions from this.\n\nIsn’t the finding in section 3.1 / FIgure 4 that high norm tokens shared the same direction already stated in Sun et al? They found that these HNs occur in the same locations in the vectors - this isn’t a new finding. The notion of “empirical high norm direction” is already considered in that work too, just under a different name.\n\n Isn’t the finding that “self attention plays an insignificant role in non-initial HN tokens” (Section 3.3) the same as the finding from Sun et al. that high-norm tokens act as fixed biases? They also found that they are independent of input values. \n\nIsn’t the finding that removing the “explosion subspace component” degrades performance essentially equivalent to the experiment of Sun et al that zeroes out the high-norm tokens?\n\nHow does the quantization procedure differ from that of LLM.int8()?\n\nThis is more of a comment, but the main thing that would change my evaluation of this paper is a better understanding of how it fits in to the broader literature and what exactly new it contributes. The analysis section as I currently understand it doesn’t demonstrate any meaningful new understanding of why high-norm tokens occur in the first place, and many of the experimental findings mirror those identified in Sun et al. What exactly does this work contribute beyond SINDER and Sun et al besides straightforward application to language models?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eFlMuuV7rD", "reviewer": "Reviewer_NnNK", "summary": "This paper investigates the phenomenon of high-norm tokens in large language models (LLMs), extending the understanding of singular defects from vision transformers (ViTs) to the context of LLMs. Unlike ViTs, where high-norm tokens have been modeled through singular vectors of linear approximations, the causes and characteristics of high-norm tokens in LLMs remain largely unexplored. The authors provide both theoretical insights and empirical validation across various recent models, leading to several key observations: \n\n1. The layer-wise singular direction predicts the abrupt explosion of token norms in LLMs.  \n2. Negative eigenvalues of a layer explain the sudden decay of high-norm tokens.  \n3. The computational pathways leading to high-norm tokens differ between initial and non-initial tokens.  \n4. High-norm tokens are triggered by the right leading singular vector of the matrix approximating the corresponding modules.\n\nThe authors demonstrate two such applications: improving quantization schemes and designing LLM signatures. The improved quantization strategy selectively preserves precision for critical layers, enhancing robustness without compromising efficiency. Meanwhile, the stable high-norm directions serve as robust signatures to trace model lineage and detect model infringement.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful comments!\nWe will incorporate the suggestions in the revision.\n\n> Is causal attention really the key factor? Why ViTs have similar high-norm tokens?\n\nWe hypothesize that causal attention excites high norms in LLM in the following way.\nIn the causal formulation, the possible output features at the initial position are finite ($=V$, the vocabulary size, e.g., 32K for LLaMA2).\nThe softmax loss increases the norm of these features for each training sample, so their norms grow fast.\nBy comparison, the possible output features at the 2nd position are 1 billion ($=V^2$).\nThe push towards increasing their norms is thus dispersed, yielding no systematic high norms in the 2nd and subsequent positions.\n\nThe high-norm tokens in ViTs could stem from different causes as implied by their drastically different patterns.\n1. They appear in the late stage of training, while High-norm tokens in LLMs appear in early stage (L324-327).\n2. They grow progressively layer by layer, while high-norm tokens in LLMs explode abruptly (L157-159).\n3. They appear at random locations, while high-norm tokens in LLMs mainly locate at the starting position (L160).\n4. They can be repaired by SINDER without affecting performance, while the presence of high-norm tokens is critical for the performance of LLMs (L310).\n5. The single-token assumption in ViTs is an ideal assumption to simplify the analysis. By contrast, in LLMs, the single-token assumption is real and valid for the initial token (L157 Column2).\n\nThese differences motivate the question \"whether the theory of singular defects can be applied to LLMs\" (L34-35 Column2) and our study gives an affirmative answer.\n\n> How about the behavior of high-norm vision tokens in the language models of VLMs? Are they working as the vision registers or behaving as the LLM high-norm tokens?\n\nTo answer this question, we forwarded an image-text mixed data using LLaVA-v1.5-7b, and located the image patch token with the highest norm in the ViT features.\nHere is its norm (round to integer) at different LLM layers:\n\n```txt\n764 (norm of input visual token), 764 (after 1st layer of LLM), 764, 765, 764, 764, 764, 762, 761, 760, 758, 756, 754, 752, 750, 748, 746, 744, 741, 740, 738, 738, 737, 736, 734, 734, 732, 732, 732, 730, 730, 729, 728\n```\n\nUnlike the high-norm text token in LLM, whose norm explodes and decays, the norm of high-norm vision token stays nearly the same throughout the LLM.\nIn addition, the angle between the high-norm vision token and the high-norm text token is kept around 87 degrees across LLM layers.\nBased on these, we hypothesize that high-norm vision tokens neither work as registers nor behave as LLM high-norm tokens.\n\n> What about the differences between the attention sink and the high-norm tokens?\n\nAttention sink refers to the phenomenon that the initial token receives large attention scores.\nIt is a side-effect emerging from the fact that the initial tokens have high norms, as discussed in the massive activation paper by Sun et al.\n\nWe believe our efforts on demystifying high-norm tokens provide a better understanding of the underlying mechanism of attention sinks.\nFor example, in Fig. 12, we show that model trained with window attention (Mistral) does not have initial high-norm tokens.\nAs a result, Mistral can process arbitrary context length without the special treatment introduced in the attention sink paper.\n\n> What is the actual role of the high-norm tokens in LLMs?\n\nIn Fig. 13, we show that high-norm tokens appear in the very early stages of training.\nWe therefore hypothesize that they are a way to accelerate network convergence by injecting some biases.\nTo verify this, we added an extra layernorm at the output of the residual branch of each attention block and that of each FFN block to suppress any possible high-norm tokens.\nWe trained the modified LLaMA2-1b model from scratch and observed that the loss decreased more slowly compared to the original structure.\n\n> Which kinds of text tokens are easier to trigger the high-norm token phenomenon?\n\nFirstly, as shown in Fig. 7, any token in the initial position will trigger the high-norm phenomenon.\nSecondly, some high-frequency tokens in the corpus such as `\\n`, `
` may appear as non-initial high-norm tokens.\nYet, the set of non-initial high-norm tokens may change during training.\nFor example, in Fig. 13 Column2, the '.' token is a non-initial high-norm token at training iteration 50k, but is no longer a high-norm token at iteration 143k.\nWe will extend Fig. 6 to more LLMs in the revision and below we summarize their non-initial high-norm tokens.\n\n|Model|Non-initial high-norm tokens\n|:-|:-\n|LLaMA2-7B-Chat|`
`, `.`, `。`, `\\n`\n|LLaMA2-7B-Code|`
`, `.`, `。`, `\\n`\n|LLaMA2-13B|`
`\n|LLaMA2-13B-Chat|`
`\n|LLaMA3-8B|`<\\|begin_of_text\\|>`\n|LLaMA3-8B-Instruct|`<\\|begin_of_text\\|>`\n|Phi3-Mini|None\n|Phi3.5-Mini|None\n|Phi3-Mini-128k|None\n|Phi3-Medium|`<\\|endoftext\\|>`\n|Qwen2-7B|None\n|Qwen2-7B-Instruct|None", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes. No issues are found yet.", "experimental_designs_or_analyses": "Yes. No issues are found yet.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n\n- I think it is a valuable paper that addresses an important yet underexplored topic: the behavior of high-norm tokens in LLMs. The findings provide a lot of new insights and greatly inspires the future works.\n\nWeaknesses:\n\n- This paper hypothesizes that the causal self-attention might be the underlying reason for the emergence of LLM's high-norm tokens. So I just wonder why ViTs (no causal attention) also have the similar high-norm tokens? Does the causal attention really the key factor of this phenomenon?\n\n- I am curious about the high-norm vision tokens in LLaVA-style Large VLMs. The paper shows the difference functions about the high-norm tokens in LLMs and ViTs. While in Large VLMs, some previous works noticed that the vision tokens in VLMs inherit the high-norm tokens greatly from the ViT (the vision encoder). How about the behaviors of these high-norm vision tokens in the language models of VLMs? Are they still working as the vision registers or behaving as the LLM high-norm tokens like claimed in this paper?\n\n- What about the differences between the attention sink[1] and the high-norm tokens discussed in this paper?\n\n- What is the actual role of the the high-norm tokens of LLMs discussed in this paper?\n\n- More discussions are expected, e.g., which kinds of text tokens are easier to trigger the high-norm token phenomenon?\n\n[1] Xiao, et al. Efficient Streaming Language Models with Attention Sinks. ICLR, 2024.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XeFuG3TEOx", "reviewer": "Reviewer_6zUQ", "summary": "This paper investigates the phenomenon of high-norm tokens in LLMs, identifying key factors that influence their behavior. These factors include singular directions, negative eigenvalues, and distinct computational pathways for initial and non-initial tokens. The study reveals that high-norm tokens are primarily driven by the leading singular vector of specific model components. These insights have practical implications for enhancing quantization schemes and designing LLM signatures.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your encouraging comments!", "claims_and_evidence": "N/A", "methods_and_evaluation_criteria": "N/A", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "N/A", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All reviewers agreed this paper should be accepted: it addresses an important problem, the analysis is carefully done, and the applications section is strong. The main concerns were around framing: reviewer 3U5B thought that the novelty claims of the paper were a bit exaggerated, and the authors clarified their contributions beyond prior work.
Decision: Accept (poster)