From Attention to Activation: Unraveling the Enigmas of Large Language Models

## Abstract

Abstract We study two strange phenomena in auto-regressive Transformers:
(1) the dominance of the first token in attention heads;
(2) the occurrence of large outlier activations in the hidden states.
We find that popular large language models,
such as Llama attend maximally to the first token in 98% of attention heads,
a behaviour we attribute to the softmax function.
To mitigate this issue,
we propose a reformulation of softmax to softmax-1 .
Furthermore, we identify adaptive optimisers, e.g. Adam, as the primary contributor to the large outlier activations and
introduce OrthoAdam , a novel optimiser that utilises orthogonal matrices to transform gradients,
to address this issue.
Finally, not only do our methods prevent these phenomena from occurring,
but additionally, they enable Transformers to sustain their performance when quantised using basic algorithms,
something that standard methods are unable to do.
In summary, our methods reduce the attention proportion on the first token from 65 65 65 65 % to 3.3 3.3 3.3 3.3 %,
the activation kurtosis in the hidden states from 1657 1657 1657 1657 to 3.1 3.1 3.1 3.1 ,
and perplexity penalty under 4-bit weight quantisation from 3565 3565 3565 3565 to 0.3 0.3 0.3 0.3 .

## 1 Introduction

Transformers have revolutionised machine learning, achieving state-of-the-art performance across diverse domains,
including natural language processing, computer vision and even protein structure prediction (OpenAI, ; Carion et al., ; Jumper et al., ).
However, the inner workings of auto-regressive Transformers remain enigmatic.
Recent studies (Elhage et al., ; Olsson et al., ; Bansal et al., ) unravelled some of their complexities,
yet our research reveals two surprising phenomena remain pervasive:

- 1.
The strong, consistent dominance of the first token in attention maps—see top of Figure 1.
- 2.
The presence of outlier activation values, across sequence position, in specific feature channels of the hidden states
(the intermediate features of each layer *after* the residual connection)
that are orders of magnitude larger than other values—see top of Figure 1.

We ask: What causes these phenomena? Are they essential to performant models? And, if not, how can we mitigate them?

These two phenomena are aesthetically curious, but also have important practical implications.
For instance,
Llama models (Touvron et al., ; Dubey et al., ) exhibit the aforementioned first token dominance of attention,
and so requiring complicated attention masking schemes to extend Llama models to tasks with
long sequences (Xiao et al., )
*i.e.* increase the maximum context length used during training.
This is particularly crucial for instruction-tuned models where long conversations are desirable (Wei et al., ; Ouyang et al., ).
Similarly, the presence of outlier activations leads to challenges in quantising large language models (LLMs).
Large outlier activations increase the required quantisation range (to capture the outliers),
resulting in low effective bits for the non-outlier activations, causing severe performance degradation post-quantisation.
To address this issue, prior work has proposed mixed-precision decomposition of LLMs (Dettmers et al., )
or complex scaling of the weights and activations which must be learnt for each model (Xiao et al., ).
Therefore, our additional motivation is to understand and mitigate these phenomena in a general manner,
such that these issues are resolved *during training*.

We begin by examining the attention mechanism, and surprisingly find, across numerous input sequences,
query tokens attend *most* to the first key token up to 98% of the time.
This is striking considering the limited semantic information the first token typically
contains—it is often a special token indicating the start of a sequence, such as <bos>.
We explore explanations for this, ruling out positional encodings, non-linearity choice, or feature normalisation.
Ultimately, we identify the softmax function used in the attention mechanism combined with causal masking as the root
cause—excessive attention on the first key token demonstrates an attention head effectively doing nothing (Bondarenko et al., ; Clark et al., ).
The first token is privileged due to causal masking; it is the only key token to which all query tokens can attend.
We propose a straightforward adjustment to softmax as a solution, *softmax-1*,
which removes first token dominance in attention (bottom of Figure 1).

Table: Table 1: Due to surprising phenomena in Transformer models, basic zeropoint 4-bit weight quantisation leads to catastrophic performance degradation. Our models trained with *softmax-1* and *OrthoAdam* exhibit improved robustness to quantisation.

Despite removing first token dominance in attention, using *softmax-1*, we find that the problem of
outlier activations in the hidden states persists.
Once again, we investigate potential causes of this issue and discover the outliers are
primarily caused by the use of adaptive optimisers, *e.g.* Adam (Kingma & Ba, ).
Specifically, our experiments show the exponential decaying averages of first and second moments of gradients result in outlier
activations.
To tackle this, we propose a novel optimiser, *OrthoAdam*, which transforms computed gradients using orthogonal matrices,
thus storing gradients in an alternative basis to the model parameters.
Our results demonstrate this optimiser eliminates the outliers in the hidden states of Transformers (bottom of Figure 1).

Our research extends beyond aesthetic curiosities.
While LLMs perform well despite first token dominance and outlier activations,
they lead to practical challenges.
Although advanced schemes have been developed to enable quantised LLMs to maintain their performance,
we show our approach enables LLMs to maintain their performance with the most basic quantisation methods,
such as per-tensor 8-bit *absmax* weight/activation quantisation and 4-bit *zeropoint* weight quantisation.
Thus, while our investigation began to better understand Transformers,
our methods offer additional practical benefits.

In summary, our contributions are as follows:

- •
We identify the dominance of the first token in attention and the occurrence of outliers in the
activations of the hidden states as significant issues in auto-regressive Transformers.
- •
We propose two simple, effective solutions: a reformulation of the softmax function, *softmax-1*,
to address the former issue, and a novel optimiser, *OrthoAdam*, to tackle the latter.
Our methods reduce first token attention from $65$% to $3.3$% and activation kurtosis from $1657$ to $3.1$.
- •
We demonstrate that these proposals not only resolve the identified problems but also lead
to practical improvements in the performance of Transformers under 8-bit weight/activation and 4-bit weight quantisation.
Our method reduces the perplexity penalty under 4-bit weight quantisation from $3565$ to $0.3$.

## 2 Problem Definition

This work investigates the two most prominent and strange phenomena of auto-regressive Transformer models:
(1) strong, consistent dominance of the first token in the attention maps;
(2) strong, consistent outlier activations in specific feature channels of the
hidden states (the intermediate features computed *immediately after* the residual connections)—see
top of Figure 1.
We aim to understand the cause of these phenomena and to propose individual
solutions for each of them.
They have been investigated or commented on previously (Bondarenko et al., ; Dettmers et al., ; Xiao et al., ),
but our work reaches different conclusions on the causes and suggests
novel solutions.
We start by describing these two anomalies in detail.

### 2.1 First Token Dominance in Attention Maps

The top of Figure 1 shows the attention map, averaged across all layers and heads, of a Transformer model,
specifically a pretrained GPT2-Medium model (Radford et al., ),
for a single real natural language sequence.
Strangely, in this average attention map the key corresponding to the first token receives the highest attention across all queries.
Quantitatively, we find the first key token is the most attended to key in $76$% of (query, head) pairs and
receives $52$% of all attention, when evaluating on the en validation split of the C4 dataset (Raffel et al., ; Dodge et al., ).
This behaviour is consistent across different LLMs, including
the Llama series (Touvron et al., ; Dubey et al., ), DeepSeek (Liu et al., ), and the GPT2 series (Radford et al., ).
See Appendix G for detailed examples of attention maps for these models.

Attention is a key component of the Transformer architecture,
and work on the interpretability of LLMs often focuses on
analysing attention (Elhage et al., ).
Moreover, many models, such as Llama2, use a special token for the beginning of a sequence (the <bos> token),
which is *always* the first token in an input sequence.
This makes first token dominance particularly
puzzling, as such models should learn the initial input structure easily.
We hypothesise that this phenomenon in the attention mechanism is a symptom of
a fundamental problem in the Transformer architecture and is not necessary for a performant auto-regressive Transformer.

### 2.2 Outlier Activations in the Hidden States

The top of Figure 1 shows the activation magnitude in the hidden states of a pretrained GPT2-Medium model.
We observe the hidden states of the Transformer model exhibit consistent outlier activations
in specific feature channels across all token positions (boxed red),
with the most extreme outliers occurring in the first token position (circled red).
Once again, this behaviour is consistent across different LLMs
and is invariant to the input sequence *i.e.* the same feature channels *always* exhibit outlier activations.
See Appendix F for examples of hidden states in pretrained models.

From a practical perspective, these outlier activations are problematic with regards
to quantising models for deployment (Lin et al., ; Dettmers et al., ).
However, from a theoretical perspective, the cause of these outlier activations is not well understood.
Previous works, have suggested these outliers are related to
first token domination in attention maps (Xiao et al., ; Bondarenko et al., ).
This is plausible for the most extreme outliers observed in the first token position,
but it does not explain the outlier activations observed across all token positions.
In this work, we show the two phenomena are unrelated and separate solutions are required
to address each.

## 3 Method: First Token Dominance of Attention Maps

We start by eliminating plausible causes of the first phenomenon of interest: first token dominance of attention maps.
We mainly consider GPT2 as a representative auto-regressive Transformer, because of its simplicity,
but also consider the more recent Llama2 model to narrow down possible causes of this phenomenon.
For all experiments, unless mentioned otherwise, we use a GPT2 model with 130M parameters,
trained on the en split of the C4 dataset.

### 3.1 Eliminating Certain Causes of First Token Dominance of Attention Maps

Both GPT2 and Llama exhibit first token dominance in attention maps.
Thus, we can rule out parts of their architecture that are different:

- •
Positional encoding. Llama models use Rotary Positional Encodings (RoPE) (Su et al., ), while GPT2 models uses learnt absolute positional encodings (Vaswani et al., ).
- •
Initial token. Llama models use a <bos> token to denote the beginning of a sequence, while GPT2 models do not.
- •
Activation function. Llama models use SiLU (Elfwing et al., ) in the feedforward layers, while GPT2 models use GeLU (Hendrycks & Gimpel, ).
- •
Feature Normalisation. Llama models use RMSNorm (Zhang & Sennrich, ), while GPT2 models use LayerNorm (Ba et al., ).

Note that Llama and GPT2 use different positional encoding,
but it is possible that any form of positional encoding might be cause of first token dominance.
To test this possibility,
we train a GPT2 model without any positional encodings and observe the attention maps.
We find equivalently trained GPT2 models with/without positional encodings exhibit first token dominance in $33$%/$20$% of (query, head) pairs and
allocate $17$%/$10$% of all attention to the first token.
Thus, we conclude that positional encodings are not the cause of these anomalies.
The models mentioned here are trained for relatively few steps and first token dominance
is more pronounced in our longer-trained models and in publicly available pretrained models.

### 3.2 Removing First Token Dominance of Attention Maps

After eliminating the above causes,
we have two aspects of Transformers that could cause first token dominance:
(1) causal masking in self-attention; and (2) softmax normalisation in attention heads.

Consider the self-attention mechanism on the initial token in a causal Transformer.
The first *query* token can only attend to its own key token and therefore it in receives an attention score of $1$,
due to softmax normalisation.
Similarly, the second query can only attend to the first two keys, whose attention scores must sum to $1$.
Prior work establishes attention heads specialise to concepts or concept groups (Bansal et al., ; Elhage et al., ).
However, given a query irrelevant to the specialisation of an attention head,
it must still allocate attention across the keys summing up to $1$.
Moreover, causal masking privileges the first key token above all others;
it is the only key token to which all tokens can attend.
This explains why the first token specifically dominates attention maps.

Clearly, a particular attention head should be able to attend nowhere if no relevant information is present.
Thus, we modify the softmax function to the following:

$$ $\displaystyle\text{softmax-1}(x_{i})$ $\displaystyle=\frac{\exp(x_{i})}{1+\sum_{j=1}^{L}\exp(x_{j})};\quad\quad\sum_{ i=1}^{L}\text{softmax-1}(x_{i})<1$ (1) $$

This modification removes the strict enforcement of attention scores summing to $1$,
allowing the model to allocate attention as it sees fit,
including having low attention scores everywhere.
From a registers/attention sink perspective (Darcet et al., ; Xiao et al., ),
the $1$ in the denominator is equivalent to a register/attention sink key token
which has 0 0 *dot product* with any query token.

Validating the hypothesis.
We train two GPT2 models, one with canonical softmax and one with softmax-1,
keeping all other variables the same.
The model trained with canonical softmax attention exhibits first token dominance;
the first key token is the most attended to key in $53$% of (query, head) pairs.
However,
the model trained with softmax-1 lowers this to just $2$%.
Furthermore, with canonical softmax $46$% of all attention is received by the first key, while using softmax-1
lowers this to $4\%$, thereby validating our idea.

Figure: Figure 2: Relaxing causal masking leads to attention domination by a different token
Refer to caption: x5.png

The difference in attention maps between canonical softmax and softmax-1 is shown in Figure 1,
which compares the attention maps of two models on the same input sequence.
Furthermore, we find using softmax-1 has no effect on training stability, convergence or model performance
(see Appendix H for the training curves of all our trained models).

What if causal masking is relaxed?
To verify the first token is privileged by causal masking, causing *first* token dominance,
we train a GPT2 model with canonical softmax in which
causal masking is removed for the first 10 tokens (the loss function is appropriately modified).
Figure 2 shows a specific token still dominating the attention map,
but it is the *fifth* one, not the first.

## 4 Method: Outlier Activations

To quantitatively establish the extent of outliers in the hidden states,
we use kurtosis.
Kurtosis, in this case, is a measure of tail heaviness of a set of activation values.
Activations which are normally distributed have a kurtosis of $\sim$$3$,
while higher kurtosis indicates a heavier-tailed distribution (*e.g.* the exponential distribution)
and lower kurtosis indicates a lighter-tailed distribution (*e.g.* the uniform distribution).
Given hidden states $\mathbf{X}\in\mathbb{R}^{M\times L\times D}$ of a Transformer model,
where $M$ is the number of layers, $L$ is the number of tokens and $D$ is the number of feature channels,
we compute the per-layer, per-position kurtosis of the hidden states as:

$$ $\displaystyle\kappa_{m,l}=\text{Kurt}_{m,l}\left[X_{m,l,d}\right]=\frac{ \mathbb{E}_{d}[(X_{m,l,d}-\mu_{m,l})^{4}]}{\mathbb{E}_{d}[(X_{m,l,d}-\mu_{m,l} )^{2}]^{2}},\quad\text{where}\quad\mu_{m,l}=\mathbb{E}_{d}[\mathbf{X}_{m,l,d}]$ (2) $$

where $X_{m,l,d}$ is the hidden state at layer $m$ at position $l$ for feature $d$,
and $\mu_{m,l}$ is the mean hidden state value at layer $m$ at position $l$.

### 4.1 Eliminating Certain Causes of Outlier Activations

We start by eliminating certain causes which could lead to the presence of outlier activations.

Feedforward Layer Biases. GPT2 uses biases in all feedforward layers,
while Llama uses none, therefore it is unlikely feedforward layer biases cause of outlier activations.

Normalisation Layers. GPT2 uses LayerNorm (Ba et al., ) while LLama uses RMSNorm (Zhang & Sennrich, ),
which both learn individual scaling parameters for each feature channel, potentially causing the outlier activations.
To remove such an effect,
we replace LayerNorm in our trained GPT2 models with an RMSNorm version which applies a *single* global scale instead of per-channel scaling,
and call it “RMSNormSingle”—similar to “Simple RMSNorm” from Qin et al. () which has no learned parameters.
We find outlier activations persist in the hidden states of a GPT2 model with RMSNormSingle.
In Table 4 we show kurtosis remains high in models trained without biases and/or with RMSNormSingle.

Optimiser.
Most Transformer models are trained with Adam (Kingma & Ba, ) or a variant.
These optimisers track the first and second moments of the computed gradients using exponential moving averages,
tracking these moments at a parameter level.
The main hyperparameters of Adam-like optimisers are $\beta_{1}$ and $\beta_{2}$,
which control the decay rates of the first and second moments, respectively.
If $\beta_{2}=0$, only the first moment of the gradients is tracked, resembling stochastic gradient descent (SGD) with momentum.
Conversely, if $\beta_{1}=0$, only the second moment of the gradients is tracked, resembling RMSProp.
We suspect that given the optimiser tracks moments in the same basis as the model parameters,
it is the most likely cause of the outlier activations in the hidden states auto-regressive Transformer models.

Validating the hypothesis.
We train a series of GPT2 models using Adam, RMSProp, SGD with and without momentum, tuning the learning rate and training schedule to encourage convergence.
The model trained with SGD has the slowest convergence and highest validation perplexity,
while the model trained with Adam converges the fastest and has the lowest perplexity.
However, we find models trained with Adam and RMSProp have high kurtosis, $140$ and $70$, respectively,
while training with SGD gives a kurtosis of $\sim$3.0.
We provide these results in our ablation study (Section 5.3).

Figure: Figure 3: Rotating vectors with dominant components leads to a reduction in the maximum absolute value.
Refer to caption: x6.png

### 4.2 OrthoAdam

The previous section leaves an important question for training Transformer models:
“How can we train a model with an optimiser which has the speed and convergence properties of Adam,
but produces activations properties similar to SGD”?

Optimisers which track exponential decaying averages of the first and/or second moments
of the gradients lead to outlier activations in the hidden states of Transformer models.
Moreover, in the models trained above,
the largest absolute *parameter* values correspond to the features which exhibit outlier activations
in the hidden states, *i.e.* if outlier activations occur in feature channel $i$ of the hidden states,
the largest model parameter values correspond to specific weights which act on
feature channel $i$ of the hidden states *e.g.* the $i$^th output channel of the output projection weights of the attention/MLP layers.
Therefore, to arrive at these large model parameter values, the optimiser (*e.g.* Adam) must
provide relatively large updates to these specific parameters and not others.
We note here that Adam and similar optimisers calculate gradient moments in the same basis as the model parameters.
Additionally, given the channels which contain outlier activations appear invariant to the input sequence,
we hypothesise that these channels are an artefact of the optimiser and do not correspond to any meaningful feature in the input
sequence—see Appendix F for plots of the hidden states of pretrained models with different input sequences.
Given these observations, we discuss an idealised case of observed hidden states below,
and show how orthogonal transformations can be used to reduce outlier activations.

Consider a $D$-dimensional vector, $\mathbf{x}=\alpha\mathbf{e}_{i}+\mathbf{z}$,
where $\mathbf{e}_{i}$ is the $i$^th unit vector in the standard basis,
$\mathbf{x}\in^{D}$, $\alpha\in^{+},\alpha\gg 1$ and $\mathbf{z}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$.
The first term represents the single outlier activation specific to the $i$^th channel
and the second term represents the “informative” activations.
The vector $\mathbf{x}$ represents the hidden states of a Transformer model with high kurtosis.
This simplified model makes two assumptions: (1) there is a single outlier activation channel; and
(2) the informative activations are normally distributed.

For values of $D$ similar to that of Transformer models, *i.e.* $D\approx[10^{3},10^{5}]$,
$\text{Kurt}[x_{j}]=O(D)$.
Therefore, we expect larger Transformer models of a given architecture to have larger kurtosis in their hidden states.
Moreover, the ratio of the $\ell_{\infty}$-norm to the $\ell_{2}$-norm of the hidden states
in our simplified model, $\frac{\left\lVert\mathbf{x}\right\rVert_{\infty}^{2}}{\left\lVert\mathbf{x}
\right\rVert_{2}^{2}}$, is close to $1$.
This ratio is another proxy for the extent of outliers.

Now we consider the effect of an appropriate orthogonal transformation on the vector $\mathbf{x}$.
Let $\mathbf{Q}\in^{D\times D}$ be an orthogonal matrix, and $\mathbf{y}=\mathbf{Q}\mathbf{x}$.
Under an particular orthogonal transformation,
$\frac{\left\lVert\mathbf{y}\right\rVert_{\infty}^{2}}{\left\lVert\mathbf{y}
\right\rVert_{2}^{2}}\approx\frac{1}{D}$
and $\text{Kurt}[y_{j}]=3$. The orthogonal transformation which achieves this is one which rotates the vector $\mathbf{x}$
such that $\mathbf{Q}\mathbf{e}_{i}=\frac{1}{\sqrt{D}}\mathbf{1}$.
Figure 3 illustrates this rotation process in 2D and 3D.
The kurtosis and norm ratio results quoted in this section are derived in Appendix C and Appendix D, respectively,
and are shown to be empirically valid for models we train from the plots in Section E.2 and Section E.3, respectively.

One option is to apply orthogonal transformations directly to the hidden states of the model during
*i.e.* make $\mathbf{Q}$ part of the model parameters but are kept fixed during training.
Instead, we propose a novel optimizer, *OrthoAdam*, which applies orthogonal transformations to incoming gradients
such that the moment calculations (which our experiments in Table 3 show are the key factor in producing outlier activations)
are performed in a different basis to the model parameters to prevent gradient updates to any particular set of parameters which lead to outlier activations.
We provide the full algorithm in Algorithm 1.

In our experiments, we randomly sample the orthogonal matrix for each parameter (which remains fixed during the training of the model).
We find that using *OrthoAdam* leads to a significant reduction in the kurtosis of hidden states
in Transformer models, effectively eliminating the outlier activations.
This is shown qualitatively at the top of Figure 1, where feature channels with high absolute activation values in the hidden states
*are no longer present* across all token positions, and quantitatively in Table 2 showing the kurtosis of hidden states
in models trained with *OrthoAdam* is close to $3$, with *no performance penalty*.

Figure: Algorithm 1 OrthoAdam, our proposed optimiser for reducing activation outliers. $\mathbf{\bar{g}}^{2}_{t}$ is the element-wise square $\mathbf{\bar{g}}_{t}\odot\mathbf{\bar{g}}_{t}$. With $\beta_{1}^{t}$ and $\beta_{2}^{t}$ we mean $\beta_{1}$ and $\beta_{2}$ taken to the power of $t$.

## 5 Experiments

Datasets.
We train all models on the en training split of the C4 dataset (Dodge et al., ; Raffel et al., ) and evaluate
on 100000 samples from the validation en split.

Models.
For our experiments, we train GPT2 models with $\sim${$60$M, $130$M, $350$M, $1.4$B} parameters
and Llama2 models with $\sim$$130$M parameters.
Apart from changing the softmax function, the only other changes we make to the model architectures are
the use of RMSNormSingle and we do not use biases in feedforward layers.
We ablate these changes in the ablation study at the end of this section.

Training.
Unless stated otherwise, we use a batch size of 512 and a cosine learning rate schedule with linear warmup for
{$1000$, $2000$, $6000$, $10000$} steps for models with {$60$M, $130$M, $350$M, $1.4$B} parameters respectively,
with a maximum learning rate of $10^{-3}$.
We train models with {$60$M, $130$M, $350$M, $1.4$B} parameters for {$160$k, $320$k, $960$k, 600k} steps respectively.
Note that we use a reduced number of steps for the $1.4$B parameter model due to computational constraints.
In the ablation study, we train GPT2 models with $130$M parameters for $40$k steps only.

Metrics.
We evaluate our experiments in the following metrics:
(1) the perplexity (PPL) of models on the validation set;
(2) the mean kurtosis across all layers of the model (evaluated separately for the first token and the remaining tokens);
(3) the maximum absolute activation across all layers of the model (again evaluated separately);
(4) the percentage of (query, head) pairs in which the first key token is the most attended to key token.
We calculate (1) to ensure our method *at least maintains* the vanilla language model performance
*i.e.* to ensure the model is not harmed by softmax-1 or OrthoAdam.
(2) and (3) show quantitatively the extent to which outlier activations are present in the hidden states.
Finally, (4) shows the extent to which the first token dominates attention in the model.

### 5.1 Main Results

We show the results of softmax-1 and OrthoAdam used to train GPT2 and Llama2 models in Table 2.
We observe that across both model architectures and all sizes,
the evaluated PPL is the same or slightly lower when comparing a model with softmax-1 and
trained with OrthoAdam to the vanilla model with neither,
indicating that our method does not change model performance.
Despite no significant change in PPL,
each of our proposed methods lead to a significant reduction in outlier activations in the hidden states
(shown by a considerably lower mean layer kurtosis
and maximum absolute activation), with the largest reduction observed when both softmax-1 and OrthoAdam are used.
In particular,
for GPT-2 models with $60$M, $130$M, $350$M and $1.4$B parameters,
the kurtosis without our modifications were $77.9$, $141.5$, $161.8$ and $351.0$,
while after our modification they drop to $7$, $7.3$, $3.1$, and $3.0$.
We observe similar results for Llama2-130M where the perplexity is around the same as the original version,
but kurtosis is reduced from $170$ to $6.9$.
Similar to kurtosis, in all cases we see a significant reduction of the mean activation value.
Furthermore, we also observe the drastic drop in first token attention.
While the vanilla versions of the model have maximal first token attention of up to $64.8$%,
after our modification, it is reduced to $1$-$3$%.

Table: Table 2: Main results showing the impact of *softmax-1* and *OrthoAdam* on trained GPT2 and Llama2 models. Utilising *softmax-1* and *OrthoAdam*, significantly reduces the kurtosis and the max activation values of hidden states. Using *softmax-1* only is sufficient to reduce first token dominance in attention. We generally find that all combinations of *softmax-1* and/or *OrthoAdam* at a given model size lead to similar performance. $\mathbb{E}_{m}\left[\kappa_{m,1}\right]$: mean kurtosis of the first token; $\mathbb{E}_{m}\left[\kappa_{m,>1}\right]$: mean kurtosis of all other tokens; $\mathbb{E}_{m}\left[\left|X_{m,1,d}\right|\right]$: mean max absolute activation value of the first token; $\mathbb{E}_{m}\left[\left|X_{m,>1,d}\right|\right]$: mean max absolute activation value of all other tokens. All values are averaged across all layers.

### 5.2 Quantisation

We quantise trained models
using *Absmax* and *Zeropoint* quantisation.
*Absmax quantisation* scales a given tensor (weight or activation) using the absolute maximum absolute value.
On the other hand,
*Zeropoint quantisation* shifts the quantised tensor such that the minimum tensor value is the minimum representable value.
See Dettmers et al. () for exact details on the quantisation schemes.

Experimental Setup.
We quantise the trained models using Absmax quantisation using 8-bit integers and the more powerful
Zeropoint quantisation using 4-bit integers.
In the case of Absmax quantisation, we use 3 different configurations:
(1) *fine* quantisation, where “per-channel” scaling is used for input activations and weights;
(2) *moderate* quantisation, with “per-tensor” scaling for input activations and weights;
and (3) *coarse* quantisation, with “per-tensor” scaling for input *and output* activations and weights.
In the case of Zeropoint quantisation, we use a single configuration where “per-channel” scaling is used for *weights only*.
Note that only linear layers are quantised the embeddings, normalisation layers and softmax functions are not quantised.

Table: Table 3: Performance of our trained models under various quantisation settings. When using *OrthoAdam* and *softmax-1* (OA + S1), the performance penalty due to quantisation is significantly reduced. The benefits of our proposed changes are more pronounced under more aggressive quantisation settings—*i.e.* 4-bit weight and coarse 8-bit weight/activation quantisation (vanilla models exhibit catastrophic performance degradation).

Results.
In Table 3 we show the results of quantising the trained models using Absmax
and Zeropoint quantisation.
We experimentally confirm that in all cases, models trained with softmax-1 and OrthoAdam are more robust to Absmax quantisation
schemes than models trained with the canonical softmax function and Adam.
The difference in performance is most pronounced when using moderate and coarse quantisation schemes—models
trained with softmax-1 and OrthoAdam are able to maintain performance while models trained with canonical
softmax and Adam suffer a significant degradation in performance.
In particular, in the coarse setting, our method outperforms the baseline by up to $36.12$ points.
For Zeropoint quantisation, we observe that all GPT2 models trained with canonical softmax and Adam
become unusable when using 4-bit integer weight quantisation,
while models trained with softmax-1 and OrthoAdam suffer only a small drop in performance.
Llama2 models in both cases remain usable after quantisation, but the performance drop
is more pronounced when using the canonical softmax function and Adam.

### 5.3 Ablation Study

Table 4 shows the results of an ablation study on GPT2 models with $130$M parameters.
As expected from the discussion in Section 3,
we find removing biases from linear layers and varying the position encodings does
not prevent first token domination—we see a small reduction in first token domination when positional
encodings are removed.

When using softmax-1, we find first token dominance is mitigated with only $\sim$2% of (query, head) pairs
having the first key token as the most attended to key token.

Switching from LayerNorm to RMSNorm with a learnt scale for each channel (RMSNorm-M, the normalisation used in Llama2) does not reduce
the prevalence of outlier activations in the hidden states.
However, switching to RMSNorm with a single learnt scale (RMSNorm-S) reduces the mean layer kurtosis and max absolute activation by $\sim$$40$%,
but still remain high.
In all of the above cases in which Adam is used as the optimiser, we observe similar perplexity to the initial model (top row).
Slight exceptions being the use of rotary and no positional encodings,
in which perplexity reduces and increases by $1.3$ and $0.5$, respectively.

Changing the optimiser to RMSProp leads to increased perplexity ($0.5$ compared to the initial model),
reduced mean layer kurtosis and max absolute activation, by $\sim$$50$% and $\sim$$30$%, respectively,
when comparing to the equivalent model trained with Adam.
In contrast to all previous cases, using SGD with/without momentum (on a longer schedule to encourage convergence),
leads to a significant decrease in mean layer kurtosis and max absolute activation, by up to 98% and 97%,
respectively, when comparing to the equivalent model trained with Adam.
However, using SGD requires a significantly longer training schedule to approach initial model performance.
Additionally, using SGD without momentum leads to a significantly higher perplexity ($6.8$ compared to the initial model).
This finding confirms the importance of the optimiser in causing outlier activations in the hidden states.

Using OrthoAdam yields the desirable results from SGD without momentum—namely
a significant decrease in mean layer kurtosis ($140$ to $3.0$) and max absolute activation ($432$ to $43.5$)
and the desirable results from Adam—namely similar perplexity to a model trained with Adam and therefore
much faster and better convergence than SGD without momentum.

The final three rows of Table 4 show that using OrthoAdam with softmax-1 and RMSNorm-S
leads to the most desirable results, and critically the removal of softmax-1 and the use of LayerNorm or RMSNorm-M
reintroduces first token attention dominance and outlier activations, respectively.

Table: Table 4: Ablation study on the impact of various architectural choices on the performance of a GPT2 model with $sim$130M parameter model. *SGD models are trained for $8\times$ longer than the others to encourage convergence.

## 6 Related Work

Language Models.
Current language models are based on Transformer models (Vaswani et al., ).
While there are Transformer-based LLMs that used the original encoder-decoder architecture such as T5 (Raffel et al., ),
researchers developed models such as BERT (Devlin et al., ) and RoBERTa (Liu et al., ), which are encoder-only.
However, most current LLMs such as the GPT series (Radford et al., ; ; Brown et al., )
and Llama series (Touvron et al., ; ; Dubey et al., ) use a decoder-only architecture.
In our work, we focus on this variant using GPT2 and Llama2 architectures.

Attention Dominance.
Bondarenko et al. () identify the dominance of bland tokens in the attention maps of the BERT encoder-only Transformer,
and suggest complex clipping schemes, additional hyperparameters, and a gating
mechanism to to mitigate this issue.
Other researchers found the same issue in long-range attention (Xiao et al., ) and found a workaround using “attention sinks” and discontinuous attention masking.
In vision Transformers, Darcet et al. () made the same observation and proposed a solution using “registers”.
In contrast to these works, we first find the root cause of this issue, the softmax mechanism in attention, and then reformulate it to prevent the first token dominance ever happening.

Outlier Activations.
Previous works have shown that in certain Transformer models which use post-normalisation
the norm of the *weights* of the learnt model must increase (Arora et al., ; Soudry et al., ).
However the same reasoning does not apply for most recent decoder-only Transformers which use pre-normalisation (Xiong et al., ) (*i.e.* normalisation
before the residual connection).
A blog-post by Elhage et al. () discusses the presence of outlier activations
in the hidden states of Transformer models and rules out numerical precision as the cause.
Another blog-post by Miller () posits the
activation outliers are caused by the attention mechanism,
however, we find outliers and attention dominance are disjoint phenomena.
He et al. () identify the presence of outliers and propose an “Outlier Protected Transformer Block” which makes many architectural changes such as
removing normalisation layers and severely downscaling the activations at the residual connection.
In our contrast, similar to first token dominance, we first find the root cause of this strange behaviour, and then fix it without doing architecture changes.

Outlier-Aware Quantisation.
The presence of outliers in the activations of the hidden states has led to
a number of works, such as LLM.int8 (Dettmers et al., ),
per-embedding group quantisation (Bondarenko et al., ), and
SmoothQuant (Xiao et al., ) propose varying
quantisation schemes to handle the presence of outliers, which require calibration.
In contrast, we eliminate the presence of outliers in our trained models thus enabling
the use of the most basic quantisation schemes such as Absmax and Zeropoint quantisation.

## 7 Conclusion

In this work, we study two surprising phenomena in large auto-regressive Transformers:
(1) the strong, consistent dominance of the first token in attention maps;
and (2) the presence of outlier activations in the hidden states.
We propose novel solutions:
(1) the softmax-1 function to remove first token dominance;
and (2) the OrthoAdam optimiser which mitigates outlier activations.
By doing so,
we reduce first token dominance of attention maps by up to $95$% and
the activation kurtosis by up to $99.8$%.
Furthermore, our work improves our understanding of Transformers but also offer practical benefits in model quantisation,
reducing the quantisation penalty by up to $99.9$%.

## Appendix A Training Details

In this section, we provide details on the training of our models.

In all experiments we use a batch size of 512 and in all experiments
using Adam or OrthoAdam as the optimiser, we use a peak learning rate of $10^{-3}$.
This excludes the experiments in Section 5.3 which use SGD as the optimiser,
which use a peak learning rate of $0.2$.
In all experiments we use a cosine learning rate schedule with linear warmup for
{1000, 2000, 6000, 10000} steps for models with {60M, 130M, 350M, 1.4B} parameters respectively.
Note that we use a reduced number of steps for the 1.4B model due to computational constraints.
For the main experimental results in Tables 2 and 3,
we train the models with {60M, 130M, 350M, 1.4B} parameters for {160k, 320k, 960k, 600k} steps respectively.
For the ablation study in Section 5.3,
we train GPT2 models with 130M parameters for 40k steps with 2000 warmup steps.
We use a maximum sequence length of 256 tokens,
which we find is sufficient to observe the anomalies of first token attention dominance and large outlier activations
found in popular pretrained models such as GPT2 (Radford et al., )
and Llama (Touvron et al., ; ; Dubey et al., ).
The result of our training setup is that models trained for the main experimental results with {60M, 130M, 350M, 1.4B} parameters
are trained on {21B, 42B, 126B, 79B} tokens respectively.
The ablation experiments are trained on 5B tokens.
We train models on 8 NVIDIA 32GB V100 GPUs using the Pytorch deep-learning framework (Paszke et al., )
and the HuggingFace Transformers library (Wolf et al., ).

## Appendix B Note on Supplementary Material

The supplementary material contains three folders:

- •
our_attention_maps/: additional attention map plots for our trained models from Section 5.1, using softmax-1 removes first token dominance.
- •
our_hidden_states/: additional hidden state plots for our trained models from Section 5.1, using OrthoAdam and softmax-1 together removes outlier activations.
- •
our_output_weights/: plots showing the maximum norm of the output features for the final feedforward layer in each attention and MLP sub-block in
our trained models from Section 5.1, using OrthoAdam and softmax-1 leads to a large reduction in the maximum norm of the output weights.

## Appendix C Kurtosis grows with the number of dimensions in Transformers

In this section, we use some observations from the hidden states of transformer models to
illustrate how the kurtosis of the hidden states grows with the number of dimensions in the hidden states.
This is something we observe empirically in the hidden states of transformer models and is a key motivation for our work.
Table 2 shows the kurtosis of the hidden states of transformer models
trained *without softmax-1 or OrthoAdam* grows as the model size increases,
as does the maximum activation value in the hidden states.

To make this mathematically rigorous, we shall consider a simple example,
in which we shall approximate the hidden states of a transformer model at a single token position
as a $D$-dimensional vector comprising of the sum of a scaled one-hot vector and a standard normal vector.

Consider a $D$-dimensional vector $\mathbf{x}$ which is the sum of two $D$-dimensional vectors
$\alpha\mathbf{e}_{i}$ and $\mathbf{z}$, where $\mathbf{e}_{i}$ is the $i$th unit vector in the standard basis,
$\mathbf{x}\in^{D}$, $\alpha\in$ and $\mathbf{z}\sim\mathcal{N}(\mathbf{0},\mathbf{I}_{D})$.
Therefore the elements of $\mathbf{x}$ are given by:

$$ $\displaystyle x_{j}$ $\displaystyle=\alpha\delta_{ij}+z_{j}\quad\text{for}\leavevmode\nobreak\ j=1,2 ,\ldots,D$ $$

where $\delta_{ij}$ is the Kronecker delta function.
The mean is given by:

$$ $\displaystyle\mu=\mathbb{E}\left[x_{j}\right]$ $\displaystyle=\mathbb{E}\left[\alpha\delta_{ij}+z_{j}\right]=\alpha\mathbb{E} \left[\delta_{ij}\right]+\mathbb{E}\left[z_{j}\right]$ $\displaystyle=\frac{\alpha}{D}+0=\frac{\alpha}{D}\quad\text{as}\leavevmode \nobreak\ \mathbb{E}\left[z_{j}\right]=0\leavevmode\nobreak\ \text{by definition of the standard normal distribution}$ $\displaystyle\mu$ $\displaystyle=\frac{\alpha}{D}$ (C.1) $$

The variance is given by:

$$ $\displaystyle\sigma^{2}$ $\displaystyle=\text{Var}\left[x_{j}\right]=\text{Var}\left[\alpha\delta_{ij}+z _{j}\right]$ $\displaystyle=\alpha^{2}\text{Var}\left[\delta_{ij}\right]+\text{Var}\left[z_{ j}\right]\quad\text{as}\leavevmode\nobreak\ \alpha\delta_{ij}\leavevmode \nobreak\ \text{and}\leavevmode\nobreak\ z_{j}\leavevmode\nobreak\ \text{are independent in our model}$ $\displaystyle=\alpha^{2}\text{Var}\left[\delta_{ij}\right]+1\quad\text{as} \leavevmode\nobreak\ \text{Var}\left[z_{j}\right]=1\leavevmode\nobreak\ \text{ by definition of the standard normal distribution}$ $\displaystyle\text{Var}\left[\delta_{ij}\right]$ $\displaystyle=\mathbb{E}\left[\delta_{ij}^{2}\right]-\left(\mathbb{E}\left[ \delta_{ij}\right]\right)^{2}=\frac{1}{D}\left(1-\frac{1}{D}\right)$ Therefore: $\displaystyle\sigma^{2}$ $\displaystyle=\frac{\alpha^{2}}{D}\left(1-\frac{1}{D}\right)+1$ (C.2) $$

The kurtosis of the elements of $\mathbf{x}$ is given by:

$$ $\displaystyle\text{Kurt}[x_{j}]$ $\displaystyle=\mathbb{E}\left[{\left(\frac{x_{j}-\mu}{\sigma}\right)}^{4} \right]=\frac{\mathbb{E}\left[\left(x_{j}-\mu\right)^{4}\right]}{\sigma^{4}}$ $$

When $j\neq i$:

$$ $\displaystyle\mathbb{E}\left[\left(x_{j}-\mu\right)^{4}\right]$ $\displaystyle=\mathbb{E}\left[\left(z_{j}-\frac{\alpha}{D}\right)^{4}\right]$ $\displaystyle=\mathbb{E}\left[z_{j}^{4}-4z_{j}^{3}\frac{\alpha}{D}+6z_{j}^{2} \left(\frac{\alpha}{D}\right)^{2}-4z_{j}\left(\frac{\alpha}{D}\right)^{3}+ \left(\frac{\alpha}{D}\right)^{4}\right]$ $\displaystyle=\mathbb{E}\left[z_{j}^{4}\right]-4\mathbb{E}\left[z_{j}^{3} \right]\frac{\alpha}{D}+6\mathbb{E}\left[z_{j}^{2}\right]\left(\frac{\alpha}{D }\right)^{2}-4\mathbb{E}\left[z_{j}\right]\left(\frac{\alpha}{D}\right)^{3}+ \left(\frac{\alpha}{D}\right)^{4}$ As $\mathbb{E}\left[z_{j}^{3}\right]=0$ and $\mathbb{E}\left[z_{j}^{4}\right]=3$ : $\displaystyle=3+6\left(\frac{\alpha}{D}\right)^{2}+\left(\frac{\alpha}{D} \right)^{4}$ $$

When $j=i$:

$$ $\displaystyle\mathbb{E}\left[\left(x_{j}-\mu\right)^{4}\right]$ $\displaystyle=\mathbb{E}\left[\left(\alpha+z_{j}-\frac{\alpha}{D}\right)^{4}\right]$ $\displaystyle=\mathbb{E}\left[\left(\alpha\left(1-\frac{1}{D}\right)+z_{j} \right)^{4}\right]$ $\displaystyle=\mathbb{E}\left[\left(\alpha\left(1-\frac{1}{D}\right)\right)^{4 }+4\left(\alpha\left(1-\frac{1}{D}\right)\right)^{3}z_{j}\right.$ $\displaystyle\left.\qquad\quad+\leavevmode\nobreak\ 6\left(\alpha\left(1-\frac {1}{D}\right)\right)^{2}z_{j}^{2}+4\left(\alpha\left(1-\frac{1}{D}\right) \right)z_{j}^{3}+z_{j}^{4}\right]$ $\displaystyle=\left(\alpha\left(1-\frac{1}{D}\right)\right)^{4}+6\left(\alpha \left(1-\frac{1}{D}\right)\right)^{2}+3$ $$

Therefore, the *overall fourth moment* of the elements of $\mathbf{x}$ is given by:

$$ $\displaystyle\mathbb{E}\left[\left(x_{j}-\mu\right)^{4}\right]$ $\displaystyle=\frac{1}{D}\left(\left(\alpha\left(1-\frac{1}{D}\right)\right)^{ 4}+6\left(\alpha\left(1-\frac{1}{D}\right)\right)^{2}+3\right)$ $\displaystyle\quad+\frac{D-1}{D}\left(3+6\left(\frac{\alpha}{D}\right)^{2}+ \left(\frac{\alpha}{D}\right)^{4}\right)$ $$

And the kurtosis of the elements of $\mathbf{x}$ is given by:

$$ $\displaystyle\text{Kurt}\left[x_{j}\right]$ $\displaystyle=\frac{\frac{1}{D}\left(\left(\alpha\left(1-\frac{1}{D}\right) \right)^{4}+6\left(\alpha\left(1-\frac{1}{D}\right)\right)^{2}+3\right)+\frac{ D-1}{D}\left(3+6\left(\frac{\alpha}{D}\right)^{2}+\left(\frac{\alpha}{D}\right )^{4}\right)}{\left(\frac{\alpha^{2}}{D}\left(1-\frac{1}{D}\right)+1\right)^{2}}$ $\displaystyle\text{Kurt}\left[x_{j}\right]$ $\displaystyle=\frac{3+\frac{\alpha^{4}}{D}+\frac{6\alpha^{2}}{D}-\frac{4\alpha ^{4}}{D^{2}}-\frac{6\alpha^{2}}{D^{2}}+\frac{6\alpha^{4}}{D^{3}}-\frac{3\alpha ^{4}}{D^{4}}}{1+\frac{2\alpha^{2}}{D}-\frac{2\alpha^{2}}{D^{2}}+\frac{\alpha^{ 4}}{D^{2}}-\frac{2\alpha^{4}}{D^{3}}+\frac{\alpha^{4}}{D^{4}}}$ (C.3) $$

At this point, we can see that Kurtosis is a function of $\alpha$ and $D$, however if
we consider the limit as $D\to\infty$, we can see that $\text{Kurt}[x_{j}]\to 3$ *i.e.* the kurtosis of a Gaussian distribution.
However, this neglects the importance of the scaling factor $\alpha$ which we know empirically is larger than the dimensionality of the hidden states.
The table below summarises the maximum activation values (analogous to $\alpha$) and the dimension of the hidden states for the models we trained.

**Table 5: Model sizes and maximum activation values for the models used in our experiments.**
| Model | #Parameters | Model Size ($D$) | Max Activation ($\alpha$) |
| --- | --- | --- | --- |
| GPT2 | 60M | 512 | 1856 |
| 130M | 768 | 7018 |  |
| 350M | 1024 | 40196 |  |
| 1.4B | 2048 | 56798 |  |
| Llama | 130M | 768 | 4623 |

Given this empirical information, we make the *conservative* assumption that $\alpha=D$.
Under this assumption which is supported by our empirical observations, Equation C.3 simplifies to:

$$ $\displaystyle\text{Kurt}\left[x_{j}\right]$ $\displaystyle=\frac{3+\frac{D^{4}}{D}+\frac{6D^{2}}{D}-\frac{4D^{4}}{D^{2}}- \frac{6D^{2}}{D^{2}}+\frac{6D^{4}}{D^{3}}-\frac{3D^{4}}{D^{4}}}{1+\frac{2D^{2} }{D}-\frac{2D^{2}}{D^{2}}+\frac{D^{4}}{D^{2}}-\frac{2D^{4}}{D^{3}}+\frac{D^{4} }{D^{4}}}$ $\displaystyle=\frac{3+D^{3}+6D-4D^{2}-6+6D-3}{1+2D-2+D^{2}-2D+1}$ $\displaystyle=\frac{D^{3}-4D^{2}+12D-6}{D^{2}}$ $\displaystyle\text{Kurt}\left[x_{j}\right]$ $\displaystyle=D-4+\frac{12}{D}-\frac{6}{D^{2}}=O(D)$ (C.4) $$

Using our conservative assumption that $\frac{\alpha}{D}=1$,
we can see that the kurtosis of the hidden states grows linearly with the dimensionality of the hidden states
when $D$ is in the region of $10^{3}-10^{5}$ as is the case for transformer models.

This simple example serves as a mathematical illustration of the empirical observations we make in the hidden states of transformer models.
We have shown that the kurtosis of the hidden states is expected to grow linearly with the dimensionality of the hidden states,
and so the issue of outlier activations is expected to grow as the hidden states of transformer models grow in size.

## Appendix D Orthogonal transformations and reduction in ℓ ∞ subscript ℓ \ell_{\infty} roman_ℓ start_POSTSUBSCRIPT ∞ end_POSTSUBSCRIPT –norm and Kurtosis

From our simple model in Appendix C we have a simplified model of Transformer hidden states, $\mathbf{x}\in^{D}$,
where the first element is $\alpha$ and the rest are standard normal random variables.

$$ $\displaystyle\mathbf{x}$ $\displaystyle=\alpha\mathbf{e}_{i}+\mathbf{z}\quad\text{where}\leavevmode \nobreak\ z_{j}\sim\mathcal{N}(0,1)$ $$

From this model, we can compute the expected $\ell_{2}$–norm:

$$ $\displaystyle\mathbb{E}\left[\left\lVert\mathbf{x}\right\rVert_{2}^{2}\right]$ $\displaystyle=\sum_{j=1}^{D}x_{j}^{2}=\alpha^{2}+\sum_{j=1}^{D}z_{j}^{2}= \alpha^{2}+D\text{Var}\left[z_{j}\right]=\alpha^{2}+D$ (D.1) $$

Using the triangle inequality, we can compute a range for the $\ell_{\infty}$–norm:

$$ $\displaystyle\mathbb{E}\left[\left\lVert\mathbf{x}\right\rVert_{\infty}\right] =\mathbb{E}\left[\max_{1\leq j\leq D}\left(\left|\alpha+z_{i}\right|,\max_{j \neq i}\left|z_{j}\right|\right)\right]$ Given $\alpha\gg 1$ , we can drop the terms for $j\neq i$ and compute the expected $\ell_{\infty}$ –norm using the $i$ th element: $\displaystyle\mathbb{E}\left[\left\lVert\mathbf{x}\right\rVert_{\infty}\right] =\mathbb{E}\left[\left|\alpha+z_{i}\right|\right]$ $\displaystyle\left|\alpha+z_{i}\right|\leq\left|\alpha\right|+\left|z_{i}\right|$ Using folded normal distribution properties, $\mathbb{E}\left[\left|z_{i}\right|\right]=\sqrt{\frac{2}{\pi}}\ll\alpha$ , therefore: $\displaystyle\mathbb{E}\left[\left\lVert\mathbf{x}\right\rVert_{\infty}\right]\approx\alpha$ $$

Given that $\alpha\gg 1$, we can safely assume that $\left\lVert\mathbf{x}\right\rVert_{\infty}^{2}\approx\alpha^{2}$.
Therefore:

$$ $\displaystyle\mathbb{E}\left[\frac{\left\lVert\mathbf{x}\right\rVert_{\infty}} {\left\lVert\mathbf{x}\right\rVert_{2}}\right]$ $\displaystyle\approx\frac{\alpha}{\sqrt{D+\alpha^{2}}}$ Note from Table 5 that the maximum activation value, $\alpha$ , is generally much larger than the model size, $D$ . $\displaystyle\mathbb{E}\left[\frac{\left\lVert\mathbf{x}\right\rVert_{\infty}} {\left\lVert\mathbf{x}\right\rVert_{2}}\right]$ $\displaystyle\approx 1$ (D.2) $$

We find this empirically to be the case in the middle layers of the Transformer models we study (see plots in Section E.3).

The $\infty$-norm of $\mathbf{x}$ can be thought of as a proxy for the extent of outliers in a vector.
If $\frac{\left\lVert\mathbf{x}\right\rVert_{2}}{\left\lVert\mathbf{x}\right\rVert
_{\infty}}\approx 1$,
then a vector has at least one large outlier and consequently a high kurtosis.

We will now show that applying an orthogonal transformation to a vector can reduce the $\ell_{\infty}$-norm constrained
to a fixed $\ell_{2}$-norm. Using the same definition of $\mathbf{x}$ as above, let $\mathbf{Q}\in^{D\times D}$ be an orthogonal matrix
and let $\mathbf{y}=\mathbf{Q}\mathbf{x}$.

$$ $\displaystyle\left\lVert\mathbf{y}\right\rVert^{2}_{2}$ $\displaystyle=\mathbf{y}^{T}\mathbf{y}=\mathbf{x}^{T}\mathbf{Q}^{T}\mathbf{Q} \mathbf{x}=\mathbf{x}^{T}\mathbf{x}=\left\lVert\mathbf{x}\right\rVert^{2}_{2}$ $\displaystyle\mathbb{E}[\left\lVert\mathbf{y}\right\rVert^{2}_{2}]$ $\displaystyle=\mathbb{E}[\left\lVert\mathbf{x}\right\rVert^{2}_{2}]=\alpha^{2}+D$ (D.3) $$

This standard proof shows that applying an orthogonal transformation to a vector does not change the $\ell_{2}$–norm of the vector.
It can however lead to a dramatic reduction in the $\ell_{\infty}$–norm of the vector.
We will now show that for a vector, $\mathbf{y}\in^{D}$, constrained to have a fixed $\ell_{2}$–norm, $\sqrt{\alpha^{2}+D}$,
the $\ell_{\infty}$–norm of a vector can be reduced significantly by applying an orthogonal transformation
such that $y_{j}=\frac{\alpha}{\sqrt{D}}+z^{\prime},\forall j\in[1,D]$, where $z^{\prime}\sim\mathcal{N}(0,1)$.

$$ $\displaystyle\mathbf{y}$ $\displaystyle=\mathbf{Q}\mathbf{x}=\mathbf{Q}\left(\alpha\mathbf{e}_{i}+ \mathbf{z}\right)=\alpha\mathbf{Q}\mathbf{e}_{i}+\mathbf{Q}\mathbf{z}$ Select $\mathbf{Q}$ such that $\mathbf{Q}\mathbf{e}_{i}=\left(\frac{1}{\sqrt{D}},\frac{1}{\sqrt{D}},\ldots, \frac{1}{\sqrt{D}}\right)$ , given $\mathbf{Q}$ is orthogonal, $\mathbf{Q}\mathbf{z}\leavevmode\nobreak\ =\leavevmode\nobreak\ \mathbf{z^{ \prime}}\leavevmode\nobreak\ \sim\leavevmode\nobreak\ \mathcal{N}(0,\mathbf{I} _{D})$ . $\displaystyle\mathbb{E}\left[\left\lVert\mathbf{y}\right\rVert_{\infty}\right]$ $\displaystyle\approx\frac{\alpha}{\sqrt{D}}+\sqrt{2\ln{D}},\quad\text{using extreme value theory\leavevmode\nobreak\ \cite[citep]{(\@@bibref{AuthorsPhrase 1Year}{cramer1946mathematical}{\@@citephrase{, }}{})}}$ $\displaystyle=\frac{\alpha+\sqrt{2D\ln{D}}}{\sqrt{D}}$ $$

The expected ratio of $\ell_{\infty}$–norm to $\ell_{2}$–norm is:

$$ $\displaystyle\mathbb{E}\left[\frac{\left\lVert\mathbf{y}\right\rVert_{\infty}^ {2}}{\left\lVert\mathbf{y}\right\rVert_{2}^{2}}\right]$ $\displaystyle=\frac{\mathbb{E}\left[\left\lVert\mathbf{y}\right\rVert_{\infty} ^{2}\right]}{\mathbb{E}\left[\left\lVert\mathbf{y}\right\rVert_{2}^{2}\right]} =\frac{\left(\alpha+\sqrt{2D\ln{D}}\right)^{2}}{D\left(\alpha^{2}+D\right)}$ Using the same conservative assumption as in Appendix C that $\alpha=D$ , Table 5 shows empirically $\alpha>D$ : $\displaystyle\mathbb{E}\left[\frac{\left\lVert\mathbf{y}\right\rVert_{\infty}^ {2}}{\left\lVert\mathbf{y}\right\rVert_{2}^{2}}\right]$ $\displaystyle=\frac{D^{2}+2D\ln{D}+2D\sqrt{2D\ln{D}}}{D^{3}+D^{2}}=\frac{D+2 \sqrt{2D\ln{D}}+2\ln{D}}{D^{2}+1}$ As $D$ grows, the last term of the numerator and the $1$ in the denominator become negligible: $\displaystyle\mathbb{E}\left[\frac{\left\lVert\mathbf{y}\right\rVert_{\infty}^ {2}}{\left\lVert\mathbf{y}\right\rVert_{2}^{2}}\right]$ $\displaystyle\approx\frac{1}{D}+\frac{D+2\sqrt{2\ln{D}}}{D^{\frac{3}{2}}}=O \left(\frac{1}{D}\right)$ $$

Therefore, under an orthogonal transformation, the $\ell_{\infty}$–norm to $\ell_{2}$–norm ratio can be reduced significantly.
It is trivial to show that $\text{Kurt}[y_{j}]=3$ and we see many of our experiments
which use OrthoAdam and softmax-1 exhibit this behaviour (see plots in Section E.2).

$$ $\boxed{\begin{aligned} \mathbf{x}=\alpha\mathbf{e}_{i}+\mathbf{z},\quad\mathbb {E}\left[\frac{\left\lVert\mathbf{x}\right\rVert_{\infty}^{2}}{\left\lVert \mathbf{x}\right\rVert_{2}^{2}}\right]\approx 1\quad&\rightarrow\quad\mathbf{y }=\mathbf{Q}\mathbf{x},\quad\mathbb{E}\left[\frac{\left\lVert\mathbf{y}\right \rVert_{\infty}^{2}}{\left\lVert\mathbf{y}\right\rVert_{2}^{2}}\right]\approx \frac{1}{D}\\ \text{Kurt}\left[x_{j}\right]=D-4+\frac{12}{D}-\frac{6}{D^{2}}=O(D)\quad& \rightarrow\quad\text{Kurt}\left[y_{j}\right]=3\end{aligned}}$ $$

The exact form of $\mathbf{Q}$ can be computed numerically or constructed using appropriately normalised Hadamard matrices (Sylvester, ).

## Appendix E Layer Progression of First Token Attention Dominance,
Kurtosis, ℓ ∞ subscript ℓ \ell_{\infty} roman_ℓ start_POSTSUBSCRIPT ∞ end_POSTSUBSCRIPT -Norm to ℓ 2 subscript ℓ 2 \ell_{2} roman_ℓ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT -Norm Ratio and Maximum Absolute Activation

For brevity,
we give metrics for the first token attention dominance,
hidden state kurtosis and absolute maximum activation *averaged over all layers*
in Table 2 which gives the results of the main experiments in our work.

However, the layer-wise progression of these metrics is also of interest and
can provide insights into the behaviour of the model.
Additionally, we provide the same metrics for popular pretrained GPT2 and Llama models
to show the similarity to our models *trained without softmax-1 and OrthoAdam*.

Finally, to establish a relationship between activation kurtosis and the $\ell_{\infty}$-norm to $\ell_{2}$-norm ratio,
we calculate the Pearson’s correlation coefficients between per-layer kurtosis and per-layer $\ell_{\infty}$-norm to $\ell_{2}$-norm ratio
for all models in our main experimental results from Table 2.

All metrics are computed on the same validation set of the C4 dataset (Raffel et al., )
as in the main paper (Section 5).

### E.1 First Token Attention Dominance

We begin by examining the progression of first token attention dominance across layers.
We calculate the percentage of (head, query) pairs where the query token attends most to the first (key) token.
Given different models have a different number of layers,
we normalise the layer index to the range $[0,1]$ for each model.

We find a general trend across our trained models which use the canonical softmax function
where the first token attention dominance begins low in the initial layers where models do
initial processing of all input tokens.
The dominance rises to a peak in the middle layers where heads specialise to specific sub-tasks
and so the first token is attended to as a default “no-op” (Bondarenko et al., ; Clark et al., ).
Finally, the dominance decreases in the final layers where the model “detokenises” the features
back into token space.

#### E.1.1 GPT2-60M

Figure: Figure 4: Layer-wise progression of first token attention dominance for GPT2-60M. The x-axis is normalised to the range $[0,1]$. S1/OA denote models trained with softmax-1 and/or OrthoAdam.
Refer to caption: x8.png

#### E.1.2 GPT2-130M

Figure: Figure 5: Layer-wise progression of first token attention dominance for GPT2-130M. The x-axis is normalised to the range $[0,1]$. S1/OA denote models trained with softmax-1 and/or OrthoAdam.
Refer to caption: x9.png

#### E.1.3 GPT2-350M and GPT2-1.4B

Figure: Figure 6: Layer-wise progression of first token attention dominance for GPT2-350M and GPT2-1.4B. The x-axis is normalised to the range $[0,1]$. S1/OA denote models trained with softmax-1 and/or OrthoAdam.
Refer to caption: x10.png

#### E.1.4 Llama-130M

Figure: Figure 7: Layer-wise progression of first token attention dominance for Llama-130M. The x-axis is normalised to the range $[0,1]$. S1/OA denote models trained with softmax-1 and/or OrthoAdam.
Refer to caption: x11.png

#### E.1.5 Popular Pretrained Models—GPT2 and Llama

Figure: Figure 8: Layer-wise progression of first token attention dominance for popular pretrained GPT2 and Llama models. The x-axis is normalised to the range $[0,1]$.
Refer to caption: x12.png

### E.2 Activation Kurtosis

Next, we examine the progression of activation kurtosis across layers.
As observable in Table 2, the kurtosis of the first hidden state is significantly higher
than the other hidden states and so we plot the kurtosis of the first hidden state only for brevity.

We observe in the plots below that models trained without OrthoAdam exhibit a general trend of increasing kurtosis
as the hidden states progress through the layers. Demonstrating that multiple layers of the model
contribute to the emergence of large activation values.
Models trained with OrthoAdam *but not softmax-1* exhibit a similar trend, but with lower kurtosis values initially.
Finally, models trained with both OrthoAdam and softmax-1 exhibit a consistent small kurtosis across layers—around
the value of 3 which is the kurtosis of a Gaussian distribution.
Interestingly, GPT2-60M and GPT2-130M show small rises in the final layers—the cause of this is left for future work.

We find that some models show a reduction in kurtosis in the final layers, we again
attribute this to the “detokenisation” of the features back into token space.

#### E.2.1 GPT2-60M

Figure: Figure 9: Layer-wise progression of activation kurtosis of the first token position for GPT2-60M. The x-axis is normalised to the range $[0,1]$. S1/OA denote models trained with softmax-1 and/or OrthoAdam.
Refer to caption: x13.png

#### E.2.2 GPT2-130M

Figure: Figure 10: Layer-wise progression of activation kurtosis of the first token position for GPT2-130M. The x-axis is normalised to the range $[0,1]$. S1/OA denote models trained with softmax-1 and/or OrthoAdam.
Refer to caption: x14.png

#### E.2.3 GPT2-350M and GPT2-1.4B

Figure: Figure 11: Layer-wise progression of activation kurtosis of the first token position for GPT2-350M and GPT2-1.4B. The x-axis is normalised to the range $[0,1]$. S1/OA denote models trained with softmax-1 and/or OrthoAdam.
Refer to caption: x15.png

#### E.2.4 Llama-130M

Figure: Figure 12: Layer-wise progression of activation kurtosis of the first token position for Llama-130M. The x-axis is normalised to the range $[0,1]$. S1/OA denote models trained with softmax-1 and/or OrthoAdam.
Refer to caption: x16.png

#### E.2.5 Popular Pretrained Models—GPT2 and Llama

Figure: Figure 13: Layer-wise progression of activation kurtosis of the first token position for popular pretrained GPT2 and Llama models. The x-axis is normalised to the range $[0,1]$.
Refer to caption: x17.png

### E.3 ℓ ∞ subscript ℓ \ell_{\infty} roman_ℓ start_POSTSUBSCRIPT ∞ end_POSTSUBSCRIPT -Norm to ℓ 2 subscript ℓ 2 \ell_{2} roman_ℓ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT -Norm Ratio

The plots below show the progression of the $\ell_{\infty}$-norm to $\ell_{2}$-norm ratio across layers.
We observe that models trained without OrthoAdam exhibit a general trend of increasing ratio
as the hidden states progress through the layers. Once again as this ratio is maximal in the first hidden state,
we plot the ratio of the first hidden state only for brevity (as done for kurtosis).

The trends are similar to the kurtosis plots and so the same commentary applies.

#### E.3.1 GPT2-60M

Figure: Figure 14: Layer-wise progression of the $\ell_{\infty}$-norm to $\ell_{2}$-norm ratio in the hidden states of the first token position for GPT2-60M. The x-axis is normalised to the range $[0,1]$. S1/OA denote models trained with softmax-1 and/or OrthoAdam.
Refer to caption: x18.png

#### E.3.2 GPT2-130M

Figure: Figure 15: Layer-wise progression of the $\ell_{\infty}$-norm to $\ell_{2}$-norm ratio in the hidden states of the first token position for GPT2-130M. The x-axis is normalised to the range $[0,1]$. S1/OA denote models trained with softmax-1 and/or OrthoAdam.
Refer to caption: x19.png

#### E.3.3 GPT2-350M and GPT2-1.4B

Figure: Figure 16: Layer-wise progression of the $\ell_{\infty}$-norm to $\ell_{2}$-norm ratio in the hidden states of the first token position for GPT2-350M and GPT2-1.4B. The x-axis is normalised to the range $[0,1]$. S1/OA denote models trained with softmax-1 and/or OrthoAdam.
Refer to caption: x20.png

#### E.3.4 Llama-130M

Figure: Figure 17: Layer-wise progression of the $\ell_{\infty}$-norm to $\ell_{2}$-norm ratio in the hidden states of the first token position for Llama-130M. The x-axis is normalised to the range $[0,1]$. S1/OA denote models trained with softmax-1 and/or OrthoAdam.
Refer to caption: x21.png

#### E.3.5 Popular Pretrained Models—GPT2 and Llama

Figure: Figure 18: Layer-wise progression of the $\ell_{\infty}$-norm to $\ell_{2}$-norm ratio in the hidden states of the first token position for popular pretrained GPT2 and Llama models. The x-axis is normalised to the range $[0,1]$.
Refer to caption: x22.png

To further clarify that in Transformer models the $\ell_{\infty}$-norm to $\ell_{2}$-norm ratio is a proxy for activation kurtosis,
we calculate the Pearson’s correlation coefficients between the two metrics for all models in our main experimental results
from Table 2 and public GPT2 and Llama models.
The results are shown in Table 6.
We find a strong positive correlation between the two metrics across all models which reinforces our intuition
that using orthogonal matrices to transform the gradients in the optimiser is an effective way to mitigate the emergence of large activation values,
as an orthogonal transformation can reduce the $\ell_{\infty}$-norm of a vector substantially for a given $\ell_{2}$-norm.

Table: Table 6: Correlation of the kurtosis and norm-ratio of the hidden states of our trained models and popular pretrained models.

### E.4 Maximum Absolute Activation

Finally, we examine the progression of the maximum absolute activation across layers.

#### E.4.1 GPT2-60M

Figure: Figure 19: Layer-wise progression of the maximum absolute activation in the hidden states of the first token position for GPT2-60M. The x-axis is normalised to the range $[0,1]$. S1/OA denote models trained with softmax-1 and/or OrthoAdam.
Refer to caption: x23.png

#### E.4.2 GPT2-130M

Figure: Figure 20: Layer-wise progression of the maximum absolute activation in the hidden states of the first token position for GPT2-130M. The x-axis is normalised to the range $[0,1]$. S1/OA denote models trained with softmax-1 and/or OrthoAdam.
Refer to caption: x24.png

#### E.4.3 GPT2-350M and GPT2-1.4B

Figure: Figure 21: Layer-wise progression of the maximum absolute activation in the hidden states of the first token position for GPT2-350M and GPT2-1.4B. The x-axis is normalised to the range $[0,1]$. S1/OA denote models trained with softmax-1 and/or OrthoAdam.
Refer to caption: x25.png

#### E.4.4 Llama-130M

Figure: Figure 22: Layer-wise progression of the maximum absolute activation in the hidden states of the first token position for Llama-130M. The x-axis is normalised to the range $[0,1]$. S1/OA denote models trained with softmax-1 and/or OrthoAdam.
Refer to caption: x26.png

#### E.4.5 Popular Pretrained Models—GPT2 and Llama

Figure: Figure 23: Layer-wise progression of the maximum absolute activation in the hidden states of the first token position for popular pretrained GPT2 and Llama models. The x-axis is normalised to the range $[0,1]$.
Refer to caption: x27.png

## Appendix F Hidden States of Pretrained Models

In this section, we present the progression of hidden states of popular pretrained models.
This shows how models establish outlier activations and how they persist in the same feature dimensions
across layers. For each model we show the *absolute* activation values in the features containing the
largest activations.
We show the mean across layers, the first layer, $\frac{1}{4}$ and $\frac{3}{4}$ of the layers.

Figure: Figure 24: Example hidden state plots for a GPT2-Small model.
Refer to caption: x28.png

Figure: Figure 25: Example hidden state plots for a GPT2-Medium model.
Refer to caption: x29.png

Figure: Figure 26: Example hidden state plots for a GPT2-Large model.
Refer to caption: x30.png

Figure: Figure 27: Example hidden state plots for a GPT2-XL model.
Refer to caption: x31.png

Figure: Figure 28: Example hidden state plots for a Llama2-7B model.
Refer to caption: x32.png

Figure: Figure 29: Example hidden state plots for a Llama3.1-8B model.
Refer to caption: x33.png

Figure: Figure 30: Example hidden state plots for a DeepSeekv2-Lite model.
Refer to caption: x34.png

## Appendix G Attention Maps of Pretrained Models

In this section, we present the attention maps of popular pretrained models.
This shows how models establish attention patterns and how they persist after initial layers.
This shows that generally after the first or second layer, first token attention dominance is highly
established and persists across layers.

We show the mean across layers, the first layer, $\frac{1}{4}$ and $\frac{3}{4}$ of the
layers—averaging over all heads in each case.

Figure: Figure 31: Example attention maps for a GPT2-Small model.
Refer to caption: x35.png

Figure: Figure 32: Example attention maps for a GPT2-Medium model.
Refer to caption: x36.png

Figure: Figure 33: Example attention maps for a GPT2-Large model.
Refer to caption: x37.png

Figure: Figure 34: Example attention maps for a GPT2-XL model.
Refer to caption: x38.png

Figure: Figure 35: Example attention maps for a Llama2-7B model.
Refer to caption: x39.png

Figure: Figure 36: Example attention maps for a Llama3.1-8B model.
Refer to caption: x40.png

Figure: Figure 37: Example attention maps for a DeepSeekv2-Lite model.
Refer to caption: x41.png

## Appendix H Training Curves

To demonstrate that our proposed methods,
*i.e.* replacing the canonical softmax function with *softmax-1*
and using our proposed optimiser, *OrthoAdam*, do not negatively impact
the training of large language models, we provide the training curves for
our models here.
One can observe that the training curves for models using either or both of our proposed changes
are stable and converge to a similar loss value as the baseline models.

### H.1 GPT2-60M

Figure: Figure 38: Training curves for GPT2-60M models with different optimisers and softmax functions. The models using *OrthoAdam* and *softmax-1* are stable and converge to a similar loss value as the baseline models. S1/OA denotes the model using softmax-1 and/or OrthoAdam.
Refer to caption: x42.png

### H.2 GPT2-130M

Figure: Figure 39: Training curves for GPT2-130M models with different optimisers and softmax functions. The models using *OrthoAdam* and *softmax-1* are stable and converge to a similar loss value as the baseline models. S1/OA denotes the model using softmax-1 and/or OrthoAdam.
Refer to caption: x43.png

### H.3 GPT2-350M and GPT2-1.4B

Figure: Figure 40: Training curves for GPT2-350M and GPT2-1.4B models with different optimisers and softmax functions. The models using *OrthoAdam* and *softmax-1* are stable and converge to a similar loss value as the baseline models. S1/OA denotes the model using softmax-1 and/or OrthoAdam.
Refer to caption: x44.png

### H.4 Llama-130M

Figure: Figure 41: Training curves for Llama-130M models with different optimisers and softmax functions. The models using *OrthoAdam* and *softmax-1* are stable and converge to a similar loss value as the baseline models. S1/OA denotes the model using softmax-1 and/or OrthoAdam.
Refer to caption: x45.png

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 5
presentation: 3
questions: 1. Why do the authors train the LLaMA2-130M model? It is strange to me.

2. Can the authors explain the how the case would change where the model is used to generate following some instruction. Will the LM attend strongly to the first generated token?


---
After rebuttal
Thanks for the authors' detailed responses. My concerns are below.

1. My first concern is the **originality of the softmax-1**. However, I failed to find the original paper when I first read the paper until I see the comments.

2. The second concern is the **contribution of the paper**. In addition, the first token attention issue was also discussed in [1]. I am afraid the story of the paper on 1 and 2 is overstated in the submission. That is why I give a lower contribution score.

3. The authors state "large language models" in their submission title, which is confusing. While the authors provide the LLaMA-8B results in the rebuttal, only a 8B LLM can not represent the entire group of LLMs. The situation can be very different when the model size comes to 70B or even 130B. A more suitable statement for the current submission would be **"autoregressive language models"**.

Therefore, I decide to keep my score.

[1] Efficient Streaming Language Models with Attention Sinks
soundness: 2
strengths: 1. The paper conducts a series of experiments to verify the phenomena, which are convincing.

2. The proposed new softmax and optimizer are simple and effective.

3. The experiments under the quantization scenario show the usefulness of the study, which potentially enlightens future research.
summary: This paper studies two phenomena in transformer-based models. The first is the strong dominance of the first token in the attention maps, and the second is the presence of outlier activations in the hidden states. To address these two issues, the paper proposes a new softmax philosophy and a new optimizer. The experiments show the advantages of the proposed methods under the quantization scenario.
weaknesses: 1. The paper states that the study is on popular large language models (LLMs). However, the experiments in the paper exclude large language models, considering that the largest-sized one in the experiments is GPT2-1.4B, not reaching the bar of a regular sized LLM (>7B). It makes the results of the paper less convincing.

2. I am not convinced that the highlighted two issues in the paper are critical for recent LMs. From the experiments in the paper, one can only see the promising usage under 4-bit/8-bit quant; in other scenarios, still not clear.

3. The paper reports PPL and other metrics to access the LM performances. However, it is not clear whether the resultant models perform well on downstream tasks and instruction following, which are important aspects to evaluate an LM/LLM. Therefore, it is not positive that the study will greatly contribute to the community.

4. Keep up with the last point, the paper does not consider (or explain) whether the study still holds under the instruction-following scenario. This situation is different from doing language modeling; or it is still the same, since the first generated token is not the start of the input sentence.

5. Minorly, it is also not mentioned in the paper the relationship of the two phenomena. It seems that they belong to two independent research aspects.

---

confidence: 4
contribution: 4
initial_rating: 5
presentation: 3
questions: - The idea of using an orthogonal basis $Q$ in Adam is very elegant, especially in the context of quantization. Nevertheless, from a modeling/theoretical viewpoint, it would be interesting to check whether the models are still using outlier "dimensions" in this orthogonal basis, that is if the projection of their activations in such an orthogonal basis still have outlier coefficients. Did you conduct such experiments? In other words, did you verify that the models are not still relying on outlier "dimensions" in a different basis than the canonical one?
- It is common to decrease the learning rates for larger models. Why did you choose a single learning rate for the different model sizes? 

**Typos / Remarks**

L419 : this sentence (*Note that only linear layers...*) seems a bit unformal.

L459: *but still remain high* -> *which remains high* ?
soundness: 2
strengths: This paper successfully addresses two issues that have been identified by extensive literature in the language modeling field when training Transformers-based models. It provides relevant methods to mitigate these issues and show that such methods can help with performance 
 (slightly) and significantly eases the quantization process.
The paper is well-written and pedagogical. The proposed solutions are elegant and simple to implement.
Overall, this paper paves the way for a better understanding of observed self-attention maps and for more expressive inductive biases for language models. It also provides immediate and substantial benefits for the field of quantization.
summary: This work studies two unexpected phenomena that occur in Transformers-based language models: the dominance of first tokens in the self-attention maps, and the presence of outlier dimensions throughout the hidden representations of these models. The authors propose a different mitigation technique for each phenomenon, and successfully mitigate both first-token dominance and outlier dimensions. They finally show that combining both mitigation techniques yields language models that can be quantized easily, with minimal performance loss.

The first mitigation technique relies on a softmax-1 activation function, that adds a bias to the denominator of the softmax function to allow for null rows in the attention map. The authors conduct experiments that show that this technique solves the first-token dominance problem but not outlier dimensions. This second problem is mitigated by OrthoAdam, a slight modification to the Adam optimizer that compute gradients in a given orthonormal basis to avoid specific canonical dimensions being used for massive coefficients.
weaknesses: The main weakness of the paper is its failure to support its main theoretical claim. From the abstract on, the authors argue that they identify "Adam as the primary contributor to the large outlier activations", but they fail to properly discuss the theoretical background of this claim. They support this claim with the following arguments, which all have flaws:
- *the optimiser tracks moments in the same basis as the model parameters* : this reason does not explain why the optimiser is pushing for these outlier activations in gradients and their moments, but only why these outlier dimensions are passed on to the model's weights;
- *Adam and RMSProp have high[er] kurtosis [than SGD]* : this indeed shows that Adam and RMSProp are particularly accelerating the emergence of outlier activations, but it should be noted that they are also accelerating convergence. Hence, this could simply imply that Adam and RMSProp are efficiently facilitating the emergence of outlier activations that could still **be caused by another element of the system** that would rely on outlier activations to increase performance. In other words, the "outlier activation" state could be a favorable state for the model in terms of performance because of another component, and this state could be reached more quickly/easily using Adam and RMSProp.

It could be argued that the fact that OrthoAdam mitigates the outlier dimension problem proves that Adam is the culprit for such a phenomenon. Nevertheless, OrthoAdam is almost explicitly designed to avoid outlier dimensions *in the canonical basis*, as discussed in section 4.2, and could be seen as a variation of Adam that enforces the absence of outlier dimensions. Hence, this just shows that it is possible to train a model under this constraint without substantial performance decrease, but it does not show that Adam is the key **causal** element in the emergence of outlier dimensions.

Another limitation of the paper lies in some parameters and models that were used to validate the approaches:
- The models were trained using a sequence length of 256, which questions the scalability of softmax-1 to longer sequences, as sequence length may be a crucial factor for this activation function;
- All resulting models seem to have a noticeably high perplexity on the validation set. As a comparison, the 70M Pythia model reaches approximately a 3.2 loss (=24 perplexity) at half-training (~150B tokens), and the 410M version reaches 2.2 for cross-entropy (=9 perplexity)(https://wandb.ai/eleutherai/pythia-extra-seeds/reports/Some-loss-curves-for-smaller-Pythia-models--Vmlldzo2NTkxNDIw). This is significantly smaller than what is described in the paper for equivalent models, which raises questions about implementation, hyperparameter choices and evaluation.

Finally, the authors do not include a discussion on training dynamics, showing how the models converge during training, and they do not provide an analysis of OrthoAdam in terms of memory requirements and training latency, which may question the scalability and technical relevance of the method. My intuition is that the cost should be tolerable, but I think this should be mentioned in the paper.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - As the authors note, Llama2 remains usable after quantization even in the vanilla version. Do the authors have some intuition regarding this?

- I had a hard time understanding the discussion in #222 around causal masking. What does it mean to relax causal masking? train with an MLM loss? and why would the fifth token dominate in this case?
soundness: 3
strengths: - Interesting and important research questions
- A well executed study. In particular, several hypotheses are considered for the phenomena in question, and the authors are able to pin-down on what seems to be their root cause 
- Very interesting findings
- Practical results that allow simpler quantization
summary: This paper studies two interesting phenomena in transformer-based LLMs. First, the large amount of attention the first token receives. Second, the emergence of outlier neurons in the transformer layer activation. They attribute the first phenomenon to the softmax function, and the second to the Adam optimizer. They propose variants of these two methods (softmax-1 and OrthoAdam), which resolve these two issues. They show that using these methods allows models to quantize far better than without them.
weaknesses: - softmax-1 has been introduced before in the streamingLLM paper (https://arxiv.org/abs/2309.17453, referred to as $Softmax_1$ or Zero-sink in that work). The authors in that work showed that "_while the zero sink alleviates the attention sink problem to some extent, the model still relies on other initial tokens as attention sinks_". This work seems to argue differently. It seems important to pin down the differences between both setups that lead to the different behaviors.

In addition, I found some of the major claims to require further discussion.

- From table 4, it seems RMSNorm-s is also important for mitigating the outlier effect. This is currently not really 
discussed, and the authors only mention softamx-1 and OrthoAdam as the important parts.

- The softmax-1 method requires some discussion and intuition. Why is the specific method appropriate here? Specifically, the authors say (#208) that it allows "having low attention scores everywhere." But standard softmax also allows that, e.g., by assigning uniform attention scores to all tokens. Is it about a better inductive bias? It would be helpful to see a more detailed comparison between the two methods. E.g., an explanation of why softmax-1 is more effective than standard softmax for this task, and/or a discussion of any potential trade-offs or downsides to using softmax-1.

- If I understand the discussion around OrthoAdam correctly, the authors say the reason for the outliers is (a) the appearance of high values of features, and (b) that optimizers like Adam lead to such high values. Is my understanding correct? If so, this should be stated more explicitly.



Minor:

- Using *x*s to contrast with *v*s in table 2 (as used in tables 3 and 4) would make it clearer than leaving it blank.

Also, there are several typos and such in the paper:

- #195: "... therefore it *in* receives ..." (drop "in")
- #306: "... under *an* particular ..." (should be "a")
- #313: " the model *during* i.e.  ..." (something's strange here, either drop during or add something after it)
- #420: "Note that only linear layers are *quantised the* embeddings, ..." (Missing period/, while)

Meta-review: The paper offers two main contributions the dominance of first tokens in the self-attention maps, and the presence of outlier dimensions throughout the hidden representations of these models.

Strengths:
Addresses known issues in the literature, offers practical techniques
Clearly written

Weaknesses:  
The main issue with the paper was giving too little credit to previous works which was revised and should be easy to fix in the camera ready.
Limited experiments scope, that if addressed could make the results more significant or convincing (e.g., a focus on perplexity rather than downstream task).

Decision: Accept (Poster)