Differential Transformer

## Abstract

Abstract Transformer tends to overallocate attention to irrelevant context. In this work, we introduce Diff Transformer, which amplifies attention to the relevant context while canceling noise. Specifically, the differential attention mechanism calculates attention scores as the difference between two separate softmax \mathrm{softmax} attention maps. The subtraction cancels noise, promoting the emergence of sparse attention patterns. Experimental results on language modeling show that Diff Transformer outperforms Transformer in various settings of scaling up model size and training tokens. More intriguingly, it offers notable advantages in practical applications, such as long-context modeling, key information retrieval, hallucination mitigation, in-context learning, and reduction of activation outliers. By being less distracted by irrelevant context, Diff Transformer can mitigate hallucination in question answering and text summarization. For in-context learning, Diff Transformer not only enhances accuracy but is also more robust to order permutation, which was considered as a chronic robustness issue. The results position Diff Transformer as a highly effective and promising architecture to advance large language models.

## 1 Introduction

Transformer [] has garnered significant research interest in recent years, with the decoder-only Transformer emerging as the de facto standard for large language models (LLMs). At the heart of Transformer is the attention mechanism, which employs the $\mathrm{softmax}$ function to weigh the importance of various tokens in a sequence.
However, recent studies [, ] show that LLMs face challenges in accurately retrieving key information from context.

As illustrated on the left side of Figure˜1, we visualize the normalized attention scores assigned to different parts of the context by a Transformer.
The task is to retrieve an answer embedded in the middle of a pile of documents.
The visualization reveals that Transformer tends to allocate only a small proportion of attention scores to the correct answer, while disproportionately focusing on irrelevant context.
The experiments in Section˜3 further substantiate that Transformers struggle with such capabilities.
The issue arises from non-negligible attention scores assigned to irrelevant context, which ultimately drowns out the correct answer.
We term these extraneous scores as attention noise.

Figure: Figure 1: Transformer often over-attends to irrelevant context (i.e., attention noise). Diff Transformer amplifies attention to answer spans and cancels noise, enhancing the capability of context modeling.
Refer to caption: x1.png

In this paper, we introduce Differential Transformer (a.k.a. Diff Transformer), a foundation architecture for large language models.
The differential attention mechanism is proposed to cancel attention noise with differential denoising.
Specifically, we partition the query and key vectors into two groups and compute two separate $\mathrm{softmax}$ attention maps. Then the result of subtracting these two maps is regarded as attention scores.
The differential attention mechanism eliminates attention noise, encouraging models to focus on critical information.
The approach is analogous to noise-canceling headphones and differential amplifiers [] in electrical engineering, where the difference between two signals cancels out common-mode noise.
In the middle of Figure˜1, we also present the normalized distribution of attention scores for Diff Transformer.
We observe that Diff Transformer assigns significantly higher scores to the correct answer and much lower scores to irrelevant context compared to Transformer.
The right side of Figure˜1 shows that the proposed method achieves notable improvements in retrieval capability.

We conduct extensive experiments on language modeling.
We scale up Diff Transformer in terms of parameter count, training tokens, and context length.
The scaling curves indicate that Diff Transformer requires only about 65% of model size or training tokens needed by Transformer to achieve comparable language modeling performance.
Moreover, Diff Transformer outperforms Transformer in various downstream tasks.
The long-sequence evaluation also shows that Diff Transformer is highly effective in utilizing the increasing context.
In addition, the experimental results demonstrate that Diff Transformer has intriguing advantages for large language models.
For example, the proposed method substantially outperforms Transformer in key information retrieval, hallucination mitigation, and in-context learning.
Diff Transformer also reduces outliers in model activations, which provides new opportunities for quantization.
The findings establish Diff Transformer as an effective and distinctive foundation architecture for large language models.

## 2 Differential Transformer

We propose Differential Transformer (a.k.a. Diff Transformer) as a foundation architecture for sequence modeling, such as large language models (LLMs).
We take a decoder-only model as an example to describe the architecture.
The model is stacked with $L$ Diff Transformer layers.
Given an input sequence $x=x_{1}\cdots x_{N}$, we pack the input embeddings into $X^{0}=[{\bm{x}}_{1},\cdots,{\bm{x}}_{N}]\in\mathbb{R}^{N\times d_{\text{model}}}$, where $d_{\text{model}}$ represents the hidden dimension of the model.
The input is further contextualized to obtain the output $X^{L}$, i.e., $X^{l}=\operatorname{Decoder}(X^{l-1}),\ l\in[1,L]$.
Each layer consists of two modules: a differential attention module followed by a feed-forward network module.
Compared to Transformer [], the main difference is the replacement of conventional $\mathrm{softmax}$ attention with differential attention while the macro layout is kept the same.
We also adopt pre-RMSNorm [] and SwiGLU [, ] as improvements following LLaMA [].

### 2.1 Differential Attention

Figure: Figure 2: Multi-head differential attention. Each head takes the difference between two $\mathrm{softmax}$ attention maps to cancel out attention noise. $\lambda$ is a learnable scalar that is initialized to $\lambda_{\text{init}}$. $\operatorname{GroupNorm}$ applies normalization to each head independently. A fixed multiplier $(1-\lambda_{\text{init}})$ is used after $\operatorname{GroupNorm}$, which aligns the gradient flow with Transformer. The code implementation is available at [https://aka.ms/Diff-Transformer](https://aka.ms/Diff-Transformer).
Refer to caption: x2.png

The differential attention mechanism maps query, key, and value vectors to outputs.
We use query and key vectors to compute attention scores, and then compute a weighted sum of value vectors.
The critical design is that we use a pair of $\mathrm{softmax}$ functions to cancel the noise of attention scores.
Specifically, given input $X\in\mathbb{R}^{N\times d_{\text{model}}}$, we first project them to query, key, and value $Q_{1},Q_{2},K_{1},K_{2}\in\mathbb{R}^{N\times d},\ V\in\mathbb{R}^{N\times 2d}$.
Then the differential attention operator $\operatorname{DiffAttn{}}(\cdot)$ computes outputs via:

$$ $\displaystyle[Q_{1};Q_{2}$ $\displaystyle]=XW^{Q},\quad[K_{1};K_{2}]=XW^{K},\quad V=XW^{V}$ (1) $\displaystyle\operatorname{DiffAttn}$ $\displaystyle(X)=(\mathrm{softmax}(\frac{Q_{1}K^{T}_{1}}{\sqrt{d}})-\lambda~\mathrm{softmax}(\frac{Q_{2}K^{T}_{2}}{\sqrt{d}}))V$ $$

where $W^{Q},W^{K},W^{V}\in\mathbb{R}^{d_{\text{model}}\times 2d}$ are parameters, and $\lambda$ is a learnable scalar.
In order to synchronize the learning dynamics, we re-parameterize the scalar $\lambda$ as:

$$ $\displaystyle\lambda=\exp(\mathbf{\lambda_{q_{1}}}\cdot\mathbf{\lambda_{k_{1}}})-\exp(\mathbf{\lambda_{q_{2}}}\cdot\mathbf{\lambda_{k_{2}}})+\lambda_{\text{init}}$ (2) $$

where $\mathbf{\lambda_{q_{1}}},\mathbf{\lambda_{k_{1}}},\mathbf{\lambda_{q_{2}}},\mathbf{\lambda_{k_{2}}}\in\mathbb{R}^{d}$ are learnable vectors, and $\lambda_{\text{init}}\in(0,1)$ is a constant used for the initialization of $\lambda$.
We empirically find that the setting $\lambda_{\text{init}}=0.8-0.6\times\exp(-0.3\cdot(l-1))$ works well in practice, where $l\in[1,L]$ represents layer index.
It is used as the default strategy in our experiments.
We also explore using the same $\lambda_{\text{init}}$ (e.g., 0.8) for all layers as another initialization strategy. As shown in the ablation studies (Section˜3.8), the performance is relatively robust to different initialization strategies.

Differential attention takes the difference between two $\mathrm{softmax}$ attention functions to eliminate attention noise.
The idea is analogous to differential amplifiers [] proposed in electrical engineering, where the difference between two signals is used as output, so that we can null out the common-mode noise of the input.
In addition, the design of noise-canceling headphones is based on a similar idea.
We can directly reuse FlashAttention [] as described in Appendix˜A, which significantly improves model efficiency.

#### Multi-Head Differential Attention

We also use the multi-head mechanism [] in Differential Transformer.
Let $h$ denote the number of attention heads. We use different projection matrices $W_{i}^{Q},W_{i}^{K},W_{i}^{V},i\in[1,h]$ for the heads.
The scalar $\lambda$ is shared between heads within the same layer.
Then the head outputs are normalized and projected to the final results as follows:

$$ $\displaystyle\mathrm{head}_{i}$ $\displaystyle=\operatorname{DiffAttn}(X;W_{i}^{Q},W_{i}^{K},W_{i}^{V},\lambda)$ (3) $\displaystyle\overline{{\mathrm{head}_{i}}}$ $\displaystyle=(1-\lambda_{\text{init}})\cdot\operatorname{LN}(\mathrm{head}_{i})$ $\displaystyle\operatorname{MultiHead}(X)$ $\displaystyle=\operatorname{Concat}(\overline{{\mathrm{head}_{1}}},\cdots,\overline{{\mathrm{head}_{h}}})W^{O}$ $$

where $\lambda_{\text{init}}$ is the constant scalar in Equation˜2, $W^{O}\in\mathbb{R}^{d_{\text{model}}\times d_{\text{model}}}$ is a learnable projection matrix, $\operatorname{LN}(\cdot)$ uses RMSNorm [] for each head, and $\operatorname{Concat}(\cdot)$ concatenates the heads together along the channel dimension.
We use a fixed multiplier $(1-\lambda_{\text{init}})$ as the scale of $\operatorname{LN}(\cdot)$ to align the gradients with Transformer.
Appendix˜F proves that the overall gradient flow remains similar to that of Transformer. The nice property enables us to directly inherit similar hyperparameters and ensures training stability.
We set the number of heads $h=d_{\text{model}}/2d$, where $d$ is equal to the head dimension of Transformer. So we can align the parameter counts and computational complexity.

Headwise Normalization
Figure˜2 uses $\operatorname{GroupNorm}(\cdot)$ [] to emphasize that $\operatorname{LN}(\cdot)$ is applied to each head independently.
As differential attention tends to have a sparser pattern, statistical information is more diverse between heads.
The $\operatorname{LN}(\cdot)$ operator normalizes each head before concatenation to improve gradient statistics [, ].

### 2.2 Overall Architecture

The overall architecture stacks $L$ layers, where each layer contains a multi-head differential attention module, and a feed-forward network module.
We describe the Differential Transformer layer as:

$$ $\displaystyle Y^{l}$ $\displaystyle=\operatorname{MultiHead}(\operatorname{LN}(X^{l}))+X^{l}$ (4) $\displaystyle X^{l+1}$ $\displaystyle=\operatorname{SwiGLU}(\operatorname{LN}(Y^{l}))+Y^{l}$ (5) $$

where $\operatorname{LN}(\cdot)$ is RMSNorm [], $\operatorname{SwiGLU}(X)=(\operatorname{swish}(XW^{G})\odot XW_{1})W_{2}$, and $W^{G},W_{1}\in\mathbb{R}^{d_{\text{model}}\times\frac{8}{3}d_{\text{model}}},W_{2}\in\mathbb{R}^{\frac{8}{3}d_{\text{model}}\times d_{\text{model}}}$ are learnable matrices.

## 3 Experiments

We evaluate Differential Transformer for large language models from the following perspectives.
First, we compare the proposed architecture with Transformers in various downstream tasks (Section˜3.1) and study the properties of scaling up model size and training tokens (Section˜3.2).
Second, we conduct a length extension to 64K and evaluate the long-sequence modeling capability (Section˜3.3).
Third, we present the results of key information retrieval, contextual hallucination evaluation, and in-context learning (Sections˜3.4, 3.6 and 3.5).
Forth, we show that Differential Transformer can reduce outliers in the model activations compared to Transformer (Section˜3.7).
Fifth, we conduct extensive ablation studies for various design choices (Section˜3.8).

### 3.1 Language Modeling Evaluation

We train 3B-size Diff Transformer language models on 1T tokens and compare with previous well-trained Transformer-based models [, , ] in various downstream tasks.
As described in Appendix˜B, we follow the same setting to train a 3B-size Transformer language model on 350B tokens.
The checkpoints are also used in the following experiments and analysis to ensure fair comparisons.

**Table 1: Eval Harness [] accuracy compared with well-trained Transformer language models [, , ]. We scale the 3B model to 1 trillion training tokens. The 1T results of StableLM-3B-4E1T are taken from its technical report [].**
| Model | ARC-C | ARC-E | BoolQ | HellaSwag | OBQA | PIQA | WinoGrande | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Training with 1T tokens |  |  |  |  |  |  |  |  |
| OpenLLaMA-3B-v2 [] | 33.9 | 67.6 | 65.7 | 70.0 | 26.0 | 76.7 | 62.9 | 57.5 |
| StableLM-base-alpha-3B-v2 [] | 32.4 | 67.3 | 64.6 | 68.6 | 26.4 | 76.0 | 62.1 | 56.8 |
| StableLM-3B-4E1T [] | — | 66.6 | — | — | — | 76.8 | 63.2 | — |
| Diff-3B | 37.8 | 72.9 | 69.0 | 71.4 | 29.0 | 76.8 | 67.1 | 60.6 |

Setup
We follow a similar recipe as StableLM-3B-4E1T [].
We set hidden size to $3072$. The number of layers is $28$.
The head dimension $d$ is $128$. The number of heads is $24$ for Transformer and $12$ for Diff Transformer, to align computation FLOPs and model size.
The total parameter count is about 2.8B.
The training sequence length is 4096.
The batch size is 4M tokens.
We train the models with 1T tokens.
We use AdamW [] optimizer with $\beta=0.9,0.95$. The maximal learning rate is 3.2e-4 with 1000 warmup steps and linearly decays to 1.28e-5.
The training corpus also follows StableLM-3B-4E1T [].
We employ tiktoken-cl100k_base tokenizer.
Detailed hyperparameters are provided in Appendix˜C.

Results
Table˜1 reports the zero-shot results on the LM Eval Harness benchmark [].
We compare Diff Transformer with well-trained Transformer-based language models, including OpenLLaMA-v2-3B [], StableLM-base-alpha-3B-v2 [], and StableLM-3B-4E1T [].
OpenLLaMA-v2-3B and StableLM-base-alpha-3B-v2 are also trained with 1T tokens.
The 1T results of StableLM-3B-4E1T are taken from its technical report [].
Experimental results show that Diff Transformer achieves favorable performance compared to previous well-tuned Transformer language models.
In addition, Appendix˜B shows that Diff Transformer outperforms Transformer across various tasks, where we use the same setting to train the 3B-size language models for fair comparisons.

### 3.2 Scalability Compared with Transformer

Figure: (a) Scaling model size ranging from 830M to 13B.
Refer to caption: x3.png

We compare the scaling properties of Diff Transformer and Transformer on language modeling.
We scale up the model size, and the number of training tokens, respectively.
We follow the augmented Transformer architecture as in LLaMA [] and use the same setting to ensure fair comparison.
Specifically, the “Transformer” models include improvements in RMSNorm [], SwiGLU [, ], and removal of bias.

Scaling Model Size
As shown in Figure˜3(a), we train language models with 830M, 1.4B, 2.8B, 6.8B, and 13.1B parameters.
The models are trained with a sequence length of 2048, and a batch size of 0.25M tokens.
We train models for 40K steps.
Detailed hyperparameters are described in Appendix˜D.
The scaling law [] empirically fits well in this configuration.
Figure˜3(a) shows that Diff Transformer outperforms Transformer in various model sizes.
The results indicate that Diff Transformer is scalable in terms of parameter count.
According to the fitted curves, 6.8B-size Diff Transformer achieves a validation loss comparable to 11B-size Transformer, requiring only 62.2% of parameters.
Similarly, 7.8B-size Diff Transformer matches the performance of 13.1B-size Transformer, requiring only 59.5% of parameters.

Scaling Training Tokens
As shown in Figure˜3(b), we evaluate the 3B language models (as presented in Appendix˜B) every 40B tokens (i.e., 10K steps) up to a total of 360B tokens (i.e., 90K steps).
The fitted curves indicate that Diff Transformer trained with 160B tokens achieves comparable performance as Transformer trained with 251B tokens, consuming only 63.7% of the training tokens.

### 3.3 Long-Context Evaluation

Figure: Figure 4: Cumulative average negative log-likelihood (lower is better) on book data. Diff Transformer leverages long context more effectively.
Refer to caption: x5.png

We extend the 3B-size language models (described in Appendix˜B) to 64K context length.
We continue training the 3B checkpoints for additional 1.5B tokens. Most hyperparameters are kept the same as in Section˜3.1.
The learning rate is 8e-5. The RoPE [] $\theta$ is increased to 640,000.
The training corpus is up-sampled according to sequence length [].

Results
Figure˜4 presents cumulative average negative log-likelihood (NLL) of the tokens at varying positions [], where lower NLL indicates better performance.
The evaluation is conducted on book data within 64K length.
We observe a consistent decrease in NLL as the context length increases.
Diff Transformer achieves lower NLL values than Transformer.
The results demonstrate that Diff Transformer can effectively leverage the increasing context.

### 3.4 Key Information Retrieval

The Needle-In-A-Haystack [] test is widely used to evaluate the ability to extract critical information embedded in a large context.
We follow the multi-needle evaluation protocol of LWM [] and Gemini 1.5 [].
The needles are inserted into varying depths within contexts of different lengths.
Each needle consists of a concise sentence that assigns a unique magic number to a specific city.
The goal is to retrieve the magic numbers corresponding to the query cities.
We position the answer needle at five different depths within the context: 0%, 25%, 50%, 75%, and 100%, while placing other distracting needles randomly.
Each combination of depth and length is evaluated using $50$ samples.
The average accuracy is reported.
Let $N$ denote the total number of number-city pairs and $R$ the number of query cities.

**Table 2: Multi-needle retrieval accuracy in 4K length, averaged over the answer needle positions. $N$ represents the number of needles, and $R$ denotes the number of query cities.**
| Model | $N=1$ | $N=2$ | $N=4$ | $N=6$ |
| --- | --- | --- | --- | --- |
| $R=1$ | $R=2$ | $R=2$ | $R=2$ |  |
| Transformer | 1.00 | 0.85 | 0.62 | 0.55 |
| Diff | 1.00 | 0.92 | 0.84 | 0.85 |

Retrieve from 4K Context Length
As shown in Table˜2, we insert $N=1,2,4,6$ needles into 4K-length contexts and retrieve $R=1,2$ needles.
We evaluate 3B-size models trained with 4K input length (Appendix˜B).
We find that both models obtain good accuracy for $N=1$ and $N=2$.
As $N$ and $R$ increase, Diff Transformer maintains a consistent accuracy, while the performance of Transformer drops significantly.
In particular, at $N=6,R=2$, the accuracy gap between the two models reaches $30\%$.
The results indicate the superior ability of Diff Transformer to retrieve key information in distracting contexts.

Figure: (a) Transformer.
Refer to caption: x6.png

Retrieve from 64K Context Length
As shown in Figure˜5, the evaluated context length ranges from 8K to 64K for the $N=8,R=1$ setting.
We evaluate the 3B-size models with length extension (Section˜3.3).
We report the accuracy across varying answer needle depths (y-axis) and context lengths (x-axis).
The bottom row is the average accuracy for all depths.
Diff Transformer maintains stable performance across different context lengths.
In contrast, Transformer’s average accuracy gradually declines as the context length increases up to the maximal length, i.e., 64K.
Besides, Diff Transformer outperforms Transformer particularly when key information is positioned within the first half of the context (i.e., 0%, 25%, and 50% depth).
In particular, when needles are placed at the 25% depth in a 64K context, Diff Transformer achieves $76\%$ accuracy improvement over Transformer.

**Table 3: Attention scores allocated to answer spans and noise context in the key information retrieval task. The target answer is inserted in varying positions (i.e., depth) of context. Diff Transformer allocates more attention scores to useful information and effectively cancels out attention noise.**
| Model | Attention to Answer $\uparrow$ | Attention Noise $\downarrow$ |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $0\%$ | $25\%$ | $50\%$ | $75\%$ | $100\%$ | $0\%$ | $25\%$ | $50\%$ | $75\%$ | $100\%$ |  |
| Transformer | 0.03 | 0.03 | 0.03 | 0.07 | 0.09 | 0.51 | 0.54 | 0.52 | 0.49 | 0.49 |
| Diff | 0.27 | 0.30 | 0.31 | 0.32 | 0.40 | 0.01 | 0.02 | 0.02 | 0.02 | 0.01 |

Attention Score Analysis
Table˜3 presents the attention scores allocated to the answer span and the noise context for the key information retrieval task.
The scores indicate the model’s ability to preserve useful information against attention noise.
We compare the normalized attention scores when key information is inserted at different positions (i.e., depths) within the context.
Compared with Transformer, Diff Transformer allocates higher attention scores to the answer span and has lower attention noise.

### 3.5 In-Context Learning

We evaluate in-context learning from two perspectives, including many-shot classification and robustness of in-context learning.
In-context learning is a fundamental capability of language models, which indicates how well a model can utilize input context.

Many-Shot In-Context Learning
As presented in Figure˜6, we compare the accuracy of many-shot classification between Transformer and our architecture.
We evaluate the 3B-size language models that support 64K input length (Section˜3.3).
We follow the evaluation protocol of [] and use constrained decoding [].
We incrementally increase the number of demonstration samples from 1-shot until the total length reaches 64K length.
Specifically, the TREC [] dataset has 6 classes, TREC-fine [] has 50 classes, Banking-77 [] has 77 classes, and Clinic-150 [] has 150 classes.
The results show that Diff Transformer consistently outperforms Transformer across datasets and varying numbers of demonstration samples.
Moreover, the improvement in average accuracy is substantial, ranging from 5.2% to 21.6%.

Figure: (a) TREC with 6 classes.
Refer to caption: x8.png

Figure: (a) Examples are randomly arranged.
Refer to caption: x12.png

Robustness of In-Context Learning
Figure˜7 compares the robustness of in-context learning between Transformer and Diff Transformer.
Given the same demonstration examples, we analyze the performance variance with order permutations.
Lower variance indicates greater robustness and less risk of catastrophic performance degradation.
The evaluation protocol is the same as above.
Figure˜7 presents the analysis on the TREC dataset.
More results are also provided in Appendix˜E.
We evaluate two prompt formats, i.e., examples are randomly arranged (Figure˜7(a)), and alternately arranged by class (Figure˜7(b)).
In both settings, Diff Transformer has much smaller performance variance compared to Transformer.
The results indicate that our approach is more robust for in-context learning.
In contrast, Transformer tends to be distracted by order permutations [], resulting in a huge margin between the best and worst results.

### 3.6 Contextual Hallucination Evaluation

We evaluate contextual hallucination of the 3B-size language models (described in Appendix˜B) on text summarization and question answering.
Notice that we focus on the cases where the input context contains correct facts, but the model still fails to produce accurate outputs.

We follow the evaluation protocol of [].
We feed the model output along with ground-truth responses to GPT-4o []. Then we ask GPT-4o to make binary judgements on whether the model outputs are accurate and free of hallucinations.
Previous studies [, ] have shown that the above hallucination evaluation protocol has relatively high agreement between GPT-4o judgments and human annotations. The automatic metric is reliable and mirrors the human evaluation.
For each dataset, the accuracy is averaged over 100 samples.

**(a) Accuracy (i.e., free of hallucinations) on text summarization datasets.**
| Model | XSum | CNN/DM | MultiNews |
| --- | --- | --- | --- |
| Transformer | 0.44 | 0.32 | 0.42 |
| Diff | 0.53 | 0.41 | 0.61 |

Summarization
Table˜4(a) presents hallucination evaluation on summarization datasets XSum [], CNN/DM [], and MultiNews [].
The task is to generate summaries for input documents.

Question Answering
As shown in Table˜4(b), we compare the hallucination rate of Diff Transformer and Transformer on both single- and multi-document question answering.
The Qasper [] dataset is single-document question answering. In contrast, HotpotQA [] and 2WikiMultihopQA [] are multi-document question answering.
The goal is to answer questions about the given context.
All evaluation examples are from LongBench [].

Compared with Transformer, our method mitigates contextual hallucination on summarization and question answering.
The performance improvement possibly stems from Diff Transformer’s better focus on essential information needed for the task, instead of irrelevant context.
This aligns with previous observation [] that one primary reason for contextual hallucination in Transformer is the misallocation of attention scores.

### 3.7 Activation Outliers Analysis

In large language models, a subset of activations manifests with significantly larger values compared to the majority, a phenomenon commonly called activation outliers [, ].
The outliers result in difficulties for model quantization during training and inference.
We demonstrate that Diff Transformer can reduce the magnitude of activation outliers, potentially allowing lower bit-widths for quantization.

**Table 5: Largest activation values in attention logits and hidden states. Top activation values are considered as activation outliers, due to their significantly higher magnitude than the median. Diff Transformer mitigates outliers compared to Transformer.**
| Model | Activation Type | Top-1 | Top-2 | Top-3 | Top-10 | Top-100 | Median |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Transformer | Attention Logits | 318.0 | 308.2 | 304.9 | 284.7 | 251.5 | 5.4 |
| Diff | Attention Logits | 38.8 | 38.8 | 37.3 | 32.0 | 27.4 | 3.3 |
| Transformer | Hidden States | 3608.6 | 3607.4 | 3603.6 | 3552.1 | 2448.2 | 0.6 |
| Diff | Hidden States | 1688.2 | 1672.5 | 1672.1 | 1624.3 | 740.9 | 1.2 |

Statistics of Largest Activation Values
Table˜5 presents the statistics of activation values collected from Transformer and Diff Transformer models trained in Appendix˜B.
We analyze two types of activations, including attention logits (i.e., pre-$\mathrm{softmax}$ activations), and hidden states (i.e., layer outputs).
The statistics are gathered from 0.4M tokens.
As shown in Table˜5, although the median values are of similar magnitude, Diff Transformer exhibits much lower top activation values compared to Transformer.
The results show that our method produces fewer activation outliers.

Figure: Figure 8: Zero-shot accuracy on the HellaSwag [] dataset. We quantize the attention logits from 16 bits (i.e., unquantized) to 8 bits, 6 bits, and 4 bits.
Refer to caption: x14.png

Quantization of Attention Logits
As shown in Figure˜8, we quantize the attention logits to lower bits.
We apply dynamic post-training quantization using absmax quantization [].
The 16-bit configuration represents the original results without quantization.
The models are progressively quantized to 8 bits, 6 bits, and 4 bits.
Figure˜8 reports the zero-shot performance on HellaSwag []. The other datasets follow a similar trend.
Diff Transformer retains high performance even at reduced bit-widths, ranging from 16 bits to 6 bits. In comparison, Transformer’s accuracy significantly drops with 6-bit quantization.
The 4-bit Diff Transformer achieves comparable accuracy as the 6-bit Transformer, and outperforms the 4-bit Transformer by about 25% in accuracy.
The results indicate that Diff Transformer natively mitigates activation outliers in attention scores, providing new opportunities for low-bit FlashAttention [] implementations.

### 3.8 Ablation Studies

**Table 6: Ablation studies of 1.4B-size models. We report language modeling loss on the validation set. We also follow Arora et al. [] to report fine-grained metrics, where “AR-Hit” evaluates $n$-grams previously seen in the context. “#Heads” is number of heads. “$d$” is head dimension. “GN” indicates whether GroupNorm is used.**
| Model | #heads | $d$ | GN | Valid. Set$\downarrow$ | Fine-Grained Slices |  |
| --- | --- | --- | --- | --- | --- | --- |
| AR-Hit$\downarrow$ | Others$\downarrow$ |  |  |  |  |  |
| Transformer | 16 | 128 | ✗ | 3.087 | 0.898 | 3.272 |
| Transformer | 8 | 256 | ✗ | 3.088 | 0.899 | 3.273 |
| $+$ GroupNorm | 8 | 256 | ✓ | 3.086 | 0.899 | 3.271 |
| Diff Transformer | 8 | 128 | ✓ | 3.062 | 0.880 | 3.247 |
| $-$ GroupNorm | 8 | 128 | ✗ | 3.122 | 0.911 | 3.309 |
| with $\lambda_{\text{init}}=0.8$ | 8 | 128 | ✓ | 3.065 | 0.883 | 3.250 |
| with $\lambda_{\text{init}}=0.5$ | 8 | 128 | ✓ | 3.066 | 0.882 | 3.251 |

We conduct ablation studies with 1.4B-size language models.
The training setup is the same as the 1.4B model in Section˜3.2.
The models have $L=24$ layers, $h=16$ heads for Transformer, and $h=8$ heads for Diff Transformer. The head dimension is $d=128$.
Detailed hyperparameters are described in Appendix˜D.

Table˜6 reports fine-grained loss on the validation set.
We follow Zoology [] and divide loss into “Ar-Hit” and “Others”.
Specifically, “Ar-Hit” considers the last token of an $n$-gram previously seen in the context, which evaluates the associative recall capability.
The “Others” slice represents the tokens that cannot be recalled from the context or frequent tokens.

As shown in Table˜6, we ablate various design choices of Diff Transformer and present several Transformer variants.
Notice that all models have comparable size and training FLOPs for fair comparisons.
The first and fourth rows are the default settings for Transformer and Diff Transformer, respectively, which are directly taken from Figure˜3(a).
Our method outperforms Transformer in terms of both overall and fine-grained loss.
As Diff Transformer halves the number of heads to match model size, the second row shows that the configuration change does not have much impact.
We ablate GroupNorm from Diff Transformer, which degrades performance due to training instability. Because multiple heads tend to have different statistics in our method, GroupNorm plays a key role in normalizing them to similar values.
In contrast, comparing the third and first rows, adding GroupNorm to Transformer has negligible effect on performance.
The results indicate that the improvements of our method come from the differential attention mechanism, instead of configurations or normalization modules.
Moreover, we compare different strategies to initialize $\lambda$.
As described in Section˜2.1, the default setting uses exponential initialization, i.e., $\lambda_{\text{init}}=0.8-0.6\times\exp(-0.3\cdot(l-1))$, where $l$ is the layer index.
The last two rows employ constant initialization with $\lambda_{\text{init}}=0.8,0.5$.
The minimal change in the validation loss suggests that the models are robust to the choice of $\lambda$ initialization.

## 4 Conclusion

In this work, we introduce Differential Transformer (a.k.a. Diff Transformer), which amplifies attention to the relevant context while canceling noise.
Experimental results on language modeling show that Diff Transformer outperforms Transformer in terms of scaling properties, long-context modeling, key information retrieval, hallucination mitigation, in-context learning, and reduction of activation outliers.
The results emphasize the importance of reducing attention noise.
Moreover, the differential attention mechanism can be easily implemented with FlashAttention [].
The findings position Diff Transformer as a distinctive and promising foundation architecture for large language models.
In the future, we can develop efficient low-bit attention kernels due to the reduced magnitude of activation outliers.
As the attention pattern becomes much sparser, we would also like to utilize the property to compress key-value caches.

## Acknowledgement

We would like to acknowledge Ben Huntley for maintaining the GPU cluster.
The long-sequence training utilizes CUBE, which is an internal version of [].

## Appendix A Implementation of Differential Attention

We present the pseudocode for $\operatorname{DiffAttn{}(\cdot)}$ and conventional $\mathrm{softmax}$ attention.

Implementation with FlashAttention
Additionally, we provide implementations with FlashAttention [].
We categorize the implementations into two types by whether it supports using different dimensions between $Q,K$ and $V$.
Specifically, let $\operatorname{FlashDiffAttn_1(\cdot)}$ denote the package that supports different dimensions (e.g., xformers(^1^11[https://github.com/facebookresearch/xformers](https://github.com/facebookresearch/xformers))), and $\operatorname{FlashDiffAttn_2(\cdot)}$ the package that does not (e.g., flash-attention(^2^22[https://github.com/Dao-AILab/flash-attention](https://github.com/Dao-AILab/flash-attention))).
We also implement a customized-flash-attention(^3^33[https://aka.ms/flash-diff](https://aka.ms/flash-diff)) package, which is modified based on the official FlashAttention2 [], in order to support different dimensions between $Q,K$ and $V$.

The code implementation is available at [https://aka.ms/Diff-Transformer](https://aka.ms/Diff-Transformer).

Efficiency
Table˜7 compares the throughput between Diff Transformer and Transformer.
For fair comparison, we use the customized-flash-attention implementation mentioned above for both methods. The experiments are conducted with Nvidia H100-80GB GPU cards.

**Table 7: Throughput is measured with number of tokens per second.**
| Model | Model Size | Length | Throughput |  |
| --- | --- | --- | --- | --- |
| Forward + Backward | Forward |  |  |  |
| Transformer | 3B | 2K | 7247 | 51228 |
| Diff | 3B | 2K | 6635 ($-9\%$) | 46811 ($-9\%$) |
| Transformer | 3B | 4K | 7491 | 48762 |
| Diff | 3B | 4K | 6718 ($-12\%$) | 44521 ($-10\%$) |
| Transformer | 13B | 2K | 998 | 14346 |
| Diff | 13B | 2K | 942 ($-6\%$) | 13653 ($-5\%$) |

As shown in Table˜7, we evaluate the settings with different model size (3B, 13B) and context length (2K, 4K).
For 3B models, there are 12 heads for Diff Transformer and 24 heads for Transformer.
For 13B model there are 20 heads for Diff Transformer and 40 heads for Transformer.
All models have the same head dimension $d=128$.
Training efficiency consists of forward and backward.
Prefill efficiency only includes forward.
Table˜7 shows that the throughput results are comparable within an acceptable range.
Notice that the customized-flash-attention implementation is built on FlashAttention2 [].
With the recent release of FlashAttention3 [], the gap of throughput can be further reduced.
More advanced kernel implementation, which is specifically designed for differential attention, can also improve throughput.

## Appendix B Language Modeling Evaluation

Following the same setting as in Section˜3.1, we train 3B-size language models on 350B tokens and compare Diff Transformer with Transformer [] in various downstream tasks.
We use the augmented Transformer architecture as in LLaMA [].
Specifically, the “Transformer” models include improvements in RMSNorm [], SwiGLU [, ], and removal of bias.

Table˜8 reports the zero-shot and 5-shot results on the LM Eval Harness benchmark [].
The results show that Diff Transformer outperforms Transformer across various tasks in both zero-shot and few-shot settings.

**Table 8: Comparison of Diff Transformer with well-trained Transformer language models on LM Eval Harness []. Diff Transformer achieves better accuracy in the zero- and few-shot settings.**
| Model | ARC-C | ARC-E | BoolQ | HellaSwag | OBQA | PIQA | WinoGrande | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Training with 350B tokens (Zero-Shot) |  |  |  |  |  |  |  |  |
| Transformer-3B | 32.2 | 66.8 | 62.9 | 63.4 | 26.2 | 74.5 | 61.6 | 55.4 |
| Diff-3B | 33.0 | 68.3 | 60.1 | 66.2 | 27.6 | 75.5 | 62.7 | 56.2 |
| Training with 350B tokens (5-Shot) |  |  |  |  |  |  |  |  |
| Transformer-3B | 34.0 | 69.5 | 65.3 | 63.4 | 25.0 | 75.2 | 62.6 | 56.4 |
| Diff-3B | 35.0 | 69.5 | 67.2 | 66.9 | 27.6 | 76.1 | 63.8 | 58.0 |

## Appendix C Hyperparameters for Section ˜ 3.1

Table˜9 presents the detailed hyperparameters for the Diff Transformer-3B models in Section˜3.1.
For Transformer-3B, the only difference is that there are 24 heads.
Notice that both Transformer-3B and Diff Transformer-3B have similar FLOPs.

**Table 9: Hyperparamters used for the Diff Transformer-3B model in Section˜3.1.**
| Params | Values |
| --- | --- |
| Layers | 28 |
| Hidden size | 3072 |
| FFN size | 8192 |
| Vocab size | 100,288 |
| Heads | 12 |
| Adam $\beta$ | (0.9, 0.95) |
| LR | $3.2\times 10^{-4}$ |
| Batch size | 4M |
| Warmup steps | 1000 |
| Weight decay | 0.1 |
| Dropout | 0.0 |

## Appendix D Hyperparameters for Section ˜ 3.2

Table˜10 reports the hidden dimension, number of layers, and number of heads of Diff Transformer for different model sizes.
For all model sizes of Transformer, we double the number of heads compared with Diff Transformer to align parameters.
The FFN size is $\frac{8}{3}\times d_{\text{model}}$, where $d_{\text{model}}$ is the hidden dimension.
The training length is set to 2048.
The batch size is set to 0.25M tokens.
We use AdamW [] with $\beta_{1}=0.9,\beta_{2}=0.98$.
The learning rate is $1.5\times 10^{-4}$ for 830M to 2.8B sizes, and $7.5\times 10^{-5}$ for 6.8B to 13.1B sizes.
The warmup step is 375 with linear rate decay.
The weight decay is set to 0.05.
We train the models with 40k steps, i.e., 10B tokens.

**Table 10: Model size and hyperparameters used for Diff Transformer in Section˜3.2.**
| Size | Hidden Dim. | #Layers | #Heads |
| --- | --- | --- | --- |
| 830M | 1536 | 24 | 8 |
| 1.4B | 2048 | 24 | 8 |
| 2.8B | 2560 | 32 | 10 |
| 6.8B | 4096 | 32 | 16 |
| 13.1B | 5120 | 40 | 20 |

## Appendix E Robustness of In-Context Learning

As described in Section˜3.5, we evaluate the robustness of in-context learning of Transformer and Diff Transformer with permutations of the same in-context examples.
We evaluate the 3B-size language models that are extended to 64K length (Section˜3.3).

Figure˜9 provides comparisons on four datasets, with in-context examples randomly arranged.
The evaluation protocol is the same as in Section˜3.5.
The variance in accuracy of Diff Transformer is consistently lower than that of Transformer, indicating greater robustness of Diff Transformer for in-context learning.

Figure: (a) TREC with 6 classes.
Refer to caption: x15.png

## Appendix F Gradient Flow of Diff Transformer

We show that the gradient flow in differential attention is similar to that of conventional $\mathrm{softmax}$ attention. With this property, the same hyperparameters used in Transformer can be applied directly to the corresponding Diff Transformer without concerns about training instability.

For differential attention, we select a single head in the proof and expand Equation˜1 and Equation˜3 as follows.
We have $X\in\mathbb{R}^{N\times d_{\text{model}}}$ as the input, $Q_{1},Q_{2},K_{1},K_{2}\in\mathbb{R}^{N\times d},\ V\in\mathbb{R}^{N\times 2d}$, and $O\in\mathbb{R}^{N\times d_{\text{model}}}$ as the output:

$$ $\begin{gathered}[Q_{1};Q_{2}]=[XW^{Q_{1}};XW^{Q_{2}}],\quad[K_{1};K_{2}]=[XW^{K_{1}};XW^{K_{2}}],\quad V=XW^{V}\\ A_{1}=\mathrm{softmax}(\frac{Q_{1}K^{T}_{1}}{\sqrt{d}}),\quad A2=\mathrm{softmax}(\frac{Q_{2}K^{T}_{2}}{\sqrt{d}})\\ O=\operatorname{GroupNorm}((A_{1}-\lambda~A_{2})V)W^{O}\end{gathered}$ (6) $$

where $W^{Q_{1}},\ W^{Q_{2}},\ W^{K_{1}},\ W^{K_{2}}\in\mathbb{R}^{d_{\text{model}}\times d},\ W^{V}\in\mathbb{R}^{d_{\text{model}}\times 2d},\ W^{O}\in\mathbb{R}^{2d\times d_{\text{model}}}$ are parameters, $\lambda$ is a learnable scalar, and GroupNorm has a fixed multiplier as scale: $\gamma=1-\lambda_{\text{init}}$. For a token $x$ in $(A_{1}-\lambda~A_{2})V$, we have $\cfrac{\partial\text{GN}(x)}{\partial x}=\Theta(\cfrac{\sqrt{2d}\cdot\gamma}{||x||_{2}})=\Theta(1)$ as $\cfrac{||x||_{2}}{\sqrt{2d}}=\Theta(1-\lambda_{\text{init}})$ at the early training stage.
With this formulation and given the gradient of $O$ as $\cfrac{\partial L}{\partial O}$, we formulate gradients of parameters as:

$$ $\displaystyle\cfrac{\partial L}{\partial W^{O}}$ $\displaystyle=\cfrac{\partial L}{\partial O}\cfrac{\partial O}{\partial W^{O}}$ (7) $\displaystyle=((A_{1}-\lambda~A_{2})V)^{\intercal}\cfrac{\partial L}{\partial O}$ $\displaystyle\cfrac{\partial L}{\partial W^{V}}$ $\displaystyle=\cfrac{\partial L}{\partial O}\cfrac{\partial O}{\partial V}\cfrac{\partial V}{\partial W^{V}}$ $\displaystyle=X^{\intercal}(A_{1}-\lambda~A_{2})^{\intercal}\cfrac{\partial L}{\partial O}(W^{O})^{\intercal}$ $\displaystyle\cfrac{\partial L}{\partial W^{Q_{1}}}$ $\displaystyle=\cfrac{\partial L}{\partial O}\cfrac{\partial O}{\partial A_{1}}\cfrac{\partial A_{1}}{\partial Q_{1}}\cfrac{\partial Q_{1}}{\partial W^{Q_{1}}}$ $\displaystyle=\cfrac{1}{\sqrt{d}}~X^{\intercal}[A_{1}\odot(\cfrac{\partial L}{\partial O}(W^{O})^{\intercal}V^{\intercal}-(A_{1}\odot(\cfrac{\partial L}{\partial O}(W^{O})^{\intercal}V^{\intercal}))J)]K_{1}$ $\displaystyle\cfrac{\partial L}{\partial W^{Q_{2}}}$ $\displaystyle=\cfrac{\partial L}{\partial O}\cfrac{\partial O}{\partial A_{2}}\cfrac{\partial A_{2}}{\partial Q_{2}}\cfrac{\partial Q_{2}}{\partial W^{Q_{2}}}$ $\displaystyle=\cfrac{-\lambda}{\sqrt{d}}~X^{\intercal}[A_{2}\odot(\cfrac{\partial L}{\partial O}(W^{O})^{\intercal}V^{\intercal}-(A_{2}\odot(\cfrac{\partial L}{\partial O}(W^{O})^{\intercal}V^{\intercal}))J)]K_{2}$ $\displaystyle\cfrac{\partial L}{\partial W^{K_{1}}}$ $\displaystyle=\cfrac{\partial L}{\partial O}\cfrac{\partial O}{\partial A_{1}}\cfrac{\partial A_{1}}{\partial K_{1}}\cfrac{\partial K_{1}}{\partial W^{K_{1}}}$ $\displaystyle=\cfrac{1}{\sqrt{d}}~X^{\intercal}[A_{1}\odot(\cfrac{\partial L}{\partial O}(W^{O})^{\intercal}V^{\intercal}-(A_{1}\odot(\cfrac{\partial L}{\partial O}(W^{O})^{\intercal}V^{\intercal}))J)]^{\intercal}Q_{1}$ $\displaystyle\cfrac{\partial L}{\partial W^{K_{2}}}$ $\displaystyle=\cfrac{\partial L}{\partial O}\cfrac{\partial O}{\partial A_{2}}\cfrac{\partial A_{2}}{\partial K_{2}}\cfrac{\partial K_{2}}{\partial W^{K_{2}}}$ $\displaystyle=\cfrac{-\lambda}{\sqrt{d}}~X^{\intercal}[A_{2}\odot(\cfrac{\partial L}{\partial O}(W^{O})^{\intercal}V^{\intercal}-(A_{2}\odot(\cfrac{\partial L}{\partial O}(W^{O})^{\intercal}V^{\intercal}))J)]^{\intercal}Q_{2}$ $$

where $J\in\mathbb{R}^{N\times N}$ is a all-one matrix.

As a comparison, we reformulate conventional $\mathrm{softmax}$ attention. For attention with $2d$ dimension, we have $X\in\mathbb{R}^{N\times d_{\text{model}}}$ as the input, $Q_{1},Q_{2},K_{1},K_{2}\in\mathbb{R}^{N\times d},\ V\in\mathbb{R}^{N\times 2d}$, and $O\in\mathbb{R}^{N\times d_{\text{model}}}$ as the output:

$$ $\begin{gathered}[Q_{1};Q_{2}]=[XW^{Q_{1}};XW^{Q_{2}}],\quad[K_{1};K_{2}]=[XW^{K_{1}};XW^{K_{2}}],\quad V=XW^{V}\\ A=\mathrm{softmax}(\frac{Q_{1}K^{T}_{1}+Q_{2}K^{T}_{2}}{\sqrt{2d}})\\ O=(AV)W^{O}\end{gathered}$ (8) $$

where $W^{Q_{1}},W^{Q_{2}},W^{K_{1}},W^{K_{2}}\in\mathbb{R}^{d_{\text{model}}\times d},W^{V}\in\mathbb{R}^{d_{\text{model}}\times 2d},W^{O}\in\mathbb{R}^{2d\times d_{\text{model}}}$ are parameters. Denote the gradient of $O$ as $\cfrac{\partial L}{\partial O}$, we formulate gradients of parameters via:

$$ $\displaystyle\cfrac{\partial L}{\partial W^{O}}$ $\displaystyle=\cfrac{\partial L}{\partial O}\cfrac{\partial O}{\partial W^{O}}$ (9) $\displaystyle=(AV)^{\intercal}\cfrac{\partial L}{\partial O}$ $\displaystyle\cfrac{\partial L}{\partial W^{V}}$ $\displaystyle=\cfrac{\partial L}{\partial O}\cfrac{\partial O}{\partial V}\cfrac{\partial V}{\partial W^{V}}$ $\displaystyle=X^{\intercal}A^{\intercal}\cfrac{\partial L}{\partial O}(W^{O})^{\intercal}$ $\displaystyle\cfrac{\partial L}{\partial W^{Q_{1}}}$ $\displaystyle=\cfrac{\partial L}{\partial O}\cfrac{\partial O}{\partial A}\cfrac{\partial A}{\partial Q_{1}}\cfrac{\partial Q_{1}}{\partial W^{Q_{1}}}$ $\displaystyle=\cfrac{1}{\sqrt{2d}}~X^{\intercal}[A\odot(\cfrac{\partial L}{\partial O}(W^{O})^{\intercal}V^{\intercal}-(A\odot(\cfrac{\partial L}{\partial O}(W^{O})^{\intercal}V^{\intercal}))J)]K_{1}$ $\displaystyle\cfrac{\partial L}{\partial W^{Q_{2}}}$ $\displaystyle=\cfrac{\partial L}{\partial O}\cfrac{\partial O}{\partial A}\cfrac{\partial A}{\partial Q_{2}}\cfrac{\partial Q_{2}}{\partial W^{Q_{2}}}$ $\displaystyle=\cfrac{1}{\sqrt{2d}}~X^{\intercal}[A\odot(\cfrac{\partial L}{\partial O}(W^{O})^{\intercal}V^{\intercal}-(A\odot(\cfrac{\partial L}{\partial O}(W^{O})^{\intercal}V^{\intercal}))J)]K_{2}$ $\displaystyle\cfrac{\partial L}{\partial W^{K_{1}}}$ $\displaystyle=\cfrac{\partial L}{\partial O}\cfrac{\partial O}{\partial A}\cfrac{\partial A}{\partial K_{1}}\cfrac{\partial K_{1}}{\partial W^{K_{1}}}$ $\displaystyle=\cfrac{1}{\sqrt{2d}}~X^{\intercal}[A\odot(\cfrac{\partial L}{\partial O}(W^{O})^{\intercal}V^{\intercal}-(A\odot(\cfrac{\partial L}{\partial O}(W^{O})^{\intercal}V^{\intercal}))J)]^{\intercal}Q_{1}$ $\displaystyle\cfrac{\partial L}{\partial W^{K_{2}}}$ $\displaystyle=\cfrac{\partial L}{\partial O}\cfrac{\partial O}{\partial A}\cfrac{\partial A}{\partial K_{2}}\cfrac{\partial K_{2}}{\partial W^{K_{2}}}$ $\displaystyle=\cfrac{1}{\sqrt{2d}}~X^{\intercal}[A\odot(\cfrac{\partial L}{\partial O}(W^{O})^{\intercal}V^{\intercal}-(A\odot(\cfrac{\partial L}{\partial O}(W^{O})^{\intercal}V^{\intercal}))J)]^{\intercal}Q_{2}$ $$

With the property of $\mathrm{softmax}$, we have $A\overset{\Theta}{=}A_{1}\overset{\Theta}{=}A_{2}\overset{\Theta}{=}A_{1}-\lambda A_{2}$, considering gradient magnitude. Therefore, the gradients of the corresponding parameters of attention and differential attention are equivalent in magnitude, differing by some constant factors, as shown in Equation˜7 and Equation˜9. When using an optimizer that is invariant to gradient magnitude, such as AdamW [], parameter updates in Diff Transformer are similar to those of Transformer. This allows us to reuse Transformer hyperparameters without risking training instability.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: NA
soundness: 3
strengths: 1. The proposed modification of attention in Diff Transformer is well-motivated

2. The experimental results are strong, with large-scale experiments up to 3B-parameter models and 350B data tokens. 

3. The pre-trained model was evaluated on multiple benchmarks, and also on long-context evaluation, retrieval-oriented tasks, many-shot in-context learning and hallucination evaluation.

4. The paper is well-written, easy to follow.
summary: The paper aims to mitigate the problem in standard softmax attention to over-allocate attention weights to irrelevant context. The proposed Diff Transformer uses the difference between two separate softmax attention scores to cancel noise assigned to irrelevant contextual tokens. Concretely, Diff Transformer first partition the query and key vectors into two groups to compute separate softmax attention scores. The final attention score is the subtraction of the two groups. One learnable parameter $\lambda$ is introduced to balance the two groups of attention scores.

Experiments were conducted on a 3B-parameter model trained on 350B data tokens. Diff Transformer outperforms Transformer on training loss and downstream benchmarks. Ablation studies investigate the importance of the RMSNorm on each attention head.
weaknesses: Some design motivation in the model is still not clear:

1. Why the learnable $\lambda$ is re-parameterized in Eq (2)?

2. Why in Eq (3) there is a term $(1 - \lambda_{init}$ for each head?

3. Why the RMSNorm for each head is so important for Diff Transformer stability? The explainable in section 3.8 is unconvincing to me.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: 1. Do both attention calculations in differential attention use RoPE positional encoding?
2. Similar to multi-query attention, can $K_1$, $K_2$, or $Q_1$, $Q_2$ share parameters?
3. Is there a significant difference in sparsity between DiffAttn and regular attention? Does the first term of DiffAttn resemble the pattern of standard attention (implying that the second negative attention term serves to cancel out noise)?
4. What is the calculation formula for Table 3? (Specifically, what is the exact normalization operation?)
5. Please discuss the relationship between differential attention and sparse attention.
soundness: 3
strengths: 1. this is a solid and well-written paper. 
2. while the technique itself is not complex, as far as I know, this is the first work to propose a new architecture using differential attention. 3. the experiments and conclusions in this paper are thorough and insightful.
summary: The paper introduces Differential Transformer, a novel architecture for LLMs that enhances attention to relevant context while canceling out noise. The core innovation is a differential attention mechanism that calculates attention scores as the difference between two softmax attention maps, promoting sparse attention patterns and reducing attention noise. The paper demonstrates through extensive experiments that the DIFF Transformer outperforms the standard Transformer in various aspects, including scaling model size, training tokens, long-context modeling, key information retrieval, hallucination mitigation, in-context learning, and reduction of activation outliers.
weaknesses: 1. the authors mention "promoting the emergence of sparse attention patterns" multiple times on Lines 13, 160, and 539, but do not provide statistics and quantification of the sparsity of attention distribution between DIFF Transformer and the general Transformer.
2. The authors did not discuss or conduct experimental comparisons with work related to the sparse attention. 
For example: 
> Efficient Content-Based Sparse Attention with Routing Transformers
>
> Generating Long Sequences with Sparse Transformers.
3. It would be even better if the effectiveness of the DIFF Transformer could be validated on image or speech modalities.
4. The absence of a *Related Work* section limits the paper's ability to contextualize its contributions within the broader field.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: I don't have questions.
soundness: 3
strengths: 1.The proposed DIFF Transformer introduces an innovative differential attention mechanism that reduces attention noise through the difference between two independent softmax mappings. This approach performs exceptionally well in long-context processing, making a significant improvement for long-text tasks.
2.DIFF Transformer demonstrates strong performance in language modeling, in-context learning, and multi-needle retrieval tasks, particularly outperforming traditional Transformers in long-context scenarios. It greatly enhances the accuracy and robustness in practical applications like question answering and text summarization, showing broad potential for various use cases.
3.The paper provides extensive experimental validation, confirming DIFF Transformer’s stability across different tasks and model parameter settings. The hyperparameter tuning is thorough, ensuring both robust performance and reliable results.
4.The paper is well-organized, with clear illustrations and analogies that present the differential attention mechanism effectively. Detailed experimental steps make it easy to understand the innovative contributions.
summary: This paper proposes a new model architecture called Differential Transformer (DIFF Transformer), which reduces noise and more accurately focuses on relevant context through a differential attention mechanism. In a series of language modeling experiments, DIFF Transformer outperforms the standard Transformer across various tasks and model sizes. The paper demonstrates the superiority of this method in long-context modeling, key information retrieval, hallucination mitigation, in-context learning, and reduction of activation outliers.
weaknesses: While the differential attention mechanism brings notable performance improvements, it also adds computational complexity. Efficiency tests indicate that DIFF Transformer’s throughput, particularly with multi-head normalization and dual softmax calculations, is slightly lower than that of traditional Transformers (by about 5%-12%). Although this impact is relatively minor in the experiments, further exploration of computational efficiency would be valuable if scaling to large-scale applications.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: - Can the authors elaborate further on why we need a fixed multiplier $(1 − \lambda_{init}) $ in Diff Transformer (Eq. 3)?
- In section 3.7, Is absmax quantization the suitable approach to adopt in this context? Why not consider quantization approaches that are robust to activation outliers, such as LLM.int8()?
soundness: 3
strengths: - The paper introduces a simple architecture tweak to mitigate the observations that Transformer models tend to incorrectly allocate excessive attentions to irrelevant contexts.
- The paper demonstrates promising empirical results of the proposed architecture through a decent suite of evaluations, ranging from language modeling capability that fits well with scaling law, in-context learning, to improving long-context capability and mitigating contextual hallucination.
- I appreciate that the authors include a variety of downstream tasks to showcase how addressing attention noises in Transformer models can lead to performance improvements.
summary: The paper introduces a new Transformer variant, called Diff Transformer. The key idea is in the design of attention mechanism, where Diff Transformer uses two softmax attention functions to cancel out potential attention noises. Through a suite of empirical studies, Diff Transformer show promising performances compared to standard Transformer architecture.
weaknesses: - Related work can be discussed more thoroughly in the paper. For example, [1] discusses that Transformer models tend to mis-allocate attention to irrelevant contexts potentially biased by their position within the context [1], and [2] also shows LLMs can be easily distracted by irrelevant contexts.
- While the proposed Diff Transformer shows promising performances, they have to be trained from scratch which can be computationally expensive compared to post-training approaches that mitigate attention noises in standard Transformer models, such as the calibration technique used in [1], and improved decoding approaches used in [3] and [4]. Currently, vanilla Transformer is the only baseline considered in the paper. However, I would appreciate if the authors can show how Diff Transformer compares to other more lightweight techniques that fix Transformer's attention noises.
- Apart from pretraineing Diff Transformer, there is no further instruction tuning experiments included in the current paper. I would be interested in seeing whether instruction tuned Diff Transformer also shows better performances than standard instruction tuned Transformers.


[1] Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization. Hsieh et al. 2024.

[2] Large Language Models Can Be Easily Distracted by Irrelevant Context. Shi et al. 2023.

[3] Trusting Your Evidence: Hallucinate Less with Context-aware Decoding. Shi et al. 2023.

[4] Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Maps. Chuang et al. 2024.

Meta-review: This paper proposes a simple (in principle) modification to the attention mechanism, coined differential attention that allows to improve the signal to noise ration in the attention maps for model with very long context windows. The proposed approach is simple, plug-in to all modern transformer-based models and shows good performance on all tasks. Given the positive reviews and interesting and simple research findings, this paper is a clear accept. Given the broad impact (generic architecture improvement) this could be an oral presentation at ICLR 2025.

Decision: Accept (Oral)