Differential Transformer
Authors: Tianzhu Ye
Li Dong
Yuqing Xia
Yutao Sun
Yi Zhu
Gao Huang
Furu Wei
Venue: ICLR 2025 Oral (2025)
Keywords: sequence modeling
language models
model architecture
Transformer
Primary area: foundation or frontier models, including LLMs

Abstract: Transformer tends to overallocate attention to irrelevant context. In this work, we introduce Diff Transformer, which amplifies attention to the relevant context while canceling noise. Specifically, the differential attention mechanism calculates attention scores as the difference between two separate softmax attention maps. The subtraction cancels noise, promoting the emergence of sparse attention patterns. Experimental results on language modeling show that Diff Transformer outperforms Transformer in various settings of scaling up model size and training tokens. More intriguingly, it offers notable advantages in practical applications, such as long-context modeling, key information retrieval, hallucination mitigation, in-context learning, and reduction of activation outliers. By being less distracted by irrelevant context, Diff Transformer can mitigate hallucination in question answering and text summarization. For in-context learning, Diff Transformer not only enhances accuracy but is also more robust to order permutation, which was considered as a chronic robustness issue. The results position Diff Transformer as a highly effective and promising architecture for large language models.

Published as a conference paper at ICLR 2025
DIFFERENTIAL TRANSFORMER
Tianzhu Ee Li Dong Yuqing Xia Yi Zhut Gao Huang Furu Weit BNRist, Tsinghua University, Beijing, China
Yutao Sun
Microsoft Research
http://aka.ms//Gnnet.
ABSTRACT
Transformer tends overallocate attention to irrelevant context. In this work, we introduce DIFF Transformer, which amplifies attention to the relevant context while canceling noise. Specifically, the differential attention mechanism calculates attention scores as the difference between two separate softmax attention maps. The subtraction cancels noise, promoting the emergence of sparse attention pat- terns. Experimental results on language modeling show that DIFF Transformer outperforms Transformer in various settings of scaling up model size and training tokens. More intriguingly. it offers notable advantages in practical applications, such as long-context modeling, key information retrieval, hallucination mitigation, in-context learning, and reduction of activation outliers. By being less distracted by irrelevant context, DIFF Transformer can mitigate hallucination in question answering and text summarization. For in-context learning, DIFF Transformer not only enhances accuracy but is also more robust to order permutation, which was considered as chronic robustness issue. The results position DIFF Transformer a highly effective and promising architecture to advance large language models.
INTRODUCTION
Transformer (Vaswani et al., 2017) has garnered significant research interest in recent years, with the decoder-only Transformer emerging as the de facto standard for large language models (LLMs). At the heart of Transformer is the attention mechanism, which employs the softmax function to weigh the importance of various tokens in sequence. However, recent studies (Kamradt, 2023; Liu et al., 2024b) show that LLMs face challenges in accurately retrieving key information from context.
As illustrated on the left side of Figure 1, we visualize the normalized attention scores assigned to different parts of the context by Transformer. The task is to retrieve an answer embedded in the middle of a pile of documents. The visualization reveals that Transformer tends to allocate only small proportion of attention scores to the correct answer, while disproportionately focusing on irrelevant context. The experiments in Section 3 further substantiate that Transformers struggle with such capabilities. The issue arises from non-negliiible attention scores assigned to irrelevant context, which ultimately drowns out the correct answer. We term these extraneous scores as attention noise.
Normalized Attention Score
Normalized Attention Score
Multi-Needle Retrieval
Transformer
Differential Transformer (This Work)
85%
High Signal to Noss Rbwio
70
0.32
Attention
0.31
Signal-to-Niiee Radio
59%
0.18
0.19
Attention Naise
0.13
0.
0.01
Accunayy
Transfarmtt
<B0S>
..ontext...
ANSWER
Context...
Query
<808>
Context....
ANSWER
Context.
Quiryy
Difoneriol
Figure 1: Transformer often over-attends to irrelevant context (i.e., attention noise). DIFF Transformer amplifies attention to answer spans and cancels noise, enhancing the capability of context modeling.
Equal contribution. Corresponding authors.

---

Published as a conference paper at ICLR 2025
In this paper, introduce Differential Transformer (a.k.a. DIFF Transformer), foundation architec- ture for large language models. The differential attention mechanism is proposed to cancel attention noise with differential denoising. Specifically, partition the query and key vectors into two groups and compute two separate softmax attention maps. Then the result of subtracting these two maps is regarded attention scores. The differential attention mechanism eliminates attention noise, encouraging models to focus on critical information. The approach is analogous to noise-canceling headphones and differential amplifiers (Laplante al., 2018) in electrical engineering, where the difference between two signals cancels out common-mode noise. In the middle of Figure 1, we also present the normalized distribution of attention scores for DIFF Transformer. We observe that DIFF Transformer assigns significantly higher scores to the correct answer and much lower scores to irrelevant context compared Transformer. The right side of Figure shows that the proposed method achieves notable improvements in retrieval capability.
We conduct extensive experiments on language modeling. We scale up DIFF Transformer in terms of parameter count, training tokens, and context length. The scaling curves indicate that DUFF Transformer requires only about 65% of model size or training tokens needed by Transformer to achieve comparable language modeling performance. Moreover, DIFF Transformer outperforms Transformer in various downstream tasks. The long-sequence evaluation also shows that DIFF Transformer is highly effective in utilizing the increasing context. In addition, the experimental results demonstrate that DIFF Transformer has intriguing advantages for large language models. For example, the proposed method substantially outperforms Transformer in key information retrieval, hallucination mitigation, in-context learning, and mathematical reasoning, DIFF Transformer also reduces outliers in model activations, which provides new opportunities for quantization. The findings establish DIFF Transformer as an effective and distinctive foundation architecture for large language models.
DIFFERENTIAL TRANSFORMER
We propose Differential Transformer (a.k.a. DIFF Transformer) as a foundation architecture for sequence modeling, such as large language models (LLMs). We take a decoder-only model as an example to describe the architecture. The model is stacked with DIFF Transformer layers. Given an input sequence r = x1 ... IN. we pack the input embeddings into X = [x], æNn] E R desda where dmodel represents the hidden dimension of the model. The input is further contextualized to obtain the output XL, i.e., X Decoder(X modules: a differential attention module followed by a feed-forward network module. Compared to Transformer (Vaswani et al., 2017), the main difference is the replacement of conventional softmax attention with differential attention while the macro layout is kept the same. We also adopt pre- RMSNorm (Zhang & Sennrich, 2019) and SwiGLU (Shazeer, 2020; Ramachandran et al., 2017) as improvements following LLaMA (Touvron et al., 2023).
1 E [1, L.J. Each layer consists of two
2.1 DIFFERENTALL ATTENTION
The differential attention mechanism maps query, key, and value vectors to outputs. We use query and key vectors to compute attention scores, and then compute a weighted sum of value vectors. The critical design is that use a pair of softmax functions scores. Specifically, given input X E N Xddate, we first project them to query, key, and value Q1. Q2, K1, K2 E IR xd V I. N22d Then the differential attention operator DiffAttn()) com- putes outputs via:
cancel the noise of attention
QQ: [2: = X
[K1; [2]= XW V XWV
1iKT ) softmax
42KT Vd
DiffAttn(X)) = (softmax(
))V
vd
where Wº, W WV E igdena x2d are parameters, and À is a learnable scalar. In order synchronize the learning dynamics, we re-parameterize the scalar À as:
A = exp(>qi
>k,) - cxp(>qp Axa)) Auni
where Aqu AKa Aags A E Rdd are learnable vectors, and Amni E (0, 1) is a constant used for the initialization of X. We empirically find that the setting Ximi 0.8 0.6 exp(-0.3 - (1) works
2

---

Published as a conference paper at ICLR 2025
Linear
def DiffAttn (X, N_q, w_k, W_v, a):
Concat (t (Axx)
Q1, Q2 = split W_q) K1, K2 = split(X (X W_k) V = x &_y
GroupNorm
# Q1, K1: [b, n, d]; V: [b, n, 2d]
s - 1 / sqrt(d)
A1 Q1 K1. transpose (-1, -2) *
[softmax( (Q:K;) À Asoftmax((2
A2 Q2 K2. te..traappseee- (-1, -2) *
h Heads
return
(softmax
softmax (A2)) 0
K1
Q2
K2
def (xl W_q. W_k, N_v, N_o, A): 0 GroupNorm (DiffAttn W_qi, M_ki, M_vi, )) for i range(h)]) 0 Ainit) return Concat (0) W_o
Linear
Linear
Linear
X
Figure 2: Multi-head differential attention. Each head takes the difference between two softmax attention maps to cancel out attention noise. is a learnable scalar that is initialized to Amit- GroupNorm applies normalization each head independently. A fixed multiplier (1 Amit is used after GroupNorm, which aligns the gradient flow with Transformer. The code implementation is available at https:: /akaaaaaa ms /Diff- Transformer.
well in practice, where E [1, L] represents layer index. It is used as the default strategy in our experiments. We also explore using the same Ainit (e.g., 0.8) for all layers as another initialization strategy. As shown in the ablation studies (Section 3.8), the performance is relatively robust to different initialization strategies.
Differential attention takes the difference between two softmax attention functions to eliminate attention noise. The idea is analogous to differential amplifiers (Laplante et al., 2018) proposed in electrical engineering, where the difference between two signals is used as output, so that we can null out the common-mode noise of the input. Naderi al. (2024) also prove that differential attention makes the spectral distribution of attention matrices more balanced, which effectively resolves rank collapse. In addition, the design of noise-canceling headphones is based on a similar idea. We can directly reuse FlashAttention (Dao et al., 2022) as described in Appendix A, which significantly improves model efficiency.
Multi- Head Differential Attention We also use the multi-head mechanism (Vaswani al., 2017) in Differential Transformer. Let & denote the number of attention heads. We use different projection matrices w W:K, wY, i E [1, h] for the heads. The scalar À is shared between heads within the same layer. Then the head outputs are normalized and projected to the final results follows:
head; DiffAttn(X; W.O, W W. A)
head; = (1 Amit) LN(haad,)
MultiHead(XX) = (Concathheadd ... head,
where Ama is the constant scalar in Equation (2), W E R dhodd is a learnable projection matrix, LN(-) uses RMSNorm (Zhang& & Sennrich, 2019) for each head, and Concat(-) concatenates the heads together along the channel dimension. We use fixed multiplier (1 - >iin) as the scale of L.(-)) to align the gradients with Transformer. Appendix G proves that the overall gradient flow remains similar to that of Transformer. The nice property enables to directly inherit similar hyperparameters and ensures training stability. We set the number of heads h = d /2d where d' is equal to the head dimension of Transformer. So we can align the parameter counts and computational complexity.
Headwise Normalization Figure 2 uses GroupNorm(() (Wu & He, 2018) to emphasize that LN(-) is applied to each head independently. As differential attention tends to have sparser pattern, statistical information is more diverse between heads. The LN(-) operator normalizes each head before concatenation improve gradient statistics (Wang al., 2023; Qin et al., 2022).
3

---

Published as a conference paper at ICLR 2025
2.2 OVERALL ARCHITECTURE
The overall architecture stacks L layers, where each layer contains multi-head differential attention module, and reed-forward network module. We describe the Differential Transformer layer
Y MuliiHead(LL((x)) X
X SwiGLU(LN(Y")) +
(5)
where LN(.) is RMSNorm (Zhang & Sennrich, 2019), SwiGLU(X) = o and W W1 E Rdanet chrudd W2 E R tmans demate are learnable matrices.
3 EXPERIMENTS
We evaluate Differential Transformer for large language models from the following perspectives. First, we compare the proposed architecture with Transformers in various downstream tasks (Section 3.1) and study the properties of scaling up model size and training tokens (Section 3.2). Second, conduct a length extension to 64K and evaluate the long-sequence modeling capability (Section 3.3). Third, we present the results of key information retrieval, contextual hallucination evaluation, and in-context learning (Sections 3.4-3.6). Forth, we show that Differential Transformer can reduce outliers in the model activations compared Transformer (Section 3.7). Fifth, we conduct extensive ablation studies for various design choices (Section 3.8).
3.1 LANGUAGE MODELING EVALUATION
We train 3B-size DIFF Transformer language models on IT tokens and compare with previous well-trained Transformer-based models (Geng & Liu, 2023; Tow, 2023; Tow et al., 2023) in various downstream tasks. As described in Appendix B, we follow the same setting to train 3B-size Transformer language model on 350B tokens. The checkpoints also used in the following experiments and analysis to ensure fair comparisons.
Setup We follow similar recipe StableLM-3B-4FF (Tow et al., 2023). We set hidden size to 3072. The number of layers is 28. The head dimension is 128. The number of heads 24 for Transformer and 12 for DIFF Transformer, align computation FLOPs and model size. The total parameter count is about 2.8B. The training sequence length is 4096. The batch size is 4M tokens. We train the models with IT tokens. We use AdamW (Loshchilov & Hutter, 2019) optimizer with 3 = 0.9, 0.95. The maximal learning rate is 3.2e-4 with 1000 warmup steps and linearly decays 1.28e-5. The training corpus also follows StableLM-3B-4EIII (Tow al., 2023). We employ t tiktoken-cll 111100kk base tokenizer. Detailed hyperparameters are provided in Appendix D.
Results Table reports the zero-shot results on the LM Eval Harness benchmark (Gao et al., 2023). We compare DIFF Transformer with well-trained Transformer-bee based language models, including OpenLLaMA-v2-3B (Geng & Liu, 2023), StableLM- Daw.M---a--aapa--------. (Tow, 2023), and StableLM- 3B-4EIT (Tow et al., 2023). DeenLLLMM-v2-331 and M aaaeeallpha-BB-2222 are also trained with IT tokens. The IT results of Bablee_M-33B-411 are taken from its technical report (Tow et al., 2023). Experimental results show that DIFF Transformer achieves favorable performance compared to previous well-tuned Transformer language models. In addition, Appendix shows that DIFF Transformer outperforms Transformer across various tasks, where use the same setting train the 3B-size language models for fair comparisons.
Model
ARC-C ARC-E BoolQ
HellaSwag
OBQA
PIQA
WinoGrande
Avg
Truining with IT tokens
Opent I aMA-3B-v2 (Geng Lin, 2023)
33.9
67.6
65.7
70.0
26.0
76.7
62.9
57.5
StableLM bbaab--s-aal casse-alpha-3B--- x2 (Tow, 2023)
32.4
67.3
64.6
68.6
26.4
76.0
62.1
56,8
BableLM-3B-4EITT (Tow et al., 2023)
66.6
76.8
Du==3B
37.8
72.9
69.0
71.4
29.0
76.8
67.1
60.6
Table 1: Eval Harness Gao et al. (2023) accuracy compared with well-trained Transformer language models (Tow et al., 2023; Tow, 2023; Geng & Liu, 2023). We scale the 3B model to trillion training tokens. The 1T results of StableLM-3B--EI11 are taken from its technical report Tow et al. (2023).
4

---

Published as a conference paper at ICLR 2025
3.15
Transformer Diff (Ours)
2.9
Transformer
Diff (Ours)
3.10
2.8
3.05
Loss
2.7
3.00
2.95
2.6
2.90
2.5
36% Fewer Tokens
10a
10
2
2
2
2
# Parameters (B) (log scale)
#Tokens (B) (log scale)
(a) Scaling model size ranging from 830M to 13B.
(b) Scaling number of training tokens for 3B models.
Figure 3: Language modeling loss of scaling up parameter count and training tokens. DIFF Trans- former requires only about 65% of model size or training tokens to match Transformer's performance.
3.2 SCALABILITY COMPARED WITH TRANSFORMER
We compare the scaling properties of DIFF Transformer and Transformer on language modeling. We scale up the model size, and the number of training tokens, respectively. We follow the augmented Transformer architecture as LLaMA (Touvron et al., 2023) and use the same setting to ensure fair comparison. Specifically, the "Transformer" models include improvements in RMSNorm (Zhang & Sennrich, 2019), SwiGLU (Shazeer, 2020; Ramachandran et al., 2017), and removal of bias.
Scaling Model Size As shown in Figure 3a, we train language models with 830M, 1.4B, 2.8B, 6.8B, and 13.1B parameters. The models are trained with a sequence length of 2048, and a batch size of 0.25M tokens. We train models for 40K steps. Detailed hyperparameters The scaling law (Kaplan et al., 2020) empirically fits well in this contiguration. Figure 3a shows that DIFF Transformer outperforms Transformer in various model sizes. The results indicate that DIFF Transformer is scalable in terms of parameter count. According to the fitted curves, 6,8e-size DIFF Transformer achieves a validation loss comparable to 11B-size Transformer, requiring only 62.2% of parameters. Similarly. 7.8B-size DIFF Transformer matches the performance of 3.1BBsize Transformer, requiring only 59.5% % of parameters.
described in Appendix E.
Scaling Training Tokens As shown in Figure 3b, we evaluate the 3B language models (as presented in Appendix B) every 40B tokens (i.e., 10K steps) up to a total of 360B tokens (i.e., 90K steps). The fitted curves indicate that DIFF Transformer trained with 160B tokens achieves comparable performance Transformer trained with 25 tokens, consuming only 63.7 % of the training tokens.
3.3 LONG-CONTEXT EVALUATION
We extend the 3B-size language models (described in Appendix B) to 64K context length. We continue train- ing the 3B checkpoints for additional 1.5B tokens. Most hyperparameters are kept the same as in Section 3. The learning rate 8e-5. The RoPE (Su al., 2021) 0 is increased to 640,000. The training corpus is sammmdee according to sequence length (Fu et al., 2024).
Negative Log-Likelihood
Transformer
Diff (Ours)
Results Figure 4 presents cumulative average nega- tive log-likelihood (NL.)) of the tokens varying po- sitions (Reid et al., 2024), where lower NLL indicates better performance. The evaluation is conducted on book data within 64K length. We observe consistent decrease in NLL as the context length increases. DIFF Transformer achieves lower NLL values than Trans- former. The results demonstrate that DIFF Transformer effectively leverage the increasing context.
100
1K
10K
100K
Sequence Position
Figure 4: Cumulative average negative log- likelihood (lower better) book data. DIFF Transformer leverages long context more effectively.
5

---

Published as a conference paper at ICLR 2025
N 8, R 1 Multi-Needle Retrieval
8, R 1 Multi-Needie Retrieval
0.96
0.88
0.52
1.00
0.44
1.00
0.66
0.58
0.16
0.58
0.86
0.64
Depth (%6
0.76
0.6
Score
Depth (%6
0.96
0.6
0.4
100
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
Avg.
0.64
Avg.
49K
56k
64k
16k
49K
58K
64k
Context Length
Context Length
(a) Transformer.
(b) DIFF Transformer.
Figure 5: Multi-needle retrieval results in 64k length.
3.4 KEY INFORMATION RETRIEVAL
The Needle- In-A In-A--aystack (Kamradt, 2023) test is widely used to evaluate the ability to extract critical information embedded a large context. We follow the multi-needle evaluation protocol of LWM (Liu et al., 2024a) and Gemini 1.5 (Reid et al., 2024). The needles are inserted into varying depths within contexts different lengths. Each needle consists of a concise sentence that assigns a unique magic number to specific city. The goal is retrieve the magic numbers corresponding to the query cities. We position the answer needle at five different depths within the context: 0%. 25%, 50%, 75%, and 100%, while placing other distracting needles randomly. Each combination of depth and length is evaluated using 50 samples. The average accuracy is reported. Let N denote the total number number-city pairs and R the number of query cities.
Retrieve from 4K Context Length As shown in Table 2, we insert N 1, needles into 4K-length contexts and retrieve R = 1,2 nee- dles. We evaluate 3B-size models trained with 4K input length (Appendix B). We find that both models obtain good accuracy for N
Model
N
= 2 N
=
N = 6
R =
R = 2
R 2
R = 2
Transformer
1.00
0.85
0.62
0.55
and N = 2. As N and R increase, DIFF Transformer maintains a consistent accuracy, while the per- formance of Transformer drops significantly, In particular, at N = 6, R = 2, the accuracy gap be- tween the two models reaches 30%. The results indicate the superior ability of DIFF Transformer retrieve key information in distracting contexts.
DIFF
1.00
0.84
0.85
Table 2: Multi-needle retricval accuracy in 4K length, averaged over the answer needle positions. N represents the number of needles, and R de- notes the number of query cities.
Retrieve from 64K Context Length As shown in Figure 5, the evaluated context length ranges from 8K 64K for the N 8, R setting. We evaluate the 3B-size models with length extension (Section 3.3). We report the accuracy across varying answer needle depths (y-axis) and context lengths (x-axis). The bottom row is the average accuracy for all depths. DIFF Transformer maintains stable performance across different context lengths. In contrast, Transformer's 's average accuracy gradually declines as the context length increases up the maximal length, i.e.. 64K. Besides, DIFF Transformer outperforms Transformer particularly when key information is positioned within the first half of the context (i.e., 0%, 25%, and 50% depth). particular, when needles are placed at the 25% depth in 64K context, DIFF Transformer achieves 76% accuracy improvement over Transformer.
itention Score Analysis Table presents the attention scores allocated to the answer span and the noise context for the key information retrieval task. The scores indicate the model's ability to preserve useful information against attention noise. We compare the normalized attention scores when key information is inserted at different positions (i.e., depths) within the context. Compared with Transformer, DIFF Transformer allocates higher attention scores to the answer span and has lower attention noise.

---

Published as a conference paper at ICLR 2025
Attention to Answer
Attention Noise +
Model
0%
25%
50%
75%
100%
0%
25%
50%
75%
100%
Transformer
0.03
0.03
0.03
0.07
0.09
0.51
0.54
0.52
0.49
0.49
DIFF
0.27
0.30
0.31
0.32
0.40
0.01
0.02
0.02
0.02
0.01
Table Attention scores allocated answer spans and noise context in the key information retrieval task. The target answer is inserted in varying positions (i.e., depth) of context. DIFF Transformer allocates more attention scores to useful information and effectively cancels out attention noise.
3.5 IN-CONTEXT LEARNING
We evaluate in-context learning from two perspectives, including many-shot classification and robustness of in-context learning. In-context learning fundamental capability of language models, which indicates how well model can utilize input context.
Many-Shot In-Context Learning As presented in Figure 6, we compare the accuracy of many-shot classification between Transformer and our architecture. We evaluate the 3B-size language models that support 64K input length (Section 3.3). We follow the evaluation protocol of (Bertsch al., 2024) and use constrained decoding (Ratner et al., 2023). We incrementally increase the number of demonstration samples from 1-shot until the total length reaches 64K length. Specifically, the TREC (Hovy et al., 2001) dataset has 6 classes, TREC-fine (Hovy et al., 2001) has 50 classes, Banking-77 (Casanueva et al., 2020) has classes, and Clinic-150 (Larson et al., 2019) has 150 classes. The results show that DIFF Transformer consistently outperforms Transformer across datasets and varying numbers of demonstration samples. Moreover, the improvement in average accuracy is substantial, ranging from 5.2% to 21.6%.
90
80
Accuracy (%)
80
+18.0
Accuracy (%)
21.66
70
60
60
50
Diff (Ours)
Diff (Ours) Transformer
Transformer
50
0
1000
2000
3000
1000
2000
3000
# Samples
Samples
(a) TREC with 6 classes.
(b) TREC-fine with 50 classes.
80
80
+10.4
+5.2
Accuracy (%)
70
Accuracy (%)
70
60
65
50
Diff (Ours)
Diff (Ours)
55
Transformer
Transformer
500
1000 1500 2000 2500 # Samples
0
1000
2000
3000
# Samples
(c) Banking-777 77 with 77 classes.
(d) Clinic- 150 with 150 classes.
Figure 6: Many-shot in-context learning accuracy four datasets. Demonstration examples increase from 1-shot until the total length reaches 64K tokens. The dashed lines represent the average accuracy after the performance becomes stable.

---

Published as a conference paper at ICLR 2025
100
90
4.00
90
13.4
85
Accuracy (96)
Accuracy (%)
80
70
75
60
56.7
19.0
50
70
Diff (Ours)
Diff (Ours)
Transformer
65
Transformer
30
3
15
25
30
Random Seed
Random Seed
(a) Examples are randomly arranged.
(h) Examples arranged alternately by class.
Figure 7: Robustness evaluation of in-context learning on the TREC dataset. Accuracy is evaluated with order permutations of demonstration examples by sweeping random seeds. 'he dash lines represent the margin between the best and worst results. Smaller margin indicates superior robustness. Two prompt formats examined.
Robustness of In-Context Learning Figure 7 compares the robustness of in-context learning between Transformer and DIFF Transformer. Given the same demonstration examples, we analyze the performance variance with order permutations. Lower variance indicates greater robustness and less risk of catastrophic performance degradation. The evaluation protocol is the same as above. Figure presents the analysis the TREC dataset. More results are also provided in Appendix F. We evaluate two prompt formats, i.c., examples are randomly arranged (Figure 7a), and alternately arranged by class (Figure 7b). In both settings, DIFF Transformer has much smaller performance variance compared to Transformer. The results indicate that our approach is more robust for in-context learning. In contrast, Transformer tends to be distracted by order permutations (Lu et al., 2022), resulting in huge margin between the best and worst results.
3.6 CONTEXTUAL HALLUCINATION EVALUATION
We evaluate contextual hallucination of the 3B-size language models (described in Appendix B) on text summarization and question answering. Notice that we focus on the cases where the input context contains correct facts, but model still fails to produce accurate outputs.
We follow the evaluation protocol of (Chuang et al., 2024). We feed the model output along with ground- truth responses to GPT-4o (OpenAI, 2024). Then we ask GPT-4o to make binary judgements on whether the model outputs accurate and free of hallucinations. Previous studies (Chuang et al., 2024; Ravi et al., 2024) have shown that the above hallucination evaluation protocol has relatively high agreement between GPT-4o judgments and human annotations. The automatic metrie is reliable and mirrors the human evaluation. For each dataset, the accuracy is averaged over 100 samples.
Summarization Table 4a presents hallucination evaluation summarization datasets XSum (Narayan et al., 2018), CNN/DM (See et al., 2017), and MultiNews (Fabbri et al., 2019). The task generate summaries for input documents.
Model
XSum CNN/DM MultiNews
Model
Qasper
HotpotQA
2WikiMQA
Transformer
0.44
0.32
0.42
Transformer
0.28
0.36
0.29
DIFF
0.53
0.41
0.61
DIFF
0.39
0.46
0.36
(a) Accuracy (i.e., free of hallucinations) marization datasets.
Accuracy (i.e., free of hallucinations) on question answering datasets.
Table 4: Evaluation of contextual hallucination text summarization and question answering, Higher accuracy indicates less hallucination. We follow Chuang et al. (2024) to employ GPT-4o to make binary judgments, which has relatively high agreement with human annotation.
8

---

Published as a conference paper at ICLR 2025
Model
Activation Type
Top-1
Top-2
Top-3
Top-10
Top-100
Median
Transformer
Attention Logits
318.0
308.2
304.9
284.7
251.5
5.4
DIFF
Attention Logits
38.8
38.8
37.3
32.0
27.4
3.3
Transformer
Hidden States
3608.6
3607.4 4
3603.6
3552.1
2448.2
0.6
DIFF
Hidden States
1688.2
1672.5
1672.1
1624.3
740.9
1.2
Table 5: Largest activation values in attention logits and hidden states. Top activation values are considered activation outliers, due to their significantly higher magnitude than the median. DIFF Transformer mitigates outliers compared to Transformer.
Question Answering As shown in Table 4b, we compare the hallucination rate of DIFF Transformer and Transformer on both single- and multi-document question answering. The Qasper (Dasigi et al., 2021) dataset single- single-document question answering. In contrast, HotporQA (Yang et al., 2018) and 2WikiMulthhoPA (Ho et al., 2020) are multi-document question answering. The goal is to answer questions about the given context. All evaluation examples are from LongBench (Bai et al., 2023).
Compared with Transformer, our method mitigates contextual hallucination on summarization and question answering. The performance improvement possibly stems from DIFF Transformer's better focus on essential information needed for the task, instead of irrelevant context. This aligns with previous observation (Huang et al., 2024) that one primary reason for contextual hallucination in Transformer is the misallocation of attention scores.
3.7 ACTIVATION OUTLIERS ANALYSIS
In large language models, subset of activations manifests with significantly larger values compared to the majority, a phenomenon commonly called activation outliers (Bondarenko et al., 2024; Sun et al., 2024). The outliers result in difficulties for model quantization during training and inference. We demonstrate that DIFF Transformer can reduce the magnitude of activation outliers, potentially allowing lower bit-widths for quantization.
Statistics of Largest Activation Values Table presents the statistics of activation values collected from Transformer and DIFF Transformer models trained in Appendix B. We analyze two types of activations, including attention logits (i.c., pre-softmax activations), and hidden states (i.c., layer outputs). The statistics are gathered from 0.4M tokens. As shown in Table 5, although the median values are of similar magnitude, DIFF Transformer exhibits much lower top activation values compared to Transformer. The results show that our method produces fewer activation outliers.
Quantization of Attention Logits As shown in Figure 8, we quantize the attention logits to lower bits. We apply dynamic post-training quantization using absmax quantization (Wan et al., 2024). The 16-bit configuration represents the original results without quantization. The models are progressively quantized to 8 bits, 6 bits, and 4 bits. Figure re- ports the zero-shot performance on HellaSwag (Gao al., 2023). The other datasets follow similar trend. DIFF Transformer retains high performance even at reduced bit-widths, ranging from 16 bits to 6 bits. In comparison, Transformer's accuracy significantly drops with 6-bit quantization. The bit DIFF Transformer achieves comparable accuracy as the 6-bit Transformer, and outperforms the 4-bit Transformer by about 25% in accuracy. The results indicate that DIFF Transformer natively mitigates activation outliers in attention scores, providing new opportunities for low-bit FlashAttention (Dao et al., 2022) implementations.
HellaSwag
60
Accuracy (%)
50-
Diff (Ours)
Transformer
40
16
Bits 6
Figure 8: Zero-shot accuracy on the Hel- laSwag (Gao et al., 2023) dataset. We quantize the attention logits from 16 bits (i.e., unquan- tized) to 8 bits, 6 bits, and 4 bits.

---

Published as a conference paper at ICLR 2025
Fine-Grained Slices
Model
#heads
GN
Valid. Set,
AR-Hitt
Others.
Transformer
16
128
X
3.087
0.898
3.272
Transformer
8
256
X
3.088
0.899
3.273
GroupNorm
8
256
V
3.086
0.899
3,271
DIFF Transformer
8
128
/
3.062
0.880
3.247
-GroupNorm
8
128
X
3.122
0.911
3,309
with Ainit = 0.8
128
3.065
0.883
3,250
with Amit 0.5 8 128
3.066
0.882
3,251
Table Ablation studies of 1.4B-size models. We report language modeling loss on the validation set. We also follow Arora et al. (2023) to report fine-grained metrics, where "AR-Hit" evaluates n-grams previously seen in the context. "#Heads" is number of heads. "d" head dimension. "GN" indicates whether GroupNorm is used.
3.8 ABLATION STUDIES
We conduct ablation studies with 1.4B-size language models. The training setup the same as the 1.4B model in Section 3.2. The models have I. 24 layers, h = 16 heads for Transformer, and heads for DIFF Transformer. The head dimension = 128. Detailed hyperparameters are described in Appendix E.
Table reports fine-grained loss on the validation set. We follow Zoology (Arora et al., 2023) and divide loss into "Ar-Hir" and "Others". Specifically, "Ar-Hit" considers the last token of an n-gram previously seen in the context, which evaluates the associative recall capability. The "Others" slice represents the tokens that cannot be recalled from the context frequent tokens.
As shown in Table 6, we ablate various design choices of DIFF Transformer and present several Transformer variants. Notice that all models have comparable size and training FLOPs for fair comparisons. The first and fourth rows are the default settings for Transformer and DIFF Transformer, respectively, which directly taken from Figure 3a. Our method outperforms Transformer in terms of both overall and fine-grained loss. As DIFF Transformer halves the number of heads to match model size, the second row shows that the configuration change does not have much impact. We ablate GroupNorm from DIFF Transformer, which degrades performance due to training instability. Because multiple heads tend have different statistics our method, GroupNorm plays key role in normalizing them to similar values. In contrast, comparing the third and first rows, adding GroupNorm Transformer has negligible effect on performance. The results indicate that the improvements of our method come from the differential attention mechanism, instead of configurations modules. Moreover, we compare different strategies to initialize X. As described in Section 1, the default setting uses exponential initialization, i.e., Amit 0.8 0.6 x exp(-0.3 -0.3 (1 1)), where / is the layer index. The last two rows employ constant initialization with Amni = 0.8, 0.5. The minimal change in the validation loss suggests that the models are robust to the choice of À initialization.
normalization
4 CONCLUSION
In this work, we introduce Differential Transformer (a.k.a. DIFF Transformer), which amplifies attention to the relevant context while canceling noise. Experimental results on language modeling show that DIFF Transformer outperforms Transformer terms of scaling properties, long-conted modeling, key information retrieval, hallucination mitigation, in-context learning, and reduction of activation outliers. The results emphasize the importance of reducing attention noise. Moreover, the differential attention mechanism can be easily implemented with Flashhttention (Dao et al., 2022). The findings position DIFF Transformer as a distinctive and promising foundation architecture for large language models. In the future, reduced magnitude of activation outliers. As the attention pattern becomes much sparser, we would also like utilize the property to compress key-value caches.
develop efficient low-bit bit attention kernels due to the
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
We would like to acknowledge Ben Huntley for maintaining the GPU cluster. The long-sequence training utilizes CUBE, which an internal version of (Lin et al., 2023).
The work is supported in part by the National Key R&D Program of China under Grant 2024YFB4708200 and National Natural Science Foundation of China under Grant U24B20173.
REFERENCES
Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. Zoology: Mcasuring and improving recall in efficient language models. arXiv preprint (eXiv:2312.0492) 2023.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308. 14508, 2023.
Amanda Bertsch, Maor Ivgi, Uri Alon, Jonathan Berant, Matthew R. Gormley, and Graham Neubig. In-context learning with long-context models: An in-depth exploration. arXiv:2405.00200 2024.
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Quantizable transformers: Removing outliers by helping attention heads do nothing. Advances Neural Information Processing Systems, 36, 2024.
Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. Efficient intent detection with dual sentence encoders. In Proceedings of the 2nd Workshop Natural Language Processing for Conversational AI, pp. 38-45, 2020.
Yung-Sung Chuang, Linlu Qiu, Cheng- Yu Hsieh, Ranjay Krishna, Yoon Kim, and James Glass. Lookback lens: Detecting and mitigating contextual hallucinations in large language models using only attention maps. arXiv preprint arXiv:2407.00071, 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Tri Dao. FlashAttention 2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.0869/1 2023.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances Neural Information Processing Systems, 16:46944-66359, 2022.
Pradeep Dasigi, Kyle Lo, Iz. Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. A dataset of information-seekking questions and answers anchored in research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4599-4610, 2021.
Alexander Richard Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. Multi-news: large- scale multi-doccumntt summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1074-1084, 2019.
Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hanna Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128k context. ArXiv, abs/2402.10177 10171, 2024.
Leo Gao. Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang. Kevin Wang. and Andy Zou. framework for few-shot language model evaluation, 12 2023.
11

---

Published as a conference paper at ICLR 2025
Xinyang Geng and Hao Liu. OpenLLaMA: An open reproduction of LLaMA. https://github com/ htth/ppplmmmeeeee resaanmcreeeearooppe.e..e 1mma, 2023.
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability Ilms via reinforcement learning. arXiv preprint arXiv:2501.12944 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fffth Conference on Neural Information Processing Systems Datasets and Benchmurks Truck (Round 2), 2021.
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 8609-6625, 2020.
Eduard Hovy, Laurie Gerber, Ulf Hermjakob, Chin- Yew Lin, and Deepak Ravichandran. Toward semantics-based answer pinpointing. In Proceedings of the first international conference on Human language technology research, 2001.
Qidong Huang, Xiaoyi Dong. Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang. Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and vetrospection-allccatinn Computer Vision and Pattern Recognition, pp. 13418-13422 2024.
Proceedings of the IEÉE/CVF Conference
Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16770, 2024.
Greg Kamradt. Needle in a Haystack pressure testing LLMs https://githin com/ gkamradt wttrrrattetLittet_eeeeeeeee InAeeslaaIAHaayttcac.. ttattttcckckeeeeeeeeee tree/mal in, 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, n language models.
Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural CoRR, 0as/200110833/. 2020.
Rik
Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. Mawps: A math word problem repository. In Proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies, pp. 1152- 1157, 2016.
Philip A Laplante, Robin Cravey, Lawrence P Dunleavy, James L. Antonakos, Rodney LeRoy, Jack East, Nicholas E Buris, Christopher J Conant, Lawrence Fryda, Robert William Boyd, et al. Comprehensive dictionary of electrical engineering. CRC Press, 2018.
Stefan Larson, Anish Mahendran, Joseph J Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K Kummerfeld, Kevin Leach, Michael A Laurenzano, Lingjia Tang. al. An evaluation dataset for intent classification and out-of-scope prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP,, pp. 1311-1316, 2019.
Haoran Li, Qingxiu Dong, Zhengyang Tang. Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shao- han Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, et al. Synthetic data (almost) from scratch: Generalized instruction tuning for language models. arXiv preprint 240. 2024.
Zhiqi Lin, Youshan Miao, Guodong Liu, Xiaoxiang Shi, Quanlu Zhang, Fan Yang. Saeed Maleki, Yi Zhu, Xu Cao, Cheng Li, Mao Yang, Lintao Zhang, and Lidong Zhou. SuperScaler: Supporting flexible DNN parallelization via unified abstraction, 2023.
Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. arXiv preprint arXiv:2402.08268, 2024a.
12

---

Published as a conference paper at ICLR 2025
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Perey Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024b. 20..4.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019.
Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. In ICLR, 2023.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pp. 8086-8098, Dublin, Ireland, May 2022. Association for Computational Linguistics.
Shen-Yur Yun Miao, Chao-Chun Liang, and Keh- Yih Su. A diverse corpus for evaluating and developing english math word problem solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 975-984, 2020.
Alireza Naderi, Thiziri Nait Saada, and Jared Tanner. Mind the gap: spectral analysis of rank collapse and signal propagation in attention layers, 2024.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguisties, 2018.
Open-R1. Openthoughts- 114k-maathh-11-t--hhha.aaaahh-, 2025. URL htt.hpps//hgggingfaa....ooeteeee 3paa==///peenhhhughht--14444
OpenAl. Hello, gpt-4o.
2024.
OpenThoouhts. Open Thoughts. https:/opee--hhouhtsaai January 2025.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nip models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 2080-2094, 2021.
Zhen Qin, Xiaodong Han, Weixuan Sun, Dongxu Li, Lingpeng Kong, Nick Barnes, and Yiran Zhong. The devil in linear transformer. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 7025-7041, 2022.
Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Swish: self-gated activation function. arXiv: Neural and Evolutionary Computing, 2017.
Nir Ratner, Yoav Levine, Yonatan Belinkov, Ori Ram, Inbal Magar, Omri Abend, Ehud Karpas, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. Parallel context windows for large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, pp. 6383-6402, 2023.
Selvan Sunitha Ravi, Bartosz Mielezarek, Anand Kannappan, Douwe Kiela, and Rebecca Qian. Lynx: An open source hallucination evaluation model. arXiv preprint arXiv:2407.0844 2024.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac. Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
Abigail See, Peter J Liu, and Christopher D Manning. Get to the point: Summarization with vointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2017.
13

---

Published as a conference paper at ICLR 2025
Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-3: Fast and accurate attention with asynchrony and low pww-precision. arXiv preprint arXiv:2407.08608, 2024.
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.052022 2020.
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, arXiv preprint arXiv: 10..220444.9944 2021.
Mingjic Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2024.
Zhengyang Tang, Xingxing Zhang, Benyou Wang, and Furu Wei. Mathscale: scaling instruction tuning for mathematical reasoning. In Proceedings of the 41st International Conference on Machine Learning, pp. 47885-47900, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302 / 13971, 2023
Jonathan Tow. StableLM Alpha v2 models. stablein-base-alle alpha- 3b-v2, 2023.
Jonathan Tow, Marco Bellagente, Dakota Mahan, and Carlos Riquelme. StableLM 3B 4EIT.
https:
/ /tt.//aaaaaaa .as Stts/ttaaleMMM 3B-4BB-- 4E1T, T, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp. 6000-6010, 2017.
Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Jiachen Liu, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, Mosharaf Chowdhury, and Mi Zhang. Efficient large language models: A survey. Transactions on Machine Learning Research, 2024.
Hongyu Wang, Shuming Ma. Shaohan Huang, Li Dong, Wenhui Wang. Zhiliang Peng, Yu Wu, Payal Bajaj, Saksham Singhal, Alon Benhaim, al. Magneto: foundation Transformer. In International Conference Machine Learning, pp. 36077-36092. PMLR, 2023.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference computer vision (ECCV), pp. 3-19, 2018
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369-2380, 2018
Beichen Zhang, Kun Zhou, Xilin Wei, Xin Zhao, Jing Sha, Shijin Wang, and Ji-Rong Wen. Evaluat- ing and improving tool-augmented computation-intensive math reasoning. Advances in Neural Information Processing Systems, 36:23570-23589, 2023.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systemx, 32, 2019.
14

---

Published as a conference paper at ICLR 2025
A
IMPLEMENTATION OF DIFFERENTIAL ATTENTION
We present the pseudocode for DiffAttn(-) and conventional softmax attention.
Attention (X, W_q, W_k, w_v):
def (iffAttn W_q. W,k, W_v, A):
q X 0 W_q
Q1, Q2 split(X 0 W_q)
X Q W_k
K1, K2 split(X @ W_k)
X Q W_v
V a W_v
q, K, V: [b, n, d]
qi, Ki: [b, n, d]; V: [b, n, 2d]
a = 1 sqrt(d)
= 1 sqrt(d)
A Q 0 K. (-am -2)
A1 = Q1 0 K1 s1.ttrnnppoee-11 (-1, -2) *
A2 = Q2 6 K2. transpose (-1, -2) *
return softmax 8
return
(softmax X softmax (A2)) V
Implementation with FlashAttention Additionally, we provide implementations with FlashAtten- tion (Dao et al., 2022). We categorize the implementations into two types by whether supports using different dimensions between Q, K and V. Specifically, let FlashDiffAttn I(-) denote the package that supports different dimensions (e.g.. xformers ), and FlashDiffAttn 2(-) the package that does not (e.g., flash-attent aattetieiinnnt iontniion We also implement cust zed aad-flash-att tish-attent ion" package, which is modified based on the official FlashAttention2 (Dao, 2023), in order to support different dimensions between Q, K and V.
The code implementation is available at https: wtt.aaaa/....ffrrrrssffoooo
FlashDiffAtcn_L (X, N_q. W_k, W_v, A): Q1, Q2 split w_q) K1, K2 split(X 0 W_k)
def FlashDiffAttn_ 2X, W_q. W_k, W_v, A): Q1, Q2 split 0 W_q)
K1, K2 split(X 0 W_k)
V I 0 W_v
V1, V2 split (X 0 W_v)
A1 Blash_attn(PP) K1, V)
A11 flash_atta K1, V1)
A12
K1, V2)
A1 Concat(A11, A12)
A2 flash_ate (Q2 K2, V)
A21 flash_attn (Q2, K2, V1)
A22 Beahh_attn((e (Q2, K2, V2)
A2 Concat(A21, A22)
return A1 À A2
return A1 À A2
Efficiency Table 7 compares the throughput between DIFF Transformer and Transformer. For fair comparison, we use the customiie zed- fl flash-ar at ent ion implementation mentioned above for both methods. The experiments are conducted with Nvidia H100-80GB GPU cards.
Throughput
Model
Model Size
Length
Forward + Backward
Forward
Transformer
3B
2K
7247
51228
DIFF
3B
2K
6635 9%)
46811 (-9%)
Transformer
3B
4K
7491
48762
DIFF
3B
4K
6718
(-12%)
44521 (-10%)
Transformer
13B
2K
998
14346
DIFF
13B
2K
942 ( 6%)
13653 5%)
Table Throughput is measured with number of tokens per second.
As shown in Table 7, we evaluate the settings with different model size (3B, 13B) and context length (2K, 4K), For 3B models, there are 12 heads for DIFF Transformer and 24 heads for Transformer. For
15

---

Published as a conference paper at ICLR 2025
13B model there 20 heads for DIFF Transformer and 40 heads for Transformer. All models have the same head dimension = 128. Training efficiency consists forward and backward. Prefill efficiency only includes forward. Table 7 shows that the throughput results are comparable within an acceptable range. Notice that the customi zed-f lash sesh-attentti on implementation built on FlashAttention2 (Dao, 2023). With the recent release of Attentonn (Shah al., 2024), the gap of throughput can be further reduced. More advanced kernel implementation, which is specifically designed for differential attention, can also improve throughput.
B LANGUAGE MODELING EVALUATION
Following the same setting as in Section 3.1, we train 3B-size language models on 350B tokens and compare DIFF Transformer with Transformer (Vaswani et al., 2017) in various downstream tasks. We use the augmented Transformer architecture as in LLaMA (Touvron al., 2023). Specif- ically, the "Transformer" models include improvements in RMSNorm (Zhang & Sennrich, 2019), SwiGLU (Shazeer, 2020; Ramachandran et al., 2017), and removal of bias.
Table 8 reports the zero-shot and 5-shot results on the LM Eval Harness benchmark (Gao al., 2023). The results show that DIFF Transformer outperforms Transformer across various tasks in both zero-shot and few-shot settings.
Model
ARC-C ARC-E
BoolQ
HellaSwag
OBQA
PIQA
WinoGrande
Avg
Training with 350B tokens (Zero-Shot)
Transformer-3B
32.2
66.8
62.9
63.4
26.2
74.5
61.6 62.7
55.4
DIFF-3B
33.0
68.3
60.1
66.2
27.6
75.5
56.2
Training with 350B tokens (5- Shot)
Transformer-3B
34.0
69.5
65.3
63.4
25.0
75.2
62.6
56.4
DIFF-3B
35.0
69.5
67.2
66.9
27.6
76.1
63.8
58.0
Table Comparison of DIFF Transformer with well-trained Transformer language models on LM Eval Harness (Gao et al., 2023). DIFF Transformer achieves better accuracy in the zero- and few-shot settings.
C EVALUATION ON MATHEMATICAL REASONING
We continue training the 3B-size language models that support 64K input length (Section 3.3) with math data to evaluate their ol-style (Jaech et al., 2024) reasoning capability. The training consists of two stages: fine-tuning with synthetic math data, and distilling from DeepSeek-R1 (Guo et al., 2025) to promote ol-style reasoning. We evaluate the models across math benchmarks: GSM-SK (Cobbe al., 2021),, MATH (Hendrycks al., 2021), SVAMP (Patel al., 3021), ASDiv (Miao et al., 2020), et al., 2016), CARP (Zhang et al., 2023), TABMWP (Lu et al., 2023), and CollegeMath (Tang et al., 2024).
MAWPS
Math Capability Evaluation In the first stage, we train both DIFF Transformer and Transformer for additional 20B tokens on synthetic math data (Li et al., 2024). The learning is 8e-5. The batch size is 4M tokens. Hy-
30
Diff (Ours)
Transformer
perparameters are kept the same as in Section 3.3. We evaluate the models every 2B tokens from 6B tokens to 20B tokens and report the average accuracy over 8 math datasets. The output length is restricted kens. As shown in Figure 9, DIFF Transformer surpasses Transformer in solving mathematical problems. DIFF Transformer starts to substantially outperform Trans- former after 15B tokens, reaching an accuracy gap of 11.3% by the end of 20B tokens. The results demon- strate that DIFF Transformer can learn to solve reasoning tasks more effectively than Transformer.
Accuracy
20
15.
to-
10-
5
10
15
20
# Tokens (B)
Figure 9: DIFF surpasses Transformer in averaged accuracy over math datasets.
16

---

Published as a conference paper at ICLR 2025
06.2
Diff (Ours)
80
70.8
75.1
Transformer
Accuracy (%)
60
55.0
42.641.2
42.5
40-
27.5
30.0
19.9
16.4
GSM 8K
MATH
SVAMP
ASDIV
MAWPS
CARP
TAMMWP
CollegeMath
Average
Figure 10: Accuracy on 8 math benchmarks with ol reasoning.
ol-style Reasoning Evaluation In the second stage, we further distill the models from DeenThoughs--114K-WM (Open-R1, 2025). The dataset is filtered from OpenThoughts-- 114K (OpenTThoughts, 2025) dataset. It consists of 89K math samples with the average length of 6K tokens. We apply supervised fine-tuning on the dataset to equip the models with ol-style reasoning capability. The learning rate is set to le-5. The batch size is 1M tokens. Other hyperpa- rameters are the same as in the first stage. We train both models for 2B tokens and select the best checkpoint for each. The output length is restricted to 16K tokens. As shown in Figure 10, DIFF Transformer outperforms Transformer on benchmarks with an average accuracy gain of 7.5%. DIFF Transformer generates reasoning process with average length of 6144 tokens, compared to 6913 for Transformer. The experimental results demonstrate the superior reasoning capability of DIFF Transformer over Transformer. It suggests that differential attention mechanism contributes to the improved performance mathematical reasoning.
17

---

Published as a conference paper at ICLR 2025
D HYPERPARAMETERS FOR SECTION 3.1
Table 9 presents the detailed hyperparameters for the DIFF Transformer-3B models in Section 3.1. For Transformer-3B, the only difference is that there 24 heads. Notice that both Transformer-3B and DIFF Transformer-3B have similar FLOPs.
Params
Values
Layers
28
Hidden size
3072
FFN size
8192
Vocab size
100,288
Heads
12
Adam
(0.9, 0.95)
LR
3.2 x 10
Batch size
4M
Warmup steps
1000
Weight decay
0.1
Dropout
0.0
Table 9: Hyperparamters used for the DIFF Transformer-3B model Section 3.1.
E HYPERPARAMETERS FOR SECTION 3.2
Table 10 reports the hidden dimension, number of layers, and number of heads of DIFF Transformer for different model sizes. For all model sizes of Transformer, we double the number of heads compared with DIFF Transformer align parameters. The FFN size is 3 x dmuoeee where dawdel is the hidden dimension. The training length is set to 2048. The batch size is set to 0.25M tokens. We use AdamW (Loshchilov & Hutter, 2019) with 31 = 0.9, 33 = 0.98. The learning rate is 1.5 x 10-4 for 830M to 2.8B sizes, and 7.5 x 10-3 for 6.8B to I3.1B sizes. The warmup step is 375 with linear rate decay. The weight decay is set to 0.05. We train the models with 40k steps, i.e. 10B tokens.
Size
Hidden Dim.
#Layers
#Heads
830M
1536
24
1.4B
2048
24
2.8B
2560
32
10
6.8B
4096
32
16
13.1B
5120
40
20
Table 10: Model size and hyperparameters used for DIFF Transformer in Section 3.2.
18

---

Published as a conference paper at ICLR 2025
F ROBUSTNESS OF IN-CONTEXT LEARNING
As described Section 3.5, we evaluate the robustness of in-context learning of Transformer and DIFF Transformer with permutations of the same in-context examples. We evaluate the 3B-size language models that are extended to 64K length (Section 3.3).
Figure 11 provides comparisons evaluation protocol is the same as in Section 3.5. The variance in accuracy of DIFF Transformer is consistently lower than that of Transformer, indicating greater robustness of DIFF Transformer for in-context learning.
four datasets, with in-context examples randomly arranged. The
90
14.0
85
80
Accuracy (96)
Accuracy
70
75-
19.0
24.0
70
Diff (Ours)
60
Transformer
Diff (Ours)
65
Transformer
2
3
7
8
8
Random Seed
Random Seed
(a) TREC with 6 classes.
(b) TREC-fine with 50 classes.
82
80
80
9.0
75
Accuracy (%6
Accuracy (%)
78
70
(Ours)
76
Transformer
12.0
74
65
13.0
Diff (Ours)
Transformer
60
70
2
3
7
2
5
8
Random Seed
Random Seed
(c) Banking-77 with 77 classes.
(d) Clinic- 150 with 150 classes.
Figure 11: Robustness evaluation of in-context learning on four datasets. Accuracy is evaluated with order permutations of demonstration examples by sweeping random seeds. The dash lines represent the margin between the best and worst results. Demonstration examples are randomly arranged the prompt.
19

---

Published as a conference paper at ICLR 2025
G GRADIENT FLOW OF DIFF TRANSFORMER
We show that the gradient flow in differential attention is similar to that of conventional softmax attention. With this property, the same hyperparameters used Transformer can be applied directly to the corresponding DIFF Transformer without concerns about training instability.
For differential attention, we select single head proof and expand Equation (1) and Equation (3) as follows. We have X E IR X as the input, Q1, Q2, K1, K2 IR V E IR x2d and E RN d as the output:
[Q:Q2]] (XWe) XW
[Ki K2] = |XWR Ki XW
V= XW
A softmax QiKi ) A2 - sofumax
Q2K2
(6)
Vd
0 = GrGrouNNorm(( A2)
where W W0,, W W B xd WV E & 2d is a learnable scalar, and GroupNorm has a fixed multiplier as scale: 7 = Amn- For token
WO R 2d are parameters,
(GN(x) O V2d ? Or
|
in (A1 À V2)V, we have
((1)
O Amit)) at the early
|
/2d
ÔL
training stage. With this formulation and given the gradient of O we formulate gradients of parameters as:
00
0L OL d0 awo DmeWo
((A1 - X V2) OL 00
AL
0O av
OW 80 OW
OL (WOT
=XT(A: - A2) 80
aL OL 80 0A, iQ1
awe A00 0Q1 OWQ
1
OL
AL (W))TT (A1 e (00 (
XT{A ① 80
Vd
AL OL 80 0A₂ 0Q2
awe: 80 800 0Q OWQ
OL W W
aL (A2 (o)TV)))J 30
vd [
00
OL OL 80 0A OK1
OWK, 800A, 0K, OWK.
1
XT(Ai @ 80
(WO (A1
OL
30 (
aL
Vd
aL ØL. 80 0A₂ 0K2
ØWK; 800 KK OWKz
- ~ X | (WD)TVT OA 30
OL
aL ( (
Vd
a0
where J E NxN is all-one matrix.
As comparison, we reformulate conventional softmax attention. For attention with 2d dimension,
have X RR X demati as the input, Q1, Q2, Ki, K₂ Nxx, V R and O R N dentel as
20

---

Published as a conference paper at ICLR 2025
the output:
[Q1: [XW xw [K] = | [ V= XWV
A softmax QiKi Q2K3
(8)
V2d
O= = (AV)W
where 1 WQ², WKi, W E Rdmeaidd WV E R Wo R2dxxoatt are parameters.
ØL
Denote the gradient of O as we formulate gradients of parameters via:
80
OL 0L 80
OWo
d0awo
(AV)T 0L
00
0L
OL00 Øv
8006 OWV
ØL XT (WO)T
00
0L
OL OA aQ)
aWQ.
800A 1Q1 OWQ.
1
XT 30
OL
(WO
OL (W)TVT))JJ
V2d
80
OL OL.00 0A aQ2
(9)
QWQ:
800 A 0Q2 @W
1
OL ( (WOTTTT (A -
OL
22d
00
00
OL
OL OA OK1
OWKi Ki
80 A 0K OWKI
1
0L XT 100 (WO (Ao W (
ØT
V21
00
0L
OL 0A OK2
OWK: K2 100AOK, OW
1
OL A
OL - (A (
(
W T 00
2d
00
With the property of softmax, we have A _ Therefore, the gradients of the corresponding parameters of attention and differential attention are equivalent in magnitude, differing by some constant factors, as shown in Equation (7) and Equation (9). When using optimizer that is invariant to gradient magnitude, such as Adam (Loshchilov & Hutter, 2019), parameter updates DIFF Transformer are similar to those of Transformer. This allows us reuse Transformer hyperparameters without risking training instability.
A1 A2 A1 XAg, considering gradient magnitude.
21


Reviews:
[{"review_id": "utzreit81w", "reviewer": "Reviewer_hbKx", "summary": "The paper aims to mitigate the problem in standard softmax attention to over-allocate attention weights to irrelevant context. The proposed Diff Transformer uses the difference between two separate softmax attention scores to cancel noise assigned to irrelevant contextual tokens. Concretely, Diff Transformer first partition the query and key vectors into two groups to compute separate softmax attention scores. The final attention score is the subtraction of the two groups. One learnable parameter $\\lambda$ is introduced to balance the two groups of attention scores.\n\nExperiments were conducted on a 3B-parameter model trained on 350B data tokens. Diff Transformer outperforms Transformer on training loss and downstream benchmarks. Ablation studies investigate the importance of the RMSNorm on each attention head.", "questions": "NA", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The proposed modification of attention in Diff Transformer is well-motivated\n\n2. The experimental results are strong, with large-scale experiments up to 3B-parameter models and 350B data tokens. \n\n3. The pre-trained model was evaluated on multiple benchmarks, and also on long-context evaluation, retrieval-oriented tasks, many-shot in-context learning and hallucination evaluation.\n\n4. The paper is well-written, easy to follow.", "weaknesses": "Some design motivation in the model is still not clear:\n\n1. Why the learnable $\\lambda$ is re-parameterized in Eq (2)?\n\n2. Why in Eq (3) there is a term $(1 - \\lambda_{init}$ for each head?\n\n3. Why the RMSNorm for each head is so important for Diff Transformer stability? The explainable in section 3.8 is unconvincing to me.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IQm76io61S", "reviewer": "Reviewer_QNSJ", "summary": "The paper introduces Differential Transformer, a novel architecture for LLMs that enhances attention to relevant context while canceling out noise. The core innovation is a differential attention mechanism that calculates attention scores as the difference between two softmax attention maps, promoting sparse attention patterns and reducing attention noise. The paper demonstrates through extensive experiments that the DIFF Transformer outperforms the standard Transformer in various aspects, including scaling model size, training tokens, long-context modeling, key information retrieval, hallucination mitigation, in-context learning, and reduction of activation outliers.", "questions": "1. Do both attention calculations in differential attention use RoPE positional encoding?\n2. Similar to multi-query attention, can $K_1$, $K_2$, or $Q_1$, $Q_2$ share parameters?\n3. Is there a significant difference in sparsity between DiffAttn and regular attention? Does the first term of DiffAttn resemble the pattern of standard attention (implying that the second negative attention term serves to cancel out noise)?\n4. What is the calculation formula for Table 3? (Specifically, what is the exact normalization operation?)\n5. Please discuss the relationship between differential attention and sparse attention.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. this is a solid and well-written paper. \n2. while the technique itself is not complex, as far as I know, this is the first work to propose a new architecture using differential attention. 3. the experiments and conclusions in this paper are thorough and insightful.", "weaknesses": "1. the authors mention \"promoting the emergence of sparse attention patterns\" multiple times on Lines 13, 160, and 539, but do not provide statistics and quantification of the sparsity of attention distribution between DIFF Transformer and the general Transformer.\n2. The authors did not discuss or conduct experimental comparisons with work related to the sparse attention. \nFor example: \n> Efficient Content-Based Sparse Attention with Routing Transformers\n>\n> Generating Long Sequences with Sparse Transformers.\n3. It would be even better if the effectiveness of the DIFF Transformer could be validated on image or speech modalities.\n4. The absence of a *Related Work* section limits the paper's ability to contextualize its contributions within the broader field.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i1FyRpD0Sy", "reviewer": "Reviewer_cGmd", "summary": "This paper proposes a new model architecture called Differential Transformer (DIFF Transformer), which reduces noise and more accurately focuses on relevant context through a differential attention mechanism. In a series of language modeling experiments, DIFF Transformer outperforms the standard Transformer across various tasks and model sizes. The paper demonstrates the superiority of this method in long-context modeling, key information retrieval, hallucination mitigation, in-context learning, and reduction of activation outliers.", "questions": "I don't have questions.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The proposed DIFF Transformer introduces an innovative differential attention mechanism that reduces attention noise through the difference between two independent softmax mappings. This approach performs exceptionally well in long-context processing, making a significant improvement for long-text tasks.\n2.DIFF Transformer demonstrates strong performance in language modeling, in-context learning, and multi-needle retrieval tasks, particularly outperforming traditional Transformers in long-context scenarios. It greatly enhances the accuracy and robustness in practical applications like question answering and text summarization, showing broad potential for various use cases.\n3.The paper provides extensive experimental validation, confirming DIFF Transformer’s stability across different tasks and model parameter settings. The hyperparameter tuning is thorough, ensuring both robust performance and reliable results.\n4.The paper is well-organized, with clear illustrations and analogies that present the differential attention mechanism effectively. Detailed experimental steps make it easy to understand the innovative contributions.", "weaknesses": "While the differential attention mechanism brings notable performance improvements, it also adds computational complexity. Efficiency tests indicate that DIFF Transformer’s throughput, particularly with multi-head normalization and dual softmax calculations, is slightly lower than that of traditional Transformers (by about 5%-12%). Although this impact is relatively minor in the experiments, further exploration of computational efficiency would be valuable if scaling to large-scale applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BMOkEH35gE", "reviewer": "Reviewer_txCS", "summary": "The paper introduces a new Transformer variant, called Diff Transformer. The key idea is in the design of attention mechanism, where Diff Transformer uses two softmax attention functions to cancel out potential attention noises. Through a suite of empirical studies, Diff Transformer show promising performances compared to standard Transformer architecture.", "questions": "- Can the authors elaborate further on why we need a fixed multiplier $(1 − \\lambda_{init}) $ in Diff Transformer (Eq. 3)?\n- In section 3.7, Is absmax quantization the suitable approach to adopt in this context? Why not consider quantization approaches that are robust to activation outliers, such as LLM.int8()?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a simple architecture tweak to mitigate the observations that Transformer models tend to incorrectly allocate excessive attentions to irrelevant contexts.\n- The paper demonstrates promising empirical results of the proposed architecture through a decent suite of evaluations, ranging from language modeling capability that fits well with scaling law, in-context learning, to improving long-context capability and mitigating contextual hallucination.\n- I appreciate that the authors include a variety of downstream tasks to showcase how addressing attention noises in Transformer models can lead to performance improvements.", "weaknesses": "- Related work can be discussed more thoroughly in the paper. For example, [1] discusses that Transformer models tend to mis-allocate attention to irrelevant contexts potentially biased by their position within the context [1], and [2] also shows LLMs can be easily distracted by irrelevant contexts.\n- While the proposed Diff Transformer shows promising performances, they have to be trained from scratch which can be computationally expensive compared to post-training approaches that mitigate attention noises in standard Transformer models, such as the calibration technique used in [1], and improved decoding approaches used in [3] and [4]. Currently, vanilla Transformer is the only baseline considered in the paper. However, I would appreciate if the authors can show how Diff Transformer compares to other more lightweight techniques that fix Transformer's attention noises.\n- Apart from pretraineing Diff Transformer, there is no further instruction tuning experiments included in the current paper. I would be interested in seeing whether instruction tuned Diff Transformer also shows better performances than standard instruction tuned Transformers.\n\n\n[1] Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization. Hsieh et al. 2024.\n\n[2] Large Language Models Can Be Easily Distracted by Irrelevant Context. Shi et al. 2023.\n\n[3] Trusting Your Evidence: Hallucinate Less with Context-aware Decoding. Shi et al. 2023.\n\n[4] Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Maps. Chuang et al. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)