Understanding Differential Transformer Unchains Pretrained Self-Attentions
Authors: Chaerin Kong
Jiho Jang
Nojun Kwak
Venue: NeurIPS 2025 poster ()
Keywords: Attention
Transformer
Language Models
Primary area: deep_learning
TL;DR: We analyze the reasons behind Differential Transformer's success, based on which we propose an efficient adaptation method to enhance pretrained LLMs.

Abstract: Differential Transformer has recently gained significant attention for its impressive empirical performance, often attributed to its ability to perform noise canceled attention. However, precisely how differential attention achieves its empirical benefits remains poorly understood. Moreover, Differential Transformer architecture demands large-scale training from scratch, hindering utilization of open pretrained weights. In this work, we conduct an in-depth investigation of Differential Transformer, uncovering three key factors behind its success: (1) enhanced expressivity via negative attention, (2) reduced redundancy among attention heads, and (3) improved learning dynamics. Based on these findings, we propose DEX, a novel method to efficiently integrate the advantages of differential attention into pretrained language models. By reusing the softmax attention scores and adding a lightweight differential operation on the output value matrix, DEX effectively incorporates the key advantages of differential attention while remaining lightweight in both training and inference. Evaluations confirm that DEX substantially improves the pretrained LLMs across diverse benchmarks, achieving significant performance gains with minimal adaptation data (< 0.01%).

Understanding Differential Transformer Unchains Pretrained Self-Attentions
Chaerin Kongl.2-
Jiho Jang2*
Nojun Kwak?
TwelveLabs
Scoul National University
chaerin k. kong@gmail com
Abstract
Differential Transformer has recently gained significant attention for its impressive empirical performance, often attributed its ability to perform noise canceled attention. However, precisely how differential attention achieves its empirical benefits remains poorly understood. Moreover, Differential Transformer archi- tecture demands large-scale training from scratch, hindering utilization of open pretrained weights. this work, we conduct an in-depth investigation of Differ- ential Transformer, uncovering three key factors behind its success: (I) enhanced expressivity via negative attention, (2) reduced redundancy among attention heads, and (3) improved learning dynamies. Based these findings, we propose DEX, novel method to efficiently integrate the advantages of differential attention into pretrained language models. By reusing the softmax attention scores and adding lightweight differential operation on the output value matrix, DEX effectively incor- porates the key advantages of differential attention while remaining lightweight in both training and inference. Evaluations confirm that DEX substantially improves the pretrained LLMs across diverse benchmarks, achieving significant performance gains with minimal adaptation data (< 0.01%)
1 Introduction
Transformer- based architectures have emerged as the cornerstone of modern deep learning across multiple domains 774 22,
41 33). With their attention mechanism, transformers effectively model oon-range dependencies, leading significant advances large
65
1
67 34 40
[1 20
language models 73 48
70
i 9]. However, a growing body of work [39 50 52 highlights that
these language models struggle with key information retrieval due to inherent attention noise.
To address this issue. Differential (DIFF) Transformer 85 introduces differential attention that computes the difference between two attention scores, thereby boosting attention on key tokens while suppressing common noise. Although its strong empirical performance has established it as promising alternative to standard transformers. how this simple architecture consistently harnesses the differential operation for effective noise cancellation without explicit guidance remains elusive. Moreover, due the gap in architecture, employing DIFF attention requires training from scratch, which prohibits utilization of open pretrained weights [73 70
48
59
27
and incurs huge cost.
In this paper, aim fill this gap by providing an in-depth analysis of the mechanisms of DIFF Transformer and presenting a method to efficiently integrate its benefits into existing pretrained transformers. Our key observations are threefold. (1) DIFF attention enhances expressivity through negative attention scores. (2) DIFF attention reduces redundancy among its attention heads. (3) DIFF Transformer exhibits improved learning dynamics.
Building on these insights, we present DEX (Differential Extension), an efficient framework that injects the strengths of DIFF Transformer into pretrained LLM without training from scratch.
Equal contributions.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

02-
02-
3.5 103 th0
160
33
55
1d m3 153
23
13
(a) Rank Correlation
(h) Pearson's Correlation
(c) JS Divergence
(d) Cosine Distance
Figure 1: Attention score comparison between the two groups in DIFF attention. Top 5% refers top-5% tokens with highest attention score in each sequence. It clearly shows that the overlap between the two attention scores is much greater in non-salient tokens.
Unlike most finetuning methods that fit the model to downstream data, DEX is an architectural adaptation strategy that introduces key mechanism from a different architecture pretrained model, conceptually similar to MHA2MLA [8) Specifically, DEX operates by reusing the pretrained softmax attention scores (softmax QK") and applying its learnable differential mechanism to the output value matrix (softmax QK" V), making the adaptation lightweight (in both training and inference) yet effective, as demonstrated empirically. To facilitate stable and performant transition, introduce additional techniques for head selection and X-annealing, which controls the critical balance between original knowledge and incoming architectural changes. We validate DEX across multiple model families (Llama-3 [26) and Qwen-2.5 [84]) and scales (0.5B-8B), using diverse benchmarks such as language modeling [25] 77 79), key information retrieval [39] and in-context
learning (6]. DEX consistently achieves significant gains using less than 0.01 % the size of the original training data (<IB tokens), without incurring nontrivial test-time overhead.
How Does Differential Transformer Work?
In this section, systematically analyze the internal mechanics of DIFF Transformer. Since the original weights are not publicly available at the time of writing, we train a DIFF Transformer on a similar data mix to carry out our analyses. Please refer to Appendix E.2 for full details.
2.1 Preliminary: DIFF Transformer
The key innovation of DIFF Transformer is replacing the softmax attentions with DIFF attentions. DIFF attention introduces mechanism designed to suppress attention noise by computing the difference between attention scores from two separate groups. Given an input sequence X e R.N xunad is first projected into queries, keys, and values as follows:
[Q Q2) XWQ, [K1; XWK: V
(1)
where Q1, Q2, KI, K2 E Nxd and V E N denote projected matrices, and Wo, WK, Wy Ramas 2d are learnable parameters. The differential attention is then computed as:
(Q.Kl - - softmax Vd
Q2222 Vd
DiffAttm (X) (sottmxx
(2)
where À is a learnable scalar. This differential mechanism enhances robustness by canceling common- mode attention noise, similar in spirit to differential amplifiers. We note that despite DIFF attention having the same number of parameters, it exhibits significantly higher compute cost and peak memory usage in practice due to enlarged dimensions (see Fig 12). Refer to the original paper [85 for details.
Higher Expressivity via Negative Attentions
The empirical success of DIFF Transformer is often attributed to its noise-canceling effect, achieved through subtraction between attention groups. Such noise cancellation is commonly hypothesized to enhance performance by inducing sparsity [85), concentrating attention on relevant context while suppressing irrelevant information. We investigate whether DIFF attention operates primarily through
this lens of conventional sparsity 72 30).

---

GLAMA
ULANA
02-
02
(a) Sparsity ratio (c=le-4) (b) Sparsity ratio (e=le-6) (c) Attention ent. (abs) (d) Attention ent. (group)
Figure 2: (a), (b): ratio of attention scores whose absolute value is smaller than Except for the bottom layers, DIFF Transformer displays lower sparsity ratio. (c), (d): Attention score entropy. Entropy in (c) measures magnitude concentration, calculated renormalized absolute values of the final differential attention scores. Group refers to the two separate attentions in DIFF.
Our analysis of DIFF attention's 's dual attention groups (Fig indeed indicates form of selective filtering. Metrics such as correlations, the groups' attention scores (computed pairwise between corresponding heads) reveal high overall similarity (blue) but notably weaker correspondence for the most salient tokens (green). This points to selective cancellation where shared, less critical attention patterns are offset by the subtraction, while distinct signals for key tokens are largely preserved or emphasized. However, this observed filtering does not directly translate to increased sparsity in its traditional definition (i.e., having many close-to-zero values). In fact, Fig aaa a) and (b) show that DIFF attention often exhibits lower sparsity ratios, while Fig c and (d) reveal higher entropy values, both indicative of lower sparsity when compared to standard softmax attention.
divergence [47], and cosine distance between
This suggests that DIFF attention's 's noise canceling embodies more nuanced mechanism than simply zeroing out non-salient contexts. As Fig shows, DIFF attention assigns negative scores to substan- tial fraction of context tokens, whose relative atten- tion magnitude generally increases in higher layers. Hence, DIFF attention does not uniformly zero out irrelevant contexts, but is capable of flexibly contex- tualizing them using these signed scores. As [53 shows, employing negative attention to explicitly model negative relevance in the query-ke (QK) cir- cuit provides greater ficxibility to the output-vali value (OV) matrix, reducing its need for implicit informa- tion filtering and thereby fostering more expressive representations. Qualitative examples in Fig 4 such as Nlown-weighte irrelevant subject in Indirect Ob- ject Identification task [80] non-literal interpre- tation in sarcasm detection, illustrate how DIFF at- tention can achieve more refined information flow using negative attention (green boxes). This con- trasts with standard softmax attention that assigns high scores even to these highly irrelevant contexts, whose -igennssnsittiitty often burdens its OV matrix with implicit information filtering [53] (Additional examples are in Appendix B.5).
(a) Proportion
(b) Magnitude
Figure 3 The proportion and average magnitude of
positive/negative attention scores.
DIFFT
LLMMA
LLaMA
Figure 4: Attention scores on Indirect Object Iden- tification (IOI, top two) and sarcastic expression (bottom two). Blue indicates negative and red repre- sents positive. Green boxes highlight the difference.
2.3 Reduced Redundancy among Attention Heads
57 7 Our analysis reveals that DIFF attention significantly reduces redundancy among attention heads. Fig presents cosine distance between per-head attention scores (higher distance relates to lower redundancy) and Centered Kernel Alignment [6T] between value-projected head features (higher alignment translates to higher redundancy). The plots clearly indicate that DIFF attention exhibits reduced redundancy at both the
Multi-head self attention is powerful but can be redundant 75
21
45 83
3

---

LLAMA
(a) Cosine distance between attention scores.
(b) CKA between attention head features.
Figure 5: (Left) Pairwise cosine distance between per-head attention scores (flattened across layers) Brighter indicates larger distance, hence less redundancy. (Right) CKA [61 between per-head features. Brighter means higher alignment, hence higher redundancy. See Appendix B.2
attention score (left) and feature (right) levels. One might attribute this DIFF having fewer effective heads. However, our experiments demonstrate that merely employing fewer, wider attention heads does not alleviate redundancy (see Appendix [B.2]. We hypothesize that the differential mechanism grants greater flexibility in controlling attention patterns, reducing inter-head redundancy.
Examining attention head importance provides further in- sights into head utilization. Figgooddmoostrteee the head importances [60 15], normalized by the maximum value and sorted. In DIFF Transformer, importance is distributed more uniformly across attention heads (Fig o blue), indi- cating that each head contributes more evenly to the final representation. Combined with the reduced redundancy, this balanced contribution allows DIFF attention to cap- ture a broader spectrum of diverse features compared conventional multi-head attention.
Figure 6: Layerwise head importance
distributions, normalized and sorted.
2.4 Improved Learning Dynamics
DIFF attention introduces novel components, including the differential operation and learnable parameter X. To understand their impact learning dynamics, we analyze the Hessian maximum
eigenvalue spectra (F following the procedure of [64] As discussed in [64] high prevalence
of negative eigenvalues indicates non-convexity in the loss landscape, which can hinder training,
particularly during early phases [63] 19 36 35). We observe significantly fewer negative eigenvalues
for DIFF Transformer compared to the standard transformer, suggesting improved optimization dynamics. Notably, this benefit is largely lost when the learnable À is removed (green line in Fig 7
Training statistics further corroborate this finding. & the language modeling loss and gradient norms for the standard and DIFF transformer. While DIFF consistently achieves lower loss and more stable grad norms, removing the learnable notably impairs optimization. We hypothesize that the learnalle À plays a key role in stabilizing training dynamics, especially during the early stages.
(a) Loss curve.
(b) Gradient norm.
Figure 7: Hessian max eigenvalue spectra. While transformer and DIFF without learnable À (DIFFT-Lam) shows a number of negative eigen- values, DIFF has much less.
Figure 8: Loss and gradient norm. DIFF shows the best dynamic while DIFFT-Lam, DIFF with non-learaable X, shows instability.

---

Out Projection
def Attn W_q, W_, K.,.I.D,,,,d
# standard softmax attention
Corcat
Q, K, eqrt
X W_q, w_k, X W_v
8
Impiiot Adaptation
A Q K. transspooe -2) ***
0 softmax (A) 0
0 A))n(()
implicit differential adaptation
fo
0 Af_D(D) do else return
def MHA (X, W_q, W_k N.v,f_D. N,, [Attn (x,,,,,,,,,..... do- hs)) for i in range (h)] hs selected heads return Concat (0) e W_o
Softmax Attention
We
W
W,
Figure 9: Differential Extension (DEX). The output value matrix O is transformed by subtracting X-modulated projection from itself. This operation targets layer-specific subset of attention heads.
3 Differential Extension
Based on the insights from See we present DEX, a framework that integrates differential mechanism into pretrained self-attentions. In designing DEX, we identify three primary desiderata: (1) effectively integrating the beneficial properties of DIFF Transformer; (2) ensuring a lightweight transition by maximally preserving and leveraging the pretrained knowledge; and (3) minimizing test-time computational or memory overhead. In the following subsections, we describe each component of our framework in detail, explicitly connecting the lessons learned from Sec 2 to satisfy these desiderata.
3.1 Implicit Differential Adaptation
Our analysis (Sec 2.2) suggests that DIFF attention's ability model negative relevance in its QK circuit enhances representational power by facilitating more nuanced information processing in the OV matrix. While this achieved in DIFF attention by explicitly subtracting two attention scores, naively retrofitting such existing heads into two groups risks significant knowledge loss and instability; duplicating them incurs prohibitive computational and parameter overhead. With DEX, we aim to achieve similar enhancements information processing, but stably and efficiently.
structure onto pretrained models can be problematic. Splitting
DEX introduces its learnable differential mechanism directly to the attention outpul instead of the query-key (QK) circuit, an approach term implicit adaptation. This strategy is motivated by the reusability of pretrained attention magnitude signals, supported by our empirical findings that the absolute scores of DIFF attention often mirror standard softmax scores (Fig Appendix [B.I). By targeting the OV matrix, which is known to control information flow and perform implicit filtering (Sec
53 23 DEX empowers the pretrained attention with improved processing of standard
attention patterns.
Formally, our implicit differential adaptation is defined follows:
QK vd
0 softmax
V.
0'=0- Af)
where fo denotes a learnable projection parameterized by Wp and is learnable sealar. This design offers several notable advantages, including lightweight adaptation through effective knowledge reuse, minimal parameter and test-time compute overhead, and high compatibility with existing transformers. We empirically demonstrate that despite being implicit, DEX effectively delivers the empirical strengths of differential attention.
3.2 Selective Adaptation
Attention heads in standard multi-head attention can be highly redundant, and their contribution to the final representation is seldom equal 775 (7). Further motivated by our findings on effective
head utilization in differential attention (Sec 2.3 propose to leverage this inherent heterogeneity
5

---

via selective adaptation, applying the implicit adaptation (Eq. B) only to subset of heads within each layer. typically targeting those identified as less critical. This selective approach enhances underutilized heads while preserving critical ones, thereby improving overall representational capacity and safeguarding vital pretrained knowledge. We introduce two data-driven head selection strategies:
Low-Importance Head Selection. The first method selects heads based on low representational importance, following headwise importance criteria established in [60] [5]. We compute importance scores and apply differential adaptation to the top-k heads with the lowest scores in each layer.
High-Entropy Head Selection. The second strategy targets attention heads with high entropy, state often associated with weaker representational focus, reduced functional specialization, or poten- tial under-utilization [89] 33 555 43]. Similarly, we select and adapt the top-k heads demonstrating the highest entropy within each layer.
3.3 Balancing Adaptation with Pretrained Knowledge via X-Annealing
Our analysis in See 2.4 reveals that adaptive modulation of the differential mechanism is critical for stable optimization. In our scenario, maintaining a careful balance between pretrained knowledge and newly introduced architectural modifications is crucial. Zero-initializing, the learnable À would be a typical way to safely introduce DEX the model to adopt the differential mechanism, À could remain near zero if the pretrained model is already strong. To facilitate a stable and effective transition, we propose a scheduled annealing of A:
B2 88 28 but that alone does not sufficiently encourage
t (
a min 1, 6
A(t) = (1 aa)
Ainit
+ a
where t is the current training step, T is the annealing duration, Ainit is a constant, and Ateam is learnable parameter initialized around zero. This schedule initiates A(t) with zero for stability, uses annealed Aimi to provide gradual learning signal for the differential mechanism (e.g.. W Wp) when f < T, and transitions control the learnable Aleam for optimal adaptation as T.
3.4
Overall Framework
The complete formulation of DEX for given head h is expressed as follows:
QK
O = softmax
O'= =-)-(())(( E HO)D(O),
(5)
where H is the set of heads selected for differential adaptation, and O' is concatenated across all heads and passed into the output projection. During training, we update Wk. Wv. and Wo along with WD and Atearn within self-attention, keeping all other parameters (e.g.. FFN) frozen. This targeted update strategy provides the necessary flexibility to integrate DEX into standard transformers, while keeping the training lightweight, especially under standard GQA 2 setting.
Experiments
DIFF Transformer has demonstrated strong performance across wide variety of tasks, including general language modeling, key information retrieval, and in-context learning. We quantitatively validate the effectiveness of DEX in integrating these strengths into pretrained LLMs. We conduct ablation experiments and analyses to further verify our design choices.
4.1 Language Modeling Evaluation
Setup We apply DEX to Llama-3. 1-8B 26 Llama- 56 and
84
As the original pretraining data for these models is unavailable, we build custom corpus of web pages, papers, encyclopedias, and code from open datasets 44 82), similar to OLMo 62). This corpus contains 887M tokens (Llama-3 tokenizer), less than 0.01% % the models' original pretraining data size. Although DEX is not presented as fine-tuning method, we compare against baselines trained on the same data inta-innluding parameter-efficien tuning (PEFT; LoRA (32), PiSSA (44) and
full fine-tuning (FT; Galore (90), APOLLO (93) to control for the influence of this corpus. Direct

---

Table 1: Language modeling benchmark scores across model variants and training methods. Green indicates improvement over the baseline, while gray indicates a decrease.
Model
Arc-C
Arc-E
BoolQ
COPA
Hellaswag
MNLI
OBQA
PIQA
WTC
Winogrande
WSC
AVG
A
Liama-3B
46.3
71.7
73.1
85.0
73.6
35.0
43.2
77.5
49.8
69.1
37.5
60,2
LoRA (r=8)
43.4
70.2
75.3
82.0
74.2
54.5
43.0
77.1
53.8
70.1
36.5
61.8
+1,6
(r-32)
72.0
76.2
83.0
74.7
46.7
43.2
77.7
55.2
70.0
61.3
+1.5
PSSSA
45.4
73.8
74.1
82.0
74.3
46.6
42.4
78.3
56.1
69.9
38.5
61.9
+1.7
FT
45.7
73.7
73.8
84.0
74.7
38.5
41.4
78.0
55.3
70.7
40.4
61.5
+1.3
GaLore
46.1
74.9
76.2
87.0
74.1
33.1
42.6
77.9
53.0
70.2
38.5
61.2
+1.0
APOLLO
45.8
74.4
73.5
84.0
74.7
35.0
42.8
77.5
56.1
70.2
45.2
61.7
+1.5
Ours
45.5
73.3
74.8
84.0
74.1
49.5
42.6
78.2
51.9
69.1
63.5
64.2
+4.0
Llama-I8
36.3
60.6
63.4
77.0
63.6
36.0
37.2
74.5
48.6
59.9
42.3
54.5
LoRA (1=8)
34.6
63.3
46.4
78.0
64.1
32.9
36.6
75.1
47.9
60.9
40.4
52.7
-1.8
LoRA (r=32)
35.9
65.4
61.5
78.0
64.4
32.6
38.2
75.1
48.7
60.3
37.5
54.3
-0.2
36.3
65.2
79.0
64.2
33.1
37.2
49.7
54.4
FT
65.5
60.7
76.0
64.5
38.2
49.2
54.9
+0.4
GaLore
36.3
65.7
60.2
77.0
64.2
34.4
37.6
75.2
50.6
60.7
36.5
34.4
-0.1
APOLLO
37.1
65.0
58.1
27.0
64.4
37.5
36.8
74.9
51.6
60.4
36.5
54.5
+0.0
Ours
35.2
64.2
57.8
79.0
64.0
38.0
51,9
60.6
48.1
55.6
Qwen-1.58
72.2
72.8
83.0
67.8
53.0
63.5
62.2
LoRA (r=8)
70.3
67.5
39.2
53.3
61.0
LaRA (r-32)
43.4
70.2
71.0
85.0
67.5
50.7
39.2
75.5
52.0
64.7
47.1
60.6
-1.6
PISSA
44.3
70.1
72.6
84.0
66.7
47.5
40.0
74.3
54.7
63.9
52.9
61.0
-1.2
FT
43.9
71.9
68.7
84.0
67.6
51.5
40.2
75.7
53.6
64.5
48.1
60.9
-1.3
GaL.rr
72.7
72.0
67.4
39.6
53.1
61.1
APOLLO
45.1
73.4
72.4
83.0
67.7
50.1
39.4
75.7
53.9
64.8
43.3
60.8
-1.4
Ours
45.3
74.1
70.1
84.0
67.8
50.2
40.8
76.4
53.3
63.2
61.6
62.4
+0.2
Qwen-0.58
31.8
58.7
62.3
74.0
52.2
38.3
35.4
69.9
49.2
56.2
41.3
51.8
LoRA (r=8)
34.3
66.1
57.2
74.0
52.3
33.9
33.6
69.4
50.0
56.2
43.3
51.8
+0.0
LoRA (r-32)
31.4
63.9
60.6
23.0
52.1
39.1
34.4
69.7
49.2
58.6
36.5
51.6
-0.2
PISSA
34.6
51.7
33.4
50.2
51.3
FT
65.6
74.0
52.3
34.0
50.8
Gal.ore
35.2
65.3
58.2
24.0
52.2
34.4
33.6
70.2
49.7
56.4
36.5
51.4
-0.4
APOLLO
35.3
65.7
58.0
72.0
52.3
34.7
34.0
70.2
50.3
$7.0
36.5
51.5
-0.3
Ours
34.8
65.2
56.5
73.0
52.3
40.1
35.4
70.1
51.6
57.6
61.5
54.4
+2.6
Liama-8B
53.6
81.1
82.1
87.0
79.0
49.7
45.0
81.3
51.9
73.3
59.6
67.6
LoRA (r=8)
53.1
79.5
78.3
89.0
80.4
62.1
44.8
80.5
57.8
74.9
54.8
68.7
+1.1
FT
52.3
80.2
80.5
91.0
80.6
60.8
45.6
81.1
58.8
73.7
57.7
69.3
+1.7
Ours
52.1
79.5
79.6
91.0
80.4
58.6
46.4
80.5
58.3
75.2
64.4
69.6
+2.0
comparison with original DIFF Transformer is limited by unavailable pretrained weights, and we defer evaluations in smaller settings to Appendix along with other details. For head selection we simply set k to be half the total number of heads for each model, and we adopt the Ainit from (85) (we provide ablations in Appendix B.3) For PiSSA we report 32 case as this yields good results.
Results We report performances on 11 widely used language modeling benchmarks 16 78
76
86
58 8 (68) using (25). As shown in Table DEX achieves significant improvements across model sizes and families. Given the discrepancy between our training corpus, original pretraining data and the downstream tasks, it is natural to observe degradation after additional training in some cases. Nevertheless, DEX demonstrates robust performance gains on the majority of benchmarks, even when other methods- all trained on the same corpus- exhibit performance drops. In particular, attribute DEX's 's strong performance on WSC to its enhanced anaphora resolution granted by the capacity to model negative relevance for incorrect antecedents, which aligns with our intuitions. Notably, although DEX only updates self-attentions, it consistently outperforms both PEFT and full fine-tuning even when full tuning steadily outperforms PEFT (e.g., Llama-IB).
4.2 Key Information Retrieval
Needle-in- -ana--ayyttcck test [39 widely adopted to assess LLM's ability to identify critical informa- tion embedded in an extensive context. Following the multi-needle retrieval setting of 49 69, 85), we place the needle at five distinct depths within the context: 0%, 25%, 50%, 75%, and 100%, accompanied by distracting needles. We note the total number of needles placed in the context as N, and the number of target needles actually being queried as R. Each combination of depth and context length is assessed using 20 samples.
7

---

Mutti ncedle eval R-1)
Multi nnedlic Retricval R-1)
231
613
633
Ei1t
EAD
360
2333
EAAt
2.0
275
140
120
200
60
69
84%
625
435
605
130
3.%
600
643
645
608
150
405
875
100
LO00UE Lengeh
C LengT)
(a) LLAMA
(b) LLAMA DEX
Figure 10: Multi-needle retrieval results. N: total number of needles, R: number of queries.
Fig 10 shows the result for N = 8, R case. DEX significantly enhances the retrieval performance of the base Llama-3B model across all context lengths and embedding depths. Notably, DEX improves the average accuracy score by 11.4% (absolute increase from 66.9% to 78.3%), highlighting its effectiveness in improving key information retrieval capabilities.
Increased attention to answer ratio Table further demonstrates that DEX effectively transfers the core capability of DIFF attention: attention noise cancel- ing. Despite being implicitly applied to the output value matrix, DEX notably alters the effective atten- tion pattern to focus relevant information. This result empirically supports our design choice, plac- ing DEX on the sweetspot between efficiency and efficacy. We provide details in Appendix E.3
Table Effective attention scores allocated the answer spans inserted at different depths in key information retrieval.
Attention to Answer
1
Model
0%
25%
50%
75%
100%
Avg
Llama DEX
0.06
0.04
0.06
0.05
0.08
0.06
0.21
0.13
0.18
0.16
0.27
0.19
4.3 In-Context Learning
Table 3: In-context learning performance.
DIFF Transformer notably enhances in-context leam- ing performance compared to standard transformer models. To validate whether DEX achieve similar improvements, we conduct a comprehensive evalua- tion using three established benchmarks: TREC (31],
N-shot
Dataset
10
100
500
1000
2000
Avg
TREC
Banking-77 12), and Clinic-150)
421 Following
Llama
20.0 71.1
88.9
93.3
88.9
93.3
75.9
the setup of 6 adopt random selection proce- dure for N-shot examples, as retrieval-based scores quickly saturate with state-of-the-as LLMs.
LoRA
20.0
68.9
93.3
93.3
91.1
93.3
76.7
16.0
76.0
92.4
91.1
93.3
75.8
26.7
84.4
93.3
88.9
93.3
78.9
Banking-77
The results summarized in Table clearly illustrate that DEX consistently delivers performance gains across all evaluated benchmarks compared to both the base Llama model and fine-tuning baselines (LoRA and FT). DEX achieves the highest average accu- racy varying N-shot settings, demonstrating its robustness and efficacy in enhancing the in-context learning capabilities of pretrained models.
Llama
24.4
35.6
55.6
86.7
91.1
91.1
64.1
LoRA
26.7
40.0
88.9
88.9
91.1
64.8
21.6
34.4
56.0
84.4
88.8
92.4
62.9
DEX
22.2
37.8
60.0
91.1
95.6
95.6
67.0
Clinic-150
Llama
15.6
44.4
60.0
82.2
95.6
95.6
65.6
LoRA
22.2
42.2
57.8
82.2
95.6
95.6
65.9
FT
22.2
42.2
60.0
82.2
93.3
95.6
65.9
DEX
22.2
40.0
57.8
82.2
97.8
97.8
66.3
4.4 Application to Instruction Tuning
We investigate whether DEX can likewise enhance performance on instruction-followinn tasks. To fairly assess the effect of DEX, adopt two complementary settings. First, apply DEX a publicly available instruction-tuned checkpoint trained on an open-source instruction corpus OpenHermes-2.55 771 using the same training data (OH-2.5). This continued instruction tuning setting climinates the confounding effect of training data and lets us verify whether DEX improves
3.1

---

Table Instruction-tuning results on 8 benchmarks. The top four rows correspond the first setting, while the bottom two rows correspond to the second.
Model
MMLU
Are-C
IFEval
MBPP++
GSM8K
AGIEval
HumanEval
Math500
AVG
A
Instructiontuned
78.3
68.3
13.4
+ LLRRA
63.1
79.5
45.7
65.3
70.3
40.6
47.0
4.0
51.9
-0.3
+ +FT
63.0
78.6
49.2
63.2
68.8
42.3
36.6
20.0
53.7
+1.5
DEX
63.1
77.7
57.2
64.8
74.3
40.7
47.6
19.2
55.6
+3,4
Pretrained
LoRA
63.7
70.5
42.0
65.3
57.4
35.4
45.7
2.0
47.8
44
DEX
63.6
77.3
$1.0
66.1
68.4
37.9
50.7
16.2
53.9
+1.7
the performance of an existing instruct model. Second, we directly apply DEX to a base pretrained model an instruction-tuning method itself, similarly using OH-2.5 but in single stage. We examine if DEX can effectively induce instruction-folloviie tuning. Note that we include FT (further fine-tuning the open-source checkpoint on the same OpenHermes data for more steps) as an additional baseline to alleviate the concern for underfitting, which clearly distinguishes the contribution of DEX from the benefit of more training steps.
capabilities without prior end-to-end instruction
Table reports results on eight representative benchmarks that span language understanding 29 commonsense reasoning [16]. instruction following [92], math [17] 29]. code generation 4 and general human task 191. We observe that DEX delivers favorable results on diverse settings, significantly outperforming baselines benchmarks like GSM8K, HumanEyal and IFEval. When directly applied to a base pretrained model, DEX achieves notably higher performance than LoRA, demonstrating comparable performance to more heavily tuned baselines (top 3 rows) without any end-to-end SFT. These results indicate DEX's s effectiveness in inducing and reinforcing instruction- following capabilities efficiently.
4.5 Ablation and Analysis
We conduct ablation experiments using Llama-3B model. We mainly focus on two critical compo- nents: head selection strategies and learnable lambda annealing. We report the average score for the 11 language modeling benchmarks (similar to Table J. Appendix B presents full results.
From Table 5 it is evident that incorporating entropy-based head selection combined with both learnable and annealed lambda methods yields the best performance, achieving the overall accu- racy of 64.2%. Removing either component from lambda leads to noticeable performance drops, in- dicating the necessity of both. Additionally, both head selection strategies outperform the config- uration without head selection, with the entropy- based strategy pushing the boundary further. The fact that choosing low entropy heads (L) under- performs further supports our design. These findings underline the complementary roles of the head
Table Ablation with head selection and lambda control strategies. imp. refers importance- based and ent. stands for entropy-based.
Model
Head Selection
A-learned A-amnealed
LM Ace (%)
DEX
all
61.9
DEX
imp.
/
/
63.9
Dex
DEX
eat. (1)
64,2
DEX
ent.
x 63.8
ent.
DEX ent.
x
62.4
selection and lambda annealing mechanisms maximizing the effectiveness DEX.
We also analyze the inner working mechanism DEX. First, we investigate how DEX modifies the original attention output via the subtracted term A AfD(O). Fig. Iaa plots cosine similarity (indicating modification direction, c.g., positive for suppression) against relative norm (modification magnitude). We observe that while the similarity distribution suggests DEX'S capacity to both reinforce and suppress features, heads exhibit distinct patterns, with some focusing on amplification (higher norm for negative cosine, left) and others on attenuation (higher norm for positive cosine, right). Second, CKA on head output features reveals that DEX notably reduces inter-head redundancy lbb), implying more diverse head specialization. Lastly, we monitor À during training Fig [ig Simply zero-initializing the learnable À (red) completely fails to introduce DEX, while removing annealing (green) results in instability at the initial phase. Our approach (blue) smoothly introduces DEX with minimal damage to the pretrained knowledge. Refer to Appendix for full results.

---

ILAMA
(a) Jointplot between (O).
(b) Head CKA.
(c) during training.
Figure 11: Analysis on DEX. (a) Cosine similarity (cosine( O, A)) vs. Relative Norm (|A||/|||))
where A = (O) (b) CKA of attention head output features (brighter means higher redundancy). (c) Training dynamics of learnable À under different mittalizaation/amnallng schemes.
We verify the test-time efficiency of DEX by comparing the throughput (tokens per second) and latency with base Llama and DIFF Transformer (3B). Fig clearly shows that both terms of throughput and latency, DEX demonstrates superior inference time efficiency. While DIFF Transformer exhibits increasing inefficiency with longer context due to its compute-heavy attention operation, DEX remains competitive the original Llama baseline thanks to its lightweight design.
25000
LLAMA DEX
Loma
20000
DIFF
Latency lms!
15000
DEX
10000
5000
5000
16
15c
10h
30C
64k
26
42
11c
1.5c
32k
64
Se geggerce Gength
Sequuence Length
(a) Throughput comparison.
(b) Latency Comparison.
Figure 12: Inference time efficiency analysis. We benchmark (a) throughput and (b) latency of three attention variants. While DIFF attention costs significantly more compute at test-time compared to the original Llama attention, DEX incurs negligible overhead thanks to its lightweight design. All benchmarks are measured on single Nvidia A100 GPU.
Finally, we evaluate the effect of training data size on the application of DEX (Fig. [3). Notable gains appear with
just 400M tokens, highlighting the lightweight nature of
DEX. Since training data lacks direct correlation with the downstream benchmarks, modest amount of data (<1B) sufficient elicit the full potential of DEX, and simply adding more general data yields diminishing returns in downstream tasks.
Figure 13: DRX with varying data size.
Conclusion
In this work, we study the internal mechanism of DIFF Transformer to identify three key factors behind its empirical success: enhanced expressivity via negative attention, reduced redundancy among attention heads and improved optimization dynamics. Based on these insights, we propose DEX, an architectural adaptation method that efficiently integrates the empirical strengths of DIFF Transformer into pretrained LLMs with standard transformer architecture. Diverse evaluation results confirm the effectiveness and versatility of DEX.
Limitations For DIFF Transformer analysis, we followed the original setup as much as possible, but different behaviors
emerge under different model scale, data composition, etc. Similarly, though DEX works well across model sizes, it has not been tested beyond 8B parameter scale. We leave it for future works.
Acknowledgement N. Kwak was supported by NRF 2021RRA2C300666 and ITTP grants (RS-2021-112 1343, RS 21-2022-111222220 RS- RS-2025-254423388 all funded by the Korean Govermment.
10

---

References
[1] Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: highly capable language model locally on your phone. arXiv preprint arXiv:2404. 14219, 2024.
[2] Joshua Ainslic, James Lee-Thorp. Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305. 13245, 2023.
[3] Ameen Ali, Tomer Galanti, and Lior Wolf. Centered self-attention layers. arXiv preprint arXiv:2306.01611 0/000, 2023.
[4] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint MxXiv:210.00773: 2021.
[5] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang. Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 2023.
[6] Amanda Bertsch, Maor Ivgi, Emily Xiao, Uri Alon, Jonathan Berant, Matthew R Gormley, and Graham Neubig. In-context learning with long-context models: An in-depth exploration. arXiv preprint 2rXiv:2405.00200 2024.
[7] Yuchen Bian, Jiaji Huang, Xingyu Cai. Jiahong Yuan, and Kenneth Church. On attention redun- dancy: A comprehensive study. In Proceedings of the 2021 conference of the north american chapter of the association for computational linguisties: human language technologies, pages 930-945, 2021.
[8] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
[9] Tom Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[10] Yueyang Cang, Yuhang Liu, Xiaoteng Zhang, Erlu Zhao, and Li Shi. Dint transformer, 2025.
URL
17486
[11] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference computer vision, pages 213-229. Springer, 2020.
[12] Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. Efficient intent detection with dual sentence encoders. arXiv preprint arXiv:2003.0480) 2020.
[13] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:1 14:15504-15097 2021.
[14] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
[15] Hongrong Cheng, Miao Zhang, and Javen Qinfeng Shi. Mini-llm: Memory-efficieen structured pruning for large language models. arXiv preprint 1rXiv:2407.11633 2024.
[16] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/ abs/1003. 2018.

---

[17] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. /4/68, 2021.
[18] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems, 35:16344-16359, 2022.
[19] Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non- convex optimization. Advances in neural information processing systems, 27, 2014.
[20] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- tre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguisties: human language technologies, volume (long and short papers), pages 4171-4186, 2019.
[21] Dujian Ding, Ganesh Jawahar, and Laks VS Lakshmanan. Pass: Pruming attention heads with almost-sure sparsity targets. Transactions on Machine Learning Research.
[22] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11929, 2020.
[23] Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodee Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tristan Hume, Chris Olah, Jared Kaplan, and Sam McCandlish. A mathematical framework for transformer circuits. https: /tt:://traaststet ormer circuits. pub/ 2021/framewone index. html December 2021.
Accessed: 2025-05-03.
[24] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.0000 2020.
[25] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL https://zeeooo. org/records/ 12608602
[26] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407:21788, 2024.
[27] Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, al. Olmo: Accelerating the science of language models. arXiv preprint arXiv:2402.0088 2024.
[28] Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv=2337.04725,,0 2023.
[29] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
[30] Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22(241):1-124, 2021.
12

---

[31] Eduard Hovy. Laurie Gerber, Ulf Hermjakob, Chin- Yew Lin, and Deepak Ravichandran. Toward comaantics-based answer pinpointing. Proceedings of the first international conference on Human language technology research, 2001.
[32] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1 (2):3, 2022.
[33] Jiho Jang, Seonhoon Kim, Kiyoon Yoo, Chaerin Kong, Jangho Kim, and Nojun Kwak. Self- distilled self-supervised representation learning. In Proceedings of the IEEE/CVE winter conference on applications of computer vision, pages 2829-2839, 2023.
[34] Jiho Jang, Chaerin Kong, Donghyeon Jeon, Seonhoon Kim, and Nojun Kwak. Unifying vision- language representation space with single-tower transformer. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 980-988, 2023.
[35] Stanislaw Jastrzebski, Maciej Szymczak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho, and Krzysztof Geras. The break-even point optimization trajectories of deep neural networks. arXiv preprint arXiv:2002.09557. 2020.
[36] Stanislaw Jastrzebski, Devansh Arpit. Oliver Astrand, Giancarlo B Kerg, Huan Wang, Caiming Xiong, Richard Socher, Kyunghyun Cho, and Krzysztof J Geras. Catastrophic fisher explosion: Early phase fisher matrix impacts generalization. In International Conference on Machine Learning, pages
PMLR, 2021.
[37] Nandan Kumar Jha and Brandon Reagen. Entropy-guided attention for private 1lms. arXiv preprint Stti::2001004499 2025,
[38] Tao Ji, Bin Guo, Yuanbin Wu, Qipeng Guo, Lixing Shen, Zhan Chen, Xipeng Qiu. Qi Zhang, and Tao Gui. Towards economical inference: Enabling deepseek's multi-head latent attention in any transformer-based llms. arXiv preprint NMNi:22502.4483337 2025.
[39] Greg Kamradt. Llmtest_neeet needleinahhystackk NeedleInAfaaystack 2023.
[40] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015-4026, 2023.
[41] Chaerin Kong, DongHyeon Jeon, Ohjoon Kwon, and Nojun Kwak. Leveraging off-the-shelf diffusion model for multi-attribun fashion image manipulation. In Proceedings of the IEEE/cvf winter conference on applications of computer vision, pages 848- 058- 2023.
[42] Stefan Larson, Anish Mahendran, Joseph J Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K Kummerfeld, Kevin Leach, Michael A Laurenzano, Lingjia Tang, et al. An evaluation dataset for intent classification and out-of-scope prediction. arXiv preprint 2019.
[43] Chong Li, Shaonan Wang, Yunhao Zhang, Jiajun Zhang, and Chengqing Zong. Interpreting and exploiting functional specialization in multi-head attention under multi-task learning. arXiv preprint 10X. 2023.
[44] Jeffrey Li, Alex Fang. Georgios Smyrnis, Maor Ivgi. Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha Sedrick Keh, Kushal Arora, Saurabh Garg. Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mereat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng- Yu Hsieh, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah Pratt, Sunny Sanyal, Gabriel II- harco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang. Khyathi Chandu, Thao Nguyen, Igor Vasiljevic, Sham Kakade, Shuran Song. Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo, Alaaeldin El-Nouby. Hadi Pouransari, Alexander Toshev, Stephanie Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alexandros G. Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar.
13

---

Datacomp-Ihe Im: In search of the next generation of training sets for language models. arXiv preprint arXiv:2406.11794, 2024.
[45] Jiaoda Li, Ryan Cotterell, and Mrinmaya Sachan. Differentiable subset pruning of transformer heads. Transactions of the Association for Computatioonll Linguistics, 2-1442-14555 2021.
[46] Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.0611// 2023.
[47] Jianhua Lin. Divergence measures based on the shannon entropy. IEEE Transactions on Information theory, 37(1):145-151, 1991.
[48] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412. 19437, 2024.
[49] Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with blockwise ringattention. arXiv preprint arXiv:2402.0827 2024.
[50] Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024.
[51] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1r1 1. 1.05101, 2017.
[52] Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint wrXiv:2/04408788 2021.
[53] Ang Lv, Ruobing Xie, Shuaipeng Li, Jiayi Liao, Xingwu Sun, Zhanhui Kang. Di Wang, and Rui Yan. More expressive attention with negative weights. arXiv preprint arXiv:2411.07 2024.
[54] Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. Advances Neural Information Processing Systems, 17112208--121772 2024.
[55] Weikang Meng, Yadan Luo, Xin Li, Dongmei Jiang, and Zheng Zhang. Polaformer: Polarity- aware linear attention for vision transformers. arXiv preprint arXiv:2501. 2025.
[56] Meta AL Llama 3.2: Connect 2024 vision for edge and mobile devices. https: ai
meta. www.blog//////cccccccocoooowooot a1aaaaaaaaaaammmmmma 3-2-connect. www.ecntt222---iieeeeeeeee -ee wwt.-eeeeeeeoobll--eeeeeeeeeee Septem- ber 2024. Accessed: 2025-04-24.
[57] Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? Advances in neural information processing systems, 32, 2019.
[58] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. In EMNLP, 2018
[59] Mistral AI Team. Mistral 7B: The best 7B model date, Apache 2.0. https://mistrall.at news/ httss//mmoonniinnnisssrrr...
September 2023. Accessed: 2025-04-09.
[60] Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11264-11272. 2019.
[61] Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. arXiv preprint arXiv:2010.15327, 2020.
[62] Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. olmo furious. arXiv preprint 2xXi:2551.00655 2024.
14

---

[63] Namuk Park and Songkuk Kim. Blurs behave like ensembles: Spatial smoothings to improve accuracy, uncertainty, and robustness. In International Conference on Machine Learning, pages 17390-17419. PMLR, 2022.
[64] Namuk Park and Songkuk Kim. How do vision transformers work? arXiv:2202.0670, 2022.
arXiv preprint
[65] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195 4205, 2023.
[66] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The Refined dataset for Falcon LLM: outperforning curated corpora with web data, and web data only. arXiv preprini arXiv:2306.01 2023. httpas:/a///axiiti
http:/bb/233..011 01116
[67] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 7748.8763. PmLR, 2021.
[68] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint
2019.
[69] Gemini Team, Petko Georgiev, Ving lan Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 2024.
[70] Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503. 19786, 2025.
[71] Teknium. OpenHermes 2.5: An open-source instruction dataset. https://buggingffcee co/ 2024. Accessed: 2025-05-12.
[72] Reginald P Tewarson and Reginald P Tewarson. Sparse matrices, volume 69. Academic press New York, 1973.
[73] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 2023.
[74] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[75] Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi- head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv:1905.09418, 2019.
[76] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NIP,, pages 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10. (6533/11/11854444
URI.https: littps://aclanne aaaaa//hoolaaaaggyoc
org/W18-
[77] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 1804.07461, 2018.
[78] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: stickier benchmark for general-puroose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc
15

---

E. Fox, and R. Gareett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL
file/ f
Paper pdf
[79]
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019.
[80] Kevin Wang,
Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: circuit for indirect object identification in gpt-2 small. arXiv preprint Xiv: 1000993, 2022.
[81] Maurice Weber, Daniel Y. Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexan- drov, Xiaozhong Lyu. Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin. Tri Dao, Percy Liang, Christopher Ré, Irina Rish, and Ce Zhang. Redpajama: an open dataset for training large language models. NeurIPS Datasets and Benchmarks Track, 2024.
[82] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Reprexentations.
[83] Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared Ilama: Accelerating language model pre-training via structured pruning. arXiv preprint arXiv:2310.06694, 2023.
[84] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412. 15115, 2024.
[85] Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, and Furu Wei. Differential transformer. arXiv preprint arXiv:2410.05255 2024.
[86] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
[87] Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Dan Busbridge, Jason Ramapuram, Yizhe Zhang, Jiatao Gu, and Joshua M Susskind. Stabilizing transformer training by preventing attention entropy collapse. In International Conference Machine Learning, pages 40770- 40803. PMLR, 2023.
[88] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference computer vision, pages 3836-3847, 2023.
[89] Zhisong Zhang, Yan Wang, Xinting Huang, Tianqing Fang, Hongming Zhang, Chenlong Deng, Shuaiyi Li, and Dong Yu. Attention entropy is key factor: An analysis of parallel context en- coding with full-attention-bsse pre-trained language models. arXiv preprint 1rX15:2412.16555. 2024.
[90] Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficteee effficient 11m training by gradient low-rank projection. arXiv preprint arXiv:2403.03507, 2024.
[91] Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models, arXiv preprint ar0iv:2304. 2tl.23304006664, 2023.
[92] Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-folloiinng evaluation for large language models. arXiv preprint arXiv:2311.07911 2023.
[93] Hanqing Zhu, Zhenyu Zhang, Wenyan Cong, Xi Liu, Sem Park, Vikas Chandra, Bo Long, David Pan, Zhangyang Wang, and Jinwon Lee. Apollo: Sgd-like memory, adamw-level performance. arXiv preprint arX1v:2412.05270, 2024.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We present analyses in Sec our extension method in Sec 3 and experiments in Sec
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We mention the limitations in Sec
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
Our work does not include theoretical results that demand disclosure of their assumptions or proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: briefly explain our experimental setup in Sec 4.1 We provide comprehen-
sive details in Appendix E
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
18

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
Some part of the code proprietary asset, which prohibits disclosure.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://niis.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ipss.cc/pill wttpcccpppppppppeeeeeeeeeeeeeeee httaa//gggieee/eeeuuebiisssssss /Cc////oddiiisssseeeeeee ssiiiiiisiyyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We present all the relevant details in Sec and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The experiments are generally too expensive multiple times and provide eтror bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
19

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide relevant information in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Our work does not involve any human subject, and we rely completely on publicly available open-source data.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss this in Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
20

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does contain any contributions that pose high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We add proper citations to all the open-source data, code and models use in this work.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
21

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our work does not include release of new assets at this point.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
22

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23

---

A Related Works
85 introduces an architecture designed to mitigate attention noise 39 50 (52) a known challenge in transformer models. Its core mechanism involves computing attention scores using two groups and then subtracting the resulting attention maps, aiming to cancel out common-mode noise components. Building on this, DINT Transformer [T]] aims to improve numerical stability and training dynamics by incorporating an integral term alongside the differential one. However, these pioneering works do not provide detailed mechanistic analyses explaining why differential attention is effective. Furthermore, both architectures inherently require computing two separate attention pathways, resulting substantial computational overhead compared standard attention. This increased cost hinders practical deployment, particularly for large-scale models. Building on top of our analysis on DIFF Transformer's success, we propose DEX that implicitly embeds the benefits of differential attention into pretrained language models without nontrivial computational overhead.
Differential Transformer
Negative Attention Scores. Several approaches explicitly introduce negative attention scores. Centered Attention (3), for instance, adds offsets to the softmax calculation, forcing attention weights per query to sum to zero (rather than one) to mitigate over-smoothing. Other methods achieve negative weighting through direct manipulations of the softmax operation or via linear attention approximations [53] 55), often demonstrating enhanced representational expressivity. However, methods that explicitly alter the core attention computation can introduce training stability challenges and often lack compatibility with highly optimized implementations like FlashAttention [18]. In contrast, DEX aims to capture the benefits of signed, differential attention implicitly. By applying its learnable transformation after the standard softmax attention calculation (i.e., to the output values), DEX avoids modifying the core QK-softmax pathway, thereby maintaining compatibility and potentially simplifying integration and training.
have shown that there is significant redundancy among attention heads in multi- head attention, and propose head pruning methods to enhance efficiency. Instead of getting rid of unimportant heads, our approach applies implicit differential adaptation to redundant heads, effectively revitalizing them and modeling richer attention representations. Meanwhile, attention entropy-baad based analyses have provided insights into the transformer attention mechanisms, 89 55) argues that excessively high attention entropy negatively impacts performance, while (37 87 associates entropy with training stability. In this
Attention Redundancy and Entropy. Numerous works [15 60 57 75
45
work, we leverage attention entropy in two ways: (1) understanding the attention score distribution
(and potential sparsity), and (2) identifying less critical attention heads.
B Ablation and Analysis
In this section, we present additional empirical results to support our design choice and analysis.
B.1 Attention Magnitude Correlation
In Fig present the correlation between attention scores from Llama and DIFF Transformer, computed layer by layer. Specifically, we compare Llama's 's soft- max scores against both the original signed scores from DIFF attention (green) and their absolute val- ues (blue). Note that while standard Llama attention scores are non-negative (due softmax), DIFF atten- tion scores can be negative. Interestingly, both rank and Pearson correlations are significantly higher when using absolute values (blue) compared to signed val- ues (green). This suggests strong correspondence in the magnitude of attention (indicating relative impor- tance), even when the signed scores differ. This observation motivates our DEX design: since the relative importance signals (magnitudes) from the standard QK/softmax pathway are largely pre- served, we reuse them and focus our adaptation efforts on enhancing the subsequent OV circuit to incorporate differential mechanism.
(a) Rank Corr.
(b) Pearson's Corr.
Figure 14: Correlation between Llama atten- tion and DIFF attention.
24

---

B.2 Attention Head Redundancy
We address the potential concern that lower inter-head redundancy in DIFF Transformer stems from its common configuration using fewer, wider attention heads (typically halving head count while doubling head dimension
We plot the average pairwise cosine distance between head attention scores per layer (5). The figure shows DIFF attention exhibiting signifccntly higher average cosine distance, indicating lower redundancy (greater pattern dissimilarity) among its heads. No- tably, merely using fewer, wider heads does not repli- cate this effect, as demonstrated by our LLAMA-half baseline (green), configured with halved head count and doubled head dimension. We hypothesize that the differential mechanism grants greater flexibilityy in controlling attention patterns, thus reducing inter- head redundancy.
Figure 15:
Mean pairwise cosine distance between attention scores from different heads.
Heatmaps visualizing the pairwise cosine distances between attention maps from different heads (Fig[16)
further corroborate our findings. They show lower inter-head distances (indicating higher similarity and redundancy) in the standard transformer (LLAMA), whereas DIFF Transformer maintains higher distances, demonstrating more diverse attention patterns.
LLAMA
DIFFT
Pairwise cosine distance between attention maps from different attention heads in each layer. Brighter color indicates larger distance, hence lower redundancy.
Figure 16:
Centered Kernel Alignment (CKA) analysis comparing attention heads before and after applying DEX (7) further confirms that DEX reduces inter-head redundancy. The results clearly show lower overall alignment between heads after adaptation in the pretrained models, indicating increased functional diversity.
25

---

Figure 17: Centered Kemel Alignment for attention heads. Brighter colors indicate higher alignment/similarty.
(Top) Llama, (Bottom) DEX.
B.3 Abalation Ainit
Table shows DEX performance on the language modeling benchmarks (average over 11 tasks from Table using Qwen-0.5B) when varying the Ainit strategy- The results indicate relative robustness different fixed scalar initializations (0.3-0.8). How- ever, adopting the initialization scheme from the orig- inal DIFF Transformer setting yields slightly the best performance. We hypothesize the layer-aware initialization is beneficial for training.
Table 6: Ablation on Amit- DIFF refers to
depth-aware initialization following (85).
Amnst
0.8
0.5
0.3
DIFF
LM Acc (%)
54.3
54.0
54.2
54.4
B.4 Ablation Head Selection
In Table 7 we present the average performance of DEX with different number of target attention heads (k) on 11 language modeling benchmarks. Select- ing too few heads (e.g., 8) provides insufficient capacity for the differential adaptation, while modify- ing too many heads risks disrupting critical pretrained knowledge, leading to performance degradation. We empirically find modifying about 50% of the attention heads tends to optimal in general (note we use Llama- IB with 32 heads for ease of demonstration).
Table 7: Ablation on head selection k.
A
8
16
24
32
LM Acc (%) 53.7
55.6
54.4
54.1
B.5 Qualitative Results
We display additional examples from qualitative analysis in Sec 2.2
DIFFT
LLaMA
Figure 18: Qualitative examples for DIFF Transformer and Llama.
with the observations from Fig 4 illustrating how DIFF attention leverages negative attention scores. The left example shows Indirect Object Identification task where DIFF Trans- former assigns negative attention score to mark the subject (i.e., John) as irrelevant. The right example shows sarcasm detection, where DIFF attention identifies the non-literal expression and explicitly allocates negative attention scores accordingly.
C Comparison to DIFF Transformer
direct comparison with the original DIFF Transformer model not possible due to unavailable weights. Therefore, to establish point of reference, we compare DEX with DIFF Transformer model
26

---

that we trained ourselves at smaller scale, following the procedures detailed in Appendix E.2 To
set up this comparison, we first train two models from scratch on the exact same training data: (1) standard transformer baseline using the Llama architecture (I (Ilama), and (2) DIFF Transformer model (DIFF). Subsequently, we apply DEX to the Llama baseline using small subset of the pretraining data (<IB tokens) to create the third model, simply noted DEX. We additionally train a separate Llama model from scratch with DEX attached from the beginning, to understand DEX'S architectural capacity beyond its original purpose of adaptation, which we refer to DEX-S. We report the performance of these four models (Llama, DIFF, DEX, DEX-S) on the 11 language modeling benchmarks Table
Table 8: Scores 11 benchmarks. Green indicates increases and gray indicates decreases. All
values rounded one decimal place.
Model
Arc-C
Are-E
BoolQ
COPA
Hellaswag
MNLI
OBQA
PIQA
WIC
Winogrande
WSC
AVG
Llama
21.8
37.0
60.5
63.0
29.0
35.1
25.2
58.4
42.4
DIFF
24.2
37.2
54.0
68.0
29.0
35.5
26.4
58.9
50.0
52.2
36.5
42.9
+0.5
DEX
22.2
37.1
60.5
64.0
29.0
35.1
25.8
58.3
50.6
51.2
36.5
42.8
+0.4
DEX-S
22.5
37.1
61.5
63.0
28.7
35.2
27.4
58.1
50.0
$1.0
36.5
42.8
+0.4
From the table, we first observe that DIFF Transformer generally outperforms standard transformer on the majority of benchmarks, which supports the strength of DIFF Transformer as a general purpose language model. Furthermore, the results clearly demonstrate that DEX, despite being lightweight both during training and inference, effectively enhances the pretrained Llama model, closing the gap between standard transformer and DIFF Transformer. DEX-S, variant of DEX applied from scratch, also delivers competitive performances beyond standard Llama model.
LLAMA
Figure 19: Head CKA comparison between Llama vs DEX vs DIFF.
Head CKA results further support the effectiveness of DEX (Fig Compared standard trans- former (left), DEX significantly reduces the inter-head redundancy (indicated by lower alignment), yielding similar results to DIFF Transformer.
D Efficiency Analysis
To evaluate inference efficiency, we benchmark throughput (tokens per second) for 3B-parameter versions of Llama, DIFF Transformer, and DEX, presenting the results in Fig 12a Context lengths were varied from 1k to 64k tokens to cover a comprehensive range of use cases. All tests were conducted single NVIDIA A100-80GB GPU, utilizing PyTorrc's 's standard scaled dot-product attention implementationf The reported throughputs are averaged over 30 batches, following an initial warm-up batches.
E Implementation Details
In this section, we provide comprehensive details for our experiments, some of which were abbreviated in the main manuscript for brevity.
wttppp/twwrrae////ttaa/////tt//a wn..oooroooo.ooooo...o..ooooee.a http://reeeeereeooaooooooocccnoo soaleddoo.. product__teee hat.uctt__attttiiohttt..
27

---

E.1 Language Modeling Evaluation
Dataset We constructed our custom training corpus using a subset of the Dolmino datase Specifi- cally, we mixed web pages, academic papers, encyclopedia entries, and code texts in approximate ratios of 74.3%, 6.5%, 7.9%, and 11.3% respectively. This resulted in corpus totaling 887M tokens (measured using the Ilama-3 tokenizer). Our data preparation generally followed the recipe of OLMo2 [62]. with the main exception being a greater upsampling the code text component.
Training All models, including baselines and DEX variants, were trained for I epoch. A context length of 32k tokens was used for all Llama and Qwen models during this training phase. We employed cosine learning rate schedule, using peak learning rate of 1 x 10-1 for partial fine-tuning methods (including DEX) and x 10-6 for full fine-tuning baselines, as these settings generally yielded the best outcomes in preliminary experiments. learning rate warm-up ratio of 0.03 was used. All experiments were conducted using 8 NVIDIA A100-80GB GPUs, with the run time ranging from 2.5-16 hours depending on the model size.
custom corpus
E,2 Training DIFF Transformer
We train our own DIFF Transformer model for analysis. This subsection details its training procedure.
Dataset We followed the recipe of DIFF Transformer and StableLM M-3B using various open-source datasets [66 24 46 81] create corpus of approximately 30 billion tokens (Llama-3 tokenizer). This corpus encompasses a diverse range of domains, including academic papers, source code, encyclopedic articles, and literature.
Model We trained a 0.4-bilionn parameter version of DIFF Transformer. Key architectural parame- ters are provided in Table
Table 9: Configuration for 0.4B DIFF Transformer.
params
values
# Layers
# Heads
16
# KV Heads
4
Hidden size
1024
FFN size
4096
Training For training, we employed the Adam W optimizer [51] with cosine learning rate schedule. The peak learning rate was set to 1 10-4, the global batch size to 256, and the learning rate warm-up to 0.1. The À parameters within the differential attention were initialized according to the exact schedule specified in the original DIFF Transformer paper [85).
E.3 Approximating Effective Attention Scores for DEX Interpretability
Because DEX directly alters the attention block's output value matrix O rather than the initial softmax scores, standard attention visualization can be misleading. To provide insight into its effective learned behavior, we propose methods approximate the effective attention scores that would yield DEX'S modified output using the original value matrix.
Least-Squares Approximation This method uses the Moore-Penrost pseudoinverse to derive effective attention scores X that best reconstruct DEX's output transformation. Specifically, let A softmax QK" /dd)) be the original softmax attention scores from a given head. V E R Nxd, be the corresponding value matrix (where N sequence length, dę key dimension, d, value dimension), and W E Xd, be the learnable weight matrix for fD in DEX (assuming X(t) is
httpe///wgguuuiteecc//eeeeeeeeep 1124 http::///gtaauoooos//aaallaa///p
28

---

absorbed into Wo or considered R for this analysis). The original head output is O = AV, and the DEX-modified output O' =O WD). We seek an effective attention score matrix X such that XV R O'.
The least-squares solution for X is:
- = AV(I- WD)V
(6)
where V denotes the Moore-Penrose pscudoinverse of V, computed numerically in practice.
This X represents the attention pattern that, if applied to the original values V, would best reconstruct DEX's modified output for that head. Since this involves an approximation and the use of a pseu- doinverse (which can be sensitive if V is ill-conditioned has significant null space), numerical considerations are important. We therefore complement and cross-check these results using a second technique.
Optimization-base Approximation As with the pseudoinverse method, we aim to find an effective attention score matrix X such that XV approximates the DEX output O' AV(T W D)) Rather than a closed-form pseudoinverse solution, this approach directly optimizes for for each input sample for which O' and V are computed. For each sample, X is typically initialized (e.g., as the original attention scores A) and then updated for 100 iterations using gradient descent (learning rate 1 1 10 9) minimize reconstruction loss with the form |XV O
The primary interpretable attention scores reported in our main analyses (e.g.. Table 2 were derived using the pseudoinverse method. This optimization- based approach served cross validation, and we confirmed strong agreement between the effective attention scores obtained from both techniques. While both methods yield approximations subject numerical precision, they offer valuable tools for understanding the internal mechanisms and effective attention patterns of DEX.
F Broader Impact
Potential Positive Societal Impacts: By improving core LLM capabilities such information retrieval, in-context learning, and overall representational quality, DEX could contribute to more effective and reliable AI systems. This includes advancements in Al-assisted education, more capable research tools, improved accessibility to information, and more helpful Al assistants. Furthermore, DEX's design emphasizes lightweight adaptation, which could make powerful LLM enhancements more essoucce-fffcceet and accessible, potentially reducing the computational burden associated with adapting large models.
Potential Negative Societal Impacts: As DEX is designed to improve the capabilities of L.MMs, shares the potential negative societal impacts inherent in more powerful language model technology. Enhancements in LLM performance and efficiency could inadvertently facilitate the creation of more sophisticated scalable misuse scenarios, such generating convineing disinformation, spam, or impersonations. If an LLM enhanced by DEX produces incorrect or biased information, its improved fluency might make such outputs seem more authoritative, potentially exacerbating harm. While DEX is foundational architectural improvement rather than a specific end-user application, the dual-use nature of advancements LLM capabilities warrants careful consideration.
We believe that continued research into robust AI safety measures, ethical development guidelines, bias detection and mitigation, and responsible deployment practices for all LLMS crucial their capabilities, including those enhanced by methods like DEX, advance.
29


Reviews:
[{"review_id": "UqCP4ST7Pl", "reviewer": "Reviewer_9JJa", "summary": "The paper introduces DEX (Differential Extension), an extension designed for the recently proposed Differential Transformer (DIFFT) framework.  The paper begins by explaining why DIFFT works in practice, empirically demonstrating that DIFFT takes advantage of: a) discrepancies between all attention (vs salient) scores, (b) negative relevance in QK values, (c) reduced redundancy in attention heads, and (d) improved learning dynamics.  DEX is then introduced, the chief innovation of which is moving trainable parameters from a second set of QK matrices (i.e., DIFFT) to the regular output matrix (OV) minus a \\lambda* parameterized-weight matrix *OV.  DEX also incorporates adaptive head selection (low-importance and high-entropy) during differential adaptation and an annealing schedule to \\lambda.  DEX performance is subsequently demonstrated over Llama3 and Qwen2.5 models of various sizes against various PEFT methods (adapted over the base model) for common reasoning benchmarks.  While improvements are not always substantial, DEX does consistently improve performance across all models, which is not the case for competitor methods (including full fine-tuning).  Additional experiments key information retrieval and ICL also demonstrate the utility of the proposed method relative to full fine-tuning/LoRA/base model performance.", "questions": "# Minor suggestions\nFor Figure 1, could the authors add up arrows to subfigures (a) and (b) and downarrows for subfigures (c) and (d) (as well as mention these trends for the respective metrics))?  This will help the reader quickly parse the results (especially since the similarity trends in (a-b) are flipped in (c-d)) to see the stated conclusion: \" It clearly shows that the overlap between the two attention scores is much greater in non-salient tokens.\"  Also, listing curve averages and contrasting \"Top 5%\" vs \"All\" will help quantify the stated conclusion.\n\n# Questions\n- \"Although DEX is not presented as a fine-tuning method, we compare against baselines trained on the same data\" <- Is DEX trained using full precision then?\n- Is there a downside to Equation 3?  It seems like there is a tradeoff between granularity for efficiency that could be mentioned.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n## Signifiance and Originality\nThe DIFFT framework has been a high impact contribution since its debut.  The fundamental idea behind DEX, i.e., removing dual QK matrices in favor of a single parameterized weight on the output-value matrix, is a very interesting and intuitive approach.\n\n## Quality\nMany of the experiments in Sections 2 and 4 were high quality, supporting the several points made in the text.\n\n# Weaknesses\n## Quality\nA critical issue is the lack of necessary experiments/benchmarks.  E.g.:\n> Direct comparison with original DIFF Transformer is limited by unavailable pretrained weights, which\n we defer to Appendix along with other details.\n\nComparison to DIFFT is necessary to show demonstrate the effectiveness of DEX relative to the work it builds on.  Even if the checkpoint is unavailable,\n the code from the Differential Transformer paper is available:\nhttps://github.com/microsoft/unilm/tree/master/Diff-Transformer\n\nFurthermore, given the custom data used for the experiments in Table 1, comparing to the DIFFT checkpoint would not be a fair comparison; to ensure a fair comparison, DIFFT would have to be trained on the same data and training recipe as used for DEX.\n\nAdditionally, one of the major claimed drawbacks to DIFFT is the following:\n> We note that despite DIFF attention having the same number of parameters, it exhibits significantly higher compute cost and peak memory usage in practice due to enlarged dimensions\n\nThis is a main criticism of DIFFT, and motivation for DEX.  Thus, a heads up comparison showing both training/testing runtime and memory usage between DIFFT and DEX is required.\n\nFinally, some presented experiments do not accurately support the claims being made.  E.g.:\n\n\"Training statistics further corroborate this finding. Fig.8 plots the language modeling loss and gradient\nnorms for the standard and DIFF transformer. While DIFF consistently achieves lower loss and more\nstable grad norms, removing the learnable λ notably impairs optimization. We hypothesize that the\nlearnable λ plays a key role in stabilizing training dynamics, especially during the early stages.\" <- At the end of the day, these do not equate to downstream performance.  Could the authors test the model (is this Llama3?  Which one?) w and w/o learnable \\lambdas on natural language tasks to show that there is some performative difference?\n\nIn Section 3.3, zero-initializing \\lambda is the not the only option.  It is unclear whether annealing is required, or some randomly (perhaps Gaussian) sampled values would be sufficient.  Furthermore, in the ablation experiments:\n> However, adopting the initialization scheme from the original DIFF Transformer setting yields slightly the best\nperformance.\n\nThus, it is thus unclear whether \\lambda annealing is thus a valid design choice.  Based on these results, \\lambda annealing is suboptimal and would ideally be replaced by the DIFFT initialization scheme in the main results.\n\n## Clarity\nThe paper has substantial room to improve in clarity.  For instance, important details are lacking when describing the experiments.  These critically limit the ability to understand these experiments.  \n- For Figure 1, can the authors please detail the experiments either in the figure caption or inline (i.e., lines 70-79).  Reading through those lines, it is not clear what dataset or models are being presented.\n- For Figure 2, could the authors please include what LLMs are being evaluated, e.g., is it Llama-3.2-1B?\n- \"Notably, this benefit is largely lost when the learnable λ is removed\" <- What does this mean?  Is \\lambda set to one in this case?\n- Need similar model/data details for the following experiments: \"Qualitative examples in Fig.4, such as down-weighting irrelevant subject in Indirect Object Identification task [77] or non-literal interpretation in sarcasm detection, illustrate how DIFF attention can achieve a more refined information flow using negative attention (green boxes).\"\n- Can the algorithms for lines 173-175 and 176-179 be written out for full clarity?\n- \"Figure 5\" <- Please include a colorbar.  Also suggest adding a down arrow to (a) and an up arrow to (b).\n\nThe writing also has significant room for improvement.  Currently, the paper feels crammed, which may be why many details are lacking for the experiments in Section 2.  Some of the text could be tightened up/condensed, e.g., lines 145-151 and 152-158, as well as lines 161-164 (which would ideally go in the discussion section).", "was_revised": true, "final_justification": "## Update during rebuttal\nThe authors have pointed to various points in both the paper and appendix to address much of my questions, with additional experiments demonstrating the better throughput achievable with DEX.  Furthermore, the authors have agreed to writing edits which quickly address these clarity issues in the main paper.", "rebuttal": "Thank you for your thoughtful review and constructive feedbacks. We are glad you found our work interesting, original and containing high quality experiments. We will address your concerns one by one.\n\n**Concern 1: Comparison to the original Differential Transformer is missing.**\n\nWe agree that comparison between our method (DEX) and DIFFT is an important part of our work. We want to gently mention that we presented 11 natural language benchmark scores in **Tab.8**, where we compare the performance of Llama, DIFFT, DEX from scratch, and DEX from Llama checkpoint (our proposed adaptation setting) all trained on the *exact same data* (see Appendix F.2) along with qualitative CKA results (**Fig.19**). We placed these results in the Appendix due to page limits, but we acknowledge that this choice has made these critical results less accessible to readers. We appreciate your constructive feedback, and promise to rearrange the tables to improve their visibility upon revision. We would appreciate if you kindly understand that we had to run these *from-scratch* pretrainings in smaller scales due to compute resource constraints (model and data specs in Appendix F). \n\nTo supplement our comparative study and further address your concerns, we present key information retrieval (KIR) and in-context learning (ICL) evaluations below:\n\n| KIR | 256 | 512 | 768 | 1024 | 1280 | 1536 | 1792 | 2048 | Avg |\n|---|---|---|---|---|---|---|---|---|---|\n| Llama | 0.09 | 0.05 | 0.04 | 0.06 | 0.11 | 0.04 | 0.02 | 0.06 | 0.059 |\n| DIFFT | 0.12 | 0.07 | 0.10 | 0.09 | 0.13 | 0.08 | 0.05 | 0.11 | 0.094 |\n| DEX | 0.11 | 0.13 | 0.08 | 0.12 | 0.08 | 0.07 | 0.07 | 0.08 | 0.093 |\n| DEX-S | 0.14 | 0.09 | 0.10 | 0.18 | 0.19 | 0.11 | 0.10 | 0.12 | **0.129** |\n\n| ICL | n=1 | 5 | 10 | 15 | 20 | 30 | 40 | 50 | AVG |\n|---|---|---|---|---|---|---|---|---|---|\n| Llama | 0.074 | 0.096 | 0.178 | 0.200 | 0.170 | 0.193 | 0.185 | 0.156 | 0.156 |\n| DIFFT | 0.074 | 0.141 | 0.148 | 0.156 | 0.207 | 0.193 | 0.207 | 0.207 | 0.167 |\n| DEX | 0.074 | 0.104 | 0.185 | 0.200 | 0.200 | 0.200 | 0.193 | 0.193 | **0.169** |\n| DEX-S | 0.067 | 0.119 | 0.185 | 0.193 | 0.185 | 0.193 | 0.200 | 0.193 | 0.167 |\n\n- DEX-S stands for DEX trained from scratch with the exact same training data as others.\n- Columns in KIR correspond to the context length, averaged across 5 different needle depths (0, 25, 50, 75, 100%) as in Fig.10.\n- Columns in ICL represent the number of shots. We average the scores on three benchmarks used in Tab.3, Trec, Banking-77 and Clinic-150.\n- Full detail in Appendix F.\n\n**Concern 2: Train/test efficiency and memory usage comparison is required.**\n\nThis is a great feedback. While we do have test-time throughput comparison in **Fig.12** (with Llama-3B model) where DEX shows far better efficiency than DIFFT, we additionally show other efficiency benchmarks (on Nvidia A100 GPU).\n\n**Test run time.**\n\n| Throughput (tokens/sec) | 1024 | 2048 | 4096 | 8192 | 16384 | 32768 | 65536 |\n|---|---|---|---|---|---|---|---|\n| Llama | 23117.8 | 23236.7 | 22731.5 | 20777.9 | 18327.2 | 14804.0 | 10272.9 |\n| DEX | 21787.5 | 22131.6 | 21775.2 | 20048.2 | 17788.0 | 14470.3 | 10127.4 |\n| DIFFT | 19474.2 | 18986.2 | 16972.2 | 13693.5 | 9766.9 | 5879.1 | 2757.1 |\n\n| Latency (ms) | 1024 | 2048 | 4096 | 8192 | 16384 | 32768 | 65536 |\n|---|---|---|---|---|---|---|---|\n| Llama | 44.3 | 88.1 | 180.2 | 394.5 | 894.4 | 2213.5 | 6379.5 |\n| DEX | 47.0 | 92.5 | 188.1 | 408.8 | 921.5 | 2264.5 | 6471.1 |\n| DIFFT | 52.6 | 107.9 | 241.3 | 598.5 | 1678.3 | 5573.6 | 23770.0 |\n\n| peak Vram (GB) | 1024 | 2048 | 4096 | 8192 | 16384 | 32768 | 65536 |\n|---|---|---|---|---|---|---|---|\n| Llama | 6.49 | 6.98 | 7.97 | 9.96 | 13.92 | 21.84 | 37.68 |\n| DEX | 6.61 | 7.11 | 8.10 | 10.08 | 14.04 | 21.96 | 37.80 |\n| DIFFT | 6.49 | 6.98 | 7.97 | 9.96 | 13.92 | 21.84 | 37.68 |\n\n- Above tables measure test-time throughput, latency and peak memory usage with varying context lengths (column) and batch size set to 1.\n- Note that DEX uses additional 120mb peak memory (constant regardless of input) due to the learnable projection parameters.\n\n**Train run time.**\n\n| Train | sec/step | total runtime | peak VRAM |\n|---|---|---|---|\n| Llama | 1.42 | 3h 33m | 68.1 |\n| DEX | 1.52 | 3h 50m | 68.8 |\n| DIFFT | 2.52 | 6h 18m | 70.7 |\n\n- 0.4B model (Appendix F), 890B training tokens, 24k context window, AdamW optimizer, 4 gpus, bf16 automatic mixed precision, pytorch SDPA implementation with no gradient checkpointing.\n\n**Concern 3a: Ablation for learnable \\lambda's on natural language tasks is needed.**\n\nWe would like to gently guide you to **Tab.4** (bottom 3 rows), where we present results with and without the learnable lambda on 11 natural language tasks (done on Llama-3B model). For additional clarity, we present full 11 benchmark scores for Llama-3B, Llama-1B and Qwen-0.5B model.\n\n| Model / Variant | Arc-C | Arc-E | BoolQ | COPA | Hellaswag | MNLI | OBQA | PIQA | WIC | Winogrande | WSC | AVERAGE |\n| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| **LLaMA-3.2 3B** | | | | | | | | | | | | |\n| +learnable | 45.5 | 73.3 | 74.8 | 84.0 | 74.1 | 49.5 | 42.6 | 78.2 | 51.9 | 69.1 | 63.5 | **64.2** |\n| -learnable | 45.5 | 73.0 | 71.3 | 83.0 | 73.4 | 49.6 | 40.8 | 77.6 | 53.6 | 67.4 | 62.5 | **63.4** |\n| **LLaMA-3.2 1B** | | | | | | | | | | | | |\n| +learnable | 35.2 | 64.2 | 57.8 | 79.0 | 64.0 | 38.0 | 38.0 | 75.0 | 51.9 | 60.6 | 48.1 | **55.6** |\n| -learnable | 34.5 | 63.1 | 54.5 | 79.0 | 63.8 | 43.3 | 37.4 | 75.0 | 53.0 | 60.3 | 36.5 | **54.6** |\n| **Qwen-0.5B** | | | | | | | | | | | | |\n| +learnable | 34.8 | 65.2 | 56.5 | 73.0 | 52.1 | 40.1 | 35.4 | 70.1 | 51.6 | 57.6 | 61.5 | **54.4** |\n| -learnable | 33.8 | 64.4 | 56.4 | 73.0 | 51.7 | 35.2 | 35.2 | 69.5 | 50.0 | 56.5 | 53.8 | **52.7** |\n\n- Here, we initialize lambda with DIFF scheme and apply lambda annealing, only changing learnable lambda to non-learnable.\n\n**Concern 3b: The effectiveness of \\lambda annealing is unclear, especially with different \\lambda initializations.**\n\nAlthough we present lambda-anneal ablation in **Tab.4** and lambda-initialization ablation in **Tab.5**, we believe that having experiments that put these two variables together would clarify the effectiveness of lambda-annealing. To address your concern, we report the following result:\n\n| Initialization | 0.8 | 0.5 | 0.3 | DIFF | N(0.5, 0.1) | Unif(0, 1) |\n| :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| +anneal | 54.3 | 54.0 | 54.2 | 54.4 | 54.3 | 54.1 |\n| -anneal | 54.1 | 54.0 | 54.1 | 54.2 | 54.0 | 53.9 |\n\n- 'DIFF' in column marks the init scheme from DIFF paper.\n- lambda is learnable.\n- We report average scores from 11 natural language benchmarks in Tab.1, where Qwen-0.5B model is used for efficiency. Same training data is used as explained in Sec.4.\n\nWe can see that lambda annealing consistently delivers performance gains with different initializations.\n\n**Concern 4: The model and dataset being evaluated is unclear in Fig 1 and Fig 2.**\n\nWe mention the model and dataset used for the analyses in Sec.2 in **L50-52**, with reference to full details in **Appendix F.2**. We use the validation split of our training data for the analyses.\n\n**Concern 5: Clarification is needed for the statement “Notably, this benefit is largely lost when the learnable λ is removed.\"**\n\nWe apologize for the simplified explanation. Specifically, we initialize lambda using the original DIFFT scheme and fix it (non-learnable), thereby only removing the “learnable” part of lambda dynamics.\n\n**Concern 6: The model and dataset being evaluated is unclear in Fig 4.**\n\nThe experimental setup for all analyses in Sec.2 is mentioned in **L50-52** and detailed in **Appendix F.2**. For IOI task, as the citation points to in **L96**, we use the data setting from the paper \"Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small\". We will supplement explanation about the specific settings for further clarity.\n\n**Concern 7: Can the algorithms for lines 173-175 and 176-179 be written out for full clarity?**\n\nAbsolutely, we will add the full algorithms in the Appendix for maximal clarity. Here, we provide brief sketches to alleviate your concerns.\n\n1. Entropy-based\n- For each head ($h$) in layer ($l$), compute its average attention entropy scores $\\bar{\\mathcal{E}}_{l,h}$, which is the mean entropy taken over all samples (we pass in 50 samples from our train data).\n- In each layer $l$, we rank all heads based on their average entropy score.\n- We select the top-$k$ heads with the **highest** entropy scores from each layer's ranked list to be modified by DEX.\n\n2. Importance-based\n- For each head ($h$) in layer ($l$), compute its head importance metric (using [57,14] in the main paper).\n- In each layer $l$, we rank all heads based on their importance metric.\n- We select the top-$k$ heads with the **lowest** importance scores from each layer as the target for DEX.\n\n**Concern 8: Adding colorbar, down arrow and up arrow can help improve Fig 5.**\n\nWe will surely do that to improve the clarity of our figure.\n\n**Question 1: Is DEX trained using full precision?**\n\nWe adopt mixed precision training using bf16, deepspeed-zero3 and AdamW optimizer.\n\n**Question 2: Is there tradeoff between granularity and efficiency for Eq.3?**\n\nDirectly modifying the attention map (QK) would be a direct and \"granular\" way to manipulate the attention forward (as in DIFFT). However, this hurts efficiency either by damaging pretrained knowledge (more training is needed) or incurring more compute (slower inference), as mentioned in **L148-150**. We show that DEX, despite being efficient, brings the strengths of full DIFFT.\n\nWe once again thank you for your thorough and constructive review. We have presented additional explanations and experiments which we believe address your concerns. We hope our response resolves your concerns and shifts your assessment of our work in a positive direction. Thank you."}, {"review_id": "rkMXg6my9o", "reviewer": "Reviewer_WqWH", "summary": "Differential transformers have recently established itself as an architecture with competitive or better performance compared to traditional transformers. This paper provides an in-depth analysis of the benefits and working mechanism of differential transformers. By showing the main benefits of differential transformers, namely 1) negative attention scores potentially provides additional expressivity, 2) it allows focusing mainly on salient tokens, 3) the additional lambda parameter helps smoother learning dynamics. \n\nThe paper then goes on to provide a method called DEX that introduces the benefits of differential transformers directly on top of pretrained transformers. Dex uses a learnable scalar lambda and additional learnable projection matrix for a fraction of all attention weights in the pretrained transformer. This allows finetuning of pretrained transformers to get benefits of differential transformers.\n\nExperiments on Llama-3.1-8B, Llama-3.2-3B/1B and Qwen-2.5-1.5B/0.5B finetuned on dataset created by the authors show clear benefit of DEX. Further experiments on key information retrieval and in-context learning shows the benefit of DEX.", "questions": "How important is the head selection, what happens if we do not use head selection in Tab. 4 and try out different combination of lambda_learned and lambda_annealing?\n\nPlease also check the weaknesses if it is possible to show experiments on other datasets", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n- The paper provides an in-depth analysis of the advantages of differential transformer compared to a standard transformer, that is 1) more expressive attention weights since attention can be negative, 2) selective attention heads for salient tokens, 3) an additional lambda parameter that helps with the learning dynamics.\n\n- The paper provides a method to obtain the benefits of a differential transformer on top of standard transformer by finetuning pretrained models. This allows preserving the pretrained knowledge as well as gain the benefits of differential transformers\n\n- The paper also creates a dataset for finetuning that shows good results on the experiments\n\n- Experimental results show improvement compared to standard finetuning of transformers\n\nWeaknesses\n- The experiments are restricted to older datasets. The models considered finetuned have been extensively evaluated on many recent datasets such as MATH500, GSM8k, MBPP, HumaEval, AIME (maybe AIME is not recommended for the selected models in the paper). Can the authors show performance on these datasets as well? The authors can also choose to finetune or apply their method on NuminaMath and evaluate on MATH500 or GSM8k (for non-overlapping cases between NuminaMath and Math500).", "was_revised": true, "final_justification": "The authors have addressed the main limitation I had raised regarding older datasets: the authors have pointed out additional experiments that were present in the appendix as well as added additional experiments.\n\nThe authors have also added ablations regarding my question head-selection.", "rebuttal": "Thank you for your helpful feedbacks and comments. We greatly appreciate your positive assessment about our work providing \"in-depth analysis of differential transformer\", proposing an effective method to \"obtain the benefits on top of standard transformers\" and showing strong experimental results. We hereby address your concerns in details.\n\n**Concern 1: Evaluation result on more recent benchmarks like MATH500, GSM8k, MBPP and HumanEval is needed.**\n\nWe believe this feedback is very relevant and insightful. We first want to gently mention that empirical results for the latest benchmarks such as GSM8k, MBPP++, AGIEval, IFEval, MMLU are presented in **Tab.7**, for both cases where DEX is applied to base pretrained checkpoint (Llama-3.1-8B) and instruction-tuned checkpoint (Llama-3.1-8B tuned on Open-Hermes-2.5 data). It is clearly visible that DEX significantly outperforms LoRA or full fine-tuning baselines across different scenarios. \n\nTo further alleviate your concerns, we ran additional evaluations for MATH-500 and HumanEval with our checkpoints used in Tab.7. We present the extended results below.\n\n| Model | MMLU | ARC-C | IFEVAL | MBPP++ | GSM8K | AGIEval | HumanEval | Math500 | AVERAGE |\n| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| **(1) Instruction-tuned** | | | | | | | | | |\n| Base | 62.9 | 78.3 | 46.8 | 68.3 | 71.1 | 32.2 | 44.5 | 13.4 | 52.2 |\n| + LoRA | 63.1 | 79.5 | 45.7 | 65.3 | 70.3 | 40.6 | 47.0 | 4.0 | 51.9 |\n| + FT | 63.0 | 78.6 | 49.2 | 63.2 | 76.5 | 42.3 | 36.6 | 20.2 | 53.7 |\n| + DEX | 63.1 | 77.7 | 57.2 | 64.8 | 74.3 | 40.7 | 47.6 | 19.2 | **55.6** |\n| **(2) Pretrained** | | | | | | | | | |\n| + LoRA | 63.7 | 70.5 | 42.0 | 65.3 | 57.4 | 35.4 | 45.7 | 2.0 | 47.8 |\n| + DEX | 63.6 | 77.3 | 51.0 | 66.1 | 68.4 | 37.9 | 50.6 | 16.2 | **53.9** |\n\n- The training with Open-Hermes-2.5 instruct data is also very lightweight, taking less than a day to tune an 8B model with a single A100 node. The number of training tokens roughly sums to 350M, which is much less than 0.01% of the 15T+ tokens of llama-3 model [1].\n- Please refer to Appendix C for full details.\n\nIt is clear that DEX delivers robust performance across a wide variety of scenarios and tasks, including broad knowledge (MMLU), instruction following (IFEval), scientific reasoning (Arc-C), math (GSM8k) and coding (MBPP++, HumanEval). It is noteworthy how DEX outperforms both heavier tuning method (FT) and lighter tuning method (LoRA), showing its fundamental advantage as an adaptation method.\n\n**Question 1: What happens if we try out different combinations of lambda learned and lambda annealing without head selection in Tab.4?**\n\nThis is an interesting comment as there are many different possible combinations for our framework. We present results from the additional experiments to address your question.\n\n| Combination | all heads (no selection) | entropy-based selection |\n| :--- | :--- | :--- |\n| +learn, +anneal | 61.9 | 64.2 |\n| +learn, -anneal | 61.7 | 63.8 |\n| -learn, +anneal | 61.6 | 63.4 |\n| -learn, -anneal | 61.3 | 62.4 |\n\n- We run these experiments using Llama-3.2-3B model.\n- We report the average score for 11 natural language tasks shown in Tab. 1.\n- For all these configurations, we initialize the lambda value using the original DIFFT scheme, and control each component.\n\nIt is clear that combining both \"learnable lambda\" and \"lambda annealing\" yields the best performance regardless of the head selection.\n\nWe once again thank you for your insightful review. We have provided additional explanations and experiments which we believe address your concerns. We hope our response resolves your concerns and further strengthens your positive assessment of our work. Thank you.\n\n[1] Dubey, Abhimanyu, et al. \"The llama 3 herd of models.\" arXiv e-prints (2024): arXiv-2407."}, {"review_id": "m0Uk8wdwJN", "reviewer": "Reviewer_WJux", "summary": "The paper analyzes why Differential Transformers (DIFF) outperform standard transformers and proposes DEX, a lightweight, efficient extension for infusing differential attention's benefits into existing pretrained models without the need for training from scratch. The core claims are that DIFF attention enables negative attention scores (higher expressivity), reduces attention head redundancy, and improves learning dynamics. DEX is evaluated on various benchmarks, demonstrating improved performance with minimal data and computational overhead", "questions": "Please see weaknesses.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- Provide detailed insights to the effective of Differential Transformers.\n- Propose DEX - a novel lightweight alternative for DIFFT, which can leverage minimal data and computational overhead.  \n- Experiments are conducted on diverse benchmarks, showcasing the benefits of DEX.\n\nWeaknesses:\n- My main concerns lie on the experimental setup. Current experiments use a **custom-mixed dataset**, not the original pretraining data for models like Llama or Qwen. Thus, the absolute improvement and generalization to other training regimes or more realistic finetuning settings are somewhat unclear. Besides, **direct comparisons to original DIFF Transformer are limited**. The authors should provide the performance of DIFF on the main language modeling benchmark (different models and scope), and other tasks (Key IR, In-Context Learning). \n\n- **Ambiguity in Head Selection Strategy**: The paper proposes two head selection strategies—low importance and high entropy—but lacks detailed comparative results clearly indicating why one strategy is preferable. Additionally, practical guidelines or robustness analyses concerning the selection of heads across different tasks and models would improve usability.\n\n- **Limited Theoretical Justification**: The paper utilizes empirical observations and visualizations (e.g., attention entropy, redundancy) to interpret the Differential Transformers. However, it provides limited theoretical insights to justify why the differential attention approach outperforms standard attention mechanisms beyond empirical observations. A deeper theoretical exploration or formal analysis would significantly enhance understanding.\n\n- **Potentially Overstated Generalization Claims:** The claim of significant gains using minimal adaptation data (<0.01%) is strong but could be overstated. Further analysis is needed regarding whether such minimal training consistently delivers strong performance across a broader range of datasets and scenarios.", "was_revised": true, "final_justification": "The author has addressed my concerns about the custom dataset used, the lack of comparison with DIFFT, the robustness analysis on head selection. I do think their reply is detailed enough. While I appreciate their efforts to ensure a fair comparison given the limited computing resources, I still believe that additional large-scale experiments for DIFFT and DEX would further strengthen their conclusions regarding the significance of DEX, and should be included in the revised version. Given that, I increase my score to 4.", "rebuttal": "We deeply appreciate your thorough review and constructive comments. We are glad to hear your positive assessment of our paper on \"providing detailed insights\", \"proposing a novel method\" and \"conducting diverse experiments\". We are happy to address your concerns through this rebuttal.\n\n**Concern 1a: Experiments use custom-mixed dataset, not the original pretraining data, which makes the absolute improvement unclear.**\n\nThis is a very relevant feedback, and we agree that this setting can potentially complicate the evaluation. However, we would be grateful if you understand that the original pretraining dataset for Llama or Qwen models are **not publicly available**, necessitating the use of custom dataset. Therefore, in order to minimize the confounding effect of the train dataset, we ran all other baselines (full tuning, LoRA, etc) on the exact same corpus. Full fine-tuning (FT) corresponds to 'continued pretraining' that maximally fits the model to the additional training data, while LoRA represents more robust finetuning method at the other end of the spectrum that preserves knowledge from the original training data. The fact that DEX consistently outperforms both methods testifies for its fundamental advantage. We ran comprehensive experiments using 5 model variants and 11 benchmarks, and DEX has delivered robust performance gains across most settings.\n\nAlso, **Tab.7** presents DEX applied to instruction tuning setting, where we start from a Llama-3.1-8B model checkpoint instruct-tuned on OpenHermes-2.5 dataset (open-source) then apply DEX and other baselines using the exact same OpenHermes-2.5 dataset. This further eliminates the effect of training data gap, robustly demonstrating the advantage of DEX.\n\n**Concern 1b: Direct comparison to original DIFFT (language modeling, Key IR, ICL) is missing.**\n\nWe believe this is a valid feedback. First, we want to gently mention that we presented 11 downstream natural language modeling benchmark scores in **Tab.8**, where we compare the performance of Llama model, DIFFT, DEX from scratch, and DEX from Llama checkpoint (our proposed adaptation setting) all trained on the exact same dataset (see Appendix F) along with qualitative CKA results (**Fig.19**). To further address your concerns, we ran additional evaluations on key information retrieval (KIR) and in-context learning (ICL) for these models, and present the results below:\n\n| KIR | 256 | 512 | 768 | 1024 | 1280 | 1536 | 1792 | 2048 | Avg |\n|---|---|---|---|---|---|---|---|---|---|\n| Llama | 0.09 | 0.05 | 0.04 | 0.06 | 0.11 | 0.04 | 0.02 | 0.06 | 0.059 |\n| DIFFT | 0.12 | 0.07 | 0.10 | 0.09 | 0.13 | 0.08 | 0.05 | 0.11 | 0.094 |\n| DEX | 0.11 | 0.13 | 0.08 | 0.12 | 0.08 | 0.07 | 0.07 | 0.08 | 0.093 |\n| DEX-S | 0.14 | 0.09 | 0.10 | 0.18 | 0.19 | 0.11 | 0.10 | 0.12 | **0.129** |\n\n| ICL | n=1 | 5 | 10 | 15 | 20 | 30 | 40 | 50 | AVG |\n|---|---|---|---|---|---|---|---|---|---|\n| Llama | 0.074 | 0.096 | 0.178 | 0.200 | 0.170 | 0.193 | 0.185 | 0.156 | 0.156 |\n| DIFFT | 0.074 | 0.141 | 0.148 | 0.156 | 0.207 | 0.193 | 0.207 | 0.207 | 0.167 |\n| DEX | 0.074 | 0.104 | 0.185 | 0.200 | 0.200 | 0.200 | 0.193 | 0.193 | **0.169** |\n| DEX-S | 0.067 | 0.119 | 0.185 | 0.193 | 0.185 | 0.193 | 0.200 | 0.193 | 0.167 |\n\n- DEX-S stands for DEX trained from scratch with the exact same training data as others.\n- Columns in KIR correspond to the context length, averaged across 5 different needle depths (0, 25, 50, 75, 100%) as in Fig.10.\n- Columns in ICL represent number of shots. We average the scores on three benchmarks used in Tab.3, Trec, Banking-77 and Clinic-150.\n\nIt is evident from these results that DEX's performance is comparable to or often better than the original DIFFT (despite being much more efficient both during training and inference), consistently outperforming Llama.\n\n**Concern 2a: Detailed comparative results for head selection strategies are required.**\n\nAlthough we do have comparison for head selection strategies at **Tab 4**, we agree that supplementing experiments on this following your feedback would be beneficial. Hence, we present additional results below, under \"concern 2b\".\n\n**Concern 2b: Practical guidelines or robustness analyses on head selection across tasks and models would improve usability.**\n\nIncluding practical guidelines or robustness analyses is a great idea to improve the usability of our method in real life. We appreciate your constructive feedback, and share two relevant results from our additional experiments: (1) head selection across model variants and (2) head selection across downstream tasks.\n\n| Model | entropy | importance | all heads (no selection) |\n| :--- | :--- | :--- | :--- |\n| Llama-3B | **64.2** | 63.9 | 61.9 |\n| Llama-1B | **55.6** | 55.1 | 54.0 |\n| Qwen-1.5B | **62.4** | 62.0 | 61.6 |\n| Qwen-0.5B | **54.4** | **54.4** | 52.8 |\n\n- The scores are averaged across 11 widely used natural language benchmarks presented in Tab.1.\n\n| Task | entropy | importance | all heads (no selection) |\n| :--- | :--- | :--- | :--- |\n| Broad Knowledge | **63.1** | **63.1** | 62.5 |\n| Instruction Following | **57.2** | 55.4 | 54.9 |\n| Math | **74.3** | **74.3** | 70.6 |\n| Science | 77.7 | **77.8** | 75.8 |\n\n- We run this experiment with Llama-3.1-8B model instruct-tuned on Open-Hermes-2.5 dataset, as detailed in Appendix C.\n- We categorize MMLU, IFEval, GSM8k and Arc-C as broad knowledge, instruction following, math and science task, respectively.\n\nIt is evident from these results that both head selection strategies perform robustly across different models and tasks (data), yielding notable improvements over the 'no selection' baseline. We would recommend using entropy-based selection for most practical uses, which delivers the best performance in most of the cases. We will include these results upon revision.\n\n**Concern 3: Theoretical justification for differential attention’s empirical advantage is limited.**\n\nDifferential Transformer is a new architecture that empirically delivers strong performances. Conceptually, the introduction of DIFF will look like \n\n”Different architecture” → “Different internal patterns/mechanisms” → “Different (better) downstream results.”\n\nHowever, the fact that little is known about the *“internal patterns/mechanisms”* of DIFFT makes this a “missing link” in our understanding of the framework, which motivates our work.\n\nOur main contributions are two-folds:\n\n1. Provide insightful analyses that identify what internal mechanisms actually make differential transformers effective (expressivity of real-valued attention, head diversity, better learning dynamics)\n2. Using the identified empirical characteristics as our inspiration, propose a data- and compute-efficient adaptation framework that injects DIFFT’s benefits (stronger language understanding, key information retrieval, in-context learning, etc) to pretrained LLMs without training from scratch (unlike the original paper)\n\nUnfortunately, providing theoretical justification for “why DIFFT *should* perform better” is out of our scope, though that will be an impactful topic for future research. We humbly ask for your understanding on this matter, and hope you find our contributions interesting and meaningful. \n\n**Concern 4: Analysis with a broader range of datasets and scenarios is needed to claim significant gains using minimal adaptation data.**\n\nIn **Tab.1**, we show DEX outperforming baselines applied to 5 different models with <0.01% original training data. In **Tab.2-3** and **Fig.10**, we demonstrate that DEX can improve both key-information retrieval and in-context learning capabilities with minimal (<0.01%) training data. We believe our evaluation at this point is comprehensive, but might not be \"general enough\" as these results all belong to DEX applied to different “pretrained” models using additional “pretraining data”. \n\nIn  **Tab.7**, we present two different scenarios: DEX applied to “pretrained” model using “instruction data” and DEX applied to “instruct-tuned” model using “instruction data” (continuous instruction tuning). We evaluate these scenarios using a wide variety of benchmark datasets spanning instruction following, scientific reasoning, math and coding, where DEX delivers impressive overall performances. We want to note that the training dataset for these experiments (Open-Hermes-2.5) spans roughly 350M tokens, which is much less than 0.01% of the 15T+ original training tokens for Llama-3 model [1]. Using DEX, it takes less than a day to tune an 8B model with a single A100 node, which further supports DEX's effectiveness under minimal training.\n\nHence, we have provided empirical results on 5 models, 3 scenarios (pretrain+pretrain, pretrain+instruct, instruct+instruct), and over 20 benchmarks (11 language modeling, 6 instruction, 3 ICL, KIR, etc). We acknowledge that extensive evaluation is indeed needed to assess generalization, but we would be grateful if you kindly consider the depth and breadth of our evaluation for this matter.\n\nWe once again thank you for your thorough and helpful review. We have provided additional explanations and experiments which we believe address your concerns. We hope our response resolves your concerns and steers your assessment of our work in a positive direction. Thank you.\n\n[1] Dubey, Abhimanyu, et al. \"The llama 3 herd of models.\" arXiv e-prints (2024): arXiv-2407."}, {"review_id": "WWqyzX3UUl", "reviewer": "Reviewer_HPZ6", "summary": "This paper presents two main contributions. Firstly, it provides an investigation into Differential Transformers, including their internal workings, the phenomenon of Negative Attention, reductions in attention head redundancy, and their training dynamics. Secondly, the authors introduce DEX, a transformer architecture that, while conceptually related to Differential Transformers, is still different. DEX offers a one advantage over Differential Transformers: its applicability to pretrained models. Through extensive experimentation on various pretrained transformer models, the authors demonstrate DEX's superior performance across multiple tasks, outperforming LoRA, full fine-tuning, and other common fine-tuning strategies. The paper is well-supported by a thorough ablation study and offers valuable initial insights into the characteristics of models fine-tuned with DEX.", "questions": "Q1: Regarding Figure 8a, the trend seems to indicate that training for more steps benefits LLaMA more than the Differential Transformer, potentially leading to a better final loss for LLaMA. Could the authors train for a longer duration and report these extended results to explore whether a regular Transformer architecture might ultimately be superior?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nW1: The paper provides detailed investigation of Differential Transformers, illustrating the components learned by this architectural design. This contributes to understanding the inner workings of this architecture.\n\nW2: The proposed DEX architecture is novel and appears to outperform existing fine-tuning approaches.\n\nW3: The ablation study is both important and well-executed.\n\nW4: The paper is clear and well-structured.\n\nWeaknesses\n\nS1: The main weakness here is that the DEX architecture is actually different from Differential Transformers, and it's not truly clear that DEX performs as Differential Transformers do. I think you should first compare Differential Transformer and DEX when trained from scratch, as was done for Section 2 when comparing to LLaMA. Then, assuming DEX and Differential Transformers behave similarly, you can apply DEX to pretrained models. Training DEX from scratch, mirroring how LLaMA and Differential Transformers were handled for Section 2, would be very beneficial.\n\nS2: A more minor weakness is that throughout the paper, all the legends in the figures are very small and very hard to read. The authors should fix this for improved clarity.", "was_revised": true, "final_justification": "I've reviewed the rebuttal and the discussions with the authors, and my score remains unchanged.", "rebuttal": "We deeply appreciate your thoughtful review and constructive feedback. We are pleased to hear that you found our work insightful, novel, well-executed and well-structured. We hereby address your concerns one by one.\n\n**Concern 1: Analyzing DEX-from-scratch as done in Sec.2 to confirm its internal resemblance to Differential Transformer would be beneficial.**\n\nWe believe this is a very insightful feedback. To provide some context for our original thinking, we did not believe side-by-side comparison between DEX and Differential Transformer (like in Sec.2) was *necessary* because our primary goal for DEX was not to replicate all the internal mechanisms of DIFF. Rather, it was to efficiently inject the downstream strengths of DIFF (as evidenced by natural language tasks, information retrieval, etc) to pretrained transformer models for practical purpose using our learned insight as design inspiration, though we do believe that there is strong correlation between the \"internal mechanism\" and the \"downstream performance\".\n\nThat being said, we agree that including the analysis would indeed add clarity and completeness to our claims about DEX. Below we present 3 key analyses introduced in Sec.2, namely (1) attention score correlation (**Fig.1**), (2) negative attention ratio (**Fig.3**), and (3) attention head CKA (**Fig.5**). We ask for your understanding on presenting these results in raw numbers, as communication through image is no longer possible.\n\n| Layer | Correlation (top-5%) | Correlation (all) | Rank Correlation (top-5%) | Rank Correlation (all) |\n| :--- | :--- | :--- | :--- | :--- |\n| 1 | 0.12 | 0.46 | 0.07 | 0.34 |\n| 2 | -0.09 | 0.14 | 0.14 | 0.47 |\n| 3 | -0.07 | 0.22 | 0.12 | 0.36 |\n| 4 | 0.09 | 0.36 | -0.07 | 0.21 |\n| 5 | 0.21 | 0.36 | 0.04 | 0.14 |\n| 6 | 0.24 | 0.36 | 0.09 | 0.16 |\n| 7 | 0.52 | 0.50 | 0.30 | 0.14 |\n| 8 | 0.58 | 0.52 | 0.39 | 0.18 |\n| 9 | 0.17 | 0.28 | -0.03 | 0.14 |\n| 10 | 0.18 | 0.17 | -0.08 | 0.05 |\n| 11 | 0.55 | 0.41 | 0.27 | 0.08 |\n| 12 | 0.10 | 0.37 | -0.26 | -0.05 |\n| 13 | 0.48 | 0.18 | 0.15 | -0.02 |\n| 14 | 0.21 | 0.21 | -0.16 | -0.05 |\n| 15 | 0.18 | 0.19 | -0.21 | -0.05 |\n| 16 | 0.26 | 0.01 | -0.09 | -0.02 |\n| **Average** | **0.23** | **0.30** | **0.04** | **0.13** |\n\nThe above shows the correlation (Pearson's) and rank correlation between the two *group*'s attention scores, measured on our 0.4B DEX model trained from scratch. We applied DEX to all attention heads because we were training from scratch anyways, and computed *effective attention scores* for this analysis as detailed in **Appendix F.3**. Note that this experimental setup applies to all three tables presented here.\n\nIt is clearly visible from the table that just like in **Fig.1**, DEX-from-scratch shows lower correspondence (overlap) on salient tokens with the highest attention scores, potentially indicating a form of selective filtering of common noise.\n\nNext, we present layer-wise positive/negative ratio for the attention scores of DEX.\n\n| Layer | Positive | Negative |\n| :--- | :--- | :--- |\n| 0 | 0.69 | 0.31 |\n| 1 | 0.61 | 0.39 |\n| 2 | 0.59 | 0.41 |\n| 3 | 0.62 | 0.38 |\n| 4 | 0.56 | 0.44 |\n| 5 | 0.59 | 0.41 |\n| 6 | 0.56 | 0.44 |\n| 7 | 0.59 | 0.41 |\n| 8 | 0.53 | 0.47 |\n| 9 | 0.52 | 0.48 |\n| 10 | 0.52 | 0.48 |\n| 11 | 0.50 | 0.50 |\n| 12 | 0.50 | 0.50 |\n| 13 | 0.51 | 0.49 |\n| 14 | 0.51 | 0.49 |\n| 15 | 0.51 | 0.49 |\n\nThis demonstrates that like Differential Transformer (in **Fig.3**), DEX induces a significant proportion of *negative attention scores* once learned. Our qualitative inspection indicates the formation of a \"more expressive attention mechanism\" thanks to this increased flexibility, as was the case with Differential Transformer (**Sec.2.2**). \n\nLastly, we show the head-CKA result that quantifies the redundancy between attention heads.\n\n| layer | Llama | DIFFT | DEX |\n| :--- | :--- | :--- | :--- |\n| 0 | 0.46 | 0.51 | 0.39 |\n| 1 | 0.41 | 0.23 | 0.38 |\n| 2 | 0.43 | 0.31 | 0.39 |\n| 3 | 0.45 | 0.38 | 0.37 |\n| 4 | 0.43 | 0.30 | 0.37 |\n| 5 | 0.38 | 0.47 | 0.33 |\n| 6 | 0.45 | 0.43 | 0.32 |\n| 7 | 0.40 | 0.31 | 0.34 |\n| 8 | 0.45 | 0.29 | 0.31 |\n| 9 | 0.56 | 0.33 | 0.35 |\n| 10 | 0.37 | 0.36 | 0.37 |\n| 11 | 0.32 | 0.35 | 0.38 |\n| 12 | 0.44 | 0.39 | 0.33 |\n| 13 | 0.57 | 0.43 | 0.36 |\n| 14 | 0.46 | 0.44 | 0.43 |\n| 15 | 0.62 | 0.52 | 0.47 |\n| **Average** | **0.45** | **0.38** | **0.37** |\n\nWe present the average head-CKA result for Llama, DIFFT and DEX (also trained from scratch), where lower score indicates \"less redundancy\" among the attention heads. It is clearly visible that both DIFFT and DEX show notably reduced attention head redundancy when compared to Llama model. We believe this enables both DIFFT and DEX to form richer representation that encodes a wider spectrum of information. Note that we compute pairwise head CKA with rbf kernel and average the scores for each layer.\n\nThese three analyses consistently highlight the mechanistic similarity between DEX and DIFF, revealing shared properties in three key areas:\n\n1. Selective filtering of attention noise;\n2. Improved expressivity through negative attention; and\n3. Reduced attention head redundancy.\n\n**Concern 2: The legends in the figures are small and hard to read.**\n\nWe apologize for this. We humbly accept your feedback and we will make sure to enlarge the legends for our figures upon revision.\n\n**Question 1: Could the authors train for a longer duration and report these extended results to explore whether a regular Transformer architecture might ultimately be superior?**\n\nThe original result that compares Differential Transformer and Llama model is introduced in **Tab.8**. To answer your question, we have run continued pretraining on the four models (Llama, DIFF, DEX, DEX-from-scratch) for additional 10k iterations (approximately 6B train tokens). We present the average scores for 11 natural language tasks used in **Tab.1** that includes Arc-C, Arc-E, BoolQ, Hellaswag, etc.\n\n| Model | Average (Tab.8) | Average (+10k steps) |\n| :--- | :--- | :--- |\n| Llama | 42.4 | 42.5 |\n| DIFF | 42.9 | 42.9 |\n| DEX | 42.8 | 42.8 |\n| DEX-scratch | 42.8 | 42.9 |\n\nWe can see that additional training gives a slight improvement on the downstream scores for Llama and DEX-scratch, but the change is insufficient to close the performance gap with DIFF. Nevertheless, we believe it was a highly relevant question to check the final performance after additional training, given the trend in Fig.8a.\n\nWe once again thank you for your insightful and constructive review. We have provided additional explanations and experiments which we believe address your concerns. We hope our response resolves these points and further strengthens your positive assessment of our work. Thank you."}]

Author rebuttal:

Meta-review: This paper investigates the Differential Transformer, whose strong empirical performance is often linked to noise-canceled attention but remains theoretically underexplored. The authors identify three factors behind its success–enhanced expressivity via negative attention, reduced redundancy among heads, and improved learning dynamics. Building on these insights, they propose DEX, a lightweight method that incorporates differential attention into pretrained language models by reusing softmax scores and applying a simple differential operation on the value matrix. DEX is efficient at both training and inference and achieves gains across diverse benchmarks.

While there are some limitations–most notably the absence of large-scale experiments for DIFFT and DEX–the reviewers agree that the paper’s in-depth analysis of differential transformers is insightful, and the proposed DEX method is novel. Since all reviewers have unanimously recommended acceptance with strong support from Reviewer WqWH, I likewise recommend acceptance of this paper.
Decision: Accept (poster)