Differential Transformer

Authors: Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, Furu Wei
Primary area: foundation or frontier models, including LLMs
Keywords: sequence modeling, language models, model architecture, transformer

Abstract:
Transformer tends to overallocate attention to irrelevant context. In this work, we introduce Diff Transformer, which amplifies attention to the relevant context while canceling noise. Specifically, the differential attention mechanism calculates attention scores as the difference between two separate softmax attention maps. The subtraction cancels noise, promoting the emergence of sparse attention patterns. Experimental results on language modeling show that Diff Transformer outperforms Transformer in various settings of scaling up model size and training tokens. More intriguingly, it offers notable advantages in practical applications, such as long-context modeling, key information retrieval, hallucination mitigation, in-context learning, and reduction of activation outliers. By being less distracted by irrelevant context, Diff Transformer can mitigate hallucination in question answering and text summarization. For in-context learning, Diff Transformer not only enhances accuracy but is also more robust to order permutation, which was considered as a chronic robustness issue. The results position Diff Transformer as a highly effective and promising architecture for large language models.

Decision: accept (oral)
Reviewer scores: 8, 8, 8, 8

Reviews and discussion:
<thread>
creation date:
1730687579654

thread_type:
official_review

<thread_object>
creator: reviewer_txcs

<content>
summary:
The paper introduces a new Transformer variant, called Diff Transformer. The key idea is in the design of attention mechanism, where Diff Transformer uses two softmax attention functions to cancel out potential attention noises. Through a suite of empirical studies, Diff Transformer show promising performances compared to standard Transformer architecture.

strengths:
- The paper introduces a simple architecture tweak to mitigate the observations that Transformer models tend to incorrectly allocate excessive attentions to irrelevant contexts.
- The paper demonstrates promising empirical results of the proposed architecture through a decent suite of evaluations, ranging from language modeling capability that fits well with scaling law, in-context learning, to improving long-context capability and mitigating contextual hallucination.
- I appreciate that the authors include a variety of downstream tasks to showcase how addressing attention noises in Transformer models can lead to performance improvements.

weaknesses:
- Related work can be discussed more thoroughly in the paper. For example, [1] discusses that Transformer models tend to mis-allocate attention to irrelevant contexts potentially biased by their position within the context [1], and [2] also shows LLMs can be easily distracted by irrelevant contexts.
- While the proposed Diff Transformer shows promising performances, they have to be trained from scratch which can be computationally expensive compared to post-training approaches that mitigate attention noises in standard Transformer models, such as the calibration technique used in [1], and improved decoding approaches used in [3] and [4]. Currently, vanilla Transformer is the only baseline considered in the paper. However, I would appreciate if the authors can show how Diff Transformer compares to other more lightweight techniques that fix Transformer's attention noises.
- Apart from pretraineing Diff Transformer, there is no further instruction tuning experiments included in the current paper. I would be interested in seeing whether instruction tuned Diff Transformer also shows better performances than standard instruction tuned Transformers.


[1] Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization. Hsieh et al. 2024.

[2] Large Language Models Can Be Easily Distracted by Irrelevant Context. Shi et al. 2023.

[3] Trusting Your Evidence: Hallucinate Less with Context-aware Decoding. Shi et al. 2023.

[4] Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Maps. Chuang et al. 2024.

questions:
- Can the authors elaborate further on why we need a fixed multiplier $(1 − \lambda_{init}) $ in Diff Transformer (Eq. 3)?
- In section 3.7, Is absmax quantization the suitable approach to adopt in this context? Why not consider quantization approaches that are robust to activation outliers, such as LLM.int8()?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730704190410

thread_type:
official_review

<thread_object>
creator: reviewer_cgmd

<content>
summary:
This paper proposes a new model architecture called Differential Transformer (DIFF Transformer), which reduces noise and more accurately focuses on relevant context through a differential attention mechanism. In a series of language modeling experiments, DIFF Transformer outperforms the standard Transformer across various tasks and model sizes. The paper demonstrates the superiority of this method in long-context modeling, key information retrieval, hallucination mitigation, in-context learning, and reduction of activation outliers.

strengths:
1.The proposed DIFF Transformer introduces an innovative differential attention mechanism that reduces attention noise through the difference between two independent softmax mappings. This approach performs exceptionally well in long-context processing, making a significant improvement for long-text tasks.
2.DIFF Transformer demonstrates strong performance in language modeling, in-context learning, and multi-needle retrieval tasks, particularly outperforming traditional Transformers in long-context scenarios. It greatly enhances the accuracy and robustness in practical applications like question answering and text summarization, showing broad potential for various use cases.
3.The paper provides extensive experimental validation, confirming DIFF Transformer’s stability across different tasks and model parameter settings. The hyperparameter tuning is thorough, ensuring both robust performance and reliable results.
4.The paper is well-organized, with clear illustrations and analogies that present the differential attention mechanism effectively. Detailed experimental steps make it easy to understand the innovative contributions.

weaknesses:
While the differential attention mechanism brings notable performance improvements, it also adds computational complexity. Efficiency tests indicate that DIFF Transformer’s throughput, particularly with multi-head normalization and dual softmax calculations, is slightly lower than that of traditional Transformers (by about 5%-12%). Although this impact is relatively minor in the experiments, further exploration of computational efficiency would be valuable if scaling to large-scale applications.

questions:
I don't have questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730896501398

thread_type:
official_review

<thread_object>
creator: reviewer_qnsj

<content>
summary:
The paper introduces Differential Transformer, a novel architecture for LLMs that enhances attention to relevant context while canceling out noise. The core innovation is a differential attention mechanism that calculates attention scores as the difference between two softmax attention maps, promoting sparse attention patterns and reducing attention noise. The paper demonstrates through extensive experiments that the DIFF Transformer outperforms the standard Transformer in various aspects, including scaling model size, training tokens, long-context modeling, key information retrieval, hallucination mitigation, in-context learning, and reduction of activation outliers.

strengths:
1. this is a solid and well-written paper. 
2. while the technique itself is not complex, as far as I know, this is the first work to propose a new architecture using differential attention. 3. the experiments and conclusions in this paper are thorough and insightful.

weaknesses:
1. the authors mention "promoting the emergence of sparse attention patterns" multiple times on Lines 13, 160, and 539, but do not provide statistics and quantification of the sparsity of attention distribution between DIFF Transformer and the general Transformer.
2. The authors did not discuss or conduct experimental comparisons with work related to the sparse attention. 
For example: 
> Efficient Content-Based Sparse Attention with Routing Transformers
>
> Generating Long Sequences with Sparse Transformers.
3. It would be even better if the effectiveness of the DIFF Transformer could be validated on image or speech modalities.
4. The absence of a *Related Work* section limits the paper's ability to contextualize its contributions within the broader field.

questions:
1. Do both attention calculations in differential attention use RoPE positional encoding?
2. Similar to multi-query attention, can $K_1$, $K_2$, or $Q_1$, $Q_2$ share parameters?
3. Is there a significant difference in sparsity between DiffAttn and regular attention? Does the first term of DiffAttn resemble the pattern of standard attention (implying that the second negative attention term serves to cancel out noise)?
4. What is the calculation formula for Table 3? (Specifically, what is the exact normalization operation?)
5. Please discuss the relationship between differential attention and sparse attention.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731216633496

thread_type:
official_review

<thread_object>
creator: reviewer_hbkx

<content>
summary:
The paper aims to mitigate the problem in standard softmax attention to over-allocate attention weights to irrelevant context. The proposed Diff Transformer uses the difference between two separate softmax attention scores to cancel noise assigned to irrelevant contextual tokens. Concretely, Diff Transformer first partition the query and key vectors into two groups to compute separate softmax attention scores. The final attention score is the subtraction of the two groups. One learnable parameter $\lambda$ is introduced to balance the two groups of attention scores.

Experiments were conducted on a 3B-parameter model trained on 350B data tokens. Diff Transformer outperforms Transformer on training loss and downstream benchmarks. Ablation studies investigate the importance of the RMSNorm on each attention head.

strengths:
1. The proposed modification of attention in Diff Transformer is well-motivated

2. The experimental results are strong, with large-scale experiments up to 3B-parameter models and 350B data tokens. 

3. The pre-trained model was evaluated on multiple benchmarks, and also on long-context evaluation, retrieval-oriented tasks, many-shot in-context learning and hallucination evaluation.

4. The paper is well-written, easy to follow.

weaknesses:
Some design motivation in the model is still not clear:

1. Why the learnable $\lambda$ is re-parameterized in Eq (2)?

2. Why in Eq (3) there is a term $(1 - \lambda_{init}$ for each head?

3. Why the RMSNorm for each head is so important for Diff Transformer stability? The explainable in section 3.8 is unconvincing to me.

questions:
NA
</content>
</thread_object>

</thread>
<thread>
creation date:
1732528812952

thread_type:
public_comment

<thread_object>
creator: ~minh_nguyen_hoang1

<content>
title:
Comparison with cosine-based transformer

comment:
Dear authors, have you considered comparing the performance of your work to cosine-based transformer. While the theoretical motivation coming from the differential amplifiers used in signal processing, the differential attention has the output range of $[-\lambda, 1]$, in contrast to the normal softmax-based attention only support positive similarity. I would love to see some comparision results between your proposed method and previous cosine-based attention methods or any discussion on how this method can surpassed cosine-based methods.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732755022467

thread_type:
public_comment

<thread_object>
creator: ~xinyu_zhao3

<content>
title:
Implementation details of Signal-to-Noise Ratio

comment:
Dear authors, for calculating the Signal-to-Noise Ratio in Fig1 and Attention Noise in Table 3, could you please explain the attention weights come from which tokens? E.g. the query tokens/the first generated token/the first answer-relevant token/all answer tokens. Will there be any difference in Attention Noise for different tokens?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733314140225

thread_type:
public_comment

<thread_object>
creator: ~kebin_wu2

<content>
title:
Other option to cancel noise

comment:
Dear authors, 

Thank you for this interesting paper.  I have two questions and would appreciate if you can provide your insights on these. 
1.   since the attentions on the irrelevant tokens are considered as noise,  why don't we use addition instead of subtraction to do the noise cancellation?   the motivation is that we can improve the signal to noise ratio with the simple averaging in the field of signal processing, where random noise after adding is still low, but the signal itself can be enhanced?

2.  The differential transformer proposed here is validated with decoding-only LLM in this paper if I am not mistaken,  have you tried applying this to encoder-based models, such as the text encoder used in the CLIP model? Thank you.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a simple (in principle) modification to the attention mechanism, coined differential attention that allows to improve the signal to noise ration in the attention maps for model with very long context windows. The proposed approach is simple, plug-in to all modern transformer-based models and shows good performance on all tasks. Given the positive reviews and interesting and simple research findings, this paper is a clear accept. Given the broad impact (generic architecture improvement) this could be an oral presentation at ICLR 2025.