DSAS: A Universal Plug-and-Play Framework for Attention Optimization in Multi-Document Question Answering
Authors: Jiakai Li
Rongzheng Wang
Yizhuo Ma
Shuang Liang
Guangchun Luo
Ke Qin
Venue: NeurIPS 2025 poster ()
Keywords: large language model
attention dilution
attention optimization
Primary area: deep_learning

Abstract: While large language models (LLMs) show considerable promise across various fields, they have notable limitations in handling multi-document question answering (Multi-doc QA) tasks. The first challenge is long-range dependency modeling, where LLMs struggle to focus on key information in long texts, which weakens important semantic connections. Second, most LLMs suffer from the ''lost-in-the-middle'' issue, where they have difficulty processing information in the middle of long inputs. Current solutions either truncate global dependencies or demand costly finetuning, ultimately lacking a universal and simple solution for these challenges. To resolve these limitations, we propose Dual-Stage Adaptive Sharpening (DSAS) containing two modules. (i) The Contextual Gate Weighting (CGW) module alleviates ''lost-in-the-middle'' by assessing paragraph relevance through layer-wise attention tracking and position-aware weighting. (ii) The Reciprocal Attention Suppression (RAS) module enhances focus on critical paragraphs by suppressing information exchange between key and irrelevant texts, thus mitigating the limitations in long-range dependency modeling. Extensive experiments on four benchmarks demonstrate DSAS's efficacy across mainstream LLMs (Llama, Qwen, Mistral, and Deepseek), with an average F1-score improvement of 4.2% in Multi-doc QA tasks on Llama-3.1-8B-Instruct and Qwen2.5-14B-Instruct. Ablation studies confirm the essential contributions of both the CGW and RAS modules. In addition, detailed discussions in the Appendix further validate the robustness and scalability of DSAS.

DSAS: A Universal Plug-and-Play Framework for Attention Optimization in Multi-Document Question Answering
Jiakai Li', Rongzheng Wang', Yizhuo Ma²,
Shuang Liang', Guangehun Luo², Ke Qin
IIstitute of Intelligent Computing. University of Electronic Science and Technology of China, Chengdu, China Schooo of Information and Software Engineering, University of Electronic Science and Technology of China, Chengdu, China 1jkēstd. uestc. edu. en, qinkeQuestc edu.
Abstract
While large language models (LLMs) show considerable promise across various fields, they have notable limitations in handling multi-document question answer- ing (Multi-doc QA) tasks. The first challenge is long-range dependency modeling, where LLMs struggle focus on key information in long texts, which weakens important semantic connections. Second, most LLMs suffer from the "lost-in-the- middle" issue, where they have difficulty processing information in the middle of long inputs. Current solutions either truncate global dependencies or demand costly finetuning, ultimately lacking universal and simple solution for these challenges. To resolve these limitations, we propose Dual-Stage Adaptive Sharpen- ing (DSAS) containing two modules. (i) The Contextual Gate Weighting (CGW) module alleviates "'ost-in-the-middlle" by assessing paragraph relevance through layer-wise attention tracking and position-aware weighting. (ii) The Reciprocal Attention Suppression (RAS) module enhances focus on critical paragraphs by suppressing information exchange between key and irrelevant texts, thus mitigating the limitations in long-range dependency modeling. Extensive experiments on four benchmarks demonstrate DSAS's efficacy across mainstream LLMs (Llama, Qwen, Mistral, and Deepseek), with an average Fi-score improvement of 4.2% in Multi-doc QA tasks on 1aama-3.1-B--nmae tion studies confirm the essential contributions of both the CGW and RAS modules. addition, detailed discussions the Appendix further validate the robustness and scalability of DSAS.
and Qwen2.5-14P Ins-tuutruct. Abla-
1 Introduction
Transformer- (34] large language models (LLMs) have demonstrated remarkable performance, which have extensively promoted various complex natural language processing applications [5] 30 Building on the progress, recent advancements have shifted research focus toward enhancing LLMs' long-context processing capabilities 23) giving rise to LLMs that significantly expand context windows from 4K tokens to 128K or even 1M tokens, e.g.,
37
[10) and
Gemini-1.5 31). These LLMs have unlocked unprecedented potential for complex tasks requiring cross-document reasoning, such as legal case analysis [8] and multi-source academic synthesis 7
*Corresponding author.
Code is available at http::///gtthbb.cc//rrrareeeteet
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Input Paragraphs
LLM
LLM-DSAS
or artias openstnng ig the illagaa about parr narthem
Parograph 1: Bllidd
Decoder Layer
prosent-day
wrttiee and disected by l'ang. and laeed Chinnse witter Liu Qingbang a short novel "Shen MuSacre Woot".
Inpat
Decoder Layes ers
Ottput
Information
?
Paregrph The Musk of Pa Munchu ava 1802 pre-Code adventtree film directed (hhdtee Brabin, writin Larse Kutc, Edgur Allan Wosll Rahmer Manchi, degeoved duugloxe
Incomation suppeesioo
Willard based
Long range Modeling
Myma his Mancho's Cerghis Khon, Lewis from
Pangrugh with noise
Spacse attention
quont for Stoae plays nenests, Dr.
Lost in- the middle
Question: Which film came out fiss, Blird Shuck The Mask Of
Puragmpt I...
Pangrrph
Supporting
penograhh
parogrehh
Queston
d'arpet
In Multi-doc QA tasks, directly processing long inputs comprising numerous paragraphs with LLMS presents two major challenges: long-range dependency modeling and "lost-in-the-me tiiddde" resulting in degraded answer quality. DSAS acts plug-in that enhances LLMs through dual- stage process: (i) reinforcing information flow between supporting passages and both the question and target, and (ii) suppressing interactions between supporting and negative paragraphs.
Figure 1:
However, simply concatenating multiple documents into these long contexts often results in degraded performance due attention dilution (27]. phenomenon where critical inter-document dependencies are overshadowed by irrelevant tokens. While solutions like StreamingLLLM 40 and Selective Self-
attention 44 are introduced, they either truncate global dependencies or lack generalizability. leaving the core challenge of context-aware attention prioritization unresolved.
As shown in Figure T] the aforementioned attention dilution phenomenon reveals two critical lim- itations in multi nouti-doccmmntt question answering (Multi-doc QA) scenarios: (i) Limited long-range dependency modeling: Despite claims of 128K-token support, RULER [[S] reveals that LLMs often struggle with real-world tasks requiring combinatorial reasoning. While recent efforts (c.g.. StreamingLLM 40), LM-Infinite (13) explore attention mechanism optimization address the challenge, they sacrifice global token interactions, compromising Multi-doc QA performance, (ii) Persistent fost-innthhe-ee middee" issue: Nelson et (24] demonstrate that LLMs perform poorly when key information appears in the middle of long inputs. Current solutions like LongAlign ( using a hybrid strategy (combining long-instruction examples with short data) require additional training using curated datasets, making them less adaptable to mainstream LLMs. These challenges highlight the critical need for universal, plug-and-play modules that enhance Multi-doc QA capabilities without architectural constraints task-specific fine-tuning.
demonstrate that attention-driven information flow analysis provides critical insights into model reasoning patterns. Inspired by this insight, we adapt the methodology to Multi-doc QA tasks. To address these issues, we introduce Dual-Stage Adaptive Sharpening (DSAS) as shown in Figure [ training- frree attention optimization mechanism comprising two modules: Contextual Gate Weighting (CGW) and Reciprocal Attention Suppression (RAS). Specifically, our method works follows: CGW tracks attention scores across selected model layers and calculates contextual gate weight for each paragraph. Additionally, CGW introduces a position-aue nosition-aware weighting mechanism to enhance focus on information in the middle. Then RAS identifies critical paragraphs and attenuates information exchange between critical paragraphs and irrelevant content. DSAS requires architectural changes or extra finetuning, serving universal plug-in for Transformer- based LLMs to strengthen Multi-doc QA capabilities.
Recent studies [35
Our contributions are as follows: (i) We systematically investigate the information flows on several LLMs in Multi-doc QA through paragraph disparity level and answer quality level analysis. (ii) Based on the findings, we propose DSAS, a training-fiee free universal plug-in for Transformer- based LLMs, which enhances focus on critical information through CGW and RAS, while suppressing irrelevant content. (iii) Extensive experiments on four public benchmark datasets demonstrate the effectiveness of DSAS for various L.LMs including Llama, Qwen, Mistral and Deepseek, achieving an average Fl-score improvement of up to 4.2% on Llama- 1.1aa-3..--8 1--B-IIntrun [10] and Qwen2.5-
(28).

---

Results HotpotQA
Results HoPA
Results MuSiQue
1.4
2.44
1.4
Information flow
1.4
Information flow
Apr.o
Information flow
122
Zet
Tall
1.0 T
1.0
Zert
1.-
n.-
0.--
0.4-
0.2
29
155
Layer
Layer
Layer
Figure 2: Layer-war Layer-wise information flows of HotpotQA, 2WikiMultiHopQA and MuSiQue tested on Llama-3. 1-18--nnttuuct, p² and p" denote supporting paragraphs and negative paragraphs, respectively. The results of Qwen2.5-7B-IInstruct are shown in Appendix A.2
Information Flow Analysis on Multi-doc QA
Identifying key factors for effective Multi-doc QA reasoning is essential. To achieve this, conduct a systematic analysis of LLMs' inference processes across three core components: input paragraphs (p). question (q), and target (t). critical step lies in selecting suitable methods to study the semantic interactions among these components. Attention score analysis [29]. method widely used to examine information flow, adapted here to investigate how LLMS integrate cross-document information.
2.1 Preliminaries
Let A AW, denote the attention score matrix and the attention weight matrix of the h-th head in the (-th layer, respectively. We obtain the layer-specific matrices AS and AW by summing across all attention heads. Here, A! (i,j) represents the information flow from the j-th to the i-th token.
To analyze interactions among three key components: (i) paragraphs (p'..... p²): Each paragraph pd spans input token indices [p" -p"" + -..". -p"") ), where p²" and p" denote the start/end positions and C' is the total paragraph count; (ii) question q: and (iii) target t: the answer generation position (i.e., the final token in the input), we propose two metrics. Ipm and Ipm are the Top-K significance of the information flow from the m-th paragraph pd to and pd to t, respectively:
Σ - Top-K ((I 4M (EDDEEE Q
1)
Ipm .
JE4
Ipm Σ Top-K (1AM (t,j)lī E p m (L.....C}.
where Q denote the token length of the question. Ipm..g and Ipm_ serve as information flow indicators analyze the inference process for I.LMs. The larger their values are, the more the generated results pay attention to the corresponding paragraphs.
Experimental Settings. We choose -lama-3.1-8B-Intttee 10 and Qwen2.5-7B-Insttuuc
as our
primary models for investigation, due to their moderate model size and strong instruction following ability. For datasets, since LongBench lacks supporting facts (labels for supporting paragraphs), we use HotpotQA [1). 2WikiMultiHopOA [4] and MuSiQue [33]. We sample 1000 examples from the training set for evaluation. Templates for constructing inputs are provided in Appendix A.I
2.2 Paragraph Disparity Level Analysis
Most LLMs adopt a multi-layer Transformer architecture, with each decoder block processing semantic information differently during inference. We intend to check layer-wise information flows of Ipm.q .q and Ipm... Figure visualizes layer-wise attention weight values. These values serve as quantified metric to display how the different paragraphs contribute to the generated answer from the paragraph disparity perspective: Information flows from distinct paragraphs diverge only slightly in the shallow layers of the L.M,, which suggests that the L.LM first establishes basic semantic understanding to support deeper processing. As layers deepen, clear divergence emerges between
3

---

Ipr.g and Zpn.q- demonstrating the LLM progressively distinguishes task-relevant paragraphs through its layer-wise semantic processing, enabling rational attention allocation. Ultimately, in the deep layers of the I.I,, L.M recognizes and utilizes key paragraphs for answer generation.
Our analysis uncovers a two-stage reasoning pattern: (i) Information initially converges the question, with supporting paragraphs exhibiting stronger information flows than negative paragraphs, confirming the model's ability to prioritize semantically relevant content. (ii) Subsequently, the infor- mation from supporting paragraphs aggregates to the target, where the LLM strategically leverages critical paragraphs to formulate answers. These observations highlight the inherent interpretability of LLMs. proving their capabilities to integrate task-specife information during inference.
2.3 Answer Quality Level Analysis
We define a reasoning process as GOOD its output exactly matches the reference answer, and BAD if it con- tains no word-level overlap with the reference answer (see Appendix A.4 for more details). To investigate the divergence between good and bad rea- soning patterns, we aggregate infor- mation flow values across all model layers and conduct a comparative anal- ysis of two distinct groups: Ip and Ip- from supporting paragraphs, along with Iy" and In- , from negative paragraphs. As shown in Figure good reasoning exhibits higher values of Zproo and Ip- and lower values of Zpm.. , and Zp The MuSiQue dataset reveals the largest gaps between good and bad reasonings across all four metrics, which is probably attributable to its multi- hop reasoning complexity.
Rosuls in Hotpor2A
Resuls in 2WiiMidiHayQA
Resali is MiSiQue
Figure 3: Comparison between mean values of the good and bad instances from the 1000 samples of Llama-3.1-8B. The results of Qwen2.5-7B
shown in Appendix [A.3]
Notably, even in bad reasoning instances, Ip- and Ip-.t for supporting paragraphs consistently exceed those of Ipn.q and Ip for negative paragraphs. This observation indicates that LLMs possess inherent discrimination capabilities between supporting and negative paragraphs eaaraggapssreeeeee regardless of reasoning quality, though the information flows of supporting paragraphs remain insufficient for optimal answer generation. Our framework addresses this core issue by introducing quantitative information flow analysis that measures Ip.q and Zp., for each paragraph. This analysis enables the selective and precise amplification of relevant semantic information, detailed in Section 3
3 Methodology
To explore the Multi-doc QA capabilities of LLMs, we analyze the attention score matrix in each layer and identify the key paragraphs through the information flows. Building on the above findings, we propose Dual-Stage Adaptive Sharpening (DSAS), which consists of two key modules. In Section we introduce Contextual Gate Weighting (CGW), which identifies key paragraphs and strengthens the attention of the question and target position toward these texts. Section 3.2 describes Reciprocal Attention Suppression (RAS), which aims to suppress interactions between key paragraphs and irrelevant information. At a high level, we strategically integrate CGW and RAS modules into the attention to implement DSAS:
computation of
-, = Vile
, A = Stack (aack(AA,,... (Af,....,, (T,)).
A, = RAS(CGW(A), A$)).
AW Softmax (A (AH, dim =1),
O = AW.
O = Connat(O1....... (O.)..,,.::)..
where W.,, W.K.
h.1: h. WV, are projection matrices of the h-th head in the 1-th layer, WDO is the output projection matrix. We STACK tensors along the first dim and CONCAT them along the last dim. The framework of DSAS is shown in 4 To avoid ambiguity, descriptions key symbols appearing in the following text are provided in Table 5 in Appendix

---

Query
Decoder Layer
Stage 1: Contectuual Gate W Weighting
DSAS S
BARE
pengeope
Towenh Teformation
Ehtruc sib-raatte
XWV
prngeops
Negattn
Update
seat night
We wiggtt
Abories peggit murti
Slage 2: Reciprocal A Utettoo Supgression
Plss
PAL-N O6 soutth und wnd (he Pau Paak
Key
Wi
Wi
Altentin oxEgi
atuu
Figure 4: Illustration of Dual-Stage Adaptive Sharpening (DSAS), including Contextual Gate Weight- ing (CGW) and Reciprocal Attention Suppression (RAS).
3.1 Stage Contextual Gate Weighting (CGW)
For each selected layer, we compute the combined information flows of each paragraph using the attention score matrix Af. Specifically, we extract the attention sub-matrix corresponding to the positions of each paragraph, question and target as follows:
M E =2Qxp" [A (g.
(t,p") (Q))
(4)
Question Matrix
Target Matrix
where Q,p" denote the token length of the question and the m-th paragraph, respectively, and (()) represents the expand operation match the question length. The combined information flow for the mntt paragraph is then calculated by averaging the column-wise Top-K values M:
2Q Σ
Iombm
Σ
Top-K
Mi.j
m E (I,...,,C,,
(5)
K
j=1
We then compute Um through Z-normalization and sigmoid scaling:
= 0.5 [Commm
jus + 0.5, m E (1.....C)
(6)
where a(-) is the sigmoid function and jur, are the mean and the standard deviation of Iomb-- The minimum value of tm is set to 0.5 to prevent overlooking the paragraphs excessively. In addition, mitigate the coss-in-the-middle issue, we introduce position-aware weighting that assigns greater weights to central key paragraphs. Hsieh et al. [16] attribute this limitation to the inherent U-shaped attention bias in LLMs, which places more emphasis on content at the beginning and end of the sequence. We correct this bias through the probability density function (PDF) of Gaussian distribution, which is defined as follows:
(xr (a-1pp)
up) 2
F(x) = _ f())dt
f(x)
exp
VpV 2T
where M4, up are the mean and the standard deviation of the input token indices 1,.,,
L
(i.e., H4 equals 0.5 (T. 1), op equals (ا-1), ), and F(x: is the cumulative distribution function
(CDF) of a Gaussian distribution. Then the positional value Ym for the m-th paragraph is computed based on the token indices [p p="" ......p." within the segment:
Up
p
Up
21
22
(8)
Op
Op
F(22)
F(21)
7m
22
21
5

---

where 21, 2g denote the normalized value of p and P: respectively. Next. we rank paragraphs by Om compute position-awas aware weights for the top 50% paragraphs, and assign value of the rest, aiming to prevent the model from attracting excessive attention to non-critical middle paragraphs. The position-aware weight Im calculated follows:
9m {
0.5C+1
mm
if rankm S 0.50,
rankon
(10)
otherwise.
The final contextual gate weight Wm is derived as:
Wm Umn
9m
(11)
W m
min (.
Wm (1
+ B, (1,...,
(12)
max max(w() min(w:)
where hyperparameters and balance positional and content relevance and determine the minimum value of Wm. respectively. The attention score matrix A is dynamically adjusted by applying con- textual gate weights of each paragraph, thereby refining the model's ' focus based on their importance:
(i.)) Wm (i.)). if E {q. j E {p P + 1, -.....,PP'
(13)
3.2 Stage 2: Reciprocal Attention Suppression (RAS)
The CGW module computes contextual gate weights for each paragraph evaluate their relevance to answer generation, enabling clear distinction between critical and irrelevant paragraphs. The RAS module aims to suppress interactions between non-critical and key paragraphs. To achieve this, we classify paragraphs using threshold- based method: paragraphs with contextual gate weights higher than the mean value of all wmn are identified as key paragraphs, while those below the threshold are labeled as irrelevant paragraphs. The key and irrelevant paragraphs are denoted as Pkey, and Pirr. in the /-th layer. Next, reciprocal attention suppression is applied between these categories. This process adjusts the attention score matrix A by suppressing values between key and irrelevant paragraphs. The reciprocal suppression is bidirectional, affecting both interactions between Pkeyd and Piryt to break cross- proas-par interference. Formally, the suppression is implemented as:
(i,)) min(Wm, Wma)) (i,j),
if j < i, iE (p .P.' +1,...,.. Pe }. and j E (u"",,'" +1,++,,.......... ve where (mi Pleyd m2 P.ra)) (m2 Pkey,t mi Pin.).
(14)
Experiments
This section assesses DSAS, aadugg-and-paay attention mechanism designed to enhance Multi-doc QA performance across diverse models and downstream tasks. Our experiments are structured follows: Section 4.1 details the implementation of DSAS, covering datasets, metries, models and hyperparameter settings. In Section 4.2] we show that DSAS achieves improvements on all Multi-doc QA benchmarks, including HotpotQA [41). 2WikiMultiHoPOA [4), MuSiQue [33), and LongBench (3, without requiring any additional training. Section 4.3 presents ablation studies to examine the effectiveness of DSAS under different variants and hyperparameters. Section 4.4 further explores the robustness of DSAS.
4.1 Implementation
Datasets & Metrics. We evaluate our method on three classic Multi-doc QA benchmarks and long-context dataset. Following previous works, we utilize the validation splits of HotpotQA [4I], 2WikiMultHopQA [4], and MuSiQue B3 to test effectiivnness, as they provide reference answers. To further assess generalization, we extend experiments to LongBench's [ corresponding subsets on Multi-doc QA tasks. Referring to HotpotQA and LongBench, we consider the F1-score as the evaluation metric for these datasets. The metrics details are shown in Appendix A.4
Baselines & Models. PINE 38 analyzes attention patterns to re-assign input paragraph positions. Given its similar operation and target to DSAS, we include PINE as baseline for comparison, along- side vanilla LLM baseline that performs inference without altering the original pipeline. We select

---

Table I: Comparsion results HotpotQA, 2WikiMultiHopAA, MuSiQue and LongBench bench- marks. 2WikiMQA denotes 2WikiMuliHHopQA. The evaluation for all tasks is assessed through the
Fl-score (%).
I LongBench
Models
Methods
HorporQA
2WikiMQA
MuSiQue
Average
HotpoiQA
2WikiMQA
MuSiQue
Vanilla
39.1
39.4
20.8
41.9
34.1
12.7
31.3
Llama-3.2-3B
PINE
39.0
39.6
21.3
42.4
35,3
12.5
31.6
DSAS
40.6 (+1.5)
39.7 (+0.3)
21.2 (+0.4)
42.7 (+0.8)
35,9 (+1.8)
13.2 (+0.5)
32.2 (+0.9)
Vanilla
32.0
19.0
24.3
39.3
16.6
22.2
25.6
Mistral-7B
PINE
32.4
19.5
24.6
40.2
16.9
24.1
26.3
DSAS
33.8 (+1.8)
20.2 (+1.2)
26.8 (+2.5)
41.9 (+2.6)
16.6
25.3 (+3.1)
27.4 (+1.))
Vanilla
42.3
46.0
30.5
55.1
50.3
28.1
42,
Qwen2.5-7B
PINE
44.5
47.2
31.4
57.3
52.3
30.4
43,9
DSAS
46.1 (43.8)
49,9 (43.9)
35,0 (44.5)
57.7 (42.6)
52.7 (+2.4)
33.4 (45.3)
45.8 (+3.7)
Vanilla
37.2
22.9
26.8
45.1
20.3
24.8
29.5
DeepSeek R1-8B
PINE
36.8
22.6
45.2
20.1
23.9
29.2
DSAS
39.0 (+1.8)
23,4 (+0.5)
28.0 (+1.2)
47.2 (+2.1)
21.2 (+0.8)
26.7 (+1.9)
30.9 (+1.4)
Vanilla
43.6
47.3
34.6
53.3
42.6
25.4
41.1
Llama-3.1-8B
PINE
46.6
49.2
37.0
54.4
46,9
30.6
DSAS
47.1 (+3.5)
50.8 (+3.5)
39.2 (+4.6)
56.5 (+3.2)
47.3 (+4.7)
32.0 (+6.6)
45.5 (+4.2)
Vanilla
48.2
55,3
38.0
57.6
53.1
32.8
47,5
Qwen2.5-14B
PINE
50.4
57.0
41.8
59.8
56.4
36.7
50.4
DSAS
51.8 (43.6)
58.2 (+2.9)
43,8 (45.8)
60.99 (43.3)
56.1 (+3.0)
39.3 (+6.5)
51.7 (+4.2)
Vanilla
48.8
60.7
42.3
58.6
48.2
35.0
48,9
Qwen2.5-32B
PINE
50.8
61.0
44.5
58.1
47.9
36.4
49.8
DSAS
50.8 (+2.0)
62.2 (+1.5)
45.4
89.5 (40.9)
50.5 (+2.3)
39.9 (44.9)
51.4 (+2.5)
six popular L.LMs, including -lama-3.2-3B-In-s. [TO), 7B-Instruct (28). -eepSecc--I--issil--LLeee
1. Qwen2.5-
(II). Llama-3.1 3.1-8B-Instein [10). Qwen2.5-14B- (28), since they are popular Transformer- decoder-only LLMs, which are convenient for exploiting and analyzing inside architectures. All models are deployed with the "bfloat16" data format due to the balance between efficiency and performance. We set the generation mode to drccoddn for all methods with deterministic sampling pa- rameters: do sample=False, temperature=0,, top_p=1, max_new how_tookes=32 (follow the setting in LongBench (3). This setting minimizes the impact of irrelevant confounders during inference, thereby ensuring that identical models and inputs always produce the same answers to fixed questions. All experiments implemented using PyTorch 2.2.1, and executed on the CPU of two 32-core Intel(R) Xeon(R) @ 2.80GHz and GPU of 8x NVIDIA A800. The codebase is compatible with Python 321 and computations were accelerated using CUDA 12.1.
Instruct [28). and Qwen2.5-32B-Instrue
Hyperparameters. The hyperparameters in Equations 5 II and are set to K = 10, a = 1, and B 0.7 for all benchmarks and LLMs. DSAS is applied to the final 50% layers of all LLMs.
4.2 Main Results
Table[[] the main experimental results, revealing three principal insights: (i) Simply replacing the original attention module with our DSAS enhances LLMs' performance across all Multi-doo QA benchmarks without requiring additional training, achieving average F1-score gains between 0.9% and 4.2%. Llama-3.1-88 and Qwen2.5-14B achieve the most significant improvements, particularly LongBench tasks. These results demonstrate that DSAS acts catalyst to enhance Multi-doc QA performance across diverse model architectures and task configurations. In contrast, although PINE (38] achieves performance gains under certain configurations, its improvements are not consistently observed. (ii) Performance improvements hold consistently across model scales ranging from 3B to 32B parameters, with medium- sized LLMs generally exhibiting greater performance gains (e.g., maximum improvement of Llama-3. 1-8B and Qwen2.5-14B. followed by Qwen2.5-7B) We attribute the pattern two main reasons. (a) Medium-siie -iittuu---ieeddeeee LLMs (e.g., Llama-3.1-8B. Qwen2.5-14B) demonstrate adequate semantic understanding yet remain vulnerable to input noise, especially in long- context scenarios. DSAS overcomes this by analyzing information flows to precisely identify critical paragraphs and amplify model focus. In contrast, smaller models (e.g.. Llama-3.2--3 comprehension capacity, while larger models (e.g., Qwen2.5-333 approach the task performance ceiling, leaving little room for further gains. (b) Medium- sized LLMs exhibit greater vulnerability to
lack sufficient
7

---

Table 2: Ablation study of DSAS HotpotQA, 2WikiMultiHopOP MuSiQue and LongBench benchmarks. 2WikiMQA denotes ZWikiMultiHoOQA, p-a denotes position-awan aware weight Im in Equation (TO).
The evaluation for all tasks is assessed through the Fi-score (%).
LongBench
Models
Variants
HotpotQA
2WikiMQA
MuSiQue
Average
HotpotQA
2WikiMQA
MuSiQue
40.6
39.7
21.2
42.7
35,9
13.2
32.2
w/o CGW
40.6
39.5
20.4
43.5
34.6
12.9
31.9
Llama-3.2-30 3,2-3B
w/o RAS
39.7
39.0
21.7
42.9
34,6
11.4
31.6
w/o
39.1
36.8
19.0
42.4
34.5
10.6
30.4
DSAS
46.1
49.9
35.0
57.7
52.7
33.4
45,8
w/o CGW
45.5
48.2
33.8
56.6
50.3
31.1
44.3
Qwen2.5-7B
w/o RAS
45.4
47.8
32.6
$6.7
51.3
32.0
44.3
w/o p-a
44.7
50.5
33.2
49.6
30.6
DSAS
47.1
50.8
39,2
47.3
32.0
CGW
38.4
46.6
30,9
Llama-3.1
w/o RAS
45.7
49.0
38.7
54.9
46.8
31.4
44.4
w/o
45.9
49.7
37.8
56.2
46.2
30.4
44.4
DSAS
51.8
58.2
43.8
60.9
56.1
39.3
51.7
w/o CGW
49.5
57.2
40.8
58.6
55.3
38.2
50.0
Qwen2.5-14B
w/o RAS
50.2
57.6
41.4
59.9
54.1
38.6
50.3
w/o p-a
51.3
58.9
43.1
61.2
55.6
39.2
51.6
the "lost-in-the-middd" phenomenon, while larger models are generally more robust to this issue. Overall, DSAS effectively unlocks the latent capabilities diverse LLMs on Multi-doc QA tasks, regardless of their architecture scale. (iii) The improvement varies across different benchmarks. For medium-sized LLMs (parameters 7B, 8B, and 14B), greater improvements emerge on MuSiQue (including its LongBench extensions) compared other benchmarks. This discrepancy suggests that DSAS particularly enhances LLMs' performance on complex tasks, where their inherent Multi-doc QA abilities be better activated. The performance gap between HotpotQA and L-HottpttQA of these LLMs may be attributed to two factors. (a) Ansong et al. (26) identify annotation inconsistencies HotpotQA that could result in unreliable assessments. (b) HotpotQA and L-HotpotQA only involve two-hop reasoning tasks with relatively lower complexity which are easy for these LLMs.
4.3
Ablation Studies
In this section, we conduct ablation studies to evaluate the effectiveness of each component in DSAS.
Table (oobee/ismmaizzss the performance of different variants on tasks.
w/o CGW. Applying contextual gate weight Wio only in RAS to regulate paragraph interactions degrades all metrics. This indicates that enhancing information propagation from key paragraphs to both the question and the target is essential for answer generation.
w/o RAS. Results reveal that removing wm during information aggregation lowers LLMs' perfor- mance, suggesting that the absence of RAS allows irrelevant content in Pirr.! to introduce noise into the semantics of key paragraphs Pkey.
w/o p-a weight. Setting a in Equation weakens resistance to the "lost-in "lost-in-the-milee tie-ttheeemmeee issue, confirming that adaptively weighting middle paragraphs through Im enhances answer quality on most benchmarks. Notably, Qwen2.5-14B with this configuration achieves competitive results relative to DSAS, indicating larger LL inherently mitigate "lost-in-the-midlle" more effectively, which aligns with the observations in LongPiBench [32]. Nevertheless, the position-awee awaree weight generally improves performance all benchmarks and LLMs, mitigating "lost-in- siintthheiiddee issuc.
Hyperparameter Study. The parameter K Equation (5) controls the number of tokens that the question and target attend to within each paragraph. Lower K values risk insufficient contextual focus on the paragraphs, while higher values may degrade performance due to interference from irrelevant content. We evaluate K with 5,10,20 and observe that K 10 achieves optimal performance of Llama-3. 1-8B, as shown Figure (a). Furthermore, we investigate the optimal number insertion layers n of DSAS. Section shows minimal divergence in shallow layers. Most LLMs have layer counts multiples of four (e.g., 28, 32). We therefore position DSAS at three types of depths: the final 25%, 50%, and 75% layers (n 25%, 50%, 75%, respectively) of the LLM. Figure 5

---

HeroagA
HeqpeRAA
MALSQQA
EisppegAA
NASIQUA
MASIQNI
LHHpprgA
L.MisQque
L-MaS
3nd
(4)
(b)
(d)
Figure 5: 1lama-3.1-8P hyperparameter study of K, n, 8 on HotpotQA, 2 2WikiMuliHopQA, MuSiQue, and LongBench benchmarks. The hyperparameter studies of 1llmm-3322-35 Qwen2.5-7B. and Qwen2.5-14B are shown in Appendix
Table Results of original and shuffled input orderings when applying DSAS.
LongBench
Models
Variants
HotporQA
2WikiMQA
MuSiQue
Average
HotpotQA
2WikiMQA
MuSiQue
Original
46.1
49,9
35.0
57.7
52.7
33.4
45.8
Qwen2.5-7B
Shuffled
46.1
49.7
35.4
57.6
53.0
33.3
Original
47.1
50.8
39.2
56.5
47.3
32.0
45,5
Llama-3.1-8B
Shuflled
47.2
51.1
39.0
56.2
47.5
31.8
45.5
Original
51.8
58.2
43.8
60.9
56.1
39.3
51.7
Qwen2.5-14B
Shuffled
51.7
58.3
43.8
60.9
56.0
39.3
51.7
demonstrates that best performance is achieved with DSAS at the final 50% layers position. This could
be because n = 25% % inadequately improves information flows, while n = 75% causes misjudgment of critical content due to the slight divergence among information flows of different paragraphs in LLMs' shallow layers. Finally, we discuss the most appropriate values for a in Equation (IT) and B in Equation (12), which used balance Umn and Im and determine the minimum value of Wms respectively. Figure (c) demonstrate a = and 3 = 0.7 are the best choices.
4.4 Further Analysis Different Subsets
Since DSAS enhances attention centrally located critical paragraphs, thereby mitigating the LLMs' inherent "lost-in-the-middles issue. It is essential to evaluate its robustness to different input orders. To this end, we conduct an experiment which we randomize the ordering of paragraphs in all test samples while deliberately increasing the probability of supporting paragraphs occurring at the edges. This shuffled configuration intentionally weakens the attention DSAS to these important information. As shown in Table results under both settings (using the fixed configuration K = 10, n=50%, a and B 0.7) reveal only minor performance fluctuations across ordering conditions. This stability demonstrates the robustness of DSAS to input order, primarily benefitting from our weighting strategy. Specifically, when evidence appears the edges (where Im. Equation (IO) is reduced), DSAS attenuates the adjustments to the model's native attention distribution. This design intentionally leverages the model's inherent capacity identify and amplify information flows from edge-positiooned evidence, which aligns with the "U-shaped attention bias" phenomenon [16].
In addition. we further analyze how paragraph counts affect DSAS. Since the MuSiQue dataset contains 2-4 supporting paragraphs per sample, we group results by supporting paragraph count and report the comparative results in Table We observe minimal performance gaps across the groups, indicating that DSAS better captures relevant information flows to improve accuracy. To examine the effect of total paragraph count, we select all samples from MuSiQue and construct inputs with 10, 20, 30, and 40 paragraphs. The 20- -0rparggr setting uses the original sample inputs; the 10 10-paragraph setting randomly removes ten of the negative paragraphs: add 10/20 paragraphs sampled from other examples. Table shows that performance generally declines the total paragraph count increases. However, DSAS degrades more slowly, since suppresses information flow from negative paragraphs.
30-paragraph and 40-paragraph settings

---

Table 4: Results different subsets of MuSiQue. #Sup, #T denote supporting paragraph count and total paragraph count, respectively.
Models
Variants
#Sup=2
#Sup=3
#Sup=4
#T=10
#T=20
#T=30
#T=40
Vanilla DSAS
30.4
30.9
30.3
32.3
30.5
29.8
29.4
Qwen2.5-7B
35.0
35.2
34.7
36.7
35.0
34.8
34.3
Vanilla DSAS
35.8
32.9
33.6
35.4
34.6
33.8
32.0
1Jama-3.
40.2
35.7
36.5
39.5
39.2
38.7
38.3
Vanilla DSAS
38.1
37.3
38.7
39.3
38.0
36,9
36.1
Qwen2.5-14D
44.0
43.3
44.3
44.7
43.8
43.5
42.9
Background and Related Works
Long-context Reasoning in LLMs. Recent advances in LLMs have spurred the development of long-context models (c.g., Llama-3. -8sstttttttt [10] and Srren2.--7BIInsttii (28). claiming 128k- token capacities). However, mainstream capacity benchmarks like Needle-in-ae aall-ini----astackkk (NIAH) [19] mainly assess simple retrieval tasks, insufficiently reflecting complex real-world applications such as Multi-doc QA requiring cross-document information aggregation and multi-step reasoning. Current methods to enhance LLMs' long-conted reasoning abilities follow two directions: (i) Positional encoding extensions (e.g., PI [6). CLEX 4. and CREAM (39) effectively expand context windows by adjusting position embeddings, but demand extra training resources and struggle to integrate with existing L.Mss with inherently large context windows. (ii) Attention mechanism optimizations reduce computational costs and support longer inputs with minimal additional training. However, most approaches prioritize efficiency over actual reasoning improvements. Both strategies leave significant gaps in enhancing LLMs' reasoning abilities within their designated context windows.
Attention mechanism. The Transformer [34] architecture LLMs currently face two challenges: quadratic computational and memory scaling with input length and the "lost-in-the-middle [24] phenomenon. To overcome these constraints, recent studies are proposed to optimize the standard
attention framework, including StreamingLLLM 44). LM-Infinite [13. H2O (5), PINE [38], and
SSC 42 StreamingLLMM and IM-Infinite optimize memory usage by retaining initial tokens and recent tokens for next-token prediction, balancing efficiency and performance. H2O preserves recent tokens while dynamically selecting critical tokens (termed heavy hitters) balance local relevance and global importance. PINE employs bidirectional inter-segment attention and re-assigns paragraph positions. SSC scales hidden states enable more balanced attention distribution across different segments, Collectively, current approaches either truncate long contexts to reduce computational and memory demands (e.g., StreamingLLMM, LM-Infinite, H2O) overlook the long-range modeling among paragraphs (e.g.. PINE, SSC). Therefore, we propose DSAS, training- free framework that adaptively optimizes attention matrices. Other approaches (e.g., prompt compression [18), retrieval -uggeeted generation (20), memory tree (5)) require external models or higher computation.
6 Conclusion
In this work, we address the critical challenges of limited long-range modeling and persistent "lost- in-the-middle" issue in Multi-doc QA for LLMs. Through systematic analysis of information flow patterns from the paragraph disparity level and the answer quality level, our study reveals layer-wise aggregation patterns in distinct information flows and differences between good and bad reasoning instances. To resolve these issues, we propose Dual-Stage Adaptive Sharpening (DSAS), a training- free, plug-and-play mechanism that adaptively sharpens attention focus through two synergistic components Contextual Gate Weighting (CGW) and Reciprocal Attention Suppression (RAS).
components:
This work highlights the unexploited potential of attention optimization for LLMs. By adaptively sharpening attention score matrices, DSAS offers practical solution for real-world applications requiring cross-document reasoning. Future directions include extending DSAS to diverse long- context scenarios, exploring its integration with eetrivall-augeent generation frameworks (e.g.. RAG 200. GraphRAG AG 19). and investigating related L.M applications (i.e., backdoor attacks [12] data distillation (22], graph reasoning (66). Our findings underscore the importance of optimal attention optimization in unlocking the full capabilities of modern LLMs for complex tasks.
10

---

Acknowledgements
This work supported by the National Natural Science Foundation of China (Grant No.62406057 and No.621760460 the Fundamental Research Funds for the Central Universities (Grant No.2YGX2002XI0422, the Noncommunicable Chronic Diseases-National Science and Technol- ogy Major Project (Grant N 106), and the Sichuan Science and Technology Program (Grant No.20024ZDZX0011 1). We gratefully acknowledge the Advanced Computing Frontier Research Center of Tianfu Jiangxi Laboratory for providing the computing resources that enabled this research, which has been successfully deployed in the laboratory's nurse agent.
References
[1] Mistral AL Distral--biinstruee Mistral-78 -lstral-77-IInttrut---o0
2024. URL httpperr/huggggggga....ooriistia http://iinnaaac...cmmsssssaaaaaa
Huggingface.
[2] Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. LongAlign: A recipe for long context alignment of large language models. In EMNLP, pages 1376-1395. Association for Computational Linguistics, 2024.
[3] Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu. Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding. In ACL, pages 31 19-31333, Bangkok, Thailand, 2024. Association for Computational Linguistics.
[4] Guanzheng Chen, Xin Li, Zaiqiao Meng, Shangsong Liang, and Lidong Bing. CLEX: con- tinuous length extrapolation for large language models. ICLR, Vienna, Austria, 2024. OpenReview.net.
[5] Howard Chen, Ramakanth Pasumuru, Jason Weston, and Asli Celikyilmaz. Walking down the memory maze: Beyond context limit through interactive reading. arXiv preprint arXiv:2310.05029, 2023.
[6] Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window large language models via positional interpolation. arXiv preprint arXiv:2306.1559 2023.
[7] Maxime Darrin, Ines Arous, Pablo Piantanida, and Jackie Chi Kit Cheung. GLIMPSE: prag- matically informative multi-document summarization for scholarly reviews. In ACL, pages 12737-12752, Bangkok, Thailand, 2024. Association for Computational Linguistics.
[8] Chenlong Deng, Kelong Mao, and Zhicheng Dou. Learning interpretable legal case retrieval via knowledge-guided case reformulation. EMNLP, pages 1253-1265, Miami, FL, USA, 2024. Association for Computational Linguistics.
[9] Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. From local to global: A graph rag approach to focru-fo summarization. arXiv preprint arXiv:2404.1611 2024.
[10] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788, 2024.
[11] Daya Guo. Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, arXiv preprint arXiv:2501. 2025.
[12] Ji Guo, Wenbo Jiang, Rui Zhang, Wenshu Fan, Jiachen Li, Guoming Lu, and Hongwei Li. Backdoor attacks against hybrid classical- neural networks. Neural Networks, 191: 107776, 2025.
[13] Chi Han, Qifan Wang, Hao Peng. Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm- infinite: Zero-shot extreme length generalization for large language models. NAACI, pages 3991-4008, Mexico City, Mexico, 2024. Association for Computational Linguistics.

---

[14] Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing A multi-hop QA dataset for comprehensive evaluation reasoning steps. In COLING, pages Barcelona, Spain, 2020.
[15] Cheng-Ping Hsich, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's 's the real context size of your long-context language models? arXiv preprint FrXiv:2404.06654 2024.
[16] Cheng- Yu Hsich, Yung-Sung Chuang, Chun-Liang Li. Zifeng Wang, Long T. Le, Abhishek Kumar, James R. Glass, Alexander Ratner, Chen- Lee, Ranjay Krishna, and Tomas Pfister. Found in the middle: Calibrating positional attention bias improves long context utilization. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 1 11-16, 2024, pages 14982 14995. Association for Computational Linguistics, 2024.
[17] Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. OPERA: alleviating hallucination in multi-modal large language models via over-trust penalty and crrospection-allocation In CVPR, pages 13418- 13427, Seattle, WA, USA. 2024. IEEE.
[18] Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Cew Lin, Yuqing Yang, and Lili Qiu. Longllmlingua: Accelerating and enhancing Ilms in long context scenarios via prompt compression. In ACL, pages 1658-1677, Bangkok, Thailand, 2024. Association for Computational Linguisties.
[19] Greg Kamradt. Needle in haystack test, 2023. URL https:// com/ gkamradt/ LLLNTest_NeedleTnnllaayyysst
GitHub repository.
[20] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval- -uggeete- generation for nooledge-intensivv NLP tasks. In NewrIPS, December 6- 12, 2020, virtual, 2020.
[21] Muquan Li, Dongyang Zhang, Tao He, Xiurui Xie, Yuan-Fang Li, and Ke Qin. Towards effective data-free knowledge distillation via diverse diffusion augmentation. In MM, Melbourne, VIC, Australia, 28 October 2024 November 2024, pages 4416 4425, ACM, 2024.
[22] Muquan Li, Hang Gou, Dongyang Zhang, Shuang Liang, Xiurui Xie, Deqiang Ouyang, and Ke Qin. Beyond random: Automatic inner-loop optimization in dataset distillation. arXiv preprint arXiv:2510.04833 2025.
[23] Jiaheng Liu, Dawei Zhu, Zhiqi Bai, Yancheng He, Huanxuan Liao, Haoran Que, Zekun Wang, Chenchen Zhang, Ge Zhang, Jiebin Zhang, et al. A comprehensive survey long context language modeling. arXiv preprint arXiv:2503. 17407, 2025.
[24] Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Trans. Assoc. Comput. Linguistics, 12:157-173 2024.
[25] Yizhuo Ma. Ke Qin, and Shuang Liang. Beta-LR: Interpretable logical reasoning based on beta distribution. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 1945-1955 Mexico City, Mexico, June 2024. Association for Computational Linguistics.
[26] Ansong Ni, Matt Gardner, and Pradeep Dasigi. Mitigating false-negative contexts in multi- document question answering with retrieval marginalization. In EMNLP, pages 6149-6161, Virtual Event Punta Cana, Dominican Republic. 2021. Association for Computational Linguis- tics.
[27] Zhen Qin, Xiaodong Han, Weixuan Sun, Dongxu Li, Lingpeng Kong, Nick Barnes, and Yiran Zhong. The devil linear transformer. EMNLP, pages 7025-7041, Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguistics.
12

---

[28] Qwen, An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang. Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang. Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. htRp:ttt::
org/ www.aaa//22..22 141111515
[29] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In ICLR, Banff, AB, Canada, 2014.
[30] Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang, Chen Lin, Yeyun Gong, Lionel M. Ni, Heung- Yeung Shum, and Jian Guo. Think-on-graph: Deep and responsible reasoning of large language model on knowledge graph. In ICLR, Vienna, Austria, 2024. Openkeview.net.
[31] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, arXiv preprint arXiv:2003005530 2024.
[32] Runchu Tian, Yanghao Li, Yuepeng Fu, Siyang Deng, Qinyu Luo, Cheng Qian, Shuo Wang. Xin Cong. Zhong Zhang, Yesai Wu, et al. Distance between relevant information pieces causes bias in long-context Iims. arXiv preprint 2rXiv:2410.14611, 2024.
[33] Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique Multihop questions via single-hop question composition. Trans. Assoc. Comput. Linguistics, 10:539-554, 2022.
[34] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurlPS, pages 5998-6008 Long Beach, CA, USA, 2017.
[35] Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. Label words are anchors: An information flow perspective for understanding in-context learning. In EMNLP, pages 9840-9855, Singapore, 2023. Association for Computational Linguistics.
[36] Rongzheng Wang, Shuang Liang, Qizhi Chen, Yihong Huang, Muquan Li, Yizhuo Ma, Dongyang Zhang, Ke Qin, and Man-Fai Leung. Graphcogent: Mitigating lims' working memory constraints via multi-agent collaboration complex graph understanding, 2025. URL https://arxiv.org: haarr//abs//0000 12379
[37] Rongzheng Wang, Shuang Liang, Qizhi Chen, Jiasheng Zhang, and Ke Qin. Graphtool- instruction: Revolutionizing graph reasoning llms through decomposed subtask instruction. In KDD, Toronto, ON, Canada, August 3-7, 2025, pages 4492-1503. ACM, 2025.
[38] Ziqi Wang, Hanlin Zhang, Xiner Li, Kuan- Huang, Chi Han, Shuiwang Ji, Sham M. Kakade, Hao Peng, and Heng Ji. Eliminating position bias of language models: A mechanistic approach. In ICLR, Singapore, April 24-28, 2025. OpenReview.net, 2025.
[39] Tong Wu, Yanpeng Zhao, and Zilong Zheng. An efficient recipe for long context extension via middle-focused positional encoding. In NeurIPS, Vancouver, BC, Canada, 2024.
[40] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In ICLR, Vienna, Austria, 2024. OpenReview.net.
[41] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhut- dinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable molti-hop question answering. In EMNLP, pages 2369-2380, Brussels, Belgium, 2018. Association for Computational Linguisties.
[42] Yijiong Yu, Huiqiang Jiang, Xufang Luo, Qianhui Wu, Chin-Yew Lin, Dongsheng Li, Yuqing Yang. Yongfeng Huang, and Lili Qiu. Mitigate position bias in 1ls via scaling a single hidden states channel. In Findings of the Association for Computational Linguistics, ACI. 2025, Vienna, Austria, pages 6092-6111. Association for Computational Linguistics, 2025.
13

---

[43] Xiaosong Yuan, Chen Shen, Shaotian Yan. Xiaofeng Zhang, Liang Xie, Wenxiao Wang, Renchu Guan, Ying Wang, and Jieping Ye. Instance adaptiiv zero-shot thain-of-thoughy prompting. In NeurlPS, Vancouver, BC, Canada, 2024.
[44] Xuechen Zhang, Xiangyu Chang, Mingchen Li, Amit K. Roy-Chowdhury. Jiasi Chen, and Samet Oymak. Selective attention: Enhancing transformer through principled context control. In NeurlPS. Vancouver, BC, Canada, 2024.
[45] Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, and al. H2O: heavy-hitter oracle for efficient generative inference of large language models. In NeurlPS, New Orleans, LA, USA, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly outline the core contributions of the paper: (I) systematic analysis information flow LLMs for Multi-doc QA tasks. (2) proposal of Dual-Stage Adaptive Sharpening (DSAS), a training-free plug-and-play framework with two modules: Contextual Gate Weighting (CGW) for alleviating "lost-in-the-middlee and Reciprocal Attention Suppression (RAS) for long-range dependency modeling optimization. (3) extensive experiments demonstrating DSAS's universality across diverse LLMs (average 4.2% Fl-score improvement on Llama-3.1-8P -.1-tBe-nsttrct and 1...--itttnnst These align with the analysis (Section methodology (Section 3 and experimental results (Section 4
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Appendix G explicitly discuss limitations, including DSAS's limited appli- cability to other long-context tasks and its high memory and computational costs when handling extremely lengthy inputs.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they seale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
15

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Our framework's 's theoretical foundation about LLMs' information flow patterns is detailed Section 2 Figures provide quantitative evidence of LLMs' information flow patterns.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The paper provides comprehensive experimental details necessary for re- producibility. Section 2 describes the experimental configuration for information flow analysis on the Multi-doc QA tasks, including dataset specifications and model implemen- tations. Supplementary details such as prompt templates are provided in Appendix [.. In Section 4.1 datasets (HotpotQA, 2WikiMultiHopQA, MuSiQue, LongBench), model architectures (Llama, Qwen, Mistral, Deepseek variants), and hyprparraeeter settings (K, a, 3) are explicitly defined. Furthermore, the implementations of "bfloat16" deployment and greedy-decoding are used to ensure consistency.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
16

---

While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c)
If the contribution is new model (e.g., . a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Datasets (HotpotQA, 2WikiMultHHopAA, MuSiQue and LongBench) are publicly available with citations provided in Section and Section 4.1 License information for all datasets is included in references.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//iips.c//pull wttpcccpppppccceeeeeeeeee..ceeet gwwp//uiieee//eeeee (Cooeeooooissssieeeceee ssiocclicyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Section 2 and and Section 4.1 describe datasets, models, hyperparameter settings (K, n, a, I), and experimental environments. A. Appendix A.44iixxAAAlilusttaaa illustrate experimental templates and assessment method, respectively.
Guidelines:
17

---

The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report enror bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The information flow analysis in Section are conducted with 1000 test instances per task. To ensure the universality of our DSAS, we incorporate multiple LLMs (Llama, Qwen, etc.) in our experiments in Section 4 Appendix D provides additional analysis of error cases.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The hardware specifications, including GPU type and memory, are detailed Additionally, the computational complexity of our method is discussed in
in Section
4.1
Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
18

---

Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://pssppappppoccccpppa///aaa
Question:
Answer: [Yes]
Justification: The work focuses on attention optimization for Multi-doc QA tasks without ethical risks. All datasets are publicly available for academic use, adhering to their original licenses.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: The paper focuses on methodological contributions for attention optimization mechanism in the Multi-doc QA tasks. Broader impacts (e.g., misuse potential) are minimal as the work is foundational.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This work uses publicly available LLMs (Llama, Qwen, etc.) and benchmark datasets (HotpotQA, 2WikiMultHHopQA, etc.). The paper does not introduce new pretrained language models, image generators, scraped datasets that pose inherent misuse risks. All experiments involve standard academic datasets and LLMS cited Section 4.1 posing no special safety risks requiring safeguards.
Guidelines:
19

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All datasets (HotpotQA, 2WikiMultiHopOO4 etc.) and LLMs (Llama, Qwen, etc.) are cited with original references (Section 4.T.. Publicly available datasets and L.MMs are used under their respective licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwpppswiptchooeo thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No datasets, models, codebases released. The paper focuses on attention optimization methodology for enhanced Multi-QA performance on existing benchmarks (HotpotQA, 2WikiMultiHopQA etc.). Experimental implementations will be shared acst-acceptance following anonymity guidelines.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
20

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: All experiments use publicly available Multi-doc QA datasets. No human subjects, crowdsourced data, or participant compensation involved in this research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not involve human subjects, biological data, sensitive per- sonal information. The research exclusively analyzes anonymized text data from academic benchmarks, exempt from IRB oversight per institutional guidelines.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMS if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: The paper proposes DSAS, which operates plug-and-play attention optimization mechanism applied to existing Transformer-based LLMs for enhanced Multi- doc QA performance. The LLMs the basis of the methodology.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
21

---

A Details of Information Flow Analysis on Multi-doc QA
A.1 Experimental Templates
Consistent with LongBench [3], the templates for HotpotQA
[1] 2WikiMultHHoQAA [14] and
MuSiQue [3]
remain the same. Each data instance includes a question and its context, with task
instructions positioned at both the start and end of the prompt to enhance model comprehension. This
template maintained consistently throughout our experiments Section
Answer the question based on the given paragraphs. Only give me the answer and do not output any other words.
The following are given paragraphs. [context] Answer the question based on the given paragraphs. Only give me the answer and do not output any other words.
Question: {question)
Answer:
A.2 Paragraph Disparity Level Analysis on Qwen
Qwen2.5-7B consists of 28 stacked decoder layers. Analytical results on diguroe@deemmnstraat similar conclusions to Llama-3.1-8 1-BB, with minimal information flow variations in shallow layers, followed by emerging divergences between Iz' and Iy- ... and finally gaps between Ip- and Ip- .t Notably, Qwen2.5 exhibits a slower progression of these divergences compared to Llama-3. --18B, with the HotpotQA and the MuSiQue datasets particularly showing significant information flow variations across the first 50% of the model's 's layers.
Results HotpotQA
Results 2WikiMultHHopOA
Results MuSiQue
2.8
20
Inta
Information flow
2.4 Tana
Information flow
1.6
Tp.
Information flow
Zen
16
22
Tp.
2
2.8
2.2
2.3
0.0-
0.8
12
16
26
20
24
12
26
20
Layer
Layor
Layer
Figure 6: Layer-wise information flows of HotpotQA, 2WikiMultiHopQA and MuSiQue tested on Qwen2.5-7B, p² and p" denote supporting paragraphs and negative paragraphs, respectively.
A.3 Answer Quality Level Analysis on Qwen
Similar to observations in Llama-3.1-80 significant differences in information flow patterns emerge between good and bad reasoning instances, a consistent phenomenon observed in LLMs that forms the basis of DSAS framework.
Results in HotpotQA
Results 2WikiMulsHiipAA
Results MuSiQue
Good
Sed
30
3pr.
Figure 7: Comparison between mean values of the good and bad instances from the 1000 samples of Q5en2.5-7B.
22

---

A.4 Answer Quality Assessment
The evaluation method employed in these Multi-doc QA benchmarks adopts the answer quality assessment approach from HotpotQA. Specifically, for each data instance, both the reference answer and response first undergo normalization, including lowercase conversion of all textual content, climination of non-essential articles (i.e., a/an/the), and redundant whitespace. Then precision and F1-score are computed between the normalized reference answer and response through exact lexical matching. This assessment ensures comparability across different benchmarks. We classify samples achieving a perfect F1-score (equal to 1) as GOOD reasoning cases, and assign those with zero precision (equal to 0) to the BAD reasoning instances. The assessment method remains the same for experiments at Section
B
Symbol Explanation
Table Symbols and Descriptions.
Symbol
Description
the attention score matrix of the h-th head in the 1-th layer the attention weight matrix of the h-th head in the 1-th layer the combined information flow for the m-th paragraph Teombon
A Ald
P1,01
the mean and the standard deviation of Teomb the normalized value of "comb""
Um
Hp: Op
the mean and the standard deviation of sequence {0, (0.1...., 1} the positional value of the m-th paragraph the position-aware weight of the m-th paragraph
Im
Im
w m' Wm
the temporary and the final contextual gate weight of the m-th paragraph the number of tokens selected to compute Zoomb"
K
the proportion of layers inserting DSAS hyperparameters for computing W/m
a, 3
C Full Hyperparameter Study
The results presented in Figures 8 9 00 indicate that DSAS maintains consistent improvement across diverse LLMs under the hyperparameter configuration in Section 4.1 (red regions the radar charts). Furthermore, observe that the robustness of DSAS to hyperparameter variations strengthens with LLMs' scale expansion (3B, 7B, 14B), with Qwen2.5- performance improvements across all benchmarks under different hyperparameter settings. We believe that this phenomenon correlates with I.LMs' scales and their intrinsic capabilities. Additionally, some hyperparameter configurations of DSAS may marginally degrade performance below baseline for Llama-3.2 3.3-B, which further validates that our method achieves maximum performance gains specifically for middle-sized LLMs (7B, 8B, 14B).
particularly showing notable and stable
HegeeQA
HegerQA
MuSFQar
LM
CHepeRIA
L-HqUAAA
CiiqeeQQ
Vallia
2 2WaittnslleoAA
L-W LSS
(a)
(b)
(d)
Figure 8: Hyperparameter study of K, n. a, B on HotpoiQA, 2WikiMultiHopQA, MuSiQue, and LongBench benchmarks of Llama-3.2-3B.
23

---

HeraagA
HeqqeRjA
EistpergA
NUSIQUA
MASIQNI
Luu
LHHepogA
L-MaSk2ae
3rd
1.20 1.1000000
20 19400iii
ESW RIManESSpN
(b)
(d)
Figure 9: Hyperparameter study of K. n, a, 3 on HotpotQA, 2WikiMuliHHopOA, MuSiQue, and LongBench benchmarks of Qwen2.5-7B.
2ATEMaHilloOA
HepwrA
HMtperQA
NUSQQU
Lou
L.HppRIA
LiiitarQA L.MEsQuee
L-iiquRQA
L-Ma3k0ac
L-HigeiQAA
L-29 AStalitiooA
LoW AiStatzhpso
(1)
(b)
(d)
Figure 10: Hyperparameter study of K, n. a, 3 on HotporQA, 2WikiMultHHopQA, MuSiQue, and LongBench benchmarks of Qwen2.5-14B.
D Error Analysis
We conduct an error analysis of DSAS with a focus attention weight interactions among paragraphs (supporting and negative paragraphs). questions, and target. Following the experimental setup in Section 2 we randomly selected 1,000 samples for confusion matrix visualization between components. The analysis conducted on the HotpotQA dataset due to standardized structure of fixed 10- paragraph inputs (including 2 supporting paragraphs), which facilitates stable observation of interaction patterns.
1.00
0.53
0.55
0.42
0.37
0.69
0:31
5.20
0.44
0.33
0.53
0.20
1.00
0.28
0.30
0.22
0.27
0.50
0.11
0.20
0.24
0.15
0.91
0.89
0.49
0.48
0.41
0.31
0.45
0.52
0.41
0.45
1.0
0.38
0.28
0.11
5.03
0.25
0.02
0.28
0.25
3.0
-0.30
0.42
0.50
0.51
1.00
0.67
0.64
2.53
3.42
0.55
0.8.
0.41
0.46
0.22
0.38
0:41
1.00
0:47
0.24
0.21
2.12
0.35
0.63
0.11
0.20
2,80
0.33
0.39
0.25
9.42
0.50
0.29
5.50
0.39
0.80
0.28
0.28
0.24
0.29
1.20
0355
2.11
0.30
0.25
0.31
0.47
0.51
0.56
8.66
0.43
m65
0.111
0.11
0.17
0.21
0.29
(66
0:44
0.45
0.34
0.55
0.57
0.50
2.55
2.58
3.00
8.87
0.55
0.52
H07
0.24
0.25
0.09
0.35
0.37
0.30
0:35
2.39
3.00
0.87
0.47
PEB
0.67
8
2.83
0.42
0.51
0.5
8.80
0.45
3.52
0.26
0.49
1.20
0.25
0.91
0.11
0.53
0.80
2.31
0.55
0.28
5.00
0.00
0.80
0.39
0.45
0.50
0.79
6.43
0.89
0.02
0.85
1.00
1.88
0.25
0.09
0.26
0.79
0.23
0.29
0.47
0.26
1.00
np8
(4)
(b)
Figure 11: Confusion matrices HotpotQA. (a) LLM, (b) LLM+DSAS. sp1, sp2 represent two supporting paragraphs. npl, np2.... np8 represent eight negative paragraphs. and denote question and target, respectively. The results are conducted on Llama-3.1-8P I-SttInntt

---

Technically, we first average attention weights across all layers to obtain global attention matrix. For pairwise component analysis (e.g., between two paragraphs), we extract corresponding sub-matrin based on token index ranges. All component pairs are analyzed following this method. Similar to Equation
5 we average the column-wise Top-K values for each sub-matrix. Finally we normalize all confusion values to 0-1. This approach effectively captures salient attention patterns while minimizing noise interference. Figure [IrreeIIppeee presents heatmap comparisons of reasoning processes using LLM and LLM+DSAS. The results demonstrate that LLM+DSAS not only strengthens the focus of questions and targets on relevant support paragraphs, but also enhances information interactions between these supporting paragraphs. Additionally, it effectively suppresses unnecessary interactions between supporting paragraphs and irrelevant negative paragraphs.
E
Complexity Analysis
Table 6 presents the complexity of DSAS. Here, L denotes the input token count, Q denotes the question token count, and C is the number of paragraphs. As illustrated in Equation 3 After computing the query, key, value matrices (Q, K, V) and attention scores A through traditional attention mechanisms, we refine A using the CGW and RAS modules. The updated scores are normalized via softmax to derive attention weights, which are then used to generate the final attention output. The core innovation of DSAS lies in computing contextual gate weight for dynamically weighting each paragraph. The time complexity scales linearly with L and Q, contributing only minor fraction of the total computation compared to the quadratic O complexity of standard attention mechanisms. Furthermore, the space complexity grows linearly with C' alone. These characteristics ensure that DSAS maintains high efficiency in practice.
Table 6: Complexity analysis of DSAS.
Time Complexity O(LQ)
Space Complexity
O(C)
Scalability
Evaluating the scalability DSAS beyond multi-document QA presents an interesting research direction. To this end, apply DSAS to several long-context tasks from LongBench [3] including summarization (GovReport, QMSum, MultiNews) and code completion (LCC, RepoBench-P) The success of information flow analysis largely relies identifying anchors, which aggregate critical information from redundant contexts to guide answer generation. While in Multi-doc QA, anchors, extending DSAS broader long-context tasks
the question and target naturally requires defining appropriate anchors for each scenario. After reviewing the prompt templates, we select the final instruction sentence the query and the final sentence as anchors for aggregating information flows (e.g.. "Now, write a one-page summary of the report. An Summary:" for GovReport; "Next line code:" for LCC).
Table 7: Comparison results on summarization and code completion tasks with hyperparameter configuration *K=10, n=50% a=1, 3=0.7". The evaluation metrics for L-GovReport, L-QMSum, L-MultiNews and L-LCC, L-RBP are Rouge-L score and Edit Sim, respectively. RBP denotes RepoBench-P.
Models
Variants
L-GovReport
L-QMSum
L-MultiNews
L-LCC
L-RBP
Average
Vanilla DSAS
33.6
22.4
23.7
53.7
48.2
36.3
Qwen2.5-7B
37.0
25.1
26.2
56.3
51.5
39,2
Vanilla DSAS
34.9
24.8
27.1
58.1
50.8
39.1
Llama-3.1-88
37.4
26.6
29.3
60.0
52.5
41.2
Vanilla DSAS
38.4
28.2
31.8
63.9
55.6
43.6
Qwen2.5- 14B
41.0
30.4
34.6
66.7
58.9
46.3
25

---

Since DSAS operates on paragraph-Ievel segments, which Multi-doc QA inherently provides, we employ fxed-length token segmentation for the other tasks. Specifically, we use 500-token segments for GovReport, QMSum, and RepoBench-P, and 200-token segments for MultiNews and L.C., ensuring a medium number (5-40) paragraphs per example. The comparison results are presented in Table 7 Based on our experimental results, we make two key observations: (i) DSAS consistently enhances performance across all five tasks under three medium-sizee model architectures; and (ii) these improvements generalize to diverse tasks (summarization and code completion), demonstrating that DSAS is effective beyond multi-doc QA.
The prompt templates used for these tasks align with those in the LongBench 3 benchmark. The sentences red in the following template boxes serve as anchors.
The template for L-Govveeort is shown below:
You are given report by government agency. Write one-page summary of the report. Report:
[context)
Now, write one-page summary of the report.
Summary:
The template for L-QMSum is shown below:
You are given meeting transcript and a query containing a question or instruction. Answer the query in more sentences.
Transcript: [context]
Now, answer the query based on the above meeting transcript in one or more sentences.
Query: (input]
Answer:
The template for L-MultiNews is shown below:
You are given several news passages. Write a one-page summary of all news.
News:
[context]
Now, write one-page summary of all the news.
Summary:
The template for L-LCC is shown below:
Please complete the code given below. (context)
Next line of code:
The template for L-RepoBench-- is shown below:
Please complete the code given below. [context) [input] Next line of code:
G Limitations
While DSAS acts as plug-and-play attention mechanism and consistently improves performance in Multi-doc QA tasks, our work has two primary limitations that warrant discussion. (1) The scalability of DSAS warrants further investigation. While our experiments on summarization and code completion tasks demonstrate consistent performance gains, the current fxee-token-coome count chunking strategy remains relatively simple. Future work could explore more refined and semantically- aware chunking methods for these tasks, which may further enhance information flow between key semantic segments and the generated answer. (2) Although DSAS enhances performance within standard L.M context windows, inherits the fundamental computational and memory limitations of conventional Transformer [34] architectures, with quadratic complexity in both memory usage and
26

---

computational cost. Consequently, our approach does not address the scalability challenges associated with processing extremely long documents (e.g.. those exceeding 100K tokens). where memory and computational demands become prohibitive. Future investigations could consider integrating sparse attention strategies or memory-efficient architectures mitigate these constraints.
These limitations highlight important directions for future research while not diminishing the ef- fectiveness of DSAS. The existing framework establishes reliable base for advancing attention optimization in long-context processing.
27


Reviews:
[{"review_id": "z8iMsi6lNE", "reviewer": "Reviewer_KV6y", "summary": "This paper presents DSAS, a universal plug-and-play framework for attention optimization in multi-document question answering. The goal is to improve the long-context understanding ability of large language models in multi-document question answering tasks by introducing some inductive biases for such tasks. Concretely, Contextual Gate Weighting (CGW) and Reciprocal Attention Suppression (RAS) are proposed and combined to address this problem. Extensive experiments on four benchmarks demonstrate DSAS's universality across mainstream LLMs.", "questions": "- How are the performance of other tasks if DSAS is applied to the LLMs? Will the performance significantly decrease, or comparable to the baseline without DSAS?\n- Can DSAS be used for context length extension? For example, suppose that a model is trained on 128K context-length, what is the performance if we apply DSAS on it to process 256K-tokens multi-document question answering tasks?", "limitations": "YES", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Pros:\n- This paper is well-written and easy to understand. The problem is clearly presented, and the motivation is well discussed. The long-context understanding ability of LLM is a very important research topic in the community, and this paper focuses on a big and typical series of subtasks (multi-document question answering tasks) in this topic.\n- The paper is well organized, especially the logic of motivation->method->evaluation and analysis. Section 2, information flow analysis on multi-doc QA, identifies key factors for effective multi-doc QA reasoning in a systematic and quantitative manner, making the proposed methods promising and intuitive.\n- The method is lightweight: it doesn't require further training on the LLMs, but just changes how the attention is calculated, based on the properties of multi-document question answering tasks.\n\nCons:\n- It would be better if the experiment and analysis section can include some case study or quantitative analysis to prove that the inherent discrimination capability is really enhanced.\n- I'd suggest adding some experiments to further analysis DSAS on different subsets. For example, how will the number of supporting/negative/total paragraphs affect the effectiveness of DSAS?", "was_revised": true, "final_justification": "The authors have well addressed my concerns. I would like to keep my positive rating.", "rebuttal": "We thank the reviewers for their insightful questions regarding the proofs for the discrimination capability, the need for further analysis on paragraph number, the scalability of DSAS on other tasks, and our applicability for context length extension.\n\n***Cons1: Proofs for the Enhanced Discrimination Capability*** \n\n(i) **We conducted an error analysis of DSAS by examining the confusion matrix visualization in Appendix C.** Figure 11 illustrates the global attention matrix among supporting/negative paragraphs, questions, and target. DSAS substantially enhances the LLM's focus on supporting paragraphs, demonstrating improved discrimination capability.\n\n(ii) In addition, to complement our findings, we have added a quantitative analysis. Specifically, Equations (1) and (2) define the information flows from the m‑th paragraph to the question and to the target, respectively. Building on these definitions, we introduce two metrics: the supporting paragraph gain ratio $\\Delta \\mathcal{I}\\_{p^s}$ and the negative paragraph suppression ratio $\\Delta \\mathcal{I}\\_{p^n}$, which are formulated as:\n$$\n\\begin{align}\n\\Delta \\mathcal{I}\\_{p^s}&=\\frac{\\left(\\frac{1}{Q}\\mathcal{I}\\_{p^s,q}^{'}+\\mathcal{I}\\_{p^s,t}^{'}\\right)-\\left(\\frac{1}{Q}\\mathcal{I}\\_{p^s,q}+\\mathcal{I}\\_{p^s,t}\\right)}{\\frac{1}{Q}\\mathcal{I}\\_{p^s,q}+\\mathcal{I}\\_{p^s,t}}, \\\\\\\\\n\\Delta \\mathcal{I}\\_{p^n}&=1-\\frac{\\frac{1}{Q}\\mathcal{I}\\_{p^n,q}^{'}+\\mathcal{I}\\_{p^n,t}^{'}}{\\frac{1}{Q}\\mathcal{I}\\_{p^n,q}+\\mathcal{I}\\_{p^n,t}},\n\\end{align}\n$$\nwhere $Q$ denotes the question token length; $\\mathcal{I}\\_{\\cdot,q}^{'}$ and $\\mathcal{I}\\_{\\cdot,t}^{'}$ denote the information flows from supporting/negative paragraphs to the question and target after using DSAS, respectively. We calculate the information flows and metrics, and observe that **LLM+DSAS evidently increases the information flows from supporting paragraphs while suppressing information flows from negative ones compared to the baseline LLM.** The results are shown in Table 1.\n\nTable 1 Results of information flows and metrics on HotpotQA, 2WikiMultiHopQA and MuSiQue of Llama-3.1-8B. \n\n|                 | $\\frac{1}{Q}\\mathcal{I}\\_{p^s,q}+ \\mathcal{I}\\_{p^s,t}$ | $\\frac{1}{Q}\\mathcal{I}\\_{p^n,q}+ \\mathcal{I}\\_{p^n,t}$ | $\\frac{1}{Q}\\mathcal{I}\\_{p^s,q}^{'}+\\mathcal{I}\\_{p^s,t}^{'}$ | $\\frac{1}{Q}\\mathcal{I}\\_{p^n,q}^{'}+ \\mathcal{I}\\_{p^n,t}^{'}$ | $\\Delta \\mathcal{I}\\_{p^s}$ | $\\Delta \\mathcal{I}\\_{p^n}$ |\n| :-: | :-: | :-: | :-: | :-: | :-: | :-: |\n| HotpotQA        | 30.36 | 11.94| 36.21| 8.47| 19.3% | 29.1% |\n| 2WikiMultiHopQA | 30.85 | 12.30| 38.48| 7.94| 24.7% | 35.4% |\n| MuSiQue         | 25.07 | 8.65 | 30.11| 6.52| 20.1% | 24.6% |\n\n(iii) Finally, we selected one example from MuSiQue and **visualized its layer-wise attention matrices** for both the baseline LLM and LLM+DSAS. The matrices for LLM+DSAS exhibit lower attention dispersion [1], indicating its ability to concentrate on consistent token positions within supporting paragraphs. In contrast, the baseline LLM fails to filter information flows from negative paragraphs. \n\n[1] Qin, Ziran, et al. \"Cake: Cascading and adaptive kv cache eviction with layer preferences.\" arXiv preprint arXiv:2503.12491 (2025).\n\n***Cons2: Further Analysis on Different Subsets*** \n\nWe have conducted a further analysis on different subsets by changing the number of supporting/negative/total paragraphs. \n\n(i) Since the MuSiQue dataset contains 2-4 supporting paragraphs per sample, we have conducted experiments on subsets grouped by supporting paragraph count. Table 2 presents the comparative performance. **We observe minimal performance gaps with varying paragraph counts**, demonstrating that the baseline LLM is able to answer the questions and DSAS better capture relevant information flows to generate more accurate answers.\n\nTable 2 Results of different supporting paragraph count setting on MuSiQue through the F1-score (%).\n\n| Models       | Method   | #Supporting=2 | #Supporting=3 | #Supporting=4 |\n| :-: | :-: | :-: | :-: | :-: |\n| Qwen-2.5-7B  | Baseline | 30.4          | 30.9          | 30.3          |\n|              | Ours     | **35.0**      | **35.2**      | **34.7**      |\n| Llama-3.1-8B | Baseline | 35.8          | 32.9          | 33.6          |\n|              | Ours     | **40.2**      | **35.7**      | **36.5**      |\n| Qwen-2.5-14B | Baseline | 38.1          | 37.3          | 38.7          |\n|              | Ours     | **44.0**      | **43.3**      | **44.3**      |\n\n(ii) Next, to examine the effect of total paragraph count, we randomly sampled 200 examples from the MuSiQue dataset and constructed inputs with 10, 20, 30, and 40 paragraphs. Since each example originally contains 20 paragraphs, the 20‑paragraph setting uses the original sample inputs. For the 10‑paragraph setting, we randomly removed ten of the negative paragraphs. For the 30‑paragraph and 40‑paragraph settings, we randomly adding 10/20 paragraphs to construct inputs, respectively, drawn from other MuSiQue samples. The results of varying total paragraph count are reported in Table 3. \n\nThe results indicate that: (a) As the total paragraph count increases (longer input sequences), performance generally declines. However, **LLM + DSAS exhibits a slower degradation trend**, as it effectively suppresses information flows from negative paragraphs. (b) **Llama‑3.1‑8B + DSAS even outperforms Qwen‑2.5‑14B under all settings**, demonstrating the effectiveness of DSAS as a plug-and-play strategy.\n\nTable 3 Results of different total paragraph count setting on MuSiQue through the F1-score (%).\n\n| Models       | Method   | #Total=10 | #Total=20 | #Total=30 | #Total=40 |\n| :-: | :-: | :-: | :-: | :-: | :-: |\n| Qwen-2.5-7B  | Baseline | 32.6      | 30.7      | 30.2      | 29.5      |\n|              | Ours     | **36.8**  | **35.1**  | **34.8**  | **34.4**  |\n| Llama-3.1-8B | Baseline | 35.4      | 34.8      | 33.9      | 32.3      |\n|              | Ours     | **39.7**  | **39.3**  | **39.0**  | **38.1**  |\n| Qwen-2.5-14B | Baseline | 39.8      | 38.4      | 37.1      | 36.6      |\n|              | Ours     | **44.9**  | **44.0**  | **43.4**  | **42.7**  |\n\n***Q1: Performance of Other Tasks*** \n\nWe appreciate the reviewer's suggestion to evaluate DSAS's scalability beyond Multi‑doc QA. Accordingly, we applied DSAS to the summarization (GovReport, QMSum, MultiNews) and code completion (LCC, RepoBench-P) tasks in the LongBench benchmark. After examining the prompt templates, we selected the final instruction sentence as the anchor (e.g., \"Now, write a one‑page summary of the report.\" for GovReport; \"Next line of code:\" for LCC). \n\nBecause DSAS requires paragraph‑level segmentation and Multi‑doc QA naturally provides this, we instead segment these new tasks by fixed token counts. Specifically, we use 500‑token segments for GovReport, QMSum, and RepoBench‑P, and 200‑token segments for MultiNews and LCC, ensuring a medium number (5-40) of paragraphs per example. The comparison results are shown in Table 4. \n\nWe observe: (a) DSAS achieves consistent performance gains **across ​​all five tasks across three medium-size model architectures.** (b) The positive results **extend to diverse tasks (summarization and code completion)**, confirming that DSAS's benefits are ​​not confined to Multi-doc QA​​. \n\nTable 4 Comparison results on summarization and code completion tasks with \"$K$=10, $n$=50%, $\\alpha$=1.0, $\\beta$=0.7\". The evaluation metrics for L-GovReport, L-QMSum, L-MultiNews and L-LCC, L-RBP are Rouge-L score and Edit Sim, respectively. RBP denotes RepoBench-P.\n\n| Models       | Method   | L-GovReport | L-QMSum  | L-MultiNews | L-LCC    | L-RBP    | Average  |\n| :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: |\n| Qwen-2.5-7B  | Baseline | 33.6        | 22.4     | 23.7        | 53.7     | 48.2     | 36.3     |\n|              | Ours     | **37.0**    | **25.1** | **26.2**    | **56.3** | **51.5** | **39.2** |\n| Llama-3.1-8B | Baseline | 34.9        | 24.8     | 27.1        | 58.1     | 50.8     | 39.1     |\n|              | Ours     | **37.4**    | **26.6** | **29.3**    | **60.0** | **52.5** | **41.2** |\n| Qwen-2.5-14B | Baseline | 38.4        | 28.2     | 31.8        | 63.9     | 55.6     | 43.6     |\n|              | Ours     | **41.0**    | **30.4** | **34.6**    | **66.7** | **58.9** | **46.3** |\n\n***Q2: DSAS for Context Length Extension***\n\n(i) Our proposed DSAS aims to address two challenges in Multi‑doc QA: limited long-range dependency and persistent ''lost-in-the-middle'' issue. Although DSAS has not been extended in scenarios exceeding the fixed window size, it offers a plug‑and‑play solution that enhances discrimination without architecture modifications.\n\n(ii) In future work, we plan to extend DSAS beyond the fixed context window by integrating it with techniques such as StreamingLLM [2], which enlarges effective context length by retaining the KV cache of initial tokens and the most recent tokens. Specifically, we will investigate methods for dynamically preserving information from key paragraphs while discarding irrelevant noise during generation, thereby supporting substantially longer contexts.\n\n[2] Xiao, Guangxuan, et al. \"Efficient streaming language models with attention sinks.\" arXiv preprint arXiv:2309.17453 (2023).\n\nWe will incorporate all the analysis and supplementary experiments into the revised manuscript."}, {"review_id": "Ex19xSandN", "reviewer": "Reviewer_Gbi5", "summary": "This paper introduces Dual-Stage Adaptive Sharpening (DSAS), a training-free framework designed to optimize attention in large language models (LLMs) for multi-document question answering. The DSAS framework consists of two main modules: Contextual Gate Weighting (CGW), which weighs paragraph relevance through attention tracking and positional weighting, and Reciprocal Attention Suppression (RAS), which suppresses unnecessary attention between key and irrelevant paragraphs. The method is applicable as a plug-and-play module requiring no architectural modification or extra parameters. The paper presents systematic analysis of information flow in LLMs and validates DSAS across four public benchmarks and several mainstream LLM architectures.", "questions": "1. The position-aware weighting, aimed at the \"lost-in-the-middle\" problem, may excessively prioritize central paragraphs even when relevant evidence occurs at the edges. While the authors mention mitigation via ranking and capping, additional discussion on failure cases of this bias (especially in non-standard input orderings) is lacking.\n2. DSAS is only evaluated on Multi-doc QA tasks, whereas LLMs are also deployed for other long-context applications like summarization. It is suggested to conduct experiments on other tasks. Moreover, the proposed method is not evaluated over general performance benchmarks, such as how DSAS affects math/code or other reasoning tasks.\n3. Figure 5 reveals non-negligible variance in gains depending on choices of K, n, α, β. These variances indicate the proposed method is unstable and could be hard to generalize to unseen domains.", "limitations": "see above.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "1. This paper is well-written and well-organized. The idea is simple and easy to follow, and is built upon in-depth analysis of layer-wise information flow.\n2. The ablations (Table 3) and hyperparameter studies (Figure 5) dissect the individual contributions of CGW, RAS, and positional weighting.", "was_revised": false, "final_justification": "", "rebuttal": "We acknowledge the reviewer's valuable insights regarding potential paragraph ordering sensitivity, DSAS's scalability, and hyperparameter configurations.\n\n***Q1: Experiments of Non-standard Orderings***\n\nAs DSAS is designed to improve attention toward centrally located paragraphs, thereby mitigating the LLMs' inherent ''lost‑in‑the‑middle'' issue. It is important to evaluate DSAS's robustness to input order. Accordingly, we conducted an experiment where we randomized the ordering of paragraphs in all test samples while deliberately increasing the probability of supporting paragraphs occurring at the edges. Table 1 presents results for both original and shuffled inputs, using the consistent configuration \"$K$=10, $n$=50%, $\\alpha$=1.0, $\\beta$=0.7\". **Results indicate minor performance fluctuations across ordering variants.** This stability demonstrates DSAS's robustness to varying input orders, primarily benefits from our weighting strategy. We think that when evidence appears at the edges​ (where $g\\_m$ in Equation (9) is reduced)​, DSAS attenuates the adjustments to the model's native attention distribution. This design intentionally leverages the model's inherent capacity​​ to identify and amplify information flows from the evidence at the edges, which aligns with the ''U-shaped attention bias'' phenomenon in [1]. \n\nWe observed one case in MuSiQue with three supporting paragraphs (evaluated using Llama-3.1-8B) that succeeded with the original ordering but failed after shuffling. Analysis revealed that shuffling positioned the supporting paragraphs more centrally. **The observed failure in this instance may stem from insufficient focus amplification on central information despite applying DSAS.** This is supported by our information flow metric $\\frac{1}{Q}\\mathcal{I}\\_{p^s,q}+\\mathcal{I}\\_{p^s,t}$, where the shuffled sample is 27.75 compared to 29.87 for the original. Nevertheless, DSAS maintains robust performance across the vast majority of samples.\n\nTable 1 Results of original and shuffled input orderings.\n\n| Models       | Method   | HotpotQA | 2WikiMultiHopQA | MuSiQue  | L-HotpotQA | L-2WikiMultiHopQA | L-MuSiQue |\n| :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: |\n| Qwen-2.5-7B  | Original | **46.1** | **49.9**        | 35.0     | **57.7**   | 52.7              | **33.4**  |\n|              | Shuffled | 46.1     | 49.7            | **35.4** | 57.6       | **53.0**          | 33.3      |\n| Llama-3.1-8B | Original | 47.1     | 50.8            | **39.2** | **56.5**   | 47.3              | **32.0**  |\n|              | Shuffled | **47.2** | **51.1**        | 39.0     | 56.2       | **47.5**          | 31.8      |\n| Qwen-2.5-14B | Original | **51.8** | 58.2            | **43.8** | **60.9**   | **56.1**          | **39.3**  |\n|              | Shuffled | 51.7     | **58.3**        | 43.8     | 60.9       | 56.0              | 39.3      |\n\n[1] Hsieh, Cheng-Yu, et al. \"Found in the middle: Calibrating positional attention bias improves long context utilization.\" arXiv preprint arXiv:2406.16008 (2024).\n\n***Q2: Performance of Other Tasks***\n\nWe appreciate the reviewer's suggestion to evaluate DSAS's scalability beyond Multi‑doc QA. Accordingly, we applied DSAS to the summarization (GovReport, QMSum, MultiNews) and code completion (LCC, RepoBench-P) tasks in the LongBench benchmark. After examining the prompt templates, we selected the final instruction sentence as the anchor (e.g., \"Now, write a one‑page summary of the report.\" for GovReport; \"Next line of code:\" for LCC). \n\nBecause DSAS requires paragraph‑level segmentation and Multi‑doc QA naturally provides this, we instead segment these new tasks by fixed token counts. Specifically, we use 500‑token segments for GovReport, QMSum, and RepoBench‑P, and 200‑token segments for MultiNews and LCC, ensuring a medium number (5-40) of paragraphs per example. The comparison results are shown in Table 2. \n\nWe observe: (a) DSAS achieves consistent performance gains **across ​​all five tasks across three medium-size model architectures​​.** (b) The positive results **extend to diverse tasks (summarization and code completion)**, confirming that DSAS's benefits are ​​not confined to Multi-doc QA​​. \n\nTable 2 Comparison results on summarization and code completion tasks with \"$K$=10, $n$=50%, $\\alpha$=1.0, $\\beta$=0.7\". The evaluation metrics for L-GovReport, L-QMSum, L-MultiNews and L-LCC, L-RBP are Rouge-L score and Edit Sim, respectively. RBP denotes RepoBench-P.\n\n| Models       | Method   | L-GovReport | L-QMSum  | L-MultiNews | L-LCC    | L-RBP    | Average  |\n| :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: |\n| Qwen-2.5-7B  | Baseline | 33.6        | 22.4     | 23.7        | 53.7     | 48.2     | 36.3     |\n|              | Ours     | **37.0**    | **25.1** | **26.2**    | **56.3** | **51.5** | **39.2** |\n| Llama-3.1-8B | Baseline | 34.9        | 24.8     | 27.1        | 58.1     | 50.8     | 39.1     |\n|              | Ours     | **37.4**    | **26.6** | **29.3**    | **60.0** | **52.5** | **41.2** |\n| Qwen-2.5-14B | Baseline | 38.4        | 28.2     | 31.8        | 63.9     | 55.6     | 43.6     |\n|              | Ours     | **41.0**    | **30.4** | **34.6**    | **66.7** | **58.9** | **46.3** |\n\n***Q3: Hyperparameter Settings***\n\n(i) As demonstrated in Figure 5 and Appendix B, we evaluated the impact of hyperparameters $K, n, \\alpha, \\beta$ across multiple models. While performance exhibits fluctuations with different hyperparameter configurations, the vast majority of settings still outperform the baseline. Notably, substantial variability occurs only with the smaller Llama‑3.2‑3B model (Figure 8), which we attribute to its relatively weaker semantic comprehension capacity and greater sensitivity to input noise, which requires more precise hyperparameter tuning. In contrast, **mainstream 7B–14B models exhibit significantly lower sensitivity to these hyperparameters.**\n\n(ii) Section 4.2 confirms that the fixed hyperparameter combination \"$K$=10, $n$=50%, $\\alpha$=1.0, $\\beta$=0.7\" demonstrates stable and effective performance. The results highlight two properties of DSAS: (a) **Model-agnostic**: It consistently improves performance across diverse architectures (Llama, Qwen, and Mistral) ranging from 3 B to 32 B parameters. (b) **Task-agnostic**: It yields uniform improvements across multiple tasks. Furthermore, Figures 2 and 6 illustrate that information flows from supporting and negative paragraphs diverge substantially in deeper layers (last 50 %), whereas shallow layers exhibit limited divergence. Figure 5 and Appendix B further confirm that $n$=50% is the optimal choice, reinforcing the robustness of the hyperparameters.\n\n(iii) In response to **Q2**, we have extended DSAS to additional tasks, including summarization and code completion. Supplementary results in Table 2 confirm that the hyperparameter configuration \"$K$=10, $n$=50%, $\\alpha$=1.0, $\\beta$=0.7\" maintains consistent efficacy **across these unseen domains.**"}, {"review_id": "JhJQjnBzRZ", "reviewer": "Reviewer_c8Si", "summary": "This paper aims to identify key issues in long-range dependency modeling, particularly in multi-document question answering, and presents some experimental techniques to improve LLM performance targeting these deficiencies. The key problems identified are that language models have difficulty processing information located in the middle of long inputs and struggle to focus on important information.\n\nThe authors propose a dual-stage mechanisms to address these issues. First, Contextual Gate Weighting: for each paragraph, the method measures information flow with respect to the question, ranks the paragraphs, and multiplies the attention scores by learned weights that combine contextual relevance and position-aware factors. The second mechanism is Reciprocal Attention Suppression, which aims to mute the attention between key and irrelevant paragraphs by downscaling the cross-attention between them.\n\nThe paper runs several experiments on open-source models and multi-document question answering datasets and reports gains in F1 scores in the majority of the settings.", "questions": "Could the authors demonstrate that the same process, with the current parameter settings, generalizes to other domains—even on a small toy dataset—to help build confidence in the method’s robustness?\n\nCould the authors provide empirical data on the computational overhead or latency introduced by the proposed method compared to standard attention?\n\nHow were the hyperparameters tuned, and what was the data splitting strategy used to avoid potential data leakage or overfitting?\n\nDo the authors have any hypotheses for why the method appears to perform best on medium-sized language models only?", "limitations": "Please check above for broad feedback, I don’t think this paper has a negative societal impact.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\nThe paper identifies key problems in the question answering field, and I agree that both of these research directions are important for the community.\n\nThe proposed method requires minimal engineering and no training runs, so it's quite easy to use.\n\nThe language of the paper is clear, and the flow is easy to understand. Attention reweighing has been studied in the community before, but the techniques presented provide an interesting additional reuse case in an experimental setting. The authors have also done a good job citing the relevant previous work in the area.\n\nThe F1 improvements, although in some cases quite marginal, are consistent and certainly improve confidence in the method.\n\n### Weaknesses\n\nThe method needs deep access to the attention stack and major changes to the inference loop. Although it is training-free, it can only be applied to open-source transformer models with the usual infrastructure. The computational overhead is also not trivial. I am also not sure how the latency will be affected.\n\nSince the paper is completely experimental and the techniques are quite arbitrary without any theoretical justification, the paper needs to clear a very high bar in the evaluation process and present a generalizable method. The current manuscript doesn’t clear that bar satisfactorily. The proposed methods seem too specific to the hyperparameters, which are not practical to tune for every specific task. This prevents the use of this method more generally. The hyperparameter tuning process needs a lot of clarification about the data splitting and tuning process.\n\nAs the authors mentioned, for some models like LLaMA (Figure 8), some choices of hyperparameters are worse than the baseline. In other scenarios, a lot of the hyperparameter settings are very close to the baseline. So, it does seem like the precise choice plays a very strong role and is quite model-dependent. “Middle size models” is a bit confusing notion and depends on the model architecture, so it's not clear whether the method will be useful or not on a particular model without having to do some kind of holdout evaluation. \n\nOverall, I agree with the problems identified in the paper, which are both important and well-motivated. However, the proposed solutions come across as somewhat ad hoc and appear to address the symptoms rather than the root causes. While I recognize that tackling the underlying issues is substantially more challenging, the current approach may reflect some degree of research overfit.", "was_revised": true, "final_justification": "I thank the authors for their detailed rebuttal. The additional results definitely help in clearing most of my apprehensions. I also appreciate the discussion around computational overhead, latency and robustness. Thus, I am happy to raise my score.", "rebuttal": "We sincerely thank the reviewer for the thoughtful comments regarding the computational overhead, theoretical justification, hyperparameter tuning, model generalization, and method design. Based on the weaknesses you identified, we have organized our responses into the following parts:\n\n***Weaknesses1: Computational Overhead and Latency*** \n\nThis discussion refers to paragraph 1 in the weaknesses the reviewer proposed.\n\n(i) We acknowledge that DSAS needs deep access to the attention stack, but it is a lightweight plugin and does not require major changes to the inference loop. DSAS intervenes only in specific layers (the final 50 %), where it computes a contextual gate weight for each paragraph based on the raw attention scores and dynamically adjusts those scores. The overall procedure preserves the standard Transformer computations and requires no architectural modifications.\n\n(ii) We acknowledge that DSAS has only been applied to open-source Transformer models. However, this constraint does not imply a technical shortcoming. In fact, existing attention optimization methods are also limited to such architectures.\n\n(iii) Regarding computational overhead, as mentioned in Appendix D, the additional operations of DSAS introduce $\\mathcal{O}(LQ)$ time complexity, where $L$ and $Q$ denote the input length and the question length, respectively. Since it scales linearly with $L$, it **constitutes a considerably smaller fraction of the total computation compared to the quadratic $\\mathcal{O}(L^2)$ complexity** of standard attention mechanisms.\n\n(iv) To evaluate inference latency, we conducted experiments on Llama-3.1-8B using a single NVIDIA A800 GPU. Table 1 reports the average per-sample inference time for a LongBench subset (L-HotpotQA). Results indicate DSAS introduces a 10% latency increase over the baseline method.  **We contend the modest overhead is worthwhile given DSAS's consistent performance gains across diverse tasks.**\n\nTable 1 Average latency comparisons on L-HotpotQA of Llama-3.1-8B.\n\n|          | Latency (s) |\n| :-: | :-: |\n| Baseline | 2.61        |\n| Ours     | 2.87        |\n\n***Weaknesses2:  Lack of Theoretical Justification***\n\nThis discussion refers to paragraph 2 in the weaknesses the reviewer proposed. We would like to offer the following clarifications on the techniques used in the manuscript:\n\n* Although the validation of DSAS is mainly experimental, the design of DSAS is firmly **grounded in an information-theoretic perspective on attention** rather than ad hoc heuristics. In Section 2, we analyze information flows at both paragraph disparity and answer quality levels, and these insights ​directly provide the basis​ for our CGW and RAS modules.\n* The used techniques align with our research goal. e.g., we adopt **Gaussian CDF to compute positional values** for enhancing the model's focus on centrally located paragraphs, due to the ''U‑shaped attention bias'' [1] in LLMs.\n* We designed DSAS as a truly plug-and-play mechanism whose generalizability is empirically validated by numerous experiments. **DSAS derives consistent improvements across diverse LLMs and tasks**. In addition, in response to ***Q1***, we have extended DSAS to additional tasks, including summarization and code completion, further demonstrating its generalizability. \n\n[1] Hsieh, Cheng-Yu, et al. \"Found in the middle: Calibrating positional attention bias improves long context utilization.\" arXiv preprint arXiv:2406.16008 (2024).\n\n***Weaknesses3:  Details of Hyperparameter Tuning***\n\nThis discussion refers to paragraph 2 in the weaknesses the reviewer proposed.\n\n(i) The hyperparameter configuration \"$K$=10, $n$=50%, $\\alpha$=1.0, $\\beta$=0.7\" is effective across diverse LLMs on various tasks (4.2% average gains on Llama-3.1-8B and Qwen2.5-14B). Our work includes the design of DSAS as well as the finding of the robust hyperparameter configuration. We believe the robustness benefits from the well-designed formulations and there is no need for hyperparameter tuning process.\n\n(ii) In response to ***Q1***, we have extended DSAS to additional tasks, including summarization and code completion. Table 1 confirms that the hyperparameter configuration \"$K$=10, $n$=50%, $\\alpha$=1.0, $\\beta$=0.7\" maintains **consistent efficacy across these unseen domains.** \n\n***Weaknesses4:  Generalization across Models***\n\nThis discussion refers to paragraph 3 in the weaknesses the reviewer proposed. We have discussed some points of the weakness. Please refer to the response to ***Weaknesses3*** above.\n\n(i) We acknowledge that LLM+DSAS underperforms the baseline under some hyperparameter settings on Llama-3.2-3B. However, this does not indicate instability of the method itself. Rather, it reflects the inherent limitations of small‑scale models, which we attribute to their relatively weaker semantic comprehension capacity and greater sensitivity to input noise, which requires more precise hyperparameter tuning. **In contrast, mainstream 7B–14B models exhibit significantly lower sensitivity to these hyperparameters (Figures 5, 9, 10).** \n\n(ii) Medium-sized LLMs refer to LLMs with parameter sizes ranging from 7B to 14B, regardless of architectural family (e.g., LLaMA-3.1-8B, Qwen2.5-7B, Mistral-7B, Qwen2.5-14B).\n\n***Weaknesses5:  Method Design***\n\nThis discussion refers to paragraph 4 in the weaknesses the reviewer proposed.\n\n(i) As noted in Introduction, existing approaches either sacrifice global token interaction or require additional training, which limits their adaptability. In contrast, DSAS does not reflect research overfit and is founded on a systematic analysis of information flows to guide attention optimization.\n\n(ii) In response to ***Weaknesses2***, we have provided a detailed explanation of the theoretical basis of DSAS.\n\n(iii) In response to ***Weaknesses1,3,4***, quantitative results confirm performance gains across diverse tasks and models while maintaining inference efficiency. The strong capability enhancement indicates DSAS addresses root causes rather than symptoms.\n\n***Q1: Robustness of DSAS*** \n\nWe appreciate the reviewer's suggestion to evaluate DSAS's scalability beyond Multi‑doc QA. Accordingly, we applied DSAS to the summarization (GovReport, QMSum, MultiNews) and code completion (LCC, RepoBench-P) tasks in the LongBench benchmark. After examining the prompt templates, we selected the final instruction sentence as the anchor (e.g., \"Now, write a one‑page summary of the report.\" for GovReport; \"Next line of code:\" for LCC). \n\nBecause DSAS requires paragraph‑level segmentation and Multi‑doc QA naturally provides this, we instead segment these new tasks by fixed token counts. Specifically, we use 500‑token segments for GovReport, QMSum, and RepoBench‑P, and 200‑token segments for MultiNews and LCC, ensuring a medium number (5-40) of paragraphs per example. The comparison results are shown in Table 2. \n\nWe observe: (a) DSAS achieves consistent performance gains **across ​​all five tasks across three medium-size model architectures​​.** (b) The positive results **extend to diverse tasks (summarization and code completion)**, confirming that DSAS's benefits are ​​not confined to Multi-doc QA​​. \n\nTable 2 Comparison results on summarization and code completion tasks with \"$K$=10, $n$=50%, $\\alpha$=1.0, $\\beta$=0.7\". The evaluation metrics for L-GovReport, L-QMSum, L-MultiNews and L-LCC, L-RBP are Rouge-L score and Edit Sim, respectively. RBP denotes RepoBench-P.\n\n| Models       | Method   | L-GovReport | L-QMSum  | L-MultiNews | L-LCC    | L-RBP    | Average  |\n| :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: |\n| Qwen-2.5-7B  | Baseline | 33.6        | 22.4     | 23.7        | 53.7     | 48.2     | 36.3     |\n|              | Ours     | **37.0**    | **25.1** | **26.2**    | **56.3** | **51.5** | **39.2** |\n| Llama-3.1-8B | Baseline | 34.9        | 24.8     | 27.1        | 58.1     | 50.8     | 39.1     |\n|              | Ours     | **37.4**    | **26.6** | **29.3**    | **60.0** | **52.5** | **41.2** |\n| Qwen-2.5-14B | Baseline | 38.4        | 28.2     | 31.8        | 63.9     | 55.6     | 43.6     |\n|              | Ours     | **41.0**    | **30.4** | **34.6**    | **66.7** | **58.9** | **46.3** |\n\n***Q2: Computational Overhead and Latency*** \n\nWe have discussed this question. Please refer to the response to ***Weaknesses1*** above.\n\n***Q3: Hyperparameter Tuning and Data Splitting Strategy*** \n\nWe have discussed this question. Please refer to the response to ***Weaknesses4*** above. \n\nFurthermore, to prevent data leakage, all experiments are conducted on the validation sets of HotpotQA, 2WikiMultiHopQA, MuSiQue datasets. For LongBench datasets, we tested all samples following the common practice.\n\n***Q4: Hypotheses for the Best Performance on Medium-sized Language Models***\n\n(i) Hypothesis 1: Balance between comprehension capacity and optimization potential. \n\nMedium-sized LLMs (e.g., Llama-3.1-8B, Qwen2.5-14B) demonstrate adequate semantic understanding yet remain vulnerable to input noise, especially in long-context scenarios. DSAS overcomes this by analyzing information flows to precisely identify critical paragraphs and amplify model focus. In contrast, smaller models (e.g., Llama-3.2-3B) lack sufficient comprehension capacity, while larger models (e.g., Qwen2.5-32B) approach the task performance ceiling, leaving little room for further gains.\n\n(ii) Hypothesis 2: Dependence on positional awareness.\n\nMedium-sized LLMs exhibit greater vulnerability to the \"lost-in-the-middle\" phenomenon. DSAS mitigates this through its position-aware weighting $g\\_m$. The experiments of ablation study reveal that removing $g\\_m$ causes a marked performance drop for Qwen2.5-7B and Llama-3.1-8B, whereas Qwen2.5-14B's exhibits a smaller decline."}, {"review_id": "57LwsiAABQ", "reviewer": "Reviewer_wMYQ", "summary": "This paper proposes a plug-and-play method to improve LLM's performance on the Multi-doc QA task. They accumulate document-wise attention weights (they call it information flow) to inspect the attention patterns on document context in a multi-doc QA task. They get two observations: 1) support documents generally receive more attention, and 2) correct answers pay more attention to the support document compared to incorrect answers. Therefore, they propose a plug-and-play method to automatically detect support documents by accumulating attention weights and then reweighting supporting documents and negative documents by the accumulated attention, which upweighs the attention on supporting ones and downweighs the attention on negative ones. Otherwise, they propose a Gaussian-based score to reward middle-position documents to deal with the lost-in-the-middle problem. They also propose a Reciprocal attention suppression module, which penalizes the inter-document attention. They conducted experiments on several multi-doc QA datasets and showed that their methods can beat vanilla LLM without their method.", "questions": "Please refer to the questions in the \"Strengths And Weaknesses\" section.", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "strengths:\n\ni. This method is plug-and-play and easy to implement. I believe it will be very attractive and interesting if we can get performance improvement by simply manipulating the attention weights.\n\nii. This method is not only applicable to multi-document question answering, but also has the potential to be applied to RAG and general long-context generation tasks.\n\nweakness:\n\ni. I believe there are some mistakes in equations 7 and 8. E(*) is the cumulative function of the half-normal distribution with mu=0 and sigma=1/sqrt(2) with support R^+. But p_e - mu and p_s - mu do not guarantee to be positive. In Figure 4, it seems that you just want to use the cumulative distribution function of a general standard normal distribution. I do not understand why you write in this way. Additionally, in equation (8), it seems that you want to estimate \\gemma by (F(x_2) - F(x_1))/2\\sqrt(2)(x_2 - x_1) where x_2, x_1 is the normalized p_e and p_s, and F is the cumulative distribution function of N(0,1). This is naive because mu is set to the half-length and the derivative is high when x=0 (i.e., p_s and p_e near the middle). But I am confused about the discounting factor 2\\sqrt(2), won't this make \\gemma a quite small number and make g_m almost 1 everywhere?\n\nii. Ambiguous symbol usage, which is confusing. For example, A^{S}_{h,l} in equation 3, mu and sigma in equation 6 and equations 7&8. \\omega_m in equation 11, A^{S}_{h,l} in 12.\nquality:1 \n\niii. Comparing only to one single baseline is not convincing enough. What are the implementation details of the only baseline method? Is it just vanilla LLM? Why do not compare with other baseline methods? I believe that there are several attention manipulation-based works in long-context generation or RAG, for example [1]. Are there any other baselines that can be implemented, such as masking documents lower than a threshold?\n\niv. In equation 1, you accumulate attention weights of question-length for each j without normalization, while in equation, there is only one attention weight for each j. However, in Figure 2, I_q and I_t show similar magnitudes. Why does this happen?\n\n[1] Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization", "was_revised": true, "final_justification": "I had a thorough discussion with the author of the paper. Given the typos and ambiguous symbols in important formulas and the omission of important baselines (which would require a complete rewrite of the entire experiments and related work sections), this paper requires significant revision for resubmission. Therefore, I give a score of 3 (Borderline reject).", "rebuttal": "We sincerely appreciate the reviewer's thorough evaluation regarding equations, symbol definitions, and methodological comparisons.\n\n***weakness i: Confusions about Equations*** \n\nWe welcome the opportunity to clarify Equations (7) and (8), which are defined in the manuscript as follows:\n$$\n\\begin{align}\nF(x) &= \\int\\_{-\\infty}^{x} f(t)\\mathrm{d}t = 0.5 \\left( 1 + E\\left( \\frac{x - \\mu}{\\hat{\\sigma} \\sqrt{2}} \\right) \\right), \\quad\nE(z) = \\frac{2}{\\sqrt{\\pi}} \\int\\_{0}^{z} e^{-t^{2}}\\mathrm{d}t \\\\\\\\\n\\gamma\\_m&=\\frac{\\hat{\\sigma}\\left(E\\left(\\frac{p^m\\_e-\\mu}{\\hat{\\sigma}\\sqrt{2}}\\right)-E\\left(\\frac{p^m\\_s-\\mu}{\\hat{\\sigma}\\sqrt{2}}\\right) \\right)}{2\\left(p^m\\_e-p^m\\_s\\right)}\n\\end{align}\n$$\n(i) As stated in line 158 of this paper, $E(\\*)$ is the **error function** and is used to define the CDF of the Gaussian distribution. While $E(\\*)$ shares the same functional form as the CDF of the half-normal distribution (with $\\mu$=0 and $\\sigma=1/\\sqrt 2$ with support $R^+$), it is defined over the entire real domain $R$.\n\n(ii) Equation (8) aims to compute a normalized difference quotient analogous to a derivative. Based on the derivation below, Equation (8) is actually equivalent to $\\frac{F(z\\_2)-F(z\\_1)}{z\\_2-z\\_1}$, which lacks $2\\sqrt 2$ in the denominator.\n$$\n\\begin{align}\nAssume& \\quad z\\_1=\\frac{p^m\\_s-\\mu}{\\hat{\\sigma}}, z\\_2=\\frac{p^m\\_e-\\mu}{\\hat{\\sigma}}. \\\\\\\\\nF(x)&=0.5\\left( 1 + E\\left( \\frac{z}{\\sqrt{2}} \\right) \\right). \\\\\\\\\n\\gamma\\_m&=\\frac{\\hat{\\sigma}\\left(E\\left(\\frac{p^m\\_e-\\mu}{\\hat{\\sigma}\\sqrt{2}}\\right)-E\\left(\\frac{p^m\\_s-\\mu}{\\hat{\\sigma}\\sqrt{2}}\\right) \\right)}{2\\left(p^m\\_e-p^m\\_s\\right)} \\\\\\\\\n&=\\frac{\\hat{\\sigma}\\left(E\\left( \\frac{z\\_2}{\\sqrt{2}} \\right)-E\\left( \\frac{z\\_1}{\\sqrt{2}} \\right)\\right)}{2((\\hat{\\sigma}z\\_2+\\mu)-(\\hat{\\sigma}z\\_1+\\mu))} \\\\\\\\\n&=\\frac{F(z\\_2)-F(z\\_1)}{z\\_2-z\\_1}\n\\end{align}\n$$\n(iii) When $p\\_s$ and $p\\_e$ reside near $\\mu$ (i.e., the midpoint of the sequence), their normalized values $z\\_1$ and $z\\_2$ approach zero. Under these conditions, the resulting derivative $\\gamma\\_m$ (i.e., the positional value of the m-th paragraph) is high. The bigger $\\gamma\\_m$ value aligns with our core design objective: **Mitigate LLMs' \"lost-in-the-middle\" by enhancing attention weights assigned to centrally located information.**\n\n(iv) Based on the derivation of Equation (8) above, the denominator excludes the $2\\sqrt 2$ factor. Consequently, this formulation does not make $\\gamma\\_{m}$ a quite small number. \n\nShould any uncertainties remain regarding this explanation, we welcome further discussion. To prevent confusion, we will enhance the textual descriptions of Equations (7) and (8) and add the complete derivation in the revised manuscript.\n\n***weakness ii: Ambiguous Symbols*** \n\n(i) $A^S\\_{h,l}$ denotes the attention score matrix of the h-th head in the l-th layer. Equation (3) first computes the initial attention scores through dot products. We then apply both CGW and RAS to update its value. Our formulation follows the expressions in [1]. Equation (12) uses the calculated contextual gate weight of each paragraph to update $A^S\\_{h,l}$.\n\n(ii) To resolve notational ambiguity, we clarify the distinct roles of $\\mu$ and $\\sigma$ across equations.\n\n$\\mu$ in Equation (6) denotes the mean of combined information flow $\\mathcal{I}\\_{comb}$, defined in Line 153; $\\mu$ in Equation (7) serves as a normalization value for scaling; $\\mu$ in Equation (8) represents the mean of sequence $\\{0,1,\\ldots, L-1\\}$, stated in Line 160.\n\nWhile $\\sigma$ commonly denotes both sigmoid function and standard deviation, we use $\\sigma$ for sigmoid function and $\\hat{\\sigma}$ for standard deviation in Equations (6)-(8) to disambiguate.\n\nHowever, the similar symbols may cause misunderstandings, so we revised Equations (6)-(8) to make them clearer:\n$$\n\\begin{align}\nv\\_m &= 0.5\\cdot \\text{Sigmoid}\\left( \\frac{\\mathcal{I}\\_{Comb^m} - \\mu\\_I}{\\sigma\\_I} \\right)+0.5,m \\in \\{1,\\ldots,C\\}, \\\\\\\\\nF(x) &= \\int\\_{-\\infty}^{x} f(t) \\, \\mathrm{d}t = 0.5 \\left( 1 + E\\left( \\frac{x - \\mu}{\\sigma \\sqrt{2}} \\right) \\right), \\quad\nE(z) = \\frac{2}{\\sqrt{\\pi}} \\int\\_{0}^{z} e^{-t^{2}} \\, \\mathrm{d}t, \\\\\\\\\n\\gamma\\_m&=\\frac{\\sigma\\_p\\left(E\\left(\\frac{p^m\\_e-\\mu\\_p}{\\sigma\\_p\\sqrt{2}}\\right)-E\\left(\\frac{p^m\\_s-\\mu\\_p}{\\sigma\\_p\\sqrt{2}}\\right) \\right)}{2\\left(p^m\\_e-p^m\\_s\\right)},\n\\end{align}\n$$\n(iii) To explicitly combines $v\\_m$ and $g\\_m$ while constraining $w\\_m$ within $[\\beta,1]$, we introduce an intermediate variable $w\\_m^{'}$ for Equations (10) and (11): \n\n$$\n\\begin{align}\n    w\\_m^{'}&=v\\_m \\cdot {g\\_m}^\\alpha, \\\\\\\\\n    w\\_m&=(1-\\beta)\\frac{w\\_m^{'}-\\text{min}(w\\_i^{'})}{\\text{max}(w\\_i^{'})-\\text{min}(w\\_i^{'})}+\\beta,i \\in \\{1,\\ldots,C\\}\n\\end{align}\n$$\n\n(iv) Lastly, To mitigate potential notational ambiguities, we will add Table 1 of key symbols in Appendix.\n\nTable 1 Symbols and descriptions.\n\n| Symbol | Description |\n| --- | --- |\n| $A^S\\_{h,l}$ | the attention score matrix of the h-th head in the l-th layer |\n| $A^W\\_{h,l}$ | the attention weight matrix of the h-th head in the l-th layer |\n| $\\mathcal{I}\\_{{comb}^m}$ | the combined information flow for the m-th paragraph |\n| $\\mu\\_I, \\sigma\\_I$  | the mean and the standard deviation of $\\mathcal{I}\\_{comb}$ |\n| $v\\_m$  | the normalized value of $\\mathcal{I}\\_{{comb}^m}$ |\n| $\\mu\\_p, \\sigma\\_p$ | the mean and the standard deviation of sequence $\\{0,1,\\ldots,L-1\\}$ |\n| $\\gamma\\_m$ | the positional value of the m-th paragraph|\n| $g\\_m$ | the position-aware weight of the m-th paragraph |\n| $w\\_m^{'}, w\\_m$ | the temporary and the final contextual gate weight of the m-th paragraph |\n| $K$  | the number of tokens selected to compute $\\mathcal{I}\\_{{comb}^m}$ |\n| $n$  | the proportion of layers inserting DSAS |\n| $\\alpha, \\beta$ | hyperparameters for computing $w\\_m$ |\n\n[1] Xiao, Da, et al. \"Improving transformers with dynamically composable multi-head attention.\" arXiv preprint arXiv:2405.08553 (2024).\n\n***weakness iii: Comparisons of Different Methods***\n\n(i) As described in line 205 of the manuscript, our baseline implementation deploys all LLMs in the ''bfloat16'' data format and employs greedy-decoding to ensure full reproducibility of generated outputs. \n\n(ii) The Calibrated attention proposed in [2] shares a similar motivation with our approach in seeking to enhance LLMs' focus on central information within long contexts. **However, DSAS extends this objective by actively promoting information flows from supporting paragraphs while suppressing information flows from irrelevant information.** Specifically, Calibrated Attention quantifies the model's positional bias by fixing a query-paragraph pair and measuring how the self-attention distribution changes as the paragraph is relocated within the input. During inference, it subtracts the measured positional bias from the original attention scores to obtain calibrated attention weights. One limitation is its requirement to precompute the positional bias for each paragraph, which may entail significant computational overhead. Since the implementation of [2] is not available, we reproduced a version guided by its core methodology. The results are summarized in Table 2.\n\n(iii) To our knowledge, existing attention efforts mainly concentrate on sparse attention mechanisms [3], context compression [4], and KV cache optimization [5], while research addressing performance improvements in long-context generation remains underexplored.\n\nTable 2 Comparison results of different methods.\n\n| Models       | Method    | HotpotQA | 2WikiMultiHopQA | MuSiQue | L-HotpotQA | L-2WikiMultiHopQA | L-MuSiQue |\n| :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: |\n| Qwen-2.5-7B  | Calibrated | 44.5 | 48.6 | 33.6 | 57.1 | 51.6 | 30.3 |\n| | Ours       | **46.1** | **49.9** | **35.0** | **57.7** | **52.7** | **33.4**  |\n| Llama-3.1-8B | Calibrated | 46.4 | 50.0 | 37.3 | 55.2 | 46.5 | 28.6 |\n| | Ours       | **47.1** | **50.8** | **39.2** | **56.5** | **47.3** | **32.0**  |\n| Qwen-2.5-14B | Calibrated | 50.4 | 57.1 | 40.4 | 58.1 | 54.9 | 36.4 |\n| | Ours       | **51.8** | **58.2** | **43.8** | **60.9** | **56.1** | **39.3**  |\n\n[2] Hsieh, Cheng-Yu, et al. \"Found in the middle: Calibrating positional attention bias improves long context utilization.\" arXiv preprint arXiv:2406.16008 (2024).\n\n[3] Jiang, Huiqiang, et al. \"Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention.\" arXiv preprint arXiv:2407.02490 (2024).\n\n[4] Zhang, Peitian, et al. \"Long context compression with activation beacon.\" arXiv preprint arXiv:2401.03462 (2024).\n\n[5] Qin, Ziran, et al. \"Cake: Cascading and adaptive kv cache eviction with layer preferences.\" arXiv preprint arXiv:2503.12491 (2025).\n\n***weakness iv: Definitions of Equations*** \n\nWe thank the reviewer for raising this point and apologize for the oversight. Equation (1) computes attention weights summed across all tokens in the question (length $Q$). In contrast, Equation (2) utilizes the attention weight for a single target token $t$. To make the equations directly comparable, our code implementation averages the accumulated attention weights over the sequence length $Q$ in Equation (1). We will correct Equation (1) accordingly in the final version. Figures 3 and 7 remain correct with this revision. The updated Equation (1) is given below:\n$$\n\\mathcal{I}\\_{p^m,q} = \\frac{1}{Q} \\sum \\text{Top-K} \\left( \\left\\\\{  \\sum\\_{i \\in q} A\\_l^W(i,j) | j \\in p^m      \\right\\\\} \\right).\n$$"}]

Author rebuttal:

Meta-review: This paper studies the long-range dependency modeling problem, particularly in multi-document question answering. To mitigate the  long texts issue and ''lost-in-the-middle'' issue the authors propose a dual-stage mechanism which first, ranks the paragraphs, and multiplies the attention scores by learned weights and then mute the attention between key and irrelevant paragraphs by downscaling the cross-attention between them. Several experiments on open-source models and multi-document question answering datasets and reports gains in F1 scores in the majority of the settings. The experiment was later expanded to 2 other tasks in the LongBench benchmark (coding, summarizations).

Reviewers recognise that this is a key challenge in  the field of LLMs and question answering. They also recognize the simplicity of the approach (no post-training) and consistent improvements in the quality. Reviewers also suggested areas to improve such as 
1) theoretical justification of the method 
2) discussions of overhead of the method and failure modes.
Decision: Accept (poster)