SinkTrack: Attention Sink based Context Anchoring for Large Language Models
Authors: Xu Liu
Guikun Chen
Wenguan Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Multimodal Large Language Models
Hallucination
Context Forgetting
Primary area: foundation or frontier models, including LLMs

Abstract: Large language models (LLMs) suffer from hallucination and context forgetting. Prior studies suggest that attention drift is a primary cause of these problems, where LLMs' focus shifts towards newly generated tokens and away from the initial input context. To address this, we make use of a related, intrinsic characteristic of LLMs: attention sink – the tendency to consistently allocate high attention to the very first token (i.e., ⟨BOS⟩) of a sequence. Concretely, we propose an advanced context anchoring method, SINKTRACK, which treats ⟨BOS⟩ as an information anchor and injects key contextual features (such as those derived from the input image or instruction) into its representation. As such, LLM remains anchored to the initial input context throughout the entire generation process. SINKTRACK is training-free, plug-and-play, and introduces negligible inference overhead. Experiments demonstrate that SINKTRACK mitigates hallucination and context forgetting across both textual (e.g., +18.9% on QuAC with Llama3.1-8B-Instruct) and multi-modal (e.g., +23.0% on M3CoT with Qwen2.5-VL-7B-Instruct) tasks. Its consistent gains across different architectures and scales underscore the robustness and generalizability. We also analyze its underlying working mechanism from the perspective of information delivery. Our source code is available at anonymous GitHub.

Published as a conference paper at ICLR 2026
SINKTRACK: ATTENTION SINK BASED CONTEXT ANCHORING FOR LARGE LANGUAGE MODELS
Xu Liu, Guikun Chen, Wenguan Wang The State Key Lab of Brain-Machine Intelligence, Zhejiang University
ABSTRACT
Large language models (LLMs) suffer from hallucination and context forgetting. Prior studies suggest that attention drift is primary cause of these problems, where LLMs' focus shifts towards newly generated tokens and away from the initial input context. To counteract this, we make use of related, intrinsic char- acteristic of LLMs: attention sink the tendency to consistently allocate high attention the very first token (i.e., (BOS)) of sequence. Concretely, pro- pose an advanced context anchoring method, SINKTRACK, which treats (BOS) information anchor and injects key contextual features (such those de- rived from the input image or instruction) into its representation. As such, LLM remains anchored to the initial input context throughout the entire generation pro- cess. SINKTRACK is training-free, plug-and-pin ference overhead. Experiments demonstrate that SINKTRACK mitigates halluci- nation and context forgetting across both textual (e.g.. +21.6% on SQuAD2.0 with Llama3. Insttrrtrcc) and multi-modal (e.g., +22.8% on M3CoT with Qwen2.5- VI--7B-Instruee tasks. Its consistent gains across different architectures and scales underscore the robustness and generalizability. We also analyze its underlying working mechanism from the perspective of information delivery. Our source code is available at GitHub
and introduces negligible in-
INTRODUCTION
Background. Large language models (LLMs)
have demonstrated impressive capabilities in tex-
Hallucination
tual and multi-mode understanding (Bevilacqua
Response:
There are many planes on the road and trees on both sides rond.
ct al. [025] Xu et al. [2024] Guan et [ Ka-
Lyuzhnaya et al. 2025 Dong et al. (023). How-
Question:
ever, their deployment in critical applications is
Describe this image.
constrained by two hard, long-standing problems:
(a) Hallucination
hallucination (Chang et al. 2024 Zhang et al.
20256 Rawte al. 2023) and context forget-
User Input:
Response:
Question:
Since there are 68 pens, 74 oranges and (omit analysis)
ting (Liu et al. 2024a; Ling et al. 2023 Li et al. 2025 Hallucination is the generation of con-
There are apples.
oranges ..... B
tent factually inconsistent with the provided in- put. For instance, when asked to describe an image of buses on the road, LLM might iden- tify "planes" that do not exist in the image (cf. Fig. D)) Context forgetting occurs during long- form tasks, where LLM may lose track of initial instructions. resulting in unexpected answers. In track of the initial instruction on answer formation (ef. F Tb).
Answer Format:
The final answer
Answer:
Forgething
(b) Context forgetting
Figure 1: Hallucination and context forgetting are pervasive obstacles in LLMs.
multi-step reasoning dialogue, LLM might lose
Motivation. Prior studies (Peysakhovich & Lerer 2023 An al. 2025 Liu al. 2024a) iden-
tify attention drift as a direct cause of hallucination and context forgetting (cf- Fig. 2}. As LLM
generates sequence, its attention progressively shifts towards newly created tokens, while atten- tion on the initial tokens which often contain the most critical input information, such as image
Corresponding Author: Wenguan Wang.

---

Published as a conference paper at ICLR 2026
features or user instructions dramatically decays. Interestingly, observe that this drift coex- ists with countervailing phenomenon: the attention sink (Xiao et al. 2024 Kaul et al. (025) While attention
most initial tokens fades, the very first token (i.e., (BOS)) intrinsically main- tains a high degree of attention throughout generation, even though it is semantically sparse (ef. st column in Fig. 2 These two patterns, one of decay and one of stability, present synergistic opportunity. The stabil- ity of the attention sink provides natural mechanism to counteract the information loss caused by attention drift. Based on this insight, our core idea is to transform the passive attention sink into an active mechanism for con- text anchoring (i.e., using (BOS) as an information an- chor), thereby mitigating the information decay caused by attention drift. Through injecting key contextual features (such as those derived from the input image or instruc- tion) into the representation of (BOS), LLM remains chored to the initial input context throughout generation.
Attention Drift
B06
Image
Other
Methodology. To implement the above idea, i.e., ing (BOS) as an information anchor, we make an iter- ative exploration. The exploration begins with hard injecctionn iiection on ($2.2). which directly replaces the Value vector of (BOS) in the KV cache with mean-pooled information vector (e.g., from image instruction features). This implementation leads to a collapse, revealing that LLM's original computational flow should be preserved. We then investigate soft injection ($2.), which blends the pooled vector into LLMs using weighted sum to respect the original flow. Despite effective, there concerns regarding its ro- bustness: relies on manually tuned hyperparameters and injects all pooled representations into (BOS) equally- which might add noise and hurt performance on long contexts. Motivated by these limitations, our final solution SINKTRACK ($2.4). which is built upon a dual-track cross- cooss-ttentton mechanism. In this design, track enables (BOS) to adaptively query and retrieve relevant infor- mation from the context (i.e., cross-attention), while a parallel self-attention track preserves LLM's native computational flow, leading to consistent performance gains.
Figure 2: Attention drift and sink (the
darker the color, the greater the weight).
Merits. Through our study of anchor formulation, injection strategies, and source integrity, we de- lightweight, adaptive enhancement with three benefits: First, it is plug-and- play. As a training-free free, inference-time enhancement, SINKTRACK can be applied to any pre-trained LLM without the risks of fine-tuning, e.g.. catastrophic forgetting and model collapse. Second, is efficient. Its core intervention is a one-off, lightweight operation performed prior to generation. Since the intervention can be stored in KV cache, SINKTRACK adds negligible computational over- head to the subsequent auto-regressive steps, thereby preserving LLMS'' original inference speed. Third, it is generalizable. SINKTRACK makes use of attention sink, a common, intrinsic character- istic of decoder-only LLMs and multi-modal LLMs (MLLMs), to mitigate hallucination and context forgetting. This principle design makes inherently generalizable across architectures, scales, and modalities, validated in experiments ($3) different LLMs and MLLMs from 3B to 12B sizes.
sign SINKTRACK
Results and Analysis. SINKTRACK demonstrates consistent performance improvements over both direct prompting and chain-of- (CoT) reasoning across various multi-modal and text-based benchmarks ($3.2). For instance, on the multi-modal M3CoT benchmark (Chen et al.] 2024b), SINKTRACK increases the accuracy of Qwen2.5-VL-7B by 22.8% over CoT, and the reading comprehension benchmark, SQuAD2.0 (Rajpurkar et al., 2018), improves Llama3. 1-B's accu- racy by 21.6%. Qualitative analysis reveals that SINKTRACK enhances the logical structure and detail focus in LLM's reasoning process ($3.3..
We also explore the underlying working mecha-
nism of SINKTRACK ($.)
2
METHODOLOGY
We first review the phenomena of attention drift and attention sink that motivate our method ($2.1 Then, present the exploration undertaken to attention sink dynamic information anchor, beginning with an initial hard injection ($2.2,, progressing to a soft injection ($2.3) dual-track mechanism ($244
and leading the development of SINKTRACK,
2

---

Published as a conference paper at ICLR 2026
Contect:
There are many buses and
Repla- cement
Exploration
BOS
KV Cache
Image
Texs
(a) Hard Injection
2.2
Vision LLM Encoder
Weighted
Fusion
Direct
Mosn-pooled Info
BOS
Hidden
Path
(b) Soft Injection
BOS
Next Decoder Layer
Aduptively Augmentted
Final Design
Cross Attention
Initial Token
Feed Forward
Track
Key Injection
Add & Norn
Track
Attention
Tokens
Rest
(c) SINKTRACK ($2.4)
Attention Output
Figure 3: An overview of our methodology exploration.
2.1 ATTENTION PATTERNS LLMS
During auto-regressive generation, LLMs exhibit two fundamental and co-existing attention patterns that motivate our approach. The first is attention drift (Peysakhovich & Lerer [2023] An et al. [2025] Liu et al., 2024a) (cf. Fig. 2). where LLMs attention progressively concentrates on more recent tokens. As the generation proceeds, attention on the initial tokens, which contain the primary context (e.g., instructions or image), dramatically decays. This progressive loss of focus on the original input a primary cause of ballucination and context forgetting. The second pattern is attention sink (Xiao (et al. 2024 Kaul al. 2025). (cf. 1at column in Fig. 2 In direct contrast to the general decay of attention initial tokens, the very first token of the sequence (i.e., (BOS) intrinsically and consistently maintains an exceptionally high degree attention throughout the generation process. (BOS) is semantically sparse, yet it functions as LLMs' most stable point of attention.
These two patterns, one of decay and one of stability, operate tandem. Surprisingly, while these phenomena have been studied independently, their synergistic potential has been overlooked. This presents a clear opportunity: rather than attempting to alter these inherent behaviors, this work, to best knowledge, the first leverage their interplay. Our idea repurpose the passive attention sink into an active mechanism to counteract drift. Detailed implementations are given next.
2.2 HARD INJECTION VIA DIRECT REPLACEMENT
This work targets decoder-only LLM, which is composed of Transformer decoder blocks stacked sequentially. The 1-th block processes the sequence of hidden states from the preceding block, ((l-1) E Raxdd where is the sequence length and is the hidden dimension. Each decoder block contains two main sub-layers. The first multi-head attention (MHA) sub-layer:
( LayerNorm (I-1) MHA(Q = (1-11) K = H (1-1, V -H((")))
(1)
The second sub-layer is feed-forward network (FFN):
H (1) LayerNorm/ ( + FFN(M
(2)
Implementation. Based on the characteristic of attention sink, the most direct and simple method to implement our idea to place information the representation of (BOS). Therefore, we first attempt hard injection (cf. Fig. Ba), which directly manipulates the embedding of (BOS). Concretely, we directly replace the Value vector (V in Eq. corresponding to (BOS) the KV cache with external information vector finto- finto is derived from the input context. For the textual task, the prompt embeddings; for the multi-modal task, is the visual features from the MLLM's vision encoder. It is then compressed via mean pooling match the dimensionality.
Results and Analysis. Unfortunately, hard in injection leads to a complete model collapse, yielding meaningless outputs. This indicates that L.L''s internal computational pathways and
3

---

Published as conference paper at ICLR 2026
82
80.4
N0,78L.
CoT
31.2
77
Seft Injection
Soft Irjjection
Qwen 2.5 7B
Liama 3.1 88
72
Gemmo
Qwen 2.5 78
67
62
62.0
60.7
38.2
53.6
57
54.7
52
652
50.7
$3.1
$1,1314
47
34.7
36.2
42
RealWorldQA (786) MMStor (1,500)
M3CoT (2,318)
POPE (9,000)
QUAC (7 (7,832)
SQuAD2.0 (11,873)
(a) Multi-modal Tasks
(b) Text-hased Tasks
Figure 4: Soft injecti on achieves considerable performance improvements ($2.3)
learned attention patterns are sensitive to direct manipulation. Abruptly overwriting values in the KV cache disrupts this finely-tuned structure, leading severe degradation. This finding estab- lishes a core principle: any effective information injection must preserve the integrity the LLM's native computational flow, motivating our development of the more nuanced soft injectior.
2.3 SOFT INJECTION VIA STATIC FUSION
Implementation. In contrast the disruptive replacement used in hard in ject ion, soft injection fuses information into LLMs' computational flow. This is achieved by fusing a mean- pooled information vector with the original hidden state of (BOS) via weighted sum (ef. Fig, 3):
H = .. + +
a.) finoo
where H is the original hidden state of the first token (BOS) at injection layer I, which corre- sponds to the output of the FFN sub-layer (Eq.2) 2). finaa E is the mean-pooke mean-pooled information vector, is the injection strength, and H is the updated hidden state after soft inject on. Such an injection strategy holds two main hyperparameters: the injection layers and the injection strength a.
Results and Analysis. We evaluate soft inject ion in four multi-modal and two text-based tasks (I.I). While soft injection achieves considerable gains (ef. Fig. 4 and validates our core premise, its static design reveals fundamental limitations. Its performance is contingent on a man- ually tuned hyperparameter (i.e., injection strength a),, restricting its applicability. Furthermore, contextual information, lacks the capacity to adaptively select most salient content. This indiscriminate fusion can introduce noise and even degrade performance, highlighting the need for a dynamic mechanism that can adaptively query the source context.
by uniformly aggregating
2.4 SINKTRACK: DUAL-TRACK CROSS-ATTENTIC MECHANISM
To overcome the limitations of soft injection, we intro- 83
Saft Injection
$1.0
81,0815S
SinkTrack
duce SINKTRACK, adaptive 78
Qwen2 5 V.
mechanism centered on dual-
Genmuó
track attention design. SINK- TRACK achieves dynamic in- formation infusion by separat- ing the attention computation for (BOS) from that of all sub- sequent tokens (cf. Fig. 3p). This separation allows SINK- TRACK to inject context adap- tively while preserving LLM's native computational flow.
73
56
63
62.5
62.062.5
60.7
53
$1.0
$1.6
51.4
30.3
RealWoridQA (786) MMStor (1,500)
M3CoT (2,318)
POPE (9,000)
Figure 5: SINK TRACK further improves performance.
Track 1: Adaptive Injection via Cross-Attention. The first track performs adaptive injection exclusively for (BOS). This is achieved by modifying MHA (Eq. [1 at designated injection layers. Concretely, the hidden state of (BOS) at injection layer I, H serves as the Query (Q), while the Key (K) and Value (V)
derived from the external, mean-pooled information vector (fffa).
4

---

Published as a conference paper at ICLR 2026
(a)
(b)
69
23
69
Pooling SinkTrock
SinkTrack
Acc. (%)
64
59
$8.8
56.1
2
(96) mmp '3V
64
59
$0,559.3
$7.9 SK.1
56. 16.1562
Acc. Gain (%)
53.2
54
53,253.6
49
49
44
17
44
19
39
37.3
37.5
39
37.3
37.5
34.7
34.7
34
34
QuAC3
QuAC4
QuAC-2
QuAC
Figure 6: Effect of mean-pooling in information injection (Eq. 4 (a) Using the mean-pooled infor- mation vector; (b) using the original (unpooled) one.
This is implemented by cross cross- attentint:
H MHA(Q = H1O K- f f
(4)
where H denotes the updated hidden state after cross-attention. While the cross-attention is de- signed for adaptive injection, one may wonder: how can the low-information query vector H retrieve information adaptively? We emphasize that this adaptive capability is a layer-wise infor- mation accumulation. At the initial injection layer, H6") aggregates a holistic representation of the
global context from fino. This aggregation infuses H with context, thereby enabling it for- mulate targeted queries in subsequent injection layers. This progression allows (BOS) to identify salient information dynamically, thus obviating the need for a manually tuned injection strength.
Track 2: Preserving Native Computation via Sell-Attention. The second track ensures that LLM's computational flow remains undisturbed. All other tokens (i.e., regular tokens) perform the standard causal self elffaateenioon computation, attending to the original, unmodified input sequence.
Sequence Reassembly. The two tracks operate in parallel within the same attention block. The final output is concatenation of the updated (BOS) representation from Track I and the regular token representations from Track 2. This dual-track mechanism allows SINKTRACK to function as an adaptive context anchor, retrieving necessary information without compromising the foundational structure of LLMs. The superiority of this design is demonstrated by its consistent performance gains on long-context benchmarks like QuAC (ef. Fig. 5)
The Effect of Mcan-Pooling. Despite outperforming CoT, we observe that SINKTRACK's per- formance gains on the long-dialogue QuAC dataset diminish the context length grows (cf. Fig(6h). We identify the cause as the mean-pooling operation, a step inherited from hard and soft injection which inevitably causes information loss by compressing variable-length quence into a fixed-size vector problem that intensifies with longer contexts. However, the cross- attention mechanism in SINKTRACK offers structural advantage, length alignment between the Query and the Key/Value. We therefore eliminate the lossy pool- ing step, instead using the complete, uncompressed sequence of contextual features as the source. This modification effectively mitigates the trend of diminishing gains and preserves performance on long-context tasks (ef. Fig(op). The complete process is described in Alg. to ease understanding.
it does not require sequence
EXPERIMENT
3.1 EXPERIMENTAL SETTING
Datasets. We select four multi-modal datasets and two text-based datasets to assess the performance of SINKTRACK for different modalities.
Multi-modal Datasets. Four multi-modal datasets are used for evaluation: RealWorldQA (X.AI, 2024), visual question answering (QA) dataset designed to assess model robustness in complex, real-world scenarios; MMStar (Chen et al. 2024a), a comprehensive benchmark for assessing multi-modal capabilities that encompasses core abilities across 18 main tasks; M3CoТ (Chen) et al. 2024b), task collection focused on multi-modal, multi-hop, complex chain-of-thougi rea- soning; and POPE (Li et al., 2023b), a benchmark specifically designed to quantify object-level hallucination in multi-modal models.
5

---

Published as a conference paper at ICLR 2026
RealWocidQA (768)
MMStar (1,500)
M3CoT (2,318)
POPE (9,000)
Average
Model
Acc
Macro-F1
Acc
Macro-Fi F1
Acc
Macro-Fi
Acc
Macro-FI
Acc
Macro-Fi
Owen2.5. 5.
Insttrct
Rai et al.
2025
Dizect
31.15
23.25
24.89 11.05
39.54 30
33.43.0.4
44,20
53.23 n3.24
69.48 10.20
35.68
44.12
Cot
35.69 1144
26,46 m.
35.24 0.33
50.68
0..5
20.69 1..1
32.04
64.58 o..22
78.10 1o.6
39.05
46.82
SERRTRACK
48.28
28.42
61.38
48.25
77.69 20.12
58.99
Genuma3-4P Iustret
Trom
2025
Direct
to.35
48.43.1.10
35.47....0
52.17.0.12
33.8
83.98.0.a
91.30..o.z
49.62
56.43
CoT
49.41arm
44,83
31.36.1110
67.52
a
50,962039
59.71
82.01
90.04 g0
58.44
65-45
SIRREXACK
L2.46.0.43
47.39
53.04.01
69.11 t0
20.01.07m
59.18
84.33.o.0 uo
91.50 c0.00
60.21
66.80
Qwen2255
VL--B---BBB- Insttrcc Bai et
2025
47.49gmm
39,93
37.29
54. no
39.20gam
$1.01
28
87.15
50.55
58.08
Coll
57.86
43.00.4.1
55 14.4...
71. 20
44.11.00
57.96 26.96..
83 83.65
90.63
60.37
65.72
SIRKTSACK
65.49
52.81 10.00
63.78
77.86.0.17
66.94.o.
79.13
85.47 uo
91.67 20.00
70.42
75.37
Gemma3 128
Instruct Team al. 2025
Year
57.34
50,96
20.89
41.90gm.m
46
84. or
91. 21
57.10
63.74
57.69
49.92.s.m
60.56
75.09
60.32
72.24
84.53
91.6 91.6aamm
65.78
72.22
SIRRTRACK
$7.91 pm
$2.54
61.60
75.81
60.57
70.97.mvv
85.40
92.13
66.37
72.86
Table I: Performance of MLLMs on multi-modal tasks in terms of Accuracy (%) and Macro-F1. The bold indicates the best performance for the base LLM.
Text-based Datasets. Two text-based datasets used for evaluation: QuAC (Choi al. 2018
conversational QA benchmark, to measure the model's 's ability to handle long-context dependencies. Its structure, where questions build upon the dialogue history, allows us to analyze performance across progressive dialogue turns (QuAC-1, -2, -3) as the context lengthens, thereby assessing the mitigation of context forgetting; and SQuAD2.0 (Rajpurkar al.) 2018), another QA benchmark to test LLMs' ability to abstain from answering when information is absent in the context, providing a direct measure of its capacity to avoid generating fabricated content.
Base LLMs. We validate the effectiveness and generalizability of SINKTRACK across mainstream. open-source LLMs and MLLMs. Specifically, Qwen2.5 (Qwen et al. 2025). MiniCPM3 (Hu al.,
2024). and Llama3. I (Grattafiori et al. 2024) are used for text-based tasks, while Qwen2.5-VL (Bai ct al.
2025) and Gemma3 (Team et al.
2025 are used for multi-modal tasks.
Competitors. Since SINKTRACK is training-free free and plug-and-play Direct queries with LLMs, and Chain-of-Thought (Wei et al., 2022) (CoT) are used as our primary competitors. Specifically, CoT guides the model to perform step-by-step reasoning by including instructions, "Let's think step by step", in the prompt, without applying any additional information injection mechanism. This comparison allows for clear isolation of the performance gains attributable to SINKTRACK.
Implementation Details. We adopt an intermittent injection strategy for both soft Injection and SINKTRACK. Specifically, the injection operation is applied intervals of every layers. This configuration is empirically determined based on our analysis (33.4.1), where we observe that such intermittent injection yields better performance compared to dense injection.
3.2 QUANTITATIVE COMPARISON RESULT
To ensure reliability and rule out the influence of generation randomness, we conduct all evaluations using three distinct random seeds. Consequently, we report the performance as the mean + variance across these independent runs. The quantitative results are presented in T multi-modal tasks and Table Saolee]fo for text- tasks. The findings show that SINKTRACE consistently outperforms both Direct and CoT by a clear margin across all tested LLMs, modalities, and benchmarks.
The results demonstrate that SINKTRACK consistently surpasses both Di rert and CoT baselines across all tested L.LMs (including Qwen2.5- and Gemma3) and benchmarks. Crucially, the low variance observed in SINKTRACK" performance indicates that anchoring the context via the atten- tion sink not only improves accuracy but also enhances the stability of the model's reasoning process compared to standard prompting methods.
Multi-modal Tasks. On benchmarks designed to assess complex reasoning and object-level hallu- cination, SINKTRACK shows substantial gains. For instance, on M3CoT, SINKTRACK boosts the accuracy of Qwen2.5-VL V-L-BB by +22.8% over CoT, On POPE, a benchmark specifically designed for hallucination detection, SINKTRACK consistently improves both accuracy and Macro-F1 scores

---

Published as a conference paper at ICLR 2026
QuAC-1 (1,000)
QuAC-2 (2,000)
Q3A003
(7,447
SQuAD2.0 (11,873)
Model
Asc
Macro.F1
Acc
Macro-Fl
Acc
Macro-FI
Acc
Macro-F1
Asc
Macro-F1
MiniCPM3-45 4B
(Hu et al.
2024
Direct
52.82
52.79
a.m
0.31
19
10.10
48.07
62.93
38.61
Co'l
42.69.n00
45.45
41,03
43.35
39.3 19.99
41.19
37
77
36,50
63.63
23.73
SINKTRACK
53.57
52.82
50.12
50.98
48.53
49.75 cm
47.29 mm
48.08
62. 22.74gr..
36.83
@wer2.5
5-7B
(Qwen et af. 2025
Direct
52.57.ua0
47.06-0.ar
52.00
48. 06
51.26
47.64....
49.47
46.71 co.1s
80.97
33.39....
CoT
38.47.155
12.11 1,300
37.30
39.12.033
36.34
37.92
36.4810.9
36.58 10.0
77.39..
28. 74
STHRTRACK
52 mo
47.52
52.98
48,08
$2.01
47.77 2m
50.25 p0
46.73 10
81.04
29. gs
1Jmma.. I.1 1-8
Instruct (Gronafori al. 2024
Direct
54,10
48.87
54.25
31.31
g3.70
21.20
$2.45
50,66
78.69 78.69....v
24.95
Coz
.4.90...12
46.09
4.00
$1.75
0.21
38.21 iacs
49.13 1x.r
34,47 1e:3
46.95 10.00
28.74
58.19 aa.
28.86 10.0
SIMKTNACK
59,40.0..0
54,06
58.05
54.25
c0
53445..
53.51
79,83
25. 25.72.0.3
Table 2: Performance of LLMs on text-based tasks in terms of Accuracy (%) and Macro-F1. The bold indicates the best performance for the base LLM.
(e.g-, 85.5% vs 83.7% accuracy and 91.8% vy 90.6% Fl score compared to CoT), confirming its effectiveness in mitigating the generation of non-existent objects.
Text-based Tasks, We use long-context conversational QA to specifically measure the mitigation of context forgetting. As the conversational context lengthens from QuAC-I 1 to QuAC, the performance of baseline methods degrades due to attention drift. In contrast, SINKTRACK maintains a strong and often widening advantage. The most significant gains are seen with Llama3.1 where on the longest-contest QuAC setting (7,354 questions). SINKTRACE achieves an accuracy of 56.2%, a 7.1% % increase CoT. On SQuAD2.0, which tests the LLM's ability abstain from answering unanswerable questions, SINK also yields consistent improvements.
Analysis of CoT Performance. It is worth noting that CoT sometimes underperforms Direct Prompting, an observation consistent with existing studies. This performance degradation is highly task-dependent. In long-context understanding tasks, such as summarization and QA, CoT can even degrade performance (Liu et al., 2025 Wang & Sun 2025). Furthermore, while CoT excels in complex reasoning (Sprague et al.) (2225) like M3COT, in non-reasoning tasks, its extended chains exacerbate attention drift and raise the risk of error accumulation, where single misstep derails the final answer (Shen et al. 2025 Liu et al. 2025).
Taking together, these results validate that SINKTRACK is a broadly effective and generalizable enhancement. It directly addresses both hallucination and context forgetting across different modal- ities, tasks, and model architectures without requiring any model training.
3.3 QUALITATIVE COMPARISON RESULT
We present qualitative comparisons to illustrate how SINKTRACK improves LLM's reasoning. In cases where CoT fails (Fig. 7 left), SINKTRAS succeeds by grounding its analysis in specific contextual details (e.g.. skis and poles) rather than ambiguous cues. This advantage also extends to failure cases where both methods produce incorrect answer due to knowledge deficits (Fig- 7 right). Even then, SINKTRACK's reasoning chain remains internally consistent, whereas CoT ex- hibits context forgetting, where its final conclusion contradicts its own intermediate steps. This indicates that SINKTRACK enhances the generation process by enforcing closer adherence to the initial context and maintaining logical integrity, even when LLM's underlying knowledge is flawed.
3.4 ANALYSIS
3.4.1
INJECTION STRATEGIES
To study the effects of two main hyperparameters: the injection layers and the injection strength (2.)), we analyze the perfor- mance of various soft injection on using Qwen2.5-VL-7B- Instruct on M3COT. The results in Table Thighlight two criti- cal findings: i) Injection Frequency. Intermittent injection (In- ject Every 5 Layers) is more effective than continuous injection (Inject Every Layer), with 9.0% absolute accuracy improve- ment. This suggests that overly frequent intervention might dis- rupt L.M''s native computational flow to some extent. ii) Injec-
Method
Ace (%)
CoT (Baseline) Tnject Every Layer Inject Every 5 Layers
43.83
$1.02
60.04
Intermittent & Increasing
59.87
Intermittent & Decaying
60.69
Table 3: Performance of different soft inject ion variants( (2.3)
7

---

Published as a conference paper at ICLR 2026
Q: What is the event that the skier is participating as?
Q: Which property do these objects have in common." ? Hanl
A. Saunbserding conctaiunnet
Ski jumping competities
Soll
Alpine skinng eniettimment
Choneconitty dking competiion
Coll Response:
Stutied dice are filled with cushioning matenial to make them
louncy casy torell, Clty flower not ouallly wall, bui fixi siightly
CoT Response: The sá resect seems to be sfnail, local one without a major puoiion stmucture The prevenee yoing people and thelosuual posunjsnggnt recreation rather Bann Sonnal au sporing evess.
millaabee Teurss bill made af sill but compresiibe clattic rubber Rubiicr bana
Arudlysi pluable syathesi rubber polymer.
Anower:
Vogue Genordliaah
So laadl sypically wilt we cau indestinn wdll)
Self Controdiction
Our Respouse: The innge abuwe a shicr wills and pulce but there sec su visile sight of sói jumping mmp or any other specitss coupmeeti typically used for ski jumping comperitors. The skier appeors be sonding on (las ground which is more characterisii of cross country skingg tuther than zlpine dillng of snow boardig
Our Respouse: Saaffia dise are made of will unsicial liee falvic or from. Clay
Fower pot is soft and milloable when wet Vomii balls are made of soft outer layer
and a nither core Itabher hands are made of nibber, which is wofh and flesible.
Lets consider the propentes Nane of tlese objects appear be hard All Sour objects
[te oteer berl adllio some Whils chyste upecced this the
Arnou:::
Detail Oriented Andlysts
propeity they slare.
the property
objects islunt))
Coneistentt Legic
Q: Look or the following schodurle Girs just miseod the
9.45 AM bus the hissoric house.
Q: Which of the fullowing is true about the tollet?
How long does she have to wait uncil the next bas?
A. has a sepurate compatteent Fx the bidet.
B. 45 minates
hour
C. 15 minures
D. 30 minutes
C. It collred towel banging beside the bider.
R. Itue low flow technoloyy ?
It locoted South America
#|1l A.M
nett it45
Response: sable shows that the ned mvstlablc bas Gins world
9-45
not located in sapariee compartment [hhre to specific lobel to confirm the soilet tocm low--llow technology The towel langing white, colonad, and there visual clues to determine the geographic location.
Response: Besad the imuge can tollct
but they see bods
Regarding the specific schedble, Cina cannor take the 9:00 AM his and the next one as 10:00AM The cime comperionn then calculated as:
0:45 PM tn 10:00 PM 45 mindee.
Chestic Lagic & Fouiity Goloulation
Thereione i1 has a reparits comparimen for the hidri
Self Controdichion
Our Response: We need to look at the schschile for the listrric hruse.
Response:
The shaws which comnected There soporate
with toilst biliet And
Since Gina mised
Thee bus times for historic house (215 A.M. (0000
A.M. bux, the nest bus 10.00 A.M.
A
indication
America. Additionalty, there
flow tochoology
towel hanging beside bitet. instead.
lucated South
(b:4 PM is 10:00 PM - 15 minstes.
Syshemotic Legit
there is white towel.
Therefore, Gisa has to wat 15 minutes for the sext bus.
Therefore (i his a sepaniee compatimest for the bides]
Cordistent Logic
Figure 7: Qualitative comparison between CoT and SINKTRACK.
tion Strength. A decaying schedule, which applies stronger injection in earlier layers, yields better performance (60.7% 59.9%). This indicates that early contextual anchoring is more beneficial than later-stage interventions. In addition, the sensitivity to injection strength motivates us to de- velop an adaptive mechanism 2.4).
3.4.2 HIERARCHICAL ATTENTION PATTERN
To analyze the structural impact of our injection, we quantify its effect on LLM's attention pattern. Concretely, we compute the Spearman's rank correlation between (BOS)'s 's attention weight vectors across all layers, before and after applying SINKTRACK, The analysis reveals a near-perfect positive correlation (p- 0.9985. p < 2.22 10 10-17), as detailed in Appendix $ This high correlation demonstrates that SINKTR Rather than disrupting LLM's s pre-trained attention hierarchy, SINK TRACK enriches the information content of (BOS) while maintaining the integrity of the original computational flow.
preserves the relative importance of attention across different layers.
3.4.3 INFORMATION FLOW
We further quantify the underlying working mechanism of SINKTRACK as an information anchor by analyzing its vertical and information flows.
Vertical Flow: Encoding via Value Vectors. The output of the attention mechanism is an aggrega- tion of value vectors weighted by attention scores:
O, = E ;=1 arj VS=
(5)
where arj represents the attention weight and V, denotes the value vector of the j-th token. Con- sequently, the information capacity of the (BOS) anchor can be quantified by calculating the L1 Norm of its value vector. As illustrated Fig. aa), a significant difference value (yellow line) is observed between the original state (Before) and the injected state (After). This variation serves as a proxy for Information Gain. The distinct divergence in LI Norm across layers confirms that the anchor information is successfully absorbed and encoded into the (BOS) representation, effectively transforming it from a standard placeholder into a semantic anchor.
Horizontal Flow: Sustained Attention Anchoring. To evaluate how this injected information is propagated during generation, we analyze the attention distribution of newly generated tokens. Specifically, we track the attention weights that subsequent tokens assign to the (BOS) anchor
8

---

Published as a conference paper at ICLR 2026
(a)
Before
After
Difference Válue
(b)
Before
After
27
26
0.46
25
24
Intormaatin Gain
Attention Weights
0.4
22
20
17
15
32
64
128
Layers
Index of Newly Generated Tokens
Figure 8: The illustration of information flow with and without SINKTRACK ($2.4)
(oH, As shown Fig. B(b), the attention weights exhibit a natural decay but maintain per- sistent magnitude, indicating that the generated sequence continuously attends to the (BOS) token. Crucially, the attention curves before and after injection (gray and red lines) overlap almost per- fectly. This phenomenon demonstrates two key findings: i) SINKTRACK successfully establishes a long-term information bridge, as the anchor remains a focal point throughout generation; and ii) The injection is non-destructive, preserving the intrinsic attention patterns of the LLMs.
RELATED WORK
Training-free free Inference-time Enhancement for LLMs. Developing blug-and-play enhancement methods improve the capabilities of LLMs has received widespread attention from the commu-
nity. These approaches can be broadly categorized into three main streams. The first category is
External Module Augmentation (Wang et al.) 2024 Ma al. 2024 Feng al. [024 Chen al.)
2022). These methods couple an LLM with external components to address limitations such as out-
dated knowledge the lack of specific skills, e.g,, retrieval-augmented (Wang al.] 2024 Feng
et al. 2024 Yu et al., (024a) and tool-augmente methods (Ma et al.) 2024] Gou et al. 2024 Lu
al.
2023 Chen al. [022 Wei et al.) 2022). The second category, Internal Reasoning Process
Enhancement (Dhuliawala et al.) [024]] Zhang et al., 2024cc Yao et al., 2023 ZZhng et al. 2024a
2025a), uses structured prompting strategies or adds reasoning steps to guide L.LM, e.g., decompos
ing complex problems into simpler sub-tasks (Khot et al.) 2023 Chen et al. [202]] Wei et al. 2022
Ning et al. 2024), generating multiple candidate outputs with verification (Gou et al. 2024
Wang
al. 2023 Zhang et al. 2024bbar a). and using non-linear reasoning structures to organize complex
inference paths (Yao et al. 2023; Roy & Roth 2015 Besta et al. 2024| Wei et al.
2022). The third
category is Internal Representation Intervention (Li et al. 2023a Zhu al.) 2025
Han al. 2025
Turner et al., [0223 Liu et al. 2024c), where methods directly modify the LLM's internal states dur-
ing inference guide its behavior, e-g., adding targeted offset to LLM's hidden states (Li al.,
2023a]Subrra Subramani et al. 2022 Turner et al. 2023) and calibrating LLM's attention (Zhu et al.] 2025 Yu et 2024b) Han et 2025 Lu al. 202I). Despite impressive, these methods often require repetitive intervention, introducing considerable computational cost at each generation step.
Unlike methods that apply continuous modifications during the decoding loop. SINKTRACK achieves training-free enhancement through a one-time intervention prior to inference. The injection of key information into the representation of (BOS) a one-off operation performed before gener- ation begins. Once this information anchor is established, the subsequent auto-regressive process is identical the standard decoding procedure, requiring no extra computational steps.
Attention Sink refers to the empirical observation that the initial token of sequence (i.e., (BOS)) consistently receives high degree of attention during auto-regressive generation. This phenomenon is considered emergent property of Transformer architectures (Xiao et al. 2024 Kaul et al. 2025 Gu et al. 2025). Proposed causes include being an artifact of the Softmax function within consequence of the token's 's stable structural position in the sequence Yu et al., 2024b] Ruscio et all, 2025; Canceddal 2024). Existing efforts
the attention mechanism (Barbero et al. 2025)
the attention sink in main ways. One line of work the initial token's stability for

---

Published as a conference paper at ICLR 2026
tasks like KV cache management (Zhang et al. 2023) and key token protection (Liu al.
2024b
Another research direction focuses on enhancing LLM's performance by modifying or calibrating the attention mechanism to synergize with this structural anchor (Han et al., 2025 Gu et al. 2025).
This work explores a new application branch for attention sink: repurposing it as an active mech- anism for context anchoring through information injection. We do not alter the attention weights themselves; instead, we modify the representation of (BOS) to form a high-information anchor that provides LLMs with high-integrity context. This shift from indirect attention calibration direct information supply reveals a new direct to address the issues of hallucination and context forgetting.
5 CONCLUSION
This work mitigates two long-standing issues (i.e., hallucination and context forgetting) in decoder- only LLMs by exploiting the synergistic relationship between attention drift and attention sink. We propose SINKTRACK, an efficient, plug-and-piln method that transforms the passive attention sink into active context anchor. Through adaptive rross-attention mechanism, SINKTRACK injects key contextual information into the first token's representation, thereby counteracting the informa- tion decay caused by attention drift during generation. Consistent performance improvements across diverse textual and multi-modal benchmarks validate its effectiveness and generalizability. and our analysis of its information delivery mechanism further substantiates the principles of SINKTRACK.
ETHICS STATEMENT
In this study, propose SINKTRACK, training-free free mechanism designed to enhance the relia- bility and reasoning capabilities of LLMs. We acknowledge that any technique that improves the fundamental capabilities of LLMs such as increasing their coherence reducing factual errors carries a dual-use potential. While our work is intended for benevolent purposes, more capable and robust models could theoretically be exploited by malicious actors generate persuasive and harder-to-detect misinformation.
Our research is focused exclusively on the positive application of mitigating object hallucinations and improving long-context reasoning on established, public academic benchmarks. The datasets and tasks used in our experiments do not involve sensitive personal data, human subjects, or the generation of socially harmful content. Our methodology is transparent and designed to be analyzed and understood by the research community.
REPRODUCIBILITY STATEMENT
All experiments are conducted using publicly available, open-source LLMs and MLLMs. Specific model checkpoints are obtained from the official Hugging Face Hub. The detailed formulation and algorithmic implementation of SINKTRACK are presented in $2] A comprehensive description of our experimental setup, including the specific datasets, evaluation metrics, and baselines used, is provided in $3] Any additional implementation details are further detailed in the Appendix. To facilitate full reproducibility. our complete source code, along with the scripts required to run all experiments and reproduce the results reported in this paper, is available at GitHub
ACKNOWLEDGEMENT
This work was supported by National Science and Technology Major Project (No. 20232D0121300), National Natural Science Foundation of China (No. 62372405), Fundamental Research Funds for the Central Universities (226-2025-00057), Jiangsu Provincial Scientific Re- search Center of Applied Mathematics (No. BK20233002), and CCF- Open Fund.
REFERENCES
Wenbin An, Feng Tian, Sicong Leng, Jiahao Nie, Haonan Lin, QianYing Wang, Ping Chen, Xiaoqin Zhang, and Shijian Lu. Mitigating object hallucinations in large vision-language models with
10

---

Published as a conference paper at ICLR 2026
assembly of global and local attention. In CVPR, pp.
2025.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint (at::2502.1392.. 2025.
Federico Barbero, Alvaro Arroyo, Xiangming Gu, Christos Perivolaropoulos, Petar Veličković, Raz- van Pascanu, and Michael M. Bronstein. Why do LLMs attend the first token? In COLM, 2025.
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gian- inazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In AAAI, pp. 17682-17690, 2024.
Marialena Bevilacqua, Kezia Oketch, Ruiyang Qin, Will Stamey, Xinyuan Zhang, Yi Gan, Kai Yang. and Ahmed Abbasi. When automated assessment meets automated content generation: Examining text quality in the era of gpts. ACM Transactions on Information Systems, 43(2);1-36, 2025.
Nicola Cancedda. Spectral filters, dark signals, and attention sinks. In ACL, pp. 1792-4802 2024.
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang. Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1-45, 2024.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language 2024a.
models? In NeurlPS, pp. 27056-27068
Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. M A novel benchmark for multi-domain multi-step multi-modal chain-of-thought. In ACL, pp. 8199-8221, 2024b.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompt- ing: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588 2022.
Eunsol Choi, He He, Mohit lyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zeuteemoer. QuAC: Question answering in context. In EMNLP, pp. 2174-2184, 2018.
Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. leain-of-verificale reduces hallucination in large language models. In Findings of ACL, pp. 3563-3578 2024.
Xin Luna Dong, Seungwhan Moon, Yifan Ethan Xu, Kshitiz Malik, and Zhou Yu. Towards next- generation intelligent assistants leveraging IIm techniques. In KDD, pp. 5792-5793, 2023.
Shangbin Feng, Weijia Shi, Yuyang Bai, Vidhisha Balachandran, Tianxing He, and Yulia Tsvetkov. Knowledge card: Filling LLMs' knowledge gaps with plug-in specialized language models. In ICLR, 2024.
Zhibin Gou, Zhihong Shao, Yeyun Gong, yelong shen, Yujuu Yang, Nan Duan, and Weizhu Chen. CRITIC: Large language models can self-correct with tool-interactive critiquing. In ICLR, 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Xiangming Gu Tianyu Pang, Chao Du, Qian Liu. Fengzhuo Zhang, Cunxiao Du Ye Wang. and Min Lin. When attention sink emerges language models: empirical view. In ICLR. 2025.
Yanchu Guan, Dong Wang, Zhixuan Chu, Shiyu Wang, Feiyue Ni, Ruihua Song. and Chenyi Zhuang. Intelligent agents with llm-based process automation. In KDD, pp. 5018-5027, 2024.

---

Published as a conference paper at ICLR 2026
Feijiang Han, Xiaodong Yu, Jianheng Tang, and Lyle Ungar. Zerotuning: Unlocking the initial to- ken's 's power to enhance large language models without training. arXiv preprint Seti:::555.1173390 2025.
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404. 06395, 2024.
Anna Kalyuzhnaya, Sergey Mityagin, Elizaveta Lutsenko, Andrey Getmanov, Yaroslav Aksenkin, Kamil Fatkhiev, Kirill Fedorin, Nikolay O Nikitin, Natalia Chichkova, Vladimir Vorona, et al. Lim agents for smart city management: Enhancing decision support through multi-agent ai sys- tems. Smart Cities (2624-6511), 8(1), 2025.
Prannay Kaul, Chengcheng Ma, Ismail Elezi, and Jiankang Deng. From attention to activation:
Unraveling the enigmas of large language models. In ICLR, 2025.
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. In ICLR, 2023.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference- time intervention: Eliciting truthful answers from language model. In NeurIPS, pp. 41451-41530, 2023a.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination large vision-language models. In EMNLP, pp. 292-305, 2023b.
Yubo Li, Xiaobin Shen, Xinyu Yao, Xueying Ding, Yidi Miao, Ramayya Krishnan, and Rema Pad- man. Beyond single-tur:: A survey on multi-turn interactions with large language models. arXiv preprint arXivv2504.0477 042750044.477, 2025.
Chen Ling, Xujiang Zhao, Jiaying Lu, Chengyuan Deng, Can Zheng, Junxiang Wang, Tanmoy Chowdhury, Yun Li, Hejie Cui, Xuchao Zhang, et al. Domain specialization as the key to make large language models disruptive: A comprehensive survey. ACM Computing Surveys, 2023.
Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. In TACL, pp. 157-173, 2024a.
Ruikang Liu, Haoli Bai, Haokun Lin, Yuening Li, Han Gao, Zhengzhuo Xu, Lu Hou, Jun Yao, and Chun Yuan. IntactKV: Improving large language model quantization by keeping pivot tokens intact. In Findings of ACL, pp. 7716-7741, 2024b.
Ryan Liu, Jiayi Geng, Addison J. Wu, Ilia Sucholutsky, Tania Lombrozo, and Thomas L. Griffiths. thouugh can reduce performance on tasks where thinking
Mind your step (by step): makes humans worse. arXiv preprint arXiv:2410.2/333 2025.
Shi Liu, Kecheng Zheng, and Wei Chen. Paying more attention to image: training-free method for alleviating hallucination in Ivims. In ECCV, pp. 125-140, 2024c.
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chamelcon: Plug-and-play compositional reasoning with large language mod- els. In NeurlPS, pp. 43447-43478, 2023.
Yu Lu, Jiali Zeng, Jiajun Zhang, Shuangzhi Wu, and Mu Li. Attention calibration for transformer in neural machine translation. In ACL, pp. 1288-1298 2021.
Yubo Ma. Zhibin Gou, Junheng Hao, Ruochen Xu, Shuohang Wang. Liangming Pan, Yujiu Yang, Yixin Cao, and Aixin Sun. SciAgent: Tool-augmented language models for scientific reasoning. In EMNLP, pp. 15701-15736, 2024.
Xuefei Ning. Zinan Lin, Zixuan Zhou, Zifu Wang, Huazhong Yang, and Yu Wang. Skeleton-of- thought: Prompting L.MSs for efficient parallel generation. In ICLR, 2024.
12

---

Published as a conference paper at ICLR 2026
Alexander Peysakhovich and Adam Lerer. Attention sorting combats recency bias in long context language models. arXiv preprint 2rX1::2310.01422,20000 2023.
Qwen, An Yang, Baosong Yang. Beichen Zhang. Binyuan Hui, Bo Zheng. Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin. Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv:2412.155 15, 2025.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for SQuAD. In ACL, pp. 784-789, 2018.
Vipula Rawte, Amit Sheth, and Amitava Das. A survey of hallucination in large foundation models. arXiv preprini urXiv:2309.0592, 2023.
Subhro Roy and Dan Roth. Solving general arithmetic word problems. In EMNLP, pp. 1743-1752, 2015.
Valeria Ruscio, Umberto Nanni, and Fabrizio Silvestri. What are you sinking? geometric approach on attention sink. arXiv preprint arXiv:2508.00 025466 2025.
Jianan Shen, Xiaolong Cui, Zhiqiang Gao, and Xuanzhu Sheng. Adaptive -faiinoo-tthught distilla- tion based on Ilm performance original problems. Mathematics, 13(22), 2025.
Zayne Rea Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. To cot or not to cor? chain-of-thought helps mainly on math and symbolic reasoning. In ICLR, 2025.
Nishant Subramani, Nivedita Suresh, and Matthew Peters. Extracting latent steering vectors from pretrained language models. In Findings of ACL, pp. 566-581, 2022.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint vrxiiv22503 2503. 2025.
Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering. arXiv preprint arXiv:2308. 10248, 2023.
Chupei Wang and Jiaqiu Vince Sun. Unable to forget: Proactive Interference reveals working mem- ory limits in Ilms beyond context length. In ICML Workshop on Long- -onttext Foundation Models, 2025.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In ICLR, 2023.
Zihao Wang, Anji Liu, Haowei Lin, Jiaqi Li, Xiaojian Ma, and Yitao Liang. Rat: Retrieval augmented thoughts elicit context-aware reasoning in fong-horizon generation. arXiv preprint arXiv:2403.05333 13, 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of- th-thughtt prompting elicits reasoning in large language models. In NeurIPS, pp. 24824-24837, 2022.
X.AI. Grok- 1.5 vision preview.
2024.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In ICLR, 2024.
Jia Xu. Weilin Du, Xiao Liu, and Xuejun Li. Llm4workflow: An Ilm based automated workflow model generation tool. In ASE, pp. 2398,, 2024.
13

---

Published as a conference paper at ICLR 2026
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In NeurIPS, 2023.
Wenhao Yu, Hongming Zhang, Xiaoman Pan, Peixin Cao, Kaixin Ma, Jian Li, Hongwei Wang, and Dong Yu. Chain-of-note: Enhancing robustness in retrieval-augmenter language models. In EMNLP, pp. 14672-14685, 2024a.
Zhongzhi Yu, Zheng Wang, Yonggan Fu, Huihong Shi, Khalid Shaikh, and Yingyan (Celine) Lin. Unveiling and harnessing hidden attention sinks: enhancing large language models without train- ing through attention calibration. ICML, 2024b.
Yongheng Zhang, Qiguang Chen, Min Li, Wanxiang Che, and Libo Qin. AutoCAP: Towards auto- matic cross-lingual alignment planning for zero-shot shain-of-thoughy In Findings of ACL, pp- 9191-9200, 2024a.
Yongheng Zhang, Qiguang Chen, Jingxuan Zhou, Peng Wang, Jiasheng Si, Jin Wang, Wenpeng Lu, and Libo Qin. "roog-of-thouughe An integrated reasoning framework with multi aulti-perspective verification and wrong information. In Findings of EMNLP, pp.
2024b.
Yongheng Zhang, Xu Liu, Ruihan Tao, Qiguang Chen, Hao Fei, Wanxiang Che, and Libo Qin. Vit- cot: Vidco-text interleaved tf-thuuhtt for boosting video understanding in large language models. In ACM MM, pp. 5267-5276, 2025a.
Yongheng Zhang, Xu Liu, Ruoxi Zhou, Qiguang Chen, Hao Fei, Wenpeng Lu, and Libo Qin. CCHall: A novel benchmark for joint cross-lingual and cross-modal hallucinations detection in large language models. In ACL, pp. 30728-30749, 2025b.
Yuansen Zhang, Xiao Wang, Tianze Chen, Jiayi Fu, Tao Gui, and Qi Zhang. Plug-and-play discrete prompting for large language models personalization. In Findings of ACL, pp. 9129- 9144, 2024c.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng. Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, al. H2o: Heavy-hitter oracle for efficient gen- erative inference of large language models. In NewrlPS, pp. 34661-34710, 2023.
Younan Zhu, Linwei Tao, Minjing Dong, and Chang Xu. Mitigating object hallucinations in large vision-language models via attention calibration. arXiv preprint arXhv:2502.01962 2025.
14

---

Published as a conference paper at ICLR 2026
SUMMARY OF THE APPENDIX
A details the pseudo-code implementation of SINKTRACK.
B provides in-depth analyses concerning the stability of the hierarchical attention pattern, resis-
tance to attention drift, and computational overhead.
discusses the limitations of the current approach.
A PSEUDO-CODE FOR SINKTRACK
To provide clear description of SINKTRACK, this section details the pseudo-code for it. As dis- cussed the main paper, the core idea is to create two parallel computational tracks within the attention module. Track (Injection Track) uses cross-attention inject external information into the initial token. Track 2 (Original Track) performs standard causal self-attention for the rest of the tokens maintain the integrity of the pre-trained sequence structure. The complete process, including the conditional check for applying the injection, is described in Algo. SI
Algorithm S1 Pseudo-code for the Dual-Track Cross-Attention Mechanism.
Lori:
infor hidden states wewe_erignnnneeeueeee (Lori Dx)) afg: conft iunituuttiion for injection rules
information thinfo Da)
Bybrid Attention (h_ori, hiinfo, efgh:
conditions tions for injaction are met do_inject Check Check conditionn icfg, h_info)
Check if condit
if do_injests
4--- PATE A: HYBRID ATTENCION FOR INJECTION ---
Project imputs
spaces
Q_ari, K_ari, V_ari info, Vinfo Project (V(h_info)
2b. Split the original Query for dual attention tracks first, 2_rest - Split Original [a_oorQ__ori,
3. Execute aual-track attencion Teack 1: INJECTION via Cross 2_first Cross (irq, K-K,info, V=V,info)
(ecss-Attent Attention
Track 2: CAUDAL SELF ATTENTION Create.Causall Mask rest) r_re
Attention rast_rest, K=K_ori, VeV_ori, masic=m)
4. Merge outputs from both tracks a combined Concatenate (a_first, e_rest)
else
STANDARD
ATTENTION
o_combined Standard Causal Attention (h_ori)
Apply final output projection hou Output Projettnrooettion (o_cembinee)
h_out
B IN-DEPTH ANALYSIS: MECHANISM AND EFFICIENCY
This section provides a deeper analysis the underlying mechanism of SINKTRACK and its com- putational implications.
B. STABILITY OF HIERARCHICAL ATTENTION PATTERN
To analyze the structural impact of our injection, we quantify its effect on the LLM's hierarchical attention pattern. Concretely, compute the Spearman's 's rank correlation between the (BOS)'s 's attention weight vectors across all layers, before and after applying SINKTRACK. As shown in Fig. SI| the analysis reveals near-perfect positive correlation (p 0.9985, p 2.22 x 10-17).
15

---

Published as a conference paper at ICLR 2026
0.9
Attention weights injection)
0.8
0.5
0.4
0.3
0.2
Spearman's 0.9985 p 2.22 x10-1?
0.1
0.2
0.4
0.6
0.8
Attention weights (w/o injection)
Figure S1: Comparison between the (BOS)'s 's attention weight vectors before and after applying SINK TRACK.
This high correlation demonstrates that SINKTRACK preserves the relative importance of atten- tion across different layers. Rather than disrupting the LLM's pre-trained attention hierarchy pat- tern, SINKTRACK enriches the information content of the initial token while maintaining the in- tegrity of the original computational flow. This finding provides a quantitative explanation for why SINKTRACK can enhance performance without causing the model collapse observed in hard injection.
B.2 VERIFICATION OF SINKTRACK'S STABILITY
To further verify that SINKTRACK remains effective over long generation sequences, we conduct "Drift Test" on Llama3. 1-8B. We feed it an input context of 8k+ tokens and track the attention weights allocated to the (BOS) token by subsequently generated tokens (steps 1024).
As presented in SI the attention weight to (BOS) remains dominant throughout the generation process. For instance, generated token index 1024 (sequence position 9106), the attention to (BOS) is 0.582, which is approximately 14x higher than the maximum attention paid to any other token. This confirms that SINKTRACK creates robust attention sink that is immune to attention drift, ensuring long-term context retention.
Table S1: SinkTrack Drift Test on Llama3. 1-8B. The attention to (BOS) consistently dominates other tokens even after generating 1024 new tokens.
Gen. Idx
Seq. Pos
Attn to (BOS)
Max Attn (Others)
Ratio
8083
0,561
0.057
9.87x
64
8146
0,570
0.048
11.97x
128
5210
0,612
0.054
11.25x
256
8338
0.560
0.050
11.24x
512
8594
0.558
0.037
15.17x
1024
9106
0.582
0.042
13.96x
B.3 COMPUTATIONAL OVERHEAD
We measure the prefill stage latency on an NVIDIA GeForce RTX 4090 to evaluate the compu- tational cost of SINKTRACK. The results, averaged over 500 samples, are presented in Table S2 Specifically, Llama3.1-8 1-8B shows a latency increase from 35.90 ms (w/o Injection) to 36.66 ms (w/ Injection), representing marginal overhead of only +0.76 ms. Similarly. Soven..5-VL77- shows an increase of +3.13 ms. These findings confirm that SINKTRACK introduces negligible latency, as the core intervention is one-off operation performed prior to the auto-regressive generation.
16

---

Published as a conference paper at ICLR 2026
Table S2: Prefill stage latency measurement on NVIDIA GeForce RTX 4090 (averaged over 500
samples). SINKTRACE introduces negligible computational overhead.
Model
w/o Injection
w/ Injection
Overhead
Llama3.1-3 1-8B
35.90 ms
36.66 ms
+0.76 ms
Qwen2.5-VL.7B
107.13 ms
110.26 ms
+3.13 ms
C
LIMITATION
This work proposes and validates SINKTRACC representing a solid step towards mitigating halluci- nation and context forgetting in LLMs. However, it is important recognize several inherent limita- tions in our current approach that open avenues for future research. Specifically, SINKTRACK'S core design anchors all contextual information the representation of single initial token. While our adaptive mechanism optimizes information retrieval, this setup introduces a potential information bottleneck, as single fixed-dimension vector has a finite capacity for representation. This may not fully capture the nuanced informational needs of tasks requiring extremely long dense contexts. For example, while effective, the diminishing performance gains we observe on the long-dialogue QuAC may be early indicator of this bottleneck. This limitation suggests need to explore fu- ture paradigms beyond single anchor, such as distributed or dynamically positioned anchoring mechanisms.
17


Reviews:
[{"review_id": "X5ijIrfrcF", "reviewer": "Reviewer_gHxw", "summary": "This paper proposes SinkTrack, a training-free, plug-and-play mechanism that enhances large language models (LLMs) by addressing two key issues: hallucination and context forgetting. The authors identify attention drift—the model’s tendency to focus increasingly on recent tokens during generation. They exploit an opposing, intrinsic behavior called the attention sink, where the first token (⟨BOS⟩) consistently receives high attention throughout decoding. SinkTrack injects key contextual information into the ⟨BOS⟩ token’s representation, and introduces a dual-track attention mechanism: one track performs adaptive cross-attention for ⟨BOS⟩, while the other maintains standard causal self-attention for other tokens. This allows ⟨BOS⟩ to query relevant information dynamically, counteracting information decay during long-context generation. The experimental results show its promise. SinkTrack requires no training, introduces negligible inference overhead, and is compatible across architectures and modalities. The authors also provide interpretability analyses of information flow, showing how the ⟨BOS⟩ token propagates injected information vertically (across layers) and horizontally (across tokens).", "questions": "Would you please provide more justification for attention anchoring, and its convergence properties?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "SinkTrack is training-free, plug-and-play, and requires only a one-time injection at inference.\n\nThe dual-track design elegantly balances adaptation (via BOS cross-attention) and model integrity (via standard causal flow), preserving pretrained representations while enhancing context retention.\n\nEvaluations span six datasets, covering both text and vision-language reasoning.", "weaknesses": "The approach is empirically strong but lacks formal justification for why attention anchoring should improve global consistency.\n\nThere’s no discussion of convergence, gradient flow, or formal guarantees that information propagation remains stable as sequence length grows.\n\nThe early versions of SinkTrack use mean-pooling for contextual information compression, which can cause information loss in very long contexts.\n\nWhile the benchmarks are diverse, all are QA or reasoning tasks. The generality claim would be stronger with results on open-ended generation, summarization, or code generation.\n\nThe writing is dense in places and could be more accessible. For instance, the transitions between hard, soft, and dual-track injection could be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wjcljtTjY4", "reviewer": "Reviewer_DYoN", "summary": "This paper proposes a training-free method to enable context anchoring (reducing hallucination) for language models, the key idea is to leverage the attention sink phenomena and fuse the hidden representation of context into the sink token to encourage more attention to the context.\n\nExperiments are conducted on three LLMs and four VLMs, showing improved performance over baseline (direct inference and CoT inference).", "questions": "* IIUC, the SINKTRACK method lets the <BOS> token attend to the rest of the tokens (instead of replacing the K,V of its own with the mean pooling of the rest of the token). If so, that should be illustrated clearly in Figure 3 (which shows the mean pooling operation).\n* I would suggest the author to present their full methods more clearly. Currently the paper is written to illustrate the \"failed\" attempt first (hard injection, soft injection, mean pooling), which makes it a bit hard to understand the actual proposed method. These comparison can be included in the ablation study.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper is written relatively clearly.\n* The idea of leveraging attention sink to encourage context anchoring is interesting.", "weaknesses": "* The baseline is limited: Using attention sink to enhance context anchoring is nice, yet this paper does not compare to prior training-free methods to mitigate halluciation. To name a few: [ACT(ICML 2024)](https://arxiv.org/pdf/2406.15765), [VCD(CVPR 2024)](https://arxiv.org/pdf/2311.16922), [OPERA(CVPR 2024)](https://arxiv.org/pdf/2311.17911), [SID(ICLR 2025)](https://arxiv.org/pdf/2408.02032), [DAC(ICML 2025)](https://arxiv.org/abs/2502.01969).\n\n* It is nice that the paper evaluates on both image and textual benchmarks, yet i found the choice of textual benchmark a bit unconventional. If the idea is to mitigate hallucination for long context, evaluation on well-studied benchmarks such as [LongBench](https://arxiv.org/abs/2308.14508) would be appropriate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UmrNKsw7gx", "reviewer": "Reviewer_Utsb", "summary": "This paper tackles hallucination and context forgetting caused by attention drift. The authors observe that, during generation, LLMs gradually shift their focus toward newly generated tokens and away from the initial context. Inspired by attention sink, they propose SinkTrack, a training-free, inference-time enhancement that treats the BOS token as an information anchor and continually injects key contextual features (e.g., image or instruction embeddings) into its representation. Experiments on both text-only and multi-modal QA show that SinkTrack suppresses drift-induced hallucinations and forgetting, outperforming both direct inference and chain-of-thought reasoning.", "questions": "See Weaknesses", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The paper highlights how attention drift triggers hallucination and context forgetting, which is a key insight for both long-context LLMs and long-context multi-modal LLMs.  \n2. The authors introduce a novel, training-free inference enhancement that alleviates the above issues with negligible overhead.  \n3. The effectiveness is demonstrated across the Qwen, Gemma, MiniCPM, and Llama families, accompanied by visualization analyses and discussions.", "weaknesses": "1. The evaluation benchmarks are somewhat outdated. While my limited knowledge of vision-related tasks and hallucination-related discussion prevents me from assessing those aspects, relying solely on QuAC and SQuAD to evaluate SinkTrack’s long-context improvements is clearly insufficient. The authors should first justify why QuAC is representative and then add results on standard long-context benchmarks such as LongBench [1], L-Eval [2], NIAH [3], and RULER [4], or more recent long-term conversation benchmarks like LongMemEval [5].\n2. The authors compare SinkTrack only against direct query and Chain-of-Thought, without comparison with the methods cited in Related Work. While three main streams of prior work are reviewed, their empirical relationship to SinkTrack has not been examined, such as an experimental comparison with a retrieval-augmented method. Including such comparisons would greatly strengthen the cohesion of the paper and the soundness of the method.\n3. The evaluation does not state the context lengths used, nor does it examine how attention drift varies as prompt length grows. Because SinkTrack itself relies on attention-based calculation, I am worried about whether the attention for SinkTrack could also drift when the prompt becomes very long. \n\n[1] LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding https://arxiv.org/abs/2308.14508\n\n[2] L-Eval: Instituting Standardized Evaluation for Long Context Language Models https://arxiv.org/abs/2307.11088\n\n[3] Needle In A Haystack - Pressure Testing LLMs https://github.com/gkamradt/LLMTest_NeedleInAHaystack\n\n[4] RULER: What's the Real Context Size of Your Long-Context Language Models? https://arxiv.org/abs/2404.06654\n\n[5] LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory https://arxiv.org/abs/2410.10813", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kFkbqKpuIL", "reviewer": "Reviewer_85VA", "summary": "This paper proposes a test-time method to mitigate reduce hallucination in MLLMs, and context forgetting on both LLMs and MLLMs. It uses soft injection to fuse information into LLM's computational flow.", "questions": "see weakness", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is generally well-written and easy to follow.\n2. The authors conducted comprehensive experiments on multiple tasks and multiple LLMs to validate their idea.", "weaknesses": "1.\tMethodological clarity:\nSome details in the method section remain unclear. For example, it is not explicitly stated how f_{\\text{info}} is obtained. From Figure 3, it appears that f_{\\text{info}} results from mean pooling over the encoder outputs; however, the architecture details of the Vision/LLM encoder are missing. The authors also mention that this work focuses only on decoder-only LLMs, which further raises questions about how the encoder component is integrated. In addition, an analysis of the extra computational overhead introduced by this module would be helpful.\n2.\tExperimental setup:\nIn Table 2, it is unclear why Chain-of-Thought (CoT) prompting performs worse than Direct Prompting, particularly on text-only tasks. Moreover, the current baselines seem rather limited. Including stronger or more diverse baselines would better demonstrate the effectiveness and generalizability of the proposed method.\n3. Definition ambiguity:\nThe distinction between vertical and horizontal information flow is not clearly justified. It is unclear whether this formulation is newly introduced by the authors or adapted from prior work. Providing clearer definitions and theoretical grounding would strengthen the analysis.\n4. Potential information loss and generalizability:\nThe proposed approach compresses all information into a single token, which may still lead to information loss. Although the reported results on the two tasks are promising, the generalizability of this information compression strategy remains uncertain. Further empirical validation on broader or more diverse tasks would make the claims more convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wRU2ShmT4A", "reviewer": "Reviewer_YQZu", "summary": "The paper proposes SINKTRACK, a training-free, dual-track cross-attention mechanism, solves the LLM problem of \"forgetting\" the original input by turning the naturally stable first token into an active information anchor that constantly and adaptively retrieves the most important details from the context.", "questions": "1. Why is the $L_2$ norm used for Information Gain in Equation 5 instead of cosine similarity or another metric?\n\n2. In \"soft injection,\" does the \"information vector\" refer to the vector corresponding to the image information?\n\n3. Does soft injection perform fusion on the $\\langle\\text{bos}\\rangle$ hidden state before the key-value projection, and how is the $\\alpha$ coefficient set?\n\n4. Why does the CoT (Chain-of-Thought) method show a decrease in performance compared to the baseline in Table 2?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents novel solutions to mitigate context forgetting and hallucination in LLMs by actively leveraging the model's intrinsic \"attention sink\" tendency towards the initial <bos> token.\n2. The quality of the work is evident through the systematic exploration of context anchoring methods, such as soft injection and SINKTRACK, which effectively enhances context coherence. The empirical results on relevant datasets confirm the method’s efficacy and significance as a solid step forward in improving LLM reliability over long contexts.\n3. This paper excels in clarity. The methodology is simple and well-motivated, and the overall organization of the paper is clear and easy to follow.", "weaknesses": "1. Sticking all the important context onto just the first token might create an information bottleneck for really long documents. The author needs to consider the impact of this.\n\n2. The 'CoT' baseline is too simple, making the new method's performance improvements look bigger than they might actually be. The author needs to compare with more advanced methods.\n\n3. It is critical to include experiments demonstrating that SINKTRACK does not hurt the model's general abilities, such as common sense reasoning and instruction following, in standard short-context benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)