Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flow
Authors: Xinlei Yu
Chengming Xu
Guibin Zhang
Yongbo He
Zhangquan Chen
Zhucun Xue
Jiangning Zhang
Yue Liao
Xiaobin Hu
Yu-Gang Jiang
Shuicheng YAN
Venue: ICLR 2026 Poster (2026)
Keywords: multi-agent system
visual hallucination snowballing
Primary area: foundation or frontier models, including LLMs
TL;DR: Using visual flow to relay information, which mitigates visual hallucination snowballing in multi-agent system.

Abstract: Multi-Agent System (MAS) powered by Visual Language Models (VLMs) enables challenging tasks but suffers from a novel failure term, multi-agent visual hallucination snowballing, where hallucinations are seeded in a single agent and amplified by following ones due to the over-reliance on textual flow to relay visual information. Through turn-, layer-, and token-wise attention analyses, we provide detailed insights into the essence of hallucination snowballing regarding the reduction of visual attention allocation. It leads us to identify a subset of vision tokens with a unimodal attention peak in middle layers that best preserve visual evidence but gradually diminish in deeper agent turns, resulting in the visual hallucination snowballing in MAS. Thus, we propose ViF, a lightweight, model-agnostic mitigation paradigm that relays inter-agent messages with Visual Flow powered by the selected visual relay tokens and applies attention reallocation to amplify this pattern. The experiment results demonstrate that our method markedly reduces hallucination snowballing, consistently improving the performance across eight benchmarks based on four common MAS structures and ten base models. The source code is publicly available at: https://github.com/YU-deep/ViF.git.

Published as a conference paper at ICLR 2026
VISUAL MULTI-AGENT SYSTEM: MITIGATING HAL- LUCINATION SNOWBALLING VIA VISUAL FLOW
Xinlei Yu'
Chengming Xu²
Guibin Zhang'
Yongbo He?
Zhangquan Chen Yu-Gang Jiang?
Zhucun Xue
Jiangning Zhang Yue Liao'
Xiaobin Hul
Shuicheng Yan'
National University of Singapore
Tencent Youtu Lab
ZZeejanng University
*Tsinghua University "Fudan University
xinlei.. yu@u. nus. edu benDxiaabinOhuueenus edu.sg
* Corresponding Author
ABSTRACT
Multi-Agent System (MAS) powered by Visual Language Models (VLMs) en- ables challenging tasks but suffers from a novel failure term, multi-agent visual hallucination snowballing, where hallucinations are seeded in single agent and amplified by following ones due to the over-reliance on textual flow to relay visual information. Through turn,, layer-, and token-wise attention analyses, detailed insights into the essence of hallucination snowballing regarding the reduc- tion of visual attention allocation. It leads to identify subset of vision tokens with a unimodal attention peak in middle layers that best preserve visual evidence but gradually diminish in deeper agent turns, resulting the visual hallucination snowballing in MAS. Thus, we propose ViF, lightweight, model-agnostic mit- igation paradigm that relays inter-agent messages with Visual Flow powered by the selected visual relay tokens and applies attention reallocation to amplify this pattern. The experiment results demonstrate that our method markedly reduces hallucination snowballing, consistently improving the performance across eight benchmarks based on four common MAS structures and ten base models. The source code is publicly available at: http:://ithuu..cmm/ wwtpsl///ooo/o///o//eeeee ViF.
provide
INTRODUCTION
MAS equipped with advanced VLMs are rapidly emerging as a solution for complex tasks, such as collaborative reasoning, multi-turn instruction following, and sophisticated multi-modal under- standing, by enabling agents to communicate and collaborate over multiple turns so as to tackle problems that are intractable for single model (Cemri et al., 2025 Li al., 2025b). However, this collaboration also exposes a fundamental reliability failure due to the problem of multi-agent visual hallucination snowballing, that is, visual misinterpretations in previous agents that are amplified as information flows through subsequent agents, producing propagatively hallucinated outputs about the visual contents and, ultimately, catastrophic hallucina- tion snowballing. This introduces new reliability and effectiveness challenges in VLM-based MAS that can not be addressed by single-agent research.
over- over- wver-refeeeence textual messages
It is noteworthy that the visual hallucination snowballing phenomenon in MAS is essentially differ- from such problem discussed in the previous works (Zhang al. 2024 Zhong et al. (024), given that the hallucination snowballing arises from two distinct but interacting mechanisms, as shown in Figure Ia (I) intrinsic hallucination, where individual VLM- agent produces erroneous tex- tual descriptions or assertions about visual contents, and (2) hallucination propagation, where the over-reliance on textual information flow, i.e, the generated text, compresses and selectively em- phasizes visual features, allowing surviving hallucinated assertions to be treated as authoritative by downstream agents. Since later agents typically accept prior textual context as strong evidence, early hallucinations are hence amplified rather than corrected, producing snowballing effect across turns. Due to the interaction between these two mechanisms, reducing per-agent hallucination alone, as focused by previous works (Wang et al. 2025 Tang al. 2025b Yin et al. 2025 Li et al.) [025c Zou et al.) 2025) Tang et al., 2025a), cannot fully solve the hallucination propagation problem, thus tailing to prevent multi-agent hallucination snowballing.

---

Published as a conference paper at ICLR 2026
Multi-Agent Hallacination Snowballing
Visual Information Relay
Agust Turn
Agull Vor Vor
Agent Pura Not
[PAC TXTI
POL EST
[PSE EXTT
Alaee
Describe
Figure
Decemnnee ifthis real calt
Food
ofrectts
(Rivating Methark)
fruit
You bezizaic mnaisssed
prodect sane
ppuddatts
liih
Visual Flaw (Curs)
2 0
Prepageiie
(a) An example multi-agen hallucination snowballing, and a comparison of visual informa- tion relay between existing iJ textual flow approaches and ii) our proposed visual flow method.
vatalle lei Alcurdest Vilee Eibasu liper
les Mlcerdest Viaia Tabesr rad keyo)
Vtrad Amertiee Klacaten Sedactiie
Agrs
Aged Fere
4gmn
Aget Fore
Agess Ta
Agens
(b) Maps of average visual attention allocation (c) Relations between visual attention alloca- across agent turns.
tion reduction and hallucination snowballing.
Figure I: Introduction to the multi-agent visual hallucination snowballing phenomenon: (a) presents an example illustrating how it happens; (b) and (c) specify the visual attention allocation reduction in different agent turns, potentially contributing to the occurrence of hallucination snowballing.
To diagnose how multi-agent pipelines lose visual fidelity across turns, we first conduct set of preliminary analyses that dissect attention dynamics among turn-wise, layer-wise, and loken-wise, through which we empirically conclude that the hallucination snowballing can be evident by the reduction of attention allocated to vision tokens over agent turns, as indicated in Figure Tb and Figure Tc Moreover, vision tokens characterized by unimodal attention peak in middle layers, as a small but vital subset of all vision tokens, can best preserve vision-specific information and whose removal most degrades visual understanding, thus being significant for enhancing the visual information flow among agents. Such token pattern, however, diminishes in deeper agent turns. im- plying the gradual dominance of textual information flow, leading to the hallucination snowballing,
Motivated by these insights, we propose an innovative mitigation strategy for multi-agent agget hallucina- tion anobballing dubbed as ViF. Instead of relying solely on textual flows, an additional visual flow is introduced to relay visual evidence by selecting subset of visual relay tokens and being textualized by previous instructions, then engaging them the process of following agents. Such design can provide downstream agents with preserved visual evidence that resists visual-to-text information loss, meanwhile preventing textual priors from entirely displacing visual signals during subsequent agent turns. In addition, an attention reallocation mechanism is introduced to amplify the ideal attention patterns and preserve visual contributions into deeper agent turns. We evaluate viF across eight benchmarks covering both comprehensive and hallucination tasks, demonstrating its striking effectiveness in alleviating hallucination snowballing in four different structures and ten base VLMs. Overall, our contributions are summarized as follows:
We formalize the multi-agent visual hallucination snowballing phenomenon and systemat- ically link it to visual attention degradation in deeper agent turns.
We provide extensive analyses that identify a subset of vision tokens that are critical for relaying visual information flow.
We introduce ViF, a model-agnostic method that optimizes inter-agent visual messages with visual flows and an attention reallocation mechanism to augment attention patterns.
Comprehensive experiments validate the efficacy of our ViF to reduce hallucination snow- balling, and additional analyses provide more convincing evidences.
2

---

Published as a conference paper at ICLR 2026
2 REQUISITE ANALYSES
As mentioned in Section 1 the hallucination snowballing can be presented by the negative correla- tion with the attention allocation to vision tokens. Quantitatively, as shown in Fig. [c the average attention allocation to vision tokens reduces from 0.165 to 0.099 at the 10th agent turn, and further to 0.063 at the 20th turn, with total 62% reduction. Furthermore, the reduction in the middle layer (-60% ) is much more remarkable than that in the first (2)1) and last (-30%) layers. For more thor- ough understanding, we mainly focus on V...VA--eeXT-77 (Liu et al. 2024bb on the POPE (Li et al. 2023) benchmark in the main paper, while Appendix B provides more detailed settings and comprehensive results on six VLMs to support the generalization ability of our claims, from which several insights are derived, thus leading to our research motivations.
conduct extensive requisite analyses among various VLMs. For simplicity,
2.1 ANALYTICAL EXPERIMENTS
Layer- Layer-Wise Attention Allocation in Different Agent Turns. To find out the underlying cause of visual hallucination snowballing MAS, we begin by measuring the trend of layer-wise attention allocation among different agent turns. In VLMs with multi-modal architectures, the decoder dy- namically allocates attention to three types textual tokens (instruction, system and output tokens) and one visual token that produced by visual encoder. Other special tokens, such start and end signals of visual input, are negligible and thus excluded.
Agunt Turn
Agent Turn
Agent Tarn
Agent Tarn 12
Agunt Tura 16
Agret Tura
g
10
15
33
35
Layaes
.6.. E. E
Layer
Layers
Layerr
Figure 2: Layer-wise attention allocation of four tokens in different agent turns.
As depicted in Figure 2 in general, when the agent turns increase, vision tokens receive gradually decreasing attention in all layers, while the attention being directed towards instruction tokens is raised accordingly; the attention allocations to system token and output token are relatively stable, without discernible trend of change. Focusing further on the middle layers of visual and instruction tokens, which are zoomed in, the opposite trend between visual and instruction tokens is more pronounced than other layers. In the first agent turn, scenario equivalent to a single-agent setting. there exists an obvious unimodal morphology peak in vision attention, and allocation to instruction is reduced conversely. However, in the 20th turn, the vision attention peak has almost disappeared and evolved into fluctuation, and is redistributed instruction tokens. Based on previous research (Yin jet al. 2025 Zhang et al. 20255) that textual and visual information are mainly fused and interacted these layers, this tendency of attention in MAS suggests that agents later turns tend largely ignore the vision tokens and over-rely on the instruction tokens, including visual contents relayed by textual output from previous agents. This preference for textual tokens, however, partially leads to the multi-agent hallucination snowballing. The previous agent may experience visual-to-ter information loss and potential cognitive bias when relaying visual evidence through textual flow. Conversely, vision tokens, as the initial visual semantic carrier, contain native and unbiased visual messages, which reduces the potential for hallucinations when relaying visual information. Based on these observations, we hypothesize: Can subset of vision tokens, acting as flow, directly relay visual information across agent turns?
Dropping Subsets of Vision Tokens in Different Layers. To intuitively verify the hypothesis, we ablate specific subsets of vision tokens in shallow/middle/deep layers (implementation modified from (Zhang et al.) 2025a)) and compare the corresponding performance degradation. We choose five subsets of vision tokens to ablate: (1) Random Tokens: randomly select tokens from the whole vision token set and maintain a relatively uniform distribution in the image. (2) Inactive Tokens:
3

---

Published as a conference paper at ICLR 2026
Table 1: Results of dropping vision token subsets in the shallow, middle, and deep layers.
Shallow 50%
Middle Layers 50%
Deep 50%
25%
75%
100%
25%
75%
100%
25%
75%
1008
wof Dropping
85,2
(4) Random
51.8 033.4
44.5
140.7
38.4
146.8
30.5
154.7
78.9
66.1
119.1
62.7
122.5
59.0
1262
84.4
10.8
83.2
2.0
82.9
82.6
12.6
(b) Inactive
55.1 (30.1
46.2
139.0
41.8
[43.4
12.5
152.7
84.3
0.9
82.9
233
81.5
3.7
78.3
6.9
35.0
10.2
84,6
0.6
84.3
10.9
84.6
10.6
29.6
20.8
81.6
(c) Fall
41.6
(43.
38.8
4644
30.7
[54.
22.5
162.7
78.3
64.8
2014
58.5
26.7
52.9
84.1
11.1
82.8
82.0
82.4
22.8
Unimodal
42.1
4331
37.6
47.6
30.0
(55.
22.8
162.4
52.9
[323
44.5
36.6
(48.6
25.3
_39.9
84,4
00.8
83.0
82.3
81.8
13.4
speiic
Al mec.iinnniieee
Figure 4: The demonstrations of selected unimodal vision tokens various cases.
select the tokens with constantly low attention and tiny fluctuation. (3) Rise Tokens and (4) Fall Tokens:
select tokens allocated with gradually upward or downward trend of attention. (5) Uni- modal Tokens: select tokens allocated with an unimodal attention peak. In terms of the unimodal tokens, we introduce parameter w to regulate the salience of the attention peak.
As listed in Table | the ablation of vi- sion tokens leads to varying degrees of performance degradation across layers. In shallow layers, all vision tokens are necessary for the visual understanding capacity; even ablating one-quarter of the tokens from any subset causes over 30% loss. On the contrary, in the deep layers, vision tokens play an insignif- icant role, with performance reduction of less than 1 % even when dropping all inactive vision tokens. Figure 3 further highlights that, compared to the results of other subsets, the dropping of uni- modal tokens in middle layers leads to more significant performance degradation. Specifically, the decrease from this subset is almost three times that of other subsets when dropping one-quarter of the tokens, and about twice when dropping half, three-quarters, and all tokens. In conclusion, the ablation study reveals that shallow and deep layers, all vision tokens almost equally important unimportant; however, in the middle layers, vision tokens with unimodal morphology play a much more crucial role in interaction information between vision and text tokens.
Drepping
do feustive
Fall Uromsdaa
65.2
(e) Reedves
Rise
-06.E
64.2943
Umg 25%
Drop 90%
Drep 75%
Dmep 1995,
Figure 3: Performance when dropping different vision to- ken subsets in middle layers.
Investigation Unimodal Tokens. To validate our previous hypothesis that certain vision tokens act visual flow for relaying visual information, visualize the unimodal vision tokens in various cases and track their ratios across agent turns. As demonstrated in Figure 4 we choose two images, each with three distinct questions, examples. The selected tokens mantically relevant and contain very few other irrelevant tokens. Besides, as depicted in Figure [5] the proportion of unimodal vision tokens continuously declines from 1.22% at the first agent turn to 0.10% at the 20th agent turn, while the percentages of the other two tokens slightly increase. The downward trend of the unimodal token proportion aligns with visual attention allocation among agent turns, which suggests that the reduction of unimodal vision tokens contributes significantly to the onset hallucination snowballing through the disappearance of the visual attention peak.
highly
4

---

Published as a conference paper at ICLR 2026
Thus, believe that the subset of uni-
Rise
Fall
Uni-mtodal
modal vision tokens
Propertia
Propertion
could meet the hy-
pothesis to relay vi-
0.4
sual evidence as vi- sual flow. ensures that semantically rel- evant visual messages
Agest Turne
Agent Tarns
Agant Turms
Figure 5: Proportions of vision tokens subsets in different agent turns.
can be relayed suffi-
ciently, while avoiding carrying substantial irrelevant vision tokens with high efficiency.
2.2 INSIGHTS
Based on experimental results and analyses, three significant insights be summarized:
The visual evidence relayed in MAS, which is typically via textual flow, potentially results in multi-agent hallucination snowballing.
When the agent turns increase, the average attention allocated to vision tokens reduces, and the attention peak in middle layers diminishes, while attention to instruction tokens increases accordingly; system and output tokens receive relatively stable attention.
In middle layers, vision tokens with unimodal attention allocation relay visual information; all vision tokens are significant in shallow layers and less significant in deep layers.
3 METHODOLOGIES
Building on the insights from the previous section, we propose a straightforward and efficient model- agnostic method named ViF to mitigate hallucination snowballing in VLM-based MAS. As demon- strated in Figure 6 our proposed method involves relaying the visual information from the previous agent via a selected subset of vision tokens, and reallocating attention in middle and deep layers to facilitate this process. Besides, provide suitable alternative for attention score based token lection, since in some recently released models with Fash-AAtentton 2/3 (Dao 2024), the attention scores not accessible.
3.1 VISUAL INFORMATION RELAY
L Leveraging the previous insights, we employ the unimodal vision tokens as additional visual flow to relay information from the previous agent. Specifically, we token-wise decompose the vision tokens V= {u1,... .... Jm according to the trend of the attention allocation in the middle layers, and select the vision tokens with unimodal morphology as initial visual relay tokens R = (fi.....Tr, rna C V, where & m. However, the original selected tokens semantically irrelevant, which only tokenized by the vision encoder, without particular semantics. Thus, we contextualize the initial visual relay tokens R with the instruction tokens I as follows:
R = f (R + [:n],
(1)
where / (-) is lightweight transformer block Mehta al. 2021), @ denotes concatenation. Here, we extract the former n component to maintain the initial length of visual relay tokens R.
To retain the spatial information of visual relay tokens, we apply the same positional encoding strategy as the previous agent. Then, the visual relay tokens will be transmitted to the subsequent agent, which will be inserted between the original vision tokens and instruction tokens, and be fed to the final LLM together with other tokens.
3.2 ATTENTION REALLOCATION
Considering the insights that tokens are of different significance in the shallow, middle, and deep layers respectively, reallocate attention optimize attention patterns. Our objectives to
5

---

Published as a conference paper at ICLR 2026
Visual Relay Tokens Generation
Attention Reallocation
Tranffomer Block
[Sytemi
Sysken hooept
Decongaation
Trauthno
Ci) Rice
on Full
tn Uniinodal
MW
[estmaatio]
Agent
Uset Instrction
Skalkve
Middic
Dusp
Afteation
Tokan
Tolenne
Tokcas
Tokenn
Tokees
Realbcation Dirctionn
Figure 6: Overview our proposed ViF, including the generation of visual relay tokens and attention reallocation to alleviate multi-agent hallucination snowballing.
activate the visual relay tokens and to optimize the distribution of attention among various tokens. Therefore, we amplify dynamic trends, both the upward and downward, of visual attention in the middle layers by adding temperature scaling to the Softmax operation in the middle layers:
exp
A Softmaz,(S)
exp
where 7 is temperature parameter, S and are attention score matrix and attention score, and A is attention matrix. It promotes the emergence of vision tokens with unimodal morphology. Besides, in the middle layers, we collect the attention of inactive vision tokens and instruction tokens, and then reallocate the collected attention to other vision tokens, which is formulated as:
= a a 8; o Mc. Mc (i,j) = ( T.j Vo) V (i T.J Z)),
(3)
= E=1
C, (i.)) =ijE
(4)
where Me and M, are the collection and reallocation mask matrices respectively, which designate the source and destination of the attention reallocation. Besides, is the reallocation coefficient, T is the whole token set, Vo C V is inactive vision token set, and V. = Ve = {UI,.... vi During the reallocation, the sum of the total attention is always 1. Additionally, in the deep layers, the reallocation from vision tokens to instruction tokens, with the same process. Thus, the two mask matrices and the reallocation coefficient modified correspondingly.
3.3 ALTERNATIVE OF ATTENTION SCORE BASED STRATEGY
To accelerate the computation and reduce memory, flash-attention (Dao 2024) mechanisms are widely used in recently released models, making the attention scores obtainable. Inspired by (Wen et al. 2025), we design a Key-Norm (L2 norm of the key matrix) based alternative for the original attention score based method. More discussions about the alternative are in Appendix C.2
4 EXPERIMENTS
We conduct experiments on three comprehensive benchmarks: MME (Yn et al. 2024), MM- Bench (Liu et al. 2024d), MM-Vet (Yu et al. 2024), and five visual hallucanation benchmarks: CHAIR (Rohrbach et al. 2018), POPE (Li et al., 2023). AMBER (Wang et al. 2023), MMHal- Bench (Sun et al.] (023), HallBench (Guan et al. (024)). Besides, we also include four bench- marks in augmented visual domains: MMIU (Meng et al. 2024), MuirBench (Wang et al. 2024a), MVBench (Li et al. 2024), and Vidco-MME (Fu et al.) 2025). For detailed experimental settings, configurations, and additional results, please refer Appendix D.
4.1 MAIN RESULTS
Performance on Comprehensive and Hallucination Benchmarks. For comprehensive assess- ments of our proposed ViF, we first compare the results six base VLMs, namely, L.aVA-v1.5-

---

Published as a conference paper at ICLR 2026
Table 2: Results across eight comprehensive and hallucination benchmarks. indicates implemen- tation with Key-Norm, while others use attention scores. The best and the second best values with our method are bolded and underlined respectively, and the rightmost column shows the average results. For identical values, we compare the following digit after the decimal point.
MAS Structure
Base Agent
MME!
MM Bencht
MM Vert
CHAIR, POPE
AMBER
Bench"
MMHal-
Hall Bench"
Avg."
LLaVA-vI.5-7B
1516.2
66.1
32.4
52.4
86.7
85.2
40.5
48.0
+Ours
15318 15566
67.8
117
34.6
12.
51.7
10.7
88.6
[19
87.8
12.6
42.8
123
50.6
12.6
3.5%
LLaVA-vL.6.7B
1511.7
68.7
45.5
86.7
88.5
43.4
52.6
+Ours
1524.3 122.6
69.4
70.7
38.2
11.
43,6
[1.
88.2
11.5
91.5 13.0
46.0
'26
54.7 12.1
1.1%
LLaVA-NeXT-7B
1567.0
72.1
47.1
44.6
88.6
87.0
45.9
52.9
Linear
+Owr
1585.2 118.2
73.5
11.4
49.3
12.2
42.9 117
90.4
1L.
89,3
12.3
48.1
12:2
55.3 12.4
3.2%
LLaVA-OV-7 OV-7B
1587.6
82.3
58.4
38.3
91.4
91.3
47.8
53.7
+Ours"
1598.8 111.2
83.4
11.1
59.9
11.5
37.2
93.0
11.6
93.9
12.6
49.6
11x
56.1
12.4
2.6%
Qwen2 VL
1686.4
81.9
63.3
38.4
90.5
91.2
48.2
$1.9
+Ours'
1699.6
133.
82.4
10.5
65.2
11.9
37.8 [0.6
91.7
11.2
94.0 12.8
50.2
f2.
54.4
12.5
2.4%
Qwen2,5-VL
1730.4
83.9
67.3
38.6
89.9
92.8
51.6
53.8
+Ours
1746.2
15.8
85,3
69.1
37.6
01.0
91.4
94.4
11.6
53.7
56.2
12.4
2.5%
LLaVA-vI. 1.5BB
1512.5
63.6
30.6
49,00
86.0
83,5
41.3
46.6
+Ours
1520.9 18.4
64.7
11.1
32.0
11/4
49.3 10.3
87.6
11.6
86.1
12.6
43.7
12.4
48.9 12.3
2.7%
LLaVA
1508.0
66.6
35.1
44.2
86.8
85.2
42.0
48.2
+Ours
1518.9 100.9
68.4
118
37.5
12.4
42.7 115
87.9
11.1
87.1
11.9
43.6
11.6
50.4
12.2
3.2%
LLaVA LLaVA-NeXT-7B
1555.2
69.2
44.0
44.2
87.0
85.6
44.7
49.5
Layered
Ours
1571.3 16.1
70.7
11.5
46.5
12.5
42.5 [1.7
89.2
12.2
87.7
12.1
47.2
12.5
51.6 12.1
3.55
LLaVA-Ov-7B
1584.2
80.6
57.9
37.6
89.9
89.1
45.2
52.7
+Ours"
1596.7 122.5
82.0
11.4
59.5
11.6
36.4 112
91.5
11.6
90.9
11.8
46.9
11.7
54.5
11.8
2.5%
Qwen2-V VL-
1679.0
79.3
37.7
88.5
45.9
49.1
+Ours"
1692.6 1336
80.6
11.3
63.4
11.9
37.1
[0.6
90.5
11.9
90.6
12.1
48.1
12.
50.9 1
1 2.5%
Qwen2.5-VL-7B
1722.5
81.0
36.8
90.1
51.2
+Ours'
1737.0 114.
82.4
11.4
63.8
11.7
36.0
[0.8
89.6
12.1
91.7 11.6
49.3
(2.
53.2 12.0
2.6%
LLaVA 7B
1519.6
67.1
33.1
49.4
88.3
89.0
44.6
52.8
+Ours
1537.6
118.
68.4
11.3
34.7 11.6
49.8
10.4
90.2
92.2 13.2
47.0
72.
55.0 (2
2.8%
LLaVA-vI.6-4
1519.8
69.0
36.9
43.9
88.6
91.3
44,3
54.9
69.7
41.8
12.1
94.0 12.7
/2.5
57.3 12.4
1576.2
73.0
49.2
43.4
90.4
89.2
47.2
55.4
Random
1596.1
75.3
50.1
10.9
41.6 11.
12.6
92.9 [3.7
49.3
58.4 [3.0
3.5%
LLaVA-OV OV
1590.1
83.7
58.5
37.7
92.5
91.9
46.3
56.2
Ours'
85.1
71.4
11.2
37.1
[0.6
94.1
94.9
13.0
48.5
59.1
12.9
Owen2-VL-78
1690.
84.1
64.4
38.1
90.8
92.3
46.5
53,7
+Ouss
1703_1 13.0
84.9
10.8
65.2
10.8
37.2
10.9
92.7
11.9
95.4
13.1
48.4
11.9
56.3
12.6
2.5%
Qwen2.5-VL-7B B
1737.7
86.5
67.0
37.8
90.4
93.6
48.9
56.8
+Ours"
1756.8
19.1
88.4
11.9
68.1
11.1
37.0 J0.8
92.8
12.4
95.8
12.2
50.6
11.
59.5
12.7
1 2.5%
LLaVA-vI.5-7B
1520.9
67.9
33.5
52.7
88.9
88.7
42.4
51.7
*Ours
1539.1 188
68.4
10.5
36.0 +2.5
51.3 11.4
90.4
FLS
91.6 12.9
44.7
12.3
54.1 12.4
3.4%
xL.677
1519.5
69.7
42.7
91.2
53.8
OOurs
1537.1 117.6
71.3
11.6
39.3
11.6
40.7
[2.0
90.1
11.6
93.8 12.6
46.0
12.6
56.1
12.3
3.5%
LLaVA-NeXT-7B
1580.5
73.2
49.5
43.0
91.0
89.4
47.9
53.1
Circular
+Ours
1599.5 19.0
74.6
11.
51.8
12.3
41.2
LLS
93.3
/2.3
92.7
13.3
51.1
13.
55.7
12.6
3,8%
LLaVA LLaVA-OV OV
1592.8
84.0
59.1
38.7
92.8
92.2
47.3
54.6
+Ours"
1606.1 113.3
84.6
10.5
60.2
11
36,9
118
94.0
112
95.0
12.8
49.4
12.1
56,9 |2:3
2.7%
Qwen2 VL-7B
1692.8
83.3
63.9
38.1
91.6
92.8
47.7
52.4
Ours'
1706.3 113.5
84.1
10.8
65.1
11.2
37.2
93.3
11.7
94.8
12.0
50.2
f2.5
54.6
f2.2
2.4%
Qwen2,5-VL. V-VL7B
1738.
85.2
66.7
38.2
91.3
93.5
50.
54.9
1756.2
86.6
114
37.4
10.8
93.4
95.9
12.4
52.5
57.3
12.4
2.6%
Table 3: Results of larger-size models circular MAS structure.
Base Agent
MMEt
Bencht
MM-Vert
CHAIRL
POPET
AMBER
MMHal- Hall Bench!
Bench?
Avg.f
LLaVA-1.5-13B
1528.7
70.2
38.3
40.8
90.0
89.6
44,7
$2.9
+Ours
1547.6 1189
71.1
189
40.5
39.1 L1.7
92.4
12.4
92.7 "3.1
47.2
12.5
55.3 12.4
3,6%
LLaVA N3XT-13B
1583.5
68.8
42.3
36.0
91.9
92.4
48.2
54.3
+Ours
1602.6 119.1
70.1
11.3
44.5
12.
34.2 11.8
23.7
11.8
95.4
+3.0
50.8
12.6
56.8 12.5
3.6%
LLaVA-NeXT-34B
1644.9
78.6
54.6
27.6
91.4
94.1
48.9
55.0
+Ours
1670.8 125.9
80.9
12.3
22
25.4
[2.2
93.6
12.2
96.3
12.
52.4
13.5
57.8 F2X
4.4%
VL.
1886.1
87.4
69.8
24.4
92.5
94.0
52.1
56.9
+Ours'
1906.22 120.
11.
12.1
11.5
(2.7
55.1
13.0
60.1
4.1%%
7B Liu et al. 2024a). LLaVA-v1.6-7B, LLaVA-NeXT-7B (Liu et al. 2024b). LLaVA-OV-7B Lii
et al.) 2025a), Qwen2-VL VL-BB (Wang al. 2024b), and Owen2.5-VL. VL-7B (Bai al. 2025). We
choose four common MAS structures, including linear (Hong et al., 2024), layered Ishibashi & Nishimura 2024). random Qian et al. 2025), and circular (Qian et al. 2025) structures. As
7

---

Published as a conference paper at ICLR 2026
Table 4: Results across four augmented visual Table 5: Evaluations of multi-agent hallucina- benchmarks on circular MAS structure, including tion snowballing with proposed HS metric. multi-image and video based scenarios.
MAS Stroenare
CHAIRJ
POPEL
AMBERJ
MMHal- Bench]
Hall Beach]
Avg.J
-eXT--NeT-77B
Agent
MMIU1
31,6
MuirBenchr"
42.6
|MVBcucht
89.2
60.4
Video-MMEI Avg.1
Lineur
117.2
25.7
1644
(9.3
26.8
[109
22.6
35.3
40.2
(154
1.3888
+Ours
33,9 123
117
$2.0 12.8
61.9115
11.
4.9
112.7
21.5
20.5
31.6
36.4
LVV7
36.9
542
56.1
67.8
Layered
10.6
(2.1
13.9
17.6
13.1
17.4
19.5
(12.)
21.3
(151
133.0%
+Ours"
39.6 127
55.5
58.3 12.
68.8
-3.8
Qwen2-VL--B VL-7B
45.5
62.8
69.8
70.6
Circular
1889
29.1
31.1
40.8
47.
+Ours
47.7 122
64.0 11.2
71.0 11.2
71.7 11.1
25
12.8
17.0
1121
17.7
1134
24.1
[167
27,8
(196
13388%
Qwen2.5-VL
47.4
61.0
22.3
2344
Random 15.5
23.4
23,8
36.8
42.5
+Ours'
49.3
119
73.6 11.3
152 15.0
(844
16.4
17.4
21.2
(156 25.6
3169 1.365%
Table 6: Comparison results of other SOTA methods and ours on LLaVA-NeXT-78 and circular MAS structure. Orig. represents the original evaluation metric, and HS is our proposed one.
CHAIR
POPE Orig.1
AMBER Orig.t
MMHHalBennch Orig:1
HallBench Ong.."
Avg. NSS
Orig.l
NSL
HSS
ISS
HS.
NSL
Orig:1
Baseline
43.0
18.9
91.0
29.1
89.4
31.1
47.9
40.8
53.1
47.4
McmVR
43,8 10.8
20.6
117
90.5
.0.5
31.2
12.1
88.9
10.5
34,4
13.3
44.8
13.1
117.
49.2
13.9
57.6
110.
2.6%
1 18.4%
VISTA
10.4
19.0
10.1 91.2
27.8
11.3
11.1
28.3
46.3
118
47.4
16.6
50.7
124 53.3
FatSight
42.1
10.9
17.7
112
91.9
0.9
22.7
16.1
91.0
11.6
26.6
145
474
10.5
42.9
12.1
51.9
112
52.4
15.0
0.5%
5.4%
DeCo
10.4
18.2
10.7
91.3
25.1
12.2
24.3
47.0
J09
13.3
50.4
[2.7
53.0
1.0%
338%
TAME
42.1 10.9
18.8
10.1 91.4
70.
22,8
16.3
21.9
12.5
22.7
18.4
46.5
114
47.8
17.0
49.9
13.2
53.8
16.4
1.6%
3.7%
41.2
11.8
12.8
(6.1 93.3
(12.1
13.3
17.7
[134
51.1
13.
[16.7
55.7
12.6
27.8
[19.6
1 3,8%
39.8%
mentioned in Appendix C. we primarily follow the multi-agent collaboration strategy with lin- increased context length, allowing the scaling of MAS. As demonstrated in Table 2 our ViF consistently enhances the average performance of the six baselines by 2.4-3.8%, verifying the com- patibility of our method on various MAS structures based on arbitrary base VLMs. Notably, on the HMMHal-Beche and HallBench benchmarks, which are more sophisticated and have unsatisfac- tory baseline performance, our ViF achieves over 4% average improvement. When applied to the circular structure, which is wwwpecinttioncocceetraa among agents, our ViF dramatically reduces hallucination snowballing and further improves the performance by 3% among the six base models, compared to the other three selected structures.
with densest collaborations and interactions
As reported Table 3. we also analyze the performance of our proposed ViF on the scaled-up models with higher parameters. It is observed that when equipped with our ViF, the larger base models featuring more than 30B parameters, e.g., LLaVA-NeXT-34B and Qwen-2.5-VL-322 hibit greater enhancement than all smaller ones, improving by more than 4% across all benchmarks. This indicates that our model-agnostic method effectively improves their comprehensive perfor- mance, likely because larger-parameter baselines possess stronger fundamental capabilities, and our approach specifically unlocks their latent potential in multi-agent scenarios.
ex-
Performance on Augmented Visual Benchmarks. We include additional four benchmarks of two augmented visual domains, including two multi-image based benchmarks: MMIU, MuirBench; video based two benchmarks: MVBench, Video-MME. As presented in Table our ViF method ex- hibits significant improvements relative to the base models across multi-image and video scenarios. Specifically, it yields an average 2.0% performance improvement across the four base models and four additional benchmarks, demonstrating robust performance in multiple visual scenarios.
Multi-Agent Hallucination Snowballing Mitigation. In addition to the results of original metrics, we attempt assess the level of hallucination snowballing in MAS quantitatively. Thus, we formally define a hallucination snowballing score (HS) as in Equation 7 measuring both the hallucination level and propagation in MAS. As reported in Table [ adding our ViF reduces at least 30% HS score on the average of five hallucination benchmarks, significantly mitigating the hallucination propagation from the textual flow of visual contents. Notably, the layered structure suffers the least from the detrimental snowballing, while in circular structure, where the initial hallucination snowballing is the most serious, the reduction of the score from our method is almost 40%.
Comparison Results We compare the results of another five model-agnostic and token- wise hallucination mitigation methodologies in multi-agent contexts, ie., MR (Zou al. 2025
8

---

Published as a conference paper at ICLR 2026
VISTA (Li et al. 2025c). FarSight (Tang al. 2025b), DeCo (Wang al. 2025 and TAME Tang et al. 2025a). Specifically, we retain the multi -gult--agent experimental settings unchanged and apply these methodologies to the base model. These counterparts restrict the deepening of intrinsic hal- lucinations in a single model to some extent, however, in multi-agent scenarios, the propagation of visual contents via textual flow still introduces vision-to-text cognitive bias and fails to restrain the snowballing of multi-agent ballucinations commendably.
As shown in Table 6 our introduced ViF approach achieves distinctly superior performance among all the benchmarks on both their original metrics and our proposed HS score. It obtains at least 4.2% enhancements in original metrics and 34.4% in HS score on average, compared other methods tailored for single model hallucination mitigation. Although these counterparts are impressively efficacious in single VI.M, their performance is compromised when applied MAS, because of the failure to deal with hallucination propagation among agents and further snowballing. Surprisingly, in MAS environments, the results of these counterparts are even inferior to the baseline, especially on challenging ones. This counterintuitive observation is likely because they modify the initial paradigm of decoding or attention in VLMs, but retain the textual flow to relay visual information, which amplifies the preference for text over vision tokens. Our method, adopting visual flow to relay information among agents, cuts the HS score almost in half and delivers tangible improvements in the mitigation of hallucination snowballing.
4.2 ADDITIONAL ANALYSES
Impact of the Number of Agent Turns. To achieve satisfactory completion, typically, MAS ne- cessitates a greater number of agent turns in more complicated and challenging tasks. However, the hallucination snowballing effect restricts the multi-turn collaboration among agents, where halluci- nations are amplified and propagated, leading to suboptimal performance. Therefore, we compare our ViF with baselines and other counterparts to assess the impact of the number of agent turns.
VISTA
butr
Bec
Dwnii S41-
Agpat Tarts
Appat Tara
Aguat Taras
Agai Tams
(a) MMHal-Bench
(b) HallBench
(c) MMHal-Bench
(d) HallBench
Figure 7: Impact of the number of agent turns. In (a) and (b), straight and dashed lines are the results with or without our ViF on various baselines and circular MAS structure, respectively, (c) and (d) show the results between other counterparts and our method based on LLaVA-NeXT-7B.
As demonstrated in Figure 7 our method maintains an upward trend in performance the number of agent turns increases, while both other contrast methods and baselines experience performance degradation instead. More precisely, when the agent turn is set to one, which equivalent to single-agent context, ViF exhibits only marginal improvement over the baselines, and falls behind some other methods designed for hallucination mitigation in a single model. As the turning trends of the baselines in Tigure/aaand (a show, the performance begins to deteriorate when the turns are only increased to 5, and at the 20th turn, their performance is even further less than that of a single agent. Further compare with other methods as illustrated in Figure 77 and [d although hallucinations mitigated in early turns to some extent, the hallucination snowballing phenomenon still suffers in later turns, essentially limiting the multi-agent collaboration and inhibiting the potential of MAS.
Ablation and Sensitivity Analyses. To verify the effectiveness each component in our ViF. we perform ablations on the visual relay tokens and the attention reallocation. As reported in Table the improvement from visual flow to relay information is prominent, and the results are still better than most comparison methods even when ablating half of the visual relay tokens, showcasing excellent robustness. The reallocation mechanism further optimizes the attention distribution among different tokens and activates visual relay tokens, which beneficial to our designs of visual relay flow.

---

Published as a conference paper at ICLR 2026
Table 8: Efficiency comparison between our ViF and the base models on the circular MAS archi- tecture. All base models are evaluated in the multi-agent environment to quantify the additional latency introduced by ViF, including the average latency (seconds), and the average floating point operations, i.e., FLOPs (T).
CHAIR
POPE
AMBER
MMHal-Berse
HallBench
Base Agent
Lotency]
FLOPs)
Laiency, FLOPs)
Lutency,
FLOPs).
Laiency)
FLOPs]
Laiency] FLOPs]
LLaVA-NeXT-7B
|3.16
157.3
2.46
103.4
2.79
127.2
3,48
184.0
3.91
248.3
+Owrs
3.47
168.5
2.79
115.7
3.10
138.9
3.83
197.6
4.23
260.3
ILaVA-NeXT 138XT-13B]
55.88
308.5
5.63
279.0
5,93
310.0
6.33
357.4
7.64
386.8
+Ours
6.17
320.6
289.6
6.25
321.2
372.8
8.03
LLaVA-NeXT-344/
8.80
417.2
8.49
387.3
8.61
408.7
9.41
444.1
11.06
478.1
+Ours
9.09
419.1
8.79
398.8
8.92
419.8
9.75
457.7
11.41
493.8
Furthermore, as listed in Table 12 13 and [1]] we conduct an analysis of the sensitivity of key hyper-parameters, Le.. the salience threshold w. the tem- perature scaling т, and the reallocation coefficient a, quantitatively assess their impact on model performance, and de- termine rational set of values.
Table 7: Ablation study on LLaVA-NeXT-7B and circular MAS structure, verifying the effectiveness of visual relay tokens and attention reallocation.
Setting
CHAIR]
POPET
AMBERT
MMHal-
Beach]
Hall
Bench]
w/o Relay Token (25%)
(+0.2) (-.4 (204) 50.7 (-0.4) 55.2(40.5)
w/o Relay Token (50%)
(2.3 (2.0 (1.6(11.1)
49,8 54.8
w/o Relay Token (75%)
(21.4) (-1.6) 91.1 (-16)
49.1 (:2.0 54.1 (41.6)
Efficiency. To assess the inference effi- ciency, particularly in multi-agent con- texts, we first compare the time and computational overheads of our pro- posed ViF with those of the base mod-
w/o Reallocation (Midklee
41.7 (40.5) (4.2) (-1
49.9(-1.2)
(4..))
we Reallocation (Deep)
(+0.2)
(-0.))
(10.))
50.9 (0.2)
55.0 (177)
we Reallocation
42.2 (1.9 (-12 49.6 (-1.5) (4.2(-).5)
Ours
41.2
93.3
92.7
51.1
55.7
els. As reported in Table g our ViF exhibits high efficiency with moderate overheads, incurring an additional 8. 1.1--3,, 13.4% inference latency and 4.8-1 1.8-11.9% computational costs (measured by FLOPs) over the base model across five selected benchmarks. These extra overheads mainly stem from the intrinsic components of ViF. Thus, the extra overhead remains stable across base models of varying scales and exhibits only slightly linear increase. Notably, the additional latency and computation are even more negligible for larger models, which are less than 4% and 3% on VL-VN-NeT--3 Fur- thermore, as presented Table T3] the time and computational overhead of our ViF remain efficient when feeding visual images with varying resolutions and across different agent turns.
5 CONCLUSION
We unveil the phenomenon of multi-agent visual hallucination snowballing existing in MAS, where subsequent agents progressively amplify errors originating in a single agent through textual infor- mation flow that relays visual messages. Based extensive analyses, the essence of hallucination snowballing lies in a subset of vision tokens with an unimodal attention peak, well-preserving the visual information, but these tokens gradually diminish with the increase in the agent turns. To al- leviate this problem. a model-agnostic method named ViF is proposed, which redefines the visual information flow in MAS. Specifically, we introduce a visual flow relay visual messages based on the selected unimodal vision tokens and utilize attention reallocation to optimize this pattern. Comprehensive experiments indicate that this novel paradigm is effective, robust, and compatible, paving the way for more efficient inter-agent visual information relay and more sophisticated MAS.
Limitations. Although we conduct experiments on a total of ten models with different sizes, which verifies the robustness of compatibility of our proposed method, more experiments are still recom- mended. For example, the results on smaller size VLMs, e.g., 3B, and also larger baselines, e.g. 72B, could provide further evidence. Besides, the inclusion of more series of basclines, such as Intern series (Chen et al. 2024b]a al Zhu et al.] (025). Llama 3 (Grattafiori et al., 2024). Ovis series (Lu et al. 2024 2025), , and MinIMCP (Hu et al. 2024), is also beneficial. Furthermore, more effective and complementary combinations of our ViF with other hallucination mitigation strategics, well more optimal vision token selection, warrant further exploration.
10

---

Published as a conference paper at ICLR 2026
6 ACKNOWLEDGEMENT
This work was supported by the National Natural Science Foundation of China under Grant No. 62320106007, and by NUS Start-up Grant A-0010106-00-00.
REFERENCES
Dosovitskiy Alexey, An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv: 2010. / 1929, 2020.
Wenbin An, Feng Tian, Sicong Leng, Jiahao Nie, Haonan Lin, Qian Wang, Guang Dai, Ping Chen, and Shijian Lu. Agla: Mitigating object hallucinations in large vasion-language models with assembly of global and local attention. arXiv preprint arXiv:2406.12718. 2024.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.1392. 139233 2025.
Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, al. Why do multi-agent Ilm systems fail? arXiv preprint arXiv:2503. 13657, 2025.
Zhangquan Chen, Manyuan Zhang, Xinlei Yu, Xufang Luo, Mingze Sun, Zihao Pan, Yan Feng, Peng Pei, Xunliang Cai, and Ruqi Huang. Think with 3d: Geometric imagination grounded spatial reasoning from limited views. arXiv preprint arXiv:2510.18622 2025a.
Zhanpeng Chen, Mingxiao Li, Ziyang Chen, Nan Du, Xiaolong Li, and Yuexian Zou. Advancing general multimodal capability of vision-language models with pyramid-descent visual position encoding. arXiv preprint arXiv:2501.10966 2025b.
The Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271,, 2024a.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguisti tasks. In Proceedings of the IEEE/CVF Conference on Computer 2024b.
Vision and Pattern Recognition (CVPR), pp.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.06261, 2025.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evalua- tion benchmark of multi-modal llms video analysis. Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pp. 24108-22222 20--241188, 2025.
Xuan Gong, Tianshi Ming. Xinpeng Wang, and Zhihua Wei. DAMRO: Dive into the attention mechanism of LVLM to reduce object hallucination. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 7696-7712, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
11

---

Published as a conference paper at ICLR 2026
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilama 3 herd of models. arXiv preprint 21xf::240..1778. 2024.
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp- 14375-14385, 2024.
Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:24022 680, 2024.
Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang. Ceyao Zhang. Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. Metagpt: Meta programming for multi-agent collabo- rative framework. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
Shengding Hu, Yuge Tu, Xu Han, Ganqu Cui, Chaoqun He, Weilin Zhao, Xiang Long, Zhi Zheng, Yewei Fang. Yuxiang Huang. Xinrong Zhang. Zhen Leng Thai, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding. Chao Jia, Guoyang Zeng. da- hai li, Zhiyuan Liu, and Maosong Sun. MiniCPM: Unveiling the potential of small language models with scalable training strategies. In First Conference on Language Modeling (COLM), 2024.
Xiaobin Hu, Yunhang Qian, Jiaquan Yu, Jingjing Liu, Peng Tang, Xiaozhong Ji, Chengming Xu, Jiawei Liu, Xiaoxiao Yan, Xinlei Yu, et al. The landscape of medical agents: A survey. Authorea Preprints, 2025.
Fushuo Huo, Wenchao Xu, Zhong Zhang, Haozhao Wang. Zhicheng Chen, and Peilin Zhao. Self- introspective decoding: Alleviating hallucinations for large vision-language models. In The Thir- teenth International Conference on Learning Representutions (ICLR), 2025.
Yoichi Ishibashi and Yoshimasa Nishimura. Self-organized agents: A Ilm multi-agent framework toward ultra large-scale code generation and optimization. arXiv preprint arXiv:2404.02183, 2024.
Seongyun Lee, Sue Hyun Park, Yongrae Jo, and Minjoon Seo. Volcano: Mitigating multimodal hallucination through self-feedback guided revision. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (ACL: Long Pupers), pp. 391-404, Mexico City, Mexico, June 2024. Association for Computational Linguistics.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. Transactions Machine Learning Research (TMLR), 2025a. ISSN 2835-8856.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 22195-22206, 2024.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hal- lucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 292-305, Singapore, December 2023.
Association for Computational Linguistics.
Yubo Li, Xiaobin Shen, Xinyu Yao, Xueying Ding, Yidi Miao, Ramayya Krishnan, and Rema Pad- man. Beyond single-turn: A survey on multi-turn interactions with large language models. arXiv preprint N0X4:22004.0717.. 2025b.
12

---

Published as a conference paper at ICLR 2026
Zhuowei Li. Haizhou Shi, Yunhe Gao, Di Liu, Zhenting Wang, Yuxiao Chen, Ting Liu, Long Zhao, Hao Wang, and Dimitris N. Metaxas. The hidden life of tokens: Reducing hallucination of large vision-language models via visual information steering. In The Forty-second International Con- ference on Machine Learning (ICMI., 2025c.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 26296-26300 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024b. URL /tttt:://
llavavvl. www.vl..ggiit.. wtt.uu...ooobb..... io/bll/bblloooo www.o///lloooeeeeeeeeooooooeeeoe 2024 01 300 llava-
Sheng Liu, Xu Zhang, Nitesh Sekhar, Yue Wu, Prateek Singhal, and Carlos Fernandez-Grande Avoiding spurious correlations via logit correction. In The Eleventh International Conference Learning Representations (ICLR), 2023.
Shi Liu, Kecheng Zheng, and Wei Chen. Paying more attention to image: A training-free free method for alleviating hallucination in Ivims. arXiv preprint arXiv:2407.21771, 2024c.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision (ECCV), pp. 216-233. Springer, 2024d.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations (ICLR), 2019.
Shiyin Lu, Yang Li, Qing-Gui Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. Ovis: Structural embedding alignment for multimodal large language model. arXiv preprint arXiv:2405.20797, 2024.
Shiyin Lu, Yang Li, Yu Xia, Yuwei Hu, Shanshan Zhao, Yanqing Ma, Zhichao Wei, Yinglun Li, Lunhao Duan, Jianshan Zhao, al. Ovis2. 5 technical report. arXiv preprint arXiv:2508. 1737,, 2025.
Sachin Mehta, Marjan Ghazvininejad, Srinivasan Iyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Delight: Deep and light- weight transformer. In International Conference on Learning Represen- tations (ICLR), 2021.
Fanqing Meng, Jin Wang, Chuanhao Li, Quanfeng Lu, Hao Tian, Jiaqi Liao, Xizhou Zhu, Jifeng Dai, Yu Qiao, Ping Luo, et al. Mmiu: Multimodal multi- image understanding for evaluating large vision-language models. arXiv preprint arXiv:2408.0277 027188 2024.
Clement Neo, Luke Ong, Philip Torr, Mor Geva, David Krueger, and Fazl Barez. Towards inter- preting visual information processing in lssounlanguage models. In The Thirteenth International Conference on Learning Representations (ICLR), 2025.
Chen Qian, Zihao Xie, YiFei Wang, Wei Liu, Kunlun Zhu, Hanchen Xia, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, Zhiyuan Liu, and Maosong Sun. Scaling large language model-based multi-agent collaboration. In The Thirteenth International Conference on Learning Representa- tions (ICLR), 2025.
Xiangfei Qiu, Jilin Hu, Lekui Zhou, Xingjian Wu, Junyang Du, Buang Zhang, Chenjuan Guo, Aoy- ing Zhou, Christian S. Jensen, Zhenli Sheng, and Bin Yang. TFB: Towards comprehensive and fair benchmarking of time series forecasting methods. In Proc. VLDB Endow., pp. 2363-2377, 2024.
Xiangfei Qiu, Xingjian Wu, Hanyin Cheng, Xvyuan Liu, Chenjuan Guo, Jilin Hu, and Bin Yang. Dbloss: Recomposition--asee loss function for time series forecasting, arXiv preprint arXiv:2510.23672. 2025a.
Xiangfei Qiu, Xingjian Wu, Yan Lin, Chenjuan Guo, Jilin Hu, and Bin Yang. DUET: Dual clustering enhanced multivariate time series forecasting. In SIGKDD, pp. 1185-1196, 2025b.
13

---

Published as a conference paper at ICLR 2026
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burs,, Trevor Darrell, and Kate Saenko. Object hal- lucination in image captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4035-4045 Brussels, Belgium, October-November 2018. Association for Computational Linguistics.
Zhiqing Sun, Sheng Shen, Shengeao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yar Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309. 14525, 2023.
Feilong Tang, Zile Huang, Chengzhi Liu, Qiang Sun, Harry Yang, and Ser-Nam Lim. Intervening anchor token: Decoding strategy in alleviating hallucinations for MILMMs. In The Thirteenth International Conference on Learning Representations (ICIR), 2025a.
Feilong Tang, Chengzhi Liu, Zhongxing Xu, Ming Hu, Zile Huang, Haochen Xue, Ziyang Chen, Zelin Peng, Zhiwei Yang, Sijin Zhou, et al. Seeing far and clearly: Mitigating hallucinations in milms with attention causal decoding. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pp. 26147-26159, 2025b.
Qwen Team. Qwen2 technical report. arXiv preprint ar arXiv:2407.1661 2024.
Chenxi Wang, Xiang Chen, Ningyu Zhang, Bozhong Tian, Haoming Xu, Shumin Deng, and Huajun Chen. Milm can see? dynamic correction decoding for hallucination mitigation. In The Thirteenth International Conference on Learning Representations (ICLR), 2025.
Fei Wang, Xingyu Fu, James Y Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, et al. Muirbench: A comprehensive benchmark for robust multi-image understanding. arXiv preprint 2rXiv:2406009411 2024a.
Junyang Wang, Yuhang Wang. Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Jiaqi Wang, Haiyang Xu, Ming Yan, Ji Zhang, al. Amber: An lim-free multi-dimennional benchmark for milms hallucination evaluation. arXiv preprint arXiv:231 1,000937 2023.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision- languge model's 's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024b.
Xintong Wang, Jingheng Pan, Liang Ding, and Chris Biemann. Mitigating hallucinations in large vision-language models with instruction contrastive decoding. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 15840-15853, Bangkok, Thailand, August 2024c. Association for Computational Linguisties.
Zichen Wen, Yifeng Gao, Shaobo Wang, Junyuan Zhang, Qintong Zhang, Weijia Li, Conghui He, and Linfeng Zhang. Stop looking for important tokens in multimodal language models: Duplica- tion matters more. arXiv preprint arXiv:2502.1 1494, 2025.
Dingchen Yang, Bowen Cao, Guang Chen, and Changjun Jiang. Pensieve: Retrospect-then-compee mitigates visual hallucination. arXiv preprint arXiv:2403. 14401, 2024.
Hao Yin, Guangzong Si, and Zilei Wang. Clearsight: Visual signal enhancement for object halluci- nation mitigation in multimodal large language models. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pp. 14625-14634, 2025.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang. Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large lan- guage models. arXiv preprint arXiv:2310, 10.1105,, 2023.
Shukang Yin. Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. National Science Review, 11 11(12):nwae403 2024.
Weihao Yu, Zhengyuan Yang, Linjie Li. Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. In The Forty-first International Conference Machine Learning (ICML), 2024.
14

---

Published as a conference paper at ICLR 2026
Xinlei Yu, Zhangquan Chen, Yudong Zhang, Shilin Lu, Ruolin Shen, Jiangning Zhang, Xiaobin Hu, Yanwei Fu, and Shuicheng Yan. Visual document understanding and question answering: A multi-agent collaboration framework with test-time scaling. arXiv preprint arXiv:2508.00 03404, 2025a.
Xinlei Yu, Chengming Xu, Guibin Zhang. Zhangquan Chen, Yudong Zhang, Yongbo He, Peng- Tao Jiang, Jiangning Zhang, Xiaobin Hu, and Shuicheng Yan. Vismem: Latent vision memory unlocks potential of vision-language models. arXiv preprint arXiv:2511.11007 2025b.
Xinlei Yu, Chengming Xu, Zhangquan Chen, Bo Yin, Cheng Yang, Yongbo He, Yihao Hu, Jiangning Zhang, Cheng Tan, Xiaobin Hu, et al. Dual latent memory for visual multi-agent system. arXiv preprint arXiv:2602.004/ 2026.
Zihao Yue, Liang Zhang, and Qin Jin. Less is more: Mitigating multimodal hallucination from an eos decision perspective. In Proceedings of the 62nd Annual Meeting the Association for Computational Linguistics (ACL.: Long Papers), pp. 11766-11781, Bangkok, Thailand, August
2024. Association for Computational Linguistics.
Bohan Zhai, Shijia Yang, Xiangchen Zhao, Chenfeng Xu, Sheng Shen, Dongdi Zhao, Kurt Keutzer, Manling Li, Tan Yan, and Xiangjun Fan. Halle-switch: Rethinking and controlling object existence hallucinations in large vision language models for detailed caption. arXiv preprint arXiv:2310.01779 2023.
Muru Zhang, Ofrr Press, William Merrill, Alisa Liu, and Noah A. Smith. How language model hallucinations can snowball. In The Forty-first International Conference Machine Learning (ICML), 2024.
Shaolei Zhang, Qingkai Fang, Zhe Yang, and Yang Feng. Llava-mini: Efficient image and video large multimodal models with one vision token. In The Thirteenth International Conference on Learning Representations (ICLR), 2025a.
Zhi Zhang, Srishti Yadav, Fengze Han, and Ekaterina Shutova. Cross-modal information flow in multimodal large language models. In Proceedings of the Computer Vision and Pattern Recogni- fion Conference (CVPR), pp. 19781-19791, 2025b.
Weihong Zhong, Xiaocheng Feng, Liang Zhao, Qiming Li, Lei Huang, Yuxuan Gu, Weitao Ma, Yuan Xu, and Bing Qin. Investigating and mitigating the multimodal hallucination snowballing in large laiiunla models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL: Long Papers), pp. 11991-12011. Association for Computa- tional Linguistics, August 2024.
Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object ballucination large vision-language models. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint (tti::2504.104799 2025.
Xin Zou, Yizhou Wang, Yibo Yan, Yuanhuiyi Lyu, Kening Zheng, Sirui Huang, Junkai Chen, Peijic Jiang, Jia Liu, Chang Tang, and Xuming Hu. Look twice before you answer: Memory-space visual retracing for hallucination mitigation multimodal large language models. The Forty- second International Conference on Machine Learning (ICML) 2025.
APPENDIX
A RELATED WORKS
Visual Hallucination. The tendency of VLMs to generate plausible but non-factual or unsupported content, iee,, visual hallucination, is well-documented in previous works. A common remedy has
15

---

Published as a conference paper at ICLR 2026
been to retrain fine-tune models to better align outputs with ground truth (Zhou al. 2024
Zhai et al. 2023 Yue et al., 2024). but these solutions often demand extensive training resources and additional data. Consequently, interest has grown in training- free techniques. including self-
feedback correction (Lee et al.] 2024 Yin et al. 2023). leveraging auxiliary models for external
knowledge integration (Yang al. 20244), and modifying decoding procedures (Wang et al. [024gc]
Zou et al.] 2025 Wang et al. 2025); Tang et al.) 2025b; Li et al., 2025c) Tang et al.) [025al] Yin
al.] 2025 Liu et al. ) In contrast these papers that focus on single VLM agent, is
not sufficient to address the failure mode of hallucination snowballing that emerges in multi-agent
collaboration, which is the core focus of our paper.
Attention in VLM-Based Agents. The hallucination problem of VLMs can be mainly attributed to and indicated by the attention mechanism. Earlier work found that LVL.Ms tend to attend to broad, global image cues and miss prompt-relevant details (Darcet et al., 2024 Gong et al. 2024 An et al.) 2024). behavior often traced to the Vision Transformer encoder (Alexey) 2020). To address this, some methods boost attention weights for pertinent image tokens (Liu et al. (024c), others select or filter informative visual features and apply contrastive decoding to suppress hallucinations (Huo eet al. 2025). Our work presents an extensive study the attention allocation token and layer- wise analysis to provide a better understanding of how multi-agent pipelines lose visual fidelity during inference turns. Based on this, we further introduce a novel visual flow alleviate hallucination snowballing in multi-agent systems.
B REQUISITE ANALYSES
B. SETTINGS
Attention Allocation. In Section 2 we first calculate the attention allocations of four tokens in different layers among different agent turns. Formally, we first denote the whole token set 'T, consisting of vision token subset V. instruction token subset I, system token subset S, and output token subset O. The attention matrix is obtained Equation and each attention score si,j indicates the attention from the ith token to the jih token. Thus, the attention allocation of a specific token type should be the sum of the attention score where the target this token, which could be calculated as follows:
Allocationttken..oo
ΣΣ Ai (i.j) Mroken JET
ΣΣ NET JET
sijs
(5)
Mioken.type I(i E: T.j Troken type)
(6)
where A denotes the average attention matrix in all attention heads of the Ith layer in this context. The attention allocation of specific tokens explicitly represents the focus in each layer of the model when understanding the task and outputting the responses.
Dropping Tokens in Certain Layers. To drop subsets of vision tokens in shallow, middle, or deep layers, set the hidden states of the subset in specific layers to instead of physical removal, because the latter changes sequence length and disrupts sequence alignment of the attention mechanism. Moreover, the implementation is mainly modified from (Liu et al. 2024b).
Token Selection. As described in Section 2 we select five subsets of vision tokens, and here we elaborate on the selection rules for each subset: (1) Random Tokens: we randomly select from all vision tokens, and limit the number of tokens in the subset to the average number in the other four subsets. Besides, we re-select the random tokens if the size of the largest connected component of selected tokens exceeds 10% of the total selected tokens, to avoid selecting centralized tokens that destroy randomness; (2) Inactive Tokens: we first calculate the average attention value across all lay- ers for each token, then select tokens whose attention values are below the lower quartile and whose fluctuation does not exceed 20%; (3) Rise Tokens and (4) Fall Tokens: we select the tokens with gradually upward or downward attention allocation in the consecutive layers. To filter out insignif- icant fluctuations and better reflect the overall attention trend each token, we utilize a tolerance threshold. When deviations from the trend in the opposite direction do not exceed this threshold, we still consider it as maintaining the original trend; (5) Unimodal Tokens: we select tokens with attention allocation of unimodal distribution, whose peak surpasses the salience threshold w.
16

---

Published as a conference paper at ICLR 2026
Linear
Layered
Random
Circular
Figure 8: The four structures of MAS in our experiments.
B.2 ADDITIONAL RESULTS
POPE (L.i et al. 2023) we use the results of the LLaVA-NeXT-7B (Liu et al., 2024b) model on example. Here, we provide results of different base VLMs to verify the generality of our insights and to avoid model-specific conclusions. The token-wise attention allocations of vision, instruction, system, and output tokens of six common VLMs are demonstrated in Figure [3] the results dropping different subsets of vision tokens in the shallow, middle and deep layers listed in Table 16 and the proportion of different vision tokens among different agent turns are demonstrated in Figure [4] These experimental results from the six models are consistent with the previous conclusions, demonstrating excellent generalization across various models.
As discussed Section 2
C METHODOLOGIES
C.I MAS STRUCTURES
Since our proposed ViF primarily centers on the snowballing of visual hallucinations multi-agent contexts, first briefly delineate the MAS structures defined herein. Existing MAS can be primar- ily categorized into two basic architectures (Guo et al.) (024): the centralized and the decentralized. The former are function-speciialized, involving intricate collaborative workflows or functional divi- sion mechanisms. To mitigate such uncertainty, we adopt the latter decentralized ones, specifically incorporating four distributed structures that feature no central agent and relatively straightforward structure; all agents are equal in status, save for sequential dependencies within the system. As il- lustrated in Figure X we include four particular sub-structures: linear structure (Hong et al., 2024) which implements linear configuration for agent-mediated interactions; layered structure (Ishibashi Nishimural (204)). which comprises multiple hierarchical layers, where agent nodes within the current layer establish connections exclusively with those in the subsequent layer; random struc- ture (Qian et al.] 2025), which establishes random connections among agent nodes, where each agent may dynamically decide to redirect to the subsequent node based current contextual infor- mation. Notably. this structure features unidirectional paths, thereby failing to ensure that a single node can reach all other nodes within the network; and cireular structure (Qian al. 2025). which utilizes fully-com cully-connected mesh, ensuring that each agent node can reach any other node in the system via least one path. Intuitively, among the four MAS structures, circular ones have the densest collaborations and interactions among agents, and theoretically, the multi-agent hallucination snow- balling effects the most serious. Thus, all the experiments except the main results are conducted on the circular structure with more obvious visual hallucination snowballing.
Primarily drawing on the multi-agent collaboration framework (Qian interactive collaboration strategy in our MAS, one anchored in topological ordering of directed acyelic graphs. This strategy employs dual-agent multi-round interaction model: within each edge of the network, in the iterative interaction, adjacent actors are assigned to nodes, and critics are assigned to edges. Specifically, the preceding actor first requests feedback: the critic then provides reflective suggestions and requests further refinement; and finally, the subsequent actor generates an optimized artifact. Through this process, the prior artifact is iteratively refined. Specifically, this collaboration expand the conventional single agent method to multi-agent environments and reduces the context length from quadratic growth to linear growth, allowing the collaborative scaling law in MAS. Consequently, this architectural design is adopted as the core MAS framework in this work.
2025). we adopt an
17

---

Published as a conference paper at ICLR 2026
Table 9: Results of the attention score based and other alternative strategies on LLaVA-NeXT-7B and circular MAS structure.
MM
MMHal-
Hall
Selection Strategy
MME1
MMtV
CHAIR
POPER
AMBER
Bencht
Bencht
Bencht
Yalue-Norm
(585.6
(-23) (-.7(22.11
43.4 (42.2) (-28( (-2.)) (-.0(:2.1) 53.61-2.1)
Yalue-Norm (+1 Buffer)
1587,4
(-26 (-19
43.6 (+2.4) (-2.) (-2.)
(-.7
(-.9
Value-Norm (+3 Buffer)
1588.9 (-10.6)
50.3 (-19)
43.1 (+1.9) (-2.) (-2.4
(1.2
53.9
Value-Nom (+5 Buffer)
1590.2 (-9.3)
50.5 (-12)
(+1.8) (-22) (-2.)
(-.7
(-.1
Value-Nont (+8 Buffer)
1589.8 (:9.7)
73.3
(-1.))
43.3 (+2.1) 90.9 (:2.4 (-1.) (-.6(-1.5)
(4.4
Key-Norm
(36.5)
(-06)
(-0.)
41.5 (10.3) 92.9 (.0.4) 92.1 (-0.6 50.6 (-0.) 55.1 (-0.6
Key-Norm (+1 Buffer)
1594,4
74.2 (-0.4) 51.7 (-0.1)
41.3 (40.1) 93.1 (-0.2) 92.4 (-0.3) 50.9 (-0.) 55.4 (-0.3)
Key- Key-Norm (+3 Buffer)
1595.9 (:3.6)
74.3
51.9 (-0.)
41.2 (0.0) 93.1 (0.2) 92.6 (.0.) 51.2 (+0.1) 55.5
Key-Norm (+5 Buffer)
1595.2 (4.3)
(.03)
51.7 (-0.1)
41.4 (10.2) 92.8 (-0.5) 92.3 (.0.4)
50.9 (-0.2 55.61-0.))
Key-Norm Buffer)
1594.7 (-4.8)
74.4 (-02)
51.5 (-0.3)
41.6 (40.4)
92,9 (.04) 92.3 (-0.4)
50.8 (40.3)
55.4
Attention Score
1599.5
74.6
51.8
41.2
93.3
92.7
51.1
55.7
Crigs
Alreaice Sear
Nee-
Key Sormet Bulles
Nec Nome Bathers
Key Nome 19 Baiter
New Norw Befters
Figure 9: Comparisons of selected tokens using the attention scores and other alternative strategies.
C.2 ALTERNATIVE OF ATTENTION SCORE BASED STRATEGY
Given that scores that are not explicitly stored and are not accessible, we design an alternative token selection strategy inspired by (Wen et al., 2025). Specifically, we utilize the L2 norm of the key to replace the attention score, which reflects the feature strength of the token; a higher value of the norm indicates that the token is relatively more prominent and has more significant semantics. Unlike the strategy introduced in (Wen et al. 2025), which adopts T, norm, we choose T-2 norm amplify the difference between tokens and promote the token selection. Statistically, the overlap of the initially selected tokens of the two strategies is more than 70%; however, the total amount of the Key-Norm based strategy is less than that of the other. Thus, we add buffer tokens of initially selected tokens, which surround the initially selected token of the 3 x space.
2/3 (Dao 2024) are widely used in the latest VLMs, resulting in attention
To verify the effectiveness of the Key-Norm, we compare it with the attention score based strategy as well as Value-Norm. which similar to our alternative. As illustrated Table 9 we observe that the Key-Norm based strategies are superior to the Value-Norm based ones. Besides, selecting by Key-Norm with three buffer tokens almost achieves the same performance as attention scores, even surpassing them in partial benchmarks. For more intuitive results, we provide the visualization comparisons of selected tokens based on these strategies from real cases. As visualized in Figure D the initial Key-Norm based selection covers the most visual relay tokens compared to the attention score based selection; however, the former one is relatively more sparse, losing partially important visual semantics. Adding buffer tokens is a good solution, which selects the surrounding tokens and supplements the information of the visual flow. It is worth noting that we add three buffer tokens when using the alternative strategy in experiments, thereby balancing accuracy and efficiency.
18

---

Published as a conference paper at ICLR 2026
D EXPERIMENTS
D.1 SETTINGS
Baselines. To verify the generality, we totally adopt ten models covering from 7B 34B in our experiments, which are listed as follows:
LLaVA-v1.5 (Liu et al., 2024a) uses a two-layer MLP to connect image features into the word embedding space, and we choose the 7B and 13B models, which have 32 and 40 layers in the decoder of the LLM, respectively.
LLaVA-v1.6 (Liu et al. 2024b) increases resolution limits for input images with augmented data. It is the early version of LLaVA-NeXT which has 32 layers in the LLM.
LLaVA-NeXT (Liu et al. 2024b) has significant improvements in reasoning, OCR, and world knowledge, remaining the same structure LLaVA-v1.6 We include 7B, 13B, and 34B models in our experiments, which have 32, 40, and 60 layers in the decoder.
LLaVA-OV et al. 2025a) the most powerful base VLM in the LLaVA family, support- ing shiggleimmaee multi multi-image, and video scenes simultaneously. It uses the Qwen2 Team 2024) as the LLM, which has 28 layers.
Qwen2-Va (Wang et al. 2024b) introduces naive dynamic resolution and modtilmodal ro- tary position embedding (M-RoPE), achieving impressive image and video understanding. Its 7B model has 28 decoder layers.
Qwen2.5-VI. (Bai et al., 2025) compresses the vision tokens with MLP-based fuser, aligns M-RoPE and absolute time, and meticulously designs a three-stage training pipeline. We select the 7B and 32B models with 32 layers of the LLM.
Benchmarks. We evaluate our method on eight widely adopted benchmarks, including three com- prehensive benchmarks and five hallucination benchmarks, which are presented as follows:
MME (Yin et al. 2024) is the first comprehensive benchmark and measures the perception and cognitive abilities of 14 challenging subtasks. Moreover, the metric is the total score across all the subtasks.
MMBench (Liu al. 2024d) consists of multiple-choiee questions to assess over twenty different ability dimensions, offering a hierarchical framework with three levels. During the assessment, GPT-4 serves the final judge. In our experiments, only include the English subset for evaluation.
MM-Vet Vet (Yu et al. 2024 a comprchensive benchmark, which defines six core capacities and assesses them complicated visual tasks. It provides a GPT-4 based evaluator for open-ended outputs.
CHAIR (Rohrbach al. 2018) a captioning hallucination assessment benchmark, com- paring the objects mentioned in the title with the objects actually existing in the image. Here, we utilize the CHAIRA AIRg metric, calculating the proportion of titles that contain at least one hallucinatory object.
POPE (Li al. 2023) merges several classic visual datasets, and generates binary questions of the existence of objects. Each image is paired with six questions, and we the accuracy metric.
AMBER (Wang et al. 2023) is tailored to assess both generative and discriminative tasks, including existence, attribute, and relation hallucination. We follow the AM BER score in the original paper.
2023) is composed of high-quality image-question pairs to mea- sure the hallucination, and the generated responses are automatically rated by GPT-4.
MMHal-Bench (Sun et al.
HallBench (Guan et al. 2024) is meticulously handcrafted by experienced human experts, and evaluated by a text-only GPT4-assisted evaluation framework.
MMIU MMen et al.] 2024h is designed to assess abilities across diverse multi- tasks, encompassing types multi-image relationships, and 11K meticulously curated multiple-choice questions.
19

---

Published as a conference paper at ICLR 2026
Table 10: Two-stage training details. VLMs utilize various strategies to project vision tokens, such as MLP-based projectors.
Stage One Pre- Pre-Training
Stage Two
Instruction Tuning
Vision Encoder
Frozen Trainable
Frozen
Projector
Trainable Truinable Trainable
Modules
Lurge Language Model
Frozen Trainable
Transformer Blnck (Ours)
Batch Size
256
256
Learning Rate
le-4
Learning Rate
le-5
Settings
Warmup Ratio
0,05
0.02
Optimizer
AdamW (oshchilov & Hutter 2019
AdamW (Loshchilov & Hutter 2019
Epoch
MuirBench (Wang et al.
2024a) consists of 12 diverse multi-image tasks, utilizing pair- wise construction approach. Each standard instance is paired with a minimally semantically distinct unanswerable variant to ensure reliable assessment.
MV-Bench (Li et al. 2024 covers 20 challenging video tasks intractable via single frames. Specifically, it transforms diverse static tasks into dynamic ones enables video tasks requir- ing broad spectrum of temporal skills.
Video-MME (Liu al., 2024d) is the first full-spectrum benchmark in video analysis, distinguished by diverse video coverage, comprehensive temporal scope, multi-modal in- tegration, and expert manual annotations, ensuring precise, reliable model assessment.
Evaluations. For the eight selected benchmarks, we largely follow their original evaluation met- rics. Besides, to assess the level of hallucination snowballing in multi-agent contexts, we propose a hallucination snowballing score (HS), which quantifies both the severity and propagation of halluci- nations. The score could be formulated as follows:
1 N
HS
1 expl D d:) h4,
where d is the hallucination propagation distance, and N are the total distance and the number of agents in the multi-agent agent system. h represents the severity of hallucination, centesimal-pote scale score produced by judge model. Here, we employ GPT-5-20250807 as the judge with the prompt as shown in the end of this paper. The average score on benchmarks measures the hallucination snowballing; the deeper and broader the snowballing of the hallucination, the higher the score.
To investigate the sensitivity of hallucination severity assessment, based on external judges, we con- duct comparative experiments employing Gemini 2.5 Pro (Comanici et al. 2025) as the judge based on POPE benchmark. We observe that over 94% of data discrepancies fall below 10%, demonstrat- ing the robustness of judge strategy. In practical applications, suffices to ensure comparisons conducted under identical judge settings to guarantee the fairness of evaluations.
Implementations. Experiments are conducted on four or eight NVIDIA H20 96G GPUs. The salience of unimodal morphology w is 0.3, the temperature scaling is 0.8. the reallocation coeffi- cient (11, Q22 in the middle and deep layers are set to 0.1 and 0.3, and the temperature of generation refer to the original paper.
set to 1 for MAS. For other configurations of baselines,
Training Pipelines. Our proposed method can be integrated with other base VLMs in MAS to alleviate the multi-agent hallucination snowballing, requiring only one additional module for visual relay selection. Following the typical training paradigm, we employ a two-stage training process including a pre-training stage and instruction tuning stage, as reported Table 10
D.2 ADDITIONAL RESULTS
Additional Analyses of Visual Relay Tokens. To further validate the effectiveness of our selected visual relay tokens in the visual flow, we conduct additional analyses with transformed visual fea- tures and different combinations of subsets of vision tokens. Specifically, employ average
20

---

Published as a conference paper at ICLR 2026
pooling, two-layer MLP, and lightweight transformer (Mehta et al. 2021 1 for visual token com- pression, respectively. We also adopt object-level visual features (Neo et al. 2025). wherein we uti- lize external segmentation model (i.c., SAM2) and incorporate the vision tokens of the predicted mask to relay visual information. Furthermore, we compare the results of different combinations of vision token subsets, defined in Table These additional tokens are randomly selected from the subsets, with the number of additional selections maintained below that of unimodal tokens.
As presented in Table II. uniformly compressed visual features fail to relay information among agents and even exacerbate visual hallucinations due to vision transformation loss, particularly in complex visual scenarios (e.g.. in MMHal-Bencch and HallBench benchmarks). Object-level visual features also yield suboptimal performance, owing to obstacles in external information selection and the introduction of additional latency. Additionally, unimodal vision tokens, adopted for visual information relay, do not gain significant benefits from the integration of other vision token subsets, while incurring extra time and computational overheads.
Correction Capability on Adversarial Visual Inputs. We assess the correction ability of our ViF in adversarial and noisy scenarios, including injecting edited images and mismatched images. The former randomly masks the area in the image, and the latter directly inputs mismatched and wrong image, both of them serve as strong adversarial scenarios. We stochastically inject the adversarial image in the 2 to 4 agent turns, and assess the performance in the following 5, 10, 15, and 20 agent turn, respectively.
As depicted in Figure 10, our ViF exhibits superior correction capability over the baseline when processing noisy and adversarial visual inputs, achieving by dynamically revising visual cognition across agent turns rather than adhering rigidly to prior outputs. As mentioned earlier, base VLMs in the multi-agent context tend to over-rely on prior texts to relay erroneous visual information. Conversely, our proposed visual flow mitigates the propagation and snowballing of hallucinations, thereby enabling enhanced correction and anti-adversarial capacities.
Combination with Other Hallucination Mitigation Strategies. To further enhance the compati- bility and applicability of our method, we evaluate the performance of combinations with existing hallucination mitigation strategies, namely Mem VR (Zou et al., (025). VISTA (Li et al., 2025c), FarSight (Tang et al., (025b), DeCo (Wang et al. 2025), and TAME (Tang et al.) 2025a). As com- pared in Figure [1] we observe that most strategies achieve further improvements when combined with our ViF in multi-agent environments.
Hlype--Paramete Analyses. There are three key arper-parameters in our proposed method, i.e., the salience of unimodal morphology w when selecting visual relay tokens with unimodal distribution, the temperature scaling t in Equation 2 and the reallocation coefficient a and a2 in Equation of the middle and deep layers. As listed Table [2] the lower the w. the more proportions of visual tokens would be included; however, excessive visual relay tokens will not bring extra performance improvement but computation costs. When w is set to 0.3, the model obtains the best results with relatively less token overhead. Besides, as shown in Table 13 and Table 14 when т, O1,, 02 are set to 0.8, 0. 0.3, our model exhibits the greatest potential.
Efficiency Analyses. As reported in Table g our proposed ViF incurs only marginal additional averagelatency
computational overhead in respect of average latency and average number of operations. Addition- ally, as listed in Table the computational overhead of our method remains relatively constant and exhibits no substantial increase with higher resolutions.
Case Study. As demonstrated in Figure 12 we visualize the generation procedure of the MAS equipped with our proposed ViF on four selected samples from two benchmarks. We observe that our method effectively mitigates the snowballing of multi-agent visual hallucinations, thereby en- hancing overall performance. As shown Example (b), although the agent outputs incorrect an- swers regarding object detection in the first turn, subsequent turns still accurately identify the per- ceptual target through visual flow information. Furthermore, as illustrated in Examples (c) and (d), misunderstandings of visual information in images lead to erroneous semantic outputs in early agent turns; however, such errors are not propagated throughout the multi-agent procedure via the visual flow, thus suppressing the snowballing of visual ballucinations.
21

---

Published as a conference paper at ICLR 2026
Table 11: Results of different transformed visual features or various combinations of subsets of vision tokens on LLaVA-NeXT-7B and circular structure, the visual flow to relay visual informa- tion. Due to the need for external model to obtain object-level visual features, we cannot calculate the end-to-en latency of this method. "Perf." indicates the quantitative performance the bench- mark.
CHAIR
POPR Perfi
AMBER
MMHal-Benne Bench HallBench
Visual Flow
Perf. Latencyl
Latency)
Porft
Latency.
Perf.t Lateney]
Perfit
Latency]
Baseline
43.00
3.16
91.0
2.46
89.4
2.79
47.9
3.48
53.1
3.91
Compressed Visual Features (Pooling)
143.9
3.33
89.6
2.61
89.2
2.95
42.5
3.64
46.7
4.05
Compressed Visual Features (MLP)
42.7
3.40
91.6
2.73
90.4
3.02
44.9
3.77
51.8
4.18
Compressed Visual Features (Transformer)
42.2
2.96
48.5
3.99
53.3
Object-level Visual Features
42.5
92.3
91.8
47.2
53.4
(e) Unimodal (a) Rise
41.0 3.55
92.7
2.87
92.1
3.20
50.6
3.87
54.8
4.33
(e) Unimodal (b) Down
42.0 3.56
92.3
2.89
91.3
3.22
49.7
3.88
53.7
4.34
(e) Unimodal (a) Rise (b) Down
41.3
3.63
93.1
2.94
92.9
3,28
50.3
3.94
54.1
4,40
(e) Unimodal (Ours)
41.2
3.47
93.3
2.79
92.7
3.10
51.1
3.83
55.7
4.23
Baseline (Edited Image)
+VI (Edited Image)
100
VIF
wie Combination
Combinating VIF
100
+VI (Mismatched Image)
Basellne (Mimatchhdd Image)
90.5
93.3
54.0
92.7
92.3
Performunc
Performance
30
10
Agent Torns
McmVK
VISTA
ParSight
DrCo
TAME
Figure 10: Analyses of cor- Figure 11: Comparisons of the combination our proposed ViF rection ability LLaVA-Next- and other hallucination mitigation strategies L.aVA-Next-7BB 7B and circular structure when and circular structure, evaluated by POPE benchmark. Specifi- feeding adversarial visual inputs, cally, incorporate five training-free methods, which are seam- evaluated by POPE benchmark. lessly integrated with our approach.
Influence of the temperature morphology on LLaVA-NeXT-7B and circu- scaling 7 on LLaVA-NeXT-7B and circu- lar MAS structure.
Table 12: Influence of the salience of unimodal Table 13:
lar MAS structure.
Ratio %
MMHal-
Hall
MMHal- Hall
CHAIR
POPET
AMBER
CHAIR
POPE
AMBER*
Bench
Bench
Bench*
Bencht
52.7
0.6 44.1
91.4
90.8
48.2
52.1
0.2
6.7
42.8
91.5
89.2
49.1
52.8
0.7 43.0
92.4
91.8
50.3
54.2
0.3
233
41.2
93.3
92.7
51.1
55.7
0.8 41.2
93.3
92.7
51.1
55.7
1.3
41.7
92.5
92.0
49.8
54.9
0.9 41.0
93.
92.2
48.9
53.8
91.1
47.9
53.0
1.0 41.7
92.5
91.6
47.7
52.8
Table 14: Influence of the reallocation coeffi- cient aj and a2 on LLaVA-NeXT-7B and cir- cular MAS structure.
MMHaa- Hall
CHAIRI
POPET
AMBER
Bench!
Bench
0.0 0.2
43.2
89.6
47.2
47.6
0.0
0.3
42.8
90.0
89.3
47.4
48.2
0.0
0.4
43.0
89,8
89.0
47.1
47.8
0.1 0.2
41.3
93.1
92.6
50.7
55.3
0.1
51.1
55.7
0.1 0.4
41.8
92.9
92.4
50.5
$4.9
0.2 0.2
42.9
90.6
89.2
47.3
48.1
0.2
0.3
43.1
89.8
88.5
46.9
48.0
0.2 0.4
42.9
89.6
88.9
46.5
47.7
22

---

Published as a conference paper at ICLR 2026
Table 15: Efficiency comparison hetween our ViF and the base models on L.aVA-NeX1-7B and the circular MAS architecture, which evaluated with different image resolutions. We employ the original-resolution samples from the CHAIR benchmark and conduct bilinear interpolation to downsample them to 1/2, 1/4, and 1/8 of the initial resolution.
1/8 Resolution
1/4 Resolution Latency] FLOPs) Latency), FLOPs) Latency). FLOPs)
1/2 Resolution
Original Resolution
Base Agent
Latency, FLOPs)
Baseline
1.90
103.5
120.8
2.69
133.6
3.16
157.3
+Ours
2.17
112.2
2.57
130.0
3.01
143.8
3.47
168.5
Example (a)
Example (b)
Are all of the cats the same color?
Is there tocthbrush in the image?
Agent Turn 1: There are many cats and they are in different colors,
Agent Turn 1: There is cal on the sink, faucets, tissue holder, mirror. But there no toothbrush. So
no.
Agent Turn 2: One orange. two grey, and mixed ones, they are not the same collos,.
There are some small pink and green brush-shaped objects, typically found in toothbrush cups beside sinks.
Agent Turn 2:
Agent It confirmed coats are diverse,
answer yes.
Agent Turn 3: Those upright, rod-like objects matched the shape and position of toothbuuhes perfectly. So the answer is yes.
Example (c)
Do the persons live in this house? Answer it and give the
Example (d)
Why does the woman have both hands air? Answer and give rationale.
Agent Turn 1: These people were dressed with great refinement, the his hand, possibly knight of noble birth. The decor of the room was exeeptionally lavish, perfectly matching their attire. So they are likely live in the house.
the right
sword
Agent Turn I: The stands basketball court with a hoop in front of her, but there is no basketball. She is dressed gorgeously and swinging her arms, she likely dancing the
Turn 2: The corridor arranged with great luxury and solemnity. And these people walked struight ahead, with general clothing, as if they were on tour or being led, rather than moving about in familiar house. the people probably
She shot baskethall because mising both hands crucial for generating power and necurncy in a jump shot or set shot.
Agent Turn 2:
Agent Turn 3: Based on the way these people dressed and also their behavior and demeanor, they are more
Agent Turn 3: She just shot the basketball because she is in the basketball context and the movements of her arms indicate shoot.
So they
here.
Figure 12: Case study of the results of ViF based on LLaVA-NeXT-TB across MM-Vet and POPE benchmarks. For clarity, we choose the linear structure and the agent turn is set to 3.
23

---

Published as a conference paper at ICLR 2026
Lapen
Ligens
LLaVA-v1.5-7B
Vilm Taaee
Oeggnn Mare
LLaVA-vI.6-7B
Vilm 7aaee
Oroper Thare
Syroe
LXXTT-
2
Video 7are
Orger
Lanen
Ligen
LLaVA-OV-7B
Oeper
Yrles
Lann
Layen
Ligen
Lagin
(e) Qwen2-VE VL
Oepenr Takm
Lapen
Ligers
Lapors
Qwen2.5-VL.*
Figure 13: Layer-wise attention allocation of six models in different agent turns, and denotes that using Key-Norm to replace the attention score.
24

---

Published as a conference paper at ICLR 2026
Table 16: Results of six VLMs when dropping different selected subsets of vision token in the shallow, middle, and deep layers.
Shallow Lavers
Middle Layers
Deep Layers 50%
25%
50%
75%
100%
25%
50%
75%
100%
25%
75%
100%
Drogping
(n) Random
49.1
41.3
36.1
28.7
71.3
64.1
60.2
56.2
81.6
81.1
80.6
80.2
LLAVA-
Inactive
53.3
44,6
38.9
29.7
82.7
78.9
80.1
76.7
52.8
81.9
82.1
81.7
Fall
10.0
36.1
28.2
19.9
75.2
62.5
55.6
50.0
82.4
81.6
81.1
80.6
Unimodal
39.6
35.9
27.9
22.4
51.3
43.1
35.0
22.6
81.6
81.3
80.5
80.9
Dropping;
Random
49.9
42.2
37.2
28.0
77.0
63.6
60.1
58.1
81.8
81.1
80.7
80.9
LLAVA-
(b) Inactive
52.1
43.8
39.5
30.5
83.6
$2.4
78.6
76.1
$2.9
83
82.7
82.6
vl.67B
Rise
39.2
33.8
28.2
18.5
78.9
61.6
54.7
51.2
81.4
80.8
80.4
80.1
39.6
36.6
57.5
82.5
(e) Unámodal
40.9
36.4
28.0
20.9
46.0
42,9
33.9
23.4
82.4
81.6
80.6
80.1
vofo
85.2
Random
LLaVA-
(b) Inactive
55.1
46.2
41.8
32.5
84.3
82.9
81.5
78.3
85.0
84.6
84.3
84.6
NcXT-7B
Rise
41.9
35.6
29.6
20.8
79
64.2
56.4
52.3
83.6
82.7
81.8
81.6
Unimodal
42.1
37.6
30.0
22.8
52.9
44.5
36.6
25.3
84.4
83.0
82.3
81.8
w/o Dropping
87.6
Random
66.8
LLaVA-
Inactive
58.8
46.5
44.9
36.4
83.6
80.2
74.4
67.8
87.7
87.3
87.2
$7.0
DB
(c) Risc
44.7
39.5
31.1
22.4
81.2
65.4
62.4
55.5
87.1
86.8
86.
86.5
Fall
45.2
42.1
33.9
24.8
62.6
66.1
59.6
53.6
87.2
86.7
86.6
86.4
33.8
26.1
39.7
wfo Dropping
85.9
Random
53.2
44.7
39.8
31.2
79.3
67.1
62.6
59.0
54.8
84.0
83.5
83.0
Qwee2-
VL-7B
(c) Rise
43.2
36.7
29.8
21.2
80.0
64.2
56.5
53.9
85.2
84.8
84.1
83.7
Fall
43,0
40.1
31.6
23.4
78.9
65.5
60.0
52.8
K4.4
84.2
84.6
83.9
Unimodal
w/o Dropping
85.6
(a) Random
$1.5
44.8
29.2
31.2
80.0
66.3
63.9
59.5
84.4
83.9
83.6
83.4
Qwen2.5
56.5
VL-7B
Rise
12.4
36.6
30.4
20.9
80.3
63.6
55.7
52.5
84.9
83.3
83.2
83.1
(d) Fall
42.7
58.8
31.0
23.0
80.0
64.2
59.0
$4.3
$4.8
83.9
83.3
82,8
Unimodal
42,1
38.1
30.4
23.1
54.0
44.7
37.0
85.0
84.8
83.6
83.4
A
Ape
Agel
Aged
(a) LLaVA-v1.5-7B
(b) 66.VA-v1.6-7B
(c) LeXTA-
(d) LLaVA-OV-7B
Age Tem
Tome
(c) Qwen2-VL- VL-7B
(f) Qwen2.5 VL-7B
Figure 14: Proportion of vision tokens subsets of six models in different agent turns.
25

---

Published as a conference paper at ICLR 2026
Prompt for Evaluating the Severity of Hallucination
You are striet viaual hallucination judge for vision- language models.
Your job is to evaluate the level of visual hallucinetion, gäven: correspanding visual inputs.
user instruction
2. the full generated output frem the model.
3. the ground-truth the instruction.
You
ha incinations and their severity
130 scale.
Definitions:
"visual hallseinatien" is: class of hallucination phenemena in which vision -language modal, conditioned in tasks that inzensistent with, incorrect sbout, entirely fabricated beyond the given visual content, thareby making its outputa unfaithful to the eviderce. Me
visual inputs, generatea visual detaila
hallucinations
facta reasoning,
tasks:
1. Decida whether the generased output contains visual hallacinations. (combining
visual inputs and ground-truth for verification! hallucinations. 3. putput a SEVERTTY score fron 0-100 (integer).
2. yea, briefly explain
Severity guidelines 10-100):
hallucination. sully consistent with references context.
1-20; Very minor fasues, local ls, small inaccuracies that do not change the main conclusion.
21-40: Cleer but localized hallucinstiens, The main cenclusion still mestly
correct.
Iaportant hallucinations significantly affect part core reasening.
61-80: hallucinarians. answer largely incorress or migleading.
B1-100: Extreme hal lucinatioo.. The answer is almost entirely fabricated or contradicis the refecences.
Output
You RUST output single valid usuN object with these fields:
"explanation short natural language explanation 11-3 sentences, can be empty there is no hallucination)
"severity"; integer in [0, 100]
Be concise but precise in your explanstion. Do NCT include any text out cutside the JS08.
26


Reviews:
[{"review_id": "CCiAJlW21s", "reviewer": "Reviewer_6R98", "summary": "This paper introduces a new task setting for VLM multi-agent system (MAS), hallucination snowballing. Hallucinated textual response from early agent turns may get passed down to subsequent turns, causing agents to receive spurious textual input and thus inducing further errors and hallucinations. Through extensive analysis of attention distribution over different input token segments, agent turns and transformer layers, the authors discovered that unimodal visual tokens in middle layers produced the most accuracy change when these tokens are removed. As such, unimodal tokens would potentially be most impactful for visual text interaction and is selected as visual relays tokens to carry visual information to subsequent turns. With this finding, the authors propose ViF, which concatenates these relay tokens with instruction and visual tokens to enhance downstream agent inference. Additionally, an attention reallocation module is designed to shift inactive attention to other regions and promote formation of unimodal token patterns. ViF method is extensively evaluated across 8 benchmarks on 6 different base models. Experimental results surpassed previous methods and proved the effectiveness of ViF.", "questions": "1. Table 2 shows experimental results for 4 different MAS structures. How are the conventional single VLM evaluation datasets such as POPE tested in multi-agent settings? Could you please provide an example of how a Yes/No question with image is processed in multi-turn agent turns? \n2. Could you please provide more experimental setup details for comparison results in table 5? How are other SOTA methods evaluated in MAS settings following circular MAS structure? \n3. There is a potential inconsistency in classifying ViF as plug-and-play. The method applies a trained transformer block to embed visual relay tokens. If this transformer block must be retained for each new base model (particularly one with a novel visual encoder), it would constitute a model-specific training phase, which is contrary to the concept of a true plug-and-play component.\n4. The paper proposes a new HS metric for snowballing hallucination. The severity of hallucination score, h, is said to be derived from a judge. Could you please provide more information about this judge and how to obtain h from the 8 benchmarks? \n5. As shown in table 1, dropping rise and fall tokens in middle layers also contributed significant accuracy change, indicating that they are also important (though not the most important) in model inferencing. Would it be possible to consider combining multiple types of visual tokens as relay tokens? How would the model perform when selecting a portion of rise, fall and unimodal tokens as relay tokens?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors introduce a new task setting for MAS hallucination and create a multi-turn evaluation metric which could be beneficial for future researchers. \n\nThis paper presents a rigorous evaluation of ViF, testing its generality across six base models of varying sizes and architectures. The authors also provide extensive ablation studies and parameter analysis to validate the method and demonstrate the impact of its components and parameter choices.\n\nThe authors also work to improve the method's speed and efficiency by maintaining compatibility with Flash Attention modules. Instead of using attention scores, alternative token selection methods based on key norms and value norms are tested and presented in Table 7 and Figure 8. These experiments support the validity of using key norms as a proxy for attention scores, thereby maintaining compatibility with Flash Attention.", "weaknesses": "While the paper is well-written with a clear logical flow, it omits crucial experimental details regarding the MAS evaluation. Specifically, it does not explain how conventional single VLM evaluation benchmarks are adapted for the MAS setting. This omission may hinder readers unfamiliar with multi-agent setups from fully understanding the significance of evaluation results with respect to snowballing hallucination. The paper would be strengthened by providing more prerequisite information on MAS evaluation protocols.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hVoNIfsvCB", "reviewer": "Reviewer_HX6A", "summary": "The critical failure of multi-agent systems lies in the compression and selective amplification of visual evidence by the \"text stream\", causing early hallucinations to spread along the information chain;ViF complements and stably transfers visual evidence through the \"visual stream\", and, in conjunction with attention redistribution, maintains visual dominance in the middle layer, significantly reducing the snowball effect in multi-round scenarios.", "questions": "1. **Cross-domain generalization:** Under video question answering, multi-image reasoning, and tool-enhanced workflows, is the benefit of ViF stable?\n\n2. The proposed ViF approach seems to mainly focus on the impact of visual tokens that have undergone an **attention sink**. Previous studies have mainly focused on the level of single agent and advocated that alleviating the phenomenon of attention sink can improve the performance of the model.[1, 2] However, some researchers, like the author, have proposed using attention sink to enhance the performance of the model. Recent Qwen3-Next series model replaces standard attention with the combination of Gated DeltaNet and Gated Attention, which enables efficient context modeling for ultra-long context length while alleviates the attention sink. If the ViF method is used on VLMs developed based on such models in the future, can similar gains still be obtained?\n\nref:\n\n[1] Huang Q, Dong X, Zhang P, et al. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 13418-13427.\n\n[2] Chen Z, Li M, Chen Z, et al. Advancing general multimodal capability of vision-language models with pyramid-descent visual position encoding[J]. arXiv preprint arXiv:2501.10967, 2025.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- **The problem definition is specific and MAS-oriented:** The paper identifies \"visual hallucination snowballing\" as a critical failure in multi-agent scenarios and makes a diagnosis from the triple perspectives of layer-round-token; this focus differs from the paradigm of traditional single-agent LVLM hallucination research. \n\n- **The design concept of the visual stream is novel:** By combining \"visual relay token + attention redistribution\", a non-textual channel is introduced at the cross-agent communication level, aiming to stably retain and amplify visual evidence, and conceptually has a concise and scalable appeal. \n\n- **Diagnosis and Ablation Adequacy:** Quantitative trends of inter-layer and round attention, hierarchical deletion of shallow/middle/deep layers, and the proportion of key subsets changing with rounds, with the overall analysis chain being complete; ablation covers the proportion of visual relay tokens and the contributions of each layer to attention redistribution, supporting the claim.\n\n- **Compatibility Emphasis:** Positioned as \"Plug and Play\", it provides Key-Norm alternatives and buffer strategies on models with FlashAttention to lower the threshold for reproducibility.", "weaknesses": "1. **Computation and latency overhead not reported:** Transmitting visual relay tokens across agents increases sequence length and attention computation, and attention redistribution also introduces additional operators. It is recommended to report the end-to-end throughput/latency curves (varying with agent rounds, image resolution, model scale, and etc.), and compare them in parallel with the baseline of text-only streams.\n\n2. **The coverage of the task domain is limited:** The main evaluation is on the understanding and question answering of static images. It is recommended to verify the generalization of ViF on agentic tasks with external tool usage, especially in multi-round planning and interaction scenarios, to examine whether visual relaying is more deeply coupled with the workflow. \n\n3. **Reliability and External Validity of the HS Metric:** HS relies on the judge model to evaluate \"hallucination severity\" and \"propagation depth\". It is recommended to conduct  reviewer consistency and model sensitivity analysis , and perform external validity testing on the  manually annotated  snowball diagnostic set; at the same time, disclose the version and prompt of the judge model. \n\n4. **Comparison with Structured Visual Features:** Besides \"visual token direct transfer\", schemes that transfering compressed/pooled spatial feature maps or object-level memory can be compared with ViF in terms of performance-overhead-robustness differences.\n\n5. **Analysis of Strong Adversarial Scenarios:** When the visual judgment of the first agent is severely deviated or deliberately injected with errors, can ViF \"correct rather than follow\"? It is recommended to construct stronger adversarial distributions to evaluatethe cross-round correction rate of early errorsandthe anti-interference ability of the visual stream.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M3Izwu4PPA", "reviewer": "Reviewer_7MHV", "summary": "The authors propose ViF, a lightweight plug-and-play method to reduce visual hallucination snowballing in Visual Multi-Agent Systems built on Vision-Language Models. They observe that as multi-agent interactions progress, agents increasingly rely on textual context while losing attention to visual evidence, causing errors to compound. To address this, ViF introduces two mechanisms: Visual Information Relay, which passes key visual tokens between agents to preserve visual grounding, and Attention Reallocation, which restores attention to visual features in middle and deep layers. Since modern VLMs use Flash-Attention, which hides attention maps, ViF uses the key norm to identify important vision tokens efficiently. Experiments on eight benchmarks and ten base models show consistent gains of about 2–3.5% in both general and hallucination-focused evaluations. Overall, ViF is an effective and easily integrable framework that improves visual consistency and reduces hallucination propagation without retraining underlying models.", "questions": "1. How were the attention reallocation parameters determined? Are they fixed hyperparameters or adaptively tuned per model and layer?\n\n2. When using key-norm as a proxy for attention importance, did you compare it empirically with models where attention maps are accessible to verify its reliability?\n\n3. Could you provide qualitative examples showing how ViF changes an agent’s reasoning trajectory or visual focus compared to the baseline?\n\n4. Since ViF passes selected vision tokens between agents, how do you prevent error propagation if the selected tokens themselves are noisy or misleading?\n\n5. Have you tested whether combining ViF with other hallucination mitigation techniques (e.g., grounding via retrieval or contrastive filtering) yields additive improvements?\n\n6. What is the computational cost (both in FLOPs and latency) introduced by ViF during inference, and is it negligible compared to the base MAS pipeline?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The proposed ViF method is lightweight and plug-and-play, requiring no model retraining or architecture modification, making it highly practical for integration into existing vision-language frameworks.\n\n2. The paper provides clear empirical evidence through attention analysis that supports the identified problem and motivates the design of ViF, showing strong interpretability and reasoning behind the approach.\n\n3. Extensive experiments across eight benchmarks and ten base models demonstrate the method’s robustness and generality, with consistent performance gains across both general and hallucination-specific evaluations.\n\n4. The work contributes to the reliability and stability of multi-agent VLM systems, an emerging but underexplored research direction, making it a meaningful step toward safer and more grounded visual reasoning in collaborative AI agents.", "weaknesses": "1. Although the paper claims ViF reduces hallucination propagation, it does not include qualitative or human evaluations to confirm whether the mitigated errors are genuinely perceptual or semantic, limiting the interpretability of improvements.\n\n2. The key-norm heuristic used as a substitute for attention may not perfectly capture true token importance, and the paper does not analyze failure cases or potential mismatches between key-norm values and actual visual relevance.\n\n3. The attention reallocation strategy is presented as effective, but the specific reallocation rule (e.g., scaling factors or redistribution formula) is not fully described, reducing reproducibility.\n\n4. The framework assumes that hallucination arises primarily from attention degradation, but other causes of hallucination are not addressed, which may limit the generality of the approach.\n\n5. The computational overhead and latency introduced by token relaying between agents are not quantified, which could be important for real-time or large-scale multi-agent applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)