Visual Multi-Agent System: Mitigating Hallucination Snowballing via
  Visual Flow
Authors: name: Xinlei Yu, name: Chengming Xu, name: Guibin Zhang, name: Yongbo He, name: Zhangquan Chen, name: Zhucun Xue, name: Jiangning Zhang, name: Yue Liao, name: Xiaobin Hu, name: Yu-Gang Jiang, name: Shuicheng Yan

Multi-Agent System (MAS) powered by Visual Language Models (VLMs) enables
challenging tasks but suffers from a novel failure term, multi-agent visual
hallucination snowballing, where hallucinations are seeded in a single agent
and amplified by following ones due to the over-reliance on textual flow to
relay visual information. Through turn-, layer-, and token-wise attention
analyses, we provide detailed insights into the essence of hallucination
snowballing regarding the reduction of visual attention allocation. It leads us
to identify a subset of vision tokens with a unimodal attention peak in middle
layers that best preserve visual evidence but gradually diminish in deeper
agent turns, resulting in the visual hallucination snowballing in MAS. Thus, we
propose ViF, a lightweight, plug-and-play mitigation paradigm that relays
inter-agent messages with Visual Flow powered by the selected visual relay
tokens and applies attention reallocation to amplify this pattern. The
experiment results demonstrate that our method markedly reduces hallucination
snowballing, consistently improving the performance across eight benchmarks
based on four common MAS structures and ten base models. The source code will
be available at: https://github.com/YU-deep/ViF.git.