Visual Document Understanding and Question Answering: A Multi-Agent
  Collaboration Framework with Test-Time Scaling
Authors: name: Xinlei Yu; name: Zhangquan Chen; name: Yudong Zhang; name: Shilin Lu; name: Ruolin Shen; name: Jiangning Zhang; name: Xiaobin Hu; name: Yanwei Fu; name: Shuicheng Yan
Published: 2025-08-05 12:52:09

Abstract: Existing vision-language models (VLMs), whether generalists or specialists,
remain constrained by their parameter scale, lack robust self-correction
capabilities, and underperform in tasks involving long visual contexts and
complex reasoning, resulting in suboptimal performance on document-based tasks.
To address this, we propose MACT, a Multi-Agent Collaboration framework with
Test-Time scaling, tailored for visual document understanding and visual
question answering (VQA). It comprises four distinct small-scale agents, i.e.,
planning, execution, judgment, and answer agents, with clearly defined roles
and effective collaboration. Notably, the judgment agent exclusively verifies
correctness and redirects to prior agents for revisions, outperforming
conventional correction strategies. To further expand the capability boundaries
of the framework, we propose mixed reward modeling that balances agent-specific
abilities and global collaboration, as well as agent-wise hybrid test-time
scaling, which customizes different scaling strategies for each agent based on
their functions. Evaluated on benchmarks spanning both document-based and
non-document-based settings, our MACT shows superior performance with a smaller
parameter scale without sacrificing the ability of general and mathematical
tasks. Especially, it stands out in benchmarks involving long visual contexts
and complicated reasoning. The three variants of MACT consistently hold the top
three positions in average scores, leading in 13 of the 15 benchmarks. Code
will be available at: https://github.com/YU-deep/MACT.git.