MATA: A Trainable Hierarchical Automaton System for Multi-Agent Visual Reasoning
Authors: Zhixi Cai
Fucai Ke
Kevin Leo
Sukai Huang
Maria Garcia de la Banda
Peter J. Stuckey
Hamid Rezatofighi
Venue: ICLR 2026 Poster (2026)
Keywords: Visual Reasoning
Agent
Neuro-Symbolic
Primary area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
TL;DR: We introduce a visual reasoning method as a hierarchical automaton with learning a high-level LLM-based transition policy over collaborating and competing agents to deliver interpretable, robust visual reasoning.

Abstract: Recent vision-language models have strong perceptual ability but their implicit reasoning is hard to explain and easily generates hallucinations on complex queries. Compositional methods improve interpretability, but most rely on a single agent or hand-crafted pipeline and cannot decide when to collaborate across complementary agents or compete among overlapping ones. We introduce MATA (Multi-Agent hierarchical Trainable Automaton), a multi-agent system presented as hierarchical finite-state automaton for visual reasoning whose top-level transitions are chosen by a trainable hyper agent. Each agent corresponds to a state in the hyper automaton, and runs a small rule-based sub-automaton for reliable micro-control. All agents read and write a shared memory, yielding transparent execution history. To supervise the hyper agent’s transition policy, we build transition-trajectory trees and transform to memory-to-next-state pairs, forming the MATA-SFT-90K dataset for supervised finetuning (SFT). The finetuned LLM as the transition policy understands the query and the capacity of agents, and it can efficiently choose the optimal agent to solve the task. Across multiple visual reasoning benchmarks, MATA achieves the state-of-the-art results compared with monolithic and compositional baselines. The code and dataset are available at https://github.com/ControlNet/MATA.

Published as a conference paper at ICLR 2026
MATA: A TRAINABLE HIERARCHICAL AUTOMATON SYSTEM FOR MULTI-AGENT VISUAL REASONING
Zhixi Cai, Fucai Ke, Kevin Leo, Sukai Huang, Maria Garcia de la Banda, Peter J. Stuckey, Hamid Rezatofighi Monash University, Australia
izhixi.cal fucai kel, kevin. n. leo, sukai. huang, mar maria. garci ade labanda, beter.stucket hamid rezatof fighi @monash. edu
Hyper Agent
Rule-based Transition
Trainable Transition
Hyper Agent
Image & Query
Control Transitions
Traverse possibll transitians
Hyper Automaton
Agents
Sub- Automata
Transition Trajectory Tree
Sub Modules
Convert SFT format
d O5//00 Sub Modules
Current Memory
Best Next State
Sub Modules
Supervised Finetune (SFT)
Hyper Agent (c) Hyper Agent (Ours) Training
(a) Linear Pipeline!
(b) Hierarchical Automaton (Ours)
Figure 1: Overview of MATA. (a) Linear pipelines (previous methods) execute modules in fixed, manually designed order. (b) MATA organizes agents as states in a hyper automaton. A trainable hyper agent learns high-level transitions between agents (blue arrows), enabling collaboration and competition, while each agent runs small rule-based sub-automaton for reliable micro-control (black arrows). (c) To train the hyper agent, expand a transition-traaector tree per image-query, score the leaves using task metries, and convert each node's snapshot into a supervised pair current memory >> best next state for supervised finetuning (SFT), forming SFTT-SFT--0K.
ABSTRACT
Recent vision-language models have strong perceptual ability but their implicit reasoning is hard to explain and easily generates hallucinations on complex queries. Compositional methods improve interpretability, but most rely on sin- gle agent or complementary agents or compete among overlapping ones. We introduce MATA (Multi-Agent hierarchical Trainable Automaton), multi-agent system presented as hierarchical finite-state automaton for visual reasoning whose top-level tran- sitions are chosen by trainable hyper agent. Each agent corresponds to state in the hyper automaton, and runs a small rule-based sub-automaton for reliable micro-control. All agents read and write shared memory, yielding transpar- ent execution history. To supervise the hyper agent's 's transition policy, we build pairs, forming
pipeline and cannot decide when to collaborate across
traniiiion-taajecton trees and transform to memory-to-next-stee the MATA-SFT-90K dataset for supervised finetuning (SFT). The finetuned LLM the transition policy understands the query and the capacity of agents, and it can efficiently choose the optimal agent to solve the task. Across multiple visual reasoning benchmarks, MATA achieves the saate-of-the-ann results compared with monolithic and compositional baselines. The code and dataset are available at https://githis wwt.:/wwww......ttt..oooooooao

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Visual reasoning is the cognitive process of interpreting and analyzing relationships among entities
in a visual scene to support decision-making and problem-solving (Ke et al., 2025b). Although recent Vision-Language Models ( (VLMs) (Liu et al. [023a] Chen et al. 2024 Bai et al., (025b) demonstrate strong perceptual ability, their implicit reasoning is difficult to audit and often causes hallucinations on complex queries involving spatial relations, spatial attributes, and counting. Com-
positional approaches (Surís et al. 2023) You et al. 2023; Ke et al., 2024) Cai et al. 2025) improve
interpretabilityy by decomposing task into planning, perception, and reasoning stages, typically em- ploying Large Language Models (LLMs) (Gemini- Teamm [2023] OpenAl 2024 DeepSeek A (025)
as planners code generators and Vision Foundation Models (VFMs) (Radford et al. 2021] Liu
eet al. 2023b Xiao et al.) 2024 Yang et al. 2024) as perceptual tools. Despite these improvements,
non-agentic compositional methods (Surís al. [023 Lu al.
2023) struggle in practice: they
limited to single-turn reasoning, thus lacking the ability to incrementally reason in a closed-loop.
Due to these limitations, agentic methods Yoou al.
2023 Ke et al.
2024 Gao al. 2024 Zhong
et al. 2025) treat visual reasoning as multi-step feedback loop in which agents actively take actions based on the current state (Ke et al. 20256).
However, most agentic systems still employ a single agent, which is often insufficient for complex reasoning (Wang et al., (025c) as different skills are required for different parts of problem. Fur- ther, in prior multi-agent methods (Hong et al.) 2023 Li et al., 2024 Nguyen et al. 2025 /Zhang (et al. 2025) (developed for other domains), collaborative agents are assigned disjoint roles for dif- ferent subtasks and are organized into hard-coded pipelines. While this is simple and interpretable, it prevents error and ballucination handling, and tends to propagate upstream mistakes through the pipeline (Gao et al., 2024: Ke et al., 2005aa.. In contrast, competition mechanism where function- ally overlapping agents for the same subtask work together is this paper, we explore compositional multi-agent visual reasoning in environment where collab- orative and competitive agents exist.
in previous work. In
Motivated by the requirements above, we cast this decision problem a finite-state automaton where the transition function picks a discrete next state and the lifecycle is naturally expressed by explicit states and transitions. This provides explainability, verifiable control flow, and modularity that yield greater versatility, reliability, and performance. A recent work (Cai et al.] (0255) also used an automaton, but its hand-written rule-based transitions are inflexible and difficult to manually define as states and transitions grow (Wang et al.) 2025a Yue et al., 2025 Dang et al.. 2025 Wan al. 2025). When new agents are added, their transitions need to be manually defined. Designing rules to select among functionally overlapping (competitive) agents is hard since the criteria are ambiguous and task-dependent, and human priors about which agents fit which tasks and queries are uncertain. We therefore design a trainable hyper agent to learn a transition policy that selects the next state. Notably, not every transition needs learning: within an agent, micro-steps (e.g., LLM/VLM prompting, verifier checks, tool I/O) follow clear procedures that are easy to define. As the number of agents grows, the main difficulty is cross-agent transition rather than agent's inside control. This motivates a hierarchical automaton in which each top-level state is agent with a small rule-base based uub-automaton, and a trainable hyper agent provides the transition function that observes the shared memory and selects the next agent. All agents read and write shared memory that records variables, tool outputs, code history, and verifier feedback, recording an explainable process. This replaces an inflexible rule-based transition policy with a data-driven, error-aware, and dynamic policy that can redirect to alternative solutions when needed. This design focuses on learning the ambiguous selection between competitive agents, while preserving reliable execution inside agents.
We introduce these ideas in MATA (Multi-Agent hierarchical Trainable Automaton), a hierarchical automaton for visual reasoning. MATA contains a specialized agent for fast, System 1 1-style percep- tion (e.g., object detection, simple question answers); a slow, System 2-style step-wise reasoner that generates and executes Python programs for multi-step inference; and one-shot workflow reasoner that solves queries without iteration.
To supervise the hyper agent, we need labeled transition decisions. We therefore run the system for each image-query pair, expand a transition trajectory tree (Kearns et al., 2002) and log the state his- tory, prompts, intermediate artifacts (detections, captions, code), feedback, and performance results.
2

---

Published as a conference paper at ICLR 2026
The leaves scored by the appropriate task performance, and each decision is labeled with the child that leads to the highest-scoring subtree. This generates memory- -o-oory-t---xx-stattee pairs (MATA- SFT-90K) for LLM supervised finetuning (SFT), shown in Figure (c).
The contributions of our paper are:
A hierarchical deterministic finite-state automaton-based system, MATA, that unifies neuro- symbolic framework with collaborative and competitive multi-agent design for visual reasoning.
Proposing (i) a learnable mechanism that trains a hyper agent as the transition policy of the hyper automaton over collaborative and competitive agents; (ii) a transition-traeectunn data generation pipeline and the dataset, MATA-SFT-90K, for supervised finetuning (SFT) of the hyper agent.
Comprehensive experiments across visual-reasoning benchmarks, with extensive ablations and analysis.
2 RELATED WORKS
Monolithic vision-language models (VLM) map images and text directly to answers with sin- gle forward pass (Xiao et al. 2024; Liu al. 2023b Li et al. 2023a by Wu al. 2023 Stanie et al. 2024 Zhu et al. (023). While these models have strong perceptual capabilities, their implicit reasoning processes are hard to explain and often degrade on queries requiring spatial relations, counting, or multi-step reasoning (Jahangard et al. 2024 2025). This motivates modular designs that expose intermediate, explainable symbolic processes (Andreas et al. 2016 Hsu et al.) 2023). Compositional methods decompose task into multiple stages (Ke al. (025b), often by having an LLM generate grounded actions (c.g.. programs or JSON) executed by tools (Gupta & Kembhavij
2023 Suris et al.] 2023 Shen al. 2023 Lu ct al. 2023). These pipelines improve interpretability and enable external tools use, but usually operate in a single forward pass with a fixed manually designed pipeline. They thus lack a flexible mechanism to engage in multi-step reasoning from feedback.
Recent works (You et al. 2023 Ke et al. 2024 Gao et al. 2024 Zhong et al. (025 explore agentic systems where an LLM/VLM VLM reasons in multiple steps and calls tools (Ke et al. (0255). However, most agentic approaches visual reasoming remain single-agent. In broader domains, multi-agent frameworks assign disjoint roles and connect them with hand-crafted collaboration patterns (Hong
2023 Li al. 2024 Nguyen et al.) 2025; Zhang al.) (025). achieving better performance al. in reasoning. However, this idea is still under-explone for visual reasoning. Notably, noise from perception and LLM/VLM hallucinations can accumulate across steps (Ke et al. (222a) from the collaborating pipelines, and most designs overlook competition between functionally overlapping agents (Wang et al., 2025c). This lack of learned transition policy limits flexibility and robustness on complex and diverse queries.
Finite-state automata as abstractions provide explicit control flow and interpretability. NAVER in- troduces probabilistic logic inside an automaton and equips modules with self-correction (Cai al. 2025), but relies on a hand-crafted transition policy that hard to manually define as states grow. HYDRA introduces
agent that includes a planner, RL controller, and a code-executing rea- soner (e et al. 2024). While data-driven, it still focuses on instruction-level planning without a learned, high-level policy for switching across qualitatively different agents on demand. By contrast, we propose MATA that explicitly learns the inter-agent transition function over a hyper-automaton whose states are agents, while keeping intra-agent nicro-steps rule-based. This learned transition function enables collaboration and competition among overlapping experts and transfers across dif- ferent domains and tasks (section 4.2), which previous visual reasoning methods with hand-written transitions or single-agent controllers do not address. States are agents; each agent runs a small, rule-based sub-automaton for reliable micro-control, while a trainable hyper agent learns cross-agent transitions over shared memory. This hierarchical view retains the interpretability of explicit state machines, avoids hand-coded transitions, and supports both collaboration and competition. Unlike prior work (Ke et al. 2024 Cai et al.) 22025, our controller is supervised -taitraidd from transition- trajectory data to transit between agents and to report a final result only when it is certain of the answer, directly addressing the gap identified above.
3

---

Published as a conference paper at ICLR 2026
Hyper Automaton (Agents)
Stepwise Reasoner
Transti
Hyper Agent
Stepwise Reasoner
inrttruction Gonerater
Inatuucionn Verifur
Godee Uanurator
Code Verffar
Cook Integposte
Hyper Agenit
Shared Memory
Bettur orged
Specialized Agent
Reasoner
by agents
Memooyy
LLM Statl
Trarsatoon
Corntroller
Perception Recalts
> Normal Tranation
Specializeg-AKgen
Omeshott Reastoeer
Tromaation
Histery
>
Verifier Foodbacks
Expert Model
Predicton
Initial
Fallurr
Task Metadata
Generalor
Vin
Intergrattr
Figure 2: Pipeline of MATA. A trainable hyper agent reads snapshot of the shared memory, pre- dicts the next state with an LLM State Controller. Its decision (blue arrows) routes control among agent states in the hyper automaton: Oneshot Reasoner, Stepwise Reasoner, and Specialized Agent. Each agent runs a rule-based sub-automaton that iterates until return to the hyper automaton. All agents read/write
append-only Shared Memory, enabling the hyper agent to access the current context for choosing the optimal next state. Lifecycle states INITIAL and FAILURE are shown out- side the agents (see subsection 3.2 for details).
METHODOLOGY
We explore multi-agent visual reasoning by learning a high-level transition function over agents within hierarchical automaton, enabling data-driven collaboration and competition among over- lapping skills and replacing inflexible hand-written pipelines.
3.1 OVERVIEW
visual reasoning instance is image-query pair (v, q) mapped to an output y (Ke al. 2025b). MATA organizes inference as a hierarchical automaton operated by a trainable hyper-agent. Infor- mally, the hyper automaton Ma a top-level automaton whose states include a set of sub-agents, with each sub-agent running a small rule-based sub-automaton, and the trainable hyper agent con- trolling the learned transition 5g. Formally, it can be described as Mealy machine (Mealy) 1955): Mo (S, Sp. Σ, A, do, r) where S denotes the set of states (containing both agent states for task execution and lifecycle states for process coordination). So the initial state where reasoning begins, Σ the inputs drawn from shared-memory snapshots (storing intermediate results from agents), A the answer space visual reasoning queries (e.g., discrete labels, bounding box coordinates, or free-text responses). da the learned transition function that determines the next state based on the current state and memory inputs, and I' the output function that generates the final answer ý once the automaton reaches terminal state. Detailed breakdowns of the states, transition mechanics, and output generation process are provided the subsequent sections (Figure
3.2 HYPER AUTOMATON
States. The finite state the union of agent states and lifecyele states: S Sagent Sure where Sagent = [ONESHO,, STEPWISE, SPECIALIZED), Site (INITIAL, FINAL, FAILURE] and the initial state Sg INITIAL. Agent states invoke concrete skills; lifecycle states orchestrate the progression and termination of the reasoning episode (c.g.. starting the task, handling uncertainty, concluding with an answer). Details of the states are shown in Table
Agents in our system are intentionally both collaborative and competitive. When control transitions from one agent to another, the successor agent reads the shared memory containing the prior history and feedback, and builds that context; this is collaboration. At the same time, multiple agents may attempt the same task; one agent stalls or fails, another can take over and complete it; this is competition. The learned transition policy dn selects among them based context (e.g., ONESHOT
4

---

Published as a conference paper at ICLR 2026
Table 1: States of the hyper automaton. The table specifies the description and the triggering condition for each state. óg: transition function of hyper automaton.
State
Description
Selected by
INITIAL
The unique state where reasoning begins.
Initial state
ONESHOT
workflow agent that executes single-pass program generation and exe- cution workflow for solvable queries, equipped with a lightweight verifier. A stepwise reasoner that produces step-wise Python programs for complex queries; code is verified and exccuted in sandboxed environment ensure correctness.
STEPWISE
80
SPECIALIZED
An expert agent that performs fast perception tasks; built-in verifiers vali- date outputs and adapt parameters.
FINAL
A terminal state in which sufficient evidence has accumulated; the output function I' is invoked when in this state to produce the final answer y). state triggered by unrecoverable errors excceding itcration limit.
FAILURE
Error occurs
STEPWISE for moderately compositional VQA; SPECIALIZED ONESHOT for grounding with simple perception). This overlap is intentional, as the three agents represent spectrum: perception (system 1), one-shot reasoning (fast thinking), and stepwise reasoning (slow thinking). Although all agents can answer all queries, each agent has different advantages and disadvantages, enabling hyper agent choose the optimal transition and re-route failure. The implementation details of agents are shown the supplementary material (Appendix B).
Shared Memory. All agents read from and write structured shared memory my the 1-th step that accumulates intermediate variables, perception results, program history, verifier feedback, and task metadata. We keep the formalism minimal: when an agent runs for one cycle, it appends its new memory Am, and m4+1 auditable and visible to the hyper agent.
U Am, Memory is append-only so the full reasoning process
Execution Step. At step the system is in (5t, m.). The hyper-agent observes the memory m, and selects the next state 81+1 via the learned transition function óø:
St+1 Ög(3;, mc), St+1 E S.
If 8t+11 Sugent, the corresponding agent executes its rule-based sub-automaton until returning to the hyper automaton and updating the memory; 8 = FINAL or t > T where T is the max step limit, the episode terminates.
The answer space A contains the required output ý for visual reasoning. For example, A {y y is text for VQA, bounding box for VG, etc). The output function N extracts the output from the memory my at FINAL state: ÿ = I'FINAL,
Output.
3.3 TRAINABLE TRANSITION FUNCTION (HYPER AGENT)
The transition function 6g in Equation implemented by a trainable LLM-based hyper agent Fe- This agent acts as the state-transition controller, selecting the next state st+1 from limited set of available candidate states. Since the I.IM requires textual input, we derive a prompt st from the shared memory The template for constructing from is shown below:
Prompt 3.1: LLM State Controller in Hyper Agent
You Al assistant control the state of multi-step visual reasoning system. Your task is to decide the next state the system should transition based on the current state and history. Dessrpeecription>> taakk title) [task desscripioon//Taskeeeeeriii Descpppisri <Query> (query) </Query>
<FFedback>> feedback jeodbback//Feebaakkk |</Feedback>
<Code>(code >/CCod>>
<Variables>> /
<Statellistor>> histthhisttttttto__ (SStteeestysttttessstoyy
5

---

Published as a conference paper at ICLR 2026
<State>(state) [state]</Staaee
<CurrentStep> (currett (rareettsstee/C/Cuieeee (/erre/tCeestteeps
Based the information ahove, determine the next state the system should transition Choose from the following states:
<StateCandidates>> [next state candidates </StaaeCandidates>>>
Return the name wrapped in <NextState> tags.
Our hyper agent Fø maps the prompt 24 to a distribution over the available states, from which 8(+1 is selected, either through greedy decoding or stochastic sampling.
The parameter of the hyper agent is supervised finetuned (SFT) on our collected transition tra- jectory dataset DD (subsectton 3.4p. Each training example provides textual memory 2t as prompt and target next state chosen by scanning branches in the trajectory tree that lead to successful and higher final scores:
0 t arg arg min (sFr
(2)
This objective guides the hyper agent on how to switch between sub-agents, and finalize the output.
3.4 DATASET GENERATION
Learning the transition policy of the hyper automaton requires examples of how agent states interact during visual reasoning. We therefore build dataset of transition trajectories. We regard the set of possible transition trajectories from an initial state as a trajectory tree Tuu: q) (Kearns et al. 2002) that records, for each node: the state history, intermediate reasoning outcomes, and final metric scores, as a textual prompt 24 based on [prompt 3.1] We collect this data by running MATA while systematically traversing each next-state option rather than committing a single path. Unlike end- to-end LLM/VLM training, this procedure explicitly explores the space of possible agent states and yields labeled decisions for our model.
Concretely, we sample images and queries from the training splits of GQA Hudson & Manning 2019) OK-VQA (Marino al., 2019), and ReTCOCCORe/CCCOO-RRIICC.
(Kzzemzadeh al. 2014 and run the hyper automaton Mo step-wise. Rather than limiting to single route, we expand bounded trajectory tree to depth T: at each node (state) the controller branches over the possible next states 81+1 E S, executes the corresponding sub-automaton, and saves a memory checkpoint Me+1. When a terminal state is reached (e.g., FINAL), which by construction corresponds to a leaf of the tree T, the output function I produces prediction ý for the given image-query pair (v, g) with ground truth y. We then compute a scalar task score for that Ieaf: for VG we use IoU(Ø, y); for VQA we use Acc(ý, y). During data collection perform near-exhauttive expansion of the transition tree to a fixed depth, which is tractable with the current three agents but, we acknowledge, grows rapidly more agents/states are added.
Bottom-up node scoring. As a result, each leaf node s E Leaves (T) is associated with a predic- tion ü. and ground truth y. from which we compute scalar score. We assign values to all nodes by propagating these scores upward from the leaves:
V(s)= metric(fe, v), maxx<Chhid(s) V(&),, otherwise.
8 E Leaves(T),
To train the LLM state controller, we convert each multi cuuti- transition into supervised examples. For every decision point state 8, with corresponding textual prompt 2t, we determine the optimal next state of by selecting the child node that leads to the subtree with the highest propagated value. Formally, for a state s; with its set of next states Child(st) S S, we choose:
of arg
max aEChild(s))
V(s).
The chosen state st becomes the label for the corresponding node prompt vt, and together they form a training example. Repeating this over all decision points produces a dataset of message histories paired with optimal next states, D = [2,, st) into instruction-co cortuctiooncomlltiion pairs suitable for supervised finetuning of LLM. Training on this dataset enables the model to learn how to control the transitions of hyper automaton. In total, we build
";))," Finally, reformat the collected examples

---

Published as a conference paper at ICLR 2026
the SFT dataset MATA-SFT-90K containing N 90,854 examples. We show the data example in Appendix H
3.5 INFERENCE
Given an image-query pair (v.q). q). we initialize the shared memory my and enter the initial state So = INITIAL. At step t, the hyper agent Fo reads the current context 2; and selects the next state 81+1 using the learned transition Equation I If 8t+11 E Sigents the corresponding sub-agent executes one cycle of its ule-based sub-automaton, appends its intermediate result to memory, and returns to the hyper automaton. If 8c+1 = FAILURE, this state indicates that the selected agent st reports unrecoverable error and the system will invoke the hyper agent to choose new state st+1 while temporarily removing the failed agent 8 from the state candidates to avoid infinite retries. If $141 FINAL the step t exceeds the limit T, the system terminates and returns the final result ý-
4 EXPERIMENTS AND RESULTS
Implementation Details. We implement MATA in PyTorch (Paszke al. 2019) and conduct all experiments on 4 RTX 4090 GPUs. The system uses interchangeable foundation models; unless otherwise stated we adopt V1.25 (8B) (Chen et al., 2025) as the VLM, Florence2-L (Xiao iet al., 2024) for object detection, DepthAnything V2 (Yang et al. 2024) for depth, and a Qwen3 (4B) Yang al., 2025) LLM for the trainable state controller in the hyper agent. The LLM is supervised finetuned on MATA-SFT-90K using AdamW, cosine decay with 5% warm-up. global batch size 64, for 8 epochs; decoding is guided at inference to ensure the output format. As MATA- SFT-90K is dataset collected by running our pipeline on multiple source datasets, "training on dataset X" means training on the subset of MATA-SFT-90K whose trajectories were generated from the training split of X. We use three SFT configurations for the hyper agent: (i) domain-specific: trained on the training split of the target dataset and evaluated on its test split; (ii) domain I trained on the dataset which is not the target dataset for evaluation; and (iii) general: trained jointly on the whole dataset. We follow the official splits of all the benchmark datasets, reporting accuracy. For faimess, when comparing with compositional baselines we keep the same foundation models, and for monolithic models we use the available public checkpoints with their official code. In the inference, we limit the max step of MATA T = 15 to avoid infinite running. The prompt template shown in the (Apenddx G in supplementary material.
Evaluation Protocol. We evaluate GQA (Hudson & Manning 2019). OK-VQA (Marino et al. ReCCoCoOReCCCCCOHRCCCCCC
2019,
(Kazemzadeh et al., 2014), and Ref-Adv (Akula et al.
2020
following the previous works (Surís et al. 2023] Ke et al.] 2024 Cai al.
2025). with
accuracy as the metric. We compare against the previous compositional methods which are training- required (Khan et al., 2024 Ke al. (025a) or training-free (Surís et al. 2023 Ke et al. 2024 Cai
let al.
2025), and mono lithic methods (Li et al.
2023b; Zhu et al. 2023 Liu et al.] 2023a
Su et al.;
2023
Han et al. 2023 Dai et al. 2023 Li al. [023a Wang al. 2024 Bai et al.] 2025b Chen
et al.
2025 /Zhu et al. 2025 Wang et al. 2025b OpenAl 2024 Tiong et al. 2022
Yang et al.
2022
Alayrac et al. 2022).
4.1
QUANTITATIVE RESULTS
Compositional Image Question Answering. On GQA (Hudson & Manning 2019). which em- phasizes complex compositional reasoning over spatial relations and attributes, MATA reaches 64.9% accuracy TTble 2). surpassing previous trainable compositional methods HYDRA and Vis- Rep, training-free baselines such as ViperGPT. It is also competitive with strong monolithic VLMs, exceeding Intern and Qwen2.5-VL. The gains stem from the learned transition policy, and the hyper agent understands the capacity of agents. Easy queries invoke SPECIALIZED perception first and escalate to ONESHOT or STEPWISE only on failure or low confidence, whereas difficult cases route directly to STEPWISE to maximize the reasoning. When the range of data is narrow and distinctive, the domain-specifie setting can calibrate priors more precisely; when compositional
Our domain--rannfal term is scoped to the hyper agent: it is trained on non-test-dataset transition trajecto- ries, and never sees the optimal trajectories in other datasets.
7

---

Published as a conference paper at ICLR 2026
Table Performance on GQA dataset.
Table 3: Performance on OK-VQA dataset.
Type
Method
Acc.
Type
Method
Acc.
BLIP-2
2023b
45.3
PNP-VQA [Tiong
2022
35.9
(IniGP Zhu 2023
30.8
PICa (Yang al
2022
43.3
LLaVA (13B) Liu et ai 2025a
41.3
BLIP-Z al
2023b
45.9
PandaGPT (13B) ctall 2024
41.6
Flamingo (9B) AAlayrac et al.
2022
44.7
Monolithic
InstructBLIP (13B) (Dai et al. 2023
ImageBind-LLMM (7B) [Ilam el ml
2023
49.5
41.2
Monolithic
MiniGPT-4 (133) (Zhu
LLaVA (13B) clall 202.0
2023
42.5
37.5
Otter (7B) ct
50.0
InstructBLIP (13B) (Da
2023
47.9
Owen2-VL((BB) ( Wang
2024
34.3
Qwen2-VL (7B) (Wang
2024
28.3
Qwen2.5-VL (7B) [5a
2025b
62,4
Qwen2.5-VL (7BT [BBt a.
202.b
71.8
Qwen2-VI. (4B) (Bai el 2025a
51.6
Qwen3-VT. (4B) (Bai et 212ha
44.4
InlemVL2.55 (8B) (Chen
2025
61.5
InternVL2.5 (8B) CChen ml
2022
75.2
I3 (8B) (hu al 2025
62.4
InternVL3 (8B) al 2025
74.7
Intemm1.33.5 (8B) [Wang
2125b
63.8
InternV1.335 (8B) Wang 2025h
75.7
GPT-40-2024-05-
OpenA)
58.5
GPT-40-2024400--1
OpenA/
33.4
IdeaIGPT You
2023
41.7
IdealGPT You
2023
19.4
Compositional
ViperGPT (Suris
20223
37.9
Compositional
ViperGPT (Suris al.
2023
40.7
VisRep Khan a.
2424
51,4
VisRep (Khun et at, 2074
46.7
HYDRA [Ke et 2024
52.8
MYDRAA(KE al 2024
59.4
MATA (Ours) (General)
64.9
DWIM (Ke al. 20725a
62.8
MATA (Ours) -oomain-Speciffe
64.7
(Ours)(Generaal)
MATA (Ours) (Domain-Spccific)
76.5
Agentic types: not-ageeticchhon-speciieee O single-agent; multi-agent.
Table 4: Quantitative comparison (aceuracy) on referring expression comprehension task on
RefCOCO, RefCOCO+, RefCOCOg (Kazemzadeh et al.) (014) and Ref-Adv (Akula et al., 2020 set. Note there is no training set Ref-Adv, so scores are domain-transfer.
Type
Method
RefCOCO
RefCOCO+
RefCOCOg
Ref-Adv
GLIP-L (Li et
2022
55.0
51.1
54.6
55.7
KOSMOS Peng et al. 2023
57.4
50.7
61.7
heng
2024
12.1
12.1
32.9
32,2
Monolithic
YOLO- World- (Cheng et al
2024
16.8
36.5
33.1
Grounding(INO-TT [iu al.
2023b
61.6
59.7
60.6
60.5
GroundingDINN-BB (Liu al.
202.5
90.8
84.6
80.3
78.0
SimVG Daii el al. 2024
94.9
91.0
88.9
74.4
Florence2-B Xiao et al.
2024
94.5
91.2
88.3
72.2
Florence2-L Xiao a.
2024
95.1
92.5
90.9
71.8
GPT--4o 15- OpenAA]
2024
30.5
26.2
Qwen2.5-VL-72B (Bai et al.
2025b
94.6
92.2
90.3
Code-bison (Stanic
22024
44.4
38.2
ViperGPT (Suns et |
68.6
73.8
68.7
58.2
Compositional
VisRep (Khan el al. 2024
55.2
51.1
HYDRA (Ke ct al. 2024
65.7
66.2
59.9
48.3
DWIM (Ke et al., 2025a
82.7
74.2
NAVER (Car al. 202S
96.2
92.8
91.6
75.4
MATA (Ours) (General)
96.3
93.8
90.7
77.3
MATA (Ours) (Domain-Specific)
96.3
93.9
90.8
Agentic types: mon-agenticcnonn-speiliee O single-agent; multi-agent.
patterns shared across sources, joint training (general) regularizes transitions and reduces over- fitting. GQA we observe the latter, many patterns appear across sources in MATA-SFT-90K, so the general setting achieves better performance.
External wewwledge-Dependenn Image Question Answering. On OK-VQA Marino et al. 2019). which requires external knowledge, MATA achieves 76.5 % accuracy Table 3), surpass- ing prior compositional systems such as DWIM (62.8%) and HYDRA (59.4%), respectively, and outperforming recent monolithic VLMs including Qwen2.5-VL VL (71.8%) and Intern (75.7%). Gains come from the learned hyper agent transition: for easy queries the hyper agent first invokes SPECIAL.IZED perception and escalates the STEPWISE ONESHOT reasoner only failure
8

---

Published as a conference paper at ICLR 2026
Table Ablation of hyper agent. In this table, we report the accuracy for all VQA and referring ex- pression comprehension benchmarks, and the inference time per query (tested on GQA). HA: Hyper Automaton. Transition: Transition policy (oo). SFT: Supervised finetuning. Refer to subsection 4.2
for details.
Components
Accuracy (*)
Time (4)
HA
Transition
SFT
GQA
OK-VQA
RefCOCO
RefCOCO+
ReCCOCOg
Ref-Adv
Avg Sec.
Exhaustive
57.7
71.3
87.7
85.6
81.7
73.1
34.58
Random
57.1
71.1
83.8
81.1
73.2
6.91
LLM
58.5
75.1
93.5
88.0
76.0
LLM
64.9
76.5
96.3
93,9
90.8
77.3
8.01
Table Generalizability results. The top-left header cell uses diagonal split indicate Training Data (rows, 1) versus Test Data (columns, ->). Diagonal values ddomain-specifee train and test on the same dataset; off- aff-diagonal values evaluate cross-domain/task transfer last row reports joint training on the whole MATA-SFT- dataset (general). Off-diagonal values are close to the diagonal ones, indicating strong generalizability of the learned transition policy.
The
VQA
Visual Grounding RefCOCO+
Test
Training
GQA
OK-VQA
RefCOCO
RefCOCOg
Ref-Adv
GQA
64.7
75.8
96.1
93.7
90.4
77.0
OK-VQA
64.1
76.5
96.2
93.8
90.5
76.9
RefCOCO
63.8
75.5
96.3
93.9
90.8
77.2
RefCOCO+
63.6
75.4
96.2
93.9
90.7
77.1
RefCOCOg
63.1
75.4
96.1
93.7
90.8
77.2
All
64.9
76.0
96.3
93.8
90.7
77.3
low confidence; for difficult queries directly selects STEPWISE for multi multi-step reasoning, with com- petitive re-entry into SPECIALIZED ONESHOT to reason combining the previous findings and new evidence. We observe the domain-speciffe setting holds small edge, likely because of the narrow diversity of the reasoning pattern required in the dataset, whereas joint training (general) slightly dilutes these knowledge.
Referring Expression Comprehension. On popular benchmarks RefCOCO, RefCOCO+, Ref- COCOg Kazzmzzaeh et al. 2014) and Ref-Adv (Akula et 2020), MATA obtains state-of-the-art these datasets, exceeding strong mono-
performance (Table 4). It sets a new state-of-the-aat lithic and compositional baselines. Notably, Ref-Adv only contains a test set, which means the MATA-SFT-90K does contain the data collected from it, showing promising domain-transke generalizability of MATA. Note that due to learned transition, short simple queries are solved by SPecIALIZED perception with verification, while complex cases trigger STEPWISE and ONESHOt reasoning. comain-specific SFT slightly stronger because the language query styles is dataset- specific.
4.2 ABLATION STUDIES
Hyper Agent. Table isolates the main contribution of the trainable hyper agent and the hierar- chical automaton design. We compare: (1) Exhaustive Ensemble without hierarchical automaton (HA): exhaustively call all sub-agents and aggregate with a VLM: (2) Random Transition: HA enabled but the next state is chosen randomly; (3) LLM without SFT: a pretrained LLM is used as the state controller (no supervised finetuning): (4) LLM SFT: supervised finetuned LLM con- woperraaeeee
trols transitions. Both the exhaustive baseline and random transition yield the weakest performance, but introducing the hyper automaton already cuts runtime significantly. Replacing random with a pretrained LLM in hyper agent improves accuracy across tasks. This suggests that (i) the hyper automaton and the LLM primarily drive effective multi-agent collaboration and competition and (ii) SFT further helps the understanding of the capacity of agents in different types of questions.

---

Published as a conference paper at ICLR 2026
GQA: Accuracy vs. Size
OKVQA: Accuracy vs. Size
GQA Accuracy vs. # #Agents
0.65
0.66
0.75
GQA Accuracy
0.60
Accuracy
Accuracy
0.64
Type
Type
Without
Without SFT
0.50
SFT GQA
0.65
SFT OK-VQA
0.62
0.45
SFT All
SFT A
8.0
Sizee (B)
Size (B)
Number of Agents
Figure 3: Results of different LLM sizes. Accuracy versus the model size (in billions of parameters) of the hyper agent's LLM state controller. Left: GQA; right: OK-VQA. X-axis: LLM size; Y-axis: accuracy.
Figure 4: Results of different numbers of sub-agents. X- axis: number of sub-agents; Y- axis: accuracy in GQA.
Generalizability, We conduct generalization analysis by training the hyper agent on GQA subset only of MATA-SFT-90E dataset, OK- VQA subset only, or the whole dataset. [aable 6 organizes re- sults by different training/evaluation types: domain-speciffe domain-transfer, and general. domain- transfer performance is strong in both directions (GQA-OK- than 1% difference. The model trained on all data reaches similar performance to the model trained on the corresponding subset only, indicating the controller learns task-agnostic transition policy with minimal negative impact. We further discuss the effects the next paragraph.
OK-VQA-CGI
with less
LLM Size. Figure: compares the sizes of the LLM state controller from 0.6B to 8B under three settings: (i) no SFT. (ii) domain-specific SFT, and (iii) SFT all. With domain-specific SFT, even small models (0.6B/1.7B) perform competitively matching 4B and 8B. When finetuned jointly on all data, small models worse than 4B/8B by few percentage points, indicating limited capacity to absorb cross-task knowledge. Without SFT, accuracy drops sharply for smaller models and improves mainly with size. Balancing accuracy and efficiency, choose 4B as default, it produces near- optimal results with substantially lower memory, while larger models yielding only marginal gains.
Number of Agents. We ablate the number of agent states to quantify benefits beyond our 3-agent design. On GQA, single Specialized agent reaches 61.5%, adding the Oneshot reasoner lifts accu- racy to 64.5%, and adding the Stepwise reasoner yields a marginal further gain to 64.9% (Figure 4). The small improvement from 2 to agents indicates diminishing improvements on current bench- marks, suggesting that the agent count is not the major factor. We therefore use three agents in MATA.
More Analysis. We discuss more analysis for generalizability in Appendix Cl hyper agent in Ap- pendix DJ efficiency in Appendix E] comparison with direct SFT in Appendix and the qualitative examples in in[pppendix Iiin supplementary materials.
CONCLUSION
We present MATA, a visual reasoning method that uses a trainable hyper agent to learn the transition policy of hierarchical finite-state automaton. By transitioning between agents based on shared memory, the system reduces hallucinations, and preserves explainability through explicit states and context. To supervise the hyper agent, we introduced the transition- teajenttorrj dataset MATA-SFT. 90K, which converts the trajectory data into a standard SFT format and adapts as agents are added. From experiments, MATA achieves state-of-the-an performance across multiple datasets. Limita- tions. The data generation pipeline performs a near-exh transition search over the state space; this is tractable with the current three agents but may become costly as the number of states grows.
ACKNOWLEDGMENTS
This research is sponsored by the DARPA Assured Neuro Symbolic Learning and Reasoning (ANSR) program under award number FA8750-23-2-1016.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Arjun Akula, Spandana Gella, Yaser Al-Onaizan, Song-Chun Zhu, and Siva Reddy. Words Aren't t Enough, Their Order Matters: On the Robustness of Grounding Visual Referring Expressions. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6535-6565, Online, July 2020. Association for Computational Linguistics. doi: 12020000////2222..mmmmamam http://c.ooooo.oo.oo.....aaaa--.
URL
586/
lean-Baptise Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc. Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Ruther- ford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Ja- cob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikołaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: a Visual Language Model for Few-Shot Learning. In Advances in Newral Infor- mation Processing Systems, volume 35, pp. 23716-23736. Curran Associates, Inc., Decem- ber 2022. URL wttp/httsss///pppreeedddooooooea neuripsssssneueeeoer.o...ms http:///........oaaoaaa/a///pp/p Cone-rooooeeeeee htm_
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural Module Networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 39- 48, 2016. URL https: http::/:////ppaccccseeee twwcth..cccoooooo com
wttpcvv...c///oeentttt.. content, _vppr_cvpr_______________ 2016/htm1/
Andreas Neura Module www__Netwwk_______
html
Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan Liu, Dunjie Lu, Ruilin Luo, Chenxu Lv, Rui Men, Lingchen Meng, Xuancheng Ren, Xingzhang Ren, Sibo Song, Yuchong Sun, Jun Tang, Jianhong Tu, Jianqiang Wan, Peng Wang. Pengfei Wang. Qi- uyue Wang, Yuxuan Wang. Tianbao Xie. Yiheng Xu, Haiyang Xu, Jin Xu, Zhibo Yang, Mingkun Yang, Jianxin Yang, An Yang, Bowen Yu, Fei Zhang, Hang Zhang, Xi Zhang, Bo Zheng, Humen Zhong, Jingren Zhou, Fan Zhou, Jing Zhou, Yuanzhi Zhu, and Ke Zhu. Qwen3-VL Technical Re- port, November 2025a. URL hhttp://arxiv /wtpvvaaavvvv/.a///////../.a.aa. wrggabs/... 2511 21631 arXiv:2511.2166 [es].
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang. Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-VL Technical Report, Febru- MtXiv:2502.1392. [cs].
ary 2025b. URL
org/ abs / 220225522222 13923
Zhixi Cai, Fucai Ke, Simindokht Jahangard, Maria Garcia de la Banda, Reza Haffari, Pe- ter Stuckey, and Hamid Rezatofighi. NAVER: Neuro-Symbolic Compositional Au- tomaton for Visual Grounding with Explicit Logic Reasoning. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 24078 24089, 2025. URL https: ftttp::/ccpeaaceese thecwf com/ content wicpeettt//c/222ce httm//aa NAVER A Neuro- Stsebooli_oooppseaaaaeaeeeeeeaee
Barii__uuutt omaton for with Explicit ICCV 2025 www.ppppprr....
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qin- glong Zhang, Xizhou Zhu, Lewei Lu, Bin Li. Ping Luo. Tong Lu, Yu Qiao, and Jifeng Dai. VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recog- nition, pp. 24185 141185-24198, 2024. URL htll/attpp////eeacsess
thecvf..eec...cm com/ content/ http:/i__Saaaanggg___a_ inn_iinn_________ cncioooooooonttiiie ooteiiiooooeeeeeeeee
CVPR2024/ 2ttp2/024/hmmmmmm and A wiggginggg for wowtoroooeeeeeeeeeeee 20_____2R__________22___ htt.:/nnnoor..earrreo./.../..220
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu. Shen- glong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu. Xuchui Wang, Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, Conghui He, Botian Shi, Xingcheng
11

---

Published as a conference paper at ICLR 2026
Zhang, Han Lv, Yi Wang, Wenqi Shao, Pei Chu, Zhongying Tu, Tong He, Zhiyong Wu, Huipeng Deng, Jiaye Ge, Kai Chen, Kaipeng Zhang, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling, January 2025. URL http: org/
abspg//////// abs 2412. 05271 arXiv:2412.05271 [cs].
Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang. and Ying Shan. YOLO-Workes Real-Time Vhcn-Vocabulary Object Detection. In Proceedings of the IEEE/CVE Confer-
ence on Computer Vision and Puttern Recognition, pp. 16901-16911, 2024. URL https:
/popnaccess. theov com/ httm///ooneet///// Swtpteeet//VR2222///// Real - Time Oeenn www.aaaaaaaaaaaaaaaeaeaaaae ooteectt_eeeetiooonooooo CVPR 2024 pww.paaper.aa. html
YOLO- World
Ming Dai, Lingfeng Yang, Yihao Xu, Zhenhua Feng, and Wankou Yang. SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusion. In Ad-
vances in Neural Information Processing Systems, volume 37, pp. 121670-121698, Decem- wwtppreepppppppppepaaeeeeeeeeeee http:pppeerflleeepppe/////0 ber 2024. URL http:///......22222222222222535.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards General-purpose Vision- Language Models with Instruction Tuning, June 2023. URL http:///axxxvvorr//aa// arXiv:2305.06500 [cs].
2305. 06500
Yufan Dang, Chen Qian, Xueheng Luo, Jingru Fan, Zihao Xie, Ruijie Shi, Weize Chen, Cheng Yang, Xiaoyin Che, Ye Tian, Xuantang Xiong, Lei Han, Zhiyuan Liu, and Maosong Sun. Multi- Agent Collaboration via Evolving Orchestration, May 2025. URL http://arxiv.. arXiv:2505. [cs].
abs/2505. 19591
DeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, January 2025. URL URL lttt:://aaxixi org/ abs//501 12948 arXiv:2501.12948 [es].
Zhi Gao, Yuntao Du, Xintong Zhang, Xiaojian Ma, Wenjuan Han, Song-Chun Chun Zhu, and Qing Li. CLOVA: A Closed-LOop Visual Assistant with Tool Usage and Update. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13258-13268, 2024. URL https: https////pppccccss.oooo thecvs..teec..... com com/ http///ooeeett///eeeeeeee http//2/0////mmaaaaaaaaa / CLOVA A Closed- LOop Litio_VVisuaa Visual Assistant with Tool Usage and Update
CVPR hasll_apperr_hhh html
Gemini-Team. Gemini: A Family of Highly Capable Multimodal Models, December 2023. URL hitt:://axivv wwt./oo///////oooo../.o.oo.. 2312 11805 arXiv:2312. [cs].
Tanmay Gupta and Aniruddha Kembhavi. Visual Programming:
Compositional Visual Reasoning Without Training. In Proceedings of the IEEE/CVF Conference Computer Vision and
Panten Recognition, pp. 14953-14962, 2023. URL htttttps://ppeeacccesstt www.:fccccssttcccccccc. com/ Coottent/CVER2223//et/ http://ttmmmmttaaaaaeaaaaaaaaaaa Visual www.orooooongg...oo.....r......e htt.r/ammmiinnnomoooiiiiiiooooao 1000233ppprrhtmrr
Visual Reasoning hatp:oonnnnniiihhttttTaaaaainnni
Jaaming Han, Renrui Zhang, Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu, Song Wen, Ziyu Guo, Xudong Lu, Shuai Ren, Yafei Wen, Xiaoxin Chen, Xiangyu Yue, Hongsheng Li, and Yu Qiao. ImageBind-LLM:: Multi-modality Instruction Tuning, September 2023. URL latt:://aat wttpv//.rgg/b/////////////////// C3905 arXiv:2309.03900 [cs].
Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. MetaGPT: Meta Programming for A Multi- Agent Col- laborative Framework. In The Twelfth International Conference on Learning Representattinss arXiv, November 2023. doi: 20.485500arXiv.2338..00 000552. URI. httpt:://///......... /ttp://///eeeeeeeeeeeeeee
Joy Hsu, Jiayuan Mao, Joshua B. Tenenbaum, and Jiajun Wu. What's Left? concept grounding with logic-enhanced foundation models. In Proceedings of the 37th International Conference on
12

---

Published as a conference paper at ICLR 2026
Neural Information Processing Systems, NIPS '23, pp. pp. Curran Associates Inc.
Red Hook, NY, USA, 2023.
Drew A. Hudson and Christopher D. Manning. GQA: A New Dataset for Real-World Vi- sual Reasoning and Compositional Question Answering. In Proceedings of the IEEE/CVF
Conference Computer Vision and Pattem Recognition, pp. 6700 2019. URL
htt.c////ooottttnnaeeeee thecvf...eeee........ com/ cttt:/tooeetttttoco CVR___CVRRR______ CC (ttR_2009/httmm/dddeseees _QA_
A NNw__aataset_foo for Rea Wor 1 Visual Reason i ng_ and totprrooonnnnnooooooooooooooooo
CVPR 2019 paper. html
Simindokht Jahangard, Zhixi Cai, Shiki Wen, and Hamid Rezatofighi. - A Multifaceted Robotic Dataset for Understanding of Context and Dynamics of Human Interactions Within Social
Groups. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 22087-22097, 2024. URL https: /ttp::::/ppeaccccsssse
thecvf. com/ cont
CVPR2024/5mmm/aaaanggee
JRDB-Social www.MMuifffftteeoo Robotic of Context and CVPR 2024 paper.htm..
Dataset for__oorrr__rr__ www.Onndeersanoo Cttpaaannnnnnnooooooootteatcattt
Simindokht Jahangard, Mehrzad Mohammadi, Yi Shen, Zhixi Cai, and Hamid Rezatofighi. JRDB- Reasoning: A Safficultt-Gradeer Benchmark for Visual Reasoning in Robotics, August 2025.
URL httt:://axiii/e org/ abs 2508. 10287 arXiv:2508. 11.225008..887777 [cs].
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. ReferltGame: Referring to Objects Photographs of Natural Scenes. In Alessandro Moschitti, Bo Pang, and Walter Daele- mans (eds.), Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNI.P), P) pp. 787-798, Doha, Qatar, October 2014. Association for Computational 1086
Linguisties. doi: 10.3115/w1/114-10000 1086. URL htt/l:/p:://allannooooooooo
Fucai Ke, Zhixi Cai, Simindokht Jahangard, Weiqing Wang, Pari Delir Haghighi, and Hamid Rezatofighi. HYDRA: A Hyper Agent for Dynamic Compositional Visual Reasoning. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol (eds.), Computer Vision ECCV 2024, pp. 149,149, Cham, 2024. Springer Nature Switzerland. ISBN 978-3-031-72661-3. doi: 10. 00000/978-3-031-26666
Fucai Ke, Vijay Kumar B. G, Xingjian Leng, Zhixi Cai, Zaid Khan, Weiqing Wang, Pari Delir Haghighi, Hamid Rezatofighi, and Manmohan Chandraker. DWIM: Towards Tool-aware Visual Reasoning via Discrepancy-aware Workflow Generation Instruct-Masking Tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3378-3389, 2025a. URL httts:://ppeacceeeeeeeeee
DWIM Towards Tool Visual Reasoning via Discrepancy-aware
Norteloow_Geeezatiinn ICCV 2025 11125_paper.htt.
Fucai Ke, Joy Hsu. Zhixi Cai, Zixian Ma. Xin Zheng. Xindi Wu, Sukai Huang, Weiging Wang, Pari Delir Haghighi, Gholamreza Haffari, Ranjay Krishna, Jiajun Wu, and Hamid Rezatofighi. Explain Before You Answer: A Survey on Compositional Visual Reasoning, August 2025b. URI. http::/arxiv. .rgg
wawog//b////ae/ abs 2508 17298 arXiv:2508.1 17298 [cs].
Michael Kearns, Yishay Mansour, and Andrew Y. Ng. A Sparse Sampling Algorithm for Near- Optimal Planning in Large Markov Decision Processes. Machine Learning, 1932):193-208, November 2002. ISSN 1573-0565. doi: 10.1023/A:101199242777 URL petttps://do.
org/10. 110.000013:::1193338180
Zaid Khan, Vijay Kumar Bg, Samuel Schulter, Yun Fu, and Manmohan Chandraker. Self- Training Large Language Models for Improved Visual Program Synthesis With Visual Reinforcement. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14344-14353, 2024. URL ltttps://
thecvf......... com/ Self-Training Large Language Models
Donttent//1P2222//tmmm/// Sttrrrrasssttttss___SSs s_tssssstttthhte__e__eee 2024 _aaper.ppe.._ htmm html
For_Improved__1 Www.aliwaaaa.... Proggrrrrrramm
Ao Li, Yuexiang Xie, Songze Li, Fugee Tsung, Bolin Ding, and Yaliang Li. Agent-Orienter Plan- ning in Multi-Agent Systems. In The Thirteenth International Conference on Learning Represen- fations, October 2024. htt.tt htttttttss///peeeeeee net. net/ffrumm? dtt.rrmmmmmi-FFllAA
13

---

Published as a conference paper at ICLR 2026
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A Multi-Modal Model with In-Context Instruction Tuning, May 2023a. URL http:://axxii.
org/ htg//bb//20 abs/2305. 2305. 03726 AxXiv:2305.03726 [cs].
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping Language- Image Pre-training with Frozen Image Encoders and Large Language Models. In Interna- tional Conference on Machine Learning. 2023b. doi: 10488500/ARXIV.3311.125 ptttps://arxiv..o www.w/orrg/////ooo//o
URL
abss 2301 2tt////l./22231.... 12597
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang. Kai-Wei Chang, and Jianfeng Gao. Grounded Ionguage-Image Pre-Training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10965-10975. 2022. URL hnttps://openaccese twwcwheeeeeeee wtt wwwp///oooteet////e/22222222 http:/tmtt////// /Ii Grounded_Langaat
Image_Pre
html
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning, April
2023a. htLLhtt:///aaee org/ http://bs//200000 08485 arXiv:2304.0445 0.8455 [cs].
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, and Lei Zhang. Grounding DINO: Marrying DINO with Grounded Pre- Pre-Training for Open-Set Object Detection, March 2023b. //tttt:::///rii org. brgg/bb.. /
1303.05499 arXiv:2303.05499 [cs].
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-Play Compositional Reasoning with Large Lan- guage Models. In Advances in Neural Information Processing Systems, volume 36, Decem- ber 2023. BtLphtts:: /ppppttse///poooednnnnsneeeeeee
fi http:///ppppeeeeeeepppep//////20 87 haap::22100011171000000002222ae0
html
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. OK-VQA: A
Visual Question Answering Benchmark Requiring External Knowledge. Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3195-3204. 2019. URL /tttss:/openaccestt
thecvf. hwtpe//ww..oooooeeeenetCCCC.
CVPR CttCCCVEER/_EEECEE 2019/html/Marla
OK VQA A_ Visuallllll..mmmmm Ltt. Bhwswering__Bnchhrrrkk_...
Know Knowledge_cvPP 2t____cc___11___p________
George H. Mealy. A method for synthesizing sequential circuits. The Bell System Technical Journal,
159 1955. ISSN 0005-8580. doi:
URL http:tttss//eeeeppphtet
Thang Nguyen, Peter Chin, and Yu-Wing Tai. MA-RAG: Multi-Agent Retrieval-Augmeenee Gener- ation via Collaborative Chain-of-Thought Reasoning, May 2025. URL httphhttt://axxx..rrr
abs/ / 2505 20096 arXiv:2505.20096 [cs].
OpenAl. GPT-4o System Card, October 2024.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL httpps://prrooeeiinnng http:p/eeeeeeeeeerrerrsceeaa/p/p
Zhiliang Peng. Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding Multimodal Large Language Models to the World,Jlyy 2023. URLhttp:
//axxiv.or//ab//23333 14824 2XXi:23306.1822 [cs].
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In Pro- ceedings of the 38th International Conference on Machine Learning, pp. 3748-8763 PMLR, July 2021. URL https///opoooooooon.ooo..r.rrrrs
ngs ml I pttss//v139//aaforrdb0 html
14

---

Published as a conference paper at ICLR 2026
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hug- gingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face, March 2023. URL htttp://axii. org/ abs 2tbs/2333..m .7580 2XXx:2303.175800 [es].
Aleksandar Stanić, Sergi Caelles, and Michael Tschannen. Towards Truly Zero-shot Composi- tional Visual Reasoning with LLMs as Programmers. Transactions on Machine Learning Re- search, January 2024. ISSN 2835-8856. URL http:::/ httpreewwwwwwweeeeeee.e feret/fooummii
WYGiqSVatK
Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. PandaGPT: One Model To Instruction-Follow Them All, May 2023. URL arXiv:2305.13355 [es].
Dídac Surís, Sachit Menon, and Carl Vondrick. ViperGPT: Visual Inference via Python Ex- ecution for Reasoning, March 2023. URL http://arxiv.orgg/ wttp://rggaaabb/....ooo
arXiv:2303.08 [es].
Anthony Meng Huat Tiong, Junnan Li, Boyang Li, Silvio Savarese, and Steven C.H. Hoi. Plug- and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Com- putational Linguistics: EMNLP 2022, pp. 951-967, Abu Dhabi, United Arab Emirates, Decem- ber 2022. Association for Computational Linguisties. doi: 10.18653/v 111116653/122222.0 1/2022 1/20222..n222....nn....... findinng-emmll.66. URL
org. 2022. windinggeeennnnpppp...... emnlp. 677
Ziyu Wan, Yunxiang Li, Xiaoyu Wen, Yan Song. Hanjing Wang, Linyi Yang, Mark Schmidt, Jun Wang, Weinan Zhang, Shuyue Hu, and Ying Wen. ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learning, May 2025. URL htttp://arxiii wwt.://gggbbb/a//eee..
09501 arXiv:2503.099 [cs].
Haotian Wang, Xiyuan Du, Weijiang Yu, Qianglong Chen, Kun Zhu, Zheng Chu, Lian Yan, and Yi Guan. Learning to break: Knowledge-enhanced reasoning in multi-agent debate system. Neurocomputing, 129063,, February 2025a. ISSN 0925-2312. doi: 10. n0..oom/ 2024. 129063. URL
5092523122401822 https::/ wwwp::/eeeeceeieieeee com/ http://cceenn//rriee//pl..
Peng Wang. Shuai Bai, Sinan Tan, Shijic Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-VL: Enhancing Vision-Language Model's Perception of the World Any Resolution, October 2024. URL httpp://axxiv.
org/abs/ 249.. 12191 arXiv:2409. 12191 [cs].
Weiyun Wang, Zhangwei Gao, Lixin Gu. Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, Zhaokai Wang. Zhe Chen, Hongjie Zhang, Gan- lin Yang, Haomin Wang, Qi Wei, Jinhui Yin, Wenhao Li, Erfei Cui, Guanzhou Chen, Zichen Ding, Changyao Tian, Zhenyu Wu, Jingjing Xie, Zehao Li, Bowen Yang. Yuchen Duan, Xue- hui Wang, Zhi Hou, Haoran Hao, Tianyi Zhang, Songze Li, Xiangyu Zhao, Haodong Duan, Nianchen Deng, Bin Fu, Yinan He, Yi Wang, Conghui He, Botian Shi, Junjun He. Yingtong Xiong, Han Lv, Lijun Wu, Wenqi Shao, Kaipeng Zhang, Huipeng Deng, Biqing Qi, Jiaye Ge, Qipeng Guo, Wenwei Zhang, Songyang Zhang, Maosong Cao, Junyao Lin, Kexian Tang. Jian- fei Gao, Haian Huang, Yuzhe Gu. Chengqi Lyu, Huanze Tang, Rui Wang, Haijun Lx, Wanli Ouyang, Limin Wang, Min Dou, Xizhou Zhu, Tong Lu, Dahua Lin, Jifeng Dai, Weijie Su, Bowen Zhou, Kai Chen, Yu Qiao, Wenhai Wang, and Gen Luo. Intern Advancing Open- Multimodal Models in Versatility, Reasoning, and Efficiency, August 2025b. URL http::/arxivv orgoorggg....... abs/ 2200 18265 arXiv:2508. 18265 [cs].
Zeqing Wang, Wentao Wan, Qiqing Lao, Runmeng Chen, Minjie Lang, Xiao Wang, Keze Wang, and Liang Lin. Towards Top-Down Reasoning: An Explainable Multi-Agent Approach for Vi- sual Question Answering, February 2025c. URL http: ptttt://arxiv.... www.a. 13331 arXiv:2311.1333 [es].
Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. NExT-GPT: Any-to- Any Multimodal LLM, September 2023. URL http: /tttpi//rxii...... hbc///bs/2250 o2g/abbs/309..5555 19Xiv:230.00559 [es].
15

---

Published as a conference paper at ICLR 2026
Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition, pp. 4818 4829, 2024. URL htt.ttts///ppeaacceeettt thw.wa.hee...ccc.m com/ wttpc////coootett/tte//tto html ao_ [aaml/xaaoFForrnccee Florence-222 20lorenne-2_Advaciinne Adwandvaaai.... eed__aneeeeeeee_eeeeeeeeeeeeeeee
Variety of Vision CVPR 2024
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng. Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge. Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang. Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng. Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 Technical Report, May 2025. URL http: http://arxvvvv... org hbtp://bb////a/.. arXiv:2505.099 009388 [cs].
Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth Anything V2, October 2024.
org/ abs/ 2406. 09414 14
Siii:220000091144
Zhengyuan Yang. Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An Empirical Study of GPT-3 for Few-Shot Knowledge-Bass
VQA. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 308 1038-3, June 2022. doi: 10.
1609/aaai, v36i3.20215. URL httth:://////// aaai. .rriiiorg//aa http:/ndexxxc// www.nndxx..pp//aaee php/AAAI///apppppppppppaaaaaII aatia//ratclle///o/
view/20213
Haoxuan You, Rui Sun, Zhecan Wang. Long Chen, Gengyu Wang, Hammad Ayyubi, Kai-Wei Chang, and Shih-Fu Chang. IdealGPT: Iteratively Decomposing Vision and Language Rea- soning via Large Language Models. In The 2023 Conference on Empirical Methods in Natu- ral Language Processing, December 2023. URL htttps://operrvviie net/ / forum?rmm? did= IwwevJHLpc
Shengbin Yue, Siyuan Wang, Wei Chen, Xuanjing Huang, and Zhongyu Wei. Synergistic Multi- Agent Framework with Trajectory Learning for Knowledge-Intensiee Tasks. Proceedings of the AAAI Conference on Arlificial Intelligence, volume 39, pp. 25796 25804, April 2025. doi: 10. (.09//aaii.33i24..444 view/34772
/otttps://ojs.aaall.. www.oorgggnnnnne. php/ article/
Wentao Zhang, Liang Zeng, Yuzhen Xiao, Yongcong Li, Ce Cui, Yilei Zhao, Rui Hu, Yang Liu, Yahui Zhou, and Bo An. AgentOrchestra: A Hierarchical Multi-Agent Framework for General- Purpose Task Solving, August 2025. URL http:://axx: wttp:/./////....o..
http::////aaattoooo//////////000 12508
arXiv:2506. 12508 [cs].
Yaoyao Zhong, Mengshi Qi, Rui Wang, Yuhan Qiu, Yang Zhang, and Huadong Ma. VloTGPT: Learning Schedule Vision Tools Towards Intelligent Video Internet of Things. In Proceed- ings of the AAA! Conference on Artificial Intelligence. volume 39, pp. 10680-10688, April 2025. doi: (0.1609/aaii.vv911.3331 RAA//article/vee//33
URL htttps://oss aaai. wwg/index. php/
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhanc- ing Vision-Language Understanding with Advanced Large Language Models, April 2023. URL
org/ abs /2004 10592 arXiv:2304.10555 [cs].
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuchui Wang, Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang, Haomin Wang, Weiye Xu, Hao Li, Jiahao Wang, Nianchen Deng, Songze Li, Yinan He, Tan Jiang, Jiapeng Luo, Yi Wang. Conghui He, Botian Shi, Xingcheng Zhang, Wengi Shao, Junjun He, Yingtong Xiong, Wenwen Qu, Peng Sun. Penglong Jiao, Han Lv, Lijun Wu, Kaipeng Zhang, Huipeng Deng, Jiaye Ge, Kai Chen, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang.
16

---

Published as a conference paper at ICLR 2026
Intern Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models, April 2025. URL
MxXv:2504.10477 [cs].
17


Reviews:
[{"review_id": "TujjujK6z0", "reviewer": "Reviewer_mEnL", "summary": "This paper addresses a key challenge in current multi-agent systems: how to train a policy to select among multiple sub-agents, rather than relying on manually handcrafted pipelines. To tackle this, the authors propose MATA (Multi-Agent hierarchical Trainable Automaton), a novel system for visual reasoning that organizes inference as a hierarchical finite-state automaton. The authors construct a transition-trajectory dataset, MATA-SFT-90K, which is used for supervised training and evaluation. The method demonstrates strong performance on a range of visual reasoning tasks.\n\nOverall, this is a solid paper with clear motivation. The method is well designed, and the experiments are conducted rigorously. One major concern is that the three-agent design is somewhat simplistic and limited in scope, as the authors themselves acknowledge in the limitations section.", "questions": "- How were the three agents chosen, and what was the rationale behind their selection?\n- How is the competition aspect justified, given that the three agents in the current setup do not appear to have significant functional overlap?\n- Although the exponential growth of the transition search space is discussed as a limitation, do the authors have any thoughts on extending the approach to more complex scenarios with significantly more agents?\n- Would it be possible that we would observe the same performance gains if the three agents are called in an exhaustive way?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Clear motivation**: The paper addresses an important limitation in current multi-agent systems: leveraging the power of multiple agents typically requires manual pipelining, which becomes unwieldy as task complexity grows. The proposal to learn a hyper-policy for agent selection is both reasonable and interesting.\n- **Principled and extensible design**: MATA’s architecture is well aligned with its motivation. It is technically sound and, importantly, not narrowly restricted to the specific visual reasoning setting or the particular sub-agents used in this paper. In principle, the approach could extend to other tasks and larger agent pools, opening up many potential research directions.\n- **Rigorous experiment design and evaluation**: The experiments are well designed and executed. For example, the three SFT configurations and their results convincingly demonstrate the benefits of the proposed method and its generalization ability, rather than mere overfitting.", "weaknesses": "### Major\n\n- **Limited applicability**: As noted in the limitations, the use of only three agents is a restricted setting. There is also a lack of detail regarding how these three agents were selected and the rationale behind their design.\n- **Unclear attribution of performance gains**: It is not clear whether the learned state transition policy is truly responsible for the observed performance improvements. For example, if all three agents were simply called exhaustively, would performance improve regardless, making the learned policy less critical?\n- **Competition mechanism not fully justified**: While the collaborative aspect of the system is well motivated, the competitive aspect is less convincing. The paper claims that “a competition mechanism where functionally overlapping agents for the same subtask work together is under-explored,” but in the current three-agent setup, the agents seem to serve distinct roles with little actual overlap. It would be more compelling to see experiments with a larger pool of agents, including multiple agents with overlapping capabilities, to better demonstrate the value of competition.\n\n### Minor\n- Line #352: “... target dataset for evaluated; ...” should be “... target dataset for evaluation; ...”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rqgES7fYna", "reviewer": "Reviewer_6jZh", "summary": "This paper proposes **MATA (Multi-Agent Trainable Automaton)**, a hierarchical framework for **multi-agent visual reasoning**. Instead of executing a fixed modular pipeline, MATA organizes several reasoning agents (one-shot, step-wise, and specialized perception agents) as states of a **finite-state hyper-automaton**. A **trainable hyper-agent** which is an LLM-based controller then learns to select the next agent state from a shared memory, enabling dynamic collaboration and competition among agents.\n\nTo supervise this transition policy, the authors construct **MATA-SFT-90K**, a dataset of memory-to-next-state pairs extracted from expanded transition-trajectory trees across visual reasoning datasets (GQA, OK-VQA, RefCOCO series). The learned controller yields interpretable reasoning traces and achieves state-of-the-art results across multiple visual reasoning and grounding benchmarks.", "questions": "Please see the above weakness.\n\n1. Do you encounter issues with the shared memory growing unboundedly during long reasoning sequences? If so, how is this mitigated?\n2. It would also be helpful if the authors could provide qualitative visualizations comparing MATA’s reasoning paths against other systems, to better illustrate how its hierarchical controller differs in practice.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "– **Clear conceptual motivation:** The paper identifies an important limitation of existing VLMs and compositional systems that the lack of a learned, flexible orchestration mechanism among reasoning agents and recasts it elegantly as a finite-state automaton control problem.\n\n– **Novel hierarchical formulation:** Treating each agent as a sub-automaton and learning high-level transitions through a hyper-agent is a conceptually clean, interpretable design that unifies rule-based micro-control with data-driven macro-control.\n\n– **Strong empirical results:** MATA attains new SOTA accuracy on GQA, OK-VQA, and RefCOCO series, outperforming both monolithic VLMs and compositional baselines.\n\n– **Generalizability and ablation rigor:** Cross-dataset transfer (Table 6) shows < 1 % drop in zero-shot settings; ablations confirm that supervised fine-tuning of the hyper-agent contributes most gains (Table 5).", "weaknesses": "– **Incremental algorithmic novelty:** While the integration is elegant, many components (agent orchestration, SFT, trajectory trees) extend known concepts from HYDRA and NAVER. The work’s originality lies more in *system design* than in theoretical innovation.\n\n– **Limited discussion of scalability:** The near-exhaustive transition expansion is tractable for 3 agents but may explode combinatorially as more states are added.\n\n– **Computational cost analysis:** Wall-clock training times and GPU usage are only qualitatively stated; quantitative comparisons would clarify efficiency relative to HYDRA or DWIM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NSlbdjDDel", "reviewer": "Reviewer_z1Pw", "summary": "MATA introduces a hierarchical finite-state automaton for visual reasoning where multiple specialized agents (specialized perception, stepwise reasoning, oneshot reasoning) collaborate and compete. The key innovation is a trainable hyper agent that learns transition policies between agents using supervised fine-tuning on a generated dataset (MATA-SFT-90K). The dataset is created by expanding transition trajectory trees, scoring leaf nodes based on task performance, and generating memory-to-next-state training pairs. MATA improves over the base internvl model on GQA, OK-VQA, RefCOCO/RefCOCO+/RefCOCOg, and Ref-Adv benchmarks.", "questions": "- What happens when you train InternVL2.5-8B directly on MATA-SFT-90K to output answers without the multi-agent architecture? This would isolate the contribution of the architecture versus the training data. Or did distillation of a large VLM to internvl2.5?\n- What is the computational overhead (inference time, memory) compared to monolithic models and single-agent methods like HYDRA?\n- How does performance scale when adding more agents? Given the acknowledged exponential growth in trajectory trees, is this approach practical beyond 3-4 agents?\n- Can you provide evidence that the models used (InternVL2.5, Florence2-L) were not trained on GQA/OK-VQA/RefCOCO to substantiate zero-shot claims?\n- Why does the architecture without SFT perform worse than the base VLM used (internvl2.5)? Why not use a VLM instead of an LLM for the state controller?\n- Have you considered more efficient alternatives to near-exhaustive tree expansion, such as Monte Carlo tree search or learned value functions to prune unpromising branches?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The combination of trainable high-level transitions with rule-based sub-automata is elegant, focusing learning on the ambiguous agent selection problem while preserving reliable execution within agents.\n- The paper provides experiments across multiple benchmarks (VQA and visual grounding), demonstrating consistent improvements over the base model.\n- The transition trajectory tree expansion provides a principled approach to generating supervision for the hyper agent, though scalability concerns remain.", "weaknesses": "- The gains appear modest (e.g., 75.2% base internvl25 used as vlm vs 76.5% theirs on AOKVQA) considering the 90K in-domain training examples generated. The paper doesn't isolate whether improvements come from multi-agent collaboration or simply additional task-specific training data.\n- Table 5 reveals that removing SFT causes performance to drop below the base internvl25 model, suggesting the architecture itself may be detrimental without training. A crucial missing experiment is training a monolithic model on the same MATA-SFT-90K data to isolate the architectural contribution.\n- The paper claims zero-shot generalization but the base models may have been pre-trained on these datasets. This undermines claims about generalization capabilities.\n- The authors admit their trajectory tree expansion becomes intractable as agents increase, yet provide no computational overhead analysis or comparison with simpler methods.\n- While claiming to address competition between functionally overlapping agents, the paper only uses three distinct agents with clearly separated roles, not truly competitive alternatives.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cUw5WHwhVg", "reviewer": "Reviewer_TrxZ", "summary": "The paper presents a multi-agent framework for visual reasoning tasks, where a learnable hyper agent coordinates three types of reasoners. During training, only the hyper agent is optimized using synthetic data. Experiments are conducted on two VQA benchmarks (GQA and OK-VQA) and several referring expression comprehension (REC) datasets.\n\n\nThe general idea of multi-agent collaboration has been extensively explored in prior works such as [1] and [2]. While this paper introduces a slightly different collaboration mechanism, the overall conceptual novelty remains limited. Therefore, the main contributions appear to be: (1) applying multi-agent collaboration to visual reasoning; and (2) learning the hyper agent using synthetic data.\n\n\nRegarding (1), the technical contribution seems incremental. The framework largely mirrors existing multi-agent reasoning pipelines, with LLMs replaced by VLMs and visual tools. Moreover, the experimental evaluation is restricted to GQA and OK-VQA, which are insufficient to demonstrate general effectiveness on broader visual reasoning tasks.\n\n\nRegarding (2), based on Table 5, the results show minimal difference between models trained with and without SFT. This weak empirical signal makes it difficult to conclude that the proposed learning scheme provides substantial benefits.\n\n\nOverall, given the limited technical novelty and unconvincing empirical validation, I lean toward a negative recommendation for this submission.\n\n[1] Weize Chen et al., AGENTVERSE: FACILITATING MULTI-AGENT COLLABORATION AND EXPLORING EMERGENT BEHAVIORS\n\n[2] Sirui Hong et al., METAGPT: META PROGRAMMING FOR A MULTI-AGENT COLLABORATIVE FRAMEWORK", "questions": "N/A", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. A multi-agent framework for visual reasoning\n2. Empirical verification of the effectiveness of the proposed framework", "weaknesses": "See the comments in \"Summary\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)