COMMA: A Communicative Multimodal Multi-Agent Benchmark

Authors: Timothy Ossowski, Jixuan Chen, Danyal Maqbool, Zefan Cai, Tyler J. Bradshaw, Junjie Hu
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Multimodality, Agent, LLM, Benchmark

Abstract:
The rapid advances of multi-modal agents built on large foundation models have largely overlooked their potential for language-based communication between agents in collaborative tasks. This oversight presents a critical gap in understanding their effectiveness in real-world deployments, particularly when communicating with humans. Existing agentic benchmarks fail to address key aspects of inter-agent communication and collaboration, particularly in scenarios where agents have unequal access to information and must work together to achieve tasks beyond the scope of individual capabilities. To fill this gap, we introduce a novel benchmark designed to evaluate the collaborative performance of multimodal multi-agent systems through language communication. Our benchmark features a variety of scenarios, providing a comprehensive evaluation across four key categories of agentic capability in a communicative collaboration setting. By testing both agent-agent and agent-human collaborations using open-source and closed-source models, our findings reveal surprising weaknesses in state-of-the-art models, including proprietary models like GPT-4o. These models struggle to outperform even a simple random agent baseline in agent-agent collaboration and only surpass the random baseline when a human is involved.

Decision: reject
Reviewer scores: 5, 5, 5, 3

Reviews and discussion:
<thread>
creation date:
1729221459167

thread_type:
official_review

<thread_object>
creator: reviewer_x6f7

<content>
summary:
Benchmarks on both multimodal models and multi-agent benchmarks have been proposed. This paper studied the interaction of the two---how about multimodal multi-agent benchmarks? The authors hand-crafted 10 tasks that target 4 different capabilities. With evaluation over various models, a series of insights were gained.

strengths:
+ Careful analysis of the experiment results provide insight into the current deficiencies of language models.

weaknesses:
+ The proposed tasks are all toys. This causes the contribution of this paper to be limited even though it provides a reasonable starting point. As a contrasting example, in the original debate paper (Du et al.) the benchmarks that were used were specific capability benchmarks popular elsewhere.

questions:
+ Is there a fundamental asymmetry between the expert and the solver? I'm asking because in the experiments, (a, b) and (b, a) are not always both tested, leading to a particularly long table 1 that is more difficult to interpret than necessary.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729378686189

thread_type:
official_review

<thread_object>
creator: reviewer_t4yj

<content>
summary:
This paper endeavors to assess the capability of multiple multimodal models to collaborate in accomplishing complex tasks when receiving different levels of information. The authors stipulate that this collaboration process should involve communication between the models through language, facilitating model-human cooperation. As a comprehensive benchmark, COMMA evaluates four types of capabilities during the collaborative task completion process. After a thorough assessment of both model-human and model-model collaborations, the authors discovered that even the most potent closed-source models performed inadequately in their tests.

strengths:
As a comprehensive benchmark, the authors have evidently dedicated substantial effort. Regarding task distribution, the authors constructed ten subtasks that nearly comprehensively cover various aspects potentially implicated in collaborative task completion. From the perspective of model testing, the authors extensively evaluated numerous existing multimodal models, both closed-source and open-source, thereby effectively highlighting the limitations of current multimodal models. Finally, the authors classified the capabilities tested by their benchmark and analyzed common error scenarios, providing guidance for the improvement of these multimodal models.

weaknesses:
The establishment of a benchmark is undeniably a demanding task. However, I wish to express a few concerns. Firstly, in this work, what is the distinction between an agent and a model? In other words, in my previous reviews, I have consistently employed the term "model" rather than "agent" as I believe that this paper is essentially evaluating the capabilities of models and not so-called agents. If you are evaluating the capabilities of agents, please provide a definition of an agent, especially in contrast to a model. If I wish to utilize your framework to evaluate the performance of well-known agents such as CAMEL AI, AutoGen, XAgent, and other open-source agents, how can I integrate these frameworks into your evaluation system? Currently, it appears to me that you are actually testing models rather than agents. It would help if you encapsulated the models within your agent framework, such as your solver and expert agents, prior to testing them rather than directly testing the capabilities of other agents.

Furthermore, I believe that the writing of this paper seems somewhat hasty. The figure in Figure 1 is not aesthetically pleasing. At the very least, you could consider using a monotone palette of black, white, and blue for the main figure to provide a clearer perspective for the reader. Another issue I noticed with the writing is that at line 115, there is an uncompiled citation that appears as a "?" symbol. While this is not a serious error, it does detract from the reader's experience. I suggest that the authors revise the writing, especially for a possible camera-ready submission.

Although I have raised these concerns, I also greatly appreciate your work. For benchmark papers, I typically take the time to review the data or even conduct a small test myself. I have noticed that you have not released the code. If possible, could you include some code or test cases during the rebuttal phase?

questions:
See the weakness please.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729753605056

thread_type:
official_review

<thread_object>
creator: reviewer_frt1

<content>
summary:
The paper presents COMMA, a benchmark to assess the two-agent collaborative ability of VLMs. The benchmark consists of 10 simulators that provide an environment for a Solver and an Expert agents. The simulators are designed in a way that each of the two agents is not able to arrive at the success state on its own. The authors show that modern VLMs acting as both a Solver and an Expert are not able to surpass the random baseline. On the contrary, a Human in the loop acting as one of the two agents in collaboration with best VLMs are able to surpass the random baseline with varied success.

strengths:
[1] A benchmark for assessment of the collaborative abilities of VLMs is very valuable. Moreover, the presented benchmark assesses VLMs, the most functional single models to date, as opposed to non multimodal text-only LLMs.

[2] The analysis of the failure cases along with figures 3 and 4 is very insightful.

[3] The choice of models to evaluate is good: QuenVL and InternVL are at the top of ​​OpenVLM Leaderboard.

weaknesses:
[1] According to Figure 5 Left, the random baseline is very strong meaning that the benchmark is not well designed. This mostly is attributed to the small number of choices and not penalizing wrong choices enough. The claim is that modern agents are not better than the random baseline is strong only if the random baseline is weak which is not the case. In general, intuitively, with random actions during “bomb defusal” I would not expect more than 5% or even 1% success rate for it to be a proper skill test.

[2] Button and Dog are not discriminative at 100% success rate almost for all combinations of agents. 

[3] Results in Table 1 need standard deviation numbers (+/- one sigma).

[4] The paper lacks the Human-Human baseline.

[5] Overall the benchmark looks quite simplistic and is very likely to be mastered by the next generation of VLMs. The benchmark lacks assessment of more sophisticated VLM assessment like reasoning about relative positions and sizes of the objects.

[6] The benchmark cannot assess LLMs since it is tailored for VLMs.

[7] The benchmark does not evaluate VLMs in more than a 2-instance setting since all games are 2-instance.

questions:
[1] How does your benchmark stop the expert from handing over the instruction to the solver?

[2] Why aren’t LLaVA, Llama 3.2 and Clause 3.5 evaluated? What about QuenVL 72B and InternVL 76 and 40B?

[3] A lot of citations are arxiv, even though they are the papers have been already published. Please update all the citations.

[4] A good example of how to make a random baseline stronger is MMLU-Pro which extends the original 4-choice MMLU to 10 choices to rebuke the random chance from 25% to 10%.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730603448514

thread_type:
official_review

<thread_object>
creator: reviewer_icwh

<content>
summary:
This paper proposes a novel benchmark to study multimodal agents that communicate with humans to solve tasks. The authors select ten different puzzles that incorporate multimodal, memory-dependent, multi-step, and real-time elements. They evaluate four different vision-language models (VLMs) across these tasks in both Human-AI and AI-AI settings. From their analysis, the authors identify four common issues for VLMs within this benchmark: miscommunication, role misunderstanding, repetition, and misinterpretation.

strengths:
The focus on Human-AI natural language communication to solve multimodal tasks addresses a valuable research problem, and this paper establishes a promising starting point of this direction using puzzles. The authors provide strong qualitative analysis, offering insights into the weaknesses of various VLMs as agent backbones. These examples help reveal common challenges among VLMs and highlight differences between them, contributing useful knowledge for future research.

weaknesses:
In the AI-human experiments, only three data points are used for each puzzle, which is insufficient, given the relatively small performance differences between models. Additionally, the tasks are not exciting enough because solving puzzles is somewhat far from real-world scenarios, so this benchmark may serve primarily as an introductory step in studying this field.

questions:
1. Could you increase the number of runs per puzzle? For AI-AI and AI-human settings, the sample sizes of 10 and 3, respectively, seem limited for evaluating a VLM on puzzle tasks.

2. The example in the bottom right corner of Figure 2 doesn’t clearly illustrate “Misinterpretation”—miscounting the number of dogs doesn’t seem to fall under this category.

3. Could you provide human-human results for reference? Including success rates and conversation lengths could offer a useful upper bound for current performance.

4. Could you compare human behavior to AI on certain challenging puzzles, such as the password puzzle? It’s unclear why most AI-AI models completely fail on this puzzle.

5. GPT-4V+human succeeded in some maze puzzles where GPT-4o+human failed completely, which makes me confused. Could you explain the reason behind it?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a new multi-agent collaboration benchmark, COMMA, in which a solver agent and an expert agent must communicate and collaborate with each other to succeed. The results showed that VLMs alone cannot solve the tasks. On the other hand, human-AI collaboration exhibited a better performance. The findings highlighted the limits of current VLMs. The multimodal aspect of the benchmark is also particularly valuable. However, there are two critical concerns about the current benchmark that I think the revision and the rebuttal have not fully addressed:

1. The tasks are not realistic enough.

2. A strong random baseline may suggest important flaws in the benchmark design.

Given this, I would recommend further improvement before publication.