COMMA: A Communicative Multimodal Multi-Agent Benchmark

## Abstract

Abstract The rapid advances of multi-modal agents built on large foundation models have largely overlooked their potential for language-based communication between agents in collaborative tasks. This oversight presents a critical gap in understanding their effectiveness in real-world deployments, particularly when communicating with humans. Existing agentic benchmarks fail to address key aspects of inter-agent communication and collaboration, particularly in scenarios where agents have unequal access to information and must work together to achieve tasks beyond the scope of individual capabilities. To fill this gap, we introduce a novel benchmark designed to evaluate the collaborative performance of multimodal multi-agent systems through language communication. Our benchmark features a variety of scenarios, providing a comprehensive evaluation across four key categories of agentic capability in a communicative collaboration setting. By testing both agent-agent and agent-human collaborations using open-source and closed-source models, our findings reveal surprising weaknesses in state-of-the-art models, including proprietary models like GPT-4o. These models struggle to outperform even a simple random agent baseline in agent-agent collaboration and only surpass the random baseline when a human is involved. 1 1 1 We will release our benchmark and evaluation code upon acceptance.

## 1 Introduction

The field of multimodal agents is experiencing rapid growth  (Xu et al., ; Xie et al., ; Cao et al., ), with research efforts expanding at an unprecedented pace. However, amidst this growth, a critical gap in research has emerged: the lack of focus on collaborative work (Gurcan, ; Park et al., ; Hong et al., ; Liu et al., ) among multiple multimodal agents. Synergistic operation of such agents is a highly promising but largely unexplored domain.
Language agents can collaboratively finish complex tasks such as software development by assuming functional roles such as system designer, function generator, etc  (Qian et al., ; Du et al., )).
Current research on multimodal agents (Xu et al., ; Xie et al., ; Cao et al., ) has predominantly focused on individual agent capabilities, neglecting the potential for inter-agent collaboration. This limitation is further compounded by existing benchmarks such as VisualWebArena (Koh et al., ) and MME-RealWorld (Zhang et al., ), which fail to adequately assess collaborative performance between agents. As a result, our ability to evaluate and improve multi-agent systems remains constrained, hindering progress in this crucial area.

Several critical questions emerge in the context of multimodal agent collaboration. How can different agents effectively communicate multimodal information through language when they have varying levels of access to information? In scenarios where different agents possess diverse task-specific capabilities, how can they collaborate to accomplish objectives that are beyond the scope of any individual agent? These research settings remain largely uncharted and present significant challenges.
Furthermore, the ability of agents to handle incomplete information is of paramount importance, particularly when working with sensitive data  (Li et al., ) (i.e. Agent application in healthcare where privacy concerns are critical  (Tang et al., )).
The exploration of these questions is crucial for advancing the field of multimodal agent collaboration. By addressing these challenges, we can expand the applicability of multimodal agents in real-world scenarios (Zhang et al., ), particularly those involving sensitive or restricted information.

Figure: Figure 1: Overview of the interaction between the Solver and Expert agents in our benchmark. The game manager presents the Solver agent with a puzzle, where the Solver can choose to interact by clicking or requesting advice from the Expert agent. The Solver is shown an image of the puzzle (a KeyPad puzzle in this instance) and makes decisions based on the possible actions. The Expert, informed by instruction manuals, provides guidance based on the Solver’s descriptions, such as advising which buttons to press. The interaction between the Solver and Expert is captured in the dialogue, reflecting the cooperation necessary to complete the task. Both agents use self-reflection on their choices by being prompted with the conversation history as it progresses.
Refer to caption: x1.png

Motivated by these aforementioned issues, we propose a novel benchmark for evaluating collaborative multi-modal multi-agent frameworks to address critical gaps in current approaches (see  Figure 1).
Our evaluation setting simulates a scenario where an in-house agent with direct access to sensitive data (i.e., the AI solver) collaborates with external expert agents (i.e., the AI expert) to analyze information without compromising privacy. This evaluation setting could revolutionize how we handle and extract insights from sensitive datasets across various domains.

We assess multi-modal multi-agent systems using a series of carefully designed collaborative puzzle games. These scenarios typically involve two-player setups where agents have access to different, complementary information. (i.e., in a bomb defusal game, one agent possesses details about the bomb, while the other has access to a disarming manual). By employing such diverse and interactive scenarios, we aim to provide a thorough assessment of multi-modal multi-agent performance.

Our benchmark includes 10 distinct, easily customizable puzzles with thousands of unique solutions. We tested two different settings (AI-AI and AI-Human) and evaluated several popular multimodal models, including closed-source models (GPT-4V, GPT-4O, and GPT-4o1) and open-source models (Qwen-VL (Bai et al., ), and InternVL(Chen et al., )). Surprisingly, the GPT series does not outperform even a simple random baseline in the AI-AI setting, highlighting a potential growth area for future model development. Our contributions are as follows:

- •
We propose an evaluation framework called COMMA, a multimodal agent benchmark focusing on language communication between multiple agents (Section 3).
- •
Using COMMA, we carefully record conversations and performance metrics between state-of-the-art multimodal models such as QWenVL, InternVL, GPT-4o, GPT-4o1, etc (Section 4).
- •
We categorize the agent capabilities tested in our model and common failure modes, providing insight into future research directions for improving inter-agent communication (Section 5).

## 2 Related Work

#### Multi-agent Frameworks:

There are many emergent agent collaboration works  (Gurcan, ; Park et al., ; Hong et al., ; Liu et al., ; Ghafarollahi & Buehler, ; Li et al., ; Wu et al., ) among multiple language agents.
Multi-agent systems arise mainly in two different scenarios: (1) role-playing different task executors (e.g., software development requiring different roles of agents, such as program manager, software architect, programmer  Du et al. (); Qian et al. (); Hong et al. (), scientific discovery simulation  Wu et al. (), and social simulation  Park et al. (); Gurcan ()); (2) communicating between agents with different pieces of information  Wu et al. (); Li et al. () (e.g., consulting experts without sharing some sensitive or confidential data. In our case, the AI solver has some private multimodal data, and the AI expert has domain-specific knowledge or instructions).

#### Instruction-based Agent Benchmarks:

Instruction-based agent benchmarks evaluate an agent’s capability of following a human instructions to finish a task (e.g., navigating on a website, interacting with an operating system  Xu et al. (); Xie et al. (); Cao et al. ()). However, our benchmark focuses more on a communication-based evaluation where two clients engage in multi-turn conversations to solve a task collaboratively.

## 3 Benchmark

### 3.1 Design Principles of the Benchmark

Our benchmark is inspired by the cooperative gameplay scenario in Keep Talking and Nobody Explodes Games (). In this game, two players work together to defuse a bomb under time pressure. One player, the defuser, can see the bomb but lacks the instructions to disarm it. The other player, the expert, has access to the bomb’s manual but cannot see the bomb itself. The players must rely on effective communication to exchange information, navigate challenges, and defuse the bomb.

We generalize this dynamic for our benchmark, shifting the focus to solving vision-language puzzles in a communication-based agent framework. As multimodal agent systems gain momentum, our goal is to create a benchmark that rigorously evaluates their reasoning, communication, and collaborative abilities. The design of our benchmark revolves around the following core principles:

#### Language communication:

A critical aspect of our benchmark is evaluating natural language communication between agents. Similar to how players in the original game exchange information verbally, agents in our framework must use language to share observations, clarify ambiguities, and reason about tasks. In order for the agents to succeed, they must display clarity, efficiency, and depth of communication, making it an essential factor in task completion.

#### Multi-agent collaboration:

In our benchmark, agents must work together, much like the two-player dynamic of Keep Talking and Nobody Explodes. The collaboration element ensures that tasks require mutual dependency, where each agent contributes unique information or capabilities that are critical to solving the puzzle. This principle highlights how well agents can cooperate and leverage each other’s strengths.

#### Multimodality:

Our benchmark emphasizes the integration of multiple sensory inputs and outputs, such as vision, language, and audio. The puzzles involve visual elements that agents must perceive, describe, and interpret, alongside linguistic interactions. This principle assesses an agent’s ability to handle and synthesize multimodal information, a skill crucial to real-world applications.

### 3.2 Categories of Agent Capability

We benchmark agents working under different roles to solve various tasks in multiple settings, each requiring different capabilities. Specifically, the Solver agent must demonstrate strong instruction-following and multimodal reasoning, while the Expert agent is expected to excel in long text summarization and information retrieval. Both agents must possess visual comprehension and descriptive skills to succeed. Below, we outline the core capabilities tested in our benchmark.

Memory Recall (MR) In many puzzles, agents must remember their previous actions to progress. This ability is also implicitly tested when agents make mistakes. A competent agent should recall instances where past actions led to errors and adapt to avoid repeating them. The capacity to learn from mistakes and leverage memory is crucial for effective problem-solving in real-world situations.

Multimodal Grounding (MG) Since the solver agent can only communicate with the expert with text, it must be able to ground relevant spans of the expert’s instructions to the image it currently sees. This grounding of language in visual context is essential for interpreting and following guidance from the expert agent effectively.

Multi-Step Reasoning (MSR)
Certain puzzles require agents to follow a sequence of actions based on step-by-step reasoning. Much like real-world tasks, such as following a recipe or placing an online order, each action must be deliberate and contribute toward the overall goal. Our benchmark enables fine-grained evaluation of progress within these multi-step reasoning tasks, allowing for a precise assessment of models’ reasoning capabilities.

Real-time Reaction (RT)
Some puzzles challenge agents to process information rapidly and act with precise timing. This is a critical skill for embodied agents operating in dynamic, real-world environments where precise timing and quick reactions are vital.

### 3.3 Tasks

We create 10 puzzles across 4 different categories briefly summarized below. A more comprehensive description along with example images and instruction manuals can be found in Appendix A.

- •
ButtonPuzzle ( RT
): The solver must hold a colored button for a specific number of seconds based on the button’s color, the strip’s color when pressed, and the time remaining on a timer.
- •
ColorPuzzle ( MR
, MSR
): The solver presses squares of the least common color in a 4x4 grid, then follows a sequence based on a table, aiming to turn all squares white.
- •
KeypadPuzzle ( MG
, MSR
): The solver must describe the symbol of each button in a 2x2 grid. The expert must then identify a column in the manual containing these four unique symbols and tell the solver to press the symbols in the correct order from top to bottom.
- •
LedPuzzle ( MR
, MSR
): The solver presses a button if the value of its letter, when multiplied by a stage’s LED color multiplier and taken modulo 26, matches the value of the letter diagonally opposite it. At each stage, the letters on the buttons change.
- •
MazePuzzle ( MG
, MSR
): The solver navigates a mouse through a maze to a colored sphere, pressing the correct button to disarm the module based on the maze’s layout.
- •
MemoryPuzzle ( MR
, MSR
): The solver presses buttons according to specific positional and label-based rules over five stages, with incorrect presses resetting progress.
- •
PasswordPuzzle ( MG
, MSR
): The solver cycles through letters to form a valid word from a predefined list, submitting the correct word to complete the puzzle.
- •
DogPuzzle ( RT
): The solver is presented with an image containing 0-4 dogs. Based on the number of dogs in the image, the solver must press the submit button when the last digit of the timer matches the number of dogs in the image.
- •
WhoPuzzle ( MG
): The solver must tell the value on a display to the expert, who will identify a button label. The solver must then tell this label to the expert, and then press the correct button based on a detailed list of instructions.
- •
WirePuzzle ( MG
): The solver must cut one of the wires on the display. There are 3 to 6 colored wires, and the correct wire to cut changes depending on the number and order of colors.

## 4 Evaluation

### 4.1 Experimental Setup

In this section, we describe the experimental settings of our multi-agent interaction environment where two distinct agents, namely the Solver agent and the Expert agent, engage in iterative dialogue sessions. The primary aim of this setup is to assess the collaborative problem-solving capabilities between different agents or multimodal large language models (MLLMs). During our experiments, we limit the number of conversation turns to 20, allowing for a unified and systematic assessment of interactions. We use greedy decoding when available to maintain consistent agent output across runs and run inference on a single NVIDIA A100 GPU with 80GB RAM. We parse the solver’s chosen actions at each conversation turn using exact string matching, and use PyAutoGUI  (Sweigart, ) to directly perform the action on the interface if the solver outputs a valid action. Our exact prompts for both the solver and expert agent can be found in Appendix D.

### 4.2 Evaluation Metrics

We meticulously recorded several key performance metrics through multiple iterations of the experiments described below:

- •
Success Rate (SR): The solver agent is assigned a 0 or 1 value for each puzzle depending on if it completed it. These values are averaged across all puzzles to obtain the success rate.
- •
Partial Success Rate (PSR): Because our benchmark includes puzzles with multi-step reasoning, some puzzles can have a more precise success rate evaluation. For these multi-step puzzles, we assign the solver a number between 0 and 100 to indicate its progress towards the solution, and average this number across puzzles to obtain partial success rate. For single-step puzzles, partial success rate is identical to success rate.
- •
Average Mistakes (AM): After an action is chosen by the solver, the environment checks if the action was a mistake. We tally up the mistakes made during each puzzle and take a global average across puzzles to obtain average mistakes.
- •
Average Conversation Length (ACL): We count the number of conversation turns the Solver took to arrive at the solution, or default to the maximum of 20 if the solver failed. This count is averaged across all puzzles to get Average Conversation Length.

### 4.3 Models

#### Open-Source Models

- •
Human: We conduct several experiments in which a human plays as the solver or expert to provide a strong baseline. As hiring participants was prohibitively expensive and time consuming, we role played as agents ourselves across 30 sampled puzzles as a preliminary study, and leave further human participation to future work.
- •
InternVL  (Chen et al., ): A vision-language model by Shanghai AI Lab, designed for cross-modal tasks like visual question answering and image-text retrieval. We evaluate both the 26b and 8b variants of the model.
- •
QwenVL  (Bai et al., ): We use version 2 of QWenVL (QWen-VL2), offering enhanced pretraining for improved performance on vision-language tasks. We use the 2b and 7b variants.

#### Closed-Source Models

- •
GPT-4V: A version of OpenAI’s GPT-4, GPT-4V incorporates visual processing, enabling it to interpret both text and images.
- •
GPT-4o: An optimized, faster, and more cost-effective variant of GPT-4, used for applications requiring speed and efficiency.
- •
GPT-4o1: The most recent version of OpenAI’s GPT series models, which claims to have improved reasoning capability via internal chain of thought.

## 5 Results and Analysis

Table: Table 1: Average Partial Success Rate of multimodal agents on each puzzle. The solver is assigned a value between 0-100 indicating how far the solver progressed through the puzzle. The partial success rate is calculated by averaging this value over 10, 3, and 100 independent runs of each puzzle for the AI-AI, AI-Human, and random settings. The overall column is an average across all the puzzles.

### 5.1 Overall Performance

Table 1 illustrates the performances of all combinations of solver and expert pairs we evaluated. We evaluate some combinations of different open-source models because they are free, and leave pairings of separate closed-source models for future work. Intrerestingly, increasing model size does not always improve performance, as QWenVL 2b (38% PSR) outperforms QWenVL 7b (32% PSR). We observe that the random agent performs well on simple puzzles which only require one correct action (e.g. 100% on WirePuzzle, 100% on WhoPuzzle) and even outperforms all of the AI-AI setting performances. However, a random agent struggles with puzzles which require several correct actions in a row, as evidenced by its 0 percent success rate and low partial success rate in Memory, Maze, Password, Keypad, and Color puzzles.

Our results show that GPT-4o1 is the most powerful agent to use, although we only run a single iteration of each puzzle with a Human-AI setting due to rate limit issues. It makes the most progress in the color puzzle, achieving a partial success rate of 44%, followed by GPT-4o at 35%. In this puzzle, we observed that GPT-4o1’s “reasoning process” made correct connections with different parts of the puzzle’s manual, though sometimes it made incorrect conclusions at the end of the process. GPT-4V is the second best agent, with an average partial score of 53% in the AI-AI setting, although the performance is still far worse than human performance. For example, switching from a gpt-4o solver to a human solver increases overall partial success rate from 53% to 74%.

### 5.2 Qualitative Analysis on Model Failures

Figure: Figure 2: Case study examples of with the InternVL 8b model playing the role of AI solver and AI expert, resulting in failure to complete the task. Top Left: An AI solver Miscommunication error causes it to ignore instructions from the human expert, causing it to try out actions without understanding the solution. Top Right: The AI expert misunderstands its role with a Roleplay error and pretends it is the Solver. Bottom Left: The solver repeats the same bad action, resulting in a Repetition Loop error. Bottom Right: The solver misinterprets the number of dogs in the image, leading to a Misinterpretation error.
Refer to caption: x2.png

In this section, we highlight key takeaways and common failure modes displayed by the agents during their conversations. We manually classify errors across 50 conversations into the following categories, and report the distribution of these errors across the benchmark in Figure 4 and Figure 4.

Figure: Figure 3: Distribution of error category across all puzzles in AI-AI (GPT-4o) setting.
Refer to caption: extracted/5914721/figures/error_gpt4o.png

- •
Roleplay: The expert thinks it is the solver or vice versa. Figure 2 illustrates how the expert can misunderstand its role assignment, leading to miscommunication and failure to solve the puzzle.
- •
Misinterpretation: The solver misunderstands the current puzzle state/signal, resulting in failure. For instance, Figure 2 showcases the solver misinterpreting the number of dogs in the image, leading to incorrect instructions from the expert.
- •
Repetition Loop: The solver sometimes repeats its past incorrect actions, even if is in a situation it has encountered before. We classify any repeated incorrect state, action pair into this category.
- •
Miscommunication: As shown in Figure 2, the agent occasionally does not listen to the expert’s instructions, attempting to solve the puzzle on its own as if it were the expert. We also observed some open source models such as LLaVA don’t have instruction following capability for this task without further finetuning. Additionally, the solver sometimes describes the puzzle incorrectly to the expert which results in failure.

#### Repeated Actions are a Common Failure for Agents

Both open-source and closed-source models often fail due to repeating bad actions. As shown in Figure 4, InternVL is more inclined to get stuck in a repetition loop compared to GPT-4o (40% vs 18.5%). This suggests a potential improvement direction by including multi-step repetitive conversations during training, in which the model should learn to break out of the loop.

#### Agents Make More Miscommunication Errors than Misinterpretation

We observe that misinterpretation accounts for a much smaller proportion of total errors compared to miscommunication related errors (9.3% vs 44.5%) for GPT-4o and (12.0% vs 30.0%) for InternVL. We hypothesize this occurs because the training data mixture for these models primarily includes high quality single-agent data from academic benchmarks such as Visual Question Answering  (Antol et al., ), Image Captioning, etc. Including tasks requiring communication may help address this issue.

Figure: Figure 5: We plot the overall success and mistake rate on our benchmark as a function of the number of allowed conversation turns. We obtain the overall success rate and mistake rate by averaging over 100 sampled instances across all puzzles for the AI-AI setting, and 30 sampled instances for the AI-Human setting. Diamond marker indicates a human is the solver agent. Square marker indicates the solver and expert are played by the same AI model. Random is a baseline where the solver agent chooses actions uniformly at random at each time step.
Refer to caption: x3.png

### 5.3 Fine-grained Analysis

#### Multimodal Agents Struggle to Learn from Past Mistakes

An important skill for humans is to learn from past mistakes to adapt to new situations. Here we analyze if agents can display a similar capability and recover when exploring a bad trajectory when solving a puzzle. Figure 5 plots the number of allowed conversation turns to solve a puzzle, along with the overall success and mistakes rate of several multimodal agents. We note the following observations. First, incorporating a human in the pipline in the form of a human solver significantly improves overall success rate, being the only agent to achieve over random baseline performance at the 20-turn conversation mark. This is also supported by the mistakes plot, in which the human solver setting generally displays lower mistakes as the conversation progresses compared to the full AI setting. In fact, the human solver, gpt-4o expert setting shows zero mistakes over the course of most conversations, with the main reason for failure being the conversation limit. Second, humans appear to have greater ability to recover, as indicated by the faster increase in their success rate as conversation length increases, as well as the fact that they make less mistakes over time in the mistakes chart.

Figure: Figure 6: Average partial success rate (Top) and mistake rate (Bottom) for puzzles based on categories with and without human involvement. Each bar is an average performance across all model combinations. Note, some puzzles fall into multiple categorie and appear in multiple subplots.
Refer to caption: x5.png

#### Performance Based on Capability

Here we group the model performance based on the category tested: Memory (MR), Grounding (MG), Reasoning (MSR), and Reaction (RT).

#### Multimodal Agents Excel at Simple Realtime Tasks

Figure 6 gives a more nuanced look at how well multimodal agents are equipped to deal with puzzles of different nature. The agents performed well in the RT category, with the Button puzzle having the highest average partial score at 95% PSR, and Dog puzzle follows closely at 94% PSR. This suggests that agents are best at real-time reaction tasks which may involve quick decision-making based on immediate visual input and not much further communication.

#### Grounding is a Challenge for Multimodal Agents

Multimodal grounding presents a significant challenge to agents in the AI-AI setting, as seen in the varied performance. This ability requires agents to interpret and connect visual stimuli with textual instructions. The stronger performance on puzzles like Wire puzzle (88% PSR) and Who puzzle (100% PSR) indicates that agents manage better when tasks are more structured or involve simpler visual-text connections. In contrast, puzzles like the Password puzzle (1% PSR) and Maze puzzle (8% PSR), which are more abstract or less structured, present greater difficulties.

#### Multimodal Agents Struggle with Memory and Multi-Step Reasoning

Memory-based puzzles present a challenge for agents. While the LED Puzzle (38% PSR) shows moderate performance, the Color puzzle highlights a significant difficulty (9% PSR). This suggests that agents may struggle with tasks requiring them to remember previous states and actions to progress or solve sequential problems efficiently like the Memory puzzle (23% PSR). The complexity of multi-stage memory tasks could explain the poor performance. In the same vein, multi-step logical reasoning puzzles require agents to think ahead and execute a series of steps to achieve the final goal. The low performance on the Color puzzle (9% PSR) and KeyPad puzzle (24% PSR) suggests that complex reasoning tasks, especially those involving multiple stages, remain a significant challenge for agents.

## 6 Conclusion

In this paper, we address a critical gap in the field of multi-modal agents by introducing a novel benchmark specifically designed to evaluate communication in a multi-modal, multi-agent system. While substantial progress has been made in developing individual multi-modal agents, collaborative frameworks remain under-explored, particularly in scenarios requiring secure communication and the handling of sensitive data. Our benchmark aims to bridge this gap by simulating real-world conditions where agents possess complementary information and must work together to achieve complex goals.
We comprehensively evaluate metrics such as partial success rate, mistake rate, and document common failure modes for both AI-AI and AI-Human interactions. Our findings show that multimodal agents struggle to communicate with each other, often falling short of even a simple random baseline due to poor communication and frequently repeated bad actions. These findings emphasize the need for deeper investigation into enhancing inter-agent collaboration.
We hope the insights from our benchmark lay the foundation for future research on multi-modal agent collaboration and inspires the community to explore innovative approaches to improve multimodal agent capabilities this emerging field of communicative multimodal systems.

## 7 Limitations

While we aim to construct a holistic framework for multimodal agent communication, our experiments may not represent all possible scenarios in our puzzles. We conduct a preliminary study by sampling puzzle configurations and conversations between agents, and we leave more comprehensive evaluation and expansion of puzzle categories to future work. Additionally, there will inevitably be a simulation-to-reality gap from our benchmark to real-world situations, thus a high score on our benchmark may not perfectly generalize to real-world communication scenarios. Lastly, we acknowledge that there is inherent risk to using multimodal agents when handling private data. Given that LLMs have been shown to be prone to jailbreaking, it is critical to take additional safety measures before deploying an agent in practice, even if it achieves a high score on our benchmark.

## 8 Acknowledgement

Research reported in this publication was partially supported by the National Institute Of Biomedical Imaging And Bioengineering of the National Institutes of Health under Award Number R01EB033782. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

## Appendix A Manuals

## Button Puzzle

If the button is yellow, hold the button and refer to the next set of instructions of when to release it.

If you start holding the button down, a colored strip will light up on the right side of the module. Based on its color, you must release the button at a specific point in time:

- •
Blue strip: release when the countdown timer has a 4 in any position.
- •
White strip: release when the countdown timer has a 1 in any position.
- •
Yellow strip: release when the countdown timer has a 5 in any position.
- •
Any other color strip: release when the countdown timer has a 1 in any position.

## Color Puzzle

Press all squares in the correct group to progress the module.
Pressing a square will cause it to light up white.
Make all squares white to disarm the module.

To begin, press the color group containing the fewest squares.
If there is a tie, you should choose the first color that appears in the list:

- •
Red
- •
Blue
- •
Green
- •
Yellow
- •
Magenta

Then use the table to determine the next group to press in each stage.
”Group” refers to all squares of a particular color, or all non-white squares in the topmost row or leftmost column containing non-white squares.
Pressing an incorrect square will result in a strike and reset the module.
White squares will remain white for the duration of the module, but non-white squares may change color in each stage.

The table below helps to choose the next subgroup to press. The numbered keys correspond to the number of currently white squares, and the ”previously pressed color” key gives you values that indicate what color to press next based on the corresponding number of white squares.

Previously Pressed Color: {Red, Blue, Green, Yellow, Magenta, Row, Column}

$$ $\displaystyle 1:$ $\displaystyle\ \{Blue,Column,Red,Yellow,Row,Green,Magenta\}$ $\displaystyle 2:$ $\displaystyle\ \{Row,Green,Blue,Magenta,Red,Column,Yellow\}$ $\displaystyle 3:$ $\displaystyle\ \{Yellow,Magenta,Green,Row,Blue,Red,Column\}$ $\displaystyle 4:$ $\displaystyle\ \{Blue,Green,Yellow,Column,Red,Row,Magenta\}$ $\displaystyle 5:$ $\displaystyle\ \{Yellow,Row,Blue,Magenta,Column,Red,Green\}$ $\displaystyle 6:$ $\displaystyle\ \{Magenta,Red,Yellow,Green,Column,Blue,Row\}$ $\displaystyle 7:$ $\displaystyle\ \{Green,Row,Column,Blue,Magenta,Yellow,Red\}$ $\displaystyle 8:$ $\displaystyle\ \{Magenta,Red,Green,Blue,Yellow,Column,Row\}$ $\displaystyle 9:$ $\displaystyle\ \{Column,Yellow,Red,Green,Row,Magenta,Blue\}$ $\displaystyle 10:$ $\displaystyle\ \{Green,Column,Row,Red,Magenta,Blue,Yellow\}$ $\displaystyle 11:$ $\displaystyle\ \{Red,Yellow,Row,Column,Green,Magenta,Blue\}$ $\displaystyle 12:$ $\displaystyle\ \{Column,Row,Column,Row,Row,Column,Row\}$ $\displaystyle 13:$ $\displaystyle\ \{Row,Column,Row,Column,Row,Column,Column\}$ $\displaystyle 14:$ $\displaystyle\ \{Column,Column,Row,Row,Column,Row,Column\}$ $\displaystyle 15:$ $\displaystyle\ \{Row,Row,Column,Row,Column,Column,Row\}$ $$

## KeyPad Puzzle

Only one column has all four symbols from the keypad.
Press the four buttons in the order their symbols appear from top to bottom within that column.

## LED Puzzle

Two to five LEDs are installed at the top of the module, representing stages. To disarm the module, these stages must be solved in order.
Four buttons with four different letters are shown. The letters change at each stage.
The current stage is indicated by a number in the top left of the module.
The current stage’s multiplier is indicated by that stage’s LED according to the following mapping:

- •
Red: 2
- •
Green: 3
- •
Blue: 4
- •
Yellow: 5
- •
Purple: 6
- •
Orange: 7

Assign each letter of the alphabet to the numbers 0-25 (A = 0, B = 1, C = 2, etc.).
A button is correct if its letter value, multiplied by the current stage’s multiplier, modulo 26, is equal to the regular value of the letter on its diagonally opposite button.
At each stage, press a correct button. There may be more than one possible answer.

## Maze Puzzle

The mouse is the grey sphere. It can only move into other white squares. Dark squares are walls and it cannot move into those. The mouse can move forward or backward or turn left or right.
To disarm the module, navigate the mouse to the accepting position and press the circular button with the labyrinth.
Pressing the button at any other location causes a strike.
The accepting position is marked with one of four colored spheres. Which one depends on the color of the torus in the middle of the maze, according to the table below.

- •
Torus Colors: Green, Blue, Red, Yellow
- •
Sphere Colors: Blue, Red, Green, Yellow

## Memory Puzzle

Press the correct button to progress the module to the next stage. Complete all stages to disarm the module.
Pressing an incorrect button will reset the module back to stage 1.
Button positions are ordered from left to right.

### Stage 1

- •
If the display is 1, press the button in the second position.
- •
If the display is 2, press the button in the second position.
- •
If the display is 3, press the button in the third position.
- •
If the display is 4, press the button in the fourth position.

### Stage 2

- •
If the display is 1, press the button labeled ”4”.
- •
If the display is 2, press the button in the same position as you pressed in stage 1.
- •
If the display is 3, press the button in the first position.
- •
If the display is 4, press the button in the same position as you pressed in stage 1.

### Stage 3

- •
If the display is 1, press the button with the same label you pressed in stage 2.
- •
If the display is 2, press the button with the same label you pressed in stage 1.
- •
If the display is 3, press the button in the third position.
- •
If the display is 4, press the button labeled ”4”.

### Stage 4

- •
If the display is 1, press the button in the same position as you pressed in stage 1.
- •
If the display is 2, press the button in the first position.
- •
If the display is 3, press the button in the same position as you pressed in stage 2.
- •
If the display is 4, press the button in the same position as you pressed in stage 2.

### Stage 5

- •
If the display is 1, press the button with the same label you pressed in stage 1.
- •
If the display is 2, press the button with the same label you pressed in stage 2.
- •
If the display is 3, press the button with the same label you pressed in stage 4.
- •
If the display is 4, press the button with the same label you pressed in stage 3.

## Password Puzzle

The buttons above and below each letter will cycle through the possibilities for that position.
Each cycle will have 3 consecutive letters.
Only one combination of the available letters will match a password from the list below.
Press the submit button once the correct word has been set.

### List of Possible Words:

- •
about, after, again, below, could, every, first, found, great, house, large, learn, never, other, place, plant, point, right, small, sound, spell, still, study, their, there, these, thing, think, three, water, where, which, world, would, write.

## Who Puzzle

1. Read the display and use step 1 to determine which button label to read.
2. Using this button label, use step 2 to determine which button to push.

### Step 1:

Based on the display, ask the SOLVER to read the label of a particular button and proceed to step 2:

- •
”YES”: Middle Left
- •
”FIRST”: Top Right
- •
”DISPLAY”: Bottom Right
- •
”OKAY”: Top Right
- •
”SAYS”: Bottom Right
- •
”NOTHING”: Middle Left
- •
”(No Text)”: Bottom Left
- •
”BLANK”: Middle Right
- •
”NO”: Bottom Right
- •
”LED”: Middle Left
- •
”LEAD”: Bottom Right
- •
”READ”: Middle Right
- •
”RED”: Middle Right
- •
”REED”: Bottom Left
- •
”LEED”: Bottom Left
- •
”HOLD ON”: Bottom Right
- •
”YOU”: Middle Right
- •
”YOU ARE”: Bottom Right
- •
”YOUR”: Middle Right
- •
”YOU’RE”: Middle Right
- •
”UR”: Top Left
- •
”THERE”: Bottom Right
- •
”THEY’RE”: Bottom Left
- •
”THEIR”: Middle Right
- •
”THEY ARE”: Middle Left
- •
”SEE”: Bottom Right
- •
”C”: Top Right
- •
”CEE”: Bottom Right

### Step 2:

Using the label from step 1, push the first button that appears in its corresponding list:

- •
”READY”: YES, OKAY, WHAT, MIDDLE, LEFT, PRESS, RIGHT, BLANK, READY, NO, FIRST, UHHH, NOTHING, WAIT
- •
”FIRST”: LEFT, OKAY, YES, MIDDLE, NO, RIGHT, NOTHING, UHHH, WAIT, READY, BLANK, WHAT, PRESS, FIRST
- •
”NO”: BLANK, UHHH, WAIT, FIRST, WHAT, READY, RIGHT, YES, NOTHING, LEFT, PRESS, OKAY, NO, MIDDLE
- •
”BLANK”: WAIT, RIGHT, OKAY, MIDDLE, BLANK, PRESS, READY, NOTHING, NO, WHAT, LEFT, UHHH, YES, FIRST
- •
”NOTHING”: UHHH, RIGHT, OKAY, MIDDLE, YES, BLANK, NO, PRESS, LEFT, WHAT, WAIT, FIRST, NOTHING, READY
- •
”YES”: OKAY, RIGHT, UHHH, MIDDLE, FIRST, WHAT, PRESS, READY, NOTHING, YES, LEFT, BLANK, NO, WAIT
- •
”WHAT”: UHHH, WHAT, LEFT, NOTHING, READY, BLANK, MIDDLE, NO, OKAY, FIRST, WAIT, YES, PRESS, RIGHT
- •
”UHHH”: READY, NOTHING, LEFT, WHAT, OKAY, YES, RIGHT, NO, PRESS, BLANK, UHHH, MIDDLE, WAIT, FIRST
- •
”LEFT”: RIGHT, LEFT, FIRST, NO, MIDDLE, YES, BLANK, WHAT, UHHH, WAIT, PRESS, READY, OKAY, NOTHING
- •
”RIGHT”: YES, NOTHING, READY, PRESS, NO, WAIT, WHAT, RIGHT, MIDDLE, LEFT, UHHH, BLANK, OKAY, FIRST
- •
”MIDDLE”: BLANK, READY, OKAY, WHAT, NOTHING, PRESS, NO, WAIT, LEFT, MIDDLE, RIGHT, FIRST, UHHH, YES
- •
”OKAY”: MIDDLE, NO, FIRST, YES, UHHH, NOTHING, WAIT, OKAY, LEFT, READY, BLANK, PRESS, WHAT, RIGHT
- •
”WAIT”: UHHH, NO, BLANK, OKAY, YES, LEFT, FIRST, PRESS, WHAT, WAIT, NOTHING, READY, RIGHT, MIDDLE
- •
”PRESS”: RIGHT, MIDDLE, YES, READY, PRESS, OKAY, NOTHING, UHHH, BLANK, LEFT, FIRST, WHAT, NO, WAIT
- •
”YOU”: SURE, YOU ARE, YOUR, YOU’RE, NEXT, UH HUH, UR, HOLD, WHAT?, YOU, UH UH, LIKE, DONE, U
- •
”YOU ARE”: YOUR, NEXT, LIKE, UH HUH, WHAT?, DONE, UH UH, HOLD, YOU, U, YOU’RE, SURE, UR, YOU ARE
- •
”YOUR”: UH UH, YOU ARE, UH HUH, YOUR, NEXT, UR, SURE, U, YOU’RE, YOU, WHAT?, HOLD, LIKE, DONE
- •
”YOU’RE”: YOU, YOU’RE, UR, NEXT, UH UH, YOU ARE, U, YOUR, WHAT?, UH HUH, SURE, DONE, LIKE, HOLD
- •
”UR”: DONE, U, UR, UH HUH, WHAT?, SURE, YOUR, HOLD, YOU’RE, LIKE, NEXT, UH UH, YOU ARE, YOU
- •
”U”: UH HUH, SURE, NEXT, WHAT?, YOU’RE, UR, UH UH, DONE, U, YOU, LIKE, HOLD, YOU ARE, YOUR
- •
”UH HUH”: UH HUH, YOUR, YOU ARE, YOU, DONE, HOLD, UH UH, NEXT, SURE, LIKE, YOU’RE, UR, U, WHAT?
- •
”UH UH”: UR, U, YOU ARE, YOU’RE, NEXT, UH UH, DONE, YOU, UH HUH, LIKE, YOUR, SURE, HOLD, WHAT?
- •
”WHAT?”: YOU, HOLD, YOU’RE, YOUR, U, DONE, UH UH, LIKE, YOU ARE, UH HUH, UR, NEXT, WHAT?, SURE
- •
”DONE”: SURE, UH HUH, NEXT, WHAT?, YOUR, UR, YOU’RE, HOLD, LIKE, YOU, U, YOU ARE, UH UH, DONE
- •
”NEXT”: WHAT?, UH HUH, UH UH, YOUR, HOLD, SURE, NEXT, LIKE, DONE, YOU ARE, UR, YOU’RE, U, YOU
- •
”HOLD”: YOU ARE, U, DONE, UH UH, YOU, UR, SURE, WHAT?, YOU’RE, NEXT, HOLD, UH HUH, YOUR, LIKE
- •
”SURE”: YOU ARE, DONE, LIKE, YOU’RE, YOU, HOLD, UH HUH, UR, SURE, U, WHAT?, NEXT, YOUR, UH UH
- •
”LIKE”: YOU’RE, NEXT, U, UR, HOLD, DONE, UH UH, WHAT?, UH HUH, YOU, LIKE, SURE, YOU ARE, YOUR

## Wire Puzzle

Here is the manual:
The WirePuzzle module can have 3-6 wires on it. Only the one correct wire needs to be cut to disarm the module. Wire ordering begins with the first on the top.

### 3 Wires:

- •
If there are no red wires, cut the second wire.
- •
Otherwise, if the last wire is white, cut the last wire.
- •
Otherwise, if there is more than one blue wire, cut the last blue wire.
- •
Otherwise, cut the last wire.

### 4 Wires:

- •
If there is more than one red wire and the last digit of the serial number is odd, cut the last red wire.
- •
Otherwise, if the last wire is yellow and there are no red wires, cut the first wire.
- •
Otherwise, if there is exactly one blue wire, cut the first wire.
- •
Otherwise, if there is more than one yellow wire, cut the last wire.
- •
Otherwise, cut the second wire.

### 5 Wires:

- •
If the last wire is black and the last digit of the serial number is odd, cut the fourth wire.
- •
Otherwise, if there is exactly one red wire and there is more than one yellow wire, cut the first wire.
- •
Otherwise, if there are no black wires, cut the second wire.
- •
Otherwise, cut the first wire.

### 6 Wires:

- •
If there are no yellow wires and the last digit of the serial number is odd, cut the third wire.
- •
Otherwise, if there is exactly one yellow wire and there is more than one white wire, cut the fourth wire.
- •
Otherwise, if there are no red wires, cut the last wire.
- •
Otherwise, cut the fourth wire.

## Dog Puzzle

A picture containing 0-5 dogs will be shown on the display. Based on the number of dogs in the image, press the submit button when the last digit of the time left matches the number of dogs in the image.

## Appendix B Puzzle Lists

- •
ButtonPuzzle: The solver is presented with an empty strip, a colored button and a timer counting down from 10 minutes. When the button is pressed and held, the strip turns a certain color. Based off of a combination of the color of the button, the color of the strip and the time on the clock, the solver has to keep the button pressed for a certain number of seconds.
- •
ColorPuzzle: The solver is presented with a 4$\times$4 grid of colored tiles. The solver must first identify the color group with the fewest squares on a 4x4 grid and press all the squares of that color to start the module. The solver then needs to refer to a table to determine the next group to press based on the current configuration. Pressing any incorrect square results in a strike and resets the module. Non-white squares may change color after each stage. The goal is to make all squares on the grid white by following the correct sequence of groups.
- •
KeypadPuzzle: The solver has to examine a 2x2 grid of unique symbols and identify which of the four columns below the grid contains all four symbols from the grid. Once the correct column is found, the solver must press the buttons in that column in the order the symbols appear from top to bottom.
- •
LedPuzzle: The solver progresses through 2 to 5 stages, each indicated by an LED color that specifies a multiplier (Red: 2, Green: 3, Blue: 4, Yellow: 5, Purple: 6, Orange: 7). Four buttons with changing letters are shown at each stage. The solver must assign values to letters (A = 0, B = 1, etc.) and press a button if its letter value, when multiplied by the stage’s multiplier and taken modulo 26, equals the value of the letter on its diagonally opposite button. Each stage requires pressing a correct button, and there may be multiple valid choices.
- •
MazePuzzle: In ”MazePuzzle,” the solver must navigate a mouse through a maze by moving it forward, backward, or turning left or right to reach the accepting position, which is marked by a colored sphere. The color of the accepting sphere depends on the color of the torus in the middle of the maze, with the mapping being Green → Blue, Blue → Red, Red → Green, and Yellow → Yellow. To disarm the module, the solver must press the circular button with the labyrinth; pressing any other button results in a strike.
- •
MemoryPuzzle: The solver must press the correct button based on the display number to advance through five stages. Incorrect presses reset the module to stage 1. Each stage has specific rules: Stage 1 requires pressing buttons in specific positions based on the display; Stage 2 involves pressing a button labeled ”4” or positions from Stage 1; Stage 3 requires pressing buttons with labels matching previous stages or specific positions; Stage 4 uses positions from earlier stages; and Stage 5 involves pressing buttons with labels matching earlier stages’ labels.
- •
PasswordPuzzle: The solver cycles through letters above and below each position to form a word. Each cycle displays three consecutive letters, and only one combination will match a predefined list of possible words. Once the correct word is set, the solver must press the submit button to complete the puzzle. The list of possible words includes terms like ”about,” ”after,” ”great,” and ”write.”
- •
SoundPuzzle: The solver listens to a sound clip and matches it to one of the options provided. Each sound clip is associated with a code made up of four symbols ($, *, &, #). After identifying the correct option from the list (e.g., Taxi Dispatch, Cow, Extractor Fan, Train Station), the solver enters the corresponding code using a four-button keypad to proceed.
- •
WhoPuzzle The solver reads a display to determine which button label to reference and then uses that label to find which button to press based on a predefined list. The process involves two steps: first, the display directs you to a specific button label according to a detailed list of instructions. Second, using that label, you select the appropriate button from a secondary list of options. Successfully following these steps in sequence will advance the module.
- •
WirePuzzle: The solver is presented with between 3 and 6 wires of different colors. Based off of the ordering and number of colors of each type, the solver has to cut the wires in a specific order. The manual lists out the different branches that can be possible for each setting.

## Appendix C Additional Statistics

Table: Table 2: Average number of mistakes the solver made for various puzzles. We average the mistakes over 10, 3, and 100 independent runs of each puzzle for the AI-AI, AI-Human, and random settings. The overall column is an average across all the puzzles.

Table: Table 3: Average conversation length for various puzzles. We average the length of agent dialogue over 10, 3, and 100 independent runs of each puzzle for the AI-AI, AI-Human, and random settings. The overall column is an average dialogue length across all the puzzles.

We report additional metrics recorded during evaluation such as Average Mistakes (Table 2) and Conversation Length (Table 3)

## Appendix D Agent Prompts

#### Solver Prompt:

You are the solver in a cooperative game involving solving puzzles. As the solver, you are presented with an image of the puzzle, along with possible actions you may take. You should only attempt some actions if you are certain of the solution. Otherwise, you should describe the image and ask the expert. When asking the expert, keep in mind the expert cannot see the image. Your description should be concise but also detailed enough to convey the details to the expert through text only. Once you are certain of the solution, respond with just the name of the action you chose. If in a puzzle you can take multiple steps to solve it, you could output a list of action names, separated by the line break \n and in the sequential order to be executed. ONLY FINISH THE SOLVER’S DIALOGUE.

#### Expert Prompt:

You are the expert in a cooperative game involving solving puzzles. As the expert, you hold the puzzle solution manual, containing vital information on various modules and their corresponding solution procedures. Your task is to listen carefully to the solver’s descriptions of the puzzles and provide clear and accurate instructions to guide them through the solution. Be as concise and precise in your instructions as possible. If the solver does not provide you with enough information, ask for clarification if needed. ONLY FINISH THE EXPERT’S DIALOGUE.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Could you increase the number of runs per puzzle? For AI-AI and AI-human settings, the sample sizes of 10 and 3, respectively, seem limited for evaluating a VLM on puzzle tasks.

2. The example in the bottom right corner of Figure 2 doesn’t clearly illustrate “Misinterpretation”—miscounting the number of dogs doesn’t seem to fall under this category.

3. Could you provide human-human results for reference? Including success rates and conversation lengths could offer a useful upper bound for current performance.

4. Could you compare human behavior to AI on certain challenging puzzles, such as the password puzzle? It’s unclear why most AI-AI models completely fail on this puzzle.

5. GPT-4V+human succeeded in some maze puzzles where GPT-4o+human failed completely, which makes me confused. Could you explain the reason behind it?
soundness: 2
strengths: The focus on Human-AI natural language communication to solve multimodal tasks addresses a valuable research problem, and this paper establishes a promising starting point of this direction using puzzles. The authors provide strong qualitative analysis, offering insights into the weaknesses of various VLMs as agent backbones. These examples help reveal common challenges among VLMs and highlight differences between them, contributing useful knowledge for future research.
summary: This paper proposes a novel benchmark to study multimodal agents that communicate with humans to solve tasks. The authors select ten different puzzles that incorporate multimodal, memory-dependent, multi-step, and real-time elements. They evaluate four different vision-language models (VLMs) across these tasks in both Human-AI and AI-AI settings. From their analysis, the authors identify four common issues for VLMs within this benchmark: miscommunication, role misunderstanding, repetition, and misinterpretation.
weaknesses: In the AI-human experiments, only three data points are used for each puzzle, which is insufficient, given the relatively small performance differences between models. Additionally, the tasks are not exciting enough because solving puzzles is somewhat far from real-world scenarios, so this benchmark may serve primarily as an introductory step in studying this field.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: [1] How does your benchmark stop the expert from handing over the instruction to the solver?

[2] Why aren’t LLaVA, Llama 3.2 and Clause 3.5 evaluated? What about QuenVL 72B and InternVL 76 and 40B?

[3] A lot of citations are arxiv, even though they are the papers have been already published. Please update all the citations.

[4] A good example of how to make a random baseline stronger is MMLU-Pro which extends the original 4-choice MMLU to 10 choices to rebuke the random chance from 25% to 10%.
soundness: 3
strengths: [1] A benchmark for assessment of the collaborative abilities of VLMs is very valuable. Moreover, the presented benchmark assesses VLMs, the most functional single models to date, as opposed to non multimodal text-only LLMs.

[2] The analysis of the failure cases along with figures 3 and 4 is very insightful.

[3] The choice of models to evaluate is good: QuenVL and InternVL are at the top of ​​OpenVLM Leaderboard.
summary: The paper presents COMMA, a benchmark to assess the two-agent collaborative ability of VLMs. The benchmark consists of 10 simulators that provide an environment for a Solver and an Expert agents. The simulators are designed in a way that each of the two agents is not able to arrive at the success state on its own. The authors show that modern VLMs acting as both a Solver and an Expert are not able to surpass the random baseline. On the contrary, a Human in the loop acting as one of the two agents in collaboration with best VLMs are able to surpass the random baseline with varied success.
weaknesses: [1] According to Figure 5 Left, the random baseline is very strong meaning that the benchmark is not well designed. This mostly is attributed to the small number of choices and not penalizing wrong choices enough. The claim is that modern agents are not better than the random baseline is strong only if the random baseline is weak which is not the case. In general, intuitively, with random actions during “bomb defusal” I would not expect more than 5% or even 1% success rate for it to be a proper skill test.

[2] Button and Dog are not discriminative at 100% success rate almost for all combinations of agents. 

[3] Results in Table 1 need standard deviation numbers (+/- one sigma).

[4] The paper lacks the Human-Human baseline.

[5] Overall the benchmark looks quite simplistic and is very likely to be mastered by the next generation of VLMs. The benchmark lacks assessment of more sophisticated VLM assessment like reasoning about relative positions and sizes of the objects.

[6] The benchmark cannot assess LLMs since it is tailored for VLMs.

[7] The benchmark does not evaluate VLMs in more than a 2-instance setting since all games are 2-instance.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: See the weakness please.
soundness: 2
strengths: As a comprehensive benchmark, the authors have evidently dedicated substantial effort. Regarding task distribution, the authors constructed ten subtasks that nearly comprehensively cover various aspects potentially implicated in collaborative task completion. From the perspective of model testing, the authors extensively evaluated numerous existing multimodal models, both closed-source and open-source, thereby effectively highlighting the limitations of current multimodal models. Finally, the authors classified the capabilities tested by their benchmark and analyzed common error scenarios, providing guidance for the improvement of these multimodal models.
summary: This paper endeavors to assess the capability of multiple multimodal models to collaborate in accomplishing complex tasks when receiving different levels of information. The authors stipulate that this collaboration process should involve communication between the models through language, facilitating model-human cooperation. As a comprehensive benchmark, COMMA evaluates four types of capabilities during the collaborative task completion process. After a thorough assessment of both model-human and model-model collaborations, the authors discovered that even the most potent closed-source models performed inadequately in their tests.
weaknesses: The establishment of a benchmark is undeniably a demanding task. However, I wish to express a few concerns. Firstly, in this work, what is the distinction between an agent and a model? In other words, in my previous reviews, I have consistently employed the term "model" rather than "agent" as I believe that this paper is essentially evaluating the capabilities of models and not so-called agents. If you are evaluating the capabilities of agents, please provide a definition of an agent, especially in contrast to a model. If I wish to utilize your framework to evaluate the performance of well-known agents such as CAMEL AI, AutoGen, XAgent, and other open-source agents, how can I integrate these frameworks into your evaluation system? Currently, it appears to me that you are actually testing models rather than agents. It would help if you encapsulated the models within your agent framework, such as your solver and expert agents, prior to testing them rather than directly testing the capabilities of other agents.

Furthermore, I believe that the writing of this paper seems somewhat hasty. The figure in Figure 1 is not aesthetically pleasing. At the very least, you could consider using a monotone palette of black, white, and blue for the main figure to provide a clearer perspective for the reader. Another issue I noticed with the writing is that at line 115, there is an uncompiled citation that appears as a "?" symbol. While this is not a serious error, it does detract from the reader's experience. I suggest that the authors revise the writing, especially for a possible camera-ready submission.

Although I have raised these concerns, I also greatly appreciate your work. For benchmark papers, I typically take the time to review the data or even conduct a small test myself. I have noticed that you have not released the code. If possible, could you include some code or test cases during the rebuttal phase?

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: + Is there a fundamental asymmetry between the expert and the solver? I'm asking because in the experiments, (a, b) and (b, a) are not always both tested, leading to a particularly long table 1 that is more difficult to interpret than necessary.
soundness: 3
strengths: + Careful analysis of the experiment results provide insight into the current deficiencies of language models.
summary: Benchmarks on both multimodal models and multi-agent benchmarks have been proposed. This paper studied the interaction of the two---how about multimodal multi-agent benchmarks? The authors hand-crafted 10 tasks that target 4 different capabilities. With evaluation over various models, a series of insights were gained.
weaknesses: + The proposed tasks are all toys. This causes the contribution of this paper to be limited even though it provides a reasonable starting point. As a contrasting example, in the original debate paper (Du et al.) the benchmarks that were used were specific capability benchmarks popular elsewhere.

Meta-review: This paper introduces a new multi-agent collaboration benchmark, COMMA, in which a solver agent and an expert agent must communicate and collaborate with each other to succeed. The results showed that VLMs alone cannot solve the tasks. On the other hand, human-AI collaboration exhibited a better performance. The findings highlighted the limits of current VLMs. The multimodal aspect of the benchmark is also particularly valuable. However, there are two critical concerns about the current benchmark that I think the revision and the rebuttal have not fully addressed:

1. The tasks are not realistic enough.

2. A strong random baseline may suggest important flaws in the benchmark design.

Given this, I would recommend further improvement before publication.

Decision: Reject