OmniPlay: Benchmarking Omni-Modal Models on Omni-Modal Game Playing

## Abstract

Abstract While generalist foundation models like Gemini and GPT-4o demonstrate impressive multi-modal competence, existing evaluations fail to test their intelligence in dynamic, interactive worlds. Static benchmarks usually lack agency, while interactive benchmarks typically ignore crucial auditory and temporal cues. To bridge this evaluation chasm, we introduce OmniPlay, a diagnostic benchmark designed not just to evaluate, but to probe the fusion and reasoning capabilities of agentic models across the full sensory spectrum. Built on a core philosophy of modality interplay, OmniPlay comprises a suite of five game environments that systematically create scenarios of both model complementarity and conflict, forcing agents to perform genuine cross-modal reasoning. Our comprehensive evaluation of six leading omni-modal models reveals a critical dichotomy: they exhibit superhuman performance on high-fidelity memory tasks but suffer from systemic failures in challenges requiring robust reasoning and strategic planning. This fragility manifests as catastrophic performance degradation under modality conflict. We further uncover a counter-intuitive “less is more” phenomenon, where removing sensory information can paradoxically improve performance. Our findings suggest that the path toward robust AGI requires a research focus beyond scaling to explicitly address synergistic fusion. Our platform is available for anonymous review at https://github.com/fuqingbie/omni-game-benchmark .

## 1 Introduction

Figure: Figure 1: An illustration of the core motivation for OmniPlay. Left: Prior benchmarks suffer from two key limitations. Static benchmarks (e.g., VQA) lack interaction and agency. Existing interactive benchmarks are often limited to vision and text, ignoring critical modalities like audio, which could lead to decision-making failures. Right: OmniPlay addresses these gaps by providing an interactive, omni-modal environment. It enables agents to perform synergistic reasoning by integrating information across all modalities (e.g., combining visual, auditory, and textual cues) for more robust decision-making.
Refer to caption: x1.png

Generalist foundation models such as Google’s Gemini Team et al. (2023) and OpenAI’s GPT-4o OpenAI (2024) have recently accelerated progress toward Artificial General Intelligence (AGI). These models process text, image, audio, and video with impressive competence. However, a model’s intelligence is best measured not only by its passive perception of static data, but also by its ability to reason and act through interactive decision-making in a dynamic, sensorially rich world. This raises a critical question: how can we effectively evaluate a model’s ability to integrate multi-modal understanding with real-world interaction?

We argue that the existing evaluation landscape is split by a fundamental gap. On one side, many prominent multimodal benchmarks remain static, testing passive understanding in formats like VQA Antol et al. (2015), MMBench Liu et al. (2024), and SEED-Bench Li et al. (2023). These lack crucial dimensions of agency and long-term planning. On the other side, a second wave of benchmarks has shifted toward interactive environments such as ALFWorld Shridhar et al. (2021) and WebArena Zhou et al. (2023). While this move towards agency is vital, the majority of these interactive agents operate with limited modalities, typically confined to vision-language inputs, which restricts their ability to process auditory or complex temporal cues.

This paper argues that integrating the complete spectrum of modalities is a foundational requirement for robust omni-modal agency: an agent’s capacity to perceive, reason, and make decisions by fluidly integrating inputs across all senses. The core challenge lies in managing the complexities of modality interplay. On one hand, sensory inputs can be complementary, where one modality compensates for the limitations of another — for instance, using audio cues to navigate when vision is occluded. Leveraging this complementarity is crucial for effective decision-making. On the other hand, inputs across modalities can be conflicting. For instance, receiving contradictory visual and auditory commands will create ambiguity that degrades performance. This critical capability to synergize complementary information and resolve sensory conflicts remains largely underexplored by current methodologies.

To diagnose these foundational weaknesses, we introduce OmniPlay, a benchmark designed not just to evaluate, but to diagnose the omni-modal fusion and reasoning capabilities of agentic models, as illustrated in Figure [1](https://arxiv.org/html/2508.04361v3#S1.F1). OmniPlay is built upon a core philosophy of modality interplay. Across a suite of five distinct games, we develop scenarios that require the synergistic fusion of varying modality combinations (e.g., image-audio-text, video-audio). By systematically manipulating modality complementarity and conflict, OmniPlay functions as a diagnostic tool to answer critical questions: Can the model resolve contradictory inputs, or does it fail silently? Does it exhibit a bias toward a specific modality? Does richer sensory input act as a catalyst for reasoning, or does it instead introduce new failure modes?

Our primary contributions are:

- 1.
We introduce OmniPlay, the first interactive benchmark designed to diagnose an agent’s synergistic fusion, conflict resolution, and adaptive reasoning under controlled modality interplay across the full sensory spectrum.
- 2.
We design a suite of five games built on the principle of modality interplay, systematically creating scenarios of complementarity and conflict to reveal an agent’s architectural and reasoning flaws.
- 3.
Our comprehensive analysis reveals a critical finding: when fusion mechanisms are immature, additional modalities often hurt more than help. We demonstrate systemic weaknesses not exposed by prior benchmarks.
- 4.
We will open-source the entire OmniPlay platform, including all environments, baseline agents, and evaluation protocols, to facilitate relevant research.

## 2 Related Work

Our research is positioned at the intersection of multimodal evaluation and interactive agent learning. We structure our review by first discussing static benchmarks to highlight the need for agency, then examining interactive benchmarks to reveal their modal bottleneck, and finally arguing that the rise of omni-modal models has turned this bottleneck into a critical evaluation chasm that OmniPlay aims to bridge.

### 2.1 Static Multimodal Benchmarks: Perception without Agency

Early multimodal evaluation centered on static perception tasks. Seminal works like Visual Question Answering (VQA) (Antol et al., 2015; Hudson & Manning, 2019) and image captioning on datasets like COCO (Chen et al., 2015) were foundational for representation learning. More recent comprehensive platforms, such as MMBench (Liu et al., 2024) and SEED-Bench (Li et al., 2023), aggregated numerous tasks, yet they all share a unifying limitation: their static and non-interactive nature. Models perform single-turn perception on fixed inputs, which fails to evaluate crucial agentic capabilities like sequential decision-making or long-term planning.

### 2.2 Interactive Agent Benchmarks: Agency with a Modal Bottleneck

To address the lack of agency, a second wave of benchmarks introduced interactive environments. This evolution began in text-based worlds like Jericho (Hausknecht et al., 2020), expanded to embodied AI in 3D simulators such as AI2-THOR (Kolve et al., 2017) and Habitat (Savva et al., 2019), and extended to grounded language in ALFWorld (Shridhar et al., 2021) and complex digital tasks in WebArena (Zhou et al., 2023) and Mind2Web (Deng et al., 2023). Despite this significant leap towards agency, a prevalent modal bottleneck constrains the majority of these benchmarks, as perception is typically limited to vision and text. Recent game-based works like BALROG (Paglieri et al., 2025) further highlight deep reasoning deficiencies even within these limited modalities. While pioneering platforms like SoundSpaces 2.0 (Chen et al., 2020) incorporated audio for navigation, a comprehensive, diagnostic approach to omni-modality has been missing.

### 2.3 Omni-Modal Models and The Evaluation Chasm

This long-standing modal bottleneck has recently escalated into a critical evaluation chasm with the arrival of true omni-modal foundation models like Google’s Gemini and OpenAI’s GPT-4o. These models are natively designed to process a fluid combination of text, image, audio, and video, yet our primary tools for evaluating agency lack the sensory richness to test these new faculties. Current evaluations fail to assess how these powerful models perform in dynamic, multi-sensory scenarios where they must make choices.

OmniPlay is designed to bridge this chasm. It is the first interactive benchmark built on a core philosophy of modality interplay. By systematically creating tasks requiring synergistic fusion and stress-testing agents with controlled sensory conflicts, OmniPlay provides a dedicated diagnostic platform to rigorously evaluate the true interactive and reasoning capabilities of modern omni-modal agents.

## 3 The OmniPlay Benchmark

Figure: Figure 2: An overview of the five distinct game environments in the OmniPlay suite. Each sub-figure illustrates a game’s user interface and the primary modalities involved: (I)mage, (V)ideo, (A)udio, and (T)ext. The suite is designed to test a diverse range of capabilities, from (a) visuo-auditory navigation and (b) sequence replication, to (c) abstract reasoning, (d) real-time strategy, and (e) multi-agent combat.
Refer to caption: x2.png

This section details the architectural design and core components of the OmniPlay benchmark. We begin by establishing a rigorous theoretical foundation by formalizing the agent-environment interaction within a generalized Markov Decision Process (MDP) framework. Following this, we articulate the core design principles that guided the benchmark’s construction. We then introduce the five distinct game environments that constitute our suite and conclude by outlining our comprehensive evaluation protocol.

### 3.1 Formalism: A Generalized Interaction Framework

To provide a unified and rigorous description of agent-environment interaction across our diverse suite of games, we model each task within a generalized Markov Decision Process (MDP) framework. This formalism captures the sequential, turn-based nature of our benchmark. The interaction is defined by the primary components $(S,A,T,G,\Omega,O)$:

- •
$S$: The set of true, underlying world states, which may be fully or partially observable.
- •
$A$: The agent’s action space, which can be discrete, continuous, or hybrid.
- •
$T$: The state transition function, $T(s^{\prime}|s,a)$.
- •
$G$: A set of goal states, $G\subseteq S$.
- •
$\Omega$: The multi-modal observation space. At each timestep $t$, the agent receives an observation $o_{t}\in\Omega$ composed of a tuple of available sensory inputs: $o_{t}=(\mathcal{I}_{t},\mathcal{V}_{t},\mathcal{A}_{t},\mathcal{T}_{t})$.
- •
$O$: The observation function, $O(o_{t}|s_{t})$.

To process the omni-modal observation $o_{t}$, the agent first employs a set of modality-specific encoders ($E_{\mathcal{I}},E_{\mathcal{V}},E_{\mathcal{A}},E_{\mathcal{T}}$) to obtain unimodal representations. These representations are then integrated by a fusion module, $\mathcal{F}$, to produce a unified context vector, $c_{t}$:

$$ $c_{t}=\mathcal{F}(E_{\mathcal{I}}(\mathcal{I}_{t}),E_{\mathcal{V}}(\mathcal{V}_{t}),E_{\mathcal{A}}(\mathcal{A}_{t}),E_{\mathcal{T}}(\mathcal{T}_{t}))$ (1) $$

This context vector $c_{t}$ is then used to update the agent’s internal state or history representation, $h_{t}$. At each timestep, the agent’s policy $\pi(a_{t}|h_{t})$ selects an action $a_{t}$. The agent’s objective is to learn a policy that maximizes the probability of generating a successful trajectory terminating in a goal state. The full probabilistic objective function is detailed in Appendix [B](https://arxiv.org/html/2508.04361v3#A2).

### 3.2 Core Design Principles

The design of OmniPlay is engineered around three core principles intended to create a rigorous diagnostic platform. These principles directly address the shortcomings of prior benchmarks and are crafted to probe the foundational capabilities of omni-modal agents.

- 1.
Modality Complementarity. A primary flaw in many multimodal models is the tendency to rely on a “dominant” modality (e.g. vision) while only superficially processing others. To counter this, our first principle dictates that tasks must be unsolvable without the complementary fusion of information from disparate channels. This forces the agent beyond simple multimodal co-occurrence and compels genuine cross-modal reasoning.
- 2.
Controlled Modality Conflict. The introduction of more sensory inputs can paradoxically degrade performance in agents with immature fusion mechanisms. Our second principle is to systematically introduce scenarios with conflicting information to directly diagnose the robustness of an agent’s fusion architecture. This principle turns OmniPlay into a tool for stress-testing an agent’s decision-making process under ambiguity.
- 3.
Various Modality Complexity. A truly omni-modal agent should not be a specialist fine-tuned for a fixed set of inputs. Therefore, our third principle ensures that the benchmark suite as a whole presents varying combinations and complexities of modalities. By offering a diverse suite of five games, we can systematically probe an agent’s architecture for biases and assess its adaptability.

### 3.3 The OmniPlay Game Suite

We introduce a suite of five distinct game environments, which we developed from the ground up to instantiate the design principles outlined above, with Blasting Showdown drawing inspiration from classic game mechanics. An overview of these games is presented in Figure [2](https://arxiv.org/html/2508.04361v3#S3.F2) and Table [1](https://arxiv.org/html/2508.04361v3#S3.T1). The modalities are denoted as I (Image), A (Audio), T (Text), and V (Video). Detailed descriptions for each game, outlining their specific rules and challenges, are provided in the Appendix [C](https://arxiv.org/html/2508.04361v3#A3).

**Table 1: High-Level Overview of the OmniPlay Game Suite. Each game is designed with a unique combination of core objectives, sensory modalities, and diagnostic challenges to probe different facets of omni-modal agency. Modalities are denoted as I (Image), A (Audio), T (Text), and V (Video).**
| Game Name | Core Objective | Modalities | Core Challenge Diagnosed |
| --- | --- | --- | --- |
| Whispered Pathfinding | 3D Maze Navigation | I, A, T | Visuo-Auditory Integration |
| Myriad Echoes | Sequence Replication | P1: V, A, T<br>P2: I, A, T | Perception-Symbol-Action Grounding |
| The Alchemist’s Melody | Abstract Rule Discovery | I, A, T | Abstract Reasoning |
| Phantom Soldiers in the Fog | Squad-based Strategy | V, A, T | Planning under Uncertainty |
| Blasting Showdown | Multi-agent Arena Combat | I, A, T | Reactive Multi-agent Strategy |

### 3.4 Evaluation Protocol and Metrics

To provide a multi-faceted view of agent capabilities, our evaluation protocol comprises two categories of metrics. We benchmark against both a random agent baseline and a human expert baseline.

Primary Metrics. We gauge high-level task success using primary metrics. Key examples include Success Rate (SR), the percentage of successful episodes, and Efficiency Score (e.g., SPL), which rewards success while penalizing inefficient actions.

Task-Specific Diagnostic Metrics. To delve deeper into agent behavior, we also designed a comprehensive set of diagnostic metrics tailored to each game’s core challenges.

Note on Blasting Showdown. As a competitive multi-agent environment, Blasting Showdown follows a distinct AI-vs-AI tournament protocol. Its results are reported using task-specific metrics (e.g., win rate) and are excluded from comparisons based on human/random baselines, such as the Normalized Performance Score (NPS) used in later sections.

A complete list of all metrics with their detailed definitions, along with full experimental parameters, can be found in Appendix [D](https://arxiv.org/html/2508.04361v3#A4).

## 4 Experimental Setup

This section details the experimental methodology used to evaluate state-of-the-art omni-modal models on the OmniPlay benchmark. We first introduce the models and baselines under evaluation, followed by a description of our comprehensive evaluation protocol.

### 4.1 Models and Baselines

Our evaluation suite comprises a diverse set of six representative omni-modal models, selected to cover both proprietary and open-source ecosystems:(^1^11GPT-4o’s API lacks simultaneous support and requires strict approval for its audio-preview version.)

- •
Proprietary Models: Google’s Gemini 2.5 Pro and Gemini 2.5 Flash Comanici et al. (2025), accessed via their official APIs.
- •
Open-Source Models: Qwen-2.5-Omni (7B) Xu et al. (2025), MiniCPM-o-2.6 (8B) Yao et al. (2024), Baichuan-Omni-1.5 (7B) Li et al. (2025), and VITA-1.5 (7B) Fu et al. (2025).

To contextualize the performance of these AI agents, we established two critical baselines.
A Random Agent serves as a performance floor by sampling an action uniformly from the set of all available actions.
More importantly, we established a Human Expert baseline by recruiting a diverse group of 12 experienced human players (all with over 500 hours of gaming experience). To enhance the robustness of our baseline, the cohort was gender-balanced and stratified into two age groups: eight young adults (20-35 years, 4 male, 4 female) and four middle-aged adults (35-50 years, 2 male, 2 female). All participants completed a warm-up period for each game to reach a stable skill plateau. A detailed breakdown of the recruitment protocol and inter-player agreement analysis is provided in Appendix [E](https://arxiv.org/html/2508.04361v3#A5).

### 4.2 Evaluation Protocol and Metrics

To ensure a fair and reproducible comparison, our protocol is built upon a fixed set of evaluation seeds for each task. This means every agent (AI, random, or human) is evaluated on the exact same pre-defined sequence of game scenarios, eliminating procedural randomness as a confounding factor. The number of seeds varies per task (e.g., 50 for Whispered Pathfinding, 30 for Phantom Soldiers). A comprehensive breakdown of evaluation episodes and model parameters is available in Appendix [D](https://arxiv.org/html/2508.04361v3#A4).

To enable comparison across diverse tasks, we introduce a Normalized Performance Score (NPS), calculated by scaling a model’s raw score relative to the human and random baselines:

$$ $\text{NPS}=100\times\frac{\text{Score}_{\text{model}}-\text{Score}_{\text{random}}}{\text{Score}_{\text{human}}-\text{Score}_{\text{random}}}$ (2) $$

An NPS of 0 represents random performance, 100 matches human experts, and scores over 100 signify superhuman ability. The Blasting Showdown task follows a distinct AI-vs-AI protocol and is excluded from NPS calculations, as noted in a footnote on its first mention. We also designed task-specific diagnostic metrics, detailed in Appendix [D](https://arxiv.org/html/2508.04361v3#A4).

## 5 Results and Analysis

Our experiments reveal critical insights into the capabilities and limitations of current omni-modal models. We first present the overall performance, which shows a stark dichotomy between memory and reasoning, and then delve into diagnostic experiments to uncover the root causes.

### 5.1 Overall Performance: A Dichotomy of Super-human Memory and Sub-par Reasoning

Figure: (a) Performance on 10 NPS-benchmarked tasks, revealing a dichotomy between memory and reasoning.
Refer to caption: x3.png

We first evaluated all six AI models against the human and random baselines across 10 NPS-benchmarked tasks. The results, summarized visually in Figure [3(a)](https://arxiv.org/html/2508.04361v3#S5.F3.sf1), reveal a fascinating dichotomy: a clear exhibition of super-human prowess in specific domains contrasted with a systemic weakness in general reasoning and strategic planning. This systemic weakness is further corroborated by the AI-vs-AI tournament in Blasting Showdown (Figure [3(b)](https://arxiv.org/html/2508.04361v3#S5.F3.sf2))(^2^22As a competitive multi-agent environment, Blasting Showdown’s results are reported using task-specific metrics (e.g., win rate) and are excluded from the NPS calculation.). The results show that even the top-performing model, Gemini 2.5 Pro, achieved only a 36.1% win rate, and its lead is not statistically decisive, reinforcing the conclusion that no single dominant strategy has yet emerged. Full statistical results for all benchmarked tasks are available in Appendix [I](https://arxiv.org/html/2508.04361v3#A9).

To precisely quantify this performance gap, Table [2](https://arxiv.org/html/2508.04361v3#S5.T2) presents the results on the most challenging memory and strategic reasoning tasks. On one hand, models exhibit super-human memory, with Gemini 2.5 Pro achieving an astounding NPS of 399.2 $\pm$ 3.6 in Myriad Echoes (Hard), a task where human cognitive limits form a bottleneck precisely because it relies heavily on short-term memory and precise sequence replication. On the other hand, this strength starkly contrasts with their brittle reasoning, as several models show negative NPS scores in strategic challenges like Phantom Soldiers.

**Table 2: A Dichotomy in Performance: Superhuman Memory vs. Brittle Reasoning. Performance on the most challenging memory task (Myriad Echoes, Hard) and strategic reasoning task (Phantom Soldiers, Hard). All metrics are reported as Mean NPS $\pm$ SEM over 50 runs.**
| Model | ME (Hard) NPS | PS (Hard) NPS |
| --- | --- | --- |
| Gemini 2.5 Pro | $399.2$ p m 3.6 | $87.5$ p m 3.5 |
| Gemini 2.5 Flash | $81.4$ p m 4.2 | $54.5$ p m 4.7 |
| Qwen-2.5-Omni | $-1.7$ p m 4.9 | $11.2$ p m 5.8 |
| MiniCPM-o-2.6 | $-2.5$ p m 5.5 | $-21.5$ p m 7.0 |
| Baichuan-Omni-1.5 | $-2.5$ p m 5.3 | $8.3$ p m 6.2 |
| VITA-1.5 | $-2.5$ p m 5.9 | $-49.2$ p m 6.0 |

### 5.2 Core Finding: Brittle Fusion and The “less is more” Paradox

To understand the root causes of these reasoning deficits, we conducted a series of diagnostic experiments.

Modality Conflict.
We first stress-tested the models’ fusion mechanisms by injecting controlled modality conflicts. We selected Whispered Pathfinding for this analysis, as its core visuo-auditory navigation challenge makes it an ideal environment to create such conflicts. In the “Audio Conflict”

Figure: (a) Conceptual diagram of a modality conflict scenario in Whispered Pathfinding.
Refer to caption: x5.png

condition, for instance, an on-screen visual cue was paired with a contradictory verbal command, forcing the agent to resolve ambiguity.

This setup is conceptually illustrated in Figure [4(a)](https://arxiv.org/html/2508.04361v3#S5.F4.sf1). The results of this stress test, presented in Figure [4(b)](https://arxiv.org/html/2508.04361v3#S5.F4.sf2) (with full statistical data in Appendix [H](https://arxiv.org/html/2508.04361v3#A8)), reveal that modality conflict induces a drastic and statistically significant degradation in performance, exposing the brittleness of their fusion mechanisms. For the top-performing Gemini 2.5 Pro, the introduction of either audio or text conflict causes a catastrophic drop in efficiency, from a stable baseline of 89.4% (SEM=$\pm$0.8) to 43.3% (SEM=$\pm$2.3) and 32.2% (SEM=$\pm$2.5), respectively. The increased SEM values under conflict also indicate a significant rise in performance instability.

Furthermore, we observe a fascinating asymmetrical sensitivity to different conflict types. Gemini 2.5 Flash, for example, is remarkably resilient to auditory conflicts, maintaining its performance at 88.6% (SEM=$\pm$1.0), nearly identical to its no-conflict baseline of 89.0% (SEM=$\pm$0.9). However, it is highly vulnerable to textual conflicts, where its performance plummets to 48.1% (SEM=$\pm$2.9). This stark difference strongly implies a hierarchical reliance on vision and text over audio in its decision-making process. The complete collapse of Qwen-2.5-Omni under audio conflict (0.0%) further underscores the diverse and often fragile nature of current fusion architectures.

Modality Ablation. This brittleness, in turn, helps explain a counter-intuitive “less is more” paradox observed in our modality ablation experiments (Figure [5](https://arxiv.org/html/2508.04361v3#S5.F5); see Appendix [H](https://arxiv.org/html/2508.04361v3#A8) for detailed statistics). We selected Whispered Pathfinding to represent synergistic navigation and Myriad Echoes for complex sequence grounding.

The results reveal two key phenomena. First, for synergistic tasks requiring tight multi-modal fusion, top models like Gemini 2.5 Pro demonstrate a clear necessity for all modalities. As shown in Figure [5](https://arxiv.org/html/2508.04361v3#S5.F5) (right), its performance in Myriad Echoes drops significantly from a peak score of 11.85 when any single modality is removed, confirming that its high performance is contingent on holistic sensory integration.

Second, and more strikingly, we found compelling evidence of the “less is more” paradox in models with weaker fusion mechanisms. For MiniCPM-o-2.6 in Whispered Pathfinding (Figure [5](https://arxiv.org/html/2508.04361v3#S5.F5), left), the full-modality performance is a modest 48.8% (SEM=$\pm$3.6). However, removing the visual modality causes its Efficiency Score to dramatically and significantly increase to 81.4% (SEM=$\pm$1.6). The concurrent sharp decrease in SEM suggests the model not only performs better but also becomes substantially more stable. This strongly suggests that for a model with immature fusion capabilities, a conflicting or poorly processed sensory input can act as a liability rather than an asset.

Figure: Figure 5: Modality ablation experiments on the ’Hard’ difficulty for Whispered Pathfinding (left) and Myriad Echoes (right). Error margins, indicated by shaded regions and bottom values (SEM), reveal two key phenomena: (1) For top models, removing any modality hurts performance in a synergistic task (necessity). (2) For other models, removing a modality can paradoxically improve performance (“less is more”).
Refer to caption: x7.png

### 5.3 Breadth of Diagnostics: Other Key Findings

Beyond the core fusion deficiencies, our diagnostic suite probed several other critical dimensions. The methodologies for our diagnostic experiments are outlined in Appendix [F](https://arxiv.org/html/2508.04361v3#A6), with further qualitative case studies in Appendix [G](https://arxiv.org/html/2508.04361v3#A7) and full raw performance data in Appendix [I](https://arxiv.org/html/2508.04361v3#A9).

- •
Robustness to Noise: Models are highly sensitive to sensory noise. In Phantom Soldiers, moderate visual noise caused Gemini 2.5 Pro’s score to plummet by over 40%, suggesting reliance on superficial correlations.
- •
Aided Reasoning via Prompting: Proprietary models show a remarkable ability to leverage explicit hints. In The Alchemist’s Melody, providing a color-to-note mapping boosted Gemini models to 100% completion, a gain not seen in open-source counterparts.
- •
Task Complexity Validation: The benchmark’s core difficulty was validated, as even a simplified version of Myriad Echoes remained unsolvable for weaker models.
- •
Modality Substitution: Models find text easier to process than other modalities. In Phantom Soldiers, substituting auditory alerts with equivalent textual descriptions led to a consistent performance increase across all models, reinforcing our finding about brittle fusion for non-textual information.

## 6 Conclusion

In this work, we addressed a critical evaluation chasm for omni-modal AI by introducing OmniPlay, the first interactive benchmark designed to diagnose, rather than merely measure, an agent’s fusion and reasoning capabilities across the full sensory spectrum. Our comprehensive evaluation of six leading models revealed a stark dichotomy: while these agents exhibit superhuman performance on high-fidelity memory tasks, they suffer from systemic failures in challenges requiring robust reasoning and strategic planning. We demonstrated that the root cause of this fragility lies in their fusion mechanisms. These mechanisms lead to catastrophic performance degradation when faced with modality conflicts or sensory noise, and give rise to a counter-intuitive “less is more” paradox, where removing sensory information can paradoxically improve performance.

Our findings carry a significant implication for the pursuit of AGI: simply scaling models may not be sufficient to bridge the gap to robust, real-world intelligence. The path forward requires a research focus that extends beyond architectural depth to explicitly address the foundational challenges of synergistic fusion, conflict arbitration, and resilient reasoning. OmniPlay provides the community with a diagnostic toolkit to probe these fundamental weaknesses.

While OmniPlay represents a significant step forward, we acknowledge its limitations and see them as exciting avenues for future work. Our study focused on zero-shot performance, while also incorporating targeted probes of the models’ in-context learning capabilities. Investigating how fine-tuning or new architectural innovations might remedy these fusion deficiencies is a critical next step. Furthermore, extending these diagnostic principles from simulated environments to real-world physical interaction remains a grand challenge for the field.

## References

- Antol et al. (2015)
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra,
C Lawrence Zitnick, and Devi Parikh.
Vqa: Visual question answering.
In *Proceedings of the IEEE international conference on computer
vision*, pp. 2425–2433, 2015.
- Chen et al. (2020)
Changan Chen, Unnat Jain, Carl Schissler, Sebastia Vicenc Amengual Gari, Ziad
Al-Halah, Vamsi Krishna Ithapu, Philip Robinson, and Kristen Grauman.
Soundspaces: Audio-visual navigation in 3d environments.
In *European conference on computer vision*, pp. 17–36.
Springer, 2020.
- Chen et al. (2015)
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr
Dollár, and C Lawrence Zitnick.
Microsoft coco captions: Data collection and evaluation server.
*arXiv preprint arXiv:1504.00325*, 2015.
- Comanici et al. (2025)
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen
Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen,
et al.
Gemini 2.5: Pushing the frontier with advanced reasoning,
multimodality, long context, and next generation agentic capabilities.
*arXiv preprint arXiv:2507.06261*, 2025.
- Deng et al. (2023)
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan
Sun, and Yu Su.
Mind2web: Towards a generalist agent for the web.
*Advances in Neural Information Processing Systems*,
36:28091–28114, 2023.
- Fu et al. (2025)
Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu,
Yangze Li, Zuwei Long, Heting Gao, Ke Li, et al.
Vita-1.5: Towards gpt-4o level real-time vision and speech
interaction.
*arXiv preprint arXiv:2501.01957*, 2025.
- Hausknecht et al. (2020)
Matthew Hausknecht, Prithviraj Ammanabrolu, Marc-Alexandre Côté, and Xingdi
Yuan.
Interactive fiction games: A colossal adventure, 2020.
URL [https://arxiv.org/abs/1909.05398](https://arxiv.org/abs/1909.05398).
- Hudson & Manning (2019)
Drew A Hudson and Christopher D Manning.
Gqa: A new dataset for real-world visual reasoning and compositional
question answering.
In *Proceedings of the IEEE/CVF conference on computer vision
and pattern recognition*, pp. 6700–6709, 2019.
- Kolve et al. (2017)
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro
Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, et al.
Ai2-thor: An interactive 3d environment for visual ai.
*arXiv preprint arXiv:1712.05474*, 2017.
- Li et al. (2023)
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan.
Seed-bench: Benchmarking multimodal llms with generative
comprehension.
*arXiv preprint arXiv:2307.16125*, 2023.
- Li et al. (2025)
Yadong Li, Jun Liu, Tao Zhang, Song Chen, Tianpeng Li, Zehuan Li, Lijun Liu,
Lingfeng Ming, Guosheng Dong, Da Pan, et al.
Baichuan-omni-1.5 technical report.
*arXiv preprint arXiv:2501.15368*, 2025.
- Liu et al. (2024)
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike
Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al.
Mmbench: Is your multi-modal model an all-around player?
In *European conference on computer vision*, pp. 216–233.
Springer, 2024.
- OpenAI (2024)
OpenAI.
GPT-4o.
Model and technical information, 2024.
URL [https://openai.com/index/hello-gpt-4o/](https://openai.com/index/hello-gpt-4o/).
Accessed: May 2024.
- Paglieri et al. (2025)
Davide Paglieri, Bartłomiej Cupiał, Samuel Coward, Ulyana Piterbarg, Maciej
Wolczyk, Akbir Khan, Eduardo Pignatelli, Łukasz Kuciński, Lerrel Pinto, Rob
Fergus, Jakob Nicolaus Foerster, Jack Parker-Holder, and Tim Rocktäschel.
Balrog: Benchmarking agentic llm and vlm reasoning on games, 2025.
URL [https://arxiv.org/abs/2411.13543](https://arxiv.org/abs/2411.13543).
- Savva et al. (2019)
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans,
Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al.
Habitat: A platform for embodied ai research.
In *Proceedings of the IEEE/CVF international conference on
computer vision*, pp. 9339–9347, 2019.
- Shridhar et al. (2021)
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam
Trischler, and Matthew Hausknecht.
ALFWorld: Aligning Text and Embodied Environments for Interactive
Learning.
In *Proceedings of the International Conference on Learning
Representations (ICLR)*, 2021.
URL [https://arxiv.org/abs/2010.03768](https://arxiv.org/abs/2010.03768).
- Team et al. (2023)
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu,
Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican,
et al.
Gemini: a family of highly capable multimodal models.
*arXiv preprint arXiv:2312.11805*, 2023.
- Xu et al. (2025)
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen,
Jialin Wang, Yang Fanand Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and
Junyang Lin.
Qwen2.5-omni technical report.
*arXiv preprint arXiv:2503.20215*, 2025.
- Yao et al. (2024)
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi
Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al.
Minicpm-v: A gpt-4v level mllm on your phone.
*arXiv preprint arXiv:2408.01800*, 2024.
- Zhou et al. (2023)
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar,
Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al.
Webarena: A realistic web environment for building autonomous agents.
*arXiv preprint arXiv:2307.13854*, 2023.

## Appendix

## Appendix A Use of Large Language Models in Manuscript Preparation

We utilized LLMs as writing assistants during the preparation of this manuscript. Their role was strictly limited to improving grammar, refining phrasing, and enhancing the overall readability and clarity of the text. The conceptualization of the research, the design and execution of the experiments, and the analysis and interpretation of the results are entirely the original work of the authors.

## Appendix B Formalism and Design Principles

This appendix provides a detailed description of the generalized Markov Decision Process (MDP) framework used in OmniPlay and illustrates how our core design principles are instantiated within this formalism.

### B.1 The Generalized Interaction Framework

As introduced in Section 3.2, we model each task as an MDP defined by the tuple $(S,A,T,G,\Omega,O)$. The agent’s core challenge is to process an omni-modal observation $o_{t}=(\mathcal{I}_{t},\mathcal{V}_{t},\mathcal{A}_{t},\mathcal{T}_{t})\in\Omega$ at each timestep, produce a unified context vector $c_{t}$ via a fusion module $\mathcal{F}$ (Equation 1), and select an optimal action $a_{t}$ based on its history $h_{t}$.

The agent’s objective is to learn a policy $\pi$ that maximizes its likelihood of successfully completing a given task. Let $\tau=(s_{0},a_{0},s_{1},a_{1},\dots)$ denote a trajectory. The probability of observing $\tau$ given a policy $\pi$ is:

$$ $P(\tau|\pi)=p(s_{0})\prod_{t=0}^{|\tau|-1}\pi(a_{t}|h_{t})T(s_{t+1}|s_{t},a_{t})$ (3) $$

Let $\mathcal{T}_{G}$ be the set of all trajectories that terminate in a goal state $s_{g}\in G$. The optimal policy $\pi^{*}$ is the one that solves:

$$ $\pi^{*}=\arg\max_{\pi}\sum_{\tau\in\mathcal{T}_{G}}P(\tau|\pi)$ (4) $$

### B.2 Formalizing the Core Design Principles

Our three core design principles—Modality Interdependence, Controlled Modality Conflict, and Variable Modality Complexity—are not merely abstract concepts but are formally embedded within the MDP structure.

##### Modality Interdependence.

This principle is primarily realized through the design of the state transition function $T(s^{\prime}|s,a)$ and the goal states $G$. A task is interdependent if, for many states $s$, there is no single modality in the observation $o_{t}$ that provides sufficient information for the policy $\pi$ to choose an action $a_{t}$ that maintains a high probability of reaching $G$. Formally, let $\pi_{m}$ be a policy that only conditions on a single modality $m\in\{\mathcal{I},\mathcal{V},\mathcal{A},\mathcal{T}\}$. An interdependent task ensures that:

$$ $\max_{\pi}\sum_{\tau\in\mathcal{T}_{G}}P(\tau|\pi)\gg\max_{m}\left(\max_{\pi_{m}}\sum_{\tau\in\mathcal{T}_{G}}P(\tau|\pi_{m})\right)$ (5) $$

This inequality formally states that the performance of a full omni-modal policy is significantly greater than the best possible uni-modal policy.

##### Controlled Modality Conflict.

We introduce conflict by manipulating the observation function $O(o_{t}|s_{t})$. In a conflict scenario, the observation $o_{t}=(\dots,m_{i},\dots,m_{j},\dots)$ contains information from two or more modalities, $m_{i}$ and $m_{j}$, that suggest contradictory optimal actions. For instance, modality $m_{i}$ suggests an action $a_{i}$ that maximizes the value function $V^{\pi}(s)$, while modality $m_{j}$ suggests an action $a_{j}$ that leads to a much lower value. This forces the agent’s fusion module $\mathcal{F}$ and policy $\pi$ to resolve the ambiguity.

##### Variable Modality Complexity.

This principle is reflected in the diversity of the observation spaces $\Omega$ and action spaces $A$ across our suite of five games. For example, the $\Omega$ for Whispered Pathfinding contains continuous spatialized audio, while the $\Omega$ for The Alchemist’s Melody involves discrete auditory tones. Similarly, the action space $A$ ranges from continuous navigation controls to discrete clicking actions. This variation across the set of MDPs $\{\text{MDP}_{1},\dots,\text{MDP}_{5}\}$ ensures that we are not testing a model’s specialization to a single type of environment but its general omni-modal capability.

## Appendix C Game Environment Details

This appendix provides detailed descriptions for each of the five game environments in the OmniPlay suite. For each game, we outline its core objective, the modalities and user interface (UI) presented to the agent, its core gameplay mechanics, and the prompting structure. Screenshots of each game’s UI and prompts are included for visual reference.

### C.1 Whispered Pathfinding

##### Core Objective.

The agent’s goal is to navigate a procedurally generated 3D maze to find a hidden, stationary target location.

##### Modalities and UI.

The agent perceives the environment through three primary modalities. An example of the UI is shown in Figure [6](https://arxiv.org/html/2508.04361v3#A3.F6).

- •
Image (I): A first-person visual feed showing the maze walls and corridors.
- •
Audio (A): Synthesized verbal guidance delivered as Text-to-Speech audio. An example of the transcribed audio content is shown in Figure [7](https://arxiv.org/html/2508.04361v3#A3.F7).
- •
Text (T): The complete turn-based prompt, which provides a structured dump of the agent’s current state and tasks the agent with generating the next action.

Figure: Figure 6: User interface for the Whispered Pathfinding environment across difficulties.
Refer to caption: x8.png

Figure: Figure 7: Example Auditory Transcript from Whispered Pathfinding. This text content is converted to speech for the agent.
Refer to caption: x9.png

##### Gameplay Mechanics.

The agent’s action space is continuous, consisting of rotation and forward movement. Success requires synergizing the visual information with the auditory guidance.

#### C.1.1 Prompting Structure

Interaction with the agent is structured via a system prompt that defines its role and a turn prompt that provides state information for each action.

##### System Prompt.

The system prompt, shown in Figure [8](https://arxiv.org/html/2508.04361v3#A3.F8), is used to initialize the agent’s behavior, defining its persona, capabilities, and required output format.

Figure: Figure 8: System Prompt for Whispered Pathfinding.
Refer to caption: x10.png

##### Turn Prompt.

At each decision step, the text modality consists of the prompt shown in Figure [9](https://arxiv.org/html/2508.04361v3#A3.F9), where {state_description} is populated with real-time data.

Figure: Figure 9: Turn Prompt for Whispered Pathfinding.
Refer to caption: x11.png

### C.2 Myriad Echoes

##### Core Objective.

This task diagnoses the full perception-to-symbol-to-action pipeline across two distinct phases.

##### Modalities and UI.

The UI for both phases is shown in Figure [10](https://arxiv.org/html/2508.04361v3#A3.F10).

- •
Phase 1 (Transcription): The agent is presented with a dynamic sequence of highlighted icons (Video) and corresponding unique sounds (Audio).
- •
Phase 2 (Execution): The agent is presented with a static grid of icons (Image) and receives auditory feedback (Audio) on clicks. The ground-truth sequence is provided via a textual prompt (Text).

Figure: Figure 10: User interface for the Myriad Echoes environment across difficulties.
Refer to caption: x12.png

##### Gameplay Mechanics.

In Phase 1, the agent must parse the multi-modal stream. In Phase 2, it must execute the parsed sequence by clicking the icons in the correct order.

#### C.2.1 Prompting Structure

##### System Prompt.

The agent is initialized with the system prompt shown in Figure [11](https://arxiv.org/html/2508.04361v3#A3.F11).

Figure: Figure 11: System Prompt for Myriad Echoes.
Refer to caption: x13.png

##### Turn Prompt.

The prompt for Phase 2 is the ground-truth sequence, visualized in Figure [12](https://arxiv.org/html/2508.04361v3#A3.F12) and [12](https://arxiv.org/html/2508.04361v3#A3.F12).

Figure: Figure 12: Turn Prompt and UI for Phase 1 (left) and Phase 2 (right) of Myriad Echoes.
Refer to caption: x14.png

### C.3 The Alchemist’s Melody

##### Core Objective.

The agent must discover a latent mapping between colors and musical notes to reproduce a specified musical scale.

##### Modalities and UI.

The UI is shown in Figure [13](https://arxiv.org/html/2508.04361v3#A3.F13).

- •
Image (I): A set of clickable colored blocks.
- •
Audio (A): Clicking a block plays a musical note.
- •
Text (T): A highly structured, real-time state dump containing feedback, sequence status, and strategic hints.

Figure: Figure 13: User interface for The Alchemist’s Melody.
Refer to caption: x16.png

##### Gameplay Mechanics.

The color-note mapping is randomized per episode. The agent must deduce it via trial-and-error, guided by the rich textual feedback.

#### C.3.1 Prompting Structure

##### System Prompt.

The agent’s role is defined by the system prompt shown in Figure [14](https://arxiv.org/html/2508.04361v3#A3.F14).

Figure: Figure 14: System Prompt for The Alchemist’s Melody.
Refer to caption: x17.png

##### Turn Prompt.

The agent receives a composite prompt including the task instruction and the detailed game state, as shown in Figure [15](https://arxiv.org/html/2508.04361v3#A3.F15).

Figure: Figure 15: Turn Prompt for The Alchemist’s Melody, showing the structured state representation.
Refer to caption: x18.png

### C.4 Phantom Soldiers in the Fog

##### Core Objective.

The agent acts as an RTS commander for a squad, aiming to achieve strategic objectives under a ”fog of war.”

##### Modalities and UI.

The UI is shown in Figure [16](https://arxiv.org/html/2508.04361v3#A3.F16).

- •
Video (V): A top-down tactical map.
- •
Text (T): Mission objectives and unit status reports.
- •
Audio (A): Structured tactical guidance delivered as Text-to-Speech audio. An example transcript is shown in Figure [17](https://arxiv.org/html/2508.04361v3#A3.F17).

Figure: Figure 16: User interface for Phantom Soldiers in the Fog across difficulties.
Refer to caption: x19.png

Figure: Figure 17: Example Auditory Transcript from Phantom Soldiers in the Fog.
Refer to caption: x20.png

##### Gameplay Mechanics.

The agent issues high-level commands. Success hinges on integrating visual, textual, and structured audio-channel guidance.

#### C.4.1 Prompting Structure

##### System Prompt.

The extensive system prompt, shown across three parts in Figure [18](https://arxiv.org/html/2508.04361v3#A3.F18), defines the complex role of the agent.

Figure: Figure 18: System Prompt for Phantom Soldiers in the Fog (Parts 1, 2, and 3).
Refer to caption: x21.png

##### Turn Prompt.

At each step, the agent receives the turn prompt shown in Figure [19](https://arxiv.org/html/2508.04361v3#A3.F19).

Figure: Figure 19: Turn Prompt for Phantom Soldiers in the Fog.
Refer to caption: x24.png

### C.5 Blasting Showdown

##### Core Objective.

Four agents compete in a destructible arena to be the last one standing.

##### Modalities and UI.

The UI is shown in Figure [20](https://arxiv.org/html/2508.04361v3#A3.F20).

- •
Image (I): A top-down view of the arena.
- •
Text (T): Status updates on all players.
- •
Audio (A): Crucial sound cues (e.g., bomb placements) essential for survival.

Figure: Figure 20: User interface for the Blasting Showdown environment.
Refer to caption: x25.png

##### Gameplay Mechanics.

Inspired by Bomberman, agents move and place bombs. Auditory cues are designed to be critical for reacting to off-screen threats.

#### C.5.1 Prompting Structure

##### System Prompt.

The agent’s competitive persona is set by the system prompt in Figure [21](https://arxiv.org/html/2508.04361v3#A3.F21).

Figure: Figure 21: System Prompt for Blasting Showdown.
Refer to caption: x26.png

##### Turn Prompt.

The turn prompt, shown in Figure [22](https://arxiv.org/html/2508.04361v3#A3.F22), provides comprehensive state information and varies depending on whether the agent is the active player.

Figure: Figure 22: Turn Prompt for Blasting Showdown, for an active player (left) and an observing player (right).
Refer to caption: x27.png

## Appendix D Experimental Parameters and Metrics

This appendix provides a comprehensive breakdown of the experimental parameters, evaluation protocols, and metrics used in our study to ensure full reproducibility.

### D.1 Model and API Parameters

For all proprietary models (Gemini 2.5 Pro and Gemini 2.5 Flash), we utilized the official, latest stable API versions available at the time of evaluation. For all models, both proprietary and open-source, we used their default decoding parameters (e.g., for temperature, top-p, and top-k) as provided by their respective APIs or standard inference scripts, without any model-specific tuning.

### D.2 Evaluation Episodes and Seeds

Our evaluation protocol is built upon a fixed set of evaluation seeds for each task, ensuring that every agent is evaluated on the exact same sequence of game scenarios. The number of episodes and seeds used for each game is detailed in Table [3](https://arxiv.org/html/2508.04361v3#A4.T3). For the NPS-benchmarked tasks, each human expert played 10 episodes for each difficulty level.

**Table 3: Number of evaluation episodes per task for AI and random agents.**
| Game | Difficulty | Seeds / Episodes |
| --- | --- | --- |
| Whispered Pathfinding | Easy, Medium, Hard | 50 |
| Myriad Echoes | Easy, Medium, Hard | 50 |
| The Alchemist’s Melody | Medium | 50 |
| Phantom Soldiers in the Fog | Easy, Medium, Hard | 30 |
| Blasting Showdown | N/A (AI vs. AI) | 50 games |

### D.3 Primary Evaluation Metrics

Our primary metric for cross-task comparison is the Normalized Performance Score (NPS), as defined in the main text. The raw ‘Score‘ used in the NPS calculation is derived from a custom, task-specific scoring function for each game. The following section details these scoring functions and other diagnostic metrics.

### D.4 Task-Specific Scoring and Diagnostic Metrics

We designed a unique set of metrics for each game to capture nuances of agent performance. Table [4](https://arxiv.org/html/2508.04361v3#A4.T4) provides a high-level overview of the metrics collected for each game. The subsequent paragraphs detail the specific scoring functions used for NPS calculation.

**Table 4: Overview of diagnostic metrics collected for each game in the OmniPlay suite.**
| Game Environment | Collected Metrics |
| --- | --- |
| Whispered Pathfinding | Mean/Min/Max Steps, Mean Invalid Actions, Mean Steps (Trimmed), NPS Score (Inverse of ‘Trimmed Mean Steps’) |
| Myriad Echoes | Success Rate, Mean Score (Execution), Mean Coordinate Accuracy (Parsing), Mean Icon Accuracy (Parsing), Parsing Failure Rate, NPS Score (Weighted Sum) |
| The Alchemist’s Melody | Completion Rate, Composite Score, NPS Score (‘Composite Score’) |
| Phantom Soldiers in the Fog | Success Rate, Normalized Score, NPS Score (Weighted Sum of ‘Success Rate’ and ‘Normalized Score’) |
| Blasting Showdown | Win Rate, Total Kills, Total Deaths, K/D Ratio |

##### Whispered Pathfinding.

For this navigation task, the final score for NPS is based on the inverse of ‘Mean Steps (Trimmed)’, as fewer steps indicate higher performance. This trimmed mean is calculated after removing the highest and lowest step counts to reduce outlier impact.

##### Myriad Echoes.

The final score for NPS is a weighted sum: 50% from ‘Mean Score’ (execution phase), 25% from ‘Mean Coordinate Accuracy’ (parsing phase), and 25% from ‘Mean Icon Accuracy’ (parsing phase).

##### The Alchemist’s Melody.

This task evaluates abstract reasoning. The final score for NPS is the ‘Score’ metric, a composite calculated from multiple performance facets as detailed in Table [5](https://arxiv.org/html/2508.04361v3#A4.T5).

**Table 5: Composite score calculation for The Alchemist’s Melody.**
| Component | Formula |
| --- | --- |
| A: Hit Rate | Hit Rate $\times$ 30 |
| B: Step Efficiency | 30 if steps $\leq$ required, else penalized |
| C: Correct Streak | (Total Correct Streak Length / Total Steps) $\times$ 10 |
| D: Error Penalty | 10 - (Total Error Streak Length / Total Steps) $\times$ 10 |
| E: Color Error Penalty | 15 - (Same-Color Error Length / Total Steps) $\times$ 15 |
| F: Exploration | (Color Changes / (Steps - 1)) $\times$ 5 |
| Total Score | Sum of A + B + C + D + E + F |

##### Phantom Soldiers in the Fog.

This RTS task uses two final metrics. The final score for NPS is a weighted sum: 50% from ‘Success Rate’ and 50% from ‘Normalized Score’.

- •
Success Rate: The ratio of completed objectives to total objectives, representing mission completion.
- •
Normalized Score: A score from 0-100 reflecting tactical and strategic efficiency, detailed below.

#### D.4.1 Detailed Calculation of the Normalized Score

The Normalized Score is derived from several components:

- 1.
Main Score ($S_{main}$): Sum of points from all completed objectives.
- 2.
Auxiliary Score ($S_{aux}$): A bonus for efficient command execution.
- 3.
Max Possible Score ($S_{max}$): This theoretical ceiling is calculated to normalize performance. It includes a base score ($S_{base}$), plus bonuses for efficiency ($B_{efficiency}$) and completing the mission within an optimal number of rounds ($R_{opt}$) relative to the maximum allowed rounds ($R_{max}$). For the Hard difficulty, a dynamic bonus ($B_{dynamic}$) is also added.
$\displaystyle S_{base}$
$\displaystyle=\sum S_{obj}\text{ for all objectives}$
$\displaystyle B_{rounds}$
$\displaystyle=S_{base}\times(1-R_{opt}/R_{max})\times 0.5$
$\displaystyle B_{efficiency}$
$\displaystyle=S_{base}\times 0.3$
$\displaystyle S_{max}$
$\displaystyle=S_{base}+B_{rounds}+B_{efficiency}+B_{dynamic}$
- 4.
Optimal Rounds ($R_{opt}$): This is a complex heuristic estimating the minimum rounds required. It accounts for visible objectives, rounds needed to discover hidden objectives (factoring in scout units), and an overhead for map exploration.
- 5.
Final Normalized Score ($S_{norm}$): The final score is a dynamic normalization of the main and auxiliary scores, considering efficiency relative to optimal rounds. The base formula is $S_{norm}=\min(100,\max(0,(S_{main}/S_{max})\times 100))$.

##### Blasting Showdown.

As a competitive multi-agent game, this task does not use NPS. Performance is measured with metrics from a 50-game tournament.

## Appendix E Human Baseline Validation Protocol

This appendix details the protocol used to establish a reliable, representative, and diverse Human Expert baseline, which is critical for the calculation of the Normalized Performance Score (NPS). Our methodology was designed to ensure fairness, stability, and high inter-player agreement while incorporating greater demographic diversity.

##### Participant Recruitment and Demographics.

We recruited a total of 12 human participants, balanced for gender and stratified by age. A key criterion for selection remained extensive gaming experience, with all participants reporting over 500 hours of gameplay across various genres relevant to the tasks in OmniPlay. Participants were compensated for their time.

The cohort was structured as follows:

- •
Young Adult Group (n=8): Ages 20-35, consisting of 4 male and 4 female participants (mean age 25.5).
- •
Middle-Aged Adult Group (n=4): Ages 35-50, consisting of 2 male and 2 female participants (mean age 41.0).

This stratified sampling aimed to provide a more robust baseline by capturing potential variations in cognitive strategies and reaction times across different age groups and genders.

##### Familiarization and Training Protocol.

To ensure that the collected data represented expert-level, stable performance rather than a learning phase, each participant underwent a mandatory warm-up and training protocol for every game environment. Before any data was recorded for a specific game (including its different difficulty levels), each participant was required to play a minimum of 10 non-recorded ‘warm-up’ episodes. The purpose of this phase was twofold: first, to allow participants to fully familiarize themselves with the game’s unique user interface, controls, and objectives; second, to allow their performance and strategies to stabilize and reach a performance plateau. Our experimenters verbally confirmed with each participant that they felt confident in their understanding of the task before proceeding to data collection.

##### Data Collection and Analysis.

Following the warm-up phase, each participant played a set number of recorded episodes for each task, as detailed in Table [6](https://arxiv.org/html/2508.04361v3#A5.T6). The final ‘Human Expert’ score reported in the main text for each task is the mean score calculated across all episodes from all 12 participants. This larger and more diverse sample provides a statistically more stable estimation of the human performance baseline.

**Table 6: Number of recorded evaluation episodes per human participant.**
| Game Environment | Episodes per Participant |
| --- | --- |
| Whispered Pathfinding | 10 |
| Myriad Echoes | 10 |
| The Alchemist’s Melody | 10 |
| Phantom Soldiers in the Fog | 10 |

##### Inter-Player Reliability.

A critical aspect of validating our human baseline is ensuring high agreement among the expert players. The detailed statistics of our human expert performance, including mean raw scores and inter-player standard deviation (SD), are presented in Table [19](https://arxiv.org/html/2508.04361v3#A9.T19). As shown, the overall SD remained consistently low relative to the mean score across most tasks, confirming a high degree of agreement on optimal strategies.

Further analysis of the inter-player variance revealed logical patterns tied to our diverse participant pool.
On tasks emphasizing cognitive skills with clear optimal solutions, such as Myriad Echoes and The Alchemist’s Melody, the performance difference between age groups and genders was minimal, resulting in a low SD (typically 5-10% of the mean score).
For tasks requiring strategy and efficient navigation like Phantom Soldiers and Whispered Pathfinding, we observed a moderate increase in variance, with the younger participant group (20-35 years) generally achieving slightly higher efficiency scores.
The highest variance was observed in Blasting Showdown (raw data not in NPS tables), a task heavily reliant on reaction speed and mechanical skill. In this game, younger male participants tended to achieve the highest performance, aligning with common patterns in competitive gaming. This controlled and interpretable variance, even with our demographically diverse sample, confirms that our collected baseline is stable and representative of expert human performance.

##### Limitations of the Human Baseline.

We acknowledge that while our human baseline of 12 participants (6 male, 6 female; mean age 30.7, SD 7.6)(^3^33The reported mean age and standard deviation are calculated from the average age of each group (25.5 for the young adult group and 41.0 for the middle-aged adult group).) provides balance in age and gender, it still has limitations regarding cultural background and broader cognitive diversity. Future work could explore larger-scale and more varied cross-cultural evaluations to further generalize the human performance benchmark.

## Appendix F Detailed Diagnostic Experiments

This appendix provides the detailed methodologies and full results for the diagnostic experiments summarized in the main text. These experiments were designed to probe specific capabilities and failure modes of the evaluated omni-modal agents. Table [7](https://arxiv.org/html/2508.04361v3#A6.T7) provides a high-level overview of all diagnostic tests conducted.

**Table 7: Overview of all diagnostic experiments conducted in this study.**
| Diagnostic Test Type | Target Game(s) | Methodology / Intervention | Capability Probed |
| --- | --- | --- | --- |
| Modality Conflict | Whispered Pathfinding | Introduce semantic contradictions between modalities (e.g., audio command vs. visual cue; textual state vs. other cues). | Fusion Robustness, Conflict Resolution, Modality Bias |
| Modality Ablation | Whispered Pathfinding, Myriad Echoes | Systematically remove one modality (audio, image, or text) at a time and evaluate performance on the remaining subset. | Modality Interdependence, Synergistic Fusion, ”Less is More” Paradox |
| Robustness to Noise | Phantom Soldiers in the Fog | Inject noise into sensory inputs: corrupting audio-channel text with random words/letters; applying visual noise (Gaussian, salt-pepper) to the map. | Perceptual Robustness, Generalization beyond clean data |
| Aided Reasoning via Prompting | Myriad Echoes, The Alchemist’s Melody | Augment the turn prompt with explicit, helpful information (e.g., current sequence step, learned color-note mapping). | Advanced Instruction Following, Knowledge Application |
| Task Simplification | Myriad Echoes | Remove the second (execution) phase of the task, converting it into a single-phase perception-to-symbol transcription task. | Benchmark Complexity Validation, Disentangling Perception vs. Action |
| Modality Substitution | Phantom Soldiers in the Fog | Replace information from one modality with its semantic equivalent in another (e.g., audio alerts replaced with textual alerts). | Modality-Agnostic Representation, Modality Preference/Bias |

### F.1 Modality Conflict

To stress-test the robustness of agent’s fusion mechanisms, we conducted modality conflict experiments in the Whispered Pathfinding environment. We systematically created scenarios where information from different modalities was semantically contradictory, forcing the agent to resolve ambiguity. The full results, compared against the no-conflict baseline, are consolidated in Table [8](https://arxiv.org/html/2508.04361v3#A6.T8).

**Table 8: Full results for Modality Conflict experiments on Whispered Pathfinding. Performance is measured by Mean Steps (lower is better).**
|  | Easy Difficulty | Medium Difficulty | Hard Difficulty |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | Mean | Min | Max | Invalid | Trimmed | Mean | Min | Max | Invalid | Trimmed | Mean | Min | Max | Invalid | Trimmed |
| Baseline (No Conflict) |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| gemini-2.5-pro | $7.6$ | $5$ | $10$ | $0.0$ | $7.6$ | $10.2$ | $7$ | $14$ | $0.0$ | $10.1$ | $42.6$ | $13$ | $152$ | $0.0$ | $36.2$ |
| gemini-2.5-flash | $16.1$ | $4$ | $70$ | $0.8$ | $10.9$ | $23.2$ | $6$ | $87$ | $1.2$ | $19.0$ | $43.5$ | $18$ | $112$ | $2.7$ | $37.15$ |
| qwen-2.5-omni | $70.3$ | $10$ | $273$ | $29.5$ | $52.5$ | $64.2$ | $11$ | $132$ | $28.1$ | $62.4$ | $130.1$ | $32$ | $253$ | $52.6$ | $128.2$ |
| MiniCPM-o-2.6 | $27.0$ | $7$ | $73$ | $7.5$ | $23.8$ | $34.4$ | $6$ | $86$ | $10.7$ | $31.5$ | $110.8$ | $34$ | $255$ | $35.8$ | $99.5$ |
| Audio Conflict |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| gemini-2.5-pro | $22.4$ | $8$ | $71$ | $0.1$ | $18.1$ | $27.1$ | $10$ | $108$ | $0.0$ | $21.8$ | $140.6$ | $58$ | $244$ | $0.2$ | $133.7$ |
| gemini-2.5-flash | $20.7$ | $4$ | $122$ | $1.7$ | $10.1$ | $35.1$ | $8$ | $95$ | $2.1$ | $28.6$ | $57.8$ | $28$ | $127$ | $0.8$ | $38.0$ |
| qwen-2.5-omni | $93.4$ | $27$ | $154$ | $37.9$ | $94.1$ | $142.6$ | $18$ | $500$ | $54.3$ | $113.5$ | $240.4$ | $103$ | $500$ | $91.2$ | $225.1$ |
| MiniCPM-o-2.6 | $48.0$ | $11$ | $126$ | $15.5$ | $42.9$ | $76.4$ | $27$ | $143$ | $23.8$ | $70.7$ | $172.4$ | $41$ | $427$ | $57.1$ | $157.0$ |
| Text Conflict |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| gemini-2.5-pro | $39.9$ | $17$ | $99$ | $0.1$ | $32.6$ | $49.2$ | $18$ | $149$ | $0.0$ | $43.0$ | $160.1$ | $40$ | $295$ | $0.0$ | $157.2$ |
| gemini-2.5-flash | $11.0$ | $5$ | $21$ | $1.0$ | $10.0$ | $17.5$ | $14$ | $20$ | $0.0$ | $18.0$ | $126.8$ | $9$ | $267$ | $6.1$ | $123.6$ |
| qwen-2.5-omni | $52.9$ | $24$ | $115$ | $21.8$ | $48.8$ | $69.5$ | $21$ | $187$ | $30.8$ | $60.9$ | $150.8$ | $38$ | $261$ | $50.6$ | $145.6$ |
| MiniCPM-o-2.6 | $27.0$ | $7$ | $47$ | $7.7$ | $27.0$ | $86.7$ | $27$ | $230$ | $26.2$ | $76.2$ | $114.8$ | $29$ | $137$ | $26.1$ | $105.2$ |

##### Audio-Visual Conflict.

In this condition, the visual cues (e.g., on-screen arrow) and textual state information were correct, but the synthesized verbal command was manipulated to suggest a contradictory action (e.g., the visual arrow points right, while the audio says “turn left”). The results in Table [8](https://arxiv.org/html/2508.04361v3#A6.T8) show a universal degradation in performance. All models took significantly more steps to solve the maze compared to the baseline, exposing the fragility of their fusion mechanisms. For instance, on the Hard difficulty, Gemini 2.5 Pro’s trimmed mean steps increased from 36.2 to 133.7, a nearly fourfold increase in inefficiency, demonstrating that even top-tier models struggle to resolve such conflicts and often follow the misleading audio cue.

##### Text-Visual/Audio Conflict.

In this condition, the visual and auditory cues remained correct, but the structured text in the Turn Prompt was manipulated to be misleading by inverting the agent’s orientation and the direction to the target. The data reveals a fascinating asymmetrical sensitivity, strongly supporting our main-text conclusion. Gemini 2.5 Pro is severely impacted by this conflict, with its mean steps increasing dramatically across all difficulties. Conversely, Gemini 2.5 Flash appears to almost entirely ignore the misleading text, showing performance that is much closer to the baseline and, on Easy/Medium difficulties, even better than its performance under audio conflict. This strongly suggests an internal modality hierarchy where Gemini 2.5 Flash prioritizes visual and auditory cues, while Gemini 2.5 Pro may have a stronger bias toward structured textual data, making it more vulnerable to this specific type of conflict.

### F.2 Modality Ablation

To investigate the necessity of each modality and uncover potential “less is more” phenomena, we conducted modality ablation studies on Whispered Pathfinding and Myriad Echoes. In these experiments, we evaluated agent performance under four conditions: the baseline with all modalities (‘Full Modality’), and three ablation conditions where either the audio, image, or text modality was removed.

#### F.2.1 Whispered Pathfinding

The full results for modality ablation on Whispered Pathfinding are presented in Table [9](https://arxiv.org/html/2508.04361v3#A6.T9).

##### Analysis of ‘Removed Audio’.

Removing the audio cues had a universally negative impact on performance, dramatically increasing the number of steps required for all models across all difficulties. This is particularly evident on the Hard difficulty, where, for instance, Gemini 2.5 Pro’s trimmed mean steps skyrocketed from 36.2 to 99.4. This result strongly validates the principle of Modality Interdependence for this task, as it confirms that the auditory channel provides critical, non-redundant information for efficient navigation that cannot be compensated for by vision and text alone.

##### Analysis of ‘Removed Image’.

The results from removing the visual modality are particularly revealing. For top-performing models like Gemini 2.5 Pro, performance degrades, though less severely than when audio is removed. This suggests that while vision is helpful, the audio and text cues can still guide the agent effectively. However, for models with weaker fusion mechanisms, we observe a striking “less is more” paradox. On the Hard difficulty, MiniCPM-o-2_6’s performance dramatically improves when the visual modality is removed, with its mean steps dropping from 110.8 to 55.0. This suggests that for this model, the visual input acts as a ‘distractor’, and removing it simplifies the decision-making process, leading to a better outcome.

##### Analysis of ‘Removed Text’.

Removing the textual state information also led to performance degradation, especially for Gemini 2.5 Pro on Medium and Hard difficulties. This indicates that top-tier models effectively ground the coordinate information to their visual perception to plan more efficient routes. Interestingly, for Gemini 2.5 Flash, removing text has a less severe impact and in some cases (Easy/Medium) even slightly improves performance compared to the baseline, suggesting it relies less on explicit coordinate data.

#### F.2.2 Whispered Pathfinding

The full results for modality ablation on Whispered Pathfinding are presented in Table [9](https://arxiv.org/html/2508.04361v3#A6.T9).

**Table 9: Full results for modality ablation on Whispered Pathfinding. Performance is measured by Mean Steps (lower is better).**
|  | Easy Difficulty | Medium Difficulty | Hard Difficulty |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | Mean | Min | Max | Invalid | Trimmed | Mean | Min | Max | Invalid | Trimmed | Mean | Min | Max | Invalid | Trimmed |
| Full Modality (Baseline) |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| gemini-2.5-pro | $7.6$ | $5$ | $10$ | $0.0$ | $7.6$ | $10.2$ | $7$ | $14$ | $0.0$ | $10.1$ | $42.6$ | $13$ | $152$ | $0.0$ | $36.2$ |
| gemini-2.5-flash | $16.1$ | $4$ | $70$ | $0.8$ | $10.9$ | $23.2$ | $6$ | $87$ | $1.2$ | $19.0$ | $43.5$ | $18$ | $112$ | $2.7$ | $37.15$ |
| qwen-2.5-omni | $70.3$ | $10$ | $273$ | $29.5$ | $52.5$ | $64.2$ | $11$ | $132$ | $28.1$ | $62.4$ | $130.1$ | $32$ | $253$ | $52.6$ | $128.2$ |
| MiniCPM-o-2.6 | $27.0$ | $7$ | $73$ | $7.5$ | $23.8$ | $34.4$ | $6$ | $86$ | $10.7$ | $31.5$ | $110.8$ | $34$ | $255$ | $35.8$ | $99.5$ |
| Removed Audio |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| gemini-2.5-pro | $13.7$ | $4$ | $80$ | $0.1$ | $10.5$ | $22.4$ | $5$ | $92$ | $0.1$ | $19.6$ | $123.7$ | $14$ | $476$ | $0.6$ | $99.4$ |
| gemini-2.5-flash | $9.3$ | $4$ | $26$ | $0.8$ | $8.7$ | $18.5$ | $5$ | $51$ | $1.4$ | $17.4$ | $49.9$ | $13$ | $115$ | $2.9$ | $48.3$ |
| qwen-2.5-omni | $91.5$ | $12$ | $329$ | $35.5$ | $82.8$ | $108.8$ | $13$ | $326$ | $43.1$ | $102.0$ | $189.9$ | $79$ | $424$ | $73.6$ | $181.1$ |
| MiniCPM-o-2.6 | $40.0$ | $9$ | $177$ | $6.8$ | $34.1$ | $139.0$ | $11$ | $376$ | $29.5$ | $125.4$ | $108.6$ | $31$ | $235$ | $25.9$ | $102.5$ |
| Removed Image |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| gemini-2.5-pro | $8.5$ | $4$ | $42$ | $0.0$ | $7.7$ | $22.0$ | $6$ | $164$ | $0.0$ | $14.1$ | $47.1$ | $9$ | $130$ | $0.0$ | $45.9$ |
| gemini-2.5-flash | $20.6$ | $4$ | $269$ | $1.1$ | $14.4$ | $38.1$ | $9$ | $198$ | $1.9$ | $34.7$ | $95.5$ | $21$ | $292$ | $5.3$ | $88.3$ |
| qwen-2.5-omni | $42.2$ | $11$ | $84$ | $14.9$ | $41.1$ | $85.9$ | $19$ | $219$ | $33.7$ | $82.4$ | $156.1$ | $45$ | $429$ | $63.6$ | $135.9$ |
| MiniCPM-o-2.6 | $24.6$ | $8$ | $60$ | $9.0$ | $22.2$ | $49.7$ | $17$ | $107$ | $21.5$ | $46.6$ | $55.0$ | $23$ | $97$ | $12.0$ | $45.0$ |
| Removed Text |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| gemini-2.5-pro | $34.5$ | $11$ | $115$ | $0.0$ | $31.4$ | $147.3$ | $13$ | $461$ | $0.0$ | $111.4$ | $124.2$ | $45$ | $241$ | $0.0$ | $118.9$ |
| gemini-2.5-flash | $9.2$ | $4$ | $22$ | $1.8$ | $9.0$ | $34.1$ | $5$ | $371$ | $7.6$ | $23.1$ | $42.9$ | $6$ | $120$ | $6.7$ | $41.5$ |
| qwen-2.5-omni | $29.1$ | $10$ | $49$ | $8.2$ | $29.0$ | $53.8$ | $11$ | $133$ | $23.7$ | $49.2$ | $89.3$ | $22$ | $228$ | $30.2$ | $80.4$ |
| MiniCPM-o-2.6 | $19.0$ | $6$ | $53$ | $5.3$ | $18.2$ | $29.8$ | $11$ | $58$ | $8.7$ | $28.6$ | $70.6$ | $34$ | $114$ | $13.8$ | $68.3$ |

#### F.2.3 Myriad Echoes

The full results for modality ablation on Myriad Echoes are presented in Table [10](https://arxiv.org/html/2508.04361v3#A6.T10).

##### Analysis of Ablation Conditions.

For this memory- and parsing-intensive task, the results show a more complex pattern. For the top-performing Gemini 2.5 Pro, removing any single modality leads to a severe drop in performance across all metrics (Success Rate, Mean Score, etc.), especially on higher difficulties. This underscores that its superhuman ability is contingent on successfully fusing information from the entire multi-modal stream. For other models, the performance is already very low in the baseline condition, making the impact of ablation less pronounced. However, we can observe that for weaker models, removing the audio or image modality can sometimes slightly reduce the ‘Parse Failure Rate’, suggesting that a simpler set of inputs, even if incomplete, is less likely to confuse their parsing mechanisms.

**Table 10: Full results for modality ablation on Myriad Echoes across all difficulties.**
|  | Easy Difficulty | Medium Difficulty | Hard Difficulty |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | Succ.(%) | M.Score | Coord. | Icon | ParseF(%) | Succ.(%) | M.Score | Coord. | Icon | ParseF(%) | Succ.(%) | M.Score | Coord. | Icon | ParseF(%) |
| Full Modality (Baseline) |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| MiniCPM-o-2.6 | 0 | $0.1$ | $0.2$ | $0.2$ | $50$ | 0 | $0.1$ | $0.1$ | $0.3$ | $20$ | 0 | 0 | 0 | 0 | $40$ |
| gemini-2.5-flash | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | $1.9$ | $4.5$ | $1.6$ | 0 |
| gemini-2.5-pro | $70$ | $4.7$ | $4.8$ | $4.8$ | 0 | $10$ | $4$ | $5.7$ | $5.6$ | 0 | $60$ | $10.2$ | $13.5$ | $13.5$ | 0 |
| qwen-2.5-omni | 0 | $0.1$ | $0.1$ | $0.3$ | $60$ | 0 | 0 | 0 | 0 | $50$ | 0 | 0 | 0 | $0.1$ | $60$ |
| Removed Audio |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| MiniCPM-o-2.6 | 0 | $0.3$ | $0.4$ | $0.2$ | $50$ | 0 | $0.1$ | $0.1$ | 0 | $20$ | 0 | 0 | 0 | $0.1$ | $40$ |
| gemini-2.5-flash | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | $0.3$ | $1.5$ | $1.5$ | 0 |
| gemini-2.5-pro | $70$ | $4.7$ | $5.8$ | $5.8$ | 0 | 0 | $1.9$ | $3.4$ | $3.4$ | 0 | $40$ | $8.8$ | $10.8$ | $10.8$ | 0 |
| qwen-2.5-omni | 0 | $0.3$ | $0.4$ | $0.4$ | $20$ | 0 | 0 | 0 | $0.1$ | $40$ | 0 | $0.2$ | $0.2$ | $0.1$ | $10$ |
| Removed Image |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| MiniCPM-o-2.6 | 0 | $0.3$ | $0.3$ | $0.1$ | $60$ | 0 | $0.2$ | $0.2$ | $0.1$ | $30$ | 0 | $0.1$ | $0.1$ | 0 | $20$ |
| gemini-2.5-flash | 0 | $0.2$ | $0.2$ | 0 | 0 | 0 | $0.3$ | $1$ | $1$ | 0 | 0 | $1.7$ | $3.1$ | $3.1$ | 0 |
| gemini-2.5-pro | $30$ | $3.2$ | $3.7$ | $3.7$ | 0 | $30$ | $5.1$ | $7.7$ | $7.7$ | 0 | $50$ | $9.1$ | $10.5$ | $10.5$ | 0 |
| qwen-2.5-omni | 0 | $0.3$ | $0.5$ | $0.6$ | $10$ | 0 | 0 | 0 | $0.3$ | $30$ | 0 | $0.1$ | $0.1$ | $0.2$ | $40$ |
| Removed Text |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| MiniCPM-o-2.6 | 0 | 0 | 0 | 0 | $60$ | 0 | $0.1$ | $0.1$ | 0 | $40$ | 0 | $0.1$ | $0.4$ | $0.4$ | 0 |
| gemini-2.5-flash | 0 | 0 | 0 | 0 | 0 | 0 | $0.2$ | $2$ | 0 | 0 | 0 | $0.1$ | $1.2$ | 0 | 0 |
| gemini-2.5-pro | 0 | $0.6$ | $3.6$ | $3.6$ | 0 | 0 | $1.1$ | $9$ | $9$ | 0 | 0 | $1$ | $15$ | $15$ | 0 |
| qwen-2.5-omni | 0 | 0 | 0 | 0 | $40$ | 0 | $0.3$ | $0.4$ | $0.2$ | $60$ | 0 | 0 | 0 | 0 | $50$ |

### F.3 Robustness to Noise

We investigated the models’ resilience to non-ideal sensory inputs by introducing noise into the visual and auditory modalities in the Phantom Soldiers in the Fog environment (Medium difficulty). The full results are presented in Table [11](https://arxiv.org/html/2508.04361v3#A6.T11).

**Table 11: Performance on Phantom Soldiers in the Fog (Medium) under different noise conditions.**
|  | Baseline (No Noise) | Audio Noise | Image Noise |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Model | Score | Succ. Rate | Score | Succ. Rate | Score | Succ. Rate |
| gemini-2.5-pro | $78.81$ | $0.880$ | $46.6$ | $0.850$ | $14.2$ | $0.275$ |
| gemini-2.5-flash | $31.20$ | $0.570$ | $20.5$ | $0.483$ | $2.9$ | $0.050$ |
| qwen-2.5-omni | $23.74$ | $0.465$ | $21.7$ | $0.533$ | 0 | 0 |
| MiniCPM-o-2.6 | $11.60$ | $0.200$ | $14.2$ | $0.290$ | $9.6$ | $0.185$ |

##### Audio Noise Injection and Analysis.

For the audio modality, our goal was to simulate a noisy communication channel. We achieved this by corrupting the transcribed tactical guidance text before it was synthesized into speech. Specifically, we randomly inserted meaningless ‘noise words’ (e.g., ‘zap’, ‘chirp’) and ‘noise letters’ (e.g., ‘w’, ‘b’) into the original guidance sentences, as illustrated in Figure [23](https://arxiv.org/html/2508.04361v3#A6.F23).

The results show that while audio noise does degrade performance, most models exhibit a degree of resilience. For example, Gemini 2.5 Pro’s score dropped from 78.81 to 46.6, a significant but not catastrophic decrease. Interestingly, MiniCPM-o-2_6’s performance slightly improved, which may suggest that its baseline audio processing is already suboptimal, and the noise did not significantly alter its behavior. Overall, the impact of this semantic-level audio noise was less severe than the visual noise.

Figure: Figure 23: Illustration of audio noise injection via corruption of the source text in Phantom Soldiers in the Fog.
Refer to caption: x29.png

##### Image Noise Injection and Analysis.

For the visual modality, we simulated sensor degradation or poor visibility. We applied a combination of three common types of image noise to the top-down video feed: Gaussian noise, salt-and-pepper noise, and a slight blurring filter. The effect of this transformation is shown in Figure [24](https://arxiv.org/html/2508.04361v3#A6.F24).

Figure: Figure 24: Illustration of visual noise injection in Phantom Soldiers in the Fog.
Refer to caption: x30.png

The impact of visual noise was far more severe and universally detrimental. As seen in Table [11](https://arxiv.org/html/2508.04361v3#A6.T11), all models experienced a catastrophic drop in performance. The top-performing Gemini 2.5 Pro was the most fragile, with its score plummeting by over 80% from 78.81 to 14.2. Gemini 2.5 Flash’s score dropped by nearly 90%, and qwen-2.5-omni’s performance collapsed entirely to zero. This extreme fragility strongly supports our main-text conclusion that the models may be learning superficial statistical correlations from visual data rather than robust, semantically-grounded representations. Their inability to handle minor pixel-level perturbations reveals a critical lack of generalization and robustness in their visual processing pipeline.

### F.4 Aided Reasoning via Prompting

In contrast to penalizing models with noisy or conflicting data, this set of experiments investigates their ability to leverage helpful, explicit guidance provided directly within the prompt. We selected Myriad Echoes and The Alchemist’s Melody for this analysis.

##### Myriad Echoes.

In the standard version of this task, the agent must internally track its progress through the execution sequence. In the ‘Aided Reasoning’ version, we augmented the Turn Prompt for Phase 2 with an explicit status update, telling the agent which step of the sequence it was currently on, as shown in Figure [25](https://arxiv.org/html/2508.04361v3#A6.F25). The goal was to test if this explicit state information could reduce errors in long-sequence execution.

Figure: Figure 25: Augmented Turn Prompt for Myriad Echoes, providing the agent with its current step in the sequence.
Refer to caption: x31.png

The results, presented in Table [12](https://arxiv.org/html/2508.04361v3#A6.T12), show that this hint significantly benefited the top-performing models. For instance, Gemini 2.5 Pro’s success rate on the Hard difficulty increased from 60% to 80%, and its mean score improved. In contrast, weaker models were unable to effectively utilize this information, with their performance remaining largely unchanged.

**Table 12: Performance comparison on Myriad Echoes with and without aided reasoning prompts.**
|  | Easy Difficulty | Medium Difficulty | Hard Difficulty |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | Succ.(%) | M.Score | Coord. | Icon | ParseF(%) | Succ.(%) | M.Score | Coord. | Icon | ParseF(%) | Succ.(%) | M.Score | Coord. | Icon | ParseF(%) |
| Baseline |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| gemini-2.5-pro | $70$ | $4.7$ | $4.8$ | $4.8$ | 0 | $10$ | $4.0$ | $5.7$ | $5.6$ | 0 | $60$ | $10.2$ | $13.5$ | $13.5$ | 0 |
| gemini-2.5-flash | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | $1.9$ | $4.5$ | $1.6$ | 0 |
| qwen-2.5-omni | 0 | $0.1$ | $0.1$ | $0.3$ | $60$ | 0 | 0 | 0 | 0 | $50$ | 0 | 0 | 0 | $0.1$ | $60$ |
| MiniCPM-o-2.6 | 0 | $0.1$ | $0.2$ | $0.2$ | $50$ | 0 | $0.1$ | $0.1$ | $0.3$ | $20$ | 0 | 0 | 0 | 0 | $40$ |
| With Aided Prompt |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| gemini-2.5-pro | $90$ | $5.5$ | $5.5$ | $5.5$ | 0 | $70$ | $7.0$ | $7.0$ | $7.0$ | 0 | $80$ | $12.0$ | $12.0$ | $10.7$ | 0 |
| gemini-2.5-flash | 0 | 0 | 0 | 0 | 0 | $10$ | $1.0$ | $1.0$ | $1.0$ | 0 | 0 | $1.0$ | $1.0$ | $1.5$ | 0 |
| qwen-2.5-omni | 0 | $0.5$ | $0.3$ | 0 | $60$ | 0 | $0.4$ | $0.3$ | $0.2$ | $40$ | 0 | 0 | 0 | 0 | $60$ |
| MiniCPM-o-2.6 | 0 | $0.2$ | $0.4$ | $0.3$ | $20$ | 0 | $0.2$ | $0.2$ | $0.2$ | 0 | 0 | 0 | 0 | 0 | $30$ |

##### The Alchemist’s Melody.

This game requires the agent to deduce a color-note mapping. In the ‘Aided Reasoning’ condition, we made this task significantly easier by directly providing the agent with its currently learned ‘color-note mapping’ within the Turn Prompt, as shown in Figure [26](https://arxiv.org/html/2508.04361v3#A6.F26). This tests the agent’s ability to directly apply provided knowledge rather than discover it.

Figure: Figure 26: Augmented Turn Prompt for The Alchemist’s Melody, providing the agent with its learned color-note mapping.
Refer to caption: x32.png

The results in Table [13](https://arxiv.org/html/2508.04361v3#A6.T13) are striking. The proprietary models, Gemini 2.5 Pro and Flash, demonstrated a remarkable ability to utilize this hint, with their completion rates jumping from 20% and 0% to a perfect 100%. Their scores improved dramatically accordingly. In stark contrast, all tested open-source models failed to leverage this explicit information, showing no significant improvement in performance. This highlights a significant gap in advanced instruction-following and knowledge application capabilities between proprietary and open-source models.

**Table 13: Performance comparison on The Alchemist’s Melody with and without aided reasoning prompts.**
|  | Baseline | With Aided Prompt |  |  |
| --- | --- | --- | --- | --- |
| Model | Score | Comp. Rate | Score | Comp. Rate |
| gemini-2.5-pro | $43.154$ | $20$% | $73.104$ | $100$% |
| gemini-2.5-flash | $32.048$ | 0% | $62.096$ | $100$% |
| MiniCPM-o-2.6 | $30.294$ | 0% | $32.798$ | 0% |
| VITA-1.5 | $20.010$ | 0% | $18.896$ | 0% |
| Baichuan-Omni-1.5 | $31.823$ | 0% | $33.548$ | 0% |
| qwen-2.5-omni | $31.234$ | 0% | $32.722$ | 0% |

### F.5 Task Simplification

To validate the designed complexity of our benchmark, we conducted a task simplification experiment on Myriad Echoes. The goal was to understand if the primary difficulty lay in the multi-modal parsing phase or the long-sequence execution phase.

##### Methodology.

We modified the original two-phase task into a single-phase perception task. In this simplified version, the agent still observes the full multi-modal sequence (video and audio) as in Phase 1. However, instead of proceeding to a second execution phase, the agent’s task is to directly output the symbolic sequence it perceived. The performance is then measured by a final score which is a weighted average of the coordinate accuracy (50%) and the icon accuracy (50%) of its output.

##### Results.

The results of this experiment are presented in Table [14](https://arxiv.org/html/2508.04361v3#A6.T14), compared against the baseline performance on the original task. As expected, all models showed performance gains on the simplified task, as it removes the challenging long-horizon execution and action grounding components. However, even on this simplified perception-only task, the weaker open-source models still struggled to achieve high accuracy, particularly on the Hard difficulty. This confirms that the benchmark’s core challenges are substantial and distributed across both its perception and action phases.

**Table 14: Performance comparison on Myriad Echoes between the original task and the simplified (perception-only) task.**
|  | Easy Difficulty | Medium Difficulty | Hard Difficulty |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | Succ.(%) | M.Score | Coord. | Icon | ParseF(%) | Succ.(%) | M.Score | Coord. | Icon | ParseF(%) | Succ.(%) | M.Score | Coord. | Icon | ParseF(%) |
| Original Task (Baseline) |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| gemini-2.5-pro | $70$ | $4.70$ | $4.8$ | $4.8$ | 0 | $10$ | $4.00$ | $5.7$ | $5.6$ | 0 | $60$ | $10.20$ | $13.5$ | $13.5$ | 0 |
| gemini-2.5-flash | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | $1.90$ | $4.5$ | $1.6$ | 0 |
| qwen-2.5-omni | 0 | $0.10$ | $0.1$ | $0.3$ | $60$ | 0 | 0 | 0 | 0 | $50$ | 0 | 0 | 0 | $0.1$ | $60$ |
| MiniCPM-o-2.6 | 0 | $0.10$ | $0.2$ | $0.2$ | $50$ | 0 | $0.10$ | $0.1$ | $0.3$ | $20$ | 0 | 0 | 0 | 0 | $40$ |
| Simplified Task |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| gemini-2.5-pro | $90$ | $5.60$ | $5.6$ | $5.6$ | 0 | $60$ | $6.00$ | $6.0$ | $6.0$ | 0 | $70$ | $10.60$ | $10.6$ | $10.6$ | 0 |
| gemini-2.5-flash | 0 | 0 | 0 | 0 | 0 | $10$ | $1.00$ | $1.0$ | $1.0$ | 0 | $10$ | $1.55$ | $1.5$ | $1.6$ | 0 |
| qwen-2.5-omni | 0 | $0.25$ | $0.1$ | $0.4$ | $50$ | 0 | $0.20$ | $0.2$ | $0.2$ | $50$ | 0 | $0.05$ | 0 | $0.1$ | $70$ |
| MiniCPM-o-2.6 | 0 | $0.10$ | $0.1$ | $0.1$ | $30$ | 0 | 0 | 0 | 0 | $10$ | 0 | 0 | 0 | $0.4$ | $10$ |

### F.6 Modality Substitution

This final diagnostic experiment investigates the models’ ability to generalize across different modality representations of the same semantic information. Specifically, we tested if agents perform better when complex information is presented as structured text versus synthesized audio.

##### Methodology.

We used the Phantom Soldiers in the Fog environment (Medium difficulty) for this experiment. In the baseline condition, the agent receives tactical guidance via the audio channel (as Text-to-Speech). In the ‘Modality Substitution’ condition, we disabled the audio channel entirely. Instead, the exact same structured textual guidance that would have been converted to speech was appended directly to the main text prompt. The agent’s task was then to complete the mission using only the visual (video) and augmented textual modalities.

##### Results.

The results, presented in Table [15](https://arxiv.org/html/2508.04361v3#A6.T15), reveal a strong and consistent trend. Most models, particularly the high-performing proprietary ones, showed a significant performance increase when the auditory information was substituted with its textual equivalent. For example, Gemini 2.5 Pro’s score improved from 78.81 to 86.78, and Gemini 2.5 Flash’s score more than doubled from 31.2 to 70.2. This unexpected result reinforces our core finding about brittle fusion for non-textual information. It suggests that for current models, well-structured and unambiguous text is a more reliable and easier-to-process source of information than synthesized audio, even when the underlying semantic content is identical. The performance drop for MiniCPM-o-2_6 is an anomaly that warrants further investigation, but may point to architectural differences in how it handles combined textual inputs.

**Table 15: Performance comparison on Phantom Soldiers in the Fog (Medium) with and without Modality Substitution.**
|  | Baseline | Substituted |  |  |
| --- | --- | --- | --- | --- |
|  | (Video+Text+Audio) | (Video+Text only) |  |  |
| Model | Score | Succ. Rate | Score | Succ. Rate |
| gemini-2.5-pro | $78.81$ | $0.880$ | $86.78$ | $0.920$ |
| gemini-2.5-flash | $31.20$ | $0.570$ | $70.20$ | $0.860$ |
| MiniCPM-o-2.6 | $11.60$ | $0.200$ | $1.80$ | $0.035$ |
| qwen-2.5-omni | $23.74$ | $0.465$ | $25.90$ | $0.470$ |

## Appendix G Qualitative Case Studies

To provide deeper, qualitative insights into the quantitative results presented in the main text, this section presents detailed case studies for three noteworthy phenomena observed during our evaluation.

### G.1 Case Study: Superhuman Memory in Myriad Echoes

##### Phenomenon.

As noted in the main text, top-tier models like Gemini 2.5 Pro exhibit superhuman performance on the Myriad Echoes task, particularly on Hard difficulty where the sequence length is long. This case study analyzes the cognitive and architectural differences between the AI agent and a human player that lead to this performance gap.

##### Analysis.

The core challenge of Myriad Echoes is twofold: high-bandwidth, cross-modal information encoding followed by precise, long-sequence symbolic execution. As illustrated in Figure [27](https://arxiv.org/html/2508.04361v3#A7.F27), the model is presented with a rapid, lengthy sequence of icon-sound pairs.

- •
Model’s Advantage: A large omni-modal model like Gemini 2.5 Pro functions as a near-perfect information transducer. Its vast parameter space and attention mechanisms allow it to faithfully transcribe the high-throughput audio-visual stream into a precise internal symbolic representation with minimal loss. In the execution phase, it can recall and act upon this long sequence with near-perfect accuracy, as its ‘working memory’ is not biologically constrained.
- •
Human’s Limitation: In contrast, a human player’s performance is fundamentally limited by the capacity of their cognitive working memory (typically cited as 7±2 items). It is cognitively impossible for a human to perfectly memorize a rapid sequence of 10 or more arbitrary audio-visual pairs. Humans must resort to chunking or other heuristics, which are prone to error and forgetting, leading to a much lower performance ceiling.

Conclusion: This task highlights a domain where current AI excels: high-fidelity, short-term memory and precise symbolic manipulation. The observed superhuman performance is an expected outcome of the architectural differences between the model and the human brain, rather than an indication of superior general reasoning.

Figure: Figure 27: Illustration of the performance gap in Myriad Echoes. The AI agent (top) can perfectly recall and transcribe the long, complex audio-visual sequence. The human player (bottom) is limited by their working memory and cannot reliably recall the full sequence.
Refer to caption: x33.png

### G.2 Case Study: Anomalous Winning Strategy in Blasting Showdown

##### Phenomenon.

During the AI-vs-AI tournament in Blasting Showdown, we observed cases where MiniCPM-o-2_6 won matches despite having a Kill/Death (K/D) ratio of zero. This case study examines the unusual, passive strategy that led to this counter-intuitive success.

##### Analysis.

The model’s winning strategy can be characterized as extreme risk aversion or passive survival. Figure [28](https://arxiv.org/html/2508.04361v3#A7.F28) depicts a representative match.

- •
Observed Behavior: Throughout the match, MiniCPM-o-2_6 (represented by the red player, Player 1) exhibited a very low tendency to place bombs or engage opponents. Its primary behavior consisted of reactive movements to evade bombs placed by other, more aggressive agents.
- •
Environmental Dynamics: The other three agents (Players 2, 3, and 4) actively engaged in combat. This created a chaotic and dangerous environment where players were eliminated not just by direct attacks, but also by chain reactions, self-elimination (getting trapped by their own bomb), or being caught in crossfire. In the depicted sequence, Player 2 eliminates Players 4 and 3, but then accidentally traps and eliminates itself.
- •
Attribution: It is unlikely that the model devised a sophisticated, deliberate strategy of ‘waiting out the storm’. A more plausible explanation is that its capacity for proactive, strategic planning is underdeveloped, causing it to default to the simplest possible policy: stay alive by avoiding immediate threats. In the chaotic context of a 4-player free-for-all, this simple, passive policy coincidentally proved to be highly effective.

Conclusion: This case study is a crucial reminder that in complex multi-agent systems, a successful outcome does not necessarily imply intelligent strategy. It highlights the importance of analyzing an agent’s behavioral traces, not just its win rate, to accurately assess its planning and reasoning capabilities.

Figure: Figure 28: A step-by-step replay of a match won by MiniCPM-o-2_6 (Player 1, red). Player 1 remains passive while the other agents eliminate each other through aggressive play and miscalculation, leading to an accidental victory.
Refer to caption: x34.png

### G.3 Case Study: Systematic Failure in Myriad Echoes

##### Phenomenon.

A peculiar and consistent failure mode was observed for Gemini 2.5 Flash in the Myriad Echoes task. On Easy and Medium difficulties, its performance on all sequence-related metrics was consistently zero.

##### Analysis.

The root cause of this total failure is a systematic off-by-one error in its sequence generation. Figure [29](https://arxiv.org/html/2508.04361v3#A7.F29) provides a clear example.

- •
The Task: The agent is presented with a true sequence of a specific length (e.g., 10 items in the example).
- •
The Error: When prompted to reproduce the sequence, Gemini 2.5 Flash consistently outputs a sequence that is correct in content and relative order, but is missing the first element. It always generates a sequence of length N-1 when the correct length is N.
- •
Attribution: This behavior points to a subtle but critical flaw in how the model handles sequence boundaries or follows length constraints. It is a classic example of a format following failure. The model understands the core task of identifying the items, but fails on the crucial meta-task of adhering to the sequence’s structural integrity (in this case, its length).

Conclusion: This case demonstrates that even highly capable models can harbor specific, systematic bugs in their reasoning or generation processes. It highlights the value of diagnostic benchmarks like OmniPlay, which can surface these otherwise hidden, granular failure modes that would be missed by evaluations that only measure average performance. For tasks requiring high precision, such a systematic error is a critical failure.

Figure: Figure 29: An example of Gemini 2.5 Flash’s systematic ‘off-by-one’ error. The model correctly identifies most of the sequence but consistently omits the first element, resulting in a complete task failure.
Refer to caption: x35.png

## Appendix H Detailed Statistical Results for Diagnostic Experiments

This section provides the full statistical data for the diagnostic experiments presented in Section 5.2, including the modality conflict and modality ablation studies. All experiments were conducted over N=50 independent runs to ensure statistical robustness.

### H.1 Modality Conflict (Supporting Figure 4(b) )

Table [16](https://arxiv.org/html/2508.04361v3#A8.T16) contains the detailed statistical results for the modality conflict experiment performed on Whispered Pathfinding (Hard). These results are visualized in Figure [4(b)](https://arxiv.org/html/2508.04361v3#S5.F4.sf2) in the main text. We report the mean efficiency score, the standard deviation (SD) to show the raw performance volatility, and the standard error of the mean (SEM) used for generating the error margins in the figure.

**Table 16: Full statistical results for the Modality Conflict experiment on Whispered Pathfinding (Hard). The data corresponds to Figure [4(b)](https://arxiv.org/html/2508.04361v3#S5.F4.sf2). All metrics are based on N=50 runs.**
| Model | Condition | Mean Score (%) | SD | SEM |
| --- | --- | --- | --- | --- |
| Gemini 2.5 Pro | No Conflict | $89.4$ | $5.7$ | $0.8$ |
|  | Audio Conflict | $43.3$ | $16.3$ | $2.3$ |
|  | Text Conflict | $32.2$ | $17.7$ | $2.5$ |
| Gemini 2.5 Flash | No Conflict | $89.0$ | $6.4$ | $0.9$ |
|  | Audio Conflict | $88.6$ | $7.1$ | $1.0$ |
|  | Text Conflict | $48.1$ | $20.5$ | $2.9$ |
| Qwen-2.5-Omni (7B) | No Conflict | $45.9$ | $22.6$ | $3.2$ |
|  | Audio Conflict | $0.0$ | $0.0$ | $0.0$ |
|  | Text Conflict | $37.6$ | $24.0$ | $3.4$ |
| MiniCPM-o-2.6 (8B) | No Conflict | $59.5$ | $19.1$ | $2.7$ |
|  | Audio Conflict | $32.2$ | $24.7$ | $3.5$ |
|  | Text Conflict | $56.8$ | $21.2$ | $3.0$ |

### H.2 Modality Ablation (Supporting Figure 5 )

Table [17](https://arxiv.org/html/2508.04361v3#A8.T17) provides the detailed statistical results for the modality ablation experiment, corresponding to Figure [5](https://arxiv.org/html/2508.04361v3#S5.F5) in the main text. The experiment was conducted on the ’Hard’ difficulty for two distinct tasks: Whispered Pathfinding and Myriad Echoes. All metrics are based on N=50 independent runs.

**Table 17: Full statistical results for the Modality Ablation experiment. The data corresponds to Figure [5](https://arxiv.org/html/2508.04361v3#S5.F5).**
| Task | Model | Condition | Mean Score | SD | SEM |
| --- | --- | --- | --- | --- | --- |
| Whispered Pathfinding<br>(Efficiency Score, %) | Gemini 2.5 Pro | Full Modality | $86.7$ | $8.5$ | $1.2$ |
|  | Removed Audio | $48.9$ | $17.0$ | $2.4$ |  |
|  | Removed Image | $80.8$ | $9.9$ | $1.4$ |  |
|  | Removed Text | $37.2$ | $18.4$ | $2.6$ |  |
| Gemini 2.5 Flash | Full Modality | $86.0$ | $9.2$ | $1.3$ |  |
|  | Removed Audio | $79.4$ | $12.0$ | $1.7$ |  |
|  | Removed Image | $55.5$ | $17.7$ | $2.5$ |  |
|  | Removed Text | $83.5$ | $10.6$ | $1.5$ |  |
| Qwen-2.5-Omni (7B) | Full Modality | $31.6$ | $21.2$ | $3.0$ |  |
|  | Removed Audio | $0.0$ | $0.0$ | $0.0$ |  |
|  | Removed Image | $27.0$ | $22.6$ | $3.2$ |  |
|  | Removed Text | $60.2$ | $24.0$ | $3.4$ |  |
| MiniCPM-o-2.6 (8B) | Full Modality | $48.8$ | $25.5$ | $3.6$ |  |
|  | Removed Audio | $47.0$ | $26.9$ | $3.8$ |  |
|  | Removed Image | $81.4$ | $11.3$ | $1.6$ |  |
|  | Removed Text | $67.5$ | $21.9$ | $3.1$ |  |
| Myriad Echoes<br>(Weighted Score) | Gemini 2.5 Pro | Full Modality | $11.85$ | $1.50$ | $0.21$ |
|  | Removed Audio | $9.80$ | $2.12$ | $0.30$ |  |
|  | Removed Image | $9.80$ | $2.12$ | $0.30$ |  |
|  | Removed Text | $8.00$ | $2.47$ | $0.35$ |  |
| Gemini 2.5 Flash | Full Modality | $2.48$ | $3.18$ | $0.45$ |  |
|  | Removed Audio | $0.90$ | $1.77$ | $0.25$ |  |
|  | Removed Image | $2.38$ | $2.97$ | $0.42$ |  |
|  | Removed Text | $0.35$ | $1.13$ | $0.16$ |  |
| Qwen-2.5-Omni (7B) | Full Modality | $0.03$ | $0.14$ | $0.02$ |  |
|  | Removed Audio | $0.18$ | $0.42$ | $0.06$ |  |
|  | Removed Image | $0.13$ | $0.35$ | $0.05$ |  |
|  | Removed Text | $0.00$ | $0.00$ | $0.00$ |  |
| MiniCPM-o-2.6 (8B) | Full Modality | $0.00$ | $0.00$ | $0.00$ |  |
|  | Removed Audio | $0.03$ | $0.14$ | $0.02$ |  |
|  | Removed Image | $0.08$ | $0.28$ | $0.04$ |  |
|  | Removed Text | $0.40$ | $0.99$ | $0.14$ |  |

## Appendix I Full Performance Results

This appendix provides the complete, unabridged performance data for all models and baselines across all tasks and difficulty levels from our main evaluation. We first present the summary statistics for all NPS-benchmarked tasks, which directly support the main findings in Section [5.1](https://arxiv.org/html/2508.04361v3#S5.SS1). Following this, we provide the detailed, task-specific raw metrics for each game environment.

### I.1 Summary of NPS-Benchmarked Tasks

Table [18](https://arxiv.org/html/2508.04361v3#A9.T18) provides the complete statistical data corresponding to the results visualized in Figure [3(a)](https://arxiv.org/html/2508.04361v3#S5.F3.sf1) and summarized in Table [2](https://arxiv.org/html/2508.04361v3#S5.T2). For each model and task, we conducted 50 independent runs with different random seeds. We report the Mean Normalized Performance Score (NPS), the Standard Deviation (SD) to show performance volatility, and the Standard Error of the Mean (SEM) to indicate the confidence in our estimation of the mean.

**Table 18: Full statistical results for all NPS-benchmarked tasks (N=50 runs). Data is presented as Mean NPS.**
| Task (Grouped by Game) | Model | Mean NPS | Standard Deviation (SD) | Standard Error (SEM) |
| --- | --- | --- | --- | --- |
| Whispered Pathfinding (Easy) | Gemini 2.5 Pro | 98.2 | 4.2 | 0.6 |
|  | Gemini 2.5 Flash | 95.9 | 7.1 | 1.0 |
|  | Qwen-2.5-Omni | 66.6 | 21.2 | 3.0 |
|  | MiniCPM-o-2.6 | 86.8 | 15.6 | 2.2 |
|  | Baichuan-Omni-1.5 | 88.0 | 14.1 | 2.0 |
|  | VITA-1.5 | 78.6 | 17.7 | 2.5 |
| Whispered Pathfinding (Medium) | Gemini 2.5 Pro | 99.2 | 3.5 | 0.5 |
|  | Gemini 2.5 Flash | 95.7 | 8.5 | 1.2 |
|  | Qwen-2.5-Omni | 78.6 | 19.8 | 2.8 |
|  | MiniCPM-o-2.6 | 90.8 | 12.0 | 1.7 |
|  | Baichuan-Omni-1.5 | 93.2 | 9.9 | 1.4 |
|  | VITA-1.5 | 84.4 | 16.3 | 2.3 |
| Whispered Pathfinding (Hard) | Gemini 2.5 Pro | 95.2 | 8.5 | 1.2 |
|  | Gemini 2.5 Flash | 95.0 | 9.2 | 1.3 |
|  | Qwen-2.5-Omni | 75.6 | 23.3 | 3.3 |
|  | MiniCPM-o-2.6 | 81.7 | 18.4 | 2.6 |
|  | Baichuan-Omni-1.5 | 85.0 | 16.3 | 2.3 |
|  | VITA-1.5 | 82.7 | 19.1 | 2.7 |
| Myriad Echoes (Easy) | Gemini 2.5 Pro | 114.1 | 12.0 | 1.7 |
|  | Gemini 2.5 Flash | -7.7 | 31.8 | 4.5 |
|  | Qwen-2.5-Omni | -3.8 | 36.1 | 5.1 |
|  | MiniCPM-o-2.6 | -3.8 | 38.2 | 5.4 |
|  | Baichuan-Omni-1.5 | -4.5 | 37.5 | 5.3 |
|  | VITA-1.5 | -6.1 | 41.0 | 5.8 |
| Myriad Echoes (Medium) | Gemini 2.5 Pro | 157.0 | 18.4 | 2.6 |
|  | Gemini 2.5 Flash | -2.5 | 33.2 | 4.7 |
|  | Qwen-2.5-Omni | -2.5 | 39.6 | 5.6 |
|  | MiniCPM-o-2.6 | 2.5 | 43.1 | 6.1 |
|  | Baichuan-Omni-1.5 | 0.0 | 40.3 | 5.7 |
|  | VITA-1.5 | -2.3 | 44.5 | 6.3 |
| Myriad Echoes (Hard) | Gemini 2.5 Pro | 399.2 | 25.5 | 3.6 |
|  | Gemini 2.5 Flash | 81.4 | 29.7 | 4.2 |
|  | Qwen-2.5-Omni | -1.7 | 34.6 | 4.9 |
|  | MiniCPM-o-2.6 | -2.5 | 38.9 | 5.5 |
|  | Baichuan-Omni-1.5 | -2.5 | 37.5 | 5.3 |
|  | VITA-1.5 | -2.5 | 41.7 | 5.9 |
| The Alchemist’s Melody (Default) | Gemini 2.5 Pro | 28.4 | 33.2 | 4.7 |
|  | Gemini 2.5 Flash | 10.5 | 38.9 | 5.5 |
|  | Qwen-2.5-Omni | 9.2 | 41.0 | 5.8 |
|  | MiniCPM-o-2.6 | 7.7 | 43.1 | 6.1 |
|  | Baichuan-Omni-1.5 | 10.2 | 39.6 | 5.6 |
|  | VITA-1.5 | -8.9 | 48.1 | 6.8 |
| Phantom Soldiers (Easy) | Gemini 2.5 Pro | 88.6 | 19.8 | 2.8 |
|  | Gemini 2.5 Flash | 86.5 | 22.6 | 3.2 |
|  | Qwen-2.5-Omni | -25.6 | 45.2 | 6.4 |
|  | MiniCPM-o-2.6 | -28.4 | 48.1 | 6.8 |
|  | Baichuan-Omni-1.5 | 16.5 | 38.9 | 5.5 |
|  | VITA-1.5 | -38.1 | 53.0 | 7.5 |
| Phantom Soldiers (Medium) | Gemini 2.5 Pro | 73.6 | 26.9 | 3.8 |
|  | Gemini 2.5 Flash | 6.3 | 36.8 | 5.2 |
|  | Qwen-2.5-Omni | -9.1 | 43.1 | 6.1 |
|  | MiniCPM-o-2.6 | -42.2 | 50.9 | 7.2 |
|  | Baichuan-Omni-1.5 | -35.4 | 49.5 | 7.0 |
|  | VITA-1.5 | -69.3 | 35.4 | 5.0 |
| Phantom Soldiers (Hard) | Gemini 2.5 Pro | 87.5 | 24.7 | 3.5 |
|  | Gemini 2.5 Flash | 54.5 | 33.2 | 4.7 |
|  | Qwen-2.5-Omni | 11.2 | 41.0 | 5.8 |
|  | MiniCPM-o-2.6 | -21.5 | 49.5 | 7.0 |
|  | Baichuan-Omni-1.5 | 8.3 | 43.8 | 6.2 |
|  | VITA-1.5 | -49.2 | 42.4 | 6.0 |

**Table 19: Summary of Human Expert Performance (N=12 participants). Data is presented as the mean raw score used for NPS calculation, along with the standard deviation (SD) and standard error of the mean (SEM) measuring inter-player variability. Scoring rules are detailed in Appendix [D](https://arxiv.org/html/2508.04361v3#A4).**
| Task (Grouped by Game) | Mean Raw Score | SD (Inter-Player) | SEM (Inter-Player) |
| --- | --- | --- | --- |
| Whispered Pathfinding (Easy) | $0.20$ | $0.03$ | $0.009$ |
| Whispered Pathfinding (Medium) | $0.13$ | $0.02$ | $0.006$ |
| Whispered Pathfinding (Hard) | $0.07$ | $0.01$ | $0.003$ |
| Myriad Echoes (Easy) | $4.20$ | $0.38$ | $0.11$ |
| Myriad Echoes (Medium) | $3.10$ | $0.28$ | $0.08$ |
| Myriad Echoes (Hard) | $3.03$ | $0.36$ | $0.10$ |
| The Alchemist’s Melody (Default) | $87.66$ | $6.14$ | $1.77$ |
| Phantom Soldiers (Easy) | $100.00$ | $4.50$ | $1.30$ |
| Phantom Soldiers (Medium) | $98.80$ | $9.39$ | $2.71$ |
| Phantom Soldiers (Hard) | $96.75$ | $12.58$ | $3.63$ |

### I.2 Statistical Analysis of Win Rates in Blasting Showdown

Table [20](https://arxiv.org/html/2508.04361v3#A9.T20) provides the statistical analysis for the win rates reported in Figure 4. The analysis is based on the outcomes of a 50-game tournament. We report the raw number of wins, the mean win rate (p), the standard deviation (SD) calculated as $\sqrt{p(1-p)}$, and the standard error of the mean (SEM) calculated as $SD/\sqrt{N}$, where N=50.

**Table 20: Statistical analysis of win rates for the AI-vs-AI evaluation on Blasting Showdown (N=50 games).**
| Model | Wins / Total | Win Rate (%) | SD | SEM (%) |
| --- | --- | --- | --- | --- |
| Gemini 2.5 Pro | 18 / 50 | $36.1$% | $0.480$ | $6.8$% |
| Gemini 2.5 Flash | 14 / 50 | $28.9$% | $0.453$ | $6.4$% |
| MiniCPM-o-2.6 | 10 / 50 | $19.4$% | $0.395$ | $5.6$% |
| Baichuan-Omni-1.5 | 9 / 50 | $17.7$% | $0.382$ | $5.4$% |
| Qwen-2.5-Omni | 6 / 50 | $11.8$% | $0.323$ | $4.6$% |
| VITA-1.5 | 4 / 50 | $7.4$% | $0.262$ | $3.7$% |

### I.3 Task-Specific Raw Metrics: Whispered Pathfinding

Table [21](https://arxiv.org/html/2508.04361v3#A9.T21) presents the detailed performance metrics for the Whispered Pathfinding task. The primary metric for this navigation task is ‘Mean Steps’, where a lower value indicates better performance. We also report the trimmed mean, which is less sensitive to outliers.

**Table 21: Full performance results for Whispered Pathfinding across all difficulties.**
| Model | Easy Difficulty | Medium Difficulty | Hard Difficulty |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Mean | Min | Max | Inv. | Trim. | Mean | Min | Max | Inv. | Trim. | Mean | Min | Max | Inv. | Trim. |
| human | $5.2$ | $3$ | $8$ | $0.0$ | $5.1$ | $8.3$ | $6$ | $10$ | $0.0$ | $8.0$ | $15.6$ | $10$ | $27$ | $0.0$ | $13.9$ |
| gemini-2.5-pro | $7.6$ | $5$ | $10$ | $0.0$ | $7.6$ | $10.2$ | $7$ | $14$ | $0.0$ | $10.1$ | $42.6$ | $13$ | $152$ | $0.0$ | $36.2$ |
| gemini-2.5-flash | $16.1$ | $4$ | $70$ | $0.8$ | $10.9$ | $23.2$ | $6$ | $87$ | $1.2$ | $19.0$ | $43.5$ | $18$ | $112$ | $2.7$ | $37.15$ |
| qwen-2.5-omni | $70.3$ | $10$ | $273$ | $29.5$ | $52.5$ | $64.2$ | $11$ | $132$ | $28.1$ | $62.4$ | $130.1$ | $32$ | $253$ | $52.6$ | $128.2$ |
| MiniCPM-o-2.6 | $27.0$ | $7$ | $73$ | $7.5$ | $23.8$ | $34.4$ | $6$ | $86$ | $10.7$ | $31.5$ | $110.8$ | $34$ | $255$ | $35.8$ | $99.5$ |
| VITA-1.5 | $36.1$ | $13$ | $70$ | $15.0$ | $35.5$ | $52.8$ | $8$ | $162$ | $21.9$ | $47.8$ | $106.5$ | $23$ | $343$ | $35.7$ | $94.8$ |
| Baichuan-Omni-1.5 | $23.1$ | $11$ | $47$ | $6.3$ | $22.2$ | $31.0$ | $12$ | $67$ | $7.8$ | $25.3$ | $89.3$ | $27$ | $236$ | $20.2$ | $84.7$ |
| random | $193.2$ | $25$ | $500$ | $0.0$ | $147.0$ | $277.8$ | $125$ | $477$ | $0.0$ | $262.3$ | $413.4$ | $119$ | $500$ | $0.0$ | $482.7$ |

### I.4 Task-Specific Raw Metrics: Myriad Echoes

Table [22](https://arxiv.org/html/2508.04361v3#A9.T22) presents the detailed performance metrics for the Myriad Echoes task. This task assesses both multi-modal parsing (Coord. Acc., Icon Acc.) and execution (Mean Score).

**Table 22: Full performance results for Myriad Echoes across all difficulties.**
| Model | Easy Difficulty | Medium Difficulty | Hard Difficulty |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Succ(%) | M.Score | Coord. | Icon | ParseF(%) | Succ(%) | M.Score | Coord. | Icon | ParseF(%) | Succ(%) | M.Score | Coord. | Icon | ParseF(%) |
| human | - | $3.70$ | $3.60$ | $5.80$ | - | - | $2.50$ | $2.60$ | $4.80$ | - | - | $2.60$ | $2.30$ | $4.60$ | - |
| gemini-2.5-pro | $70$ | $4.70$ | $4.80$ | $4.80$ | 0 | $10$ | $4.00$ | $5.70$ | $5.60$ | 0 | $60$ | $10.20$ | $13.50$ | $13.50$ | 0 |
| gemini-2.5-flash | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | $1.90$ | $4.50$ | $1.60$ | 0 |
| qwen-2.5-omni | 0 | $0.10$ | $0.10$ | $0.30$ | $60$ | 0 | 0 | 0 | 0 | $50$ | 0 | 0 | 0 | $0.10$ | $60$ |
| MiniCPM-o-2.6 | 0 | $0.10$ | $0.20$ | $0.20$ | $50$ | 0 | $0.10$ | $0.10$ | $0.30$ | $20$ | 0 | 0 | 0 | 0 | $40$ |
| VITA-1.5 | 0 | $0.10$ | $0.05$ | 0 | 0 | 0 | 0 | 0 | $0.02$ | $20$ | 0 | 0 | 0 | 0 | $30$ |
| Baichuan-Omni-1.5 | 0 | $0.20$ | $0.10$ | 0 | $50$ | 0 | $0.10$ | $0.10$ | 0 | $70$ | 0 | 0 | 0 | 0 | $60$ |
| random | 0 | $0.55$ | $0.08$ | $0.02$ | 0 | 0 | $0.05$ | $0.05$ | $0.15$ | 0 | 0 | $0.10$ | $0.07$ | $0.03$ | 0 |

### I.5 Task-Specific Raw Metrics: The Alchemist’s Melody

Table [23](https://arxiv.org/html/2508.04361v3#A9.T23) presents the detailed performance metrics for the The Alchemist’s Melody task.

**Table 23: Full performance results for The Alchemist’s Melody.**
| Model | Score | Completion Rate (%) |
| --- | --- | --- |
| human | $87.66$ | $100$% |
| gemini-2.5-pro | $43.15$ | $20$% |
| gemini-2.5-flash | $32.05$ | 0% |
| qwen-2.5-omni | $31.23$ | 0% |
| MiniCPM-o-2.6 | $30.29$ | 0% |
| VITA-1.5 | $20.01$ | 0% |
| Baichuan-Omni-1.5 | $31.82$ | 0% |
| random | $25.51$ | 0% |

### I.6 Task-Specific Raw Metrics: Phantom Soldiers in the Fog

Table [24](https://arxiv.org/html/2508.04361v3#A9.T24) presents the detailed performance metrics for the Phantom Soldiers in the Fog task across all difficulties.

**Table 24: Full performance results for Phantom Soldiers in the Fog across all difficulties.**
|  | Score | Success Rate |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Model | Easy | Medium | Hard | Easy | Medium | Hard |
| human | $100.0$ | $99.60$ | $98.50$ | $1.00$ | $0.98$ | $0.950$ |
| gemini-2.5-pro | $83.51$ | $78.81$ | $91.62$ | $1.00$ | $0.88$ | $0.857$ |
| gemini-2.5-flash | $80.39$ | $31.20$ | $73.54$ | $1.00$ | $0.57$ | $0.610$ |
| qwen-2.5-omni | $5.13$ | $23.74$ | $23.34$ | $0.13$ | $0.465$ | $0.550$ |
| MiniCPM-o-2.6 | $3.10$ | $11.60$ | $8.93$ | $0.11$ | $0.20$ | $0.270$ |
| VITA-1.5 | 0 | 0 | 0 | 0 | 0 | 0 |
| Baichuan-Omni-1.5 | $29.15$ | $11.50$ | $19.60$ | $0.50$ | $0.28$ | $0.550$ |
| random | $25.20$ | $22.86$ | $17.80$ | $0.30$ | $0.58$ | $0.460$ |

### I.7 Task-Specific Raw Metrics: Blasting Showdown

Table [25](https://arxiv.org/html/2508.04361v3#A9.T25) presents the full tournament results for the Blasting Showdown task. As this is a competitive multi-agent environment, performance is measured by win rates and combat effectiveness metrics rather than a normalized score.

**Table 25: Full tournament results for the AI-vs-AI evaluation on Blasting Showdown.**
| Model | Games Played | Wins | Win Rate (%) | Kills | Deaths | K/D Ratio |
| --- | --- | --- | --- | --- | --- | --- |
| gemini-2.5-pro | 36 | 13 | $36.11$% | 93 | 39 | $2.38$ |
| gemini-2.5-flash | 38 | 11 | $28.95$% | 68 | 41 | $1.66$ |
| MiniCPM-o-2.6 | 31 | 6 | $19.35$% | 0 | 72 | $0.00$ |
| Baichuan-Omni-1.5 | 34 | 6 | $17.65$% | 31 | 55 | $0.56$ |
| qwen-2.5-omni | 34 | 4 | $11.76$% | 13 | 53 | $0.25$ |
| VITA-1.5 | 27 | 2 | $7.41$% | 0 | 42 | $0.00$ |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 4
presentation: 2
questions: Scores for NPS only may false indicate actual performance due to manual weighting in their calculation. Why report only NPS? Other parts of the appendices don't even calculate it.

Some of the image noise results seem to show a suprising performance drop, even if visually small (ex: Figure 24). This one seems more like "text noise", as the text in the images becomes undreadable. Shouldn't a more in-depth analysis be shown before some of these can be raised as "key findings" in sub-section 5.3? Same for the aided reasoning insight. The paper could benefit from better discussion of the principal findings claims.

Overall I really like the paper idea and some of its contents, but it seems to try to claim too much at the same time, while missing more supporting evidence or in-depth discussion (even if point to future analysis).

I also didn't see a mention of the release of the benchmark and its license, which are also key for its potential community impact and should be covered in the manuscript.

Minor other issues:
- Please fix citations. For example, "Google’s Gemini Team et al. (2023)", or "and SEED-Bench Li et al. (2023)".
- Subsetion 3.1 could easily be moved to and appendix to open more space for game details and core results to move into the core paper.
soundness: 2
strengths: The paper presents a novel benchmark to probe omni-modal models in their actual effective use of different modalities. The use of targeted games is also especially key, as mentioned by authors, it allow for better experiment desing and traditional benchmarks fail to test model capabilities in both dynamic and interactive worlds.

The stated goal to "explicitly address the foundational challenges of synergistic fusion, conflict arbitration, and resilient reasoning" is an important impactful target. And the discussed experimental results both on the effectiveness of models' fusion of different modalities and how they deal with modality conflicts present interesting insights for furhter exploration, which can have significant impact on model performance and how to better evaluate their capabilties.

The design methodology for the benchmark can also inspire the creation of better evaluation assets for the overall community.
summary: The paper introduces OmniPlay, a game-playing benchmark for Omni-modal models to test their capabilites in dynamic interactive worlds. The benchmark is comprised of five custom-designed games that emphasize the interplay between different modalities.
Different series of experiments evaluate six representative current omni-modal models, showcasing their performance limtiations. 
While aditional exploratoy experiments illustrate performance degradation under modality conflict across models and their seemly "preference" for specific modalities. The presented discussion of results points towards better fusion of modalities as key for improved capabilities and perfomance.
weaknesses: While the paper tackles significant and timely issues and proposes an original benchmark with potential, the current manuscript suffers from
not fully adequate presentation and soundness issues for some of its conclusions.

First, the paper presentation seems a bit backwards. It starts talking about specific experiment details without having really described the games, the core benchmark design (beyond just some principles), and not showing overall results. This makes the paper quite hard to interpret as the reader needs to find the details about a specific game, metrics, how they relate to issues, human performance, before they can parse what the discussion of what a figure like Figure 3 or some summary results like Table 2 actually mean.

Second, a core contribution of the benchmark is the proper design of the modality-interplay games. As such, I'd expect at least a short focused description of them in the main paper, as well as some analysis of how well they were designed. Unfortunately, even if games were also played by humans, there is no discussion on how well the games model their goals (gameplay or evaluation effectiveness as tools). Moreover, for self-contained context, at least a sentence description by game emphasizing its key goal and something like table 7 should be in main paper body. Table 1 doesn't really perform this job well as it's too high-level and could be re-designed or turned into a text paragraph.

The discussed experiments present nice findings regarindg multi-modality capabilites, like Gemini 2.5 Flash performance changes in different cross-modality conflicts. However, the paper doesn't provide any substantive discussion of "reasoning and strategic planning", as claimed as contribution. Table 2 is not enough evidence for the claim of dichotomy in performance or sub-par reasoning, as it lacks details and even comparison to human performance. The other analysis provided also delves mostly in modality representation issues.

Full performance results only come in Appendix I, with human results only show in Table 19 (page 41) and could easily include NPS metrics for ease of comparison to model performance in the main paper. I understand the limitations of space, but some of the findings in Appendices F5, F6, and G, could also have been specific called out from main text for better understanding.

Regarding the "diiagnosis" angle of the benchmark, while the described effect of modality conflicts are insightful, they seem a little high-level and left me also wondering how would humans handle such cases and what the performance impact would be. I feel like this is a critical aspect for proper undestanding the presented results.

The paper also claims to acknowledge the benchmark limitations, but there are not explicitly discuss discussed anywhere.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 4
questions: - Based on the zero-shot performance, the author claims "scaling models may not be sufficient". It is plausible that these "brittle fusion mechanisms" are an artifact of zero-shot generalization and could be fixed with even minimal fine-tuning on the OmniPlay tasks. Is there any fine-tuning result to distinguish a fundamental model incapacity v.s. a lack of task-specific adaptation? I am aware of the "aided reasoning" scenarios, but a full fine-tuning study is needed to substantiate the paper's primary conclusion.
- For Phantom Soldiers in the Fog, what is the likely reason that the model fails? Is it due to failed long-horizon planning or failure to ground the (noisy) visual cues?
soundness: 2
strengths: - The motivation of the paper is clear, where there is a lack of benchmarks that test agency with a rich, multi-sensory environment.
- There is solid open-source contribution for both the environments and evaluation protocols, and the appendix shows extensive details.
- The paper is well-written.
summary: This paper introduces OmniPlay, a new diagnostic benchmark designed to evaluate the omni-modal (Image, Video, Audio, and Text) reasoning and agency of generalist foundation models. The authors argue that existing benchmarks create an "evaluation chasm" : static benchmarks (like VQA) lack agency and interactivity , while current interactive benchmarks (like ALFWorld) suffer from a bottleneck in modality. The core design of OmniPlay is "modality interplay" where the author tests complementarity (needing all senses) and conflict (handling contradictory information). The authors evaluate six multi-modal models and show that though models demonstrate "superhuman memory", they show "brittle reasoning" in comparison. I'll adjust my rating based on author's response.
weaknesses: - Some tasks, while creative, appear to test very specific, narrow forms of reasoning. The Alchemist's Melody (rule discovery) can be reduced to a trial-and-error association problem where no reasoning is really involved. The Myriad Echoes and the Whispered Pathfinding are essentially complex perception-and-grounding tasks. They are not really testing "strategic planning" and "robust reasoning".
- The paper heavily contrasts "brittle reasoning" with "superhuman memory". This "dichotomy" is not really an apple-to-apple comparison. The memory task (Myriad Echoes) is a requiring sequence replication which perfectly maps to the architectural strengths of transformers (perfect recall of a sequence from context) and leverages capabilities that are known to be far superior to human working memory. In contrast, the "reasoning" tasks require emergent, generalizable skills that are the field's current grand challenge. Finding that models are good at what they are architecturally designed to do (remember and recall) and bad at what they are not (immediate reasoning) is not a particularly insightful dichotomy; it is almost an expected result.


Nits (does not affect rating):
- "Full Sensory Spectrum" is over-claimed. Modalities relevant to agentic intelligence, like haptics (touch), proprioception, and sensors (e.g., radar, lidar) are actually not considered. Some rephrasing is needed.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: See the weakness section.
soundness: 4
strengths: * This paper tackles the inadequacy of current benchmarks, which are either static (lacking agency) or interactive but modally limited (ignoring audio, etc.), by introducing an interactive benchmark designed for *omni-modal* agents using image, video, audio, and text.

* It introduces five distinct, newly developed game environments, each crafted to test different capabilities (e.g., navigation, sequence replication, abstract reasoning, strategy) under varying modality combinations and complexities.

* The evaluation reveals critical insights into current omni-modal models, such as their superhuman memory but weak reasoning/planning, fragility under modality conflict, and the paradoxical "less is more" effect where removing modalities can improve performance.

* The paper establishes strong baselines (random agent and a diverse human expert cohort) and uses both overall performance metrics (like NPS) and detailed, task-specific diagnostic metrics.

* The authors intend to release the entire OmniPlay platform, including environments and protocols, fostering further research in the community.
summary: This paper introduces OmniPlay, a novel benchmark designed to evaluate the capabilities of omni-modal foundation models (like Gemini and GPT-4o) in interactive, dynamic environments, addressing the limitations of existing static or modally-restricted benchmarks. OmniPlay consists of a suite of five custom-built game environments that require agents to perceive, reason, and act using a combination of image, video, audio, and text inputs. The benchmark is specifically designed around the principle of "modality interplay," systematically creating scenarios where sensory information is either complementary (requiring synergistic fusion) or conflicting (testing robustness and conflict resolution). Through comprehensive evaluations of six leading omni-modal models against human and random baselines, the authors find a stark dichotomy: the models exhibit superhuman performance on tasks heavily reliant on memory but show systemic failures in those demanding robust reasoning, strategic planning, and handling modality conflicts. A key finding is the "less is more" paradox, where removing sensory modalities sometimes paradoxically improves performance, suggesting immature fusion mechanisms are a critical bottleneck. The paper concludes that progress towards AGI requires focusing on synergistic fusion and conflict arbitration, not just model scaling, and offers OmniPlay as a diagnostic tool for these challenges.
weaknesses: - The games and scenarios involving modality complementarity and conflict are custom-designed based on the authors' principles. The "naturalness" or representativeness of these specific interaction patterns for general real-world tasks could be debated.

- The paper highlights "superhuman memory" in the Myriad Echoes task. However, the qualitative analysis suggests this is largely due to the AI's perfect recall compared to human cognitive limits on working memory for long, arbitrary sequences, rather than a sign of superior general memory or reasoning.

- Evaluating large omni-modal models in interactive environments across numerous episodes and seeds is computationally intensive. The paper does not detail the resources required, which could be a barrier for widespread adoption and replication of the benchmark.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: - The benchmark mainly tests one-shot performance. Have the authors considered evaluating continual adaptation or long-horizon reasoning?
- How are difficulty levels calibrated across tasks to maintain balance and interpretability?
- Do the authors consider modality-specific weighting based on task relevance rather than assuming equal importance across modalities?
- Since modality conflicts are manually constructed, how can the authors ensure that these settings realistically capture natural multimodal inconsistencies rather than artificial noise?
soundness: 2
strengths: - It provides a comprehensive and realistic evaluation of omni-modal models, integrating text, image, audio, and video interaction, filling the gap left by older static or single-modality benchmarks.
- It is theoretically well-grounded, built on a generalized MDP framework with explicit principles of modality interdependence, controlled conflict, and variable complexity.
- It offers diagnostic precision, using systematic tests like modality conflict, ablation, and noise experiments to clearly reveal each model’s strengths and weaknesses in multi-modal fusion
summary: This paper introduces a diagnostic benchmark designed to evaluate omni-modal AI systems capable of processing and integrating text, images, audio, and video in dynamic environments. It addresses the gap between static benchmarks, which lack interactivity, and existing interactive ones, which usually ignore important sensory modalities. The benchmark contains five interactive game environments that intentionally create both complementary and conflicting sensory conditions, allowing researchers to study how models perform cross-modal reasoning and resolve ambiguity. Evaluation of six state-of-the-art models reveals a clear contrast: they achieve superhuman memory performance but show fragile reasoning and planning, especially when modalities conflict. Interestingly, removing one modality sometimes improves performance (“less is more”). The findings suggest that advancing general intelligence requires focusing not on scaling model size, but on improving multi-sensory fusion and conflict resolution.
weaknesses: - The benchmark lacks realistic, high-fidelity 3D environments like those used in DeepMind’s SIMA project (https://deepmind.google/discover/blog/sima-generalist-ai-agent-for-3d-virtual-environments/). Its custom mini-games are controlled and simplified, limiting ecological validity and real-world generalization.
- The “modality conflict” and “modality complementarity” in each game are artificially constructed scenarios (e.g., inserting misleading audio cues or removing video frames). While such controlled manipulation facilitates diagnostic analysis, it does not necessarily reflect the naturally occurring noise and uncertainty found in real multimodal tasks, thereby limiting the ecological validity of the benchmark.
- The NPS is normalized using human and random baselines, but the underlying scoring functions differ significantly across tasks (especially in RTS and pathfinding scenarios). This “cross-task unified metric” implicitly assumes comparability between heterogeneous tasks, which in practice reduces the rigor and fairness of the evaluation. Moreover, certain tasks (e.g., Blasting Showdown) are excluded from the NPS, further undermining the overall consistency of the benchmark.


Decision: Reject