Multi-Agent Game Generation and Evaluation via Audio-Visual Recordings
Authors: name: Alexia Jolicoeur-Martineau

While AI excels at generating text, audio, images, and videos, creating
interactive audio-visual content such as video games remains challenging.
Current LLMs can generate JavaScript games and animations, but lack automated
evaluation metrics and struggle with complex content that normally requires
teams of humans working for many months (multi-shot, multi-agents) using assets
made by artists. To tackle these issues, we built a new metric and a
multi-agent system.
  We propose AVR-Eval, a relative metric for multimedia content quality using
Audio-Visual Recordings (AVRs). An omni-modal model (processing text, video,
and audio) compares the AVRs of two contents, with a text model reviewing
evaluations to determine superiority. We show that AVR-Eval properly identifies
good from broken or mismatched content.
  We built AVR-Agent, a multi-agent system generating JavaScript code from a
bank of multimedia assets (audio, images, 3D models). The coding agent selects
relevant assets, generates multiple initial codes, uses AVR-Eval to identify
the best version, and iteratively improves it through omni-modal agent feedback
from the AVR.
  We run experiments on games and animations with AVR-Eval (win rate of content
A against B). We find that content generated by AVR-Agent has a significantly
higher win rate against content made through one-shot generation. However,
models struggle to leverage custom assets and AVR feedback effectively, showing
no higher win rate. This reveals a critical gap: while humans benefit from
high-quality assets and audio-visual feedback, current coding models do not
seem to utilize these resources as effectively, highlighting fundamental
differences between human and machine content creation approaches.