VideoGameQA-Bench: Evaluating Vision-Language Models for Video Game
  Quality Assurance
Authors: name: Mohammad Reza Taesiri, name: Abhijay Ghildyal, name: Saman Zadtootaghaj, name: Nabajeet Barman, name: Cor-Paul Bezemer

With video games now generating the highest revenues in the entertainment
industry, optimizing game development workflows has become essential for the
sector's sustained growth. Recent advancements in Vision-Language Models (VLMs)
offer considerable potential to automate and enhance various aspects of game
development, particularly Quality Assurance (QA), which remains one of the
industry's most labor-intensive processes with limited automation options. To
accurately evaluate the performance of VLMs in video game QA tasks and
determine their effectiveness in handling real-world scenarios, there is a
clear need for standardized benchmarks, as existing benchmarks are insufficient
to address the specific requirements of this domain. To bridge this gap, we
introduce VideoGameQA-Bench, a comprehensive benchmark that covers a wide array
of game QA activities, including visual unit testing, visual regression
testing, needle-in-a-haystack tasks, glitch detection, and bug report
generation for both images and videos of various games. Code and data are
available at: https://asgaardlab.github.io/videogameqa-bench/