VideoGameBunny: Towards vision assistants for video games
Authors: name: Mohammad Reza Taesiri; name: Cor-Paul Bezemer
Published: 2024-07-21 23:31:57

Abstract: Large multimodal models (LMMs) hold substantial promise across various
domains, from personal assistance in daily tasks to sophisticated applications
like medical diagnostics. However, their capabilities have limitations in the
video game domain, such as challenges with scene understanding, hallucinations,
and inaccurate descriptions of video game content, especially in open-source
models. This paper describes the development of VideoGameBunny, a LLaVA-style
model based on Bunny, specifically tailored for understanding images from video
games. We release intermediate checkpoints, training logs, and an extensive
dataset comprising 185,259 video game images from 413 titles, along with
389,565 image-instruction pairs that include image captions, question-answer
pairs, and a JSON representation of 16 elements of 136,974 images. Our
experiments show that our high quality game-related data has the potential to
make a relatively small model outperform the much larger state-of-the-art model
LLaVa-1.6-34b (which has more than 4x the number of parameters). Our study
paves the way for future research in video game understanding on tasks such as
playing, commentary, and debugging. Code and data are available at
https://videogamebunny.github.io/