Two Giraffes in a Dirt Field: Using Game Play to Investigate Situation
  Modelling in Large Multimodal Models
Authors: name: Sherzod Hakimov; name: Yerkezhan Abdullayeva; name: Kushal Koshti; name: Antonia Schmidt; name: Yan Weiser; name: Anne Beyer; name: David Schlangen
Published: 2024-06-20 06:56:19

Abstract: While the situation has improved for text-only models, it again seems to be
the case currently that multimodal (text and image) models develop faster than
ways to evaluate them. In this paper, we bring a recently developed evaluation
paradigm from text models to multimodal models, namely evaluation through the
goal-oriented game (self) play, complementing reference-based and
preference-based evaluation. Specifically, we define games that challenge a
model's capability to represent a situation from visual information and align
such representations through dialogue. We find that the largest closed models
perform rather well on the games that we define, while even the best
open-weight models struggle with them. On further analysis, we find that the
exceptional deep captioning capabilities of the largest models drive some of
the performance. There is still room to grow for both kinds of models, ensuring
the continued relevance of the benchmark.