KOFFVQA: An Objectively Evaluated Free-form VQA Benchmark for Large
  Vision-Language Models in the Korean Language
Authors: name: Yoonshik Kim, name: Jaeyoon Jung

The recent emergence of Large Vision-Language Models(VLMs) has resulted in a
variety of different benchmarks for evaluating such models. Despite this, we
observe that most existing evaluation methods suffer from the fact that they
either require the model to choose from pre-determined responses, sacrificing
open-endedness, or evaluate responses using a judge model, resulting in
subjective and unreliable evaluation. In addition, we observe a lack of
benchmarks for VLMs in the Korean language, which are necessary as a separate
metric from more common English language benchmarks, as the performance of
generative language models can differ significantly based on the language being
used. Therefore, we present KOFFVQA, a general-purpose free-form visual
question answering benchmark in the Korean language for the evaluation of VLMs.
Our benchmark consists of 275 carefully crafted questions each paired with an
image and grading criteria covering 10 different aspects of VLM performance.
The grading criteria eliminate the problem of unreliability by allowing the
judge model to grade each response based on a pre-determined set of rules. By
defining the evaluation criteria in an objective manner, even a small
open-source model can be used to evaluate models on our benchmark reliably. In
addition to evaluating a large number of existing VLMs on our benchmark, we
also experimentally verify that our method of using pre-existing grading
criteria for evaluation is much more reliable than existing methods. Our
evaluation code is available at https://github.com/maum-ai/KOFFVQA