Automated Generation of Challenging Multiple-Choice Questions for Vision
  Language Model Evaluation
Authors: name: Yuhui Zhang; name: Yuchang Su; name: Yiming Liu; name: Xiaohan Wang; name: James Burgess; name: Elaine Sui; name: Chenyu Wang; name: Josiah Aklilu; name: Alejandro Lozano; name: Anjiang Wei; name: Ludwig Schmidt; name: Serena Yeung-Levy
Published: 2025-01-06 18:57:31

Abstract: The rapid development of vision language models (VLMs) demands rigorous and
reliable evaluation. However, current visual question answering (VQA)
benchmarks often depend on open-ended questions, making accurate evaluation
difficult due to the variability in natural language responses. To address
this, we introduce AutoConverter, an agentic framework that automatically
converts these open-ended questions into multiple-choice format, enabling
objective evaluation while reducing the costly question creation process. Our
experiments demonstrate that AutoConverter can generate correct and challenging
multiple-choice questions, with VLMs demonstrating consistently similar or
lower accuracy on these questions compared to human-created ones. Using
AutoConverter, we construct VMCBench, a benchmark created by transforming 20
existing VQA datasets into a unified multiple-choice format, totaling 9,018
questions. We comprehensively evaluate 33 state-of-the-art VLMs on VMCBench,
setting a new standard for scalable, consistent, and reproducible VLM
evaluation.