VisionArena: 230K Real World User-VLM Conversations with Preference
  Labels
Authors: name: Christopher Chou; name: Lisa Dunlap; name: Koki Mashita; name: Krishna Mandal; name: Trevor Darrell; name: Ion Stoica; name: Joseph E. Gonzalez; name: Wei-Lin Chiang
Published: 2024-12-11 18:59:46

Abstract: With the growing adoption and capabilities of vision-language models (VLMs)
comes the need for benchmarks that capture authentic user-VLM interactions. In
response, we create VisionArena, a dataset of 230K real-world conversations
between users and VLMs. Collected from Chatbot Arena - an open-source platform
where users interact with VLMs and submit preference votes - VisionArena spans
73K unique users, 45 VLMs, and 138 languages. Our dataset contains three
subsets: VisionArena-Chat, 200k single and multi-turn conversations between a
user and a VLM; VisionArena-Battle, 30K conversations comparing two anonymous
VLMs with user preference votes; and VisionArena-Bench, an automatic benchmark
of 500 diverse user prompts that efficiently approximate the live Chatbot Arena
model rankings. Additionally, we highlight the types of question asked by
users, the influence of response style on preference, and areas where models
often fail. We find open-ended tasks like captioning and humor are highly
style-dependent, and current VLMs struggle with spatial reasoning and planning
tasks. Lastly, we show finetuning the same base model on VisionArena-Chat
outperforms Llava-Instruct-158K, with a 17-point gain on MMMU and a 46-point
gain on the WildVision benchmark. Dataset at https://huggingface.co/lmarena-ai