BigCodeArena: Unveiling More Reliable Human Preferences in Code
  Generation via Execution
Authors: name: Terry Yue Zhuo; name: Xiaolong Jin; name: Hange Liu; name: Juyong Jiang; name: Tianyang Liu; name: Chen Gong; name: Bhupesh Bishnoi; name: Vaisakhi Mishra; name: Marek Suppa; name: Noah Ziems; name: Saiteja Utpala; name: Ming Xu; name: Guangyu Song; name: Kaixin Li; name: Yuhan Cao; name: Bo Liu; name: Zheng Liu; name: Sabina Abdurakhmanova; name: Wenhao Yu; name: Mengzhao Jia; name: Jihan Yao; name: Kenneth Hamilton; name: Kumar Shridhar; name: Minh Chien Vu; name: Dingmin Wang; name: Jiawei Liu; name: Zijian Wang; name: Qian Liu; name: Binyuan Hui; name: Meg Risdal; name: Ahsen Khaliq; name: Atin Sood; name: Zhenchang Xing; name: Wasi Uddin Ahmad; name: John Grundy; name: David Lo; name: Banghua Zhu; name: Xiaoning Du; name: Torsten Scholak; name: Leandro von Werra
Published: 2025-10-09 18:01:47

Abstract: Crowdsourced model evaluation platforms, such as Chatbot Arena, enable
real-time evaluation from human perspectives to assess the quality of model
responses. In the coding domain, manually examining the quality of
LLM-generated content is extremely challenging, as it requires understanding
long chunks of raw code and deliberately simulating code execution. To this
end, we introduce BigCodeArena, an open human evaluation platform for code
generation backed by a comprehensive and on-the-fly execution environment.
Built on top of Chatbot Arena, BigCodeArena enables the execution of
LLM-generated code and allows humans to interact with the execution process and
outcomes. We collected over 14,000 raw code-centric conversation sessions
across 10 widely used LLMs, spanning 10 languages and 8 types of execution
environments. Among these conversations, we identified more than 4,700
multi-turn samples with pairwise human preferences. Further analysis uncovers
underexplored preferences of LLMs in fine-grained domains characterized by
tasks, languages, and frameworks. To systematically examine code understanding
and generation capabilities of frontier LLMs, we curated two benchmarks based
on the collected data, namely BigCodeReward and AutoCodeArena. For
BigCodeReward, we post-processed the 4,700 conversations and evaluated the
consistency between reward models and human preferences. The evaluation shows
that most LLMs have superior performance in judging coding preferences when the
execution results are available. Inspired by these findings, we propose
AutoCodeArena, an automatic Elo rating benchmark designed to assess the coding
quality of LLMs without human involvement. We find that proprietary LLMs like
GPT-5, Claude-Sonnet-4, and Claude-Opus-4 still lead in code generation
performance among recent emerging models.