Title: Vibe Checker: Aligning Code Evaluation with Human Preference
Primary area: datasets and benchmarks
Keywords: Vibe Check, Vibe Coding, Code Evaluation, Instruction Following, Human Preference

Abstract: Large Language Models (LLMs) have catalyzed vibe coding, where users leverage LLMs to generate and iteratively refine code through natural language interactions until it passes their *vibe check*. *Vibe check* is tied to real-world human preference and goes beyond functionality: the solution should feel right, read cleanly, preserve intent, and remain correct. However, current code evaluation remains anchored to pass@k and captures only functional correctness, overlooking the non-functional instructions that users routinely apply. In this paper, we hypothesize that instruction following is the missing piece underlying *vibe check* that represents human preference in coding besides functional correctness. To quantify models' code instruction following capabilities with measurable signals, we present **VeriCode**, a taxonomy of 30 verifiable code instructions together with corresponding deterministic verifiers. We use the taxonomy to augment established evaluation suites, resulting in **Vibe Checker**, a testbed to assess both code instruction following and functional correctness. Upon evaluating 31 leading LLMs, we show that even the strongest models struggle to comply with multiple instructions and exhibit clear functional regression. Most importantly, *a composite score of functional correctness and instruction following correlates the best with human preference*, with the latter emerging as the primary differentiator among advanced LLMs on real-world programming tasks. Our work identifies core factors of the *vibe check*, providing a concrete path for benchmarking and developing models that better align with user preferences in coding.

TL;DR: This paper introduces Vibe Checker, a new code evaluation testbed, showing that real-world human preference aligns best with a mix of functional correctness and instruction following.