VibeCheck: Discover and Quantify Qualitative Differences in Large Language Models
Authors: Lisa Dunlap
Krishna Mandal
Trevor Darrell
Jacob Steinhardt
Joseph E. Gonzalez
Venue: ICLR 2025 Poster (2025)
Keywords: large language models
evaluation
Primary area: foundation or frontier models, including LLMs
TL;DR: Find interesting subjective properties that differentiate models

Abstract: Large language models (LLMs) often exhibit subtle yet distinctive characteristics in their outputs that users intuitively recognize, but struggle to quantify. These "vibes" -- such as tone, formatting, or writing style -- influence user preferences, yet traditional evaluations focus primarily on the singular vibe of correctness.
We introduce $\textbf{VibeCheck}$, a system for automatically comparing a pair of LLMs by discovering identifying traits of a model ("vibes") that are well-defined, differentiating, and user-aligned. VibeCheck iteratively discovers vibes from model outputs and then utilizes a panel of LLM judges to quantitatively measure the utility of each vibe. 
We validate that the vibes generated by VibeCheck align with those found in human discovery and run VibeCheck on pairwise preference data from real-world user conversations with Llama-3-70b vs GPT-4. VibeCheck reveals that Llama has a friendly, funny, and somewhat controversial vibe. These vibes predict model identity with 80% accuracy and human preference with 61% accuracy. Lastly, we run VibeCheck on a variety of models and tasks, including summarization, math, and captioning to provide insight into differences in model behavior. VibeCheck discovers vibes like Command X prefers to add concrete intros and conclusions when summarizing in comparison to TNGL, Llama-405b often overexplains its thought process on math problems compared to GPT-4o, and GPT-4 prefers to focus on the mood and emotions of the scene when captioning compared to Gemini-1.5-Flash.

Published as a conference paper at ICLR 2025
DISCOVER & QUANTIFY QUALITATIVE DIFFERENCES IN LARGE LANGUAGE MODELS
Lisa Dunlap UC Berkeley
Krishna Mandal UC Berkeley
Trevor Darrell UC Berkeley
Jacob Steinhardt UC Berkeley
Joseph Gonzalez UC Berkeley
ABSTRACT
Large language models (LLMs) often exhibit subtle yet distinctive characteristics in their outputs that users intuitively recognize, but struggle to quantify. These "vibes" - such as tone, formatting, or writing style influence user preferences, yet traditional evaluations focus primarily on the singular vibe of correctness. We introduce VibeCheck, system for automatically comparing pair of LLMs by discovering identifying traits of a model ("vibes"") that are well-defined, differenti- ating. and user-aligned. VibeCheck iteratively discovers vibes from model outputs and then utilizes a panel of LLM judges to quantitatively measure the utility of each vibe. We validate that the vibes generated by VibeCheck align with those found in human discovery and run VibeCheck on pairwise preference data from real-world user conversations with Llama-3-70b GPT-4. VibeCheck reveals that Llama has a friendly, funny, and somewhat controversial vibe. These vibes predict model iden- tity with 80% accuracy and human preference with 61% accuracy. Lastly, we run VibeCheck on a variety of models and tasks including summarization. math, and captioning to provide insight into differences in model behavior. VibeCheck dis- covers vibes like Command X prefers to add concrete intros and conclusions when summarizing in comparison to TNGL, Llama-405h often overexplains its thought process math problems compared to GPT-4o, and GPT-4 prefers to focus the mood and emotions of the scene when captioning compared to Gemini-1.5-Fll Code can be found at ht https: : /tt///gtttt//tt com/ lisadunlapp hataannnaaa////eeeeeeee
INTRO
vibe check A process by which group obtains subjective assessment of another person, place, or thing.
-Urban Dictionary
How a large language model writes story, explains concept, edits essay can be evaluated along many different dimensions such as creativity, formatting, and writing style. However, most evaluations focus dimension: "correcmess" State-of-the-are in evaluation methods remain largely focused on measuring accuracy for question answering and analytical reasoning tasks (Hendrycks et al., 2021a la; Wang al., 2019b;a; Hendrycks et al., 2021c), and methods which aim to provide a more holistic view of LLMs (Zhang et al., 2024; Padlewski et al., 2024; Mehri & Eskenazi, 2020b) rely on predefined concepts like conciseness, clarity, and trustworthiness to measure model's performance. These evaluation approaches fail to capture the open-ended nature of LLM applications and the critical dependence on subjective user preferences and context of the task. For instance, tone and creativity might be crucial in creative writing, whereas efficiency and readability are crucial in coding tasks. To best inform users of which model would be best for their needs, we require flexible evaluation methods that can both discover and measure the relevant axes to evaluate for given task.
When interacting with a set of LLMs for an extended period, a user can often tell which model generated particular response by looking at certain traits of the outputs. We define these identifying traits of models as "vibes" ". For instance, users have found Llama-3 outputs tend to be more friendly compared outputs from GPT-4 and Claude which tend to be more formal (see Figure 1); in other words, Llama-3 ranks high on the friendliness vibe, defined by the axis formal - friendly ly Using these insights, we might select Llama for customer service tasks and Claude for coding tasks.

---

Published as a conference paper at ICLR 2025
Understanding these vibes helps inform the development and deployment of models, but discovering and validating them for each model can be consuming and difficult. To address this, we outline how one can find and, more importantly, measure LLM's 's vibe by formalizing three necessary and quantifiable traits of a useful vibe: well-defined (agreement among multiple users), disferentiating (ability to distinguish between models), and user-aligned (predictive of user preferences).
We introduce VibeCheck, a system which qualitatively analyzes pairs of models by automatically finding well-defined, differentiating, and user-aligned vibes. Motivated by recent work in using LLM's in lieu of human judgment (Zheng et al., 2023; Zhang et al., 2024; Zhong et al., 2023; 2022; Dubois et al.. 2023), VibeCheck models the qualitative analysis process by identifying the axes on which these model ouiputs differ to obtain a core set of vibes (e.g friendliness). Once these vibes are obtained, VibeCheck employs panel of L.MM judges (Verga et al., 2024) to determine where each model's output falls on this vibe (e.g. more formal or more friendly) in order to obtain numeric scores which are then used to measure vibe on each of our 3 key criteria.
We run VibeCheck several datasets to evaluate its effectiveness across different scenarios in Section 5. First, we validate that the vibes discovered by VibeCheck align well with human-annolated differences between ChatGPT and human responses using the Human ChatGPT Comparison Corpus (HC3). Next, we demonstrate that VibeCheck outperforms a predefined list of vibes in predicting user preferences
real-world comparison data from Chatbot Arena, achieving 80% accuracy at predicting model identity and 61% % accuracy and predicting user preference. Inspecting the vibes of VibeCheck, we find that Llama-70b uses more typographic emphasis, more examples, and is funnier than GPT-4 and Claude-3-Opus. Conversely, we find that GPT-4 and Claude comment much more on ethics and limitations than Llama, which is more willing to give controversial responses.
Lastly, in Section 6 we apply VibeCheck to several applications: text summarization on CNN/Daily- Mail, math groblem-solving on MATH, and image captioning on COCO. Using VibeCheck, we find insightful qualitative differences between models with similar accuracy on correctness metrics but differing user preferences. For instance. Command X prefers add concrete intros and conclusions when summarizing in comparison to TNGL., Llama-405b often overexplains its thought process on math problems, and GPT-4 prefers focus on the mood and emotions of the scene when captioning.
2 RELATED WORK
Aspect-based evaluations. The number of benchmarks in the NLP community has exploded in recent years, with a growing body of work on exploring more holistic evaluation of language models. Several works (Pang et al., 2020; Banerjee & Lavie, 2005; Sellam et al., 2020) aim to improve on automatic metrics like BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004) scores to better measure how well a models output aligns with the ground truth by incorporating more nuanced evaluation criteria like factual accuracy, fluency, and conciseness. Similarly, efforts have been made (Liang et al., 2023; bench authors, 2023; Kiela et al., 2021; Wang et al., 2019b;a) to standardize model evaluation by evaluating models on many of these metrics across various tasks.
Moving away from measuring model outputs on ground truth responses, work from Mehri & Eskenazi (2020b); Zhang et al. (2024); Li et al. (2019); Mehri & Eskenazi (2020a); Gehrmann et al. (2021) evaluate model outputs on criteria like helpfulness and clarity using LLM judges on more open ended tasks like dialogue, role-play, and summarization. While these efforts supply a great foundation for measuring correctness, they all define the axes on what makes something correct beforehand. In contrast, VibeCheck aims to automatically discover these axes (vibes) and verify their utility to the user by measuring the correlation between vibes and human preference.
Pairwise comparison of LLMs. HCI tools like Google's AutoSxS (Google Cloud, 2024) and L.M.Comprrtorr (Kahng et al., 2024) explores the current state of human powered LLM qualitative evaluation through interviews with data analysts. These works find that practitioners often eyeball individual examples to interpret and look at qualitative differences between the outputs of two models, and develop an interactive web based application for users to inspect side-by-side LLM outputs with an LLM based rationale as to why one output is preferred over another. While these works are focused more software tools rather than a pipeline which can be quantitavely verified, these HCI findings inform VibeCheck's 's vibe discovery mechanism to align with the human-poweree powered qualitative process. Moreover, many NL. works (Zheng et al., 2023; Verga et al., 2024; Li et al., 2023; Park
2

---

Published as a conference paper at ICLR 2025
What is vibe?
Prompt: What the best coffee?
Vibe (low high)
Output A: What bold question! After considering various declare...
do two differ?
Friendliness formal + friendly
Output B: Identifying the "best" coffee is challenging because taste is susjective...
How vibes?
How do quantify vibe utility?
Prompt:
If was mouse...
Judge
Judge
Aug Soore
Proference
Output you mouse, way communicate effectively...
Viin, Ob) Ve Os)
1
A
(p Ox, Ob)
Va(ps, Os, On)
-1
Output B: Ahahahal Oh, what delightful pun!
Judge/
"Which ranks higher friendlnness Respond with B, or equal"
Well Defined Agreement Judiee
Differentiating Abiliy to predict model from friendliness 55%
Vipp, O., 0₂) B =-1
User-Allgned + Ablity to predict preference from friendinass + 55%
Figure 1: Core components of VibeCheck. vibe is an axis along which a pair of outputs differ: for example, in the top panel, output A is more friendly while output more formal, defining friendliness vibe. To score prompt output triplet, a panel of LLM judges are used to determine which output falls higher on the vibe, resulting score of (A),-1(B) -BB), or O(tie). Finally, the scores obtained over large set of outputs along with preference labels are used to compute vibe utility.
et al., 2024; Liusie et al., 2024) have explored using LLMs to predict user preference given responses from two models, showing these preference predictions often align with the judgements of human annotators. While these efforts focus more on the user experience, it does not provide an interpretable view of exactly why these users prefer one output over the other.
Discovering separable traits in unstructured data. In parallel to works in the machine learning community on LLM evaluation, there has been fantastic efforts the HCI community on comparing generative model outputs well as using L.Mss for qualitative analysis. Works like Torii et al. (2024); Byun et (2023) use LLMs to generate discussions from qualitative research data to automate the data analysis process, but note the lack of comprehensive evaluation metries. Automated data analysis on unstructured data has also been explored in Zhong et al. (2022; 2023); Dunlap et al. (2024b), which LLMs and VLMs propose and validate candidate differences between two sets of text or images the form of "set contains more X", and Chiquier et al. (2024) employs evolutionary algorithm to find text descriptions which best separates image classes to assist in zero-shot classification. We extend these works to pairwise inputs and introduce metrics success which can better verify the separabilityy consistency, and alignment of these differences.
VIBE-BASED EVALUATIONS
We define a vibe as an axis along which pair of texts can differ (e.g., "formal - friendly") that is perceptible humans. A vibe is represented by description of the axis along with a definition of what it means to be high or low on this axis (e.g. "Tone: low formal, high friendly", see Figure 1). Identifying vibes aids users in selecting models that best suit their specific tasks. this work, we focus comparing the vibes of two models by discovering the axes which their outputs differ and quantifying the utility of these vibes.
Consider dataset D composed of triples (p, o%.0%))) and preference labels yp. where is prompt and of are the outputs from models and B. For each triple, a judge (human or LLM) assigns a score for vibe v, denoted v(p, of og)) {-1, 1]. which indicates whether model A scores lower (-1), similarly (0), or higher (1) than model B on this vibe. Thus, a vibe imposes an ordering on model outputs.
3

---

Published as a conference paper at ICLR 2025
We define 3 key criteria of useful vibe; it should be well-defined, differentiating. and user-aligned.
Well-defined: multiple evaluators agree on the ordering of outputs along the vibe. We quantify this by having two different judges (typically LLMs) compute r(p, o og)) across dataset D and report Cohen's 's Kappa to assess agreement.
model's outputs consistently rank higher this vibe compared to the other's across a set of prompts. We quantify this by calculating separability score for each vibe, which measures how consistently the vibe distinguishes between the two models across all samples.
Differentiating:
sep (ep.score(v)
Σ PED
r(p. 0
D
To measure separability across a set of vibes, we fix pair of models (A, B) and measure the accuracy of using a, oB) classify which output came from which model. We also more generally measure separability for a set of vibes V/,.... Pkk by using Visk (P, OA, 0B) as a k-dimensional feature vector, model B, and reporting accuracy on held-out
then training a linear classifier to predict model set. We refer to this metric as model-man -matching accuracy.
User-aligned. One potential use of vibes is to better understand human preferences. While vibe like "frequent use of the letter 'e' may be differentiating, it is unlikely predictive of human preferences. We assume our tuples (p. (p.o², 0A of. are annotated with user preferences y E 1-1,+13, indicating which model's output is preferred. We train a logistic regression classifier to predict y using the same feature set Pick as above, reporting held-out accuracy. We refer to this metric as preference prediction accuracy. We can measure the influence of a single vibe on preferences by examining the coefficients and p-values of the preference prediction model.
VibeCheck automatically finds high-scoring vibes across the three criteria through iterative process: (1) discovering vibes, (2) computing their scores, (3) selecting those meeting all criteria, and (4) focusing on tuples (p, o ob) where existing vibes fail to differentiate the two models. We repeat this process to extract new, more distinguishing vibes, thus optimizing for the three key criteria while continuously refining the set of vibes.
4 VIBECHECK
VibeCheck consists stages: vibe discovery, vibe validation, and vibe iteration. Further details on the method implementation and prompts used are located in the Section D.
Vibe discovery. Similar to how a data scientist would inspect a subset of examples to discover discover vibes by having LLM (GPT-4o (OpenAI, 2024))
qualitative differences in outputs, examine the differences seen in a random subset of d prompt triplets. We first split the d prompt triplets into smaller batches of size batch and prompt GPT-4o find differences between model A and model B across the set {Pi, 2), o,),,..
bi),.... .... (Pbatch, batch obatch)). To encourage the vibes to be well-defined and user-aligned, we prompt GPT-4o to generate differences that are human-interpretable and informative for understanding the overall behaviors of A and B. Below is paraphrased system prompt used by the proposer.
You are a machine Learning researcher analyzing cutputs from two LLMs on the same input, identify differences along specific, mutually exclusive, and clearly defined axes that are easily interpretable by humans. For each axis, provide a concise description of what it moans for an output to be "Low" and "High" on this axis.
example axis generated in this step might be 'Tone: Low: formal; High: friendly'. We repeat this proposal step for [d/ [d/batch] sets of triplets, obtaining final set of vibes (vI,.., PM by taking the union of the vibes generated in each batch. We found that GPT-4o generates 5-10 axes of variation (vibes) for each sample, vibes which appear most often in (p,,..,'M}>
summarize vibes across all samples in Daicovery to find set of K
Vibe validation. Given vibe from the discovery phase, first apply each vibe to set of validation tuples, then use this validation set to score vibes and compute inter-annotator agreement, model-matching accuracy, and preference prediction accuracy and filter vibes with low scores.
4

---

Published as a conference paper at ICLR 2025
To apply vibes on the validation set, assign score to each pair of outputs my o..
{-1.0, 1), indicating whether model A scores lower (-1), similarly (0), or higher (1) than model B on the vibe. A score of is assigned if the outputs equal this vibe or if the vibe is not applicable (e.g., the vihe is about coding style but neither output contains code); otherwise, we compute the score using set of L.LM judges (GPT-40-mini (OpenAI, 2024) and Llama-3-70b (AI@ Meta, 2024)). We average the score of the 2 judges and then round to -1, O, or (so 0,5 is rounded to I and -0.5 to -1). To avoid position bias (Zheng et al., 2023), we run each LLM judge twice on each sample, swapping the order of the outputs. If the judge's Sigge'ssee decision is dependent on the position of the output, we deem this pair of outputs as having similar vibe and assign a score of for that judge.
Next, we use these scores to quantify each vibe on our 3 criteria and filter out any which are not well-defined, differentiating, and user-aligned. We ensure each vibe is well-defined by computing the inter-annotator agreement (Cohen's 's Kappa) for each Vj across vvalidation and remove any with Cohen's Kappa less than 0.2, which indicates a weak agreement among judges. To ensure each vibe is differentiating, we compute the separability score and discard any vibes with score below 0.05. As we explicitly prompt the model to produce vibes which provide useful insights into the behavior of language models, we assume these vibes are already aligned with users. Using the remaining k features, we run logistic regression using the scores 1/1:1 (p, OA, 0g) as features to obtain our model matching and preference prediction models.
Vibe iteration. The filtered vibes generated in the initial vibe discovery set may not capture all the differences that contribute to user preference, resulting in low model matching and preference prediction accuracy. We address this by iteratively refining our vibes based on tuples (p, P oß) that were misclassified by our prior differentiation stages. Specifically, we take the prompt output triplets that were misclassified by the model matching model and ask an LLM to find new axes on which these misclassified prompts vary. which are also not represented in the current set of vibes. We then perform the same summarization/redduee append the resulting new vibes to the existing set of vibes. We repeat this process for a fixed number of iterations j. In practice we find that after 3-5 iterations the discovery process does not find any additional vibes that significantly reduce the error rate of the model matching predictor.
procedure as before, run vibe validation/fllteingg and
RESULTS
We first validate VibeCheck by comparing its discovered vibes to those identified by human anno- tators in Section 5.1. Next, we evaluate VibeCheck on real-world user-LLM conversations with pairwise preference data, measuring the vibes' well-defined, differentiating, and user-aligned through inter-annotator agreement, model matching accuracy, and preference prediction accuracy on a heldout set. In Section 5.2 compare the discovered vibes' performance against an predefined list of common qualitative analysis criteria. Lastly, in Section 6, we demonstrate VibeCheck's 's broader applicabil- ity by analyzing model differences across summarization (Hermann al., 2015), mathematical reasoning (Hendrycks et al., 2021c), and image captioning (Lin et al., 2014: Chen et al., 2023).
Experimental setup. Unless otherwise stated, we run VibeCheck for 3 iterations, use proposer batch size of 5, and set ddisccvvey to be 20 samples per iteration. Some datasets such as MATH, CN- Maia, and COCO captions have no pre-computed preference labels; to simulate preferences, we apply LLM-as-a-judge and ensemble GPT-4o and Claude 3.5 Sonnet as judge using a similar procedure to (Zheng et al., 2023), removing any samples declared a tie. Additional details on the experimental setup and hyperparameters are given in the Section A.
We compute average Cohen's 's Kappa, model matching accuracy, and preference prediction accuracy on the top 10 vibes generated by VibeCheck a held-out set of prompt tuples with preference labels. To obtain the top 10 vibes, we apply least-angle regression on the full set of vibes returned by VibeCheck to predict model identity, then sort by the separability score. The full list of vibes discovered, LR coefficients and p-values from the model matching and preference prediction models, Cohen's 's kappa per vibe, and separability scores are in the Section G.
List of predefined Vibes, As a baseline, we prompt GPT-4o to generate a set of 10 vibes shown in Figure and Table 6 which represent common axes which LLM outputs differ.
5

---

Published as a conference paper at ICLR 2025
5.1 MEASURING VIBECHECK'S 'S ALIGNMENT WITH HUMAN DISCOVERY
In this section, we compare the findings from VibeCheck to findings obtained via human discovery to ensure that the vibes discovered and measured by LLM's align with humans. We utilize previous work (Guo et al., 2023), which collects responses written by humans and GPT-3.5 (Schulman et al., 2022) for the same list of questions and then recruits 200 annotators to look at 100-200 prompt output triples presenting the characteristics of both human responses and ChatGPT answers. This results in a set of 10 insights (vibes) which are listed in detail in Section B.
show summarization of the top 10 vibes found by VibeCheck along with the corresponding insight found by humans which align with each vibe meaning. We see that VibeCheck uncovers most of the same vibes as the human annotators, aside from (1) GPT fabricates facts and (2) GPT focuses on a literal interpretation of the question while humans address different aspects of the question and can infer hidden meaning. The inability to find these vibes is likely a weakness of our GPT proposer, as these vibes relate to the inherent weaknesses of GPT. The complete table of VibeCheck outputs is located in Figure
In Table
VibeCheck Vibes
Human Discovered Vibes
Humans include more refereaces and citations
Humans include detailed citations of papers and books.
GPT is more formal/academic, Humans are more casual/ conversational
GPT answers are typically formal, humans' are more colloquial
GPT includes disclaimers about advioe limitations
GPT refuses answer questions outside its knowledge
GPT coutions give advice, emplasizes socking professional help
GPT shows less bias harmful information
GPT has cohesive, fluid responses with clear sentence structure
GPT writes in an organized manner with clear logic
GPT strictly informarive, humans include personal ancodotes
GPT gives objective answers, hamans use subjective expressions
GPT has less emotional engagement, humans' acknowledge emotions
GPT expresses less emation, humans convey their feelings
GPT has longer, more informative responses
GPT longer more detailed responses.
GPT has more thorough & detailed responses
GPT has longer more detailed responses.
GPT has more comprehensive responses
GPT longer more detailed responses.
GPT strictly focused question. bumans diverge shift topics
GPT may fabricate facts
Table I: Comparison of VibeCheck vibes to human labels. Complete table in Figure 7, We see that the vibes discovered by VibeCheck closely align with vibes found through human analysis.
5.2 DESCRIBING USER PREFERENCE ON CHATBOT ARENA
On April 18th 2024, Meta released their ween- large language model Llama 3. On benchmarks like MMLU, Llama-3-70b outperforms Claude-3-Sonnet and Gemini 1.5. It had even stronger results on Chatbot Arena (Chiang et al., 2024), a popular platform for community-driven LLMs where users submit prompt, receive responses from 2 anonymous models, and vote on which output they prefer. On this leaderboard, Llama-3-70b ranked similarly the top proprietary models like GPT-4 and Claude3-Opus. This has led to speculation on whether there are qualitative properties of Llama that make it popular among users (Dunlap et al., 2024a).
In this section, we analyze the qualitative differences between Llama-3-700 and other top models using pairwise comparisons from Chatbot Arena. We run VibeCheck on set of combined battles (pairwise comparisons) between Llama-3-70b VS GPT-4 and Llama-3-70b VS Claude3-Opus' under three settings: using the entire dataset, and using 2 subsets of the data: STEM prompts (including coding) and Writing prompts, which include creative writing, humanities questions, and general chatting. We obtain these subsets by using GPT-4o-mint to categorize the questions as a STEM Question, Writing/Chatting prompt, or neither. The size of each subset be found in Section A.
We compare the vibes found by VibeCheck to a list of predefined vibes (Table 6) of common differences between language models which a user may be interested in. Table 2 shows that ViheCheck achieves higher model matching accuracy than the predefined vibes all categories and more iterations improve model matching and preference prediction accuracy. Furthermore, Figure shows that the vibes are more fine-grained. We summarize our other findings below:
Data: htcp::://hugginfffeeece/eaaeaeee
/.lamaa 3- 1--7-bbbtttl battles

---

Published as a conference paper at ICLR 2025
Comparing MM and PP accuracy across topics. Table shows that MM and PP accuracy is lower for STEM questions compared to writing or overall prompts. We suspect this because Llama's qualitative traits (friendliness, humor, safety, etc.) are less relevant for objective questions like coding and math, and user preferences here are influenced more by factual accuracy than stylistic traits. Conversely, VibeCheck best predicts preferences for writing-oriented prompts, style is often more important for these open ended tasks.
To understand how user preferences for these vibes vary across task domains and contexts, we analyze separability scores and preference prediction coefficients for predefined vibes in Figure 3. For writing tasks, formality, humor, and expressive emotional content positively correlate with user preference, while these traits negatively correlate with STEM tasks, where logical rigor is the most influential on preference. Conversely, logical rigor has minimal impact on preferences for writing tasks. While our dataset does not directly compare individual judgments, treating STEM and writing task users as distinct groups provides preliminary evidence of task-specific preferences. Additionally, lower separability scores for STEM tasks indicate less stylistic divergence in model outputs for objective questions like coding and math, making model identity harder to predict, consistent with Table 2.
Notable vibes for Llama-3 70B. The top 10 vibes uncovered by VibeCheck (Figure 2) highlight Llama's 's use of formatting, willingness to engage with sensitive topics, less emphasis on ethics, and a conversational, humorous style. Finer-grained vibes include Llama's 's of bold/italics to emphasize points and increased use of personal pronouns, with 'I,' 'we," and "you' appearing 3x more in Llama outputs than GPT/Claude conversations. The preference prediction coefficeients in Figure 2 show Chatbot Arena users tend to prefer outputs which are less focused on ethics, employ markdown and typographic emphasis highlight key points, and employ humor engage the user, all of which vibes which laaa possesses. We believe that this correlation between vibes and user preference can explain some of the discrepancy seen in llamas high ranking on the leaderboard comparison to models like GPT-4 which often outperform Llama.
Vibe (low >> high)
Sep Score (-04.44)
PP Coef [45,05]
Cohen
Language Professional, straightforward
Enthusiastic, friendly tonc.
Minimal use of typographic emphasis, letting the text stand alone. > Usee lypographic emphasis like bold italics highlight key points.
Typographic Emphasis.
0.64
Interactivity. Provides information passively without engaging the Encourages interaction, such as posing questions or euggeeting actione.
Formatting Completeness.
Responses are minimally formatted, relying on plain text, ab Responses include comprehensiive formatting. such as Markdown additional stylitiic elemants.
0.57
Examples Illustrations. Minimal examples Provides muttiple examples.
0.61
Use of Humor. Maintains scrious tone without humorous elements. >> Employs humor frequently to engage
0.82
Use Personal Pronouns, Rerely never uses pronouns. Frequently uses personal pramouns (), we, you).
Providee factual information without commenting on ethics. > Offers ethical considerations in responses.
Ethical Consideration.
0.53
Humility. Projects confidence and completeness without discussing limitations, Frequently acknowledges iimittions in the resporse or arees of uncertainty.
Formality Level. Uses informal or conversational language. @ Uses formal language and expressions.
0.45
Figure 2: Comparing Llama-3-70b VS GPT-4& & Claude-3-Opus on Chatbot Arena. Negative separability scores indicate Llama-3-70B aligns with the low (red) description, while negative preference coefficients show alignment with low descriptions is preferred. We see that Llama is more humorous, utilizes more formatting, provides than GPT and Claude: all attributes which correlate positively with human preference.
examples, and comments much less ethics
APPLICATIONS
We next apply VibeCheck to discover qualitative differences between models' behavior on three open-ended tasks: text summarization, math problem-solving, and image captioning. We use CNN/DailyMail (Hermann et al., 2015) for text summarization, MATH (Hendrycks et al., 2021b) with Shain-of-thoughee prompting for problem-solving. and COCO for image captioning. For CNN
7

---

Published as a conference paper at ICLR 2025
Method
Overall
STEM
Writing
M.M.
P.P.
C.K.
M.M.
P.P.
C.K.
M.M.
P.P.
C.K.
VibeCheck [1 iter]
68.68
60.00
0.42
65.20
55.99
0.44
74.09
60.58
0.51
VibeCheck [3 iter]
80.34
59.34
0.46
68.71
57.31
0.45
77.19
62.04
0.49
Predefined Vibes
72.10
61.11
0.51
65.94
58.38
0.45
75.00
59.49
0.52
Table 2: Comparing Llama-3 to GPT and Claude on Chathot Arena. We report Model Matching Accuracy (M.M). Preference Prediction Accuracy (P.P.), and average Cohen's 's Kappa (C.K) for the full dataset (Overall) and STEM and Writing categories. VibeCheck achieves higher model matching accuracy than Predefined Vibes and similar preference prediction accuracy. VibeCheck obtains the largest improvements over predefined vibes in the writing category, suggesting that for open-ended prompts, model styles differ significantly, and style has a greater influence on preference.
(low high)
Sep Score 0.4 STEM (top) Wiiting (botoo)
Coef (-0.8, 08 STEM (top) Winting (botom)
Creativity and Originality. Sticks to standard, predictable answers. > Provides responses novel imaginative scenarios.
Detail and Elaboration. Gives brief shallow responses. a Provides thorough,
nuanced, and expansive information.
Humor and Playfulness. Responds in straightforward and serious manner.
Uses humor, playful language, or wordplay to make the resporse engaging.
Formalness, Uses casuel, conversational, informal language. >> Uses formal and sophisticated vocabulary and sentence structure.
Assertiveness, Uses tentative or uncertain language. + Uses definitive, confident
Conciseness. verbose language excessive delails. minimal
words to convey point clearly.
Logical Rigor. Provides conclusions without thorough justi austfiicaiion.. Constructs well- well-supported arguments with clear reasoning.
Explicitness, Uses vague or implicit language. > States things directly and unambiguously.
Engagement. Presents information passively. Actively engages the reader using thetorical questions or interactive phrasing.
Emotional Remains neutral detached. Infuses responses with expressive emotion, making the tone enthusiastic or empathetic.
Figure 3: Comparing user preference and separability across STEM and writing tasks. On predefined list of vibes referenced in Table 2. Negative preference coefficients indicate preference for descricciittoon vibes, while negative separability scores show Llama responses align more with the low description than Claude GPT responses. For writing tasks, detailed explanations, humor, and expressive emotion positively correlate with human preference, while these traits negatively correlate with STEM tasks. Conversely, logical rigor has stronger positive impact on preference for STEM tasks. These trends are reflected in separability scores, with less separability on STEM tasks for vibes like humor and emotional tone, and more separability for logical rigor.
and MATH we use cached model predictions downloaded from HELM (Liang et al., 2023) and intentionally choose models which are ranked similarly to each other, but when running L.LM as judge to get preference labels, one model is more heavily preferred. For captioning, we generate captions on a random subset of 1000 COCO images. The vibes for each application in Section G.
6.1 WHAT DO DIFFERENT MODELS FOCUS ON WHEN SUMMARIZING?
We compare the summary styles of TNLG v2 (Smith et al., 2022) (530B) to Cohere's Command large Beta (Inc., 2023) on the CNN/Daily Mail dataset. While these models achieve a similar mean win rate on the HELM leaderboard, we see when using LLM as a preference judge, Command X has win-rate of 71.12%. Looking at the top 5 vibes located in Figure 14, we find that (1) Command X clearly states an introduction and conclusion while TNLG utilizes choppy sentences without an either (2) Command provides specific examples or anecdotes to illustrate points and (3) Command is able to capture multiple viewpoints and emotional aspects of a story while TNLG is more objective. We these qualities positively correlated with human preference, which may explain the disparity
8

---

Published as a conference paper at ICLR 2025
between correctness metrics and preference metrics. With these vibes, achieve model matching accuracy of 71.29% % and preference prediction accuracy of 61.42%.
Prompt: Summarize the article in 3 sentences.
Command X: giant filter-feeding 'lobster' took the place of whales 480 million years new fossil find has shown. The 6.5ft (two-metre) prehistoric creature was early ancestors modern crustaceans, insects spiders. At length it would have been comparable in size to athlete Usain Bolt. The remains were unearthed in Morocco and it is thought to be the oldest giant filter-feeder
(two metro) prehistoric creature TNLGv2: giant filter-feoding 'lobster took place whales 480 million years ago
belonged to the family of anomalocarididds It was 'gentle giant'.
6.2 How DO DIFFERENT LLMS SOLVE MATH PROBLEMS?
Objective tasks like math have single final answer, but the way model explains its thought process varies across models. We run VibeCheck on the MATH dataset (Hendrycks et al., 2021c) using chain-of-thouge prompting to discover how GPT-4o and Llama-405b differ their thought process and presentation. To reduce the variance seen from incorrect examples, run VibeCheck only on the questions where both models answered correctly and aim discover why GPT-4o favored in 76% of conversations. Inspecting the top 5 vibes in Figure 4, we observe that Llama-405b organizes its responses under markdown headings, adopts a more conversational tone, and includes overly detailed step-by-step explanations, as illustrated below. Examining the coefficients of the preference prediction model, we find that formal tone and frequent use of notation positively correlate with preference, while over-explaining the reasoning process negatively correlates with preference. These vibes achieve a model-matching accuracy of 97.09 % and a preference prediction accuracy of 72.79%.
Prompt: Find the positive difference between 2(6)
8(3+3)
GPT-4o:
Liama-405b:
To find the positive difference between 2(6) 4(3) and 3), first need to evaluate each expression separately.
## Step Evaluate the expression 2(6) + 4(3)
First, need evaluste the expression 216) 4(3) by following the order of operations (PEMDAS/BODMAS)
We start by multiplying and 6, which gives us 12. Then, we multiply and gives Finally, we add the two results togethar, giving us 12 12 24.
4(3)
2(6) + 4(3) 2-6 6 1- 3 21
## Step 2: Evaluate the expression 8(3+3)
Next, evaluate 813+31:
Next, we need to evaluate the expression 8(3+3] 31 following the order of operations (PEMDAS/BODMAS), We by evaluating expression inside the parentheses, which gives us 3 6. Then, we muitiply by 6, which gives us 48.
8(3 + 8-
Vibe (low -> high)
Sep Score 1-0.8,0.31
PP Coef [006,0.6)
Cohen
Explenation and Step-by-Step Detail, Continuous narrative, no explicit step labels, less granularity. Detailed labels (e.g. We Step granular breakdown.
0.90
Structural Formatting. Continuous without explicit structuring. Uses headings, subheadings, numbered lists.
Conciseness. Extended discussions, unnecessary commentary, contains repettion. > Brief, to-the- explanations, no unnecessary repettion.
0.51
Efficiency of Steps. Detailed internediary steps, broader explanations, Concise, straightforward solution steps.
Mathematical Notation Use. More written explanations, fewer symbols, Frequent use symbols, LaTeXIMethML formatting.
0.35
Figure 4: Top 5 vibes comparing GPT-4o to Llama-3-405B on MATH COT. Negative separability indicate GPT--4o aligns with the low (red) description, while negative preference coefficients show alignment with low descriptions is preferred. GPT-4o outputs contain more LaTeex/MathML. for- matting which positively correlated with human preference while 1-ma-3-405B has very structured and overly-detailed responses, which is negatively correlated with preference.
6.3 WHAT ARE VLM'S CAPTIONING STYLE?
Image captioning is one of the most popular use cases for Vision and Language models, but different captioning models focus on different image properties. We run VibeCheck captions generated by GPT-4V (Chen et al., 2023; OpenAI, 2023) and Gemini-1.5-Flaa (Reid et al., 2024) on 1000 COCO images and find that GPT-4V uses more poetic language and structures its captions as dynamic

---

Published as a conference paper at ICLR 2025
story, inferring the personality and emotions of the subjects in the image while Gemini sticks to more literal descriptions (Figure 16). The top 10 vibes generated by VibeCheck are able to achieve near perfect 99.13% model matching accuracy and 89.02% preference prediction accuracy. Although we compared the captions without the image in this experiment due to cost, the VibeCheck framework can be easily adapted to the multimodal setting.
Prompt: Describe the following image.
In the image, a young man is caught in the midst of a daring skateboarding trick on a concrete lediee at beach. His black t-shirt, adorned with white logo, contrasts with his black shorts and vibrant red shoes. He an impressive pose, balancing on the skateboard with his left foot while his right foot hovers in the air. The skateboard, tilted towards the ground, seems be defying gravity under his control.
GPT-4V:
The beach forms a livelly backdroop to this scene. People can be seen enjoying their time, adding to the dynamic atmosphere. A red and white striped awning stands out, adding a splash of color to the scene. The image captures a moment of thrill and skill, set against the relaxed ambiance of beach day.
Gemini- 1.5-Flsh:: The image shows a young man skateboarding skatepark.
wearing black shirt with white skull on it, black shorts, and red and white sneakers, He is in mid-air with his skateboard underneath him. The background is beach, and there people distance. The focused his skateboarding, and he appears to be having fun.
Creative Writing: GPT-4V: Contains narrative embellishments and subjective interpretations. Gemini- 1.m-ii-1.5-Faaatt Straightforward reporting without creative embellishments
Color and Atmosphere Description: GPT-4V: Emphasizes mood and overall ambiance, including colors and lighting. Gemini-1,5-
Flash: Focuses on identifying and detailing objects without describing the ambiance. Descriptive Richness: GPT-4V: Rich, detailed scene description with story element. Demini-1.55-Flashe Simple, factual description without additional details.
LIMITATIONS
Although VibeCheck quantifies the impact of each vibe on model identity and user preference, is challenging to disentangle whether a specific vibe directly influences human preference or if other confounding factors are at play. For example, a model might exhibit vibe of being more engaging. but its preference by users could stem from its factual accuracy, where accurate outputs incidentally appear more engaging due to their clarity or relevance. Furthermore, the -.aeMd vibe discovery process may not capture all relevant differences between models. This is particularly problematic when there's a significant discrepancy in model accuracy, as the discovered vibes may focus primarily accuracy-relate aspects. VibeCheck is also costly to validate, each judge will have to evaluate each sample in D valiidation on each vibe. In order for this to be feasible, our method uses relatively inexpensive models such as GPT-44-mini, but these judge models are often incorrect in their predictions, as shown in Figure 5. LLM judges also have biases (Zheng et al., 2023), like favoring their own outputs, which may affect the scoring. Lastly, running VibeCheck multiple times can lead to different vibes and different results, making it harder to reproduce findings exactly.
8 CONCLUSION
may seem unconventional to focus on vibes instead of concrete metrics of correctness, but these qualitative properties have measurable impact how people judge models. VibeCheck provides a valuable addition to existing metrics for correctness by capturing these qualitative aspects that influence human preference. As LLM usage expands, we anticipate increased focus on evaluating vibes to better align with user preferences. Moreover, this approach can be extended to other modalities, such audio or visual content, and can be applied to compare any pairwise set of texts, making it versatile tool for model evaluation. In future work, we hope to explore extending this framework to compare a larger number of models along with developing interventions which can use these vibes to improve human preference for given models.
Acknowledgments. We thank Ruiqi Zhong for introducing us to the joys of automated data analysis and Ion Stoica for insightful rants on evaluations beyond accuracy, as well as their feedback on the manuscript. We also thank Wei-Lin Chiang, Evan Frick, Tianle Li, and Issac Ong for co-authoring a blog post on the behaviors of Llama-3, which inspired one of the coolest experiments in this paper. Lastly, Lisa personally extends her appreciation to Joey, Jacob, and Trevor for embracing the writing of a paper that unironically uses the word "vibe" over 290 times. This paper has spawned many amusing quotes, such as: "Can we put confidence intervals on vibes?", "What if we call it 'No Numbers Just Vibes", and we replace all numbers with emojis?", and of course "I' m all vibed-out".
10

---

Published as a conference paper at ICLR 2025
REFERENCES
AI@ Meta. Llama 3 model card. 2024. URL httpt: http:////ggghhub...mmmmamaaaaaaa http///etta-llamLLLaaaaaa
blob/man www.aa/nnaaaa.. CARD mdd
Satanjeev Banerjee and Alon Lavie. METEOR:
An automatic metric for MT evaluation with improved correlation with human judgments. In Jade Goldstein, Alon Lavie, Chin-Yew Lin, and Clare Voss (eds.), Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pp. 65-72, Ann Arbor, Michigan, June 2005, Association for Computational Linguistics. URL https: https////aaaatnoloy.oooooo orgg/ W0.5 0909.
BIG bench authors. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions Machine Learning Research, 2023. ISSN 2835-8856. URL
net/ forum? www.ee/oooriddullnai....... id-uyILSSvosj.
Courtni Byun, Piper Vasicek, and Kevin Seppi. Dispensing with humans in human-computer interaction research. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, CHI EA '23, New York, NY. USA, 2023. Association for Computing Machinery, ISBN 978 45039922.. doi: 10.1445/3544499.5527749 URL httph::///ddle /tt:///oiii/ org/ 10. 10.1115//33 04444555444444 94..5444444933.7774..3
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12799, 2023.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating Iims by human preference, 2024.
Mia Chiquier, Utkarsh Mall, and Carl Vondrick. Evolving interpretable visual classifiers with large language models. European Conference on Computer Vision (ECCV), 2024.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback, 2023.
Lisa Dunlap, Evan Frick, Tianle Li, Isaac Ong. Joseph E. Gonzalez, and Wei-Lin Chiang. What's up with llama 3? arena data analysis, May 2024a. URL httts:://blog 1marena al/blog/ 2024/10 ///mmmm3///
Lisa Dunlap, Yuhui Zhang, Xiaohan Wang, Ruiqi Zhong, Trevor Darrell, Jacob Steinhardt, Joseph E. Gonzalez, and Serena Yeung-Levy. Describing differences in image sets with natural language. In Conference on Computer Vision and Pattern Recognition (CVPR), 2024b.
Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, An- uoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriam Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondfej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Majumder, Pedro Henrique Martins, Angelina -cMillan-Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Andre Niyongabo Rubungo, Salomey Osei, Ankur Parikh, Laura Perez- Beltrrbeltr, Niranjan Ramesh Rao. Vikas Raunak, Juan Diego Rodriguez, Sashank San- thanam, João Sedoe, Thibault Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. The GEM benchmark: Natural language generation, its evaluation and metrics. In Antoine Bosselut, Esin Durmus, Varun Prashant Gangal, Sebastian Gehrmann, Yacine Jernite, Laura Beltrahellr,,iini, Samira Shaikh, and Wei Xu (eds.), Proceedings of the 1st Workshop Natural Language Generation, Evaluation, and Metrics (GEM 2021). pp. 96-120, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18853/v 1/2021.gems 1.10. URL https: / antho ogy. ort.org///000 2021 gem- 10.
11

---

Published as a conference paper at ICLR 2025
Google Cloud. Perform automatic side-by-side evaluation, 2024. URL hRL google. woon/vetex-ai//eee /etee//eeeeeiiieeeeeeeeeeeeeeeee htteeaaaammeeeiidmiaaaeeeaeee//e sites/ssiiiibbdiiseeeeeeeeeeeeee
Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding. Jianwei Yue, and Yupeng Wu. How close is chatgpt human experts? comparison corpus, evaluation, and detection. arXiv preprini arxiv:2301.077 0.75977 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021a.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 2021b. URL https:://oi... /ttp:://doo..gggg////// org/10, 200550/aaXi..22. 03874.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021c.
Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines read and comprehend. In NIPS, pp. 1693-1701, 2015.
Cohere Inc.
Command-r and command-r+ models. CohereForAt/c4a1 httrr//caaccoommn---eaeeea command- 2023. Accessed: 2024-10-02.
https:/ httpss:////ugggggfac.e. co/
Minsuk Kahng, Ian Tenney, Mahima Pushkarna, Michael Xieyang Liu, James Wexler, Emily Reif, Krystal Kallarackal, Minsuk Chang, Michael Terry, and Lucas Dixon. Llm comparator: Visual analyties for side-by-side evaluation large language models. ArXiv, abs/2402.10524 2024. URL https: /appi./pppppppp swwant wes.anntiscchhaa. org. Cttp/CorpsIID::i : 7474999.
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. Dynabench: Rethinking benchmarking in nlp. NAACL, 2021.
Margaret Li, Jason Weston, and Stephen Roller. Acute-eval: Improved dialogue evaluation with optimized questions and multi-turn comparisons, 2019. URL https: /ttp:v///////////aeeeaeeaee.avea orw. /bbs/ 1909. 03087.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-follooingg models. https://githhee
wwoo/tatte -aaitssuuaaaaaaaaaee eaa, 2023.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang. Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas,, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam. Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgum, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models. Transactions on Machine Learning Research, 08 2023.
Chin- Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https: /aa/cclaan aalaaaaanannooyyoooaaaaoaaaallll .rg. W04 1013.
Tsung- Yi Lin. Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, C Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context. European conference on computer vision, pp. 740-755, 2014.
12

---

Published as a conference paper at ICLR 2025
Adian Liusie, Potsawee Manakul, and Mark J. F. Gales. Lim comparative assessment: Zero-shot nlg evaluation through pairwise comparisons using large language models. EACL, 2024.
Shikib Mehri and Maxine Eskenazi. Unsupervised evaluation of interactive dialog with DialoGPT. In Olivier Pietquin, Smaranda Muresan, Vivian Chen, Casey Kennington, David Vandyke, Nina Dethlefs, Koji Inoue, Erik Ekstedt, and Stefan Ultes (eds.), Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp. 225-235, 1st virtual meeting, July 2020a. Association for Computational Linguistics. doi: 10. 10.18653/v 1/2020. 1.28. URL https: acl htt.aaaannnntttolooooooooooo org/ 2020. s si gdial-1.2221 28.
Shikib Mehri and Maxine Eskenazi. USR:
An unsupervised and reference free evaluation metric for dialog generation. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 681-707, Online, July 2020b. Association for Computational Linguisties. doi: 10. 10.18653/v1/2020. acl-main.64. URL hitps: : /ttp//aaaatttttt wwtpllog.orrg/2220 .al.aaclllo. acl-main. 64.
OpenAl. Gpt-4 technical report, 2023.
OpenAl. Hello gpt-4o. https: http://owaa..ooo/nde////aaalla-o 2024 on 06/05/2024).
2024. (Accessed
Piotr Padlewski, Max Bain, Matthew Henderson, Zhongkai Zhu, Nishant Relan, Hai Pham, Donovan Ong, Kaloyan Aleksiev, Aitor Ormazabal, Samuel Phua, Ethan Yeo, Eugenie Lamprecht, Qi Liu, Yuqi Wang, Eric Chen, Deyu Fu, Lei Li, Che Zheng, Cyprien de Masson d' Autume, Dani Yogatama, Mikel Artetxe, and Yi Tay. Vibe-eval: A hard evaluation suite for measuring progress of multimodal language models, 2024.
Bo Pang, Erik Nijkamp. Wenjuan Han, Linqi Zhou, Yixian Liu, and Kewei Tu. Towards holistic and automatic evaluation of open-domain dialogue generation. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pp. 3619-3629, Online, 2020. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Anmual Meeting of the Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, 2002. Association for Computational Linguistics.
ChaeHun Park, Minseok Choi, Dohyun Lee, and Jaegul Choo. Paireval: Open-domain dialogue evaluation with pairwise comparison. arXiv preprint arXiv:2404.01015. 2024.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap. Captita Alayrac. Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser. et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055300 2024.
John Schulman, Barret Zoph, Christina Kim, Jacob Hilton, Jacob Menick, Jiayi Weng, Juan Fe- lipe Ceron Uribe, Liam Fedus, Luke Metz, Michael Pokorny, et al. Chatgpt: Optimizing language models for dialogue. OpenAl blog, 2022.
Thibault Sellam, Dipanjan Das, and Ankur P Parikh. Bleurt: Learning robust metrics for text generation. In Proceedings of ACL, 2020. URL
http:/axwii..gaa//////aaaa////// 04696.
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zheng, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song. Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, and Bryan Catanzaro. Using deepspeed and megatron train megatron-turing NLG 530b, A large-scale generative language model. CoRR, abs/2201. 19900 2022. URL https: / arwiv orggabb//200..1 11990.
Maya Grace Torii, Takahito Murakami, and Yoichi Ochiai. Expanding horizons in hei research through llm-driven qualitative analysis, 2024.
13

---

Published as a conference paper at ICLR 2025
Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhang- orodsky, Minjie Xu, Naomi White, and Patrick Lewis. Replacing judges with juries: Evaluating Ilm generations with panel of diverse models, 2024.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. SuperGLUE: A stickier benchmark for general-purpor language understanding systems. arXiv preprint 1905.00537, 2019a.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. Inter- national Conference on Learning Representations (ICLR), 2019b.
Yian Zhang, Yifan Mai, Josselin Somerville Roberts, Rishi Bommasani, Yann Dubois, and Percy Liang. Helm instruct: A multidimensional instruction following evaluation framework with absolute ratings, February 2024. URL https: htt.://rf.m.m/.em//e/em/m.e.. stonhstrafrrrr edu/2024/02/188
welm-instruce html.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. NeurlPS, 2023.
Ruiqi Zhong, Charlic Snell, Dan Klein, and Jacob Steinhardt. Describing differences between text distributions with natural language. In ICML, 2022.
Ruiqi Zhong, Peter Zhang, Steve Li, Jinwoo Ahn, Dan Klein, and Jacob Steinhardt. Goal driven discovery of distributional differences via language descriptions. arXiv preprint Att::2302.142333 2023.
14

---

Published as a conference paper at ICLR 2025
A EXPERIMENTAL DETAILS & DATASET STATISTICS
Dataset
# Train
# Test
Human VS ChatGPT
250
250
Chatbot Arena All
839
839
Chatbot Arena STEM
346
347
Chatbot Arena Writing
278
277
CNN/DailyMail
444
346
MATH
218
218
COCO w/ ShareGPT-4V Captions
323
346
Table 3: Dataset Statistics
Dataset
Model A
Model B
Model A Win Rate
Human VS ChatGPT
Humans
GPT-3.5
Chatbot Arena All
Instruct
GPT-4 + Claude-3-Opus
50%
Chatbot Arena STEM
Llama3-700 7amtaa3------ntree
GPT-4 + Claude-3-Opus
44%
Chatbot Arena Writing
Llama3-70b-Instre
GPT-4 + Claude-3-Opus
57%
CNN/DailyMail
Cohere Command X
TNLGv2
71.12%
MATH
GPT-4o
1lama3-4055
76%
COCO w/ ShareGPT-4V Captions
GPT-4V
Gemini- 1.5-Flash
80%
Table 4: Model Win Rates
Dataset
batch
nwm wew.eeeallvvele
nemm.finaal vibess
ileralions
Human VS ChatGPT
40
5
10
10
3
Chatbot Arena All
3
Chatbot Arena STEM
20
5
10
10
3
Chatbot Arena Writing
20
3
CNN/DailyMail
20
2
10
10
3
MATH
COCO
20
5
10
10
1
Table 5: VibeCheck Hyperparameters
num eval vibes number of vibes to validate at every iteration
= number prompt output triples to use in each iteration of the vibe discovery phase batch number of triples to feed into the prompt of the discovery LLM at once.
ilerations number of vibe iterations to perform
mum_final_ iiiees = number of vibes to evaluate at the end of all the iterations. This can be set to false, in which case all the vibes collected the iteration
We take the 1000 captions generated by GPT-4V from the ShareGPT-4V dataset Chen et al. (2023) and generate captions for the same images using the same captioning prompt using Gemini-1.5-Flas
15

---

Published as a conference paper at ICLR 2025
B GOLD STANDARD LABELS
Below is summary of key differences found by human evaluators in the HC3 dataset Guo al. (2023) listed in their paper.
Characteristics of ChatGPT
(a) Responses are well-organized, often starting with definition of key concepts before providing a step-by-step explanation and concluding with summary.
(b) Answers tend be detailed and extensive.
(c) ChatGPT generally minimizes bias and avoids generating harmful content.
(d) It refrains from responding to queries beyond its scope of knowledge.
(e) In some cases, it may generate incorrect or fabricated information.
Differences Between Human and ChatGPT Responses
(a) ChatGPT remains strictly on topic, while human responses may shift toward related or tangential subjects.
(b) It tends to provide objective, fact-based answers, whereas human responses often include personal opinions or subjective elements.
(c) ChatGPT's tone is typically formal and structured, while human speech is more conversa- tional and informal.
(d) Unlike humans, ChatGPT does not express emotions, relying solely on linguistic structure rather than emotional cues like punctuation or tone variations.
C GENERATING PRESET VIBES
Vibe
Axis Definition (low - high)
Assertiveness
Uses tentative or uncertain language. >> Uses definitive, confident statements.
Detail
Gives brief or shallow responses. > Provides thorough, nuanced, and expansive information.
Elaboration
Formality
casual, conversational, or informal language. >> formal, sophisticated language and sentence structure.
Emotional Tone
Remains neutral or detached. + Infuses responses with expressive emotion and
enthusiastic or empathetic tone.
Creativity & Originality Explicitness Humor and Playfulness Engagement
Sticks standard, predictable answers. Provides responses with novel ideas imaginative scenarios.
Uses vague or implicit language. - States things directly and unambiguously. Responds in a straightforward and serious manner. - Uses humor, playful language, or wordplay.
Presents information passively. + Actively engages the reader using rhetorical questions or interactive phrasing.
Logical Rigor
Provides conclusions without thorough justification. - Constructs well-supported arguments with clear reasoning. Uses verbose language and excessive details. Uses minimal words to convey point clearly.
Conciseness
Table 6: Predefined vibes. We prompt GPT-4o to generate a of 10 vibes which represent common axes which LLM outputs differ.
We generate our list of 10 preset vibes by prompting GPT-40 with the following:
16

---

Published as a conference paper at ICLR 2025
Preset Vibe Generation Prompt
machine learning researcher trying to figure out the major differences between the behavior of different large language models. Can you list cemmon ways which two language models can differ in their outputs?
Please output 1ist di fferences between these sets of oucputs with relation to specific axes of variation. Try to give axes that human could easily interpret and they could understand what it means be higher or lower on that specific axis. Flease ensure that the distinct
concepts used to explain what is high and low on the axis and mutually exclusive such that given any tuple of text outputs, a human could easily and reliably determine which model is higher lower on that axis.
The format should be {axis 1}: (difference) (difference)
(axis 2}:
Please output differences which have possibility of showing up in future unseen data and which would be useful for a human to know about when deciding with LLM to use. For each axis, define clearly and succinctly what constitutes a high or low score, ensuring these definitions mutually exclusive. Please give 10 differences
D ADDITIONAL VIBECHECK DETAILS
D.1 VIBE DISCOVERY
Below is the user prompt we use for vibe discovery.
Vibe Discovery Prompt
The following are the results of asking set language models generate answer for the same questions:
[PROMPT] [OUTPUT 1] [OUTPUT 2] I machine learning researcher trying to figure out the major differences between these two LLM outputs so I can better compare the behavior of these models. Are there any variations you notice in the outputs?
di fferences between these relation to specific axes of variation. Try to give axes that human could easily interpret and they could understand what it be higher or lower on that specifle axis. Flease ensure that the concepts used to explain what is high and low on the axis are distinet and mutually exclusive such that given human could easily and reliably determine which model is higher or lower that axis.
Please output
of outputs with
tuple of text outputs,
The format should be: ((axis)): low: {[1ow description]}; High: ([high description))
Vibe Summarization. To summarize the set of vibes found in the vibe discovery process, We cluster the axes using agglomerative clustering on the embeddings of the axes generated by the "tkunlp/inttuutor--"' model, and prompt GPT-40 to reduce this set by removing any vibes which similar. After this stage we are left with a set of less than 20 vibes which we use to score the outputs of each model.
17

---

Published as a conference paper at ICLR 2025
Vibe Reduction Prompt
Below 1ist of axes with description what makes a piece of text low or high on this axis. Are there any axes that have similar meanings based off their low and high descriptions? Are there any sets of axes that would convey the same information to user (e.g. level of detail)? Could any of the low and high descriptions simplified to make them easter to understand?
Please remove any axes with roughly the same neaning and simplify the descriptions what makes piece of text low high this axis. Please ensure that the descriptions of what makes a piece of text low high this axis Given any piece of text, human should be able to easily and reliably determine if this text falls high low each axis. Here is the list of axes: faxes}
distinct, useful, and mutually exclusive.
Please return the simplified 1ist of axes and the descriptions of what makes piece of text low or high on this axis. These axes should contain only one concept and should be human interpretable. Some examples bad axes include:
"Configuration Clarity: High: clearly defined structure and purpose. Low: Vaguely defined, minimal purpose." -> This axes is bad because it is not clear what clearly defined purpose means what a vaugely defined purpose means.
"Language and Communication: High: Varied/precise, complex structure. Low: Straight simple or general language." -> bad because combines multiple concepts into one axis. "Content Quality: High: High quality, engaging, informative. Low: Low quality, unengaging, uninformative." -> This axes is bad because it is not clear what high quality means nor what low quality means.
This
Some examples of good axes include:
"Corplexity: High: Complex, multi-layered, intricate. Low: Simple, fotoraarr easy to understand."
"Efficiency (coding): High: Code optimized for runtime, minimal memory usage. Low: Code inefficient, high memory usage."
Some examples of which should be combined include:
"Emotional Tone: High: Contains enotionally charged language. Low: Maintains neutral tone." and "Empathy: High: Shows empathy. Low: Only factual answers without empathy." are redundant because they both measure the enotionsl content of the text. If two similar axes are found, keep the one that is more informative or mere specitic.
Please maintain the format of the original axes and return list like ["(axis name): High: (high description) Low: {low description)", ...]. should be able to parse this output into string using ast. ast. If the original list does not contain any redundant axes, please return the original list.
the number of vibes after the first reduction step is > K, we prompt GPT-4o to reduce the set further with the final reducer prompt.
18

---

Published as a conference paper at ICLR 2025
Final Vibe Reducer Prompt
Below 1ist axes with description what makes a piece of text low or high on this axis. I would like to summarizo this 1ist to at most number representative axes.
Here the list axes: [VIBES]
These axes should contain only one concept and should be human interpretable. Some examples of bad axes include:
"Configuration clarity: High: Clearly defined structure and purpose. Low: Vaguely defined, minimal purpose." This axis is bad because it is not clear what clearly defined purpose means nor what vaguely defined purpose means.
"Language and Communication: High: Varied/precise, complex structure.
foraight simple or general language." " This axis is bad because it combines muitiple concepts into one axis. "Content Quality: High: High quality, engaging, informative. Low: Low quality, unengaging, uninformative."" -> This axis is bad because it is not clear what high quality means nor what low quality means. Some examples of good axes include:
"Complexity; High: Complex, multi-layezed, intricate. Low: Simple, straightforward, easy to understand."
"Efficiency (coding): High: Code optimized for runtime, minimal
memory usage. Low: Code inefficient, high memory usage."
Some examples of which should be combined include:
"Emotional Tone: High: Contains enotionally charged language. Low; Maintains neutral tone. and "Empathy: High: Shows emрathy. Low: Only factual answers without empathy." " measure the emstional content of the text.
redundant because they both two similar axes are
found, keep the one that is more informative or more specific.
Please return the simplified iist of <=[K] axes with any redundant
axes removed and the description of what makes piece of text low or high on this axis simplified. Are there any axes which convey roughly the information? Are there any axes where almost all samples which score highly on one axis would also sccre highly on the other?
Please maintain the format of the original axes and return numbered list. Each element should be structured as follows: "faxis name):
High: fhigh description| Low: {low description]"
D.2 VIBE VALIDATION
Prompt for ranker judge
want to compare the outputs of two language models and B) for the same prompt. would like you to evaluate where each output falls on the following axis: [V]BEI.
If you had to choose which output is higher the axis, which would you choose? Here the prompt and the outputs A and respectively;
[PROMPT] [OUTPUT A] [CUTPUT Bi
Please respend with which model you think is higher the axis and explain your reasoning. If this axis does not apply to these examples or these outputs are roughly equal on this axis, return "N/A".
19

---

Published as a conference paper at ICLR 2025
D.3 VIBE ITERATION
At iteration step i, are left with distinct vibes which well-defined and differentiating along with their scores Visk (P, о,, oB) Using these scores, we train a LR model to predict L.LM identity (i.e. "Is the response shown first LLM A LLM B?") and get the predictions on our entire set D. Assuming we have not hit the max iteration steps set by the user, we iterate if the number of samples misclassified by the model matching predictor greater than the number of prompts to perform discovery on (d). In iteration step + 1, we take these misclassified prompt output triples in batches of size batch along with the current set of vibes Vi....: 2y. and prompt the LLM to generate new differences between outputs what are not represented in the current vibes. These vibes are then reduced using the same procedure as the vibe discovery process. practice we found that often some of the reduced vibes from the discovery phase at + 1 were redundant with existing axis, preform one more deduplication step using the prompt below.
Vibe Discovery Iteration step
Given set of respenses, your task is to expand on the set of axes which have been previ ously identified by finding other clear differences between the responses that not captured by the existing axes. The expanded axes should be any differences between responses that exhaustive as possible in listing differences on as many different axes you can think of, and be specific about what constitutes high and low on each axis.
not clearly captured by the existing axes.
Your axis should be interpretable: human should easily and reliably determine which response higher, lower, or even on this axis when given a new set of responses. Please do not make your axes too broad and list as many axes as you can think of that are not covered by the existing axes. Most of these new axes should be either completely different from the existing axes or should highlight a
finegrained difference which an existing axis might broadly
cover. For instance, if an existing axis is "Enthusiasm: High: enthusiastic, Low: unenthustastic", Exclamation Points", or if an existing axis is "Cultural Context: High: culturally relevant, Low: culturally irrelevant", a new axis might be "Use of Slang". ", new axis might be "Use of Exclamation Points", or if an existing axis "Context", a new axis might be "*
new axis might be "Use of
Please think through the axes carefully and make sure they are clear, concise, and do not overlap with eachother the existing axes. Do not include any of the existing axes in your response. Your output should be in this format:
New Axes:
axis 1:
High: description of high
Low: description of low
axis 2:
High: description of high
Low: description of low
Do not incl ude any other information in your response.
20

---

Published as a conference paper at ICLR 2025
Vibe deduplication in iteration step /
Here a 1ist of axes on which two strings may vazy. Each axis has a description of what makes a string high or low on that axis.
[EXISTING AXES] INEW AXES]
It is likely that several of these axes measure similar lar things. Your task is remove any redundant axes. Think about if would gain any new information from seeing both axes, For example, "Emotional Tone: Righ: Contains emotionally charged language. Low: Maintains neutral tone." and "Empathy: High: Shows empathy. Low: Only factual answers without empathy."
redundant because they both measure the emotional content of the text. If two similar axes are found, keep the one that more informative.
Output the reduced 1ist axes, separated by a newline. All of the axes should maintain the sane format they have in the list of (axis): High: [high} Low: [low]
D.4 GENERATING PREFERENCE LABELS
prompt for generating preference labels
Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants (A and B) to the user question displayed below. You should choose the assistant that follows the user's s instructions and answers the user's question better. Your evaluation should consider factors such the helpfulness, relevance, accuracy, depth, creativity, and level of detail of their responses. Regin your evaluation by comparing the two responses and provide short explanation. Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses to influence your not favor certain objective as possible.
evaluation.
the assistants. Be
Here is the prompt and the outputs of A and B respectively:
[PROMPT] (OUTPUT |CUTPUT
Please respond with the model which contains a higher quality response. Based your analysis, please explain your reasoning before assigning a score. Use the following format for your response: Analysis: (reasoning)
Model: (A, 2, tie)
E FURTHER RELATED WORKS
Automatic metrics for benchmark evaluations. The number of benchmarks in the NLP community
has exploded in recent years, with a wealth work on providing a more holistic evaluation of language models beyond just accuracy. Several works Pang et al. (2020); Banerjee & Lavie (2005); Sellam et al. (2020), aim to improve on automatic metrics like BLEU Papineni et al. (2002) and ROUGE Lin (2004) scores to better measure how well a models output aligns with the ground truth by incorporating more nuanced evaluation criteria like factual accuracy, fluency, and conciseness. Similarly, efforts have been made I Liang et al. (2023) standardize model evaluation by evaluating models on many of these metrics.
21

---

Published as a conference paper at ICLR 2025
F LIMITATIONS
Prompt:
The following equation describes parabola. is negative, which way does the parabola open? ay"2
Liama: nice simple question! The correct answer is down.
((a)) negative, parabola opens
Technical Accuracy and Terminology: Factually correct, precise terminology Factual inaccuracies, vague terminology.
Ranker Decision: LLama Correct Answer: Left (GPT's answer)
Figure 5: Weaknesses in the mathematical abilities the LLM judge (GPT-4o-mini).
Prompt: European Thyroid Journal only
mixed open subscription?
Llamar... the European Thyroid Journal (ETJ) a hybrid open- access journal, which meens it offers both open-access and subscription based models.
GPT: of my last update 2023, European Thyroid Journal operates under an Open Access model.
Vibe: Technical Accurscy and Termino ogy: High: Factually correct with precise terminology, Low: Factual inacouracies and vague terminology.
Figure 6:
The answer to certain questions changes depending on the following parameters:
(1) When was the question asked?
(2) What is the knowledge cutoff of Model and Model B?
(3) What is the knowledge cutoff of the LLM ranker ensemble?
These types questions lead to unreliable ranker evaluations and reduced inter-annotator agreement.
G VIBES FROM EACH APPLICATION
Vibe high)
(0.5,0.5)
Goef [3.6,3.6)
Cohen
Conciseness. Elaborate and lengthy explanations. Short and the point
0.37
Citation and References, Avoids citalions, smoother text flow. Includes references and citalions crediblitt.
straighfforward. amotional engagement. Usses comforting language, acknowladges emotional challenges.
Emotional Empathy.
0.46
Technical Dopth. Simplifid, general, and basic technical explanations. + Detalled, formal, and technical explenations.
0.85
consistently include disclaimers, disclaimers or notes about advice limilations.
Legal Safety
0.29
Contextual Information. Focuses strictly the topic. a Provides additional irrelevant context and discussion.
0.39
directly, emphasis professional help. Pracicall caulious advice, emphasizes seeking professional help.
Practical Advice Safety. Addresses
Detail Orientation, Concise and limited responses covering fower aspects. A Thorough and comprehensive responses covering multiple aspects.
0.55
Response Length, Short, to-the-point reeponses. Long, informative responses.
0.50
Formality and Tone. Casual, reloxed tone with conversational language. > Formal, academic tone throughout.
0.84
Figure 7: Human VS ChatGPT outputs on HC3 (Guo et al., 2023)
22

---

Published as a conference paper at ICLR 2025
Vibe (low high)
Sep Score
(0.3,0.3)
Coef (0.7.771
Cohen
Engagement. Presents information passively. <M Actively engages the reader using thetorical
questions or interactive phrasing.
4888
Emotional Tone. Remains neutral detached. infuses responses with expressive emotion, making tone enthusiastic empathetic.
Humor Playfulness. Responds straighttioward serious manner. humor, playfu anguage, of wordplay to make the response engaging.
0.84
Creativity and Originality, Sticks standard, predictable answers. > Provides responses with imsginative
0.51
Detail exparsive information.
responses. thorough,
0.80
Assertiveness. tentative
language. detinitive, confident statements.
Explicitness. Uses vague implist language. > States things directly and unambiguously
0.43
Logical Rigor.
conclusions thorough juetification. Constructs well-supported
arguments with clear reasoning.
Conciseness. Uses verbose language and excessive detaile. >> Usee minimal words to convey a clearly.
0.40
Formalness. Uses casual, conversational, intormal language. tormal sophisticated vocabulery and sentence structure.
Figure 8: Preset vibes on Chatbot Arena [veeallerall]]
Vibe (low high)
Sop Score (-04,0.4)
[08,0.]
Cohen
Language and Tone. Professional, straightttorward tone. Enthusiastic, friendly tone.
0.51
Typogrephic Emphasis. Minimal typographic emphasis, letting typographic emphusis lkk bold or italics to highlight key points.
Interactivity. Provides informetion pessively without engeging the user. > Encourages user interaction such posing questions suggesting actions.
0.44
Formatting Completeness. Responses minimally formatted, ralying text. Responses include comprehensive formatting, such as Markdown additional stylittic elements.
Examples and Illustrations. Minimal examples. Provides multiple examples.
0.61
Humor. Maintains serious tone humorous elements. Employs humor fraquently engage the reader.
0.82
Use of Personal Pronouns. Rarely never uses personal pronouns, > Frequently uses personal you).
0.32
Ethical Consideration. Provides factuel information
Offers ethical
considerations in its responses.
Humility. Projects confidence and completeness without discussing limitations. > Frequently scknowledges imitations
uncertainty.
0.41
Formality Level. Uses informal conversational language. Uses formal language expressions.
Figure 9: VibeCheck vibes on Chatbot Arena[Overall
23

---

Published as a conference paper at ICLR 2025
Vibe (low high)
Sep Score
|.020022)
PP Coef
[-08,00.8]
Cohen
Assertiveness, tentative language. definitive, confident
Conciseness. Uses verbose language and excessive details. Uses minimal words convey a point clearly.
I
0.34
Creativity and Originality. Sticks standard, predictable answers. Provides responses with
imaginative sconarios.
Detail and Elaboration. Gives brief shallow responses. Provides thorough, nuenced, and expensive information.
0.62
Emotional Tone. Remains neutral or detached. >> infuses responses with expressive emotion, making enthusiastic empathetic.
0.45
Engagement. Presents information passively. Actively engages the reader using questions or interactive phrasing.
0.35
Explicitness. Uses vague implicit language. Statos things directly and unambiguousiy.
0.36
Formalness. Uses casual, conversational, or informal language, > Uses formal and sophisticated
vocabulary and sentence structure.
0.56
Humor and Playfulness. Responds in straightforward and serious manner. >> Uses humor, playfu response engaging.
0.59
anguage, wordplay
Logical Rigor. Provides conclusions without through justitication. Constructs well-supported arguments with clear reascning.
0.45
Figure 10: Preset vibes Chatbot Arena[STEM]
high)
[0.3,0.3)
Goef [0.0...
Cohen
Engagement Enthusiasm. response
engaging anguaaee >> The response exudes enthusiasm and engages the reader, oftnn employing exclamation points, friendly tone, and casual conversational remarks.
neutral,
0.43
Error Handling. Minimal or error handling, assumes ideal scenerios. >> Includes comprehensive inout
0.33
handling
Handling of Uncertain Information, States information definitively without diadieimers. Clearyy indicates uncertainly assumptions.
0.38
Interactivity Engagement.
focused ciarity. Engeging futorial-like,
Jargon and Terminology. Uses general language and avoids jargon. Uses specialized jargan and complex terms.
0.37
Safety and Accuracy Emphasis. Lacks explisi emphasis safety or ethice. Includes disclaimers, emphasizes derations.
Tone and Enthusiasm. Noutral, utlitarian. Engaging. onthusiastic.
0.44
Figure 11: VibeCheck vibes on Chatbot Arena [STEM]. Note that we only find vibes which achieve separability score on the training set about the 0.05 threshold.
24

---

Published as a conference paper at ICLR 2025
Vibe (low high)
Sep Score 1-04..04)
PP Coef
[-06,0..]]
Cohen
Assertiveness,
language. definilive, confident
Conciseness. Uses verbose language and excessive details. Uses minimal words convey a point clearly.
0.36
Provides with Detail and Eleboration. Gives brief shallow responses. Provides thorough, nusnced, and expensive information.
Creativity and Originality. Sticks standard, predictable ideas imaginative scenarios.
0.84
Emotional Tone. Remains noutral or detached. > infuses responses with expressive emotion, making
empathetic.
0.55
Engagement.
Presents information passively. Actively engages the reader using rhotorical questions or interactive phrasing.
0.55
Explicitness, vague implict language. things directly unembiguousy.
Formalness. Uses casual, conversational, or informal language, > Uses formal and sophisticated
vocabulary and sentence structure.
0.60
Humor and Playfulness. Responds straightforward and serious manner. Uses humor, playfu laguuae, wordplay
response engaging.
Logical Rigor. Provides conclusions without throughh justification. Constructs well-oupported arguments with clear reascring.
0.45
Figure 12: Preset vibes on Chatbot Arena [Writing]
Vibe (low high)
Sep Score 1-04..04)
PP Coef 607,771
Cohen
Hlumanees/RRlltabbien Formal technical language. Relatable and human-like language.
0.40
monotonous. Injects
Humor. Remains serious or formal, with no attempt humor even suitable contexts. Incorporates humar light hearted elements that enhance the response and ft the context.
0.55
Creativity. Predictable storylines. Unique and imaginative ideas.
Structural Organization. Unorganized respanses lacking clear structure. Clearly structured responses with headings list..
0.55
Empathy. Detached and indifferent. Deep understanding emotions.
0.53
Consistency of Persona. Displays inconsistency in lone and Maintains a consistent voice style throughout.
Ethical Nuance. Offers black-ard white viewpoints. > Considers moral complexitis.
0.52
Formality. Relles informal, casual, conversational language, relexed Inconsistent
Uses structured professional, end polished language, maintsining formal tone throughout
Caution. Offers bold risky suggestions without considering polantial drawbacks limitations, Provides careful, messured responees that consider potential risks consequences, showing prudence.
0.45
Figure 13: VibeCheck vibes on Chatbot Arena [Writing]
25

---

Published as a conference paper at ICLR 2025
Vibe high)
Sep Score [-04,04]
PP Coef [22323]]
Cohen
Conciseness. Detailed and sometimes overly risking redundancy. Summeries are concise and clear with minimal detall.
Tone on Emotional Aspects. Objective tone, factual summaries without emotion. > Capturas emotional sspects, includes quoles.
0.44
Personal Details, Omits personal details, summanizes key facts, Includes names and direct quotes individuals.
Specificity Examples. Lacks concrete examples, speaks in generalities. Indludes specific
0.45
examples or aneodotes ilustrate points.
Emphasis on Cause and Effect. Focuses event sequence, less cierity in causelity... relationships clearly.
Highlights cause
0.26
Coverage Muttiple Viewpoints, multiple perspectives or verwpoints.
single perspective.
0.28
Infroduction and Contextual Background. Minimal or absent introduction; reads like bullet points. Provides broad context setting intraductory sentences.
0.37
Contextual Emphasis.
narrowly
Emphasizes
elaments and contexts.
0.44
Depth of Explanation. Offers surtace-lavel explanations, lacks depth. Provides deep, thorough explanations.
0.48
Conclusion Ends abruptly lacks conclusive statements. iplications at the end.
Figure 14: VibeCheck vibes comparing TNLGv2 to Command Large Beta on CNN//Dallyail Summarization (Hermann et al., 2015).
Vibe high)
Score (0.8.099)
Coef [00,00
Cohen
Mathematical Notation More written explanations, symbols. symbols, LaTeXIMethML formatting.
Efficiency of Steps. Detalled internediary steps. broader explanations, > Concise, straightforward solution steos.
0.42
Extended discussions, unnecessary commentary, contains repetition. Brief, to the- point explanations, no unnocassary repetiion.
Concisoness.
Continuous prose without explioit structuring. Uses headings. subheadings, numbered lists.
Structural Formatting,
0.70
Explanation and Step-by-Step Detail, Continuous narrative, explici step labels, granulerity. steps, (e.g., Step 1'), granular breakdown.
Figure 15:
VibeCheck vibes comparing GPT-4o to Llama-3-405B on MATH CoT (Hendrycks et al., 1c). We only find 5 vibes because the vibe reduction step is not required to return < 10 vibes and in this case found only 5 distinct vibes which are able to almost perfectly separate model outputs.
26

---

Published as a conference paper at ICLR 2025
Vibe (low high)
Sep Score
1-1.0.100)
Coef [15.6115.)
Cohen
Detail Orientation, Gives general or minimal description; lacks specificity... Provides intricate details; describe specific elemants.
0.32
Background Details, Provides minimal no information about the background. Offers exlensive descriptinn surroundings.
0.56
Emotion Relationships. Provides neutral descriptions rel reletionships, > Infers emolions and relationships between subjects.
attributing feolings
0.60
Color and Atmosphere Description. Focuses on identfyiing and detailnng objects witouu describing emblance. induding colore lighting.
0.63
Space.
spatial depth
sctting.
0.42
Focus on Context. Focuses enumerating visual elements without added context, > Provides added context narraiive elements.
0.43
Contextual Storytelling. Sticks appearance narretive. and interpretstions.
context
Artistic or Poetic Language, Employs straightforward factual, and plain language. Uses vivid, maginative, and expressive language.
0.37
Descriptive Richness, Simple, factual description without additional details. > Rich, detailed scene description story elements.
0.36
Straightforward reporting without creative embellishments. Contains nerrative embellishmnnt and subjectivv interpretations.
Creative Writing.
0.42
Figure 16: VibeCheck vibes comparing Gemini-1.5. 1.5-Flash to GPT-4V COCO Captions (Lin et al., 2014).
27

---

Published as a conference paper at ICLR 2025
H
MORE LLAMA VS GPT EXAMPLES
Prompt: If was mouse would be squeaking terms?
Liama-3-70b: Anahahal Oh, what delightful punt As a conversational AI, I'd say we're definitely on squeaking temis... should say, paws paws-Itive terms? Squeak squeak!
GPT-4: you were mouse, we'd i find way communicate effectively, whether that invoives squesking,
gestures, or any form of interaction that works for both of us. I'm here to assist, regardless of your form!
Use Humor: Llama- 3-70: Employs without humorous elements.
frequently engage reader. GPT-4: Maintains serious tone
Formality Level: Liama-3-70b: Uses informal or conversational language. GPT-4: Uses formal language.
Language and Tone: 3-7ma--700: Enthusiastic, friendly tone. GPT-4: Professional, straightforward tone.
Figure 17: Squirrel example from Chatbot Arena
Prompt: what happens to olympic medals when othelets are caught disping after the event?
1ama---700: When on athicte cought doptng oht an Olympic event, the tnte notioaal Otymplc Committee UDCi has process in paace dest with the otucdon Here's what typically happons:
=*irvssligatonn and Testingnn: The IOC, colaboration with the World Anti Doping Agency (WADA) and the interaionall Fedoration oFI the sport involved, conducts an mestigttin and besting confirm the dosing via ation
""Daccul dication
locationn) dsp atton a contimed,
diquustiie the atitete from the event and stripe their medal. The Ioc then realiocatis raturn
estatum Mede The disqueiliin wihien required ritumn their medel IDC, the ath recnume
the IDc take fur action, Inclucingg anwicnn *6G-sstrcation Canemory th in come coses, a new medal ceremoty may the to award the meds to the new reciplentt)) Otympic records results
Comectionns:
updates
changes.
Some nosabli exomples.
munnnc Aerratinng (Ccyling,
stripped pranze model in incividtaal time after ectrllting dasping.
2000/ "Ruueann Olympi Team (i01e Saate Weneer Clyyp Mu athinas disqueiti striped medals due state-sponmeeee program.
golc
doping.
Itss worts roting that the 100 har a statute of Iiiitations for doping vistor ons, which currentty set at years from the date of the Olympic Gomee, This meane that it doolng otton year perfod, (DC may
diocovered
The 100 anti doping efforts aim protect the integrity of the Olympic Games and ensure level playing flold for athrenes.
3 Opaac attictes are caught ng disque flad after Olympic event, following typioaily happers models
two sitdele must return thei mecel International Dlympic Comm JOc3
The IDC then real acatre the medals to the next elginle orrietee based on the updated resuits.
The athictes originalt ty finished bohi the oisqual compettorr owarded the appropriate medals.
If the attiste who discua riedd won gold. the ver medalsst is upge aded to godd, the bronze medelit is upgreded to silver, and the fourth ii soe finishur receives the bronze medal discual sth won altver or brenae, medata are rea localed accor dingly the finahers event.
medat ceremories conducted present the new medals the rightful winners.
Ii's Importtnn that the prooees an medal medlocation, and suboequent cewmontee ee can cometimee tike years, dependinng the timing the deping revelation completion egal proceetings appear..
Exampies:: Liama-3-7000 70t: Broad range of detailne examples. Claude O-Opux: Fevver less deta led ecampies.
Typographic Emphasie: Llama-3-700 Uses typographic emphasis
italics highlight key points, Claude-3-Opus
of typographic
amphasis, letting the
Depth of Anelysis: Liamne3 70b: Detailed and in-depth analysis with numerous dislinct points. Cloude-3 Opus: Concise enelysis with fewer, general points.
Figure 18: Olympics example from Chatbot Arena
28

---

Published as a conference paper at ICLR 2025
Prompt: Is Iranian supreme leader irl Dr. Evill
Llmma-3-70b: delightful question!
While Iranian Supreme Leader Ali Khamenel may controversial figure, I'm sfraid he's not quite the same Evil, the fictional vilain Austin movie series.
Here few
1. No cat- ft--stroong"*: * Unlike Dr. EVil, Khamenel hasn't been known to stroke a cat menacingly while piotting world domination. He's religious and political leader, not Bond viliain.
2. "No outlandish schemas**: While Khamenel has involved various poltical and military conflicts, there's evidence
concocting absurd plans like Dr. Evil's moon-based laser beem or sharks with frickin' laser beams attached to their heads.
3. *NNo comically exaggerated persona**: Khamenei is serious political figure, caricatured villain with entrances, over-the- too monotogues, or outrageous fashion sense.
for dramatic
4. *NNo global domination ambitions** While iran's supreme leader may heve regional and ideological ambitions, there's no indication he world hold
seeks take
(unlike Evil's constant threats destroy world unless demands
In all seriousness, Ali Knamenei is complex figure with significant impact on Iranian pelitics and international relations. While his views have widaly criticized, essential approach discussions about him with issues involved.
respect complexity
Khamenel may
Evil, hope response brought
your
GPT-4: No, it would be inacourate and inappropriste to equate tran's Supreme Leader with fictional villain like Dr. Evil from the Austin Powers movies, While tran's government leaders to over top comedic characters trivializes real world complexities and sansitivities around this topic.
and face significant criticism internationally, making glib comparisons
Iran's Supreme Leader religious political position currently held Ali Khamerai. The role wields signif power tran's theocratic system. There many valid concerns and debates regarcing Iran's human rights record, nuclear program, regional influence and tensions with other countries. However, these issues deserve to be discussed seriously with factual information, not facile pop culture references.
Whenever discussing foreign leaders and govemments, especially those that are viewed negatvely by many. it's importent to thoughtful and avoid hyperbale caricature. Even when we disagree with a regime, should strive to understand the historizal context and represent objectively
Demonizing others, those oppose, rarely
greatar understanding positiva change.
Ethical Considerations: Llama-3-70b: Provides factual information without commenting on ethics.
Claude-3-Opus: Offers ethical consideration in its responses.
Typographic Emphasis: Llama-3-70b: Uses typographic emphasis like bold or italics to highlight key points.
Claude-3- 3-ouss: Minimal use of typographic emphasis, letting the text stand alone.
Depth Analysis: Llama-3-70bs Detalled in-depth analysis numerous distinct points.
Claude-3- :-ode-:--pus:::: Concise analysis with fewer, general points.
Figure 19: Supreme Leader example from Chatbot Arena
29


Reviews:
[{"review_id": "hRwoi1Lrmb", "reviewer": "Reviewer_CAD4", "summary": "The paper’s goal is to discover LLM vibes, i.e. traits of a model which are well-defined, differentiating and user-aligned. The main contribution of the paper is a method (VibeCheck) for quantifying vibes of LLM models. The method first employs a LLM model to extract a set of vibes, then decide which model outputs are conforming to that respective vibe based on a panel of LLM judges. The authors validate to what extent their method aligns with human responses and user preferences in the context of three different tasks: text summarization, math problem solving and image captioning. This approach allows them to quantify vibes of popular LLM models, finding that Llama3 outputs tend to be more friendly in nature compared to outputs from GPT-4 and Claude, while the latter two models tend to be more formal in nature.", "questions": "Your definition of vibes is similar to how concept axes are defined in interpretability, have you tried connecting the two?\n\nWhy discover vibes and validate on gold labels when you can actually use the gold labels directly?\n\nHave you considered the impact of the prompts you are using on eliciting specific model vibes? \n\nFigure 1: It is not clear from the example provided for “What is a vibe?” that Output A differs from Output B in the friendliness direction (formal → friendly). Which answer is more formal and which answer is more friendly? Also why Vi() takes values of 1 and -1 when the prompt for the judge is asking the model to answer with A, B or equal? Figure 1 caption needs more explanation.\n\nTable 4 - is the top table for overall results? Please indicate what the abbreviations mean - Model Matching (MM), Preference Prediction (PP)\n\nLine 336 - typo “sensitive”", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is making an important observation that current LLM evaluations are mainly focusing on one textual dimension (for eg., correctness) at the expense of other aspects that matter for the end user, failing to capture the open-ended nature of LLM applications, nuanced qualitative aspects and their dependence on subjective user preferences; LLM evaluation is indeed an open and important research topic\n\nThe authors propose a definition of vibes and discuss what makes a vide valid accounting for aspects such as well-defined, differentiating, then validate the identified vibes against human gold labels and aim to quantify the differences between popular LLM models using their method", "weaknesses": "“Vibes” are subjective and not a well-defined concept; the paper defines vibes as identifying traits of LLM models and in contrast to concrete evaluation measures which measure objective aspects of the text, evaluating subjectivity is less clear and straightforward\n\nThe authors consider a vibe to be well-defined if multiple LLM judges agree, however it is possible that those LLM judges all have the same limitations; it is also unclear to what extent LLM judges decisions on what constitutes a valid vibe aligns with humans\n\nThe paper acknowledges that vibes are user and task dependent, however there is no accounting for these aspects in the experiments\n\nThe paper would benefit from more structure, more in-depth analysis and more detailed figure/table captions. The extra space could be used for including a discussion section to summarize all the findings and takeaways.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iDThd4aB30", "reviewer": "Reviewer_HgoV", "summary": "The paper introduces VibeCheck, an evaluation pipeline to identify and compare the qualitative characteristics (\"vibes\") of LLMs. They employed three stages - (1) vibe discovery using thoroughly designed prompts; (2) vibe validation; and (3) vibe iterations. They also propose a method to quantify the usefulness of VibeCheck by calculating several metrics such as agreement, model matching accuracy, and preference prediction scores. Then, the paper tested VibeCheck across various tasks such as summarization, math, and image captioning, and it revealed how different models exhibit unique behaviors across different domains.", "questions": "- Sec 3: it could strengthen your claim of selecting those three evaluation criteria based on solid literature work. At least, it could be better to provide why these three criteria are important in measuring such qualitative behaviors of LLMs. \n- L220-225: what are the criteria for those threshold setups (0.2, 0.05)? \n- Please provide a better visualization of Table 3. It is hard to interpret the scores and their relationship with llama. \n- Appendix B: as mentioned these are direct quotes from the original HC3 paper. Please put a citation of this paper; otherwise, it could be regarded as flagging behavior for ethics review. \n- Please combine the main text and appendix into one.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This work provides a novel ground to assess the subjective characteristics of LLM outputs that usually differ by task purposes. Not much work has focused on this angle of LLM evaluation. \n- The paper attempted to provide a comprehensive framework to develop an automatic detection of \"vibes\" in different LLM outputs and how aligned these outputs are with human preferences.", "weaknesses": "- Overall, the paper is hard to read and understand. A large amount of effort is needed to restructure the organization of the paper.\n- Too many typos and grammatical errors are found in both the paper and the appendix. Needs to be corrected. \n- Results seem too weak to claim the usefulness of VibeCheck. For example, in Table 4, VibeCheck (even 3 runs) does not perform better than the pre-defined baseline, increasing doubt about whether this framework does better than a naive GPT4o model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yAgpPRZZna", "reviewer": "Reviewer_513D", "summary": "LLM-generated responses are mostly evaluated on their correctness, dismissing other important dimensions such as its tone or other subjective dimensions. \n\nDistinctive traits of an LLM are called _vibes_ in this paper. The paper introduces an approach, VibeCheck, to finding and measuring such vibes. It consists of an LLM that qualitatively compares a pair of models to find distinctive vibes between the two. It identifies which dimensions the generated outputs of the models differ on to get a set of vibes. LLM judges are then used to determine where the model's output falls on the vibe.  \n\nThis is evaluated on three fronts: \n1. well-defined (agreement among multiple users)\n2. differentiating (ability. to distinguish between LLMs) \n3. user-aligned (predictive of user preferences) \n\nThe authors conduct multiple experiments to evaluate the method.", "questions": "In the vibe iteration process, how many more vibes were found?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem this paper addresses is very important and the approach is creative. \n2. Table 1 shows the utility of the paper, it shows how similar the vibes are that VibeCheck finds to the ones that humans find. \n3. The authors conduct extensive experiments to show how it performs on different datasets. \n4. The paper is well-written and easy to follow.", "weaknesses": "1. A batch is used to find the vibes, however, there is not much discussion about strategies to select this batch. If the batch contains similar prompts and is low in diversity, then not all vibes might be considered. E.g. for vibes on STEM prompts to work, it will need to see something similar in the vibe batch, however, it is not ensured that this will be the case. \n2. The LLMs used are still on the larger side (GPT4o-mini).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v7KZDKWT3k", "reviewer": "Reviewer_a334", "summary": "This paper aims to automate the vibes-based evaluation that is common among LLM practitioners. They first define three characteristics of a useful vibe and then use an LLM-aided process to summarize model output into different axes of vibes and iteratively refine them with model misclassification based on these models. They then conduct validation using both a data with gold answer as well as open-ended metrics. Afterwards, they apply VibeCheck to tasks including summarization, mathematics and image captioning.", "questions": "1) Could you inspect the cases that, despite all these vibes being generally predictive, the model matching prediction or the preference predictions are wrong? Could you try to “VibeCheck” those cases?\n\n2) In the CNN/DailyMail experiment, the preference prediction accuracy is as low as 61.42%, which is not a whole lot better than random. Why should I still believe, then, that the vibes discovered in this case are “real” and actually useful?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "This paper is well-motivated, as it basically provides an automatic way of conducting the “vibe-based” evaluation which is very commonly practiced nowadays, as the evaluation of LLMs have become difficult. The writing of the paper is very clear and easy to follow. The results presented are overall convincing. Also, it is very nice to review some HCI literature in the paper.", "weaknesses": "1) the author did not engage with the line of work that tries to characterize human preference into high-level principles (e.g. https://arxiv.org/abs/2406.06560, https://aclanthology.org/2024.acl-long.99/). It would be much appreciated if the authors could situate their work around these and articulate their contributions more clearly. \n***\n2) In Table 4, we see that the predefined vibes get to almost the same preference as VibeCheck. Thus, I am not entirely convinced whether we would need to discover these vibe aspects at run time, as something that could be done a-priori is already quite good, also given the compute overhead as mentioned? Perhaps you could convince me if you show that individual user considers drastically different vibes or the vibes considered for different application are extremely different.\n***\n3) More analysis on model matching and preference predicting accuracy would be great – why are these results so drastically different for each task? Does this also point to a limitation of the applicability of your proposed method, the fact that the \"explained variance\" of human preference by vibes are so different across tasks?\n***\n4) I think it is important to note somewhere in the paper that these vibes (and the way you conduct experiment) are post-hoc explanations that are correlated with user preference. However, they are not necessarily the reason that a user actually prefers one option over the other in real time. In other words, one limitation is that your study lacks ecological validity. However, I think this is ok in this context, as long as you acknowledge it. Of course, if you could show that these vibes are also things the users actually consider when they indicate their preference real-time, I will also be convinced.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)