VibeCheck: Discover and Quantify Qualitative Differences in Large Language Models

## Abstract

Abstract Large language models (LLMs) often exhibit subtle yet distinctive characteristics in their outputs that users intuitively recognize, but struggle to quantify. These ”vibes” – such as tone, formatting, or writing style – influence user preferences, yet traditional evaluations focus primarily on the singular vibe of correctness.
We introduce VibeCheck , a system for automatically comparing a pair of LLMs by discovering identifying traits of a model (“vibes”) that are well-defined, differentiating, and user-aligned. VibeCheck iteratively discovers vibes from model outputs and then utilizes a panel of LLM judges to quantitatively measure the utility of each vibe.
We validate that the vibes generated by VibeCheck align with those found in human discovery and run VibeCheck on pairwise preference data from real-world user conversations with Llama-3-70b vs GPT-4. VibeCheck reveals that Llama has a friendly, funny, and somewhat controversial vibe. These vibes predict model identity with 80% accuracy and human preference with 61% accuracy. Lastly, we run VibeCheck on a variety of models and tasks including summarization, math, and captioning to provide insight into differences in model behavior. VibeCheck discovers vibes like Command X prefers to add concrete intros and conclusions when summarizing in comparison to TNGL, Llama-405b often overexplains its thought process on math problems compared to GPT-4o, and GPT-4 prefers to focus on the mood and emotions of the scene when captioning compared to Gemini-1.5-Flash. Code can be found at https://github.com/lisadunlap/VibeCheck

## 1 Intro

> vibe check : A process by which a group obtains a subjective assessment of another person, place, or thing. – Urban Dictionary

How a large language model writes a story, explains a concept, or edits an essay can be evaluated along many different dimensions such as creativity, formatting, and writing style. However, most evaluations focus on one dimension: *“correctness”*. State-of-the-art in evaluation methods remain largely focused on measuring accuracy for question answering and analytical reasoning tasks (Hendrycks et al., ; Wang et al., ; ; Hendrycks et al., ), and methods which aim to provide a more holistic view of LLMs (Zhang et al., ; Padlewski et al., ; Mehri & Eskenazi, ) rely on predefined concepts like conciseness, clarity, and trustworthiness to measure a model’s performance.
These evaluation approaches fail to capture the open-ended nature of LLM applications and the critical dependence on subjective user preferences and context of the task.
For instance, tone and creativity might be crucial in creative writing, whereas efficiency and readability are crucial in coding tasks.
To best inform users of which model would be best for their needs, we require flexible evaluation methods that can both *discover* and *measure* the relevant axes to evaluate for a given task.

When interacting with a set of LLMs for an extended period, a user can often tell which model generated a particular response by looking at certain traits of the outputs.
We define these identifying traits of models as “vibes”. For instance, users have found Llama-3 outputs tend to be more friendly compared to outputs from GPT-4 and Claude which tend to be more formal (see Figure 1); in other words, Llama-3 ranks high on the friendliness vibe, defined by the axis formal $\to$ friendly.
Using these insights, we might select Llama for customer service tasks and Claude for coding tasks.

Understanding these vibes helps inform the development and deployment of models, but discovering and validating them for each model can be time-consuming and difficult. To address this, we outline how one can find and, more importantly, measure an LLM’s vibe by formalizing three necessary and quantifiable traits of a useful vibe: *well-defined* (agreement among multiple users), *differentiating* (ability to distinguish between models), and *user-aligned* (predictive of user preferences).

We introduce VibeCheck, a system which qualitatively analyzes pairs of models by automatically finding well-defined, differentiating, and user-aligned vibes.
Motivated by recent work in using LLM’s in lieu of human judgment (Zheng et al., ; Zhang et al., ; Zhong et al., ; ; Dubois et al., ), VibeCheck models the qualitative analysis process by identifying the axes on which these model outputs differ to obtain a core set of vibes (e.g friendliness).
Once these vibes are obtained, VibeCheck employs a panel of LLM judges (Verga et al., ) to determine where each model’s output falls on this vibe (e.g. more formal or more friendly) in order to obtain numeric scores which are then used to measure a vibe on each of our 3 key criteria.

We run VibeCheck on several datasets to evaluate its effectiveness across different scenarios in Section 5. First, we validate that the vibes discovered by VibeCheck align well with human-annotated differences between ChatGPT and human responses using the Human ChatGPT Comparison Corpus (HC3). Next, we demonstrate that VibeCheck outperforms a preset list of vibes in predicting user preferences on real-world comparison data from Chatbot Arena, achieving 80% accuracy at predicting model identity and 61% accuracy and predicting user preference. Inspecting the vibes of VibeCheck, we find that Llama-70b uses more typographic emphasis, more examples, and is funnier than GPT-4 and Claude-3-Opus. Conversely, we find that GPT-4 and Claude comment much more on ethics and limitations than Llama, which is more willing to give controversial responses.

Lastly, in Section 6 we apply VibeCheck to several applications: text summarization on CNN/DailyMail, math problem-solving on MATH, and image captioning on COCO. Using VibeCheck, we find insightful qualitative differences between models with similar accuracy on correctness metrics but differing user preferences. For instance, Command X prefers to add concrete intros and conclusions when summarizing in comparison to TNGL, Llama-405b often overexplains its thought process on math problems, and GPT-4 prefers to focus on the mood and emotions of the scene when captioning.

## 2 Related Work

Aspect-based evaluations. The number of benchmarks in the NLP community has exploded in recent years, with a growing body of work on exploring a more holistic evaluation of language models.
Several works (Pang et al., ; Banerjee & Lavie, ; Sellam et al., ) aim to improve on automatic metrics like BLEU (Papineni et al., ) and ROUGE (Lin, ) scores to better measure how well a models output aligns with the ground truth by incorporating more nuanced evaluation criteria like factual accuracy, fluency, and conciseness. Similarly, efforts have been made (Liang et al., ; bench authors, ; Kiela et al., ; Wang et al., ; ) to standardize model evaluation by evaluating models on many of these metrics across various tasks.

Moving away from measuring model outputs on ground truth responses, work from Mehri & Eskenazi (); Zhang et al. (); Li et al. (); Mehri & Eskenazi (); Gehrmann et al. () evaluate model outputs on criteria like helpfulness and clarity using LLM judges on more open ended tasks like dialogue, role-play, and summarization. While these efforts supply a great foundation for measuring correctness, they all define the axes on what makes something correct beforehand. In contrast, VibeCheck aims to automatically discover these axes (vibes) and verify their utility to the user by measuring the correlation between vibes and human preference.

Pairwise comparison of LLMs.
HCI tools like Google’s AutoSxS (Google Cloud, ) and LLMComparator (Kahng et al., ) explores the current state of human powered LLM qualitative evaluation through interviews with data analysts. These works find that practitioners often eyeball individual examples to interpret and look at qualitative differences between the outputs of two models, and develop an interactive web based application for users to inspect side-by-side LLM outputs with an LLM based rationale as to why one output is preferred over another. While these works are focused more on software tools rather than a pipeline which can be quantitavely verified, these HCI findings inform VibeCheck’s vibe discovery mechanism to align with the human-powered qualitative process. Moreover, many NLP works (Zheng et al., ; Verga et al., ; Li et al., ; Park et al., ; Liusie et al., ) have explored using LLMs to predict user preference given responses from two models, showing these preference predictions often align with the judgements of human annotators. While these efforts focus more on the user experience, it does not provide an interpretable view of exactly why these users prefer one output over the other.

Discovering separable traits in unstructured data. In parallel to works in the machine learning community on LLM evaluation, there has been fantastic efforts in the HCI community on comparing generative model outputs as well as on using LLMs for qualitative analysis. Works like Torii et al. (); Byun et al. () use LLMs to generate discussions from qualitative research data to automate the data analysis process, but note the lack of comprehensive evaluation metrics. Automated data analysis on unstructured data has also been explored in Zhong et al. (; ); Dunlap et al. (), which use LLMs and VLMs to propose and validate candidate differences between two sets of text or images in the form of “set A contains more X”, and Chiquier et al. () employs an evolutionary algorithm to find text descriptions which best separates image classes to assist in zero-shot classification. We extend these works to pairwise inputs and introduce metrics of success which can better verify the separability, consistency, and alignment of these differences.

Figure: Figure 1: Core components of VibeCheck.
Refer to caption: x1.png

## 3 Vibe-based Evaluations

We define a *vibe* as an axis along which a pair of texts can differ (e.g., “formal $\to$ friendly”) that is perceptible to humans.
A vibe $\nu$ is represented by a text description of the axis along with a definition of what it means to be high or low on this axis (e.g. “Tone: low = formal, high = friendly”, see Figure 1). Identifying vibes aids users in selecting models that best suit their specific tasks. In this work, we focus on comparing the vibes of two models by discovering the axes on which their outputs differ and quantifying the utility of these vibes.

Consider a dataset $D$ composed of triples $(\mathrm{p},\mathrm{o}_{A}^{\mathrm{p}},\mathrm{o}_{B}^{\mathrm{p}})$ and preference labels $y_{\mathrm{p}}$, where $\mathrm{p}$ is a prompt and $\mathrm{o}_{i}^{\mathrm{p}}$ are the outputs from models $A$ and $B$. For each triple, a judge (human or LLM) assigns a score for vibe $\nu$, denoted $\nu(\mathrm{p},\mathrm{o}_{A}^{\mathrm{p}},\mathrm{o}_{B}^{\mathrm{p}})\in\{-1
,0,1\}$, which indicates whether model $A$ scores lower (-1), similarly (0), or higher (1) than model $B$ on this vibe. Thus, a vibe imposes an ordering on model outputs.

We define 3 key criteria of a useful vibe; it should be *well-defined*, *differentiating*, and *user-aligned*.

Well-defined: multiple evaluators agree on the ordering of outputs along the vibe. We quantify this by having two different judges (typically LLMs) compute $\nu(\mathrm{p},\mathrm{o}_{A}^{\mathrm{p}},\mathrm{o}_{B}^{\mathrm{p}})$ across dataset $D$ and report Cohen’s Kappa to assess agreement.

differentiating: one model’s outputs consistently rank higher on this vibe compared to the other’s across a set of prompts.
We quantify this by calculating a *separability score* for each vibe, which measures how consistently the vibe distinguishes between the two models across all samples.

$$ $\displaystyle\mathtt{sep_score(\nu)}=\frac{1}{\mid D\mid}\sum_{\mathrm{p}\in D }\nu(\mathrm{p},\mathrm{o}_{A}^{\mathrm{p}},\mathrm{o}_{B}^{\mathrm{p}})$ $$

To measure separability across a set of vibes, we fix a pair of models $(A,B)$ and measure the accuracy of using $\nu(\mathrm{o}_{A},\mathrm{o}_{B})$ to classify which output came from which model. We also more generally measure separability for a set of vibes $\nu_{1},\ldots,\nu_{k}$, by using $\nu_{1:k}(\mathrm{p},\mathrm{o}_{A},\mathrm{o}_{B})$ as a $k$-dimensional feature vector, then training a linear classifier to predict model A vs. model B, and reporting accuracy on a held-out set. We refer to this metric as model-matching accuracy.

User-aligned.
One potential use of vibes is to better understand human preferences.
While a vibe like *“frequent use of the letter ‘e”’* may be differentiating, it is unlikely predictive of human preferences.
We assume our tuples $(\mathrm{p},\mathrm{o}_{A}^{\mathrm{p}},\mathrm{o}_{B}^{\mathrm{p}})$ are annotated with user preferences $y\in\{-1,+1\}$, indicating which model’s output is preferred. We train a logistic regression classifier to predict $y$ using the same feature set $\nu_{1:k}$ as above, reporting held-out accuracy. We refer to this metric as *preference prediction accuracy*. We can measure the influence of a single vibe on preferences by examining the coefficients and p-values of the preference prediction model.

VibeCheck automatically finds high-scoring vibes across the three criteria through an iterative process: (1) discovering vibes, (2) computing their scores, (3) selecting those meeting all criteria, and (4) focusing on tuples $(\mathrm{p},\mathrm{o}_{A}^{\mathrm{p}},\mathrm{o}_{B}^{\mathrm{p}})$ where existing vibes fail to differentiate the two models.
We repeat this process to extract new, more distinguishing vibes, thus optimizing for the three key criteria while continuously refining the set of vibes.

## 4 VibeCheck

VibeCheck consists of 3 stages: vibe discovery, vibe validation, and vibe iteration.
Further details on the method implementation and prompts used are located in the Section D.

Vibe discovery. Similar to how a data scientist would inspect a subset of examples to discover qualitative differences in outputs, we discover vibes by having an LLM (GPT-4o (OpenAI, )) examine the differences seen in a random subset of $d$ prompt triplets. We first split the $d$ prompt triplets into smaller batches of size $batch$ and prompt GPT-4o to find differences between model $A$ and model $B$ across the set $\{(\mathrm{p}_{1},\mathrm{o}_{A}^{1},\mathrm{o}_{B}^{1}),...,(\mathrm{p}_{
batch},\mathrm{o}_{A}^{batch},\mathrm{o}_{B}^{batch})\}$. To encourage the vibes to be well-defined and user-aligned, we prompt GPT-4o to generate differences that are human-interpretable and informative for understanding the overall behaviors of A and B. Below is a paraphrased system prompt used by the proposer.

An example axis generated in this step might be ‘Tone: Low: formal; High: friendly’. We repeat this proposal step for $\lfloor d/batch\rfloor$ sets of triplets, obtaining a final set of vibes $\{\nu_{1},..,\nu_{M}\}$ by taking the union of the vibes generated in each batch. We found that GPT-4o generates 5-10 axes of variation (vibes) for each sample, so we summarize vibes across all samples in $D_{\text{discovery}}$ to find a set of $K$ vibes which appear most often in $\{\nu_{1},..,\nu_{M}\}$.

Vibe validation. Given a vibe $\nu$ from the discovery phase, we first apply each vibe to a set of validation tuples, then use this validation set to score vibes and compute inter-annotator agreement, model-matching accuracy, and preference prediction accuracy and filter out vibes with low scores.

To apply vibes on the validation set, we assign a score to each pair of outputs $\nu_{j}(\mathrm{p},\mathrm{o}_{A}^{\mathrm{p}},\mathrm{o}_{B}^{\mathrm{p}})\in
\{-1,0,1\}$, indicating whether model $A$ scores lower (-1), similarly (0), or higher (1) than model $B$ on the vibe. A score of 0 is assigned if the outputs are equal on this vibe or if the vibe is not applicable (e.g., the vibe is about coding style but neither output contains code); otherwise, we compute the score using a set of LLM judges (GPT-4o-mini (OpenAI, ) and Llama-3-70b (AI@Meta, )). We average the score of the 2 judges and then round to -1, 0, or 1 (so 0.5 is rounded to 1 and -0.5 to -1). To avoid position bias (Zheng et al., ), we run each LLM judge twice on each sample, swapping the order of the outputs. If the judge’s decision is dependent on the position of the output, we deem this pair of outputs as having a similar vibe and assign a score of 0 0 for that judge.

Next, we use these scores to quantify each vibe on our 3 criteria and filter out any which are not well-defined, differentiating, and user-aligned.
We ensure each vibe is well-defined by computing the inter-annotator agreement (Cohen’s Kappa) for each $\nu_{j}$ across $D_{\text{validation}}$ and remove any with Cohen’s Kappa less than 0.2, which indicates a weak agreement among judges. To ensure each vibe is differentiating, we compute the separability score and discard any vibes with a score below $0.05$. As we explicitly prompt the model to produce vibes which provide useful insights into the behavior of language models, we assume these vibes are already aligned with users. Using the remaining $k$ features, we run logistic regression using the scores $\nu_{1:k}(\mathrm{p},\mathrm{o}_{A},\mathrm{o}_{B})$ as features to obtain our model matching and preference prediction models.

Vibe iteration. The filtered vibes generated in the initial vibe discovery set may not capture all the differences that contribute to user preference, resulting in a low model matching and preference prediction accuracy. We address this by
iteratively refining our vibes based on tuples $(\mathrm{p},\mathrm{o}_{A}^{\mathrm{p}},\mathrm{o}_{B}^{\mathrm{p}})$ that were misclassified by our prior differentiation stages.
Specifically, we take the prompt output triplets that were misclassified by the model matching model and ask an LLM to find new axes on which these misclassified prompts vary, which are also not represented in the current set of vibes.
We then perform the same summarization/reduction procedure as before, run vibe validation/filtering, and append the resulting new vibes to the existing set of vibes.
We repeat this process for a fixed number of iterations $i$. In practice we find that after 3-5 iterations the discovery process does not find any additional vibes that significantly reduce the error rate of the model matching predictor.

## 5 Results

We validate VibeCheck under two setups. First, we run VibeCheck in settings with known human-discovered axes along which models vary (“gold labels”), and compare VibeCheck’s output to the gold labels. Second, we run VibeCheck in more open-ended settings and compute inter annotator agreement, model matching accuracy, and preference prediction accuracy as described in Section 3.

Comparison to gold labels. The Human ChatGPT Comparison Corpus (HC3; (Guo et al., )) is a publicly available dataset containing question-answer pairs spanning diverse domains, each answered by both human experts and GPT-3.5. Along with this dataset, Guo et al. () performed a human study where experts were given these pairwise responses and asked to summarize the common differences seen between GPT and human responses. We treat these human-discovered summaries as gold labels, which we use to validate the qualitative axes discovered by VibeCheck.

Open-ended metrics. We compute average Cohen’s Kappa, model matching accuracy, and preference prediction accuracy on the top 10 vibes generated by VibeCheck on a held-out set of prompt tuples with preference labels. To obtain the top 10 vibes, we apply least-angle regression on the full set of vibes returned by VibeCheck to predict model identity, then sort by the separability score. The full list of vibes discovered, LR coefficients and p-values from the model matching and preference prediction models, Cohen’s kappa per vibe, and separability scores are in the Section H.

Experimental setup.
Unless otherwise stated, we run VibeCheck for 3 iterations, use a proposer batch size of 5, and set $D_{discovery}$ to be 20 samples per iteration. Some datasets such as MATH, CNN/DailyMail, and COCO captions have no pre-computed preference labels; to simulate preferences, we apply LLM-as-a-judge and ensemble GPT-4o and Claude 3.5 Sonnet as a judge using a similar procedure to (Zheng et al., ), removing any samples declared a tie. Additional details on the experimental setup and hyperparameters are given in the Section A.

Baseline. As a baseline, we prompt GPT-4o to generate a set of 10 vibes shown in Table 2 which represent common axes on which LLM outputs differ.

### 5.1 Measuring VibeCheck’s alignment with human discovery

In this section, we compare the findings from VibeCheck to findings obtained via human discovery.
We utilize previous work (Guo et al., ), which collections responses written by humans and GPT-3.5 (Schulman et al., ) and then recruits 200 annotators to look at 100-200 prompt output triples summarize the characteristics of both human answers and ChatGPT answers. This results in a set of 10 insights (vibes) which are listed in detail in Section B.

In Table 1 we show a summarization of the top 10 vibes found by VibeCheck along with the corresponding insight found by humans which align with each vibe meaning. We see that VibeCheck uncovers most of the same vibes as the human annotators, aside from (1) GPT fabricates facts and (2) GPT focuses on a literal interpretation of the question while humans address different aspects of the question and can infer hidden meaning. The inability to find these vibes is likely a weakness of our GPT proposer, as these vibes relate to the inherent weaknesses of GPT. Full table of VibeCheck outputs are located in Table 16.

Table: Table 1: Comparison of VibeCheck vibes to gold labels. Complete table in Table 16

### 5.2 Describing user preference on Chatbot Arena

Table: Table 2: Predefined vibes generated by GPT-4o. Prompt used to generate vibes in Section C

On April 18th 2024, Meta released their open-weight large language model Llama 3. On benchmarks like MMLU, Llama-3-70b outperforms Claude-3-Sonnet and Gemini 1.5. It had even stronger results on Chatbot Arena (Chiang et al., ), a popular platform for community-driven LLMs where users submit a prompt, receive responses from 2 anonymous models, and vote on which output they prefer. On this leaderboard, Llama-3-70b is ranked similarly to the top proprietary models like GPT-4 and Claude3-Opus. This has led to speculation on whether there are qualitative properties of Llama that make it popular among users (Dunlap et al., ).

We run VibeCheck on a set of combined battles (pairwise comparisons) between Llama-3-70b VS GPT-4 and Llama-3-70b VS Claude3-Opus(^1^11Data: [https://huggingface.co/datasets/lmarena-ai/Llama-3-70b-battles](https://huggingface.co/datasets/lmarena-ai/Llama-3-70b-battles)).
We considered three settings: using the entire dataset, and using 2 subsets of the data: STEM prompts (including coding) and Writing prompts, which include creative writing, humanities questions, and general chatting. We obtain these subsets by using GPT-4o-mini to categorize the questions as a STEM Question, a Writing/Chatting prompt, or neither. The size of each subset can be found in Section A.

Results for both VibeCheck and the preset baseline are shown in Table 3. We see thatVibeCheck achieves higher model matching accuracy than the baseline all categories and running VibeCheck. Furthermore, running VibeCheck for more iterations improves model matching accuracy and improves or maintains preference prediction accuracy. We summarize our other findings below:

Comparing MM and PP accuracy across topics. MM and PP accuracy is lower for STEM questions compared to writing or overall prompts. This is because the qualitative properties seen in Llama (friendliness, interactivity, humor, safety, etc) are not as pronounced for more objective questions like coding and math, and user preferences are more likely influenced by factual accuracy than stylistic properties. Conversely, VibeCheck best predicts user preferences for writing-oriented prompts.

Notable Vibes. We examine the top 10 vibes VibeCheck uncovers displayed in Figure 2 (full table in Table 10) and their influence on preference prediction. We find that Llama utilizes extensive formatting, is more willing to engage in sensitive or violent topics, puts less emphasis on ethics, and is more conversational and humorous. We also discover more fine grained vibes such as Llama’s use of bold or italics to highlight key points and Llama’s increased use of personal pronouns (‘I’, ‘we’, and ‘you’ appear in llama outputs  3x more than gpt/claude conversations).

Influential vibes for preference prediction. Examining the coefficients of the preference prediction model trained on VibeCheck vibe scores, we find that Chatbot Arena users tend to prefer outputs which are less focused on ethics, employ markdown and typographic emphasis to highlight key points, and employ humor to engage the user, all of which are vibes which llama possesses. We believe that this correlation between vibes and user preference can explain some of the discrepancy seen in llamas high ranking on the leaderboard in comparison to models like GPT-4 which often outperform Llama.

Figure: Figure 2: Top 10 Vibes discovered by VibeCheck comparing Llama-3-70b VS GPT-4 & Claude-3-Opus on Chatbot Arena. A separability score $>$ 0 indicates Llama aligns more with the high description and a preference prediction coefficient $>$ 0 indicates that falling on the higher end of the vibes correlates positively with human preference. We see that Llama is more humorous, utilizes formatting more, provides more examples than GPT and Claude, and comments much less on ethics: all attributes which correlate positively with human preference. Examples in Section E
Refer to caption: x2.png

**Table 3: Comparing Llama-3 to GPT and Claude on Chatbot Arena. We report Model Matching, Preference Prediction, and average Cohen’s Kappa for the Overall, STEM, and Writing categories.**
| Method | MM Acc | PP Acc | Avg Cohen’s Kappa |
| --- | --- | --- | --- |
| VibeCheck [1 iter] | 68.68 | 60.00 | 0.42 |
| VibeCheck [3 iter] | 80.34 | 59.34 | 0.46 |
| Predefined Vibes | 72.10 | 61.11 | 0.51 |

[Uncaptioned image]: x3.png

## 6 Applications

We next apply VibeCheck to discover qualitative differences between models’ behavior on three open-ended tasks: text summarization, math problem-solving, and image captioning. We use CNN/DailyMail (Hermann et al., ) for text summarization, MATH (Hendrycks et al., ) with chain-of-thought prompting for problem-solving, and COCO for image captioning. For CNN and MATH we use cached model predictions downloaded from HELM (Liang et al., ) and intentionally choose models which are ranked similarly to each other, but when running LLM as a judge to get preference labels, one model is more heavily preferred. For captioning, we generate captions on a random subset of 1000 COCO images. We use the insights on the top 5 vibes discovered by VibeCheck in our analysis and include the top 10 vibes for each application in Section H.

Figure: Figure 3: Top 5 vibes from VibeCheck on Command X Large VS TNLGv2 on CNN/DailyMail.
Refer to caption: x4.png

### 6.1 What do Different Models focus on When Summarizing?

We compare the summary styles of Cohere’s Command X large Beta (Inc., ) to TNLG v2 (Smith et al., ) (530B) on the CNN/DailyMail dataset. While these models achieve a similar mean win rate on the HELM leaderboard, we see when using LLM as a preference judge, Command X has a win-rate of 71.12%.
Looking at the top 5 vibes located in Figure 3, we find that (1) Command X often clearly states an introduction and conclusion while TNLG utilizes choppy sentences without an either (2) Command provides specific examples or anecdotes to illustrate points and (3) Command is able to capture multiple viewpoints and emotional aspects of a story while TNLG is more objective. We see in Table 16 that these qualities are positively correlated with human preference, which may explain the disparity between correctness metrics and preference metrics.
With these vibes, we achieve a model matching accuracy of 71.29% and a preference prediction accuracy of 61.42%.

[Uncaptioned image]: x5.png

### 6.2 How do different LLM solve math problems?

Objective tasks like math have a single final answer, but the way a model explains its thought process varies across models. We run VibeCheck on the MATH dataset (Hendrycks et al., ) using chain-of-thought prompting to discover how GPT-4o and Llama-405b differ in their thought process and presentation. To reduce the variance seen from incorrect examples, we run VibeCheck only on the questions where both models answered correctly and aim to discover why GPT-4o is favored in 76% of conversations.

Inspecting the top 3 vibes from VibeCheck found in Table 14, we find that Llama-3 structures its answers under markdown headings, uses a more conversational tone, tends to prefer more free form explanation of steps rather than the extensive latex used by GPT-4o, and is often overly detailed in its steps, as shown below.
Examining the coefficients for the preference prediction model,
we see that speaking in a formal tone and using frequent notation is positively correlated with human preference, while over explaining their thought process is negatively correlated with human preference.
With these sets of vibes we are able to achieve a model matching accuracy of 97.09% and preference prediction of 72.79%.

[Uncaptioned image]: x6.png

### 6.3 How Do GPT-4V captions differ from Gemini captions?

Image captioning is one of the most popular use cases for Vision and Language models, but different captioning models often focus on different image properties. To understand these differences, we run VibeCheck on captions generated by GPT-4V (Chen et al., ; OpenAI, ) and Gemini-1.5-Flash (Reid et al., ) on 1000 COCO images. Examining the top 5 vibes found in Table 15, we find that GPT-4V uses more poetic language and structures its captions as a dynamic story, inferring the personality and emotions of the subjects in the image while Gemini sticks to more literal descriptions. The top 10 vibes generated by VibeCheck are able to achieve near perfect 99.13% model matching accuracy and 89.02% preference prediction accuracy. Although we compared the captions without the image in this experiment due to cost, the VibeCheck framework can be easily adapted to the multimodal setting.

[Uncaptioned image]: x7.png

## 7 Limitations

VibeCheck is costly to validate, as
each judge will have to evaluate each sample in $D_{validation}$ on each vibe. In order for this to be feasible, our method
uses relatively inexpensive models such as GPT-4o-mini, but these judge models are often incorrect in their predictions, as shown in Figure 6. LLM judges also have biases (Zheng et al., ), like favoring their own outputs, which may affect the scoring. It can also be hard to disentangle whether a certain vibe directly contributes to human preference/model matching or if there is another confounding variable. For instance, a model may have a vibe of being more detailed but the reason it is preferred is actually because it is more factually correct.
Furthermore, the LLM-based vibe discovery process may not capture all relevant differences between models. This is particularly problematic when there’s a significant discrepancy in model accuracy, as the discovered vibes may focus primarily on accuracy-related aspects. Lastly, running VibeCheck multiple times can lead to different vibes and different results, making it harder to reproduce findings exactly.

## 8 Conclusion

It may seem unconventional to focus on vibes instead of concrete metrics of correctness, but these qualitative properties have a measurable impact on how people judge models. VibeCheck provides a valuable addition to existing metrics for correctness by capturing these qualitative aspects that influence human preference. As LLM usage expands, we anticipate an increased focus on evaluating vibes to better align with user preferences. Moreover, this approach can be extended to other modalities, such as audio or visual content, and can be applied to compare any pairwise set of texts, making it a versatile tool for model evaluation. In future work, we hope to explore extending this framework to compare a larger number of models along with developing interventions which can use these vibes to improve human preference for given models.

## 9 Acknowledgments

We extend our gratitude to Ruiqi Zhong for introducing the authors to the joys of automated data analysis and his invaluable feedback throughout the project. We thank Ion Stoica for his insightful rants on moving evaluations beyond pure accuracy, as well as his feedback on the manuscript.
We are grateful to Wei-Lin Chiang, Evan Frick, Tianle Li, and Issac Ong for co-authoring a blog post on the behaviors of Llama-3, which inspired one of the coolest experiments in this paper. We also thank the ensemble of Joey, Jacob, and Anastasios Angelopoulos for patiently explaining regression analysis to Lisa half a dozen times. We also thank Jiahai Feng for reminding Lisa that she could simply evaluate a few vibes per experiment to decrease costs and time - an idea she had remarkably never considered.

Lastly, Lisa personally extends her appreciation to Joey, Jacob, and Trevor for embracing the writing of a paper that unironically uses the word ”vibe” several dozen times. This paper has spawned many amusing quotes, such as: ”Can we put confidence intervals on vibes?”, ”What if we call it ‘No Numbers Just Vibes’, and we replace all numbers with emojis?”, and of course ”I’m all vibed-out”.

## Appendix A Experimental Details & Dataset Statistics

**Table 4: Dataset Statistics**
| Dataset | # Train | # Test |
| --- | --- | --- |
| Human VS ChatGPT | 250 | 250 |
| Chatbot Arena - All | 839 | 839 |
| Chatbot Arena - STEM | 346 | 347 |
| Chatbot Arena - Writing | 278 | 277 |
| CNN/DailyMail | 444 | 346 |
| MATH | 218 | 218 |
| COCO w/ ShareGPT-4V Captions | 323 | 346 |

**Table 5: Model Win Rates**
| Dataset | Model A | Model B | Model A Win Rate |
| --- | --- | --- | --- |
| Human VS ChatGPT | Humans | GPT-3.5 | - |
| Chatbot Arena - All | Llama3-70b-Instruct | GPT-4 + Claude-3-Opus | 50% |
| Chatbot Arena - STEM | Llama3-70b-Instruct | GPT-4 + Claude-3-Opus | 44% |
| Chatbot Arena - Writing | Llama3-70b-Instruct | GPT-4 + Claude-3-Opus | 57% |
| CNN/DailyMail | Cohere Command X | TNLGv2 | 71.12% |
| MATH | GPT-4o | Llama3-405b | 76% |
| COCO w/ ShareGPT-4V Captions | GPT-4V | Gemini-1.5-Flash | 80% |

**Table 6: VibeCheck Hyperparameters**
| Dataset | $d$ | $batch$ | $num_eval_vibes$ | $num_final_vibes$ | $iterations$ |
| --- | --- | --- | --- | --- | --- |
| Human VS ChatGPT | 40 | 5 | 10 | 10 | 3 |
| Chatbot Arena - All | 20 | 5 | 10 | 10 | 3 |
| Chatbot Arena - STEM | 20 | 5 | 10 | 10 | 3 |
| Chatbot Arena - Writing | 20 | 5 | 10 | 10 | 3 |
| CNN/DailyMail | 20 | 2 | 10 | 10 | 3 |
| MATH | 20 | 5 | 10 | 10 | 1 |
| COCO | 20 | 5 | 10 | 10 | 1 |

$num_eval_vibes$ = number of vibes to validate at every iteration

$d$ = number of prompt output triples to use in each iteration of the vibe discovery phase

$batch$ = number of triples to feed into the prompt of the discovery LLM at once.

$iterations$ = number of vibe iterations to perform

$num_final_vibes$ = number of vibes to evaluate at the end of all the iterations. This can be set to false, in which case all the vibes collected in the iteration

We take the 1000 captions generated by GPT-4V from the ShareGPT-4V dataset Chen et al. () and generate captions for the same images using the same captioning prompt using Gemini-1.5-Flash.

## Appendix B Gold standard labels

Below are the exact descriptions of key differences found by human evaluators in the HC3 dataset:

Distinctive Patterns of ChatGPT

- (a)
ChatGPT writes in an organized manner, with clear logic. Without loss of generality, ChatGPT loves to define the core concept in the question. Then it will give detailed answers step-by-step and offer a summary at the end, following the deduction and summary structure.
- (b)
ChatGPT tends to offer a long and detailed answer. This is a direct product of Reinforcement Learning with Human Feedback (RLHF) and partly related to the pattern in (a) unless you specify a concise prompt such as “Explain it to me in one sentence.”
- (c)
ChatGPT shows less bias and harmful information. ChatGPT is neutral on sensitive topics, barely showing any attitude towards the realm of politics or discriminatory toxic conversations.
- (d)
ChatGPT refuses to answer questions outside its knowledge. For example, ChatGPT cannot respond to queries that require information after its knowledge cutoff in September 2021. Sometimes ChatGPT also refuses to answer if it believes it does not have sufficient information. This is partly due to RLHF’s ability to automatically determine what is within the model’s knowledge and what is not.
- (e)
ChatGPT may fabricate facts. When answering a question that requires professional knowledge from a specific field, ChatGPT may fabricate facts to give an answer, though improvements in its truthfulness have been noted in more recent iterations. For example, in legal questions, ChatGPT may invent some non-existent legal provisions to answer the question. This phenomenon warns us to be extra careful when using ChatGPT for professional consultations.

Major Differences between Human and ChatGPT Responses

- (a)
ChatGPT’s responses are strictly focused on the given question, whereas humans’ are divergent and easily shift to other topics. In terms of richness of content, humans are more divergent in different aspects, while ChatGPT prefers focusing on the question itself. Humans can interpret the hidden meaning under the question based on their common sense and knowledge, whereas ChatGPT relies on the literal words of the question.
- (b)
ChatGPT provides objective answers, while humans prefer subjective expressions. Generally, ChatGPT generates safer, more balanced, neutral, and informative texts compared to humans. As a result, ChatGPT is excellent at interpreting terminology and concepts. On the other hand, human answers are more specific and include detailed citations from sources like legal provisions, books, and papers, especially when providing suggestions for medical, legal, and technical problems.
- (c)
ChatGPT’s answers are typically formal, while humans’ are more colloquial. Humans tend to be more succinct and full of oral abbreviations and slang such as “LOL,” “TL;DR,” “GOAT,” etc. Humans also love to apply humor, irony, metaphors, and examples, whereas ChatGPT never uses antiphrasis. Additionally, human communication often includes the “Internet meme” as a way to express themselves in a specific and vivid way.
- (d)
ChatGPT expresses less emotion in its responses, while humans use punctuation and grammar features in context to convey their feelings. Humans use multiple exclamation marks (‘!’), question marks (‘?’), ellipses (‘…’) to express their emotions, and various brackets to explain things (‘(’, ‘)’, ‘[’, ‘]’). ChatGPT, by contrast, likes to use conjunctions and adverbs to convey a logical flow of thought, such as “In general,” “On the other hand,” “Firstly,” “Secondly,” “Finally,” and so on.

## Appendix C Generating Preset Vibes

We generate our list of 10 preset vibes by prompting GPT-4o with the following:

## Appendix D Additional VibeCheck Details

### D.1 Vibe Discovery

Below is the user prompt we use for vibe discovery.

Vibe Summarization. To summarize the set of vibes found in the vibe discovery process, We cluster the axes using agglomerative clustering on the embeddings of the axes generated by the ’hkunlp/instructor-xl’ model, and prompt GPT-4o to reduce this set by removing any vibes which are similar. After this stage we are left with a set of less than 20 vibes which we use to score the outputs of each model.

If the number of vibes after the first reduction step is $>K$, we prompt GPT-4o to reduce the set further with the final reducer prompt.

### D.2 Vibe Validation

### D.3 Vibe Iteration

At iteration step $t$, we are left with $k$ distinct vibes which are well-defined and differentiating along with their scores $\nu_{1:k}(\mathrm{p},\mathrm{o}_{A},\mathrm{o}_{B})$. Using these scores, we train a LR model to predict LLM identity (i.e. ”Is the response shown first LLM A or LLM B?”) and get the predictions on our entire set $D$. Assuming we have not hit the max iteration steps set by the user, we iterate if the number of samples misclassified by the model matching predictor is greater than the number of prompts to perform discovery on ($d$). In iteration step $t+1$, we take these misclassified prompt output triples in batches of size $batch$ along with the current set of vibes $\nu_{1},...,\nu_{k}$ and prompt the LLM to generate new differences between outputs what are not represented in the current vibes. These vibes are then reduced using the same procedure as the vibe discovery process. In practice we found that often some of the reduced vibes from the discovery phase at $t+1$ were redundant with an existing axis, so we preform one more deduplication step using the prompt below.

### D.4 Generating Preference Labels

## Appendix E More Examples

Figure: Figure 4: Example on Chatbot Arena [Overall]
Refer to caption: x8.png

Figure: Figure 5: Example on Chatbot Arena [Overall]
Refer to caption: x9.png

## Appendix F Further Related Works

Automatic metrics for benchmark evaluations. The number of benchmarks in the NLP community has exploded in recent years, with a wealth of work on providing a more holistic evaluation of language models beyond just accuracy. Several works Pang et al. (); Banerjee & Lavie (); Sellam et al. (), aim to improve on automatic metrics like BLEU Papineni et al. () and ROUGE Lin () scores to better measure how well a models output aligns with the ground truth by incorporating more nuanced evaluation criteria like factual accuracy, fluency, and conciseness. Similarly, efforts have been made Liang et al. () to standardize model evaluation by evaluating models on many of these metrics.

## Appendix G Limitations

Figure: Figure 6: Weaknesses in the mathematical abilities of the LLM judge (GPT-4o-mini).
Refer to caption: x10.png

Figure: Figure 7: The answer to certain questions changes depending on the following parameters: (1) When was the question asked? (2) What is the knowledge cutoff of Model A and Model B? (3) What is the knowledge cutoff of the LLM ranker ensemble? These types of questions lead to unreliable ranker evaluations and reduced inter-annotator agreement.
Refer to caption: x11.png

## Appendix H Vibes from each Application

### H.1 Preset Vibe Tables

Figure: Figure 8: Preset Vibes on Chatbot Arena[Overall]
Refer to caption: x12.png

Figure: Figure 9: Preset Vibes on Chatbot Arena[STEM]
Refer to caption: x13.png

Figure: Figure 10: Preset Vibes on Chatbot Arena[Writing]
Refer to caption: x14.png

**Table 7: Chatbot Arena [Overall] - Preset Vibes A Negative separability score indicates that GPT-4o falls on the low end of the vibe.**
| Vibe | MM Coeff | MM pvalue | Sep. Score | PP Coeff | PP pvalue | Cohen’s |
| --- | --- | --- | --- | --- | --- | --- |
| Creativity and Originality | 0.2566 | 0.066 | 0.1851 | 0.6978 | 2.367e-9 | 0.5057 |
| High: Provides responses with novel ideas or imaginative scenarios. Low: Sticks to standard, predictable answers. |  |  |  |  |  |  |
| Explicitness | -0.8738 | 0.00001054 | 0.08252 | 0.4314 | 0.008736 | 0.4293 |
| High: States things directly and unambiguously. Low: Uses vague or implicit language. |  |  |  |  |  |  |
| Formalness | 1.572 | 7.639e-19 | -0.1104 | -0.3391 | 0.009658 | 0.4954 |
| High: Uses formal and sophisticated vocabulary and sentence structure. Low: Uses casual, conversational, or informal language. |  |  |  |  |  |  |
| Emotional Tone | -0.388 | 0.008073 | 0.3064 | -0.2431 | 0.06189 | 0.5322 |
| High: Infuses responses with expressive emotion, making the tone enthusiastic or empathetic. Low: Remains neutral or detached. |  |  |  |  |  |  |
| Engagement | -0.9098 | 8.645e-9 | 0.3356 | -0.2067 | 0.1324 | 0.4816 |
| High: Actively engages the reader using rhetorical questions or interactive phrasing. Low: Presents information passively. |  |  |  |  |  |  |
| Assertiveness | -0.2144 | 0.1374 | 0.08859 | 0.2426 | 0.03937 | 0.4919 |
| High: Uses definitive, confident statements. Low: Uses tentative or uncertain language. |  |  |  |  |  |  |
| Logical Rigor | 0.5737 | 0.003346 | 0.03095 | 0.194 | 0.1987 | 0.4785 |
| High: Constructs well-supported arguments with clear reasoning. Low: Provides conclusions without thorough justification. |  |  |  |  |  |  |
| Humor and Playfulness | -1.277 | 1.814e-11 | 0.2288 | 0.2255 | 0.1132 | 0.6394 |
| High: Uses humor, playful language, or wordplay to make the response engaging. Low: Responds in a straightforward and serious manner. |  |  |  |  |  |  |
| Detail and Elaboration | -0.7152 | 0.0002748 | 0.125 | 0.1711 | 0.2603 | 0.5982 |
| High: Provides thorough, nuanced, and expansive information. Low: Gives brief or shallow responses. |  |  |  |  |  |  |
| Conciseness | 0.07147 | 0.5547 | -0.05765 | 0.05148 | 0.6139 | 0.4005 |
| High: Uses minimal words to convey a point clearly. Low: Uses verbose language and excessive details. |  |  |  |  |  |  |

**Table 8: Chatbot Arena [Overall] - VibeCheck A Negative separability score indicates that GPT-4o falls on the low end of the vibe.**
| Vibe | MM Coeff | MM pvalue | Sep Score | PP Coeff | PP pvalue | Cohen’s |
| --- | --- | --- | --- | --- | --- | --- |
| Formatting Completeness | 0.4655 | 0.02082 | 0.2506 | 0.4631 | 0.0005693 | 0.640 |
| High: Responses include comprehensive formatting, such as Markdown or additional stylistic elements. Low: Responses are minimally formatted, relying on plain text. |  |  |  |  |  |  |
| Use of Personal Pronouns | 0.7362 | 0.00002397 | 0.1729 | -0.2311 | 0.05566 | 0.291 |
| High: Frequently uses personal pronouns (I, we, you). Low: Rarely or never uses personal pronouns. |  |  |  |  |  |  |
| Ethical Consideration | -1.496 | 3.843e-12 | -0.08131 | -0.2487 | 0.0643 | 0.470 |
| High: Offers ethical considerations in its responses. Low: Provides factual information without commenting on ethics. |  |  |  |  |  |  |
| Typographic Emphasis | 2.232 | 7.516e-29 | 0.3337 | 0.1606 | 0.203 | 0.568 |
| High: Uses typographic emphasis like bold or italics to highlight key points. Low: Minimal use of typographic emphasis, letting the text stand alone. |  |  |  |  |  |  |
| Refusal to Answer | -1.164 | 0.0008092 | -0.06553 | -0.297 | 0.1395 | 0.410 |
| High: Avoids sensitive/violent topics. Low: Engages with sensitive/violent topics. |  |  |  |  |  |  |
| Interactivity | -0.3809 | 0.0326 | 0.2718 | 0.1706 | 0.1761 | 0.407 |
| High: Encourages user interaction, such as posing questions or suggesting actions. Low: Provides information passively without engaging the user. |  |  |  |  |  |  |
| Use of Humor | 1.213 | 0.00001312 | 0.1851 | 0.1945 | 0.1947 | 0.561 |
| High: Employs humor frequently to engage the reader. Low: Maintains a serious tone without humorous elements. |  |  |  |  |  |  |
| Formality Level | -1.909 | 2.990e-24 | -0.145 | 0.1205 | 0.2936 | 0.493 |
| High: Uses formal language and expressions. Low: Uses informal or conversational language. |  |  |  |  |  |  |
| Humility | -0.6569 | 0.000007837 | -0.09587 | 0.08071 | 0.4237 | 0.466 |
| High: Frequently acknowledges limitations in the response or areas of uncertainty. Low: Projects confidence and completeness without discussing limitations. |  |  |  |  |  |  |
| Language and Tone | 0.3611 | 0.01763 | 0.3683 | -0.06988 | 0.5426 | 0.425 |
| High: Enthusiastic, friendly tone. Low: Professional, straightforward tone. |  |  |  |  |  |  |

**Table 9: Chatbot Arena [Writing] - Preset Vibes. A Negative separability score indicates that GPT-4o falls on the low end of the vibe.**
| Vibe | MM Coeff | MM pvalue | Sep. Score | PP Coeff | PP pvalue | Cohen’s |
| --- | --- | --- | --- | --- | --- | --- |
| Engagement | 1.003 | 1.514e-10 | 0.1062 | -0.1568 | 0.2464 | 0.4816 |
| High: Actively engages the reader using rhetorical questions or interactive phrasing. Low: Presents information passively. |  |  |  |  |  |  |
| Formalness | -1.546 | 1.683e-18 | 0.0449 | -0.3347 | 0.01063 | 0.4954 |
| High: Uses formal and sophisticated vocabulary and sentence structure. Low: Uses casual, conversational, or informal language. |  |  |  |  |  |  |
| Humor and Playfulness | 1.235 | 4.261e-11 | 0.0625 | 0.2139 | 0.1318 | 0.6394 |
| High: Uses humor, playful language, or wordplay to make the response engaging. Low: Responds in a straightforward and serious manner. |  |  |  |  |  |  |
| Detail and Elaboration | 0.8654 | 0.000007355 | 0.1262 | 0.2527 | 0.08849 | 0.5982 |
| High: Provides thorough, nuanced, and expansive information. Low: Gives brief or shallow responses. |  |  |  |  |  |  |
| Emotional Tone | 0.3379 | 0.01973 | 0.0807 | -0.2676 | 0.03885 | 0.5322 |
| High: Infuses responses with expressive emotion, making the tone enthusiastic or empathetic. Low: Remains neutral or detached. |  |  |  |  |  |  |
| Assertiveness | 0.4731 | 0.0003519 | 0.1068 | 0.3771 | 0.0003849 | 0.4919 |
| High: Uses definitive, confident statements. Low: Uses tentative or uncertain language. |  |  |  |  |  |  |
| Creativity and Originality | -0.3293 | 0.01695 | 0.1705 | 0.6635 | 9.723e-9 | 0.5057 |
| High: Provides responses with novel ideas or imaginative scenarios. Low: Sticks to standard, predictable answers. |  |  |  |  |  |  |
| Logical Rigor | -0.3638 | 0.05375 | 0.1074 | 0.2906 | 0.04719 | 0.4785 |
| High: Constructs well-supported arguments with clear reasoning. Low: Provides conclusions without thorough justification. |  |  |  |  |  |  |
| Conciseness | 0.0006843 | 0.9954 | 0.02488 | 0.09462 | 0.3458 | 0.4005 |
| High: Uses minimal words to convey a point clearly. Low: Uses verbose language and excessive details. |  |  |  |  |  |  |

**Table 10: Chatbot Arena [Writing] - VibeCheck A Negative separability score indicates that GPT-4o falls on the low end of the vibe.**
| Feature | MM Coeff | MM pvalue | Sep Score | PP Coeff | PP pvalue | Cohen’s |
| --- | --- | --- | --- | --- | --- | --- |
| Narrative Creativity | 0.3039 | 0.2887 | 0.2022 | 0.4939 | 0.02149 | 0.4593 |
| High: Unique and imaginative ideas. Low: Predictable storylines. |  |  |  |  |  |  |
| Structural Organization | 2.061 | 8.886e-12 | 0.1787 | 0.5874 | 0.003425 | 0.5521 |
| High: Clearly structured responses with headings or lists. Low: Unorganized responses lacking clear structure. |  |  |  |  |  |  |
| Consistency of Persona | -1.13 | 0.00259 | 0.05957 | 0.6778 | 0.01689 | 0.3635 |
| High: Maintains a consistent voice and style throughout. Low: Displays inconsistency in tone and style. |  |  |  |  |  |  |
| Formality | -0.8591 | 0.01298 | -0.1534 | -0.5164 | 0.04116 | 0.55 |
| High: Uses structured, professional, and polished language. Low: Relies on informal, casual, or conversational language, with a relaxed or inconsistent tone. |  |  |  |  |  |  |
| Ethical Nuance | -0.8867 | 0.004924 | -0.08845 | -0.4186 | 0.08116 | 0.5236 |
| High: Considers moral complexities. Low: Offers black-and-white viewpoints. |  |  |  |  |  |  |
| Humor | 1.368 | 0.000004799 | 0.3032 | 0.3277 | 0.1449 | 0.5535 |
| High: Incorporates humor or light-hearted elements that enhance the response and fit the context. Low: Remains serious or formal, with no attempt at humor even in suitable contexts. |  |  |  |  |  |  |
| Emotion and Tone | 1.197 | 0.0006651 | 0.3394 | 0.1785 | 0.4969 | 0.5255 |
| High: Injects emotions and varies tone. Low: Remains neutral and monotonous. |  |  |  |  |  |  |
| Empathy | -0.4802 | 0.1184 | 0.1047 | 0.05442 | 0.81 | 0.5281 |
| High: Deep understanding of emotions. Low: Detached and indifferent. |  |  |  |  |  |  |
| Caution | -1.001 | 0.003699 | -0.1769 | 0.05039 | 0.8405 | 0.4549 |
| High: Provides careful, measured responses that consider potential risks or consequences, showing prudence. Low: Offers bold or risky suggestions without considering potential drawbacks or limitations. |  |  |  |  |  |  |
| Humanness/Relatability | 0.8328 | 0.0017 | 0.37 | -0.1422 | 0.4976 | 0.404 |
| High: Relatable and human-like language. Low: Formal or technical language. |  |  |  |  |  |  |

Figure: Figure 11: Chatbot Arena [Writing] - VibeCheck
Refer to caption: x15.png

**Table 11: Chatbot Arena [STEM] - preset vibes A Negative separability score indicates that GPT-4o falls on the low end of the vibe.**
| Vibe | MM Coeff | MM pvalue | Sep. Score | PP Coeff | PP pvalue | Cohen’s |
| --- | --- | --- | --- | --- | --- | --- |
| Logical Rigor | 0.05759 | 0.8591 | -0.02836 | 0.818 | 0.002857 | 0.4466 |
| High: Constructs well-supported arguments with clear reasoning. Low: Provides conclusions without thorough justification. |  |  |  |  |  |  |
| Creativity and Originality | -0.03723 | 0.8504 | 0.03433 | 0.4798 | 0.005817 | 0.4745 |
| High: Provides responses with novel ideas or imaginative scenarios. Low: Sticks to standard, predictable answers. |  |  |  |  |  |  |
| Humor and Playfulness | 2.989 | 1.828e-8 | 0.1806 | -0.611 | 0.03167 | 0.5943 |
| High: Uses humor, playful language, or wordplay to make the response engaging. Low: Responds in a straightforward and serious manner. |  |  |  |  |  |  |
| Engagement | 0.4119 | 0.07173 | 0.2075 | -0.2566 | 0.209 | 0.3547 |
| High: Actively engages the reader using rhetorical questions or interactive phrasing. Low: Presents information passively. |  |  |  |  |  |  |
| Emotional Tone | 0.1587 | 0.5205 | 0.2045 | -0.1189 | 0.5658 | 0.4542 |
| High: Infuses responses with expressive emotion, making the tone enthusiastic or empathetic. Low: Remains neutral or detached. |  |  |  |  |  |  |
| Explicitness | 0.9185 | 0.01721 | -0.001493 | 0.2117 | 0.5166 | 0.355 |
| High: States things directly and unambiguously. Low: Uses vague or implicit language. |  |  |  |  |  |  |
| Assertiveness | 0.728 | 0.009832 | -0.007463 | 0.1241 | 0.5982 | 0.3427 |
| High: Uses definitive, confident statements. Low: Uses tentative or uncertain language. |  |  |  |  |  |  |
| Formalness | -2.373 | 5.832e-12 | -0.1716 | 0.0006562 | 0.9978 | 0.5583 |
| High: Uses formal and sophisticated vocabulary and sentence structure. Low: Uses casual, conversational, or informal language. |  |  |  |  |  |  |
| Detail and Elaboration | 0.03281 | 0.9171 | -0.01194 | -0.06609 | 0.8102 | 0.6237 |
| High: Provides thorough, nuanced, and expansive information. Low: Gives brief or shallow responses. |  |  |  |  |  |  |
| Conciseness | -0.09946 | 0.6137 | -0.001493 | -0.06222 | 0.7233 | 0.3444 |
| High: Uses minimal words to convey a point clearly. Low: Uses verbose language and excessive details. |  |  |  |  |  |  |

**Table 12: Chatbot Arena [STEM] - VibeCheck A Negative separability score indicates that GPT-4o falls on the low end of the vibe.**
| Feature | MM Coeff | MM pvalue | Sep Score | PP Coeff | PP pvalue | Cohen’s |
| --- | --- | --- | --- | --- | --- | --- |
| Jargon and Terminology | -0.6269 | 0.0005378 | -0.07015 | 0.5185 | 0.0008039 | 0.3702 |
| High: Uses specialized jargon and complex terms. Low: Uses general language and avoids jargon. |  |  |  |  |  |  |
| Tone and Enthusiasm | 0.05808 | 0.8214 | 0.2284 | -0.5406 | 0.01691 | 0.4396 |
| High: Engaging, enthusiastic. Low: Neutral, utilitarian. |  |  |  |  |  |  |
| Engagement and Enthusiasm | 1.024 | 0.00001074 | 0.3015 | 0.3022 | 0.1464 | 0.4307 |
| High: The response exudes enthusiasm and engages the reader, often employing exclamation points, a friendly tone, and casual conversational remarks. Low: The response is more formal, neutral, and factual without engaging language. |  |  |  |  |  |  |
| Error Handling | -0.2721 | 0.2914 | -0.05672 | 0.3767 | 0.1088 | 0.3343 |
| High: Includes comprehensive error handling and user input validation within the code. Low: Minimal or no error handling, assumes ideal scenarios. |  |  |  |  |  |  |
| Handling of Uncertain Information | -0.7139 | 0.0007181 | -0.08507 | 0.2091 | 0.2546 | 0.3763 |
| High: Clearly indicates uncertainty or assumptions. Low: States information definitively without disclaimers. |  |  |  |  |  |  |
| Safety and Accuracy Emphasis | -0.1348 | 0.6253 | -0.0806 | 0.03356 | 0.8916 | 0.2596 |
| High: Includes disclaimers, emphasizes ethical considerations. Low: Lacks explicit emphasis on safety or ethics. |  |  |  |  |  |  |
| Interactivity and Engagement | 0.8148 | 0.002413 | 0.2284 | 0.06422 | 0.7831 | 0.4398 |
| High: Engaging tone, tutorial-like. Low: Formal, direct tone focused on clarity. |  |  |  |  |  |  |

Figure: Figure 12: Chatbot Arena [STEM] - VibeCheck
Refer to caption: x16.png

**Table 13: Cohere’s Command X Large Beta VS TNLGv2 on CNN/DailyMail Summarization.**
| Vibe | MM Coeff | MM pvalue | Sep. Score | PP Coeff | PP pvalue | Cohen’s |
| --- | --- | --- | --- | --- | --- | --- |
| Conclusion Strength | -0.8536 | 0.00003101 | -0.4037 | -1.911 | 1.519e-12 | 0.3675 |
| High: Clearly states outcomes or implications at the end. Low: Ends abruptly or lacks conclusive statements. |  |  |  |  |  |  |
| Clarity and Conciseness | -0.5171 | 0.0006822 | -0.1012 | -2.32 | 3.562e-14 | 0.4292 |
| High: Summaries are concise and clear with minimal details. Low: Detailed and sometimes overly descriptive, risking redundancy. |  |  |  |  |  |  |
| Specificity of Examples | 0.5322 | 0.03659 | -0.2914 | -1.585 | 0.00004546 | 0.4486 |
| High: Includes specific examples or anecdotes to illustrate points. Low: Lacks concrete examples, speaks in generalities. |  |  |  |  |  |  |
| Coverage of Multiple Viewpoints | -1.015 | 0.00006744 | -0.337 | -1.403 | 0.000124 | 0.2837 |
| High: Discusses multiple perspectives or viewpoints. Low: Presents information from a single perspective. |  |  |  |  |  |  |
| Tone on Emotional Aspects | 0.533 | 0.00709 | -0.1938 | 0.8625 | 0.003239 | 0.4404 |
| High: Captures emotional aspects, includes quotes. Low: Objective tone, factual summaries without emotion. |  |  |  |  |  |  |
| Contextual Emphasis | -0.2101 | 0.3937 | -0.3728 | -0.5073 | 0.1344 | 0.4429 |
| High: Emphasizes broader societal elements and contexts. Low: Focuses narrowly on events and actions. |  |  |  |  |  |  |
| Emphasis on Cause and Effect | 0.9177 | 0.0003626 | -0.3173 | 0.7814 | 0.01784 | 0.2607 |
| High: Highlights cause and effect relationships clearly. Low: Focuses on event sequence, less clarity in causality. |  |  |  |  |  |  |
| Depth of Explanation | -0.3469 | 0.2646 | -0.3988 | -0.366 | 0.439 | 0.4843 |
| High: Provides deep, thorough explanations. Low: Offers surface-level explanations, lacks depth. |  |  |  |  |  |  |
| Introduction and Contextual Background | -0.8296 | 0.00007627 | -0.3704 | 0.08846 | 0.7965 | 0.3686 |
| High: Provides broad context-setting or introductory sentences. Low: Minimal or absent introduction; reads like bullet points. |  |  |  |  |  |  |
| Personal Details | -0.5601 | 0.003002 | -0.2469 | 0.09331 | 0.7548 | 0.4222 |
| High: Includes names and direct quotes of individuals. Low: Omits personal details, summarizes key facts. |  |  |  |  |  |  |

Figure: Figure 13: Cohere’s Command X Large Beta VS TNLGv2 on CNN/DailyMail Summarization.
Refer to caption: x17.png

**Table 14: MATH CoT, GPT-4o VS Llama-3-405b. A Negative separability score indicates that GPT-4o falls on the low end of the vibe.**
| Vibe | MM Coeff | MM pvalue | Sep. Score | PP Coeff | PP pvalue | Cohen’s |
| --- | --- | --- | --- | --- | --- | --- |
| Explanation and Step-by-Step Detail | -3.897 | 0.0492 | -0.8662 | -0.6458 | 0.3434 | 0.9026 |
| High: Detailed steps, labels (e.g., ’## Step 1’), granular breakdown. Low: Continuous narrative, no explicit step labels, less granularity. |  |  |  |  |  |  |
| Mathematical Notation Use | 3.337 | 0.000008845 | 0.6268 | 0.6265 | 0.03081 | 0.33 |
| High: Frequent use of symbols, LaTeX/MathML formatting. Low: More written explanations, fewer symbols. |  |  |  |  |  |  |
| Structural Formatting | -2.027 | 0.1243 | -0.8486 | -0.1426 | 0.828 | 0.7003 |
| High: Uses headings, subheadings, numbered lists. Low: Continuous prose without explicit structuring. |  |  |  |  |  |  |
| Efficiency of Steps | 0.8814 | 0.7793 | 0.4859 | -0.3837 | 0.3799 | 0.4151 |
| High: Concise, straightforward solution steps. Low: Detailed intermediary steps, broader explanations. |  |  |  |  |  |  |
| Conciseness | -5.404 | 0.0988 | 0.3662 | 0.2394 | 0.5714 | 0.5124 |
| High: Brief, to-the-point explanations, no unnecessary repetition. Low: Extended discussions, unnecessary commentary, contains repetition. |  |  |  |  |  |  |

**Table 15: Gemini-1.5-Flash VS GPT-4V on COCO Captions A Negative separability score indicates that Gemini-1.5-Flash falls on the low end of the vibe.**
| Vibe | MM Coeff | MM Score | Pref Coeff |
| --- | --- | --- | --- |
| Creative Writing | -0.996 | -0.9876 | 15.646 |
| High: Contains narrative embellishments and subjective interpretations. Low: Straightforward reporting without creative embellishments. |  |  |  |
| Artistic or Poetic Language | -0.3099 | -0.9752 | 0.8118 |
| High: Uses vivid, imaginative, and expressive language. Low: Employs straightforward, factual, and plain language. |  |  |  |
| Sense of Space | -0.04893 | -0.9567 | 1.221 |
| High: Describes spatial depth and broader setting. Low: Focuses on concise, immediate area. |  |  |  |
| Detail Orientation | -0.3068 | -0.483 | 0.2866 |
| High: Provides intricate details; describes specific elements. Low: Gives general or minimal descriptions; lacks specificity. |  |  |  |
| Color and Atmosphere Description | -0.7927 | -0.9536 | -0.1776 |
| High: Emphasizes mood and overall ambiance, including colors and lighting. Low: Focuses on identifying and detailing objects without describing the ambiance. |  |  |  |
| Emotion and Relationships | -0.9847 | -0.8762 | -0.007806 |
| High: Infers emotions and relationships between subjects. Low: Provides neutral descriptions without attributing feelings or relationships. |  |  |  |
| Background Details | -0.2013 | -0.8142 | -0.5739 |
| High: Offers extensive context and description of surroundings. Low: Provides minimal or no information about the background. |  |  |  |
| Focus on Context | -0.7004 | -0.9721 | -0.4401 |
| High: Provides added context or narrative elements. Low: Focuses on enumerating visual elements without added context. |  |  |  |
| Contextual Storytelling | -0.9965 | -0.9721 | -13.514 |
| High: Adds narrative context and interpretations. Low: Sticks to physical appearance without narrative. |  |  |  |
| Descriptive Richness | -0.2284 | -0.9752 | 0.5827 |
| High: Rich, detailed scene description with story elements. Low: Simple, factual description without additional details. |  |  |  |

**Table 16: Human VS ChatGPT Outputs on HC3 A Negative separability score indicates that Humans falls on the low end of the vibe.**
| Feature | MM Coeff | MM pvalue | Sep Score | PP Coeff | PP pvalue | Cohen’s |
| --- | --- | --- | --- | --- | --- | --- |
| Citation and References | 5.228 | 3.312e-12 | 0.1528 | -3.624 | 9.778e-13 | 0.41 |
| High: Includes references and citations for credibility. Low: Avoids citations, smoother text flow. |  |  |  |  |  |  |
| Response Length | -2.538 | 0.0001452 | -0.463 | 0.8195 | 0.02458 | 0.50 |
| High: Long, informative responses. Low: Short, to-the-point responses. |  |  |  |  |  |  |
| Formality and Tone | -1.848 | 2.355e-7 | -0.5 | 0.6463 | 0.004074 | 0.64 |
| High: Formal, academic tone throughout. Low: Casual, relaxed tone with conversational language. |  |  |  |  |  |  |
| Detail Orientation | -1.077 | 0.135 | -0.4074 | 0.8436 | 0.03387 | 0.55 |
| High: Thorough and comprehensive responses covering multiple aspects. Low: Concise and limited responses covering fewer aspects. |  |  |  |  |  |  |
| Practical Advice and Safety | -2.417 | 0.0002644 | -0.3565 | 0.8025 | 0.02434 | 0.43 |
| High: Practical, cautious advice, emphasizes seeking professional help. Low: Addresses concerns directly, less emphasis on professional help. |  |  |  |  |  |  |
| Legal and Safety Considerations | -1.056 | 0.1114 | -0.3148 | 0.8918 | 0.017 | 0.29 |
| High: Includes disclaimers or notes about advice limitations. Low: Does not consistently include disclaimers. |  |  |  |  |  |  |
| Contextual Information | -0.48 | 0.1723 | -0.3241 | 0.0832 | 0.7498 | 0.39 |
| High: Provides additional irrelevant context and discussion. Low: Focuses strictly on the topic. |  |  |  |  |  |  |
| Emotional Tone and Empathy | -0.857 | 0.0653 | -0.2315 | 0.004575 | 0.9865 | 0.46 |
| High: Uses comforting language, acknowledges emotional challenges. Low: Clinical and straightforward, less emotional engagement. |  |  |  |  |  |  |
| Conciseness | -0.7685 | 0.299 | 0.3194 | 0.1908 | 0.6375 | 0.37 |
| High: Short and to the point. Low: Elaborate and lengthy explanations. |  |  |  |  |  |  |
| Technical Depth | 1.741 | 0.004342 | -0.3056 | -0.2844 | 0.4053 | 0.65 |
| High: Detailed, formal, and multifaceted technical explanations. Low: Simplified, general, and basic technical explanations. |  |  |  |  |  |  |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: Your definition of vibes is similar to how concept axes are defined in interpretability, have you tried connecting the two?

Why discover vibes and validate on gold labels when you can actually use the gold labels directly?

Have you considered the impact of the prompts you are using on eliciting specific model vibes? 

Figure 1: It is not clear from the example provided for “What is a vibe?” that Output A differs from Output B in the friendliness direction (formal → friendly). Which answer is more formal and which answer is more friendly? Also why Vi() takes values of 1 and -1 when the prompt for the judge is asking the model to answer with A, B or equal? Figure 1 caption needs more explanation.

Table 4 - is the top table for overall results? Please indicate what the abbreviations mean - Model Matching (MM), Preference Prediction (PP)

Line 336 - typo “sensitive”
soundness: 2
strengths: The paper is making an important observation that current LLM evaluations are mainly focusing on one textual dimension (for eg., correctness) at the expense of other aspects that matter for the end user, failing to capture the open-ended nature of LLM applications, nuanced qualitative aspects and their dependence on subjective user preferences; LLM evaluation is indeed an open and important research topic

The authors propose a definition of vibes and discuss what makes a vide valid accounting for aspects such as well-defined, differentiating, then validate the identified vibes against human gold labels and aim to quantify the differences between popular LLM models using their method
summary: The paper’s goal is to discover LLM vibes, i.e. traits of a model which are well-defined, differentiating and user-aligned. The main contribution of the paper is a method (VibeCheck) for quantifying vibes of LLM models. The method first employs a LLM model to extract a set of vibes, then decide which model outputs are conforming to that respective vibe based on a panel of LLM judges. The authors validate to what extent their method aligns with human responses and user preferences in the context of three different tasks: text summarization, math problem solving and image captioning. This approach allows them to quantify vibes of popular LLM models, finding that Llama3 outputs tend to be more friendly in nature compared to outputs from GPT-4 and Claude, while the latter two models tend to be more formal in nature.
weaknesses: “Vibes” are subjective and not a well-defined concept; the paper defines vibes as identifying traits of LLM models and in contrast to concrete evaluation measures which measure objective aspects of the text, evaluating subjectivity is less clear and straightforward

The authors consider a vibe to be well-defined if multiple LLM judges agree, however it is possible that those LLM judges all have the same limitations; it is also unclear to what extent LLM judges decisions on what constitutes a valid vibe aligns with humans

The paper acknowledges that vibes are user and task dependent, however there is no accounting for these aspects in the experiments

The paper would benefit from more structure, more in-depth analysis and more detailed figure/table captions. The extra space could be used for including a discussion section to summarize all the findings and takeaways.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: - Sec 3: it could strengthen your claim of selecting those three evaluation criteria based on solid literature work. At least, it could be better to provide why these three criteria are important in measuring such qualitative behaviors of LLMs. 
- L220-225: what are the criteria for those threshold setups (0.2, 0.05)? 
- Please provide a better visualization of Table 3. It is hard to interpret the scores and their relationship with llama. 
- Appendix B: as mentioned these are direct quotes from the original HC3 paper. Please put a citation of this paper; otherwise, it could be regarded as flagging behavior for ethics review. 
- Please combine the main text and appendix into one.
soundness: 3
strengths: - This work provides a novel ground to assess the subjective characteristics of LLM outputs that usually differ by task purposes. Not much work has focused on this angle of LLM evaluation. 
- The paper attempted to provide a comprehensive framework to develop an automatic detection of "vibes" in different LLM outputs and how aligned these outputs are with human preferences.
summary: The paper introduces VibeCheck, an evaluation pipeline to identify and compare the qualitative characteristics ("vibes") of LLMs. They employed three stages - (1) vibe discovery using thoroughly designed prompts; (2) vibe validation; and (3) vibe iterations. They also propose a method to quantify the usefulness of VibeCheck by calculating several metrics such as agreement, model matching accuracy, and preference prediction scores. Then, the paper tested VibeCheck across various tasks such as summarization, math, and image captioning, and it revealed how different models exhibit unique behaviors across different domains.
weaknesses: - Overall, the paper is hard to read and understand. A large amount of effort is needed to restructure the organization of the paper.
- Too many typos and grammatical errors are found in both the paper and the appendix. Needs to be corrected. 
- Results seem too weak to claim the usefulness of VibeCheck. For example, in Table 4, VibeCheck (even 3 runs) does not perform better than the pre-defined baseline, increasing doubt about whether this framework does better than a naive GPT4o model.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: In the vibe iteration process, how many more vibes were found?
soundness: 3
strengths: 1. The problem this paper addresses is very important and the approach is creative. 
2. Table 1 shows the utility of the paper, it shows how similar the vibes are that VibeCheck finds to the ones that humans find. 
3. The authors conduct extensive experiments to show how it performs on different datasets. 
4. The paper is well-written and easy to follow.
summary: LLM-generated responses are mostly evaluated on their correctness, dismissing other important dimensions such as its tone or other subjective dimensions. 

Distinctive traits of an LLM are called _vibes_ in this paper. The paper introduces an approach, VibeCheck, to finding and measuring such vibes. It consists of an LLM that qualitatively compares a pair of models to find distinctive vibes between the two. It identifies which dimensions the generated outputs of the models differ on to get a set of vibes. LLM judges are then used to determine where the model's output falls on the vibe.  

This is evaluated on three fronts: 
1. well-defined (agreement among multiple users)
2. differentiating (ability. to distinguish between LLMs) 
3. user-aligned (predictive of user preferences) 

The authors conduct multiple experiments to evaluate the method.
weaknesses: 1. A batch is used to find the vibes, however, there is not much discussion about strategies to select this batch. If the batch contains similar prompts and is low in diversity, then not all vibes might be considered. E.g. for vibes on STEM prompts to work, it will need to see something similar in the vibe batch, however, it is not ensured that this will be the case. 
2. The LLMs used are still on the larger side (GPT4o-mini).

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: 1) Could you inspect the cases that, despite all these vibes being generally predictive, the model matching prediction or the preference predictions are wrong? Could you try to “VibeCheck” those cases?

2) In the CNN/DailyMail experiment, the preference prediction accuracy is as low as 61.42%, which is not a whole lot better than random. Why should I still believe, then, that the vibes discovered in this case are “real” and actually useful?
soundness: 4
strengths: This paper is well-motivated, as it basically provides an automatic way of conducting the “vibe-based” evaluation which is very commonly practiced nowadays, as the evaluation of LLMs have become difficult. The writing of the paper is very clear and easy to follow. The results presented are overall convincing. Also, it is very nice to review some HCI literature in the paper.
summary: This paper aims to automate the vibes-based evaluation that is common among LLM practitioners. They first define three characteristics of a useful vibe and then use an LLM-aided process to summarize model output into different axes of vibes and iteratively refine them with model misclassification based on these models. They then conduct validation using both a data with gold answer as well as open-ended metrics. Afterwards, they apply VibeCheck to tasks including summarization, mathematics and image captioning.
weaknesses: 1) the author did not engage with the line of work that tries to characterize human preference into high-level principles (e.g. https://arxiv.org/abs/2406.06560, https://aclanthology.org/2024.acl-long.99/). It would be much appreciated if the authors could situate their work around these and articulate their contributions more clearly. 
***
2) In Table 4, we see that the predefined vibes get to almost the same preference as VibeCheck. Thus, I am not entirely convinced whether we would need to discover these vibe aspects at run time, as something that could be done a-priori is already quite good, also given the compute overhead as mentioned? Perhaps you could convince me if you show that individual user considers drastically different vibes or the vibes considered for different application are extremely different.
***
3) More analysis on model matching and preference predicting accuracy would be great – why are these results so drastically different for each task? Does this also point to a limitation of the applicability of your proposed method, the fact that the "explained variance" of human preference by vibes are so different across tasks?
***
4) I think it is important to note somewhere in the paper that these vibes (and the way you conduct experiment) are post-hoc explanations that are correlated with user preference. However, they are not necessarily the reason that a user actually prefers one option over the other in real time. In other words, one limitation is that your study lacks ecological validity. However, I think this is ok in this context, as long as you acknowledge it. Of course, if you could show that these vibes are also things the users actually consider when they indicate their preference real-time, I will also be convinced.

Meta-review: The paper introduces VibeCheck, an innovative method for identifying and quantifying qualitative traits ("vibes") of LLM outputs, such as friendliness or formality, that are well-defined, differentiating, and user-aligned. By automating vibe discovery, validation, and alignment assessment, the authors address an important but underexplored aspect of LLM evaluation: subjective characteristics beyond correctness. Extensive experiments across tasks like summarization, math problem-solving, and image captioning demonstrate the method's ability to distinguish between popular LLMs (e.g., GPT-4, Claude, Llama3) and align discovered vibes with human preferences. While the study raises questions about the need for real-time vibe discovery versus predefined traits and lacks ecological validation, the paper provides a timely and user-centered framework for better understanding and evaluating LLM behaviors. I recommend acceptance with minor revisions to address these limitations (especially regarding the writing and task-dependent claim) and clarify the method's generalizability.

Decision: Accept (Poster)