VibeCheck: Discover and Quantify Qualitative Differences in Large Language Models

Authors: Lisa Dunlap, Krishna Mandal, Trevor Darrell, Jacob Steinhardt, Joseph E. Gonzalez
Primary area: foundation or frontier models, including LLMs
Keywords: large language models, evaluation

Abstract:
Large language models (LLMs) often exhibit subtle yet distinctive characteristics in their outputs that users intuitively recognize, but struggle to quantify. These "vibes" -- such as tone, formatting, or writing style -- influence user preferences, yet traditional evaluations focus primarily on the singular vibe of correctness.
We introduce $\textbf{VibeCheck}$, a system for automatically comparing a pair of LLMs by discovering identifying traits of a model ("vibes") that are well-defined, differentiating, and user-aligned. VibeCheck iteratively discovers vibes from model outputs and then utilizes a panel of LLM judges to quantitatively measure the utility of each vibe. 
We validate that the vibes generated by VibeCheck align with those found in human discovery and run VibeCheck on pairwise preference data from real-world user conversations with Llama-3-70b vs GPT-4. VibeCheck reveals that Llama has a friendly, funny, and somewhat controversial vibe. These vibes predict model identity with 80% accuracy and human preference with 61% accuracy. Lastly, we run VibeCheck on a variety of models and tasks, including summarization, math, and captioning to provide insight into differences in model behavior. VibeCheck discovers vibes like Command X prefers to add concrete intros and conclusions when summarizing in comparison to TNGL, Llama-405b often overexplains its thought process on math problems compared to GPT-4o, and GPT-4 prefers to focus on the mood and emotions of the scene when captioning compared to Gemini-1.5-Flash.

Decision: accept (poster)
Reviewer scores: 3, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730579131983

thread_type:
official_review

<thread_object>
creator: reviewer_a334

<content>
summary:
This paper aims to automate the vibes-based evaluation that is common among LLM practitioners. They first define three characteristics of a useful vibe and then use an LLM-aided process to summarize model output into different axes of vibes and iteratively refine them with model misclassification based on these models. They then conduct validation using both a data with gold answer as well as open-ended metrics. Afterwards, they apply VibeCheck to tasks including summarization, mathematics and image captioning.

strengths:
This paper is well-motivated, as it basically provides an automatic way of conducting the “vibe-based” evaluation which is very commonly practiced nowadays, as the evaluation of LLMs have become difficult. The writing of the paper is very clear and easy to follow. The results presented are overall convincing. Also, it is very nice to review some HCI literature in the paper.

weaknesses:
1) the author did not engage with the line of work that tries to characterize human preference into high-level principles (e.g. https://arxiv.org/abs/2406.06560, https://aclanthology.org/2024.acl-long.99/). It would be much appreciated if the authors could situate their work around these and articulate their contributions more clearly. 
***
2) In Table 4, we see that the predefined vibes get to almost the same preference as VibeCheck. Thus, I am not entirely convinced whether we would need to discover these vibe aspects at run time, as something that could be done a-priori is already quite good, also given the compute overhead as mentioned? Perhaps you could convince me if you show that individual user considers drastically different vibes or the vibes considered for different application are extremely different.
***
3) More analysis on model matching and preference predicting accuracy would be great – why are these results so drastically different for each task? Does this also point to a limitation of the applicability of your proposed method, the fact that the "explained variance" of human preference by vibes are so different across tasks?
***
4) I think it is important to note somewhere in the paper that these vibes (and the way you conduct experiment) are post-hoc explanations that are correlated with user preference. However, they are not necessarily the reason that a user actually prefers one option over the other in real time. In other words, one limitation is that your study lacks ecological validity. However, I think this is ok in this context, as long as you acknowledge it. Of course, if you could show that these vibes are also things the users actually consider when they indicate their preference real-time, I will also be convinced.

questions:
1) Could you inspect the cases that, despite all these vibes being generally predictive, the model matching prediction or the preference predictions are wrong? Could you try to “VibeCheck” those cases?

2) In the CNN/DailyMail experiment, the preference prediction accuracy is as low as 61.42%, which is not a whole lot better than random. Why should I still believe, then, that the vibes discovered in this case are “real” and actually useful?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730672898783

thread_type:
official_review

<thread_object>
creator: reviewer_513d

<content>
summary:
LLM-generated responses are mostly evaluated on their correctness, dismissing other important dimensions such as its tone or other subjective dimensions. 

Distinctive traits of an LLM are called _vibes_ in this paper. The paper introduces an approach, VibeCheck, to finding and measuring such vibes. It consists of an LLM that qualitatively compares a pair of models to find distinctive vibes between the two. It identifies which dimensions the generated outputs of the models differ on to get a set of vibes. LLM judges are then used to determine where the model's output falls on the vibe.  

This is evaluated on three fronts: 
1. well-defined (agreement among multiple users)
2. differentiating (ability. to distinguish between LLMs) 
3. user-aligned (predictive of user preferences) 

The authors conduct multiple experiments to evaluate the method.

strengths:
1. The problem this paper addresses is very important and the approach is creative. 
2. Table 1 shows the utility of the paper, it shows how similar the vibes are that VibeCheck finds to the ones that humans find. 
3. The authors conduct extensive experiments to show how it performs on different datasets. 
4. The paper is well-written and easy to follow.

weaknesses:
1. A batch is used to find the vibes, however, there is not much discussion about strategies to select this batch. If the batch contains similar prompts and is low in diversity, then not all vibes might be considered. E.g. for vibes on STEM prompts to work, it will need to see something similar in the vibe batch, however, it is not ensured that this will be the case. 
2. The LLMs used are still on the larger side (GPT4o-mini).

questions:
In the vibe iteration process, how many more vibes were found?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730687418346

thread_type:
official_review

<thread_object>
creator: reviewer_hgov

<content>
summary:
The paper introduces VibeCheck, an evaluation pipeline to identify and compare the qualitative characteristics ("vibes") of LLMs. They employed three stages - (1) vibe discovery using thoroughly designed prompts; (2) vibe validation; and (3) vibe iterations. They also propose a method to quantify the usefulness of VibeCheck by calculating several metrics such as agreement, model matching accuracy, and preference prediction scores. Then, the paper tested VibeCheck across various tasks such as summarization, math, and image captioning, and it revealed how different models exhibit unique behaviors across different domains.

strengths:
- This work provides a novel ground to assess the subjective characteristics of LLM outputs that usually differ by task purposes. Not much work has focused on this angle of LLM evaluation. 
- The paper attempted to provide a comprehensive framework to develop an automatic detection of "vibes" in different LLM outputs and how aligned these outputs are with human preferences.

weaknesses:
- Overall, the paper is hard to read and understand. A large amount of effort is needed to restructure the organization of the paper.
- Too many typos and grammatical errors are found in both the paper and the appendix. Needs to be corrected. 
- Results seem too weak to claim the usefulness of VibeCheck. For example, in Table 4, VibeCheck (even 3 runs) does not perform better than the pre-defined baseline, increasing doubt about whether this framework does better than a naive GPT4o model.

questions:
- Sec 3: it could strengthen your claim of selecting those three evaluation criteria based on solid literature work. At least, it could be better to provide why these three criteria are important in measuring such qualitative behaviors of LLMs. 
- L220-225: what are the criteria for those threshold setups (0.2, 0.05)? 
- Please provide a better visualization of Table 3. It is hard to interpret the scores and their relationship with llama. 
- Appendix B: as mentioned these are direct quotes from the original HC3 paper. Please put a citation of this paper; otherwise, it could be regarded as flagging behavior for ethics review. 
- Please combine the main text and appendix into one.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731438629235

thread_type:
official_review

<thread_object>
creator: reviewer_cad4

<content>
summary:
The paper’s goal is to discover LLM vibes, i.e. traits of a model which are well-defined, differentiating and user-aligned. The main contribution of the paper is a method (VibeCheck) for quantifying vibes of LLM models. The method first employs a LLM model to extract a set of vibes, then decide which model outputs are conforming to that respective vibe based on a panel of LLM judges. The authors validate to what extent their method aligns with human responses and user preferences in the context of three different tasks: text summarization, math problem solving and image captioning. This approach allows them to quantify vibes of popular LLM models, finding that Llama3 outputs tend to be more friendly in nature compared to outputs from GPT-4 and Claude, while the latter two models tend to be more formal in nature.

strengths:
The paper is making an important observation that current LLM evaluations are mainly focusing on one textual dimension (for eg., correctness) at the expense of other aspects that matter for the end user, failing to capture the open-ended nature of LLM applications, nuanced qualitative aspects and their dependence on subjective user preferences; LLM evaluation is indeed an open and important research topic

The authors propose a definition of vibes and discuss what makes a vide valid accounting for aspects such as well-defined, differentiating, then validate the identified vibes against human gold labels and aim to quantify the differences between popular LLM models using their method

weaknesses:
“Vibes” are subjective and not a well-defined concept; the paper defines vibes as identifying traits of LLM models and in contrast to concrete evaluation measures which measure objective aspects of the text, evaluating subjectivity is less clear and straightforward

The authors consider a vibe to be well-defined if multiple LLM judges agree, however it is possible that those LLM judges all have the same limitations; it is also unclear to what extent LLM judges decisions on what constitutes a valid vibe aligns with humans

The paper acknowledges that vibes are user and task dependent, however there is no accounting for these aspects in the experiments

The paper would benefit from more structure, more in-depth analysis and more detailed figure/table captions. The extra space could be used for including a discussion section to summarize all the findings and takeaways.

questions:
Your definition of vibes is similar to how concept axes are defined in interpretability, have you tried connecting the two?

Why discover vibes and validate on gold labels when you can actually use the gold labels directly?

Have you considered the impact of the prompts you are using on eliciting specific model vibes? 

Figure 1: It is not clear from the example provided for “What is a vibe?” that Output A differs from Output B in the friendliness direction (formal → friendly). Which answer is more formal and which answer is more friendly? Also why Vi() takes values of 1 and -1 when the prompt for the judge is asking the model to answer with A, B or equal? Figure 1 caption needs more explanation.

Table 4 - is the top table for overall results? Please indicate what the abbreviations mean - Model Matching (MM), Preference Prediction (PP)

Line 336 - typo “sensitive”
</content>
</thread_object>

</thread>
<thread>
creation date:
1732651536907

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
An overdue reply to reviewers

comment:
We apologize for our delayed response and understand if the reviewers are unable to fully engage before the discussion period ends.

We sincerely appreciate the quality of our reviewers’ feedback! We agree with most of your concerns and are grateful for the references to relevant HCI and interpretability literature, as well as the requests for additional analysis on how vibes and preferences vary across different users and tasks. We have corrected our (many) typos and have updated the paper with more thorough analysis of the results. Below we address some shared concerns across multiple reviewers:

## Proof that vibes are user and task dependent:

We demonstrate in the appendix that (1) the vibes identified for STEM tasks differ from those found in Writing tasks, and (2) user preferences for the same vibe differ between STEM and Writing tasks. Based on the reviewers’ feedback, we now realize this is a crucial experiment and will include it in the main paper.

### Task Dependence 
While some vibes are shared across both question types, others are specific to the question type. Below are vibes which are unique to Writing and STEM questions. For instance, narrative creativity, consistency of persona, and humor are exclusive to Writing tasks, whereas error handling, a tutorial-like tone, and complex jargon are unique to STEM tasks. 

| **Writing Vibes**                          | **STEM Vibes**                                |
|--------------------------------------------|-----------------------------------------------|
| Humor: Remains serious or formal, with no attempt at humor even in suitable contexts. → Incorporates humor or light-hearted elements that enhance the response and fit the context. | Interactivity and Engagement: Formal, direct tone focused on clarity. → Engaging tone, tutorial-like. |
| Narrative Creativity: Predictable storylines. → Unique and imaginative ideas. | Error Handling: Minimal or no error handling, assumes ideal scenarios. → Includes comprehensive error handling and user input validation within the code. |
| Consistency of Persona: Displays inconsistency in tone and style. → Maintains a consistent voice and style throughout.  | Jargon and Terminology: Uses general language and avoids jargon. → Uses specialized jargon and complex terms. | |

### Task and User Dependence
The table below presents the preference prediction scores for each vibe across writing and STEM tasks. A positive preference prediction (PP) coefficient indicates that responses aligned with the “high” description of a vibe are positively associated with preferred responses. The results highlight that detailed explanations, humor, and expressive emotion are positively correlated with human preference in writing tasks. However, these same qualities negatively correlate with user preference in STEM questions. Conversely, logical rigor is highly valued for STEM tasks but has little impact on preferences for writing tasks. Although we lack do not have dataset comparing individual user judgments, treating the users who ask STEM questions and those who ask writing questions as distinct groups provides some evidence of user-specific preferences.


| Vibe (low -> high)                                                                                                                                   | PP Coef     Writing    |        PP Coef     STEM          |
|:-----------------------------------------------------------------------------------------------------------------------------------------------------|:---------------:|:---------------:|
| Detail and Elaboration. Gives brief or shallow responses. -> Provides thorough, nuanced, and expansive information.                                   | **0.65**            | **-0.07**           |
| Humor and Playfulness. Responds in a straightforward and serious manner. -> Uses humor, playful language, or wordplay to make the response engaging. | **0.38**            | **-0.61**           |
| Formalness. Uses casual, conversational, or informal language. -> Uses formal and sophisticated vocabulary and sentence structure.                    | **-0.54**           | **0**               |
| Conciseness. Uses verbose language and excessive details. -> Uses minimal words to convey a point clearly.                                            | **0.32**            | **-0.06**           |
| Logical Rigor. Provides conclusions without thorough justification. -> Constructs well-supported arguments with clear reasoning.                     | **0.26**            | **0.82**            |
| Emotional Tone. Remains neutral or detached. -> Infuses responses with expressive emotion, making the tone enthusiastic or empathetic.               | **0.07**            | **-0.12**           | |
</content>
</thread_object>

</thread>
<thread>
creation date:
1733169654374

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Nearing the end of the discussion period

comment:
Good afternoon!

As we are nearing the end of the discussion period, please let us know if there is anything else we can do to address your questions or concerns.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces VibeCheck, an innovative method for identifying and quantifying qualitative traits ("vibes") of LLM outputs, such as friendliness or formality, that are well-defined, differentiating, and user-aligned. By automating vibe discovery, validation, and alignment assessment, the authors address an important but underexplored aspect of LLM evaluation: subjective characteristics beyond correctness. Extensive experiments across tasks like summarization, math problem-solving, and image captioning demonstrate the method's ability to distinguish between popular LLMs (e.g., GPT-4, Claude, Llama3) and align discovered vibes with human preferences. While the study raises questions about the need for real-time vibe discovery versus predefined traits and lacks ecological validation, the paper provides a timely and user-centered framework for better understanding and evaluating LLM behaviors. I recommend acceptance with minor revisions to address these limitations (especially regarding the writing and task-dependent claim) and clarify the method's generalizability.