Title: Unsupervised Evaluation of Multi-Turn Objective-Driven Interactions
Primary area: foundation or frontier models, including LLMs
Keywords: large language models, unsupervised learning, large language model evaluation

Abstract: Large language models (LLMs) have seen increasing popularity in enterprise applications where AI agents and humans engage in objective-driven interactions. However, these systems are difficult to evaluate: data may be complex and unlabeled; human annotation is often impractical at scale; custom metrics can monitor for specific errors, but not previously-undetected ones; and LLM judges can produce unreliable results. We introduce the first set of unsupervised metrics for objective-driven interactions, leveraging statistical properties of unlabeled interaction data and using fine-tuned LLMs to adapt to distributional shifts. We develop metrics for labeling user goals, measuring goal completion, and quantifying LLM uncertainty without grounding evaluations in human-generated ideal responses. Our approach is validated on open-domain and task-specific interaction data.

TL;DR: Unsupervised metrics for evaluating the quality of objective-driven human-LLM conversations without human annotation or LLM judges.