Potential and Perils of Large Language Models as Judges of Unstructured
  Textual Data
Authors: name: Rewina Bedemariam; name: Natalie Perez; name: Sreyoshi Bhaduri; name: Satya Kapoor; name: Alex Gil; name: Elizabeth Conjar; name: Ikkei Itoku; name: David Theil; name: Aman Chadha; name: Naumaan Nayyar
Published: 2025-01-14 14:49:14

Abstract: Rapid advancements in large language models have unlocked remarkable
capabilities when it comes to processing and summarizing unstructured text
data. This has implications for the analysis of rich, open-ended datasets, such
as survey responses, where LLMs hold the promise of efficiently distilling key
themes and sentiments. However, as organizations increasingly turn to these
powerful AI systems to make sense of textual feedback, a critical question
arises, can we trust LLMs to accurately represent the perspectives contained
within these text based datasets? While LLMs excel at generating human-like
summaries, there is a risk that their outputs may inadvertently diverge from
the true substance of the original responses. Discrepancies between the
LLM-generated outputs and the actual themes present in the data could lead to
flawed decision-making, with far-reaching consequences for organizations. This
research investigates the effectiveness of LLMs as judge models to evaluate the
thematic alignment of summaries generated by other LLMs. We utilized an
Anthropic Claude model to generate thematic summaries from open-ended survey
responses, with Amazon's Titan Express, Nova Pro, and Meta's Llama serving as
LLM judges. The LLM-as-judge approach was compared to human evaluations using
Cohen's kappa, Spearman's rho, and Krippendorff's alpha, validating a scalable
alternative to traditional human centric evaluation methods. Our findings
reveal that while LLMs as judges offer a scalable solution comparable to human
raters, humans may still excel at detecting subtle, context-specific nuances.
This research contributes to the growing body of knowledge on AI assisted text
analysis. We discuss limitations and provide recommendations for future
research, emphasizing the need for careful consideration when generalizing LLM
judge models across various contexts and use cases.