I Think, Therefore I Am Under-Qualified? A Benchmark for Evaluating
  Linguistic Shibboleth Detection in LLM Hiring Evaluations
Authors: name: Julia Kharchenko; name: Tanya Roosta; name: Aman Chadha; name: Chirag Shah
Published: 2025-08-06 23:51:03

Abstract: This paper introduces a comprehensive benchmark for evaluating how Large
Language Models (LLMs) respond to linguistic shibboleths: subtle linguistic
markers that can inadvertently reveal demographic attributes such as gender,
social class, or regional background. Through carefully constructed interview
simulations using 100 validated question-response pairs, we demonstrate how
LLMs systematically penalize certain linguistic patterns, particularly hedging
language, despite equivalent content quality. Our benchmark generates
controlled linguistic variations that isolate specific phenomena while
maintaining semantic equivalence, which enables the precise measurement of
demographic bias in automated evaluation systems. We validate our approach
along multiple linguistic dimensions, showing that hedged responses receive
25.6% lower ratings on average, and demonstrate the benchmark's effectiveness
in identifying model-specific biases. This work establishes a foundational
framework for detecting and measuring linguistic discrimination in AI systems,
with broad applications to fairness in automated decision-making contexts.