Title: Labelling Data with Unknown References
Primary area: interpretability and explainable AI
Keywords: evaluation, LLMs-as-judges, algorithms, meta-evaluation

Abstract: An evaluator is trustworthy when there exists some agreed-upon way to measure its performance as a labeller. The two ways to establish trustworthiness are either by testing it, or by assuming the evaluator 'knows' somehow the way to label the corpus. However, if labelled references (e.g., a development set) are unavailable, neither of these approaches work: the former requires data, and the latter is an assumption, not evidence. To address this, we introduce an algorithm (the 'No-Data Algorithm') by which to establish trust in an evaluator without any existing references. Our algorithm works by successively posing challenges to said evaluator. We show that this is sufficient to establish trustworthiness w.h.p., in such a way that when the evaluator actually knows the way to label the corpus, the No-Data Algorithm accepts its output; and, conversely, flags untrustworthy evaluators when these are unable to prove it. We present formal proofs of correctness, empirical tests, and applications to LLMs-as-judges on low-resource languages.

TL;DR: We introduce an algorithm based on cryptography by which to establish trust on an evaluator when labels do not exist--for example, an LLM-as-a-judge in low-resource languages--with formal guarantees of correctness.