Prompt Candidates, then Distill: A Teacher-Student Framework for
  LLM-driven Data Annotation
Authors: name: Mingxuan Xia, name: Haobo Wang, name: Yixuan Li, name: Zewei Yu, name: Jindong Wang, name: Junbo Zhao, name: Runze Wu

Recently, Large Language Models (LLMs) have demonstrated significant
potential for data annotation, markedly reducing the labor costs associated
with downstream applications. However, existing methods mostly adopt an
aggressive strategy by prompting LLM to determine a single gold label for each
unlabeled sample. Due to the inherent uncertainty within LLMs, they often
produce incorrect labels for difficult samples, severely compromising the data
quality for downstream applications. Motivated by ambiguity aversion in human
behaviors, we propose a novel candidate annotation paradigm wherein large
language models are encouraged to output all possible labels when incurring
uncertainty. To ensure unique labels are provided for downstream tasks, we
develop a teacher-student framework CanDist that distills candidate annotations
with a Small Language Model (SLM). We further provide a rigorous justification
demonstrating that distilling candidate annotations from the teacher LLM offers
superior theoretical guarantees compared to directly using single annotations.
Extensive experiments across six text classification tasks validate the
effectiveness of our proposed method. The source code is available at
https://github.com/MingxuanXia/CanDist.