Can Large Language Models Capture Human Annotator Disagreements?
Authors: name: Jingwei Ni, name: Yu Fan, name: Vilém Zouhar, name: Donya Rooein, name: Alexander Hoyle, name: Mrinmaya Sachan, name: Markus Leippold, name: Dirk Hovy, name: Elliott Ash

Human annotation variation (i.e., annotation disagreements) is common in NLP
and often reflects important information such as task subjectivity and sample
ambiguity. While Large Language Models (LLMs) are increasingly used for
automatic annotation to reduce human effort, their evaluation often focuses on
predicting the majority-voted "ground truth" labels. It is still unclear,
however, whether these models also capture informative human annotation
variation. Our work addresses this gap by extensively evaluating LLMs' ability
to predict annotation disagreements without access to repeated human labels.
Our results show that LLMs struggle with modeling disagreements, which can be
overlooked by majority label-based evaluations. Notably, while RLVR-style
(Reinforcement learning with verifiable rewards) reasoning generally boosts LLM
performance, it degrades performance in disagreement prediction. Our findings
highlight the critical need for evaluating and improving LLM annotators in
disagreement modeling. Code and data at
https://github.com/EdisonNi-hku/Disagreement_Prediction.