Open-Source Large Language Models Outperform Crowd Workers and Approach
  ChatGPT in Text-Annotation Tasks
Authors: name: Meysam Alizadeh; name: Maël Kubli; name: Zeynab Samei; name: Shirin Dehghani; name: Juan Diego Bermeo; name: Maria Korobeynikova; name: Fabrizio Gilardi
Published: 2023-07-05 10:15:07

Abstract: This study examines the performance of open-source Large Language Models
(LLMs) in text annotation tasks and compares it with proprietary models like
ChatGPT and human-based services such as MTurk. While prior research
demonstrated the high performance of ChatGPT across numerous NLP tasks,
open-source LLMs like HugginChat and FLAN are gaining attention for their
cost-effectiveness, transparency, reproducibility, and superior data
protection. We assess these models using both zero-shot and few-shot approaches
and different temperature parameters across a range of text annotation tasks.
Our findings show that while ChatGPT achieves the best performance in most
tasks, open-source LLMs not only outperform MTurk but also demonstrate
competitive potential against ChatGPT in specific tasks.