Advancing Annotation of Stance in Social Media Posts: A Comparative Analysis of Large Language Models and Crowd Sourcing
Authors: Mao Li, Frederick Conrad


Reviewer paper summaries: The paper aims to analyze the performance of LLMs for annotating data and investigate the conditions for agreement with human annotators. The dataset used is a curated subset of a large tweets collections, filtered to mention the 2020 US Census. The data was annotated on AMT for two tasks: the knowledge of the author that a citizenship question has been banned from inclusion in the census and the stance of the author towards including such a question in the census.
Several LLMs (both open-source such as Llama and proprietary such as ChatGPT) are benchmarked on the two tasks. Then, 3 hypothesis proposed by the authors are investigated, regarding behaviors comparisons for data containing implicit vs explicit stances.
Summary of strengths: The introduction of a new stance detection dataset, alongside LLM benchmarking for both zero-shot and few-shot setups, could be valuable for the community. The idea of studying the distinction between explicit and implicit stances is also interesting.
Summary of weaknesses: While the authors make generic statements about the behavior on "System 1" and "System 2" tasks, the analysis is only conducted on a single domain (the citizenship question in the context of the us census). I am not sure the results can be generalized to any system 1 and system 2 tasks and corpuses, as different domains could comprise different types of formulations, different ways of implying a stance, etc.
For example, the authors make a clear distinction between the explicitness of an individual text and the explicitness of an entire corpus (e.g. L211, explicit stance in the domain of abortion rights and implicit stance in the domain of race and gender). However, the explicitness of a corpus or a domain is not properly addressed in the results, as only the proposed dataset is analyzed.
Comments, suggestions and typos: L311: how were the thresholds 17 and 50 chosen?
Overall assessment scores: 3