LLMs as Workers in Human-Computational Algorithms? Replicating
  Crowdsourcing Pipelines with LLMs
Authors: name: Tongshuang Wu; name: Haiyi Zhu; name: Maya Albayrak; name: Alexis Axon; name: Amanda Bertsch; name: Wenxing Deng; name: Ziqi Ding; name: Bill Guo; name: Sireesh Gururaja; name: Tzu-Sheng Kuo; name: Jenny T. Liang; name: Ryan Liu; name: Ihita Mandal; name: Jeremiah Milbauer; name: Xiaolin Ni; name: Namrata Padmanabhan; name: Subhashini Ramkumar; name: Alexis Sudjianto; name: Jordan Taylor; name: Ying-Jui Tseng; name: Patricia Vaidos; name: Zhijin Wu; name: Wei Wu; name: Chenyang Yang
Published: 2023-07-19 17:54:43

Abstract: LLMs have shown promise in replicating human-like behavior in crowdsourcing
tasks that were previously thought to be exclusive to human abilities. However,
current efforts focus mainly on simple atomic tasks. We explore whether LLMs
can replicate more complex crowdsourcing pipelines. We find that modern LLMs
can simulate some of crowdworkers' abilities in these "human computation
algorithms," but the level of success is variable and influenced by requesters'
understanding of LLM capabilities, the specific skills required for sub-tasks,
and the optimal interaction modality for performing these sub-tasks. We reflect
on human and LLMs' different sensitivities to instructions, stress the
importance of enabling human-facing safeguards for LLMs, and discuss the
potential of training humans and LLMs with complementary skill sets. Crucially,
we show that replicating crowdsourcing pipelines offers a valuable platform to
investigate (1) the relative strengths of LLMs on different tasks (by
cross-comparing their performances on sub-tasks) and (2) LLMs' potential in
complex tasks, where they can complete part of the tasks while leaving others
to humans.