CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation
Authors: Minzhi Li
Taiwei Shi
Caleb Ziems
Min-Yen Kan
Nancy F. Chen
Zhengyuan Liu
Diyi Yang
Venue: EMNLP 2023 Main (2023)
Keywords: Human–Artificial Intelligence Collaboration
Large Language Model
Data Annotation
Weak Supervision

Abstract: Annotated data plays a critical role in Natural Language Processing (NLP) in training models and evaluating their performance. Given recent developments in Large Language Models (LLMs), models such as ChatGPT demonstrate zero-shot capability on many text-annotation tasks, comparable with or even exceeding human annotators. Such LLMs can serve as alternatives for manual annotation, due to lower costs and higher scalability. However, limited work has leveraged LLMs as complementary annotators, nor explored how annotation work is best allocated among humans and LLMs to achieve both quality and cost objectives. We propose CoAnnotating, a novel paradigm for Human-LLM co-annotation of unstructured texts at scale. Under this framework, we utilize uncertainty to estimate LLMs' annotation capability. Our empirical study shows CoAnnotating to be an effective means to allocate work from results on different datasets, with up to 21% performance improvement over random baseline. For code implementation, see https://github.com/SALT-NLP/CoAnnotating.

CoAnnotating:
Uncertainty-Guided Work Allocation between
Human and Large Language Models for Data Annotation
Minzhi Li f8
Taiwei Shi
Caleb Ziems
Min- Yen Kan 1 Nancy F. Chen
Zhengyuan Liu Diyi Yang
Nattoona University of Singapore
Institute for Infocomm Research (IRR), A*STAR
FUniversity of Southern California
Stanford University
1i. nus. edu taiweish@usc. edu edu cziems@stanford.c edu
nf nfychene22r.c a-star edu. sg
1iu stu_zhengyannii2..o star edu. sg diyiy@cs. stanford. edu
kanmy@comp.c nus. www.nuu.uuuee.e
Abstract
Different Datasets
Performance Comparison
Annotation Decision
Annotated data plays critical role in Natural Language Processing (NLP) in training models and evaluating their performance. Given re- cent developments in Large Language Models (LLMs), models such as ChatGPT demonstrate zero-shot capability on many text-annolation tasks, comparable with man annotators. Such LLMs can serve as al- ternatives manual annotation, due lower costs and higher scalability. However, limited work has leveraged LLMs as complementary annotators, explored how annotation work is best allocated among humans and I.LSs to achieve both quality and cost objectives. We propose CoAnnotating, novel paradigm for Human-LLM co-annotation of unstructured texts at scale. Under this framework, we uti- lize uncertainty to estimate LLMs' annotation capability. Our empirical study shows CoAnno- tating to an effective means allocate work from results on different datasets, with up to 21% performance improvement over random baseline. For code implementation, see https: //gi teib.hubccom www.tttw.w.....eeee Comm//ALTTNLP//Coontttiin
Evaluations (Task-level)
exceeding hu-
Uncertainty Computation
Expertise Estimation
Annotation Deciston
Dataset
using different prompts
Threshold
CoAnnotating Framework (Instance-level)
:AAAAA
Entropy
ARABB
Entropy 0.97 0.97
Figure CoAnnotating framework. differs from previous work by considering how allocate data within the same dataset humans and ChatGPT by obtaining responses from CharGPT using different variations of prompts and estimating ChatGPT's annotation expertise with the use uncertainty metrics such as entropy.
quality datasets and improve the performance fine-tuned models.
1 Introduction
Labeled data plays a critical role in establishing benchmarks and developing models for Natural Language Processing (NLP). Although Large Lan- guage Models (LLMs) like ChatGPT have demon- strated their strong zero-shot performance in var- ious tasks such as question answering, reasoning, natural language inference, sentiment analysis, and named entity recognition, results obtained by fine- tuned language models still outperform LLMs on most of these tasks (Qin et al., 2023; Zhong et al., 2023; Ziems et al., 2023). Therefore, collecting labeled data for model training and fine-tuning still valuable. Instead of deploying LLMs directly for downstream uses, worthwhile investi- gate how researchers can leverage LLMs' zero-shot capability in labeling text data to construct high-
Typically, researchers recruit human annotators such as experts or crowd workers to perform data annotation (Kittur et al., 2008; Snow et al., 2008). Some challenges in manual annotation includes high costs of recruiting and training annotators, annotation inconsistency and human subjectivity (Lingren et al., 2014; Grosman et al., 2020). Re- cent work explored how LLMs perform relative to crowd workers (Ding et al., 2022) and results showed that it is possible for LLMs like ChatGPT to replace large-scale manual annotation (Huang et al., 2023; Kuzman et al., 2023). In some cases, LLMs' annotation quality even outperforms human annotators on certain tasks (Gilardi et al., 2023). Given the much lower annotation cost than crowd workers, LLMs are considered to have great poten-

---

tial to increase the cost efficiency of the data annota- tion process. However, some studies also show that, relative to human performance, LLMs' zero-shot performance falls short on more difficult and prag- matic tasks (Wang et al., 2021; Kocoń et al., 2023). They suggest that practitioners should use caution when using LLMs to annotate data (Reiss, 2023; Huang et al., 2023). Such prior works view humans and LLMs competitors, measuring the accuracy of LLM labels as a replacement for human anno- tation, rather than considering how humans and LLMs might collaborate in an efficient manner. is Human-LLM collaboration that motivates this work. We propose the CoAnnotating framework, which aims to balance the complementary profiles of humans and LLMs in terms of their respective annotation quality and cost.
2 Related Work
2.1 Weak Supervision
In a traditional supervised learning setting, every training data point is labeled by human annota- tors. However, acquiring manually annotated la- bels for training data can be prohibitively costly and time-consuming. Weak supervision helps to address the challenge using partially and imper- fectly labeled data for training (Zhang et al., 2022). Weak supervision techniques obtain these noisy la- bels by tapping into heuristics (Ratner et al., 2017; Meng et al., 2018; Awasthi et al., 2020), feature annotation (Mann and McCallum, 2010), external knowledge bases (Hoffmaan al., 201 Min al., 2013), pretrained models (Bach al., 2019; Zhang et al., 2021) and third-party tools (Lison et al., 2020). Moreover, weak supervision can be com- bined with the active learning framework (Gonsior et al., 2020) to select the most informative data to be annotated by humans and utilize weak supervi- sion to decide noisy labels. Given LLMs' stumning zero-shot capabilities, our work explores the pos- sibility of using them as a more efficient labeling source, thus freeing up resources to be reinvested in the research pipeline.
Our work tackles the problem of Human-LLM co-annotation from resource allocation perspec- tive. Following Gentile et al. (2022), Diao al. (2023) and Wang et al. (2021), we consider model confidence as a reliable signal for the model's ex- pected performance. As we consider allocating a given datapoint for an LLM to annotate, we can use the inverse of the model's 's uncertainty to estimate our confidence in that allocation. Under CoAnno- tating, quantify LLMs' annotating expertise the instance-level (estimating how well LLMs can annotate the given data point) beyond task-level (evaluating how LLMs performs on overall for each dataset). As such, a more informed allocation de- cision can be made with this fine-grained and con- textualized instance-level perspective, rather than broad and coarse dataset-level expertise.
2.2 LLMs for Annotation
Most prior works frame the decision for human LLM annotation one of competition rather than collaboration between these modes. These show that LLMs like GPT-3 davinci-003 have strong zero-shot sentiment analysis performance (Ding et al., 2022). ChatGPT (gpt-3. 5-turbo) (gpt-335--tubr performs surprisingly well on automatic genre detection in under-resourced languages like Slovenian (Kuz- man et al., 2023). ChatGPT can even achieve high accuracy on some of the most nuanced tasks like implicit hate speech detection (Huang et al., 2023). Similarly, GPT-4 is able to annotate texts that quire reasoning and contextual knowledge and pro- vide explanations that could facilitate interpretive rescarch (Törnberg, 2023). These results show the great potential of LLMs as data annotation tools with just simple prompt design and without much manual labeling efforts (Kuzman et al., 2023).
We show that our proposed method using the un- certainty of responses can achieve more efficient and more accurate work allocation than the random allocation baseline. Our results also show that con- fidence scores generated by LLMs are generally well-calibrated but not always reliable. It is possi- ble to outsource some annotation work to achieve heman-level performance for more straightforward tasks like topic understanding. On the other hand, a tradeoff between annotation quality and anno- tation cost is inevitable for more nuanced tasks. Our framework establishes guide to effectively allocate Al and human efforts in collaborative anno- tation, and in doing so, provides key insights into the capacities of LLMs, as well as the nature of the tasks and data that remain outside these capacities.
However, there is still room to close significant performance gaps between LLMs' performance and existing fine-tuned baselines on some chal- lenging tasks. LLMs struggle with named entity recognition (Ding et al., 2022; Qin et al., 2023), relational reasoning (Bang et al., 2023), affective

---

tasks (Kocoñ et al., 2023; Amin al., 2023) and semantic similarity tasks (Kocmi and Federmann, 2023; Wang al., 2023). Moreover, does not out- perform fine-tuned baselines for generation tasks like question answering and text summarization (Tan et al., 2023; Wang et al., 2023). These works all take the perspective that LLMs and humans are competitors, making task-level comparisons between LLMs and humans/fine-tunel models for each dataset. Our work views LLMs and humans as potential collaborators, with the possibility to work with each other to annotate the same dataset.
TTex = Sentencel:
[sentence1]
Sentence2: (sentence2)
Prompt
Type
Please label if the following two sen- tences are paraphrases of each other. Please give your answer as "paraphrase" "not paraphrase". (TText)
Instruction
(Text) Please label the two sentences above are paraphrases each other. Please give your answer as "paraphrase" or "not paraphrase"..
Sequence Swapping
Given the following please classify the relationship of the following two sentences as "paraphrase" "not paraphrase". (Text)
sentences,
Paraphrase
2.3 Human-Machine Collaboration for Dataset Creation
Is it true that the following two sen- tences are/are not paraphrases of each other? Give your answer as "true" or "false". (Text)
True/False
The quality of the dataset and the cost of creating a dataset are two important but sometimes conflict- ing objectives in dataset creation. Previous work suggests human-Al collaborative framework that utilizes language models' generation capability and human revision and evaluation skills (Tekiroglu et al., 2020; Yuan et al., 2021; Bartolo et al., 2021; Liu et al., 2022) to create valuable datasets of high quality. For cost efficiency, some have proposed av- eraging or majority vote over human and machine outputs (Chaganty al., 2018; Ziems et al., 2023) and some initial empirical explorations such alyzing the random combination of distillation of LLM and manual annotation (Kang et al., 2023) as well as active labeling assignments via the logit outputs (Wang et al., 2021). Our framework takes both quality and cost into consideration by using uncertainty metrics to make informed human-Al work-allocation decisions to ensure cost efficiency without compromising quality.
What relationship do the following two sentences have? Is it "puraphrase" or "not paraphrase"? (TTxtt
Question Answering
Please choose option that best de- scribes the relationship between the fol- lowing two sentences. (Text]
Multiple Choice Question
(A) Paraphrase
(B) Not paraphrase
think the following two sentences are/are not paraphrases of each other.
Question with Confirmation Bias
Do you agree? (Text)
Table I: Examples of our designed prompt types ask- ing ChatGPT to annotate each instance for the concrete task of paraphrase detection.
3.1 Prompt Construction
Previous work shows that LLMss performance can be highly sensitive to perturbations in input (Jang and Lukasiewicz, 2023). Therefore, in- troduce set of diverse types of prompts P. [pi1, Pi2, P.k) for each instance 1, Besides the (1) basic instruction format, we vary the prompts by swapping its sequence of sentences (2; symmetric perturbation), paraphrasing the instruction (3; se- mantic perturbation), enquiring in various question formats (4; True/False, 5: Textual Short Responses 6; Multiple Choice Question) and asking with con- firmation bias (7: negation perturbation).
3 CoAnnotating Framework
Our CoAnnotating framework sets up guide for annotating text data collaboratively (Figure 2). For a given unlabeled train dataset D. = {41, m. where t is the i-th instance in the dataset, our framework automatically decides whether each data instance should be annotated by human or by the LLMs (Section 3.3) by computing the certainty level of the LLMs's annotations for each instance (Section 3.2), with the goal of achieving a higher annotation quality and a lower annotation cost for a given dataset (Section 3.4).
3.2 Uncertainty Computation
In real-world setting, there no gold data on which to gauge the model's expected accuracy and thus decide on the optimal annotation strat-

---

$
Original
LLM Annotation for All Data
Econamical
Suthuman-Level Performence
CoAnnotating Pipeline
Manual Annotation for Data
90
Costyy
Human Lovel Performance
Random sampling
Expertise Estimation
Allocation
Allocation Analysis
tifiiency
Pilot Dataset
Uncertainty Computation
Pliot Dataset sorted Entropy (ms:)
Text XXX
Entropy
Text 100 XXX
Coclest
XXX
Prompting Ways
Entropy
Text 500 XXX
Around
Text 500 XXX
Entropy 0.76
Text XXX
performance.
4. Apply Step 182 Original
Figure 2: Workflow of CoAnnotating. The framework consists of uncertainty-guiie expertise estimation, work allocation, and cost performance Pareto analysis. With insights gained from Pareto analysis the pilot dataset, uncertainty-guiin work allocation can applied on the original unlabeled dataset to achieve greater cost efficiency.
egy. However, model confidence can serve as re- liable signal for model performance (Gentile et al., 2022; Diao et al., 2023; Wang al., 2021). There- fore we compute the LLM uncertainty u; to guide the work-allocation process. We compute u, in two ways which are easy to implement and have proven effectiveness in previous literature (Diao et al., 2023): (1) self-evaluation and (2) entropy. In each case, for t by prompting LLMs & times with different prompts in P; we get & annotations A, {ai1, 0:2,--., 0 for each instance. As ablation study (5.4), we also prompt LLMs times with the same prompt get annotations study the effect of prompt perturbations.
larger the entropy value, the more uncertain the responses are. We can use this metric to estimate the uncertainty level:
& Σ ==1
Polauglp)) Po(aiglps)
u, =
where Po(asj Pis)) is caleulated the frequency of certain prediction among predictions.
3.3 Work Allocation Strategies
Building upon the aforementioned uncertainty level estimation, we can then use the uncertainty level 0g to guide the work allocation.
vvaluation. Previous work shows that LLMs are well calibrated and can provide informa- tion about their uncertainty themselves (Wang et al., 2021; Kadavath et al., 2022; Diao et al., 2023). We ask the model to directly output its confidence score (Lin et al., 2022) by postpending the phrase "and please give a confidence score on scale of 0 to for your prediction". The uncertainty for t; is calculated by:
Random Allocation. Random allocation is cho- baseline strategy for comparison. This is the strategy that randomly samples instances (0 S S m) in D, to be annotated by LLMs while the remaining m data is annotated by humans.
Self- Evaluatto Guided Allocation. Wang et al. (2021) introduces active label assignment ap- proach that ranks outputs by their logits. Not all mod-
LLM APIs support this computation, ify this baseline with our self-evaluation approach. sorting instances by the self-reported confidence scores in decreasing order. We then select the top n. instances (0 < n S m) in D, with the lowest level of uncertainty as the best candidates for LLM annotation. The remaining m data points are allocated to human annotators.
&
= Po(aiglpg)
where Pa(a.j) Pij) is the probability of class label being annotated by ChatGPT given the prompt Pij- We obtain its value by extracting the confidence score provided by LLMs directly.
Entropy Guided Allocation. it not possible entirely ensure the reliability of black box LLMS self-reported confidence. Therefore, we also pro- pose the use of entropy across LLMs' responses
Entropy. Entropy is a measure of the impurity in a set of data and can be used to quantify the uncertainty associated with the class labels. The

---

gauge their certainty and reliability. We sort the instances by their respective entropy values in increasing order and select the top n instances (0 S n I m) in Dr with the lowest level of un- certainty to be annotated by LLMs. Again, the remaining m data points with inconsistent re- sponses will be allocated for human annotation.
of the train data too large, we will take a stratified random sampling for approximately 1000 samples.
Topic Classification is a challenging task for large pretrained language models like GPT-3 (Wang et al., 2021). We choose two representa- tive datasets: TREC (Li and Roth, 2002) and AG News (Zhang et al., 2015). AG News contains news titles and their descriptions, which were gathered by an academic news search engine, and which span four topics: world, sports, business, and sci- ence/technology. TREC contains of English ques- tions with six manually labeled class labels: ab- breviation; entity; description and abstract concept; human being; location; and numeric value.
3.4 Strategy Selection
We frame the co-annotation process as a multi- objective optimization problem with two main ob- jectives, maximizing annotation quality and min- imizing annotation cost. We can determine anno- tation quality by the classification performance of model fine-tuned using a certain co-annotation strategy. The total annotation cost the sum of manual annotation costs and those incurred by the LLM. Inspired by Kang et al. (2023), we apply the Pareto efficiency concept in strategy selection. Here, the Pareto efficient scenario refers to the situ- ation where it is impossible to increase the classifi- cation performance of the fine-tuned model with- out incurring higher annotation cost. By adopting different allocation strategies and setting different proportions of data allocated to LLMs, we get var- ious allocation patterns with different annotation qualities and costs. We can then plot the perfor- mances of each quality-cost combination and ap- proximate the Pareto frontier by interpolating the discrete data points (Abdolrashidi et al., 2021; ; Tre- plot annotation
Semantic Similarity known to challenge Chat- GPT (Jang and Lukasiewicz, 2023). We select MRPC (Dolan and Brockett, 2005) and Tem- WiC (Loureiro et al., 2022) as two representa- tive datasets for semantic similarity understanding. MRPC is corpus of sentence pairs extracted from online news and annotated by humans for whether the sentences are semantically equivalent. Tem- WoC contains annotated tweet pairs for whether there is a meaning shift of the target word.
Nuanced Comprehension We also experiment with Tweet Stance Detection (Mohammad et al., 2016a) and Conversation Gone Awry (Zhang et al., 2018) explore the collaboration paradigm on tasks requiring more nuanced comprehension. Tweet Stance Detection in SemEval-2211 (Moham- mad et al., 2016b) is a dataset of tweets annotated with the author's 's stance (favorable, neutral, and negative) toward certain topic and we select the topic of abortion.
viso al., 2022). Practitioners quality against the cost for pilot data to gain a bet- ter understanding of this tradeoff, and they can use the Pareto efficient points to decide which ratio of data they should outsource to LLMs their desired budget level.
4 Experiments
4.1 Datasets
4.2 LLM Annotation
We use six classification datasets for different types tasks. Since LLM inference costs much less than a human salary, we know the simple allocation de- cision is to choose LLMs over humans whenever an LLM achieves utility greater than or equal to that of human annotators. For a more challenging set- ting, we identify tasks in which LLMs are known struggle with discriminating the underlying con- structs (Pikuliak, 2023; Wang et al., 2021). In such cases, there is tradeoff between annotation quality and annotation cost and CoAnnotating facilitates better decision-min mekiing in such contexts, If the size
We obtain responses from ChatGPT (gpt-3.5-turbo) due to its high-quality an- notations and low inference cost (Kuzman et al., 2023) using different prompts carefully crafted in Table 1. If the response is an ambiguous answer such as "I cannot determine the class of the text", encode it new class label which can result in higher uncertainty metrics. The uncertainty computation decides whether annotation will be finally allocated to ChatGPT, and if so, we decide the final label with a majority vote across ChatGPT's generations (Wang et al., 2022).

---

Entropy(Different Prompts)
Entropy(Same Prompt)
Self-Evaluation
Noces
HIP:
Stance Detectiin
1003
200
603
Freqprce
000
603
242
0.0
0.2
044
0.8
1.0
0.0
0.2
0.4
1.0
TempolWCC
Conzersatio Come Awsy
345
202
Free en1
032
200
232
200
132
0.0
0.2
0.4
0.5
0.8
1,0
0.6
0.8
Value
Value
Vilue
Figure 3: Distribution of entropy and confidence values.
4.3 Evaluation
ent thresholds for entropy (lower than an entropy threshold) and self-confidenee score (higher than confidence threshold) select data that Chat- GPT is more certain about. For those instances selected, we evaluate ChatGPT's annotation qual- ity by calculating its alignment with the gold label (human annotation). Figure 4's decreasing trends for sutropy-guided allocation (green and blue dots) on all datasets validate our hypothesis of an inverse relationship between uncertainty and annotation quality. justifies the helpfulness of using the tropy of ChatGPT's annotations as an estimate for its annotating expertise. Importantly, we observe that ChatGPT's self-reported confidence scores (or- ange dots) are not consistently a good estimate for its annotation quality. For some datasets such as AG News (top left), most of the data (94.3% with calculation) has high self-reported confidence rang- ing from 0.8 to 1, which leads weak separation of data in terms of annotation quality. For MRPC (top middle), there decreasing trend where data instances with higher confidence scores in fact have a poorer alignment with gold labels. This shows that the reliability of using self-reported confidence from LLMs is not guaranteed.
To evaluate the quality of datasets annotated with different strategies, we fine-tune the same RoBERTa base classifier and calculate macro F1 scores test data for a fair comparison. We re- port macro F1 the performance due the unbalanced nature of LLMs' annotations for some datasets.
accurate representation of
In terms of cost, we only consider monetary cost in this work. We calculate human annotation costs based what was reported in the dataset paper. If the information is not applicable, each instance is annotated by 5 independent notators with a wage of $15/hour. We calculate ChatGPT annotation cost using the product of the token length of the input prompt and the price of calling API for (gpt-3.55 33. 5- tutbbo) ($0.002/16 k to- kens) at the time of experimentation.
assume
5 Results
5.1 Strategy Comparison
We plot the histograms for distribution of uncer- tainty metrics (entropy with different prompts and same prompt as well as confidence score). From Figure 3, we can observe that the model tends to be confident with its predictions with a skewed dis- tribution towards high confidence value although we ask ChatGPT normalize its answer.
The purpose of achieving a higher quality for train data is to ensure that it teach the classi- fier accurate information through fine-tuning. In Table carry out comparisons of different allo- cation strategies in terms of test performance after fine-tuning with such data labeled. We see that holding the proportion of data allocated to Chat-
We hypothesize that lower level of uncertainty in ChatGPT's response indicates a higher degree reliability in the label. Therefore, we set differ-

---

Self-vvalustion
Entropy Prompt)
Entropy
Prompts)
AG News
MRPC
Stance Detection
1.00
0.80
0.98
0.75
0.82
0.70
0.96
Allgnment with
0.80
0.65
0.94
0.78
0.92
0.76
0.8
2.0
TREC
TempoWiC
Conversation (Sone Awry
1.00
1.00
1.0
Aighnent with Human Label.
0.95
0.95
0.90
0.80
0.85
0.50
0.80
0.8
0.75
0.50
0.70
0.75
0.65
0.70
0.60
3.8
0.65
0.0
Figure 4: Scatter plots of the average alignment of ChatGPT's annotation with human annotation for train data aguinst the threshold. We vary the threshold for different metrics during work allocation to investigate the effectiveness of different metries in quantifying ChatGPT's annotation capability.
AGNews
TREC
Stance Delection
& ChatGPT
0
20
40
60
80
100
0
20
40
60
80
100
0
20
40
60
80
100
Strutegies
Macro F1
Random
88.2
87.9
85.8
79.8
81.8
82.6
92.1
88.1
86.1
81.6
76.4
75.8
60.2
53.9
53.6
55.0
50.4
536
Self Evaluation
88.2
36.0
84.9
84.1"
82.1
82.1
92.1
91.5
87.2
86.5
76.4
74.3
60.2
56.9
54.8
54.4
52.8
52.9
Entropy (DiI. Prompas)
88.2
$5.4
88.2
87.4"
84.0
82.6
92.1
91.9
87.4
80.8
79.2
75.8
60.2
58.2
55.1
56.8
54.7
53.6
Entropy Prompe)
TempoWIC
Stralegies
Random
$7.5
55.9
53.2
462
50.3
42.0
83.4
78.6
74.4
70.3
65.8
65.9
71.
63.1
54.5
57.1
50.0
54.1
Self-Evalattion
$7.5
57.8
55.9"
51.8"
52.9"
43.0
83.4
79.7
77.8
71.5
63.9
98.6
71.3
70.1"
62.6
64.2"
5044
50.7
Entropy (DiT. Prompts)
57,5
58.4
56.9"
55.9"
53.8"
42.0
83.4
80.0
79.8"
76.6"
73.1"
65.9
71.3
66.5
64.2"
62.6
56.2'
54.1
Entropy (Same Prompt)
$7.5
$6.3
53.5
52.9*
43.8
42.0
83.4
79.2
72.7
67.7
68.6
65.7
71.3
55.4
55.4
$1.1
54.8
54.6
Table 2: Test performance of fine-tuned RoBERTa under different allocation strategies. We vary the percentage data allocated to ChatGPT for anmotation and carry out finetuning using train data annotated under different strategies for all six datasets. Figure with superscript means the result under that strategy is significantly better than baseline strategy at 10% significance level.
GPT fixed (e.g., taking the setup of 40% for Tem- poWIC as an example), our proposed uncertainty- guided allocation using self-evaluation and entropy results in a better-annor anttoa-annottee dataset, reflected by its higher test F1 (56.9) than the random alloca- tion baseline (53.2). More often than not, entropy- guided allocation is better than seffidenne-guidedd allocation. This is probably due to the skewed dis- tribution of self-reported confidence, resulting in
poorer distinguishability between instances LLMs are better or worse annotating.
5.2 Pareto Efficient Allocation
By plotting test performance against annotation cost (Figure 5), practitioners can visualize the trade- off in annotation quality achievable at different budgets with collaboration between human and an LLM like ChatGPT by studying the Pareto frontier.

---

AGI Roos
HFPC
Stance Detection
Deray
Dravpy
eualuation
SuT Cuilaatoon
5e Eeuuuitee
42
50
P4
100
152
TamgssitC
Rractr
67.6
15.
300
426
55
450
426
0
100
123
44
10
130
200
200
300
250
Figure 5; Pareto curves under different allocation strategies (random, entropy guided, self-evaluation guided). The Pareto frontier is highlighted, illustrating the optimal choices that are Pareto efficient.
Dataset
Text
Groundtruth
ChatGPT
News
Tide: Sprint Set to Dehut Video Streaming Cell Phone Description: OVERLAND PARK. Kan. (AP) Channel surfing is moving the couch Sprint Corp...
Sci/Tock
Business
TREC
What does A&W beer
for?
Ahbreviation
Entiy
Stance Detection
As former fetus oppose Proolie Youth #SemST
Negative
Neutral
Zainab Yusef and not Zainab Khan? Blucbolt94; Does the credits the end of the episode say "Zainab Yusef"? No they say "Zainab Khan" and Yusef called her "Mrs, Khan" during the episode. So no, her name is "Zainah Khan" AnemomePrajectors: The Khans are clearly not traditional as Masoods, or Aña would have heen called Yusef. We already what GS said. Watch the show proprly P
rjoccolenty: Shouldn't her
Conversation
True
False
Sentencel:
At p.m. EITT, Henri had maximum sustained winds near 50 mph. with some gusts reaching mph. Sentence2:
MRPC
p.m. Friday. . Henri was hecoming disorganized, but still had maximum Not paraphrase
Parpphrase
sustained winds near 50 mph with stronger gusts.
TempoWiC
tweet 1: If you need some to watch on Netflix, containment is so good. tweet 2: have lot of questions
Same
Different
about containment series. farget word: containment
Table 3: Specific instances with high entropy values for ChatGPT annotations.
Points along the highlighted Pareto frontier mean it is theoretically impossible to achieve a higher test accuracy without increasing the budget, and it is also impossible to reduce the cost but achieve the same level of annotation quality. Furthermore, provides information on the approximate propor- tion that can be outsourced to ChatGPT to achieve human-level performance. For more straightfor- ward tasks like topic classification, part of the an- notation work could be potentially outsourced to ChatGPT and lead to cost reduction (c.g., AG News: 33%) by ensuring human-level annotation performance. For datasets requiring nuanced com- prehensions like Stance Detection and Conversa-
tion Gone Awry, any level of outsourcing to the current version of ChatGPT compromises annota- tion quality. Practitioners can choose among the Pareto efficient points based on their budgets.
5.3 Qualitative Analysis
We select some instances with entropy values higher than 0.8 from each dataset (Table 3) to un- derstand the current challenges faced by ChatGPT in annotating data. We find that ChatGPT has high uncertainty for instances containing sarcasm and incomplete sentences that require more inference during opinion mining. For example, in deciding the stance towards abortion for the tweet "as a for-

---

mer fetus I opposse", the incomplete nature of this sentence causes confusion to ChatGPT. Also, struggles with numerical reasoning as seen from its inability to compare wind speed during paraphrase detection and may be misled by some keywords ("Corp") related to other incorrect classes ("busi- ness") in topic classification.
where LLMs has seen some datasets in our experi- ment, making entropy values obtained for LLMs' responses lower. As an initial exploration of the co- annotating concept, this work aims for human-level performance in annotating datasets. does not con- sider the scope of superhuman-level performance where we treat human annotation in each dataset as gold labels. Future work can further investigate the instances where LLMSs actually annotates better than humans. We consider annotating profiles of human and LLMs as two groups but this frame- work can be further enriched by taking variations within each group (expert, crowd workers, differ- ent LLMs) into considerations. More exploration can also be carried out to investigate how to design prompts in a way that can increase LLMs's anno- tating expertise that more annotation work be outsourced LLMs for greater cost efficiency, Moreover, this work only did experiments for clas- sification tasks and English datasets. However, the idea of CoAnnotating is generalizable to generation tasks and datasets in other languages as well, which are meaningful to study in future work.
5.4 Ablation Study
We carry out inferences with the same instruction- formatted prompt for the same number of times and compute the entropy for ChatGPT's responses, From Figure 4, we observe some extent of ef- fectiveness of computing entropy using the same prompt in quantifying ChatGPT's capability, as re- flected by a decreasing pattern of alignment with the increased threshold. However, serves as a much weaker method to quantify expertise com- pared with our method with different prompt de- signs since the majority of the data has zero entropy (see Figure 3). This suggests that CharGPT's re- sponses are generally consistent within multiple applications of the same prompt. In Table 2, the test performance of entropy antropy-guided allocation un- der different prompts consistently higher than when based on single prompt. The performance gap gives strong evidence of the utility of applying different prompt types in Table
Ethical Statement
We are aware of the potential ethical concerns of using LLMs potential labelers in the data annota- tion process in terms of the perpetuation of existing biases in LLMs. Since LLMs are trained on vast amounts of texts on the Internet, they can unavoid- ably incorporate the biases present in these data sources. Such biases could be under-repressentation of certain demographic groups, cultural stereotypes as well as linguistic biases. However, we believe that the benefit of proposing collaborative co- annotation framework outweighs the potential risks related to the framework.
6
Conclusion
This work introduces CoAnnotating, a framework which takes a collaborative angle to view the re- lationship between humans and LLMs when an- notating each dataset. Under this framework, we uncertainty metries to estimate LLMs' anno- tating capability and guide effective work alloca- tion. Moreover, we apply the Pareto efficiency concept for practitioners to compare strategies and understand cost cost-performance tradeoffs. The em- pirical results demonstrate the effectiveness of our proposed framework in achieving greater cost effi- ciency. Overall, our framework provides important insights around the reliability of self-reported confi- dence score by LLMs, the sensitivity of ChatGPT's responses to prompt variations as well the extent which human resources can freed by LLMs to be put on more meaningful areas.
Acknowledgement
We are thankful to the members of SALT Lab and WING Lab as well as anonymous EMNLP review- ers for their helpful feedback. Minzhi Li is sup- ported by the A STAR Computing and Information Science (ACIS) Scholarship. Caleb Ziems is sup- ported by the NSF Graduate Research Fellowship under Grant No. DGE-2039655. We would like to acknowledge grant from the Office of Naval Research DY, and National Research Founda- tion, Singapore under its AT Singapore Programme (AISG Award No: AISG2-GC 2--2-GCC-0222000
7 Limitations
Since LLMs has been trained on a large num- ber of datasets, there may be data leakage issue

---

References
Bill Dolan and Chris Brockett. 2005. Automati- cally constructing a corpus of sentential paraphrases. In Third International Workshop Paraphrasing (IWP2005).
AmirAli Abdolrashidi, Lisa Wang, Shivani Agrawal, Jonathan Malmaud, Oleg Rybakov, Chas Leich- ner, and Lukasz Lew. 2021. Pareto-optimal quan- tized resnet is mostly 4-bit. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tem Recognition (CVPR) Workshops, pages 3091 3099.
Claudio Gentile, Zhilei Wang, and Tong Zhang. 2022. Fast rates in pool-based batch active learning. arXiv preprint arXiv:2202.05448
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms annotation tasks. arXiv preprint arXiv:2303. 15056.
Mostafa M Amin, Erik Cambria, and Bjöm W Schuller. 2023. Will affective computing emerge from foun- dation models and general ai? first evaluation on chatgpt. arXiv preprint 1rXiv:2303.03//6
for text-
Julius Gonsior, Maik Thiele, and Wolfgang Lehner. 2020. Weakal: Combining active learning and weak supervision. In Discovery Science: 23rd Interna- tional Conference, DS 2020, Thessaloniki, Greece, October 19-21, 2020, Proceedings 23, pages 34-49. Springer.
Rie Kubota Ando and Tong Zhang. 2005. framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6:1817-1853.
Galen Andrew and Jianfeng Gao. 2007. Scalable train- ing of l reggullarie log-linear models. In Proceed- ings of the 24th International Conference on Machine Learning. pages 33 40.
Jonatas S Grosman, Pedro HT Furtado, Ariane MB Rodrigues. Guilherme G Schardong, Simone DJ Bar- bosa, and Hélio CV Lopes. 2020. Eras: Improving the quality control in the annotation process for natu- ral language processing tasks. Information Systems, 93: 10:1515.3.
Abhijeet Awasthi, Sabyasachi Ghosh. Rasna Goyal, and Sunita Sarawagi. 2020. Learning from rules generalizing labeled exemplars. arXiv preprint
Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and Daniel S Weld. 2011. Knowledge- based weak supervision for information extraction of overlapping relations. In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pages 541- 550.
arXiv:2004.06025.
Stephen H Bach, Daniel Rodriguez, Yintao Liu, Chong Luo, Haidong Shao, Cassandra Xia, Souvik Sen, Alex Ratner, Braden Hancock, Houman Alborzi, al. 2019. Snorkel drybell: A case study in deploying weak supervision at industrial scale. In Proceedings the 2019 International Conference on Management of Data, pages 362-375.
Fan Huang, Haewoon Kwak, and Jisun An. 2023. Is chatgpt better than human annotators? potential and limitations of chatgpt in explaining implicit hate speech. arXiv preprint 20X.:v2302.07736
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wen- liang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multi- task, multilingual, multimodal evaluation of chatgpt on reasoning. hallucination, and interactivity. arXiy preprint arXiv:2302.cm 04023.
Myeongjun Jang and Thomas Lukasiewicz. 2023. Consistency analysis of chatgpt. arXiv preprint arXiv:3303.06273.
Max Bartolo, Tristan Thrush, Sebastian Riedel, Pon- tus Stenetorp, Robin Jia, and Douwe Kiela. 2021. Models in the loop: Aiding crowdworkers with generative annotation assistants. arXiv preprint arXiv:2112.090x
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.0522 052.222......
Arun Tejasvi Chaganty, Stephen Mussman, and Percy Liang. 2018. The price of debiasing automatic met- ries in natural language evaluation. arXiv preprint D.XIv:1807.02000
Junmo Kang, Wei Xu, and Alan Ritter. 2023. Distill or annotate? cost-efficient fine-tuning of compact models. arXiv preprini 231::2005.01645
Shizhe Diao, Pengeheng Wang, Yong Lin, and Tong Zhang. 2023. Active prompting with chain-of- thought for large language models. arXiv preprint 12242
Aniket Kittur, Ed H Chi, and Bongwon Suh. 2008. Crowdsoureing user studies with mechanical turk. In Proceedings of the SIGCHI conference on human factors in computing systems, pages 453
Bosheng Ding, Chengwei Qin, Linlin Liu, Lidong Bing, Shafiq Joty, and Boyang Li. 2022. gpt-3 good data annotator? arXiv preprini arXiv:2212. 10450.
Tom Koemi and Christian Federmann. 2023. Large language models are state-of-ther th-the-art evaluators of translation quality. arXiv preprint arXiv:2302. 145200.

---

Jan Kocoñ, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydlo, Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz. Janz, Kamil Kanclerz, al. 2023. Chatgpt: Jack of all trades, master of none. arXiv preprint arXiv:2302. 10724.
Saif Mohammad, Svetlana Kiritchenko, Parinaz Sob- hani, Xiaodan Zhu, and Colin Cherry. 2016a. Semeval-2016 task 6: Detecting stance in tweets. In Proceedings of the 10th international workshop on semantic evaluation (SemEval-2016, pages 31 31-41.
Taja Kuzman, Nikola Ljuhešié, and Igor Mozetič. 2023. Chatgpt: Beginning of an end of manual annotation? use case of automatic genre identification. arXiv preprint arXiv:2303.0393
Saif Mohammad, Svetlana Kiritchenko, Parinaz Sob- hani, Xiaodan Zhu, and Colin Cherry. 2016b. Sem task 6: Detecting stance in tweets. In Proceedings of the 10th International Workshop on Semantic Evaluation SSemEval-2010 pages 31- 41, San Diego, California. Association for Computa- tional Linguistics.
Xin Li and Dan Roth. 2002. Learning question clas- sifiers. In COLING 2002: The 19th International Conference on Computational Linguistics.
Matóš vey:
Chatgpt sur- Performance on nlp datasets. http://wwwooopppmsaa...cccccooos seurus
Pikuliak. 2023.
Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205.1433:
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt general-purpose natural language process- ing task solver? arXiv preprint arXiv:2302.0m 06476.
Todd Lingren, Louise Deleger, Katalin Molnar, Hai- jun Zhai, Jareen Meinzen-Derr, Megan Kaiser, Laura Stoutenborough, Qi Li, and Imre Solti. 2014. Eval- uating the impact of pre-annotation on annotation speed and potential bias: natural language processing gold standard development for clinical named entity recognition in clinical trial announcements. Journal of the American Medical Informatics Association, 21 2164:4406-413.
Mohammad Sadegh Rasooli and Joel R. Tetreault. 2015. Yara parser: fast and accurate dependency parser. Computing Research Repository, arXiv: 1 1503.06733. Version 2.
Alexander Ratner, Stephen Bach, Henry Ehrenberg. Jason Fries, Sen Wu, and Christopher Ré. 2017. Snorkel: Rapid training data creation with weak pervision. In Proceedings of the VLDB Endowment. International Conference on Very Large Data Bases, volume 11, page 269. NIH Public Access.
Pierre Lison, Aliaksandr Hubin, Jeremy Barnss, and Samia Touileb. 2020. Named entity recognition without labelled data: weak supervision approach. arXiv preprint arXiv:2004.1472. 14723..
Alisa Liu, Swabha Swayamdipta, Noah A Smith, and Yejin Choi. 2022. Wanli: Worker and ai collaboration for natural language inference dataset creation. arXiv preprint arXiv:2201.0599
Michael Reiss. 2023. Testing the reliability of chatgpt for text annotation and classification: A cautionary remark.
Rion Snow, Brendan O' O'connor, Dan Jurafsky, and An- drew Y Ng. 2008. Cheap and fast-but is it good? evaluating non-expert annotations for natural lan- guage tasks. In Proceedings of the 2008 conference empirical methods in natural language processing, pages 254-263
Daniel Loureiro, Aminette D'Souza, Areej Nasser Muhajab, Isabella A White, Gabriel Wong, Luis Es- pinosa Anke, Leonardo Neves, Francesco Barbieri, and Jose Sanacho-Coolldoor 2022. Tempowic: An evaluation benchmark for detecting meaning shift in social media. arXiv preprint arxiv:2209.072.1 07216..
Yiming Tan, Dehai Min. Yu Li, Wenbo Li, Nan Hu, Yongrui Chen, and Guilin Qi. 2023. Evalu- ation of chatgpt as question answering system for answering complex questions. arXiv preprint arXiv:2303.07992
Gideon S Mann and Andrew McCallum. 2010. General- ized expectation criteria for semi-supervised learning with weakly labeled data, Journal of machine learn- ing research, ((2).
Serra Sinem Tekiroglu, Yi-Ling Chung, and Marco Guerini. 2020. Generating counter narratives against online hate speech: Data and strategies. arXiv preprint arXiv:2004.042/1:
Yu Meng, Jiaming Shen, Chao Zhang, and Jiawei Han. 2018. feakly-supervised neural text classification. In proceedings of the 27th ACM International Con- ference on information and knowledge management, pages 983-992
Petter Tömberg, 2023. Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning. arXiv preprint arXiv:2304.06588.
Bonan Min. Ralph Grishman, Li Wan, Chang Wang. and David Gondek. 2013. Distant supervision for re- lation extraction with an incomplete knowledge base. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 777-782.
Marcos Treviso, António Góis, Patrick Fernandes, Er- ick Fonseca, and Andre Martins. 2022. Predicting attention sparsity transformers. Proceedings of the Sixth Workshop on Structured Prediction for

---

NLP. pages 67-81, Dublin, Ireland. Association for Computational Linguistics.
Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang
Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023. Is chatgpt good nlg evaluator? a preliminary study. arXiv preprint arXiv:2303.0404:
Shuohang Wang, Yang Liu, Yichong Xu, Chenguang
Zhu, and Michael Zeng. 2021. Want reduce labeling cost? gpt-3 can help. arXiv preprint
arXiv:2108. 13487.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Lc, Ed Chi, and Denny Zhou. 2022. Self-consistency im- proves chain of thought reasoning language mod- cls. arXiv preprint arXiv:2203.11 11171..
Ann Yuan, Daphne Ippolito, Vitaly Nikolaev, Chris Callison-Bured Andy Coenen, and Sebastian Gehrmann. 2021. Synthbio: case study human- ai collaborative curation of text datasets. arXiv preprint 1.1662711.06467.
Jieyu Zhang, Cheng-Yo Yu Hsieh, Yue Yu, Chao Zhang, and Alexander Ratner. 2022. A survey on pro- grammatic weak supervision. arXiv preprint arXiv:2202.05333.
Jieyu Zhang. Bohan Wang, Xiangehen Song, Yujing Wang, Yaming Yang, Jing Bai, and Alexander Rat- ner. 2021. Creating training sets via weak indirect supervision. arXiv preprint arXiv:21 10.03484.
Justine Zhang, Jonathan Chang, Cristian Danescu- Niculescu-Mizil, Lucas Dixon, Yiqing Hua, Nithum Thain, and Dario Taraborelli. 2018. Conversations gone awry: Detecting early signs of conversational failure. arxiv preprint arXiv: 1r5vv1885.00535.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classi- fication. Advances neural information processing systems, 28.
Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2023. Can chatgpt understand too? a comparative study on chatgpt and fine-tuned bert. arXiv preprini urXiv:2302.1 10/98.
Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2023. Can large lan- guage models transform computational social sci- ence? arXiv preprint 07X5542305.03351.

---

A Model Setting
Wall Street's dwindling band of ultra-cynies, are seeing green again."
ChatGPT parameters: temperature = 0.7:
max max_tokens- = 800; top_p 0.95;
True/False 3. Based the language used, is it true that the following news title and description belongs to the topic of "business"?
fyequency_penalty- 0; presence__eenally
= 0; oppeaaiappi_eersslee
Text: "Title: Wall St. Bears Claw Back Into the Black (Reuters) Description: Reuters Short-sellers, Wall Street's dwindling band of ultra-cynics, are sceing green again."
RoBERTa base parameters: Adam optimizer, learning.rate-2e55, correct_bias True
B Prompts
True/False 4. Based on the language used, is it true that the following news title and description belongs to the topic of "sci/tech"?
In this section, we present specific prompts used to obtain annotations from ChatGPT for each dataset during our experiments.
Text: "Title: Wall St. Bears Claw Back Into the Black (Reuters) Description: Reuters Short-sellers, Wall Street's dwindling band of ultra-cynics, are seeing green again."
B.1 AG News
Instruction. Please label the topic of the follow- ing title and description "world", "sports", "business" or "sci/tech".
Question Answering. What topic does the fol- lowing news title and description belong to? Is it "world", "sports", "business" "sci/tech"? Text: "Title: Wall St. Bears Claw Back Into the Black (Reuters) Description: Reuters Short-sellers. Wall Street's 's dwindling band of ultra-cynics, are sceing green again."
Text: "Title: Wall St. Bears Claw Back Into the Black (Reuters) Description: Reuters Short-sellers, Wall Street's dwindling band of ultra-cynics, are seeing green again."
Sequence Swapping. Text: "Title: Wall St. Bears Claw Back Into the Black (Reuters) Descrip- tion: Reuters Short-sellers, Wall Street's dwin- dling band of ultra-cynics, are seeing green again." Please label the topic of the following news title and description as "world", "sports", "business" or "sci/tech"'
Multiple Choice Question. Please choose one option that best describes the topic of the news title and description.
Text: "Title: Wall St. Bears Claw Back Into the Black (Reuters) Description: Reuters Short-sellers, Wall Street's dwindling band of ultra-cynics, seeing green again."
Given the following text, please classify the topic of the following news title and description as "world", "sports", "business" or "sci/tech".
Paraphrasing.
Options:
(A) World
Text: "Title: Wall St. Bears Claw Back Into the Black (Reuters) Description: Reuters Short-sellers, Wall Street's dwindling band of ultra-cynics, are seeing green again."
B Sports
Business
(D)Sci/tech
Confirmation Bias 1. think the following news title and description belongs to the "world" topic. Do you agree? Please give your answer as either yes or no.
True/False 1. Based the language used, is it that the following news title and description belongs the topic "world"?
Text: "Title: Wall St. Bears Claw Back Into the Black (Reuters) Description: Reuters Short-sellers, Wall Street's dwindling band of ultra-cynics, are seeing green again."
Text: "Title: Wall St. Bears Claw Back Into the Black (Reuters) Description: Reuters Short-sellers. Wall Street's dwindling band of ultra-cynics, are seeing green again."
Based on the language used, it true that the following news title and description belongs the topic "sports""
True/False 2.
Confirmation Bias 2. think the following news title and description belongs to the "sports" topic. Do you agree? Please give your answer as either yes or no.
Text: "Title: Wall St. Bears Claw Back Into the Black (Reuters) Description: Reuters Short-sellers,
Text: "Title: Wall St. Beary Claw Back Into the

---

Black (Reuters) Description: Reuters Short-sellers, Wall Street's dwindling band of ultra-cynics, are seeing green again."
True/False 3. Based on the language used, is it true that the following question belongs to the class of description and abstract concept? Please give your answer as either "true" or "false". Text: "What makes a clitoris sensitive?"
Confirmation Bias 3. I think the following news title and description belongs the "business" topic. Do you agree? Please give your answer as either yes or no.
True/False 4. Based on the language used, is it true that the following question belongs to the class of human being? Please give your answer as either "true" or "false".
Text: "Title: Wall St. Bears Claw Back Into the Black (Reuters) Description: Reuters Short-sellers, Wall Street's dwindling band of ultra-cynics, are seeing green again."
Text: "What makes a clitoris sensitive?"
True/False 5. Based on the language used, is it
Confirmation Bias 4. I think the following news title and description belongs to the "sci/tech" topic. Do you agree? Please give your answer either yes
true that the following question belongs to the class of location? Please give your answer as either "true" or "false".
Text: "What makes a clitoris sensitive?"
Text: "Title: Wall St. Bears Claw Back Into the Black (Reuters) Description: Reuters Short-sellers, Wall Street's dwindling band of ultra-cynics, are sceing green again." "
True/False 6. Based on the language used, is it true that the following question belongs to the class of numeric value? Please give your answer as either "true" or "false".
B.2 TREC
Text: "What makes clitoris sensitive?"
Instruction. Please label the type of the follow- ing question "abbreviation", "'nniit", "descrip- tion and abstract concept". "human being", "loca- tion", or "numeric value".
Question Answering. Which class does the fol- lowing question belong to? Is "abbreviation", "entity", "description and abstract concept", "hu- man being", "location", or "numeric value"? Text: "What makes clitoris sensitive?"
Text: "What makes a clitoris sensitive?"
Sequence Swapping. toris sensitive?"
Text: "What makes a cli-
Multiple Choice Question. Please choose one option that best describes the class of the following question.
Please label the type of the following question as "abbreviation", "entity", "description and abstract concept", "human being", "location", or "numeric value".
Text: "What makes clitoris sensitive?"
Options:
Ab)bbbrvviition
(B) Entity
Paraphrasing. Given the following question, please classify the type of the following question as "abbreviation", "entity", "description and abstract concept", "human being", "location". or "numeric value".
Descrriptoo and abstract concept
Hu)mmman being
Location
NF)Numrric value
Text: "What makes clitoris sensitive?"
Confirmation Bias think the following ques- tion belongs to the abbreviation topic. Do you agree? Please give your answer as either "yes" or "no".
True/False 1. Based on the language used, is it true that the following question belongs to the class of abbreviation? Please give your answer either "true" or "false"
Text: "What makes clitoris sensitive?"
Text: "What makes a clitoris sensitive?"
Confirmation Bias 2. think the following ques- tion belongs to the entity topic. Do you agree? Please give your answer as either "yes" or "no". Text: "What makes a clitoris sensitive?"
True/False Based on the language used, that the following question belongs the class of entity? Please give your answer as either "true" or "false".
Confirmation Bias 3. think the following ques- tion belongs to the description and abstract concept
Text: "What makes a clitoris sensitive?"

---

topic. Do you agree? Please give your answer as either "yes" or "no". Text: "What makes a clitoris sensitive?"
said Vicki Saporta , president of the National
Abortion Federation
Is it true that the following two sentences are paraphrases of each other? Give your answer "true" or "false".
True/False 1.
Confirmation Bias 4. I think the following ques- tion belongs to the human being topic. Do you agree? Please give your answer as either "yes" or "no".
Text: "Sentencel: They are trying turn him into martyr, said Vicki Saporta president of the National Abortion Federation which tracks abortion-related violence.
Text: "What makes clitoris sensitive?"
Confirmation Bias 5. I think the following ques- tion belongs to the location topic. Do you agree? Please give your answer as either "yes" "no""
Sentence2: We need to take these threats seriously said Vicki Saporta, president of the National Abortion Federation
Text: "What makes a clitoris sensitive?"
Is true that the following two sentences are not paraphrases of each other? Give your answer as "true" or "false".
True/False 2.
Confirmation Bias 6. T think the following ques- tion belongs to the numeric value topic. Do you agree? Please give your answer as either "yes" or "no".
Text: "Sentence They are trying to turn him into martyr, said Vicki Saporta president of the National Abortion Federation, which tracks abortion-related violence.
Text: "What makes clitoris sensitive?"
B.3 MRPC
Sentence2: We need to take these threats seriously said Vicki Saporta, president of the National Abortion Federation
Instruction. Please label if the following two sentences are paraphrases of each other. Please give your answer as "paraphrase" or "not paraphrase". Text: "Sentencel: " They are trying to turn him into a martyr, , " said Vicki Saporta, president of the National Abortion Federation which tracks abortion-related violence.
Question Answering. What relationship do the
following two sentences have? Is it "paraphrase" or "not paraphrase"?
Text: "Sentencel: They are trying to turn him into martyr, said Vicki Saporta, president of the National Abortion Federation which tracks abortion-related violence.
Sentence2: We need to take these threats seriously said Vicki Saporta, president of the National Abortion Federation."
Sentence2: We need to take these threats seriously said Vicki Saporta, president of the National Abortion Federation."
Sequence Swapping. Text: "Sentencel: " "The are trying to turn him into a martyr, " said Vicki Saporta, president of the National Abortion Feder- ation, which tracks moortion-relttee violence
Multiple Choice Question. Please choose one option that best describes the relationship between the following two sentences.
Sentence2: We need to take these threats seriously said Vicki Saporta, president of the National Abortion Federation."
Text: "Sentencel: They are trying to turn him into martyr, said Vicki Saporta president of the National Abortion Federation which tracks cooution-related violence.
Please label if the following two sentences are para- phrases of each other. Please give your answer as "paraphrase" or "not paraphrase".
Sentence2: We need to take these threats seriously said Vicki Saporta, president of the National Abortion Federation Options: (A))araphrase B Not paraphrase
Paraphrasing. Given the following two sen- tences, please classify the relationship of the fol- lowing two sentences as "paraphrase" or "not para- phrase".
Text: "Sentencel: " They are trying turn him into a martyr, said Vicki Saporta, president of the National Abortion Federation which tracks coortion-related violence.
Confirmation Bias 1. I think the following two sentences are paraphrases of each other. Do you agree? Please give your answer in "yes" or "no". Text: "Sentencel: "They are trying to turn him
Sentence2: : We need to take these threats seriously

---

into a martyr, " said Vicki Saporta, president of the National Abortion Federation, which tracks abortion-related violence.
redlining mortgages, and he is conservative. If the dems think the only way you can beat trump is with a republican, then you deserve trump, and you show just how weak the dems are. #NotMeUs
Sentence2: " WWe need to take these threats seriously said Vicki Saporta, president of the National Abortion Federation." ."
True/False 1. Is it true that the meaning of the word frisk in the following 2 sentences is the same? Please give your answer in yes no.
Confirmation Bias I think the following two sentences are not paraphrases of each other. Do you agree? Please give your answer in "yes" or "no".
Sentence I: imagine seeing qoute from cave story making into smash as dlc character instead of frisk or sans Imao
Text: "Sentencel: " They are trying to turn him into a martyr, , said Vicki Saporta, president of the National Abortion Federation, which tracks abortion-related violence.
Sentence 2: Bloomberg? Are you people for real?16 cases of sexual harrassment, stop and frisk, redlining mortgages, and he is a conservative. If the dems think the only way you can beat trump is with republican, then you deserve trump, and you show just how weak the dems are. #NotMeUs
Sentence2: " We need take these threats seriously said Vicki Saporta, president of the National Abortion Federation."
True/False 2. Is it true that the meaning of the word frisk in the following 2 sentences is different? Please give your answer as yes no.
B.4 TeMpoWiC
Instruction. Please label the meaning of the word frisk in the following sentences the same different.
Sentence 1: imagine seeing qoute from cave story making into smash as a dlc character instead of frisk or sans Imao
Sentence imagine seeing qoute from cave story making it into smash as dle character instead of frisk or sans Imao
Sentence 2: Bloomberg? Are you people for real?16 cases of sexual harrassment, stop and frisk, redlining mortgages, and he is conservative. If the dems think the only way you beat trump is with republican, then you deserve trump, and you show just how weak the dems are. #NotMeUs
Sentence 2: Bloomberg? Are you people for real?16 cases of sexual harrassment, stop and frisk, redlining mortgages, and he is conservative. If the dems think the only way you can beat trump is with republican, then you deserve trump, and you show just how weak the dems are. #NotMeUs
Question Answering. Considering the meaning of the word frisk in the following 2 sentences, is it the same or different? Please give your answer "same" or "different". Sentence 1: imagine seeing qoute from cave story making it into smash as dic character instead of frisk or sans Imao
Sequence Swapping. Sentence 1: imagine see- ing qoute from cave story making it into smash dic character instead of frisk or sans Imao Sentence Bloomberg? Are you people for real?16 cases of sexual harrassment, stop and frisk, redlining mort- gages, and he is a conservative. If the dems think the only way you can beat trump is with repub- lican, then you deserve trump, and you show just how weak the dems are. #NotMeUs
Sentence 2: Bloomberg? Are you people for real?16 cases of sexual harrassment, stop and frisk, redlining mortgages, and he is a conservative. If the dems think the only way you can beat trump is with a republican, then you deserve trump, and you show just how weak the dems are. #NotMeUs
Please label the meaning of the word frisk in the following sentences as the same different.
Multiple Choice Question. Please choose one option that best describes the meaning of the word frisk in the following 2 sentences:
Paraphrasing. Given the following text, please classify if the meaning of the word frisk in the fol- lowing 2 sentences is the or different. Sentence 1: imagine seeing qoute from cave story making it into smash dle character instead of frisk or sans Imao
Sentence 1: imagine seeing qoute from story making into smash as a dlc character instead of frisk or sans Imao
Sentence 2: Bloomberg? Are you people for real?16 cases of sexual harrassment, stop and frisk, redlining mortgages, and he is a conservative. If
Sentence 2: Bloomberg? Are you people for real?16 cases of sexual harrassment, stop and frisk,

---

the dems think the only way you can beat trump is with republican, then you deserve trump, and you show just how weak the dems are, #NotMeUs
True/False Based the language used, deter- mine whether the following text holds favourable stance towards abortion. Please give your answer as either "true" or "false".
(A) Same
(B) Different
Text: "we remind ourselves that love means to be willing to give until it hurts Mother Teresa"
Confirmation Bias 1. I think the meaning of the word frisk the following sentences the same, do you agree?
Based the language used, de- termine whether the following text holds neutral stance towards abortion. Please give your answer as either "true" or "false".
True/False 2.
Sentence 1: imagine seeing qoute from cave story making it into smash as dlc character instead of frisk or sans Imao
Text: "we remind ourselves that love means to be willing to give until it hurts Mother Teresa"
Sentence 2: Bloomberg? Are you people for real?16 cases of sexual harrassment, stop and frisk, redlining mortgages, and he is conservative. If the dems think the only way you can beat trump with a republican, then you deserve trump, and you show just how weak the dems are. #NotMeUS
True/False 3. Based the language used, deter- mine whether the following text holds a negative stance towards abortion. Please give your answer as either "true" or "false".
Text: "we remind ourselves that love means to be willing to give until it hurts Mother Teresa"
Confirmation Bias 2. I think the meaning of the word frisk the following 2 sentences is different, do you agree?
Question Answering. What type of stance wards abortion is expressed in the following text? Is favourable, neutral negative?
Sentence 1: imagine seeing qoute from cave story making it into smash as dlc character instead of frisk or sans Imao
Text: "we remind ourselves that love means to be willing to give until hurts Mother Teresa"
Sentence 2: Bloomberg? Are you people for real?16 cases of sexual harrassment, stop and frisk, redlining mortgages, and he is conservative. If the dems think the only way you can beat trump is with a republican, then you deserve trump, and you show just how weak the dems are. #NotMeUs
Multiple Choice Question. Please choose option that best describes the stance towards abor- tion of the text.
Text: "we remind ourselves that love means be willing to give until hurts Mother Teresa"
Options:
(A) Favourable
B.5 Stance Detection
(B Neutral
Instruction. Please label the stance towards abor- tion of the following text as "favourable", "neutral" "negative".
(e)Negative
Confirmation Bias 1. think the following text has favourable stance towards abortion. Do you agrec? Please give your answer as either "yes" or "no".
Text: "we remind ourselves that love means to be willing give until it hurts Mother Teresa"
Sequence Swapping. Text: "we remind our- selves that love means to be willing to give until it hurts Mother Teresa"
Text: "we remind ourselves that love means to be willing to give until it hurts Mother Teresa"
Confirmation Bias 2. think the following text has a neutral stance towards abortion. Do you agrec? Please give your answer as either "yes" or "no".
Please label the stance towards abortion of the fol- lowing text as "favourable", "neutral" "nega- tive".
Text: "we remind ourselves that love means to be willing to give until it hurts Mother Teresa"
Paraphrasing. Given the following text, please classify the position towards abortion demonstrated in the following text "favourable". "neutral" or "negative""
Confirmation Bias 3. I think the following has a negative stance towards abortion. Do you agrec? Please give your answer as either "yes" or "no".
Text: "we remind ourselves that love means to be willing to give until it hurts Mother Teresa"

---

Text: "we remind ourselves that love means to be willing give until it hurts Mother Teresa"
miles per hour. not correct? Greswik: You don't It seem know the difference between mph and kph. 25 knots is 46 km/h, as the article read already.
B.6 Conversation Gone Awry
Instruction. Please label if the conversation be- low will eventually derail into personal attack. Even if you uncertain, you must pick either "True" or "False" without using any other words. Can.u.spel: im sorry to have been mistaken but whoever wrote that article made a miscalculation when determining the speed from knots to kph, knot is 1.85 of a mile. therefore 10 knots 18.5 miles per hour. am i not correct?
Is true that the conversation be- low will not eventually derail into personal at- tack? Even if you are uncertain, you must pick either "True" or "False" without using any other words.
True/False 2.
Can.u.spel: im sorry to have been mistaken but whoever wrote that article made miscalculation when determining the speed from knots to kph, knot 1.85 of a mile. therefore 10 knots is 18.5 miles per hour. am not correct?
Greswik: You don't tt seem to know the difference between mph and kph. 25 knots is 46 km/h, as the article read already.
Greswik: You don't t seem to know the difference between mph and kph. 25 knots is 46 km/h, as the article read already.
Sequence Swapping. Can.u.spel: im sorry to have been mistaken but whoever wrote that arti- cle made miscaleulation when determining the speed from knots to kph, knot is 1.85 of mile. therefore 10 knots is 18.5 miles per hour. not correct?
Question Answering. Considering the conversa- tion below, will it eventually derail into a personal attack? Even if you are uncertain, you must pick either "True" or "False" without using any other words.
Greswik: You don't seem to know the difference between mph and kph. 25 knots is 46 km/h, as the article read already. Please label if the conversation above will eventu- ally derail into personal attack. Even if you are uncertain, you must pick either "True" or "False" without using any other words.
Can.u.spel: im sorry to have been mistaken but whoever wrote that article made miscalculation when determining the speed from knots to kph, knot is 1.85 of mile. therefore 10 knots is 18.5 miles per hour. Greswik: You don't seem to know the difference between mph and kph. 25 knots is 46 km/h, as the article read already.
not correct?
Paraphrasing Given the following conversation, please classify if the conversation below will even- tually derail into hate speech. Even if you are uncer- tain, you must pick either "True" or "False" without using any other words.
Multiple Choice Question. Please choose one option that best describes the possible continuation of the following conversation. Can.u.spel: im sorry to have been mistaken but whoever wrote that ar- ticle made a miscalculation when determining the speed from knots to kph, knot is 1.85 of a mile. therefore 10 knots 18.5 miles per hour. am not correct? Greswik: You don't seem to know the difference between mph and kph. 25 knots is 46 km/h, as the article read already.
C.neu: im sorry to have been mistaken but whoever wrote that article made a miscalculation when determining the speed from knots to kph, knot is 1.85 of mile. therefore 10 knots is 18.5 miles per hour. am i not correct?
Greswik: You don't 't seem to know the difference between mph and kph. 25 knots is 46 km/h, as the article read already.
(A) it will eventually derail into a personal attack (B) it will not eventually derail into a personal at- tack
True/False 1. Is it true that the conversation be- low will eventually derail into a personal attack? Even if you are uncertain, you must pick either "True" or "False" without using any other words. Can.u.spel: im sorry have been mistaken but whoever wrote that article made a miscalculation when determining the speed from knots to kph, knot is 1,85 of a mile. therefore 10 knots 18.5
Confirmation Bias 1.
I think the conversation below will eventually derail into a personal attack, do you agree? Please give your answer in yes no.
Can.u.spel: im sorry to have been mistaken but whoever wrote that article made a miscalculation

---

when determining the speed from knots to kph, knot is 1.85 of mile. therefore 10 knots is 18.5 miles per hour. am not correct?
Greswik: You don't seem to know the difference between mph and kph. 25 knots is 46 km/h, as the article read already.
Confirmation Bias 2. think the conversation below will not eventually derail into a personal attack, do you agree? Please give your answer in yes or no.
Can.u.spel: im sorry to have been mistaken but whoever wrote that article made a miscalculation when determining the speed from knots to kph, knot is 1.85 of mile. therefore 10 knots is 18.5 miles per hour. am not correct?
Greswik: You don't seem know the difference between mph and kph. 25 knots is 46 km/h, as the article read already.


Reviews:
[{"review_id": "ZS0SHt502K", "reviewer": "Reviewer_mCqk", "summary": "", "questions": "Question A) How are you computing the Pareto frontier in Figure 4? I think the reader needs to understand how that is being arrived at in order to give context to how close the proposed framework is to that frontier.\n\nQuestion B) How would you extend the framework to take into account the inference cost of different quality LLMs?\n\nQuestion C) GIven the large similarity to active learning in general, why not compare it to other active learning methods such as query-by-committee or density-based approaches?  Only comparing to random allocation seems a simple baseline to improve upon.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful reviews and acknowledgement of our work as one which identifies a timely problem of how to use LLMs effectively.\n\n### Response to ‘Reasons to Reject’: \n1) **Difference between our work and active learning:** Our work is *“theoretically orthogonal to active learning”* (see Reviewer rzwv). Active learning is the process of selecting worthy training data which can provide the most information to the model to be labeled (usually by human annotators). It helps to save time and resources by reducing the number of examples to be labeled. On the other hand, our framework is an annotation allocation strategy for human-LLM teams working on a fixed set of data points to be labeled. It can be integrated with active learning but it is not a type of active learning approach. Using our experimental setting as one example, one can apply active learning approaches to select a subset of unlabeled data that is challenging to a RoBERTa model and then use our CoAnnotating framework to allocate the annotations between human labelers and large language models.  \n2) **Baseline selection:** As explained above, our work is not studying active learning strategies for large language models so we did not use traditional active learning strategies like query-by-committee and density-based approaches as baselines. The main research focus of our work is to tackle the problem of how to effectively assign annotation work to humans and large language models. Previous work aiming to solve this assignment problem only considered (1) random allocation [Kang et al., 2023] and (2) using LLMs’ self-reported confidence as allocation criteria [Wang et al., 2021], so we adopted these two methods as baselines. They are more suitable baselines since they cover the same research problem as our work.\n3) **Consideration of different LLMs beyond ChatGPT:** Thank you for pointing out this interesting point for future work! We actually mentioned it in our Limitations section (line 532-536): *“We consider annotating profiles of human and LLMs as two groups but this framework can be further enriched by taking variations within each group (expert, crowdworkers, different LLMs) into considerations.”* \n\n### Response to ‘Questions for Authors’:\n**Question A:** We plot the performances of each quality-cost combination and approximate the Pareto frontier by interpolating the discrete data points [Abdolrashidi et al., 2021] [Treviso et al., 2020]. We will add the method and its relevant references to the camera-ready to make it clearer. \\\n**Question B:** Our methods of estimating LLMs’ expertise by uncertainty and cost computations can generalize to different LLMs. Here is one possible way our framework can be extended: we can still apply the uncertainty method proposed in our work to estimate the annotation expertise of different models and rank them based on entropy (ascending order) for each model. If an instance appears in the top k% of the sorted lists for several models, we will label it using the model with the lowest inference cost. If an instance does not appear in the top k% of the sorted list for any model, we will assign it to a human annotator. This is a straightforward extension of our current framework with minimal modifications and it can be further improved with other justified modifications in future work. \\\n**Question C:** As explained in our response 2 to ‘Reasons to Reject’, our work is theoretically orthogonal to active learning so we did not set traditional active learning strategies like query-by-committee and density-based approaches as baselines. Strategies we selected as baselines (random allocation and using LLMs’ self-reported confidence) are existing previous methods which share the same research focus of our work so they are more suitable. However, we appreciate your suggestions and think other active strategies you suggested can inspire future extensions of our proposed methods to effectively allocate work between human and large language models.\n\n\n[Kang et al., 2023] Kang, J., Xu, W., & Ritter, A. (2023). Distill or annotate? cost-efficient fine-tuning of compact models. arXiv preprint arXiv:2305.01645.\n\n[Wang et al., 2021] Wang, S., Liu, Y., Xu, Y., Zhu, C., & Zeng, M. (2021). Want to reduce labeling cost? GPT-3 can help. arXiv preprint arXiv:2108.13487.\n\n[Abdolrashidi et al., 2021] Abdolrashidi, A., Wang, L., Agrawal, S., Malmaud, J., Rybakov, O., Leichner, C., & Lew, L. (2021). Pareto-optimal quantized resnet is mostly 4-bit. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3091-3099).\n\n[Treviso et al., 2020] Treviso, M., Góis, A., Fernandes, P., Fonseca, E., & Martins, A. F. (2021). Predicting attention sparsity in transformers. arXiv preprint arXiv:2109.12188.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper identifies a timely problem, how to use LLMs as annotators effectively, and proposes and demonstrates a solution framework to address the LLM annotation problem.  The results convincingly show that the proposed framework is better than random allocation with many points showing 10% significance level.  They also present evidence showing it is also close to Pareto frontier for many of the datasets evaluated.\n\nWhile uncertainty based methods have been studied in other work as a method of choice for getting a human label (see missing references), I think the results of uncertainty as applied to ChatGPT are in themselves interesting and have value.", "reasons_to_reject": "While this paper is well done and it's possible applying these ideas to the latest LLMs like ChatGPT is novel, I'm still concerned that the approach is not very novel in general - please see papers in the missing references section related to active learning.  Using uncertainty to select which machine labeled samples to have humans label is a very established idea.  For example consider this paper:\nQiu, Hang, Krishna Chintalapudi, and Ramesh Govindan. \"MCAL: Minimum Cost Human-Machine Active Labeling.\" The Eleventh International Conference on Learning Representations. 2022.  https://openreview.net/forum?id=1FxRPKrH8bw \nThat paper presents a very simlar system, but for CV applications.  This new application area is very relevant now and so the analysis is interesting, but my concern is if you replaced ChatGPT with a BERT or RNN or X older model the rest of the approach/framework seems very standard and well studied. \n\nRelated to above the only baseline compared to is random allocation, but there is a lot of other active learning strategies in literature such as query-by-committee and desnity-based approaches.  Having these more creative baselines may have been more interesting than the simple random allocation baseline.\n\nA limitation not identified in the paper is that only a single LLM model was considered, but there are many choices for such an LLM and that choice is also based on cost (although not as much as human vs LLM).  For example choosing between ChatGPT and GPT-4 has a significant cost impact at current prices with quality tradeoff at large scale.  It would have been interesting to also have that choice in the framework, perhaps that's an obvious extension.", "questions_to_authors": "Question A) How are you computing the Pareto frontier in Figure 4? I think the reader needs to understand how that is being arrived at in order to give context to how close the proposed framework is to that frontier.\n\nQuestion B) How would you extend the framework to take into account the inference cost of different quality LLMs?\n\nQuestion C) GIven the large similarity to active learning in general, why not compare it to other active learning methods such as query-by-committee or density-based approaches?  Only comparing to random allocation seems a simple baseline to improve upon.", "paper_topic_and_main_contributions": "A common problem in NLP is obtaining sufficiently high-quality training data at a reasonable cost.  Because LLMs have strong zero-shot performance and are much cheaper to use instead of paying a human's salary, many researchers may consider using them for annotation instead.  This paper proposes an alternative, the CoAnnotating Framework, which describes a set of techniques for choosing when to use LLM annotation and when to use the more expensive human annotation when the LLM is uncertain.  The paper discusses a few different methods for determining the uncertainty of the LLM: self-evaluation, where the model is asked to report its own uncertainty, entropy using samples from the same prompt, and entropy using samples from different prompts.  The different uncertainty methods are compared by looking at the performance of a downstream task-specific BERT model trained on the respective annotations.  Three tasks are considered: topic classification using TREC and AG News datasets, semantic similarity using MRPC dataset, and nuanced comprehension using the Tweet Stance Detection and Conversation gone awry datasets.  The methods are compared to a random allocation baseline.  Results show significant improvement for many of the tasks by using CoAnnotation Framework vs a random allocation, including many that are at a 10% significance level.  Additionally, they show how the methods compare to the optimal Pareto frontier in terms of F1/Cost($) tradeoff, where the methods closely track the frontier, especially as compared to random allocation.  The main limitations identified include potential leakage of test datasets with LLM training sets, lack of detailed breakdown of the annotation groups, prompt design, and how it is focused on English language annotation.  Ethical concerns identified include perpetuating bias in LLMs to the resulting datasets.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "No comments on style/grammer/presentation.", "Justification_for_Ethical_Concerns": "N/A"}, "extra_text": {"Missing_References": "This work seems very related to active learning in general and I would have expected more discussion on that point.  A few examples of a very similar systems include: \nQiu, Hang, Krishna Chintalapudi, and Ramesh Govindan. \"MCAL: Minimum Cost Human-Machine Active Labeling.\" The Eleventh International Conference on Learning Representations. 2022.  https://openreview.net/forum?id=1FxRPKrH8bw \nBeluch, William H., et al. \"The power of ensembles for active learning in image classification.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.\nEstablished literature for active learning includes\n S. Tong. Active learning: theory and applications. Stanford University, 2001."}}, {"review_id": "Jxpj8NO0uL", "reviewer": "Reviewer_rzwv", "summary": "", "questions": "1. Could it be considered to add some results of LLMs solving these problems under zero-shot and few-shot settings? It's possible that using ChatGPT itself may achieve better results than fine-tuning RoBERTa with annotated data.\n2. Could evaluations on different LLMs be added? For example, using other LLMs, would there be some differences in cost and annotation quality?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful reviews and appreciation of the considerable practical value of our work.\n\n### Response to ‘Reasons to Reject’:\n1) **Task selection:** To demonstrate the general capability, we selected tasks from a diversity of domains. We did not opt for sequence tagging tasks, as there exists some difficulty by directly applying the framework. It wouldn’t make sense to split the work at individual tags since both an LLM and a human annotator need to consider the whole sequence to provide the tag. And it might be difficult to compare performance at the level of sentences. If we need a distinct prompt for each tag, the length of the prompt would be variable and thus the cost of inference would be variable. We also chose not to consider some traditional NLI tasks like MNLI and QNLI [LLM+NLU], as LLMs can already achieve a very high F1 score and there will be no significant gap between human labeling and LLM labeling. Data leakage may be one explanation for LLMs’ strong performance on NLI; future work can consider newer unleaked NLI datasets if they are released.\n2) **Integration with active learning:** Thanks for pointing out active learning as one orthogonal area that our framework can be integrated with. We did not combine them in order to just study the effectiveness of our proposed framework, but it is a promising future extension to combine these two components to further improve the efficiency of the whole annotation pipeline. \n3) **Testing few-shot capability:** We did not experiment with the few-shot capability of LLMs because few-shot settings require annotated data (usually by human). We want to best understand the orthogonal contributions of human annotation and LLM annotation by disentangling the two. It will be interesting for follow-up works to build upon our framework to understand their interactions.\n\n### Response to ‘Questions for Authors’:\n**Question 1:** Here are some findings in previous work:\n1) AGNews: GPT3-Shot1 achieves 0.74 accuracy [Wang et al., 2021] which is worse than fine-tuning on annotated data (0.94) \n2) TREC: GPT3-Shot1 achieves 0.75 accuracy  [Wang et al., 2021] which is worse than fine-tuning on annotated data (0.95)\n3) MRPC: ChatGPT achieves 0.72 F1 score [Zhong et al., 2023] which is worse than fine-tuning on annotated data (0.93) \n4) TempoWIC: ChatGPT achieves 0.56 accuracy which is worse than fine-tuning on annotated data (0.66) [Ziems et al., 2023]\n5) TweetStance: ChatGPT achieves 0.56 accuracy [Kocoń et al., 2023] which is worse than fine-tuning on annotated data (0.69)\n6) Conversation Gone Awry: ChatGPT achieves 0.53 accuracy which is worse than fine-tuning on annotated data (0.65) [Ziems et al., 2023]\n\n**Question 2:** Yes, different models could be added. Although the empirical analysis is done for one representative of large language models - ChatGPT for a consistent comparison and analysis, we design CoAnnotating as a general framework that can be applied to any large language model with varied annotation cost and quality. This framework can still hold when new models with higher capability come out in future and we encourage future work to adopt our framework with different models.\n\n[Wang et al., 2021] Wang, S., Liu, Y., Xu, Y., Zhu, C., & Zeng, M. (2021). Want to reduce labeling cost? GPT-3 can help. arXiv preprint arXiv:2108.13487.\n\n[Kocoń et al., 2023] Kocoń, J., Cichecki, I., Kaszyca, O., Kochanek, M., Szydło, D., Baran, J., ... & Kazienko, P. (2023). ChatGPT: Jack of all trades, master of none. Information Fusion, 101861.\n\n[Zhong et al., 2023] Zhong, Q., Ding, L., Liu, J., Du, B., & Tao, D. (2023). Can chatgpt understand too? a comparative study on chatgpt and fine-tuned bert. arXiv preprint arXiv:2302.10198.\n\n[Ziems et al., 2023] Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2023). Can Large Language Models Transform Computational Social Science?. arXiv preprint arXiv:2305.03514.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed annotation framework demonstrates considerable practical value in alleviating the lack of supervised data across various NLP tasks.\n2. Comprehensive experiments conducted include annotating diverse datasets and fine-tuning a RoBERTa model for evaluation. These thorough experimental designs and results clearly showcase the superiority and cost-effectiveness of this novel approach.\n3. The paper is well-written and well-structured.\n4. The appendix is abundant and provides prompts suitable for various tasks.\n", "reasons_to_reject": "1. Expanding the evaluation to include sequence labeling, NLI, and other diverse NLP tasks beyond just classification-based tasks would better demonstrate the general applicability of the proposed framework across different task types. \n2. The proposed framework is not combined with active learning paradigm. Active learning is a very natural solution that can be used to improve labeling efficiency and reduce labeling costs when it comes to human-AI collaboration. The proposed annotation workflow is theoretically orthogonal to active learning.\n3. The few-shot capability of LLMs is not utilized. It has been proven that LLMs can significantly improve performance given a small number of demonstrations. Perhaps in the framework proposed in this paper, using the human annotated data as demonstrations and inputting them into the LLM could further improve the results.", "questions_to_authors": "1. Could it be considered to add some results of LLMs solving these problems under zero-shot and few-shot settings? It's possible that using ChatGPT itself may achieve better results than fine-tuning RoBERTa with annotated data.\n2. Could evaluations on different LLMs be added? For example, using other LLMs, would there be some differences in cost and annotation quality?", "paper_topic_and_main_contributions": "This paper focuses on data annotation challenges in NLP tasks, and proposes a co-annotation framework for human-AI collaboration during the annotation process. The framework aims to achieve higher annotation quality and lower annotation costs for a given dataset. It does this by automatically deciding whether each data instance should be annotated by a human or by LLM. Experiments using annotated data to fine-tune RoBERTa on multiple benchmark datasets demonstrate that the proposed method achieves Pareto efficient results in balancing cost and annotation quality. \n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The format of Figure 3 and Table 2 can be further optimized."}, "extra_text": {}}, {"review_id": "ehaehHuo2F", "reviewer": "Reviewer_Tt3q", "summary": "", "questions": "A. Can you elaborate on how the Pareto frontier was calculated?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful reviews and appreciation of the thorough analysis of our work.\n\n### Response to ‘Reasons to Reject’: \nThank you for your suggestion to split the work to a long and short paper. We included the Pareto efficiency concept to ensure the completeness of the whole pipeline. In the camera-ready, we will better integrate this section with the first half of the paper.\n\n### Response to ‘Questions for Authors’: \nWe plot the performances of each quality-cost combination and approximate the Pareto frontier by interpolating the discrete data points [Abdolrashidi et al., 2021] [Treviso et al., 2020]. We will add the method and relevant references to the camera-ready to make it clearer.\n\n[Abdolrashidi et al., 2021] Abdolrashidi, A., Wang, L., Agrawal, S., Malmaud, J., Rybakov, O., Leichner, C., & Lew, L. (2021). Pareto-optimal quantized resnet is mostly 4-bit. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3091-3099).\n\n[Treviso et al., 2020] Treviso, M., Góis, A., Fernandes, P., Fonseca, E., & Martins, A. F. (2021). Predicting attention sparsity in transformers. arXiv preprint arXiv:2109.12188.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The topic of using LLMs as annotators is a very timely topic, and this paper does a very thorough job in its analysis. The analysis has enough content as a main contribution. Additionally, the paper is structured nicely and the visualizations are helpful for understanding the content.", "reasons_to_reject": "There is a lot of content in this paper, and it might benefit from being split into a long and short paper. Since the Pareto efficiency work wasn't mentioned until the latter half of the paper, that could perhaps go into a shorter paper geared more towards an industry audience.", "questions_to_authors": "A. Can you elaborate on how the Pareto frontier was calculated?", "paper_topic_and_main_contributions": "This paper proposes a new paradigm for LLM-human collaboration in annotation tasks since that topic has not been fully explored yet. Using model confidence as a proxy for model competence, it claims that this new method can achieve improved resource allocation where costs and quality are well-balanced. Since model confidence is not always reliable, the authors test different prompts and allocation strategies to achieve a more reliable signal. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "For Figure 3, the bracketed numbers are a little distracting. Could you bin the proportion of samples and use different marker shapes to indicate them or use marker size?\n\nFor Table 2, I personally think highlighting the cells would make the pattern easier to see in the table."}}, {"review_id": "eIH9YDRhvv", "reviewer": "Reviewer_HARE", "summary": "", "questions": "1- How do you ensure that the quality of annotations is not compromised while focusing on cost reduction in the CoAnnotating framework?\n2- What is the rationale behind the selection of the work allocation strategies presented in the paper, and can you provide a more detailed comparison of their performance?\n3- How does the CoAnnotating framework perform when considering other LLMs or smaller models? Would the same uncertainty metrics be applicable?\n4- Are there any specific domains or applications where the CoAnnotating framework may be particularly effective?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful reviews and acknowledgement of our work as one which addresses a significant research gap with thorough analysis and interesting insights.\n\n### Response to ‘Reasons to Reject’:\n1) **Challenges in manual annotation:** Some challenges in manual annotation include subjectivity, annotation inconsistency, cost in terms of recruiting and training annotators. We will add these aspects in the camera-ready.\n2) **Too much focus on cost reduction:** In CoAnnotating framework, we did not prioritize one over the other. Instead, we identified both quality and cost as core objectives. As illustrated in Section 3.4 and Figure 4, we set up a multi-objective optimization process which aims to maximize annotation quality while minimizing annotation costs at the same time.\n3) **Rationale in selecting the work allocation strategies:** We talked about the rationales in selecting the work allocation strategies in Section 3.2, which are (1) easy implementation and (2) proven effectiveness in previous literature. We made detailed comparisons for different strategies in Section 5.1, using the subsection talking about relative performance of different working allocations in terms of alignment with human label, annotation quality as well as annotation costs.\n4) **Computational method explanation and discussion of their effectiveness:** We explained the computational method of self-evaluation and entropy calculation by defining mathematical variables and explaining formulae for each of them in Section 3.2. For more mathematical details, we will include them in the appendix. We also discussed their relative effectiveness through visualizations (Figure 3 and Figure 4) and words (Line 405-417, Line 435-441).\n5) **Task selection:** To demonstrate the general capability, we selected tasks from a diversity of domains. We did not opt for sequence tagging tasks, as there exists some difficulty by directly applying the framework. It wouldn’t make sense to split the work at individual tags since both an LLM and a human annotator need to consider the whole sequence to provide the tag. And it might be difficult to compare performance at the level of sentences. If we need a distinct prompt for each tag, the length of the prompt would be variable and thus the cost of inference would be variable.\n6) **Ablation study:** We actually included the ablation study to study the impact of different prompt designs in Section 5.4 and managed to show that the design of different prompts in our framework is effective in improving work allocation. We used the trend in Figure 3 and numerical results in Table 2 to support our claim by comparing performance with the same prompt and different prompts. For different uncertainty metrics, we did not include that in the main section because our main focus is to show that our framework using uncertainty metrics as a guide works better than previous random allocation methods. Comparing different uncertainty metrics is less of a focus here but it will be interesting to explore it in future work.\n\n### Response to ‘Questions for Authors’:\n**Question 1:** As mentioned in Section 5.2 (Line 454-464), it is possible to reduce costs while ensuring human-level performance for more straightforward tasks like topic classification. On the other hand, it is still inevitable to compromise the annotation quality while cutting costs for tasks requiring nuanced comprehension. Practitioners are advised to choose among the Pareto efficient points based on their specific budgets. Therefore, the compromise is sometimes inevitable but the use of Pareto efficiency concept helps practitioners to achieve the highest possible annotation quality for a specific budget. \\\n**Question 2:** We mentioned the rationales behind the selection of the work allocation strategies (1. Easy implementation [Bansiya et al., 1999] 2. Proven effectiveness in previous literature [Diao et al., 2023]) in Section 3.2. We made detailed comparisons in Section 5.1, talking about relative performance of different working allocations in terms of alignment with human label, annotation quality as well as annotation costs. \\\n**Question 3:** Our CoAnnotating framework serves as a general framework that can be applied to any large language model with varied annotation costs and quality. The models may evolve over time but our framework can still hold. Practitioners can follow the same flow including computing the same uncertainty metrics for other models or new models that come out in future. \\\n**Question 4:** Thanks for the thought-provoking question! The CoAnnotating framework can be particularly effective for tasks requiring expert annotation (e.g. finance, legal domain) but under constrained budgets. This is because expert annotations are more costly, which may make having all data annotated by humans impossible if the budget is limited. Our proposed framework can suggest the best allocation pattern using Pareto efficiency frontier and help practitioners achieve the highest possible annotation quality given a constrained budget.\n\n[Bansiya et al., 1999] Bansiya, J., Davis, C., & Etzkorn, L. (1999). An entropy‐based complexity measure for object‐oriented designs. Theory and practice of object systems, 5(2), 111-118.\n\n[Diao et al., 2023] Diao, S., Wang, P., Lin, Y., & Zhang, T. (2023). Active prompting with chain-of-thought for large language models. arXiv preprint arXiv:2302.12246.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors propose a compelling approach to address the challenges in finding a balance between annotation quality, cost, and scalability. The framework is well-motivated and addresses a significant gap in the literature, as limited work has been done to leverage LLMs as complementary annotators and explore the allocation of annotation work between humans and LLMs. The paper should be accepted due to following reasons:\n1- The CoAnnotating framework introduces a new paradigm in the annotation process by utilizing uncertainty metrics to estimate LLMs annotation capability. This enables an effective work allocation between humans and LLMs, which can help practitioners to achieve higher annotation quality and lower annotation costs.\n2- The authors conduct experiments on six classification datasets, covering a wide range of tasks and difficulties. These experiments provide a robust evaluation of the proposed framework and demonstrate its effectiveness in various scenarios.\n3- The authors compare the performance of their proposed framework with several baseline allocation strategies, such as random allocation, self-evaluation guided allocation, and entropy guided allocation. This thorough comparison showcases the advantages of the CoAnnotating approach over existing methods.\n4- The paper offers an in-depth analysis of the results, highlighting the reliability of self-reported confidence scores by LLMs, the sensitivity of LLM responses to prompt variations, and the extent to which human resources can be freed by LLMs. The authors also acknowledge the limitations of their work, providing a foundation for future research directions.\nOverall, the paper is well-written, thoroughly evaluated, and provides valuable insights into the potential of human-LLM collaboration. I believe that the findings presented in this paper will have a considerable impact on the development of future annotation systems and contribute to the ongoing conversation around the role of LLMs in the annotation process. ", "reasons_to_reject": "While the paper proposes an innovative framework for collaborative annotation between humans and large language models, there are several concerns that warrant reconsideration and refinement before the paper can be accepted. I recommend revising the paper to address the following issues:\n1- The paper's motivation and problem statement are not clearly defined. The authors should provide a more detailed background on the challenges faced in manual annotation and the potential benefits of using LLMs as complementary annotators to set the stage for their proposed framework.\n2- The primary focus of the paper is on cost reduction through the collaboration of humans and LLMs. However, the quality of the annotations is equally important in many scenarios. The authors should provide a more balanced discussion of both cost and quality aspects in their framework.\n3- The paper presents several work allocation strategies but its lacking a clear rationale for their selection and a detailed comparison of their performance. The authors should provide a more comprehensive analysis of the allocation strategies and their benefits and drawbacks.\n4- The uncertainty computation methods used in the framework, namely self-evaluation and entropy, are not adequately explained. The authors should provide a more detailed description of these methods and discuss their effectiveness in the context of the proposed framework.\n5- The evaluation of the framework is limited to six classification datasets. To demonstrate the robustness and generalizability of the proposed framework, the authors should consider extending their evaluation to other tasks, such as sequence tagging, and a wider range of datasets.\n6-\tThe authors do not provide sufficient ablation studies to understand the individual contributions of different components of the framework. A deeper analysis of the various aspects of the framework, such as the impact of different prompt designs and uncertainty computation methods, would strengthen the paper.\nOverall, the CoAnnotating framework presents a promising approach to human-LLM collaboration in annotating unstructured text. However, the paper requires significant revisions to address the concerns mentioned above and provide a more comprehensive and well-rounded evaluation. I recommend revising the paper to address these issues.\n", "questions_to_authors": "1- How do you ensure that the quality of annotations is not compromised while focusing on cost reduction in the CoAnnotating framework?\n2- What is the rationale behind the selection of the work allocation strategies presented in the paper, and can you provide a more detailed comparison of their performance?\n3- How does the CoAnnotating framework perform when considering other LLMs or smaller models? Would the same uncertainty metrics be applicable?\n4- Are there any specific domains or applications where the CoAnnotating framework may be particularly effective?", "paper_topic_and_main_contributions": "The paper introduces CoAnnotating, a novel paradigm for human and large language model collaboration in annotating unstructured text at scale. The framework uses uncertainty metrics to estimate LLMs' annotation capability and guide effective work allocation between humans and LLMs. CoAnnotating aims to achieve higher annotation quality and lower annotation cost for a given dataset. The authors use diverse prompts to compute uncertainty and explore various work allocation strategies, such as random allocation, self-evaluation guided allocation, and entropy guided allocation. Experiments were conducted on six classification datasets, and results demonstrate the effectiveness of the CoAnnotating framework in achieving greater cost efficiency. The study also provides insights into the reliability of self-reported confidence scores by LLMs and the sensitivity of LLM responses to prompt variations. While the framework focuses on human-level performance and classification tasks in English, the concept is generalizable to generation tasks and datasets in other languages, offering potential for future research.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "There is typo in page 4 Section 3.3 first line: \"Building upon the aforementioned uncertainty level estimation, we can then such uncertainty level ui to guide the work allocation.\" Replace \"such\" with perhaps \"use\"."}}]

Author rebuttal:

Meta-review: This paper proposes a framework for annotating data using both humans and LLMs. The core idea is to use the – cheaper – LLM to generate an annotation for examples where it has high certainty, and to use a human annotator on examples where the LLM has low certainty. 

The paper discusses several different approaches to measuring uncertainty, and carry out experiments with each. Testing on three different tasks, the authors demonstrate that their approaches result in much stronger model performance after training on the produced data, when compared to a random allocation of examples between humans and LLMs. 

Beyond implications for dataset construction, this paper is also interesting purely for the proposals on how to assess LLM uncertainty.
Decision: Accept-Main