ARAIDA: Analogical Reasoning-Augmented Interactive Data Annotation (2024)

Abstract: Human annotation is a time-consuming task that requires a significant amount of effort. To address this issue, interactive data annotation utilizes an annotation model to provide suggestions for humans to approve or correct. However, annotation models trained with limited labeled data are prone to generating incorrect suggestions, leading to extra human correction effort. To tackle this challenge, we propose ARAIDA, an analogical reasoning-based approach that enhances automatic annotation accuracy in the interactive data annotation setting and reduces the need for human corrections. ARAIDA involves an error-aware integration strategy that dynamically coordinates an annotation model and a k-nearest neighbors (KNN) model, giving more importance to KNN’s predictions when predictions from the annotation model are deemed inaccurate. Empirical studies demonstrate that ARAIDA is adaptable to different annotation tasks and models. On average, it reduces human correction labor by 11.02% compared to vanilla interactive data annotation methods.

heading: 1 Introduction
text: Data annotation is a challenging task that involves a tradeoff between annotation quality and budget. While some platforms offer a cost-effective solution by relying on ML models to annotate data automatically 1, the quality of such annotations is often compromised (Wang et al., 2022a). It is particularly true in the limited data annotation scenario where the annotation budget is limited or when unlabeled data are scarce (Ringger et al., 2007; Chaudhary et al., 2021; Huang et al., 2024).
Human-machine interactive annotation methods were introduced to reduce annotation effort while maintaining annotation quality (Klie et al.,
*Correspondence to Wenqiang Lei. 1For example, https://aws.amazon.com/sagemaker/
groundtruth/.
2018, 2020; Le et al., 2021). As illustrated in Fig. 1, these methods introduce an annotation model to suggest labels (model annotations) to human annotators. The annotators accept a suggested label if it is correct. Otherwise, they have to correct the label manually. Compared to manual annotation, interactive annotation requires less human effort because human annotators only have to verify the model annotations instead of coming up with an answer from scratch, leading to potential speedup of the annotation process (Klie et al., 2020).
Evidently, the annotation model’s accuracy is crucial because incorrect suggestions require additional human effort to rectify. Existing methods update the annotation model based on previously accepted or corrected data (ground-truth annotation), aiming to reduce human corrections by improving prediction accuracy at each iteration (Klie et al., 2020; Wu et al., 2022). However, in the context of limited data annotation, the annotation model lacks sufficient labeled training data to reach a reasonable accuracy and is prone to providing incorrect suggestions (Rietz and Maedche, 2021). For example, in the span relation annotation example shown in Fig. 2 (blue), the annotation model continues to make mistakes on similar examples ([car, tyre]) even after the human annotator corrects the label ‘[tree, leaf ]=>component’. As a result, this leads to more human corrections. Such a problem is crucial for interactive annotation and has been identified by recent work (Rietz and Maedche, 2021),
10660
but it has yet to be addressed. Inspired by cognitive studies on efficient learning (Lake et al., 2017, 2015; Mitchell, 2021), finding that the human brain can learn from a few examples because our brain is continuously building analogies during the learning process of concepts to facilitate comprehension, we propose Analogical Reasoning-Augmented Interactive Data Annotation (ARAIDA), which is designed to improve interactive annotation under the limited data annotation setting. ARAIDA provides an annotation reference to the annotation model by retrieving previously human-labeled examples in the proximity of the example in consideration using the k-nearest neighbors (KNN) method. As illustrated in Fig. 2(red), the final suggestion combines the model annotation and the annotation reference provided by KNN via an error-aware integration strategy. This strategy dynamically coordinates the annotation model and KNN, giving more importance to KNN’s prediction if the predicted label from the annotation model is estimated to be inaccurate.
We conduct simulated experiments for the limited data annotation task and estimate the human annotation effort based on the number of human corrections (or the number of suggestion errors) following Hwa (2000) and Kristjansson et al. (2004). We test ARAIDA on different wordlevel and sentence-level annotation tasks, combining with different annotation models (i.e., classic and LLM-based models). The result shows that ARAIDA consistently improves different annotation models’ accuracy across various tasks. On average, it reduces human corrections by 11.02%. Further analysis attributes this improvement to the few-shot capability of the KNN module and the error-aware integration strategy that effectively synergizes complementary annotations. In summary,
our contributions are as follows:
• Calling attention to the limited data annotation scenario. We highlight the under-trained problem of the annotation model, which is crucial in practice but overlooked in interactive annotation.
• Introducing ARAIDA that involves a KNN module and an error-aware integration strategy to alleviate the under-trained problem by facilitating coordination between the two model annotators (i.e., the vanilla annotation model and KNN).
• Demonstrating the efficacy of ARAIDA in enhancing suggestion accuracy, reducing human corrections, and showcasing its flexibility to combine with various annotation models through extensive experiments.

heading: 2 Related Work
text: Our research is tied to interactive data annotation, human analogical reasoning (KNN), and retrievalbased language models. We provide a literature review and highlight our differences.
Interactive Data Annotation. Interactive data annotation aims to reduce human annotation effort by incorporating an annotation model that suggests labels to human annotators during an interactive process (Klie et al., 2018, 2020; Le et al., 2021). The annotation model must be sample-efficient because, when we start a new annotation task, there are few labeled examples to learn from. Current studies focus on employing active learning (Klie et al., 2018; Laws et al., 2011; Casanova et al., 2020; Li et al., 2021; Huang et al., 2023) to prioritize annotating examples more likely to improve model accuracy. While active learning can reduce the required training data to some extent, it may not be effective in limited data annotation scenarios
or when complex hypotheses or semantics are to be learned (Dasgupta, 2005; Rietz and Maedche, 2021). Another approach is to employ LLMs for automatic data annotation, which have demonstrated strong performance under zero-shot and few-shot settings (He et al., 2023; Gilardi et al., 2023). However, such performance might not be consistent for difficult tasks, as they may even perform worse than fine-tuned small language models (Xiao et al., 2023). Regardless of whether we use active learning and whether we use a classic or LLM-based annotation model, our empirical evidence demonstrates that ARAIDA can effectively decrease the amount of human corrections required.
KNN and Analogical Reasoning. While KNN has been extensively utilized in NLP community (Wang et al., 2019; Liu et al., 2023; Wang et al., 2022b), its underlying mechanism is often overlooked. To shed light on this, cognitive studies (Lake et al., 2017, 2015; Mitchell, 2021) revealed that the KNN inference process aligns with human analogical reasoning, enabling efficient learning (Lake et al., 2017, 2015). In particular, analogical reasoning establishes connections between relevant aspects of the current task and past experiences, forming abstractions that enhance human reasoning capabilities (Mitchell, 2021). In this context, KNN facilitates sample-efficient learning by leveraging similarities between the example to be labeled and previously annotated examples, resulting in exemplary solutions (Bautista et al., 2016), which reduce the training data requirement.
Retrieval-Based Language Models. There is growing interest in enhancing the output of language models by incorporating a retrieval module (usually KNN or alike) that interpolates with a datastore built from the training data (Khandelwal et al., 2019; Kassner and Schütze, 2020). Compared to vanilla language models, retrieval-based models ground the predictions in labeled training examples, potentially yielding better explainability and sample efficiency (Asai et al., 2023). This approach has shown promising results in tasks such as machine translation (Khandelwal et al., 2021; Liu et al., 2023), named entity recognition (Wang et al., 2022b), and question answering (Kassner and Schütze, 2020). While some studies have explored the use of dynamically adjusted combination weights between the language model and the retrieval module (Wang et al., 2021; Zheng et al., 2021; Jiang et al., 2021), our method differs signifi-
cantly for two main reasons: 1) Different tasks. We are the pioneers in introducing KNN to the interactive data annotation task, whereas these methods are primarily designed for machine translation. 2) Different techniques. We adjust the weight by estimating the error of model predictions for each data point (e.g., sentence), whereas these methods learn the weight for each token without error estimation.

heading: 3 ARAIDA: The Proposed Method
text: We present ARAIDA, an analogical reasoningbased method for interactive data annotation that provides an annotation reference to the annotation model by retrieving previously human-labeled examples in the proximity of the example in consideration. We detail the KNN inference module in Section 3.1 and the error-aware integration strategy in Section 3.2. Finally, the optimization details are provided in Section 3.3.
Task Formalization and Overview. Let X denote the dataset that needs to be annotated, with C being the number of classes. Given a data batch xt at time t, the annotation model ft predicts label vectors ft(xt) ∈ R|xt|×C , and the KNN module gt infers label vectors gt(xt) ∈ R|xt|×C using previously annotated data stored in a datastore At. Then, we estimate the probability λt ∈ R|xt|×1 of the annotation model’s predictions ft(xt) being reliable, i.e., argmax(ft(xt)) = yt, where argmax(·) returns indices of the classes with the highest predicted probability and yt ∈ R|xt|×1 are the ground truth labels. Finally, we use λt to weigh the two predictions ft(xt) and gt(xt) through a linear weighted combination:
Ft(xt) = λt · ft(xt) + (1− λt) · gt(xt). (1)
Notably, closed-source language models such as ChatGPT produce discrete labels rather than predicted distributions. Therefore, we cannot combine its predictions with KNN’s using linear combination. In such case, we use binary values (0 or 1) for λt, which acts as a function allocation to determine whether ft or gt should apply to each example. Once the human approves or corrects the final suggestions, the datastore At is updated with the newly arrived data batch and its corresponding labels. In addition, the annotation model ft (if applicable), KNN module gt, and weighting strategy λt are updated via back-propagation.

heading: 3.1 KNN Inference
text: We utilize a weighted KNN to perform inference, defined as gt(xit) = ∑
a∈ρi waya∑ a∈ρi wa , where ρi ∈ At is the k nearest neighbors of each example xit, ya corresponds to the human annotation of each neighbor a ∈ ρi. The similarity between xit and a is measured by wa = 1d(xit,a) , where d(xit, a) = ∥wknn(xit − a)∥2 is a distance metric parameterized by wknn. We use the similarity measure to retrieve ρi. To avoid overconfidence in KNN inference, we apply label smoothing to the labels of the retrieved neighbors. Specifically, we set ya = ya(1− α) + α/C, where α = 1− 1C . Datastore Maintenance Strategy. The datastore At consisting of historically annotated data grows in size as the interactive annotation continues, causing the KNN’s retrieval to be less time-efficient. To address this issue, we impose a constraint on the maximum datastore size using a pre-defined hyperparameter. We propose a class-aware maintenance strategy. Precisely, if At exceeds its budget, data that is from the majority class 2 and most similar to its class prototype3 is discarded first. This strategy ensures that the datastore contains as many labeled data from different classes as possible while minimizing the impact on the class prototype. Appendix A.1 and A.3 present experiments using different datastore sizes and maintenance strategies.

heading: 3.2 Error-aware Integration Strategy
text: Motivation. A popular method to combine annotations from two models (i.e., annotation model and KNN) is to use a weighted linear combination with a constant weight 4 (Liu et al., 2023; Wang et al., 2022b). However, assuming that one model consistently outperforms the other on all unlabeled data is unrealistic. Furthermore, both models are updated with each new batch of data in interactive data annotation, and their relative performance will alter, making it infeasible to find the optimal weight through a one-off hyperparameter tuning. To address this issue, we propose an error-aware integration strategy that automatically assigns weights to different models, relying more on KNN inference when the annotation model’s prediction is estimated to be inaccurate.
2The majority class refers to the class with the highest frequency in At.
3The class prototypes are the average of the feature vectors in At that belong to each class.
4Equivalent to when λt(xt) in Eq.1 is a constant.
Error Estimation of Model Annotation. We base on the intuition that if the model ft consistently makes mistakes on previous examples similar to the current data point xit, then its prediction ft(x i t) will likely be incorrect. To achieve this, we parameterize the integration strategy λt as a neural network to learn from the customized input.
• Customized Inputs. For each data point xit, we derive the input xit to the integration strategy λt, which considers the local error estimation Eit and local density Dit. Specifically, E i t is a vector
with elements eit,j = 1[argmax(ft(aj)) = yaj ] indicates if the annotation model ft predicted correctly on each annotated example aj ∈ ρi in the k nearest neighbors of xit. The local density Dit is a distance vector, with each element being d(xit, aj). These two vectors are combined using the element-wise multiplication operator ⊙ to create the input: xit = D i t ⊙Eit −Dit ⊙ (1−Eit). Notably, xit measures the error regularity of x i t among its neighborhood, as the more positive values in the vector xit, the less likely ft would make an error on xit. • Learning Objectives. We collect the ground truth labels yt through human feedback. To optimize the error-aware integration strategy λt, we use a mean squared error (MSE) loss, denoted as ℓtd(yt, ft(xt), λt) = MSE(1[argmax(ft(xt)) = yt], λt(xt)), where 1[·] indicates whether the ground truth labels yt are the same as the predictions ft(xt). The purpose of this loss function is to guide λt by encouraging it to predict errors made by ft.

heading: 3.3 Optimization of ARAIDA
text: To simplify the optimization process, we independently optimize the annotation model ft, KNN model gt, and error-aware integration strategy λt. We treat human feedback yt as the ground truth following previous studies on interactive annotation (Klie et al., 2018, 2020; Le et al., 2021). It is worth noting that ARAIDA supports any taskspecific annotation model and uses its corresponding loss function ℓf to update the parameters. Combining the ℓd loss and the negative log-likelihood loss ℓg to optimize KNN, we formulate the final loss function as follows:
L(f, g, λ) = Bt∑
i=1
ℓf (yi, ft(xi)) + ℓg(yi, gt(xi))
+ ℓd(yi, ft(xi), λt),
(2)
where Bt represents the total data accumulated until round t. There are two challenges to optimizing this objective function. Firstly, the operator used in KNN to retrieve the k nearest neighbors is not differentiable. To address this problem, we utilize the Gumbel-softmax-based reparameterization trick (Jang et al., 2016) to facilitate the optimization process. Secondly, the loss function L presents a bi-level optimization problem, where the optimization of λt is nested within the optimization problems of ft and gt. As a result, we update ft, gt, and λt iteratively using a coordinate-descent approach. Formally, at each optimization iteration k, we have network parameters θkf , θ k g , and θ k λ corresponding to f
k, gk, and λk. The update procedures are as follows:
θk+1f = θ k f −▽fL(f, gk, λk), θk+1g = θ k g −▽gL(fk, g, λk), θk+1λ = θ k λ −▽λL(fk+1, gk+1, λ).
(3)

heading: 4 Experiments
text: We conduct extensive experiments to assess ARAIDA’s effectiveness in the limited data annotation scenario. Our primary focus is to assess whether ARAIDA can decrease the human effort required for corrections by providing more precise annotations at various stages of the annotation process (see Section 4.2). Furthermore, we perform a comprehensive examination to investigate the behavior and impact of KNN and the error-aware integration strategy (see Section 4.3). Additional analysis of our error-aware integration strategy is presented in Section 4.4. Lastly, we analyze the sensitivity of the parameters in Appendices A.

heading: 4.1 Experimental Setup
text: Tasks & Datasets. We experiment with wordand sentence-level annotation tasks. These tasks have been highlighted as crucial in various web applications (Yao et al., 2021; Marcos-Pablos and García-Peñalvo, 2020; Lee et al., 2022). To simulate the scenario of limited data annotation, we follow Dou et al. (2019) by imposing dataset size restrictions, ranging from 1K to 5K. Table 1 overviews the dataset statistics.
• Word-level annotation. We focus on the knowledge graph completion task, which annotates the semantic classes of input word pairs (e.g., ‘[tree, leaf ]=>component’). We use two benchmark knowledge graph datasets in our experi-
ments, namely the WN18RR (Dettmers et al., 2018)5 and Freebase (Bollacker et al., 2008)6 dataset. We experiment with the eight most frequent classes for each dataset.
• Sentence-level annotation. We consider the sentiment classification task and experiment on two benchmark datasets, including SST-5 (Socher et al., 2013)7, and IMDB (Maas et al., 2011)8. SST-5 dataset contains categories on a scale of 1-5 while IMDB contains two categories (positive/negative).
Evaluation Metric. We aim to minimize the total human corrections (i.e., the total model suggestion errors) annotating a given amount of data using the interactive annotation process. Therefore, we report the Machine Cumulative Accuracy (MCA), defined as the total correct suggestions divided by the total suggestions for different dataset sizes. To assess the performance of each method in the limited data annotation scenario, we present the mean and the corresponding standard deviation by varying the dataset size ({1K, 2K, 3K, 4K, 5K}).
Annotation Models. To verify the generalizability of ARAIDA, we apply it in conjunction with different annotation models:
• Classic annotation models. We utilize lightweight annotation models following
5https://paperswithcode.com/dataset/wn18rr 6https://www.microsoft.com/en-us/download/
details.aspx?id=52312 7https://nlp.stanford.edu/sentiment/code.html 8https://www.kaggle.com/ datasets/lakshmi25npathi/ imdb-dataset-of-50k-movie-reviews
previous works (Desmond et al., 2021; Chen et al., 2020; Hedderich et al., 2021). Specifically, for word-level tasks, we use a distributional model (Roller et al., 2014; Kober et al., 2021) with pretrained GloVe word embeddings embedding (Pennington et al., 2014)9. For sentence-level tasks, we use FastText (Joulin et al., 2017) to derive the sentence embeddings. We denote this baseline as Dist./FT.
• LLM-based annotators. We also use large language models (LLMs) as annotation models, whose few-shot and in-context learning capabilities might help with the limited data annotation process. We experiment with LLaMa27B (Touvron et al., 2023) and ChatGPT (Ouyang et al., 2022) 10. We use zero-shot and few-shot prompts for ChatGPT (denoted as ChatGPTzero and ChatGPTfew). Detailed prompts can be found in Table 7. For LLaMa2, we consider both the vanilla LLaMa2 that uses the same zero-shot prompts as ChatGPTzero and LLaMa2sft, which is fine-tuned using an open-source toolkit11 during the interactive annotation process. Finetuning data examples can be found in Table 8.
Impact of Active Learning. Regardless of the annotation model, the sequence of the data to annotate also affects the amount of human corrections. Intuitively, if we show unambiguous examples first, few corrections are needed. However, the annotation model and KNN may not learn to handle more challenging examples and may subsequently
9We did not use contextualized embeddings because the word-level task in our experiment has no context.
10The gpt-3.5-turbo checkpoint in OpenAI’s API (https://platform.openai.com/docs/models/ gpt-3-5).
11https://github.com/Alpha-VLLM/ LLaMA2-Accessory
make more mistakes. Previous studies focused on applying active learning in interactive annotation to enhance the annotation models’ sample efficiency (Laws et al., 2011; Klie et al., 2018; Li et al., 2021). To study the impact of active learning in the limited data annotation scenario, we compare an uncertainty-based active learning method with random data ordering for different annotation models with and without ARAIDA. Note that we omit the ChatGPT with AL results because we are unable to estimate its prediction uncertainty accurately.
Implementation Details. All experiments are carried out on a machine with Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz and a GeForce RTX 3090 GPU. For simplicity, we implement our integration strategy using a three-layer, fully-connected network with ReLu activation and dropout. For KNN, we set k = 20 for ρt. KNN runs in the embedding space of text-embedding-ada-002 (Neelakantan et al., 2022) when combining with ChatGPT. For Dist./FT and LLaMa2 models, KNN runs in the corresponding model’s embedding space. Moreover, we leave the details of LLM prompts and mode fine-tuning examples in Appendix D.

heading: 4.2 Main Result
text: We evaluate the effectiveness of ARAIDA in enhancing the model annotation quality, hence reducing human correction effort. Table 2 and Figure 3 show the machine cumulative accuracy scores averaged across varying amounts of data ({1K, 2K, 3K, 4K, 5K}) for different experimental settings. We make the following observations:
ARAIDA reduces human corrections consistently. As illustrated in Table 2 and Figure 3, ARAIDA consistently improves the model suggestion accuracy across different annotation models and annotation tasks. Specifically, it achieves a
99.62% 47.52% 52.48% 27.45% 72.55% Action Sensitivity Consistent Recommendation Inconsistent Recommendation Diversity (inconsistent but accurate) Sensitivity (inconsistent and inaccurate)
99.76%
44.36%
55.64%
12.97%
87.03%
(a) Redial
(b) OpendialKG
99.62% 47.52% 52.48% 27.45% 72.55% (a) Redial 99.76% 44.36% 55.64% 12.97% 87.03% (b) OpendialKG
Action Sensitivity Consistent Recommendation Inconsistent Recommendation Diversity (inconsistent but accurate) Sensitivity (inconsistent and inaccurate)
Action Sensitivity Consistent RecommendationInconsistent Recommendation Diversity (inconsistent but accurate)Sensitivity (inconsistent and inaccurate)
47 .6
2
38 .8
5 9 2.
44
41 .35
0. 42
43 .1
1 93 .5
2
48 .4
1
51 .3
51 .6
94 .7
8
53 .8
5
52 .5
3
52 .2
6
94 .9
2
55 .1
2 35 45 55 65 75 85 95
WN18RR FreeBase IMDB SST-5
Word-level Annotation Sentence-level Annotation
ChatGPTzero ARAIDA+ChatGPTzero ChatGPTfew ARAIDA+ChatGPTfew hat PTzero ChatGPTzero + ARAIDA t f ChatGPTfew + ARAIDA
Figure 3: MCA scores using ChatGPT-based methods. We omit the ChatGPT with AL results because we are unable to estimate its prediction uncertainty. ARAIDA can further improve ChatGPT’s performance
13.06% performance gain in MCA on Dist./FT, 3.85% on LLaMa2sft, 16.80% on Dist./FT with AL, 2.61% on LLaMa2sft with AL, 30.48% on LLaMa2, 8.81% on ChatGPTzero, and 1.55% on ChatGPTfew. On average, it achieves an 11.02% performance gain in model annotation accuracy, translating into a reduction in human correction effort in practice.
Active learning does not always help. When comparing the annotation models with and without active learning, we observe that active learning does not always improve model performance and can even harm the performance in some instances (e.g., Dist./FT). We hypothesize that the more challenging cases selected by active learning might require more training data for the models to correct their predictions (Dasgupta, 2005; Rietz and Maedche, 2021), which are unavailable in limited data annotation scenarios. Instead of relying on an annotation model alone, ARAIDA acts as a posthoc “plug-in” that fixes the annotation model’s mistakes using retrieved annotation references and yields a robust improvement under various settings with and without active learning.
LLMs are strong annotation models. ARAIDA can improve them further. LLMs perform better than classic distributional models, especially for sentence-level tasks. Furthermore, LLaMa2 with fine-tuning consistently outperforms the vanilla LLaMa2 model, and few-shot ChatGPT consistently beats its zero-shot counterpart. Interestingly, comparing these two pairs of annotation models, we observe that ARAIDA brings a more substantial improvement to weaker models. When the annotation models are already strong (in the case of LLaMa2sft and ChatGPTfew), ARAIDA is
more conservative in making corrections, yielding a smaller but consistent improvement. It demonstrates ARAIDA’s robustness to combine with annotations models with different performances.
Annotation Model Word-level Annotation Sentence-level Annotation WN18RR FreeBase IMDB SST-5 Dist./FT ARAIDA 52.16±1.37 43.02±6.43 79.33±2.81 37.21±3.03 - w/o KNN 50.44±1.02 32.47±12.37 70.18±8.43 36.02±3.14 - w/o f(·) 50.28±3.01 41.52±5.87 78.48±0.68 35.02±1.37 - w/ const. 51.85±4.77 41.78±6.78 78.58±3.94 37.07±2.68 LLaMa2sft ARAIDA 60.74±2.33 55.23±1.46 95.15±9.32 49.62±5.98 - w/o KNN 58.24±2.79 53.11±1.38 94.06±9.02 46.84±6.30 - w/o f(·) 45.23±2.58 39.82±3.41 89.13±0.55 42.93±1.71 - w/ const. 58.24±2.79 53.11±1.38 94.06±9.02 46.84±6.30 ChatGPTzero ARAIDA 50.42±1.37 43.11±1.83 93.52±1.24 48.41±1.06 - w/o KNN 47.62±2.89 38.85±2.23 92.44±0.95 41.30±1.72 - w/o f(·) 48.73±2.64 41.30±5.04 90.61±0.26 45.84±1.12 - w/ const. 48.73±2.64 41.30±5.04 92.44±0.95 45.84±1.12
ChatGPTfew ARAIDA 52.53±1.67 52.26±3.44 94.92±1.02 55.12±1.36 - w/o KNN 51.30±1.72 51.60±3.15 94.78±0.99 53.85±2.03 - w/o f(·) 48.73±2.64 41.30±5.04 90.61±0.26 45.84±1.12 - w/ const. 51.30±1.72 51.60±3.15 94.78±0.99 53.85±2.03
Table 3: Ablation study on the KNN and the error-aware integration strategy modules. We report the MCA scores using various methods, averaging results with different amounts of data. Error-aware integration strategy effectively coordinates the two annotators.

heading: 4.3 Ablation Study
text: This section aims to perform a comprehensive examination to investigate the behavior and impact of the KNN and out integration strategy. We conduct an ablation study to analyze the effectiveness of each component. In particular, we consider the following baselines:
• ARAIDA w/o KNN: Using the annotation model alone to suggest labels. Equivalent to λt(·) = 1.
• ARAIDA w/o f(·): Using KNN alone to suggest labels. Equivalent to λt(·) = 0.
• ARAIDA w/ const.: Using a constant λ∗t value for all examples, in contrast to ARAIDA, which varies λt for different examples. In our experiments, we report the result with the best λ∗t tuned on the validation set.
The ablation test results are presented in Table 3. Due to limited space, we omit the result for vanilla LLaMa2, which is much weaker than other LLMbased baselines. The detailed observations are provided below.
KNN is a strong stand-alone annotator. Table 3 reveals that although KNN (ARAIDA w/o f(·))
does not match the performance of ARAIDA, it obtains comparable accuracy as using the annotation model alone (ARAIDA w/o KNN). This result is surprising, given KNN’s simplicity compared to the annotation models (including LLMs). Such the results also highlight that significance and effectiveness of analogical reasoning, allowing humans to reason more effectively (Mitchell, 2021).
Error-aware integration strategy effectively coordinates the two annotators. Table 3 shows that ARAIDA’s error-aware integration strategy achieves consistent performance gain compared to using a constant weight λ∗t . This in turn also confirms the original intention behind our construction of the error-aware integration strategy: it is infeasible to find the optimal weight through a one-off hyperparameter tuning. Here, it is worth noting that ChatGPT outputs discrete labels rather than probabilistic vectors. In this case, the constant strategy reduces into an indicator function: if λ∗t > 0.5, Ft(x) = ft(x), ∀xt; otherwise, Ft(x) = gt(x),∀xt. In this case, ARAIDA w/ const. prefers ChatGPT or KNN based on their performance. However, it cannot improve the annotation quality beyond the individual components (i.e., the KNN and the annotation model) due to the discrete output of ChatGPT.

heading: 4.4 Qualitative Analysis
text: To shed light on how the error-aware integration strategy works, we measure the cumulative accuracy of the annotator model and KNN throughout the annotation process for each dataset and present the result with the Dist./FT annotation model in Figure 4. We also separate the cases where λ > 0.5 (higher weights assigned to the annotation model f(·)) and λ ≤ 0.5 (higher weights assigned to KNN). Our observations are as follows.
Firstly, we observe that there is a substantial gap between the two solid lines (MCAf on λ > 0.5
and MCAf on λ ≤ 0.5), showing that our error estimation model effectively identifies cases where f(·) is likely to error and assigns it a lower weight. Secondly, KNN reaches a reasonable accuracy much faster than the annotation model at the initial stage of the interactive annotation process. Even as the number of annotated examples increases, its accuracy is still higher than the annotation model f(·) when λ ≤ 0.5. This result reveals that KNN compensates for the performance deficiencies of f(·) on data where it is more likely to make a mistake. Therefore, combining KNN using the error-aware integration strategy in ARAIDA leads to an overall improvement in annotation quality, hence reducing human correction effort.

heading: 5 Conclusion
text: In interactive data annotation, an annotation model suggests labels to human annotators to verify. However, the annotation model is prone to errors when trained on limited labeled data. To tackle this challenge, we proposed ARAIDA, an approach inspired by analogical reasoning, to compensate for the performance deficiencies of the annotation model and correct its mistakes using an error-aware integration strategy. Extensive experiments demonstrated that ARAIDA is flexible to combine with different annotation models across various tasks and yields consistent improvement in label suggestion accuracy, which leads to a reduction of human correction effort. In this study, our method explores a new solution to bring more flexibility by allowing the human to design any preferred annotation model according to different annotation tasks. We are devoted to optimizing human-machine utilities by emphasizing the learning of task-specified concepts efficiently from a few human demonstrations. In future work, we plan to extend ARAIDA to other annotation tasks and develop it as a general toolkit that can benefit the NLP community.

heading: 6 Limitations
text: Human Studies. This work aims to reduce human correction effort in interactive data annotation. We follow previous work (Hwa, 2000; Kristjansson et al., 2004) to use the number of model suggestion errors to approximate the human correction effort needed. However, the actual effort needed depends on the particular example and the type of errors (e.g., whether it is obvious). Ideally, we would involve human annotators and measure the saving of annotation time. However, due to the large number of experimental settings, conducting human studies with each annotation model and ablation baseline was infeasible.
Error-Prone Human Annotation. This paper treats human annotations as ground truth following previous studies in interactive data annotation (Klie et al., 2018; Le et al., 2021). However, uncertainty and inconsistency of human annotations do occur. We refer readers to the literature on handling errorprone human annotation, such as crowd-sourced data annotation (Larson et al., 2020).
Although human annotation errors are not the focus of this work, we explore ARAIDA’s performance under synthesized label noise conditions. We consider the crowd-sourced data annotation scenario and assume that each human annotator hi makes mistakes with the latent probability pie ∼ (0, 0.3). We set the total number of annotators O = 10 and sample their corresponding error probabilities Pe = {p1e, p2e, ..., pOe }. Then, we sample ui from a uniform distribution U(0, 1) for each annotation. If ui ≤ pie, hi assigns a randomly sampled incorrect label; otherwise, it assigns the correct one. We use majority voting of the 10 annotators to obtain the final annotations following Shirani et al. (2019).
We slightly modified ARAIDA’s datastore maintenance strategy. When the datastore exceeds its budge, we discard the data from the majority class and most dis-similar to its class prototype instead of removing the most similar one (as in the original ARAIDA strategy). This strategy may help remove incorrectly labeled data. The experimental result in Figure 5 shows that ARAIDA still outperforms the baseline. The modified datastore maintenance strategy (ARAIDA-dis) further improves the performance by a slight margin. Further research and more rigorous experiments are required to address the human annotation noise problem in interactive annotation.
0% 10% 20% 30% 40% 50% 60%
1K 2K 3K 4K 5K
(AL) ARIA (AL) Baseline (PL) ARIA (PL) Baseline
50% 55% 60% 65% 70% 75% 80% 85%
1K 2K 3K 4K 5K
(AL) ARIA (AL) Baseline (PL) ARIA (PL) Baseline
19%
24%
29%
34%
39%
1K 2K 3K 4K 5K
0%
10%
20%
30%
40%
50%
1K 2K 3K 4K 5K
RIA + Dist. w/ AL Dist. w/ AL ARIA + Dist. w/ PL Dist. w/ PL
RIA + FastText w/ AL FastText w/ AL ARIA + FastText w/ PL FastText w/ PL
18%
20%
22%
24%
26%
28%
1 6 11 16 21 26 31 36
(AL) Baseline (AL) ARIA (AL) ARIA-dis
20%
25%
30%
35%
40%
1 6 11 16 21 26 31 36
Dist./FT ARAIDA ARAIDA-dis
1K 2K 3K 4K 5K
1K 2K 3K 4K 5K
Figure 5: MAC scores of various methods with synthesized label noise on the SST-5 dataset. Dist./FT is used as the annotation model. ARAIDA-dis refers to ARAIDA with a modified datastore maintenance strategy.
Latency. The KNN component requires retrieving similar examples as the input data, which may limit our method’s time efficiency when the datastore size is large. To address this problem, besides the proposed datastore management strategy, we can also employ an efficient similarity search library such as FAISS 12 to speed up the retrieval.

heading: Acknowledgements
text: This work was supported in part by the National Natural Science Foundation of China (No. 62272330); in part by the Fundamental Research Funds for the Central Universities (No. YJ202219).

heading: A Hyperparameter Analysis
text: In this section, we provide a detailed hyperparameter analysis of ARAIDA, including the datastore size At, the number of neighbors k for ρt, and the datastore maintenance strategy. We show results only for the Dist./FT annotation model for brevity.
A.1 Datastore Size
We tune the size of the datastore At in {100, 500, 1000, 2000} and evaluate the machine cumulative accuracy of ARAIDA with or without active learning. We illustrate the results in Fig.6. A larger datastore size generally brings higher annotation performance since it allows us to maintain more data from past human-machine interactions. However, it also requires larger memory usage and causes longer latency. We found that a datastore size of 1000 is a reasonable tradeoff, which we utilize in our main experiments.
A.2 Top k for ρt We tune the number of neighbors At in {5, 10, 20, 50} and evaluate the machine cumulative accuracy of ARAIDA with or without active learning. As shown in Fig.7, k = 20 seems to perform well for all tasks.
A.3 Datastore Maintenance Strategy
We propose a class-aware datastore maintenance strategy for ARAIDA, which removes labeled examples from the majority class most similar to their class prototype. Compared to the conventional First-In-First-Out (FIFO) strategy (Dekel et al., 2005), our method ensures that 1) the datastore contains as much data from different classes as possible, and 2) the class prototype is least affected after the removal. We compare with two variants of ARAIDA, including ARAIDA w/ FIFO and ARAIDA w/ class FIFO. The former discards the oldest example regardless of the class; the latter discards the oldest example belonging to the majority class in the datastore.
As shown in Fig.8, ARAIDA w/ FIFO can suffer from a sudden decrease in performance, as it may remove important examples arbitrarily. After integrating the class information, ARAIDA w/ class FIFO removes the oldest analogy from the majority class, achieving a comparative performance to ARAIDA. However, ARAIDA still performs better when the number of classes increases (e.g., FreeBase).

heading: B Results on Comparing to Fully Fine-tuned Model
text: We utilized up to 5K of data to simulate the limited data annotation task, and we updated the parameters of the annotation model as the interactive annotation process progressed. To validate the effectiveness of the fully fine-tuned model, we use the remaining data from the original dataset, excluding the 5K data and the validation data, as the training data to fully fine-tune the model. Next, we employ this model for interactive annotation. Since it is already fully fine-tuned, we do not update the model parameters during the annotation process. Due to time constraints, we are currently only considering scenarios where active learning has not been adopted. Additionally, we only fine-tune small annotation models (i.e., BERT and Dist./FT).
Based on the results in the Table 5, it is evident that the annotation performance of the fully finetuned model surpasses that of our method. This suggests that while our method currently offers a lightweight approach to aid data annotation, which can to some extent enhance the sample efficiency of the annotation model, there is still potential for improvement in future research.Nevertheless, it’s important to emphasize that in real-world data an-
21%
26%
31%
36% 41%
1 4 7 10 13 16 19 22 25 28 31 34 37
SST
50%
60%
70%
80% 90%
1 4 7 10 13 16 19 22 25 28 31 34 37
IMDB
0%
10%
20%
30%
40%
50%
1 4 7 10 13 16 19 22 25 28 31 34 37
FreeBase
0% 10% 20% 30% 40% 50% 60%
1 4 7 10131619222528313437
WN
ARAIDA-100 ARAIDA-500 ARAIDA-1000 ARAIDA-2000
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K
21%
26%
31%
36% 41%
1 4 7 10 13 16 19 22 25 28 31 34 37
SST with AL
1K 2K 3K 4K 5K 50%
55%
60%
65%
70%
75% 80%
1 5 9 13172125293337
IMDB with AL
1K 2K 3K 4K 5K
0%
10%
20%
30%
40%
50%
1 4 7 10 13 16 19 22 25 28 31 34 37
FreeBase-AL
1K 2K 3K 4K 5K 0%
10%
20%
30%
40%
50%
60%
1 4 7 10131619222528313437
WN
ARAIDA-100 ARAIDA-500 ARAIDA-1000 ARAIDA-2000
1K 2K 3K 4K 5K
21%
26%
31%
36% 41%
1 4 7 10 13 16 19 22 25 28 31 34 37
SST
50%
60%
70%
80% 90%
1 4 7 10 13 16 19 22 25 28 31 34 37
IMDB
0%
10%
20%
30%
40%
50%
1 4 7 10 13 16 19 22 25 28 31 34 37
FreeBase
0% 10% 20% 30% 40% 50% 60%
1 4 7 10131619222528313437
WN
ARIA-100 ARIA-500 ARIA-1000 ARIA-2000
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K
21%
26%
31%
36% 41%
1 4 7 10 13 16 19 22 25 28 31 34 37
SST with AL
1K 2K 3K 4K 5K 50%
55%
60%
65%
70%
75% 80%
1 5 9 13172125293337
IMDB with AL
1K 2K 3K 4K 5K
0%
10%
20%
30%
40%
50%
1 4 7 10 13 16 19 22 25 28 31 34 37
FreeBase-AL
1K 2K 3K 4K 5K 0%
10%
20%
30%
40%
50%
60%
1 4 7 10131619222528313437
WN
ARIA-100 ARIA-500 ARIA-1000 ARIA-2000
1K 2K 3K 4K 5K
(a) WN18RR (b) WN18RR + AL (c) FreeBase (d) FreeBase + AL
21%
26%
31%
36%
41%
1 4 7 10 13 16 19 22 25 28 31 34 37
SST
50%
60%
70%
80%
90%
1 4 7 10 13 16 19 22 25 28 31 34 37
IMDB
0%
10%
20%
30%
40%
50%
1 4 7 10 13 16 19 22 25 28 31 34 37
FreeBase
0% 10% 20% 30% 40% 50% 60%
1 4 7 10131619222528313437
WN
ARIA-100 ARIA-500 ARIA-1000 ARIA-2000
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K
21%
26%
31%
36%
41%
1 4 7 10 13 16 19 22 25 28 31 34 37
SST with AL
1K 2K 3K 4K 5K 50%
55%
60%
65%
70%
75%
80%
1 5 9 13172125293337
IMDB with AL
1K 2K 3K 4K 5K
0%
10%
20%
30%
40%
50%
1 4 7 10 13 16 19 22 25 28 31 34 37
FreeBase-AL
1K 2K 3K 4K 5K 0%
10%
20%
30%
40%
50%
60%
1 4 7 10131619222528313437
WN
ARIA-100 ARIA-500 ARIA-1000 ARIA-2000
1K 2K 3K 4K 5K
21%
26%
31%
36%
41%
1 4 7 10 13 16 19 22 25 28 31 34 37
SST
50%
60%
70%
80%
90%
1 4 7 10 13 16 19 22 25 28 31 34 37
IMDB
0%
10%
20%
30%
40%
50%
1 4 7 10 13 16 19 22 25 28 31 34 37
FreeBase
0% 10% 20% 30% 40%
50%
60%
1 4 7 10131619222528313437
WN
ARIA-100 ARIA-500 ARIA-1000 ARIA-2000
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K
21%
26%
31%
36%
41%
1 4 7 10 13 16 19 22 25 28 31 34 37
SST with AL
1K 2K 3K 4K 5K 50%
55%
60%
65%
70%
75%
80%
1 5 9 13172125293337
IMDB with AL
1 2 3 4K 5K
0%
10%
20%
30%
40%
50%
1 4 7 10 13 16 19 22 25 28 31 4 37
Fr eBase-AL
2 3 4K 5K 0%
10%
20%
30%
40%
50%
60%
1 4 7 10131619 2 528 1 437
WN
ARIA-100 ARIA-500 ARIA-1 00 ARIA-2000
1K 2K 3K 4K 5K
(e) IMDB (f) IMDB + AL (g) SST-5 (h) SST-5 + AL
Figure 6: Machine Cumulative Accuracy of ARAIDA with different datastore sizes.
WN18RR FreeBase Raw Class Mapped Class Raw Class Mapped Class
_hypernym hypernym /location/location/contains contains _derivationally_related_form derivation /olympics/olympic_sport/athletes./olympics/olympic_athlete_affiliation/country country _member_meronym member /music/performance_role/track_performances./music/track_contribution/role track_role _has_part component /people/person/profession profession _synset_domain_topic_of synset /music/performance_role/regular_performances./music/group_membership/role group_role _instance_hypernym instance of hypernym /location/location/adjoin_s./location/adjoining_relationship/adjoins adjoins _also_see synonym /film/film/release_date_s./film/film_regional_release_date/film_release_region film_release _verb_group verb group /food/food/nutrients./food/nutrition_fact/nutrient nutrient
Table 4: Class mapping details for WN18RR and FreeBase
0% 10% 20% 30% 40% 50% 60% 70%
1 5 9 13172125293337
WN
ARAIDA-5 ARAIDA-10 ARAIDA-20 ARAIDA-50
15%
20%
25%
30%
35%
40%
1 5 9 13172125293337
SST
1K 2K 3K 4K 5K
0
0.1
0.2
0.3
0.4
0.5
0.6
1 5 9 13172125293337
FB
50% 55% 60% 65% 70% 75% 80% 85%
1 5 9 13172125293337
imdb
1K 2K 3K 4K 5K
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K
0% 10% 20% 30% 40% 50% 60% 70%
1 5 9 13 72125293337
WN AL
ARAIDA-5 ARAIDA-10 ARAIDA-20 ARAIDA-50
15%
20%
25%
30%
35%
40%
1 5 9 13172125293337
SST AL
50%
55%
60%
65%
70%
75%
80%
1 5 9 13172125293337
IMDB AL
0%
10%
20%
30%
40%
50%
1 5 9 13172125293337
fb
1K 2K 3K 4K 5K1K 2K 3K 4 5K
1K 2K 3K 4K 5K1K 2K 3K 4 5K
0% 10% 20% 30% 40% 50% 6 70
1 5 9 13172125293337
WN
ARIR-5 ARIR-10 ARIR-20 ARIR-50
15%
20%
25%
30%
35%
40%
1 5 9 13172125293337
SST
1K 2K 3K 4K 5K
0%
10%
20%
30%
40%
50%
60
1 5 9 13172125293337
FB
50% 55% 60% 65% 70% 75% 80% 85%
1 5 9 13172125293337
imdb
1K 2K 3K 4K 5K
1 2K 3K 4K 5K 1K 2K 3K 4K 5K
0% 10% 20% 30% 40% 50% 60% 70%
1 5 9 13172125293337
WN AL
ARIR-5 ARIR-10 ARIR-20 ARIR-50
15%
20%
25%
30%
35%
40%
1 5 9 13172125293337
SST AL
55%
60%
65%
70%
75%
80%
1 5 9 13172
IMD AL
0%
10%
20%
30%
40%
50%
1 5 9 1317212529 37
f
1K 2K 3K 4K 5K1K 2K 3K 4K 5K
1K 2K 3K 4K 5K1K 2K 3
(a) WN18RR (b) WN18RR + AL (c) FreeBase (d) FreeBase + AL
0% 10% 20% 30% 40% 50% 60% 70%
1 5 9 13172125293337
WN
ARIR-5 ARIR-10 ARIR-20 ARIR-50
15
20%
25%
30
35%
40
7
SST
1K 2K 3K 4K 5K
0
0.1
0.2
0.3
0.4
0.5
0.6
1 5 9 13172125293337
FB
50% 55% 60% 65% 70% 75% 80% 85%
1 5 9 13172125293337
imdb
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K
0% 10% 20% 30% 40% 50% 60% 70%
1 5 9 13172125293337
N AL
ARIR-5 ARIR-10 ARIR-20 ARIR-50
15%
20%
25%
30%
35%
40%
1 5 9 131721252933 7
SST AL
50%
55%
60%
65%
70%
75%
80%
1 5 9 13172125293337
I DB AL
0%
10%
20%
30%
40%
50%
1 5 9 13172125293337
fb
K 2K 3K 4K 5K1K 2K 3K 4K 5K
1K 2K 3K 4K 5K1K 2K 3K 4K 5K 0%
10% 20% 30% 40% 50% 60% 70%
1 5 9 13172125293337
WN
ARIR-5 ARIR-10 ARIR-20 ARIR-50
15%
20%
25%
30%
35%
40%
1 5 9 1317212529 337
SST
1K 2K 3K 4K 5K
0
0.1
0.2
0.3
0.4
0.5
0.6
1 5 9 13172125293337
FB
50% 55% 60% 65% 70% 75%
80%
85%
1 5 9 13172125293337
imdb
1 2 3 4 5
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K
0% 10% 20% 30% 40% 50% 60% 70%
5 9 1317212529 37
WN AL
ARIR-5 ARIR-10 ARIR-20 ARIR-50
15%
20%
25%
30%
35%
40%
1 5 9 13172125293337
SST AL
50%
55%
60%
65%
70%
75%
80%
1 5 9 13172125293337
IMDB AL
0%
10%
20%
30%
40%
50%
1 5 9 13172125293337
fb
1K 2K 3K 4K 5K1K 2K 3K 4K 5K
1K 2K 3K 4K 5K1K 2K 3K 4K 5K
(e) IMDB (f) IMDB + AL (g) SST-5 (h) SST-5 + AL
Figure 7: Machine Cumulative Accuracy of ARAIDA with different k for ρt.
notation scenarios, we typically don’t have access to fully fine-tuned models initially, as we lack labeled datasets. While one might turn to LLMs like ChatGPT, our findings in Figure 3 indicate that our
method could potentially improve upon ChatGPT even further.
20%
25%
30%
35%
40% 45%
1 5 9 13172125293337
SST
50% 55% 60% 65% 70% 75% 80% 85%
1 5 9 13172125293337
IMDB
0%
10%
20%
30%
40% 50% 60%
1 5 9 13172125293337
FB
0% 10% 20% 30% 40% 50% 60% 70%
1 5 9 13172125293337
WN
ARAIDA ARAIDA w/ FIFO ARAIDA w/class FIFO
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K
20%
25%
30%
35%
40% 45%
1 5 9 13172125293337
SST-AL
0.5
0.55
0.6
0.65
0.7 0.75 0.8
1 5 9 13172125293337
IMDB
0%
10%
20%
30%
40%
50%
1 5 9 13172125293 7
FB
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K
1K 2K 3K 4K 5K 0%
10%
20%
30%
40%
50%
60%
1 5 9 13172125293337
WN
ARAIDA ARAIDA w/ FIFO ARAIDA w/class FIFO 1K 2K 3K 4K 5K 20% 25%
30%
35%
40%
45%
1 5 9 13172125293337
SST
50% 55% 60% 65% 70% 75% 80% 85%
1 5 9 13172125293337
IMDB
0%
10%
20%
30%
40%
50%
60%
1 5 9 13172125293337
FB
0% 10% 20% 30% 40% 50% 60% 70%
1 5 9 13172125293337
WN
ARIA ARIA w/ FIFO ARIA w/class FIFO
1K 2K 3 4 5K 1K 2K 3K 4K 5K
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K
20%
25%
30%
35%
40% 45%
1 5 9 13172125293337
SST-AL
0.5
0.55
0.6
0.65
0.7 0.75 0.8
1 5 9 13172125293337
IMDB
0%
10%
20%
30%
40%
50%
1 5 9 13172125293337
FB
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K
1K 2K 3K 4K 5K 0%
10%
20%
30%
40%
50%
60%
1 5 9 13172125293337
WN
ARIA ARIA w/ FIFO ARIA w/class FIFO
1K 2K 3K 4K 5K
(a) WN18RR (b) WN18RR + AL (c) FreeBase (d) FreeBase + AL
2
25%
30%
35
40%
45
1 5 9 13172125293337
SST
50% 55% 60% 65% 70% 75% 80% 85%
1 5 9 13172125293337
IMDB
0%
10%
20%
30%
40%
50%
60%
1 5 9 13172125293337
FB
0% 10% 20% 30% 40% 50% 60% 70%
1 5 9 13172125293337
WN
ARIA ARIA w/ FIFO ARIA w/class FIFO
1 2K 3K 4K 5K 1K 2K 3K 4K 5K
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K 20%
25%
30%
35%
40%
45%
1 5 9 13172125293 7
SST-AL
50%
55%
60%
65%
70%
75%
80%
1 5 9 13172125293337
IMDB
0%
10%
20%
30%
40%
50%
1 5 9 13172125293337
FB
1K 2K 3K 4K 5K 1K 2K K 4K 5K
1K 2K 3K 4K 5K 0%
10%
20%
30%
40%
50%
60%
1 5 9 13172125293337
WN
ARIA
ARIA w/ FIFO
ARIA w/class FIFO
1K 2K 3K 4K 5K
20%
25
30%
35
40%
45%
1 5 9 13172125293337
SST
50% 55% 60% 65% 70% 75%
80%
85%
1 5 9 13172125293337
IMDB
0%
10%
20%
30%
40%
50%
60%
1 5 9 13172125293337
FB
0% 10% 20% 30% 40% 50%
60%
70%
1 5 9 13172125293337
WN
ARIA ARIA w/ FIFO ARIA w/class FIFO
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K
20%
25%
30%
35%
40%
45%
1 5 9 13 72125293337
SST-AL
0.5
0.55
0.6
0.65
0.7
0.75
0.8
1 5 9 13172125293337
IMDB
0%
10%
20%
30%
40%
50%
1 5 9 13172125293337
FB
1K 2K 3 4 5 1K 2K 3K 4K 5K
1K 2K 3K 4K 5K 0%
10%
20%
30%
40%
50%
60%
1 5 9 13172125293337
WN
ARIA ARIA w/ FIFO ARIA w/class FIFO
1K 2K 3K 4K 5K
(e) IMDB (f) IMDB + AL (g) SST-5 (h) SST-5 + AL
Figure 8: Machine Cumulative Accuracy of ARAIDA with different datastore maintenance strategies.
Table 5: Results on comparing to fully fine-tuned models
Model WN18RR FreeBase IMDB SST-5 BERT(fully finetuned) 75.13±1.12 60.47±1.24 90.04±0.42 46.59±1.20 BERT+ARAIDA 54.41±1.51 50.30±2.25 88.79±3.54 43.81±3.12 Dist./FT(fully finetuned) 72.64±1.05 58.50±1.36 84.87±0.81 42.12±1.44 Dist./FT+ARAIDA 52.16±1.37 43.02±6.43 79.33±2.81 37.21±3.03

heading: C Results on Smaller-sized Transformer-based Model
text: With a wide array of language models to choose from, we faced the challenge of not being able to test all available models. To address this, we selected three prominent models (GloVe, LLaMa2, and ChatGPT) based on our hardware resource capabilities in our main experiments. While open to conducting further experiments, our focus was limited by time constraints, leading us to concentrate solely on fine-tuning BERT (w/o AL), with the outcomes detailed in Table 6. After implementing ARAIDA, we observed a significant enhancement in the quality of annotations by using our ARAIDA.

heading: D Prompts and Fine-Tuning Examples for LLM-Based Annotation Models
text: Table 7 presents prompts used for zero-shot and few-shot annotation. Table 8 shows fine-tuning examples for LLaMa2sft.

heading: Future Work
text: This suggests that while our method currently offers a lightweight approach to aid data annotation, which can to some extent enhance the sample efficiency of the annotation model, there is still potential for improvement in future research. Nevertheless, it’s important to emphasize that in real-world data an-
21%
26%
31%
36% 41%
1 4 7 10 13 16 19 22 25 28 31 34 37
SST
50%
60%
70%
80% 90%
1 4 7 10 13 16 19 22 25 28 31 34 37
IMDB
0%
10%
20%
30%
40%
50%
1 4 7 10 13 16 19 22 25 28 31 34 37
FreeBase
0% 10% 20% 30% 40% 50% 60%
1 4 7 10131619222528313437
WN
ARAIDA-100 ARAIDA-500 ARAIDA-1000 ARAIDA-2000
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K
21%
26%
31%
36% 41%
1 4 7 10 13 16 19 22 25 28 31 34 37
SST with AL
1K 2K 3K 4K 5K 50%
55%
60%
65%
70%
75% 80%
1 5 9 13172125293337
IMDB with AL
1K 2K 3K 4K 5K
0%
10%
20%
30%
40%
50%
1 4 7 10 13 16 19 22 25 28 31 34 37
FreeBase-AL
1K 2K 3K 4K 5K 0%
10%
20%
30%
40%
50%
60%
1 4 7 10131619222528313437
WN
ARAIDA-100 ARAIDA-500 ARAIDA-1000 ARAIDA-2000
1K 2K 3K 4K 5K
21%
26%
31%
36% 41%
1 4 7 10 13 16 19 22 25 28 31 34 37
SST
50%
60%
70%
80% 90%
1 4 7 10 13 16 19 22 25 28 31 34 37
IMDB
0%
10%
20%
30%
40%
50%
1 4 7 10 13 16 19 22 25 28 31 34 37
FreeBase
0% 10% 20% 30% 40% 50% 60%
1 4 7 10131619222528313437
WN
ARIA-100 ARIA-500 ARIA-1000 ARIA-2000
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K
21%
26%
31%
36% 41%
1 4 7 10 13 16 19 22 25 28 31 34 37
SST with AL
1K 2K 3K 4K 5K 50%
55%
60%
65%
70%
75% 80%
1 5 9 13172125293337
IMDB with AL
1K 2K 3K 4K 5K
0%
10%
20%
30%
40%
50%
1 4 7 10 13 16 19 22 25 28 31 34 37
FreeBase-AL
1K 2K 3K 4K 5K 0%
10%
20%
30%
40%
50%
60%
1 4 7 10131619222528313437
WN
ARIA-100 ARIA-500 ARIA-1000 ARIA-2000
1K 2K 3K 4K 5K
(a) WN18RR (b) WN18RR + AL (c) FreeBase (d) FreeBase + AL
21%
26%
31%
36%
41%
1 4 7 10 13 16 19 22 25 28 31 34 37
SST
50%
60%
70%
80%
90%
1 4 7 10 13 16 19 22 25 28 31 34 37
IMDB
0%
10%
20%
30%
40%
50%
1 4 7 10 13 16 19 22 25 28 31 34 37
FreeBase
0% 10% 20% 30% 40% 50% 60%
1 4 7 10131619222528313437
WN
ARIA-100 ARIA-500 ARIA-1000 ARIA-2000
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K
21%
26%
31%
36%
41%
1 4 7 10 13 16 19 22 25 28 31 34 37
SST with AL
1K 2K 3K 4K 5K 50%
55%
60%
65%
70%
75%
80%
1 5 9 13172125293337
IMDB with AL
1K 2K 3K 4K 5K
0%
10%
20%
30%
40%
50%
1 4 7 10 13 16 19 22 25 28 31 34 37
FreeBase-AL
1K 2K 3K 4K 5K 0%
10%
20%
30%
40%
50%
60%
1 4 7 10131619222528313437
WN
ARIA-100 ARIA-500 ARIA-1000 ARIA-2000
1K 2K 3K 4K 5K
21%
26%
31%
36%
41%
1 4 7 10 13 16 19 22 25 28 31 34 37
SST
50%
60%
70%
80%
90%
1 4 7 10 13 16 19 22 25 28 31 34 37
IMDB
0%
10%
20%
30%
40%
50%
1 4 7 10 13 16 19 22 25 28 31 34 37
FreeBase
0% 10% 20% 30% 40%
50%
60%
1 4 7 10131619222528313437
WN
ARIA-100 ARIA-500 ARIA-1000 ARIA-2000
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K
21%
26%
31%
36%
41%
1 4 7 10 13 16 19 22 25 28 31 34 37
SST with AL
1K 2K 3K 4K 5K 50%
55%
60%
65%
70%
75%
80%
1 5 9 13172125293337
IMDB with AL
1 2 3 4K 5K
0%
10%
20%
30%
40%
50%
1 4 7 10 13 16 19 22 25 28 31 4 37
Fr eBase-AL
2 3 4K 5K 0%
10%
20%
30%
40%
50%
60%
1 4 7 10131619 2 528 1 437
WN
ARIA-100 ARIA-500 ARIA-1 00 ARIA-2000
1K 2K 3K 4K 5K
(e) IMDB (f) IMDB + AL (g) SST-5 (h) SST-5 + AL
Figure 6: Machine Cumulative Accuracy of ARAIDA with different datastore sizes. WN18RR FreeBase Raw Class Mapped Class Raw Class Mapped Class
_hypernym hypernym /location/location/contains contains _derivationally_related_form derivation /olympics/olympic_sport/athletes. /olympics/olympic_athlete_affiliation/country country _member_meronym member /music/performance_role/track_performances. /music/track_contribution/role track_role _has_part component /people/person/profession profession _synset_domain_topic_of synset /music/performance_role/regular_performances. /music/group_membership/role group_role _instance_hypernym instance of hypernym /location/location/adjoin_s. /location/adjoining_relationship/adjoins adjoins _also_see synonym /film/film/release_date_s. /film/film_regional_release_date/film_release_region film_release _verb_group verb group /food/food/nutrients. /food/nutrition_fact/nutrient nutrient
Table 4: Class mapping details for WN18RR and FreeBase
0% 10% 20% 30% 40% 50% 60% 70%
1 5 9 13172125293337
WN
ARAIDA-5 ARAIDA-10 ARAIDA-20 ARAIDA-50
15%
20%
25%
30%
35%
40%
1 5 9 13172125293337
SST
1K 2K 3K 4K 5K
0
0. 1
0. 2
0. 3
0. 4
0. 5
0. 6
1 5 9 13172125293337
FB
50% 55% 60% 65% 70% 75% 80% 85%
1 5 9 13172125293337
imdb
1K 2K 3K 4K 5K
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K
0% 10% 20% 30% 40% 50% 60% 70%
1 5 9 13 72125293337
WN AL
ARAIDA-5 ARAIDA-10 ARAIDA-20 ARAIDA-50
15%
20%
25%
30%
35%
40%
1 5 9 13172125293337
SST AL
50%
55%
60%
65%
70%
75%
80%
1 5 9 13172125293337
IMDB AL
0%
10%
20%
30%
40%
50%
1 5 9 13172125293337
fb
1K 2K 3K 4K 5K1K 2K 3K 4 5K
1K 2K 3K 4K 5K1K 2K 3K 4 5K
0% 10% 20% 30% 40% 50% 6 70
1 5 9 13172125293337
WN
ARIR-5 ARIR-10 ARIR-20 ARIR-50
15%
20%
25%
30%
35%
40%
1 5 9 13172125293337
SST
1K 2K 3K 4K 5K
0%
10%
20%
30%
40%
50%
60
1 5 9 13172125293337
FB
50% 55% 60% 65% 70% 75% 80% 85%
1 5 9 13172125293337
imdb
1K 2K 3K 4K 5K
1 2K 3K 4K 5K 1K 2K 3K 4K 5K
0% 10% 20% 30% 40% 50% 60% 70%
1 5 9 13172125293337
WN AL
ARIR-5 ARIR-10 ARIR-20 ARIR-50
15%
20%
25%
30%
35%
40%
1 5 9 13172125293337
SST AL
55%
60%
65%
70%
75%
80%
1 5 9 13172
IMD AL
0%
10%
20%
30%
40%
50%
1 5 9 1317212529 37
f
1K 2K 3K 4K 5K1K 2K 3K 4K 5K
1K 2K 3K 4K 5K1K 2K 3
(a) WN18RR (b) WN18RR + AL (c) FreeBase (d) FreeBase + AL
0% 10% 20% 30% 40% 50% 60% 70%
1 5 9 13172125293337
WN
ARIR-5 ARIR-10 ARIR-20 ARIR-50
15
20%
25%
30
35%
40
7
SST
1K 2K 3K 4K 5K
0
0. 1
0. 2
0. 3
0. 4
0. 5
0. 6
1 5 9 13172125293337
FB
50% 55% 60% 65% 70% 75% 80% 85%
1 5 9 13172125293337
imdb
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K
0% 10% 20% 30% 40% 50% 60% 70%
1 5 9 13172125293337
N AL
ARIR-5 ARIR-10 ARIR-20 ARIR-50
15%
20%
25%
30%
35%
40%
1 5 9 131721252933 7
SST AL
50%
55%
60%
65%
70%
75%
80%
1 5 9 13172125293337
I DB AL
0%
10%
20%
30%
40%
50%
1 5 9 13172125293337
fb
K 2K 3K 4K 5K1K 2K 3K 4K 5K
1K 2K 3K 4K 5K1K 2K 3K 4K 5K 0%
10% 20% 30% 40% 50% 60% 70%
1 5 9 13172125293337
WN
ARIR-5 ARIR-10 ARIR-20 ARIR-50
15%
20%
25%
30%
35%
40%
1 5 9 1317212529 337
SST
1K 2K 3K 4K 5K
0
0. 1
0. 2
0. 3
0. 4
0. 5
0. 6
1 5 9 13172125293337
FB
50% 55% 60% 65% 70% 75%
80%
85%
1 5 9 13172125293337
imdb
1 2 3 4 5
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K
0% 10% 20% 30% 40% 50% 60% 70%
5 9 1317212529 37
WN AL
ARIR-5 ARIR-10 ARIR-20 ARIR-50
15%
20%
25%
30%
35%
40%
1 5 9 13172125293337
SST AL
50%
55%
60%
65%
70%
75%
80%
1 5 9 13172125293337
IMDB AL
0%
10%
20%
30%
40%
50%
1 5 9 13172125293337
fb
1K 2K 3K 4K 5K1K 2K 3K 4K 5K
1K 2K 3K 4K 5K1K 2K 3K 4K 5K
(e) IMDB (f) IMDB + AL (g) SST-5 (h) SST-5 + AL
Figure 7: Machine Cumulative Accuracy of ARAIDA with different k for ρt. notation scenarios, we typically don’t have access to fully fine-tuned models initially, as we lack labeled datasets. While one might turn to LLMs like ChatGPT, our findings in Figure 3 indicate that our
method could potentially improve upon ChatGPT even further. 20%
25%
30%
35%
40% 45%
1 5 9 13172125293337
SST
50% 55% 60% 65% 70% 75% 80% 85%
1 5 9 13172125293337
IMDB
0%
10%
20%
30%
40% 50% 60%
1 5 9 13172125293337
FB
0% 10% 20% 30% 40% 50% 60% 70%
1 5 9 13172125293337
WN
ARAIDA ARAIDA w/ FIFO ARAIDA w/class FIFO
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K
20%
25%
30%
35%
40% 45%
1 5 9 13172125293337
SST-AL
0. 5
0. 55
0. 6
0. 65
0. 7 0. 75 0. 8
1 5 9 13172125293337
IMDB
0%
10%
20%
30%
40%
50%
1 5 9 13172125293 7
FB
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K
1K 2K 3K 4K 5K 0%
10%
20%
30%
40%
50%
60%
1 5 9 13172125293337
WN
ARAIDA ARAIDA w/ FIFO ARAIDA w/class FIFO 1K 2K 3K 4K 5K 20% 25%
30%
35%
40%
45%
1 5 9 13172125293337
SST
50% 55% 60% 65% 70% 75% 80% 85%
1 5 9 13172125293337
IMDB
0%
10%
20%
30%
40%
50%
60%
1 5 9 13172125293337
FB
0% 10% 20% 30% 40% 50% 60% 70%
1 5 9 13172125293337
WN
ARIA ARIA w/ FIFO ARIA w/class FIFO
1K 2K 3 4 5K 1K 2K 3K 4K 5K
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K
20%
25%
30%
35%
40% 45%
1 5 9 13172125293337
SST-AL
0. 5
0. 55
0. 6
0. 65
0. 7 0. 75 0. 8
1 5 9 13172125293337
IMDB
0%
10%
20%
30%
40%
50%
1 5 9 13172125293337
FB
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K
1K 2K 3K 4K 5K 0%
10%
20%
30%
40%
50%
60%
1 5 9 13172125293337
WN
ARIA ARIA w/ FIFO ARIA w/class FIFO
1K 2K 3K 4K 5K
(a) WN18RR (b) WN18RR + AL (c) FreeBase (d) FreeBase + AL
2
25%
30%
35
40%
45
1 5 9 13172125293337
SST
50% 55% 60% 65% 70% 75% 80% 85%
1 5 9 13172125293337
IMDB
0%
10%
20%
30%
40%
50%
60%
1 5 9 13172125293337
FB
0% 10% 20% 30% 40% 50% 60% 70%
1 5 9 13172125293337
WN
ARIA ARIA w/ FIFO ARIA w/class FIFO
1 2K 3K 4K 5K 1K 2K 3K 4K 5K
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K 20%
25%
30%
35%
40%
45%
1 5 9 13172125293 7
SST-AL
50%
55%
60%
65%
70%
75%
80%
1 5 9 13172125293337
IMDB
0%
10%
20%
30%
40%
50%
1 5 9 13172125293337
FB
1K 2K 3K 4K 5K 1K 2K K 4K 5K
1K 2K 3K 4K 5K 0%
10%
20%
30%
40%
50%
60%
1 5 9 13172125293337
WN
ARIA
ARIA w/ FIFO
ARIA w/class FIFO
1K 2K 3K 4K 5K
20%
25
30%
35
40%
45%
1 5 9 13172125293337
SST
50% 55% 60% 65% 70% 75%
80%
85%
1 5 9 13172125293337
IMDB
0%
10%
20%
30%
40%
50%
60%
1 5 9 13172125293337
FB
0% 10% 20% 30% 40% 50%
60%
70%
1 5 9 13172125293337
WN
ARIA ARIA w/ FIFO ARIA w/class FIFO
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K
1K 2K 3K 4K 5K 1K 2K 3K 4K 5K
20%
25%
30%
35%
40%
45%
1 5 9 13 72125293337
SST-AL
0. 5
0. 55
0. 6
0. 65
0. 7
0. 75
0. 8
1 5 9 13172125293337
IMDB
0%
10%
20%
30%
40%
50%
1 5 9 13172125293337
FB
1K 2K 3 4 5 1K 2K 3K 4K 5K
1K 2K 3K 4K 5K 0%
10%
20%
30%
40%
50%
60%
1 5 9 13172125293337
WN
ARIA ARIA w/ FIFO ARIA w/class FIFO
1K 2K 3K 4K 5K
(e) IMDB (f) IMDB + AL (g) SST-5 (h) SST-5 + AL
Figure 8: Machine Cumulative Accuracy of ARAIDA with different datastore maintenance strategies. Table 5: Results on comparing to fully fine-tuned models
Model WN18RR FreeBase IMDB SST-5 BERT(fully finetuned) 75. 13±1. 12 60. 47±1. 24 90. 04±0. 42 46. 59±1. 20 BERT+ARAIDA 54. 41±1. 51 50. 30±2. 25 88. 79±3. 54 43. 81±3. 12 Dist. /FT(fully finetuned) 72. 64±1. 05 58. 50±1. 36 84. 87±0. 81 42. 12±1. 44 Dist. /FT+ARAIDA 52. 16±1. 37 43. 02±6. 43 79. 33±2. 81 37. 21±3. 03