Assisted Text Annotation Using Active Learning to Achieve High Quality with Little Effort Franziska Weeber∗, Felix Hamborg†, Karsten Donnay‡† and Bela Gipp§ ∗University of Konstanz, Germany, †Heidelberg Academy of Sciences and Humanities, Germany, ‡University of Zurich, Switzerland, §University of Wuppertal, Germany franziska.weeber@uni-konstanz.de Abstract—Large amounts of annotated data have become more important than ever, especially since the rise of deep learning techniques. However, manual annotations are costly. We propose a tool that enables researchers to create large, high-quality, annotated datasets with only a few manual annotations, thus strongly reducing annotation cost and effort. For this purpose, we combine an active learning (AL) approach with a pre- trained language model to semi-automatically identify annotation categories in the given text documents. To highlight our research direction’s potential, we evaluate the approach on the task of identifying frames in news articles. Our preliminary results show that employing AL strongly reduces the number of annotations for correct classiﬁcation of even these complex and subtle frames. On the framing dataset, the AL approach needs only 16.3% of the annotations to reach the same performance as a model trained on the full dataset. Index Terms—active learning, deep learning, Longformer, text classiﬁcation, data annotation I. INTRODUCTION AND RELATED WORK With the rise of machine learning and recently deep learn- ing, annotated data have become more important than ever before. While traditional machine learning relies mostly on handcrafted rules and features, recent approaches use large amounts of annotated data. Not only computer scientists rely on these extensive labeled corpora, they are also needed in other disciplines, e.g., for the large-scale analysis of media bias in political science, genome diagnostics in the medical and biological sciences, and other practical applications such as online customer support bots. Large amounts of unlabeled data are available for most tasks, but creating the annota- tions needed to use them as training sets is costly [4]. A solution is active learning (AL), where a classiﬁer is trained over multiple iterations by querying batches of only those instances for manual annotation from a large unlabeled pool that are considered most informative for the model [4]. AL can substantially reduce the number of labeled observations needed to classify the full pool, especially when classes are unbalanced because the minority class will be overrepresented in the queried training data [4], [6]. The AL process is structured into two phases: seed set se- lection and iterative query selection plus training. During seed set selection, the initial training set is chosen, e.g., randomly, manually, or based on other criteria such as seed words or cluster membership [6]. Humans label these instances, and then the model is trained on them. In query selection, each iteration entails: predicting the class probabilities for the data pool, querying new instances for labeling, and retraining the model. The ﬁrst task aims to determine which instances to query for labeling next using a measure of informativeness, e.g., the uncertainty of the prediction. Afterward, the classiﬁer is retrained on the updated labeled set until a performance threshold is reached or a predeﬁned number of texts have been annotated [6]. Previous work on semi-automated data annotation employ- ing AL suffers from the following shortcomings: (1) super- ﬁcial or poor results and (2) lack of easy-to-use annotation interfaces. First, prior methods using AL often use outdated methods, such as bag of word representations and SVMs [5], [6], not reaching the performance of language models such as BERT or its successors. Second, most approaches do not provide an interface for the annotation [5] or still require users to have programming experience [3]. Despite these shortcomings, the usefulness