Experiments	Here we compare the efficiency of our system with four widely used annotation tools. We extract 100 sentences from CoNLL 2003 English NER  training data, with each sentence containing at least 4 entities. Two undergraduate students without any experience on those tools are invited to annotate those sentences. Their average annotation time is shown in Figure FIGREF25 , where “Yedda+R” suggests annotation using Yedda with the help of system recommendation. The inter-annotator agreements for those tools are closed, which around 96.1% F1-score. As we can see from the figure, our Yedda system can greatly reduce the annotation time. With the help of system recommendation, the annotation time can be further reduced. We notice that “Yedda+R” has larger advantage with the increasing numbers of annotated sentences, this is because the system recommendation gives better suggestions when it learns larger annotated sentences. The “Yedda+R” gives 16.47% time reduction in annotating 100 sentences.

YEDDA: A Lightweight Collaborative Text Span Annotation Tool	In this paper, we introduce \textsc{Yedda}, a lightweight but efficient and comprehensive open-source tool for text span annotation. \textsc{Yedda} provides a systematic solution for text span annotation, ranging from collaborative user annotation to administrator evaluation and analysis. It overcomes the low efficiency of traditional text annotation tools by annotating entities through both command line and shortcut keys, which are configurable with custom labels. \textsc{Yedda} also gives intelligent recommendations by learning the up-to-date annotated text. An administrator client is developed to evaluate annotation quality of multiple annotators and generate detailed comparison report for each annotator pair. Experiments show that the proposed system can reduce the annotation time by half compared with existing annotation tools. And the annotation time can be further compressed by 16.47\% through intelligent recommendation.

The recommending system is designed as “pluggable” which ensures that the recommending algorithm can be easily extended into other sequence labeling models, such as Conditional Random Field (CRF)  . The recommendation can be controlled through two buttons “RMOn” and “RMOff”, which enables and disables the recommending function, respectively. It is inevitable that the annotator or the recommending system gives incorrect annotations or suggestions. Based on our annotation experience, we found that the time cost of annotation correction cannot be neglected. Therefore, Yedda provides several efficient modification actions to revise the annotation: • INLINEFORM0 Action withdraw: annotators can cancel their previous action and let system return to the last status by press the shortcut key Ctrl+z. • INLINEFORM0 Span label modification: if the selected span has the correct boundary but receives an incorrect label, annotator only needs to put the cursor inside the span (or select the span) and press the shortcut key of the right label to correct label. • INLINEFORM0 Label deletion: similar to the label modification, the annotator can put the cursor inside the span and press shortcut key q to remove the annotated (recommended) label.

For example, command “ INLINEFORM3 ” represents annotating following 2 characters as label ` INLINEFORM4 ' (mapped into a specific label name), the following 3 characters as label ` INLINEFORM5 ' and 2 characters further as label ` INLINEFORM6 '. It has been shown that using pre-annotated text and manual correction increases the annotation efficiency in many annotation tasks  ,  . Yedda offers annotators with system recommendation based on the existing annotation history. The current recommendation system incrementally collects annotated text spans from sentences that have been labeled, thus gaining a dynamically growing lexicon. Using the lexicon, the system automatically annotates sentences that are currently being annotated by leveraging the forward maximum matching algorithm. The automatically suggested text spans and their types are returned with colors in the user interface, as shown in green in Figure FIGREF4 . Annotators can use the shortcut to confirm, correct or veto the suggestions. The recommending system keeps online updating during the whole annotation process, which learns the up-to-date and in-domain annotation information.

To keep annotation efficient and accurate, Yedda provides systematic solutions across the whole annotation process, which includes the shortcut annotation, batch annotation with a command line, intelligent recommendation, format exporting and administrator evaluation/analysis. Figure FIGREF1 shows the general framework of Yedda. It offers annotators with a simple and efficient Graphical User Interface (GUI) to annotate raw text. For the administrator, it provides two useful toolkits to evaluate multi-annotated text and generate detailed comparison report for annotator pair. Yedda has the advantages of being: • INLINEFORM0 Convenient: it is lightweight with an intuitive interface and does not rely on specific operating systems or pre-installed packages. • INLINEFORM0 Efficient: it supports both shortcut and command line annotation models to accelerate the annotating process. • INLINEFORM0 Intelligent: it offers user with real-time system suggestions to avoid duplicated annotation. • INLINEFORM0 Comprehensive: it integrates useful toolkits to give the statistical index of analyzing multi-user annotation results and generate detailed content comparison for annotation pairs. This paper is organized as follows: Section 2 gives an overview of previous text annotation tools and the comparison with ours.

WebAnno  ,  supports both the automatic annotation suggestion and annotation quality monitoring such as inter-annotator agreement measurement, data curation, and progress monitoring. It compares the annotation disagreements only for each sentence and shows the comparison within the interface, while our system can generate a detailed disagreement report in .pdf file through the whole annotated content. Besides, those web-based annotation tools need to build a server through complex configurations and some of the servers cannot be deployed on Windows systems. The differences between Yedda and related work are summarised in Table TABREF2 . Here “Self Consistency” represents whether the tool works independently or it relies on pre-installed packages. Compared to these tools, Yedda provides a lighter but more systematic choice with more flexibility, efficiency and less dependence on system environment for text span annotation. Besides, Yedda offers administrator useful toolkits for evaluating the annotation quality and analyze the detailed disagreements within annotators.

Yedda provides the function of annotating text span by selecting using mouse and press shortcut key to map the selection into a specific label. It is a common annotation process in many annotation tools  ,  . It binds each label with one custom shortcut key, this is shown in the “Shortcuts map Labels” part of Figure FIGREF4 . The annotator needs only two steps to annotate one text span, i.e. “select and press”. The annotated file updates simultaneously with each key pressing process. Yedda also support the command line annotation function (see the command entry in the bottom of Figure FIGREF4 ) which can execute multi-span annotation at once. The system will parse the command automatically and convert the command into multi-span annotation instructions and execute in batch. It is quite efficient for the tasks of character-based languages (such as Chinese and Japanese) with high entity density. The command follows a simple rule which is INLINEFORM0 , where ` INLINEFORM1 ' are the length of the entities and ` INLINEFORM2 ' is the corresponding shortcut key.

Administrator Toolkits	For the administrator, it is important and necessary to evaluate the quality of annotated files and analyze the detailed disagreements of different annotators. Shown in Figure FIGREF13 , Yedda provides a simple interface with several toolkits for administrator monitoring the annotation process. To evaluate and monitor the annotation quality of different annotators, our Multi-Annotator Analysis (MAA) toolkit imports all the annotated files and gives the analysis results in a matrix. As shown in Figure FIGREF16 , the matrix gives the F1-scores in full level (consider both boundary and label accuracy) and boundary level (ignore the label correctness, only care about the boundary accuracy) of all annotator pairs. If an administrator wants to look into the detailed disagreement of annotators, it is quite convenient by using the Pairwise Annotators Comparison (PAC). PAC loads two annotated files and generates a specific comparison report file for the two annotators. As shown in Figure FIGREF21 , the report is mainly in two parts: • INLINEFORM0 Overall statistics: it shows the specific precision, recall and F1-score of two files in all labels.

Annotator Client	The client is designed to accelerate the annotation process as much as possible. It supports shortcut annotation to reduce the user operation time. Command line annotation is designed to annotate multi-span in batch. In addition, the client provides system recommendations to lessen the workload of duplicated span annotation. Figure FIGREF4 shows the interface of annotator client on an English entity annotation file. The interface consists of 5 parts. The working area in the up-left which shows the texts with different colors (blue: annotated entities, green: recommended entities and orange: selected text span). The entry at the bottom is the command line which accepts annotation command. There are several control buttons in the middle of the interface, which are used to set annotation model. The status area is below the control buttons, it shows the cursor position and the status of recommending model. The right side shows the shortcut map, where shortcut key “a” or “ INLINEFORM0 ” means annotating the text span with “Artificial” type and the same for other shortcut keys. The shortcut map can be configured easily. Details are introduced as follows.

Introduction	Natural Language Processing (NLP) systems rely on large-scale training data  for supervised training. However, manual annotation can be time-consuming and expensive. Despite detailed annotation standards and rules, inter-annotator disagreement is inevitable because of human mistakes, language phenomena which are not covered by the annotation rules and the ambiguity of language itself  . Existing annotation tools  ,  ,  ,  mainly focus on providing a visual interface for user annotation process but rarely consider the post-annotation quality analysis, which is necessary due to the inter-annotator disagreement. In addition to the annotation quality, efficiency is also critical in large-scale annotation task, while being relatively less addressed in existing annotation tools  ,  . Besides, many tools  ,  require a complex system configuration on either local device or server, which is not friendly to new users. To address the challenges above, we propose Yedda , a lightweight and efficient annotation tool for text span annotation. A snapshot is shown in Figure FIGREF4 . Here text span boundaries are selected and assigned with a label, which can be useful for Named Entity Recognition (NER)  , word segmentation  , chunking  ,etc.

Related Work	There exists a range of text span annotation tools which focus on different aspects of the annotation process. Stanford manual annotation tool is a lightweight tool but does not support result analysis and system recommendation. Knowtator  is a general-task annotation tool which links to a biomedical onto ontology to help identify named entities and relations. It supports quality control during the annotation process by integrating simple inter-annotator evaluation, while it cannot figure out the detailed disagreed labels. WordFreak  adds a system recommendation function and integrates active learning to rank the unannotated sentences based on the recommend confidence, while the post-annotation analysis is not supported. Web-based annotation tools have been developed to build operating system independent annotation environments. Gate  includes a web-based with collaborative annotation framework which allows users to work collaboratively by annotating online with shared text storage. Brat  is another web-based tool, which has been widely used in recent years, it provides powerful annotation functions and rich visualization ability, while it does not integrate the result analysis function. Anafora  and Atomic  are also web-based and lightweight annotation tools, while they don't support the automatic annotation and quality analysis either.

Acknowledgements	We thank Yanxia Qin, Hongmin Wang, Shaolei Wang, Jiangming Liu, Yuze Gao, Ye Yuan, Lu Cao, Yumin Zhou and other members of SUTDNLP group for their trials and feedbacks. Yue Zhang is the corresponding author. Jie is supported by the YEDDA grant 52YD1314.

Yedda	Yedda is developed based on standard Python GUI library Tkinter, and hence needs only Python installation as a prerequisite and is compatible with all Operating System (OS) platforms with Python installation. It offers two user-friendly interfaces for annotators and administrator, respectively, which are introduced in detail in Section SECREF9 and Section SECREF19 , respectively.

Conclusion and Future Work	We have presented a lightweight but systematic annotation tool, Yedda, for annotating the entities in text and analyzing the annotation results efficiently. In order to reduce the workload of annotators, we are going to integrate active learning strategy in our system recommendation part in the future. A supervised sequence labeling model (such as CRF) is trained based on the annotated text, then unannotated sentences with less confidence (predicted by this model) are reordered in the front to ensure annotators only annotate the most confusing sentences.

As the annotated file is saved in .ann format, Yedda provides the “Export” function which exports the annotated text as standard format (ended with .anns). Each line includes one word/character and its label, sentences are separated by an empty line. The exported label can be chosen in either BIO or BIOES format  .

It also gives the three accuracy indexes on overall full level and boundary level in the end. • INLINEFORM0 Content comparison: this function gives the detailed comparison of two annotated files in whole content. It highlights the annotated parts of two annotators and assigns different colors for the agreed and disagreed span.

Section 3 describes the architecture of Yedda and its detail functions. Section 4 shows the efficiency comparison results of different annotation tools. Finally, Section 5 concludes this paper and give the future plans.

Question: what dataset was used in their experiment?
Answer: CoNLL 2003 English NER