Introduction
Natural Language Processing (NLP) systems rely on large-scale training data BIBREF0 for supervised training. However, manual annotation can be time-consuming and expensive. Despite detailed annotation standards and rules, inter-annotator disagreement is inevitable because of human mistakes, language phenomena which are not covered by the annotation rules and the ambiguity of language itself BIBREF1 .
Existing annotation tools BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 mainly focus on providing a visual interface for user annotation process but rarely consider the post-annotation quality analysis, which is necessary due to the inter-annotator disagreement. In addition to the annotation quality, efficiency is also critical in large-scale annotation task, while being relatively less addressed in existing annotation tools BIBREF6 , BIBREF7 . Besides, many tools BIBREF6 , BIBREF4 require a complex system configuration on either local device or server, which is not friendly to new users.
To address the challenges above, we propose Yedda , a lightweight and efficient annotation tool for text span annotation. A snapshot is shown in Figure FIGREF4 . Here text span boundaries are selected and assigned with a label, which can be useful for Named Entity Recognition (NER) BIBREF8 , word segmentation BIBREF9 , chunking BIBREF10 ,etc. To keep annotation efficient and accurate, Yedda provides systematic solutions across the whole annotation process, which includes the shortcut annotation, batch annotation with a command line, intelligent recommendation, format exporting and administrator evaluation/analysis.
Figure FIGREF1 shows the general framework of Yedda. It offers annotators with a simple and efficient Graphical User Interface (GUI) to annotate raw text. For the administrator, it provides two useful toolkits to evaluate multi-annotated text and generate detailed comparison report for annotator pair. Yedda has the advantages of being:
• INLINEFORM0 Convenient: it is lightweight with an intuitive interface and does not rely on specific operating systems or pre-installed packages.
• INLINEFORM0 Efficient: it supports both shortcut and command line annotation models to accelerate the annotating process.
• INLINEFORM0 Intelligent: it offers user with real-time system suggestions to avoid duplicated annotation.
• INLINEFORM0 Comprehensive: it integrates useful toolkits to give the statistical index of analyzing multi-user annotation results and generate detailed content comparison for annotation pairs.
This paper is organized as follows: Section 2 gives an overview of previous text annotation tools and the comparison with ours. Section 3 describes the architecture of Yedda and its detail functions. Section 4 shows the efficiency comparison results of different annotation tools. Finally, Section 5 concludes this paper and give the future plans.
Related Work
There exists a range of text span annotation tools which focus on different aspects of the annotation process. Stanford manual annotation tool is a lightweight tool but does not support result analysis and system recommendation. Knowtator BIBREF6 is a general-task annotation tool which links to a biomedical onto ontology to help identify named entities and relations. It supports quality control during the annotation process by integrating simple inter-annotator evaluation, while it cannot figure out the detailed disagreed labels. WordFreak BIBREF3 adds a system recommendation function and integrates active learning to rank the unannotated sentences based on the recommend confidence, while the post-annotation analysis is not supported.
Web-based annotation tools have been developed to build operating system independent annotation environments. Gate BIBREF11 includes a web-based with collaborative annotation framework which allows users to work collaboratively by annotating online with shared text storage. Brat BIBREF7 is another web-based tool, which has been widely used in recent years, it provides powerful annotation functions and rich visualization ability, while it does not integrate the result analysis function. Anafora BIBREF4 and Atomic BIBREF5 are also web-based and lightweight annotation tools, while they don't support the automatic annotation and quality analysis either. WebAnno BIBREF12 , BIBREF13 supports both the automatic annotation suggestion and annotation quality monitoring such as inter-annotator agreement measurement, data curation, and progress monitoring. It compares the annotation disagreements only for each sentence and shows the comparison within the interface, while our system can generate a detailed disagreement report in .pdf file through the whole annotated content. Besides, those web-based annotation tools need to build a server through complex configurations and some of the servers cannot be deployed on Windows systems.
The differences between Yedda and related work are summarised in Table TABREF2 . Here “Self Consistency” represents whether the tool works independently or it relies on pre-installed packages. Compared to these tools, Yedda provides a lighter but more systematic choice with more flexibility, efficiency and less dependence on system environment for text span annotation. Besides, Yedda offers administrator useful toolkits for evaluating the annotation quality and analyze the detailed disagreements within annotators.
Yedda
Yedda is developed based on standard Python GUI library Tkinter, and hence needs only Python installation as a prerequisite and is compatible with all Operating System (OS) platforms with Python installation. It offers two user-friendly interfaces for annotators and administrator, respectively, which are introduced in detail in Section SECREF9 and Section SECREF19 , respectively.
Annotator Client
The client is designed to accelerate the annotation process as much as possible. It supports shortcut annotation to reduce the user operation time. Command line annotation is designed to annotate multi-span in batch. In addition, the client provides system recommendations to lessen the workload of duplicated span annotation.
Figure FIGREF4 shows the interface of annotator client on an English entity annotation file. The interface consists of 5 parts. The working area in the up-left which shows the texts with different colors (blue: annotated entities, green: recommended entities and orange: selected text span). The entry at the bottom is the command line which accepts annotation command. There are several control buttons in the middle of the interface, which are used to set annotation model. The status area is below the control buttons, it shows the cursor position and the status of recommending model. The right side shows the shortcut map, where shortcut key “a” or “ INLINEFORM0 ” means annotating the text span with “Artificial” type and the same for other shortcut keys. The shortcut map can be configured easily. Details are introduced as follows.
Yedda provides the function of annotating text span by selecting using mouse and press shortcut key to map the selection into a specific label. It is a common annotation process in many annotation tools BIBREF7 , BIBREF11 . It binds each label with one custom shortcut key, this is shown in the “Shortcuts map Labels” part of Figure FIGREF4 . The annotator needs only two steps to annotate one text span, i.e. “select and press”. The annotated file updates simultaneously with each key pressing process.
Yedda also support the command line annotation function (see the command entry in the bottom of Figure FIGREF4 ) which can execute multi-span annotation at once. The system will parse the command automatically and convert the command into multi-span annotation instructions and execute in batch. It is quite efficient for the tasks of character-based languages (such as Chinese and Japanese) with high entity density. The command follows a simple rule which is INLINEFORM0 , where ` INLINEFORM1 ' are the length of the entities and ` INLINEFORM2 ' is the corresponding shortcut key. For example, command “ INLINEFORM3 ” represents annotating following 2 characters as label ` INLINEFORM4 ' (mapped into a specific label name), the following 3 characters as label ` INLINEFORM5 ' and 2 characters further as label ` INLINEFORM6 '.
It has been shown that using pre-annotated text and manual correction increases the annotation efficiency in many annotation tasks BIBREF14 , BIBREF7 . Yedda offers annotators with system recommendation based on the existing annotation history. The current recommendation system incrementally collects annotated text spans from sentences that have been labeled, thus gaining a dynamically growing lexicon. Using the lexicon, the system automatically annotates sentences that are currently being annotated by leveraging the forward maximum matching algorithm. The automatically suggested text spans and their types are returned with colors in the user interface, as shown in green in Figure FIGREF4 . Annotators can use the shortcut to confirm, correct or veto the suggestions. The recommending system keeps online updating during the whole annotation process, which learns the up-to-date and in-domain annotation information. The recommending system is designed as “pluggable” which ensures that the recommending algorithm can be easily extended into other sequence labeling models, such as Conditional Random Field (CRF) BIBREF15 . The recommendation can be controlled through two buttons “RMOn” and “RMOff”, which enables and disables the recommending function, respectively.
It is inevitable that the annotator or the recommending system gives incorrect annotations or suggestions. Based on our annotation experience, we found that the time cost of annotation correction cannot be neglected. Therefore, Yedda provides several efficient modification actions to revise the annotation:
• INLINEFORM0 Action withdraw: annotators can cancel their previous action and let system return to the last status by press the shortcut key Ctrl+z.
• INLINEFORM0 Span label modification: if the selected span has the correct boundary but receives an incorrect label, annotator only needs to put the cursor inside the span (or select the span) and press the shortcut key of the right label to correct label.
• INLINEFORM0 Label deletion: similar to the label modification, the annotator can put the cursor inside the span and press shortcut key q to remove the annotated (recommended) label.
As the annotated file is saved in .ann format, Yedda provides the “Export” function which exports the annotated text as standard format (ended with .anns). Each line includes one word/character and its label, sentences are separated by an empty line. The exported label can be chosen in either BIO or BIOES format BIBREF16 .
Administrator Toolkits
For the administrator, it is important and necessary to evaluate the quality of annotated files and analyze the detailed disagreements of different annotators. Shown in Figure FIGREF13 , Yedda provides a simple interface with several toolkits for administrator monitoring the annotation process.
To evaluate and monitor the annotation quality of different annotators, our Multi-Annotator Analysis (MAA) toolkit imports all the annotated files and gives the analysis results in a matrix. As shown in Figure FIGREF16 , the matrix gives the F1-scores in full level (consider both boundary and label accuracy) and boundary level (ignore the label correctness, only care about the boundary accuracy) of all annotator pairs.
If an administrator wants to look into the detailed disagreement of annotators, it is quite convenient by using the Pairwise Annotators Comparison (PAC). PAC loads two annotated files and generates a specific comparison report file for the two annotators. As shown in Figure FIGREF21 , the report is mainly in two parts:
• INLINEFORM0 Overall statistics: it shows the specific precision, recall and F1-score of two files in all labels. It also gives the three accuracy indexes on overall full level and boundary level in the end.
• INLINEFORM0 Content comparison: this function gives the detailed comparison of two annotated files in whole content. It highlights the annotated parts of two annotators and assigns different colors for the agreed and disagreed span.
Experiments
Here we compare the efficiency of our system with four widely used annotation tools. We extract 100 sentences from CoNLL 2003 English NER BIBREF8 training data, with each sentence containing at least 4 entities. Two undergraduate students without any experience on those tools are invited to annotate those sentences. Their average annotation time is shown in Figure FIGREF25 , where “Yedda+R” suggests annotation using Yedda with the help of system recommendation. The inter-annotator agreements for those tools are closed, which around 96.1% F1-score. As we can see from the figure, our Yedda system can greatly reduce the annotation time. With the help of system recommendation, the annotation time can be further reduced. We notice that “Yedda+R” has larger advantage with the increasing numbers of annotated sentences, this is because the system recommendation gives better suggestions when it learns larger annotated sentences. The “Yedda+R” gives 16.47% time reduction in annotating 100 sentences.
Conclusion and Future Work
We have presented a lightweight but systematic annotation tool, Yedda, for annotating the entities in text and analyzing the annotation results efficiently. In order to reduce the workload of annotators, we are going to integrate active learning strategy in our system recommendation part in the future. A supervised sequence labeling model (such as CRF) is trained based on the annotated text, then unannotated sentences with less confidence (predicted by this model) are reordered in the front to ensure annotators only annotate the most confusing sentences.
Acknowledgements
We thank Yanxia Qin, Hongmin Wang, Shaolei Wang, Jiangming Liu, Yuze Gao, Ye Yuan, Lu Cao, Yumin Zhou and other members of SUTDNLP group for their trials and feedbacks. Yue Zhang is the corresponding author. Jie is supported by the YEDDA grant 52YD1314.

Question: how many sentences did they annotate?
Answer: 100 sentences

100 sentences

Question: what dataset was used in their experiment?
Answer: CoNLL 2003 English NER

CoNLL 2003 English NER BIBREF8

Question: what are the existing annotation tools?
Answer: BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6, BIBREF7

existing annotation tools BIBREF6 , BIBREF7