Question: What systematic solutions were provided by Yedda across the whole annotation process?
Evidence: Natural Language Processing (NLP) systems rely on large-scale training data BIBREF0 for supervised training. However, manual annotation can be time-consuming and expensive. Despite detailed annotation standards and rules, inter-annotator disagreement is inevitable because of human mistakes, language phenomena which are not covered by the annotation rules and the ambiguity of language itself BIBREF1 .
Existing annotation tools BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 mainly focus on providing a visual interface for user annotation process but rarely consider the post-annotation quality analysis, which is necessary due to the inter-annotator disagreement. In addition to the annotation quality, efficiency is also critical in large-scale annotation task, while being relatively less addressed in existing annotation tools BIBREF6 , BIBREF7 . Besides, many tools BIBREF6 , BIBREF4 require a complex system configuration on either local device or server, which is not friendly to new users.
To address the challenges above, we propose Yedda , a lightweight and efficient annotation tool for text span annotation. A snapshot is shown in Figure FIGREF4 . Here text span boundaries are selected and assigned with a label, which can be useful for Named Entity Recognition (NER) BIBREF8 , word segmentation BIBREF9 , chunking BIBREF10 ,etc. To keep annotation efficient and accurate, Yedda provides systematic solutions across the whole annotation process, which includes the shortcut annotation, batch annotation with a command line, intelligent recommendation, format exporting and administrator evaluation/analysis.
Figure FIGREF1 shows the general framework of Yedda. It offers annotators with a simple and efficient Graphical User Interface (GUI) to annotate raw text. For the administrator, it provides two useful toolkits to evaluate multi-annotated text and generate detailed comparison report for annotator pair. Yedda has the advantages of being:
• INLINEFORM0 Convenient: it is lightweight with an intuitive interface and does not rely on specific operating systems or pre-installed packages.
• INLINEFORM0 Efficient: it supports both shortcut and command line annotation models to accelerate the annotating process.
• INLINEFORM0 Intelligent: it offers user with real-time system suggestions to avoid duplicated annotation.
• INLINEFORM0 Comprehensive: it integrates useful toolkits to give the statistical index of analyzing multi-user annotation results and generate detailed content comparison for annotation pairs.
This paper is organized as follows: Section 2 gives an overview of previous text annotation tools and the comparison with ours. Section 3 describes the architecture of Yedda and its detail functions. Section 4 shows the efficiency comparison results of different annotation tools. Finally, Section 5 concludes this paper and give the future plans.
To keep annotation efficient and accurate, YEDDA provides systematic solutions across the whole annotation process, which includes the shortcut annotation, batch annotation with a command line, intelligent recommendation, format exporting and administrator evaluation/analysis.
Answer: Includes the shortcut annotation, batch annotation with a command line, intelligent recommendation, format exporting and administrator evaluation/analysis.