A Web-based Tool for the
Integrated Annotation of Semantic and Syntactic Structures
Richard Eckart de Castilho† and Éva Mújdricza-Maydt∗and Seid Muhie Yimam‡ and Silvana Hartmann†and Iryna Gurevych† and Anette Frank∗ and Chris Biemann‡† Ubiquitous Knowledge Processing Lab‡ FG Language Technology

Department of Computer Science
Technische Universität Darmstadt

Department of Computer Science
Technische Universität Darmstadt∗ Department of Research Training Group AIPHES

Computational Linguistics
Heidelberg University

Heidelberg University andTechnische Universität Darmstadt

Abstract
We introduce the third major release of WebAnno, a generic web-based annotation tool for
distributed teams. New features in this release focus on semantic annotation tasks (e.g. semanticrole labelling or event annotation) and allow the tight integration of semantic annotations withsyntactic annotations. In particular, we introduce the concept of slot features, a novel constraint
mechanism that allows modelling the interaction between semantic and syntactic annotations,as well as a new annotation user interface. The new features were developed and used in anannotation project for semantic roles on German texts. The paper briefly introduces this projectand reports on experiences performing annotations with the new tool. On a comparative evaluation,
our tool reaches significant speedups over WebAnno 2 for a semantic annotation task.

1

Introduction

As natural language processing pushes towards natural language understanding, i.e. the ability for amachine to process the meaning of language rather than just its structure, there is a growing need forcorpora analysed on the semantic level. Semantic structures pose different requirements for annotation
tools than morphological or syntactic annotation. For example, the rich sets of semantic categories used intasks such as semantic role labelling (SRL) or event annotation require special support to avoid the choicebecoming a burden to the annotator. Also, due to the interaction of semantics with other levels of linguistic
analysis, particularly syntax, it is desirable to work with generic annotation tools that simultaneously
support multiple levels of annotation.
Generic web-based annotation tools do not sufficiently support the annotation of semantic structureswith their rich tagsets. Also, specialised annotation tools, in particular for SRL, are not flexibly adaptableto other annotation schemes, and many of them are technologically outdated.
WebAnno 31 is the third major release of the web-based annotation tool WebAnno (Yimam et al., 2013;
Yimam et al., 2014) introducing new functionalities enabling the annotation of semantic structures:
1. Slot-features allow the appropriate modelling of predicate-argument structures for SRL. We also
support the following additional semantic annotation types: participants and circumstances for event
annotation, n-ary relations for relation extraction, and slot-filling tasks for information extraction.
2. Constraints help annotators by performing a context-sensitive filtering of the rich semantic tagsets.
For example, the sense of a semantic predicate determines available argument roles. Such a filtering isnecessary to avoid loosing valuable time by having annotators search through a large number of tags
or to manually type in tags. Constraint rules can be defined manually or they can be automaticallyThis work is licensed under a Creative Commons Attribution 4.0 International Licence. License details: http://creativecommons.
org/licenses/by/4.0/
1 https://webanno.github.io/webanno/

76
Proceedings of the Workshop on Language Technology Resources and Tools for Digital Humanities (LT4DH),
pages 76–84, Osaka, Japan, December 11-17 2016.

generated, e.g. from machine-readable lexical resources. To our knowledge, there is no otherweb-based annotation tool offering a comparable functionality.
3. An improved annotation interface for a streamlined annotation process using a permanently visible
sidebar instead of a pop-up dialog for editing annotations and their features.
These new functionalities integrate well with the existing functionalities in WebAnno 2, in particular its
support for the annotation of syntactic structures, thus enabling semantic annotation in coordination withsyntactic annotation. To our knowledge, WebAnno 3 is presently the only web-based and team-oriented
annotation tool to support both, the annotation of semantic as well as syntactic structures. This includes,
but is not limited to the tools mentioned in Section 2.
WebAnno 3 was developed and implemented in close coordination with users in the context of anannotation project (cf. Mújdricza-Maydt et al. (2016)) for word sense disambiguation (WSD) and SRL on
German texts and driven by its practical requirements. SRL is the task of identifying semantic predicates,
their arguments, and assigning roles to these arguments. It is a difficult task usually performed by experts.
Examples of well-known SRL schemes motivated by different linguistic theories are FrameNet (Baker
et al., 1998), PropBank (Palmer et al., 2005), and VerbNet (Kipper Schuler, 2005). SRL annotation
is typically based on syntactic structures obtained from treebanks, such as the constituent-based Penn
Treebank (for PropBank annotation), or the German TIGER treebank for FrameNet-style annotation(Burchardt et al., 2009). An argument is typically identified by the span of its syntactic head or syntactic
constituent. For some annotation schemes (e.g. FrameNet), the task also includes WSD. In this case, thesense label typically determines the available argument slots. The example below shows an annotation
using FrameNet; the predicate ask receives the frame label Questioning (corresponding to its word sense)
and its arguments are annotated as Addressee, Speaker, Message, and Iterations:(1)

FredRole:Addressee didn’t answer.

(2)

While IRole:Speaker had askedFrame:Questioning this questionRole:Message twiceRole:Iterations before.

Note that there are multiple interdependencies between annotations involved here. E.g. the available
sense labels for the semantic predicate depend on its lemma. Also, the available argument roles depend on
the sense label. We further describe the annotation project and the application of WebAnno 3 within the
project in Section 4. While joint WSD and SRL annotations are conveniently supported and facilitatedusing constraints, they can also be performed separately and independently of one another.

2

Related Work

We briefly review presently available annotation tools that could be used for annotating semantic structures.
Since we aim to support geographically distributed annotation teams, we consider recent generic webbased annotation tools. Additionally, we examine tools from earlier semantic annotation projects that are
specialised for SRL but not web-based.
2.1

Web-based Annotation Tools

Anafora by Chen and Styler (2013) is a recent web-based annotation tool for event-like structures.
Specifically, it supports the annotation of spans and n-ary relations. Spans are anchored on text while
relations exist independently from the text and consist of slots that can be filled with spans. Annotations
are visualised using a coloured text background. Selecting a relation highlights the participating spans byplacing boxes around them. Anafora is not suited for annotation tasks that require an alignment of thesemantic structures with syntactic structures such as constituent or dependency parse trees.
brat by Stenetorp et al. (2012) is another web-based annotation tool with a focus on collaborative
annotation. The tool supports spans and n-ary relations (also called events). Annotations are visualised
as boxes and arcs above the text. Multiple annotators can simultaneously work on the same annotations
instead of being isolated from each other. However, this removes the ability to calculate inter-annotator
agreement. All annotation actions are performed through a pop-up dialog, which necessitates manyactions even for simple annotations. While in principle the support for semantic annotation in brat through77the n-ary relations is good, there is no support for guiding the user through rich semantic tagsets, e.g. byshowing only applicable tags based on the annotation’s context.
WebAnno 2 (Yimam et al., 2014), for which we present an updated version here, is a generic andflexible annotation tool for distributed teams. To visualise the text and annotations, it uses the JavaScriptbased annotation visualisation from brat.
The annotation of semantic structures can only be realised with great difficulty in WebAnno 2. It requires
the creation of a custom span layer to represent semantic predicates and arguments. An additional custom
relation layer is required to connect the arguments to the predicate. The distinction between predicates andarguments needs to be made through a feature on the span layer (e.g. semanticType=pred|arg), because
the use of two distinct span layers for predicates and their arguments is not possible. The reason is that
WebAnno 2 only allows relations to be created between annotations on the same layer. With such a setup,annotators have to take great care not to create invalid semantic structures, e.g. by linking predicates toother predicates or arguments to other arguments while the annotation guidelines only allow links between
predicates and arguments. This complicates and slows down the annotation process and requires post-hoc
consistency checks. Finally, WebAnno 2 offers no provisions to guide annotators through rich semantic
tagsets. Due to the complex setup, annotating semantic structures also requires too many user interactions(clicks), making it also a very tedious task. This problem is aggravated by an annotation dialog popping
up for each action.
2.2

Semantic Role Annotation Tools

SALTO by Burchardt et al. (2006) supports the annotation of constituency treebanks with FrameNet
categories. Once a frame for a predicate has been selected, applicable roles from FrameNet can be assigned
to nodes in the parse tree by drag-and-drop. SALTO supports discontinuous annotations, multi-token
annotations, and cross-sentence annotations. It also offers basic team management functionalities including
workload assignment and curation. However, annotators cannot correct mistakes in the underlying treebank
because the parse tree is not editable. This is problematic for automatically preprocessed input. The final
release of SALTO was in 2012.
Jubilee and Cornerstone by Choi et al. (2010) are tools for annotating PropBank. Jubilee supports theannotation of PropBank instances, while its sister tool Cornerstone allows editing the frameset XML files
that provide the annotation scheme to Jubilee. The user interface (UI) of Jubilee displays a treebank view
and allows annotating nodes in the parse tree with frameset senses and roles. Jubilee supports annotation
and adjudication of annotations in small teams. It is a Java application that stores all data on the filesystem. Thus, the annotation team needs to be able to access a common file system, which does not meet
our needs for a distributed team. Both tools appear to be no longer being developed since 2014.
2.3

Requirements of Semantic Annotation

The annotation of semantic structures imposes two main requirements on annotation tools: 1) the flexibility
to support multiple layers of annotation including syntactic and semantic layers using freely configurable
annotation schemes and 2) the ability to handle large, interdependent tagsets.
Flexible multi-layer annotation. While the usage-driven design of dedicated SRL annotation tools
allows for a very efficient annotation, users face a serious lack of flexibility when trying to combine
different annotation schemes (e.g. GermaNet senses (Hamp and Feldweg, 1997) and VerbNet roles), or
when trying to use data preprocessed in different ways (e.g. for a crowdsourcing approach, automatically
pre-annotating predicate and argument spans can be helpful, while experts may find pre-annotateddependency relations beneficial). This is not supported by current web-based annotation tools.
Handling rich annotation schemes. Tools need to specifically support rich semantic annotation
schemes—like FrameNet—with interdependent labels (i.e. sense labels determine available argument
roles). Manually typing sense and role labels is error-prone, and selecting them from a long list is
cumbersome for the annotator. The tool must guide the annotator, e.g. by filtering or reordering possible
sense labels based on the lemma and do the same for role labels based on the selected sense. E.g. SALTO
only offers frames in the spectrum of the lemma under annotation.
78

3

Semantic Annotation using WebAnno 3

To meet the requirements described above, WebAnno 3 introduces support for arbitrary semantic rolelabelling schemes by geographically distributed teams. The great challenge in building WebAnno 3 wasto identify a way of implementing support for semantic annotation not only without adversely affecting
existing features of WebAnno 2, but also by capitalising on the existing features. The data model forannotations used by WebAnno is based on feature structures, i.e. typed key-value pairs. First, ourtool adds a new type of features for use in these structures, namely slot features, that enable SRL andother slot-filling annotation tasks. Next, we introduce constraint rules to allow the modelling of tag
interdependencies. Constraint rules can be used not only to provide context-sensitive assistance forsemantic annotations, but in general for all kinds of annotations supported by WebAnno 3, including thoseinherited from WebAnno 2. Finally, an improved and streamlined user interface was introduced to enableannotation to be performed without disruptive pop-up dialogs.
3.1

Semantic Annotation using Slot Features

WebAnno provides a flexible configuration system for annotation layers supporting spans and relations.
However, as outlined in Section 2.1, the provided features are insufficient for semantic annotation.
WebAnno 3 introduces a new feature type into the feature-structure-based system—the so-called slotfeatures—to support semantic annotations like SRL. Slot features can be added to span layers and allow
one span annotation (slot owner) to link to one or more other span annotations (slot fillers). Each link isqualified with a role, which may optionally be restricted by a tagset. Slot features can be configured toallow arbitrary slot fillers or only such pertaining to a specific layer. Figure 1 illustrates the annotation ofword sense and semantic roles for ask as introduced in Example (1).
Declaration. We model the semantic predicate using a span layer SemPred with a slot feature calledarguments. A second span layer—SemArg—represents the slot fillers.2
Interaction. To use the slot feature, the user first creates the slot-owning annotation, here a SemPredannotation. Then, the user selects a role to add a new slot to the feature arguments. Selecting a span thenautomatically creates the slot filler annotation (SemArg) and completes the link.
Visualisation. The links between slot owner and slot fillers are visualised as arcs in the colour of theslot owner. A special dashed style is used to distinguish them from other types of relations.
Agreement. With the introduction of slot features, we completely reimplemented the calculation ofinter-annotator agreement. A single feature of a layer can be selected for agreement calculation (i.e.
the word sense feature or the semantic argument feature). The first step in the calculation of agreement
identifies which annotations actually need to be compared to each other. To address this, we first derivecoordinates from the annotated data (Table 1). For regular span features, a coordinate is the tuple of<layer, feature, char offsets(begin-end)> and for relation features <layer, feature, char offsets(srcBegin-srcEnd,
tgtBegin-tgtEnd)>. Then, agreement is calculated by comparing the labels assigned by each user for eachcoordinate.
We implemented two modes of calculating agreement for slot features. The first mode role-as-label
calculates agreement based on the slot’s role. Consider asked as the semantic predicate and Fred as a slot
filler. Two annotators agree if they assigned the same role to Fred, e.g. Addressee. It is a disagreement if
one annotator assigns the role Speaker to Fred and another assigns the role Addressee (cf. Table 1 d/e). In
this mode, the slot filler is part of the annotation coordinates (cf. Table 1 column extra).
In the second mode, target-as-label, agreement is calculated based on the slot filler. Two annotators
agree if they use the same slot filler for a slot with a given role. So consider again asked as the semantic
predicate and Addressee as the role to be filled. In this case, there is an agreement if two annotators choose
Fred as a slot filler and a disagreement if one annotator fills the slot with Fred and the other one with I. In
this mode, the role label is part of the annotation coordinates (cf. Table 1 column extra).
Missing annotations are very common with slot features. For example, cases d) and e) in Table 1 differ
in their coordinates, namely in the extra column which is used to discriminate between slots, but not in
2 The layer names proposed here are meant as examples motivated by our use case. In WebAnno, the name of the layers andtheir features can in general be freely defined by the annotation project’s creator.

79

Feature typea) string on spanb) string on relation*c) slot (role-as-label)d) slot (target-as-label)e) slot (target-as-label)*

User
A
A
A
A
B

layer
Lemma
Dependency
SemPredSemPredSemPredCoordinates
featurechar offsets
value
32-37 (asked)
DependencyType 26-27 (I), 32-37 (asked)
arguments
32-37 (asked)
arguments
32-37 (asked)
arguments
32-37 (asked)

extra
0-4 (Fred)
Addressee
Speaker

Label
ask
nsubj
Addressee
0-4 (Fred)
0-4 (Fred)

Table 1: Examples of coordinates and labels for different layer and feature types in Figure 1 (except *).

Figure 1: Cross-sentence annotation/curation with slot features and constraint rules.
their label. This makes them two entirely different annotations which the respective other user simply did
not annotate rather than a case of disagreement. To cope with such missing annotations, Krippendorff’s
Alpha (Krippendorff, 1970) was added to the list of supported agreement measures. This measure isconceptually able to handle missing annotations.
The level of detail on agreement calculation provided by WebAnno 2 proved to be insufficient. It is
often not clear how the agreement values related to the actual annotations. Thus, while reimplementing
the agreement calculation to support for slot features into the agreement mechanism, we also greatly
enhanced the level of detail in the UI to explain which data was (not) used for calculation and why it may
not have been possible to calculate agreement. Finally, we added the ability to export raw agreement data
as CSV files for detailed inspection and external agreement calculation.
3.2

Handling Rich Tagsets via Constraints

Semantic annotation tasks typically operate with a very large set of labels. While a typical part-of-speechtagset contains around 50 tags, FrameNet contains over 800 frame labels. We currently observe over
1,000 role labels and more than 8,000 frame-role combinations in the FrameNet lexicon. However, givena specific lemma, there is only a small set of applicable frame labels that it evokes (e.g. ask can evoke theframes Questioning and Request).
We introduce a generic mechanism that allows formalising such interdependencies as constraint rules.
80

Listing 1: Example FrameNet-style constraints
SemPred {/ / Rule 1@Lemma . v a l u e = ” a s k ” −> s e n s e I d = ” Q u e s t i o n i n g ” | s e n s e I d = ” R e q u e s t ” | s e n s e I d = ”XXX” ;/ / . . o t h e r lemmata/ / Rule 2
s e n s e I d = ” Q u e s t i o n i n g ” −>/ / core ro les
a r g u m e n t s . r o l e = ” A d d r e s s e e ” ( ! ) | a r g u m e n t s . r o l e = ” Message ” ( ! ) | a r g u m e n t s . r o l e = ” S p e a k e r ” ( ! ) |/ / non−c o r e r o l e s
a r g u m e n t s . r o l e = ” Time ” | a r g u m e n t s . r o l e = ” I t e r a t i o n s ” ;/ / . . other senses}
Dependency {g o v e r n o r . p o s . v a l u e = ”NN” & d e p e n d e n t . p o s . v a l u e = ”DET” −> d e p e n d e n c y T y p e = ”DET ” ;}Listing 2: Grammar for constraint rules/ / B a s i c s t r u c t u r e −−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−<f i l e >: : = <i m p o r t>∗ | <s c o p e>∗<s c o p e>
: : = <s h o r t L a y e r N a m e> ”{” <r u l e s e t > ”}”<r u l e s e t >: : = <r u l e >∗<i m p o r t>
: : = ” i m p o r t ” <q u a l i f i e d L a y e r N a m e>
” a s ” <s h o r t L a y e r N a m e><r u l e>: : = <conds> ”−>” <a c t i o n s > ” ; ”/ / C o n d i t i o n s −−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−<conds>: : = <cond> | <cond> ”&” <conds><cond>: : = <p a t h> ”=” <v a l u e>
<p a t h>: : = <f e a t u r e N a m e> | <s t e p> ” . ” <p a t h><s t e p>: : = <f e a t u r e N a m e> | <l a y e r S e l e c t o r ><l a y e r S e l e c t o r >: : = <l a y e r O p e r a t o r >? <s h o r t L a y e r N a m e><l a y e r O p e r a t o r >: : = ”@” / / s e l e c t a n n o t a t i o n i n l a y e r X

|
|
|
|
|
|
|
|
|
|
|
|
|
|

/ / R e s t r i c t i o n s −−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−<a c t i o n s >: : = <a c t i o n > |<a c t i o n > ” | ” <a c t i o n s ><a c t i o n >: : = <a c t i o n P a t h > ”=” <v a l u e>( ” ( ” <f l a g s > ” ) ” )<a c t i o n P a t h >: : = <f e a t u r e N a m e> |<a c t i o n P a t h > ” . ” <f e a t u r e N a m e>
<f l a g s >::= ”!” / / core role

Rules are typically defined over annotations and their labels, but may also access the annotated text. A
rule consists of a left-hand side expressing the conditions under which the rule applies and a right-handside expressing the effect of the rule. A logical conjunction of multiple conditions can be specified. If
multiple effects for a rule are specified, they are considered to be alternatives. The mechanism does not
limit the annotator’s choice. It only reorders the list of labels the users can choose from, showing first
those labels covered by constraints in a bold font, then the rest of the labels. While constraint rules can be
written manually, they can also be generated automatically from machine-readable schemes. We provide
suitable conversions from FrameNet and GermaNet (cf. Section 4).
Listing 1 shows example constraint rules for a FrameNet-like annotation style. These apply to annotations on the layer SemPred. Rule 1 reorders the choice of sense IDs for the SemPred annotation: if a
Lemma annotation with value ask exists at the same character offsets as the SemPred (Figure 1 1 ), thenthe sense IDs Questioning, Request, and XXX (other/unknown) should be offered as top labels in the sense
ID dropdown selection (Figure 1 2 ). Similar rules for other lemmata would normally follow.
Rule 2 in Listing 1 illustrates constraints on the roles of slot features. If the senseId Questioning ischosen, Addressee, Message, Speaker, Time, and Iterations are offered as top labels in the argument role
dropdown box (Figure 1 3 ). Slots for core roles (marked with “!”) are added automatically while slots
for other roles need to be added manually.
The constraint mechanism is generic. It uses a path-like notation to navigate along features pointing toother annotations (e.g. governor.pos.value), as illustrated in the dependency rule. Within conditions, apath component may also be @shortLayerName, meaning “select all annotations of the specified layer
at the same offsets as the current annotation”. We use this in Listing 1 to access information from theLemma layer. Listing 2 shows the full constraint rule grammar in a BNF-like notation.
An indicator (Figure 1 4 ) next to a feature in the editor panel shows the status of constraints. The
indicator is green if there are matching rules (e.g. there is a Lemma annotation with the value ask andthere is a rule with a condition @Lemma.value=”ask”). It is orange if there are rules that affect thefeature but none matches (e.g. there is at least one rule with a condition on the value of a Lemma but nonethat matches on the specific given value for the present annotation). Finally, it is red if there are rules, but
the feature is not restricted by a tagset. In the latter case, the rules have no effect since without a tagset,81there are no labels to be reordered.
3.3

Redesigned Annotation User Interface

The annotation user interfaces of WebAnno 2 and brat use a pop-up dialog whenever an annotation iscreated or modified. This was perceived as a major impediment to efficient annotation by our annotation
team, not only because the dialog obscures the underlying annotation interface, but mainly because thedialog needs to be closed by an explicit user action. Hence, we implemented a dialog-less annotation userinterface, similar to the one in the Anafora tool. Along with the new interface, additional user-expressedneeds for semantic annotation were addressed. These include better support for keyboard-based annotation,
the ability to create zero-length annotations to represent slot fillers not realised in the text, the display ofdescriptive tooltips explaining semantic tags, the indication of tokens transitively covered by dependency
relations, and the ability for the curation of cross-sentence annotations.
Dialog-less annotation. The UI elements used for editing the annotation feature values have been
moved from a dialog (cf. brat) to a permanently visible detail panel on the side of the screen (Figure 1 5 ).
Selecting a span of text or drawing an arc now immediately creates an annotation on the active layer
and loads its features in the detail panel. Changes to the feature values are immediately reflected in the
annotation view. For tasks that require repeatedly annotating the same feature values (e.g. annotating
persons in a named entity task), the option save feature can be enabled for each feature individually. New
annotations are then created with the same feature values as the previously selected annotation.
Keyboard-based annotation. Keyboard-based annotation can significantly speed up the annotation
process. For annotation layers with a single tagset-controlled feature (e.g. part-of-speech), we introduce a
new keyboard-based forward annotation mode (Figure 1 6 ). In this mode, pressing a key selects the firsttag starting with the respective character and pressing it again selects the next. As annotators memorise thenumber of key presses for the tags, we expect their efficiency to improve. Once a tag is chosen, pressing
space automatically selects the next token for annotation.
Zero-width annotations. Zero-width span annotations can now be created (at arbitrary positions in
the text), e.g. to instantiate semantic predicates or arguments that are not realised in the text.
Tooltips. Tag descriptions are now displayed as tooltips (Figure 1 7 ) during annotation. In particular
for rich semantic tagsets, this is an important improvement. The tag descriptions are part of the tagset
definitions which can be edited via the user interface in the project settings. Alternatively, they canbe externally generated and imported as JSON files. For the purpose of our annotation project, we
automatically derived tagset files from VerbNet and GermaNet.
Relation yield highlighting. Moving the mouse cursor over a span with outgoing relations now
displays a tooltip showing the yield of the relation (Figure 1 8 ). E.g. for dependency relations, theyield represents all the tokens subsumed by the respective dependency node. This feature facilitates
the coordination of syntactic and semantic annotations when using syntactic heads as slot fillers. In the
annotation project, it was determined that dependency-based annotation, enhanced with highlighting of
the dependency relation yield proved superior to span-based annotation, which is more error prone.
Cross-sentence curation. We extended the curation component of WebAnno to support the adjudication of multiple annotations for individual sentences. This was necessary in order to curate cross-sentence
semantic relations, e.g. to use parts of a previous sentence as slot fillers. An example of such a case isprovided in Figure 1 9 . Curators can now define the number of sentences to visualise and curate at atime. Additionally, an indicator is displayed for any relation annotations that are not rendered because oneof their endpoints is realised outside of the window of currently displayed sentences (Figure 1 10 ).

4

Experiences and Evaluation

4.1

Annotation Study

Our new tool was tested in a large-scale semantic annotation study on German standard as well as nonstandard texts (cf. Mújdricza-Maydt et al. (2016)). An annotation team performed a combination of word
sense and semantic role annotation based on GermaNet senses and VerbNet-style role labels. As basis
for the annotation, the new slot features were used to model predicate-argument structures. Compared to
82

Doc

a) WebAnno 2

1
2
3
4
All

14:58
12:21
05:19
10:40
43:18

b) WebAnno 3
no constraints
10:07
10:29
04:16
08:22
33:14

c) WebAnno 3
constraints
08:57
08:16
03:47
07:22
28:22

Table 2: Annotation times in minutes:seconds for WebAnno 2 and 3 with and without constraints.
earlier experiences with predicate-argument annotation using relation layers in WebAnno 2, the annotationteam reported improved comfort and efficiency with the new interface. Having the annotation detailpanel visible all the time and no longer being forced to constantly switch between annotation dialog
and annotation view was perceived as an important factor for this improvement. This study also madeextensive use of constraint rules and relied heavily on the new tooltips explaining senses and roles to theannotators. In a pilot study for the project, different strategies of selecting slot fillers were examined.
The strategy that was eventually selected relied on pre-annotated dependency relations and required theannotator to select the syntactic head of a phrase to be used as a slot filler (e.g. house instead of the house).
The new ability to display the yield of the dependency relations proved to be instrumental in this task tolocate the correct head words.
To facilitate SRL annotation projects for others, we provide in addition to the tool itself 1) the setup for
our annotation study as a demo project; 2) configuration files for different predicate sense and semanticrole labeling frameworks (VerbNet, FrameNet, and PropBank), including their respective tagsets andconstraint rules.
4.2

Comparative Evaluation

After the development has been finalised, we conducted a comparative evaluation study in order tomeasure whether our efforts in tool engineering translate into annotation time speedups, which directly
influences annotation cost. For this, we chose the following setup: an annotator was presented with a static
display of gold standard predicate-argument annotations like the ones shown in Figure 1. In this way,
we minimise fluctuations caused by cognitive load for choosing the right labels, simulating a maximallytrained annotator. The annotator had ample experience with both the annotation tools, but not with thespecific settings. Thus, for each setting, the annotator became familiar with the setup on a small training
document. We conducted this annotation in three different settings: a) using WebAnno 2 (version 2.3.1),
modelling the annotation task as outlined in Section 2.1, b) using WebAnno 3 without constraints andc) using WebAnno 3 with constraints for rules that re-order semantic predicates according to the lemmavalue, as exemplified in (Figure 1 4 ). To avoid interpersonal differences in annotation speed, the sameannotator worked on all three settings. The data set on which time was measured included four documentswith a total of 64 semantic predicates, filling a total of 115 slots with overall 94 semantic arguments(SemArg in Figure 1). Note that the same semantic argument can fill slots of different predicates.
Table 2 shows the times in minutes and seconds measured for the four documents. The improvements of
the user interface in setting b) already results in a speedup of about 23% less time compared to WebAnno 2.
In combination with constraints, a speedup of over 34% is reached in setting c), meaning that the sameannotator can produce 50% more annotations in the same time. The annotator reported that she needed toperform considerably fewer interactions in setting c).

5

Conclusion

We have introduced WebAnno 3,3 the third major release of WebAnno. The new version introduces
major improvements that significantly go beyond the state-of-the-art: they enable the annotation ofsemantic structure and the handling of rich semantic tagsets. The new version was developed in close
coordination with an associated annotation project performing combined WSD and SRL annotation on
German texts. Moreover, the tool can be used for many non-linguistic annotation tasks in various domains,
3 https://webanno.github.io/webanno/

83

e.g. the annotation of metaphors and ambiguities (Digital Humanities), skills and qualifications (Human
Resources), sentiment in product reviews (Advertisement/Marketing), etc. A comparative evaluation
showed considerable speedups over previous tools and validates both, our adaptation to the user interface
and the incorporation of the constraint language to facilitate the annotation of large, interdependent tagsets.

Acknowledgments
The work presented in this paper was funded by a German BMBF grant to the CLARIN-D project, by theGerman Research Foundation as part of the Research Training Group Adaptive Preparation of Informationfrom Heterogeneous Sources (AIPHES) under grant No. GRK 1994/1, and by the Volkswagen Foundationas part of the Lichtenberg-Professorship Program under grant No. I/82806. We also thank Aakash Sharma
and Sarah Holschneider for their active support in the development of WebAnno, the documentation of
the tool, and for conducting the comparative evaluation study.

References
Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. The Berkeley FrameNet project. In Proc. of
COLING-ACL 1998, pages 86–90, Montreal, Canada.
Aljoscha Burchardt, Katrin Erk, Anette Frank, Andreas Kowalski, and Sebastian Padó. 2006. SALTO – A Versatile
Multi-Level Annotation Tool. In Proc. of LREC 2006, pages 517–520, Genoa, Italy.
Aljoscha Burchardt, Katrin Erk, Anette Frank, Andrea Kowalski, Sebastian Padó, and Manfred Pinkal. 2009.
Using FrameNet for the Semantic Analysis of German: Annotation, Representation and Automation. In Hans C.
Boas, editor, Multilingual FrameNets – Practice and Applications, pages 209–244. Mouton de Gruyter, Berlin,
Germany.
Wei-Te Chen and Will Styler. 2013. Anafora: A web-based general purpose annotation tool. In Proc. of the
NAACL HLT 2013 Demonstration Session, pages 14–19, Atlanta, GA, USA.
Jinho Choi, Claire Bonial, and Martha Palmer. 2010. Multilingual PropBank annotation tools: Cornerstone andJubilee. In Proc. of the NAACL HLT 2010 Demonstration Session, pages 13–16, Los Angeles, CA, USA.
Birgit Hamp and Helmut Feldweg. 1997. GermaNet - a lexical-semantic net for German. In Automatic Information Extraction and Building of Lexical Semantic Resources for NLP Applications, pages 9–15, Madrid, Spain.
Karin Kipper Schuler. 2005. VerbNet: A Broad-coverage, Comprehensive Verb Lexicon. Ph.D. thesis, Universityof Pennsylvania, Philadelphia, PA, USA.
Klaus Krippendorff. 1970. Estimating the reliability, systematic error and random error of interval data. Educational. and Psych. Measurement, 30(1):61–70.
Éva Mújdricza-Maydt, Silvana Hartmann, Iryna Gurevych, and Anette Frank. 2016. Combining Semantic Annotation of Word Sense & Semantic Roles: A Novel Annotation Scheme for VerbNet Roles on German Language
Data. In Proc. of LREC 2016, pages 3031–3038.
Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The Proposition Bank: An annotated corpus of semanticroles. Computational Linguistics, 31(1):71–105.
Pontus Stenetorp, Sampo Pyysalo, Goran Topić, Tomoko Ohta, Sophia Ananiadou, and Jun’ichi Tsujii. 2012. brat:a web-based tool for NLP-assisted text annotation. In Proc. of EACL 2012, pages 102–107, Avignon, France.
Seid Muhie Yimam, Iryna Gurevych, Richard Eckart de Castilho, and Chris Biemann. 2013. WebAnno: A
flexible, web-based and visually supported system for distributed annotations. In Proc. of ACL 2013: System
Demonstrations, pages 1–6, Sofia, Bulgaria.
Seid Muhie Yimam, Richard Eckart de Castilho, Iryna Gurevych, and Chris Biemann. 2014. Automatic annotationsuggestions and custom annotation layers in WebAnno. In Proc. of ACL 2014: System Demonstrations, pages
91–96, Baltimore, MD, USA.

84