Introduction
Among the several senses that The Oxford English Dictionary, the most venerable dictionary of English, provides for the word event, are the following.
Although an event may refer to anything that happens, we are usually interested in occurrences that are of some importance. We want to extract such events from textual documents. In order to extract important events or events of a specific type, it is likely that we have to identify all events in a document to start with.
Consider the first paragraphs of the article on the Battle of Fredericksburg in the English Wikipedia, accessed on May 5, 2012. We have highlighted the “events" in the paragraph.
The Battle of Fredericksburg was fought December 11–15, 1862, in and around Fredericksburg, Virginia, between General Robert E. Lee's Confederate Army of Northern Virginia and the Union Army of the Potomac, commanded by Maj. Gen. Ambrose E. Burnside. The Union army's futile frontal assaults on December 13 against entrenched Confederate defenders on the heights behind the city is remembered as one of the most one-sided battles of the American Civil War, with Union casualties more than twice as heavy as those suffered by the Confederates.
The paragraph contains two fairly long sentences with several “events", mentioned using the following words: fought, commanded, assaults, entrenched, remembered, casualties and suffered. Some of these “events" are described in terms of verbs whereas the others are in terms of nouns. Here fought, commanded, assaults, battles definitely seem to be “events" that have durations or are durative. Entrenched seems to talk about a state, whereas it is possible that suffered talks about something punctual (i.e., takes a moment or point of time) or can be durative (i.e., takes a longer period of time) as well. The act of remembering by an individual is usually considered to happen momentarily, i.e., forgotten things come back to mind at an instant of time. But, in this paragraph it is given in passive voice and hence, it is unclear who the actor is, possibly a lot different people at different points of time.
Thus, depending on who is asked, the “events” picked out may be slightly different, but the essence is that there are several events mentioned in the paragraph and the objective in event extraction is to extract as many of them as possible in an automated fashion. For example, someone may not pick out remembered as an event that took place. Some others may not want to say that entrenched is an event. In addition, if one is asked to pick an important event, responses may vary from person to person. Finally, if one is asked to summarize the paragraph, depending on the person asked the summary may vary. A summary prepared by the author of this article is given below.
The Battle of Fredericksburg, fought December 11-12, 1862, was one of the most one-sided battles of the American Civil War, with heavy Union casualties.
Obviously, there are many other possibilities for summarization. However, the idea is that identification of events and their participants may play a significant role in summarizing a document.
This paper discusses the extraction of events and their attributes from unstructured English text. It is an survey of research in extracting event descriptions from textual documents. In addition, we discuss how the idea of event extraction can be used in application domains such as summarization of a document. We also discuss application of event extraction in the biomedical domain and in the context of Twitter messages.
The rest of the paper is organized in the following manner. Section "Events in Linguistics and Philosophy" provides a description of research in linguistics and philosophy. The author believes that such a background, at least at a basic level, is necessary to understand and develop the approaches and algorithms for automatic computational detection and extraction of events and their participants from textual documents. Section "Extracting Events from Textual Documents" discusses approaches used in extracting events from textual documents. Most approaches these days use machine learning techniques.
Events in Linguistics and Philosophy
Real world events are things that take place or happen. In this section, we present an overview of how real events are represented in terms of language. In particular, we discuss classification of events and features necessary for such classification. We follow this by presenting the preferred way among philosophers to represent events in terms of logic. We bring this section to an end by presenting some of the structures ascribed to events by linguists or philosophers working at an abstract level.
The reason for the inclusion of this section in the paper is to set the context for the discussions in the following sections on the practical task of extracting events. Practical systems do not usually follow linguistic niceties although they draw inspiration from linguistics or philosophy.
Classifying Events
There have been many attempts at classifying linguistic events. Below, we briefly discuss a few. The primary focus when linguists discuss events is on the verb present in a sentence. Nouns, adjectives and other elements present in a sentence provide arguments for the verb.
Aristotle (as presented in BIBREF0 ) classified verbs that denote something happening into three classes: actuality, movement and action. An actuality represents the existence of a thing or things; this is called state by others (e.g., BIBREF1 ). An examples of actuality can be seen in the sentence Jon is ill. A movement is an incomplete process or something that takes time but doesn't have an inherent end. An example of movement is seen in the sentence Jon is running. An action is something that takes time and has an inherent end. An example of an action is present in the sentence Jon is building a house. In other words, Aristotle distinguished between states and events and then events.
BIBREF2 lists verbs that belong to the three Aristotelian classes and develops membership criteria for the classes. Kenny renamed the classes as states, activities (actions without inherent end) and performances (actions with inherent ends). Kenny's membership criteria are based on semantic entailments about whether the event can be considered to have taken place when it is still in progress. For example, during any point when we say Jon is running, we can consider that the activity of running has taken place. In other words Jon is running entails Jon has run. Thus, run is an activity. In contrast, when we say Joh is taking the final, we cannot say that Jon has taken the final. In other words, the first does not entail the second. Thus, the main difference between an activity and a performance is what is called delimitation. A delimited event has a natural end.
BIBREF3 developed a 4-way classification scheme for linguistic events and BIBREF4 developed a set of criteria for membership in the classes. The classes enumerated by Dowty are: states, activities, achievements and accomplishments. The definitions are given below.
Activities: Events that take place over a certain period of time, but do not necessarily have a fixed termination point. Examples; Jon walked for an hour, and Jon is driving the car.
Accomplishments: Events that happen over a certain period of time and then end. Examples: Jon built a house in a month, and Jon is taking the final.
Achievements: These are events that occur instantaneously and lack continuous tenses. Examples: Jon finished the final in 45 minutes and The vase broke.
States: These are non-actions that hold for a certain period of time, but lack continuous tenses. Examples: Jon knows the answer and Jon likes Mary.
BIBREF5 adopts the same classification as Vendler and Dowty, but divides achievements into two classes. The first one is still called achievements, but the second one is called semelfactives. In this new scheme, achievements are instantaneous (that is, the beginning of the event is the same as its end) culminating events, but semelfactives are events with no duration that result in no change of state. An example of a semelfactive is: Jon knocked on the door.
Table 1 presents the nomenclatures introduced by various linguists in one place. There are many variations of the schemes given here, although we do not discuss them in this paper.
In the early work on event classification, Aristotle, Vendler and others assume that what needs to be classified is the verb. However, many have concluded that it is impossible to classify a verb into a specific class. It is more appropriate to say that a clause containing an event has a class, and the classification of such a clause depends not only upon the verb, but also on other material present in the clause BIBREF6 , BIBREF4 , BIBREF7 , BIBREF8 . In other words, the classification must be compositional or must depend on various features of the clause, not exclusively verb-based. There is also substantial evidence that sentence material other than the verb can change the overall event type. For example, addition of a direct object can change an activity to an accomplishment BIBREF1 , as in the following examples.
Bill ran for five minutes/*in five minutes: activity
Bill ran the mile *for 5 minutes/in 5 minutes: accomplishment
Parameters of Event Classes
Many authors in linguistics have delved deeper into the nature of event classes and have tried to come up with features or characteristics that can be used to identify whether something (verb or a clause) belongs to a particular event class or not. These features or characteristics are necessary to describe the structure of events in a theoretical sense. Description of event structure usually refers to the actual words used (lexical features or characteristics) and also the structure of clause or sentence (syntactic features or characteristics). Identification of such features may be described as finding parameters of event types or parameterization of event types.
A lot of the work on parameterization of event types/classes use the classes espoused by Vendler. These include BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 , BIBREF15 and others. We will only briefly touch upon such work in this paper. Our objective is to impress upon the reader that identification of features of event classes is considered an important task by linguists.
For example, BIBREF9 describes Vendler's classes with two binary features or parameters: continuousness: whether an event has duration, and boundedness: whether an event has a (natural) terminal point or endpoint. Using these two features, the four Vendler classes can be parameterized as follows.
: -bounded, -continuous
: -bounded, +continuous
: +bounded, -continuous
: +bounded, +continuous
BIBREF12 , BIBREF13 introduce the notion of countability while discussing event classes. This is similar to the mass-count opposition in nouns. Terminating events can be counted, but non-terminating processes cannot. Hoeksema introduces two binary features: count and duration to obtain Vendler's classes as seen below. The feature duration refers to whether the event takes place over time.
: -count, -duration
: -count, +duration
: +count, -duration
: +count, +duration
BIBREF11 refines Vendler's classes by adding a class much like Smith's semelfactives BIBREF5 . He suggests that, in addition to states, there are four event types: culmination, culminated process, point, and process. He uses two binary features or parameters: consequence identifying termination or culmination, and atomic or non-atomic (which Moens called extended). Atomic is also called momentous or pointed. Moen's classification is given below, along with the features and examples.
: +consequence, +atomic (examples: recognize, win the race)
: +consequence, -atomic (examples: build a house)
: -consequence, +atomic (example: hiccup, tap, wink)
: -consequence, -atomic (example: run, swim, play the piano)
: (examples: understand, love, resemble)
Moens also claims that culminated process is an event class whose members are made up of smaller atomic units. In particular, a culminated process is a process with a consequent state. This insight that events can be decomposed into sub-events was used later by others working on the lexical analysis of events e.g., BIBREF16 , BIBREF17 . Others such as BIBREF18 , BIBREF19 , BIBREF20 have claimed that arguments of verbs are related to sub-events.
We summarize the various features that linguists have used to classify events in Table 2 . Of course, we do not discuss many other proposals for features in this brief discussion.
Classification of events and their parameterization of verbs or predicates (or clauses) are only the first steps in developing a deeper linguistic understanding of events. In particular, in order to understand the linguistic representation of events, linguists need to go beyond classification schemes.
Events in Logical Representation of Semantics
Mathematical logic is used to represent the semantics of language. In particular, we use logic to represent the meaning of single sentences. Early work on events, e.g., Panini (as discussed by BIBREF21 and BIBREF22 ) stated that language encodes two kinds of information–actions and non-actions. Verbs represent actions and nouns represent non-actions or things.
BIBREF23 proposes that one needs an event variable $e$ to represent events in mathematical logic. This variable $e$ is used to represent relations represented by the event denoted by the verb and other constituents in the sentence, such as modifiers. Davidson claims that logically speaking, events are like things in that they can be represented by a variable and this variable can be modified and quantified. A question that arises is: how many arguments should an event predicate (in logic) take BIBREF2 ? Just like nominal modifiers modify nouns, event modifiers can modify event predicates. An event predicate can take any number of modifiers just like noun (nominal) modifiers. Examples of event modifiers are: time, place, manner and instrument. Davidson proposed that an event predicate may take one or more required arguments (is this true?) and any number of adjuncts or optional modifiers. Consider the following examples from BIBREF23 . The English sentence and the corresponding logical representation or logical form is given for each example.
John buttered the toast.
$\exists e \; buttered (Jones, the\_toast, e)$
John buttered the toast slowly.
$\exists e \; buttered (Jones, the\_toast, e) \wedge slowly (e)$
John buttered the toast slowly, in the bathroom.
$\exists e \; buttered (Jones, the\_toast, e) \wedge slowly (e) \wedge in\_the\_bathroom (e)$
John buttered the toast slowly, in the bathroom, with a knife.
$\exists e \; buttered (Jones, the\_toast, e) \wedge slowly (e) \wedge in\_the\_bathroom (e) \wedge with\_a\_knife (e)$
John buttered the toast slowly, in the bathroom, with a knife, at midnight.
$\exists e \; buttered (Jones, the\_toast, e) \wedge slowly (e) \wedge in\_the\_bathroom (e) \wedge with\_a\_knife (e) \wedge at\_midnight (e)$
Thus we can see that Davidson's approach places the event variable $e$ in the main predicate of a clause and distributes it among the modifiers of the clause in logical representation. In writing the meaning in Davidsonian logic, the author creates predicates such as $the\_toast$ and $in\_the\_bathroom$ , just for illustration, without going into details.
Davidsonian representation allows events to be represented in logic (logical semantics) without requiring verbs to have multiple arities, i.e., without taking different arguments in different situations. Because the event is represented as a variable, the event variable $e$ can be included in the representation of logical meaning of each modifier or adjunct. Another benefit is that using Davidson's representation, one can analyze events represented syntactically as nouns (nominals) or verbs BIBREF21 . For example, one can refer to an event using the verb to burn or the noun a burn. Parsons also observes that using a variable to represent an event allows quantification over events the same way quantification applies to things. The following examples are from BIBREF21 .
In every burning, oxygen is consumed.
$\forall e \; burning (e) \rightarrow \exists e^{\prime } (consuming (e^{\prime }) \wedge object (e, oxygen) \wedge in (e,e^{\prime })$
Agatha burned the wood.
$\exists e \; burning (e) \wedge subject (e, Agatha) \wedge object (e, wood)$
Oxygen was consumed.
$\exists e^{\prime } \; consuming (e^{\prime }) \wedge object (e^{\prime }, oxygen)$
We do not go into details of containment of events as expressed by $in$ in the first example above, and also the representation of passives as in the third example above. In these three examples, the author uses predicates such as object and subject which represent more fine-grained relationship with the main predicate (corresponding to the verb usually) than the examples earlier. Extending this work, BIBREF21 , BIBREF24 , BIBREF25 have demonstrated that using Davidson's $e$ variable allows one to express tense dependency between perception verbs and their infinitival compliments in a natural way.
BIBREF21 extends Davidson's approach to logical representation by adding an extra term corresponding to the event type of the predicate. He distinguishes between two types of eventualities: eventualities that culminate called Cul containing achievements and accomplishments, and those that do not, called Hold containing states and activities.
John buttered the toast.
$\exists e \; buttering (e) \wedge agent (e, Jones) \wedge theme (e, toast) \wedge (\exists t \; (t <now \wedge Cul (e,t))$
Mary knows Fred.
$\exists e \; knowing (e) \wedge experiencer (e, Mary) \wedge theme (e, Fred) \wedge Hold (e, now))$
In the logical representation in these examples, the author uses predicates such as theme, agent and experiencer which are usually are called cases in linguistics BIBREF26 . In addition, the author uses a variable $t$ to express time. $now$ is a special indexical variable. We do not give detailed discussions of these fine points here.
BIBREF27 also proposes a logical form based on Davidson's approach. The main motivation behind Hobb's approach is to be able to produce satisfactory semantic representation when an event is expressed as a noun, or when we want to express the meaning of tenses, modalities, and adverbial modifiers. He also explains how so-called opaque adverbials like almost in the sentence, John is almost a man. can be represented by the Davidsonian approach, which Hobbs extends. He also shows how the ambiguity between de re and de dicto meanings of sentences BIBREF28 that discuss beliefs can be explained by his approach to logical form representation of sentences. The representation by Hobbs is quite similar to other such representations based on Davidson, although there are some fine points of differences, that we do not discuss here. From a practical point of view, several research efforts in computational linguistics have adopted Hobb's logical form, and one such recent approach is by BIBREF29 who attempt to map Stanford dependency parses BIBREF30 into Hobbsian logical form, and discover that sometimes it is possible to do so, but in other cases the mapping requires semantic information that is not present in such dependencies indentified by the Stanford parser.
 Event structure
Early efforts at identification of event structure in linguistics was usually limited to explaining essential grammatical phenomena. However, others later proposed complex structures that go beyond simple structures such as Davidson's approach of representing an event by a single logical variable and its components by additional predicates. Understanding the structure of an event entails (i) understanding the argument structure of the word (or, phrase) used to express the event in surface form, (ii) understanding the components in the conceptual or semantic description of an event, and (iii) understanding the relation or mapping between syntactic realization of an event and its conceptual components. In fact, analysis of argument structure includes all three steps and requires finding the relation between meaning of a verb (or a clause) and the syntactic realization of arguments. BIBREF19 , BIBREF31 introduce argument structure as a distinct level of representation in linguistics. Other prominent representations proposed include f-structures BIBREF32 , linear event structures BIBREF18 , lexical conceptual structures (LCS) BIBREF33 , BIBREF34 and two related structures: event structures and qualia structures for arguments BIBREF17 .
There are two sides to event structure: syntactic and semantic. When specifying event structure, at the semantic level, the description must be richer than semantic role descriptions BIBREF35 , BIBREF36 . BIBREF37 argues that named roles or thematic roles are too coarse-grained to provide useful semantic interpretation of a sentence. It is also necessary to capture semantic distinctions in a much more fine-grained manner compared to prior theories of BIBREF38 , BIBREF39 , BIBREF40 . ***A sentence or two on these theories*** By this time it was clear that sophisticated approaches to specifying event structure must build upon the rich taxonomy of verb classes BIBREF37 and descriptive vocabulary work BIBREF41 and BIBREF33 .
BIBREF41 discusses the systematic relations in language between meaning and surface expression. In particular, Talmy focuses on verbs and in particular, verbs that describe motion or location. He sketches a “motion" event in order to explore issues in lexicalization. The basic motion event consists of one object called figure moving or located with respect to another object called the referent or the ground. The motion event has additional components such as path and motion, manner and cause. Talmy gives examples of cases where the verb at once can express, in addition to the action or motion, one or more of figure, path, manner or cause. If a sematic component such as manner or cause is expressed directly by the verb, it is called conflation of manner (or cause) into the verb. Some verbs incorporate aspect, which represents the “pattern of distribution of at ion though time." In some languages, verbs can incorporate personation as well. Personation is a specification of the person involved, e.g., self or non-self. Some verbs incorporate what is called valence, where in conceptualizing an event that involves several different entities in distinct roles, a verb is able to direct greater attention to some one of these entities that to the others, or perhaps adopt a specific perspective. Sometimes, semantic components are not incorporated into the verb, but are expressed through what Talmy calls satellites. A satellite is an immediate constituent of the verb root other than inflections, auxiliaries or nominal arguments.
Talmy enumerates 35 different semantic components. In addition to the six listed above, these include main purpose, result, polarity, aspect, personation, temporal and spatial setting, gender, valence, etc. Talmy also isolates surface elements within a verb complex such as the root verb, inflections, appositions, subordinate clauses and satellites. He then examines which semantic elements are expressed by which surface elements. He finds that the relationship is mostly not one-to-one. A combination of semantic elements may be expressed by a single surface element, or a single semantic element by a combination of surface elements. In a similar manner, semantic elements of different types can be expressed by the same type of surface elements or by several different ones.
Talmy's work does not enumerate lexical entries for specific verbs, but provides detailed discussion on semantic facets of meanings of a verb. The main thrust of Talmy's work is to demonstrate that semantic elements and surface elements relate to each other in specific patterns, both typological and universal. In work prior to Talmy's, most work has treated language's lexical elements as atomic givens, without involving semantic components that comprise them. These studies treated the properties that such whole forms can manifest, in particular, word order, grammatical relations and case roles. Talmy's cross-linguistic study determines semantic components' surface presence, site (their host constituent or grammatical relation) and combination within a site. In addition, Talmy's tracing of surface occurrence patterns extends beyond treating single semantic component at a time to treating a concurrent set of components.
Lexical semantics must strive to represent at least some of the various semantic components that Talmy enumerates. In addition, it must incorporate ways of mapping from syntax to semantics or vice versa. In a very simple system, a set of detailed rules may be able to enumerate the mappings from syntax to semantics and vice versa. In a complex modern system, it is necessary that a machine learning technique will automatically acquire the mappings. This usually requires a lot of labeled examples for a machine learning program to learn such mappings. We discuss some such as efforts later in the paper.
Consider the following illustrative examples.
Mary walked.
Mary walked to the store.
Mary walked for 30 minutes.
Sentence a) describes a process, which is an activity of of indefinite length, i.e., the sentence does not say how long the activity of walking took. Although b) does not give an explicit time duration for the walking event, it depicts an accomplishment and provides a logical culmination to the duration of the event of walking because the event is over when Mary reached the store. Sentence c) talks about a bounded process in which, the event of walking terminating although it does not provide an explicit termination point, but provides a bound to the time extent of the activity in terms of a duration adverbial. This example motivates the observation that the use of prepositional phrases or duration adverbials can change the (aspectual) classification of an event. To explain such phenomena better, it is beneficial to have more complex event structures or lexical analysis of event words. BIBREF18 hypothesizes that the direct object plays a role in delimitation of an event, i.e., whether it has a culmination or not. BIBREF16 , BIBREF17 builds upon such observations and hypothesizes that it is necessary to know the how an event can be broken down into sub-events. He provides the following reasons for sub-eventual analysis.
Sub-eventual analysis of predicates allows verbal decomposition leading to more complex lexical semantics.
Scope of adverbial modification, for some adverbials, can be explained better using event sub-structures.
Semantic arguments of items within a complex event structure can be mapped onto argument structures better.
Pustejovsky describes a generative lexicon in the sense that meanings are described in terms of a limited number of so-called generative devices or primitives by drawing upon Aristotle's species of opposition BIBREF42 . For example, to express the meaning of the word closed as in The door is closed, The door closed or John closed the door, one needs the concept of opposition between closed and not-closed. This essential opposition in the meaning of a lexical item is described by Pustejovsky in terms of what is called the qualia structure of the lexical item. Thus, there are three primary components to the event structure proposed by Pustejovsky.
Event type: The event type of the lexical item is given in terms of the classification schemes discussed earlier.
Rules for event composition: Since an event may be expressed by more than a single verb, the meanings of several lexical items may have to be composed to obtain a description. For example, how does PP attachment change the meaning of the central event in context?
Mapping rules from event structure to argument structure: Pustejovsky describes a number of rules or principles for such mapping. These rules describe how how semantic participants are realized syntactically.
Pustejovsky provides lexical meaning in terms of four separate structures.
Argument structure: The behavior of a word as a function, with its arity. This provides the predicate argument structure for a word, which specifies how it maps to syntax.
Event structure: It identifies a specific event type for a word or a phrase, following BIBREF3 .
Qualia structure: It provides the essential attributes of an object that need to be expressed lexically.
Inheritance structure: It specifies how the word is globally related to other concepts in the lexicon.
In summary, Pustejovsky endows complexity to lexical entries for verbs as well as non-verbs so that semantic weight does not fall on verbs alone in the lexicon and when composing the meaning of a sentence from its constituents. Pustejovsky's approach also reduces the number of lexical entries necessary for individual verbs because the lexical entries become more general. Pustejovosky focuses on creating more muscular compositional semantics rather than decomposing a verb's meaning into a specified number of primitives.
Semantic Arguments and Syntactic Positions
Frequently, specific semantic arguments of a verb (also called thematic arguments) appear in characteristic syntactic positions. This has led to theories or proposals regarding mapping between the two. These theories state that specific semantic arguments belong in specific syntactic positions and that there is 1-1 relationship between semantic argument and (initial) syntactic position. Such proposals or theories include the Universal Alignment Hypothesis BIBREF43 and Uniformity of Theta Assignment Hypothesis BIBREF44 . These are supposed to be universal in that they applied across languages and across verbs. For example, agents appear in subject positions across languages and verbs. This mapping is thus universal. However, other mappings are not so universal. For example, the theme can appear in object, subject or indirect object position; and the experiencer can appear in subject or object position.
A theory that explains lexicon-to-syntax mapping also needs to explain the existence of argument alterations. In other words, it should explain the possibility that the same semantic role can appear in different syntactic positions for the same verb. Usually, linguists classify verbs into a number of semantic classes (different from the ones we talked about earlier) and for each class, a set of mapping relations and a set of argument alterations are specified BIBREF45 , BIBREF46 , BIBREF47 . However, other researchers claim that such semantic classification is difficult to obtain because semantically similar verbs may behave differently across languages BIBREF48 , a given verb in a language may have multiple syntactic realizations BIBREF48 , BIBREF6 , and semantically similar verbs may allow several syntactic realizations BIBREF6 .
Lexical Resources for Action or Event Representation
The discussions on lexical representation of verbs so far have been based on efforts where a small number of examples were studied intently by linguists before making the various proposals. Starting the 1980s but more so in the 1990s, when computer scientists started to focus more on analysis of large text corpora, it became evident to some that the lexical analysis of pure linguists can be extended by knowledge gathered from such corpora. This led to development of the Comlex lexicon BIBREF49 , WordNet BIBREF50 , BIBREF51 , VerbNet BIBREF52 , FrameNet BIBREF53 , BIBREF54 , BIBREF55 , BIBREF56 and other resources. Some of these may have started without an automatic analysis of corpora, but soon corpora were used to refine and enhance the initial lexical resources. Comlex was a substantial resource whose creators spent a lot of effort in enumerating subcategorization features. WordNet is a large lexical resource or ontology, which encompasses words from all categories. WordNet includes verbs, but is not verb-specific. VerbNet, of course, is focussed on verbs alone. FrameNet is also focussed on verbs. Both VerbNet and FrameNet attempt to represent all verbs, not only those which are used to represent “events". However, the term event itself is not clearly defined and most anything that is described by a verb can be considered an event in some context or another.
Comlex and Nomlex Lexicons
Comlex was created at New York University as a computational lexicon providing detailed syntactic information on approximately 38,000 words BIBREF49 . Of course, not all of these were verbs or words that describe actions. The feature set Comlex provided were more detailed than commerically available dictionaries at the time such as the Oxford Advanced Learner's Dictionary (OALD) BIBREF57 and Longman's Dictionary of Contemporary Englisch (LDOCE) BIBREF58 . The initial word list was derived from OALD. The lexicon used a Lisp-like notation for dictionary entries. We see some sample entries for verbs in Comlex in Figure 1 .
Comlex paid particular attention to providing detailed subcategorization or complement information for verbs, and nouns and adjectives that take complements. Comlex was influenced by prior work on lexicon such as the Brandeis Verb Lexicon BIBREF59 , the ACUILEX project BIBREF60 , the NYU Lingustic String Project BIBREF61 , the OALD and the LDOCE, and it incorporated distinctions made in these dictionaries. Comlex had 92 different subcategorization features for verbs. The features recorded differences in grammatical functions as well as constituent structure. In particular, Comlex captured four different types of control: subject control, object control, variable control and arbitrary control. It was also able to express the fact that a verb may have different control features for different complement structures, or different prepositions within the complement. Figure 2 shows a few complements used in Comlex. Here :cs is the constituent structure, :gs is the grammatical structure and :ex are examples. The authors created a initial lexicon manually and then refined it using a variety of sources, both commercial and corpus-based.
The Nomlex dictionary of nominalizations was also developed at NYU BIBREF62 , BIBREF63 . It enumerated allowed complements for nominalizations, and also related nominal complements of the corresponding verbs. A nominalization is the noun form of a verb. For example, the verb promote is nominalized as nominalization. Similarly, the nominalizations of the verb appoint are appointment and appointee. Nomlex entries are similar in syntax to Comlex entries. Each Nomlex entry has a :nom-type feature which specifies four types of nominalizations: action (appointment, destruction) or state (knowledge), subject (teacher), object (appointee) and verb-part for those nominalizations that incorporate a verbal particle (takeover). Meyers et al. BIBREF63 presented a procedure what mapped syntactic and semantic information for an active clause containing a verb e.g., (IBM appointed Alice Smith as vice president) into a set of patterns for nominalization (IBM's appointment of Alice Smith as vice president or Alice Smith's appointment as vice president). The lexical entry for the verb appoint used in Comlex is given in Figure 1 . The lexical entry in Nomlex for the action nominalization appointment is given in Figure 3 .
Levin's Verb Classes
Levin's verb classes BIBREF45 explicitly provide the syntax for each class, but do not provide semantic components. The classes are based on the ability or inability of a verb to occur in pairs of syntactic frames, with the assumption that syntactic frames reflect the underlying semantics. For example, break verbs and cut verbs are similar because they can all take part in transitive and middle constructions. However, only break verbs can occur in simple intransitive constructs. Similarly, cut verbs can occur in conative constructs and break verbs cannot. The explanation given is that cut describes a sequence of actions that result in the goal of separating an object into pieces. It is possible that one can perform the actions without achieving the result (e.g., John cut at the loaf). For break, the result is a changed state where the object becomes separated into pieces. If the result is not achieved, we cannot say that the action of breaking took place. The examples below are taken from BIBREF64 .
Transitive Construction: (a) John broke the window., (b) John cut the bread.
Middle Construction: (a) Glass breaks easily., (b) This loaf cuts easily.
Intransitive Construction: (a) The window broke., (b) *The bread cut.
Conative Construction: (a) *John broke at the window., (b) John valiantly cut at the frozen loaf, but his knife was too dull to make a dent in it.
Levin's original classes had some inconsistencies. For example, many verbs were listed in multiple classes, some of which had conflicting syntactic frames. BIBREF65 refined the original classification to remove some of these problems to build a more fine-grained, syntactically and semantically coherent refined class called intersective Levin classes. Levin's classes also are focussed mostly on verbs taking noun (NP) and prepositional phrase (PP) complements, and are weak on coverage of ADJP, ADVP, sentential complement, etc. VerbNet is built using these classes.
Organization of verbs into such classes capture generalizations about their properties. Such classes also help create better NLP systems. Many NLP systems benefit from using the mapping from surface realization of arguments to predicate-argument structure that is available in such classes. These classes also capture abstractions (e.g., syntactic and semantic properties) and as a result, they are helpful in many operational contexts where the available corpora are small in size and thus, it is not possible to extract detailed lexical information about verbs from such small corpora. The predictive power of the classes can compensate for the lack of sufficient data. Lexical classes have been helpful in tasks such as subcategorization acquisition BIBREF66 , BIBREF67 , BIBREF68 , automatic verb acquisition BIBREF69 , semantic role labeling BIBREF70 , and word sense disambiguation BIBREF71 . ***Add newer citations for application. Look at after 2004 proceedings of NAACL-HLT***
WordNet
The WordNet project BIBREF50 , BIBREF51 started in the mid-1980s at Princeton University and over time, has become the most widely used lexical resource in English, especially when one needs a lexical resource that can be used by a program. Wordnet was primarily designed as a semantic network and later modified to be a lexical database.
WordNet groups words into synsets (synonym set) and contains relations among these synsets. A synset contains all the word forms that can refer to a given concept or sense. For each sense of each word, WordNet also provides a short, general definition called its gloss and example usages.
As the name hints, the WordNet can be thought of as a large graph where the words and synsets are nodes. These nodes linked by edges that represent lexical and semantic-conceptual links, which we discuss briefly below. Individual words may also be linked with antonym links. Superclass-subclass relations link entire synsets. WordNet has entries for verbs, nouns, adjectives and adverbs.
To get a better feel for what WordNet is like, let us look at the online version of WordNet at Priceton University. When we search for the word assault in the online WordNet, the results come in two parts: noun and verb, because assault can be either a verb or a noun. The results that show up for verb are given in Figure 4 . The verb senses of assault belongs to three synsets. In other words, it has three senses or can refer to three different concepts. Each sunset is composed of several verbs. The second of these synsets contains one sense of each of the verbs assail, set on and attack.
A verb may have four types of entries in WordNet: hypernyms, toponyms, entailment and coordinate terms. These terms are defined here. A verb $Y$ is a hypernym of the verb $X$ if the activity $X$ is a (kind of) $Y$ . For example, to perceive is an hypernym of to listen. A verb $Y$ is a troponym of the verb $X$ if the activity $Y$ is doing $X$ in some manner. For example, to lisp is a troponym of to talk. A verb $Y$ is entailed by $X$ if by doing $X$0 one must be doing $X$1 . For example, to sleep is entailed by to snore. Coordinate terms are those verbs that share a common hypernym, e.g., to lisp and to yell. If we want to see the direct troponym of the second synset for the verb meaning of assault, we get what we see in Figure 5 .
WordNet has been used in many applications. However, it is most commonly used as a computational lexicon or “ontology" of English (or, another language) for word sense disambiguation, a task that assigns the most appropriate senses (i.e. synsets) to words in specific contexts. Although WordNet is large and detailed, WordNet does not have information required by NLP applications such as predicate-argument structure. Although WordNet contains a sufficiently wide range of common words, it does not cover special domain vocabulary. It is general in nature, and therefore difficult to use if specialized vocabulary is needed. Also, WordNet senses are sometimes overly fine-grained even for human beings and as a results, some researcher argue that it cannot achieve very high performance in the tasks where it is applied. Although WordNet is the most widely used online lexical database in NLP applications, is also limited in its coverage of verbs.
The English WordNet currently contains approximately 117,659 synsets, each sunset corresponding to a sense of a word. It has 11,529 verbs that belong to 13,767 synsets. It also contains 117,798 nouns that belong to 82,115 synsets. WordNets have been developed or are being developed in a large number of languages such as Catalan, French, Spanish, Japanese, Chinese, Danish, Korean and Russian. Notable collaborative efforts include Euro Wordnet BIBREF72 , BIBREF73 , BIBREF74 , Asian Wordnet BIBREF75 , BIBREF76 , BIBREF77 and Indo WordNet BIBREF78 projects. The Indo WordNet focuses on 18 major languages of India. For example, as of June 2012 there are 15,000 synsets in the Assamese WordNet, 24,000 in Bengali, 16,000 in Bodo, 27,000 in Gujarati, and 31,500 in Oriya. WordNets in most other languages are not as sophisticated as the one in English.
FrameNet
FrameNet BIBREF79 , BIBREF55 , BIBREF80 is another substantial publicly available lexical resource that has come into existence independently. It is based on the theory of frame semantics BIBREF81 , BIBREF82 , BIBREF53 , BIBREF54 , BIBREF56 where a frame corresponds to a stereo-typical scenario involving an interaction and participants, where participants play some kind of roles. The idea is that the meanings of most words are best understood in context. FrameNet proposes a small context, called a semantic frame, a description of a type of event, relation or entity and the participants in it. A frame has a name and this name is used to identify a semantic relation that groups together the semantic roles.
Although frames mostly correspond to verbs, there are frames that can be identified by nouns and adjectives. FrameNet also has a large number of annotated sentences. Each annotated sentence exemplifies a possible syntactic realization of the semantic role associated with a frame for a given target word. FrameNet extracts syntactic features and corresponding semantic roles from all annotated sentences in the FrameNet corpus, it builds a large set of rules that encode possible syntactic realizations of semantic frames.
FrameNet aims to document the range of semantic and syntactic combinatory possibilities— valences–of each word in each of its senses, through computer-assisted annotation of example sentences and automatic tabulation of the annotation results. The FrameNet lexical database, currently contains more than 10,000 lexical units (defined below), more than 6,000 of which are fully annotated, in nearly 800 hierarchically-related semantic frames, exemplified in more than 170,000 annotated sentences. See the FrameNet website for the latest statistics. FrameNet has been used as a semantic role labeling, used in applications such as information extraction, machine translation, event recognition, sentiment analysis, etc., like the other publicly available lexical resources.
An example of a frame is Attack. This frame has several frame elements. The core frame elements are assailant and victim. There are a large number of non-core frame elements. These include Circumstances, Containing_event, Direction, Duration, Explanation, Frequency, Manner, Means, Place, Purpose, Result, Time, Weapon, etc. For each of these frame elements there can be seen in one or more annotated sentences. Here is an example annotated sentence.
$$[_{Assailant} \; The \; gang] \; ASSAULTED [_{Victim} \; him] \\ [_{Time} \; during \; the \;drive \; to \; Rickmansworth] [_{Place} \; in \;Hertfordshire] ...$$   (Eq. 74)
The frame Attack is associated with a large number of associated units. These include verbs and nouns. Example verbs are ambush, assail, assault, attack, bomb, bombard, charge, hit, infiltrate, invade, raid, storm and strike. Examples of nouns are airstrike, ambush, assailant, assault, attack, etc. The frame Attack inherits from a frame called Intentionally_affect. It is inherited by frames Besieging, Counterattack, Invading and Suicide_attack.
FrameNet annotates each frame element (or its representation, actually) in at least three layers: a frame element name (e.g., Food), a grammatical function (e.g., Object) and a phrase type (e.g., NP). Only the frame elements are shown in the Web-based interface to reduce visual clutter, although all three are available in the XML downloads. FrameNet has defined more than 1000 semantic frames. These frames are linked together using frame relations which relate more general frames to specific ones. This allows for reasoning about events and intentional actions.
Because frames are semantic, they are often similar across languages. For example, frames about buying and selling involve frame elements Buyer, Seller, Goods and Money in every language. FrameNets have been developed for languages such as Portuguese, German, Spanish, Chinese, Swedish and Japanese.
At the current time, there are 1159 frames in FrameNet. There are approximately 9.6 frame elements per frame. There are 12595 lexical units of which 5135 are nouns, 4816 are verbs, 2268 are adjectives. There are 12.1 lexical units per frame.
There have been some attempts at extending the coverage of FrameNet. One such effort is by BIBREF83 who use a new broad-coverage lexical-semantic resource called PPDB to add lemmas as pontential triggers for a frame and to automatically rewrite existing example sentences with these new triggers. PPDB, The Paraphrase Database, is a lexical, phrasal and syntactic paraphrase database BIBREF84 . They use PPDB's lexical rules along with a 5-gram Kneser-Ney smoothed language model trained using KenLM BIBREF85 on the raw English sequence of the Annotated Gigaword corpus BIBREF86 .
PropBank
PropBank BIBREF87 , BIBREF88 , BIBREF89 is an annotated corpus of verb propositions and their arguments. PropBank does not annotate events or states of affairs described using nouns. PropBank-style annotations usually are closer to the syntactic level, whereas FrameNet-style annotations are more semantically motivated although, as discussed earlier, FrameNet provides layers of annotations including syntactic parses. PropBank annotates one million words of the Wall Street Journal portion of the Penn Treebank BIBREF90 with predicate-argument structure for verbs using semantic role labels for each verb argument.
Although the same tags are used across all verbs (viz., Arg0, Arg1, $\cdots $ , Arg5), these tags have verb-specific meaning. FrameNet requires that the use of a given argument label is consistent across different uses of a specific verb, including its syntactic alternations. Thus, Arg1 (italicized) in “John broke the window broke" is the same window that is annotated as the Arg1 in “The window broke" even though it is the syntactic subject in one case and the syntactic object in another. FrameNet does not guarantee that an argument label is used consistently across different verbs. For example, Arg2 is used as label to designate the destination of the verb “bring", but the extent of the verb “rise". Generally, the arguments are simply listed in the order of their prominence for each verb. However, PropBank tries to use Arg0 as the consistent label for the “prototypical agent" and Arg1 for the “prototypical patient" as discussed in BIBREF7 .
PropBank divides words into lexemes using a very coarse-grained sense disambiguation scheme. Two senses are considered distinct only if their argument labels are different. In PropBank each word sense is called a frameset. PropBank's model of predicate-argument structure differs from dependency parsing. In dependency parsing, each phrase can be dependent only on one other phrase. But, in PropBank, a single phrase can be arguments to several predicates. PropBank provides a lexicon which divides each word into coarse-grained senses or framesets, and provides examples usages in a variety of contexts. For example, the to make an attack, criticize strongly sense of the predicate lemma (or, verb) attack is given in Table 3 along with an example.
PropBank tries to keep rolesets consistent across related verbs. Thus, for example, the buy roleset is similar to the purchase and sell rolesets. See Table 4 , taken from BIBREF87 .
One can clearly see that it may be possible to merge such similar framesets together to obtain something similar to the verb roles in FrameNet's Commerce frameset.
Although similar, PropBank differs from FrameNet we have discussed earlier in several ways. PropBank is a resource focussed on verbs whereas FrameNet is focussed on frame semantics that generalizes descriptions across similar verbs as well as nouns and other words (e.g., adjectives) as discussed earlier. PropBank was created with the idea of serving as training data to be used with machine learning algorithms for the task of semantic role labeling. It requires all arguments to a verb to be syntactic constituents in nature. In addition, PropBank differentiates among senses of a verb if the senses take different sets of arguments. There is a claim that due to such differences, semantic role labeling is easier using a corpus annotated with PropBank type annotation compared to FrameNet type annotation.
VerbNet
VerbNet BIBREF91 , BIBREF64 , BIBREF92 attempts to provide a definitive resource for lexical entries for English verbs. It is compatible with WordNet, but has lexical entries with explicit syntactic and semantic information about verbs, using Levin's verb classes BIBREF45 . It uses verb classes to capture generalizations and for efficient encoding of the lexicon. Its syntactic frames for verb classes are represented using a fine-grained variation of Lexicalized Tree Adjoining Grammers BIBREF93 , BIBREF94 , BIBREF65 augmented with semantic predicates, allowing for creating compositional meanings for more complex constituents such as phrases and clauses. VerbNet provides traditional semantic information such as thematic roles and semantic predicates, with syntactic frames and selectional restrictions. it also allows for extension of verb meaning through adjunction of particular syntactic phrases.
A verb entry corresponds to a set of classes, corresponding to the different senses of the verb. For each verb sense, there is a verb class as well as specific selectional restrictions and semantic characteristics that may not be captured by class membership. VerbNet also contains references to WordNet synsets. Verb classes capture generalizations about verb behavior. Each verb class lists the thematic roles that the predicate-argument structure of its members allows, and provides descriptions of the syntactic frames corresponding to allowed constructs, with selectional restrictions given for each argument in each frame. Verb classes are hierarchically organized. It required some manual restructuring of Levin's classes. Each event $E$ is decomposed into a three-part structure according to BIBREF11 , BIBREF95 . VernNet uses a time function for each predicate specifying whether the predicate is true during the preparatory, culmination or consequent/result stage of an event. This structure allows VerbNet to express the semantics of classes of verbs like Change of State verbs. For example, in the case of the verb break, it is important to distinguish between the state of the object before the end of the action and the new state that results afterwards.
Table 5 is an example of a simplified VerbNet entry from its website. The original VerbNet was extended using extensions proposed by BIBREF68 . This resulted in the addition of a large number of new classes, and also a much more comprehensive coverage of English verbs. Table 6 provides statistics of VerbNet's coverage in its initial version, VerbNet as described in BIBREF91 , BIBREF64 , BIBREF96 , and its current version as in its official Website.
The absence of any lexicon or resource that provides for accurate and comprehensive predicate-argument structure (or semantic role labels) for English verbs has been long considered a critical element that was needed to produce robust natural language processors. This was shown clearly by BIBREF97 who evaluated an English-Korean machine translation system. The authors showed that among several factors impacting on the low quality of translations, one that was most influential was the inability to predicate-argument structure. Even with a grammatical parse of the source sentence ad complete vocabulary coverage, the translation was frequently bad. This is because, the authors found, that although the parser recognized the constituents that are verb arguments, it was unable to precisely assign the arguments to appropriate positions. This led to garbled translations. Simply preserving the proper argument position labels and not changing other things, resulted in substantial improvement in acceptable translations. When using one parser, the improvement was 50%; with a second parser, the improvement was dramatic 300%. Thus, the purpose in developing lexical resources such as FrameNet and PropBank, PropBank especially so, is to provide for training data annotated with predicate-argument positions with labels. Such data can be used with machine learning techniques.
Combining FrameNet, VerbNet and WordNet
There have been attempts to integrate lexical resources to obtain more robust resources with wider coverage. We discuss one such effort here. BIBREF98 integrate FrameNet, VerbNet and WordNet discussed earlier into a single and richer resource with the goal of enabling robust semantic parsing. The reason for building connections among the three lexical resources is that similar syntactic patterns often introduce different semantic interpretations and similar meanings can be realized in many different ways. The improved resource provides three enhancements: (1) It extends the coverage of FrameNet, (2) It augments VerbNet's lexicon with frame semantics, and (3) It implements selectional restrictions using WordNet semantic classes. They use knowledge about words and concepts from WordNet, information about different situations from FrameNet, and verb lexicon with selectional restrictions from VerbNet. They extract syntactic features and corresponding semantic roles from all annotated sentences in FrameNet to build a large set of rules that encode the possible syntactic realization of semantic frames. They identify the VerbNet verb class that corresponds to a FrameNet frame and this allows them to parse sentences that include verbs not covered by FrameNet. This they do by exploiting a transitivity relation via VerbNet classes: verbs that belong to the same Levin classes are likely to share the same FrameNet frame, and their frame semantics can be analyzed even if not explicitly defined in FrameNet. They use information from WordNet in several stages in the parsing process. The argument constraints encoded in VerbNet (e.g., $+animate, +concrete$ ) are mapped to WordNet semantic classes, to provide selectional restrictions for better frame selection and role labeling in a semantic parser. In addition, the mapping between WordNet verb entries and FrameNet lexical units allows them to extend the parser coverage, by assigning common frames to verbs that are related in meaning according to the WordNet semantic hierarchies. The authors found that their mapping algorithms produced 81.25% correct assignment of VerbNet entries with a correct FrameNet frame. They also were able to map 78.22% VerbNet predicate-argument structures with some syntactic features and selectional restrictions to the corresponding FrameNet semantic roles.
OntoNotes and Other Large-scale Annotated Corpora
The OntoNotes project BIBREF99 , BIBREF100 , BIBREF101 has created an infrastructure for much richer domain independent representation of shallow meaning for use in natural language processing tasks, including event detection and extraction, in English, Chinese and Arabic. OntoNotes annotates documents at several layers: syntax, propositions, word senses including nominalizations and eventive noun senses, named entities, ontology linking and co-reference. It has been designed to be a well-annotated large-scale corpus from which machine learning programs can learn many different aspects of meaning felicitously.
OntoNotes uses Penn TreeBank parses BIBREF102 , PropBank propositional structures BIBREF87 , BIBREF88 , BIBREF89 on top of Penn Treebank, and uses the Omega ontology BIBREF103 for word sense disambiguation. As we know, the Penn Treebank is annotated with information from which one can extract predicate-argument structures. The developers of OntoNotes use a parser that recovers these annotations BIBREF104 . The Penn Treebank also has markers for “empty" categories that represent displaced constituents. Thus, to create OntoNotes, its developers use another parser BIBREF105 , BIBREF106 to extract function words. They also use a maximum entropy learner and voted preceptons to recover empty categories. PropBank, as we know, annotates the one-million word Wall Street Journal part of the Penn Treebank with semantic argument structures for verbs. As we have noted earlier, the creators of OntoNote and others have discovered that WordNet's very fine grained sense distinctions make inter-annotator agreement or good tagging performance difficult. To achieve better performance, OntoNotes uses a method BIBREF107 , BIBREF108 for sense inventory creation and annotation that includes links between grouped word senses and the Omega ontology BIBREF103 . OntoNotes represents sense distinctions in a hierarchical structure, like a decision tree, where coarse-grained distinctions are made at the root and increasingly fine-grained restrictions until reaching WordNet senses at the leaves. Sets of senses under specific nodes of the tree are grouped together into single entries, along with syntactic and semantic criteria for their groupings; these are presented to annotators for improved annotation agreement, obtaining up to 90% inter-annotator agreement. OntoNote follows a similar method for annotation of nouns.
To allow access to additional information such as subsumption, property inheritance, predicate frames from other sources, links to instances and so on, OntoNotes also links to an ontology. This requires decomposing the hierarchical structure of OntoNotes into subtrees which then can be inserted at the appropriate conceptual node in the ontology. OntoNotes represents its terms in the Omega ontology BIBREF103 . Omega has been assembled by merging a variety of sources such as WordNet, Mikrokosmos BIBREF109 , and a few upper ontologies such as DOLCE BIBREF110 , SUMO BIBREF111 , and Penman Upper Model BIBREF112 . OntoNote also includes and cross-references verb frames from PropBank, FrameNet, WordNet and Lexical Conceptual Structures BIBREF113 . OntoNotes also has coreferences. It connects coreferring instances of specific referring expressions, primarily NPs that introduce or access a discourse entity.
For the purpose of our paper, it is important to know that OntoNotes tries to annotate nouns that carry predicate structure, e.g., those whose structure is derived from their verbal form. In particular, OntoNotes annotates nominalization and eventive senses of nouns. OntoNotes applies two strict criteria for identifying a sense of a noun as a nominalization BIBREF101 .
The noun must relate transparently to a verb, and typically display a nominalizing morpheme such as -ment (govern/government), -ion (contribute/contribution), though it allows some zero-derived nouns such as $kill$ , the noun derived from $kill$ , the verb.
The noun must be able to be used in a clausal noun phrase, with its core verbal arguments related by semantically empty or very “light" licensers, such as genitive markers (as in The Roman's destruction of the city.. or with the verb's usual particle or prepositional satellites as in John's longing for fame and fortune...
Just like nominalization senses, OntoNotes has strict definition of eventive senses. They have two definitional criteria (1) and (2), and a diagnostic test (3), for determining if a noun sense is eventive.
Activity causing a change of state: A noun sense is eventive when it refers to a single unbroken activity or process, occurring during a specific time period, that effects a change in the world of discourse.
Reference to activity proper: The noun must refer to the actual activity or process, not merely to the result of the activity or the process.
The noun patterns with eventive predicates in the “have" test: BIBREF114 describes the following heuristic lexico-syntactic diagnostic test to apply to many nouns. The test has four parts to it as discussed briefly below.
Create a natural sounding sentence using the construction X had <NP> where <NP> is a noun phrase headed by the noun in question, e.g., John had a party.
Check if the sentence can be used in present progressive as in John is having a party. If the sentence is felicitous, it adds to the noun being inventive. If it sounds odd, it adds to the evidence that the noun is stative.
Check if the sentence can be used in a pseudo-cleft construction such as What John did was have a party. If it is felicitous, the noun is more likely to be eventive. If not, it is more likely to be stative.
Check if the sentence suggests iterative or habitual action using the simple present such as John has a party every Friday. If so, it adds evidence that the noun is eventive. If the sentence suggests that the situation is taking place at that very moment that it is uttered, it adds evidence that the noun is stative as in John has a cold.
In addition to OntoNotes, there have been other efforts at obtaining large-scale annotated corpora such at the GLARF project BIBREF115 that tries to capture information from various Treebanks and superimpose a predicate argument structure. The Unified Linguistic Annotation (ULA) project BIBREF116 is a collaborative effort that aims to merge PropBank, NomBank, the Penn Discourse Treebank BIBREF117 and TimeBank BIBREF118 with co-reference information.
Extracting Events from Textual Documents
Different models of events have been used in computational linguistics work geared toward information extraction.
We discuss TimeML events next followed by events. We discuss biomedical event extraction in Section , and extraction of events from Twitter in Section .
TimeML Events
TimeML is a rich specification language for event and temporal expressions in natural language text. In the TimeML BIBREF119 , BIBREF120 annotation scheme, an event is a general term for situations that happen or occur. Events can be punctual or momentary, or last for a period of time. Events in TimeML format may also include predicates describing states or circumstances in which something holds true. Only those states that participate in an opposition structure, as discussed in Subsection UID42 , are annotated. In general, an event can be expressed in terms of verbs, nominalizations, adjectives, predicative clauses, or prepositional phrases. TimeML allows an event, annotated with the EVENT tag, to be one of seven types: occurrence, state, report, i-action, i-state, aspectual and perception. The first five are special cases. The last two, Occurrence and State are used for general cases that do not fall in the special ones.
Reporting: A reporting event describes an action declaring something, narrating an event, informing about a situation, and so on. Some verbs which express this kind of event are say, report, tell, explain, and state. An example sentence with the verb say is Punongbayan said that the 4,795-foot-high volcano was spewing gases up to 1,800 degrees.
I-Action: I stands for intensional. According to the TimeML annotation guidelines, an i-action is a dynamic event that takes an event-denoting argument, which must be explicitly present in the text. Examples of verbs that are used to express i-actions include attempt, try, promise and offer. An example sentence with the verb try is Companies such as Microsoft or a combined worldcom MCI are trying to monopolize Internet access.
I-State: I-State stands for intensional state. Like an I-Action, an I-State event takes an argument that expresses an event. Unlike an I-Action, the I-State class is used for events which are states. An example sentence that uses the verb believe is We believe that his words cannot distract the world from the facts of Iraqi aggression. Other verbs used to express i-states include intend, want, and think.
Aspectual: An aspectual predicate takes an event as an argument, and points to a part of the temporal structure of the event. Such a part may be the beginning, the middle or the end of an event. Verbs such as begin, finish and continue are such aspectual predicates. An example sentence with the verb begin is All non-essential personnel should begin evacuating the sprawling base.
Perception: This class includes events involving the physical perception of another event. Such events are typically expressed by verbs such as see, watch, glimpse, hear, listen, and overhear. An example sentence with the verb see is Witnesses tell Birmingham police they saw a man running.
Occurrence: An occurrence is a general event that occurs or happens in the world. An example of an occurrence is given in the following sentence: The Defense Ministry said 16 planes have landed so far with protective equipment against biological and chemical warfare. The occurrence has been highlighted in bold.
State: A state describes circumstances in which something obtains or holds true. An example sentence that shows two states is It is the US economic and political embargo which has kept Cuba in a box.
TimeML allows one to mark up temporal expressions using the TIMEX3 tag. Temporal expressions are of three types: (a) Fully specified temporal expressions such as June 11, 2013, (b) Underspecified temporal expressions such as Monday, (c) Durations such as three days. TimeML uses the SIGNAL tag to annotate sections of text, usually function words, that indicate how temporal objects are related to each other. The material marked by SIGNAL may contain different types of linguistic elements: indicators of temporal relations such as prepositions such as on and during, other temporal connectives such as when, etc. The TIMEX3 and SIGNAL tags were introduced by BIBREF122 , BIBREF123 .
A major innovation of TimeML is the LINK tags that encode relations between temporal elements of a document and also help establish ordering between the events in a document. There are three types of links: TLINK showing temporal relationships between events, or between an event and a time; SLINK or a subordination link to show context that introduces relations between two events, or an event and a signal; ALINK or an aspectual link to show relationship between an aspectual event and its argument event. TLINK allows for 13 temporal relations introduced by BIBREF124 , BIBREF125 . SLINK is used to express contexts such as use of modal verbs, negatives, positive and negative evidential relations, factives which require the event argument to be true, and counterfactives which require the event argument to be false. ALINK expresses initiation, culmination, termination or continuation relationships between an event and its argument event. Finally, TimeML is able to express three types of causal relations: an event causing an event, an entity causing an event, and the special situation where the use of the discourse marker and as a signal to introduce a TLINK indicating that one event happened before another as in He kicked the ball and it rose into the air.
The creators of TimeML have spent significant efforts to develop a fairly large corpus annotated with TimeML tags. This corpus is called the TIMEBANK corpus BIBREF118 and has 300 annotated articles. This corpus has been used to learn to extract events and temporal relations among events.
ACE Events
In the ACE model, only “interesting” events are annotated in corpora and thus extracted by a trained program. ACE annotators specify the event types they want to be extracted. For example, in one information extraction contest, an ACE 2005 event was of 8 types, each with one has one or more sub-types. The types are given below. ***Maybe, give some examples***
Life: Be-born, marry, divorce, injure and die
Movement: Transport
Transaction: Transfer-ownership, Transfer money
Business: Start-organization, Merge-organization, Declare-bankruptcy
Contact: Meet, Phone-write
Conflict: Attack, demonstrate
Personnel: Start position, End position, Nominate, Elect, and
Justice: Arrest-Jail, Release-Parole, Trial-Hearing, Charge-Indict, Sue, Convict, Sentence, Fine, Execute, Extradite, Acquit, Appeal, Pardon.
Each event also has four categorial attributes. The attributes and their values are given below.
Modality: Asserted and Other where Other includes, but is not limited to: Believed events; Hypothetical events; Commanded and requested events; Threatened, Proposed and Discussed events; and Promised events.
Polarity: Positive and Negative.
Genericity: Specific, Generic
Tense: Past, Present, Future and Unspecified.
ACE events have arguments. Each event type has a set of possible argument roles, which may be filled by entities, time expressions or other values. Each event type has a set of possible argument roles. There are a total of 35 role types although no single event can have all 35 roles. A complete description of which roles go with which event type can be found in the annotation guidelines for ACE 2005 events. In an ACE event, time is noted if when explicitly given. Others have defined events or event profiles themselves to suit their purpose. For example, Cybulska and Vossen BIBREF126 , BIBREF127 describe an historical information extraction system where they extract event and participant information from Dutch historical archives. They extract information using what they call profiles. For example, they have developed 402 profiles for event extraction although they use only 22 of them in the reported system. For extraction of participants, they use 314 profiles. They also 43 temporal profiles and 23 location profiles to extract temporal and locational information. Profiles are created using semantic and syntactic information as well as information gleaned from Wordnet BIBREF50 .
The ACE annotation scheme, discussed earlier, was developed by NIST in 1999, and the ERE (Entities, Relations and Events) scheme was defined as a simpler version of ACE BIBREF128 . One of ERE's goals is also to make annotating easier and annotations more consistent across annotators. ERE attempts to achieve these goals by removing the most problematic annotations in ACE and consolidating others. We will discuss the three types annotations now: Entities, Relations and Events.
For example, consider Entities. ACE and ERE both have Person, Organization, Geo-Political Entity and Location as types of entities. ACE has two additional types, Weapon and Vehicle, which ERE does not have. ERE doesn't distinguish between Facility and Location types and merge them into Location. ERE has a type called Title for titles, honorifics, roles and professions. ACE has subtypes for entity mentions, which ERE does not. In addition to subtypes, ACE classifies entity mentions into classes (e.g., Specific, Generic and Underspecified), ERE has only Specific. ACE and ERE also have differences in how extents and heads are marked, and levels of entity mentions.
The purpose of Relation annotation in both ACE and ERE is to extract a representation of the meaning of the text, not necessarily tied to the underlying syntactic or lexical representation. Both schemes include Physical, Part-Whole, Affiliation and Social relations although the details are a bit different. Both tag relations inside a single sentence and tags only explicit mentions. Nesting of tags is not allowed. Each relation can have up to two ordered Argument slots. Neither model tags negative relations. However, ERE annotates only asserted ("real") events whereas ACE allows others as well, e.g., Believed Events, Hypothetical Events, Desired Events and Requested Events. There is no explicit trigger word in ACE, which annotates the full clause that serves as the trigger for a relation whereas ERE attempts to minimize the annotated span by allowing for the tagging of an optional trigger word or phrase. ACE justifies tagging of each Relation by assigning Syntactic Clauses to them, such as Possessive, PreMod and Coordination. The three types of Relations inn ERE and ACE have sub-types: Physical, Part-Whole, and Social and Affiliation, but ERE collapses ACE types and sub-types to make them more concise, possibly less specific. BIBREF128 discuss the similarities and differences between ACE and ERE in detail.
Events in both ACE and ERE are defined as `specific occurrences' involving `specific participants'. Like entities and relations, ERE is less specific and simplified compared to ACE. Both annotation schemes annotate the same event types: Life, Movement, Transaction, Business, Conflict, Contact, Personnel, and Justice.
BIBREF129 use another annotation scheme called Richer Event Description (RED), synthesizing co-reference BIBREF130 , BIBREF131 and THYME-TimeML temporal relations BIBREF132 . BIBREF129 discusses challenges in annotating documents with the RED schema, in particular cause-effect relations. The usual way to annotate cause-effect relations is using the counter-factual definition of causation in philosophy BIBREF133 , BIBREF134 :
“X causes Y" means if X had not occurred, Y would not have happened.
However, BIBREF129 found that this definition leads to many difficult and sometimes erroneous annotations, and that's why while performing RED annotations, they used another definition BIBREF135 , BIBREF136 which treats causation as “a local relation depending on intrinsic properties of the events and what goes on between then, and nothing else". In particular, the definition is
“X causes Y" means Y was inevitable given X.
In fact, in the annotations performed by BIBREF129 , they use the new definition to make judgements, but use the old definition as a precondition to the new one.
The Knowledge Base Population Track (TAC-BKP) was started by NIST in 2009 to evaluate knowledge bases (KBs) created from the output of information extraction systems. The primary tasks are a) Entity linking–linking extracted entities to entities in knowledge bases, and b) Slot filling–adding information to entity profiles, information that is missing from the knowledge base BIBREF137 . Wikipedia articles have been used as reference knowledge bases in evaluating TAC-KBP tasks. For example, given an entity, the goal is to identify individual nuggets of information using a fixed list of inventory relations and attributes. For example, given a celebrity name, the task is to identify attributes such as schools attended, occupations, important jobs held, names of immediate family members, etc., and then insert them into the knowledge base. Many people compare slot filling to answering a fixed set of questions, obtaining the answers and filling in the appropriate slots in the knowledge base. Slot filling in TAC-KBP differs from extraction in ACE and ERE notations in several ways such as TAC-KBP seeks out information for named entities only, chiefly PERs and ORGs, TAC-KBP seeks to obtain values for slots and not mentions, and events are handled as uncorrelated slots, and assessment is like in question-answering.
Our focus on this paper has been on extracting events, and we know that to extract events properly, we need to explicitly extract event mentions, and also extract associated attributes such as agents, locations, time of occurrence, duration, etc. Rather than explicitly modeling events, TAC-KBP does so implicitly as it captures various relations associated with for example the agent of the event. For example, given a sentence “Jobs is the founder and CEO of Apple", TAC-KBP may pick "Apple" as the focal entity and identify "Jobs" as the filler of its founder slot, and "Jobs" as the filler of its CEO slot. However, an ACE or ERE annotation program will ideally pick the event as Founding, with Jobs as an argument (say the first argument or arg1, or the Actor) of the event, and "Apple" as another argument, say arg2.
Extracting Events
Many even extraction systems have been built over the years. A big motivator for development of event extraction systems seem to be various contests that are held every few years, although there has been considerable amount of non-contest related research as well. Although we discuss extraction of events represented by various formats, the methods are not really different from each other. That is why we discuss TimeML events in more detail and present the others briefly in this section.
We describe a few of the approaches that have been used for extracting TimeML type events. Quite a few papers that attempt to do so have been published BIBREF121 , BIBREF138 , BIBREF139 , BIBREF140 , BIBREF141 , and we pick just a few representative papers.
: BIBREF121 implemented an event and event feature extraction system called EVITA and showed that a linguistically motivated rule-based system, with some help using statistical disambiguation perfumed well on this task. Evita is claimed to be a unique tool within the TimeML framework in that it is very general, being not based on any pre-established list of event patterns and being domain-independent. Evita can also identify, based on linguistic cues, grammatical information associated with event referring expressions, such as tense, aspect, polarity and modality, as stated in the TimeML specification. Evita does not directly identify event participants, but can work with named entity taggers to link arguments to events.
Evita breaks down the event recognition problem to a number of sub-tasks. Evita preprocesses the input text using the Alembic Workbench POS tagger, lemmatizer to find lexical stems, and chunkier to obtain phrase chunks, verbal, nominal and adjectival, the three that are commonly used as event referring expressions BIBREF142 . For each subtask after pre-processing, it combines linguistic- and statistically-based knowledge. Linguistic knowledge is used in local and limited contexts such as verb phrases and to extract morphological information. Statistical knowledge is used to disambiguate nominal events. The sub-tasks in event recognition in Evita are: determination of event candidates and then the events, identification of grammatical features of events, additional clustering of event chunks for event detection and grammatical feature identification in some situations.
For event identification, Evita looks at the lexical items tagged by the preprocessing step. It uses different strategies for identifying events in the three categories: verbs, nouns and adjectives. For identifying events in a verbal chunk, Evita performs lexical look-up and limited contextual parsing in order to exclude weak stative predicates such as be and generics such as verbs with bare plural subjects. Identifying events expressed by nouns involves a phase of lexical look-up and disambiguation using WordNet, and by mapping events SemCor and TimeBank 1.2 to WordNet synsets. Evita consults 25 subtrees from WordNet where all the synsets denote events. One of these, the largest, is the tree underneath the sunset that contains the word event. If the result of this lexical look-up is not conclusive (i.e., if a nominal occurs as both event and non-event in WordNet), a disambiguation step is applied, based on rules learned by a Bayesian classifier trained on SemCor. To identify events from adjectives, Evita uses a conservative approach, where it tags only those adjectives that were annotated as such in TimeBank 1.2, when such adjectives occur as the head of a predicative complement.
To identify grammatical features (e.g., tense, aspect, modality, polarity and non-finite morphology) of events, Evita uses different procedures based on the part of speech of the event denoting expression. But, in general it involves using morphology, pattern matching, and applying a large number (e.g., 140 such rules for verbal chunks) simple linguistic rules. However, to identify the event class, it performs lexical look-up and word sense disambiguation. Clustering is used to identify chunks from the preprocessing stage, that contribute information about the same event, e.g., when some modal auxiliaries and use of copular verbs. Clustering is activated by specific triggers such as the presence of a chunk headed by an auxiliary verb or a copular verb.
Evaluation of Evita was performed by comparing its performance against TimeBanck 1.2. The reported performance was that Evita had 74.03% precision, 87.31% recall and an F-measure of 80.12% in event detection. Accuracy (precision?) for polarity, aspect and modality was over 97% in each case.
: BIBREF138 use TimeBank-annotated events and identify which words and phrases are events. They consider event identification as a classification task that works on word-chunks. They use the BIO formulation that augments each class label with whether the word is the Beginning, Inside or Outside of a chunk BIBREF143 .
They use a number of features, categorized into various classes, for machine learning. These include affix features (e.g., three or four characters from the beginning and end of each word), morphological features (e.g., base form of the word, and base form of any verb associated with the word if the word is a noun or gerund, for example), word-class features (e.g., POS tags, which noun or verb cluster a word belongs to where the clusters are obtained using co-occurrence statistics in the manner of BIBREF144 ), governing features (e.g., governing light verb, determiner type—cardinal or genitive, for example), and temporal features (e.g., a BIO label indicating whether the word is contained inside a TIMEX2 temporal annotation, a governing temporal preposition like since, till, before, etc.). They also use negation features and Wordnet hypernyms as features. For classification, they use the TinySVM implementation of SVM by BIBREF145 .
They perform experiments with TimeBank documents using a 90% stratified sampling for training and 10% for testing. They obtained 82% precision and 71% recall, with an F-measure of 0.759. They did compare their algorithm with an version of Evita they programmed themselves; this system obtained 0.727 F-measure, and thus Bethard and Martin's approached performed about 4% better. When Bethard and Martin's system was extended to identifying semantic class of an event, it did not perform as well, obtaining precision of 67%, recall of 51%, and F-measure of 0.317. However, the system was much better at identifying the classes of verbs with F-measure of 0.707 compared to finding classes of nouns with an F-measure of 0.337 only.
TIPSem (Temporal Information Processing based on Semantic information) is a system that participated in the TemEval-2 Competition BIBREF146 in 2010, which presented several tasks to participants, although we are primarily interested in the event extraction task. TIPSem achieved the best F1 score in all the tasks in TempEval-2 for Spanish, and for English it obtained the best F1 metric in the task of extracting events, which required the recognition and classification of events as defined by TimeML EVENT tag.
TIPSem learns Conditional Random Field (CRF) models using features for different language analysis levels, although the approach focuses on semantic information, primarily semantic roles and semantic networks. Conditional Random Fields present a popular and efficient machine learning technique for supervised sequence labeling BIBREF147 .
The features used for training the CRF models are similar to one used by others such as Bethard and Martin, although details vary. However, they add semantic role labels to the mix of features. In particular, they identify roles for each governing verb. Semantic role labeling BIBREF148 , BIBREF149 , BIBREF150 identifies for each predicate in a sentence, semantic roles and determine their arguments (agent, patient, etc.) and their adjuncts (locative, temporal, etc.). The previous two features were combined in TIPSem to capture the relation between them. The authors think this combination introduces additional information by distinguishing roles that are dependent on different verbs. The importance of this falls especially on the numbered roles (A0, A1, etc.) meaning different things when depending on different verbs.
The test corpus consists of 17K words for English and 10K words for Spanish, provided by the organizers of TempEval-2. For English, they obtained precision of 0.81, recall of 0.86 and F-measure of 0.83 for recognition with event classification accuracy of 0.79; for Spanish the numbers were 0.90, 0.86, 0.88 for recognition and 0.66 for classification accuracy. We provide these numbers although we know that it is difficult to compare one system with another, for example Bethard and Martin's system with TIPSem since the corpora used are difference.
As in TempEval-2, TempEval-3 BIBREF151 participants took part in a task where they had to determine the extent of the events in a text as defined by the TimeML EVENT tag. In addition, systems may determine the value of the features CLASS, TENSE, ASPECT, POLARITY, MODALITY and also identify if the event is a main event or not. The main attribute to annotate is CLASS.
The TempEval-3 dataset was mostly automatically generated, using a temporal merging system. The half-million token text corpus from English Gigaword2 was automatically annotated using TIPSem, TIPSem-B BIBREF140 and TRIOS BIBREF152 . These systems were re-trained on the TimeBank and AQUAINT corpus, using the TimeML temporal relation set. The outputs of these three state-of-the-art system were merged using a merging algorithm BIBREF151 . The dataset used comprised about 500K tokens of “silver" standard data and about 100K tokens of “gold" standard data for training, compared to the corpus of roughly 50K tokens corpus used in TempEval 1 and 2.
There were seven participants and all the participants except one used machine learning approaches. The top performing system was ATT-1 BIBREF153 with precision 81.44, recall 80;67 and F1 of 81.05 for event recognition, and 71.88 for event classification. Close behind was the ATT-2 system BIBREF153 with precision, recall and F-1 of 81.02, 80.81 and 80.92 for event recognition respectively, and 71.10 for event classification. Both systems used MaxEnt classifiers with
Obviously, different sets of features impact on the performance of event recognition and classification BIBREF154 , BIBREF155 , BIBREF156 . In particular, BIBREF157 also examined performance based on different sizes of n-grams in a small scale (n=1,3). Inspired by such work, in building the ATT systems, the creators intended to systematically investigate the performance of various models and for each task, they trained twelve models exploring these two dimensions, three of which we submitted for TempEval-3, and of these three performed among the top ten in TempEval-3 Competition.
The ATT-1 models include lexical, syntactic and semantic features, ATT-2 models include only lexical and syntactic features, and ATT-3 models include only lexical features, i.e., words. They experimented with context windows of 0, 1, 3, and 7 words preceding and following the token to be labeled. For each window size, they trained ATT-1, ATT-2 and ATT-3 models. The ATT-1 models had 18 basic features per token in the context window for up to 15 tokens, so up to 270 basic feaures for each token to be labeled. The ATT-2 models had 16 basic features per token in the context window, so up to 240 basic features for each token to be labeled. The ATT-3 models had just 1 basic feature per token in the context window, so up to 15 basic features for each token to be labeled.
For event extraction and classification, and event feature classification, they used the efficient binary MaxEnt classifier for multi-class classification, available in the machine learning toolkit LLAMA BIBREF158 . They also used LLAMA's pre-processor to build unigram, bigram and trigram extended features from basic features.
For event and time expression extraction, they trained BIO classifiers. It was found that the absence of semantic features causes only small changes in F1. The absence of syntactic features causes F1 to drop slightly (less than 2.5% for all but the smallest window size), with recall decreasing while precision improves somewhat. F1 is also impacted minimally by the absence of semantic features, and about 2-5% by the absence of syntactic features for all but the smallest window size.1
A was surprising that that ATT-3 models that use words only performed well, especially in terms of precision (precision, recall and F2 of 81.95, 75.57 and 78.63 for event recognition, and 69.55 F1 for event classification) . It is also surprising that the words only models with window sizes of 3 and 7 performed as well as the models with a window size of 15. These results are promising for “big dataÓ text analytics, where there may not be time to do heavy preprocessing of input text or to train large models.
We have already discussed several approaches to extraction of events represented by TimeML representation. Extracting events that use other representation is not very different, but different representations have existed and exist, and therefore we briefly present some such attempts. Some of these predate the time TimeML became popular. For example, the various Message Understanding Conferences (MUCs, seven were organize by DARPA from 1987 to 1997), asked participants to extract a small number of relations and events. For instance, MUC-7, the last one, called for the extraction of 3 relations (person-employer, maker-product, and organization-location) and 1 event spacecraft launches.
For example, the MUC-7 and ACE events did not attempt to cover all events, but a limited number of pre-specified event types or classes that participants need to detect during a contest period, based on which the contestants submit papers for publication. The number and the type of arguments covered are also limited and are pre-specified before the competitions start.
BIBREF159 discuss a relation and event extraction system covering areas such as political, financial, business, military, and life-related topics. The system consists of tagging modules, a co-reference resolution module, and a temple generation module. They store the events generated in MUC-7 BIBREF160 format, which is not very unlike the ACE format.
Events are extracted along with their event participants, e.g., who did what to whom when and where? For example, for a BUYING event, REES extracts the buyer, the artifact, the seller, and the time and location of the BUYING event. REES covers 61 types of events. There are 39 types of relations.
The tagging component consists of three modules: NameTagger, NPTagger and EventTagger. Each module relies on the same pattern-based extraction engine, but uses different sets of patterns. The NameTagger recognizes names of people, organizations, places, and artifacts (only vehicles in the implemented system). The NPTagger then takes the output of the NameTagger and first recognizes non-recursive Base Noun Phrase (BNP) BIBREF143 , and then complex NPs for only the four main semantic types of NPs, i.e., Person, Organization, Location, and Artifact (vehicle, drug and weapon). The EventTagger recognizes events applying its lexicon-driven, syntactically-based generic patterns.
REES uses a declarative, lexicon-driven approach. This approach requires a lexicon entry for each event-denoting word, which is generally a verb. The lexicon entry specifies the syntactic and semantic restrictions on the verb's arguments. After the tagging phase, REES sends the output through a rule-based co-reference resolution module that resolves: definite noun phrases of Organization, Person, and Location types, and singular personal pronouns. REES outputs the extracted information in the form of either MUC-style templates or XML.
One of the challenges of event extraction is to be able to recognize and merge those event descriptions which refer to the same event. The Template Generation module uses a set of declarative, customizable rules to merge co- referring events into a single event.
The system's recall, precision, and F-Measure scores for the training set (200 texts) and the blind set (208 texts) from about a dozen news sources. On the so-called training set, the system achieved F-measure of 64.75 for event extraction and 75.35 for relation extraction. The blind set F-Measure for 31 types of relations (73.95
As seen earlier in Subsection "ACE Events" , the way ACE events are specified, they have a lot of details that need to be extracted. BIBREF161 follows several steps to extract events and uses machine learning algorithms at every step. The steps are pre-processing of text data, identifying anchors, assigning event types, extracting arguments identifying attributes of events such as modality, polarity, genericity and tense, and finally identifyings event co-referents of the same individuated event. In other words, Ahn attempts to cover all the steps sequentially, making the simplifying assumption that they are unrelated to each other.
A single place in a textual document which may be considered the primary place of reference or discussion about an event is called the event anchor. Ahn treats finding the anchor for an event within a document as a word classification task, using a two-stage classification process. He uses a binary classifier to classify a word as being an event anchor or not. He then classifies those identified as event anchors into one of the event classes. Ahn used one classifier for binary classification and then another classifiers to classify only the positive instances.
Ahn treats identifying event arguments as a pair classification task. Each event mention is paired with each of the entity, time and value mentions occurring in the same sentence to form a single classification instance. There were 35 role types in the ACE 2006 task, but no event type allows arguments of all types. Each event type had its own set of allowable roles. The classification experiment run was a multi-class classification where a separate multi-class classifier was used for each event type. Ahn trains a separate classifier for each attribute. Genericity, modality, and polarity are each binary classification tasks, while tense is a multi-class task. For event coreference, Ahn follows the approach given in BIBREF162 . Each event mention in a document is paired with every other event mention, and a classifier assigns to each pair of mentions the probability that the paired mentions corefer. These probabilities are used in a left-to-right entity linking algorithm in which each mention is compared with all already-established events (i.e., event mention clusters) to determine whether it should be added to an existing event or start a new one.
Ahn experimented with various combinations of a maximum entropy classifier MegaM BIBREF163 and a memory-based nearest neighbor classifier called TIMBL BIBREF164 , for the various tasks.
The ACE specification provided a way to measure the performance of an event extraction system. The evaluation called ACE value is obtained by scoring each of the component tasks individually and then obtaining a normalized summary value. Overall, using the best learned classifiers for the various subtasks, they achieve an ACE value score of 22.3%, where the maximum score is 100%. The value is low, but other systems at the time had comparable performance.
BIBREF165 describe an approach to classify sentences in a document as specifying one or more events from a certain ACE 2006 class. They classify each sentence in a document as containing an instance of a certain type or not. Unlike BIBREF161 , they are not interested in identifying arguments or any additional processing. Also, unlike Ahn who classifies each word as possibly being an event anchor for a specific type of ACE event, Naughton et al. perform a classification of each sentence in a document as being an on-event sentence or an off-event sentence. An on-event sentence is a sentence that contains one or more instances of the target event type. An off-event sentence is a sentence that does not contain any instances of the target event type. They use several approaches to classify a sentence as on-event or off-event. These include the following: SVM-based machine learning BIBREF166 , language modeling approaches using count smoothing, and a manual approach which looks for Wordnet synonyms or hypernyms of certain trigger words in a sentence.
Naughton et al. found that 1) use of a large number of features to start but then reduction of these features using information gain, and 2) use of SVM produces the best results although all versions of SVM (i.e., with all features with no reduction, just the terms without complex features, or a selection of terms and other features) all work very well. A “surprising" finding was that the “manual" trigger-based classification approach worked almost as well as the SVM based approaches.
Determining Event Coreference
When an event is mentioned in several places within a document, finding which references are to the same event is called determining event coreference. These are co-referents to the event. Determining when two event mentions in text talk about the same event or co-refer is a difficult problem. As BIBREF167 point out that the events may be actual occurrences or hypothetical events.
BIBREF162 present a statistical language-independent framework for identifying and tracking named, nominal and pronominal references to entities wi

Question: Which datasets are used in this work?
Answer: GENIA corpus