Introduction
The Semantic Web BIBREF0 , BIBREF1 and the growing popularity of Linked Data (data published using Semantic Web technologies) have renewed interest in concept-to-text generation BIBREF2 , especially text generation from ontologies BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 . An ontology provides a conceptualization of a knowledge domain (e.g., wines, consumer electronics) by defining the classes and subclasses of the individuals (entities) in the domain, the possible relations between them etc. The current standard to specify Semantic Web ontologies is owl BIBREF12 , which is based on description logics BIBREF13 , rdf, and rdf schema BIBREF14 . owl2 is the latest version of owl BIBREF15 . Given an owl ontology for a knowledge domain, one can publish on the Web machine-readable data pertaining to that domain (e.g., catalogues of products, their features etc.), with the data having formally defined semantics based on the ontology.
Several equivalent owl syntaxes have been developed, but people unfamiliar with formal knowledge representation have difficulties understanding them BIBREF16 . For example, the following statement defines the class of St. Emilion wines, using the functional-style syntax of owl, one of the easiest to understand.
SubClassOf(:StEmilion
ObjectIntersectionOf(:Bordeaux
ObjectHasValue(:locatedIn :stEmilionRegion)
ObjectHasValue(:hasColor :red)
ObjectHasValue(:hasFlavor :strong)
ObjectHasValue(:madeFrom :cabernetSauvignonGrape)
ObjectMaxCardinality(1 :madeFrom)))
To make ontologies easier to understand, several ontology verbalizers have been developed BIBREF17 , BIBREF18 , BIBREF7 , BIBREF19 , BIBREF20 , BIBREF21 , BIBREF22 , BIBREF8 , BIBREF23 , BIBREF9 . Although verbalizers can be viewed as performing a kind of light natural language generation (nlg), they usually translate the axioms (in our case, owl statements) of the ontology one by one to controlled, often not entirely fluent English statements, typically without considering the coherence of the resulting texts. By contrast, more elaborate nlg systems BIBREF3 , BIBREF24 , BIBREF11 can produce more fluent and coherent multi-sentence texts, but they need domain-specific linguistic resources. For example, Naturalowl BIBREF11 , a publicly available nlg system for owl ontologies, produces the following description of St. Emilion wines from the owl statement above. It needs, however: a sentence plan for each relation (e.g., :locatedIn) of the ontology, i.e., a linguistically annotated template showing how to express the relation; a natural language name for each class and individual, i.e., a linguistically annotated noun phrase to be used as the name of the class or individual; a text plan specifying the order in which relations should be expressed etc. Similar domain-specific linguistic resources are used in most concept-to-text systems BIBREF2 . Manually constructing resources of this kind, however, can be tedious and costly.
St. Emilion is a kind of red, strong Bordeaux from the St. Emilion region. It is made from exactly one grape variety: Cabernet Sauvignon grapes.
Instead of requiring domain-specific linguistic resources, simpler verbalizers use the owl identifiers of classes and individuals (e.g., :cabernetSauvignonGrape) typically split into tokens as their natural language names, they express relations using phrases obtained by tokenizing the owl identifiers of the relations (e.g., :hasColor), they order the resulting sentences following the ordering of the corresponding owl statements etc. Without domain-specific linguistic resources, Naturalowl behaves like a simple verbalizer, producing the following lower quality text from the owl statement above. A further limitation of using tokenized owl identifiers is that non-English texts cannot be generated, since owl identifiers are usually English-like.
St Emilion is Bordeaux. St Emilion located in St Emilion Region. St Emilion has color Red. St Emilion has flavor Strong. St Emilion made from grape exactly 1: Cabernet Sauvignon Grape.
Previous experiments BIBREF11 indicate that the texts that Naturalowl generates with domain-specific linguistic resources are perceived as significantly better than (i) those of swat, one of the best available owl verbalizers BIBREF23 , BIBREF10 , and (ii) those of Naturalowl without domain-specific linguistic resources, with little or no difference between (i) and (ii). The largest difference in the perceived quality of the texts was reported to be due to the sentence plans, natural language names, and (to a lesser extent) text plans.
In this paper, we present methods to automatically or semi-automatically extract from the Web the natural language names and sentence plans required by Naturalowl for a given ontology. We do not examine how other types of domain-specific linguistic resources (e.g., text plans) can be generated, leaving them for future work. We base our work on Naturalowl, because it appears to be the only open-source nlg system for owl that implements all the processing stages of a typical nlg pipeline BIBREF2 , it supports owl2, it is extensively documented, and has been tested with several ontologies. The processing stages and linguistic resources of Naturalowl, however, are typical of nlg systems BIBREF25 . Hence, we believe that our work is also applicable, at least in principle, to other nlg systems. Our methods may also be useful in simpler verbalizers, where the main concern seems to be to avoid manually authoring domain-specific linguistic resources. Experiments show that texts generated using linguistic resources extracted by our methods with minimal human involvement are perceived as being almost as good as texts generated using manually authored linguistic resources, and much better than texts produced by using tokenized owl identifiers.
Section SECREF2 below provides background information about Naturalowl, especially its natural language names and sentence plans. Sections SECREF3 and SECREF4 then describe our methods to extract natural language names and sentence plans, respectively, from the Web. Section SECREF5 presents our experimental results. Section SECREF6 discusses related work. Section SECREF7 concludes and suggests future work.
Background information about NaturalOWL
Given an owl ontology and a particular target individual or class to describe, Naturalowl first scans the ontology to select statements relevant to the target. It then converts each relevant statement into (possibly multiple) message triples of the form INLINEFORM0 , where INLINEFORM1 is an individual or class, INLINEFORM2 is another individual, class, or datatype value, and INLINEFORM3 is a relation (property) that connects INLINEFORM4 to INLINEFORM5 . For example, the ObjectHasValue(:madeFrom :cabernetSauvignonGrape) part of the owl statement above is converted to the message triple INLINEFORM6 :StEmilion, :madeFrom, :cabernetSauvignonGrape INLINEFORM7 . Message triples are similar to rdf triples, but they are easier to express as sentences. Unlike rdf triples, the relations ( INLINEFORM8 ) of the message triples may include relation modifiers. For example, the ObjectMaxCardinality(1 :madeFrom) part of the owl statement above is turned into the message triple INLINEFORM9 :StEmilion, maxCardinality(:madeFrom), 1 INLINEFORM10 , where maxCardinality is a relation modifier. In this paper, we consider only sentence plans for message triples without relation modifiers, because Naturalowl already automatically constructs sentence plans for triples with relation modifiers from sentence plans for triples without them.
Having produced the message triples, Naturalowl consults a user model to select the most interesting ones that have not been expressed already, and orders the selected triples according to manually authored text plans. Later processing stages convert each message triple to an abstract sentence representation, aggregate sentences to produce longer ones, and produce appropriate referring expressions (e.g., pronouns). The latter three stages require a sentence plan for each relation ( INLINEFORM0 ), while the last stage also requires natural language names for each individual or class ( INLINEFORM1 or INLINEFORM2 ).
The natural language names of NaturalOWL
In Naturalowl, a natural language (nl) name is a sequence of slots. The contents of the slots are concatenated to produce a noun phrase to be used as the name of a class or individual. Each slot is accompanied by annotations specifying how to fill it in; the annotations may also provide linguistic information about the contents of the slot. For example, we may specify that the English nl name of the class :TraditionalWinePiemonte is the following.
[ ] INLINEFORM0 [traditional] INLINEFORM1 [wine] INLINEFORM2 [from] INLINEFORM3 [ ] INLINEFORM4 [Piemonte] INLINEFORM5 [region] INLINEFORM6
The first slot is to be filled in with an indefinite article, whose number should agree with the third slot. The second slot is to be filled in with the adjective `traditional'. The third slot with the neuter noun `wine', which will also be the head (central) noun of the noun phrase, in singular number, and similarly for the other slots. Naturalowl makes no distinctions between common and proper nouns, but it can be instructed to capitalize particular nouns (e.g., `Piemonte'). In the case of the message triple INLINEFORM0 :wine32, instanceOf, :TraditionalWinePiemonte INLINEFORM1 , the nl name above would allow a sentence like “This is a traditional wine from the Piemonte region” to be produced.
The slot annotations allow Naturalowl to automatically adjust the nl names. For example, the system also generates comparisons to previously encountered individuals or classes, as in “Unlike the previous products that you have seen, which were all traditional wines from the Piemonte region, this is a French wine”. In this particular example, the head noun (`wine') had to be turned into plural. Due to number agreement, its article also had to be turned into plural; in English, the plural indefinite article is void, hence the article of the head noun was omitted.
As a further example, we may specify that the nl name of the class FamousWine is the following.
[ ] INLINEFORM0 [famous] INLINEFORM1 [wine] INLINEFORM2
If INLINEFORM0 :wine32, instanceOf, :TraditionalWinePiemonte INLINEFORM1 and INLINEFORM2 :wine32, instanceOf, :FamousWine INLINEFORM3 were both to be expressed, Naturalowl would then produce the single, aggregated sentence “This is a famous traditional wine from the Piemonte region”, instead of two separate sentences “This is a traditional wine from the Piemonte region” and “This is a famous wine”. The annotations of the slots, which indicate for example which words are adjectives and head nouns, are used by the sentence aggregation component of Naturalowl to appropriately combine the two sentences. The referring expression generation component also uses the slot annotations to identify the gender of the head noun, when a pronoun has to be generated (e.g., “it” when the head noun is neuter).
We can now define more precisely nl names. A nl name is a sequence of one or more slots. Each slot is accompanied by annotations requiring it to be filled in with exactly one of the following:
(i) An article, definite or indefinite, possibly to agree with another slot filled in by a noun.
(ii) A noun flagged as the head. The number of the head noun must also be specified.
(iii) An adjective flagged as the head. For example, the nl name name of the individual :red may consist of a single slot, to be filled in with the adjective `red'; in this case, the adjective is the head of the nl name. The number and gender of the head adjective must also be specified.
(iv) Any other noun or adjective, (v) a preposition, or (vi) any fixed string.
Exactly one head (noun or adjective) must be specified per nl name. For nouns and adjectives, the nl name may require a particular inflectional form to be used (e.g., in a particular number, case, or gender), or it may require an inflectional form that agrees with another noun or adjective slot.
When providing nl names, an individual or class can also be declared to be anonymous, indicating that Naturalowl should avoid referring to it by name. For example, in a museum ontology, there may be a coin whose owl identifier is :exhibit49. We may not wish to provide an nl name for this individual (it may not have an English name); and we may want Naturalowl to avoid referring to the coin by tokenizing its identifier (“exhibit 49”). By declaring the coin as anonymous, Naturalowl would use only the nl name of its class (e.g., “this coin”), simply “this”, or a pronoun.
The sentence plans of NaturalOWL
In Naturalowl, a sentence plan for a relation INLINEFORM0 specifies how to construct a sentence to express any message triple of the form INLINEFORM1 . Like nl names, sentence plans are sequences of slots with annotations specifying how to fill the slots in. The contents of the slots are concatenated to produce the sentence. For example, the following is a sentence plan for the relation :madeFrom.
[ INLINEFORM0 ] INLINEFORM1 [make] INLINEFORM2 [from] INLINEFORM3 [ INLINEFORM4 ] INLINEFORM5
Given the message triple INLINEFORM0 :StEmilion, :madeFrom, :cabernetSauvignonGrape INLINEFORM1 , the sentence plan would lead to sentences like “St. Emilion is made from Cabernet Sauvignon grapes”, or “It is made from Cabernet Sauvignon grapes”, assuming that appropriate nl names have been provided for :StEmilion and :cabernetSauvignonGrape. Similarly, given INLINEFORM2 :Wine, :madeFrom, :Grape INLINEFORM3 , the sentence plan above would lead to sentences like “Wines are made from grapes” or “They are made from grapes”, assuming again appropriate nl names. As another example, the following sentence plan can be used with the relations :hasColor and :hasFlavor.
[ INLINEFORM0 ] INLINEFORM1 [be] INLINEFORM2 [ INLINEFORM3 ] INLINEFORM4
Given the message triples INLINEFORM0 :StEmilion, :hasColor, :red INLINEFORM1 and INLINEFORM2 :StEmilion, :hasFlavor, :strong INLINEFORM3 , it would produce the sentences “St. Emilion is red” and “St. Emilion is strong”, respectively.
The first sentence plan above, for :madeFrom, has four slots. The first slot is to be filled in with an automatically generated referring expression (e.g., pronoun or name) for INLINEFORM0 , in nominative case. The verb of the second slot is to be realized in passive voice, present tense, and positive polarity (as opposed to expressing negation) and should agree (in number and person) with the referring expression of the first slot ( INLINEFORM1 ). The third slot is filled in with the preposition `from', and the fourth slot with an automatically generated referring expression for INLINEFORM2 , in accusative case.
Naturalowl has built-in sentence plans for domain-independent relations (e.g., isA, instanceOf). For example, INLINEFORM0 :StEmilion, isA, :Bordeaux INLINEFORM1 is expressed as “St. Emilion is a kind of Bordeaux” using the following built-in sentence plan; the last slot requires the nl name of INLINEFORM2 without article.
[ INLINEFORM0 ] INLINEFORM1 [be] INLINEFORM2 [“a kind of”] INLINEFORM3 [ INLINEFORM4 ] INLINEFORM5
Notice that the sentence plans of Naturalowl are not simply slotted string templates (e.g., “ INLINEFORM0 is made from INLINEFORM1 ”). Their linguistic annotations (e.g., pos tags, agreement, voice, tense, cases) along with the annotations of the nl names allow Naturalowl to produce more natural sentences (e.g., turn the verb into plural when the subject is also plural), produce appropriate referring expressions (e.g., pronouns in the correct cases and genders), and aggregate shorter sentences into longer ones. For example, the linguistic annotations of the nl names and sentence plans allow Naturalowl to produce the aggregated sentence “St. Emilion is a kind of red Bordeaux made from Cabernet Sauvignon grapes” from the triples INLINEFORM2 :StEmilion, isA, :Bordeaux INLINEFORM3 , INLINEFORM4 :StEmilion, :hasColor, :red INLINEFORM5 , INLINEFORM6 :StEmilion, :madeFrom, :cabernetSauvignonGrape INLINEFORM7 , instead of three separate sentences.
We can now define more precisely sentence plans. A sentence plan is a sequence of slots. Each slot is accompanied by annotations requiring it to be filled in with exactly one of the following:
(i) A referring expression for the INLINEFORM0 (a.k.a. the owner) of the message triple in a particular case.
(ii) A verb in a particular polarity and form (e.g., tense), possibly to agree with another slot.
(iii) A noun or adjective in a particular form, possibly to agree with another slot.
(iv) A preposition, or (v) a fixed string.
(vi) A referring expression for the INLINEFORM0 (a.k.a. the filler) of the message triple.
More details about the nl names and sentence plans of Naturalowl and their roles in sentence aggregation, referring expressions etc. can be found elsewhere BIBREF11 . Both sentence plans and nl names were so far authored manually, using a Protégé plug-in (Fig. FIGREF9 ).
Our method to extract natural language names from the Web
Given a target class or individual INLINEFORM0 that we want to produce an nl name for, we first extract from the Web noun phrases that are similar to the owl identifier of INLINEFORM1 . The noun phrases are ranked by aligning their words to the tokens of the identifier. The top-ranked noun phrases are then enhanced with linguistic annotations (e.g., pos tags, agreement, number), missing articles etc., turning them into nl names. We aim to identify the best few (up to 5) candidate nl names for INLINEFORM2 . In a fully automatic scenario, the candidate nl name that the method considers best for INLINEFORM3 is then used. In a semi-automatic scenario, the few top (according to the method) nl names of INLINEFORM4 are shown to a human author, who picks the best one; this is much easier than manually authoring nl names.
Extracting noun phrases from the Web
We first collect the owl statements of the ontology that describe INLINEFORM0 , the individual or class we want to produce an nl name for, and turn them into message triples INLINEFORM1 , as when generating texts. For example, for the class INLINEFORM2 :KalinCellarsSemillon of the Wine Ontology, one of the ontologies of our experiments, three of the resulting message triples are:
INLINEFORM0 :KalinCellarsSemillon, isA, :Semillon INLINEFORM1
INLINEFORM0 :KalinCellarsSemillon, :hasMaker, :KalinCellars INLINEFORM1
INLINEFORM0 :KalinCellarsSemillon, :hasFlavor, :Strong INLINEFORM1
For each collected message triple INLINEFORM0 , we then produce INLINEFORM1 and INLINEFORM2 , where INLINEFORM3 is the tokenized identifier of INLINEFORM4 . From the three triples above, we obtain:
tokName(:KalinCellarsSemillon) INLINEFORM0 “Kalin Cellars Semillon” , tokName(:Semillon) INLINEFORM1 “Semillon”
tokName(:KalinCellars) INLINEFORM0 “Kalin Cellars”, tokName(:Strong) INLINEFORM1 “Strong”
Subsequently, we attempt to shorten INLINEFORM0 , i.e., the tokenized identifier of the individual or class we wish to produce an nl name for, by removing any part (token sequence) of INLINEFORM1 that is identical to the tokenized identifier of the INLINEFORM2 of any triple INLINEFORM3 that we collected for INLINEFORM4 . If the shortened tokenized identifier of INLINEFORM5 is the empty string or contains only numbers, INLINEFORM6 is marked as anonymous (Section SECREF3 ). In our example, where INLINEFORM7 :KalinCellarsSemillon, the tokenized identifier of INLINEFORM8 is initially INLINEFORM9 “Kalin Cellars Semillon”. We remove the part “Semillon”, because of the triple INLINEFORM10 :KalinCellarsSemillon, :isA, :Semillon INLINEFORM11 and the fact that tokName(:Semillon) INLINEFORM12 “Semillon”, as illustrated below. We also remove the remaining part “Kalin Cellars”, because of INLINEFORM13 :KalinCellarsSemillon, :hasMaker, :KalinCellars INLINEFORM14 and the fact that tokName(:KalinCellars) INLINEFORM15 “Kalin Cellars”. Hence, :KalinCellarsSemillon is marked as anonymous.
Anonymizing :KalinCellarsSemillon causes Naturalowl to produce texts like (a) below when asked to describe :KalinCellarsSemillon, rather than (b), which repeats “Semillon” and “Kalin Cellars”:
(a) This is a strong, dry Semillon. It has a full body. It is made by Kalin Cellars.
(b) Kalin Cellars Semillon is a strong, dry Semillon. It has a full body. It is made by Kalin Cellars.
Similarly, if INLINEFORM0 :SouthAustraliaRegion and we have collected the following message triple, the tokenized identifier of INLINEFORM1 would be shortened from “South Australia Region” to “South Australia”. We use altTokName to denote the resulting shortened tokenized identifiers.
INLINEFORM0 :SouthAustraliaRegion, :isA, :Region INLINEFORM1
tokName(:SouthAustraliaRegion) INLINEFORM0 “South Australia Region”, tokName(:Region) INLINEFORM1 “Region”
altTokName(:SouthAustraliaRegion) INLINEFORM0 “South Australia”
Also, if INLINEFORM0 :exhibit23 and we have collected the following triple, altTokName(:exhibit23) would end up containing only numbers (“23”). Hence, :exhibit23 is marked as anonymous.
INLINEFORM0 :exhibit23, :isA, :Exhibit INLINEFORM1
tokName(:exhibit23) INLINEFORM0 “exhibit 23”, tokName(:Exhibit) INLINEFORM1 “exhibit”
We then collect the tokenized identifiers of all the ancestor classes of INLINEFORM0 , also taking into account equivalent classes; for example, if INLINEFORM1 has an equivalent class INLINEFORM2 , we also collect the tokenized identifiers of the ancestor classes of INLINEFORM3 . For INLINEFORM4 :KalinCellarsSemillon, we collect the following tokenized identifiers, because :Semillon, :SemillonOrSauvignonBlanc, and :Wine are ancestors of INLINEFORM5 .
tokName(:Semillon) INLINEFORM0 “Semillon”, tokName(:SemillonOrSauvignonBlanc) INLINEFORM1 “Semillon Or Sauvignon Blanc”, tokName(:Wine) INLINEFORM2 “Wine”
If INLINEFORM0 does not contain any of the collected tokenized identifiers of the ancestor classes of INLINEFORM1 , we create additional alternative tokenized identifiers for INLINEFORM2 , also denoted INLINEFORM3 , by appending to INLINEFORM4 the collected tokenized identifiers of the ancestor classes of INLINEFORM5 . For example, if INLINEFORM6 :red and :Color is the parent class of INLINEFORM7 ( INLINEFORM8 :red, isA, :Color INLINEFORM9 ), we also obtain “red color”:
tokName(:red) INLINEFORM0 “red”, tokName(:Color) INLINEFORM1 “color”, altTokName(:red) INLINEFORM2 “red color”
By contrast, if INLINEFORM0 :KalinCellarsSemillon, no INLINEFORM1 is produced from the ancestors of INLINEFORM2 , because INLINEFORM3 “Kalin Cellars Semillon” contains tokName(:Semillon) INLINEFORM4 “Semillon”, and :Semillon is an ancestor of :KalinCellarsSemillon.
Furthermore, we create an additional INLINEFORM0 by removing all numbers from INLINEFORM1 ; for example, from INLINEFORM2 “Semillon 2006” we obtain INLINEFORM3 “Semillon”. Lastly, if INLINEFORM4 contains brackets, we create an INLINEFORM5 for each part outside and inside the brackets; for example, from “gerbil (dessert rat)” we get “gerbil” and “dessert rat”.
Subsequently, we formulate a Boolean Web search query for INLINEFORM0 (e.g., “South” AND “Australia” AND “Region”) and each INLINEFORM1 (e.g., “South” AND “Australia”); recall that INLINEFORM2 is the individual or class we wish to produce an nl name for. We convert the retrieved pages of all the queries to plain text documents and parse every sentence of the text, if any stemmed word of the sentence is the same as any stemmed word of any INLINEFORM3 or INLINEFORM4 . We then extract the noun phrases (nps) from every parsed sentence. For example, from the sentence “the Naples National Archaeological Museum houses some of the most important classical collections” we extract the nps “the Naples National Archaeological Museum”, “some of the most important classical collections”, and “the most important classical collections” (Fig. FIGREF19 ).
For each extracted np, we compute its similarity to INLINEFORM0 and each INLINEFORM1 . Let np be an extracted np and let name be INLINEFORM2 or an INLINEFORM3 . To compute the similarity between np and name, we first compute the character-based Levenshtein distance between each token of np and each token of name; we ignore upper/lower case differences, articles, and connectives (e.g. “or”), which are often omitted from owl identifiers. In the following example, np INLINEFORM4 “the Naples National Archaeological Museum” (but “the” is ignored) and name = “national arch napoli museum”; this name is an INLINEFORM5 produced by appending to INLINEFORM6 the tokenized identifier of the parent class (:Museum) of INLINEFORM7 (Section UID14 ). The Levenshtein distance between “national” and “National” is 0 (upper/lower case differences are ignored). The distance between “napoli” and “Naples” is 4; a character deletion or insertion costs 1, a replacement costs 2.
[column sep=0em,row sep=.4in] outer sep=0pt,anchor=base] (T) ; outer sep=0pt,anchor=base] (A) national; outer sep=0pt,anchor=base] (B) arch; outer sep=0pt,anchor=base] (C) napoli; outer sep=0pt,anchor=base] (D) museum;
outer sep=0pt,anchor=base] (t) (the); outer sep=0pt,anchor=base] (a) Naples; outer sep=0pt,anchor=base] (b) National; outer sep=0pt,anchor=base] (c) Archaeological; outer sep=0pt,anchor=base] (d) Museum;
; (A) – (b) node[draw=none,fill=none,font=,midway,left] 0; (B) – (c) node[draw=none,fill=none,font=,midway,right] 10; (C) – (a) node[draw=none,fill=none,font=,midway,above] 4; (D) – (d) node[draw=none,fill=none,font=,midway,left] 0;
We then form pairs of aligned tokens INLINEFORM0 , where INLINEFORM1 , INLINEFORM2 are tokens from name, np, respectively, such that each token of name is aligned to at most one token of np and vice versa, and any other, not formed pair INLINEFORM3 would have a Levenshtein distance (between INLINEFORM4 , INLINEFORM5 ) larger or equal to the minimum Levensthein distance of the formed pairs. In our example, the pairs of alinged tokens are INLINEFORM6 “national”, “National” INLINEFORM7 , INLINEFORM8 “arch”, “Archaeological” INLINEFORM9 , INLINEFORM10 “napoli”, “Naples” INLINEFORM11 , INLINEFORM12 “museum”, “Museum” INLINEFORM13 .
The similarity between np and name is then computed as follows, where INLINEFORM0 is the set of aligned token pairs, INLINEFORM1 is the Levenshtein distance (normalized to INLINEFORM2 ) between the INLINEFORM3 and INLINEFORM4 of pair INLINEFORM5 , and INLINEFORM6 , INLINEFORM7 are the lengths (in tokens) of np and name, respectively. DISPLAYFORM0
For each extracted np of INLINEFORM0 , we compute its similarity to every possible name, i.e., INLINEFORM1 or INLINEFORM2 , as discussed above, and we assign to the np a score equal to the largest of these similarities. Finally, we rank the extracted nps of INLINEFORM3 by decreasing score. If two nps have the same score, we rank higher the np with the fewest crossed edges in its best alignment with a name. If two nps still cannot be distinguished, we rank them by decreasing frequency in the parsed sentences of INLINEFORM4 ; and if their frequencies are equal, we rank them randomly.
Turning the extracted noun phrases into natural language names
The extracted nps are not yet nl names, because they lack the linguistic annotations that Naturalowl requires (e.g., pos tags, agreement, number); they may also lack appropriate articles. To convert an np to an nl name, we first obtain the pos tags of its words from the parse tree of the sentence the np was extracted from. For example, the np “the Red Wine” becomes:
the INLINEFORM0 Red INLINEFORM1 Wine INLINEFORM2
For every noun, adjective, article, preposition, we create a corresponding slot in the nl name; all the other words of the np become slots containing the words as fixed strings (Section SECREF3 ). For nouns and adjectives, the base form is used in the slot (e.g., “wine” istead of “wines”), but slot annotations indicate the particular inflectional form that was used in the np; e.g., the nn pos tag shows that “wine” is singular. A named-entity recognizer (ner) and an on-line dictionary are employed to detect nouns that refer to persons and locations. The genders of these nouns are determined using the on-line dictionary, when possible, or defaults otherwise (e.g., the default for person nouns is a `person' pseudo-gender, which leads to “he/she” or “they” when generating a pronoun). Nouns not referring to persons and locations are marked as neuter. Since the nps are extracted from Web pages, there is a risk of wrong capitalization (e.g., “the RED wine”). For each word of the nl name, we pick the capitalization that is most frequent in the retrieved texts of the individual or class we generate the nl name for. Hence, the np “the Red Wines” becomes:
[] INLINEFORM0 [red] INLINEFORM1 [wine] INLINEFORM2
which requires a definite article, followed by the adjective “red”, and the neuter “wine” in singular.
A dependency parser is then used to identify the head of each nl name (Section SECREF3 ) and to obtain agreement information. Adjectives are required to agree with the nouns they modify, and the same applies to articles and nouns. At this stage, the np “the Red Wines” will have become:
[] INLINEFORM0 [red] INLINEFORM1 [wine] INLINEFORM2
We then consider the main article (or, more generally, determiner) of the nl name, i.e., the article that agrees with the head (e.g., “a” in “a traditional wine from the Piemonte Region”). Although the nl name may already include a main article, it is not necessarily an appropriate one. For example, it would be inappropriate to use a definite article in “The red wine is a kind of wine with red color”, when describing the class of red wines. We modify the nl name to use an indefinite article if the nl name refers to a class, and a definite article if it refers to an individual (e.g., “the South Australia region”). The article is omitted if the head is an adjective (e.g., “strong”), or in plural (e.g., “Semillon grapes”), or if the entire nl name (excluding the article, if present) is a proper name (e.g., “South Australia”) or a mass noun phrase without article (e.g., “gold”). Before inserting or modifying the main article, we also remove any demonstratives (e.g., “this statue”) or other non-article determiners (e.g., “some”, “all”) from the beginning of the nl name . In our example, the nl name is to be used to refer to the class :RedWine, so the final nl name is the following, which would lead to sentences like “A red wine is a kind of wine with red color”.
[] INLINEFORM0 [red] INLINEFORM1 [wine] INLINEFORM2
Recall that Naturalowl can automatically adjust nl names when generating texts (Section SECREF3 ). For example, in a comparison like “Unlike the previous red wines that you have seen, this one is from France”, it would use a definite article and it would turn the head noun of the nl name to plural, also adding the adjective “previous”. The resulting nl names are finally ranked by the scores of the np s they were obtained from (Section UID16 ).
Inferring interest scores from natural language names
The reader may have already noticed that the sentence “A red wine is a kind of wine with red color” that we used above sounds redundant. Some message triples lead to sentences that sound redundant, because they report relations that are obvious (to humans) from the nl names of the individuals or classes. In our example, the sentence reports the following two message triples.
INLINEFORM0 :RedWine, isA, :Wine INLINEFORM1 , INLINEFORM2 :RedWine, :hasColor, :Red INLINEFORM3
Expressed separately, the two triples would lead to the sentences “A red wine is a kind of wine” and “A red wine has red color”, but Naturalowl aggregates them into a single sentence. The “red color” derives from an INLINEFORM0 of :Red obtained by considering the parent class :Color of :Red (Section UID14 ). It is obvious that a red wine is a wine with red color and, hence, the two triples above should not be expressed. Similarly, the following triple leads to the sentence “A white Bordeaux wine is a kind of Bordeaux”, which again seems redundant.
INLINEFORM0 :WhiteBordeaux, isA, :Bordeaux INLINEFORM1
Naturalowl provides mechanisms to manually assign interest scores to message triples BIBREF11 . Assigning a zero interest score to a triple instructs Naturalowl to avoid expressing it. Manually assigning interest scores, however, can be tedious. Hence, we aimed to automatically assign zero scores to triples like the ones above, which report relations that are obvious from the nl names. To identify triples of this kind, we follow a procedure similar to the one we use to identify individuals or classes that should be anonymous (Section UID12 ). For each INLINEFORM0 triple that involves the individual or class INLINEFORM1 being described, we examine the nl names of INLINEFORM2 and INLINEFORM3 . If all the (lemmatized) words of the phrase produced by the nl name of INLINEFORM4 (e.g., “a Bordeaux”), excluding articles, appear in the phrase of the nl name of INLINEFORM5 (e.g., “a white Bordeaux”), we assign a zero interest score to INLINEFORM6 . INLINEFORM7
Our method to automatically extract sentence plans from the Web
To produce a sentence plan for a relation, we first extract slotted string templates (e.g., “ INLINEFORM0 is made from INLINEFORM1 ”) from the Web using seeds (values of INLINEFORM2 ) from the ontology. We then enhance the templates by adding linguistic annotations (e.g., pos tags, agreement, voice, tense) and missing components (e.g., auxiliary verbs) turning the templates into candidate sentence plans. The candidate sentence plans are then scored by a Maximum Entropy classifier to identify the best few (again up to 5) candidate sentence plans for each relation. In a fully automatic scenario, the sentence plan that the classifier considers best for each relation is used. In a semi-automatic scenario, the few top sentence plans of each relation are shown to a human author, who picks the best one.
Extracting templates from the Web
For each relation INLINEFORM0 that we want to generate a sentence plan for, our method first obtains the owl statements of the ontology that involve the relation and turns them into message triples INLINEFORM1 , as when generating texts. For example, if the relation is :madeFrom, two of the triples may be:
INLINEFORM0 :StEmilion, :madeFrom, :cabernetSauvignonGrape INLINEFORM1 , INLINEFORM2 :Semillon, :madeFrom, :SemillonGrape INLINEFORM3
To these triples, we add more by replacing the INLINEFORM0 , INLINEFORM1 , or both of each originally obtained triple by their classes (if INLINEFORM2 or INLINEFORM3 are individuals), their parent classes, or their equivalent classes. For example, from INLINEFORM4 :StEmilion, :madeFrom, :cabernetSauvignonGrape INLINEFORM5 we also obtain the following three triples, because Wine is a parent class of StEmilion, and Grape is a parent class of :cabernetSauvignonGrape.
INLINEFORM0 :Wine, :madeFrom, :cabernetSauvignonGrape INLINEFORM1
INLINEFORM0 :StEmilion, :madeFrom, :Grape INLINEFORM1 , INLINEFORM2 :Wine, :madeFrom, :Grape INLINEFORM3
We obtain the same additional triples from INLINEFORM0 :Semillon, :madeFrom, :SemillonGrape INLINEFORM1 , because Wine and Grape are also parent classes of Semillon and SemillonGrape, but we remove duplicates.
Each INLINEFORM0 triple is then replaced by a pair INLINEFORM1 , where INLINEFORM2 is a word sequence generated by the nl name of INLINEFORM3 , and similarly for INLINEFORM4 . We assume that the nl names are manually authored, or that they are generated by our method of Section SECREF3 . In the latter case, we keep only one nl name per individual or class, the one selected by the human author (in a semi-automatic setting of nl name generation) or the top ranked nl name (in a fully automatic setting). The five triples above become the following pairs. We call pairs of this kind seed name pairs, and their elements seed names. If a seed name results from a class, parent-class, or an equivalent class of the original INLINEFORM5 or INLINEFORM6 , we consider it a secondary seed name.
INLINEFORM0 “St. Emilion”, “Cabernet Sauvignon grape” INLINEFORM1 , INLINEFORM2 “Semillon”, “Semillon grape” INLINEFORM3
INLINEFORM0 “wine”, “Cabernet Sauvignon grape” INLINEFORM1 , INLINEFORM2 “St. Emilion”, “grape” INLINEFORM3 , INLINEFORM4 “wine”, “grape” INLINEFORM5
We then retrieve Web pages using the seed name pairs (of the relation that we want to generate a sentence plan for) as queries. For each seed name pair, we use the conjunction of its seed names (e.g., “St. Emilion” AND “Cabernet Sauvignon grape”) as a Boolean query. We convert all the retrieved pages (of all the seed name pairs) to plain text documents and parse every sentence of the retrieved documents, if at least one stemmed word from each seed name of a particular pair is the same as a stemmed word of the sentence. We then keep every parsed sentence that contains at least two nps matching a seed name pair. For example, the sentence “obviously Semillon is made from Semillon grapes in California” contains the nps “Semillon” and “Semillon grapes” that match the seed name pair INLINEFORM0 “Semillon”, “Semillon grape” INLINEFORM1 (Fig. FIGREF32 ). Two nps of a sentence match a seed name pair if the similarity between any of the two nps and any of the two seed names (e.g., the first np and the second seed name) is above a threshold INLINEFORM2 and the similarity between the other np and the other seed name is also above INLINEFORM3 . The similarity between an np and a seed name is computed as their weighted cosine similarity, with INLINEFORM4 weights, applied to stemmed nps and seed names, ignoring stop-words. The INLINEFORM5 of a word of the np or seed name is the frequency (usually 0 or 1) of the word in the np or seed name, respectively; the INLINEFORM6 is the inverse document frequency of the word in all the retrieved documents of the relation. We call np anchor pair any two nps of a parsed sentence that match a seed name pair, and np anchors the elements of an np anchor pair.
From every parsed sentence that contains an np anchor pair, we produce a slotted string template by replacing the first np anchor by INLINEFORM0 , the second np anchor by INLINEFORM1 , including between INLINEFORM2 and INLINEFORM3 in the template the words of the sentence that were between the two np anchors, and discarding the other words of the sentence. In the example of Fig. FIGREF32 , we would obtain the template “ INLINEFORM4 is made from INLINEFORM5 ”. Multiple templates may be extracted from the same sentence, if a sentence contains more than one np anchor pairs; and the same template may be extracted from multiple sentences, possibly retrieved by different seed name pairs. We retain only the templates that were extracted from at least two different sentences. We then produce additional templates by increasingly extending the retained ones up to the boundaries of the sentences they were extracted from. In Fig. FIGREF32 , if the template “ INLINEFORM6 is made from INLINEFORM7 ” has been retained, we would also produce the following templates.
obviously INLINEFORM0 is made from INLINEFORM1 obviously INLINEFORM2 is made from INLINEFORM3 in obviously INLINEFORM4 is made from INLINEFORM5 in California
INLINEFORM0 is made from INLINEFORM1 in INLINEFORM2 is made from INLINEFORM3 in California
Again, we discard extended templates that did not result from at least two sentences.
Turning the templates into candidate sentence plans
The templates (e.g., “ INLINEFORM0 is made from INLINEFORM1 ”) are not yet sentence plans, because they lack the linguistic annotations that Naturalowl requires (e.g., pos tags, agreement, voice, tense, cases), and they may also not correspond to well-formed sentences (e.g., they may lack verbs). The conversion of a template to a (candidate) sentence plan is similar to the conversion of Section SECREF22 . We start by obtaining pos tags from the parse trees of the sentences the template was obtained from. Recall that a template may have been extracted from multiple sentences. We obtain a pos tag sequence for the words of the template from each one of the sentences the template was extracted from, and we keep the most frequent pos tag sequence. We ignore the pos tags of the anchor nps, which become INLINEFORM2 and INLINEFORM3 in the template. For example, the template “ INLINEFORM4 is made from INLINEFORM5 ” becomes:
INLINEFORM0 is INLINEFORM1 made INLINEFORM2 from INLINEFORM3 INLINEFORM4
For every verb form (e.g., “is made”), noun, adjective, and preposition, we create a corresponding slot in the sentence plan. For verbs, nouns, and adjectives, the base form is used in the slot; each verb slot is also annotated with the voice and tense of the corresponding verb form in the template. If a negation expression (e.g., “not”, “aren't”) is used with the verb form in the template, the negation expression is not included as a separate slot in the sentence plan, but the polarity of the verb slot is marked as negative; otherwise the polarity is positive. We determine the genders and capitalizations of nouns (and proper names) as in Section SECREF22 . The INLINEFORM0 and INLINEFORM1 are also replaced by slots requiring referring expressions. For example, the template “ INLINEFORM2 is made from INLINEFORM3 ” becomes:
[ INLINEFORM0 ] INLINEFORM1 [make] INLINEFORM2 [from] INLINEFORM3 [ INLINEFORM4 ] INLINEFORM5
Agreement and case information is obtained using a dependency parser. The parser is applied to the sentences the templates were extracted from, keeping the most frequent parse per template. Referring expressions obtained from np anchors that were verb subjects are marked with nominative case, and they are required to agree with their verbs. Referring expressions corresponding to verb objects or preposition complements are marked with accusative case (e.g., “from him”). Referring expressions corresponding to np anchors with head nouns in possessive form (e.g., “Piemonte's”) are marked with possessive case. In our example, we obtain:
[ INLINEFORM0 ] INLINEFORM1 [make] INLINEFORM2 [from] INLINEFORM3 [ INLINEFORM4 ] INLINEFORM5
Any remaining words of the template that have not been replaced by slots (e.g., “obviously” in “obviously INLINEFORM0 is made from INLINEFORM1 ”) are turned into fixed string slots. Subsequently, any sentence plan that has only two slots, starts with a verb, or contains no verb, is discarded, because sentence plans of these kinds tend to be poor. Also, if a sentence plan contains a single verb in the past participle, in agreement with either INLINEFORM2 or INLINEFORM3 , followed by a preposition (e.g. “ INLINEFORM4 made in INLINEFORM5 ”), we insert an auxiliary verb to turn the verb form into present passive (e.g., “ INLINEFORM6 is made in INLINEFORM7 ”); in domains other than those of our experiments, a past passive may be more appropriate (“ INLINEFORM8 was made in INLINEFORM9 ). Similarly, if a single verb appears in the present participle (e.g. “ INLINEFORM10 making INLINEFORM11 ”), we insert an auxiliary verb to obtain a present continuous form. Both cases are illustrated below.
Lastly, we filter the remaining sentence plans through a Web search engine. For this step, we replace referring expression slots by wildcards, we generate the rest of the sentence (e.g., “* is made from *”), and we do a phrase search. If no results are returned, the sentence plan is discarded.
Applying a Maximum Entropy classifier to the candidate sentence plans
The retained candidate sentence plans are then scored using a Maximum Entropy (MaxEnt) classifier. The classifier views each candidate sentence plan INLINEFORM0 for a relation INLINEFORM1 as a vector of 251 features, and attempts to estimate the probability that INLINEFORM2 is a good sentence plan (positive class) for INLINEFORM3 or not (negative class). The 251 features provide information about INLINEFORM4 itself, but also about the templates, seed name pairs, and np anchor pairs of INLINEFORM5 , meaning the templates that INLINEFORM6 was obtained from, and the seed name pairs and np anchor pairs (Fig. FIGREF32 ) that matched to produce the templates of INLINEFORM7 .
The productivity of the INLINEFORM0 -th seed name pair INLINEFORM1 (e.g., INLINEFORM2 INLINEFORM3 “Semillon”, INLINEFORM4 “Semillon grape” INLINEFORM5 ) of a relation INLINEFORM6 (e.g., INLINEFORM7 :madeFrom) is defined as follows: DISPLAYFORM0
where: INLINEFORM0 is the number of times INLINEFORM1 matched any np anchor pair of the parsed sentences of INLINEFORM2 , counting only matches that contributed to the extraction (Section SECREF29 ) of any template of INLINEFORM3 ; INLINEFORM4 is the total number of seed name pairs of INLINEFORM5 ; and INLINEFORM6 is the INLINEFORM7 -th seed name pair of INLINEFORM8 . The intuition behind INLINEFORM16 is that seed name pairs that match np anchor pairs of many sentences of INLINEFORM17 are more likely to be indicative of INLINEFORM18 . When using the MaxEnt classifier to score a sentence plan INLINEFORM19 for a relation INLINEFORM20 , we compute the INLINEFORM21 of all the seed name pairs INLINEFORM22 of INLINEFORM23 , and we use the maximum, minimum, average, total, and standard deviation of these productivity scores as five features of INLINEFORM24 .
The productivity of a seed name INLINEFORM0 (considered on its own) that occurs as the first element of at least one seed name pair INLINEFORM1 of a relation INLINEFORM2 is defined as follows: DISPLAYFORM0
where: INLINEFORM0 is the number of seed name pairs INLINEFORM1 of INLINEFORM2 that have INLINEFORM3 as their first element; INLINEFORM4 is the number of times INLINEFORM5 (as part of a seed name pair INLINEFORM6 of INLINEFORM7 ) matched any element of any np anchor pair of the parsed sentences of INLINEFORM8 and contributed to the extraction of any template of INLINEFORM9 ; INLINEFORM10 and INLINEFORM11 are as in Eq. EQREF36 . Again, when using the classifier to score a sentence plan INLINEFORM12 for a relation INLINEFORM13 , we calculate the INLINEFORM14 values of all the (distinct) seed names INLINEFORM15 that occur as first elements in the seed name pairs of INLINEFORM16 , and we use the maximum, minimum, average, total, and standard deviation of these productivity scores as five more features of INLINEFORM17 . We define similarly INLINEFORM18 for a seed name INLINEFORM19 that occurs as the second element in any seed name pair INLINEFORM20 of INLINEFORM21 , obtaining five more features for INLINEFORM22 .
Similarly to Eq. EQREF36 , we define the productivity of the INLINEFORM0 -th NP anchor pair INLINEFORM1 (e.g., INLINEFORM2 INLINEFORM3 “Semillon”, INLINEFORM4 “Semillon grapes” INLINEFORM5 in Fig. FIGREF32 ) of a relation INLINEFORM6 as follows: DISPLAYFORM0
where: INLINEFORM0 is the number of times a seed name pair of INLINEFORM1 matched INLINEFORM2 in the parsed sentences of INLINEFORM3 and contributed to the extraction of any template of INLINEFORM4 ; and INLINEFORM5 is the total number of np anchor pairs of INLINEFORM6 . As with INLINEFORM14 , the intuition behind INLINEFORM15 is that np anchor pairs that match many seed name pairs of INLINEFORM16 are more indicative of INLINEFORM17 . When using the classifier to score a sentence plan INLINEFORM18 for a relation INLINEFORM19 , we compute the INLINEFORM20 of all the np anchor pairs of INLINEFORM21 , and we use the maximum, minimum, average, total, and standard deviation of these scores as five additional features of INLINEFORM22 .
Similarly to Eq. EQREF38 , the productivity of an NP anchor INLINEFORM0 (considered on its own) that occurs as the first element of at least one np anchor pair INLINEFORM1 of INLINEFORM2 is defined as follows: DISPLAYFORM0
where: INLINEFORM0 is the number of np anchor pairs INLINEFORM1 of INLINEFORM2 that have INLINEFORM3 as their first element; INLINEFORM4 is the number of times INLINEFORM5 (as part of an np anchor pair INLINEFORM6 of INLINEFORM7 ) matched any element of any seed name pair of INLINEFORM8 and contributed to the extraction of any template of INLINEFORM9 ; and INLINEFORM10 , INLINEFORM11 are as in Eq. EQREF39 . Again, we calculate the INLINEFORM12 values of all the (distinct) np anchors INLINEFORM13 that occur as first elements in the np anchor pairs of INLINEFORM14 , and we use the maximum, minimum, average, total, and standard deviation of these productivity scores as five more features of INLINEFORM15 . We define similarly INLINEFORM16 for a seed name INLINEFORM17 that occurs as the second element in any np anchor pair INLINEFORM18 of INLINEFORM19 , obtaining five more features for INLINEFORM20 .
The productivity of a template INLINEFORM0 (e.g., “ INLINEFORM1 is made from INLINEFORM2 ”) of a relation INLINEFORM3 is defined as follows: DISPLAYFORM0
where: INLINEFORM0 is the number of times the particular template INLINEFORM1 was extracted from any of the parsed sentences of INLINEFORM2 ; INLINEFORM3 is the total number of templates of INLINEFORM4 ; and INLINEFORM5 is the INLINEFORM6 -th template of INLINEFORM7 . The intuition is that templates that are produced more often for INLINEFORM8 are more indicative of INLINEFORM9 . Again, we calculate the INLINEFORM10 of all the templates INLINEFORM11 of INLINEFORM12 , and we use the maximum, minimum, average, total, and standard deviation of these productivity scores as five more features of INLINEFORM13 .
The productivity of a parsed sentence INLINEFORM0 (e.g., “obviously Semillon is made from Semillon grapes in California”) of a relation INLINEFORM1 is defined as follows: DISPLAYFORM0
where: INLINEFORM0 is the number of times any template of INLINEFORM1 was extracted from the particular parsed sentence INLINEFORM2 ; INLINEFORM3 is the total number of parsed sentences of INLINEFORM4 ; and INLINEFORM5 is the INLINEFORM6 -th parsed sentence of INLINEFORM7 . The intuition is that sentences that produce more templates for INLINEFORM8 are more indicative of INLINEFORM9 . Again, we calculate the INLINEFORM10 of all the parsed sentences INLINEFORM11 of INLINEFORM12 , and we use the maximum, minimum, average, total, and standard deviation of these productivity scores as features of INLINEFORM13 .
The joint productivity of a seed name pair INLINEFORM0 and a template INLINEFORM1 of a relation INLINEFORM2 is: DISPLAYFORM0
where: INLINEFORM0 is the number of times the particular seed name pair INLINEFORM1 matched any np anchor pair of the parsed sentences of INLINEFORM2 and contributed to the extraction of the particular template INLINEFORM3 ; and INLINEFORM4 are again the total numbers of seed name pairs and templates, respectively, of INLINEFORM5 . Again, when scoring a sentence plan INLINEFORM6 for a relation INLINEFORM7 , we calculate the INLINEFORM8 of all the combinations of seed name pairs INLINEFORM9 and templates INLINEFORM10 that led to INLINEFORM11 , and we use the maximum, minimum, average, total, and standard deviation of these scores as features of INLINEFORM12 . We define very similarly INLINEFORM13 , INLINEFORM14 , INLINEFORM15 , INLINEFORM16 , INLINEFORM17 , INLINEFORM18 INLINEFORM19 , INLINEFORM20 , INLINEFORM21 , obtaining five additional features of INLINEFORM22 from each one. We also define: DISPLAYFORM0
where: INLINEFORM0 is the number of times INLINEFORM1 matched the np anchor pair INLINEFORM2 in a parsed sentence of INLINEFORM3 contributing to the extraction of template INLINEFORM4 ; INLINEFORM5 are the numbers of seed name pairs, np anchor pairs, templates of INLINEFORM6 . We define similarly INLINEFORM7 and INLINEFORM8 , obtaining five features from each type of productivity score.
For each productivity version of Section UID35 , we define a prominence variant. For example, based on the productivity of a seed name pair INLINEFORM0 of a relation INLINEFORM1 (Eq. EQREF36 , repeated as Eq. EQREF47 ), DISPLAYFORM0
we define the prominence of a candidate sentence plan INLINEFORM0 with respect to the seed name pairs of INLINEFORM1 : DISPLAYFORM0
where: the INLINEFORM0 denotes 1 if condition INLINEFORM1 holds and 0 otherwise; INLINEFORM2 (in the numerator) is the number of times INLINEFORM3 matched any np anchor pair of the parsed sentences of INLINEFORM4 , counting only matches that contributed to the extraction of a template of INLINEFORM5 that led to the particular sentence plan INLINEFORM6 ; by contrast, INLINEFORM7 (in the denominator) is the number of times INLINEFORM8 matched any np anchor pair of the parsed sentences of INLINEFORM9 , counting only matches that contributed to the extraction of any template of INLINEFORM10 ; and INLINEFORM11 is the total number of seed name pairs of INLINEFORM12 . In other words, we count how many (distinct) seed name pairs of INLINEFORM13 produced INLINEFORM14 , dividing by the number of (distinct) seed name pairs of INLINEFORM15 that produced at least one template of INLINEFORM16 . The intuition is that the more seed name pairs of INLINEFORM17 lead to the sentence plan INLINEFORM18 , the better INLINEFORM19 is.
In a similar manner, we define INLINEFORM0 based on Eq. EQREF39 , and similarly for all the other productivity versions of Section UID35 . We obtain one feature for the candidate sentence plan INLINEFORM1 from each prominence variant, i.e., we do not compute any maximum, minimum, average, sum, standard deviation values, unlike the productivity versions, which lead to five features each.
To estimate the extent to which two seed names INLINEFORM0 , INLINEFORM1 of a relation INLINEFORM2 co-occur when they match np anchors to produce templates of INLINEFORM3 , we use a Pointwise Mutual Information (pmi) score: DISPLAYFORM0
The second factor of the right-hand side of Eq. EQREF50 is the standard pmi definition, using productivity scores instead of probabilities. The first factor normalizes the pmi scores to INLINEFORM0 ( INLINEFORM1 if INLINEFORM2 never co-occur when producing templates of INLINEFORM3 , 0 if they are independent, 1 if they always co-occur). Intuitively, if INLINEFORM4 co-occur frequently when they produce templates of INLINEFORM5 , they are strongly connected and, hence, they are more indicative of INLINEFORM6 . Again, when using the classifier to score a sentence plan INLINEFORM7 for a relation INLINEFORM8 , we calculate INLINEFORM9 for all the seed name pairs INLINEFORM10 of INLINEFORM11 , and we use the maximum, minimum, average, total, and standard deviation of these pmi scores as five more features of INLINEFORM12 . We define similarly INLINEFORM13 , INLINEFORM14 , INLINEFORM15 , INLINEFORM16 , INLINEFORM17 , INLINEFORM18 , INLINEFORM19 , INLINEFORM20 , INLINEFORM21 , INLINEFORM22 , obtaining five features for INLINEFORM23 from each one.
These features view seed names, np anchors, templates, and owl identifiers as sequences of tokens.
For each seed name INLINEFORM0 and np anchor INLINEFORM1 that matched (as first elements of a seed name pair INLINEFORM2 and np anchor pair INLINEFORM3 ) to produce a particular sentence plan INLINEFORM4 , we calculate their cosine similarity INLINEFORM5 with INLINEFORM6 weights (defined as in Section SECREF29 ). We then use the maximum, minimum, average, total, and standard deviation of these cosine similarities as features of INLINEFORM7 . Intuitively, they show how good the matches that produced INLINEFORM8 were. We repeat for each seed name INLINEFORM9 and np anchor INLINEFORM10 that matched (as second elements of their pairs) to produce INLINEFORM11 , this time computing INLINEFORM12 , obtaining five additional features of INLINEFORM13 .
We do the same using INLINEFORM0 , defined below, instead of INLINEFORM1 : DISPLAYFORM0
where: INLINEFORM0 , INLINEFORM1 are the lengths (in tokens) of INLINEFORM2 ; INLINEFORM3 , INLINEFORM4 are the token sequences of INLINEFORM5 , respectively; INLINEFORM6 is the probability of encountering token INLINEFORM7 in a parsed sentence of INLINEFORM8 ; and INLINEFORM9 is the probability of encountering both INLINEFORM10 and INLINEFORM11 in the same parsed sentence of INLINEFORM12 ; we use Laplace estimates for these probabilities. Again, we compute INLINEFORM13 for every seed name INLINEFORM14 and np anchor INLINEFORM15 that matched to produce a particular sentence plan INLINEFORM16 , and we use the maximum, minimum, average, total, and standard deviation of these scores as features of INLINEFORM17 . We repeat using INLINEFORM18 instead of INLINEFORM19 , obtaining five more features.
Similarly, we compute INLINEFORM0 and INLINEFORM1 for each np anchor INLINEFORM2 or INLINEFORM3 (left, or right element of an np anchor pair) and template INLINEFORM4 (ignoring the INLINEFORM5 and INLINEFORM6 ) that led to a particular sentence plan INLINEFORM7 , and we use their maximum, minimum, average, total, and standard deviation as ten additional features of INLINEFORM8 . We also compute INLINEFORM9 and INLINEFORM10 for each template INLINEFORM11 and tokenized identifier INLINEFORM12 of a relation INLINEFORM13 (e.g., INLINEFORM14 :madeFrom becomes INLINEFORM15 “made from”) that led to the sentence plan INLINEFORM16 , obtaining ten more features. Finally, we compute INLINEFORM17 , INLINEFORM18 , and INLINEFORM19 for all the INLINEFORM20 and INLINEFORM21 np anchors (first or second elements in their pairs) of INLINEFORM22 , obtaining fifteen more features of INLINEFORM23 . Although they may look strange, in effect INLINEFORM24 and INLINEFORM25 examine how strongly connected the words inside each np anchor ( INLINEFORM26 or INLINEFORM27 ) are.
Another group of features try to estimate the grammaticality of a candidate sentence plan INLINEFORM0 . Let us assume that INLINEFORM1 is for relation INLINEFORM2 . For every seed name pair of INLINEFORM3 (not only seed name pairs that led to INLINEFORM4 ), we generate a sentence using INLINEFORM5 ; we ignore only seed name pairs that produced no sentence plans at all, which are assumed to be poor. For example, for the seed name pair INLINEFORM6 INLINEFORM7 “Semillon”, INLINEFORM8 “Semillon grape” INLINEFORM9 of the relation INLINEFORM10 :madeFrom and the following candidate sentence plan:
[ INLINEFORM0 ] INLINEFORM1 [make] INLINEFORM2 [from] INLINEFORM3 [ INLINEFORM4 ] INLINEFORM5
the sentence “Semillon is made from Semillon grapes” is generated. We do not generate referring expressions, even when required by the sentence plan (e.g., [ INLINEFORM0 ] INLINEFORM1 ); we use the seed names instead. We obtain confidence scores for these sentences from the parser, and we normalize these scores dividing by each sentence's length. The maximum, minimum, average, and standard deviation of these scores are used as features of INLINEFORM2 .
Some additional features for a candidate sentence plan INLINEFORM0 follow:
True if INLINEFORM0 contains a present participle without an auxiliary; otherwise false.
True if INLINEFORM0 has a main verb in active voice; otherwise false.
True if INLINEFORM0 contains a referring expression for INLINEFORM1 before a referring expression for INLINEFORM2 ; otherwise false. Sentence plans that refer to INLINEFORM3 before INLINEFORM4 are usually simpler and better.
True if a referring expression of INLINEFORM0 is the subject of a verb of INLINEFORM1 ; otherwise false. This information is obtained from the parsed sentences that led to INLINEFORM2 . We use the most frequent dependency tree, if INLINEFORM3 was derived from many sentences. Sentence plans with no subjects are often ill-formed.
True if a referring expression of INLINEFORM0 is the object of a verb of INLINEFORM1 ; otherwise false. Again, we consider the parsed sentences that led to INLINEFORM2 using the most frequent dependency tree. Sentence plans with no objects are often ill-formed, because most relations are expressed by transitive verbs.
True if all the sentences INLINEFORM0 was derived from were well-formed, according to the parser.
True if INLINEFORM0 required a repair at the end of the sentence plan generation (Section SECREF33 ); otherwise false. Repaired sentence plans can be poor.
The number of slots of INLINEFORM0 , the number of slots before the slot for INLINEFORM1 , the number of slots after the slot for INLINEFORM2 , the number of slots between the slots for INLINEFORM3 and INLINEFORM4 (4 features).
The maximum, minimum, average, total, standard deviation of the ranks of the Web pages (returned by the search engine, Section SECREF29 ) that contained the sentences INLINEFORM0 was obtained from. Sentences from higher-ranked pages are usually more relevant to the seed name pairs we use as queries. Hence, sentence plans obtained from higher-ranked Web pages are usually better.
The number of Web pages that contained the sentences from which INLINEFORM0 was obtained. Uncommon sentences often lead to poor sentence plans.
Ranking the candidate sentence plans
Each candidate sentence plan of a relation INLINEFORM0 is represented as a feature vector INLINEFORM1 , containing the 251 features discussed above. Each vector is given to the MaxEnt classifier to obtain a probability estimate INLINEFORM2 that it belongs in the positive class INLINEFORM3 , i.e., that the sentence plan is correct for INLINEFORM4 . The candidate sentence plans of each relation INLINEFORM5 are then ranked by decreasing estimated (by the classifier) INLINEFORM6 . We call sp our overall sentence plan generation method that uses the probability estimates of the classifier to rank the candidate sentence plans.
In an alternative configuration of our sentence plan generation method, denoted sp*, the probability estimate INLINEFORM0 of each candidate sentence plan is multiplied by its coverage. To compute the coverage of a sentence plan for a relation INLINEFORM1 , we use the sentence plan to produce a sentence for each seed name pair of INLINEFORM2 (as when computing the grammaticality of a sentence plan in Section SECREF34 ). Subsequently, we use each sentence as a phrase query in a Web search engine. The coverage of the sentence plan is the number of seed name pairs for which the search engine retrieved at least one document containing the search sentence (verbatim), divided by the total number of seed name pairs of INLINEFORM3 . Coverage helps avoid sentence plans that produce very uncommon sentences. Computing the coverage of every candidate sentence plan is time consuming, however, because of the Web searches; this is also why we do not include coverage in the features of the classifier. Hence, we first rank the candidate sentence plans of each relation INLINEFORM4 by decreasing INLINEFORM5 , and we then re-rank only the top ten of them (per INLINEFORM6 ) after multiplying the INLINEFORM7 of each one by its coverage.
In both sp and sp*, in a semi-automatic scenario we return to a human inspector the top five candidate sentence plans per relation. In a fully automatic scenario, we return only the top one.
Experiments
We now present the experiments we performed to evaluate our methods that generate nl names and sentence plans. We first discuss the ontologies that we used in our experiments.
The ontologies of our experiments
We used three ontologies: (i) the Wine Ontology, one of the most commonly used examples of owl ontologies; (ii) the m-piro ontology, which describes a collection of museum exhibits, was originally developed in the m-piro project BIBREF27 , was later ported to owl, and accompanies Naturalowl BIBREF11 ; and (iii) the Disease Ontology, which describes diseases, including their symptoms, causes etc.
The Wine Ontology involves a wide variety of owl constructs and, hence, is a good test case for ontology verbalizers and nlg systems for owl. The m-piro ontology has been used to demonstrate the high quality texts that Naturalowl can produce, when appropriate manually authored linguistic resources are provided BIBREF28 . We wanted to investigate if texts of similar quality can be generated with automatically or semi-automatically acquired nl names and sentence plans. The Disease Ontology was developed by biomedical experts to address real-life information needs; hence, it constitutes a good real-world test case.
The Wine Ontology contains 77 classes, 161 individuals, and 14 relations (properties). We aimed to produce nl names and sentence plans for the 49 classes, 146 individuals, and 7 relations that are directly involved in non-trivial definitions of wines (43 definitions of wine classes, 52 definitions of wine individuals), excluding classes, individuals, and relations that are only used to define wineries, wine-producing regions etc. By “non-trivial definitions” we mean that we ignored definitions that humans understand as repeating information that is obvious from the name of the defined class or individual (e.g., the definition of :RedWine in effect says that a red wine is a wine with red color).
The m-piro ontology currently contains 76 classes, 508 individuals, and 41 relations. Many individuals, however, are used to represent canned texts (e.g., manually written descriptions of particular types of exhibits) that are difficult to generate from symbolic information. For example, there is a pseudo-individual :aryballos-def whose nl name is the fixed string “An aryballos was a small spherical vase with a narrow neck, in which the athletes kept the oil they spread their bodies with”. Several properties are also used only to link these pseudo-individuals (in effect, the canned texts) to other individuals or classes (e.g., to link :aryballos-def to the class :Aryballos); and many other classes are used only to group pseudo-individuals (e.g., pseudo-individuals whose canned texts describe types of vessels all belong in a common class). In our experiments, we ignored pseudo-individuals, properties, and classes that are used to represent, link, and group canned texts, since we focus on generating texts from symbolic information. We aimed to produce nl names and sentence plans for the remaining 30 classes, 127 individuals, and 12 relations, which are all involved in the definitions (descriptions) of the 49 exhibits of the collection the ontology is about.
The Disease Ontology currently contains information about 6,286 diseases, all represented as classes. Apart from is-a relations, synonyms, and pointers to related terms, however, all the other information is represented using strings containing quasi-English sentences with relation names used mostly as verbs. For example, there is an axiom in the ontology stating that the Rift Valley Fever (doid_1328) is a kind of viral infectious disease (doid_934). All the other information about the Rift Valley Fever is provided in a string, shown below as `Definition'. The tokens that contain underscores (e.g., results_in) are relation names. The ontology declares all the relation names, but uses them only inside `Definition' strings. Apart from diseases, it does not define any of the other entities mentioned in the `Definition' strings (e.g., symptoms, viruses).
Name: Rift Valley Fever (doid_1328)
is-a: viral infectious disease (doid_934)
Definition: A viral infectious disease that results_in infection, has_material_basis_in Rift Valley fever virus, which is transmitted_by Aedes mosquitoes. The virus affects domestic animals (cattle, buffalo, sheep, goats, and camels) and humans. The infection has_symptom jaundice, has_symptom vomiting blood, has_symptom passing blood in the feces, has_symptom ecchymoses (caused by bleeding in the skin), has_symptom bleeding from the nose or gums, has_symptom menorrhagia and has_symptom bleeding from venepuncture sites.
We defined as individuals all the non-disease entities mentioned in the `Definition' strings, also adding statements to formally express the relations mentioned in the original `Definition' strings. For example, the resulting ontology contains the following definition of Rift Valley Fever, where :infection, :Rift_Valley_fever_virus, :Aedes_mosquitoes, :jaundice etc. are new individuals.
SubClassOf(:DOID_1328
ObjectIntersectionOf(:DOID_934
ObjectHasValue(:results_in :infection)
ObjectHasValue(:has_material_basis_in :Rift_Valley_fever_virus)
ObjectHasValue(:transmitted_by :Aedes_mosquitoes)
ObjectHasValue(:has_symptom :jaundice)
ObjectHasValue(:has_symptom :vomiting_blood)
ObjectHasValue(:has_symptom :passing_blood_in_the_feces)
ObjectHasValue(:has_symptom :ecchymoses_(caused_by_bleeding_in_the_skin))
ObjectHasValue(:has_symptom :bleeding_from_the_nose_or_gums)
ObjectHasValue(:has_symptom :menorrhagia)
ObjectHasValue(:has_symptom :bleeding_from_venepuncture_sites)))
The new form of the ontology was produced automatically, using patterns that searched the definition strings for relation names (e.g., results_in), sentence breaks, and words that introduce secondary clauses (e.g., “that”, “which”). Some sentences of the original definition strings that did not include declared relation names, like the sentence “The virus affects...and humans” in the `Definition' string of Rift Valley Fever above, were discarded during the conversion, because it was not always possible to reliably convert them to appropriate owl statements.
The new form of the Disease Ontology contains 6,746 classes, 15 relations, and 1,545 individuals. We aimed to automatically produce nl names and sentence plans for the 94 classes, 99 individuals, and 8 relations that are involved in the definitions of 30 randomly selected diseases. We manually authored nl names and sentence plans for the same classes, individuals, and relations, to be able to compare the quality of the resulting texts. Manually authored nl names and sentence plans for the Wine and m-piro ontologies are also available (they are included in the software of Naturalowl).
We note that the relations (properties) INLINEFORM0 of our experiments are all used in message triples INLINEFORM1 , where INLINEFORM2 is an individual or class, i.e., they are object properties in owl's terminology. Datatype properties, where INLINEFORM3 is a datatype value (e.g., integer, string, date), can in principle be handled using the same methods, but appropriate recognizers may be needed to obtain appropriate anchors, instead of np anchors. For example, a datatype property may map persons to dates of birth; then a recognizer of dates would be needed to extract (and possibly normalize) appropriate date anchors from Web pages, since a parser may not treat dates as np s.
Experiments with automatically or semi-automatically produced NL names
We now present experiments we performed with our method that generates nl names (Section SECREF3 ).
In a first experiment, we measured how well our nl names method determines which individuals and classes should be anonymous (Sections SECREF3 and UID12 ). We compared the decisions of our method against the corresponding anonymity declarations in the manually authored nl names of the three ontologies. Table TABREF72 summarizes the results of this experiment. Precision is the total number of individuals and classes our nl names method correctly (in agreement with the manually authored nl names) declared as anonymous, divided by the total number of individuals and classes our method declared as anonymous. Recall is the total number of individuals and classes our nl names method correctly declared as anonymous, divided by the total number of individuals and classes (among those we aimed to produce nl names for) that the manually authored nl names declared as anonymous. For the Disease Ontology, the manually authored nl names and our nl names method agreed that no individuals and classes (that we aimed to produce nl names for) should be anonymous, which is why precision and recall are undefined. Accuracy is the number of correct decisions (individuals and classes correctly declared, or correctly not declared as anonymous), divided by the total number of individuals and classes (that we aimed to produce nl names for).
The anonymity decisions of our method were perfect in the m-piro ontology and Disease Ontology. In the Wine Ontology, the precision of our method was also perfect, i.e., whenever our method decided to declare an individual or class as anonymous, this was a correct decision; but recall was lower, i.e., our method did not anonymize all the individual and classes that the manually authored nl names did. The latter is due to the fact that the manually authored nl names of the Wine ontology also anonymize 14 individuals and classes with complex identifiers (e.g., :SchlossVolradTrochenbierenausleseRiesling) to produce more readable texts. By contrast, our method declares individuals and classes as anonymous only to avoid redundancy in the generated texts (Section UID12 ), hence it does not anonymize the 14 individuals and classes.
We then invoked our nl name generation method for the individuals and classes it had not declared as anonymous (160 in the Wine Ontology, 108 in the m-piro ontology, 195 in the Disease Ontology), using the top 10 returned documents per Web search (or top 20, when the search engine proposed spelling corrections – see Section UID16 ). We ranked the produced nl names (as in Sections UID16 and SECREF22 ), and kept the top 5 nl names per individual or class. The first author then inspected the resulting nl names and marked each one as correct or incorrect. An nl name was considered correct if and only if: (i) it would produce morphologically, syntactically, and semantically correct and unambiguous noun phrases (e.g., “Cabernet Sauvignon grape” is correct for :CabernetSauvignonGrape, but “Cabernet Sauvignon wine”, “Cabernet Sauvignon”, or “grape” are incorrect); and (ii) its slot annotations (e.g., pos tags, gender, agreement) were all correct.
Table TABREF74 shows the results of this experiment. The “1-in-1” score is the ratio of individuals and classes for which the top returned nl name was correct. The “1-in-3” score is the ratio of individuals and classes for which there was at least one correct nl name among the top three, and similarly for “1-in-5”. The “1-in-1” score corresponds to a fully automatic scenario, where the top nl name is used for each individual or class, without human intervention. By contrast, the “1-in-3” and “1-in-5” scores correspond to a semi-automatic scenario, where a human inspects the top three or five, respectively, nl names per individual or class, looking for a correct one to select. The mean reciprocal rank (mrr) is the mean (over all the individuals and classes we asked our method to produce nl names for) of the reciprocal rank INLINEFORM0 , where INLINEFORM1 is the rank (1 to 5) of the top-most correct nl name returned for the INLINEFORM2 -th individual or class; if no correct nl name exists among the top five, then INLINEFORM3 . mrr rewards more those methods that place correct nl names towards the top of the five returned ones. The weighted scores of Table TABREF74 are similar, but they weigh each individual or class by the number of owl statements that mention it in the ontology.
The results of Table TABREF74 show that our nl names method performs very well in a semi-automatic scenario. In a fully automatic scenario, however, there is large scope for improvement. We note, though, that our definition of correctness (of nl names) in the experiment of this section was very strict. For example, an nl name with only a single error in its slot annotations (e.g., a wrong gender in a noun slot) was counted as incorrect, even if in practice the error might have a minimal effect on the generated texts that would use the nl name. The experiment of Section UID77 below, where nl names are considered in the context of generated texts, sheds more light on this point.
By inspecting the produced nl names, we noticed that our method is very resilient to spelling errors and abbreviations in the owl identifiers of individuals and classes. For example, it returns nl names producing “a Côte d'Or wine” for :CotesDOr, and “the Naples National Archaeological Museum” for :national-arch-napoli. Several wrongly produced nl names are due to errors of tools that our method invokes (e.g., parser, ner). Other errors are due to over-shortened altTokNames (Section SECREF10 ); e.g., one of the altTokNames of :CorbansDryWhiteRiesling was simply “Dry White”, which leads to an nl name that does not identify the particular wine clearly enough. Finally, in the Disease Ontology, our automatic conversion of the `Description' strings produced many individuals whose identifiers are in effect long phrases (see, for example, the owl description of :doid_1328 in Section SECREF67 ). Our nl names method manages to produce appropriate nl names (with correct slot annotations etc.) for some of them (e.g., :mutation_in_the_SLC26A2_gene), but produces no nl names in other cases (e.g., :infection_of_the_keratinized_layers). Some of these errors, however, may not have a significant effect on the generated texts (e.g., using the tokenized identifer “infection of the keratinized layers”, which is the default when no nl name is provided, may still lead to a reasonable text). Again, the experiment of Section UID77 below sheds more light on this point.
The top five automatically produced nl names of each individual and class were also shown to a second human judge. The second judge was a computer science researcher not involved in nlg, fluent in English, though not a native speaker. For each individual or class, and for each one of its top five nl names, the judge was shown a phrase produced by the nl name (e.g., “Cabernet Sauvignon”), an automatically generated sentence about the individual or class (expressing a message triple of the ontology) illustrating the use of the nl name (e.g., “Cabernet Sauvignon is a kind of wine.”), and a sentence where the nl name had been automatically replaced by a pronoun (e.g., “It is a kind of wine.”) to check the gender of the nl name. The judge was asked to consider the phrases and sentences, and mark the best correct nl name for each individual or class. The judge could also mark more than one nl names for the same individual or class, if more than one seemed correct and equally good; the judge was instructed not to mark any of the five nl names, if none seemed correct. The judge completed this task in 49, 45, and 75 minutes for the Wine, m-piro, and Disease Ontology (727, 540, 965 candidate nl names), respectively; by contrast, manually authoring the nl names of the three ontologies took approximately 2, 2, and 3 working days, respectively. These times and the fact that the second judge was not aware of the internals of Naturalowl and its resources suggest that the semi-automatic authoring scenario is viable and very useful in practice.
Table TABREF76 compares the decisions of the second judge, hereafter called INLINEFORM0 , to those of the first author, hereafter called INLINEFORM1 . INLINEFORM2 was able to view the full details of the nl names using Naturalowl's Protégé plug-in, unlike INLINEFORM3 who viewed only phrases and example sentences. For the purposes of this study, INLINEFORM4 marked all the correct nl names (not only the best ones) among the top five of each individual or class. In Table TABREF76 , micro-precision is the number of nl names (across all the individuals and classes) that were marked as correct by both INLINEFORM5 and INLINEFORM6 , divided by the number of nl names marked as correct by INLINEFORM7 , i.e., we treat the decisions of INLINEFORM8 as gold. Macro-precision is similar, but we first compute the precision of INLINEFORM9 against INLINEFORM10 separately for each individual or class, and we then average over all the individuals and classes. INLINEFORM11 1-in-5 is the percentage of individuals and classes for which INLINEFORM12 marked at least one nl name among the top five as correct, and similarly for INLINEFORM13 1-in-5. Pseudo-recall is the number of individuals and classes for which both INLINEFORM14 and INLINEFORM15 marked at least one nl name as correct, divided by the number of individuals and classes for which INLINEFORM16 marked at least one nl name as correct; this measure shows how frequently INLINEFORM17 managed to find at least one correct (according to INLINEFORM18 ) nl name, when there was at least one correct nl name among the top five. Computing the true recall of the decisions of INLINEFORM19 against those of INLINEFORM20 would be inappropriate, because INLINEFORM21 was instructed to mark only the best nl name(s) of each individual and class, unlike INLINEFORM22 who was instructed to mark all the correct ones. We also calculated Cohen's Kappa between INLINEFORM23 and INLINEFORM24 ; for each individual or class, if INLINEFORM25 had marked more than one nl names as correct, we kept only the top-most one, and similarly for INLINEFORM26 , hence each judge had six possible choices (including marking no nl name) per individual and class. The results of Table TABREF76 indicate strong inter-annotator agreement in the semi-automatic authoring of nl names in all three ontologies.
In order to examine how the produced nl names affect the perceived quality of the generated texts, we showed automatically generated texts describing individuals and classes of the three ontologies to six computer science students not involved in the work of this article; they were all fluent, though not native, English speakers. We generated texts using Naturalowl configured in four ways. The no-nln configuration uses no nl names; in this case, Naturalowl uses the tokenized owl identifiers of the individuals and classes as their names. manual-nln uses manually authored nl names. auto-nln uses the top-ranked nl name that our nl names method produces for each individual and class. Finally, semi-auto-nln uses the nl name (of each individual or class) that a human inspector (the first author of this article) selected among the top five nl names produced by our method. Additionally, both auto-nln and semi-auto-nln use the methods of Sections UID12 and SECREF27 to anonymize individuals or classes and to infer interest scores from nl names, whereas manual-nln uses the anonymity declarations and interest scores of the manually authored linguistic resources, and no-nln uses no anonymity declarations and no interest scores. Apart from the nl names, anonymity declarations, and interest scores, all four configurations use the same, manually authored other types of linguistic resources (e.g., sentence plans, text plans to order the message triples). Below are example texts generated from the three ontologies by the four configurations.
manual-nln: This is a moderate, dry Zinfandel. It has a medium body. It is made by Saucelito Canyon in the city of Arroyo Grande.
semi-auto-nln: This is a moderate, dry Zinfandel wine. It has a medium body. It is made by the Saucelito Canyon Winery in the Arroyo Grande area.
auto-nln: This is a dry Zinfandel and has the medium body. It is the moderate. It is made by Saucelito Canyon in Arroyo Grande.
no-nln: Saucelito Canyon Zinfandel is Zinfandel. It is Dry. It has a Medium body. It is Moderate. It is made by Saucelito Canyon. It is made in Arroyo Grande Region.
manual-nln: This is a statue, created during the classical period and sculpted by Polykleitos. Currently it is exhibited in the National Archaeological Museum of Napoli.
semi-auto-nln: This is a statue, created during the classical period and sculpted by the sculptor polyclitus. Currently it is exhibited in the Naples National Archaeological Museum.
auto-nln: This is a statue, created during classical and sculpted by the polyclitus. Currently it is exhibited in national arch napoli.
no-nln: Exhibit 4 is statue, created during classical period and sculpted by polyclitus. Today it is exhibited in national arch napoli.
manual-nln: Systemic mycosis is a kind of fungal infectious disease that affects the human body. It results in infection of internal organs. It is caused by fungi.
semi-auto-nln: A systemic mycosis is a kind of fungal infectious disease that affects human body. It results in infections of internal organs and it is caused by the fungi.
auto-nln: A systemic mycosis is fungal that affects human body. It results in infections of internal organs and it is caused by the fungi.
no-nln: Systemic mycosis is a kind of fungal infectious disease. It affects human body. It results in infection of internal organs. It is caused by Fungi.
We note that some nl names of semi-auto-nln and auto-nln can be easily improved using the Protégé plug-in of Naturalowl. For example, the nl name of the human body can be easily modified to include a definite article, which would improve the texts of semi-auto-nln and auto-nln in the Disease Ontology examples above (“affects the human body” instead of “affects human body”). Nevertheless, we made no such improvements.
Recall that there are INLINEFORM0 non-trivial definitions of wine classes and wine individuals in the Wine Ontology, 49 exhibits in the m-piro ontology, and that we randomly selected 30 diseases from the Disease Ontology (Section SECREF67 ). Hence, we generated INLINEFORM1 texts from the Wine Ontology (with the four configurations of Naturalowl), INLINEFORM2 texts from the m-piro ontology, and INLINEFORM3 texts from the Disease ontology. For each individual or class, the message triples of its definition (regardless of their interest scores) along with the corresponding texts were given to exactly one student. The four texts of each individual or class were randomly ordered and the students did not know which configuration had generated each one of the four texts. For each individual or class, the students were asked to compare the four texts to each other and to the message triples, and score each text by stating how strongly they agreed or disagreed with statements INLINEFORM4 – INLINEFORM5 below. A scale from 1 to 5 was used (1: strong disagreement, 3: ambivalent, 5: strong agreement). Examples and more detailed guidelines were also provided to the students.
( INLINEFORM0 ) Sentence fluency: Each sentence of the text (on its own) is grammatical and sounds natural.
( INLINEFORM0 ) Clarity: The text is easy to understand, provided that the reader is familiar with the terminology and concepts of the domain (e.g., historical periods, grape varieties, virus names).
( INLINEFORM0 ) Semantic correctness: The text accurately conveys the information of the message triples.
( INLINEFORM0 ) Non-redundancy: There is no redundancy in the text (e.g., stating the obvious, repetitions).
Tables TABREF80 – TABREF82 show the scores of the four configurations of Naturalowl, averaged over the texts of each ontology. For each criterion, the best scores are shown in bold. In each criterion (row), we detected no statistically significant differences between scores marked with the same superscript; all the other differences (in the same row) were statistically significant. Overall, the manually authored nl names led to the best (near-perfect) scores, as one might expect. The scores of semi-auto-nln were overall slightly lower, but still high (always INLINEFORM2 ) and no statistically significant differences to the corresponding scores of manual-nln were detected. These findings confirm that our nl names method performs very well in a semi-automatic scenario, where a human inspects and selects among the top-ranked automatically produced nl names. By contrast, auto-nln performed overall much worse than semi-auto-nln and manual-nln, and often worse than no-nln, which again indicates that our nl names method cannot be used in a fully automatic manner.
The no-nln configuration, which uses tokenized identifiers of individuals and classes, performed overall much worse than manual-nln and semi-auto-nln in the Wine and m-piro ontologies, which shows the importance of nl names in the perceived quality of generated texts. The differences between no-nln, semi-auto-nln, and manual-nln were smaller in the Disease Ontology, where no statistically significant differences between the three configurations were detected. These smaller differences are due to the fact that the conversion of the Disease Ontology (Section SECREF67 ) produced many individuals whose owl identifiers are in effect long phrases, easily readable, and sometimes better than then top-ranked nl names of our methods; furthermore, our nl names method does not manage to produce any nl names for many of these individuals and, hence, semi-auto-nln ends up using their tokenized identifiers, like no-nln. We also note that there are very few redundant message triples and no anonymous individuals or classes in the Disease Ontology, which explains the higher non-redundancy scores of no-nln in the Disease Ontology, compared to the much lower non-redundancy scores of no-nln in the other two ontologies.
Experiments with automatically or semi-automatically produced sentence plans
We now present the experiments we performed to evaluate our method that generates sentence plans (Section SECREF4 ). Recall that our method employs a MaxEnt classifier to predict the probability that a candidate sentence plan is correct (positive class) or incorrect (negative class).
To create training instances for the MaxEnt classifier, we used our sentence plan generation method without the classifier to obtain candidate sentence plans (as in Sections SECREF29 and SECREF33 ) from Wikipedia for the seven relations of the Wine Ontology (Section SECREF67 ). We used the manually authored nl names of the Wine Ontology to obtain seed names, and the top 50 Wikipedia articles of each search query. We searched Wikipedia exclusively at this stage, as opposed to querying the entire Web, to obtain high quality texts and, hence, hopefuly more positive training examples (correct sentence plans). The first author then manually tagged the resulting 655 candidate sentence plans as positive or negative training instances, depending on whether or not they were correct. A candidate sentence plan was considered correct if and only if: (i) it would produce morphologically, syntactically, and semantically correct sentences; and (ii) the annotations of its slots (e.g., pos tags, voice, tense, agreement) were all correct. To compensate for class imbalance in the training set (16% positive vs. 84% negative candidate sentence plans), we replicated all the positive training instances (over-sampling) to obtain an equal number of positive and negative training instances.
Figure FIGREF89 shows the error rate of the classifier on (i) unseen instances (test error) and (ii) on the instances it has been trained on (training error). To obtain the curves of Fig. FIGREF89 , we performed a leave-one-out cross validation on the 655 instances (candidate sentence plans) we had constructed, i.e., we repeated the experiment 655 times, each time using a different instance as the only test instance and the other 654 instances as the training dataset. Within each repetition of the cross-validation, we iteratively trained the classifier on 10%, 20%, ..., 100% of the training dataset (654 instances, with over-sampling applied to them). The training error counts how many of the instances that were used to train the classifier were also correctly classified by the classifier. The test error counts how many of the test (unseen) instances (one in each repetition of the cross-validation) were correctly classified by the classifier (trained on the corresponding percentage of the training dataset). The error rates of Fig. FIGREF89 are averaged over the 655 repetitions of the cross-validation. The training error curve can be thought of as a lower bound of the test error curve, since a classifier typically performs better on the instances it has been trained on than on unseen instances. The two curves indicate that the classifier might perform slightly better with more training data, though the test error rate would remain above INLINEFORM1 . The relatively small distance of the right ends of the two curves indicates only mild overfitting when the entire training dataset is used.
To assess the contribution of the 251 features (Section SECREF34 ), we ranked them by decreasing Information Gain (ig) BIBREF29 computed on the 655 instances. Table TABREF90 shows the maximum, minimum, and average ig scores of the features in each group (subsection) of Section SECREF34 . On average, the pmi and token-based features are the best predictors, whereas the prominence features are the worst. The maximum scores, however, show that there is at least one good feature in each group, with the prominence features being again the worst in this respect. The minimum scores indicate that there is also at least one weak feature in every group, with the exception of the pmi features, where the minimum ig score (0.21) was much higher. Figure FIGREF91 shows the ig scores of all the features in each group, in ascending order. There are clearly many good features in every group, with the prominence features again being the weakest group overall.
We then iteratively trained the classifier on the entire training dataset ( INLINEFORM0 ), removing at each iteration the feature (among the remaining ones) with the smallest ig score. Figure FIGREF92 shows the resulting test and training error rate curves, again obtained using a leave-one-out cross-validation. As more features are removed, the distance between the training and test error decreases, because of reduced overfitting. When very few features are left (far right), the performance of the classifier on unseen instances becomes unstable. The best results are obtained using all (or almost all) of the features, but the test error is almost stable from approximately 50 to 200 removed features, indicating that there is a lot of redundancy (e.g., correlated features) in the feature set. Nevertheless, we did not remove any features in the subsequent experiments, since the overfitting was reasonably low and the training and test times of the MaxEnt classifier were also low (performing a leave-one-out cross-validation on the 655 instances with all the features took approximately 6 minutes). We hope to explore dimensionality reduction further (e.g., via pca) in future work.
In a subsequent experiment, the classifier was trained on the 655 instances (candidate sentence plans) of the previous section; recall that those instances were obtained from Wikipedia articles for Wine Ontology relations. The classifier was then embedded (without retraining) in our overall sentence plan generation method (sp or sp*, see Section SECREF66 ). The sentence plan generation method was then invoked to produce sentence plans from the entire Web, not just Wikipedia, and for the relations of all three ontologies, not just those of the Wine Ontology. We kept the top 10 returned documents per Web search (Section SECREF29 ), to reduce the time to complete them. The first author inspected the top 5 sentence plans per relation (as ranked by sp or sp*), marking them as correct or incorrect (as in Section UID86 ). We then computed the 1-in1, 1-in-5, and mrr scores of the produced sentence plans per ontology, along with weighted variants of the three measures. All six measures are defined as in Section UID73 , but for sentence plans instead of nl names; the weighted variants weigh each relation by the number of owl statements that mention it in the ontology.
Tables TABREF95 – TABREF97 show the results for the three ontologies. The configurations “with seeds of manual-nln” use the nl names from the manually authored linguistic resources to obtain seed names (Section SECREF29 ); by contrast, the configurations “with seeds of semi-auto-nln” use the semi-automatically produced nl names to obtain seed names. Recall that sp* reranks the candidate sentence plans using their coverage (Section SECREF66 ). Tables TABREF95 – TABREF97 also include results for a bootstrapping baseline (boot), described below. For each measure, the best results are shown in bold.
Overall, sp* performs better than sp, though the scores of the two methods are very close or identical in many cases, and occasionally sp performs better. Also, sp and sp* occasionally perform better when semi-automatically produced nl names are used to obtain seed names, than when manually authored nl names are used. It seems that manually authored nl names occasionally produce seeds that are uncommon on the Web and, hence, do not help produce good sentence plans, unlike semi-automatically produced nl names, which are extracted from the Web (and then manually filtered). The 1-in-5 results of Tables TABREF95 – TABREF97 show that our sentence plan generation method (especially sp*) performs very well in a semi-automatic scenario, especially if the weighted measures are considered. By contrast, our method does not always perform well in a fully automatic scenario (1-in-1 results); the Disease Ontology was the most difficult in that respect. Overall, sp* with seeds of semi-auto-nln seems to be the best version. The mrr scores of our method (all versions) were higher in the Wine Ontology and lower in the other two ones, which may be due to the fact that the classifier was trained for Wine Ontology relations (but with texts from Wikipedia).
While inspecting the sentence plans, we noticed several cases where the owl identifier of the relation was poor (e.g., the :locatedIn relation of the Wine Ontology connects wines to the regions producing them, but our method produced good sentence plans (e.g., [ INLINEFORM0 ] [is produced] [in] [ INLINEFORM1 ]). On the other hand, our method (all versions) produced very few (or none) sentence plans for relations with fewer than 10 seed name pairs. Also, the most commonly produced sentence plans are [ INLINEFORM2 ] [is] [ INLINEFORM3 ] and [ INLINEFORM4 ] [is] [ INLINEFORM5 ]. While the former may be appropriate for a message triple INLINEFORM6 , the latter is almost never appropriate, so we always discard it.
The top five sentence plans of sp* for each relation were also shown to the second human judge ( INLINEFORM0 ) who had examined the automatically produced nl names in the experiments of Section UID75 . For each relation, and for each one of its top five sentence plans, the second judge was shown a template view of the sentence plan (e.g., “ INLINEFORM1 is made from INLINEFORM2 ”), and an automatically generated sentence illustrating its use (e.g., “Cabernet Sauvignon is made from Cabernet Sauvignon grapes.”). The judge was asked to consider the templates and sentences, and mark the best correct sentence plan for each relation. The judge could also mark more than one sentence plans for the same relation, if more than one seemed correct and equally good; the judge was instructed not to mark any of the five sentence plans, if none seemed correct. The second judge completed this task (inspecting 40, 29, and 40 candidate sentence plans of the Wine, m-piro, and Disease Ontology, respectively) in approximately 5 minutes per ontology (15 minutes for all three ontologies); by contrast, manually authoring the sentence plans took approximately one working day per ontology. Again, these times suggest that the semi-automatic authoring scenario is viable and very useful in practice.
We also measured the agreement between the second judge ( INLINEFORM0 ) and the first author ( INLINEFORM1 ) in the semi-automatic authoring (selection) of sentence plans, as in Section UID75 . The results, reported in Table TABREF99 , show perfect agreement in the Wine and Disease Ontologies. In the m-piro ontology, the agreement was lower, but still reasonably high; the pseudo-recall score shows INLINEFORM2 did not select any sentence plan for some relations where INLINEFORM3 believed there was a correct one among 

Question: what ontologies did they use?
Answer: Wine Ontology, m-piro ontology, Disease Ontology

Wine Ontology, m-piro ontology, Disease Ontology