Accuracy is the number of correct decisions (individuals and classes correctly declared, or correctly not declared as anonymous), divided by the total number of individuals and classes (that we aimed to produce nl names for). The anonymity decisions of our method were perfect in the m-piro ontology and Disease Ontology. In the Wine Ontology, the precision of our method was also perfect, i.e., whenever our method decided to declare an individual or class as anonymous, this was a correct decision; but recall was lower, i.e., our method did not anonymize all the individual and classes that the manually authored nl names did. The latter is due to the fact that the manually authored nl names of the Wine ontology also anonymize 14 individuals and classes with complex identifiers (e.g., :SchlossVolradTrochenbierenausleseRiesling) to produce more readable texts. By contrast, our method declares individuals and classes as anonymous only to avoid redundancy in the generated texts (Section UID12 ), hence it does not anonymize the 14 individuals and classes.

This difference is mostly due to the fact that semi-auto did not always manage to produce sentence plans to convey correctly the semantics of the message triples, because of too few seeds, as in the experiments of the previous section. This also affected the clarity score of semi-auto in the m-piro ontology. The scores of auto were much lower, again indicating that our methods cannot be used in a fully automatic scenario. The scores of verbalizer were overall much lower than those of manual, again showing the importance of linguistic resources when generating texts from ontologies. The high non-redundancy score of verbalizer in the Disease Ontology is due to the fact that there are very few redundant message triples and no anonymous individuals or classes in the Disease Ontology. Hence, verbalizer, which treats all the message triples as important and does not anonymize any individuals or classes performs well in terms of non-redundancy. We made a similar observation in Section UID77 .

Again, these times suggest that the semi-automatic authoring scenario is viable and very useful in practice. We also measured the agreement between the second judge ( INLINEFORM0 ) and the first author ( INLINEFORM1 ) in the semi-automatic authoring (selection) of sentence plans, as in Section UID75 . The results, reported in Table TABREF99 , show perfect agreement in the Wine and Disease Ontologies. In the m-piro ontology, the agreement was lower, but still reasonably high; the pseudo-recall score shows INLINEFORM2 did not select any sentence plan for some relations where INLINEFORM3 believed there was a correct one among the top five. As a baseline, we also implemented a sentence plan generation method that uses a bootstrapping template extraction approach. Bootstrapping is often used in information extraction to obtain templates that extract instances of a particular relation (e.g., :makeFrom) from texts, starting from seed pairs of entity names (e.g., INLINEFORM0 “cream”, “milk” INLINEFORM1 ) for which the relation is known to hold  ,  ,  ,  .

The best results are obtained using all (or almost all) of the features, but the test error is almost stable from approximately 50 to 200 removed features, indicating that there is a lot of redundancy (e.g., correlated features) in the feature set. Nevertheless, we did not remove any features in the subsequent experiments, since the overfitting was reasonably low and the training and test times of the MaxEnt classifier were also low (performing a leave-one-out cross-validation on the 655 instances with all the features took approximately 6 minutes). We hope to explore dimensionality reduction further (e.g., via pca) in future work. In a subsequent experiment, the classifier was trained on the 655 instances (candidate sentence plans) of the previous section; recall that those instances were obtained from Wikipedia articles for Wine Ontology relations. The classifier was then embedded (without retraining) in our overall sentence plan generation method (sp or sp*, see Section SECREF66 ). The sentence plan generation method was then invoked to produce sentence plans from the entire Web, not just Wikipedia, and for the relations of all three ontologies, not just those of the Wine Ontology.

Some sentences of the original definition strings that did not include declared relation names, like the sentence “The virus affects...and humans” in the `Definition' string of Rift Valley Fever above, were discarded during the conversion, because it was not always possible to reliably convert them to appropriate owl statements. The new form of the Disease Ontology contains 6,746 classes, 15 relations, and 1,545 individuals. We aimed to automatically produce nl names and sentence plans for the 94 classes, 99 individuals, and 8 relations that are involved in the definitions of 30 randomly selected diseases. We manually authored nl names and sentence plans for the same classes, individuals, and relations, to be able to compare the quality of the resulting texts. Manually authored nl names and sentence plans for the Wine and m-piro ontologies are also available (they are included in the software of Naturalowl). We note that the relations (properties) INLINEFORM0 of our experiments are all used in message triples INLINEFORM1 , where INLINEFORM2 is an individual or class, i.e., they are object properties in owl's terminology.

Furthermore, manual uses manually authored intrest scores, verbalizer uses no interest scores, whereas auto and semi-auto use interest scores obtained from the (top-ranked or selected) nl names using the method of Section SECREF27 . All the other linguistic resources (most notably, text plans) are the same (manually authored) across all four configurations. We did not experiment with the boot and sp sentence plan generation methods in this section, since sp* performed overall better in the previous experiments. Below are example texts generated from the three ontologies by the four configurations we considered. manual: This is a moderate, dry Chenin Blanc. It has a full body. It is made by Foxen in the Santa Barbara County. semi-auto: This is a full, dry Chenin Blanc wine. It is moderate. It is made at the Foxen Winery in the Santa Barbara region. auto: This is dry Chenin Blanc and is the full. It is the moderate. It is made at Foxen and it is Santa Barbara. verbalizer: Foxen Chenin Blanc is Chenin Blanc. It has sugar Dry, it has maker Foxen and it has body Full.

auto-sp* uses the sp* method (Section SECREF66 ) with no human selection of sentence plans, i.e., the top-ranked sentence plan of each relation. We did not consider sp in this experiment, since the previous experiments indicated that sp* was overall better. In semi-auto-sp*, a human inspector (the first author) selected the best sentence plan of each relation among the five top-ranked sentence plans of sp*. Similarly, auto-boot and semi-auto-boot use the boot baseline of Section UID100 with no human selection or with a human selecting among the top five, respectively. Apart from the sentence plans, all six configurations use the same, manually authored other types of linguistic resources (e.g., nl names, interest scores, text plans to order the message triples). Below are example texts generated from the three ontologies by the six configurations. manual-sp: This is a moderate, dry Zinfandel. It has a full body. It is made by Elyse in the Napa County. semi-auto-sp*: This is a full, dry Zinfandel. It is moderate. It is made at Elyse in the Napa County.

obviously INLINEFORM0 is made from INLINEFORM1 obviously INLINEFORM2 is made from INLINEFORM3 in obviously INLINEFORM4 is made from INLINEFORM5 in California  INLINEFORM0 is made from INLINEFORM1 in INLINEFORM2 is made from INLINEFORM3 in California  Again, we discard extended templates that did not result from at least two sentences.

( INLINEFORM0 ) Clarity: The text is easy to understand, provided that the reader is familiar with the terminology and concepts of the domain (e.g., historical periods, grape varieties, virus names). ( INLINEFORM0 ) Semantic correctness: The text accurately conveys the information of the message triples.  Tables TABREF108 – TABREF110 show the scores of the six configurations of Naturalowl, averaged over the texts of each ontology. For each criterion, the best scores are shown in bold. In each criterion (row), we detected no statistically significant differences between scores marked with the same superscript; all the other differences (in the same row) were statistically significant. manual-sp was the best overall configuration, as one would expect, but semi-auto-sp* performed only slightly worse, with no detected statistically significant difference between the two configurations in most cases. The only notable exception was the semantic correctness of the m-piro ontology, where the difference between semi-auto-sp* and manual-sp was larger, because for some relations semi-auto-sp* produced sentence plans that did not correctly express the corresponding message triples, because of too few seeds. These findings confirm that sp* performs very well in a semi-automatic scenario.

This may help avoid concept drift, i.e., gradually obtaining templates that are more appropriate for other relations that share seed name pairs with the relation we wish to generate templates for. We did not use Eq. EQREF102 within each iteration, because in our experiments very few iterations (most often only the initial one) were needed. Also, using a function like Eq. EQREF102 within each iteration requires a threshold INLINEFORM0 , to discard templates with INLINEFORM1 at the end of each iteration, which is not trivial to tune. Similar functions can be used to score the new seed name pairs within each iteration or at the end of the bootstrapping. Since very few iterations (most often only the initial one) were needed in our experiments, we ended up using mostly (and most often only) the initial seed name pairs, which are known to be correct; hence, scoring the seed name pairs seemed unnecessary. Tables TABREF95 – TABREF97 show that the results of boot are consistently worse than the results of sp and sp*. As already noted, for most relations more than INLINEFORM0 templates had been produced at the end of the first iteration (with the initial seed name pairs) of boot.

For each individual or class, the message triples of its definition (regardless of their interest scores) along with the corresponding texts were given to exactly one student. The four texts of each individual or class were randomly ordered and the students did not know which configuration had generated each one of the four texts. For each individual or class, the students were asked to compare the four texts to each other and to the message triples, and score each text by stating how strongly they agreed or disagreed with statements INLINEFORM4 – INLINEFORM5 below. A scale from 1 to 5 was used (1: strong disagreement, 3: ambivalent, 5: strong agreement). Examples and more detailed guidelines were also provided to the students. ( INLINEFORM0 ) Sentence fluency: Each sentence of the text (on its own) is grammatical and sounds natural. ( INLINEFORM0 ) Clarity: The text is easy to understand, provided that the reader is familiar with the terminology and concepts of the domain (e.g., historical periods, grape varieties, virus names). ( INLINEFORM0 ) Semantic correctness: The text accurately conveys the information of the message triples.

The ontologies of our experiments	We used three ontologies: (i) the Wine Ontology, one of the most commonly used examples of owl ontologies; (ii) the m-piro ontology, which describes a collection of museum exhibits, was originally developed in the m-piro project  , was later ported to owl, and accompanies Naturalowl  ; and (iii) the Disease Ontology, which describes diseases, including their symptoms, causes etc. The Wine Ontology involves a wide variety of owl constructs and, hence, is a good test case for ontology verbalizers and nlg systems for owl. The m-piro ontology has been used to demonstrate the high quality texts that Naturalowl can produce, when appropriate manually authored linguistic resources are provided  . We wanted to investigate if texts of similar quality can be generated with automatically or semi-automatically acquired nl names and sentence plans. The Disease Ontology was developed by biomedical experts to address real-life information needs; hence, it constitutes a good real-world test case. The Wine Ontology contains 77 classes, 161 individuals, and 14 relations (properties).

Additional iterations were used only for 5 relations of the m-piro ontology. Hence, the differences in the performance of boot compared to sp and sp* are almost entirely due to the fact that boot uses the confidence function of Eq. EQREF102 instead of the MaxEnt classifier (and the coverage of the sentence plans, in the case of sp*). Hence, the MaxEnt classifier has an important contribution in the performance of sp and sp*. Tables TABREF95 – TABREF97 show that this contribution is large in all three ontologies, despite the fact that the classifier was trained on Wine Ontology relations only (but with texts from Wikipedia). To examine how sentence plans produced by different methods affect the perceived quality of generated texts, we showed automatically generated texts describing individuals and classes of the three ontologies to six computer science students, the same students as in the experiments of Section UID77 . We used six configurations of Naturalowl in this experiment. The no-sp configuration is given no sentence plans; in this case, Naturalowl automatically produces sentence plans by tokenizing the owl identifiers of the relations, acting like a simple verbalizer. manual-sp uses manually authored sentence plans.

The no-nln configuration, which uses tokenized identifiers of individuals and classes, performed overall much worse than manual-nln and semi-auto-nln in the Wine and m-piro ontologies, which shows the importance of nl names in the perceived quality of generated texts. The differences between no-nln, semi-auto-nln, and manual-nln were smaller in the Disease Ontology, where no statistically significant differences between the three configurations were detected. These smaller differences are due to the fact that the conversion of the Disease Ontology (Section SECREF67 ) produced many individuals whose owl identifiers are in effect long phrases, easily readable, and sometimes better than then top-ranked nl names of our methods; furthermore, our nl names method does not manage to produce any nl names for many of these individuals and, hence, semi-auto-nln ends up using their tokenized identifiers, like no-nln. We also note that there are very few redundant message triples and no anonymous individuals or classes in the Disease Ontology, which explains the higher non-redundancy scores of no-nln in the Disease Ontology, compared to the much lower non-redundancy scores of no-nln in the other two ontologies.

Introduction	The Semantic Web  ,  and the growing popularity of Linked Data (data published using Semantic Web technologies) have renewed interest in concept-to-text generation  , especially text generation from ontologies  ,  ,  ,  ,  ,  ,  ,  ,  . An ontology provides a conceptualization of a knowledge domain (e.g., wines, consumer electronics) by defining the classes and subclasses of the individuals (entities) in the domain, the possible relations between them etc. The current standard to specify Semantic Web ontologies is owl  , which is based on description logics  , rdf, and rdf schema  . owl2 is the latest version of owl  . Given an owl ontology for a knowledge domain, one can publish on the Web machine-readable data pertaining to that domain (e.g., catalogues of products, their features etc.), with the data having formally defined semantics based on the ontology. Several equivalent owl syntaxes have been developed, but people unfamiliar with formal knowledge representation have difficulties understanding them  . For example, the following statement defines the class of St. Emilion wines, using the functional-style syntax of owl, one of the easiest to understand.

semi-auto-boot performed clearly worse than semi-auto-sp* in this scenario. In the fully automatic scenario, with no human selection of sentence plans, auto-sp* was overall better than auto-boot, but still not good enough to be used in practice. The no-sp configuration, which uses sentence plans constructed by tokenizing the identifiers of the owl relations, obtained much lower sentence fluency and clarity scores than manual-sp, which shows the importance of sentence plans in the perceived quality of the texts. The semantic correctness scores of no-sp were also much lower than those of manual-sp in the Wine and m-piro ontologies, but the difference was smaller (with no detected statistically significant difference) in the Disease Ontology, because tokenizing the owl identifiers of the relations of the Disease Ontology (e.g., :has_symptom) leads to sentences that convey the correct information in most cases, even if the sentences are not particularly fluent and clear. The sentence fluency and clarity scores of no-sp in the Disease Ontology were also higher, compared to the scores of no-sp in the other two ontologies, for the same reason.

no-nln: Systemic mycosis is a kind of fungal infectious disease. It affects human body. It results in infection of internal organs. It is caused by Fungi.  We note that some nl names of semi-auto-nln and auto-nln can be easily improved using the Protégé plug-in of Naturalowl. For example, the nl name of the human body can be easily modified to include a definite article, which would improve the texts of semi-auto-nln and auto-nln in the Disease Ontology examples above (“affects the human body” instead of “affects human body”). Nevertheless, we made no such improvements. Recall that there are INLINEFORM0 non-trivial definitions of wine classes and wine individuals in the Wine Ontology, 49 exhibits in the m-piro ontology, and that we randomly selected 30 diseases from the Disease Ontology (Section SECREF67 ). Hence, we generated INLINEFORM1 texts from the Wine Ontology (with the four configurations of Naturalowl), INLINEFORM2 texts from the m-piro ontology, and INLINEFORM3 texts from the Disease ontology.

We kept the top 10 returned documents per Web search (Section SECREF29 ), to reduce the time to complete them. The first author inspected the top 5 sentence plans per relation (as ranked by sp or sp*), marking them as correct or incorrect (as in Section UID86 ). We then computed the 1-in1, 1-in-5, and mrr scores of the produced sentence plans per ontology, along with weighted variants of the three measures. All six measures are defined as in Section UID73 , but for sentence plans instead of nl names; the weighted variants weigh each relation by the number of owl statements that mention it in the ontology. Tables TABREF95 – TABREF97 show the results for the three ontologies. The configurations “with seeds of manual-nln” use the nl names from the manually authored linguistic resources to obtain seed names (Section SECREF29 ); by contrast, the configurations “with seeds of semi-auto-nln” use the semi-automatically produced nl names to obtain seed names. Recall that sp* reranks the candidate sentence plans using their coverage (Section SECREF66 ).

no-sp: Molluscum contagiosum is a kind of viral infectious disease and it located in the skin. It results in infections. It has symptom lesions. It transmitted by fomites and contact with the skin, and it has material basis in the molluscum contagiosum virus.  As in the corresponding experiment with nl names (Section UID77 ), we generated INLINEFORM0 texts from the Wine ontology (this time with six configurations), INLINEFORM1 texts from the m-piro ontology, and INLINEFORM2 texts from the Disease ontology. The students were asked to score each text by stating how strongly they agreed or disagreed with statements INLINEFORM3 – INLINEFORM4 below; the non-redundancy criterion was not used in this experiment, because all six configurations used the same (manually authored) nl names and interest scores. Otherwise, the experimental setup was the same as in the corresponding experiment with nl names (Section UID77 ). ( INLINEFORM0 ) Sentence fluency: Each sentence of the text (on its own) is grammatical and sounds natural.

Extracting Linguistic Resources from the Web for Concept-to-Text Generation	Many concept-to-text generation systems require domain-specific linguistic resources to produce high quality texts, but manually constructing these resources can be tedious and costly. Focusing on NaturalOWL, a publicly available state of the art natural language generator for OWL ontologies, we propose methods to extract from the Web sentence plans and natural language names, two of the most important types of domain-specific linguistic resources used by the generator. Experiments show that texts generated using linguistic resources extracted by our methods in a semi-automatic manner, with minimal human involvement, are perceived as being almost as good as texts generated using manually authored linguistic resources, and much better than texts produced by using linguistic resources extracted from the relation and entity identifiers of the ontology.

Conclusions and future work	Concept-to-text generation systems typically require domain-specific linguistic resources to produce high quality texts, but manually constructing these resources can be tedious and costly. Focusing on Naturalowl, a publicly available state of the art natural language generator for owl ontologies, we proposed methods to automatically or semi-automatically extract from the Web sentence plans and natural language names, two of the most important types of domain-specific generation resources. We showed experimentally that texts generated using linguistic resources produced by our methods in a semi-automatic manner, with minimal human involvement, are perceived as being almost as good as texts generated using manually authored linguistic resources, and much better than texts produced by using linguistic resources extracted from the relation and entity identifiers of the ontologies. Using our methods, constructing sentence plans and natural language names requires human effort of a few minutes or hours, respectively, per ontology, whereas constructing them manually from scratch is typically a matter of days. Also, our methods do not require any familiarity with the internals of Naturalowl and the details of its linguistic resources. Furthermore, unlike previous related work, no parallel corpus of sentences and semantic representations is required.

Experiments	We now present the experiments we performed to evaluate our methods that generate nl names and sentence plans. We first discuss the ontologies that we used in our experiments.

Several properties are also used only to link these pseudo-individuals (in effect, the canned texts) to other individuals or classes (e.g., to link :aryballos-def to the class :Aryballos); and many other classes are used only to group pseudo-individuals (e.g., pseudo-individuals whose canned texts describe types of vessels all belong in a common class). In our experiments, we ignored pseudo-individuals, properties, and classes that are used to represent, link, and group canned texts, since we focus on generating texts from symbolic information. We aimed to produce nl names and sentence plans for the remaining 30 classes, 127 individuals, and 12 relations, which are all involved in the definitions (descriptions) of the 49 exhibits of the collection the ontology is about. The Disease Ontology currently contains information about 6,286 diseases, all represented as classes. Apart from is-a relations, synonyms, and pointers to related terms, however, all the other information is represented using strings containing quasi-English sentences with relation names used mostly as verbs.

Hence, we believe that our work is also applicable, at least in principle, to other nlg systems. For example, ontosum  , which generates natural language descriptions of individuals, but apparently not classes, from rdf schema and owl ontologies, uses similar processing stages, and linguistic resources corresponding to nl names and sentence plans. Reiter et al.  discuss the different types of knowledge that nlg systems require and the difficulties of obtaining them (e.g., by interviewing experts or analyzing corpora). Unlike Reiter et al., we assume that domain knowledge is already available, in the form of owl ontologies. The domain-specific linguistic resources of Naturalowl belong in the `domain communication knowledge' of Reiter et al., who do not describe particular corpus-based algorithms to acquire knowledge. Ngonga Ngomo et al.  discuss sparql2nl, a system that translates sparql queries to English. sparql2nl uses techniques similar to those of simple ontology verbalizers. To express the rdf triples INLINEFORM0 that are involved in a sparql query, it assumes that the labels (e.g., rdfs:label, perhaps also identifiers) of the relations are verbs or nouns.

The top five sentence plans of sp* for each relation were also shown to the second human judge ( INLINEFORM0 ) who had examined the automatically produced nl names in the experiments of Section UID75 . For each relation, and for each one of its top five sentence plans, the second judge was shown a template view of the sentence plan (e.g., “ INLINEFORM1 is made from INLINEFORM2 ”), and an automatically generated sentence illustrating its use (e.g., “Cabernet Sauvignon is made from Cabernet Sauvignon grapes.”). The judge was asked to consider the templates and sentences, and mark the best correct sentence plan for each relation. The judge could also mark more than one sentence plans for the same relation, if more than one seemed correct and equally good; the judge was instructed not to mark any of the five sentence plans, if none seemed correct. The second judge completed this task (inspecting 40, 29, and 40 candidate sentence plans of the Wine, m-piro, and Disease Ontology, respectively) in approximately 5 minutes per ontology (15 minutes for all three ontologies); by contrast, manually authoring the sentence plans took approximately one working day per ontology.

We then invoked our nl name generation method for the individuals and classes it had not declared as anonymous (160 in the Wine Ontology, 108 in the m-piro ontology, 195 in the Disease Ontology), using the top 10 returned documents per Web search (or top 20, when the search engine proposed spelling corrections – see Section UID16 ). We ranked the produced nl names (as in Sections UID16 and SECREF22 ), and kept the top 5 nl names per individual or class. The first author then inspected the resulting nl names and marked each one as correct or incorrect. An nl name was considered correct if and only if: (i) it would produce morphologically, syntactically, and semantically correct and unambiguous noun phrases (e.g., “Cabernet Sauvignon grape” is correct for :CabernetSauvignonGrape, but “Cabernet Sauvignon wine”, “Cabernet Sauvignon”, or “grape” are incorrect); and (ii) its slot annotations (e.g., pos tags, gender, agreement) were all correct. Table TABREF74 shows the results of this experiment.

For example, there is an axiom in the ontology stating that the Rift Valley Fever (doid_1328) is a kind of viral infectious disease (doid_934). All the other information about the Rift Valley Fever is provided in a string, shown below as `Definition'. The tokens that contain underscores (e.g., results_in) are relation names. The ontology declares all the relation names, but uses them only inside `Definition' strings. Apart from diseases, it does not define any of the other entities mentioned in the `Definition' strings (e.g., symptoms, viruses). Name: Rift Valley Fever (doid_1328) is-a: viral infectious disease (doid_934) Definition: A viral infectious disease that results_in infection, has_material_basis_in Rift Valley fever virus, which is transmitted_by Aedes mosquitoes. The virus affects domestic animals (cattle, buffalo, sheep, goats, and camels) and humans.

The judge was asked to consider the phrases and sentences, and mark the best correct nl name for each individual or class. The judge could also mark more than one nl names for the same individual or class, if more than one seemed correct and equally good; the judge was instructed not to mark any of the five nl names, if none seemed correct. The judge completed this task in 49, 45, and 75 minutes for the Wine, m-piro, and Disease Ontology (727, 540, 965 candidate nl names), respectively; by contrast, manually authoring the nl names of the three ontologies took approximately 2, 2, and 3 working days, respectively. These times and the fact that the second judge was not aware of the internals of Naturalowl and its resources suggest that the semi-automatic authoring scenario is viable and very useful in practice. Table TABREF76 compares the decisions of the second judge, hereafter called INLINEFORM0 , to those of the first author, hereafter called INLINEFORM1 . INLINEFORM2 was able to view the full details of the nl names using Naturalowl's Protégé plug-in, unlike INLINEFORM3 who viewed only phrases and example sentences.

Experiments with automatically or semi-automatically produced sentence plans	We now present the experiments we performed to evaluate our method that generates sentence plans (Section SECREF4 ). Recall that our method employs a MaxEnt classifier to predict the probability that a candidate sentence plan is correct (positive class) or incorrect (negative class). To create training instances for the MaxEnt classifier, we used our sentence plan generation method without the classifier to obtain candidate sentence plans (as in Sections SECREF29 and SECREF33 ) from Wikipedia for the seven relations of the Wine Ontology (Section SECREF67 ). We used the manually authored nl names of the Wine Ontology to obtain seed names, and the top 50 Wikipedia articles of each search query. We searched Wikipedia exclusively at this stage, as opposed to querying the entire Web, to obtain high quality texts and, hence, hopefuly more positive training examples (correct sentence plans). The first author then manually tagged the resulting 655 candidate sentence plans as positive or negative training instances, depending on whether or not they were correct.

For example, in a museum ontology, there may be a coin whose owl identifier is :exhibit49. We may not wish to provide an nl name for this individual (it may not have an English name); and we may want Naturalowl to avoid referring to the coin by tokenizing its identifier (“exhibit 49”). By declaring the coin as anonymous, Naturalowl would use only the nl name of its class (e.g., “this coin”), simply “this”, or a pronoun.

SubClassOf(:StEmilion  ObjectIntersectionOf(:Bordeaux  ObjectHasValue(:locatedIn :stEmilionRegion)  ObjectHasValue(:hasColor :red)  ObjectHasValue(:hasFlavor :strong)  ObjectHasValue(:madeFrom :cabernetSauvignonGrape)  ObjectMaxCardinality(1 :madeFrom)))  To make ontologies easier to understand, several ontology verbalizers have been developed  ,  ,  ,  ,  ,  ,  ,  ,  ,  . Although verbalizers can be viewed as performing a kind of light natural language generation (nlg), they usually translate the axioms (in our case, owl statements) of the ontology one by one to controlled, often not entirely fluent English statements, typically without considering the coherence of the resulting texts. By contrast, more elaborate nlg systems  ,  ,  can produce more fluent and coherent multi-sentence texts, but they need domain-specific linguistic resources. For example, Naturalowl  , a publicly available nlg system for owl ontologies, produces the following description of St. Emilion wines from the owl statement above.

Apart from the configurations of Naturalowl, the experimental setup was the same as in Sections UID77 and UID104 . We generated INLINEFORM0 texts from the Wine ontology, INLINEFORM1 texts from the m-piro ontology, and INLINEFORM2 texts from the Disease ontology. The students were now asked to score each text for sentence fluency, clarity, semantic correctness, and non-redundancy, by stating how strongly they agreed or disagreed with statements INLINEFORM3 – INLINEFORM4 of Section UID77 . Tables TABREF113 – TABREF115 show the scores of the four configurations of Naturalowl, averaged over the texts of each ontology. For each criterion, the best scores are shown in bold. In each criterion (row), we detected no statistically significant differences between scores marked with the same superscript; all the other differences (in the same row) were statistically significant. manual had the best overall scores, as one would expect, but the scores of semi-auto were close, in most cases with no detected statistically significant difference, despite the combined errors of the methods that produce nl names and sentence plans. The biggest difference between semi-auto and manual was in the semantic correctness criterion of the m-piro ontology.

The seed pairs are used as queries in a search engine to obtain documents that contain them in the same sentence (e.g., “cream is made from milk”). Templates are then obtained by replacing the seeds with slots in the retrieved sentences (e.g., “ INLINEFORM2 is made from INLINEFORM3 ”). The templates (without their slots, e.g., “is made from”) are then used as phrasal search queries to obtain new sentences (e.g., “gasoline is made from petroleum”), from which new seed pairs ( INLINEFORM4 “gasoline”, “petroleum” INLINEFORM5 ) are obtained. A new iteration can then start with the new seed pairs, leading to new templates, and so on. Given a relation INLINEFORM0 , our baseline, denoted boot, first constructs seed name pairs using the ontology and the nl names, as in Section SECREF29 ; we used only manually authored nl names in the experiments with boot. Then, boot uses the seed name pairs to obtain templates (“ INLINEFORM1 is made from INLINEFORM2 ”) from the Web, again as in Section SECREF29 .

SubClassOf(:DOID_1328  ObjectIntersectionOf(:DOID_934  ObjectHasValue(:results_in :infection)  ObjectHasValue(:has_material_basis_in :Rift_Valley_fever_virus)  ObjectHasValue(:transmitted_by :Aedes_mosquitoes)  ObjectHasValue(:has_symptom :jaundice)  ObjectHasValue(:has_symptom :vomiting_blood)  ObjectHasValue(:has_symptom :passing_blood_in_the_feces)  ObjectHasValue(:has_symptom :ecchymoses_(caused_by_bleeding_in_the_skin))  ObjectHasValue(:has_symptom :bleeding_from_the_nose_or_gums)  ObjectHasValue(:has_symptom :menorrhagia)  ObjectHasValue(:has_symptom :bleeding_from_venepuncture_sites)))  The new form of the ontology was produced automatically, using patterns that searched the definition strings for relation names (e.g., results_in), sentence breaks, and words that introduce secondary clauses (e.g., “that”, “which”).

Extracting noun phrases from the Web	We first collect the owl statements of the ontology that describe INLINEFORM0 , the individual or class we want to produce an nl name for, and turn them into message triples INLINEFORM1 , as when generating texts. For example, for the class INLINEFORM2 :KalinCellarsSemillon of the Wine Ontology, one of the ontologies of our experiments, three of the resulting message triples are:  INLINEFORM0 :KalinCellarsSemillon, isA, :Semillon INLINEFORM1   INLINEFORM0 :KalinCellarsSemillon, :hasMaker, :KalinCellars INLINEFORM1   INLINEFORM0 :KalinCellarsSemillon, :hasFlavor, :Strong INLINEFORM1  For each collected message triple INLINEFORM0 , we then produce INLINEFORM1 and INLINEFORM2 , where INLINEFORM3 is the tokenized identifier of INLINEFORM4 .

In order to examine how the produced nl names affect the perceived quality of the generated texts, we showed automatically generated texts describing individuals and classes of the three ontologies to six computer science students not involved in the work of this article; they were all fluent, though not native, English speakers. We generated texts using Naturalowl configured in four ways. The no-nln configuration uses no nl names; in this case, Naturalowl uses the tokenized owl identifiers of the individuals and classes as their names. manual-nln uses manually authored nl names. auto-nln uses the top-ranked nl name that our nl names method produces for each individual and class. Finally, semi-auto-nln uses the nl name (of each individual or class) that a human inspector (the first author of this article) selected among the top five nl names produced by our method. Additionally, both auto-nln and semi-auto-nln use the methods of Sections UID12 and SECREF27 to anonymize individuals or classes and to infer interest scores from nl names, whereas manual-nln uses the anonymity declarations and interest scores of the manually authored linguistic resources, and no-nln uses no anonymity declarations and no interest scores.

It needs, however: a sentence plan for each relation (e.g., :locatedIn) of the ontology, i.e., a linguistically annotated template showing how to express the relation; a natural language name for each class and individual, i.e., a linguistically annotated noun phrase to be used as the name of the class or individual; a text plan specifying the order in which relations should be expressed etc. Similar domain-specific linguistic resources are used in most concept-to-text systems  . Manually constructing resources of this kind, however, can be tedious and costly. St. Emilion is a kind of red, strong Bordeaux from the St. Emilion region. It is made from exactly one grape variety: Cabernet Sauvignon grapes.  Instead of requiring domain-specific linguistic resources, simpler verbalizers use the owl identifiers of classes and individuals (e.g., :cabernetSauvignonGrape) typically split into tokens as their natural language names, they express relations using phrases obtained by tokenizing the owl identifiers of the relations (e.g., :hasColor), they order the resulting sentences following the ordering of the corresponding owl statements etc.

Our method to automatically extract sentence plans from the Web	To produce a sentence plan for a relation, we first extract slotted string templates (e.g., “ INLINEFORM0 is made from INLINEFORM1 ”) from the Web using seeds (values of INLINEFORM2 ) from the ontology. We then enhance the templates by adding linguistic annotations (e.g., pos tags, agreement, voice, tense) and missing components (e.g., auxiliary verbs) turning the templates into candidate sentence plans. The candidate sentence plans are then scored by a Maximum Entropy classifier to identify the best few (again up to 5) candidate sentence plans for each relation. In a fully automatic scenario, the sentence plan that the classifier considers best for each relation is used. In a semi-automatic scenario, the few top sentence plans of each relation are shown to a human author, who picks the best one.

The mrr scores of our method (all versions) were higher in the Wine Ontology and lower in the other two ones, which may be due to the fact that the classifier was trained for Wine Ontology relations (but with texts from Wikipedia). While inspecting the sentence plans, we noticed several cases where the owl identifier of the relation was poor (e.g., the :locatedIn relation of the Wine Ontology connects wines to the regions producing them, but our method produced good sentence plans (e.g., [ INLINEFORM0 ] [is produced] [in] [ INLINEFORM1 ]). On the other hand, our method (all versions) produced very few (or none) sentence plans for relations with fewer than 10 seed name pairs. Also, the most commonly produced sentence plans are [ INLINEFORM2 ] [is] [ INLINEFORM3 ] and [ INLINEFORM4 ] [is] [ INLINEFORM5 ]. While the former may be appropriate for a message triple INLINEFORM6 , the latter is almost never appropriate, so we always discard it.

We do not examine how other types of domain-specific linguistic resources (e.g., text plans) can be generated, leaving them for future work. We base our work on Naturalowl, because it appears to be the only open-source nlg system for owl that implements all the processing stages of a typical nlg pipeline  , it supports owl2, it is extensively documented, and has been tested with several ontologies. The processing stages and linguistic resources of Naturalowl, however, are typical of nlg systems  . Hence, we believe that our work is also applicable, at least in principle, to other nlg systems. Our methods may also be useful in simpler verbalizers, where the main concern seems to be to avoid manually authoring domain-specific linguistic resources. Experiments show that texts generated using linguistic resources extracted by our methods with minimal human involvement are perceived as being almost as good as texts generated using manually authored linguistic resources, and much better than texts produced by using tokenized owl identifiers. Section SECREF2 below provides background information about Naturalowl, especially its natural language names and sentence plans.

On the downside, our methods do not perform sufficiently well in a fully-automatic scenario, with no human involvement during the construction of the linguistic resources. The processing stages and linguistic resources of Naturalowl are typical of nlg systems. Hence, we believe that our work is also applicable, at least in principle, to other nlg systems. Our methods may also be useful in simpler ontology verbalizers, where the main concern seems to be to avoid manually authoring domain-specific linguistic resources, currently at the expense of producing texts of much lower quality. Future work could aim to improve our methods to allow using them in a fully automatic manner. Further work could also explore how other kinds of domain-specific linguistic resources for nlg, most importantly text plans, could be constructed automatically or semi-automatically. Another future goal might be to consider languages other than English.

Some of these errors, however, may not have a significant effect on the generated texts (e.g., using the tokenized identifer “infection of the keratinized layers”, which is the default when no nl name is provided, may still lead to a reasonable text). Again, the experiment of Section UID77 below sheds more light on this point. The top five automatically produced nl names of each individual and class were also shown to a second human judge. The second judge was a computer science researcher not involved in nlg, fluent in English, though not a native speaker. For each individual or class, and for each one of its top five nl names, the judge was shown a phrase produced by the nl name (e.g., “Cabernet Sauvignon”), an automatically generated sentence about the individual or class (expressing a message triple of the ontology) illustrating the use of the nl name (e.g., “Cabernet Sauvignon is a kind of wine.”), and a sentence where the nl name had been automatically replaced by a pronoun (e.g., “It is a kind of wine.”) to check the gender of the nl name.

We aimed to produce nl names and sentence plans for the 49 classes, 146 individuals, and 7 relations that are directly involved in non-trivial definitions of wines (43 definitions of wine classes, 52 definitions of wine individuals), excluding classes, individuals, and relations that are only used to define wineries, wine-producing regions etc. By “non-trivial definitions” we mean that we ignored definitions that humans understand as repeating information that is obvious from the name of the defined class or individual (e.g., the definition of :RedWine in effect says that a red wine is a wine with red color). The m-piro ontology currently contains 76 classes, 508 individuals, and 41 relations. Many individuals, however, are used to represent canned texts (e.g., manually written descriptions of particular types of exhibits) that are difficult to generate from symbolic information. For example, there is a pseudo-individual :aryballos-def whose nl name is the fixed string “An aryballos was a small spherical vase with a narrow neck, in which the athletes kept the oil they spread their bodies with”.

Apart from the nl names, anonymity declarations, and interest scores, all four configurations use the same, manually authored other types of linguistic resources (e.g., sentence plans, text plans to order the message triples). Below are example texts generated from the three ontologies by the four configurations. manual-nln: This is a moderate, dry Zinfandel. It has a medium body. It is made by Saucelito Canyon in the city of Arroyo Grande. semi-auto-nln: This is a moderate, dry Zinfandel wine. It has a medium body. It is made by the Saucelito Canyon Winery in the Arroyo Grande area. auto-nln: This is a dry Zinfandel and has the medium body. It is the moderate. It is made by Saucelito Canyon in Arroyo Grande. no-nln: Saucelito Canyon Zinfandel is Zinfandel. It is Dry. It has a Medium body. It is Moderate. It is made by Saucelito Canyon. It is made in Arroyo Grande Region.

Tables TABREF95 – TABREF97 also include results for a bootstrapping baseline (boot), described below. For each measure, the best results are shown in bold. Overall, sp* performs better than sp, though the scores of the two methods are very close or identical in many cases, and occasionally sp performs better. Also, sp and sp* occasionally perform better when semi-automatically produced nl names are used to obtain seed names, than when manually authored nl names are used. It seems that manually authored nl names occasionally produce seeds that are uncommon on the Web and, hence, do not help produce good sentence plans, unlike semi-automatically produced nl names, which are extracted from the Web (and then manually filtered). The 1-in-5 results of Tables TABREF95 – TABREF97 show that our sentence plan generation method (especially sp*) performs very well in a semi-automatic scenario, especially if the weighted measures are considered. By contrast, our method does not always perform well in a fully automatic scenario (1-in-1 results); the Disease Ontology was the most difficult in that respect. Overall, sp* with seeds of semi-auto-nln seems to be the best version.

By contast, we aim to assign a single high quality nl name to each individual or class of a given ontology. Furthermore, our nl names are not simply strings, but contain additional information (e.g., head, gender, number, agreement) that helps produce high quality texts.

Background information about NaturalOWL	Given an owl ontology and a particular target individual or class to describe, Naturalowl first scans the ontology to select statements relevant to the target. It then converts each relevant statement into (possibly multiple) message triples of the form INLINEFORM0 , where INLINEFORM1 is an individual or class, INLINEFORM2 is another individual, class, or datatype value, and INLINEFORM3 is a relation (property) that connects INLINEFORM4 to INLINEFORM5 . For example, the ObjectHasValue(:madeFrom :cabernetSauvignonGrape) part of the owl statement above is converted to the message triple INLINEFORM6 :StEmilion, :madeFrom, :cabernetSauvignonGrape INLINEFORM7 . Message triples are similar to rdf triples, but they are easier to express as sentences. Unlike rdf triples, the relations ( INLINEFORM8 ) of the message triples may include relation modifiers. For example, the ObjectMaxCardinality(1 :madeFrom) part of the owl statement above is turned into the message triple INLINEFORM9 :StEmilion, maxCardinality(:madeFrom), 1 INLINEFORM10 , where maxCardinality is a relation modifier.

Then, given an unseen set of attribute-value pairs, multiple alternative dependency trees are constructed in a top-down manner, using beam search and the maximum entropy model to estimate the probabilities of the trees being constructed. The most probable tree that expresses all the attribute-value pairs is eventually chosen, and the corresponding phrase is returned. In later work  , the generated dependency trees are further altered by a set of hand-crafted rules that add unmentioned attributes, and the trees are also ranked by language models. In our case, where we aim to express multiple message triples INLINEFORM1 all describing an individual or class INLINEFORM2 , we can think of the message triples as attribute-value pairs INLINEFORM3 . To apply the methods of Ratnaparkhi, however, a parallel training corpus with sets of attribute-value pairs (or message triples) and the corresponding target texts would be needed; and corpora of this kind are difficult to obtain. By contrast, our methods require no parallel corpus and, hence, can be more easily applied to ontologies of new domains.

mrr rewards more those methods that place correct nl names towards the top of the five returned ones. The weighted scores of Table TABREF74 are similar, but they weigh each individual or class by the number of owl statements that mention it in the ontology. The results of Table TABREF74 show that our nl names method performs very well in a semi-automatic scenario. In a fully automatic scenario, however, there is large scope for improvement. We note, though, that our definition of correctness (of nl names) in the experiment of this section was very strict. For example, an nl name with only a single error in its slot annotations (e.g., a wrong gender in a noun slot) was counted as incorrect, even if in practice the error might have a minimal effect on the generated texts that would use the nl name. The experiment of Section UID77 below, where nl names are considered in the context of generated texts, sheds more light on this point. By inspecting the produced nl names, we noticed that our method is very resilient to spelling errors and abbreviations in the owl identifiers of individuals and classes.

Experiments with automatically or semi-automatically produced NL names	We now present experiments we performed with our method that generates nl names (Section SECREF3 ). In a first experiment, we measured how well our nl names method determines which individuals and classes should be anonymous (Sections SECREF3 and UID12 ). We compared the decisions of our method against the corresponding anonymity declarations in the manually authored nl names of the three ontologies. Table TABREF72 summarizes the results of this experiment. Precision is the total number of individuals and classes our nl names method correctly (in agreement with the manually authored nl names) declared as anonymous, divided by the total number of individuals and classes our method declared as anonymous. Recall is the total number of individuals and classes our nl names method correctly declared as anonymous, divided by the total number of individuals and classes (among those we aimed to produce nl names for) that the manually authored nl names declared as anonymous. For the Disease Ontology, the manually authored nl names and our nl names method agreed that no individuals and classes (that we aimed to produce nl names for) should be anonymous, which is why precision and recall are undefined.

Without domain-specific linguistic resources, Naturalowl behaves like a simple verbalizer, producing the following lower quality text from the owl statement above. A further limitation of using tokenized owl identifiers is that non-English texts cannot be generated, since owl identifiers are usually English-like. St Emilion is Bordeaux. St Emilion located in St Emilion Region. St Emilion has color Red. St Emilion has flavor Strong. St Emilion made from grape exactly 1: Cabernet Sauvignon Grape.  Previous experiments  indicate that the texts that Naturalowl generates with domain-specific linguistic resources are perceived as significantly better than (i) those of swat, one of the best available owl verbalizers  ,  , and (ii) those of Naturalowl without domain-specific linguistic resources, with little or no difference between (i) and (ii). The largest difference in the perceived quality of the texts was reported to be due to the sentence plans, natural language names, and (to a lesser extent) text plans. In this paper, we present methods to automatically or semi-automatically extract from the Web the natural language names and sentence plans required by Naturalowl for a given ontology.

( INLINEFORM0 ) Non-redundancy: There is no redundancy in the text (e.g., stating the obvious, repetitions). Tables TABREF80 – TABREF82 show the scores of the four configurations of Naturalowl, averaged over the texts of each ontology. For each criterion, the best scores are shown in bold. In each criterion (row), we detected no statistically significant differences between scores marked with the same superscript; all the other differences (in the same row) were statistically significant. Overall, the manually authored nl names led to the best (near-perfect) scores, as one might expect. The scores of semi-auto-nln were overall slightly lower, but still high (always INLINEFORM2 ) and no statistically significant differences to the corresponding scores of manual-nln were detected. These findings confirm that our nl names method performs very well in a semi-automatic scenario, where a human inspects and selects among the top-ranked automatically produced nl names. By contrast, auto-nln performed overall much worse than semi-auto-nln and manual-nln, and often worse than no-nln, which again indicates that our nl names method cannot be used in a fully automatic manner.

The infection has_symptom jaundice, has_symptom vomiting blood, has_symptom passing blood in the feces, has_symptom ecchymoses (caused by bleeding in the skin), has_symptom bleeding from the nose or gums, has_symptom menorrhagia and has_symptom bleeding from venepuncture sites.  We defined as individuals all the non-disease entities mentioned in the `Definition' strings, also adding statements to formally express the relations mentioned in the original `Definition' strings. For example, the resulting ontology contains the following definition of Rift Valley Fever, where :infection, :Rift_Valley_fever_virus, :Aedes_mosquitoes, :jaundice etc. are new individuals.

Extracting templates from the Web	For each relation INLINEFORM0 that we want to generate a sentence plan for, our method first obtains the owl statements of the ontology that involve the relation and turns them into message triples INLINEFORM1 , as when generating texts. For example, if the relation is :madeFrom, two of the triples may be:  INLINEFORM0 :StEmilion, :madeFrom, :cabernetSauvignonGrape INLINEFORM1 , INLINEFORM2 :Semillon, :madeFrom, :SemillonGrape INLINEFORM3  To these triples, we add more by replacing the INLINEFORM0 , INLINEFORM1 , or both of each originally obtained triple by their classes (if INLINEFORM2 or INLINEFORM3 are individuals), their parent classes, or their equivalent classes. For example, from INLINEFORM4 :StEmilion, :madeFrom, :cabernetSauvignonGrape INLINEFORM5 we also obtain the following three triples, because Wine is a parent class of StEmilion, and Grape is a parent class of :cabernetSauvignonGrape.

For example, it returns nl names producing “a Côte d'Or wine” for :CotesDOr, and “the Naples National Archaeological Museum” for :national-arch-napoli. Several wrongly produced nl names are due to errors of tools that our method invokes (e.g., parser, ner). Other errors are due to over-shortened altTokNames (Section SECREF10 ); e.g., one of the altTokNames of :CorbansDryWhiteRiesling was simply “Dry White”, which leads to an nl name that does not identify the particular wine clearly enough. Finally, in the Disease Ontology, our automatic conversion of the `Description' strings produced many individuals whose identifiers are in effect long phrases (see, for example, the owl description of :doid_1328 in Section SECREF67 ). Our nl names method manages to produce appropriate nl names (with correct slot annotations etc.) for some of them (e.g., :mutation_in_the_SLC26A2_gene), but produces no nl names in other cases (e.g., :infection_of_the_keratinized_layers).

Related work	Simple ontology verbalizers  ,  ,  ,  ,  ,  ,  ,  ,  typically produce texts describing individuals and classes without requiring manually authored domain-dependent linguistic resources. They usually tokenize the owl identifiers or labels (e.g., rdfs:label) of the individuals or classes to obtain nl names and sentence plans. Androutsopoulos et al.  showed that the texts of the swat verbalizer  ,  , one of the best publicly available verbalizers, are perceived as being of significantly lower quality compared to texts generated by Naturalowl with domain-dependent linguistic resources; nl names, sentence plans, and (to a lesser extent) text plans were found to contribute most to this difference. Without domain-dependent linguistic resources, Naturalowl was found to generate texts of the same quality as the swat verbalizer. Naturalowl is based on ideas from ilex  and m-piro  ,  . Excluding simple verbalizers, it is the only publicly available nlg system for owl, which is why we based our work on it. Nevertheless, its processing stages and linguistic resources are typical of nlg systems  ,  .

Pseudo-recall is the number of individuals and classes for which both INLINEFORM14 and INLINEFORM15 marked at least one nl name as correct, divided by the number of individuals and classes for which INLINEFORM16 marked at least one nl name as correct; this measure shows how frequently INLINEFORM17 managed to find at least one correct (according to INLINEFORM18 ) nl name, when there was at least one correct nl name among the top five. Computing the true recall of the decisions of INLINEFORM19 against those of INLINEFORM20 would be inappropriate, because INLINEFORM21 was instructed to mark only the best nl name(s) of each individual and class, unlike INLINEFORM22 who was instructed to mark all the correct ones. We also calculated Cohen's Kappa between INLINEFORM23 and INLINEFORM24 ; for each individual or class, if INLINEFORM25 had marked more than one nl names as correct, we kept only the top-most one, and similarly for INLINEFORM26 , hence each judge had six possible choices (including marking no nl name) per individual and class. The results of Table TABREF76 indicate strong inter-annotator agreement in the semi-automatic authoring of nl names in all three ontologies.

Joint experiments with extracted NL names and sentence plans	In a final set of experiments, we examined the effect of combining our methods that produce nl names and sentence plans. We experimented with four configurations of Naturalowl. The auto configuration produces nl names using the method of Section SECREF3 ; it then uses the most highly ranked nl name of each individual or class to produce seed name pairs, and invokes the sp* method of Section SECREF4 to produce sentence plans; it then uses the most highly ranked sentence plan for each relation. semi-auto also produces nl names using the method of Section SECREF3 , but a human selects the best nl name of each individual or class among the five most highly ranked ones; the selected nl names are then used to produce seed name pairs, and the sp* method is invoked to produce sentence plans; a human then selects the best sentence plan of each relation among the five most highly ranked ones. The manual configuration uses manually authored nl names and sentence plans. In the verbalizer configuration, no nl names and sentence plans are provided to Naturalowl; hence, acting like a simple verbalizer, Naturalowl produces nl names and sentence plans by tokenizing the owl identifiers of individuals, classes, and relations.

Konstas and Lapata  use a probabilistic context-free grammar to convert a given set of database entries to a single sentence (or phrase). Roughly speaking, in each parse tree of the grammar, the leaves are the words of a sentence, and the internal nodes indicate which database entries are expressed by each subtree. The grammar is constructed using hand-crafted templates of rewrite rules and a parallel training corpus of database entries and sentences; a generative model based on the work of Liang et al.  is employed to estimate the probabilities of the grammar. Subsequently, all the parse trees of the grammar for the sentences of the training corpus and the corresponding database entries are represented as a weighted directed hypergraph  . The hypergraph's weights are estimated using the inside-outside algorithm  on the training corpus. Following Huang and Chiang  , the hypergraph nodes are then integrated with an INLINEFORM0 -gram language model trained on the sentences of the corpus. Given a new set of database entries, the most probable derivation is found in the hypergraph using a INLINEFORM1 -best Viterbi search with cube pruning  and the final sentence is obtained from the derivation.

We then use the maximum, minimum, average, total, and standard deviation of these cosine similarities as features of INLINEFORM7 . Intuitively, they show how good the matches that produced INLINEFORM8 were. We repeat for each seed name INLINEFORM9 and np anchor INLINEFORM10 that matched (as second elements of their pairs) to produce INLINEFORM11 , this time computing INLINEFORM12 , obtaining five additional features of INLINEFORM13 . We do the same using INLINEFORM0 , defined below, instead of INLINEFORM1 : DISPLAYFORM0  where: INLINEFORM0 , INLINEFORM1 are the lengths (in tokens) of INLINEFORM2 ; INLINEFORM3 , INLINEFORM4 are the token sequences of INLINEFORM5 , respectively; INLINEFORM6 is the probability of encountering token INLINEFORM7 in a parsed sentence of INLINEFORM8 ; and INLINEFORM9 is the probability of encountering both INLINEFORM10 and INLINEFORM11 in the same parsed sentence of INLINEFORM12 ; we use Laplace estimates for these probabilities.

Again, we compute INLINEFORM13 for every seed name INLINEFORM14 and np anchor INLINEFORM15 that matched to produce a particular sentence plan INLINEFORM16 , and we use the maximum, minimum, average, total, and standard deviation of these scores as features of INLINEFORM17 . We repeat using INLINEFORM18 instead of INLINEFORM19 , obtaining five more features. Similarly, we compute INLINEFORM0 and INLINEFORM1 for each np anchor INLINEFORM2 or INLINEFORM3 (left, or right element of an np anchor pair) and template INLINEFORM4 (ignoring the INLINEFORM5 and INLINEFORM6 ) that led to a particular sentence plan INLINEFORM7 , and we use their maximum, minimum, average, total, and standard deviation as ten additional features of INLINEFORM8 . We also compute INLINEFORM9 and INLINEFORM10 for each template INLINEFORM11 and tokenized identifier INLINEFORM12 of a relation INLINEFORM13 (e.g., INLINEFORM14 :madeFrom becomes INLINEFORM15 “made from”) that led to the sentence plan INLINEFORM16 , obtaining ten more features.

It determines if a relation label is a verb or noun using hand-crafted rules and the pos tags of the label's synonyms in WordNet  . It then employs manually authored templates, corresponding to our sentence plans, to express the relation; e.g., the template “ INLINEFORM1 writes INLINEFORM2 ” is used for a triple involving the relation :write, since “write” is a verb, but “ INLINEFORM3 's author is INLINEFORM4 ” is used for the relation :author, since “author” is a noun. To express the INLINEFORM5 or INLINEFORM6 of a triple, sparql2nl tokenizes the label (or identifier) of the corresponding individual or class, pluralizing the resulting name if it refers to a class. Ratnaparkhi  aims to express a set of attribute-value pairs as a natural language phrase; e.g., INLINEFORM0 becomes “flights from Athens to New York on Wednesday”. A parallel training corpus containing sets of attribute-value pairs, the corresponding phrases, and their dependency trees is required. A maximum entropy model is trained on the corpus, roughly speaking to be able to estimate the probability of a dependency tree given a set of attribute-value pairs.

Ranking the candidate sentence plans	Each candidate sentence plan of a relation INLINEFORM0 is represented as a feature vector INLINEFORM1 , containing the 251 features discussed above. Each vector is given to the MaxEnt classifier to obtain a probability estimate INLINEFORM2 that it belongs in the positive class INLINEFORM3 , i.e., that the sentence plan is correct for INLINEFORM4 . The candidate sentence plans of each relation INLINEFORM5 are then ranked by decreasing estimated (by the classifier) INLINEFORM6 . We call sp our overall sentence plan generation method that uses the probability estimates of the classifier to rank the candidate sentence plans. In an alternative configuration of our sentence plan generation method, denoted sp*, the probability estimate INLINEFORM0 of each candidate sentence plan is multiplied by its coverage. To compute the coverage of a sentence plan for a relation INLINEFORM1 , we use the sentence plan to produce a sentence for each seed name pair of INLINEFORM2 (as when computing the grammaticality of a sentence plan in Section SECREF34 ). Subsequently, we use each sentence as a phrase query in a Web search engine.

For example, the template “ INLINEFORM4 is made from INLINEFORM5 ” becomes:   INLINEFORM0 is INLINEFORM1 made INLINEFORM2 from INLINEFORM3 INLINEFORM4  For every verb form (e.g., “is made”), noun, adjective, and preposition, we create a corresponding slot in the sentence plan. For verbs, nouns, and adjectives, the base form is used in the slot; each verb slot is also annotated with the voice and tense of the corresponding verb form in the template. If a negation expression (e.g., “not”, “aren't”) is used with the verb form in the template, the negation expression is not included as a separate slot in the sentence plan, but the polarity of the verb slot is marked as negative; otherwise the polarity is positive. We determine the genders and capitalizations of nouns (and proper names) as in Section SECREF22 . The INLINEFORM0 and INLINEFORM1 are also replaced by slots requiring referring expressions.

If the number of obtained templates is smaller than INLINEFORM3 , the templates (without their slots) are used as phrasal search queries to obtain new documents and new sentences (from the documents) that match the templates. For each new sentence (e.g., “gasoline is made from petroleum”), boot finds the np s (“gasoline”, “petroleum”) immediately before and after the search phrase, and treats them as a new seed name pair, discarding pairs that occur in only one retrieved document. The new pairs are then used to obtain new templates, again discarding templates occurring in only one document. This process is repeated until we have at least INLINEFORM4 templates for INLINEFORM5 , or until no new templates can be produced. In our experiments, we set INLINEFORM6 to obtain approximately the same number of templates as with sp and sp*.

When using the classifier to score a sentence plan INLINEFORM18 for a relation INLINEFORM19 , we compute the INLINEFORM20 of all the np anchor pairs of INLINEFORM21 , and we use the maximum, minimum, average, total, and standard deviation of these scores as five additional features of INLINEFORM22 . Similarly to Eq. EQREF38 , the productivity of an NP anchor INLINEFORM0 (considered on its own) that occurs as the first element of at least one np anchor pair INLINEFORM1 of INLINEFORM2 is defined as follows: DISPLAYFORM0  where: INLINEFORM0 is the number of np anchor pairs INLINEFORM1 of INLINEFORM2 that have INLINEFORM3 as their first element; INLINEFORM4 is the number of times INLINEFORM5 (as part of an np anchor pair INLINEFORM6 of INLINEFORM7 ) matched any element of any seed name pair of INLINEFORM8 and contributed to the extraction of any template of INLINEFORM9 ; and INLINEFORM10 , INLINEFORM11 are as in Eq. EQREF39 .

Again, we calculate the INLINEFORM10 of all the templates INLINEFORM11 of INLINEFORM12 , and we use the maximum, minimum, average, total, and standard deviation of these productivity scores as five more features of INLINEFORM13 . The productivity of a parsed sentence INLINEFORM0 (e.g., “obviously Semillon is made from Semillon grapes in California”) of a relation INLINEFORM1 is defined as follows: DISPLAYFORM0  where: INLINEFORM0 is the number of times any template of INLINEFORM1 was extracted from the particular parsed sentence INLINEFORM2 ; INLINEFORM3 is the total number of parsed sentences of INLINEFORM4 ; and INLINEFORM5 is the INLINEFORM6 -th parsed sentence of INLINEFORM7 . The intuition is that sentences that produce more templates for INLINEFORM8 are more indicative of INLINEFORM9 . Again, we calculate the INLINEFORM10 of all the parsed sentences INLINEFORM11 of INLINEFORM12 , and we use the maximum, minimum, average, total, and standard deviation of these productivity scores as features of INLINEFORM13 .

[ ] INLINEFORM0 [famous] INLINEFORM1 [wine] INLINEFORM2  If INLINEFORM0 :wine32, instanceOf, :TraditionalWinePiemonte INLINEFORM1 and INLINEFORM2 :wine32, instanceOf, :FamousWine INLINEFORM3 were both to be expressed, Naturalowl would then produce the single, aggregated sentence “This is a famous traditional wine from the Piemonte region”, instead of two separate sentences “This is a traditional wine from the Piemonte region” and “This is a famous wine”. The annotations of the slots, which indicate for example which words are adjectives and head nouns, are used by the sentence aggregation component of Naturalowl to appropriately combine the two sentences. The referring expression generation component also uses the slot annotations to identify the gender of the head noun, when a pronoun has to be generated (e.g., “it” when the head noun is neuter). We can now define more precisely nl names. A nl name is a sequence of one or more slots.

At this stage, the np “the Red Wines” will have become: [] INLINEFORM0 [red] INLINEFORM1 [wine] INLINEFORM2  We then consider the main article (or, more generally, determiner) of the nl name, i.e., the article that agrees with the head (e.g., “a” in “a traditional wine from the Piemonte Region”). Although the nl name may already include a main article, it is not necessarily an appropriate one. For example, it would be inappropriate to use a definite article in “The red wine is a kind of wine with red color”, when describing the class of red wines. We modify the nl name to use an indefinite article if the nl name refers to a class, and a definite article if it refers to an individual (e.g., “the South Australia region”). The article is omitted if the head is an adjective (e.g., “strong”), or in plural (e.g., “Semillon grapes”), or if the entire nl name (excluding the article, if present) is a proper name (e.g., “South Australia”) or a mass noun phrase without article (e.g., “gold”).

The genders of these nouns are determined using the on-line dictionary, when possible, or defaults otherwise (e.g., the default for person nouns is a `person' pseudo-gender, which leads to “he/she” or “they” when generating a pronoun). Nouns not referring to persons and locations are marked as neuter. Since the nps are extracted from Web pages, there is a risk of wrong capitalization (e.g., “the RED wine”). For each word of the nl name, we pick the capitalization that is most frequent in the retrieved texts of the individual or class we generate the nl name for. Hence, the np “the Red Wines” becomes: [] INLINEFORM0 [red] INLINEFORM1 [wine] INLINEFORM2  which requires a definite article, followed by the adjective “red”, and the neuter “wine” in singular. A dependency parser is then used to identify the head of each nl name (Section SECREF3 ) and to obtain agreement information. Adjectives are required to agree with the nouns they modify, and the same applies to articles and nouns.

The productivity of a seed name INLINEFORM0 (considered on its own) that occurs as the first element of at least one seed name pair INLINEFORM1 of a relation INLINEFORM2 is defined as follows: DISPLAYFORM0  where: INLINEFORM0 is the number of seed name pairs INLINEFORM1 of INLINEFORM2 that have INLINEFORM3 as their first element; INLINEFORM4 is the number of times INLINEFORM5 (as part of a seed name pair INLINEFORM6 of INLINEFORM7 ) matched any element of any np anchor pair of the parsed sentences of INLINEFORM8 and contributed to the extraction of any template of INLINEFORM9 ; INLINEFORM10 and INLINEFORM11 are as in Eq. EQREF36 . Again, when using the classifier to score a sentence plan INLINEFORM12 for a relation INLINEFORM13 , we calculate the INLINEFORM14 values of all the (distinct) seed names INLINEFORM15 that occur as first elements in the seed name pairs of INLINEFORM16 , and we use the maximum, minimum, average, total, and standard deviation of these productivity scores as five more features of INLINEFORM17 .

Before inserting or modifying the main article, we also remove any demonstratives (e.g., “this statue”) or other non-article determiners (e.g., “some”, “all”) from the beginning of the nl name . In our example, the nl name is to be used to refer to the class :RedWine, so the final nl name is the following, which would lead to sentences like “A red wine is a kind of wine with red color”. [] INLINEFORM0 [red] INLINEFORM1 [wine] INLINEFORM2  Recall that Naturalowl can automatically adjust nl names when generating texts (Section SECREF3 ). For example, in a comparison like “Unlike the previous red wines that you have seen, this one is from France”, it would use a definite article and it would turn the head noun of the nl name to plural, also adding the adjective “previous”. The resulting nl names are finally ranked by the scores of the np s they were obtained from (Section UID16 ).

At the end of the bootstrapping, instead of using a MaxEnt classifier (Sections SECREF34 and SECREF66 ), boot scores the templates of each relation INLINEFORM0 using the following confidence function: DISPLAYFORM0  where INLINEFORM0 is a template being scored, INLINEFORM1 is the number of (distinct) np anchor pairs of INLINEFORM2 extracted by INLINEFORM3 (from the documents retrieved by all the seed name pairs of INLINEFORM4 ), INLINEFORM5 is the number of (distinct) np anchor pairs of INLINEFORM6 not extracted by INLINEFORM7 (from the same documents), and INLINEFORM8 is the number of sentences (of the same documents) that match INLINEFORM9 . The five templates with the highest INLINEFORM10 (in a semi-automatic scenario) or the single template with the highest INLINEFORM11 (in a fully automatic scenario) are then converted to sentence plans, as in Section SECREF33 . Functions like Eq. EQREF102 can also be applied within each iteration of the bootstrapping, not only at the end of the entire bootstrapping, to keep only the best new templates of each iteration.

Sentence plans that refer to INLINEFORM3 before INLINEFORM4 are usually simpler and better. True if a referring expression of INLINEFORM0 is the subject of a verb of INLINEFORM1 ; otherwise false. This information is obtained from the parsed sentences that led to INLINEFORM2 . We use the most frequent dependency tree, if INLINEFORM3 was derived from many sentences. Sentence plans with no subjects are often ill-formed. True if a referring expression of INLINEFORM0 is the object of a verb of INLINEFORM1 ; otherwise false. Again, we consider the parsed sentences that led to INLINEFORM2 using the most frequent dependency tree. Sentence plans with no objects are often ill-formed, because most relations are expressed by transitive verbs. True if all the sentences INLINEFORM0 was derived from were well-formed, according to the parser. True if INLINEFORM0 required a repair at the end of the sentence plan generation (Section SECREF33 ); otherwise false. Repaired sentence plans can be poor.

It is caused by the Bundibugyo ebolavirus, a Côte d’Ivoire ebolavirus, the Sudan ebolavirus and the Zaire ebolavirus. verbalizer: Ebola hemorrhagic fever is a kind of viral infectious disease. It has symptom muscle aches, sore throat, fever, weakness, stomach pain, red eyes, joint pain, vomiting, headache, rash, internal and external bleeding, hiccups and diarrhea. It transmitted by infected medical equipment, contact with the body fluids of an infected animal, and contaminated fomites and it has material basis in Bundibugyo ebolavirus, Cote d'Ivoire ebolavirus, Sudan ebolavirus and Zaire ebolavirus.  Again, some nl names and sentence plans of semi-auto and auto can be easily improved using the Protégé plug-in of Naturalowl. For example, the sentence plan that reports the historical period of the exhibit in the second semi-auto example above can be easily modified to use the simple past tense (“was produced” instead of “is produced”). Nevertheless, we made no such improvements.

The productivity of the INLINEFORM0 -th seed name pair INLINEFORM1 (e.g., INLINEFORM2 INLINEFORM3 “Semillon”, INLINEFORM4 “Semillon grape” INLINEFORM5 ) of a relation INLINEFORM6 (e.g., INLINEFORM7 :madeFrom) is defined as follows: DISPLAYFORM0  where: INLINEFORM0 is the number of times INLINEFORM1 matched any np anchor pair of the parsed sentences of INLINEFORM2 , counting only matches that contributed to the extraction (Section SECREF29 ) of any template of INLINEFORM3 ; INLINEFORM4 is the total number of seed name pairs of INLINEFORM5 ; and INLINEFORM6 is the INLINEFORM7 -th seed name pair of INLINEFORM8 . The intuition behind INLINEFORM16 is that seed name pairs that match np anchor pairs of many sentences of INLINEFORM17 are more likely to be indicative of INLINEFORM18 . When using the MaxEnt classifier to score a sentence plan INLINEFORM19 for a relation INLINEFORM20 , we compute the INLINEFORM21 of all the seed name pairs INLINEFORM22 of INLINEFORM23 , and we use the maximum, minimum, average, total, and standard deviation of these productivity scores as five features of INLINEFORM24 .

If a seed name results from a class, parent-class, or an equivalent class of the original INLINEFORM5 or INLINEFORM6 , we consider it a secondary seed name.  INLINEFORM0 “St. Emilion”, “Cabernet Sauvignon grape” INLINEFORM1 , INLINEFORM2 “Semillon”, “Semillon grape” INLINEFORM3   INLINEFORM0 “wine”, “Cabernet Sauvignon grape” INLINEFORM1 , INLINEFORM2 “St. Emilion”, “grape” INLINEFORM3 , INLINEFORM4 “wine”, “grape” INLINEFORM5  We then retrieve Web pages using the seed name pairs (of the relation that we want to generate a sentence plan for) as queries. For each seed name pair, we use the conjunction of its seed names (e.g., “St. Emilion” AND “Cabernet Sauvignon grape”) as a Boolean query. We convert all the retrieved pages (of all the seed name pairs) to plain text documents and parse every sentence of the retrieved documents, if at least one stemmed word from each seed name of a particular pair is the same as a stemmed word of the sentence.

For example, for the seed name pair INLINEFORM6 INLINEFORM7 “Semillon”, INLINEFORM8 “Semillon grape” INLINEFORM9 of the relation INLINEFORM10 :madeFrom and the following candidate sentence plan: [ INLINEFORM0 ] INLINEFORM1 [make] INLINEFORM2 [from] INLINEFORM3 [ INLINEFORM4 ] INLINEFORM5  the sentence “Semillon is made from Semillon grapes” is generated. We do not generate referring expressions, even when required by the sentence plan (e.g., [ INLINEFORM0 ] INLINEFORM1 ); we use the seed names instead. We obtain confidence scores for these sentences from the parser, and we normalize these scores dividing by each sentence's length. The maximum, minimum, average, and standard deviation of these scores are used as features of INLINEFORM2 . Some additional features for a candidate sentence plan INLINEFORM0 follow: True if INLINEFORM0 contains a present participle without an auxiliary; otherwise false. True if INLINEFORM0 has a main verb in active voice; otherwise false. True if INLINEFORM0 contains a referring expression for INLINEFORM1 before a referring expression for INLINEFORM2 ; otherwise false.

Intuitively, if INLINEFORM4 co-occur frequently when they produce templates of INLINEFORM5 , they are strongly connected and, hence, they are more indicative of INLINEFORM6 . Again, when using the classifier to score a sentence plan INLINEFORM7 for a relation INLINEFORM8 , we calculate INLINEFORM9 for all the seed name pairs INLINEFORM10 of INLINEFORM11 , and we use the maximum, minimum, average, total, and standard deviation of these pmi scores as five more features of INLINEFORM12 . We define similarly INLINEFORM13 , INLINEFORM14 , INLINEFORM15 , INLINEFORM16 , INLINEFORM17 , INLINEFORM18 , INLINEFORM19 , INLINEFORM20 , INLINEFORM21 , INLINEFORM22 , obtaining five features for INLINEFORM23 from each one. These features view seed names, np anchors, templates, and owl identifiers as sequences of tokens. For each seed name INLINEFORM0 and np anchor INLINEFORM1 that matched (as first elements of a seed name pair INLINEFORM2 and np anchor pair INLINEFORM3 ) to produce a particular sentence plan INLINEFORM4 , we calculate their cosine similarity INLINEFORM5 with INLINEFORM6 weights (defined as in Section SECREF29 ).

Turning the extracted noun phrases into natural language names	The extracted nps are not yet nl names, because they lack the linguistic annotations that Naturalowl requires (e.g., pos tags, agreement, number); they may also lack appropriate articles. To convert an np to an nl name, we first obtain the pos tags of its words from the parse tree of the sentence the np was extracted from. For example, the np “the Red Wine” becomes:  the INLINEFORM0 Red INLINEFORM1 Wine INLINEFORM2  For every noun, adjective, article, preposition, we create a corresponding slot in the nl name; all the other words of the np become slots containing the words as fixed strings (Section SECREF3 ). For nouns and adjectives, the base form is used in the slot (e.g., “wine” istead of “wines”), but slot annotations indicate the particular inflectional form that was used in the np; e.g., the nn pos tag shows that “wine” is singular. A named-entity recognizer (ner) and an on-line dictionary are employed to detect nouns that refer to persons and locations.

The intuition is that the more seed name pairs of INLINEFORM17 lead to the sentence plan INLINEFORM18 , the better INLINEFORM19 is. In a similar manner, we define INLINEFORM0 based on Eq. EQREF39 , and similarly for all the other productivity versions of Section UID35 . We obtain one feature for the candidate sentence plan INLINEFORM1 from each prominence variant, i.e., we do not compute any maximum, minimum, average, sum, standard deviation values, unlike the productivity versions, which lead to five features each. To estimate the extent to which two seed names INLINEFORM0 , INLINEFORM1 of a relation INLINEFORM2 co-occur when they match np anchors to produce templates of INLINEFORM3 , we use a Pointwise Mutual Information (pmi) score: DISPLAYFORM0  The second factor of the right-hand side of Eq. EQREF50 is the standard pmi definition, using productivity scores instead of probabilities. The first factor normalizes the pmi scores to INLINEFORM0 ( INLINEFORM1 if INLINEFORM2 never co-occur when producing templates of INLINEFORM3 , 0 if they are independent, 1 if they always co-occur).

Our sentence plan generation method contains a template extraction stage (Section SECREF29 ), which is similar to methods proposed to automatically obtain templates that extract instances of particular relations from texts. We discussed bootstrapping in Section UID100 . Xu et al.  adopt a similar bootstrapping approach with templates obtained from dependency trees. Bootstrappoing has also been used to obtain paraphrasing and textual entailment rules  ,  . The sentence plans we produce are not just templates (e.g., “ INLINEFORM0 bought INLINEFORM1 ”), but include additional annotations (e.g., pos tags, agreement, voice, tense, cases). Furthermore, they are not intended to capture all the alternative natural language expressions that convey a particular relation, unlike information extraction, paraphrase and textual entailment recognition; our goal is to obtain a single sentence plan per relation that leads to high quality texts. Bootstrapping approaches have also been used to obtain templates that extract named entities of a particular semantic class (e.g., person names) from texts  ,  . Methods of this kind aim to extract all the named entities of a particular class from a corpus.

The training error counts how many of the instances that were used to train the classifier were also correctly classified by the classifier. The test error counts how many of the test (unseen) instances (one in each repetition of the cross-validation) were correctly classified by the classifier (trained on the corresponding percentage of the training dataset). The error rates of Fig. FIGREF89 are averaged over the 655 repetitions of the cross-validation. The training error curve can be thought of as a lower bound of the test error curve, since a classifier typically performs better on the instances it has been trained on than on unseen instances. The two curves indicate that the classifier might perform slightly better with more training data, though the test error rate would remain above INLINEFORM1 . The relatively small distance of the right ends of the two curves indicates only mild overfitting when the entire training dataset is used. To assess the contribution of the 251 features (Section SECREF34 ), we ranked them by decreasing Information Gain (ig)  computed on the 655 instances. Table TABREF90 shows the maximum, minimum, and average ig scores of the features in each group (subsection) of Section SECREF34 .

Datatype properties, where INLINEFORM3 is a datatype value (e.g., integer, string, date), can in principle be handled using the same methods, but appropriate recognizers may be needed to obtain appropriate anchors, instead of np anchors. For example, a datatype property may map persons to dates of birth; then a recognizer of dates would be needed to extract (and possibly normalize) appropriate date anchors from Web pages, since a parser may not treat dates as np s.

Furthermore, the methods of Ratnaparkhi aim to produce a single sentence per set of attribute-value pairs, whereas we produce linguistic resources that are used to generate multi-sentence texts (e.g., our nl names and sentence plans include annotations used in sentence aggregation and referring expression generation). Angeli et al.  generate multi-sentence texts describing database records. Their methods also require a parallel training corpus consisting of manually authored texts and the database records (and particular record fields) expressed by each text. The generative model of Liang et al.  is applied to the training corpus to align the words of each text to the database records and fields it expresses. Templates are then extracted from the aligned texts, by replacing words aligned to record fields with variables. To generate a new text from a set of database records, the system generates a sequence of phrases. For each phrase, it first decides which records and fields to express, then which templates to generate the phrase with, and finally which template variables to replace by which record fields. These decisions are made either greedily or by sampling probability distributions learnt during training. This process is repeated until all the given record fields have been expressed.

Lu et al.  generate natural language sentences from tree-structured semantic representations  . Given a parallel training corpus of sentences and tree-structured semantic representations, hybrid trees are created by expanding the original semantic representation trees of the corpus with nodes standing for phrases of the corresponding sentences. To generate a new sentence from a tree-structured semantic representation, a set of candidate hybrid trees is initially produced based on predefined tree patterns and a crf model trained on the hybrid trees of the parallel corpus. A sentence is then obtained from the most probable candidate hybrid tree. In later work, Lu and Ng  extend their hybrid trees to support formal logic (typed lambda calculus) semantic representations. A synchronous context free grammar is obtained from the extended hybrid trees of the parallel corpus. The grammar is then used to map formal logic expressions to new sentences. We note that owl is based on description logic  and, hence, methods similar to those of Lu et al. could in principle be used to map owl statements to sentences, though the hybrid trees would have to be modified for description logic. A parallel training corpus of texts and description logic expressions (or corresponding owl statements) would again be needed, however, and only single sentences would be obtained.

the owner) of the message triple in a particular case. (ii) A verb in a particular polarity and form (e.g., tense), possibly to agree with another slot. (iii) A noun or adjective in a particular form, possibly to agree with another slot. (iv) A preposition, or (v) a fixed string. (vi) A referring expression for the INLINEFORM0 (a.k.a. the filler) of the message triple. More details about the nl names and sentence plans of Naturalowl and their roles in sentence aggregation, referring expressions etc. can be found elsewhere  . Both sentence plans and nl names were so far authored manually, using a Protégé plug-in (Fig. FIGREF9 ).

Similarly, if INLINEFORM0 :SouthAustraliaRegion and we have collected the following message triple, the tokenized identifier of INLINEFORM1 would be shortened from “South Australia Region” to “South Australia”. We use altTokName to denote the resulting shortened tokenized identifiers.  INLINEFORM0 :SouthAustraliaRegion, :isA, :Region INLINEFORM1  tokName(:SouthAustraliaRegion) INLINEFORM0 “South Australia Region”, tokName(:Region) INLINEFORM1 “Region” altTokName(:SouthAustraliaRegion) INLINEFORM0 “South Australia”  Also, if INLINEFORM0 :exhibit23 and we have collected the following triple, altTokName(:exhibit23) would end up containing only numbers (“23”). Hence, :exhibit23 is marked as anonymous.

Applying a Maximum Entropy classifier to the candidate sentence plans	The retained candidate sentence plans are then scored using a Maximum Entropy (MaxEnt) classifier. The classifier views each candidate sentence plan INLINEFORM0 for a relation INLINEFORM1 as a vector of 251 features, and attempts to estimate the probability that INLINEFORM2 is a good sentence plan (positive class) for INLINEFORM3 or not (negative class). The 251 features provide information about INLINEFORM4 itself, but also about the templates, seed name pairs, and np anchor pairs of INLINEFORM5 , meaning the templates that INLINEFORM6 was obtained from, and the seed name pairs and np anchor pairs (Fig. FIGREF32 ) that matched to produce the templates of INLINEFORM7 .

The number of slots of INLINEFORM0 , the number of slots before the slot for INLINEFORM1 , the number of slots after the slot for INLINEFORM2 , the number of slots between the slots for INLINEFORM3 and INLINEFORM4 (4 features). The maximum, minimum, average, total, standard deviation of the ranks of the Web pages (returned by the search engine, Section SECREF29 ) that contained the sentences INLINEFORM0 was obtained from. Sentences from higher-ranked pages are usually more relevant to the seed name pairs we use as queries. Hence, sentence plans obtained from higher-ranked Web pages are usually better. The number of Web pages that contained the sentences from which INLINEFORM0 was obtained. Uncommon sentences often lead to poor sentence plans.

Finally, we compute INLINEFORM17 , INLINEFORM18 , and INLINEFORM19 for all the INLINEFORM20 and INLINEFORM21 np anchors (first or second elements in their pairs) of INLINEFORM22 , obtaining fifteen more features of INLINEFORM23 . Although they may look strange, in effect INLINEFORM24 and INLINEFORM25 examine how strongly connected the words inside each np anchor ( INLINEFORM26 or INLINEFORM27 ) are. Another group of features try to estimate the grammaticality of a candidate sentence plan INLINEFORM0 . Let us assume that INLINEFORM1 is for relation INLINEFORM2 . For every seed name pair of INLINEFORM3 (not only seed name pairs that led to INLINEFORM4 ), we generate a sentence using INLINEFORM5 ; we ignore only seed name pairs that produced no sentence plans at all, which are assumed to be poor.

Similarly, given INLINEFORM2 :Wine, :madeFrom, :Grape INLINEFORM3 , the sentence plan above would lead to sentences like “Wines are made from grapes” or “They are made from grapes”, assuming again appropriate nl names. As another example, the following sentence plan can be used with the relations :hasColor and :hasFlavor. [ INLINEFORM0 ] INLINEFORM1 [be] INLINEFORM2 [ INLINEFORM3 ] INLINEFORM4  Given the message triples INLINEFORM0 :StEmilion, :hasColor, :red INLINEFORM1 and INLINEFORM2 :StEmilion, :hasFlavor, :strong INLINEFORM3 , it would produce the sentences “St. Emilion is red” and “St. Emilion is strong”, respectively. The first sentence plan above, for :madeFrom, has four slots. The first slot is to be filled in with an automatically generated referring expression (e.g., pronoun or name) for INLINEFORM0 , in nominative case.

The verb of the second slot is to be realized in passive voice, present tense, and positive polarity (as opposed to expressing negation) and should agree (in number and person) with the referring expression of the first slot ( INLINEFORM1 ). The third slot is filled in with the preposition `from', and the fourth slot with an automatically generated referring expression for INLINEFORM2 , in accusative case. Naturalowl has built-in sentence plans for domain-independent relations (e.g., isA, instanceOf). For example, INLINEFORM0 :StEmilion, isA, :Bordeaux INLINEFORM1 is expressed as “St. Emilion is a kind of Bordeaux” using the following built-in sentence plan; the last slot requires the nl name of INLINEFORM2 without article. [ INLINEFORM0 ] INLINEFORM1 [be] INLINEFORM2 [“a kind of”] INLINEFORM3 [ INLINEFORM4 ] INLINEFORM5  Notice that the sentence plans of Naturalowl are not simply slotted string templates (e.g., “ INLINEFORM0 is made from INLINEFORM1 ”).

Again, we calculate the INLINEFORM12 values of all the (distinct) np anchors INLINEFORM13 that occur as first elements in the np anchor pairs of INLINEFORM14 , and we use the maximum, minimum, average, total, and standard deviation of these productivity scores as five more features of INLINEFORM15 . We define similarly INLINEFORM16 for a seed name INLINEFORM17 that occurs as the second element in any np anchor pair INLINEFORM18 of INLINEFORM19 , obtaining five more features for INLINEFORM20 . The productivity of a template INLINEFORM0 (e.g., “ INLINEFORM1 is made from INLINEFORM2 ”) of a relation INLINEFORM3 is defined as follows: DISPLAYFORM0  where: INLINEFORM0 is the number of times the particular template INLINEFORM1 was extracted from any of the parsed sentences of INLINEFORM2 ; INLINEFORM3 is the total number of templates of INLINEFORM4 ; and INLINEFORM5 is the INLINEFORM6 -th template of INLINEFORM7 . The intuition is that templates that are produced more often for INLINEFORM8 are more indicative of INLINEFORM9 .

The joint productivity of a seed name pair INLINEFORM0 and a template INLINEFORM1 of a relation INLINEFORM2 is: DISPLAYFORM0  where: INLINEFORM0 is the number of times the particular seed name pair INLINEFORM1 matched any np anchor pair of the parsed sentences of INLINEFORM2 and contributed to the extraction of the particular template INLINEFORM3 ; and INLINEFORM4 are again the total numbers of seed name pairs and templates, respectively, of INLINEFORM5 . Again, when scoring a sentence plan INLINEFORM6 for a relation INLINEFORM7 , we calculate the INLINEFORM8 of all the combinations of seed name pairs INLINEFORM9 and templates INLINEFORM10 that led to INLINEFORM11 , and we use the maximum, minimum, average, total, and standard deviation of these scores as features of INLINEFORM12 . We define very similarly INLINEFORM13 , INLINEFORM14 , INLINEFORM15 , INLINEFORM16 , INLINEFORM17 , INLINEFORM18 INLINEFORM19 , INLINEFORM20 , INLINEFORM21 , obtaining five additional features of INLINEFORM22 from each one.

Our method to extract natural language names from the Web	Given a target class or individual INLINEFORM0 that we want to produce an nl name for, we first extract from the Web noun phrases that are similar to the owl identifier of INLINEFORM1 . The noun phrases are ranked by aligning their words to the tokens of the identifier. The top-ranked noun phrases are then enhanced with linguistic annotations (e.g., pos tags, agreement, number), missing articles etc., turning them into nl names. We aim to identify the best few (up to 5) candidate nl names for INLINEFORM2 . In a fully automatic scenario, the candidate nl name that the method considers best for INLINEFORM3 is then used. In a semi-automatic scenario, the few top (according to the method) nl names of INLINEFORM4 are shown to a human author, who picks the best one; this is much easier than manually authoring nl names.

The “1-in-1” score is the ratio of individuals and classes for which the top returned nl name was correct. The “1-in-3” score is the ratio of individuals and classes for which there was at least one correct nl name among the top three, and similarly for “1-in-5”. The “1-in-1” score corresponds to a fully automatic scenario, where the top nl name is used for each individual or class, without human intervention. By contrast, the “1-in-3” and “1-in-5” scores correspond to a semi-automatic scenario, where a human inspects the top three or five, respectively, nl names per individual or class, looking for a correct one to select. The mean reciprocal rank (mrr) is the mean (over all the individuals and classes we asked our method to produce nl names for) of the reciprocal rank INLINEFORM0 , where INLINEFORM1 is the rank (1 to 5) of the top-most correct nl name returned for the INLINEFORM2 -th individual or class; if no correct nl name exists among the top five, then INLINEFORM3 .

For example, the template “ INLINEFORM2 is made from INLINEFORM3 ” becomes: [ INLINEFORM0 ] INLINEFORM1 [make] INLINEFORM2 [from] INLINEFORM3 [ INLINEFORM4 ] INLINEFORM5  Agreement and case information is obtained using a dependency parser. The parser is applied to the sentences the templates were extracted from, keeping the most frequent parse per template. Referring expressions obtained from np anchors that were verb subjects are marked with nominative case, and they are required to agree with their verbs. Referring expressions corresponding to verb objects or preposition complements are marked with accusative case (e.g., “from him”). Referring expressions corresponding to np anchors with head nouns in possessive form (e.g., “Piemonte's”) are marked with possessive case. In our example, we obtain: [ INLINEFORM0 ] INLINEFORM1 [make] INLINEFORM2 [from] INLINEFORM3 [ INLINEFORM4 ] INLINEFORM5  Any remaining words of the template that have not been replaced by slots (e.g., “obviously” in “obviously INLINEFORM0 is made from INLINEFORM1 ”) are turned into fixed string slots.

The natural language names of NaturalOWL	In Naturalowl, a natural language (nl) name is a sequence of slots. The contents of the slots are concatenated to produce a noun phrase to be used as the name of a class or individual. Each slot is accompanied by annotations specifying how to fill it in; the annotations may also provide linguistic information about the contents of the slot. For example, we may specify that the English nl name of the class :TraditionalWinePiemonte is the following. [ ] INLINEFORM0 [traditional] INLINEFORM1 [wine] INLINEFORM2 [from] INLINEFORM3 [ ] INLINEFORM4 [Piemonte] INLINEFORM5 [region] INLINEFORM6  The first slot is to be filled in with an indefinite article, whose number should agree with the third slot. The second slot is to be filled in with the adjective `traditional'. The third slot with the neuter noun `wine', which will also be the head (central) noun of the noun phrase, in singular number, and similarly for the other slots. Naturalowl makes no distinctions between common and proper nouns, but it can be instructed to capitalize particular nouns (e.g., `Piemonte').

INLINEFORM0 :WhiteBordeaux, isA, :Bordeaux INLINEFORM1  Naturalowl provides mechanisms to manually assign interest scores to message triples  . Assigning a zero interest score to a triple instructs Naturalowl to avoid expressing it. Manually assigning interest scores, however, can be tedious. Hence, we aimed to automatically assign zero scores to triples like the ones above, which report relations that are obvious from the nl names. To identify triples of this kind, we follow a procedure similar to the one we use to identify individuals or classes that should be anonymous (Section UID12 ). For each INLINEFORM0 triple that involves the individual or class INLINEFORM1 being described, we examine the nl names of INLINEFORM2 and INLINEFORM3 . If all the (lemmatized) words of the phrase produced by the nl name of INLINEFORM4 (e.g., “a Bordeaux”), excluding articles, appear in the phrase of the nl name of INLINEFORM5 (e.g., “a white Bordeaux”), we assign a zero interest score to INLINEFORM6 . INLINEFORM7

Each slot is accompanied by annotations requiring it to be filled in with exactly one of the following: (i) An article, definite or indefinite, possibly to agree with another slot filled in by a noun. (ii) A noun flagged as the head. The number of the head noun must also be specified. (iii) An adjective flagged as the head. For example, the nl name name of the individual :red may consist of a single slot, to be filled in with the adjective `red'; in this case, the adjective is the head of the nl name. The number and gender of the head adjective must also be specified. (iv) Any other noun or adjective, (v) a preposition, or (vi) any fixed string. Exactly one head (noun or adjective) must be specified per nl name. For nouns and adjectives, the nl name may require a particular inflectional form to be used (e.g., in a particular number, case, or gender), or it may require an inflectional form that agrees with another noun or adjective slot. When providing nl names, an individual or class can also be declared to be anonymous, indicating that Naturalowl should avoid referring to it by name.

In later work, Konstas and Lapata  find the most probable derivation in the hypergraph by forest reranking, using features that include the decoding probability of the derivation according to their previous work, the frequency of rewrite rules in the derivation, as well as lexical (e.g., word INLINEFORM2 -grams) and structural features (e.g., INLINEFORM3 -grams of record fields). The weights of the features are estimated with a structured perceptron  on the training corpus. Apart from simple verbalizers, all the other related methods discussed above require a parallel training corpus of texts (or sentences, or phrases) and their semantic representations, unlike our work. A further difference from our work is that all the previous methods assume that the English names of the various entities (or individuals and classes) are already available in the semantic representations of the texts to be generated, or that they can be directly obtained from the identifiers of the entities in the semantic representations. By contrast, we also proposed methods to produce appropriate nl names for individuals and classes, and we showed experimentally (Section UID77 ) that without nl names the perceived quality of the generated texts is significantly lower.

Inferring interest scores from natural language names	The reader may have already noticed that the sentence “A red wine is a kind of wine with red color” that we used above sounds redundant. Some message triples lead to sentences that sound redundant, because they report relations that are obvious (to humans) from the nl names of the individuals or classes. In our example, the sentence reports the following two message triples.  INLINEFORM0 :RedWine, isA, :Wine INLINEFORM1 , INLINEFORM2 :RedWine, :hasColor, :Red INLINEFORM3   Expressed separately, the two triples would lead to the sentences “A red wine is a kind of wine” and “A red wine has red color”, but Naturalowl aggregates them into a single sentence. The “red color” derives from an INLINEFORM0 of :Red obtained by considering the parent class :Color of :Red (Section UID14 ). It is obvious that a red wine is a wine with red color and, hence, the two triples above should not be expressed. Similarly, the following triple leads to the sentence “A white Bordeaux wine is a kind of Bordeaux”, which again seems redundant.

A candidate sentence plan was considered correct if and only if: (i) it would produce morphologically, syntactically, and semantically correct sentences; and (ii) the annotations of its slots (e.g., pos tags, voice, tense, agreement) were all correct. To compensate for class imbalance in the training set (16% positive vs. 84% negative candidate sentence plans), we replicated all the positive training instances (over-sampling) to obtain an equal number of positive and negative training instances. Figure FIGREF89 shows the error rate of the classifier on (i) unseen instances (test error) and (ii) on the instances it has been trained on (training error). To obtain the curves of Fig. FIGREF89 , we performed a leave-one-out cross validation on the 655 instances (candidate sentence plans) we had constructed, i.e., we repeated the experiment 655 times, each time using a different instance as the only test instance and the other 654 instances as the training dataset. Within each repetition of the cross-validation, we iteratively trained the classifier on 10%, 20%, ..., 100% of the training dataset (654 instances, with over-sampling applied to them).

On average, the pmi and token-based features are the best predictors, whereas the prominence features are the worst. The maximum scores, however, show that there is at least one good feature in each group, with the prominence features being again the worst in this respect. The minimum scores indicate that there is also at least one weak feature in every group, with the exception of the pmi features, where the minimum ig score (0.21) was much higher. Figure FIGREF91 shows the ig scores of all the features in each group, in ascending order. There are clearly many good features in every group, with the prominence features again being the weakest group overall. We then iteratively trained the classifier on the entire training dataset ( INLINEFORM0 ), removing at each iteration the feature (among the remaining ones) with the smallest ig score. Figure FIGREF92 shows the resulting test and training error rate curves, again obtained using a leave-one-out cross-validation. As more features are removed, the distance between the training and test error decreases, because of reduced overfitting. When very few features are left (far right), the performance of the classifier on unseen instances becomes unstable.

A language model is also employed to ensure that the transitions between phrases sound natural. As with the work of Ratnaparkhi, the methods of Angeli et al. could in principle be applied to express message triples describing an individual or class, but again a parallel training corpus containing texts and the database records and fields expressed by each text would be needed. Wong and Mooney  ,  employ Statistical Machine Translation (smt) methods to automatically obtain formal semantic representations from natural language sentences. They automatically construct a synchronous context-free grammar, by applying a statistical word alignment model to a parallel training corpus of sentences and their semantic representations. The grammar generates both natural language sentences and their semantic representations. Given a new sentence, the grammar produces candidate semantic representations, and a maximum-entropy model estimates the probability of each canidate representation. Chen and Mooney  use the same methods in the reverse direction, to convert formal semantic representations to single sentences. In principle, similar smt methods could be employed to generate sentences from message triples. However, a parallel corpus of texts and message triples would again be needed. Furthermore, smt methods produce a single sentence at a time, whereas our work concerns multi-sentence texts.

The sentence plans of NaturalOWL	In Naturalowl, a sentence plan for a relation INLINEFORM0 specifies how to construct a sentence to express any message triple of the form INLINEFORM1 . Like nl names, sentence plans are sequences of slots with annotations specifying how to fill the slots in. The contents of the slots are concatenated to produce the sentence. For example, the following is a sentence plan for the relation :madeFrom. [ INLINEFORM0 ] INLINEFORM1 [make] INLINEFORM2 [from] INLINEFORM3 [ INLINEFORM4 ] INLINEFORM5  Given the message triple INLINEFORM0 :StEmilion, :madeFrom, :cabernetSauvignonGrape INLINEFORM1 , the sentence plan would lead to sentences like “St. Emilion is made from Cabernet Sauvignon grapes”, or “It is made from Cabernet Sauvignon grapes”, assuming that appropriate nl names have been provided for :StEmilion and :cabernetSauvignonGrape.

In the case of the message triple INLINEFORM0 :wine32, instanceOf, :TraditionalWinePiemonte INLINEFORM1 , the nl name above would allow a sentence like “This is a traditional wine from the Piemonte region” to be produced. The slot annotations allow Naturalowl to automatically adjust the nl names. For example, the system also generates comparisons to previously encountered individuals or classes, as in “Unlike the previous products that you have seen, which were all traditional wines from the Piemonte region, this is a French wine”. In this particular example, the head noun (`wine') had to be turned into plural. Due to number agreement, its article also had to be turned into plural; in English, the plural indefinite article is void, hence the article of the head noun was omitted. As a further example, we may specify that the nl name of the class FamousWine is the following.

Their linguistic annotations (e.g., pos tags, agreement, voice, tense, cases) along with the annotations of the nl names allow Naturalowl to produce more natural sentences (e.g., turn the verb into plural when the subject is also plural), produce appropriate referring expressions (e.g., pronouns in the correct cases and genders), and aggregate shorter sentences into longer ones. For example, the linguistic annotations of the nl names and sentence plans allow Naturalowl to produce the aggregated sentence “St. Emilion is a kind of red Bordeaux made from Cabernet Sauvignon grapes” from the triples INLINEFORM2 :StEmilion, isA, :Bordeaux INLINEFORM3 , INLINEFORM4 :StEmilion, :hasColor, :red INLINEFORM5 , INLINEFORM6 :StEmilion, :madeFrom, :cabernetSauvignonGrape INLINEFORM7 , instead of three separate sentences. We can now define more precisely sentence plans. A sentence plan is a sequence of slots. Each slot is accompanied by annotations requiring it to be filled in with exactly one of the following: (i) A referring expression for the INLINEFORM0 (a.k.a.

In this paper, we consider only sentence plans for message triples without relation modifiers, because Naturalowl already automatically constructs sentence plans for triples with relation modifiers from sentence plans for triples without them. Having produced the message triples, Naturalowl consults a user model to select the most interesting ones that have not been expressed already, and orders the selected triples according to manually authored text plans. Later processing stages convert each message triple to an abstract sentence representation, aggregate sentences to produce longer ones, and produce appropriate referring expressions (e.g., pronouns). The latter three stages require a sentence plan for each relation ( INLINEFORM0 ), while the last stage also requires natural language names for each individual or class ( INLINEFORM1 or INLINEFORM2 ).

From the three triples above, we obtain: tokName(:KalinCellarsSemillon) INLINEFORM0 “Kalin Cellars Semillon” , tokName(:Semillon) INLINEFORM1 “Semillon” tokName(:KalinCellars) INLINEFORM0 “Kalin Cellars”, tokName(:Strong) INLINEFORM1 “Strong”  Subsequently, we attempt to shorten INLINEFORM0 , i.e., the tokenized identifier of the individual or class we wish to produce an nl name for, by removing any part (token sequence) of INLINEFORM1 that is identical to the tokenized identifier of the INLINEFORM2 of any triple INLINEFORM3 that we collected for INLINEFORM4 . If the shortened tokenized identifier of INLINEFORM5 is the empty string or contains only numbers, INLINEFORM6 is marked as anonymous (Section SECREF3 ). In our example, where INLINEFORM7 :KalinCellarsSemillon, the tokenized identifier of INLINEFORM8 is initially INLINEFORM9 “Kalin Cellars Semillon”.

We remove the part “Semillon”, because of the triple INLINEFORM10 :KalinCellarsSemillon, :isA, :Semillon INLINEFORM11 and the fact that tokName(:Semillon) INLINEFORM12 “Semillon”, as illustrated below. We also remove the remaining part “Kalin Cellars”, because of INLINEFORM13 :KalinCellarsSemillon, :hasMaker, :KalinCellars INLINEFORM14 and the fact that tokName(:KalinCellars) INLINEFORM15 “Kalin Cellars”. Hence, :KalinCellarsSemillon is marked as anonymous.           Anonymizing :KalinCellarsSemillon causes Naturalowl to produce texts like (a) below when asked to describe :KalinCellarsSemillon, rather than (b), which repeats “Semillon” and “Kalin Cellars”:  (a) This is a strong, dry Semillon. It has a full body. It is made by Kalin Cellars. (b) Kalin Cellars Semillon is a strong, dry Semillon. It has a full body. It is made by Kalin Cellars.

The coverage of the sentence plan is the number of seed name pairs for which the search engine retrieved at least one document containing the search sentence (verbatim), divided by the total number of seed name pairs of INLINEFORM3 . Coverage helps avoid sentence plans that produce very uncommon sentences. Computing the coverage of every candidate sentence plan is time consuming, however, because of the Web searches; this is also why we do not include coverage in the features of the classifier. Hence, we first rank the candidate sentence plans of each relation INLINEFORM4 by decreasing INLINEFORM5 , and we then re-rank only the top ten of them (per INLINEFORM6 ) after multiplying the INLINEFORM7 of each one by its coverage. In both sp and sp*, in a semi-automatic scenario we return to a human inspector the top five candidate sentence plans per relation. In a fully automatic scenario, we return only the top one.

INLINEFORM0 :exhibit23, :isA, :Exhibit INLINEFORM1  tokName(:exhibit23) INLINEFORM0 “exhibit 23”, tokName(:Exhibit) INLINEFORM1 “exhibit” We then collect the tokenized identifiers of all the ancestor classes of INLINEFORM0 , also taking into account equivalent classes; for example, if INLINEFORM1 has an equivalent class INLINEFORM2 , we also collect the tokenized identifiers of the ancestor classes of INLINEFORM3 . For INLINEFORM4 :KalinCellarsSemillon, we collect the following tokenized identifiers, because :Semillon, :SemillonOrSauvignonBlanc, and :Wine are ancestors of INLINEFORM5 .

tokName(:Semillon) INLINEFORM0 “Semillon”, tokName(:SemillonOrSauvignonBlanc) INLINEFORM1 “Semillon Or Sauvignon Blanc”, tokName(:Wine) INLINEFORM2 “Wine” If INLINEFORM0 does not contain any of the collected tokenized identifiers of the ancestor classes of INLINEFORM1 , we create additional alternative tokenized identifiers for INLINEFORM2 , also denoted INLINEFORM3 , by appending to INLINEFORM4 the collected tokenized identifiers of the ancestor classes of INLINEFORM5 .

It is transmitted by infected medical equipment, contact with the body fluids of an infected animal, and contaminated fomites and it is caused by Bundibugyo ebolavirus, Cote d'Ivoire ebolavirus, Sudan ebolavirus and Zaire ebolavirus. semi-auto: An Ebola hemorrhagic fever is a kind of viral Infectious disease. It often causes a muscle ache, a sore throat, a fever, weakness, stomach pain, a red eye symptom, joint pain, vomiting, a headache, rash, a severe internal bleeding, hiccups and diarrhea. It is transmissible by contaminated medical equipment, the direct contact with infected animals, and the contaminated fomite and it is caused by the species Bundibugyo ebolavirus, the Côte d’Ivoire ebolavirus, the Sudan ebolavirus and the Zaire ebolavirus. auto: An Ebola hemorrhagic fever is viral. It is a muscle aches, contaminated medical equipment, a sore throat, a fever, weakness, stomach pain, a red eye, joint pain, a vomiting, a headache, rash, a content internal bleeding symptom, a hiccups and diarrhea.

It located in Santa Barbara Region and it has flavor Moderate.  manual: This is a portrait that portrays Alexander the Great and it was created during the Roman period. It is made of marble and today it is exhibited in the Archaeological Museum of Thassos. semi-auto: This is a portrait. It is thought to be Alexander the Great, it is produced during the Roman period and it was all hand carved from marble. It is found in the Archaeological Museum of Thasos. auto: This is a portrait. It is thought to be Alexander the Great, it handles from Roman and it is marble. It derives from the Thasos Archaeological Museum. verbalizer: Exhibit 14 is portrait. It exhibit portrays alexander the great. It creation period roman period. It made of marble. It current location thasos archaelogical.  manual: Ebola hemorrhagic fever is a kind of viral infectious disease. Its symptoms are muscle aches, sore throat, fever, weakness, stomach pain, red eyes, joint pain, vomiting, headaches, rashes, internal and external bleeding, hiccups and diarrhea.

Infections are caused by molluscum contagiosum. Molluscum contagiosum often causes lesions. It is transmissible by fomites and contact with the skin, and it is caused by the molluscum contagiosum virus. semi-auto-boot: Molluscum contagiosum is a kind of viral infectious disease that occurs when the skin. Infections are caused by molluscum contagiosum. Molluscum contagiosum can cause lesions. Fomites and contact with the skin can transmit molluscum contagiosum. Molluscum contagiosum is caused by the molluscum contagiosum virus. auto-sp*: Molluscum contagiosum is a kind of viral infectious disease that is the skin. It is infections. It is lesions. It is fomites and contact with the skin, and it is caused by the molluscum contagiosum virus. auto-boot: Molluscum contagiosum is a kind of viral infectious disease that is the skin. It is infections. It is lesions. It is fomites, the molluscum contagiosum virus and contact with the skin.

semi-auto-boot: This is a kantharos, produced during the Hellenistic period and almost certainly from Amphipolis. It is among the Archaeological Museum of Kavala. auto-sp*: This is a kantharos that handles from the Hellenistic period and is almost certainly from Amphipolis. It derives from the Archaeological Museum of Kavala. auto-boot: This is a kantharos that is the Hellenistic period and is Amphipolis. It is the Archaeological Museum of Kavala. no-sp: This is a kantharos. It creation period the Hellenistic period, it original location Amphipolis and it current location the Archaeological Museum of Kavala.  manual-sp: Molluscum contagiosum is a kind of viral infectious disease that affects the skin. It results in infections and its symptom is lesions. It is transmitted by fomites and contact with the skin, and it is caused by the molluscum contagiosum virus. semi-auto-sp*: Molluscum contagiosum is a kind of viral infectious disease that can occur in the skin.

semi-auto-boot: This is a full, dry Zinfandel. It is moderate. Elyse produced it and the Napa County is Home to it. auto-sp*: This is a full, dry Zinfandel. It is moderate. It is made at Elyse and it is the Napa County. auto-boot: This is a full, dry Zinfandel. It is moderate. It is Elyse and the Napa County. no-sp: This is a Zinfandel. It has sugar dry, it has body full and it has flavor moderate. It has maker Elyse and it located in the Napa County.  manual-sp: This is a kantharos, created during the Hellenistic period and it originates from Amphipolis. Today it is exhibited in the Archaeological Museum of Kavala. semi-auto-sp*: This is a kantharos, produced during the Hellenistic period and almost certainly from Amphipolis. It is found in the Archaeological Museum of Kavala.

manual-nln: This is a statue, created during the classical period and sculpted by Polykleitos. Currently it is exhibited in the National Archaeological Museum of Napoli. semi-auto-nln: This is a statue, created during the classical period and sculpted by the sculptor polyclitus. Currently it is exhibited in the Naples National Archaeological Museum. auto-nln: This is a statue, created during classical and sculpted by the polyclitus. Currently it is exhibited in national arch napoli. no-nln: Exhibit 4 is statue, created during classical period and sculpted by polyclitus. Today it is exhibited in national arch napoli.  manual-nln: Systemic mycosis is a kind of fungal infectious disease that affects the human body. It results in infection of internal organs. It is caused by fungi. semi-auto-nln: A systemic mycosis is a kind of fungal infectious disease that affects human body. It results in infections of internal organs and it is caused by the fungi. auto-nln: A systemic mycosis is fungal that affects human body. It results in infections of internal organs and it is caused by the fungi.

For the purposes of this study, INLINEFORM4 marked all the correct nl names (not only the best ones) among the top five of each individual or class. In Table TABREF76 , micro-precision is the number of nl names (across all the individuals and classes) that were marked as correct by both INLINEFORM5 and INLINEFORM6 , divided by the number of nl names marked as correct by INLINEFORM7 , i.e., we treat the decisions of INLINEFORM8 as gold. Macro-precision is similar, but we first compute the precision of INLINEFORM9 against INLINEFORM10 separately for each individual or class, and we then average over all the individuals and classes. INLINEFORM11 1-in-5 is the percentage of individuals and classes for which INLINEFORM12 marked at least one nl name among the top five as correct, and similarly for INLINEFORM13 1-in-5.

EQREF47 ), DISPLAYFORM0  we define the prominence of a candidate sentence plan INLINEFORM0 with respect to the seed name pairs of INLINEFORM1 : DISPLAYFORM0  where: the INLINEFORM0 denotes 1 if condition INLINEFORM1 holds and 0 otherwise; INLINEFORM2 (in the numerator) is the number of times INLINEFORM3 matched any np anchor pair of the parsed sentences of INLINEFORM4 , counting only matches that contributed to the extraction of a template of INLINEFORM5 that led to the particular sentence plan INLINEFORM6 ; by contrast, INLINEFORM7 (in the denominator) is the number of times INLINEFORM8 matched any np anchor pair of the parsed sentences of INLINEFORM9 , counting only matches that contributed to the extraction of any template of INLINEFORM10 ; and INLINEFORM11 is the total number of seed name pairs of INLINEFORM12 . In other words, we count how many (distinct) seed name pairs of INLINEFORM13 produced INLINEFORM14 , dividing by the number of (distinct) seed name pairs of INLINEFORM15 that produced at least one template of INLINEFORM16 .

We also define: DISPLAYFORM0  where: INLINEFORM0 is the number of times INLINEFORM1 matched the np anchor pair INLINEFORM2 in a parsed sentence of INLINEFORM3 contributing to the extraction of template INLINEFORM4 ; INLINEFORM5 are the numbers of seed name pairs, np anchor pairs, templates of INLINEFORM6 . We define similarly INLINEFORM7 and INLINEFORM8 , obtaining five features from each type of productivity score. For each productivity version of Section UID35 , we define a prominence variant. For example, based on the productivity of a seed name pair INLINEFORM0 of a relation INLINEFORM1 (Eq. EQREF36 , repeated as Eq.

We define similarly INLINEFORM18 for a seed name INLINEFORM19 that occurs as the second element in any seed name pair INLINEFORM20 of INLINEFORM21 , obtaining five more features for INLINEFORM22 . Similarly to Eq. EQREF36 , we define the productivity of the INLINEFORM0 -th NP anchor pair INLINEFORM1 (e.g., INLINEFORM2 INLINEFORM3 “Semillon”, INLINEFORM4 “Semillon grapes” INLINEFORM5 in Fig. FIGREF32 ) of a relation INLINEFORM6 as follows: DISPLAYFORM0  where: INLINEFORM0 is the number of times a seed name pair of INLINEFORM1 matched INLINEFORM2 in the parsed sentences of INLINEFORM3 and contributed to the extraction of any template of INLINEFORM4 ; and INLINEFORM5 is the total number of np anchor pairs of INLINEFORM6 . As with INLINEFORM14 , the intuition behind INLINEFORM15 is that np anchor pairs that match many seed name pairs of INLINEFORM16 are more indicative of INLINEFORM17 .

Subsequently, any sentence plan that has only two slots, starts with a verb, or contains no verb, is discarded, because sentence plans of these kinds tend to be poor. Also, if a sentence plan contains a single verb in the past participle, in agreement with either INLINEFORM2 or INLINEFORM3 , followed by a preposition (e.g. “ INLINEFORM4 made in INLINEFORM5 ”), we insert an auxiliary verb to turn the verb form into present passive (e.g., “ INLINEFORM6 is made in INLINEFORM7 ”); in domains other than those of our experiments, a past passive may be more appropriate (“ INLINEFORM8 was made in INLINEFORM9 ). Similarly, if a single verb appears in the present participle (e.g. “ INLINEFORM10 making INLINEFORM11 ”), we insert an auxiliary verb to obtain a present continuous form. Both cases are illustrated below.     Lastly, we filter the remaining sentence plans through a Web search engine. For this step, we replace referring expression slots by wildcards, we generate the rest of the sentence (e.g., “* is made from *”), and we do a phrase search. If no results are returned, the sentence plan is discarded.

Turning the templates into candidate sentence plans	The templates (e.g., “ INLINEFORM0 is made from INLINEFORM1 ”) are not yet sentence plans, because they lack the linguistic annotations that Naturalowl requires (e.g., pos tags, agreement, voice, tense, cases), and they may also not correspond to well-formed sentences (e.g., they may lack verbs). The conversion of a template to a (candidate) sentence plan is similar to the conversion of Section SECREF22 . We start by obtaining pos tags from the parse trees of the sentences the template was obtained from. Recall that a template may have been extracted from multiple sentences. We obtain a pos tag sequence for the words of the template from each one of the sentences the template was extracted from, and we keep the most frequent pos tag sequence. We ignore the pos tags of the anchor nps, which become INLINEFORM2 and INLINEFORM3 in the template.

We call np anchor pair any two nps of a parsed sentence that match a seed name pair, and np anchors the elements of an np anchor pair. From every parsed sentence that contains an np anchor pair, we produce a slotted string template by replacing the first np anchor by INLINEFORM0 , the second np anchor by INLINEFORM1 , including between INLINEFORM2 and INLINEFORM3 in the template the words of the sentence that were between the two np anchors, and discarding the other words of the sentence. In the example of Fig. FIGREF32 , we would obtain the template “ INLINEFORM4 is made from INLINEFORM5 ”. Multiple templates may be extracted from the same sentence, if a sentence contains more than one np anchor pairs; and the same template may be extracted from multiple sentences, possibly retrieved by different seed name pairs. We retain only the templates that were extracted from at least two different sentences. We then produce additional templates by increasingly extending the retained ones up to the boundaries of the sentences they were extracted from. In Fig. FIGREF32 , if the template “ INLINEFORM6 is made from INLINEFORM7 ” has been retained, we would also produce the following templates.

We then keep every parsed sentence that contains at least two nps matching a seed name pair. For example, the sentence “obviously Semillon is made from Semillon grapes in California” contains the nps “Semillon” and “Semillon grapes” that match the seed name pair INLINEFORM0 “Semillon”, “Semillon grape” INLINEFORM1 (Fig. FIGREF32 ). Two nps of a sentence match a seed name pair if the similarity between any of the two nps and any of the two seed names (e.g., the first np and the second seed name) is above a threshold INLINEFORM2 and the similarity between the other np and the other seed name is also above INLINEFORM3 . The similarity between an np and a seed name is computed as their weighted cosine similarity, with INLINEFORM4 weights, applied to stemmed nps and seed names, ignoring stop-words. The INLINEFORM5 of a word of the np or seed name is the frequency (usually 0 or 1) of the word in the np or seed name, respectively; the INLINEFORM6 is the inverse document frequency of the word in all the retrieved documents of the relation.

INLINEFORM0 :Wine, :madeFrom, :cabernetSauvignonGrape INLINEFORM1   INLINEFORM0 :StEmilion, :madeFrom, :Grape INLINEFORM1 , INLINEFORM2 :Wine, :madeFrom, :Grape INLINEFORM3  We obtain the same additional triples from INLINEFORM0 :Semillon, :madeFrom, :SemillonGrape INLINEFORM1 , because Wine and Grape are also parent classes of Semillon and SemillonGrape, but we remove duplicates. Each INLINEFORM0 triple is then replaced by a pair INLINEFORM1 , where INLINEFORM2 is a word sequence generated by the nl name of INLINEFORM3 , and similarly for INLINEFORM4 . We assume that the nl names are manually authored, or that they are generated by our method of Section SECREF3 . In the latter case, we keep only one nl name per individual or class, the one selected by the human author (in a semi-automatic setting of nl name generation) or the top ranked nl name (in a fully automatic setting). The five triples above become the following pairs. We call pairs of this kind seed name pairs, and their elements seed names.

DISPLAYFORM0  For each extracted np of INLINEFORM0 , we compute its similarity to every possible name, i.e., INLINEFORM1 or INLINEFORM2 , as discussed above, and we assign to the np a score equal to the largest of these similarities. Finally, we rank the extracted nps of INLINEFORM3 by decreasing score. If two nps have the same score, we rank higher the np with the fewest crossed edges in its best alignment with a name. If two nps still cannot be distinguished, we rank them by decreasing frequency in the parsed sentences of INLINEFORM4 ; and if their frequencies are equal, we rank them randomly.

such that each token of name is aligned to at most one token of np and vice versa, and any other, not formed pair INLINEFORM3 would have a Levenshtein distance (between INLINEFORM4 , INLINEFORM5 ) larger or equal to the minimum Levensthein distance of the formed pairs. In our example, the pairs of alinged tokens are INLINEFORM6 “national”, “National” INLINEFORM7 , INLINEFORM8 “arch”, “Archaeological” INLINEFORM9 , INLINEFORM10 “napoli”, “Naples” INLINEFORM11 , INLINEFORM12 “museum”, “Museum” INLINEFORM13 . The similarity between np and name is then computed as follows, where INLINEFORM0 is the set of aligned token pairs, INLINEFORM1 is the Levenshtein distance (normalized to INLINEFORM2 ) between the INLINEFORM3 and INLINEFORM4 of pair INLINEFORM5 , and INLINEFORM6 , INLINEFORM7 are the lengths (in tokens) of np and name, respectively.

anchor=base] (C) napoli; outer sep=0pt,anchor=base] (D) museum; outer sep=0pt,anchor=base] (t) (the); outer sep=0pt,anchor=base] (a) Naples; outer sep=0pt,anchor=base] (b) National; outer sep=0pt,anchor=base] (c) Archaeological; outer sep=0pt,anchor=base] (d) Museum; ; (A) – (b) node[draw=none,fill=none,font=,midway,left] 0; (B) – (c) node[draw=none,fill=none,font=,midway,right] 10; (C) – (a) node[draw=none,fill=none,font=,midway,above] 4; (D) – (d) node[draw=none,fill=none,font=,midway,left] 0;  We then form pairs of aligned tokens INLINEFORM0 , where INLINEFORM1 , INLINEFORM2 are tokens from name, np, respectively,

To compute the similarity between np and name, we first compute the character-based Levenshtein distance between each token of np and each token of name; we ignore upper/lower case differences, articles, and connectives (e.g. “or”), which are often omitted from owl identifiers. In the following example, np INLINEFORM4 “the Naples National Archaeological Museum” (but “the” is ignored) and name = “national arch napoli museum”; this name is an INLINEFORM5 produced by appending to INLINEFORM6 the tokenized identifier of the parent class (:Museum) of INLINEFORM7 (Section UID14 ). The Levenshtein distance between “national” and “National” is 0 (upper/lower case differences are ignored). The distance between “napoli” and “Naples” is 4; a character deletion or insertion costs 1, a replacement costs 2. [column sep=0em,row sep=.4in] outer sep=0pt,anchor=base] (T) ; outer sep=0pt,anchor=base] (A) national; outer sep=0pt,anchor=base] (B) arch; outer sep=0pt,

Subsequently, we formulate a Boolean Web search query for INLINEFORM0 (e.g., “South” AND “Australia” AND “Region”) and each INLINEFORM1 (e.g., “South” AND “Australia”); recall that INLINEFORM2 is the individual or class we wish to produce an nl name for. We convert the retrieved pages of all the queries to plain text documents and parse every sentence of the text, if any stemmed word of the sentence is the same as any stemmed word of any INLINEFORM3 or INLINEFORM4 . We then extract the noun phrases (nps) from every parsed sentence. For example, from the sentence “the Naples National Archaeological Museum houses some of the most important classical collections” we extract the nps “the Naples National Archaeological Museum”, “some of the most important classical collections”, and “the most important classical collections” (Fig. FIGREF19 ). For each extracted np, we compute its similarity to INLINEFORM0 and each INLINEFORM1 . Let np be an extracted np and let name be INLINEFORM2 or an INLINEFORM3 .

For example, if INLINEFORM6 :red and :Color is the parent class of INLINEFORM7 ( INLINEFORM8 :red, isA, :Color INLINEFORM9 ), we also obtain “red color”: tokName(:red) INLINEFORM0 “red”, tokName(:Color) INLINEFORM1 “color”, altTokName(:red) INLINEFORM2 “red color” By contrast, if INLINEFORM0 :KalinCellarsSemillon, no INLINEFORM1 is produced from the ancestors of INLINEFORM2 , because INLINEFORM3 “Kalin Cellars Semillon” contains tokName(:Semillon) INLINEFORM4 “Semillon”, and :Semillon is an ancestor of :KalinCellarsSemillon. Furthermore, we create an additional INLINEFORM0 by removing all numbers from INLINEFORM1 ; for example, from INLINEFORM2 “Semillon 2006” we obtain INLINEFORM3 “Semillon”. Lastly, if INLINEFORM4 contains brackets, we create an INLINEFORM5 for each part outside and inside the brackets; for example, from “gerbil (dessert rat)” we get “gerbil” and “dessert rat”.

Sections SECREF3 and SECREF4 then describe our methods to extract natural language names and sentence plans, respectively, from the Web. Section SECREF5 presents our experimental results. Section SECREF6 discusses related work. Section SECREF7 concludes and suggests future work.

Question: what ontologies did they use?
Answer: Wine Ontology, m-piro ontology, Disease Ontology