# Natural Language Interaction with the Web of Data by Mining its Textual Side

Elena Cabrio, Julien Cojan, Alessio Palmero Aprosio, Fabien Gandon

► To cite this version:

Elena Cabrio, Julien Cojan, Alessio Palmero Aprosio, Fabien Gandon. Natural Language Interaction with the Web of Data by Mining its Textual Side. Intelligenza Artificiale, 2012, Special Issue on Natural Language Processing in the Web Era, pp.137-154. hal-01171879

HAL Id: hal-01171879

https://inria.hal.science/hal-01171879v1

Submitted on 27 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Natural Language Interaction with the Web of Data by Mining its Textual Side

Elena Cabrio<sup>a</sup>, Julien Cojan<sup>a</sup>, Alessio Palmero Aprosio<sup>b</sup> and Fabien Gandon<sup>a</sup>

AbstractThe semantic web is an extension of the classical web. The data and schemas it adds coexist with the documents that were already linked and available. This not only allows interoperability, reusability and potentially unforeseen applications of opened data, but it also creates a unique situation of availability on the web of huge collections of the same pieces of information available at the same time as text and as structured data. An interesting example is the couple Wikipedia-DBpedia: exploiting these interlinked structured and unstructured data sources in parallel can offer a great potential for both Natural Language Processing and Semantic Web applications. Starting from these observations, this paper addresses the problem of enhancing interactions between non-expert users and data available on the Web. In particular, we present QAKiS, a system for open domain Question Answering over linked data, that addresses the problem of question interpretation as a relation-based match, where fragments of the question are matched to binary relations of the triple store, using relational textual patterns automatically collected. In the current version, the relational patterns are automatically extracted from Wikipedia, while DBpedia is the data set to be queried using a natural language interface.

Keywords: Question Answering, Linked Data, Wikipedia, DBpedia

## 1. Web of documents and web of data: jointly exploiting two facets of the web

In the early days of the web this child of hyper-text systems and network applications was often presented using the metaphor of a huge online library of interlinked documents. This documentary facet of the web is the most ancient and most persistent perception of it, and sometimes still hides the true nature of the web, which is in fact a huge network of computational resources, algorithms and data. Among the side effects of this biased perception of the web, for a long time Natural Language Processing (NLP) community only saw the web as a huge corpus (e.g. [20], [19]), where data are stored in standard HTML documents, and presented to the users in human readable representations rendered through the web technical stack and architecture. But in fact one could argue

that there are no documents on the web and there never were. We only have ephemeral representations transferred through networks and produced on demand by distributed programs using distributed data.

What is now called the web of data can been seen as the first wave of the deployment of the Semantic Web (SW), and makes very visible the fact that the web is not only publishing linked documents, but also linked data and linked applications. The standardization of the frameworks of the semantic web aims at transforming the access to information by adding machine-readable linked data and the semantics of their schema to the already visible textual content, to facilitate automated processing and integration of the vast amount of available information on the web.

An important point to make here is that the Semantic Web is an extension of the classical web and that the data and schemas it adds coexist with the document-

<footnote><sup>a</sup> INRIA Sophia Antipolis, 2004 Route des Lucioles BP93, 06902 Sophia Antipolis, France.</footnote>

<footnote>E-mail: {elena.cabrio,julien.cojan,fabien.gandon}@inria.fr</footnote>

<footnote><sup>b</sup> FBK, Via Sommarive 18, 38100 Povo-Trento, Italy.</footnote>

<footnote>Università degli Studi di Milano, Via Festa del Perdono 7, 20122 Milano, Italy.</footnote>

<footnote>E-mail: aprosio@fbk.eu</footnote>
tary representations that were already available and
linked on the web. Moreover, more and more web sites
are adding direct access to the data they use to gen-
erate their pages and support the services they offer.
This not only allows interoperability, reusability and
potentially unforeseen applications of opened data, but
it also creates a unique situation of availability on the
web of huge collections of the same pieces of infor-
mation available at the same time in textual represen-
tations and as data structured by primitives defined
in public formal schemas. For the NLP field this cre-
ates an unprecedented situation where very large bod-
ies of linguistic resources are available together with
corresponding semantically structured data ready to be
used to bootstrap, improve and feed NLP approaches
and applications. In addition, since the web is a hybrid
space where software agents and people interact, there
is a growing interest in methods bridging the semantic
divide between the places on the web where users pub-
lish, exchange free content (e.g. forums, wikis, micro-
blogs) and places where software exchanges data. NLP
has a lot to provide here, for instance in supporting
natural language interactions with the growing web of
data and web of services.

NLP and Semantic Web research fields are therefore
becoming more and more interconnected, even if they
address, in a way, opposite perspectives. NLP tools fo-
cus on unstructured information (e.g. text documents),
while Semantic Web tools typically deal with more
structured information on a more granular level. Many
important problems span the two worlds of structured
and unstructured information where the combination
of NLP and Semantic Web tools would be highly com-
plementary. For instance, in the use of NLP tools to au-
tomatically tag web pages with RDF descriptions (i.e.
Resource Description Framework, a standard model
for data interchange on the Web)<fnref n="1" />, or the possibility to
query the SW to retrieve object related services using
current search engines [10].

A very important case of such web sites offering
strongly tied texts and data is the couple Wikipedia-
DBpedia. Collaboratively constructed resources such
as Wikipedia, have grown into central knowledge
sources providing a vast amount of updated informa-
tion accessible on the Web essentially as pages for hu-
man consumption. From such corpora, structured in-
formation has been extracted and stored into knowl-
edge bases (e.g. DBpedia), that cover a wide range

of different domains and connect entities across them.
Exploiting these interlinked structured and unstruc-
tured data sources in parallel offers a great potential
for both NLP and Semantic Web applications. Many
researchers have indeed used this kind of resources on
the web as a basis for training statistical algorithms,
and we should expect that much more brilliant results
can be obtained if the learning base can rely on the
fine grained grid proposed by the SW [10]. Moreover,
NLP applications such as information extraction have
in fact advanced to the point where they can ease the
knowledge acquisition bottleneck, by creating seman-
tic annotations from text. Such techniques are there-
fore able to provide the requisite RDF data for the SW
from existing unstructured text databases in the web.
At the same time, NLP systems can also consume se-
mantic web data and schemas, providing for instance
natural language query systems that take advantage of
semantic web meta-data to provide the answers to a
user question.

In line with the direction of filling the gap between
NLP and Semantic Web, in this paper we address the
problem of enhancing interactions between non-expert
users and data available on the Web. This research
challenge can be broken down into the following sub-
questions: i) how to automatically extract structured
data from unstructured documents to populate RDF
triple stores? and ii) in a Question Answering setting,
how to map natural language expressions (e.g. user
queries) with concepts and relations in a structured
knowledge base?

Given the complexity of the research question,
our work narrows its scope focusing on the couple
Wikipedia-DBpedia as a case study of parallel sources
of structured and unstructured data. In particular, we
present our ongoing works on i) the WikiFramework,
i.e. a methodology to collect relational patterns in sev-
eral languages, for the relations defined in DBpedia
ontology, and ii) QAKiS, a system for open domain
Question Answering over linked data. A crucial is-
sue in Question Answering over linked data, is the
interpretation of the question in order to convert it
into a corresponding query in a formal language (e.g.
SPARQL). Most of current approaches (e.g. Power-
Aqua, Freya [22]) base this conversion on some form
of flexible matching between words of the question
and names of concepts and relations of a triple store.
One of the limitations however, is that a word-based
match may fail to detect the relevant context around a
word, without which the match might be wrong.

<footnote n="1">¹http://www.w3.org/RDF/</footnote>
The generic approach behind QAKiS addresses the task of QA over structured knowledge bases (e.g. DBpedia) where the relevant information is expressed also in unstructured form (e.g. Wikipedia pages). Its major novelty is to implement a relation-based match, where fragments of the question are matched to relational textual patterns automatically collected from Wikipedia (i.e. the WikiFramework repository). The underlying intuition is that such relation-based matching would provide more precision with respect to the matching on single tokens, as done by current QA systems on linked data.

The reminder of the paper is as follows. Section 2 introduces the sources of unstructured and structured information on which we focus in our case study, i.e. Wikipedia and DBpedia. Section 3 presents the WikiFramework. Sections 4 and 5 describe QAKiS system architecture, workflow and our participation to the Question Answering over Linked Data Challenge. Section 6 compares the proposed approach to the related work. Conclusions (Section 7) end the paper, discussing interesting perspectives for future work.

## 2. Wikipedia and DBpedia: rich content fraternal twin sites

As introduced before, the work presented in this paper relies on the intuition that exploiting interlinked structured and unstructured data sources in parallel can offer a great potential for both NLP and Semantic Web applications. As a case study, we focus on the couple Wikipedia-DBpedia. As we all know, Wikipedia<fnref n="2" /> is a collaboratively constructed online encyclopedia: its content is free to reuse, and can be edited and improved by anyone following the rules of the edition process. In the last years, researchers involved in NLP tasks have taken advantage of Wikipedia as a vast resource of textual documents, in particular for Information Extraction (IE). In the first part of this section (Section 2.1) we briefly introduce Wikipedia as a knowledge collection, i.e. a huge textual resource to mine information exploited by NLP applications. From such corpus, structured information has been extracted and stored into a knowledge base, i.e. DBpedia<fnref n="3" />, to make Wikipedia content machine-readable. In the second part of this section (Section 2.2), we therefore focus on the principles underlying DBpedia creation process and structure.

### 2.1. Wikipedia: knowledge collection and information extraction

Wikipedia contains a huge amount of completely free data in various languages. Moreover, its crowdsourcing model and the very productive community around it assure that the stored information is very precise and updated. As a result, beside becoming a reference knowledge source for people in their everyday life, Wikipedia properties are more and more exploited for various NLP tasks.<fnref n="4" /> Researchers started to use such resource to substitute or integrate conventional lexical semantic resources such as WordNet [12] or linguistically annotated corpora in different NLP tasks (e.g. word sense disambiguation, semantic role labeling, information retrieval, text categorization). In particular, for information extraction applications, links between Wikipedia pages in different languages can be used to calculate cross-language text similarity [28], while argument categories and lists of pages can set up the basis to build elementary taxonomies [27].

Recently, a valuable new feature has been added to Wikipedia pages, the *infobox*, i.e. a table summarizing the key concepts related to the subject of an article (see Figure 1). For example, biographies in Wikipedia have a specific *infobox* usually containing factual information about the person described in the article, e.g. date of birth, nationality, activity, and so on. Originally, it was designed for an aesthetic purpose (it fills the top right-hand corner box often present in Wikipedia pages) but then it became a very important source of structured information for research purposes. The data in the *infoboxes* can be easily extracted to build a knowledge base, and some research projects are born with this aim, such as DBpedia,<fnref n="5" /> FreeBase,<fnref n="6" /> and Yago.<fnref n="7" /> Freebase [4] contains data from structured sources, like Wikipedia and MusicBrainz, as well as hand-crafted data from its users. Yago [29] is automatically built [17] from Wikipedia, while GeoNames<fnref n="8" /> and

<footnote n="4">⁴Among other initiatives, the workshop on “the People’s Web meets NLP: Collaboratively Constructed Semantic Resources and their Applications to NLP” (at its third edition in 2012) gathers works around this topic http://www.ukp.tu-darmstadt.de/scientific-community/acl-2012-workshop</footnote>

<footnote n="5">⁵http://dbpedia.org/About</footnote>

<footnote n="6">⁶http://www.freebase.com/</footnote>

<footnote n="7">⁷http://www.mpi-inf.mpg.de/yago-naga/yago/</footnote>

<footnote n="8">⁸http://www.geonames.org/</footnote>

<footnote n="2">²http://www.wikipedia.org/</footnote>

<footnote n="3">³http://dbpedia.org</footnote>
WordNet [12] using an unsupervised approach. DBpedia [24] knowledge base lies in the middle, since it includes a manually-built ontology populated using articles in Wikipedia by a rule-based semi-automatic approach (see Section 2.2).

Infoboxes can also be used as structured information to which the distant supervision paradigm can be applied [26]. This new approach consists in a semi-supervised learning algorithm trained starting from a knowledge base instead of an annotated corpus, and it is convenient for natural language applications, where annotated data is often not readily available and more expensive to be produced in terms of human effort and speed. Differently from current approaches (see Section 6), in our work we explore the use of the distant supervision in a different framework, i.e. question answering. In particular, we rely on the DBpedia knowledge base (described in the next section) as training set: its entries are searched in Wikipedia articles to extract textual patterns expressing the DBpedia relations.

## 2.2. DBpedia principles

DBpedia [3] is a database extracted from Wikipedia. Since Wikipedia pages contain a lot of data potentially useful for automatic programs, the Semantic Web community started an effort to extract data from Wikipedia and then to publish them in a structured format (following the open standards of the Semantic Web) to make them machine-readable. Taking benefit from Wikipedia wide corpus, DBpedia 3.8 release [2] describes:

> 3.77 million things, [...] including 764,000 persons, 573,000 places (including 387,000 populated places), 333,000 creative works (including 112,000 music albums, 72,000 films and 18,000 video games), 192,000 organizations (including 45,000 companies and 42,000 educational institutions), 202,000 species and 5,500 diseases.

Moreover, data is also extracted from the equivalent articles from 111 languages, as well as links to 1.300.000 links to images and 6.600.000 external web pages.

In the following, this section proposes a small survey to investigate the potentiality of these sources of interlinked structured and unstructured data, focusing in particular on i) the methodologies to extract data from Wikipedia, ii) the naming convention and data structure, and iii) the normalization strategies (required by the noisy vocabulary used in Wikipedia).

### 2.2.1. Data extraction from Wikipedia

Figure 1 shows an example of Wikipedia article, pointing at the elements from which data are typically extracted. The title of the page is used as identifier, and in case it carries multiple meanings a disambiguation page redirects the user to the pages corresponding to each meaning. For instance, Jaguar_(disambiguation) lists several articles, among which the page about the animal (i.e. Jaguar) and the one describing the car maker (i.e. Jaguar Cars). Articles may contain links within the same Wikipedia chapter (i.e. internal links), links towards other chapters (i.e. inter-language links) and links towards other web pages (i.e. external links). Inter-language links connect articles about the same subject in different languages.

As introduced before, Wikipedia articles can also contain structured elements like the infobox, providing factual data in the form of values associated to fields, that can be easily extracted. Categories define the topics that are used to index the articles (they are also written as Wikipedia pages, in a separate namespace from the articles), and are organized in a generalization hierarchy. For instance, the category Bridges in the San Francisco Bay Area is a subcategory of Bridges in California. Articles are tagged with a category and linked to the category page. For instance, the article Golden Gate Bridge is tagged with the categories Bridges in the San Francisco Bay Area and 1937 in California. As can be seen from the last example, the thematic tagging of the articles according to their categories cannot be used as classification (e.g. "1937 in California" is not a set of this that the Golden Gate Bridge belongs to), but can be exploited to generate one [29]. In subsection 2.2.3, we explain how DBpedia classification is created.

### 2.2.2. Naming conventions and data structure

DBpedia describes the subjects of Wikipedia articles. They are identified by URIs defined by replacing the beginning of the URIs of Wikipedia articles by http://dbpedia.org/resource/. For instance, the Golden Gate Bridge is identified by http://dbpedia.org/resource/Golden_Gate_Bridge.

The data is stored in DBpedia in the RDF format [36], which is a Semantic Web standard for data representation, and is made of triples (subject, predicate, value). A sample of the generated triples<fnref n="9" />

<footnote n="9">⁹We take the usual prefix for DBpedia resources, dbpedia: as a short version for http://dbpedia.org/resource/. The Golden Gate Bridge URI is then written as</footnote>
<figure><img src="image_2.png" /><figcaption>Figure 1. Extracted elements from Golden Gate Bridge Wikipedia article.</figcaption></figure>

<table><caption>Table 1
Triples extracted from the page Golden Gate Bridge.</caption><thead><tr><td>Subject</td><td>Predicate</td><td>Value</td></tr></thead><tbody><tr><td>dbpedia:Golden_Gate_Bridge</td><td>rdfs:label</td><td>"Golden Gate Bridge"@en</td></tr><tr><td>dbpedia:Golden_Gate_Bridge</td><td>dbpedia-prop:carries</td><td>"6 lanes of US 101, pedestrians and bicycles"@en</td></tr><tr><td>dbpedia:Golden_Gate_Bridge</td><td>dbpedia-prop:length</td><td>8,981</td></tr><tr><td></td><td>...</td><td></td></tr><tr><td>dbpedia:Golden_Gate_Bridge</td><td>dbpedia-prop:crosses</td><td>dbpedia:Golden_Gate</td></tr><tr><td>dbpedia:Golden_Gate_Bridge</td><td>dbpedia-prop:designer</td><td>dbpedia:Joseph_Strauss_(engineer)</td></tr><tr><td>dbpedia:Golden_Gate_Bridge</td><td>dbpedia-prop:designer</td><td>dbpedia:Irving_Morrow</td></tr><tr><td>dbpedia:Golden_Gate_Bridge</td><td>dbpedia-prop:designer</td><td>dbpedia:Charles_Alton_Ellis</td></tr></tbody></table>

is provided in Table 1. These triples express property values of the Golden Gate Bridge. The predicate rdfs:label is part of a standard vocabulary to describe resources in RDF; the predicates, e.g.

dbpedia-prop:carries, dbpedia-prop:length and so on, are defined from the field name of the infobox (i.e. *carries* and *total length* respectively, see Figure 1). When the value of a property is a link to another Wikipedia page, it is replaced by the URI of the corresponding resource (the last four triples in the Table 1). Since these triples create additional relations among resources, the database can then be seen as a graph with the resources as edges and relations as

<footnote>dbpedia:Golden_Gate_Bridge. Other prefixes used in the following are dbpedia-prop: for http://dbpedia.org/property/ and dbpedia-owl: for http://dbpedia.org/ontology/.</footnote>
wedges. The connection between resources provide access to the data of the connected resource (for instance, given the Brookling Bridge example, a user can be interested in knowing where the architect that designed the bridge was born).

With respect to more technical aspects, DBpedia data are available under the license CC-BY-SA<fnref n="10" />. The resource can also be dereferenced, which means that a http request on a resource URI will return a description of the resource, i.e. the triples in which it appears in the subject or in the object position. The format of the data returned depend on the http header: by default it is an html page, but other formats can be requested (e.g. RDF/XML, JSON, N-triples). It is then possible to navigate through the data with the “follow your nose principle”: the description of the dereferenced resource contains the URIs of the related resources, that can be dereferenced in turn, and so on. In addition, the database can be queried<fnref n="11" /> with the language SPARQL [37].

### 2.2.3. Alignment to a normalized vocabulary

The predicates described in the previous subsection are generated from the field names in the infoboxes, so they strongly depend on the terms used in Wikipedia. If different terms are used to express the same property in two different infoboxes, different predicates will be generated. For instance, the field *designer* in Infobox Bridge (Figure 1) is used to specify the architect of the bridge, while in the Infobox building the field *architect* is used instead. Applications that want to use DBpedia data must therefore handle this problem of terms variability. In addition, the field names change from one language to another.

To overcome this issue, an ontology is edited by the DBpedia community<fnref n="12" /> in order to define a normalized vocabulary. The ontology contains a class hierarchy built from the Infobox types. For instance, it states that a *Bridge* is a *Route of transportation*, which is itself an *Infrastructure*. The ontology contains also the properties definition. The correspondence between the terms used in Wikipedia and the ontology terms are also edited collaboratively as one mapping per infobox, so the mapping is given in the context of one infobox. For instance, Table 2 shows the correspondences obtained for our example.

In this section, we introduced and described a very rich synchronized source of text and data. In the next section, we will consider how this coupling Wikipedia-DBpedia can be exploited to collect natural language relational patterns corresponding to data schema relational patterns.

## 3. WikiFramework: collecting relational patterns

Relational patterns capture different ways to express a certain relation in a given language. For instance, the relation `birthDate(Person, Date)` can be expressed in English by the following relational patterns: `[Person was born in Date]`, `[Person, (Date)]`, `[Person, whose date of birth is Date]`.

The goal of the WikiFramework [23] we present in this section is to establish a robust methodology to collect relational patterns in several languages, for the relations defined in DBpedia ontology (similarly to [15], [38]) that where presented in the previous section.

We assume that there is a high probability that the structured information present in the *infobox* is also expressed using natural language sentences in the same Wikipedia page. Therefore, we collect all the triples for the 1296 DBpedia ontology relations, and for each relation we apply the following procedure:

1. the subject of each triple is extracted (e.g. the instance Golden Gate Bridge);

2. each DBpedia relation is automatically matched with the Wikipedia pages whose topic is the same as the subject, and in which such relation is reported in the *infobox*. For instance, given the relation *crosses* reported above, the Wikipedia page about the Golden Gate Bridge is selected (Figure 2);

3. all the sentences in the selected Wikipedia pages where both the strings of subject and object of the relation match are collected. For instance, in the page about the Golden Gate Bridge (Figure 2), the sentence "The Golden Gate Bridge is a suspension bridge spanning the Golden Gate" is detected, since both entities match (i.e. domain: Golden Gate Bridge, range: Golden Gate);

4. such matching sentences are extracted and the subject and object are substituted with the corresponding DBpedia ontology classes (i.e. for the relation *crosses*, *Bridge* is the domain and *River* is the range)

<footnote n="10">¹⁰http://dbpedia.org/Downloads.</footnote>

<footnote n="11">¹¹http://dbpedia.org/sparql</footnote>

<footnote n="12">¹²http://mappings.dbpedia.org</footnote>
<table><caption>Table 2
Correspondences between terms in Wikipedia and ontology terms.</caption><thead><tr><td>Subject</td><td>Predicate</td><td>Value</td></tr></thead><tbody><tr><td>dbpedia:Golden_Gate_Bridge</td><td>rdf:type</td><td>dbpedia-owl:Bridge</td></tr><tr><td>dbpedia:Golden_Gate_Bridge</td><td>rdf:type</td><td>dbpedia-owl:RouteOfTransportation</td></tr><tr><td>dbpedia:Golden_Gate_Bridge</td><td>dbpedia-owl:bridgeCarries</td><td>"6 lanes of US 101, pedestrians and bicycles"@en</td></tr><tr><td>dbpedia:Golden_Gate_Bridge</td><td>dbpedia-owl:crosses</td><td>dbpedia:Golden_Gate</td></tr><tr><td>dbpedia:Golden_Gate_Bridge</td><td>dbpedia-owl:architect</td><td>dbpedia:Joseph_Strauss_(engineer)</td></tr><tr><td>dbpedia:Golden_Gate_Bridge</td><td>dbpedia-owl:architect</td><td>dbpedia:Irving_Morrow</td></tr><tr><td>dbpedia:Golden_Gate_Bridge</td><td>dbpedia-owl:architect</td><td>dbpedia:Charles_Alton_Ellis</td></tr></tbody></table>

<figure><img src="image_3.png" /><figcaption>Figure 2. An example of Wikipedia page with infobox.</figcaption></figure>

5. the pattern [The Bridge is a suspension bridge spanning the River] is obtained and stored in a pattern repository.

To increase the recall of the pattern extraction algorithm outlined above, we apply different matching strategies. The first one involves the exact match of the entire string (the subject), as provided by Wikipedia. If the algorithm does not find such string, we clean it deleting the expressions between brackets, used to disambiguate pages with the same title, as in “Carrie (novel)” and “Carrie (1976 film)”. In the final step, the original string is tokenized and lemmatized using Stanford CoreNLP<fnref n="13" /> and, given the set of obtained tokens,

a new string is built by combining them (preserving the original word order). For instance, starting from “John Fitzgerald Kennedy”, we obtain the new strings “John Fitzgerald”, “John Kennedy”, “Fitzgerald Kennedy”, “John”, “Fitzgerald” and “Kennedy”. In case of numeric or time ranges, we apply the entity extractor of Stanford CoreNLP, and match the value in the extracted string with the given normalized value.

Once the patterns for all relations are collected, we cluster them according to the lemmas that are present between the domain and the range. Then, we sort such patterns according to the frequency they appeared in Wikipedia pages, and we wipe out: i) the ones whose frequency is less than 2, and ii) the ones that contain only non discriminative words (e.g. punctuation marks, prepositions, articles, etc. as in the pattern [Person (Date)]). Table 3 reports an extract of the

<footnote n="13">¹³http://nlp.stanford.edu/software/corenlp.shtml</footnote>
<table><caption>Table 3
Examples of relational patterns.</caption><thead><tr><th>Relation</th><th>Patterns</th><th>Relation</th><th>Patterns</th></tr></thead><tbody><tr><td>spouse</td><td>Person is the wife of Person<br>Person married Person<br>Person, the husband of Person</td><td>birthDate</td><td>Person was born on Date<br>Person (Date<br>Person (born Date</td></tr><tr><td>crosses</td><td>Bridge spans the River<br>Bridge is a bridge over the River<br>Bridge crossing River</td><td>location</td><td>Thing, located in PopulatedPlace<br>Thing founded in PopulatedPlace<br>Thing corporate headquarters in PopulatedPlace</td></tr><tr><td>author</td><td>Person wrote Work<br>Work is a novel by Person<br>Work was written by Person</td><td>numberOf<br>Students</td><td>University student population is Number<br>University school enrolls Number<br>Number students attend University</td></tr></tbody></table>

set of patterns collected for a sample of the DBpedia relations.

This section showed how the coupling of documents and data in a web source can be mined to collect natural language relational patterns corresponding to data schema relational patterns. In the next section, we will show how these patterns can be used to enhance natural language interactions with the data collection, i.e. how the WikiFramework pattern repository is integrated and exploited by the QAKiS system for answer retrieval.

## 4. QAKiS: a system for data answer retrieval from natural questions

QAKiS (Question Answering with Framework-based System), is a question answering system that allows end users to submit a query to an RDF triple store in English and obtain the answer in the same language, hiding the complexity of the non intuitive formal query languages involved in the resolution process. At the same time, the expressiveness of these standards is exploited to scale to the huge amounts of available semantic data. In its current implementation, QAKiS addresses the task of QA over structured Knowledge Bases (KBs) (e.g. DBpedia) where the relevant information is expressed also in unstructured form (e.g. Wikipedia pages). It implements a relation-based match for question interpretation, to convert the user question into SPARQL. The underlying intuition is that a relation-based matching would provide more precision with respect to matching on single tokens, as done by current QA systems.

In this section we present QAKiS architecture, and a step by step description of the system work-flow. A demo of QAKiS based on Wikipedia for patterns extraction and on DBpedia as RDF data set to be queried

using a natural language interface is available online<fnref n="14" />, and shown in Figure 3.

QAKiS is composed of two main modules (Figure 4): i) the *query generator* takes the user question as input, generates the typed questions to be matched with the patterns, and then generates the SPARQL queries from the retrieved patterns; ii) the *pattern matcher* takes as input a typed question, and retrieves the patterns (among those stored in the pattern repository) that match such question with the highest similarity.

The current version of QAKiS targets questions containing a Named Entity (NE) that is related to the answer through one property of the ontology, as Which river does the Brooklyn Bridge cross?. According to our WikiFramework-based approach, each question matches a single pattern (i.e. one relation). As explained in the following, before running the pattern matcher component, we replace i) the NE present in the question by its types, and ii) the question keywords by the Expected Answer Type (EAT), obtaining what we call a *typed question* (e.g. [River] does the [Bridge] cross?). The answer can then be retrieved with a SPARQL query over a single triple.

### 4.1. EAT and NE identification

Before running the *pattern matcher* component, we identify the target of the question with a NER (Named Entity Recognition) tool. We apply the Stanford Core NLP NE Recognizer together with a set of strategies based on the comparison with the labels of the instances in the DBpedia ontology. We plan to test the use of other NER tools in the future. At the same time, simple heuristics are applied to infer the EAT from the question keyword, e.g. if the question starts with “When”, the EAT is [Date]

<footnote n="14">¹⁴http://dbpedia.inria.fr/qakis/</footnote>
<figure><img src="image_4.png" /></figure>

Question Answering
WikiFramework-based
System

Which river does the Brooklyn Bridge cross?

Get answers!

Clear

Your asked: Which river does the Brooklyn Bridge cross?

**Pattern matching:**

typed question: [River] does [Bridge] cross ?

had a best match with pattern: crosses [D:Bridge] international border cross [R:River]

with score 10.1366333333333

**The query generated is:**

select distinct * where { <http://dbpedia.org/resource/Brooklyn_Bridge> <http://dbpedia.org/ontology/crosses> ?v . ?v rdf:type <http://dbpedia.org/ontology/River> . OPTIONAL {?v <http://www.w3.org/2000/01/rdf-schema#label> ?I filter (lang(?l)="en") } } limit 20

<table><caption>Figure 3. QAKiS demo interface. The user can either write a question (or select among a list of examples) and click on *Get Answers!*. QAKiS outputs: i) the user question (the recognized Named Entity (NE) is linked to its DBpedia page), ii) the generated typed question, iii) the pattern matched, iv) the SPARQL query sent to the DBpedia SPARQL endpoint, and v) the answer (below the green rectangle results).</caption><thead><tr><td>#</td><td>result</td></tr></thead><tbody><tr><td>1</td><td>East River</td></tr></tbody></table>

<figure><img src="image_5.png" /><figcaption>Figure 4. QAKiS workflow</figcaption></figure>

or [Time], with “Who”, the EAT is [Person] or [Organisation] and so on.

## 4.2. Typed questions generation

We generate a *typed question* by replacing the question keywords (e.g. who, where) and the NE by the types and supertypes. Given the question “Who is the husband of Amanda Palmer?” 9 typed questions are generated, since i) both [Person] or [Organisation] (subclasses of [owl:Thing]) are considered as EAT, and ii) [MusicalArtist], [Artist] and [owl:Thing] are the types of the NE Amanda Palmer.

## 4.3. WikiFramework pattern matching

The typed questions are lemmatized, tokenized, and stopwords are removed. The same procedure is applied

to the patterns stored in the WikiFramework repository, and three sets of keywords for each relation are created, respectively for most frequent tokens, lemmas and stems. Each set contains 20 words, sorted by the frequency of presence in the collected patterns. To further improve recall, we append to the sets of keywords the tokens, lemmas and stems extracted from the CamelCase name of the relation (e.g. the tokens “birth” and “date” are added to the keywords of the relation `birthDate`). A Word Overlap algorithm is then applied to match the typed questions with such patterns for each relation. A similarity score is provided for each match: the highest represents the most likely relation.

## 4.4. Query selector

A set of patterns (max. 5) is retrieved by the pattern matcher component for each typed question, and sorted by decreasing matching score. For each of
them, one or two SPARQL queries are generated, either *i*) *select ?s where(?s <property> <NE>)*, *ii*) *select ?s where(<NE> <property> ?s)* or *iii*) both, according to the compatibility between their types and the property domain and range. Such queries are then sent to the SPARQL endpoint for answer retrieval. If the query produces no results, we try with the next pattern, until a successful query is found or no more patterns are retrieved.

In the previous sections we described step by step the complex work-flow that allowed us to build a question answering system on top of a collectively maintained encyclopedia. In the next section, we experiment the combination all those processes, and we evaluate the quality of the results on a standard data set.

## 5. Experimental evaluation

In order to evaluate our system on a comparable setting, we took part into the Question Answering over Linked Data (QALD-2)<fnref n="15" /> evaluation campaign. The challenge is aimed at any kind of QA system that mediates between a user, expressing his or her information need in natural language, and semantic data. Two data sets are provided by the organizers: *i*) a training set of 100 natural language questions annotated with corresponding SPARQL queries and correct answers, and *ii*) a test set of 100 non-annotated natural language questions, to be filled with the answers provided by the participating systems. Table 1 reports QAKiS's results on the QALD-2 data sets (DBpedia track) [5]. As introduced before, the current version of QAKiS targets only questions containing a Named Entity that is related to the answer through one property of the ontology (we are currently working on improving the algorithm to take into consideration the syntactic structure of the sentence, to tackle more complex questions). However, QAKiS performances are in line with the results obtained by the other participating systems (avg. precision, recall and F-measure on the test set are respectively 0.40, 0.42 and 0.41)¹⁶.

Most of QAKiS' mistakes concern wrong relation assignment (i.e. wrong pattern matching). We plan to replace the Word Overlap algorithm with approaches

considering the syntactic structure of the question. Another issue concerns questions ambiguity, i.e. the same surface forms can in fact refer to different relations in the DBpedia ontology (e.g. Who is the owner of Universal Studios? and Who owns Aldi? relies respectively on the relations *owner* and *keyPerson* for answer retrieval). We plan to cluster relations with several patterns in common, to allow QAKiS to search among all the relations in the cluster.

The partially correct answers concern questions involving more than one relation: the current version of the algorithm detects indeed only one of them. We plan to target questions as *Give me all people that were born in Vienna and died in Berlin* in a short time, since the two relations are easily separable and can be joint using a JOIN function in the SPARQL query. On the contrary, we need more complex strategies to answer questions with nested relations (e.g. Who is the daughter of Bill Clinton married to?), for which at the moment we answer with the name of Bill Clinton's wife (we match only the relation *spouse*). We plan short term solutions also for boolean questions, as *Is the wife of president Obama called Michelle?* (id=7, training set): we correctly match the relation *spouse* but we provide as answer *Michelle_Obama*, instead of the boolean *true*.

## 6. Related work

Concerning Wikipedia-based knowledge acquisition, distant supervision has been successfully used in various IE systems. The *Intelligence in Wikipedia* project [35] first explored this approach basing on Wikipedia and DBpedia, in order to improve infoboxes coverage for cases in which they lack expected information. In [18] the same idea is used to mine the web and automatically collect unknown relations starting from unlabeled structured data, such as tables and lists extracted from the web.

In the SW community, the University of Leipzig developed BOA [15], a system that uses information from linked data knowledge bases to extract patterns expressing such relations. The extracted patterns are then applied to web articles to extract new instances of the relation described by the patterns. This knowledge is finally put back into the knowledge base, closing the loop. In practice, BOA extends the idea underlying *Intelligence in Wikipedia* [35] to the web-scale. The distant supervision paradigm has also been implemented for different purposes, e.g. temporally anchored rela-

<footnote n="15">¹⁵http://greententacle.techfak.uni-bielefeld.de/~cunger/qald/

¹⁶The precision, recall and F-measure of the winning system [1] on the test set are, respectively, 0.44, 0.48 and 0.46.</footnote>
<table><caption>Table 4
QAKiS performances on DBpedia data sets (participation to QALD-2 [5])</caption><thead><tr><th></th><th>Precision</th><th>Recall</th><th>F-measure</th><th># answered</th><th># right answ.</th><th># partially right</th></tr></thead><tbody><tr><td><strong>train</strong></td><td>0.476</td><td>0.479</td><td>0.477</td><td>40/100</td><td>17/40</td><td>4/40</td></tr><tr><td><strong>test</strong></td><td>0.39</td><td>0.37</td><td>0.38</td><td>35/100</td><td>11/35</td><td>4/35</td></tr></tbody></table>

tion extraction [14], slot filling tasks [30] and sentiment classification [16].

Concerning Question Answering, a survey on the QA research field is provided in [22], with a focus on ontology-based QA. Moreover, they examine the potential of the open user friendly interfaces for the SW to support end users in reusing and querying the SW content. State of the art QA systems over Linked Data generally address the issue of question interpretation mapping a natural language question to a triple-based representation. For instance, Freya [7] is an interactive Natural Language Interface for querying ontologies. It uses syntactic parsing in combination with the ontology-based lookup for question interpretation, partly relying on the user's help in selecting the entity that is most appropriate as match for some natural language expression. One of the problem of that approach is that often end-users can be unable to help, in case they are not informed about the modeling and vocabulary of the data. PowerAqua [21] accepts user queries expressed in NL and retrieves answers from multiple semantic sources on the SW. It follows a pipeline architecture, according to which the question is *i*) transformed by the linguistic component into a triple based intermediate format, *ii*) passed to a set of components to identify potentially suitable semantic entities in various ontologies, and then *iii*) the various interpretations produced in different ontologies are merged and ranked for answer retrieval. The major shortcoming of PowerAqua is its limited linguistic coverage, that we address in our system exploiting the variability of relational patterns extracted from Wikipedia.

Pythia [34] relies on a deep linguistic analysis to compositionally construct meaning representations using a vocabulary aligned to the vocabulary of a given ontology. While it can handle linguistically complex questions, Pythia's major drawback is that it requires a lexicon, which up to this moment has to be created manually. It therefore fails to scale to very large datasets. More recently, Unger and colleagues [33] present an approach more similar to the one we adopt in QAKiS. Their system (based on Pythia [34]) relies on a linguistic parse of the question to produce a SPARQL template that directly mirrors the internal

structure of the question (i.e. a SPARQL template with slots that need to be filled with URIs). This template is then instantiated using statistical entity identification and predicate detection (e.g. applying string similarity as well as natural language patterns extracted from structured data and text documents). However, differently from the other two approaches mentioned before, the last one has not yet been evaluated on the standard data sets of the QALD challenge.

The QALL-ME Framework [13] is a reusable architecture for building multi-and cross-lingual QA systems working on structured data modeled by an ontology, applying a relation-based match similar to the one we proposed for QAKiS. An example of its implementation is provided for a closed domain, i.e. movies/events in the tourism domain.

While most of the approaches focus on the problem of deriving a structured query for a given natural language question, [11] addresses the problem from the opposite perspective. More specifically, they propose a preliminary approach to query verbalization, i.e. the translation of a structured query into natural language expressions that are readable and understandable by the human day-to-day user.

Beside QA, keyword-based approaches have been developed over the past years to query the SW, leading to semantic search engines. Among the others, Swoogle [9], Sigma [31], Watson [8], and Sindice [32], aim to index RDF across the Web and make it available for entity search.

## 7. Conclusions and future perspectives

With the goal of pushing forward with bridging NLP and Semantic Web, in this paper we have discussed various issues concerning the enhancement of the interactions between non-expert users and data available on the Web. In particular, we started from the observations that exploiting the interlinked structured and unstructured data sources available on the web in parallel offers a great potential for both Natural Language Processing and Semantic Web applications, and we have investigated such intuition following two (converging)
directions. On the one hand, we described the Wikiframework, i.e. a robust methodology to automatically collect relational patterns in several languages, for the relations defined in DBpedia ontology. On the other hand, we presented our ongoing work on QAKiS, a QA system over linked data, that takes advantage of the relational patterns resulting from the application of the WikiFramework approach. The results we obtained with QAKiS at the Interacting with Linked Data Challenge are in line with other systems' results, fostering future research in this direction.

Several research lines can be considered for future work. With respect to the WikiFramework, we plan to improve the pattern extraction algorithm following [38], and to extend it to collect patterns for multiple languages, for instance French and Italian. As discussed, multilingual versions of the DBpedia data set are now being released as stable versions, and can therefore be used to mine multilingual information. The variability (both cultural and of knowledge) coming from the exploitation of such multilingual and different data sets can be used to enrich the current knowledge bases (e.g. with pieces of information present for a certain language and not for another). We also plan to extend our pattern repository recursively applying and matching the patterns we collected and the instances present in DBpedia on other corpora on the web, to acquire more variability (in particular, for relations expressed in a standard way in Wikipedia such as *BirthDate*). Moreover, other social and collaborative content (e.g. *Wiktionary*<fnref n="17" />) can be mined for constructing and extending structured lexical semantic resources (following e.g. [39]). We are currently considering to publish the WikiFramework relational patterns as RDF triples, organized according to an RDF vocabulary describing the structure of the patterns (similarly to [15]).

Concerning QAKiS, we are planning to: i) investigate the applicability of the Textual Entailment approach (a framework for applied semantics, where linguistic objects are mapped by means of semantic inferences at a textual level [6]) to improve the question-pattern matching algorithm; ii) improve the system coverage, addressing boolean and n-relation questions (considering also the syntactic structure of the question); iii) explore the use of Spotlight<fnref n="18" /> [25] or OpenCalais<fnref n="19" /> tools to recognize Named Entities in the ques-

tion, and to annotate mentions of DBpedia resources in text. We are also currently exploring QAKiS applicability in real application scenarios.

## References

[1] Nitish Aggarwal and Paul Buitelaar. A system description of natural language query over dbpedia 96-99. In *Proceedings of the Interacting with Linked Data (ILD) Workshop at ESWC 2012*, Heraklion, Greece, 2012.

[2] Christian Bizer. DBpedia 3.8 released, including enlarged Ontology and additional localized Versions, August 2012.

[3] Christian Bizer, Jens Lehmann, Georgi Kobilarov, Sören Auer, Christian Becker, Richard Cyganiak, and Sebastian Hellmann. DBpedia - a crystallization point for the web of data. *Web Semant.*, 7(3):154–165, September 2009.

[4] Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. Freebase: a collaboratively created graph database for structuring human knowledge. In *SIGMOD '08: Proceedings of the 2008 ACM SIGMOD international conference on Management of data*, pages 1247–1250, New York, NY, USA, 2008. ACM.

[5] Elena Cabrio, Alessio Palmero Aprosio, Julien Cojan, Bernardo Magnini, Fabien Gandon, and Alberto Lavelli. Qakis qald-2. In *Proceedings of the Interacting with Linked Data (ILD) Workshop at ESWC 2012*, Heraklion, Greece, 2012.

[6] Ido Dagan, Bill Dolan, Bernardo Magnini, and Dan Roth. Recognizing textual entailment: Rational, evaluation and approaches. *Natural Language Engineering (JNLE)*, 15(Special Issue 04):i–xvii, 2009.

[7] Danica Damljanovic, Milan Agatonovic, and Hamish Cunningham. Freya: an interactive way of querying linked data using natural language. In *Proceedings of the 8th international conference on The Semantic Web*, ESWC'11, pages 125–138, Berlin, Heidelberg, 2012. Springer-Verlag.

[8] Mathieu d'Aquin, Enrico Motta, Marta Sabou, Sofia Angeletou, Laurian Gridinoc, Vanessa Lopez, and Davide Guidi. Toward a new generation of semantic web applications. *IEEE Intelligent Systems*, 23(3):20–28, May 2008.

[9] Li Ding, Tim Finin, Anupam Joshi, Rong Pan, R. Scott Cost, Yun Peng, Pavan Reddivari, Vishal Doshi, and Joel Sachs. Swoogle: a search and metadata engine for the semantic web. In *Proceedings of the thirteenth ACM international conference on Information and knowledge management*, CIKM '04, pages 652–659, New York, NY, USA, 2004. ACM.

[10] Luca Dini. Nlp technologies and semantic web: Risks, opportunities and challenges. *Intelligenza Artificiale*, 1(1):67–71, 2004.

[11] Basil Ell, Denny Vrandecic, and Elena Simperl. Spartiqulation: Verbalizing sparql queries. In *Proceedings of the Interacting with Linked Data (ILD) Workshop at ESWC 2012*, Heraklion, Greece, 2012.

[12] Christiane Fellbaum. Wordnet: An electronic lexical database. In *Language, Speech and Communication*, MIT Press, 1998.

[13] Óscar Ferrández, Christian Spurk, Milen Kouylekov, Iustin Dornescu, Sergio Ferrández, Matteo Negri, Rubén Izquierdo, David Tomás, Constantin Orasan, Guenter Neumann, Bernardo Magnini, and José Luis Vicedo González. The qall-me frame-

<footnote n="17">¹⁷www.wiktionary.org/</footnote>

<footnote n="18">¹⁸http://spotlight.dbpedia.org</footnote>

<footnote n="19">¹⁹opencalais.com/</footnote>
work: A specifiable-domain multilingual question answering
architecture. *J. Web Sem.*, 9(2):137–145, 2011.

[14] Guillermo Garrido, Anselmo Peñas, Bernardo Cabaleiro, and Álvaro Rodrigo. Temporally anchored relation extraction. In *ACL (1)*, pages 107–116. The Association for Computer Linguistics, 2012.

[15] Daniel Gerber and Axel-Cyrille Ngonga Ngomo. Bootstrapping the linked data web. In *Proceedings of the 1st Workshop on Web Scale Knowledge Extraction ISWC 2011*, Bonn, Germany, 2011.

[16] Alec Go, Richa Bhayani, and Lei Huang. Twitter sentiment classification using distant supervision. *Processing*, pages 1–6, 2009.

[17] Johannes Hoffart, Fabian M. Suchanek, Klaus Berberich, and Gerhard Weikum. Yago2: A spatially and temporally enhanced knowledge base from wikipedia. *Artificial Intelligence*, 2012.

[18] Raphael Hoffmann, Congle Zhang, and Daniel S. Weld. Learning 5000 relational extractors. In *ACL*, pages 286–295, 2010.

[19] Andrew Kehoe and Antoinette Renouf. Webcorp: Applying the web to linguistics and linguistics to the web. In *Proceedings of the WWW2002 Conference*, Honolulu, Hawaii, 2002.

[20] Adam Kilgarriff and Gregory Grefenstette. Introduction to the special issue on the web as corpus. *Comput. Linguist.*, 29(3):333–347, September 2003.

[21] Vanessa Lopez, Victoria S. Uren, Marta Sabou, and Enrico Motta. Cross ontology query answering on the semantic web: an initial evaluation. In *K-CAP*, pages 17–24, 2009.

[22] Vanessa Lopez, Victoria S. Uren, Marta Sabou, and Enrico Motta. Is question answering fit for the semantic web?: A survey. *Semantic Web*, 2(2):125–155, 2011.

[23] Rahmad Mahendra, Lilian Wanzare, Raffaella Bernardi, Alberto Lavelli, and Bernardo Magnini. Acquiring relational patterns from wikipedia: A case study. In *Proceedings of the 5th Language and Technology Conference*, Poznan, Poland, 2011.

[24] Pablo Mendes, Max Jakob, and Christian Bizer. DBpedia: A multilingual cross-domain knowledge base. In *Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Mehmet Ugur Dogan, Bente Maegaard, Joseph Mariani, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12)*, Istanbul, Turkey, may 2012. European Language Resources Association (ELRA).

[25] Pablo N. Mendes, Max Jakob, Andrés García-Silva, and Christian Bizer. DBpedia spotlight: shedding light on the web of documents. In *Proceedings of the 7th International Conference on Semantic Systems*, I-Semantics ’11, pages 1–8, New York, NY, USA, 2011. ACM.

[26] Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. Distant supervision for relation extraction without labeled data. In *Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2 - Volume 2*, ACL ’09, pages 1003–1011, Stroudsburg, PA, USA, 2009. Association for Computational Linguistics.

[27] Vivi Nastase and Michael Strube. Decoding wikipedia categories for knowledge acquisition. In *Proceedings of the*

23rd national conference on Artificial intelligence - Volume 2,
AAAI'08, pages 1219–1224. AAAI Press, 2008.

[28] Martin Potthast, Benno Stein, and Maik Anderka. A wikipedia-based multilingual retrieval model. In *Craig Macdonald, Iadh Ounis, Vassilis Plachouras, Ian Ruthven, and Ryen White, editors, Advances in Information Retrieval*, volume 4956 of *Lecture Notes in Computer Science*, pages 522–530. Springer Berlin / Heidelberg, 2008. 10.1007/978-3-540-78646-7_51.

[29] Fabian M. Suchanek, Gjergji Kasneci, and Gerhard Weikum. Yago: a core of semantic knowledge. In *Proceedings of the 16th international conference on World Wide Web*, WWW ’07, pages 697–706, New York, NY, USA, 2007. ACM.

[30] Mihai Surdeanu, David McClosky, Julie Tibshirani, John Bauer, Angel X. Chang, Valentin I. Spitkovsky, and Christopher D. Manning. A simple distant supervision approach for the TAC-KBP slot filling task. In *Proceedings of the Third Text Analysis Conference (TAC 2010)*, Gaithersburg, Maryland, USA, November 2010.

[31] Giovanni Tummarello, Richard Cyganiak, Michele Catasta, Szymon Danielczyk, Renaud Delbru, and Stefan Decker. Sig.ma: Live views on the web of data. *J. Web Sem.*, 8(4):355–364, 2010.

[32] Giovanni Tummarello, Renaud Delbru, and Eyal Oren. Sindice.com: weaving the open linked data. In *Proceedings of the 6th international The semantic web and 2nd Asian conference on Asian semantic web conference, ISWC’07/ASWC’07*, pages 552–565, Berlin, Heidelberg, 2007. Springer-Verlag.

[33] Christina Unger, Lorenz Bühmann, Jens Lehmann, Axel-Cyrille Ngonga Ngomo, Daniel Gerber, and Philipp Cimiano. Template-based question answering over rdf data. In *Proceedings of the 21st international conference on World Wide Web*, WWW ’12, pages 639–648, New York, NY, USA, 2012. ACM.

[34] Christina Unger and Philipp Cimiano. Pythia: Compositional meaning construction for ontology-based question answering on the semantic web. In *NLDB*, pages 153–160, 2011.

[35] Daniel S. Weld, Fei Wu, Eytan Adar, Saleema Amershi, James Fogarty, Raphael Hoffmann, Kayur Patel, and Michael Skinner. Intelligence in wikipedia. In *Proceedings of the 23rd national conference on Artificial intelligence - Volume 3*, AAAI’08, pages 1609–1614. AAAI Press, 2008.

[36] W3C RDF working group. Resource Description Framework (RDF). http://www.w3c.org/RDF/, February 2004.

[37] W3C SPARQL working group. SPARQL Query Language for RDF - W3C recommendation. http://www.w3.org/TR/rdf-sparql-query/, January 2008.

[38] Fei Wu and Daniel S. Weld. Open information extraction using wikipedia. In *Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics*, ACL ’10, pages 118–127, Stroudsburg, PA, USA, 2010. Association for Computational Linguistics.

[39] Torsten Zesch, Christof Müller, and Iryna Gurevych. Extracting lexical semantic knowledge from wikipedia and wikitionary. In *Proceedings of the Conference on Language Resources and Evaluation (LREC), electronic proceedings*. Ubiq-uitious Knowledge Processing, Universität Darmstadt, Mai 2008.