Competency Questions and SPARQL-OWL Queries Dataset and Analysis

Abstract: Competency Questions (CQs) are natural language questions outlining and constraining the scope of knowledge represented by an ontology. Despite that CQs are a part of several ontology engineering methodologies, we have observed that the actual publication of CQs for the available ontologies is very limited and even scarcer is the publication of their respective formalisations in terms of, e.g., SPARQL queries. This paper aims to contribute to addressing the engineering shortcomings of using CQs in ontology development, to facilitate wider use of CQs. In order to understand the relation between CQs and the queries over the ontology to test the CQs on an ontology, we gather, analyse, and publicly release a set of 234 CQs and their translations to SPARQL-OWL for several ontologies in different domains developed by different groups. We analysed the CQs in two principal ways. The first stage focused on a linguistic analysis of the natural language text itself, i.e., a lexico-syntactic analysis without any presuppositions of ontology elements, and a subsequent step of semantic analysis in order to find patterns. This increased diversity of CQ sources resulted in a 5-fold increase of hitherto published patterns, to 106 distinct CQ patterns, which have a limited subset of few patterns shared across the CQ sets from the different ontologies. Next, we analysed the relation between the found CQ patterns and the 46 SPARQL-OWL query signatures, which revealed that one CQ pattern may be realised by more than one SPARQL-OWL query signature, and vice versa. We hope that our work will contribute to establishing common practices, templates, automation, and user tools that will support CQ formulation, formalisation, execution, and general management.

Within the field of ontology engineering, Competency Questions (CQs) BIBREF0 are natural language questions outlining the scope of knowledge represented by an ontology. They represent functional requirements in the sense that the developed ontology or an ontology-based information system should be able to answer them; hence contain all the relevant knowledge. For example, a CQ may be What are the implementations of C4.5 algorithm?, indicating that the ontology needs to contain classes, such as Algorithm and C4.5 as subclass of Algorithm, and something about implementations such that the answer to the CQ will be non-empty.
CQs are a part of several ontology engineering methodologies, yet the actual publication of CQs for the available ontologies is rather scarce. Even more scarce is the publication of the CQs' respective formalisation in terms of, e.g., SPARQL queries. This suggests CQs are not used widely as intended. We hypothezise that it may be due to the lack of common practices, templates, automation, and user tools that would support CQ formulation, formalisation, execution, and general management; or: it is still a fully manual process. For instance, even if one has specified CQs, there is no automatic way to translate it to, say, a SPARQL-OWL BIBREF1 query (for validation and verification), and not even a systematic manual way either.
There have been few attempts to analyse CQs. Ren et al. BIBREF2 analysed CQs and their patterns to determine CQ archetypes, as tried BIBREF3 . Those patterns have a limited coverage, however, for they are based on CQ sets of at most two ontologies (Pizza and Software), which thus may contain domain bias, CQ author bias, and `prejudiced' patterns as the Pizza CQs were created after the ontology. As simple example of the latter issue, one could create a CQ Which pizza has hot as spiciness? that neatly fits with Pizza's hasSpiciness data property, or a more natural phrase Which pizzas are hot? that is fully agnostic of how it is represented in the ontology, be it with a data property, object property, or a class. More generally, it suggests that Ren et al.'s CQ patterns, formulated alike “Which [CE1] [OPE] [CE2]?”, may not be appropriate as CQ pattern, as it presupposes which kind of element it would be in an ontology. The manual process and `free form' formulation of CQs by domain experts also runs onto problems that some turn out not translatable into a test over the ontology for various reasons. For instance, the CQ How can I get problems [with X] fixed? of the Software Ontology cannot be answered by a declarative specification that the ontology is, or take the CQ for the DMOP ontology BIBREF4 : Given a data mining task/data set, which of the valid or applicable workflows/algorithms will yield optimal results (or at least better results than the others)?: assuming that the question may deal with an arbitrary (not pre-defined upfront) dataset, this CQ may only be answered via performing data mining experiments and not by the ontology itself. Therefore, without a clear guidelines of what kind of CQs may be meaningfully expressed and used as requirement specification for an ontology's content, their uptake and usage likely will remain limited. This paper aims to contribute to addressing the engineering shortcomings of using CQs in ontology development.
To clear up the CQ muddle and trying to understand the relation between CQs and the queries over the ontology to test the CQs on an ontology, we gather, analyse, and publicly release a larger set of CQs and their translations to SPARQL-OWL for several ontologies in different domains developed by different groups. For the analysis in particular, we seek to address the following research questions:
A total of 234 CQs for 5 ontologies have been collected and translated into SPARQL-OWL queries, and made available as a data resource. We analysed them in two principal ways. The first stage focused on a linguistic analysis of the natural language text itself, i.e., a lexico-syntactic analysis without any presuppositions of ontology elements, and a subsequent step of semantic analysis. This revealed 17 CQ patterns at the natural language layer. While a few patterns occur in multiple CQ sets, there are also patterns unique to a CQ set, supporting the expectation that a broad sampling is required to obtain a more representative set of patterns. The second phase consists of designing SPARQL-OWL queries for all CQs, where possible, and examining the signature of the queries. We found 46 query signatures resulting from the collected 131 SPARQL-OWL queries. The third step consists of the analysis of the relation between the CQ patterns and the SPARQL-OWL query signatures. This is, as hypothesised, a INLINEFORM0 : INLINEFORM1 relation, or: one CQ pattern may be realised by more than one SPARQL-OWL query and there may be more than one CQ pattern for a SPARQL-OWL query signature.
The remainder of the paper is structured as follows. We first discuss related works on CQs and CQ patterns in Section SECREF2 . Section SECREF3 is devoted to the linguistic analysis of CQs and Section SECREF4 to the generation and analysis of the SPARQL-OWL queries. We discuss and return to the research questions in Section SECREF5 and conclude in Section SECREF6 . The data is available from a Git repository at https://github.com/CQ2SPARQLOWL/Dataset.
The aim of the analysis of the CQs is to examine whether there are some popular linguistic structures that can be reused to specify requirements for, and validate, new and existing ontologies. This section describes the collection of the materials, the methods, and subsequently the results of the CQ analysis.
We describe and motivate the materials first and then proceed to the methods and motivations thereof.
There are multiple ontologies available over internet with competency questions provided, but since the focus of our research is on SPARQL-OWL queries, we selected only those ontologies with CQs stated against ontology schema (T-Box). As a result we selected 5 ontologies with 234 competency questions in total. Table TABREF8 summarizes our dataset size and source of each ontology.
The Software Ontology (SWO) BIBREF5 is included because its set of CQs is of substantial size and it was part of Ren et al.'s set of analysed CQs. The CQ sets of Dem@Care BIBREF8 and OntoDT BIBREF9 were included because they were available. CQs for the Stuff BIBREF6 and African Wildlife (AWO) BIBREF7 ontologies were added to the set, because the ontologies were developed by one of the authors (therewith facilitating in-depth domain analysis, if needed), they cover other topics, and are of a different `type' (a tutorial ontology (AWO) and a core ontology (Stuff)), thus contributing to maximising diversity in source selection.
In this section, we carry out and examine the `translation' of CQs to a form that can be evaluated against an ontology.
As first preliminary observation, we observe that an OWL ontology can be serialized as an RDF/XML graph BIBREF10 and thus queried using SPARQL Query Language BIBREF11 . In its base form SPARQL is basically a pattern matching language and as such does not provide any reasoning capabilities; however, it is possible to introduce these by using SPARQL Entailment Regimes BIBREF12 . In particular, we employ OWL 2 Direct Semantics Entailment Regime. Intuitively, it allows us to construct a SPARQL query such that its WHERE clause contains OWL axioms, possibly with some of its IRIs and literals replaced by SPARQL variables. The results of the execution of such a query are all the variable mappings such that the axioms obtained by applying these mapping to the axioms in the query, are entailed by the queried ontology. SPARQL, being a query language for RDF, employs Turtle syntax BIBREF13 to express Basic Graph Patterns (BGPs) and this convention is kept also for expressing OWL axioms, i.e., their RDF representation is used BIBREF10 . This is consistent with how the only available implementation behaves BIBREF1 , BIBREF14 .
The second preliminary comment is that we note that, unlike Dennis et al. BIBREF15 's claim, CQs do not have to have specific presuppositions other than vocabulary, but queries do, for it is the queries that are specific to the ontology and the modelling style used and other modelling decisions made. We can make this distinction here, because of the separation of concerns between the linguistics of the CQs on the one hand and the queries and ontology how it it realised on the other hand, rather than having the two combined as in BIBREF3 , BIBREF2 , BIBREF15 .
This work was partly supported by the Polish National Science Center (Grant No 2014/13/D/ST6/02076). Jedrzej Potoniec acknowledges support from the grant 09/91/DSPB/0627.

Figure and table captions:
Table 1: Competency questions dataset summary
Table 3: Replacing complex entity expressions with single identifier.
Table 2: Normalization of words into common forms. REMOVED means that given text is deleted from pattern.
Table 4: Number of pattern candidates and actual patterns
Table 5: Patterns that are shared by CQ sets of multiple ontologies.
Table 6: Higher Level Patterns that occur in more than one CQ set.
Figure 1: Average CQs covered by a single pattern per given CQ set of an ontology.
Figure 2: Average CQs covered by a single higher level pattern per given CQ set of an ontology.
Table 7: Prefixes used thruought the paper in the SPARQL-OWL queries
Table 8: Translatability of competency questions .
Table 9: Keywords usage among SPARQL-OWL queries .
Table 10: The signatures that are common for at least three queries in the dataset. In the column Ontologies listed are ontologies from which the queries originated along with the number of queries having that signature.
Table 11: The signatures that are shared between queries coming from at least two ontologies. In the column Ontologies listed are ontologies from which the queries originated along with the number of queries having that signature.
Table 12: Frequent signal phrase with most frequently SPARQL-OWL queries cooccurrences count .
Table 13: Signal words .

Questions about this paper:
How many domains of ontologies do they gather data from?

Answers:
5 domains: software, stuff, african wildlife, healthcare, datatypes