What ’ s Wrong with OWL Benchmarks ?

This paper is motivated by experiences in course of developing an ontology-based application within a real-world setting. When comparing these experiences with results of well-known benchmarks we found out that current benchmarks are not well suited to provide helpful hints for application developers who seek for a reasoning system matching typical real-world needs. This work aims at identifying requirements to make future benchmarks more useful for application developers. 1 On Benchmarking OWL Reasoners Reasoning with OWL Lite as well as OWL DL is known to be of high worstcase complexity. By using the “right” combination of costly expressions, one can intentionally or incidentally create even a very small ontology whose complexity will make practical reasoning impossible. On the other hand, recent implementations have shown significant progress with respect to the scalability of instance retrieval over ABoxes containing several hundred thousands of individuals. These observations cannot be seen as separate issues. Quite the contrary, they are opposing landmarks on the map of complexity results whose topology is characterized by a trade-off between expressivity and efficiency. This trade-off typically comes as a relatively sharp border between tractable and effectively un-tractable ontologies, the so called computational cliff, and is caused by an increase in language expressiveness [1]. Although a worst-case result, in practice this behavior is often observed with tableaux based systems when adding language constructs that prohibit the use of highly efficient optimization techniques (i. e. inverse roles, nominals, etc.). Eventually, a multiplier to this effect is the size of the ABox. Having knowledge about the rough landscape of practically tractable ontologies as well as reasoning approaches for particular query requirements is of fundamental importance when building real-world applications. By real-world we mean ontology-based applications with an expressivity beyond ALC, containing thousands of individuals and an inference response time of less than a second, even in a dynamical setting. For instance, context-aware applications want to offer services to users based on their actual situation. Experiences in the course of operating a context-aware application for mobile users [2] clearly have shown that the quality of this application significantly depends on the availability of scalable reasoning services able to deal with dynamic data. Within this application user situations are computed by applying Description Logic reasoning to classify qualitative context elements gathered from various sources and represented as OWL individuals [3]. Since changes in the user’s environment occur frequently the corresponding ABoxes have to be updated rather often. Unfortunately, none of the current benchmarks or system comparisons either draw a clear picture of effectively tractable language fragments or ABox sizes, give insights into pros and cons of different reasoning approaches, or consider technical issues such as serialization, interfaces, or ABox updates. For instance, many benchmarks consist of synthetical generated and sparsely interrelated data using inexpressive ontology languages such as the widely used Lehigh University Benchmark (LUBM) [4]. In case of the LUBM an incomplete query answering procedure exploiting told ABox information from a classified TBox is sufficient to answer most of the given queries correctly. Obviously, this cannot be considered as a meaningful benchmark and consequently has let to exceptional performance for inherently incomplete reasoning systems. On the other hand, a demonstration of the steep slope of the computational cliff is the University Ontology Benchmark (UOBM) [5], which is a direct extension of the LUBM. The UOBM extends the LUBM by adding extra TBox axioms making use of all of OWL Lite (UOBM Lite) and OWL DL (UOBM DL). In addition, the ABox is enriched by interrelations between individuals of formerly separated units, which then requires ABox reasoning to answer the set of given UOBM queries. Not surprisingly, it turned out that incomplete systems such as OWLIM, DLDB, or Minerva now can only answer a fraction even of the OWL Lite queries completely. Only one theoretically sound and complete approach, namely Pellet [6], was able to handle about a tenth of the number of individuals compared to the LUBM within our time limit. All other systems failed either with a timeout or due to the lack of memory. Furthermore, discussing inherent drawbacks and advantages of different approaches with respect to diverse application tasks has been largely neglected in recent benchmarks or system comparisons. However, application developers need to be aware of the trade-off when using a rule-based materialization approach such as in OWLIM [7], a Datalog-driven engine such as KAON2 [8], or tableau-based reasoners such as Pellet, RacerPro [9] or FaCT++ [10]. To give an example, a materialization approach has to exhaustively compute and explicitly store all implicit information of an ontology typically at load-time. On the contrary, traditional tableau-based systems will explicitly compute such information only on demand. A Datalog transformation approach, on the other hand, is known to perform worse in the presence of numbers greater than two in cardinality restrictions. Therefore, a serious benchmark has to discuss its results with respect to different approaches in order to provide helpful hints to users. Another performance related issue deals with the way of feeding the systems with large amounts of data. Our selective tests have shown that for some systems not only the transmission format (RDF/XML or DIG [11]) is of importance, but also the way data is encoded (e. g. deep vs. flat serialization). A real-world requirement which has not been taken into account in any benchmark so far is concerned with dynamic data. The ABox is not only expected to be the largest part of an ontology but is also subject to frequent changes. In order to serve as an efficient reasoning component within a realistic setting it is necessary to perform well under small ABox updates. First results in this research direction, e. g. [12], need to be evaluated by appropriate benchmarks. Finally, all benchmark results need to be weighted with respect to soundness and completeness of the underlying inference procedure. Assuming that soundness and completeness is an indispensable requirement for knowledge-based applications — of which we think it is — many of the existing benchmark results are not helpful at all. Some of our randomly selected tests showed that even systems assumed to implement a sound and complete calculus fail on a number of OWL Lite test cases. In fact, only RacerPro and KAON2 passed all tests of the OWL fragment they claim to support completely. The purpose of this paper is to identify weaknesses of current OWL benchmarks and system evaluations. In the following section, we analyze existing benchmark suites, discuss corresponding results and compare them with our own tests. As a result we compiled a collection of requirements (Section 3) to make future benchmarks more useful for application developers. Section 4 summarizes the findings presented in this report. 2 Benchmarking Experiences This section tries to roughly draw a picture of practically tractable OWL repositories with current reasoning systems. This is done by gathering data from different existing as well as own benchmarks. The collected results are reviewed with respect to the system, i. e. the underlying approach, as well as the kind of test ontologies. Our overall goal is to qualitatively analyze various benchmark suites and results in order to identify requirements for a comprehensive benchmark suite suitable to allow ontology-based application developers to pick the right system for their individual task. A common benchmark for today’s DL reasoners is to measure the performance in processing huge ontologies. Ontologies with relatively small and inexpressive TBoxes and ABoxes containing hundreds of thousands or even millions of individuals and relations are predominantly used in benchmarking tests. It is assumed that real world applications will also exhibit the described characteristics. A selection of such “real world ontologies” (e.g. Gene Ontology or Airport Codes Ontology) which are used for benchmarking can be found in [13]. Nowadays, the Lehigh University Benchmark (LUBM) is the de facto standard when it comes to reasoning with large ontologies [14,15,6,16,17]. But as mentioned before, many reasoners that achieved excellent results when benchmarked with LUBM, failed to reason about other ABox or TBox tests (cf. results 3 http://archive.godatabase.org/ 4 http://www.daml.ri.cmu.edu/ont/AirportCodes.daml for OWLIM and KAON2 from Sections 2.1 and 2.2). We therefore experimented with a set of tests of a different kind using both existing and newly created benchmarks. In the following, we will present some of these measurements, whereas the aim of these tests was not to simply nominate the fastest or most reliable reasoner. Instead of overloading this report with a complete set of all of our measurements we will highlight some results that demonstrate the necessity and requirements for a comprehensive benchmark that goes beyond the LUBM. In the following, we will present selected results for KAON2 (built 05-122005), Pellet 1.3, OWLIM 2.8. 3, and RacerPro 1.9. 0. We divided the whole process of loading an ontology, any preprocessing as applicable and processing of a query into two separate measurement stages: Loading and preprocessing the ontology This stage summarizes the measurements for the process of feeding the ontology into the reasoner and any parsing as required by the interface. Also any preprocessing that is either done automatically or can be started manually is included into this measure. Query processing This stage measures t