# Metrics-Based Comparison of OWL and XML for Representing and Querying Cognitive Radio Capabilities
## 5 Evaluation
### Experiments
#### 5.5.2 Evaluation Results and Analysis  
In the results of the experiments, the ground truths of most of the requests are empty, which indicates that no matching results existed for these requests. This was expected, since device descriptions and queries were generated randomly and independently. What we were most interested in was how OWL inference affects the quality of the query results. We expected to see differences in the matching results of the two approaches. The queries that did not return any matches were not used in the comparisons of the two approaches. This section shows the quality metrics and analysis. The raw data for the metrics results are available online ([[https://github.com/YankeeChen/evaluator/tree/master/evaluation/results](https://github.com/YankeeChen/evaluator/tree/master/evaluation/results)]([https://github.com/YankeeChen/evaluator/tree/master/evaluation/results](https://github.com/YankeeChen/evaluator/tree/master/evaluation/results)), accessed on 15 November 2022). Figure 5a-c show the average recall, precision, and F-Measure metrics of the two approaches for each of the four batches of queries. For each batch, six results are shown. The five bars show, from left to right, the associated metrics for the OWL-based approach with the progressively less complete test ontologies (\(B_{1}\) to \(B_{5}\), from left to right). The rightmost bar shows the associated metric obtained using the XML-based approach. \begin{table}
\begin{tabular}{c c c c c c} \hline
**Axiom Type** & \(B_{0}\) & \(B_{1}\) & \(B_{2}\) & \(B_{3}\) & \(B_{4}\) \\ \hline SubClassOf & ✓ & ✓ & ✓ & & \\ EquivalentClasses & ✓ & ✓ & ✓ & ✓ & ✓ \\ DisjointClasses & ✓ & ✓ & ✓ & ✓ & ✓ \\ DisjointUnion & ✓ & ✓ & ✓ & ✓ & ✓ \\ SubObjectPropertyOf & ✓ & ✓ & & & \\ EquivalentObjectProperties & ✓ & ✓ & ✓ & ✓ & ✓ \\ DisjointObjectProperties & ✓ & ✓ & ✓ & ✓ & ✓ \\ InverseObjectProperties & ✓ & ✓ & ✓ & ✓ & ✓ \\ ObjectPropertyDomain & ✓ & ✓ & ✓ & ✓ & ✓ \\ ObjectPropertyRange & ✓ & ✓ & ✓ & ✓ & \\ FunctionalObjectProperty & ✓ & ✓ & ✓ & ✓ & ✓ \\ SymmetricObjectProperty & ✓ & ✓ & ✓ & ✓ & ✓ \\ TransitiveObjectProperty & ✓ & ✓ & ✓ & ✓ & ✓ \\ SubDataPropertyOf & ✓ & & & & \\ EquivalentDataProperties & ✓ & ✓ & ✓ & ✓ & ✓ \\ DataPropertyDomain & ✓ & ✓ & ✓ & ✓ & ✓ \\ DataPropertyRange & ✓ & ✓ & ✓ & ✓ & ✓ \\ FunctionalDataProperty & ✓ & ✓ & ✓ & ✓ & ✓ \\ ClassAsertion & ✓ & ✓ & ✓ & ✓ & ✓ \\ ObjectPropertyAssertion & ✓ & ✓ & ✓ & ✓ & ✓ \\ DataPropertyAssertion & ✓ & ✓ & ✓ & ✓ & ✓ \\ \hline \end{tabular}
\end{table} 
Table 4: Axiom coverage of the ontologies. For the recall metric, we conclude that the XML-based approach provides matching results that are approximately 34% less complete than those produced with the OWL-based approach with the relatively most complete test ontology \(B_{1}\). The result is roughly the same regardless of the number of devices. For instance, the average recall metric value of 5000 queries against 4000 devices in the XML-based approach is 65.75%, which is 34.25% lower than that produced with \(B_{1}\), and the results are about the same for other batches. It can also be observed from the results that the XML-based approach derives exactly the same results as the OWL-based approach with the least complete ontology \(B_{5}\) for each batch. This is as expected, since \(B_{5}\) does not include any OWL axioms except for vocabularies (see in Table 4), and thus no new facts can be derived by the OWL reasoner. Therefore, it derives exactly the same matching devices as the XML-based approach. When it comes to the comparison of the OWL-based approach with different versions of test ontologies, as expected, a more specific ontology (more precise definitions of classes) provides more power (more axioms) to the OWL reasoner, which results in more complete matching. Figure 5: Metrics evaluation results (**a**) Average recall comparison; (**b**) Average precision comparison; (**c**) Average F-Measure comparison; (**d**) Device registration time comparison; (**e**) Average query response time comparison.