# Metrics-Based Comparison of OWL and XML for Representing and Querying Cognitive Radio Capabilities
## 5 Evaluation
### Evaluation Process  
A high-level representation of the evaluation process is shown in Figure 2. This section describes the evaluation process in more detail. Figure 3 shows the concrete data flow in the evaluation process with the notations introduced in Section 4.1. Knowledge about a set of devices \(D\) is modeled as the most complete ontology \(B_{0}\). The method first automatically generates device descriptions in RDF/XML [43] from \(B_{0}\) using RODG. Since RDF/XML constitutes a base for OWL, while at the same time its syntax is XML, the device descriptions in RDF/XML can serve as data representations of \(D\) for both the OWL-and XML-based approaches denoted by \(D^{owl}\) and \(D^{xml}\), respectively. Next, the method generates query expressions, \(Q^{owl}\) and \(Q^{xml}\), and requests about device capabilities for the two approaches. This is achieved through two procedures. First, the method automatically generates a set of random SPARQL queries as \(Q^{owl}\) from \(B_{0}\) and \(D^{owl}\) using SQG. Then, the set of XQuery queries \(Q^{xml}\) is obtained by converting a SPARQL query \(Q^{owl}\) into an equivalent XQuery query in \(Q^{xml}\) using the open source component named xql2 xquery [32; 33].  
The queries are then submitted to the \"matchers\". The method uses DeVISor as the OWL-based matcher and BaseX as the XML-based matcher. The matchers include device registration. Note that, in the OWL-based approach, apart from the device descriptions, a background knowledge test (\(B_{i}\in B\), \(1\leq i\leq k\)) needs to be loaded into the OWL-based matcher (DeVISor) to derive inferred facts. Then, the processing of each query is executed for both approaches by the implemented query functions \(f^{owl}\) and \(f^{xml}\).  
The matching results of each query expression returned from the matchers are evaluated by comparing them with the ground truth of the matching devices against the query. This is achieved by implementing all metric functions of the two approaches (\(m^{xml}\) and \(m^{owl}\)). The implementation of the metric functions is shown in Algorithms 1-3. Additionally, the method collects metrics on the performance during the matching process. Finally, the metric values are derived and stored in an external file for further analysis.  
Figure 3: Data flow diagram of the evaluation process with the notations.  
```
Input:\(D_{m}\): matching devices of a query; \(D_{g}\): Ground truth of the query Output:\(m_{s}\): precision metric value \(\in[0,1]\) if\(|D_{m}|\
eq 0\)then
2\(count\gets 0\) foreach\(D_{i,m}\in D_{m}\)do
3 if\(D_{i,m}\in D_{g}\)then
4\(count\gets count+1\)
5
6\(m_{c}\leftarrow\frac{count}{|D_{g}|}\)
7 elseif\(|D_{m}|=0\)then
8\(m_{c}\gets 1\)
9 else
10\(m_{c}\gets 0\)
```  
**Algorithm 1**CALC-RECALL(\(D_{m}\), \(D_{g}\)).  
```
Input:\(D_{m}\): matching devices of a query; \(D_{g}\): Ground truth of the query Output:\(m_{s}\): precision metric value \(\in[0,1]\) if\(|D_{m}|\
eq 0\)then
1\(count\gets 0\) foreach\(D_{i,m}\in D_{m}\)do
2 if\(D_{i,m}\in D_{g}\)then
3\(count\gets count+1\)
4\(m_{s}\leftarrow\frac{count}{|D_{m}|}\)
5 elseif\(|D_{g}|=0\)then
6\(m_{s}\gets 1\)
7 else
8\(m_{s}\gets 0\)
```  
**Algorithm 2**CALC-PRECISION(\(D_{m}\), \(D_{g}\)).  
```
Input:\(m_{s}\): precision metric value \(\in[0,1]\); \(m_{c}\): recall metric value \(\in[0,1]\) Output:\(m_{f}\): F-measure metric value \(\in[0,1]\) if\(m_{s}\
eq 0\)or\(m_{c}\
eq 0\)then
9\(m_{f}\leftarrow\frac{2\times m_{s}\times m_{c}}{m_{s}+m_{c}}\)
10 else
11\(m_{f}\gets 0\)
```  
**Algorithm 3**CALC-F-MEASURE(\(m_{s}\), \(m_{c}\)).  
Some more details about the implementation of the evaluation process are presented in the next section.