## 4 Implementation and Experiments
### Results and Evaluation
#### 4.2.2 Qualitative Results  
The qualitative evaluation was conducted using the Fleiss' kappa measure, shown in Equation (9). The Fleiss' kappa is an extension of Cohen's kappa which is a measure of the agreement between two raters, where agreement due to chance is factored out. This case, the number of raters can be more than two. As for Cohen's kappa, no weighting is used and the categories are considered to be unsorted:  
\[\kappa=\frac{\bar{p}-\bar{p_{e}}}{1-\bar{p_{e}}} \tag{9}\]  
where \(\bar{p}\) defines the actual observed agreement and \(\bar{p_{e}}\) represents chance agreement. The factor \(\bar{p}-\bar{p_{e}}\) represents the degree of agreement actually achieved above chance and the factor \(1-\bar{p_{e}}\) represents the degree of agreement that is attainable above chance. \(\kappa\) takes the value of 1 if the raters are in complete agreement.  
The results obtained by the human raters in the quantitative evaluation are analyzed with the Fleiss' kappa measure. Table 13 shows the results for each topic and for each seed URLs source (Google and Wikipedia). Table 14 shows the interpretation agreement between raters. According with these values, Wikipedia's seed URLs obtained a substantial agreement between the human raters; meanwhile, Google's seed URLs obtained a moderate agreement (computer science and politics) and substantial agreement for diabetes. The diabetes corpus was consistent in the qualitative evaluation in both cases (Wikipedia and Google).  
\begin{table}
\begin{tabular}{l c c c c c} \hline \hline \multirow{2}{*}{**Category**} & \multicolumn{5}{c}{**Levels**} \\ \cline{2-6}  & **0** & **1** & **2** & **3** & **4** \\ \hline Computer science & 44 & 1151 & 6487 & 26,730 & 79,845 \\ Diabetes & 145 & 202 & 336 & 349 & 357 \\ Politics & 63 & 1717 & 17,346 & 86,260 & 291,615 \\ \hline \hline \end{tabular}
\end{table}
Table 12: The number of Wikipedia pages by category level.  
Discussion  
In accordance with the results from Tables 10 and 11, the average HR for each topic is above 70%, that is, the accepted or downloaded web pages are relevant to the corresponding topic. The computer science and politics topics got an average HR under the 80% since both topics are broader than the diabetes topic, i.e., the computer science and politics topics contain several Wikipedia pages as is pointed out in Table 12, e.g., the fourth level for computer science contains 79,845 Wikipedia pages and the diabetes topic contains 357 Wikipedia pages at the same level.  
The SFWC relies on the proposed KRS to describe the content of a corpus from any topic. In the evaluation, the corpus size does not determine the quality of the crawling results. The quality was determined by the content of the corpus, and the selection of the most representative enriched nouns for each corpus in the KRS. For example, the diabetes corpus size is 202, and it was the topic with the best results in the quantitative and qualitative analysis. However, the computer science and politics topics could improve the results if the corpus increases their size since the number of Wikipedia pages per level has a significance difference, as it is shown in Table 12.