## 6 System Evaluation
### Artificial Data Based Evaluation  
Given the shortage of real data, we consider the decision trees designed by the experts to be the golden knowledge (ground truth) of our domain problem. So, to evaluate our system, we did the following:  
Using the provided data from the experts (decision trees, symptoms per disease, and their weights), we generated an artificial dataset of test cases. Since real cases were not available, this dataset was crucial to test and implement additional ranking methods based on machine learning techniques and to evaluate all the implemented methods. More specifically, the dataset generation algorithm generates random combinations of the input values (symptoms and environmental parameters), and for each one of them, it applies the blocking rules to isolate the valid diseases. If there is more than one possible disease available, the algorithm runs two ranking methods (based on the similarity of the symptoms and their assigned weights) and combines the results into a single score value. Then, it randomly selects one of the diseases, considering their scores (the disease with the higher score has the highest probability of being selected, but it is not guaranteed). The algorithm continues to generate cases until the specified maximum number of instances has been reached for all the diseases. The final dataset has 1793 cases, each one related to 36 features (diagnostic parameters) plus the class feature (disease). Using a subset of the generated dataset (50%), we trained a rule-based expert system with certainty factors (CFs) using the tool and methodology of ACRES. The implemented Expert System can take as input a case and return a ranked list of the diseases, accompanied by a CF value. Thus, the overall system for preliminary diagnosis based on symptoms, given a new case, isolates the possible diseases and performs three separate ranking approaches (case-based, weighted, and with the trained ACRES expert system) that are combined into a final one via a voting method. The entire generated dataset was used to evaluate the system. Since the overall diagnostic process results in the three most probable diseases, we are interested in evaluating whether the process includes the right disease within that result. In Table 6, we present the evaluation results for each one of the ranking methods. The 'Accuracy' column shows the rate of the dataset cases that were correctly classified by their disease (the disease was the first one in the ranked list). The 'Top-3 Accuracy' column shows the rate of cases in which the disease was ranked in one of the first three positions, whereas the 'Average Rank' column shows the average rank of the correct disease in the list of predictions (a lower average rank indicates better performance). The results in bold indicate that the aggregate method performed better than the individual ranking methods. They also show that although ACRES is better at diagnosing the right disease as first on the list, it is not in the top 3 cases where CBO does better. Table 7 displays the results of the aggregate ranking for each one of the individual diseases. It shows (results in bold) that 'pasteridiasi' is the most difficult to diagnose disease in both cases, as first and in the top 3. This is consistent with the fact that its decision tree is one of the most complicated ones. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Method** & **Accuracy** & **Top-3 Accuracy** & **Average Rank** \\ \hline CBO & 0.511 & 0.919 & 1.825 \\ \hline WBM & 0.435 & 0.897 & 1.982 \\ \hline ACRES & 0.622 & 0.909 & 1.743 \\ \hline AGM & **0.658** & **0.940** & **1.569** \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Methods evaluation results. It is important to note that the CBO and WBM approaches perform similarly among different randomly generated datasets. On the other hand, the ACRES expert system (ES) performance depends on the similarity of the test dataset with the training dataset used to train the ES.