## 4 Results
### Accuracy results versus LIDC data subsets  
By applying the active-DECORATE to the new LIDC dataset (Tables 4 and 5), the classification accuracy was on average 70.48% (Table 6) with an average number of iterations equal to 37 and average number of instances added at each iteration equal to 123. The results were substantially lower than on the previous available LIDC dataset (LIDC85-only 85 cases out of which only 60 cases were rated by at least one radiologist) for which the average accuracy was 88.62%. Looking at the ratings distributions of the training datasets (nodules on which at least three radiologists agree) for the LIDC and LIDC85 datasets (Table 5), we noticed that the distributions for the LIDC dataset were strongly skewed in the direction of one dominant class for almost each characteristic and therefore, produced unbalanced datasets when experiment \begin{table}
\begin{tabular}{c c c c} \hline \hline  & **LIDC** & **LIDC85** & **LIDC\_B** \\ \hline Instances & 2,204 & 379 & 912 \\ Nodules & 914 & 149 & 542 \\ Cases/Patients & 207 & 60 & 179 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: LIDC datasets overview; LIDC_B is a balanced data set. To validate the effect of the unbalanced data on the accuracy of the classifier, we evaluated further the ensemble of classifier on another balanced dataset. The second subset (LIDC_B) was formed by randomly removing nodules from the most dominant class/rating such that the most dominant class has almost the same number of nodules as the second most dominant class. Furthermore, when comparing our proposed approach with the traditional decision trees applied as single classifiers per characteristic, our approach notably outperforms the traditional approach by 24% to 45% accuracy, depending on the characteristics of the data subsets (Table 7). While all of the data instances were involved in the creation of both the decision trees and the ensemble from Tables 6 and 7, we also wanted to test further the performance of our algorithm on \"unseen\" data. We reserved 10% of our data set to be completely unavailable (\"unseen\") for the creation of the classifiers. This 10% was chosen to be similar to the entire data set with respect to \begin{table}
\begin{tabular}{l l l l l l l l l l} \hline \hline  & \multicolumn{3}{c}{**LIDC**} & \multicolumn{3}{c}{**LIDC85**} & \multicolumn{3}{c}{**LIDC\_B**} \\  & \multicolumn{3}{c}{**(80\% Confidence level)**} & \multicolumn{3}{c}{**(60\% Confidence level)**} & \multicolumn{3}{c}{**(80\% Confidence level)**} \\ \hline \multirow{3}{*}{**Characteristics**} & **\#of** & **\#of** & **Accuracy** & **\#of** & **\#of** & **Accuracy** & **\#of** & **\#of** & **Accuracy** \\  & **ITR** & **IAL** & & **ITR** & **IAL** & & **ITR** & **IAL** & \\ \hline Lobulation & 68 & 196 & 54.53\% & 10 & 1 & 81.00\% & 33 & 24 & 83.56\% \\ Malignancy & 18 & 136 & 89.89\% & 8 & 1 & 96.31\% & 12 & 170 & 89.38\% \\ Margin & 34 & 112 & 75.67\% & 5 & 8 & 98.68\% & 16 & 6 & 93.58\% \\ Sphericity & 33 & 49 & 87.47\% & 9 & 9 & 91.03\% & 23 & 24 & 85.86\% \\ Spiculation & 30 & 117 & 50.17\% & 15 & 13 & 63.06\% & 34 & 33 & 82.5\% \\ Subtlety & 29 & 86 & 81.73\% & 7 & 4 & 93.14\% & 18 & 55 & 93.93\% \\ Texture & 44 & 163 & 53.9\% & 9 & 0 & 97.10\% & 11 & 61 & 95.83\% \\
**Average** & **36.6** & **122.7** & **70.48\%** & **9** & **5.14** & **88.62\%** & **21** & **53** & **89.23\%** \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Classification accuracies of the ensemble of classifiers built using decision trees; the number of classifiers ( _Csize_ ) was set to 10 and number of artificially generated examples ( _Rsize_ ) to 1; #of ITR stands for number of iterations of Active-DECORATE, and #of IAL stands for number of instances added to the training data later (those that did not reach the confidence threshold).