## 4 Results
### Ensemble of classifiers' predictions versus expert panel agreement  
Furthermore, we measured the agreement between the panel of experts and our ensemble of classifiers using both kappa and weighted kappa statistics for different levels of agreement. The results (Table 12) show that higher levels of agreement yield higher kappa statistics. Furthermore, we noticed that weighted kappa statistics better captured the level of agreement than the non-weighted kappa statistic across different reference truths in the sense of being more consistent when going from one level of agreement to another. With the exception of spiculation and texture, the weighted kappa statistics for all the other five characteristics for the entire LIDC dataset showed that the ensemble of classifiers was in'moderate' agreement or better ('substantial' or 'almost perfect') with the LIDC panel of experts when there were at least three radiologists who agreed on the semantic characteristics. Furthermore, when analyzing these five semantic characteristics with respect to the other two reference truths, we learned that the ensemble of classifiers was in 'fair' or'moderate' agreement with the panel of experts. \begin{table}
\begin{tabular}{l c c c c c c} \hline  & \multicolumn{3}{c}{**SVM**} & \multicolumn{3}{c}{**SVM ensemble**} \\ \cline{2-7}  & **full** & **original** & **reserved** & **Full** & **original** & **reserved** \\ \hline Lobulation & 60.02\% & 63\% & 55.02\% & 67.64\% & 69.87\% & 66.98\% \\ Malignancy & 50.45\% & 51.28\% & 36.15\% & 77.49\% & 78.16\% & 62.91\% \\ Margin & 45.68\% & 45.69\% & 17.51\% & 63.83\% & 61.9\% & 37.32\% \\ Sphericity & 42.96\% & 42.16\% & 33.18\% & 64.15\% & 64.45\% & 53.98\% \\ Spiculation & 69.23\% & 69.54\% & 56.54\% & 80.8\% & 80.42\% & 59.49\% \\ Subtlety & 45.64\% & 45.24\% & 24.19\% & 66.28\% & 66.41\% & 54.83\% \\ Texture & 73.69\% & 75.98\% & 57.51\% & 88.92\% & 89.06\% & 69.94\% \\
**Average** & **55.38**\% & **55.78**\% & **40.01**\% & **72.73**\% & **72.89**\% & **57.92**\% \\ \hline \end{tabular}
\end{table} 
Table 11: Classification Accuracy of support vector machines on full, original and reserved data sets (single classifier _vs._ ensemble of classifiers). Figures 6-12 present a visual overview of the ensemble of classifiers' agreement with the panel of experts' opinions. In this visualization, we were interested not only in the \"absolute\" accuracy of the classifier, but also in how the classifier did with regard to rater disagreement. For each semantic characteristic, we have displayed four graphs. Each one of these graphs corresponds to a distinct number of raters. That is, we show one graph for nodules rated by one radiologist (upper left graph in each figure), one graph for nodules rated by two radiologists (upper right graph in each figure), one graph for nodules rated by three radiologists (lower left graph in each figure) and one graph for nodules rated by four radiologists (lower right graph in each figure). In each graph, we have a bar corresponding to the number of radiologists which our algorithm predicted correctly. (Thus the graphs with more radiologists have more bars.) The height of the bars shows how many nodules there were in each level of prediction success. Looking at just the height of these bars, we can see that our classifier's success was quite good with respect to most of the semantic characteristics - these characteristics present very right-skewed distributions. Lobulation, spiculation and texture present more uniform distribution, meaning our classifier was less successful at predicting the radiologists' labels. We present one further visualization in these graphs-each bar is gray-coded to indicate the radiologists' level of agreement among themselves. (Thus, for example, the upper left graph, one radiologist, has no gray-coding, as a radiologist will always agree with himself.) This gray-coding allows us to see that the approach is much better at matching radiologists when the radiologists agree with themselves.