## 3 Experiment and Results
### Experiment 4: Whole Medial Tibia Cartilage Prediction  
In this experiment, we combined the feature space of the three experiments above to train a single model for the whole medial tibia compartment. For CDI, all the nine CDI locations were included and for manual segmentation, all locations from the three slices were used to generate samples. Table 7 listed the distribution of the 'change' and 'no-change' categories for the whole dataset. Table 8 summarized the prediction performance and Figure 6 plotted the ROC curves of the four models. CDI still achieved better performance than manual segmentation in both the raw dataset and the rebalanced dataset. Rebalancing strategies helped improve the performance of CDI to AUC = 0.720 but did not help the manual segmentation. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multicolumn{2}{c}{**Quantification Method**} & \multicolumn{2}{c}{**Precision**} & \multicolumn{1}{c}{**Recall**} & \multicolumn{1}{c}{**F-Measure**} & \multicolumn{1}{c}{**MCC**} & \multicolumn{1}{c}{**AUC**} \\ \hline CDI & 0.500 & 0.053 & 0.095 & 0.091 & 0.671 \\ \hline Manual Segmentation & 0.186 & 0.178 & 0.182 & \(-\)0.029 & 0.438 \\ \hline CDI with rebalance & 0.382 & 0.684 & 0.491 & 0.254 & **0.694** \\ \hline Manual Segmentation with rebalance & 0.168 & 0.304 & 0.216 & \(-\)0.079 & 0.415 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Prediction performance of ANN using three top CDI locations and manual segmentation on X3 slice. \begin{table}
\begin{tabular}{c c c c} \hline \hline \multicolumn{2}{c}{**Quantification Method**} & \multicolumn{2}{c}{**‘Changed’**} & \multicolumn{1}{c}{**‘No Change’**} \\ \hline \multirow{4}{*}{CDI} & \multirow{2}{*}{Original} & Train & 98 & 415 \\ \cline{3-4}  & & Test & 43 & 173 \\ \cline{3-4}  & & Train & 392 & 415 \\ \cline{3-4}  & & Test & 43 & 173 \\ \hline \multirow{3}{*}{Manual Segmentation} & \multirow{2}{*}{Original} & Train & 370,438 & 15,112 \\ \cline{3-4}  & & Test & 157,591 & 62,009 \\ \cline{1-1} \cline{2-4}  & & Train & 15,112 & 15,112 \\ \cline{1-1} \cline{2-4}  & & Test & 157,591 & 62,009 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Sample distribution in experiment 4. Figure 5: ROC plot of top CDI (#7, #8, and #9) and manual segmentation on X3 slice. In addition, we compared the four groups of experiments longitudinally. Such a comparison gave us a better idea of how different feature spaces performed using the same quantification method. The best performance from the above four groups of the experiment was summarized into Table 9. For CDI, the central CDI locations achieved the best AUC = 0.766 on the rebalanced dataset, which indicated that the pattern of cartilage changing was stronger in the central region of medial tibia compartment. For manual segmentation, the combination of all slices of a knee MR sequence generated the best AUC = 0.656 than using only the individual slice. Overall, using CDI identifies a stronger pattern of cartilage change than using manual segmentation, no matter employing subregion features or the whole feature space. Figure 7 plotted the ROC curves of the models using different feature spaces. Finally, we compared the performance of ANN with a simple baseline classifier (ZeroR) using whole medial tibia data. ZeroR [23] is the simplest classification method which relies on predicting the majority class while ignoring the other class on the binary problem. Despite the fact that there is no predictability power in ZeroR, it is helpful for deciding a baseline performance on the imbalanced classification problems and compare its performance with our classification method. The performance of ZeroR was same before and after using rebalance. Table 10 compared the performance of the two classifiers and as the results showed, ANN significantly outperformed the baseline classifier.