## 4 Experimental Results
The NSCLC datasets contain a different number of samples for each class, and the class distribution is profoundly different, as described in Figure 12. In this figure, it is evident that there were few samples for class 6 (T5) and class 7 (T0/Tis). After removing the missing-class samples, the total number of samples from three NSCLC datasets became 672 and they were used for staging. Before we started the experiment, we performed the preparation of the data for training and testing. As the data in the real world is fuzzy in nature, we validated our CAD using repeated random trials. Figure 13 shows the preparation of data for repeated random trials in both first and second stage classifications. For first stage classification, the samples in the LIDC-IDRI dataset (888 samples) were randomly divided into five splits 50%-50%, 60%-40%, 70%-30%, 80%-20% and 90%-10% for training and testing. Similarly, for the second stage classification, samples in the NSCLC datasets (672 samples) were randomly divided into five trials. In each split, we repeatedly selected random samples \(k=100\) times as described in the figure. Since the samples were randomly selected, each iteration \(k\) generated \begin{table}
\begin{tabular}{l c c} \hline \hline
**Dataset** & **Descriptions** & **Number of Exams Used** \\ \hline \multirow{6}{*}{LIDC-IDRI} & Contains 1010 thoracic CT scans of lung cancer patients and an associated XML file that shows the true nodule annotations. & 888 \\  & Contains 89 CT scans from non-small cell lung cancer patients who underwent surgery. The clinical data of tumor stages is also available. & 89 \\  & Contains 422 CT scans from non-small cell lung cancer patients. & \\ NSCLC-Radiomics & Three-dimensional manual delineation and clinical results by radiologist are available. & 422 \\  & Contains 211 CT, PET/CT scans of non-small cell lung cancer patients. & \\ NSCLC Radiogenomics & Both annotation files and clinical data are also provided. & 211 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Detailed descriptions of the datasets applied. Figure 12: Class distribution of NSCLC datasets. different samples for training and testing. We trained and tested five different classifiers namely the Decision Tree (DT), K-nearest neighbor (KNN), Support Vector Machine (SVM), Ensemble Tree (ET), and the Back Propagation Neural Network (BPNN) on these random samples. In each iteration we calculated the performance measures of each classifier then calculated the mean and standard deviation of performance measures for each split. The performance measurements which were calculated for each classifier are accuracy, sensitivity, specificity, precision, and the F-score. They can be computed by:  
\[accuracy=\ \frac{TP+TN}{TP+TN+FP+FN} \tag{6}\]  
\[sensitivity(recall)=\ \frac{TP}{TP+FN} \tag{7}\]  
\[specificity=\ \frac{TN}{FP+TN} \tag{8}\]  
\[precision=\ \frac{TP}{FP+TP} \tag{9}\]  
\[F1-score=\ 2*\frac{Precision*recall}{Precision+recall} \tag{10}\]  
Nonetheless, the second stage of classification is a multi-classification problem unlike the first; it is therefore more complex to evaluate the performance. To solve this problem, we used the \"one versus the rest method\", which calculates the performance of the classifier for an individual class and then calculates the overall performance. For example, \(C_{i}\ (i=1,2,\ \ldots,l)\) is an individual class of a multi-classification problem, and \(l\) is the total number of classes. From the counts of \(Ci\), the assessments \(TP_{i}\), \(TN_{i}\), \(FP_{i}\) and \(FN_{i}\) were obtained. Then, the overall performance of the classifier for all classes was calculated by:  
\[accuracy=\ \frac{\sum_{i=1}^{l}\frac{TP_{i}+TN_{i}}{TP_{i}+TN_{i}+FP_{i}+FN_{i}}}{l} \tag{11}\]  
\[sensitivity=\ \frac{\sum_{i=1}^{l}\frac{TP_{i}}{TP_{i}+FN_{i}}}{l} \tag{12}\]  
\[specificity=\ \frac{\sum_{i=1}^{l}\frac{TN_{i}}{TN_{i}+FP_{i}}}{l} \tag{13}\]  
Figure 12:0 Data preparation for repeated random trials.