## 4 Modeling and Validation
### Validation Results  
All models were evaluated and compared based on classification accuracy. Accuracy is defined as the number of correct predictions divided by the total number of predictions. The classification error rate is the proportion of observations that have been misclassified: Error rate = 1 - Accuracy. Model accuracy can be computed from the model's resulting confusion matrix. The confusion matrix is a table used to evaluate the performance of a classification model. It provides a summary of the number of correct and incorrect predictions made by the model compared to the actual outcomes (i.e., ground truth). A confusion matrix typically consists of four parts: true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN):  
* _True positives (TPs)_: These are the cases where the model correctly predicts a positive outcome when the actual outcome is positive. * _True negatives (TNs)_: These are the cases where the model correctly predicts a negative outcome when the actual outcome is negative. * _False positives (FPs)_: These are the cases where the model incorrectly predicts a positive outcome when the actual outcome is negative. * _False negatives (FNs)_: These are the cases where the model incorrectly predicts a negative outcome when the actual outcome is positive. Using the above definitions from the confusion matrix, the accuracy is calculated by:  
\[\text{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN} \tag{1}\]  
Figure 7 shows the confusion matrix for the _Support Vector Classifier_ model's results from the factor analyzed _random forest top feature reduced dataset_ (68 Factor Features). The confusion matrix reveals that there were 1065 correct predictions (15 wrong) out of 1080 classification predictions. This equates to an accuracy of 99%. The chart in Figure 8 graphically displays the accuracies of all the models and the various gene datasets. The nine dataset categories are along the x-axis. The color lines depict the accuracies of the different models (see legend). The vertical bars show the gene/factor counts for the datasets. The left y-axis shows the model accuracy scale and the right y-axis shows the gene/factor count scale for the datasets. Figure 6: Ensemble model majority voting flow scheme. The _baseline gene dataset_ with its 58,544 features yielded an accuracy of 98% with the logistic regression model. This set the accuracy baseline to measure the other datasets against. The logistic regression model was a steady good performer across all reduced gene feature datasets, with consistent accuracies of 98%. The _k-Nearest Neighbor_ and the _decision tree_ models had the worst accuracy performance and larger accuracy fluctuations between the different reduced gene feature datasets. The other four models, including the _ensemble_ model, performed in the middle of the pack with accuracies between 96 and 97% for the larger gene feature datasets. However, when the feature size dropped, all the models' accuracy performance increased. The _Support Vector Classification_ model yielded the overall best accuracy of 99% with the 68 factors dataset from the random forest top features. The consistent accuracy performance of the top models demonstrates the effectiveness of the employed dimension reduction techniques. This improved performance can partially be attributed to the reduced feature datasets preventing model over-fitting. Large feature datasets can result in a model that is too complex and learns irrelevant information or \"noise\" within the dataset. When the model memorizes the noise and fits too closely to the training set, it becomes \"overfitted\" and cannot generalize well to new data. The smaller datasets with highly correlated features performed better than the larger datasets with more features. Moreover, the power of the exploratory factor analysis and its ability to represent latent relationships between features is proven by the exceptional 99% accuracy. Model performance was further evaluated by assessing execution times on the changing feature set sizes.