# Application of Machine Learning Algorithms for Prediction of Tumor T-Cell Immunogens
## 3 Results and Discussion
This process involved randomly shuffling the target pairs in the training data, constructing new models, and subsequently evaluating them on the test data. This iterative procedure was repeated 100 times, and the average accuracy value was computed. The accuracies for each model hovered around 0.5 (0.5075 for the SVM model, 0.5182 for the RF, and 0.4987 for the XGBoost), which is a noticeable deterioration in performance close to that of a random classifier (which has an accuracy score of 0.5 for binary class classification). The resulting poor performance of the model with the shuffled data compared to the good performance of the original data demonstrates the robustness of the models. Next, we undertook an analysis to determine the significance of each feature in influencing the performance of the chosen models. Employing two feature importance techniques on the test set--permutation feature importance and drop-column feature importance--we evaluated the impact of each feature on the accuracy score of the three models (Supplementary File S1). Each feature was assessed by comparing the baseline accuracy of the model with the accuracy when the feature was altered. Positive values \begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline
**Model** & _TN_ & _FN_ & _TP_ & _FP_ & **Sensitivity** & **Specificity** & **Accuracy** & _AROC_ & _MCC_ \\ \hline kNN & 49 & 9 & 161 & 121 & 0.95 & 0.29 & 0.62 & 0.62 & 0.31 \\ LDA & 92 & 58 & 112 & 78 & 0.66 & 0.54 & 0.60 & 0.61 & 0.20 \\ QDA & 115 & 105 & 65 & 55 & 0.38 & 0.68 & 0.53 & 0.53 & 0.06 \\ SVM & 138 & 38 & 132 & 32 & 0.78 & 0.81 & 0.80 & 0.88 & 0.59 \\ RF & 153 & 48 & 122 & 17 & 0.72 & 0.90 & 0.81 & 0.87 & 0.63 \\ XGBoost & 129 & 35 & 135 & 41 & 0.79 & 0.76 & 0.78 & 0.86 & 0.55 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Summary of the performance of the machine learning (ML) models on the training set (10-fold cross-validation). _TN_—true negatives; _FN_—false negatives; _TP_—true positives; _FP_—false positives; _AROC_—area under the ROC curve (sensitivity vs. 1-specificity); _MCC_—[PERSON]’s correlation coefficient. \begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline
**Model** & _TN_ & _FN_ & _TP_ & _FP_ & **Sensitivity** & **Specificity** & **Accuracy** & _AROC_ & _MCC_ \\ \hline \hline \(k\)NN & 9 & 4 & 38 & 33 & 0.90 & 0.21 & 0.56 & 0.56 & 0.16 \\ LDA & 22 & 16 & 26 & 20 & 0.62 & 0.52 & 0.57 & 0.57 & 0.14 \\ QDA & 26 & 26 & 16 & 16 & 0.38 & 0.62 & 0.50 & 0.50 & 0.00 \\ SVM & 36 & 9 & 33 & 6 & 0.79 & 0.86 & 0.82 & 0.83 & 0.64 \\ RF & 39 & 10 & 32 & 3 & 0.76 & 0.93 & 0.85 & 0.80 & 0.70 \\ XGBoost & 32 & 9 & 33 & 10 & 0.79 & 0.76 & 0.77 & 0.83 & 0.55 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Summary of the performance of the machine learning (ML) models on the test set. _TN_—true negatives; _FN_—false negatives; _TP_—true positives; _FP_—false positives; _AROC_—area under the ROC curve (sensitivity vs. 1-specificity); _MCC_—[PERSON]’s correlation coefficient. indicated that altering (permuting or dropping) the feature decreased the model's performance, suggesting the feature's importance. Conversely, zero or negative values suggested that altering the feature either had no effect or even improved the model's performance, indicating a negative correlation with model performance. However, the observed values were generally minimal, precluding definitive conclusions regarding feature importance. After that we observed the common features between the different models. Table 4 delineates the top 10 most important features for each model. The _ACC_ features are represented as follows: the first numerical index represents the \(E\)-descriptor of the first amino acid, the second index stands for the \(E\)-descriptor of the second amino acid, and the third one, for the _lag_-value. Both permutation feature importance and drop-column feature importance methods identified four attributes as significant for the SVM model, namely _ACC145_, _ACC147_, _ACC313_, and _ACC234_.