# Application of Machine Learning Algorithms for Prediction of Tumor T-Cell Immunogens
## 2 Materials and Methods
### Machine Learning Models Validation  
The machine learning (ML) models developed in the present study underwent validation through a rigorous process, encompassing 10-fold cross-validation and assessment against an independent test set. Cross-validation is a technique for estimating the performance of a predictive model [36]. In this approach, the training set is split into \(k\) smaller sets (in our case \(k=10\)). For each fold, a model is trained using \(k-1\) folds as training data and the resulting model is validated on the remaining part of the data (i.e., it is used as a test set). The performance measure reported by \(k\)-fold cross-validation is then the average of the values computed in the loop. This approach can be computationally expensive but does not waste too much data (as is the case when fixing an arbitrary validation set), which is a major advantage in problems such as an inverse inference where the number of samples is very small. The predictive performance of each model was evaluated through Receiver Operating Characteristic (ROC) statistics, which categorize outcomes into four possibilities as follows: true positives (TP, accurately predicting an immunogen as an immunogen), true negatives (TN, accurately predicting a non-immunogen as a non-immunogen), false positives (FP, inaccurately predicting a non-immunogen as an immunogen), and false negatives (FN, inaccurately predicting an immunogen as a non-immunogen). Based on these outcomes, key parameters were computed [37]. _Sensitivity_, (3), measures the proportion of correctly identified positive samples out of the total positive samples, offering insight into a model's ability to detect immunogenic peptides. Conversely, _specificity_ (4) gauges the accuracy of identifying negative samples out of the total negative samples, indicating the model's proficiency in recognizing non-immunogenic peptides. _Accuracy_ (5), a widely utilized metric, quantifies the ratio of correctly classified samples to the total sample size, providing an overall assessment of classification performance. The _Area Under the ROC Curve (AROC)_ assesses predictive efficacy, with values ranging from 0.5 for random prediction to 1.0 for perfect prediction. _Matthew's Correlation Coefficient (MCC)_ (6) evaluates the correlation between the observed and predicted classifications, accounting for imbalanced data. An MCC of +1 denotes a perfect prediction, \(-1\) signifies complete disagreement between predictions and true values, while zero indicates performance no better than random guessing. \[Sensitivity=\frac{TP}{TP+FN} \tag{3}\]  
\[Specificity=\frac{TN}{TN+FP} \tag{4}\]  
\[Accuracy=\frac{TP+TN}{TP+TN+FP+FN} \tag{5}\]  
\[MCC=\frac{TP\times TN-FP\times FN}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}} \tag{6}\]  
To assess the robustness of the model predictions and ascertain that they are not mere chance occurrences, a Y-scrambling method was employed [38]. This method involves the following steps: Initially, the model is trained using the original feature-target pairs, and its performance is noted. Subsequently, for a specified number of iterations, the target column is systematically shuffled, and the model is retrained on the newly formed feature-target pairs. The expectation is that the model exhibits proficient performance on the original data while manifesting diminished efficacy on the shuffled data. If this anticipated behavior is not observed, and the metric displays minimal variance, it suggests that the predictions lack robustness and may be attributed to chance. We evaluated the attribute performance for each of the top three performing models, namely SVM, RF, and XGBoost. This assessment utilized two model inspection techniques to gauge the contribution of individual features to the statistical performance of a fitted model on a given dataset. Firstly, permutation feature importance, implemented using the scikit-learn library, involved randomly shuffling the values of a single feature and observing the resultant degradation of the model's score.