## 3 Results
### Multi-Phase CT Radionic Features-Based Machine Learning Model  
We calculated the average test performance of each model in 10-fold cross validations for a hold-out test set, and then we compared the test performance across various machine learning algorithms based on multiple metrics. In terms of the average Area Under Receiver Operating Characteristic Curve (AU-ROC) and the average Area Under the Precision-Recall Curve (AU-PRC), the Random Forest model achieved better performance (AU-ROC: mean = 0.725, standard error = 0.023; AU-PRC: mean = 0.693, standard error = 0.015) than the other ML models for classifying malignant renal tumors using principal components derived from selected radiomic features (Table 1, Figure 1A,B). An average accuracy of 0.778 was obtained on evaluation with the hold-out test set (Table 2). At the optimal threshold, the Random Forest model showed an F1 score of 0.746, precision of 0.862, sensitivity of 0.657, specificity of 0.651, and Negative Predictive Value (NPV) of 0.364 (Table 2B, Figure 1C). Meanwhile, Rbf SVM achieved the second-highest AU-ROC of 0.702, and it showed better performance than the Random Forest model in the following metrics at its optimal probability threshold: F1 score of 0.781, precision of 0.885, sensitivity of 0.699, specificity of 0.698, and NPV of 0.411. These results are summarized in Table 2.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **AU-ROC** & **AU-PRC** & **ACC** \\ \hline Linear SVM & \(0.625\pm 0.031\) & \(0.577\pm 0.017\) & \(0.765\pm 0.005\) \\ Rbf SVM & \(0.702\pm 0.034\) & \(0.649\pm 0.028\) & \(0.77\pm 0.011\) \\ XGBost & \(0.679\pm 0.017\) & \(0.662\pm 0.016\) & \(0.79\pm 0.006\) \\ Random Forest & \(0.725\pm 0.023\) & \(0.693\pm 0.015\) & \(0.778\pm 0.009\) \\ \hline  & **Probability** & \multirow{2}{*}{**F1 Score**} & \multirow{2}{*}{**Precision**} & **Specificity** \\  & **Threshold** & & & **(PPV)** \\ \hline Linear SVM & \(0.7555\) & \(0.661\pm 0.003\) & \(0.808\pm 0.037\) & \(0.559\pm 0.180\) & \(0.558\pm 0.209\) & \(0.276\pm 0.028\) \\ Rbf SVM & \(0.8044\) & \(0.781\pm 0.008\) & \(0.885\pm 0.030\) & \(0.699\pm 0.155\) & \(0.698\pm 0.082\) & \(0.411\pm 0.065\) \\ XGBost & \(0.715\) & \(0.704\pm 0.003\) & \(0.837\pm 0.011\) & \(0.608\pm 0.031\) & \(0.605\pm 0.036\) & \(0.317\pm 0.017\) \\ Random Forest & \(0.79\) & \(0.746\pm 0.005\) & \(0.862\pm 0.019\) & \(0.657\pm 0.036\) & \(0.651\pm 0.065\) & \(0.364\pm 0.022\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Model performance of multi-phase CT radiomic feature-based machine learning models for malignant renal tumor classification.  
Figure 1: Malignant renal tumor classification performance of the Random Forest model. (**A**) Receiver Operating Characteristic curve and (**B**) Precision-Recall curve of the best model for classifying malignant renal tumors. (**C**) The difference in the prediction performance of the best model according to the decision threshold. (**D**) Confusion Matrix at the optimal decision threshold.  
We compared the model performance of four different types of machine learning algorithms for classifying malignant renal tumors. Sensitivity, precision, specificity (Positive Predictive Value; PPV), and Negative Predictive Value (NPV) were calculated at the optimal probability threshold for each model. The average and standard errors over test performance of each model in 10-fold cross-validations for a hold-out test set were calculated.