## 3 Results  
### Model Calculation  
An overview of the performance parameters of the different calculated models during the nested resampling procedure can be found in Table 2. With a balanced accuracy of 77.6% and an AUC of 0.87, the support vector machines (SVM) outperformed all the other ML algorithms (see Table 2).  
The absolute and relative distribution of the 10 most predictive variables identified during nested resampling and used for the model buildings can be seen in Table 3. They can be grouped in the following two areas: (1) problematic or antisocial behavior during current hospitalization, and (2) psychopathology. In the initial model including ten variables, the _time spent at a high-security level during current forensic hospitalization_ was the dominant variable. However, the variable was omitted as it was considered circular. Further analysis was, therefore, conducted with the remaining nine most predictive variables.  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Statistical** & **Balanced** & **AUC** & **Sensitivity** & **Specificity** & **PPV (\%)** & **NPV (\%)** \\ \hline Logistic & & & & & & \\ Regression & 74.9 & 0.85 & 77.8 & 72.1 & 85.7 & 60.6 \\ \hline Tree & 74.7 & 0.80 & 72.6 & 76.8 & 86.2 & 56.7 \\ \hline Random & 75.3 & 0.83 & 74.9 & 74.9 & 87.3 & 59.9 \\ Forest & & & & & & \\ \hline Gradient & & & & & & \\ Boosting & & & & & & \\ \hline KNN & 77.7 & 0.85 & 78.6 & 76.8 & 88.0 & 63.1 \\ \hline
**SVM** & **77.6** & **0.87** & **78.2** & **66.9** & **87.3** & **66.3** \\ \hline Naive Bayes & 75.9 & 0.85 & 87.9 & 76.1 & 87.8 & 59.8 \\ \hline \hline \end{tabular}  
* AUC = area under the curve (level of discrimination); PPV = positive predictive value; NPV = negative predictive value; KNN = k-nearest neighbors; SVM = support vector machines.  
\end{table}
Table 2: Machine learning models and performance in nested cross-validation on training dataset 1.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Variable Code** & **Variable Description** & **Aggressive Incidents** &
\begin{tabular}{c} **No** \\ **Aggressive Incidents** \\ \end{tabular} \\ \hline DZ1 & Did the patient complain about the & 73/111 (65.8) & 45/238 (18.9) \\ \hline DZ2 & Did the patient show negative behavior & 76/112 (67.9) & 40/237 (16.9) \\ \hline DZ7 & Did the patient show dis/antisocial behavior? & 90/111 (81.1) & 73/238 (30.7) \\ \hline DZ10 & Did the patient break the rules of the ward & 61/112 (54.5) & 36/238 (15.1) \\ \hline R22c (mean, SD) & Time spent at a high-security level during & 48.36 (59.65)) & 33.84 (45.22) \\  & current forensic hospitalization & 30.19 (12.34) & 22.05 (11.35) \\ \hline PA7 & Adapted PANSS at admission: Hostility & & & \\ \hline  & symptom absent & 27/113 (23.9) & 160/238 (67.2) \\ \hline  & symptom discreet & 22/113 (19.5) & 45/238 (18.9) \\ \hline  & symptom substantial & 64/113 (56.6) & 33/238 (13.9) \\ \hline PA18 & Adapted PANSS at admission: Tension & & & \\ \hline  & symptom absent & 25/113 (22.1) & 131/238 (55) \\ \hline  & symptom discreet & 25/113 (22.1) & 54/238 (22.7) \\ \hline  & symptom substantial & 63/113 (55.8) & 53/238 (22.3) \\ \hline PA22 & Adapted PANSS at admission: & & & \\ \hline \hline \end{tabular}  
* AUC = area under the curve (level of discrimination); PPV = positive predictive value; NPV = negative predictive value; KNN = k-nearest neighbors; SVM = support vector machines.  
\end{table}
Table 3: Absolute and relative distribution of relevant predictor variables 1.  
The quality of the final model in the validation step is shown in Table 4. As expected, the balanced accuracy of 73.5 and the AUC of 0.84 were less than the results of the initial training model but they were still meaningful. With a sensitivity of 84% and a specificity of 59%, the patients involved in aggressive incidents were identified correctly in almost three-quarters of events, while three-thirds of all the non-aggressive patients were identified correctly (see Table 4).