## 2 Materials and Methods
### Algorithms  
We employed various ML algorithms that have been applied for similar applications in the past, such as skill classification based on kinematic data [28] and regression tasks in the clinical surgery domain [29; 30]. For the two classification experiments we employed the k Nearest Neighbor (kNN), Linear Discriminant Analysis (LDA), Gaussian Naive Bayes (GNB), Quadratic Discriminant Analysis (QDA), Logistic Regression (LR), Random Forest (RF), and Support Vector Machine (SVM) methods. For the regression experiment, we employed the following regression algorithms: Support Vector Regressor (SVR) with three kernels (Linear, RBF, and Polynomial), Linear Regression (LinR), Second-degree Polynomial Regression (PolR), Random Forest (RF), \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Description** & **Abbreviation** & **Task 5** & **Task 6** & **Task 7** \\ \hline Accuracy rate—cuts without injury (\%) & CutsNoInj & & & ✓ \\ Average speed of right instrument movement (cm/s) & Speed-R & ✓ & ✓ & \\ Average speed of left instrument movement (cm/s) & Speed-L & & & ✓ \\ Economy of movement -grasper (\%) & EOM-G & ✓ & & \\ Economy of movement—left instrument (\%) & EOM-L & & ✓ & \\ Ideal path length of clipper (cm) & IPL-C & ✓ & & \\ Ideal path length of grasper (cm) & IPL-G & ✓ & & \\ Ideal path length of left instrument (cm) & IPL-L & & ✓ & \\ Number of exposed green balls that are collected & \#GreenB & & ✓ & \\ Number of clipped ducts & \#ClipD & ✓ & & \\ Number of movements of left instrument & \#Move-L & ✓ & & ✓ \\ Number of movements of right instrument & \#Move-R & & ✓ & \\ Number of lost balls which miss the basket & \#LoSt & & ✓ & \\ Number of cutting maneuvers performed without causing injury & \#CutM-NoInj & & & ✓ \\ Relevant path length—clipper (cm) & RPL-C & ✓ & & \\ Safe retraction—overstretch (\%) & SafeRetr & & & ✓ \\ Trial number & TN & & & ✓ \\ \hline \hline \end{tabular}
\end{table} 
Table 3: ‘Best features’ selected via the [PERSON]’s score. ‘✓’ denotes whether the metric is included in the corresponding task. kNN regression (kNNR), and XGBoost (XGB). In addition, we applied a deep learning approach based on the LSTM network, which has shown promising performance in similar predictions tasks in surgical research [31]. The input (feature vectors) and output (RNT) were the same as that of the ML models and the RNT predictions were also generated in'real time' (i.e., without using information from forthcoming trials). A significant advantage of LSTM is that it can handle long-term dependencies by removing or adding information to its internal cell state as the information flows from one trial to the next one. In this study, we employed an LSTM network with 256 units and a dense layer with one node and linear activation for RNT prediction. The dense layer's weights are shared across the multiple trial steps, whereas at each trial, the LSTM output was forwarded to the dense layer for generating the current prediction. For kNN, we used the Euclidian distance with k = 1 (first nearest neighbor). For RF, the number of estimators was set to one hundred while leaving the decision tree's maximum depth unspecified. For SVM, we employed two kernels for the classification task (linear and radial basis function (RBF)) and three kernels for the regression task (linear, RBF, and polynomial), leading essentially to two and three SVM variations, respectively. The kernel parameter gamma was set to 1/(# features) and the penalty C = 1 in all runs. For XGB, the maximum depth was set at 6 and the learning rate at 0.3. The LTSM network was trained for 30 epochs with batch size 1, learning rate = 0.001, Adam optimization, and the Huber loss function. Due to the limited size of the dataset, we did not employ an extra validation set for parameter fine tuning. Instead, the parameters of the models were set according to those proposed in similar studies, and when not available, to the default ones in the employed scikit-learn library.