# Machine Learning Method for Prediction of Hearing Improvement After Stapedotomy
## 2 Materials and Methods
### Modelling  
Several models were used and compared in our approach. L1 (Lasso) [14] and L2 (Ridge) [15] regularization variants of linear regression were employed. Regularization enhances linear regression by penalizing the coefficient values, resulting in reduced overfitting and increased generalizability of the models.  
K-nearest neighbours (KNNs) models [16] utilize the features of other samples in their prediction. To predict the target value of an unseen sample, the targets of samples with similar features (nearest neighbours) are considered.  
The Random Forest (RF) regressor is an ensemble method. It constructs several random, uncorrelated decision trees in training [17]. The prediction results from different decision trees are aggregated, usually by majority voting [18].  
The models listed were trained separately for each postoperative target hearing intensity, i.e., for each component of the output vector a separate regressor was built and trained. In this way, the models specialize in predicting a single hearing intensity, primarily to allow comparison of underlying factors between different targets. The individual predictions can be used to construct a hearing intensity vector prediction.  
The mentioned models were used to assess different types of modeling approaches (linear, ensemble, neighbor-based) on the given problem domain. Furthermore, we opted for models that are suitable for smaller datasets. Considering ensemble methods, RF has been shown to be more stable on smaller datasets [19]. To prevent overfitting, ensemble method RF was chosen over decision tree, since combining multiple decision trees generally leads to more robust predictions. Regularization was employed to mitigate the effects of multicollinearity and also reduce overfitting. Another important factor in choosing the mentioned models was their previous prediction usefulness in related approaches [4; 5; 6; 7; 8; 9].  
Hyperparameter tuning and feature selection were also determined separately for each target.  
The hyperparameters were determined by cross-validation (CV). The hyperparameters considered were \(\lambda\) for Ridge and Lasso and k for the number of neighbours for KNNs. The maximum tree depth, the maximum number of features considered in the splitting, and the number of estimators were estimated for RF.  
In the testing process, the RF model with all features (RF_all) was added for feature selection assessment and comparison. RF was chosen for this task because the aggregation of multiple models is inherently robust to irrelevant features, randomly selecting a subset of features at each split. Linear regression without feature selection (LR_all) was also added to assess the feature selection and regularization effect of other linear models (Lasso, Ridge).  
\begin{table}
\begin{tabular}{c c} \hline
**Feature** & **Value 1** \\ \hline Hearing intensity means (air) & \\
125\_250 & 53.76 \(\pm\) 14.37 \\
500\_1000 & 57.05 \(\pm\) 14.07 \\
1500\_2000 & 54.02 \(\pm\) 16.36 \\
3000\_4000 & 56.83 \(\pm\) 20.21 \\
6000\_8000 & 61.08 \(\pm\) 21.41 \\ Hearing intensity means (bone) & \\
500\_1000 & 19.51 \(\pm\) 9.47 \\
1500\_2000 & 28.90 \(\pm\) 13.55 \\
300\_4000 & 31.67 \(\pm\) 15.84 \\ Other & \\ Air mean & 56.55 \(\pm\) 14.79 \\ Bone mean & 26.69 \(\pm\) 11.65 \\ Bone/Air & 0.46 \(\pm\) 0.12 \\ Bone\(\times\)Air & 1651.6 \(\pm\) 1173.78 \\ ABG & 30.95 \(\pm\) 8.46 \\ \hline \end{tabular} \({}^{\dagger}\) In dB, except Bone/Air and Bone\(\times\)Air.  
\end{table}
Table 3: Constructed features. Values are presented as mean \(\pm\) standard deviation. ABG—air-bone gap.