# Data Mining Techniques for Endometriosis Detection in a Data-Scarce Medical Dataset
## 2 Materials and Methods
### Dataset from Multiple Sample Types  
In Section 2.1, we present a data analysis focused on the GIT and FRT regions. This section also explains how we implemented the proposed method. There were regions with missing data for some sample types due to a variety of factors, including the complexity and/or invasiveness of the sample collection process, contamination, or the degradation of sample quality. As a result, we only took into account sample types that had complete datasets for each patient and omitted the others from the study. Table 14 shows that the GIT region had a very limited sample size and an unbalanced distribution of patients with EM disease. For the GIT region, we could use k = 3, so each part would contain 3 samples. However, this led to an insufficient number of samples per class for classification purposes, resulting in the lowered reliability of the evaluation metrics, such as accuracy or the F1 score. Therefore, we did not evaluate the classification performance for the GIT region in this study. The FRT could be partitioned into 7 subsets, each containing 3 samples that could be used to conduct an evaluation. Therefore, a new dataset was required that aggregated features by patient. The merging method presented in Algorithm 7 transformed a dataset with a list of features into a new dataset, where each patient had a single row with all their features. For the initial part of the execution, we created a list of columns for all the types and features of the samples, plus a dictionary of datasets for each \"PatientID\". The next step was to check whether there were any missing sample types for each patient. If the patient was missing a sample type, then they were skipped. Separately, the algorithm added the column \"IsEndometriosis\" and grouped the dataset by \"SampleType\". Next, the values of each type of sample were appended to an array; the array was scaled and then added to the list of rows. The final output was a dataset with these rows and columns. A total of 438 bacterial taxa were provided as input data before the merge. The merge operation increased the number of features to 1314, as each feature had 3 values for the FRT, 1 for each sample type. Nevertheless, applying the filtering described in Section 2.2.1 reduced the number of features to 716, so 54.49% of the columns were discarded. The algorithm for FRT classification is based on the previous methods presented in Algorithm 8. It first combines the data and then applies a supervised machine learning method to classify the patients. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Region** & **Number of Patients** & **Patients with EM** \\ \hline GIT (oral + fecal) & 08 & 3 (37.50\%) \\ FRT (EB + EF + vaginal) & 21 & 7 (33.33\%) \\ FRT2 (EB + vaginal) & 21 & 7 (33.33\%) \\ \hline \hline \end{tabular}
\end{table} 
Table 14: Patients per sample type. ```

**Algorithm 8**classify_frt(dataset, features, k, classifier)

### Hyperparameter Optimization

Hyperparameter optimization of the selected classifier may not necessarily improve its performance. Furthermore, the influence of hyperparameter optimization could be affected by the variability induced by the random_state hyperparameter, controlling both the model and the cross-validation process. Therefore, 100 different values of random_state were tested in order to achieve the statistical stability of the results of each classification task. For the SVC algorithm, the regularization hyperparameter C was adjusted following a manual grid search [46] to find the optimal values. The regularization strength varied inversely with C, which required a strictly positive value. Following the standard approach, we set the initial value of C to a default of 1 and then evaluated powers of 10, namely, 1, 10, and 100, for comparison. The parameter \(\gamma\) in the rbf kernel was considered by default since preliminary experimentation showed that, in the specified range, the performance did not vary significantly.