## 4 Hyperparameter Tuning
### Selection of the Relevant Set  
This experiment determines how different criteria for the construction of relevant sets influence the treatment prediction performance described using the metrics (defined above) of accuracy concerning patients/treatments.  
We tried the following two alternatives.  
1. Relevant Patients Based on Similarity: For each target patient \(p\), select at least 200 patients in descending order of similarity based on accumulated abnormal evaluation.
2. Relevant Patients Based on Uniform Choice: For each target patient \(p\), randomly and uniformly select at least 200 patients, regardless of their similarities.  
The intuition behind the Similarity Selection of relevant patients is that building a model for treatment recommendations for patient \(P\) based on patients having similar abnorms can lead to better predictions of treatments, especially rare ones.  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Strategy** & **Encoding Variant** & **Abnormals** & \begin{tabular}{c} **Relevant Patient** \\ **Selection Method** \\ \end{tabular} & **Precision** & **Recall** & **F1-score** \\ \hline Abnormal-hot-accumulated & \begin{tabular}{c} Abnormality-Hot \\ Encoding \\ Reduced \\ \end{tabular} & Accumulated & Similarity & 0.71 & 0.42 & 0.53 \\ \begin{tabular}{c} Instantaneous \\ Uniform-umap-accumulated \\ \end{tabular} & Instantaneous & Similarity & 0.78 & 0.61 & 0.68 \\ \begin{tabular}{c} Abnormal-umap-accumulated \\ \end{tabular} & \begin{tabular}{c} Reduced \\ Dimensionality \\ **Reduced** \\ **Dimensionality** \\ \end{tabular} & None & Uniform choice & 0.96 & 0.60 & 0.74 \\ \
\begin{tabular}{c} **accumulated** \\ \end{tabular} &
\begin{tabular}{c} **Accumulated** \\ \end{tabular} & **Similarity** & **0.93** & **0.64** & **0.76** \\ \hline \hline \end{tabular}
\end{table}
Table 2: The effect of different training-parameter experiments on the overall recommendation performance on non-sequestered data when recommending treatments to use within \(h=2\) h of the target patient’s medical state. The last row which is bolded shows the best choice of hyperparameters.  
The intuition behind uniform Random Selection of relevant patients is to eliminate bias.  
The results show that the overall F1-score of Uniform Selection and Similarity Selection are very similar, as shown in Section 2, where the strategy uniform-umap-accumulated (UUA) corresponds to Uniform Selection and the strategy abnormal-umap-accumulated (AUA) corresponds to Similarity Selection. This suggests that for common treatments, the uniform Random Selection of relevant patients is as good as Similarity Selection of relevant patients.  
By contrast, the metric of accuracy by treatments summarized in Figure 6 with line graphs (red for strategy UUA and blue for strategy AUA) shows that the Similarity Selection of relevant patients performs much better than Uniform Selection for rare treatments. In our dataset, among treatments in the lowest 75% of frequency with a mean number of 571 applications (which is 10 times less than the 5194 applications of frequent treatments), Similarity Selection achieves an accumulated F1-score of 66%, whereas Uniform Selection achieves an accumulated F1-score only of 59%. For the least frequent 50% of treatments, the F1-score when using Similarity Selection is 43%, while the F1-score when using Random Selection is 29%. Admittedly, neither method does very well for rare treatments, but Similarity Selection performs much better for such treatments.