Utilizing Semantic Textual Similarity for Clinical Survey Data Feature Selection
Authors: Benjamin C Warner, Ziqi Xu, Simon Haroutounian, Thomas Kannampallil, Chenyan Lu
Accepted at: acl25

Utilizing Semantic Textual Similarity for Clinical Survey Data Feature Selection

Benjamin C. Warner, Ziqi Xu, Simon Haroutounian, Thomas Kannampallil, Chenyang Lu
Washington University in St. Louis
{b.c.warner, ziqixu, sharout, thomas.k,lu}@wustl.edu

Abstract
Surveys are widely used to collect patient data in healthcare, and there is significant clinical interest in predicting patient outcomes using survey data. However, surveys often include numerous features that lead to high-dimensional inputs for machine learning models. This paper exploits a unique source of information in surveys for feature selection. We observe that feature names (i.e., survey questions) are often semantically indicative of what features are most useful. Using language models, we leverage semantic textual similarity (STS) scores between features and targets to select features. The performance of STS scores in directly ranking features as well as in the minimal-redundancy-maximal-relevance (mRMR) algorithm is evaluated using survey data collected as part of a clinical study on persistent post-surgical pain (PPSP) as well as an accessible dataset collected through the NIH All of Us program. Our findings show that features selected with STS can result in higher performance models compared to traditional feature selection algorithms.

1 Introduction
Survey data can have many features and a relatively low number of examples, particularly in human subjects research where there is a high cost of recruiting participants. Small and novel populations combined with multiple externally-validated questionnaires can easily result in high-dimensional data. A typical solution is to apply feature selection using statistical measures, but because this relies upon the same high-dimensional data being filtered, suboptimal selections can be made. Conversely, features can be manually selected for inclusion in a model, but this can be limited as it involves testing individual combinations of features. An ideal solution would combine the empirical ranking of the former approach with the domain-informed selection offered by the latter approach.

We explore this problem through two clinical datasets, a private dataset from a study examining persistent post-surgical pain (PPSP) and the All of Us dataset. Both of these datasets collect answers to hundreds of questions filled out by clinicians and participants covering a broad range of topics. For the PPSP dataset, there are only 617 examples, which limits the amount of information available for feature selection.

1.1 Proposed Approach
Survey questions are, in practice, semantically related to a target outcome and semantically similar to or distinct from other questions. Semantic textual similarity (STS) could be an analogue to statistical measures that capture relationships between features, such as mutual information (MI). STS may then be useful for determining which questions are relevant to predicting a target question, or which questions are redundant to one another. To this end, we evaluate the use of STS scores directly to determine the most relevant features, and test STS scores both as a direct replacement and as a complement in algorithms utilizing statistical scores, such as minimal-redundancy-maximal-relevance (mRMR). Our overall approach, outlined in Figure 1, is to first score features using a language model (LM) that produces STS scores, statistical scores, or a linear combination of both; and then use a general selection algorithm to pick features from these scores.

There appears to be nearly no literature examining feature selection with embeddings or STS. The closest match to our knowledge examined the usage of word2vec continuous bag-of-words embeddings (Mikolov et al., 2013) trained upon Twitter data to select Google search query trends matching the embeddings of a target concept (Lampos et al., 2017). Our approach differs in several key ways, with the principal difference being that we use STS scores to make comparative selections with limited tab-
ular data, whereas (Lampos et al., 2017) use the distribution of STS scores produced by thousands of candidate time-series features to make selections. In addition, we compare the performance of different selection algorithms, pre-trained LMs, fine-tuning datasets and scoring combinations as they relate to feature selection performance.

We present several empirical contributions:

• An examination of how STS scores generated by LMs between feature-feature and feature-target question pairs can help select features, either alone or in combination with statistical measures such as MI.

• A comparison of different LM scoring configurations as they relate to classifier performance.

• A demonstration of how STS-based feature selection algorithms can reduce overfitting using a gated-access and private clinical dataset relating to persistent post-surgical pain.

We make our code available at https://github.com/bcwarner/sts-select and via pip install sts-select. Additionally, a Hugging Face collection with the best performing models for each dataset can be found through our GitHub.

2 Preliminaries

2.1 Feature Selection

Fitting high-dimensional data is particularly difficult when the number of examples is low since a model can easily overfit on the training data. To counter this, we can utilize feature selection, where a subset of the overall features in a dataset are selected for learning.

Feature selection methods can be divided into three categories: embedded, wrapper, and filter methods. Embedded methods incorporate feature selection as a part of training, while wrapper methods interact in a feedback loop with the learning model. Filter methods select a subset of features based on properties of the dataset before the model is trained, which differs from embedded and wrapper methods in that they do not form a feedback loop with the model (Guyon et al., 2008). Because of their independence, they tend to generalize well (Remeseiro and Bolon-Canedo, 2019).

Feature selection methods used in clinical survey data cover a broad range of techniques. A study examining autism spectrum disorder (ASD) survey data, examined feature selection using principal component analysis, t-distributed stochastic neighbor embedding, and denoising autoencoders; and also found that survey features targeting ASD tend to have high levels of redundancy (Washington et al., 2019). Some of the other feature selection methods found for models involve questionnaires include wrapper models based on random forests (Niemann et al., 2020), bootstrapped feature selection (Abbas et al., 2018), principal component analysis, multicluster feature selection (Saridewi and Sari, 2020), permutation importance (Chen et al., 2023), and ReliefF (Abut et al., 2016).

One particularly useful feature selection technique is minimal-redundancy-maximal-relevance (mRMR), which aims to maximize the relevance of features to the target, while minimizing the redundancy between selected features. This is particularly useful when we have a small number of features that are correlated and want to ensure a model incorporates as broad as a set of information as possible (Peng et al., 2005; Ramírez-Gallego et al., 2017).

Underpinning the mRMR objective function is the mutual information (MI) between classes and features, which measures the amount of shared information between two distributions. In addition to MI, Pearson’s r and F-statistic p-values, can also be used in the mRMR algorithm.

2.2 Semantic Textual Similarity

Semantic textual similarity (STS) is a task where a LM is used to score the semantic similarity of two sentences, generally by evaluating the differences between embeddings generated by a model. Cosine similarity is one typical function used to measure the difference between embeddings (Reimers and Gurevych, 2019; Oniani et al., 2022).

STS scores can be produced with language models (LMs) that follow the pre-training/foundation model paradigm. These latter models are trained with a self-supervised learning task, and then modified and trained to complete a supervised task (Devlin et al., 2018; Radford et al., 2019). Pre-training is particularly useful since it results in better generalization (Erhan et al., 2010), and because it allows computationally expensive models to be reused for different tasks (Wolf et al., 2019). Some large language models (LLMs) have demonstrated capabilities at many reasoning tasks involving semantic meaning (Singhal et al., 2022; Wei et al., 2023),
Figure 1: Overview of the STS-based feature selection process. Feature-feature and feature-label relationships are first scored, using statistical algorithms and/or models fine-tuned to produce STS scores. These scores are then used to select the best subset of features for a downstream model.

and are highly applicable since survey features may involve complex word relationships.

Clinical language involves vocabulary and semantic meaning that is often not present in non-clinical texts, and various pre-trained architectures exist to fill this gap. To date, there are over 80 clinical LMs available (Wornow et al., 2023), with a diverse set of architectural and training designs. Clinical LMs can perform better than general LMs on tasks specific to the clinical domain (Alsentzer et al., 2019), and can do so more efficiently than a general-purpose models (Lehman et al., 2023).

3 Methodology

3.1 Scoring & Selection

A distinction can be made between the scoring of features, where we measure a feature’s relationship to a target or another feature, and the selection of features, where we then use these scores to select the appropriate features. We evaluate the performance of STS, as well as three baseline scoring methods: MI, Pearson’s r coefficients, and F-statistic p-values. We also evaluate the linear combination of them, with the coefficient for STS being a hyperparameter \( \alpha \), which we test over a logarithmically-spaced range of 30 values from \( [10^{-2}, 10^2] \). For selection algorithms with these scores we evaluate selecting the top \( N \) feature-target scores, selecting feature-target scores above a given standard deviation \( k \), and selection of \( N \) features using these scores in mRMR.

We evaluate several baseline feature selectors from the filter method category because they do not have a feedback loop from which overfitting could occur (Remeseiro and Bolon-Canedo, 2019) which makes them appropriate for small datasets. We use selection based on the best weights from a linear support vector machine (SVM) model and an XGBoost model (Chen and Guestrin, 2016) using SelectFromModel from scikit-learn (Pedregosa et al., 2011), and recursive feature elimination (RFE). The linear SVM model is tested with \( C \) over 10 logarithmically spaced values from \( [10^{-2}, 1] \), while the XGBoost instance used in SelectFromModel has the default settings. RFE is also evaluated using a linear SVM model with \( C = 1 \) to select features.

For all of the aforementioned selection algorithms and the baseline algorithms we compare against, we select 40 features. For selection by standard deviations, all features with scores above \( \mu + k\sigma \) are selected, where \( k = 1 \).

Other than formatting to support the one-hot vectorization of categorical features, the feature names are unmodified from their original text. In addition to each of the features used to assign the target label, we also use the name persistent_pain to the target, and average their STS scores together in evaluation. Details for how each dataset derived the target label can be found in Sections 4.1.1 and 4.1.2. The MI, and Pearson’s r and F-statistic approxima-
<table>
  <tr>
    <th>Dataset</th>
    <th>Type</th>
    <th>Pairs</th>
  </tr>
  <tr>
    <td>bio_simlex (Chiu et al., 2018)</td>
    <td>Clin.</td>
    <td>987</td>
  </tr>
  <tr>
    <td>bio_sim_verb (Chiu et al., 2018)</td>
    <td>Clin.</td>
    <td>1,000</td>
  </tr>
  <tr>
    <td>mayosrs (Pedersen et al., 2007)</td>
    <td>Clin.</td>
    <td>101</td>
  </tr>
  <tr>
    <th>Total Clinical</th>
    <td></td>
    <td>2,088</td>
  </tr>
  <tr>
    <td>sts-companion (Cer et al., 2017)</td>
    <td>Gen.</td>
    <td>5,289</td>
  </tr>
  <tr>
    <td>stsb_multi_mt;en (May, 2021)</td>
    <td>Gen.</td>
    <td>5,749</td>
  </tr>
  <tr>
    <th>Total General</th>
    <td></td>
    <td>11,038</td>
  </tr>
  <tr>
    <th>Total</th>
    <td></td>
    <td>13,126</td>
  </tr>
</table>

Table 1: Datasets used for training and fine-tuning selected models.

tion methods used in this paper are implemented from scikit-learn (Pedregosa et al., 2011).

3.2 LMs and Fine-Tuning Datasets

For generating STS scores, we evaluate the performance of several different pre-trained LMs using Hugging Face’s transformers (Wolf et al., 2019) with the sentence_transformers library (Reimers and Gurevych, 2019) to fine-tune and produce STS scores. We evaluate two models trained on general vocabulary, namely a11-MiniLM-L12-v2 (Reimers and Gurevych, 2019) and bert-base-uncased (Devlin et al., 2018). We also evaluate the performance of models pre-trained on clinical/scientific text, including Bio_ClinicalBERT (Alsentzer et al., 2019), BioMed-RoBERTa-base (Gururangan et al., 2020), PubMedBERT (Gu et al., 2021), BioGPT (Luo et al., 2022), and GatorTron (Yang et al., 2022).

To fine-tune these models, we train the model to predict cosine similarity on one of two combined STS datasets, which we group into a general sentence-level and clinical phrase-level set. We also evaluate the performance of fine-tuning on the combination of both the general and clinical STS datasets. Table 1 highlights the composition of the selected fine-tuning datasets.

Additionally, ClinicalSTS (Xiong et al., 2020) and MedSTS (Wang et al., 2020) are two clinical sentence-pair datasets we were unable to obtain for fine-tuning. To overcome this limitation, we also evaluate the performance of a Bio_ClinicalBERT model fine-tuned on just the ClinicalSTS dataset (Mulyar et al., 2019).

4 Experiments

4.1 Data

To evaluate our proposed approach, we utilize two datasets, a private dataset examining PPSP and a cohort subset from the All of Us gated-access dataset from the National Institutes of Health. Both datasets use the Research Electronic Data Capture (REDCap) system (Harris et al., 2009) for collecting and organizing patient survey data.

4.1.1 PPSP Dataset

The PPSP dataset is a partially complete set of participants from the P5: Personalized Prediction of Persistent Postsurgical Pain study (IRB #202101123) conducted at the Washington University School of Medicine in St. Louis/BJC HealthCare system. This dataset attempts to examine who will get PPSP, the phenomenon of a patient experiencing surgically-related pain for a longer duration of time than expected (Vila et al., 2020).

The dataset contains 1631 partial responses from a total 617 participants as a part of a final goal of 2,000 participants from the BJC HealthCare system. When exported from REDCap, the combination of these questionnaires results in 458 named features, ranging from demographic features to various measures of psychological and physical pain and correlated variables. Table 4 in Appendix A outlines the key characteristics of the dataset, including number of examples and general demographics.

4.1.2 All of Us Dataset

The All of Us dataset is a gated-access dataset of electronic health records, surveys, and other medical data collected by the National Institutes of Health for a broad range of biomedical research. The All of Us program collects extensive survey data capturing patient history.

To parallel our private dataset, we utilize the All of Us Registered Tier Dataset v7 and select a cohort of patients who experienced “Persistent pain following procedure” or “Pre-surgery evaluation” and use the presence of the former condition as the label for that dataset. We only include patients who have the “Contains Surveys Codes” condition, resulting in 32,631 patients.

4.2 Data & Model Preparation

To deal with missing entries in survey data, several imputation strategies are applied. For columns with numerical types of data, null entries are replaced with the mean value, and then have the \( L_2 \) norm applied to that column. Date/time types are dropped for simplicity. String types—which we are treating as categorical types given the previous filtering of unique values—will be imputed with the
most common value, and then split up into one-hot columns.

Classifier models tested include linear SVM, multilayer perceptron, Gaussian Naïve Bayes (Gaussian NB), and \( k \)-nearest neighbors (\( k \)-NN). For linear SVM, we test over 10 values of \( C \) logarithmically-spaced from 0.01 to 1. For \( k \)-NN, we evaluate 3, 5, and 7 neighbors, and for MLP, we evaluate tanh and ReLU activations with \( \alpha \) logarithmically-spaced from 0.01 to 1 over 10 steps.

An 80%/20% train/test split is used for evaluating overall performance, and 5-fold flat cross-validation (CV) is employed to both select hyper-parameters and evaluate the overall performance of the dataset. Nested CV is typically employed for evaluating model selection with small datasets, but experimentally may not be necessary with low numbers of hyperparameters while using specific model types, such as gradient boosted trees (Wainer and Cawley, 2021). For this reason, and due to the fact that nested CV with \( K \) outer-folds would incur a proportional increase in run-time, flat 5-fold CV is used. For randomization in NumPy and PyTorch and any of their dependencies, we use the seeds 278797835 and 424989.

To emulate the low-dimensional setting found in the PPSP dataset, the All of Us dataset has examples randomly dropped such that the ratio of features to examples is 1:1 (i.e. 1466 examples with 1466 features). The final dimensionality of the PPSP dataset after preprocessing is 617 examples with 269 features.

5 Results

We evaluated 1108 different feature selection and classifier pairings for the PPSP dataset, and 821 pairings for the All of Us dataset. Tables 2 and 3 show the the performance in area under the receiver-operator curve (AUROC) and area under the precision-recall curve (AUPRC) for each of the STS-based feature selectors compared to all baselines using the Gaussian NB classifier, which was the most significantly improved classifier for both datasets. Two-tailed t-test \( p \)-values between those groups, which were corrected for multiple hypotheses using the Benjamini–Yekutieli method with \( \alpha = 0.05 \) (Benjamini and Yekutieli, 2001; Virtanen et al., 2020; Seabold and Perktold, 2010), are also shown. Full results stratified by all classifiers tested can be found in Tables 9 to 12 in Appendix C. Detailed summaries of training results with the Gaussian NB can be found in Tables 13 and 14.

![MI feature-feature and feature-target pairings from the PPSP dataset.](page_1012_186_377_282.png)

Figure 2: MI feature-feature and feature-target pairings from the PPSP dataset.

5.1 Scoring Performance

When evaluating the performance between the STS and non-STS scorers, as shown in Tables 2 and 3, we find that STS tends to slightly drop in test performance for the All of Us dataset and slightly improve in test performance for the PPSP dataset. We find in both cases the test-train gap improves with both STS alone or in combination with the majority of the baseline scoring methods compared to the baseline feature selectors.

We compare performance for each of these hyperparameters overall using ANOVA and Tukey’s honestly significant difference (HSD) test in Tables 5 to 8 in Appendix B (Tukey, 1949; Seabold and Perktold, 2010). For training the STS scoring models, we find that the differences between a clinical and general fine-tuning dataset appear to be significant for the All of Us dataset but negligible with the PPSP dataset, as seen in Tables 5 and 7 in Appendix B. Similarly, for fine-tuned scoring models, we find that for the majority of pairings, there is no significant difference in model performance, as seen in Tables 6 and 8 in Appendix B.

5.2 Feature Scoring Differences

When evaluating the selected features, we see a noticeable difference between those selected using MI and STS scores. Figures 2 and 4, which shows MI, and Figures 3 and 5, which shows selected STS scores for the features and targets for the dataset, highlights why this difference exists, as the STS scores are capable of highlighting more relationships between features and targets. The performance of Pearson’s \( r \) and F-statistic scores
Table 2: Results stratified by scorer measured by AUROC using Gaussian NB, with average baseline performance (\( \mu_B \)), average performance for a given scorer (\( \mu_S \)), average difference between baseline and scorer (\( \Delta \)), and the t-test \( p \) value corrected with the Benjamini–Yekutieli method.

<table>
  <tr>
    <th rowspan="2">Hyperparameters</th>
    <th colspan="4">Test - Train AUROC</th>
    <th colspan="4">Test AUROC</th>
  </tr>
  <tr>
    <th>D.</th>
    <th>Scorer</th>
    <th>\( \mu_B \)</th>
    <th>\( \mu_S \)</th>
    <th>\( \Delta \)</th>
    <th>\( p \)</th>
    <th>\( \mu_B \)</th>
    <th>\( \mu_S \)</th>
    <th>\( \Delta \)</th>
    <th>\( p \)</th>
  </tr>
  <tr>
    <td rowspan="5"><i>All of Us</i></td>
    <td>All STS</td>
    <td></td>
    <td><b>−0.057</b></td>
    <td>0.066</td>
    <td><b>0.003</b></td>
    <td></td>
    <td>0.578</td>
    <td>−0.050</td>
    <td>0.852</td>
    <td></td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td>−0.040</td>
    <td>0.084</td>
    <td>0.077</td>
    <td></td>
    <td>0.705</td>
    <td>0.078</td>
    <td>0.234</td>
    <td></td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td><b>−0.059</b></td>
    <td>0.064</td>
    <td><b>0.009</b></td>
    <td></td>
    <td>0.628</td>
    <td><b>0.551</b></td>
    <td><b>0.076</b></td>
    <td><b>0.002</b></td>
  </tr>
  <tr>
    <td>Pearson’s r & STS</td>
    <td></td>
    <td>−0.055</td>
    <td>0.068</td>
    <td>0.027</td>
    <td></td>
    <td>0.592</td>
    <td>−0.035</td>
    <td>1.000</td>
    <td></td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td>−0.062</td>
    <td>0.061</td>
    <td>0.013</td>
    <td></td>
    <td>0.548</td>
    <td>−0.080</td>
    <td>0.002</td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="5"><b>PPSP</b></td>
    <td>All STS</td>
    <td></td>
    <td>0.060</td>
    <td>0.085</td>
    <td>7.0 · 10<sup>−05</sup></td>
    <td></td>
    <td>0.840</td>
    <td>0.098</td>
    <td>8.7 · 10<sup>−04</sup></td>
    <td></td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td>0.065</td>
    <td>0.089</td>
    <td>1.5 · 10<sup>−05</sup></td>
    <td></td>
    <td>0.839</td>
    <td>0.097</td>
    <td>0.024</td>
    <td></td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td>−0.024</td>
    <td>0.050</td>
    <td>9.6 · 10<sup>−04</sup></td>
    <td></td>
    <td>0.742</td>
    <td>0.816</td>
    <td>0.074</td>
    <td>0.261</td>
  </tr>
  <tr>
    <td>Pearson’s r & STS</td>
    <td></td>
    <td>0.066</td>
    <td>0.090</td>
    <td>4.6 · 10<sup>−06</sup></td>
    <td></td>
    <td>0.857</td>
    <td>0.115</td>
    <td>1.7 · 10<sup>−04</sup></td>
    <td></td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td>0.060</td>
    <td>0.084</td>
    <td>7.3 · 10<sup>−06</sup></td>
    <td></td>
    <td>0.849</td>
    <td>0.107</td>
    <td>1.8 · 10<sup>−06</sup></td>
    <td></td>
  </tr>
</table>

Table 3: Results stratified by scorer measured by AUPRC using Gaussian NB.

<table>
  <tr>
    <th rowspan="2">Hyperparameters</th>
    <th colspan="4">Test - Train AUPRC</th>
    <th colspan="4">Test AUPRC</th>
  </tr>
  <tr>
    <th>D.</th>
    <th>Scorer</th>
    <th>\( \mu_B \)</th>
    <th>\( \mu_S \)</th>
    <th>\( \Delta \)</th>
    <th>\( p \)</th>
    <th>\( \mu_B \)</th>
    <th>\( \mu_S \)</th>
    <th>\( \Delta \)</th>
    <th>\( p \)</th>
  </tr>
  <tr>
    <td rowspan="5"><i>All of Us</i></td>
    <td>All STS</td>
    <td></td>
    <td><b>−0.011</b></td>
    <td>0.025</td>
    <td>4.0 · 10<sup>−05</sup></td>
    <td></td>
    <td>0.027</td>
    <td>−0.006</td>
    <td>1.000</td>
    <td></td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td>−0.026</td>
    <td>0.010</td>
    <td>0.972</td>
    <td></td>
    <td>0.046</td>
    <td>0.013</td>
    <td>0.173</td>
    <td></td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td><b>−0.007</b></td>
    <td>0.029</td>
    <td><b>7.3 · 10<sup>−08</sup></b></td>
    <td></td>
    <td>0.033</td>
    <td><b>0.023</b></td>
    <td><b>−0.010</b></td>
    <td><b>1.9 · 10<sup>−06</sup></b></td>
  </tr>
  <tr>
    <td>Pearson’s r & STS</td>
    <td></td>
    <td>−0.012</td>
    <td>0.023</td>
    <td>0.039</td>
    <td></td>
    <td>0.030</td>
    <td>−0.003</td>
    <td>1.000</td>
    <td></td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td>−0.007</td>
    <td>0.028</td>
    <td>7.3 · 10<sup>−08</sup></td>
    <td></td>
    <td>0.023</td>
    <td>−0.010</td>
    <td>1.9 · 10<sup>−06</sup></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="5"><b>PPSP</b></td>
    <td>All STS</td>
    <td></td>
    <td>0.102</td>
    <td>0.065</td>
    <td>0.002</td>
    <td></td>
    <td>0.423</td>
    <td>0.126</td>
    <td>4.3 · 10<sup>−04</sup></td>
    <td></td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td>0.106</td>
    <td>0.069</td>
    <td>0.012</td>
    <td></td>
    <td>0.410</td>
    <td>0.114</td>
    <td>0.056</td>
    <td></td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td>0.037</td>
    <td>0.087</td>
    <td>0.050</td>
    <td>0.065</td>
    <td>0.296</td>
    <td>0.406</td>
    <td>0.110</td>
    <td>0.056</td>
  </tr>
  <tr>
    <td>Pearson’s r & STS</td>
    <td></td>
    <td>0.115</td>
    <td>0.078</td>
    <td>2.9 · 10<sup>−05</sup></td>
    <td></td>
    <td>0.440</td>
    <td>0.143</td>
    <td>1.9 · 10<sup>−04</sup></td>
    <td></td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td>0.100</td>
    <td>0.063</td>
    <td>0.003</td>
    <td></td>
    <td>0.435</td>
    <td>0.138</td>
    <td>3.5 · 10<sup>−05</sup></td>
    <td></td>
  </tr>
</table>

Figure 3: STS scoring with microsoft/biogpt fine-tuned on both the general and clinical vocab pairings from the PPSP dataset.

![STS scoring with microsoft/biogpt fine-tuned on both the general and clinical vocab pairings from the PPSP dataset.](page_104_1042_495_246.png)

Figure 4: MI feature-feature and feature-target pairings from the All of Us dataset.

![MI feature-feature and feature-target pairings from the All of Us dataset.](page_104_1342_495_246.png)

Figure 5: STS scoring with UFNLP/gatortron-base fine-tuned on both the general and clinical vocab pairings from the All of Us dataset.

![STS scoring with UFNLP/gatortron-base fine-tuned on both the general and clinical vocab pairings from the All of Us dataset.](page_670_1042_495_246.png)

shows a similar trend, which can be seen in Figures 6a, 6b and 7a in Appendix D.

6 Discussion

6.1 Feature Selection Performance

Using STS to select features appears to offer significant performance benefits over traditional feature selection methods. One main reason that the usage of STS appears to work better than statistical scoring methods is that STS scores are not affected by the curse of dimensionality. As can be seen with
MI in Figures 2 and 4, complex relationships can cause individual features to share little information, and features that we would expect to be more relevant than others will be scored only marginally better than those that would not be relevant.

STS also provides a contrasting, non-correlated, source of information compared to statistical measures like MI. This can be seen from the strong contrast in highlighted regions between Figs. 2 and 3. This is particularly useful for when statistical scores that result from the training dataset fails to match what we might expect from the population sampled, as STS is not vulnerable to differences in the sample and population distributions. STS is clearly more able to highlight redundant regions, especially near the diagonal, and is more able to distinguish between relevant and irrelevant features than traditional statistical scores.

Inspection of the features selected with STS and statistically scored models highlights these issues. For example, in the All of Us dataset, when examining the top N features scored with MI, we find that the majority of features are variations of “Can’t Afford Care: Skipped Med to Save Money”, “Can’t Afford Care: Took Less Med To Save Money”, etc. In contrast, the top N features for PubMedBERT begin with questions such as “Are you still seeing a doctor or health care provider for a hernia?” and “Are you still seeing a doctor or health care provider for reactions to anesthesia (such as hyperthermia)?”. While the first set of features is more strongly correlated to the labels in the training set, the second set of features are more semantically related to the label, and are more likely to reflect ground-truth explanations.

6.2 Implications on Survey Writing

The surveys used here were not designed in anticipation of this type of feature selection algorithm, but subsequent questionnaires—or other data with descriptive feature names—may be designed with STS-based feature selection in mind. We offer several guidelines for feature name writing.

A key consideration is that of the limitations of the LMs’ vocabulary. Not all possible words or sub-words will be present, and furthering this, not all words will appear in the pre-training corpora or fine-tuning data. However, as long as either the pre-training or fine-tuning data contains references to a desired relationship, the LM should be able to produce a meaningful STS score, as the LM will have learned word relationships in its vocabulary during pre-training.

Furthering these limitations are features with hidden semantics, which may not be clearly related to the target, may be mistaken for other concepts, or may be a component of another concept. A clear example of this is the group of questions with “Color-Word Score (CW).” These particular questions have hidden semantics as they involve a phrase that is unlikely to be interpreted as the Stroop Color-Word test often used for assessing cognition (Jensen and Jr, 1966; Bjekić et al., 2018; Martinsen et al., 2014). Because this question was written for a clinician to fill out, key contextual information is absent. This particular phrase is picked up when using MI to score, but is missed when using STS for features with more direct connections to the targets. To deal with this, we suggest that phrases should be expanded to include words that connect it to the target outcome.

Another issue we do not examine in-depth is how writing style affects the outcome of the features chosen. While parts of the feature name are intrinsically linked to the outcome of the feature itself, the overall writing style (i.e., word ordering, word choice, verbosity, etc.) can be more random. Although LMs can be used to detect differences in writing style (Ríos-Toledo et al., 2022; Yamshchikov et al., 2021), the fine-tuning datasets we use here incorporate smaller positive scores for two sentence pairs sharing the same topic, and for this reason it is likely that the impact of writing style could be a smaller concern than the topics of the feature names. However, to minimize variability in selections, we still recommend adopting a uniform and detailed writing style across a set of feature names.

Although the evidence presented here suggests that there may be a negligible difference between the performance of tokenizers—as the models tested use different tokenizers—subtle differences in tokenization may change the way that semantic relationships are captured. For example, (Bostrom and Durrett, 2020) note that byte-pair encoding (Gage, 1994; Sennrich et al., 2016) has weaker performance than unigram language modeling (Kudo, 2018) with respect to morphological segmentations, implying that features tokenized with the latter algorithm may produce more meaningful STS scores.

Overall, our recommendations can be summarized as follows:
• Common and meaningful vocabulary should be prioritized to ensure an LM recognizes it.

• Acronyms should be expanded where possible, and rare or ambiguous phrases should be qualified with supporting contextual details where possible.

• Feature names should be as stylistically consistent and detailed throughout.

• Attention should be paid with the algorithm used to tokenize feature names.

6.3 Future Work

One area of future work is evaluating the performance of other measures, as the scorers evaluated here both have limitations. MI, Pearson’s \( r \), and the F-statistic are incapable of measuring the true amount of information a feature contains in context with other features, and STS only represent semantic relationships without any regard to their underlying statistical relationship.

Future work could also consider the choice of other LMs, or alternatives to fine-tuning such as adapters (Houlsby et al., 2019). Resource limitations prevented us from evaluating the performance of LMs in the billion-parameter range, although the evidence presented here suggests that there may be marginal benefits to different scoring models.

Future work regarding STS-based feature selection could also consider the use of semantic pairs that specifically rate relevancy and redundancy between pairs of questions rather than similarity. Relevant and redundant questions may not always be semantically similar, and a model fine-tuned for this may select better features than those for STS. Similarly, the performance of other selection algorithms with STS could also be evaluated.

Another potential area of future work would be to serialize the the feature selection task into a text prompt. Serialization of tabular data into a question prompt for a LLM can achieve high performance in a few-shot learning context (Hegselmann et al., 2022), and serialization of a feature selection objective may also be able to capture further semantic relationships between features.

7 Conclusion

Overall, we demonstrate that our proposed approach of using STS to score features—either alone or with statistically-based scorers—can be effective in the context of clinical survey feature selection. We discussed the intrinsic differences between STS-based and statistical scoring that result in the observed performance difference on two different datasets, as well as how different fine-tuning hyperparameters affect the performance. Finally, we suggest various considerations for writing survey questions that work with this approach, and potential areas of future research with regards to STS-based feature selection.

Limitations

One key limitation is that we only evaluate our results on one non-public and another gated-access dataset of protected health information (PHI). We were unable to find other fully-public datasets that were similar in dimensionality with descriptive feature names.

Another limitation is that many types of data were dropped from the PPSP dataset for simplicity, such as non-categorical string types. The pre-processing steps we use reduce the overall feature count from 458 features to 269 features, before any further feature selection is applied.

For the All of Us dataset, we did not include microsoft/biogpt and the ClinicalSTS fine-tuning of Bio_CinicalBERT that dataset due to package incompatibilities.

The feature selectors evaluated here are a subset of possible approaches, and future work may evaluate other baselines and STS-based selection algorithms. Further limitations that could be areas of future work are discussed further in Section 6.3.

Ethical Considerations

The data from the PPSP dataset is a part of the P5: Personalized Prediction of Persistent Postsurgical Pain study (IRB #202101123) performed at the Washington University School of Medicine in St. Louis/BJC HealthCare system. All code and model artifacts we release are not derived from any PHI collected in this study.

The features selected from such a model are not comprehensive, particularly in a clinical context. The example features we discuss are thus relative, and should not be used to inform any clinical decision-making.
Acknowledgments

This work was made possible through the support of the Department of Defense Congressionally Directed Medical Research Programs award W81XWH-21-1-0736, as well as the National Institute of Health through their support of the All of Us program. The authors would like to also thank Hanyang Liu for his valuable guidance on the final writing and publishing of this project. Finally, we would also like to thank the participants in both the P5 study and the All of Us program whose partnership made this research possible.

References

Halim Abbas, Ford Garberson, Eric Glover, and Dennis P Wall. 2018. Machine learning approach for early detection of autism by combining questionnaire and home video screening. Journal of the American Medical Informatics Association, 25(8):1000–1007. Publisher: Oxford University Press.

Fatih Abut, Mehmet Fatih Akay, and James George. 2016. Developing new VO2max prediction models from maximal, submaximal and questionnaire variables using support vector machines combined with feature selection. Computers in biology and medicine, 79:182–192. Publisher: Elsevier.

Emily Alsentzer, John R Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. 2019. Publicly available clinical BERT embeddings. arXiv preprint arXiv:1904.03323.

Yoav Benjamini and Daniel Yekutieli. 2001. The control of the false discovery rate in multiple testing under dependency. The Annals of Statistics, 29(4).

Jovana Bjekić, Marko Živanović, Danka Purić, Joukje M. Oosterman, and Saša R. Filipović. 2018. Pain and executive functions: a unique relationship between Stroop task and experimentally induced pain. Psychological Research, 82(3):580–589.

Kaj Bostrom and Greg Durrett. 2020. Byte pair encoding is suboptimal for language model pretraining. arXiv preprint arXiv:2004.03720.

Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14, Vancouver, Canada. Association for Computational Linguistics.

Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785–794.

Yao Chen, Ke Wang, and Jian John Lu. 2023. Feature selection for driving style and skill clustering using naturalistic driving data and driving behavior questionnaire. Accident Analysis & Prevention, 185:107022. Publisher: Elsevier.

Billy Chiu, Sampo Pyysalo, Ivan Vulić, and Anna Korhonen. 2018. Bio-SimVerb and Bio-SimLex: wide-coverage evaluation sets of word similarity in biomedicine. BMC bioinformatics, 19(1):1–13. Publisher: BioMed Central.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.

Dumitru Erhan, Aaron Courville, Yoshua Bengio, and Pascal Vincent. 2010. Why does unsupervised pre-training help deep learning? In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 201–208. JMLR Workshop and Conference Proceedings.

Philip Gage. 1994. A New Algorithm for Data Compression. C Users J., 12(2):23–38. Place: USA Publisher: R & D Publications, Inc.

Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH), 3(1):1–23. Publisher: ACM New York, NY.

Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks. In Proceedings of ACL.

Isabelle Guyon, Steve Gunn, Masoud Nikravesh, and Lofti A Zadeh. 2008. Feature extraction: foundations and applications, volume 207. Springer.

Paul A Harris, Robert Taylor, Robert Thielke, Jonathon Payne, Nathaniel Gonzalez, and Jose G Conde. 2009. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. Journal of biomedical informatics, 42(2):377–381. Publisher: Elsevier.

Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. 2022. TabLLM: Few-shot Classification of Tabular Data with Large Language Models. arXiv preprint arXiv:2210.10723.

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gessmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-Efficient Transfer Learning for NLP. ArXiv:1902.00751 [cs, stat].
Axthijr R Jensen and William D Rohwer Jr. 1966. The Stroop Color-Word Test: A Review. Acta Psychologica, 25.

Taku Kudo. 2018. Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75, Melbourne, Australia. Association for Computational Linguistics.

Vasileios Lampos, Bin Zou, and Ingemar Johansson Cox. 2017. Enhancing feature selection using word embeddings: The case of flu surveillance. In Proceedings of the 26th International Conference on World Wide Web, pages 695–704.

Eric Lehman, Evan Hernandez, Diwakar Mahajan, Jonas Wulff, Micah J Smith, Zachary Ziegler, Daniel Nadler, Peter Szolovits, Alistair Johnson, and Emily Alsentzer. 2023. Do We Still Need Clinical Language Models? arXiv preprint arXiv:2302.08091.

Renqian Luo, Lai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu. 2022. BioGPT: generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformatics, 23(6). Publisher: Oxford Academic.

Sofia Martinsen, Pär Flodin, Jonathan Berrebi, Monika Löfgren, Indre Bileviciute-Ljungan, Martin Ingvar, Peter Fransson, and Eva Kosek. 2014. Fibromyalgia Patients Had Normal Distraction Related Pain Inhibition but Cognitive Impairment Reflected in Caudate Nucleus and Hippocampus during the Stroop Color Word Test. PLoS ONE, 9(10):e108637.

Philip May. 2021. Machine translated multilingual STS benchmark dataset.

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.

Andriy Mulyar, Elliot Schumacher, and Mark Dredze. 2019. semantic-text-similarity.

Uli Niemann, Petra Brueggemann, Benjamin Boecking, Birgit Mazurek, and Myra Spiliopoulou. 2020. Development and internal validation of a depression severity prediction model for tinnitus patients based on questionnaire responses and socio-demographics. Scientific reports, 10(1):4664. Publisher: Nature Publishing Group UK London.

David Oniani, Sonish Sivarajkumar, and Yanshan Wang. 2022. Few-Shot Learning for Clinical Natural Language Processing Using Siamese Neural Networks. arXiv preprint arXiv:2208.14923.

Ted Pedersen, Serguei VS Pakhomov, Siddharth Patwardhan, and Christopher G Chute. 2007. Measures of semantic similarity and relatedness in the biomedical domain. Journal of biomedical informatics, 40(3):288–299. Publisher: Elsevier.

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, and others. 2011. Scikit-learn: Machine learning in Python. the Journal of machine Learning research, 12:2825–2830. Publisher: JMLR.org.

Hanchuan Peng, Fuhui Long, and Chris Ding. 2005. Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy. IEEE Transactions on pattern analysis and machine intelligence, 27(8):1226–1238. Publisher: IEEE.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and others. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.

Sergio Ramírez-Gallego, Iago Lastra, David Martínez-Rego, Verónica Bolón-Canedo, José Manuel Benítez, Francisco Herrera, and Amparo Alonso-Betanzos. 2017. Fast-mRMR: Fast minimum redundancy maximum relevance algorithm for high-dimensional big data. International Journal of Intelligent Systems, 32(2):134–152. Publisher: Wiley Online Library.

Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084.

Beatriz Remeseiro and Veronica Bolon-Canedo. 2019. A review of feature selection methods in medical applications. Computers in biology and medicine, 112:103375. Publisher: Elsevier.

Germán Ríos-Toledo, Juan Pablo Francisco Posadas-Durán, Grigori Sidorov, and Noé Alejandro Castro-Sánchez. 2022. Detection of changes in literary writing style using N-grams as style markers and supervised machine learning. PLOS ONE, 17(7):e0267590.

Valentina Siwi Saridewi and Riri Fitri Sari. 2020. Feature selection in the human aspect of information security questionnaires using multicluster feature selection. International Journal of Advanced Science and Technology, 29(7 Special Issue):3484–3493. Publisher: Science and Engineering Research Support Society.

Skipper Seabold and Josef Perktold. 2010. Statsmodels: Econometric and Statistical Modeling with Python. pages 92–96, Austin, Texas.

Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, and others. 2022. Large Language Models Encode Clinical Knowledge. arXiv preprint arXiv:2212.13138.

John W. Tukey. 1949. Comparing Individual Means in the Analysis of Variance. Biometrics, 5(2):99.

Molly R Vila, Marko S Todorovic, Cynthia Tang, Marilee Fisher, Aaron Steinberg, Beverly Field, Michael M Botros, Michael S Avidan, and Simon Haroutounian. 2020. Cognitive flexibility and persistent post-surgical pain: the FLEXCAPP prospective observational study. British journal of anaesthesia, 124(5):614–622. Publisher: Elsevier.

Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. Van Der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, António H. Ribeiro, Fabian Pedregosa, Paul Van Mulbregt, SciPy 1.0 Contributors, Aditya Vijaykumar, Alessandro Pietro Bardelli, Alex Rothberg, Andreas Hilboll, Andreas Klocekner, Anthony Scopatz, Antony Lee, Ariel Rokem, C. Nathan Woods, Chad Fulton, Charles Masson, Christian Häggström, Clark Fitzgerald, David A. Nicholson, David R. Hagen, Dmitrii V. Pasechnik, Emanuele Olivetti, Eric Martin, Eric Wieser, Fabrice Silva, Felix Lenders, Florian Wilhelm, G. Young, Gavin A. Price, Gert-Ludwig Ingold, Gregory E. Allen, Gregory R. Lee, Hervé Audren, Irvin Probst, Jörg P. Dietrich, Jacob Silterra, James T Webber, Janko Slavić, Joel Nothman, Johannes Buchner, Johannes Kulick, Johannes L. Schönberger, José Víncius De Miranda Cardoso, Joscha Reimer, Joseph Harrington, Juan Luis Cano Rodriguez, Juan Nunez-Iglesias, Justin Kuczynski, Kevin Tritz, Martin Thoma, Matthew Newville, Matthias Kümmerrer, Maximilian Bolingbroke, Michael Tartre, Mikhail Pak, Nathaniel J. Smith, Nikolai Nowaczyk, Nikolay Shebanov, Oleksandr Pavlyk, Per A. Brodtkorb, Perry Lee, Robert T. McGibbon, Roman Feldbauer, Sam Lewis, Sam Tygier, Scott Sievert, Sebastiano Vigna, Stefan Peterson, Surhud More, Tadeusz Pudlik, Takuya Oshima, Thomas J. Pingel, Thomas P. Robitaille, Thomas Spura, Thouis R. Jones, Tim Cera, Tim Leslie, Tiziano Zito, Tom Krauss, Utkarsh Upadhyay, Yaroslav O. Halchenko, and Yoshiaki Vázquez-Baeza. 2020. SciPy 1.0: fundamental algorithms for scientific computing in Python. Nature Methods, 17(3):261–272.

Jacques Wainer and Gavin Cawley. 2021. Nested cross-validation when selecting classifiers is overzealous for most practical applications. Expert Systems with Applications, 182:115222. Publisher: Elsevier.

Yanshan Wang, Naveed Afzal, Sunyang Fu, Liwei Wang, Feichen Shen, Majid Rastegar-Mojarad, and Hongfang Liu. 2020. MedSTS: a resource for clinical semantic textual similarity. Language Resources and Evaluation, 54:57–72. Publisher: Springer.

Peter Washington, Kelley Marie Paskov, Haik Kalantar-ian, Nathaniel Stockham, Catalin Voss, Aaron Kline, Ritik Patnaik, Brianna Chrismann, Maya Varma, Qandeel Tariq, and others. 2019. Feature selection and dimension reduction of social autism data. In Pacific Symposium on Biocomputing 2020, pages 707–718.

Chengwei Wei, Yun-Cheng Wang, Bin Wang, and C-C Jay Kuo. 2023. An Overview on Language Models: Recent Developments and Outlook. arXiv preprint arXiv:2303.05759.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierre Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and others. 2019. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.

Michael Wornow, Yizhe Xu, Rahul Thapa, Birju Patel, Ethan Steinberg, Scott Fleming, Michael A Pfiffer, Jason Fries, and Nigam H Shah. 2023. The shaky foundations of large language models and foundation models for electronic health records. npj Digital Medicine, 6(1):135. Publisher: Nature Publishing Group UK London.

Ying Xiong, Shuai Chen, Qingcai Chen, Jun Yan, Buzhou Tang, and others. 2020. Using character-level and entity-level representations to enhance bidirectional encoder representation from transformer-based clinical semantic textual similarity model: ClinicalSTS modeling study. JMIR Medical Informatics, 8(12):e23357. Publisher: JMIR Publications Inc., Toronto, Canada.

Ivan P. Yamshchikov, Viacheslav Shibaev, Nikolay Khlebnikov, and Alexey Tikhonov. 2021. Style-transfer and Paraphrase: Looking for a Sensible Semantic Similarity Metric. Proceedings of the AAAI Conference on Artificial Intelligence, 35(16):14213–14220.

Xi Yang, Nima PourNejatian, Hoo Chang Shin, Kaleb E Smith, Christopher Parisien, Colin Compas, Cheryl Martin, Mona G Flores, Ying Zhang, Tanja Magoc, and others. 2022. GatorTron: A Large Language Model for Clinical Natural Language Processing. medRxiv, pages 2022–02. Publisher: Cold Spring Harbor Laboratory Press.

A Private PPSP Dataset Demographics

The demographics of the PPSP dataset are reported in Table 4.
<table>
  <tr>
    <th>Name</th>
    <th>Value</th>
  </tr>
  <tr>
    <th colspan="2">PPSP Characteristics</th>
  </tr>
  <tr>
    <td>Completed Cases</td>
    <td>617</td>
  </tr>
  <tr>
    <td>Responses by Individual (mean)</td>
    <td>2.64</td>
  </tr>
  <tr>
    <td>Responses by Individual (std. dev.)</td>
    <td>1.54</td>
  </tr>
  <tr>
    <td>Responses by Individual (max)</td>
    <td>10</td>
  </tr>
  <tr>
    <td>Positive Cases</td>
    <td>25</td>
  </tr>
  <tr>
    <td>Negative Cases</td>
    <td>592</td>
  </tr>
  <tr>
    <th colspan="2">Race</th>
  </tr>
  <tr>
    <td>Caucasian</td>
    <td>497</td>
  </tr>
  <tr>
    <td>American Indian / Alaskan Native</td>
    <td>7</td>
  </tr>
  <tr>
    <td>Asian</td>
    <td>4</td>
  </tr>
  <tr>
    <td>Black / African Heritage</td>
    <td>98</td>
  </tr>
  <tr>
    <td>Hawaiian Native / Other Pacific Islander</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Other</td>
    <td>9</td>
  </tr>
  <tr>
    <td>Prefer not to answer</td>
    <td>7</td>
  </tr>
  <tr>
    <th colspan="2">Sex Assigned at Birth</th>
  </tr>
  <tr>
    <td>Female</td>
    <td>425</td>
  </tr>
  <tr>
    <td>Male</td>
    <td>185</td>
  </tr>
  <tr>
    <td>(No Answer)</td>
    <td>7</td>
  </tr>
  <tr>
    <th colspan="2">Age</th>
  </tr>
  <tr>
    <td>Age (min)</td>
    <td>19</td>
  </tr>
  <tr>
    <td>Age (mean)</td>
    <td>52.47</td>
  </tr>
  <tr>
    <td>Age (std. dev.)</td>
    <td>13.52</td>
  </tr>
  <tr>
    <td>Age (max)</td>
    <td>75</td>
  </tr>
</table>

Table 4: Demographics of the partial PPSP dataset.

B Performance By Feature Selector Hyperparameter

Tables 5 and 6 highlights the performance of all classifiers tested as fine-tuning dataset, feature selector, and STS scoring models are changed, respectively, for the All of Us dataset.

Tables 7 and 8 highlights the performance of all classifiers tested as fine-tuning dataset, feature selector, and STS scoring models are changed, respectively, for the PPSP dataset.

C Performance By Classifier

Tables 13 and 14 summarizes the performance of individual configurations of Gaussian NB for the All of Us and PPSP dataset respectively.

D Baseline Feature Scoring Performance

Figures 6a, 6b, 7a and 7b show the feature and target scorings using Pearson’s r and F-statistic for both the All of Us and Pearson’s r datasets.
Table 5: Results for fine-tuning dataset grouped by test AUROC for the All of Us dataset. One-way ANOVA \( p = 0.011 \), significant Tukey HSD results highlighted.

<table>
  <tr>
    <th>Fine-Tuning Dataset</th>
    <th>Mean</th>
    <th>Std. Dev.</th>
    <th>(1)</th>
    <th>(2)</th>
    <th>(3)</th>
  </tr>
  <tr>
    <td>Clinical Pairs (1)</td>
    <td>0.572</td>
    <td>0.091</td>
    <td>-</td>
    <td><b>0.026</b></td>
    <td><b>0.024</b></td>
  </tr>
  <tr>
    <td>Combined Pairs (2)</td>
    <td>0.594</td>
    <td>0.095</td>
    <td><b>0.026</b></td>
    <td>-</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>General Pairs (3)</td>
    <td>0.594</td>
    <td>0.101</td>
    <td><b>0.024</b></td>
    <td>1.000</td>
    <td>-</td>
  </tr>
</table>

Table 6: Results for scoring model grouped by test AUROC for the All of Us dataset. One-way ANOVA \( p = 0.001 \), significant Tukey HSD results highlighted.

<table>
  <tr>
    <th>Scoring Model</th>
    <th>Mean</th>
    <th>Std. Dev.</th>
    <th>(1)</th>
    <th>(2)</th>
    <th>(3)</th>
    <th>(4)</th>
    <th>(5)</th>
    <th>(6)</th>
  </tr>
  <tr>
    <td>Bio_ClinicalBERT (1)</td>
    <td>0.569</td>
    <td>0.080</td>
    <td>-</td>
    <td><b>0.012</b></td>
    <td>0.966</td>
    <td>1.000</td>
    <td>0.369</td>
    <td><b>0.033</b></td>
  </tr>
  <tr>
    <td>bert-base-uncased (2)</td>
    <td>0.608</td>
    <td>0.093</td>
    <td><b>0.012</b></td>
    <td>-</td>
    <td>0.121</td>
    <td><b>0.010</b></td>
    <td>0.747</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>all-MiniLM-L12-v2 (3)</td>
    <td>0.578</td>
    <td>0.097</td>
    <td>0.966</td>
    <td>0.121</td>
    <td>-</td>
    <td>0.953</td>
    <td>0.862</td>
    <td>0.243</td>
  </tr>
  <tr>
    <td>biomed_roberta_base (4)</td>
    <td>0.568</td>
    <td>0.099</td>
    <td>1.000</td>
    <td><b>0.010</b></td>
    <td>0.953</td>
    <td>-</td>
    <td>0.332</td>
    <td><b>0.027</b></td>
  </tr>
  <tr>
    <td>gatortron-base (5)</td>
    <td>0.592</td>
    <td>0.101</td>
    <td>0.369</td>
    <td>0.747</td>
    <td>0.862</td>
    <td>0.332</td>
    <td>-</td>
    <td>0.903</td>
  </tr>
  <tr>
    <td>PubMedBERT (6)</td>
    <td>0.604</td>
    <td>0.100</td>
    <td><b>0.033</b></td>
    <td>1.000</td>
    <td>0.243</td>
    <td><b>0.027</b></td>
    <td>0.903</td>
    <td>-</td>
  </tr>
</table>

Table 7: Results for fine-tuning dataset grouped by test AUROC for the PPSP dataset. One-way ANOVA \( p = 0.027 \), significant Tukey HSD results highlighted.

<table>
  <tr>
    <th>Fine-Tuning Dataset</th>
    <th>Mean</th>
    <th>Std. Dev.</th>
    <th>(1)</th>
    <th>(2)</th>
    <th>(3)</th>
    <th>(4)</th>
  </tr>
  <tr>
    <td>Combined Pairs (1)</td>
    <td>0.709</td>
    <td>0.107</td>
    <td>-</td>
    <td>0.994</td>
    <td>0.061</td>
    <td>0.457</td>
  </tr>
  <tr>
    <td>Clinical Pairs (2)</td>
    <td>0.707</td>
    <td>0.105</td>
    <td>0.994</td>
    <td>-</td>
    <td><b>0.043</b></td>
    <td>0.308</td>
  </tr>
  <tr>
    <td>ClinicalSTS (3)</td>
    <td>0.749</td>
    <td>0.089</td>
    <td>0.061</td>
    <td><b>0.043</b></td>
    <td>-</td>
    <td>0.292</td>
  </tr>
  <tr>
    <td>General Pairs (4)</td>
    <td>0.720</td>
    <td>0.101</td>
    <td>0.457</td>
    <td>0.308</td>
    <td>0.292</td>
    <td>-</td>
  </tr>
</table>

Table 8: Results for scoring model grouped by test AUROC for the PPSP dataset. One-way ANOVA \( p = 0.799 \), significant Tukey HSD results highlighted.

<table>
  <tr>
    <th>Scoring Model</th>
    <th>Mean</th>
    <th>Std. Dev.</th>
    <th>(1)</th>
    <th>(2)</th>
    <th>(3)</th>
    <th>(4)</th>
    <th>(5)</th>
    <th>(6)</th>
    <th>(7)</th>
  </tr>
  <tr>
    <td>PubMedBERT (1)</td>
    <td>0.710</td>
    <td>0.106</td>
    <td>-</td>
    <td>0.998</td>
    <td>1.000</td>
    <td>0.996</td>
    <td>1.000</td>
    <td>0.941</td>
    <td>0.999</td>
  </tr>
  <tr>
    <td>all-MiniLM-L12-v2 (2)</td>
    <td>0.717</td>
    <td>0.097</td>
    <td>0.998</td>
    <td>-</td>
    <td>1.000</td>
    <td>1.000</td>
    <td>0.999</td>
    <td>0.943</td>
    <td>0.943</td>
  </tr>
  <tr>
    <td>bert-base-uncased (3)</td>
    <td>0.712</td>
    <td>0.098</td>
    <td>1.000</td>
    <td>1.000</td>
    <td>-</td>
    <td>0.999</td>
    <td>1.000</td>
    <td>0.973</td>
    <td>0.996</td>
  </tr>
  <tr>
    <td>biomed_roberta_base (4)</td>
    <td>0.718</td>
    <td>0.109</td>
    <td>0.996</td>
    <td>1.000</td>
    <td>0.999</td>
    <td>-</td>
    <td>1.000</td>
    <td>1.000</td>
    <td>0.923</td>
  </tr>
  <tr>
    <td>gatortron-base (5)</td>
    <td>0.713</td>
    <td>0.107</td>
    <td>1.000</td>
    <td>1.000</td>
    <td>1.000</td>
    <td>-</td>
    <td>1.000</td>
    <td>0.987</td>
    <td>0.990</td>
  </tr>
  <tr>
    <td>biogpt (6)</td>
    <td>0.723</td>
    <td>0.100</td>
    <td>0.941</td>
    <td>0.999</td>
    <td>0.973</td>
    <td>1.000</td>
    <td>0.987</td>
    <td>-</td>
    <td>0.701</td>
  </tr>
  <tr>
    <td>Bio_ClinicalBERT (7)</td>
    <td>0.705</td>
    <td>0.111</td>
    <td>0.999</td>
    <td>0.943</td>
    <td>0.996</td>
    <td>0.923</td>
    <td>0.990</td>
    <td>0.701</td>
    <td>-</td>
  </tr>
</table>
Table 9: t-test results for scorer grouped by test - train AUROC.

<table>
  <tr>
    <th rowspan="2">Selection Hyperparameters</th>
    <th colspan="4">Test - Train AUROC</th>
  </tr>
  <tr>
    <th>Classifier</th>
    <th>Scorer</th>
    <th>\( \mu_B \)</th>
    <th>\( \mu_S \)</th>
    <th>\( \Delta \)</th>
    <th>\( p \)</th>
  </tr>
  <tr>
    <td colspan="7"><i>All of Us</i></td>
  </tr>
  <tr>
    <td rowspan="5">All</td>
    <td>All STS</td>
    <td></td>
    <td><b>-0.085</b></td>
    <td><b>0.091</b></td>
    <td></td>
    <td><b>5.9 \cdot 10^{-09}</b></td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td><b>-0.057</b></td>
    <td><b>0.120</b></td>
    <td></td>
    <td><b>5.9 \cdot 10^{-09}</b></td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td><b>-0.176</b></td>
    <td><b>-0.087</b></td>
    <td><b>0.089</b></td>
    <td><b>8.0 \cdot 10^{-07}</b></td>
  </tr>
  <tr>
    <td>Pearson’s r & STS</td>
    <td></td>
    <td><b>-0.105</b></td>
    <td><b>0.072</b></td>
    <td><b>2.6 \cdot 10^{-04}</b></td>
    <td></td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td><b>-0.081</b></td>
    <td><b>0.095</b></td>
    <td><b>1.5 \cdot 10^{-07}</b></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="5">Linear SVM</td>
    <td>All STS</td>
    <td></td>
    <td><b>-0.067</b></td>
    <td><b>0.071</b></td>
    <td></td>
    <td><b>0.025</b></td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td><b>-0.011</b></td>
    <td><b>0.127</b></td>
    <td></td>
    <td><b>3.0 \cdot 10^{-04}</b></td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td><b>-0.137</b></td>
    <td><b>-0.074</b></td>
    <td><b>0.064</b></td>
    <td><b>0.183</b></td>
  </tr>
  <tr>
    <td>Pearson’s r & STS</td>
    <td></td>
    <td><b>-0.098</b></td>
    <td><b>0.039</b></td>
    <td><b>0.645</b></td>
    <td></td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td><b>-0.066</b></td>
    <td><b>0.072</b></td>
    <td><b>0.085</b></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="5">MLP</td>
    <td>All STS</td>
    <td></td>
    <td><b>-0.120</b></td>
    <td><b>0.106</b></td>
    <td></td>
    <td><b>0.009</b></td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td><b>-0.087</b></td>
    <td><b>0.139</b></td>
    <td></td>
    <td><b>0.004</b></td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td><b>-0.226</b></td>
    <td><b>-0.119</b></td>
    <td><b>0.107</b></td>
    <td><b>0.025</b></td>
  </tr>
  <tr>
    <td>Pearson’s r & STS</td>
    <td></td>
    <td><b>-0.150</b></td>
    <td><b>0.076</b></td>
    <td><b>0.196</b></td>
    <td></td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td><b>-0.114</b></td>
    <td><b>0.112</b></td>
    <td><b>0.019</b></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="5">Gaussian NB</td>
    <td>All STS</td>
    <td></td>
    <td><b>-0.057</b></td>
    <td><b>0.066</b></td>
    <td></td>
    <td><b>0.003</b></td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td><b>-0.040</b></td>
    <td><b>0.084</b></td>
    <td></td>
    <td><b>0.077</b></td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td><b>-0.123</b></td>
    <td><b>-0.059</b></td>
    <td><b>0.064</b></td>
    <td><b>0.009</b></td>
  </tr>
  <tr>
    <td>Pearson’s r & STS</td>
    <td></td>
    <td><b>-0.055</b></td>
    <td><b>0.068</b></td>
    <td><b>0.027</b></td>
    <td></td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td><b>-0.062</b></td>
    <td><b>0.061</b></td>
    <td><b>0.013</b></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="5">k-NN</td>
    <td>All STS</td>
    <td></td>
    <td><b>-0.095</b></td>
    <td><b>0.119</b></td>
    <td></td>
    <td><b>4.2 \cdot 10^{-04}</b></td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td><b>-0.081</b></td>
    <td><b>0.133</b></td>
    <td></td>
    <td><b>0.003</b></td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td><b>-0.214</b></td>
    <td><b>-0.095</b></td>
    <td><b>0.119</b></td>
    <td><b>0.003</b></td>
  </tr>
  <tr>
    <td>Pearson’s r & STS</td>
    <td></td>
    <td><b>-0.115</b></td>
    <td><b>0.099</b></td>
    <td><b>0.009</b></td>
    <td></td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td><b>-0.083</b></td>
    <td><b>0.131</b></td>
    <td><b>0.002</b></td>
    <td></td>
  </tr>
  <tr>
    <td colspan="7"><i>PPSP</i></td>
  </tr>
  <tr>
    <td rowspan="5">All</td>
    <td>All STS</td>
    <td></td>
    <td><b>-0.070</b></td>
    <td><b>0.038</b></td>
    <td></td>
    <td><b>0.569</b></td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td><b>-0.091</b></td>
    <td><b>0.016</b></td>
    <td></td>
    <td><b>1.000</b></td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td><b>-0.108</b></td>
    <td><b>-0.085</b></td>
    <td><b>0.022</b></td>
    <td><b>1.000</b></td>
  </tr>
  <tr>
    <td>Pearson’s r & STS</td>
    <td></td>
    <td><b>-0.056</b></td>
    <td><b>0.051</b></td>
    <td><b>0.171</b></td>
    <td></td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td><b>-0.046</b></td>
    <td><b>0.062</b></td>
    <td><b>0.066</b></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="5">Linear SVM</td>
    <td>All STS</td>
    <td></td>
    <td><b>0.008</b></td>
    <td><b>0.039</b></td>
    <td></td>
    <td><b>0.187</b></td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td><b>-0.010</b></td>
    <td><b>0.021</b></td>
    <td></td>
    <td><b>1.000</b></td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td><b>-0.031</b></td>
    <td><b>-0.012</b></td>
    <td><b>0.019</b></td>
    <td><b>1.000</b></td>
  </tr>
  <tr>
    <td>Pearson’s r & STS</td>
    <td></td>
    <td><b>0.022</b></td>
    <td><b>0.053</b></td>
    <td><b>0.108</b></td>
    <td></td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td><b>0.032</b></td>
    <td><b>0.063</b></td>
    <td><b>0.017</b></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="5">MLP</td>
    <td>All STS</td>
    <td></td>
    <td><b>-0.074</b></td>
    <td><b>0.017</b></td>
    <td></td>
    <td><b>1.000</b></td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td><b>-0.121</b></td>
    <td><b>-0.030</b></td>
    <td></td>
    <td><b>1.000</b></td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td><b>-0.091</b></td>
    <td><b>-0.113</b></td>
    <td><b>-0.021</b></td>
    <td><b>1.000</b></td>
  </tr>
  <tr>
    <td>Pearson’s r & STS</td>
    <td></td>
    <td><b>-0.056</b></td>
    <td><b>0.036</b></td>
    <td><b>1.000</b></td>
    <td></td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td><b>-0.005</b></td>
    <td><b>0.086</b></td>
    <td><b>0.053</b></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="5">Gaussian NB</td>
    <td>All STS</td>
    <td></td>
    <td><b>0.060</b></td>
    <td><b>0.085</b></td>
    <td></td>
    <td><b>7.0 \cdot 10^{-08}</b></td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td><b>0.065</b></td>
    <td><b>0.089</b></td>
    <td></td>
    <td><b>1.5 \cdot 10^{-05}</b></td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td><b>-0.024</b></td>
    <td><b>0.050</b></td>
    <td><b>0.074</b></td>
    <td><b>9.6 \cdot 10^{-04}</b></td>
  </tr>
  <tr>
    <td>Pearson’s r & STS</td>
    <td></td>
    <td><b>0.066</b></td>
    <td><b>0.090</b></td>
    <td><b>4.6 \cdot 10^{-06}</b></td>
    <td></td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td><b>0.060</b></td>
    <td><b>0.084</b></td>
    <td><b>7.3 \cdot 10^{-06}</b></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="5">k-NN</td>
    <td>All STS</td>
    <td></td>
    <td><b>-0.273</b></td>
    <td><b>0.011</b></td>
    <td></td>
    <td><b>1.000</b></td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td><b>-0.299</b></td>
    <td><b>-0.015</b></td>
    <td></td>
    <td><b>1.000</b></td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td><b>-0.284</b></td>
    <td><b>-0.267</b></td>
    <td><b>0.017</b></td>
    <td><b>1.000</b></td>
  </tr>
  <tr>
    <td>Pearson’s r & STS</td>
    <td></td>
    <td><b>-0.257</b></td>
    <td><b>0.027</b></td>
    <td><b>1.000</b></td>
    <td></td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td><b>-0.269</b></td>
    <td><b>0.014</b></td>
    <td><b>1.000</b></td>
    <td></td>
  </tr>
</table>
Table 10: t-test results for feature selector grouped by test AUROC.

<table>
  <tr>
    <th rowspan="2">Selection Hyperparameters</th>
    <th colspan="5">Test AUROC</th>
  </tr>
  <tr>
    <th>Classifier</th>
    <th>Scorer</th>
    <th>\( \mu_B \)</th>
    <th>\( \mu_S \)</th>
    <th>\( \Delta \)</th>
    <th>\( p \)</th>
  </tr>
  <tr>
    <td colspan="7"><i>All of Us</i></td>
  </tr>
  <tr>
    <td rowspan="4">All</td>
    <td>All STS</td>
    <td></td>
    <td>0.586</td>
    <td>−0.009</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td>0.589</td>
    <td>−0.006</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td>0.595</td>
    <td>0.014</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>Pearson’s \( r \) & STS</td>
    <td></td>
    <td>0.598</td>
    <td>0.003</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td>0.579</td>
    <td>−0.016</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td rowspan="4">Linear SVM</td>
    <td>All STS</td>
    <td></td>
    <td>0.648</td>
    <td>−0.005</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td>0.627</td>
    <td>−0.026</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td>0.653</td>
    <td>0.048</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>Pearson’s \( r \) & STS</td>
    <td></td>
    <td>0.663</td>
    <td>0.010</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td>0.647</td>
    <td>−0.006</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td rowspan="4">MLP</td>
    <td>All STS</td>
    <td></td>
    <td>0.615</td>
    <td>0.006</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td>0.576</td>
    <td>−0.033</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td>0.609</td>
    <td>0.013</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>Pearson’s \( r \) & STS</td>
    <td></td>
    <td>0.631</td>
    <td>0.022</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td>0.618</td>
    <td>0.009</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td rowspan="4">Gaussian NB</td>
    <td>All STS</td>
    <td></td>
    <td>0.578</td>
    <td>−0.050</td>
    <td>0.852</td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td>0.705</td>
    <td>0.078</td>
    <td>0.234</td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td>0.628</td>
    <td><b>0.551</b></td>
    <td><b>−0.076</b></td>
    <td><b>0.002</b></td>
  </tr>
  <tr>
    <td>Pearson’s \( r \) & STS</td>
    <td></td>
    <td>0.592</td>
    <td>−0.035</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td>0.548</td>
    <td>−0.080</td>
    <td>0.002</td>
  </tr>
  <tr>
    <td rowspan="4"><i>k</i>-NN</td>
    <td>All STS</td>
    <td></td>
    <td>0.504</td>
    <td>0.010</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td>0.506</td>
    <td>0.012</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td>0.494</td>
    <td>0.010</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>Pearson’s \( r \) & STS</td>
    <td></td>
    <td>0.506</td>
    <td>0.012</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td>0.503</td>
    <td>0.009</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td colspan="7">PPSP</td>
  </tr>
  <tr>
    <td rowspan="4">All</td>
    <td>All STS</td>
    <td></td>
    <td>0.714</td>
    <td>0.026</td>
    <td>0.773</td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td>0.703</td>
    <td>0.015</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td>0.688</td>
    <td>0.012</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>Pearson’s \( r \) & STS</td>
    <td></td>
    <td>0.730</td>
    <td>0.042</td>
    <td>0.160</td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td>0.722</td>
    <td>0.035</td>
    <td>0.261</td>
  </tr>
  <tr>
    <td rowspan="4">Linear SVM</td>
    <td>All STS</td>
    <td></td>
    <td>0.710</td>
    <td>−0.019</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td>0.718</td>
    <td>−0.011</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td>0.729</td>
    <td>−0.040</td>
    <td>0.261</td>
  </tr>
  <tr>
    <td>Pearson’s \( r \) & STS</td>
    <td></td>
    <td>0.726</td>
    <td>−0.003</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td>0.706</td>
    <td>−0.023</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td rowspan="4">MLP</td>
    <td>All STS</td>
    <td></td>
    <td>0.658</td>
    <td>−0.001</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td>0.638</td>
    <td>−0.021</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td>0.659</td>
    <td>−0.014</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>Pearson’s \( r \) & STS</td>
    <td></td>
    <td>0.668</td>
    <td>0.009</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td>0.680</td>
    <td>0.020</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td rowspan="4">Gaussian NB</td>
    <td>All STS</td>
    <td></td>
    <td>0.840</td>
    <td>0.098</td>
    <td><b>8.7 \cdot 10^{-04}</b></td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td>0.839</td>
    <td>0.097</td>
    <td><b>0.024</b></td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td>0.742</td>
    <td>0.074</td>
    <td>0.261</td>
  </tr>
  <tr>
    <td>Pearson’s \( r \) & STS</td>
    <td></td>
    <td>0.857</td>
    <td>0.115</td>
    <td><b>1.7 \cdot 10^{-04}</b></td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td>0.849</td>
    <td>0.107</td>
    <td><b>1.8 \cdot 10^{-06}</b></td>
  </tr>
  <tr>
    <td rowspan="4"><i>k</i>-NN</td>
    <td>All STS</td>
    <td></td>
    <td>0.647</td>
    <td>0.027</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td>0.616</td>
    <td>−0.004</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td>0.620</td>
    <td>0.029</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>Pearson’s \( r \) & STS</td>
    <td></td>
    <td>0.667</td>
    <td>0.047</td>
    <td>0.463</td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td>0.655</td>
    <td>0.035</td>
    <td>1.000</td>
  </tr>
</table>
Table 11: t-test results for scorer grouped by test - train AUPRC.

<table>
  <tr>
    <th rowspan="2">Selection Hyperparameters</th>
    <th colspan="4">Test - Train AUPRC</th>
  </tr>
  <tr>
    <th>Classifier</th>
    <th>Scorer</th>
    <th>\( \mu_B \)</th>
    <th>\( \mu_S \)</th>
    <th>\( \Delta \)</th>
    <th>p</th>
  </tr>
  <tr>
    <td colspan="7"><i>All of Us</i></td>
  </tr>
  <tr>
    <td rowspan="5">All</td>
    <td>All STS</td>
    <td></td>
    <td>-0.076</td>
    <td>0.107</td>
    <td>1.2 \cdot 10^{-08}</td>
    <td></td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td>-0.083</td>
    <td>0.100</td>
    <td>1.3 \cdot 10^{-04}</td>
    <td></td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td>-0.183</td>
    <td>0.122</td>
    <td>1.6 \cdot 10^{-08}</td>
    <td></td>
  </tr>
  <tr>
    <td>Pearson's r & STS</td>
    <td></td>
    <td>-0.105</td>
    <td>0.078</td>
    <td>0.002</td>
    <td></td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td>-0.058</td>
    <td>0.125</td>
    <td>1.2 \cdot 10^{-08}</td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="5">Linear SVM</td>
    <td>All STS</td>
    <td></td>
    <td>-0.106</td>
    <td>0.123</td>
    <td>0.002</td>
    <td></td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td>-0.094</td>
    <td>0.134</td>
    <td>0.009</td>
    <td></td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td>-0.228</td>
    <td>0.145</td>
    <td>0.002</td>
    <td></td>
  </tr>
  <tr>
    <td>Pearson's r & STS</td>
    <td></td>
    <td>-0.160</td>
    <td>0.069</td>
    <td>0.328</td>
    <td></td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td>-0.082</td>
    <td>0.147</td>
    <td>0.002</td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="5">MLP</td>
    <td>All STS</td>
    <td></td>
    <td>-0.141</td>
    <td>0.205</td>
    <td>3.5 \cdot 10^{-05}</td>
    <td></td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td>-0.136</td>
    <td>0.209</td>
    <td>0.002</td>
    <td></td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td>-0.345</td>
    <td>0.224</td>
    <td>1.3 \cdot 10^{-04}</td>
    <td></td>
  </tr>
  <tr>
    <td>Pearson's r & STS</td>
    <td></td>
    <td>-0.190</td>
    <td>0.155</td>
    <td>0.010</td>
    <td></td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td>-0.113</td>
    <td>0.232</td>
    <td>9.2 \cdot 10^{-05}</td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="5">Gaussian NB</td>
    <td>All STS</td>
    <td></td>
    <td>-0.011</td>
    <td>0.025</td>
    <td>4.0 \cdot 10^{-05}</td>
    <td></td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td>-0.026</td>
    <td>0.010</td>
    <td>0.972</td>
    <td></td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td>-0.036</td>
    <td>0.029</td>
    <td>7.3 \cdot 10^{-08}</td>
    <td></td>
  </tr>
  <tr>
    <td>Pearson's r & STS</td>
    <td></td>
    <td>-0.012</td>
    <td>0.023</td>
    <td>0.039</td>
    <td></td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td>-0.007</td>
    <td>0.028</td>
    <td>7.3 \cdot 10^{-08}</td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="5">k-NN</td>
    <td>All STS</td>
    <td></td>
    <td>-0.041</td>
    <td>0.069</td>
    <td>3.5 \cdot 10^{-05}</td>
    <td></td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td>-0.047</td>
    <td>0.063</td>
    <td>0.037</td>
    <td></td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td>-0.110</td>
    <td>0.080</td>
    <td>8.1 \cdot 10^{-06}</td>
    <td></td>
  </tr>
  <tr>
    <td>Pearson's r & STS</td>
    <td></td>
    <td>-0.059</td>
    <td>0.051</td>
    <td>0.039</td>
    <td></td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td>-0.029</td>
    <td>0.081</td>
    <td>9.1 \cdot 10^{-06}</td>
    <td></td>
  </tr>
  <tr>
    <td colspan="7"><b>PPSP</b></td>
  </tr>
  <tr>
    <td rowspan="5">All</td>
    <td>All STS</td>
    <td></td>
    <td>-0.051</td>
    <td>0.045</td>
    <td>0.250</td>
    <td></td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td>-0.075</td>
    <td>0.022</td>
    <td>1.000</td>
    <td></td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td>-0.097</td>
    <td>0.032</td>
    <td>0.941</td>
    <td></td>
  </tr>
  <tr>
    <td>Pearson's r & STS</td>
    <td></td>
    <td>-0.037</td>
    <td>0.060</td>
    <td>0.096</td>
    <td></td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td>-0.030</td>
    <td>0.067</td>
    <td>0.030</td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="5">Linear SVM</td>
    <td>All STS</td>
    <td></td>
    <td>0.011</td>
    <td>0.052</td>
    <td>0.023</td>
    <td></td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td>-0.016</td>
    <td>0.024</td>
    <td>1.000</td>
    <td></td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td>0.006</td>
    <td>0.047</td>
    <td>0.297</td>
    <td></td>
  </tr>
  <tr>
    <td>Pearson's r & STS</td>
    <td></td>
    <td>0.027</td>
    <td>0.068</td>
    <td>0.035</td>
    <td></td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td>0.029</td>
    <td>0.070</td>
    <td>0.021</td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="5">MLP</td>
    <td>All STS</td>
    <td></td>
    <td>-0.087</td>
    <td>0.067</td>
    <td>0.458</td>
    <td></td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td>-0.132</td>
    <td>0.022</td>
    <td>1.000</td>
    <td></td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td>-0.154</td>
    <td>0.019</td>
    <td>1.000</td>
    <td></td>
  </tr>
  <tr>
    <td>Pearson's r & STS</td>
    <td></td>
    <td>-0.059</td>
    <td>0.095</td>
    <td>0.264</td>
    <td></td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td>-0.023</td>
    <td>0.131</td>
    <td>0.021</td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="5">Gaussian NB</td>
    <td>All STS</td>
    <td></td>
    <td>0.102</td>
    <td>0.065</td>
    <td>0.002</td>
    <td></td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td>0.106</td>
    <td>0.069</td>
    <td>0.012</td>
    <td></td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td>0.037</td>
    <td>0.050</td>
    <td>0.065</td>
    <td></td>
  </tr>
  <tr>
    <td>Pearson's r & STS</td>
    <td></td>
    <td>0.115</td>
    <td>0.078</td>
    <td>2.9 \cdot 10^{-05}</td>
    <td></td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td>0.100</td>
    <td>0.063</td>
    <td>0.003</td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="5">k-NN</td>
    <td>All STS</td>
    <td></td>
    <td>-0.232</td>
    <td>-0.003</td>
    <td>1.000</td>
    <td></td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td>-0.256</td>
    <td>-0.026</td>
    <td>0.627</td>
    <td></td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td>-0.229</td>
    <td>0.014</td>
    <td>1.000</td>
    <td></td>
  </tr>
  <tr>
    <td>Pearson's r & STS</td>
    <td></td>
    <td>-0.232</td>
    <td>-0.003</td>
    <td>1.000</td>
    <td></td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td>-0.224</td>
    <td>0.005</td>
    <td>1.000</td>
    <td></td>
  </tr>
</table>
Table 12: t-test results for scorer grouped by test AUPRC.

<table>
  <tr>
    <th rowspan="2">Selection Hyperparameters</th>
    <th colspan="4">Test AUPRC</th>
  </tr>
  <tr>
    <th>Classifier</th>
    <th>Scorer</th>
    <th>\( \mu_B \)</th>
    <th>\( \mu_S \)</th>
    <th>\( \Delta \)</th>
    <th>\( p \)</th>
  </tr>
  <tr>
    <td colspan="7"><i>All of Us</i></td>
  </tr>
  <tr>
    <td rowspan="5">All</td>
    <td>All STS</td>
    <td></td>
    <td>0.034</td>
    <td>−0.001</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td>0.034</td>
    <td>−0.002</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td>0.036</td>
    <td>−0.002</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>Pearson’s r & STS</td>
    <td></td>
    <td>0.037</td>
    <td>0.002</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td>0.033</td>
    <td>−0.002</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td rowspan="5">Linear SVM</td>
    <td>All STS</td>
    <td></td>
    <td>0.045</td>
    <td>−0.004</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td>0.038</td>
    <td>−0.011</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td>0.049</td>
    <td>0.046</td>
    <td>−0.003</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>Pearson’s r & STS</td>
    <td></td>
    <td>0.050</td>
    <td>0.001</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td>0.044</td>
    <td>−0.005</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td rowspan="5">MLP</td>
    <td>All STS</td>
    <td></td>
    <td>0.040</td>
    <td>0.001</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td>0.035</td>
    <td>−0.004</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td>0.039</td>
    <td>0.040</td>
    <td>0.001</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>Pearson’s r & STS</td>
    <td></td>
    <td>0.043</td>
    <td>0.004</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td>0.042</td>
    <td>0.003</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td rowspan="5">Gaussian NB</td>
    <td>All STS</td>
    <td></td>
    <td>0.027</td>
    <td>−0.006</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td>0.046</td>
    <td>0.013</td>
    <td>0.173</td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td>0.033</td>
    <td><b>0.023</b></td>
    <td><b>−0.010</b></td>
    <td><b>1.9 \cdot 10^{-06}</b></td>
  </tr>
  <tr>
    <td>Pearson’s r & STS</td>
    <td></td>
    <td>0.030</td>
    <td>−0.003</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td><b>0.023</b></td>
    <td><b>−0.010</b></td>
    <td><b>1.9 \cdot 10^{-06}</b></td>
  </tr>
  <tr>
    <td rowspan="5"><i>k</i>-NN</td>
    <td>All STS</td>
    <td></td>
    <td>0.025</td>
    <td>0.003</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td>0.022</td>
    <td>0.001</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td>0.021</td>
    <td>0.003</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>Pearson’s r & STS</td>
    <td></td>
    <td>0.027</td>
    <td>0.005</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td>0.025</td>
    <td>0.004</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td colspan="7"><b>PPSP</b></td>
  </tr>
  <tr>
    <td rowspan="5">All</td>
    <td>All STS</td>
    <td></td>
    <td>0.273</td>
    <td>0.027</td>
    <td>0.673</td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td>0.253</td>
    <td>0.007</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td>0.246</td>
    <td>0.267</td>
    <td>0.021</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>Pearson’s r & STS</td>
    <td></td>
    <td>0.287</td>
    <td>0.041</td>
    <td>0.190</td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td>0.285</td>
    <td>0.039</td>
    <td>0.190</td>
  </tr>
  <tr>
    <td rowspan="5">Linear SVM</td>
    <td>All STS</td>
    <td></td>
    <td>0.252</td>
    <td>−0.019</td>
    <td>0.838</td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td>0.245</td>
    <td>−0.026</td>
    <td>0.695</td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td>0.271</td>
    <td>0.244</td>
    <td>−0.027</td>
    <td>0.673</td>
  </tr>
  <tr>
    <td>Pearson’s r & STS</td>
    <td></td>
    <td>0.271</td>
    <td>−0.000</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td>0.247</td>
    <td>−0.024</td>
    <td>0.695</td>
  </tr>
  <tr>
    <td rowspan="5">MLP</td>
    <td>All STS</td>
    <td></td>
    <td>0.214</td>
    <td>−0.021</td>
    <td>0.915</td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td>0.191</td>
    <td>−0.045</td>
    <td>0.134</td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td>0.236</td>
    <td>0.209</td>
    <td>−0.027</td>
    <td>0.673</td>
  </tr>
  <tr>
    <td>Pearson’s r & STS</td>
    <td></td>
    <td>0.228</td>
    <td>−0.008</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td>0.230</td>
    <td>−0.006</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td rowspan="5">Gaussian NB</td>
    <td>All STS</td>
    <td></td>
    <td>0.423</td>
    <td>0.126</td>
    <td><b>4.3 \cdot 10^{-04}</b></td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td>0.410</td>
    <td>0.114</td>
    <td>0.056</td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td>0.296</td>
    <td>0.406</td>
    <td>0.110</td>
    <td>0.056</td>
  </tr>
  <tr>
    <td>Pearson’s r & STS</td>
    <td></td>
    <td>0.440</td>
    <td>0.143</td>
    <td><b>1.9 \cdot 10^{-04}</b></td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td>0.435</td>
    <td>0.138</td>
    <td><b>3.5 \cdot 10^{-05}</b></td>
  </tr>
  <tr>
    <td rowspan="5"><i>k</i>-NN</td>
    <td>All STS</td>
    <td></td>
    <td>0.204</td>
    <td>0.023</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>F-Score & STS</td>
    <td></td>
    <td>0.167</td>
    <td>−0.015</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>MI & STS</td>
    <td></td>
    <td>0.182</td>
    <td>0.211</td>
    <td>0.029</td>
    <td>0.954</td>
  </tr>
  <tr>
    <td>Pearson’s r & STS</td>
    <td></td>
    <td>0.211</td>
    <td>0.030</td>
    <td>0.673</td>
  </tr>
  <tr>
    <td>STS</td>
    <td></td>
    <td>0.229</td>
    <td>0.047</td>
    <td>0.673</td>
  </tr>
</table>
Table 13: Results for Gaussian NB with baseline feature selection methods and using UFNLP/gatortron-base to score features for the All of Us dataset. * is best test AUROC for given feature selector among ours. † is smallest test-train AUROC difference for given feature selector among ours.

<table>
  <tr>
    <th rowspan="2">Selection Hyperparameters</th>
    <th rowspan="2">Feature Selector</th>
    <th rowspan="2">Scorer</th>
    <th rowspan="2">FT Dataset</th>
    <th colspan="3">AUROC</th>
    <th colspan="3">AUPRC</th>
  </tr>
  <tr>
    <th>Test</th>
    <th>Train</th>
    <th>Δ</th>
    <th>Test</th>
    <th>Train</th>
    <th>Δ</th>
  </tr>
  <tr>
    <td>mRMR</td>
    <td>Pearson's r & STS</td>
    <td></td>
    <td>General Pairs</td>
    <td><b>0.823</b></td>
    <td>0.761</td>
    <td>0.062</td>
    <td><b>0.076</b></td>
    <td>0.093</td>
    <td>-0.017</td>
  </tr>
  <tr>
    <td>Top N†</td>
    <td>Pearson's r & STS</td>
    <td></td>
    <td>General Pairs</td>
    <td>0.580</td>
    <td>0.591</td>
    <td>-0.011</td>
    <td>0.024</td>
    <td>0.028</td>
    <td>-0.004</td>
  </tr>
  <tr>
    <td>Top N</td>
    <td>Pearson's r & STS</td>
    <td></td>
    <td>Combined Pairs</td>
    <td>0.628</td>
    <td>0.718</td>
    <td>-0.090</td>
    <td>0.031</td>
    <td>0.071</td>
    <td>-0.040</td>
  </tr>
  <tr>
    <td>mRMR†</td>
    <td>Pearson's r & STS</td>
    <td></td>
    <td>Combined Pairs</td>
    <td>0.540</td>
    <td>0.556</td>
    <td>-0.016</td>
    <td>0.022</td>
    <td>0.026</td>
    <td>-0.004</td>
  </tr>
  <tr>
    <td>Std. Dev.</td>
    <td>Pearson's r & STS</td>
    <td></td>
    <td>Clinical Pairs</td>
    <td>0.613</td>
    <td>0.711</td>
    <td>-0.099</td>
    <td>0.027</td>
    <td>0.039</td>
    <td>-0.013</td>
  </tr>
  <tr>
    <td>Std. Dev.†</td>
    <td>MI & STS</td>
    <td></td>
    <td>Clinical Pairs</td>
    <td>0.568</td>
    <td>0.664</td>
    <td>-0.096</td>
    <td>0.024</td>
    <td>0.034</td>
    <td>-0.010</td>
  </tr>
  <tr>
    <td>Std. Dev.†</td>
    <td>STS</td>
    <td></td>
    <td>Clinical Pairs</td>
    <td>0.568</td>
    <td>0.664</td>
    <td>-0.096</td>
    <td>0.024</td>
    <td>0.034</td>
    <td>-0.010</td>
  </tr>
  <tr>
    <td>mRMR</td>
    <td>Pearson's r</td>
    <td></td>
    <td>-</td>
    <td>0.745</td>
    <td>0.743</td>
    <td><b>0.003</b></td>
    <td>0.045</td>
    <td>0.068</td>
    <td>-0.023</td>
  </tr>
  <tr>
    <td>Top N</td>
    <td>MI</td>
    <td></td>
    <td>-</td>
    <td>0.689</td>
    <td>0.647</td>
    <td>0.042</td>
    <td>0.034</td>
    <td>0.036</td>
    <td><b>-0.002</b></td>
  </tr>
  <tr>
    <td>Std. Dev.</td>
    <td>MI</td>
    <td></td>
    <td>-</td>
    <td>0.685</td>
    <td>0.688</td>
    <td>-0.003</td>
    <td>0.033</td>
    <td>0.038</td>
    <td>-0.004</td>
  </tr>
  <tr>
    <td>mRMR</td>
    <td>MI</td>
    <td></td>
    <td>-</td>
    <td>0.670</td>
    <td>0.748</td>
    <td>-0.079</td>
    <td>0.042</td>
    <td>0.067</td>
    <td>-0.026</td>
  </tr>
  <tr>
    <td>SFM-XGBoost</td>
    <td>-</td>
    <td></td>
    <td>-</td>
    <td>0.654</td>
    <td>0.800</td>
    <td>-0.147</td>
    <td>0.056</td>
    <td><b>0.165</b></td>
    <td>-0.108</td>
  </tr>
  <tr>
    <td>Top N</td>
    <td>Pearson's r</td>
    <td></td>
    <td>-</td>
    <td>0.648</td>
    <td>0.759</td>
    <td>-0.111</td>
    <td>0.029</td>
    <td>0.086</td>
    <td>-0.056</td>
  </tr>
  <tr>
    <td>Identity</td>
    <td>-</td>
    <td></td>
    <td>-</td>
    <td>0.602</td>
    <td>0.790</td>
    <td>-0.188</td>
    <td>0.026</td>
    <td>0.053</td>
    <td>-0.027</td>
  </tr>
  <tr>
    <td>RFE</td>
    <td>-</td>
    <td></td>
    <td>-</td>
    <td>0.574</td>
    <td><b>0.830</b></td>
    <td>-0.256</td>
    <td>0.026</td>
    <td>0.085</td>
    <td>-0.059</td>
  </tr>
  <tr>
    <td>mRMR</td>
    <td>F-statistic</td>
    <td></td>
    <td>-</td>
    <td>0.552</td>
    <td>0.758</td>
    <td>-0.206</td>
    <td>0.027</td>
    <td>0.067</td>
    <td>-0.041</td>
  </tr>
  <tr>
    <td>SFM-LinearSVM</td>
    <td>-</td>
    <td></td>
    <td>-</td>
    <td>0.547</td>
    <td>0.775</td>
    <td>-0.228</td>
    <td>0.023</td>
    <td>0.052</td>
    <td>-0.029</td>
  </tr>
  <tr>
    <td>Std. Dev.</td>
    <td>Pearson's r</td>
    <td></td>
    <td>-</td>
    <td>0.536</td>
    <td>0.720</td>
    <td>-0.183</td>
    <td>0.022</td>
    <td>0.040</td>
    <td>-0.018</td>
  </tr>
</table>

Table 14: Results for Gaussian NB with baseline feature selection methods and using emilyalsentzer/Bio_Clinical-BERT to score features for the PPSP dataset.

<table>
  <tr>
    <th rowspan="2">Selection Hyperparameters</th>
    <th rowspan="2">Feature Selector</th>
    <th rowspan="2">Scorer</th>
    <th rowspan="2">FT Dataset</th>
    <th colspan="3">AUROC</th>
    <th colspan="3">AUPRC</th>
  </tr>
  <tr>
    <th>Test</th>
    <th>Train</th>
    <th>Δ</th>
    <th>Test</th>
    <th>Train</th>
    <th>Δ</th>
  </tr>
  <tr>
    <td>Std. Dev.</td>
    <td>F-statistic & STS</td>
    <td></td>
    <td>General Pairs</td>
    <td><b>0.914</b></td>
    <td>0.828</td>
    <td>0.085</td>
    <td>0.499</td>
    <td>0.309</td>
    <td><b>0.190</b></td>
  </tr>
  <tr>
    <td>Std. Dev.†</td>
    <td>STS</td>
    <td></td>
    <td>General Pairs</td>
    <td>0.840</td>
    <td>0.811</td>
    <td>0.029</td>
    <td>0.458</td>
    <td>0.329</td>
    <td>0.129</td>
  </tr>
  <tr>
    <td>mRMR†</td>
    <td>F-statistic & STS</td>
    <td></td>
    <td>General Pairs</td>
    <td>0.615</td>
    <td>0.616</td>
    <td><b>-0.001</b></td>
    <td>0.140</td>
    <td>0.117</td>
    <td>0.022</td>
  </tr>
  <tr>
    <td>Top N</td>
    <td>Pearson's r & STS</td>
    <td></td>
    <td>Combined Pairs</td>
    <td>0.906</td>
    <td>0.798</td>
    <td>0.109</td>
    <td>0.495</td>
    <td>0.373</td>
    <td>0.122</td>
  </tr>
  <tr>
    <td>Top N†</td>
    <td>MI & STS</td>
    <td></td>
    <td>Clinical Pairs</td>
    <td>0.852</td>
    <td>0.776</td>
    <td>0.076</td>
    <td>0.443</td>
    <td>0.339</td>
    <td>0.104</td>
  </tr>
  <tr>
    <td>mRMR</td>
    <td>Pearson's r & STS</td>
    <td></td>
    <td>Clinical Pairs</td>
    <td>0.841</td>
    <td>0.793</td>
    <td>0.048</td>
    <td>0.333</td>
    <td>0.269</td>
    <td>0.064</td>
  </tr>
  <tr>
    <td>Top N</td>
    <td>F-statistic</td>
    <td></td>
    <td>-</td>
    <td>0.907</td>
    <td>0.814</td>
    <td>0.093</td>
    <td>0.482</td>
    <td>0.324</td>
    <td>0.157</td>
  </tr>
  <tr>
    <td>Top N</td>
    <td>Pearson's r</td>
    <td></td>
    <td>-</td>
    <td>0.907</td>
    <td>0.814</td>
    <td>0.093</td>
    <td>0.482</td>
    <td>0.324</td>
    <td>0.157</td>
  </tr>
  <tr>
    <td>Std. Dev.</td>
    <td>Pearson's r</td>
    <td></td>
    <td>-</td>
    <td>0.845</td>
    <td>0.764</td>
    <td>0.080</td>
    <td>0.291</td>
    <td>0.196</td>
    <td>0.095</td>
  </tr>
  <tr>
    <td>SFM-XGBoost</td>
    <td>-</td>
    <td></td>
    <td>-</td>
    <td>0.817</td>
    <td>0.828</td>
    <td>-0.011</td>
    <td>0.434</td>
    <td>0.391</td>
    <td>0.043</td>
  </tr>
  <tr>
    <td>mRMR</td>
    <td>F-statistic</td>
    <td></td>
    <td>-</td>
    <td>0.790</td>
    <td>0.852</td>
    <td>-0.062</td>
    <td>0.393</td>
    <td>0.415</td>
    <td>-0.022</td>
  </tr>
  <tr>
    <td>mRMR</td>
    <td>Pearson's r</td>
    <td></td>
    <td>-</td>
    <td>0.770</td>
    <td>0.753</td>
    <td>0.017</td>
    <td>0.220</td>
    <td>0.178</td>
    <td>0.043</td>
  </tr>
  <tr>
    <td>Top N</td>
    <td>MI</td>
    <td></td>
    <td>-</td>
    <td>0.738</td>
    <td>0.879</td>
    <td>-0.140</td>
    <td>0.384</td>
    <td>0.396</td>
    <td>-0.012</td>
  </tr>
  <tr>
    <td>Std. Dev.</td>
    <td>MI</td>
    <td></td>
    <td>-</td>
    <td>0.737</td>
    <td><b>0.879</b></td>
    <td>-0.142</td>
    <td>0.383</td>
    <td>0.391</td>
    <td>-0.008</td>
  </tr>
  <tr>
    <td>mRMR</td>
    <td>MI</td>
    <td></td>
    <td>-</td>
    <td>0.719</td>
    <td>0.736</td>
    <td>-0.017</td>
    <td>0.219</td>
    <td>0.190</td>
    <td>0.029</td>
  </tr>
  <tr>
    <td>RFE</td>
    <td>-</td>
    <td></td>
    <td>-</td>
    <td>0.622</td>
    <td>0.623</td>
    <td>-0.002</td>
    <td>0.142</td>
    <td>0.119</td>
    <td>0.022</td>
  </tr>
  <tr>
    <td>Std. Dev.</td>
    <td>F-statistic</td>
    <td></td>
    <td>-</td>
    <td>0.615</td>
    <td>0.616</td>
    <td>-0.002</td>
    <td>0.140</td>
    <td>0.117</td>
    <td>0.022</td>
  </tr>
  <tr>
    <td>SFM-LinearSVM</td>
    <td>-</td>
    <td></td>
    <td>-</td>
    <td>0.598</td>
    <td>0.758</td>
    <td>-0.160</td>
    <td>0.153</td>
    <td>0.203</td>
    <td>-0.050</td>
  </tr>
  <tr>
    <td>Identity</td>
    <td>-</td>
    <td></td>
    <td>-</td>
    <td>0.583</td>
    <td>0.645</td>
    <td>-0.062</td>
    <td>0.130</td>
    <td>0.126</td>
    <td><b>0.005</b></td>
  </tr>
</table>
(a) Pearson’s \( r \) from the PPSP dataset.

(b) F-statistic from the PPSP dataset.

Figure 6: Feature-feature and feature-target pairings from the PPSP.

(a) Pearson’s \( r \) from the All of Us dataset.

(b) F-statistic from the All of Us dataset.

Figure 7: Feature-feature and feature-target pairings from the All of Us dataset.

Reviewer paper summaries: The paper uses a semantic similarity based method to select features for supervised prediction models trained on survey data. Surveys and questionaires are a very common study type in epidemiology and health research, in which patients are asked questions on their condition, lifestyle, treatment etc. Answers are used to build models predictive of some class, such as the outcome of an intervention. Surveys may be administered multiple times in a study, and can contain large number of questions each wit many possible answers. This leads to a high dimensional feature space, and typically manual or statistical techniques are used to reduce this through relevant feature selection before training models. The authors' intuition is that the language used in the question itself contains information that could be useful in this feature selection task. They propose an approach that uses the semantic similarity between questions and between questions and outcomes to decide which questions (features) are most relevant. Semantic similarity is computed using language models, fine tuned on semantic text similarity data. Multiple models are evaluated over two survey datasets, using multiple classifiers.
Summary of strengths: A well written paper on an interesting and important topic. The authors introduce a nove, semantic similarity based measure for feature selection, which give performance benefits.

Well designed experiments with good baselines, and one large dataset for replication.

Thorough evaluation of multiple classifiers and models - though see also below under weaknesses.

The authors go beyond seeing this as a set of NLP experiments in isolation from their use-case, making suggestions as to how survey writing might be adapted to make better use of semantic similarity techniques.
Summary of weaknesses: In total the authors evaluated 1929 feature selection / classifier pairings. This is a lot, and I think stretches the credibility of the statistics even with correction for multiple testing. What is the power of the analysis? Can you comment further on this?

The work is restricted to health survey data. It would be interesting to hear how this work might be applied to other survey areas, and potentially other areas where there is a text key (questions) with values (answers).
Comments, suggestions and typos: I struggled to find any minor errors - very well written.

I found lines 259 to 272 confusing, and needed to go back and forth to table 1 to understand what you meant. Having read this section, I expected to find discussion of results for clinical, general and combined pairs but couldn't see any outside of the appendices and one mention on line . Which pair trained models did you use in the experiments reported in Tables 2 and 3?

**Comments after Author Discussion**
I have read the author response. It would be good if the authors could include something on statistical power in the final version of the paper. I have not changed my scores.
Overall assessment scores: 4.5