LLM-Select: Feature Selection with Large Language Models
Authors: Daniel P Jeong
Zachary Chase Lipton
Pradeep Kumar Ravikumar
Venue: Accepted by TMLR ()

Abstract: In this paper, we demonstrate a surprising capability of large language models (LLMs): given only input feature names and a description of a prediction task, they are capable of selecting the most predictive features, with performance rivaling the standard tools of data science. Remarkably, these models exhibit this capacity across various query mechanisms. For example, we zero-shot prompt an LLM to output a numerical importance score for a feature (e.g., ``blood pressure'') in predicting an outcome of interest (e.g., ``heart failure''), with no additional context. In particular, we find that the latest models, such as GPT-4, can consistently identify the most predictive features regardless of the query mechanism and across various prompting strategies. We illustrate these findings through extensive experiments on real-world data, where we show that LLM-based feature selection consistently achieves strong performance competitive with data-driven methods such as the LASSO, despite never having looked at the downstream training data. Our findings suggest that LLMs may be useful not only for selecting the best features for training \textit{but also for deciding which features to collect in the first place}. This could potentially benefit practitioners in domains like healthcare and the social sciences, where collecting high-quality data comes at a high cost.

Published in Transactions on Machine Learning Research (04/2025)
LLM-Select: Feature Selection with Large Language Models
Daniel P. Jeong!
danielje@cs.com cmm...ccccccooommmoo edu
Zachary C. Lipton
zlipton @cmu. edu
Pradeep Ravikumar'
prodeepr cma. cmma edu
Maahine Learning Deportment, Carnegie Mellon University
Abbridge AT
Reviewed on OpenReview: https: openrevee.. net/ forum? id- 16f7ea133
Abstract
In this paper, we demonstrate surprising capability of large language models (LLMs): given only input feature names and a description of a prediction task, they are capable of selecting the most predictive features, with performance rivaling the standard tools of data science. Remarkably, these models exhibit this capacity across various query mechanisms. For example, we zero-shot prompt an LLM to output a numerical importance score for a feature (e-g., "blood pressure") in predicting an outcome of interest (e.g., "heart failure"), with no additional context. In particular, we find that the latest models, such as GPT-4, consistently identify the most predictive features regardless of the query mechanism and across various prompling strategies. We illustrate these findings (hrough extensive exper- iments on real-world data, where we show that LLM-based feature selection consistently achieves strong performance competitive with data-driven methods such as the LASSO, despite never having looked at the downstream training data. Our findings suggest that LLMS may be useful not only for selecting the best features for training but also for deciding which features to collect in the first place. This could benefit practitioners in domains like healthcare and the social sciences, where collecting high-quality data comes at high cost.
1 Introduction
Transformer-baseer large language models (LLMs) pretrained massive text corpora for next-word predic- tion exhibit the remarkable capability to generalize to unseen tasks, simply by conditioning an input prompt that contains task-relevant instructions and a small number of examples (Vaswani et al., 2017; Rad- ford et al., 2019; Brown et al., 2020). With sufficient model scale and an appropriate prompting strategy, these models demonstrate strong performance on various commonsense, symbolic, and arithmetic reasoning tasks (Lewkowyez et al., 2022; Wei al., 2022b; Kojima et al., 2022; Suzgun et al., 2023; Anil al., 2023) and complex ionstionn-n and prediction lasks that require real-world knowledge (Petroni et al., 2019; Liévin et al., 2022; Singhal et al., 2023; Manikandan et al., 2023). Such findings suggest that by pretraining on vast amounts of text from various domains, LLMs encode rich knowledge about rcal-world relationships, which they can leverage for performing various downstream lasks (Choi et al., 2022; Moor et al., 2023).
In this paper, we demonstrate that LLMs are capable of performing feature selection for supervised learning tasks. Given that we are often aware of the real-world semantics associated with the input features (c.g., "blood pressure") and the target outcome (e.g., "heart failure") in a downstream training dataset, we investi- gate effective ways of prompting an LLM to identify the most informative features for predicting the outcome (Figure 1(a)). For example, prompt the LLM with "Rank the following features by their importance for predicting the incidence of heart failure: blood pressure, ... creatinine.' "1 and select the top-ranked features
We note that the examples here are simplifications of exact prompts used in our study. See Appendices C-F for details.

---

Published in Transactions on Machine Learning Research (04/2025)
(a)
(b)
Best Data-Driven Baseline
LLM-Score (Durs)
Random Se selection
LLM-Sotre
Candidate Features hright, sice, history failing...
"Pooide
for each festure in predicting heart faiture"
"Rank the tollowing foatures by their importance in predicting heart fullure."
LLM W
0.7
MLM Seg
Ranked Features heartrate,
0.6
"Slven selected features, output
feature add predicting
pressure, creadirine...
0.5 Small-Scale Catasets Small Dasasata
PIMICIV
1400
LUN
Figure 1: Selecting features by zero-shot prompting an LLM leads to strong downstream predictive perfor- mance, competitive with data-driven feature selection methods. (a) Overview of our proposed LLM-SCORE, LLM-RANK, and LLM-SEQ methods (Section 3). (b) Average test AUROC (higher is better) on classifica- tion datasets when selecting the top 30% of features according to the best-performing data-driven bascline each dataset (in red), LLM-SCORE based on GPT-4 (in blue), and a random feature selection baseline (in black). Error bars indicate standard eтror across datasets in each group.
to train a downstream prediction model. Alternatively, for each candidate feature, we prompt the LLM with "Provide a feature importance score between 0 and 1 for (candidate feature) for predicting the incidence of heart failure." and select the features with the highest LLM-generated scores for training.
Surprisingly, find that even without looking at the training data, these models capable of identifying the most predictive features, with performance rivaling those of data-driven feature selection methods such as the LASSO (Tibshirani, 1996) (Figure 1(b)). Such result is counterintuilive; given that datasets with identical input feature names can correspond to arbitrarily different data distributions, due to factors such as selection bias and confounding, it is not a priori obvious that selecting features only based on their names will be effective. Remarkably, we find that for the latest models such as GPT-4, even zero-shol prompting the L.M to generate a numerical feature importance score one-featurr-att---imme can lead to strong feature selection performance. Moreover, we find that such LLM-generated feature importance scores are correlated with several commonly used feature importance metrics (e.g., Fisher score (Gu et al., 2011), SHAP (Lundberg & Lee, 2017)), suggesting that LLMs are capable of distilling the real-world relationships encoded in their parameters into statistically meaningful scores. We demonstrate these findings by stress-testing our LLM- based feature selection methods against traditional data-driven methods on real-world datasets from diverse domains (e.g., finance, healthcare, criminal justice), using both closed-source (GPT-4 (OpenAI, 2023), GPT- 3.5 (Brown et al., 2020)) and open-source LLMs (Llama-2 (Touvron et al., 2023)) of various sizes.
Our main ributiinionns can be summarized follows:
We propose three approaches to prompting an LLM for feature selection: (i) selecting features with the highest LLM-generated feature importance scores (LLM-SCORE): (ii) selecting features based on an LLM-generated ranking (LLM-RANK): and (iii) sequentially selecting features in dialogue with LLM (LLM-SBR) (Section 3)-
2. We show that even without access to the downstream training data, LLMs of sufficient scale achieve strong feature selection performance real-world datascts, often competitive with traditional data- driven feature selection methods such as the LASSO (Result 2, Section 4.1; Section 4.2).
3. We comprehensively assess the sensitivity of LLM- feature selection methods to various prompt- ing strategies and demonstrate that even zero-shot prompting the LLM with no additional context about the downstream data can elicit strong feature selection performance (Result 3, Section 4.1).
4. We show that model scale increases, LLM-generated importance scores generally exhibit higher rank correlation with commonly used feature importance metrics (Result 4, Section 4.1).
Our findings suggest that LLMs may be useful not only for selecting the most predictive features after data collection, but also for deciding what features to collect in the first place. This could benefit
2

---

Published in Transactions on Machine Learning Research (04/2025)
practitioners in domains like healthcare and the social sciences, where obtaining high-quality data (e.g.. running medical tests for patients, designing survey questions) can be expensive and time-consuming.
2 Related Work
2.1 Prompting LLMs
Prompting an effective method for adapting off-the-shelf LLMs to perform new tasks unseen during training, without explicit gradient updates to the model parameters (Radford al., 2019; Liu et al., 2023). In a standard prompting setup, output is autoregressively sampled from an LLM conditional on text descriptions of a desired task and optionally a set of in-context input-output examples (Brown et al., 2020) and used as solution for the given task. Even without task-specific fine-tuning, such a zero-shot or few-shot in-context learning approach can be surprisingly effective for adapting pretrained LLMs towards a wide range of natural-languaae tasks (Hendrycks et al., 2021; Lin et al., 2022; Patel & Pavlick, 2022; Srivastava et al., 2023), given a language model of sufficient scale (Wei al., 2022a).
Meanwhile, several works show that LLM outputs can be highly sensitive to the specifics of the input prompt and choice of decoding strategy (Jiang et al., 2020; Zhao et al., 2021; Sclar et al., 2024; Jeong et al., 2024a;bb. As such, choosing an appropriate prompting strategy is crucial, especially for challenging reasoning tasks. In our experiments, we mainly consider two prompting vechiqqes-chain-----tee chain-of-thought prompting (CoT; Wei et al., 2022b) and self-consistency decoding (Wang et al., 2023). We focus on these methods as they often dramatically boost performance on tasks that require multi-step reasoning (Kojima et al., 2022; Lewkowycz et al., 2022; Chen et al., 2023), which we hypothesizod to be important for feature selection.
Chain-of-thought prompting (CoT). CoT prompting (Wei et al., 2022b) is few-shot prompting method that augments each input-output example with a chain-of-thought- coherent scries of natural- language reasoning steps leading to the correct answer. Wang et al. (2023) show that given a large-enough model and an appropriately designed input prompt, CoT prompting can elicit logically consistent step-by- step solutions from the LLM and substantially boost performance on complex tasks such as solving math problems (Lewkowyez et al., 2022; Imani et al., 2023) and answering commonsense (Suzgun al., 2023; Anil et al., 2023) and knowledge-intensiie reasoning questions (Liévin et al., 2022; Singhal et al., 2023).
Self-consistency decoding. A common sampling strategy used for LLMs is greedy decoding (Radford et al., 2019; Brown et al., 2020; Chowdhery et al., 2022), where at each Loken-generation step, the Loken with the highest probability is taken as the output. In self-consistency decoding (Wang et al., 2023) on the other hand, multiple outputs are randomly sampled from the LLM (via c.g., temperature sampling (Ackley et al., 1985)) and marginalized generate the final prediction. Prior works suggest that for CoT prompting, self- consistency decoding can significantly boost performance and that it is especially beneficial when diverse set of reasoning paths are possible for solving given task (Wang et al., 2023; Lewkowyez et al., 2022).
Extensions of CoT prompting. More recently, oree-of (ToT; Yao et al., 2023) and graph-of- thoughts (GoT; Besta et al., 2024) prompting have been proposed as generalizations of CoT prompting (with self-consistency decoding) to further enhance the exploration and evaluation of multiple reasoning paths. ToT prompting formulates reasoning as traversal over tree of plausible partial solutions, allowing for explicit branching and backtracking of diverse reasoning paths in a hierarchical manner. GoT prompting further generalizes ToT prompting by formulating reasoning via graph, additionally allowing for exploration and dependencies across different reasoning paths. In our paper, we focus on standard CoT prompting (Wei et al., 2022b) for an initial investigation on L.LM-based feature selection in the simplest setting. We leave an in-depth investigation of whether extensions such as ToT and GoT prompting can improve LLM-based feature selection performance as future work.
2.2 Feature Selection
Feature selection a classical machine learning problem, where given a set of candidate features, the goal is to select the most informative feature subset that is predictive of outcome of interest (Blum & Langley,
3

---

Published in Transactions on Machine Learning Research (04/2025)
1997; Guyon & Elissceff, 2003; Chandrashekar & Sahin, 2014; Li et al., 2017). Feature selection methods can generally be grouped into three categories: filter, wrapper, and embedded methods. Filter methods (Tazar et al., 2012) select features by ranking them according to a statistical or information- criterion e.g., mutual information (Lewis, 1992; Ding & Peng, 2005; Bennasar et al., 2015), Fisher score (Duda et al., 2001; Gu et al., 2011), maximum mean discrepancy (Song et al., 2012) and choosing the top ranked features, independent of the downstream learning algorithm. Wrapper methods identify a locally optimal feature subset that maximizes the performance of the downstream prediction model (Kohavi & John, 1997; Yamada et al., 2014; Chen et al, 2017; Climente- González et al., 2019), often by employing heuristic search strategy (e.g., sequential selection (Ferri et al., 1994; Luo & Chen, 2014; Yasuda et al., 2023). recursive feature elimination (RFE; Guyon et al., 2002)). Embedded methods select features as part of the model learning process, most commonly based on regularization techniques that encourage feature sparsity (Tibshirani, 1996; Yuan & Lin, 2006; Feng & Simon, 2017; Lemhadri et al., 2021), and others based on specialized neural network architectures (Lu et al., 2018; Balm et al., 2019; Yamada et al., 2020). In our experiments, we compare our methods against traditional feature selection baselines from all three calegories.
LLMs for feature selection. In a prior work most similar to ours, Choi et al. (2022) propose the LMPriors framework, where they prompt the davinci-instruct-bet. whether each candidate feature should be used to predict the target outcome, and select features whose difference in log- probabbbiilsss for generating "Y" (Yes) or "N" (No) token crosses predefined threshold. Our work differs from theirs in two key aspects. First, we propose three different feature selection methods which all directly use the generated text output and not the associated token probabilities, which are often not directly accessible in closed-source, proprietary LLMs. Second, we provide a more comprehensive evaluation across various model scales and prompting strategies on a larger collection of datasets and derive practical insights. Our proposed feature selection methods perform as strongly as theirs even without the same level of access into the LLM (Appendix A.3.3).
variant of GPT-3 (Brown et al., 2020) to answer
3 Selecting Features with LLMs
We address the standard supervised learning setup where, given labeled data D = {(x10, y with x(9) e Rd and (()) E V, our goal is to learn prediction model f E F such that f f= arg minses Eo(C(f. D)) for some model class F and loss function L. We assume access to concepls e = .c....., cal for the input features and cy for the prediction target, which are text descriptions that capture their real-world semantics. For example, when predicting heart failure (1 if positive, O otherwise) given a patient's blood pressure and weight measurements, = I'blood pressure", "weight"), Cy = "heart failure", and æ = [.1, z2] denotes the numerical measurements used to Iearn f. Concept annotations are widely available in many practical settings, e.g., as column names in labular dalasets via datasheets that contain auxiliary metadata. For feature selection, our goal is to find a subset S C {l....,d} d} of size k < d such that a model fs trained on (1) )())
Ds
{(=))
where z (i)
las. ......... (1)] achieves strong performance under a budget on k.
3.1 Feature Selection with LLMs
To leverage a pretrained LLM M for feature selection, we prompt M with the input concepts and target concept Cy and select features based on the generated output. We consider the following three approaches: (i) selecting features based on LLM-generated feature importance scores; (ii) selecting features based on an L.M-generrted ranking; and (iii) sequentially selecting features in a dialogue with an LLM. We design separate prompt templates for each approach and denote them by prompt promptranks and prompt respectively. Each prompt template can be viewed as a function of the input and target concepts which out- puts a set of natural- languae instructions tailored to the corresponding selection strategy. While generally, text outputs generated from an LLM need to be processed further to extract the relevant information, we omit such steps in the notation below for simplicity.
Selection based on LLM-generated feature importance scores (LLM-SCORE). In this approach, we prompt M for a set of numerical feature importance scores [$1.-.., sa] with 8y E [0, 1] Vj (1.....d) d).
4

---

Published in Transactions on Machine Learning Research (04/2025)
where high 8j indicates that an imput concept сj is closely related to cy. Formally, we can represent this
8j ((prompl.caroe)..com Cy), Vj E (1.....d].
(1)
We then define S to be the set of indices of the top-k concepts with the highest importance scores and use Ds to learn a downstream prediction model fs. Given that the LLM is only given a single input concept ej and the target concept by each time it is prompted, we hypothesize that 8 captures the maryinal importance of each feature for predicting the target, as informed by the knowledge encoded in M. We note that the feature importance scores are directly parsed from the text output and do not correspond to the token probabilities associated with generating the text output. We also note that the score range of [o, 1] is an arbilrary choice and therefore evaluate the sensitivily of LLM-SCORE to different choices in Appendix A.3.4.
Selection based on an LLM--enerated feature ranking (LLM-RANK). In this approach, we prompt M for a ranking r = [er,,..,, Carl of all input concepts, where the input concepts e are ordered by their conceptual relevance to cy. Formally, we can represent this as
r= Mc Cy)).
(2)
We define 8 to be the set of indices of the top-k highest ranked concepts and use Ds to learn a downstream prediction model fs. We hypothesize that the rank of each input concept reflects its relative importance for predicting the target, with respect to all of the other input concepts in c.
Sequential selection in a dialogue with an LLM (LLM-SEQ). In this approach, we consider selection strategy analogous to sequential selection methods. We start with an empty set of concepts and iteratively add a new concept by prompting the LLM to select a candidate concept that would maximally improve the cross-validatie performance of a downstream prediction model. Formally, assuming that our goal is to select concepts, at each iteration k, we have
Mppromphac(CSs
(3)
where c(t) denotes the f-th selected input concept, S. S {1.....d} d} denotes the subset of concept indices selected up to the t-th iteration, and So = 0. We then use Ds. to train a downstream prediction model fs.. where we tune the hyperparameters via 5-fold cross- vrlssvaaliitionn. The cross-validatte validation performance of fs. on Ds, is then appended to the prompt used for the next iteration (Appendix E). For the initialization of the feature subset, we also consider starting with S = (arg max, 85) containing the concept with the highest score from Equation (1) and iterating over t = ,,....,:: However, we focus on the former approach in the main text, as it empirically performs better than the latter (we compare the two approaches in Appendix A.3.5). We hypothesize that this approach encourages the LLM to select a feature that is maximally informative with respect to the feature subset already selected, each iteration of the algorithm. Meanwhile, we note that since LLM-SEQ is a greedy sequential selection approach, the resulting top-k feature subset S is not guaranteed to be globally optimal. This limitation arises as at each iteration t, the next feature to add is determined with respect to St-1, which precludes an exploration all possible subsets of k.
For instantiating each method, we use LLMs that have been fine-tuned via instruction tuning and reinforce- ment learning from human feedback (RLHF) (Christiano et al., 2017; Stiennon et al., 2022; Ouyang et al., 2022), which are generally better at following instructions and capable of handling conversational contexts. However, emphasize that the LLMs are not fine-tuned in any way on the downstream dataset D.
4 Experiments
In this section, we demonstrate the effectiveness of the three LLM-based feature selection methods introduced in Section 3 on various real-world prediction tasks. For all of our experiments, we use the following LLMs:
1. GPT-4 (OpenAT, 2023): 11.7T parameters,
2. GPT-3.5 (Brown et al., 2020): ~17B parameters,
5

---

Published in Transactions on Machine Learning Research (04/2025)
3. Llama-2 (Touvron al., 2023): 70B parameters,
4. Llama-2 (Touvron et al., 2023): 13B parameters,
Llama-2 (Touvron et al., 2023): 7B parameters.
For GPT-4 and GPT-3.55 we the gpt-4-0613 and gpt-3. models available via the OpenAI APL. We clarify that for both models, the official parameter counts have not been disclosed by OpenAl, and that the approzimate (~) number of parameters listed here are rumored estimates. For Llama-2, we use the 2aam--2-1bcchat-hae and the vLLM framework (Kwon et al., 2023) to increase throughput and speed up output generation.
HuggingPace checkpoints 2a7ma---00ccht--ti
and use
Prompt design. We provide all of the prompl templates used in our experiments in Appendices C-E. The prompt templates were carefully constructed to reliably elicit the desired response in the correct format from each LLM. In the default template, provide a feature importance score between 0 and 1 for predicting (target concept) and a reasoning behind how the importance score was assigned."), (ii) output format instructions (e.g.. "Output your answer in a JSON format."), and (iii) the main user prompt (e.g., "Provide score and reasoning for (concept)."). We emphasize that the default prompts are not "fine-tuned" on each dataset in any way, as they only embed the imput and target concepts and no other dataset-specific information. Meanwhile, we examine how the following changes to the input prompt affect feature selection performance:
only include (i) the main system prompt (e.g., "Your task is to
1. Adding dataset-specific context: When prompling the LLM to select features for dataset D, we investigate whether adding auxiiaary information about D (e.g., data collection process, cohort) helps better contextualize the importance cach fcature and improve feature selection performance.
2. Adding few-shot examples: We investigate whether adding few-shot examples improves LLM- based feature selection performance via in-context learning. For instance, when generating feature importance scores (as in Equation (1)), we include example concepls from along with their human- annotated feature importance scores (e.g., ["blood pressure", 0.9]) in prompt
Adding CoT explanations: Given the empirical success of CoT prompting (Wei et al., 2022b) in improving the reasoning capabilities of LLMs in few-shot settings, we investigate whether adding CoT reasoning into the few-shot examples (e.g,, ["blood pressure", "Blood pressure is important for... Thus, the score is 0.9.,, 0.9) improves the performance of LLM-based feature selection.
For all three changes, we manually construct the relevant inputs via human annotation. For datasct-spocific context, we manually sunmarize the metadata associated with each dataset (e.g., sourced from the "Data Card" of Kaggle dataset). For selecting the few-shot examples and generating their CoT explanations, we prioritize features whose semantic relevance the target is unambiguous and least open subjective interpretation, based on common sense and relevant domain knowledge. For given dataset, the few-shot examples and CoT explanations provided to the LLMs remain fixed across all queries. See Appendix C for full details on the examples and CoT explanations for each dataset considered in Sections 4.1 4.2.
Concretely, we consider the following six variations of the prompt template in our experiments:
1. Default (No change),
2. Default + Examples,
3. Default + Examples with CoT,
Default + Context,
Default + Context + Examples,
6. Default + Context + Examples with CoT.
For LLM-RANK and LLM-SEQ, these variations are less straightforward to implement, due to e.g. limited context windows or ambiguity in constructing a valid example. For example, for LLM-RANK, the full list of concepts must be added, along with the main system prompt and output format instructions. We thus focus L.L.-SCORE. for exploring how these variations impact feature selection performance.

---

Published in Transactions on Machine Learning Research (04/2025)
Decoding. By default, greedy decoding (i.e., sampling with temperature T= 0), given its straight- forward and deterministic behavior. For GPT-4 and GPT-3.5, we account for their inherent non-determinism at the time of writing by sampling 5 responses with T = 0 and averaging the feature importance scores parsed from the text generations. Meanwhile, as there is no clear notion of ground truth when assigning feature importance (e.g,, what the ground-truth importance score for "blood pressure" when predicting "heart failure"?) and multiple reasoning paths are possible for determining importance, we also consider self-consistency decoding (Wang al., 2023). For the latter, we set T 0.5 and average across 5 samples.
Source code. To ensure the reproducibility of our results, we open-source the source code used for all of our evaluations detailed below via our Gut repository?.
4.1 Evaluation on Small-Scale Datasets
We compare LLM-SCORE, LLM-RANK, and LLM-SBQ against several feature selection methods using small-scale, low-dimensional datasets from various domains (e.g., healthcare, criminal justice), each with ~10-70 features after preprocessing. Here, we focus on the siaall-scale setting to stress-test the LLM- based feature selection methods in various ways and to allow comparison with methods that are less scalable to high-dimensional settings (e.g,, sequential selection baselines). We use seven binary classifi- cation datasets (Credit-G, Bank, Give Me Some Credit, COMPAS Recidivism, Pima Indians Diabetes, AUS Cars* YouTube*) and seven regression datasets (CA Housing, Diabetes Progression, Wine Quality, Miami Housing, Used Cars, NBA*, NYC Rideshare"), where those marked with an asterisk (*) are datasets published after the pretraining data cutoff dates of all of the LLMs evaluate". We include these datasets to ensure that findings generalize to datasets that the LLMs have not been trained on.
We provide the remaining details on all datascts in Appendix A.1.
We evaluate each feature selection method by measuring how the test performance a downstream prediction model changes as we vary the proportion features selected from 10% to 100%, in approximately 10% increments. On cach dataset and cach proportion, we measure the test performance of an Lz-penalizod logistic/linear regression model trained using the selected features. For each training run, we perform model selection via grid search with 5-fold volssvaaii We use the area under the ROC curve (AUROC) and mean absolute error (MAE) to measure performance classification and regression tasks, respectively. For LLM- based feature selection, we select the concepts c in 10% increments, which may each correspond to more than one feature after preprocessing if the concept is categorical (e.g., one-hot encoding for "et.hicitty").
Baselines. We compare the LLM-based feature selection methods against the following baselines:
1. LassoNet (Lemhadri et al., 2021).
2. the LASSO (Tibshirani, 1996),
3. forward sequential selection,
4. backward sequential selection,
recursive feature elimination (RFE; Guyon et al., 2002),
6. minimum redundancy maximum relevance (MRMR; Ding & Peng, 2005),
filtering by mutual information (MI; Lewis, 1992).
8. HSIC-Lasso (Yamada et al., 2014),
Concrete Autoencoder (CAE; Balm et al., 2019),
10. Sequential Attention (SA; Yasuda et al., 2023),
11. random feature selection.
Phttps://githu wtt.ccooooomooooooo..eo /tt///t///llmlmseett For GPT-4 and GPT-3.5, the cutoff date Sep., 2021 the time of writing (reference). For Llama-2, the cutoff date is Sep., 2022 (reference). As L.Mss regularly updated, the knowledge cutoff dates may be different. more recent versions.
7

---

Published in Transactions on Machine Learning Research (04/2025)
0.9
Ranking by MAE
12.5
0.B
AUROC
10.0
(a)
0.7
7.5
5.0
2.5
0.5
0.0
MI
LASSO Forward
HStC-Lasso
GPT-4 1-1.7T)
CAE
MRMR
L Llama-2 (13B)
1778
Backward
Llama-2 -2 (7B)
Llama-2 (70B)
RFE
LassoNet Random
SA
Forward
MRMR
LASSO
HSIC-Lassso Lasso
GPT4 (-1.7T)
MI
GPT-3.8 5 (-1758)
CAE
LassoNet
Liama- (70B)
Backward
RFE
Liama-2 (78)
Llama-2 (13B)
Random
SA
GPT-3 5
LLM-Score (GPT-4)
LLM-Score
LLM-Score
LLM-Score
LLM-Score
- Best Baseline
(GPT-3.5)
(Llama-2, 70B)
(Llama22, 138)
(Llama-2, 78)
Random
(b)
YouTube*
NBA*
(c)
YouTube*
NBA
3
AUROC
0.8
AUROC
0.8
MAE
22
MAE
0.7
0.6
0.6
0.4 0.6 1.0
0.4
0.6
0.8
1.0
0.8 Fraction Selected
0.2
0.6
0.2 0.6 0.8
Fraction Selected
Fraction Selected
Fraction Selected
Figure 2: LLM-SCORE shows competitive feature selection performance against data-driven baselines, given an LLM of sufficient scale. (a) Average AUROC (left; higher is better) and ranking by MAE (right; lower is better) across all datasets when selecting the top 30% of features. (b) Feature selection paths for LLM- SCORE (GPT-4), the best-performing baseline, and random selection on dalasels published after the LLM cutoff dates. (c) Feature selection paths for LLM-SCORE on the same datasets, with varying LLM scale.
For LassoNet and the LASSO, we first compute the regularization paths with warm starts (Friedman et al., 2010) to identify the regularization coefficients corresponding each feature proportion, and then train a separate downstream model as described above. We take this two-step approach to docouple the effects of feature selection and regularization on the downstream test performance. For forward forward/backward baakwardd sequen- tial selection, greedily add/remove new feature at each iteration based the 5-fold cross-validation performance resulting from addingg/rmovvnng features with the smallest weights in a gggistc/llneaa regression model selected via grid search with 5-fold cross-validatiin using all features. For MI, we select features with the highest marginal mutual information with the target variable based on the training data. When continuous features and/or labels are present, we use the nearest-ncighbor approximations (Kraskov et al., 2004; Ross, 2014) available in scikit-learn to estimate the empirical mutual information. For HSIC-Lasso, we compute the full kernel matrix if the number of training examples is less than 1000, but otherwise use the block-wise approximation method by Climente- Glnnelte-Gonzélzz et al. (2019), using the recommended block size B 20 and number of permutations M = 3. For CAE, we use multi- perceptron (MLP) decoder with hidden layers of width 256 and train the model for a maximum of 1000 epochs with a supervised learning objective (as in Appendix F of Balm al. (2019)). For SA, we use an MLP with 1 hidden layer of width 67 and ReLU activation, following Yasuda al. (2023). For all experiments, we repeat and average the results over random seeds, which control the train-validation splits used for cross-validation and the behavior of random feature selection, ensure the robustness of results. We include the remaining details in Appendix A.2.
each candidate fcature. For RFE, we recursively climinate
Result 1: LLM-based feature selection methods achieve strong performance competitive with data-driven baselines, with sufficient LLM scale (Figure 2). Figure 2(a) shows the downstream test performance, averaged over all of the classification (left) and regression (right) datasets, when selecting the top 30% of features according to each baseline and LLM-SCCRE based on GPT-4, GPT-3.5 and Llama-2.
We the architecture in the official repository for CAF: httpps:/ghthub..omm/ http://mmmaaannneenteeeeeee CConrreteeCeeeeeeeee ConereeeeAuttenncodee Autoencaders.
8

---

Published in Transactions on Machine Learning Research (04/2025)
GPF-4 GPI4(-1.77),
GPE4[-1.77
(8)
CreditGG
Bank
Give Me Some Credi
COMPAS Recidivam
Pima Indianns Dlobetos
AUS Carst
Yo.Tlbel
0.00
AUNDC
0.18
0.80
70
MRRCC
2.00
AUROC
800
AUROC
AUROC
0.88
2.00
0.74
0.12
0.82
0.70
03 14 6.6 08
22. 04 0.6 0. 1-4 Praction Selected
62
04 06 04 1.0
32
244 6- 644 1.5
02
24 6-6 6-4 1-4
02. 0.1 6-6 0.6 1.
32 0.0 0.6
Fractian Selected
Fracikon Selected
Fractam Selected
Praction Selecied
Fraction S leceedd
Fractkon Sellcceed
C4 Hausing
Dabette Progression
0.00
Wine Quality
Miami Houshn
Used Chrs
KBA*
NYC Rideshhre"
525
$2.0
2.00
1.00
47.5
45.0
2.50
0.2
9.4
2:30
2.0
01 04 06 06 3.2
32
24
0.6 0, 3.3
02 64 355 0.0 10
31 14 0.6 1.0 1.0
02 04
85 06 1.
3:00
0.2
04 0.5 0.6
022 04 06 00 L0
fraction Selected
Traction Selected
Fraction Selected
Trackarn Selected
fraction Selecied
Fraction lecteed
Fraction Selecced
(-1/58)
(b)
0.75
0.00
C 6
Bank
3.00
Give Me Some Credi
COMPAS Redidiiin
0.62
Pina Indions Diobetos
AUS Cars
Yo DuuTube
AONOC
PURDE
2o
3.50
AUROC
0.70
0.80
AUROC
AURCC
3.00
0.76
3.70
0.14
2.5
54 0.6 6- 15
22 04 688 0.
62 04
36
08 1/0
32 S4 65 048 1.5
52 14 6. 6.
02 6.6 0. 1.3
32 0.4 0.6
Fraction Selected
fraction Selected
Fracton Selected
Fraction Selected
Fraction Selected
Fraction Se lected
Fraction Selected
CA Housing
Diocaess Progression
Wine Quality
Miami Housing
Used Cars
KEA
NYC Rid:shore*
0.48
MAC
2.35
MAE
MAE
3:20
2:50
3:5
02 04 06 08 1.5 Fraction Selected
24 06 02
428
32 5. 06 18
08 1.5
0.2 0. 0.
622 0. 1e Fraction Selected
Praction Selected
Fraction Selected
Fraction Selected
Fraction Selected
Fraction Se letteed
Figure 3: Feature selection paths for LLM-SCORE, LLM-RANK, and LLM-SEQ based on (a) GPT-4 and (b) GPT-3.5 on all classification and regression datasets. Within each panel, the top row shows the results on the classification datasets, and the bottom row shows the results on the regression datasets. GPT-4- based methods all show consistently strong performance across datasets, showing substantial overlap in their corresponding feature selection paths. GPT-3.5- methods also show similar trends, which are albeit less pronounced. Dalasets marked with an asterisk (*) were published after the LLM culoff dales.
For the regression datasets, we report the average ranking based on test MAE to account for differences in the scale of MAE values across datasets. For the larger models, notably GPT-4 (in blue) and GPT- 3.5 (in green), selecting features based on L.I.-generated importance scores leads to strong rewnstream performance on average, competitive with the data-driven baselines. We observe similar results for LLM- RANK and LLM-SEQ (Figure A1). As an example, in Figure 2(b), we show the feature selection paths (i.c., "test performance vs. fraction of features selected" curves) for the best-performing baseline on each dataset, LLM-SCORE based GPT-4, and random selection on the YouTube* and NBA* datasets published after the LLM cutoff dates. Here, GPT-4-based LLM-SCORE outperforms the best baseline (LASSO) on the YouTube* dataset, achieving higher AUROC overall, and performs as strongly as that (HSIC-Lasso) on the NBA* * dataset. Meanwhile, performance is less consistent with smaller LLMs (Figure 2(c)). For example, LLM-SCORE based on Llama-2 (7B) (in yellow) performs well on the YouTube* datasct but close to random on the *BA* dataset. These results demonstrate that using LLMs for feature selection can be effective, but suggest that a sufficiently large model may be required for reliable performance. We include the full results comparing LLM-SCORE, LLM-RANK, and LLM-SEQ to the data-driven basclincs in Appendix A.3.1.
Result 2: All three LLMebased feature selection methods achieve similarly strong performance (Figure 3). The feature selection paths for LLM-SCORE, LLM-RANK, and LLM-SEQ based on GPT- 4 overlap significantly on all datasets except. Diabetes Progression and NYC Rideshare* (Figure 3())). Figures 2 and 3(a) together illustrate that GPT-4 achieves consistently strong feature selection performance regardless of the selection mochanism. Notably, even the simple strategy of querying for an importance score one-feature-ate aaneture-at-a-tinne a-ime can be just as effective as those that account for the other features. We overall observe similar trend in the feature selection paths for GPT-3.5 (Figure 3(b)); which is albeit less pronounced.
9

---

Published in Transactions on Machine Learning Research (04/2025)
20
Average Improvement (%)
[Defsult)"
Two (Examples)
10
T-o (Exampics with CoT)
T-0 Examples)
THO [Comest Examples with (aT)
Tw0.5
T-0.5 T- (Examples)
10
Te0.5 (Exemples CoT)
(Contee)
Te0.5 (Contedd Esamples)
-20
(-1-7T-1777)
GPT-3.5 (-1758)
Liama-2 (70B)
Llama-2 (138)
Llama-2 (7B)
Tw0.5 (Contest + Esamplas with CoT)
Figure 4: Changes in average improvement (%) in LLM-ScORE feature selection performance as we vary the decoding strategy (T 0: greedy, T = 0.5: self-consistency) and prompt design (in parentheses), compared to the performance achieved under the default prompting setup (in bold; see Section 4). Error bars indicate standard error across datasets. On average, no approach substantially improves over the default setting.
Meanwhile, for the smaller Llama-2 models, the performances LLM-SCORE, LLM-RANK, and LLM-SEQ are less consistent across datasets (Figures A3-A5), and no single method necessarily outperforms the others. These results suggest that while all three LLM-based feature selection methods can be similarly effective, their effectiveness varies more significantly across datasets as the model size decreases.
Result 3: Zero-shot prompting with no dataset-specific context and greedy decoding results in strong feature selection performance (Figure 4). To assess the sensitivity of LLM-SCORE to prompt design and decoding strategy variations, we evaluate the downstream test performance across six prompt designs ("prompt design" in Section 4) and two decoding strategies change in the area under the feature selection paths to mcasure the impact of each variation. For classifi- cation tasks, an increase in the area indicates improved performance, it suggests that fewer features are needed to achieve high AUROC. Conversely, for regression tasks, a decrease in the area indicates improve- ment. Thus, we quantify improvement in feature selection performance by computing the % increase for the classification tasks and the % decrease for the regression tasks. Figure 4 shows the average improvement for each (prompt design, decoding strategy) wair-with respect to that achieved with the default prompt and greedy decoding (T = 0). For the settings where we add examples, we only consider a one-shot setting given the relatively small number of features in the datasets. We find that no prompting strategy consistently im- proves feature selection performance across all LLMs, sometimes even degrading it (e.g., GPT-3.5, Llama-2 (13B)). Meanwhile, find that the largest. and most capable GPT-4 is substantially less sensitive to the prompting strategy and generally benefits from additional context, albeit to a limited extent. These obser- vations suggest that zero-shot prompting without dataset-specific context, combined with greedy decoding, is a strong baseline prompting strategy for LLM-based feature selection.
We use the
Result 4: LLM-SCORE exhibits higher correlation with widely used feature importance metrics as model scale increases (Figure 5). To probe the semantics of L.M feature importance metries, we measure the alignment between LLM--CCOE and the following feature importance metrics:
SHAP (Lundberg Lee, 2017),
Fisher score (Duda et al., 2001),
mutual information,
Pearson correlation,
Spearman correlation,
permutation importance (Breiman, 2001).
We compute the Kendall's т coefficient e [-1,1] (Kendall, 1938) to quantify the agreement in pairwise For SHAP, we compute the mean absolute
orderings, where +1/-1 indicates perfect Shapley value for each feature using all test samples, after training an XGBoost model (Chen Guestrin, 2016). For permutation importance, we measure the average drop in test performance of an Ieppenalized
10

---

Published in Transactions on Machine Learning Research (04/2025)
Average Rank Correlation
0.5
0.0
-0.5
GPT-4 (-1.7T)
GPT-3.5 (-175B)
Llama-2 (70B)
Llama-2 (138)
Llama-2 (7B)
SHAP: XG8oost
Fisher Score
Mutual Information
Pearson
Spearman
Permutation
Figure 5: Average rank correlation (Kendall's r) between each feature importance metric and LLM-SCORE based on GPT-4, GPT-3.5, and Llama-2. Error bars indicate standard error across datasets. LLM-SCORE generally exhibits higher rank correlation with standard feature importance metrics as model scale increases, but does not uniquely align to specific notion of feature importance.
logistie/linear regression model when randomly shuffling the values of each feature 30 times. In Figure 5, we observe that there is no specific notion of importance that LLM-SCORE consistently aligns to. Meanwhile, as model scale increases, the LLM-gencrated scores generally exhibit higher correlation with the importance metrics considered (e-g., SHAP, Spearman). An interesting future research direction may be to investigate prompting strategies that increase the alignment of LLM-SCORE to desired notions of importance. We include additional results on investigating the semantics of LLM-SCORE in Appendix A.3.7.
4.2
Evaluation on Large-Scaat -cale Datasets
We show that LLM-based feature selection also achieves strong performance on the more complex large-scale, higher dimensional datasets, each with ~30000 features after preprocessing. Here, we focus on GPT-4- -4-based LLM-SCORE with the default prompting setup, given its consistently strong performance on small-scale datasets and scalability a large number of features. We construct supersets of the Income, Employment, Public Coverage, and Mobility datasets from folktables (Ding et al., 2021) by extracting all features available from the 2018 American Community Survey data for California while removing features that lead to label leakage. We also manually extract three datasets from the MIMIC-IV database (Johnson et al., 2023) for classifying whether an ICU patient was diagnosed with chronic kidney disease (CKD), chronic obstructive pulmonary discase (COPD), and heart failure (HF). Importantly, we note that MIMIC-IV is not publicly available (access requires special credentials via PhysioNet (Goldberger el. al., 2000)) and that the datasets we manually derived are not based on existing data preprocessing pipelines (e.g., MIMIC-Exxtade (Wang al., 2020), FIDDLE (Tang et al., 2020)). As such, these exact datasets were not part of the LLM pretraining corpora. We provide the remaining dataset details in Appendix B.1.
We evaluate each feature selection method by measuring how the test performance of a downstream prediction model changes when we select the top 10%, 30%, 50%, 70% or 100% of all input concepts. We train the following models for downstream prediction: LightGBM (Ke et al., 2017), MLP, and Læ-penalized logistic regression. For each feature selection method and dataset, we perform model selection via a random search with 40 hyperparameter samples. We average the test AUROC over 5 random seeds, which control the trainivalidation splits, the hyperparameter samples, and the initialization of model parameters.
Baselines.
We compare LLM-SCORE based on GPT-4 against the following feature selection baselines:
LassoNet (Lemhadri et al., 2021),
the LASSO (Tibshirani, 1996),
3. minimum redundancy maximum relevance (MRMR; Ding & Peng, 2005),
HSIC-Lasso (Yamada et al., 2014),
5. Concrete Autoencoder (CAE; Balm et al., 2019),
6. random feature selection.
11

---

Published in Transactions on Machine Learning Research (04/2025)
(a)
folktables
folktables
AUROC
0.8
AUROC
0.7
0.5
LIM-Gcore
MAVR
CAT
HSID-
Random
LASGO
Lasnobir:
MRNB
LASSD
LLM-ScoM
HSICAas
Lxouhet
Random
LLM-Sccre
-
Best Baseline
Random
(b)
folktables::
folktables:
(c)
IV CKD
MINIC.IV COPD
0.50
0.800
0.85
8
AURICC
0.68
0.750
0.65
0.54
0.80
0.64
0,700
0.62
9.675
0.60
0.78
0.650
0.6
0.8
Fraction Selected
Fraction Selected
Selected
Selected
Figure 6: LLM-SCORE based on GPT-4 shows competitive feature selection performance against data-driven basclines on the larger folktables and MIMIC-IV datasets, each with ~03000 features. For all plots, we show the results with GBM as the prediction model. (a) Average AUROC across all folktables (left) and MIMIC-IV (right) datasets and downstream prediction models when selecting the top 30% of features. Error bars indicate the standard error across datasets. (b) Feature selection paths for LLM-SCORE, the best-performing baseline, and random selection on the Income and Employment datasets (folktables). (c) Feature selection paths for the same feature selection methods the CKD and COPD datasets (MIMIC-IV).
Unlike for the small-scale dataset experiments, we exclude forward sequential selection, backward sequential selection, recursive feature elimination (Guyon al., 2002), and Sequential Attention (Yasuda et al., 2023), given the high computational cost of training O(kd) separate models when sclecting k out of d candidate fea- tures. For LassoNet, the LASSO, and the random feature selection baseline, set up each method to select features at the comcept level (e.g.. "ethnicity" instead of the one-hot encoded features "ethnicity "Asian or "ethnicity Hispanic") to match the behavior of LLM-SCORE (e.g., via group-wise feature sparsity (Yuan & Lin, 2006)). For LassoNet and the LASSO, we compute the regularization paths with warm starts (Friedman et al., 2010) (Figure B1) and use the regularization coefficient that selects about the same number of features as LLM-SCORE in each ovaluation setting. For MRMR, HSIC-Lasso, and CAE, we seleet exactly the same number of features LLM-SCORE in each setting. We use the same setup for HSIC-Lasso as in Section 4.1 and additionally subsample down to 60000 training examples if the training set size exceeds it to avoid out-of-memory errors (Employment and Public Coverage; Appendix B.1). For CAE, we adopt the sctup in Section 4.1 but increase the hidden layer widths for the MLP decoder from 256 to 512. We provide the remaining details in Appendix B.2.
Result. Figure 6(a) shows the downstream test AUROC, averaged over all of the folktables (left) and MIMIC-IV (right) datasets, when (i) selecting the top 30% of features according to each baseline and LLM- ScoRE based on GPT-4 and (ii) using LightGBM as the downstream prediction model. Here, we show the performance when selecting the top 30% features, as the test performance roughly saturates to that of using all features when selecting more than 30% of features for all feature selection methods (Figure B2(a)). We observe that GPT-4-based LLM-SCORE overall performs the best on the folktables datasets, and performs on par with HSIC-Lasso and significantly better than LassoNet and the random feature selection baseline the MIMIC-IV datasets. In Figures 6(b-c), show the full feature selection paths for the best- performing bascline (MRMR), LLM-SCORE, and random feature selection on the Income and Employment datasets from folktables and the CKD and COPD dalasets from MIMIC-IV. We observe that for most of these datasets, only a small subset of features are highly predictive of the target outcome, as indicated by the high test AUROC of the best-performing bascline and the low test AUROC of the random feature selection baseline at the 10% and 30% marks. We find that LLM-SCORE is effective selecting these highly predictive features, with performance rivaling that of the best-performing data-driven baseline and
12

---

Published in Transactions on Machine Learning Research (04/2025)
substantially better than random when selecting the top 10% and 30% of features. Notably, the strong performance on the MIMIC-IV datasets suggests that LLM-SCOOE can be effective even in highly specialized domains like healtheare where, without access to sufficient training data, substantial domain knowledge would be required for feature selection (Section B.1.2). We include the full results for all downstream prediction models on all datasets Figure B2 in Appendix B.3.
5 Discussion and Conclusion
In this work, we demonstrated that LLMs are capable of performing feature selection for supervised learn- ing tasks, even without access the dounstream training data. We proposed three different approaches to prompting LLMs for feature Sitoction-LLLM-SE
LLM-RANK, and LLM-SEQ (Section 3). We found that with sufficient LLM scale, even zero-shot prompting an LLM to select features can result in strong downstream predictive performance, often competitive with data-driven methods such as the LASSO (Tīb- shirami, 1996) (Results 1 & 3, Section 4.1). For the latest models such as GPT-4, we found that all three LLM-driven methods achieve similarly strong performance and that even the simple strategy of querying for an importance score one-frature-at-a-time (LLM-SCORE) can be as effective as those that account for other available features (Result 2, Section 4.1; Section 4.2). In particular, while the LLM-generated scores do not necessarlly align to a specific notion of feature importance (e.g., Fisher score, mutual information), we found that their rank correlations tend to be higher as model scale increases (Result 4, Section 4.1), suggesting that the most capable LLMs are able to translate the real-world relationships encoded in their parameters into stalistically meaningful scores. Our findings suggest. that LLMs may be useful not only for selecting the best features post data collection, but also for deciding what features to collect in the first place. This could potentially benefit practitioners in domains like healthcare and the social sciences, where collecting high-quality data can be expensive and time-consuminng
Limitations. First, while we have demonstrated the effectiveness of LLM-based feature selection methods in various domains (e.g., finance, healtheare, criminal justice), is possible that they show limited per- formance in extremely specialized rare domains. Second, as I.L.M-based feature selection methods rely textual semantics (e.g., column names in tabular datasets) to identify predictive features, they may not be effective on (i) datasets without sufficient text annotations (Appendix A.3. or (ii) datasets where the statistical relationships between features and targets do not align with the semantic associations inferred by the LLM (due to e.g., selection bias). Third, LLMs may exhibit undesirable biases inherited from their pretraining data (Gallegos et al., 2024), which can result in downstream performance disparities across data subpopulations. In the data-driven setting, it is possible to mitigate such issues by selecting features for each subpopulation independently or modifying the training objective to account for group fairness (Zemel et al., 2013; Sagawa et al., 2020; Izmailov et al., 2022). However, it is not immediately obvious how to incorpo- rate similar notions of group fairness into LLMebased feature selection methods, especially for closed-source LLMs which can only be accessed via prompting. As such, combining LLM--riven feature selection with data-driven methods or using it in a human-in-the-loop setup may be a more reliable approach for mitigat- ing bias concerns, especially in safety-critical domains. We leave an in-depth investigation of methods for incorporating group fairness to LLM-based feature selection for future work.
Acknowledgements
We gratefully acknowledge DARPA 2-A8750-2322-10533 ONR (N00014-23-1-2369)) NSF (1152211955), UPMC, Highmark Health, Abridge, Ford Rescarch, Mozilla, the PwC Center, Amazon AI, JPP Morgan Chase, the Block Center, the Center for Machine Learning and Health, and the CMU Software Engineering Institute (SEI) via Department of Defense contract FA8702-15-D-0005 for their generous support of our research. We also thank Michael Oberst, Goutham Rajendran, Bingbin Liu, and Che-Ping Tsai for helpful feedback and comments on earlier versions of this work.
13

---

Published in Transactions on Machine Learning Research (04/2025)
References
David H. Ackley, Geoffrey E. Hinton, and Terrence J. Sejnowski. A Learning Algorithm for Boltzmann Machines. Cognitive Science, 9(1):147 169, 1985.
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Morcira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Jun- whan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo Aakanksha Chowdhery, Clí- ment Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lu- cas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittychoriah, Matthew Jagiclski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sueha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacck, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. PaLM Technical Report, 2023.
Muhammed Fatih Bahn, Abubakar Abid, and James Zou. Concrete Autoencoders: Differentiable Feature Selection and Reconstruction. In International Conference Machine Learning, 2019.
Mohamed Bennasar, Yulia Hicks, and Rossitza Setchi. Feature Selection using Joint Mutual Information Maximisation. Expert Systems with Applications, 42(22):8520 8532, December 2015.
Macicj Besta, Nils Blach, Ales Kubicok, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyezyk, and Torsten Hoefler. Graph of Thoughts: Solving Elaborate Problems with Large Language Models. AAAI Conference on Artificial Intelligence, 2024.
Avrim L. Blum and Pat Langley. Selection of Relevant Features and Examples in Machine Learning. Artificial Intelligence, 97(1):245-277 1997.
Leo Breiman. Random Forests. Machine Learning, 45(1):5-32) 32, 2001.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christo- pher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodci. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Syslems, 2020.
Girish Chandrashekar and Ferat Sahin. A Survey on Feature Selection Methods. Computers & Electrical Engineering, 40(1):16-25 28, 2014.
Z. Che, S. Purushotham, K. Cho, D. Sontag, and Y. Liu. Recurrent Neural Networks for Multivariate Time Series with Missing Values. Scientific Reports, 8(6085), 2018
Jianbo Chen, Mitchell Stern, Martin J Wainwright, and Michael Jordan. Kernel Feature Selection via Conditional Covariance Minimization. In Advances in Neural Information Processing Systems, 2017.
14

---

Published in Transactions on Machine Learning Research (04/2025)
Jiuhai Chen, Lichang Chen, Heng Huang, and Tianyi Zhou. When Do You Need Chain-of-Thought Prompt- ing for ChatGPT? Xiv preprint ar arXiv:2304.0322 2023.
Tianqi Chen and Carlos Guestrin. XGBoost: Scalable Tree Boosting System. In ACM SIGKDD Interna- tional Conference on Knowledge Discovery and Data Mining, 2016.
Kristy Choi, Chris Cundy, Sanjari Srivastava, and Stefano Ermon. LMPriors: Pre-Trained Language Models Task-Specific Priors. In Advances in Newral Information Processing Systems 2022 Foundation Models for Decision Muking Workshop, 2022.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbbrr,, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bremnan Saela, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. PaLM: Scaling Language Modeling with Pathways. arXiv preprint ar "xx::2204002311 2022.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep Rein- forcement Learning from Human Preferences. In Advances in Newrul Information Processing Systems, 2017.
Héctor Seinente-Gonzálee Chloé-Agathe Azencott, Samuel Kaski, and Makoto Yamada. Block HSIC Lasso: Model-Free Biomarker Detection for Ultra-High Dimensional Data. Bioinformatics, 35(14):1477-4435 2019.
Paulo Cortez, A. Cerdeira, F. Almeida, T. Matos, and J. Reis. Wine Quality. UCI Machine Learning Repository, 2009.
Aaron Defazio, Francis Bach, and Simon Lacoste-Julien, SAGA: A Fast Incremental Gradient Method With Support for Ntn-Strongly Convex Composite Objectives. In Advonces in Neural Information Processing Systems, 2014.
Chris Ding and Hanchuan Peng. Minimum Redundancy Feature Selection From Microarray Gene Expression Data. Journal of Bioinformatics and Computational Biology, 3(2):185-205, 2005.
Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring Adult: New Datasets for Fair Machine Learning. In Advances in Neural Information Processing Systems, 2021.
R.O. Duda, P.E. Hart, and D.G. Stork. Pattern Classification. John Wiley & Sons, New York, 2 edition, 2001.
Bradley Efron, Trevor Hastic, lain Johnstone, and Robert Tibshirani. Least Angle Regression. The Annals of Statistics, 32(2):407-451, 2004.
Jean Feng and Noah Simon. Sparse-input, Neural Networks for High-dimensional Nonparametric Regression and Classification. arXiv preprint arXio:1711.075500 2017.
F.J. Ferri, P. Pudil, M. Hatef, and J. Kittler. Comparative Study of Techniques for Large-scale Feature Se- lection. In Pattern Recognition in Practice IV, volume 16 of Machine Intelligence and Pattern Recognition, pp. 403-413. North-Hollande 1994.
Jerome Friedman, Robert Tibshirani, and Trevor Hastie. Regularization Paths for Generalized Linear Models via Coordinate Descent. Journal of Slatistical Software, 33(1):1-22) 2010.
15

---

Published in Transactions on Machine Learning Research (04/2025)
Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mchrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. Bias and Fairness in Large Language Models: A Survey. arXiv:2309.0077 2024.
Ary L. Goldberger, Lais A. N. Amaral, Leon Glass, Jeffrey M. Hausdorff, Plamen Ch. Ivanov, Roger G. Mark, Joseph E. Mictus, George B. Moody, Shung-Kang Peng, and H. Eugene Stanley. PhysioBank, PhysioToolkit, and PhysioNet:: Components of a New Research Resource for Complex Physiologic Signals. Circulation, 101(23):e215-e22) June 2000.
Leo Grinsztajn, Edouard Oyallon, and Gael Varoquaux. Why Do Tree-based Models Still Outperform Deep Learning on Typical Tabular Data? In Advances in Neural Information Processing Systems 2022 Datasets and Benchmarks Track, 2022.
Quanquan Gu, Zhenhui Li, and Jiawei Han. Generalized Fisher Score for Feature Selection. In Uncertainty in Artificial Intellignnee, 2011.
Isabelle Guyon and André Elisseeff. An Introduction to Variable and Feature Selection. Journal of Machine Learning Research, 3: 3:1137-1182, 2003.
Isabelle Guyon, Jason Weston, Stephen Barnhill, and Vladimir Vapnik. Gene Selection for Cancer Classifi- cation Using Support Vector Machines. Machine Learning, 46:380-422, 2002.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. Mcasuring Massive Multitask Language Understanding. International Conference Learning Representations, 2021.
Hans Hofmann. Statlog (German Credit Data). UCI Machine Learning Repository, 1994.
Shima Imani, Liang Du, and Harsh Shrivastava. MathPrompter: Mathematical Reasoning using Large Lan- guage Models. In International Conference on Learning Representations 2023 Workshop on Trustworthy and Reliable Large-Scale Machine Learning Models, 2023.
Pavel Izmailov, Polina Kirichenko, Nate Gruver, and Andrew G Wilson. On Feature Learning in the Presence of Spurious Correlations. In Advances in Newral Information Processing Systems, 2022.
Daniel P. Jeong, Saurabh Garg, Zachary C. Lipton, and Michael Oberst. Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress? In Conference on Empirical Methods in Natural Language Processing, 2024a.
Daniel P. Jeong, Pranav Mani, Saurabh Garg, Zachary C. Lipton, and Michael Oberst. The Limited Impact of Medical Adaptation of Large Language and Vision-Language Models. 2rXiv:2411.0888 2024b.
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. How Can We Know What Language Models Know? Transactions of the Association for Computational Linguistics, 0:423-435 2020.
Alistair E. W. Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J. Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, Li-wei H. Lehman, Leo A. Celi, and Roger G. Mark. MIMIC-IV, A Freely Accessible Electronic Health Record Datasct. Scientific Data, 10(1), 2023.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tic-Yan Liu. LBgh A Highly Efficient Gradient Boosting Decision Tree. In Advances in Neural Information Processing Systems, 2017.
M. G. Kendall. A New Measure of Rank Correlation. Biometrika, 30(1/2):81-93, 1938.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations, 2013.
Ron Kohavi and George H. John. Wrappers for Feature Subset Selection. Artificial Intelligence, 97(1): 273 324, 1997.
16

---

Published in Transactions on Machine Learning Research (04/2025)
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large Language Models are Zero-Shot Reasoners. In Advances in Newral Information Processing Systems, 2022.
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating Mutual Information. Physical Review. E, Statistionl, nonlinear, and soft malter physics, 69:066138, 07 2004.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph F. Gonzalez, Hao Zhang, and lon Stoica. Efficient Memory Management for Large Language Model Serving with PagodAttention. In ACM Symposium on Operating Systems Principles, 2023.
Jeff Larson, Julia Angwin, Lauren Kirchner, and Surya Mattu. How We Analyzed the COMPAS Recidivism Algorithm. ProPublica, May 2016.
Cosmin Lazar, Jonatan Taminau, Stijn Meganck, David Steenhoff, Alain Coletta, Colin Molter, Virginic de Schaetzen, Robin Duque, Hugues Bersini, and Ann Nowe. A Survey on Filter Techniques for Feature Selection in Gene Expression Microarray Analysis. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 944::1106 1119, 2012.
Ismacl Lemhadrí, Feng Ruan, Louis Abraham, and Robert Tibshirani. NessoNet: A Neural Network with Feature Sparsity, Journal of Mochine Learning Research, 22(12)):1-29, 2021.
David D. Lewis. Feature Selection and Feature Extraction for Text Categorization. In Speech and Natural Language, 1992.
Aitor Lewkowycz, Anders Johan Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Venkatesh Ramasesh, Ambrose Slone, Cem Anil, Tmanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving Quantitative Reasoning Problems with Language Models. In Advances in Neural Information Processing Systems, 2022.
Jundong Li, Kewei Cheng, Suhang Wang, Fred Morstatter, Robert P. Trevino, Jiliang Tang, and Huan Liu. Feature Selection: A Data Perspective. ACM Compuling Surveys, 50(6), 2017.
Liam Li, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Jonathan Ben-tzur, Moritz Hardt, Ben- jamin Recht, and Ameet Talwalkar. A System for Massively Parallel Hyperparameter Tuning, In Confer- ence on Systems and Machine Learning, 2020.
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring How Models Mimic Human False- hoods. In Annual Meeting of the Association for Computational Linguistics, 2022.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pro-Train, Prompt, and Predict: Systematic Survey of Prompting Methods in Natural Language Processing, ACM Computing Surveys, 55()), 2023.
Valentin Liévin, Christoffer Egeberg Hother, and Ole Winther. Can Large Language Models Reason About Medical Questions? arXiv preprint arXiv:2207.08144 2022.
Yang Lu, Yingying Fan, Jinchi Lv, and William Stafford Noble. DeepPINK: Reproducible Feature Selection in Deep Neural Networks. In Advances in Neural Information Processing Systems, 2018.
Scott. M. Lundberg and Su-In Lee. A Unified Approach to Interpreting Model Predictions. In Advances in Newral Information Processing Systems, 2017.
Shan Luo and Zchua Chen. Sequential Lasso Cum EBIC for Feature Selection With Ultra-High Dimensional Feature Space. Journal of the American Statistical Association, 109(507):1229-1444) 2014.
Hariharan Manikaudan, Yiding Jiang, and J Zico Kolter. Language Models Are Weak Learners. ar preprint arXiv:2306.14101, 2023.
Michael Moor, Oishi Banerjee, Zahra Shakeri, Harlan Krumholz, Jure Leskovee, Eric Topol, and Pranav Rajpurkar. Foundation Models for Generalist Medical Artificial Intelligence. Nature, 616:259- 225, 2023.
17

---

Published in Transactions on Machine Learning Research (04/2025)
S. Moro, P. Rita, and P. Cortez. Bank Marketing. UCI Machine Learning Repository, 2012.
OpenAL. GPT-4 Technical Report. arXiv preprint arXim:2903. 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training Language Models to Follow Instructions with Human Feedback. In Advances in Neural Information Processing Systems, 2022.
R. Kelley Pace and Ronald Barry. Sparse Spatial Autoregressions. Statistics Probability Letters, 33(3): 291-297, 1997.
Roma Patel and Ellie Pavlick. Mapping Language Models to Grounded Conceptual Spaces. In International Conference on Learning Representations, 2022.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexan- der Miller. Language Models as Knowledge Bases? In Empirical Methods in Natural Language Processing and the International Joinl. Conference on Natural Language. Processing, 2019.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Llya Sutskever. Language Models are Unsupervised Multitask Learners. In OpenAT Blog, 2019.
Brian C. Ross. Mutual Information Between Discrete and Continuous Data Sets. PLoS ONE, 9, 2014.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization In International Conference on Learning Representations, 2020.
Melanie Sclar, Yejin Choi, Yulia. Tsvetkov, and Alane Suhr. Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting. In International Conference on Learning Representations, 2024.
Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Mahdavi, Jason Wei, Hyung Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Abubakr Babiker, Nathanael Schärli, Aakanksha Chowdhery, Philip Mansfield, Dina Demner-Fushine and Vivek Natarajan. Large Language Models Encode Clinical Knowledge. Nature, 620:1-9, 2023.
Jack W. Smith, James E. Everhart, William C. Dickson, William C. Knowler, and Richard S. Johannes. Using the ADAP Learning Algorithm Forccast the Onset of Diabetes Mellitus. Annual Symposium on Computer Ayplicalions in Medical Care, 1988.
Le Song, Alex Smola, Arthur Gretton, Justin Bedo, and Karsten Borgwardt. Feature Selection via Depen- dence Maximization. Journal of Machine Learning Research, 13(47):1393-15 1434, 2012.
Aarohi Srivastava al. Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models. Transactions on Machine Learning Research, 2023.
Nisan Stiemnon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to Summarize from Human Feedback. arXiv:2009. 05:::20000012,, 2022.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them. In Findings of the Association for Computational Lin- guistics, 2023.
Shengpu Tang, Parmida Davarmancsh, Yanmong Song, Danai Koutra, Michael W Sjoding, and Jenna Wiens. Democratizing EHR Analyses with FIDDLE; A Flexible Pata-Driven Preprocessing Pipeline for Structured Clinical Data. Journal of the American Medical Informatics Association, 10 2020.
18

---

Published in Transactions on Machine Learning Research (04/2025)
Robert Tibshirani. Regression Shrinkage and Selection via the Lasso. Journal of the Royal Statistical Society (Series B), 58:267-2888 288, 1996.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Fer- rer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Maric-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Raujan Subra- manian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iiyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Fduunv, and Thomas Scialom. Llama 2: Open Foundation and Models. arxiv preprint arXiv:2307 0990888 2023.
Chat
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkorcit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All You Need. In Advances in Newral Information Processing Systems, 2017.
Shirly Wang, Matthew B. A. McDermott, Geeticka Chauhan, Marzyeh Ghassemi, Michael C. Hughes, and Tristan Naumann. MIMIC-Extratt: A Data Extraction, Preprocessing, and Representation Pipeline for MIMIC-III. In Conference on Health, Inference, and Learning, 2020.
Xuezhi Wang, Jason Wei, Dalc Schuurmans, Quoc V Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self- Consistency Improves Chain of Thought Reasoning in Language Models. In International Conference on Learning Representations, 2023.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent Abilities of Large Language Models. Transactions on Machine Learning Research, 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoe V Le, and Denny Zhou. Chain-of- Prompting Elicits Reasoning in Large Language Models. In Advances in Newral Information Processing Systems, 2022b.
Makoto Yamada, Wittawat Jitkrittum, Leonid Sigal, Eric Xing, and Masashi Sugiyama. gigh-Dimensional Feature Selection by Feature- Wrstere-Wiie Kernelized Lasso. Neural Computation, 26(1):185 207,, 01 2014.
Yutaro Yamada, Ofir Lindenbaum, Sahand Negahban, and Yuval Kluger. Feature Selection using Stochastic Gates, In International Conference on Machine Learning, 2020.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of Thoughtss Deliberate Problem Solving with Large Language Models. In Advances in Neural Information Processing Systems, 2023.
Taisuke Yasuda, Mohanmadhossein Bateni, Lin Chen, Matthew Fahrbach, Gang Fu, and Vahab Mirrokni. Sequential Attention for Feature Selection. In International Conference on Learning Representations, 2023.
Ming Yuan and Yi Lin. Model Selection and Estimation in Regression with Grouped Variables. Journal of the Royal Statistical Society Series B: Statistical Methodology, 68(1):49 67, 2006.
Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning Fair Representations. In International Conference on Machine Learning, 2013.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate Before Use: Improving Few-shot Performance of Language Models. In International Conference Machine Learning, 2021.
19

---

Published in Transactions on Machine Learning Research (04/2025)
Ciyou Zhu, Richard H. Byrd, Peihuang Lu, and Jorge Nocedal. Algorithm 778: L-BFGS-B: Fortran Subrou- tines for Large-Scale Constraind Optimization. ACM Transactions on Mathemalical Software, 23 (4):550 560, 1997.
20

---

Published in Transactions on Machine Learning Research (04/2025)
Appendix for LLM-Select: Feature Selection with Large Language Models
Table of Contents
A Additional Details Small-Scale Datasct Experiments
24
A.1 Dalasets
24
A.2 Feature Sclection, Model Training, and Hyperparameter Optimization
................................
25
A.2.1 Additional Details on Feature Selection Methods
26
A.2.2
Downstream Model Training and Hyperparameter Optimization
27
A.3
Additional Experimental Results
27
A.3.1
Comparison of LIM-SCORE, LLM-RANK, and LLM-SEQ to Basclines
27
A.3.2
Comparison of LLM-SCORE, LLM-RANK, and LLM-SEQ Across LLMs of Varying Scale
29
A.3.3
Comparison of LLM-SCORE, LLM-RANK, and LLM-SEQ LMPriors (Choi et al., 2022)
30
A.3.4
Sensitivity of
to Different Score Ranges
31
A.3.5
Feature Subset Initialization for LLM-SEQ
32
A.3.6
Importance of Text Semantics for ALM-based Feature Selection
34
A.3.7
Investigation of the Semantics of LLM-SCORE
35
B Additional Details on Large-Scale Dataset Experiments
45
B.1
Datasets
................................
45
B.1.1 folktables Datasets
45
B.1.2 MIMIC-IV Datasets
46
B.2 Feature Selection, Model Training, and Hyperparameter Optimization
................................
50
B.2.1 Additional Details on Feature Selection Methods
51
B.2.2 Downstream Model Training and Hyperparameter Optimization
................................
52
B.3 Additional Experimental Results
52
C Prompting for Feature Importance Scores
54
C.1 LLM-SCORE Template for Credit-G
54
C.2 LLM-SCORE Template for Bank
................................
55
C.3 LLM-SCORE Template for Give Some Credit
C.4 LLM-SCORE Template for COMPAS Recidivism
56
C.5 LLM-SCORE Template for Pima Indians Diabetes
57
C.6 LLM-SCORE Template for AUS Cars*
57
C.7 LLM-SCORE Template for YouTube*
58
C.8 LLM-SCORE Template for Housing
58
C.9 L.M-SCORE: Template for Diabetes Progression
59
21

---

Published in Transactions on Machine Learning Research (04/2025)
C.10 LLM-SCORE Template for Miami Housing
59
C.11 LLM-SCORE Template for Wine Quality
60
C.12 LLM-SCORE Template for Used Cars
60
C.13 LLM-SCORE Template for NBA*
61
C.14 LLM-SCORE Template for NYC Rideshare
C.15 LLM-SCORE Template for Income
62
C.16 LLM-SCORE Template for Employment
63
C.17 LLM-SCORE Template for Public Coverage
64
C.18 LLM-SCORE Template for Mobility
64
C.19 LLM-SCORE Template for CKD
65
C.20 LLM-SCORE Template for COPD
66
C.21 LLM-SCORE Template for HF
67
D Prompting for Feature Rankings
69
D.1 LLM-RANK Template for Credit-G
D.2 LLM-RANK Template for Bank
D.3 LLM-RANK Template for Give Some Credit
D.4 LLM-RANK Template for COMPAS Recidivism
70
D.5 LLM-RANK Template for Pima Indians Diabetes
70
D.6 LLM-RANK Template for AUS Cars
70
D.7 LLM-RANK Template for YouTube"
70
D.8 LLM-RANK Template for CA Housing
71
D.9 LLM-RANK Template for Diabetes Progression
71
D.10 LLM-RANK Template for Wine Quality.
D.11 LLM-RANK Template for Miami Housing
72
D.12 LLM-RANK Template for Used Cars
72
D.13 LLM-RANK Template for NBA*
72
D.14 LLM-LAMRA Template for NYC Rideshare*
72
E Prompting for Sequential Feature Selection
74
E.1
LLM-SEQ Template for Credit-G
74
E.2
LLM-SEQ Template for Bank
75
E.3
LLM-SEQ Template for Give Me Some Credit
75
E.4
LLM-SEQ Template for COMPAS Recidivism
75
E.5
LLM--ERQ Template for Pima Indians Diabetes
E.6
LLM-SEQ Template for AUS Cars*
76
E.7
LLM-SEQ Template for YouTube*
76
22

---

Published in Transactions on Machine Learning Research (04/2025)
E.8
LLM-SEQ Template for CA Housing
76
E.9
LLM-SEQ Template for Diabetes Progression
76
E.10
LLM-SEQ Template for Wine Quality
76
E.11
LLM-SEQ Template for Miami Housing
E.12
LLM-SEQ Template for Used Cars
77
E.13
LLM-SEQ Template for NBA
77
E.14 LLM-SEQ Template for NYC Rideshare"
77
23

---

Published in Transactions on Machine Learning Research (04/2025)
A
Additional Details on Scall-Scale Dataset Experiments
Here, we provide additional details and results for the small-scale dataset experiments in Section 4.1.
A.1 Datasets
For all datasets, we report both the total number of features after preprocessing and the total number of concepts (iee, feature names), which may differ due to the one-hot encoding of categorical features. For example, the Credit-G dataset contains 61 features after one-hot encoding the categorical features and only 20 concepts. In this case, when feature selection is performed at the concept. level, we select & of the 20 concopts; when feature selection is performed at the feature level, we select of the 61 preprocessed features.
For each dataset, we randomly shuffle and take a 80-20 train test split. We then take 5-fold split of the training set for validaaali,,,,,,, where the 5-fold splits vary across the random seeds (=(1,2,3,44,5) used throughout the experiments. The test set remains fixed and does not vary with the random seed used. For classification datasels,
always lake a stratified split to preserve the label proportions across the train, validation, and test sets. We standardize all of the mumerical features to have zero mean and unit variance, and one-hot encode all categorical features. Below, we provide the remaining details for each dataset.
Classification Datasets:
Credit-G (Hofmann, 1994) is a UCI dataset where the goal is to predict whether a client at bank carries high credit risk, given set of attributes about the client (e.g., credit history, savings account status). The dataset contains 1000 samples and 61 features after preprocessing (20 concepts), with 700 positive samples and 300 negative samples.
Bank (Moro et al., 2012) is a UCT dataset where the goal to predict whether a client at a bank will subscribe a term deposit, given data collected from a telemarketing campaign at a Portuguese banking institution from 2008 to 2013. The dataset contains 45211 1 samples and 51 features after preprocessing (16 concepts), with 5289 positive samples and 39922 negative samples.
Give Me Some Credit is a Kaggle dataset?, where the goal is to predict whether an individual is likely to experience significant financial distress/delingt /deliqquenyy within the next two years, given information about the individual's financial status (e.g., debt ratio, monthly income). The dataset contains 120269 samples and 10 features after preprocessing (10 concepts), with 8357 positive samples and 111912 negative samples.
COMPAS Recidivism a dataset collected from 2016 study on the racial biases present in the Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) algorithm (Lar- son al., 2016). The goal is to predict whether a criminal defendant carries high risk of recidivism, given their criminal history and demographic attributes. The dataset contains 6172 samples and 23 features after preprocessing (14 concepts), with 2751 positive samples and 3421 negative samples.
Pima Indians Diabetes (Smith et al., 1988) a Kaggle datasct where the goal is to predict whether female adult patient of Pima Indian heritage has diabetes, given set of clinical measure- ments and demographics. The dataset contains 768 samples and 8 features after preprocessing (8 concepts), with 268 positive samples and 500 negative samples.
AUS Cars a Kaggle dataset 10 published in 2023, where the goal is to predict whether the price of a car in Australia is above $30000, given set of attributes about the car (e.g., year of manufacture, number of doors in the car, fuel consumption rate). The dataset contains 14283 samples and 33 features after preprocessing (11 concepts), with 6396 positive samples and 7887 negative samples.
9httpe://archivee.ee uci http://aawrr/ee//ssaaigg////aere
24

---

Published in Transactions on Machine Learning Research (04/2025)
YouTube* is Kaggle dataset" published in 2023, where the goal is to predict whether YouTube channel has more than 20 million subscribers, given set of attributes about the channel (e.g.: total number of videos uploaded on the channel, date when the channel was created). The dataset contains 588 samples and 72 features after preprocessing (22 concepts), with 268 positive samples and 320 negative samples.
Regression Datasets:
CA Housing (Pace & Barry, 1997) a StatTib dataset 12 where the goal is to predict the median housing price of a US Census block group in California, given data collected from the 1990 US Census. A block group, typically population of 600 to 3000 people, is the smallest goographical unit. for which the US Census Bureau publishes sample data. The dalaset contains 20640 samples and 8 features after preprocessing (8 concepts).
Diabetes Progression is dataset used in a study by Efron et al. (2004), where the goal is to predict the discase progression level in diabetic patients, given their baseline blood serum measure- ments from the previous year and demographic information. The dataset contains 442 samples and 10 features after preprocessing (10 concepts).
Wine Quality (Cortez et al., 2009) is a UCI dataset 14 collected from red and white vinho verde wine samples from northern Portugal, where the goal is to predict whether a wine is high or low quality, given its various physicochemical measurements (e.g., acidity, density). The dataset contains 6497 samples and 11 features after preprocessing (11 concepts).
Miami Housing (Grinsztaju et al., 2022) is an OpenML 'ataset where the goal is to predict the selling price of house in Miami in 2016, given structural (e.g., area, structure quality) and geographic (e.g., longitude, latitude) information about each house. The dataset contains 13932 samples and 15 features after preprocessing (15 concepts).
Used Cars is a Kaggle "6 where the goal is to predict the selling price of a used car, given a set of attributes about each car (e.g., age, fuel type, number of previous owners). The dataset contains 301 samples and 11 features after preprocessing (7 concepts).
NBA is a Kaggle dataset published in 2023, where the goal is to predict the number of points per game for an NBA basketball player, given a set of attributes and statistics (from the NBA season) about the player (e.g., blocks per game, position, team). The dataset contains 388 samples and 61 features after preprocessing (27 concepts).
NYC Rideshare* a Kaggle 1ataset published in 2023, where the goal to predict the total pay given to a rideshare-app driver in NYC after trip, given information about the trip (e.g., time elapsed from ride request to dropoff, total distance of the trip). The original dataset on Kaggle contains some data from 2022, we only extract the data rideshare data from 2023 for preprocessing and evaluation. The dataset contains 5000 samples and 19 features after preprocessing (15 concepts).
A.2 Feature Selection, Model Training, and Hyperparameter Optimization
As described in Section 4.1 of the main text, we evaluate the effectiveness of each feature selection method by measuring how the test performance of a downstream La-penalized logistic regression (for classification tasks) linear regression (for regression tasks) model changes as vary the proportion of features selected from 10% to 100%, in approximately 10% increments. For logistic regression, we minimize the negative log-likeling logelioel using the L-BFGS optimizer (Zhu et al., 1997) and use importance weighting to balance the
html
dtt.ssttttttttttt-rrrrro- the-2023-2024
25

---

Published in Transactions on Machine Learning Research (04/2025)
weights of the positive and negative samples, some of the datasets exhibit label imbalance. For linear regression, we minimize the mean squared error (MSE) using the L--BFGS optimizer. For all experiments, we aggregate the results over 5 random seeds (=11,2,3,4,55), which control the arain-validation splits and the behavior of random feature selection, in order to ensure the robustness of results.
A.2.1 Additional Details on Feature Selection Methods
LassoNet. For feature selection with LassoNet (Lemhadri et al., 2021), we use multi-layer perceptron (MLP) with 1 hidden layer and 100 hidden units for all datasets and fix the hicrarchy coefficient M to the recommended value of 10. For a given dataset, we take 80-20 split of the training set (split controlled by the random seed = [1, 2, 3, 4, 5)), using the latter 20% split for validation. For classification tasks, we minimize the binary cross-entropy loss using the Adam optimizer (Kingma & Ba, 2015) with the default learning rate 10-3 and the default momentum hyperparameters of 31 = 0.9, 32 = 0.999, and e = 10 10- For regression tasks, we minimize the MSE using the Adam optimizer with the same configuration used for classification tasks. Following Lemhadri et al. (2021), we compute the dense-to sp-rsse regularization paths with warm starts (Friedman et al., 2010). We first train unregularized MLP for 10 epochs, using the validation split for early stopping with a patience of 3 epochs. We then increase the regularization coefficient starting from À = 10-6, itcratively multiplying powers of 1.02 until all of features become inactive (i.c., A = 10-6, 1.02 10-6, 1.022 10-6 .).,,, where for each value of X, we further train the model for 3 epochs. After computing the full regularization path, we identify the regularization strengths that select 10% to 100% of all fcatures in approximately 10% increments.
LASSO. For feature selection with the LASSO (Tibshirani, 1996) on each dataset, we first train an L1- penalized logistic regression model with inverse regularization coofficient C 10-4 (for classification tasks) or Li-penalized linear regression model with regularization coefficient. À = 10a (for regression (asks) To compute the sparse-to-dent dense regularization paths with warm starts (Friedman et al., 2010), then gradually increase C' by iteratively multiplying powers of 1.02 (i.c., C = 10-4, 1.02 - 10-4, 1.022 .1--....) and gradually decrease À by iteratively multiplying powers of 1.02 (i.e., À = 1002 1..2-".10000 1.02 1002-21100 2..0".... until all features become active. For logistic regression, we train each model by minimizing the negative log- lig-likel using the SAGA optimizer (Defazio et al., 2014). For linear regression, we train each model by minimizing the MSE using the SAGA optimizer. After computing the full regularization path, we identify the regularization strengths that select 10% to 100% of all features in approximately 10% increments.
1.02
Forward/ serward/backe bbackward sequential selection. For forward or backward sequential selection, we first run a grid search with 5-fold cross-vale validatiin to select the best hyperparameter to use for training Ioz-ennalizdd logistic/ /lieea regression model for sequential selection. For classification tasks, sweep through the inverse regularization coefficients C = [0.1, 0.5, 1, 5, 10, 50, 100] and select the value of C' that leads to the highest cross-validation area. under the ROC curve (AUROC). For regression tasks, we sweep through the regular- ization coefficients X = [0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1,5, 10] and select the value of À that Icads to the lowest cross- gross-validation mean absolute error (MAE). We use the selected hyperparameter value for training a model at each iteration of forward or backward sequential selection. For forward sequential selection, we start with an empty feature subset and iteratively add new feature which maximizes the validaaiiot performance when included in the feature subset. For backward sequential selection, we start with all fea- tures and iteratively remove feature such that the cross- validatio performance degrades minimally when excluding it from the feature subset.
Random feature selection. For random feature selection, we randomly select the input concepts in approximately 10% increments, as in LLM-based feature selection. For example, on the Credit-G datasct, which contains a total of 20 concepts (61 features after preprocessing), selecting 10% via random feature selection equivalent to selecting 2 out of the 20 concepts, which may correspond to more than 6 R 61.0.1 features. The randomness in the feature subset controlled by the random seeds (=(1,2,3,445) used throughout the experiments.
For logistic regression, configure the inverse regularization coefficient C instead of the regularization coefficient. A, as the scikit-learo implementation logistic regression uses the former.
26

---

Published in Transactions on Machine Learning Research (04/2025)
(a)
12.5
Ranking by MAE
10.0
AUROC
5.0
2.5
GPT-3.5 (-1758)
MI
LASSO
Forward
HSIC-Lasso
GPT-4 (1)
CAE
Liama-2 (78)
Liama-2 (70B)
MRMR
Liama-2 (138)
Backward
RFE
LassoNet
SA
Random
Forward
LASSO
HRMR
HSIC-Lassio
GPT-4 1-1.TT)
GPT-3. (-1758)
Ml
Llama-2 (708)
Llama-2 (238)
CAE
Lassohiet
Liama-2 (7B)
Backward
AFE
Random
SA
(b)
0.9
12.5
Ranking by MAE
0.8
10.0
AUROC
7.5
0.7
5.0
0.6
2.5
0.5
0.0
MI
LASSO
GPT4 (~1.7T)
Forward
HSIC-Lasso
GPT-3.5 (-1758)
CAE
MRMR
Liama (7B)
Liama-2 (708)
Backward
Liama-2 (138)
RFE
LassoNet
SA
Random
MRMR
Forward
GPT-4 (-1.7T)
LASSO
HSiC-Laso
GPT-3 (-175B)
MI
CAE
LassoNet
Liama (78)
Liama-2 (708)
Backward
Llama-2 (138)
RFE
Random
54
Figure A1: LLM-RANK and LLM-SEQ achieve strong feature selection performance competitive with data- driven baselines, given LLM of sufficient scale. (a) Average AUROC (left; higher is better) and ranking by MAE (right; lower is better) across all datasets when selecting the top 30% of features with LLM-RANK. (b) Average AUROC (left; higher is better) and ranking by MAE (right; lower is better) across all datasets when selecting the top 30% of features with LLM-SEQ.
A.2.2 Downstream Model Training and Hyperparameter Optimization
For each dataset and at each proportion of features selected by each method, we grid search with 5-fold cross-validation to select the best hyperparameter to use for training and evaluation of the downstream prediction model. For classification tasks, we sweep through the inverse regulariza- tion coefficients C = [0.1, 1, 5, 10, 50, 100] and select the hyperparameter value that leads to the highest cross-validatton AUROC. For rogression tasks, we sweep through the rogularization coefficients
x = [0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1, 5, 10] and select the hyperparameter value that leads to the lowest voss-validation MAE. We then train final eggisti/llneaa regression model with the selected regularization strength using the full training set, and measure the performance of the final model on the test set. We repeat this process 5 times with different random seeds (=(1,2,3.4451) and average the results.
A.3 Additional Experimental Results
A.3.1 Comparison of LLM-SCORE, LLM-RANK, and LLM-SEQ to Baselines
In this section, we include additional experimental results that compare the performances of LLM-SCORE, LLM-RANK, and LLM-SEQ against those of the baselines described in Section 4.1 of the main text. We note that all of the results shown in this section are based on the LLM outputs generated from the defoult prompt, which does not include any datasct-spccifie context or few-shot examples, and with greedy decoding. For LLM-SEQ, we present the results for the setup where we start with an emply feature subset. As described in Section 4.1 of the main text, we evaluate the effectiveness of each feature selection method by measuring how the test performance 10% to 100%, in approximately 10% increments. Intuitively, an effective feature selection method should be able to identify highly informative ive subset of features and enable strong downstream predictive performance,
downstream model changes as we vary the proportion of features selected from
27

---

Published in Transactions on Machine Learning Research (04/2025)
-
LassoNet
Forward Badkward
RFE
-
MI
CAE
-
Random
Llama-2 (13B)
GPT:3.5 (-1758)
Liama-2 (7B)
Llama-2 (70B)
(-1.7T)
Credit-G
Bank
Give Me Some Credit
COMPAS Recidivism Pima In Indians Diabetes
AUS Cars
YouTube
0.80
0.85
0.80
AUROC
0.70
AUROC
0.8
0.7
0.75
0.70
0.75
80
By soe
AUROC
0.70
0.75
AURDE
AUROC
0.8
0.7
(a)
0.65
0.03
0.03
0.00
0.60
0.60
0.6
0.55
0.2 0.60, 1.0
0.20.40.000 0.8 1.0
0.2 0.6 0.9 1.0
0.2040:60.800
0.2 0.4 0.5 0.8 1.0
0.2 0.4 0.60.8 10
0.2040.60.514
Fraction Selected
Fraction Selected
Fraction Selected
Fraction Selected
Fraction Selected
Fraction Selected
Fraction Selected
CA Housing
Diabetes Progression
Wine Quality
Miami Housing
Used Cars
NBA
NYC Rideshare*
0.9
0.66
3.5
0.40
3.0
MAE
MAE
0.62
2.5
MAE
MAE
0.7
0.58
1.5
0.2 0.4
0.2.4.00..... Fraction Selected
0.20
0.0.4.6...81 Fraction Selected
1.0
060800
0.2.44
Fraction Selected
Fraction Selected
Fraction Selected
Fraction Selected
Fraction Selected
Bank
Give Some Credit
COMPAS Recidivism
Pima Indians Diabetes
AUS Cars
YouTube
0.80
0.9
0.80
0.80
AUROC
0.8
0.75
0.80
0.75
0.75
AUROC
AUROC
0.8
0.70
0.7
0.70
0.70
(b)
0.65
0.65
0.7
0.60
0.5
0.60
0.55
0.60
0.6
0.
0,204060810
0.20.4.60.8..0 Fraction Selected
1.0
0.60..660
0.204668810
Fraction Selected
Praction Selected
Fraction Selected
Praction Selected
Fraction Selected
Fraction Selected
Housing
Diabetes Progression on
Wine Quality
Miamn Housing
Used Cars
NBA
NYC Rideshare*
0.9
0.66
0.40
MAE
0.62
0.35
MAE
2.5
MAE
MAE
0.7
0.60
0.30
2.0
0.58
0.20
0.7 9.4 0.8
02 0.4 08 1.0 Praction Selected
0.2 84
0.4.5.88..0
6.2 0.4.0.8. 1.0
0.0..6.88.0
0.2 0.2 0.8
Praction Selected
Fraction Selected
Fraction Selected
Praction Selected
Fraction Selected
Fraction Selected
Credit-G
Bank
Give Me Some Credit
COMPAS Recidivism
Pima Indians Diabetes
AUS Cars
You Tube
0.80
0.9
0.80
0.70
0.75
AUROC
0.8
0.75
0.80
0.75
0.75
AUROC
AUROC
0.8
0.05
0.7
0.70
(c)
0.05
0.65
0.65
0.7
0.55
0.60
0.60
0.60
0.0
0.6
0.55
0.20.40.6...1
0.201060810
0.2
1.0
0.6.8....
0.2 0.4 0.6 0.8 1.0
0.2 0.4...8..1
0.204060.81
Fraction Selected
Fraction Selected
Fraction Selected
Fraction Selected
Fraction Selected
Fraction Selected
Fraction Selected
CA Housing
Diabetes Progression on
Wine Quality
Miami Housing
Used Cars
NBA*
NYC Rideshare*
0.9
0.64
0.40
3.0
10
MAE $
0.35
MAE
2.5
MAE
MAE
0.7
0.60
0.30
2.0
0.58
0.25
0.2 0.4 0.6 1.0 Fraction Selected
0.2 0.4 0.6 0.6 1.0 Fraction Selected
0.8 1.0 Fraction Selected
0.2 0.4 0.6
0.2 0.60..11...... 1.0 Fraction Selected
0.2 0.4 1.0 Fraction Selected
0.2 6.4.0.660.8 1.0 Fraction Selected
0.20.4 0.60... Fraction Selected
Figure A2: Feature selection paths for all baselines and the (a) LLM-SCORE, (b) LLM-RANK, and (c) LLM-SEQ methods (ours, solid lines) basod on GPT-4 GPT-3.5, and the Llama-2 models. In cach panel, the top row shows the test AUROC (higher is better) on the classification datasets and the bottom row shows the test. MAE (lower is better) on the regression datasets. Dalasets marked with asterisk (*) were published after the LLM cutoff dates. On most datasets, LLM-based feature selection methods show strong performance comparable to the baselines. Larger models, especially GPT-4, show consistently strong performance across all datasets.
even when the number of selected features is relatively small. For all experiments, the results are averaged over 5 random seeds (=,,,,,3,4,55), which control the train-validation splits and random feature selection.
28

---

Published in Transactions on Machine Learning Research (04/2025)
Figure A1 shows the downstream test performance, averaged over all of the classification (left) and regression (right) datasets, when selecting the top 30% of features according to each baseline, and (a) LLM-RANK and (b) LLM-SEQ, based on GPT-4, GPTT-35,, and Llama-2. For the regression datasets, we report the average ranking based on test MAE to account for differences in the scale of MAE values across datascts. We observe that LLM-RANK and LLM-SEQ both show strong feature selection performance on average, competitive with the data-driven basclines.
Figure A2 shows the feature selection paths for the basclines and for LLM-SCORE, LLM-RANK, and LLM- SEQ based on GPT-4, GPT-3.5, and the Llama-2 models. Table Al shows the area under the feature selection paths in Figure A2 for the basclines and for LLM-SCORE, LLM-RANK, and LLM-SEQ based on GPT-4, GPT-3.5,, and the Llama-2 models. As discussed in Result 3, Section 4.1 of the main text, the area under each feature selection path serves as a quantitative summary of feature selection performance, where a higher area is desirable for classification tasks and lower area is desirable for regression tasks. For each datasct, we highlight in bold the area of the best-performing bascline and the area of the best-performing LLM-based feature selection method for each LLM.
In Figure A2, observe that LLM-SCORE, LLM-RANK, and LLM-SEQ all achieve strong feature selection performance competitive with the data-driven baselines. For the larger models, notably GPT-4 (in blue) and GPT-3.5 (in green), we consistently observe such strong performance across many datasets. For the smaller models, notably the Llama-2 models (in orange, pink, and yellow), we observe that feature selection perfor- mance is generally more sensitive to the choice of selection strategy (i.e., LLM-RANK or LLM-SEQ), but that performance can be strong when an adequate selection strategy is used. For example, on the Credit-G and COMPAS Recidivism datasets, we see that using LLM--ANK signíficantly improves the performances of GPT-3.5 and the Llama-2 models over that achieved with LLM-SCORE (shown in Figure 2) from being close to the average performance of random feature selection (in black) to being on par or better than those of the best-performing baselines. For LLM-SEQ, we observe a similar improvement on the Pima Indians Diabetes and COMPAS Recidivism datascts.
Table A1 further illustrates these findings from Figure A2. We see that for the larger GPT-4 and GPT-3.5 models, the area of the best-performing LLM-based feature selection method is often on par with the area of the best-performing bascline, across most datascts. For example, the Credit-G dataset, best-performing baseline (MRMR) achieves an area of 0.7496, while the best-performing selection strategy for GPT-4 (LLM-SEq) outperforms it with an area of 0.7710, and the best-performing selection strategy for GPT-3.5 (LLM-RANK) performs as well with an area of 0.7495. We observe that these models achieve strong performance on the other datasets as well (e.g., COMPAS Recidivism, Pima Indians Diabetes, Wine Quality). Meanwhile, for the smaller Llama-2 models, we see that the area of the best-performing selection strategy can be on par with that of the best-performing baseline but that feature selection performance more sensitive to the choice of selection strategy. For example, the Pima Indians Diabetes dataset, we see that the areas of the best- pess-performing selection strategies for the three Llama-2 models are 0.7728 (70B), 0.7744 (13B), and 0.7709 (7B), respectively, which are on par with the area of 0.7855 for the best-performing baseline (RFE). However, as noted in Result 1, Section 4.1 of the main text, the performance of the smaller models are less consistent across datasets and the three LLM-based feature selection methods.
that the
The above results demonstrate that all thrce LLM-based feature scloction methods can be competitive with traditional baselines but that a sufficiently large model may be required to ensure reliable performance.
A.3.2 Comparison of LLM-SCORE, LLM-RANK, and LLM-SEQ Across LLMs of Varying Scale
In this section, we include additional results that compare the feature selection behaviors and performances of LLM-SCORE, LLM-RANK, and LLM-SEQ across LLMs of varying scale. We note that all of the results shown in this section are based on the LLM outputs generated from the default prompt, which does not include any dalaset-specific context or few-shot examples, and with greedy decoding. As done for GPT-4 and GPT-3.5 in Result 2, Section 4.1 of the main text (Figure 3), we show the changes in average test performance for LLM-SCORE, LLM-SEQQ and LLM-RANK for GPT-3.5 and the Llama-2 models in Figures A3-A5. In Figure A6, we also show the average rank correlation (Kendall's 7) between (i) LLM-SCORE and L.LM-RANK (left) and that between (ii) LLM--SCORR and LLM--SQQ (right) to show how well the orderings
29

---

Published in Transactions on Machine Learning Research (04/2025)
Table A1: Area under the feature selection paths for the baselines and our LLM-SCORE, LLM-RANK, and LLM-SEQ methods based on GPT-4, GPT-3.5, and the Llama-2 models. For classification datascts, higher is better (t). For regression datasets, lower is better (4). Give Me Credit and Pima Diabetes are shorthands for the Give Me Some Credit and Pima Indians Diabetes datasets, respectively. For cach dataset, we
boldface the area of the best-performing baseline and that of the best-performing LLM-based method for each LLM. Datasets marked with an asterisk (*) were published after the LLM cutoff dates.
Clocsiication Datasiss
Regresioo Dutasets 1
Give Ne
CIMPAS
Piea
CA
Dsabutes
Mise
Misms
NYC
Credit-O
Bank
Credit
Recidiviox
Disbetes
AUB Caru*
YouTcbe*
Bousing
Frogression
Quality
Housing
Uned Cars
MBA
Ridesbaree
LessoNet
0.7358
0.7915
0.7889
0.7704
0,7564
0.7818
0.6005
0.0721
44.0832
0.0776
0.2288
1.5074
2.5073
3.7089
LASSO
0.7904
0.5819
0.7753
0.8050
0.7829
0.9144
0.5264
0,5726
40.0336
0.5727
0.8533
0.0025
Furwaed
0.7556
0.6817
0.7881
0.8257
0.78/2
0.9204
0.5711
$1.5665
0.2278
1.51.85
0.0610
1.5367
Backword
0.7201
0.7722
0.7463
0.7722
0.7188
0,8030
0.0634
0.7274
45.4677
0.6363
0.2965
2,0084
0.1074
3.6240
MRMR
0.7496
0.8855
0.7881
08190
0.7850
08751
0.71977
0.5935
45.2877
0.5821
0.228/
1,5787
0.0655
1.750S
MI
0.7121
0.6757
0.7882
0.8277
0.7736
0.9119
0.823
0.0221
15.0250
0.5400
0.2129
1.5027
0.0051
1.6716
RFC
0.6817
0.7986
0.7782
0.7508
0.7855
D8111
0.7884
0.6312
44.7423
0.5780
0.2282
2,3120
1.2152
3.4855
HSSC-LLao
07766
0.8712
0.7857
0.8288
0.7596
0.9100
0.7174
0.0001
44.1273
0.5500
0.2345
1,3156
0.0571
1.9081
CAE
0.7357
1.5730
0.7790
0.8211
0.9130
0.8215
0.0854
4572770
0.5825
1.73112
0,3000
1,5895
0.7122
0.7921
0.7485
0.733
0.7120
0.458)
0.7338
0.7300
0.6120
0.2545
2,0027
1,0880
3,3182
Random
0.6365
0.7408
0.7038
0.7636
0.7265
0.7861
0.6948
6.7566
2130
0.6162
0.2780
2,4023
1.1800
3.1502
GPT-4 (LLM Scont)
0,7336
0.8G86
0.7504
0.192
0.7813
0
0.8474
0.5030
9091
0.5004
0.2353
1,063
0.1376
1.9757
GPT-4 (LL.M-RASS))
0,7570
0.9667
0.7717
0.1132
0.7821
0.0021
0.8547
0.5948
5616
0.578G
0.2365
1,4261
0.1201
2,0082
GPT-4 (LLM Stoe
0,7710
0.5641
0,7417
0.8201
0.7794
0.9022
0.8452
C.5S4
45.3253
0,5800
0,2382
1.4334
0.0600
1.7038
GPT-3.5 (LLM Scoaa)
0,7158
0
0.7481
0.11311
0.7803
0.7709
0.6822
29 6221
0.5559
0.2506
1.5020
0,5125
1,9633
GPT-355 J.LM-RANK)
0.7495
0.8730
0.7710
0L8288
0.7811
0.8924
0.8508
0.5945
3033-08
0.5802
0.2510
1.37.5
0.5101
2.5767
GPT-8.5 (LLMM-S9))
0.6952
0.643
0.7838
0.7970
0.777
2,0076
0.1628
100m-22-708 (LLM-S)O
0.7258
0.7797
0.7910
0.7512
1.7045
0.5205
1.7974
Llumu-2-7088 (LLMM- RANK)
0.7618
B.738
0.7706
0.003
0.7728
0.9016
0.8506
0.5941
48.1246
0.5972
0.3577
1,4991
0.8164
1.6438
10mm-2-70B (L..M-SHG)
0.7263
07436
0.7217
0.8138
0.7587
0.842
0.0965
0.6338
3056
0.6312
0.2649
1,5000
0.29552
2.39/5
Damo-2-133 (LLM SCoMCo))))
0.7002
0.279
0.7281
0.7033
0,7300
0.8537
0.8.977
0.6325
56 30:3
0J6036
0-2676
1,5727
0,2452
3,0830
E000-2-155 (T..M-RANK)
018631
0. 0.7003
0.7095
0.7744
0.89336
0.8661
0.5925
47 5013
0,0245
0.3/35
1,6183
1,0390
1.5407
11amm-2-13B (LLM-SRQ)
0.6088
0.8470
0.7746
0.7974
0.7744
0.7:25
0.7840
0.6576
44.9851
0.0167
0.5307
1,9731
0.2974
3.5621
1lmm-2.7R (L..M-Smne))
0.731
0.7033
0.7358
0.7017
0.7152
0.8527
0.8107
0.5952
50.90 9002
0.6061
0,2535
2.2008
1.0096
1,6089
Llao (LLM RANK)
0.7671
0.8676
0.7615
0.8115
0,7061
0.8924
0.8185
0.5825
0.6117
0,3001
2.8061
0,7841
1.7905
Llmo-2:70 (L..M-Ses)
077122
0.7854
0.7451
0.8205
0.770M
0,5691
0.8216
0.0388
1.8047
0,2413
2,0030
of the features by their relevance to the prediction target agree between each pair selection strategies being compared. As described in Result 4, Section 4.1 of the main text, the Kendall's т cocfficient E [-1,1] quantifies the agreement in pairwise orderings, where +1/-1 indicates perfect agreement/ deinnn////ssaarement....
Together with Figure 3 discussed in Result 2, Section 4.1 of the main text, Figures A3 A5 show that the feature selection paths of LLM-SCORE, LLM-RANK, and LLM-SEQ are generally more consistent with one another (i.e., exhibits less variability) with increasing model scale, with GPT-4 showing almost perfect alignment on all datasets except Diabetes Progression. Figure A6 also shows that the average rank correlation between LLM-SCORE and LLM-RANK and that between LLM-SCORE and LLM-SEQ both tend to increase with increasing model scale. Table A1 further illustrates this finding, where we see that with increasing model scale, the areas under the feature selection paths for LLM-SCORE, LLM-RANK, and LLM-SEQ tend to be more similar.
The above results suggest that the assignment. of feature importance tends to be more consistent across LLM-SCORE, LLM-RANK, and LLM-SEQ for the larger models. Meanwhile, for the smaller models, we see that choosing different selection strategy leads to qualitatively distinct results, translating to increased variability in the average Lest performance of the downstream prediction model.
A.3.3 Comparison of LLM-SCORE, LLM-RANK, and LLM-SEQ to LMPriors (Choi et al., 2022)
In this section, we include additional results that compare the average feature selection performance of LLM- SCORE, LLM-RANK, and LLM-SEQ to that of the LMPriors framework (Choi et al., 2022). As discussed in Section 2.2 methods are more broudly applicable than the LMPriors framework, which requires access to the token probabilities often not provided in statc-of-tho-art proprictary LLMs such as GPT-4 and GPT-3.5. Meanwhile, the original implementation of LMPriors uses the Davinci-instruut-beer (Brown et al., 2020), which is no longer serviced by OpenAL. We therefore replicate their setup using the open-source Llama-2 models (Touvron et al., 2023), where we do have full access to the token probabilitios. We note that all of the results shown in this section are based on the LLM outputs generated from the defaull prompt, which does not include any dataset-specific context or few-shot examples, and with greedy
variant of GPT-3
30

---

Published in Transactions on Machine Learning Research (04/2025)
2 (70B), Score
Liama (708), Rank
Uame-22 (70B), LEQ
Coedi-G
Bank
Glue Me Some Credit
COMPAS Becidlitem
Pims Indiars Diabetes
AUS
YorTube
0.00
6.75
AUROC
AUMOC
AUROC
AEROC
AUB-OC
0:70
775
22 24 06 1d
04 14
BJ 65 56 1X Fraction Selocted
022 0. S6 0X Frattio Sclected
013
344
66: 6X 1/3
037 08 0x 01 Fraction Selected
244 Froction Solected
Roction on Selected
Faction Sc ccted
Fraction Selected
C4 Housiingg
Dapetees Progression
Wine Quality
Miami Housing
Uted Cars
NRA*
NYT Ridestarre
2.45
2.0
15
07 EX on Selected
8J 0. Da 1-5 10 Fraction Selected
0:2 24 Fractio Salected
03 Di
07 0.0 n. 01 Fraction Selected
2-2 F-d
Fraction So ected
Fraction Selected
Fraction Solected
Figure A3: Feature selection paths for LLM-SCORE, LLM-SEQ, and LLM-RANK based on Llama-2 with 70B paramcters. Datasets marked with an asterisk (*) were published after the LLM cutoff dates.
Liame-2 2 (13B1, LLM-Score
Liama:2 2 (139). Lank
Liama-2 (138), LLM-Seq -
8:00
0.70
AUBOR
AURDC
AUROC
AUBOCIC
AUROC
AUROC
2.43
Faation on Selecced
Raction Se ected
Fraction Selected
Factorr Selected
Fraction Selected
Factin Selected
Fraction Solected
Housing
Progression
Qualtt
6.0
246
MONE
2.59
35
Pact on Selectod
Fraction Se octed
Fraction Selected
Fraction Salected
Praction Selected
Faction Selected
Fraction Selected
Figure A4: Feature selection paths for LLM-SCORE, LLM-SEQ, and LLM-RANK based on Llama-2 with 13B paramcters. Datasets marked with an asterisk (*) were published after the LLM cutoff dates.
Laama 2 (7B), LLM Score
Liama 2 (78), LLM-Rank
Lamaa (78). LUM-SSe
C-edi-G
Clus Some Credit
COMPAS Beridfutem
Pima Indiarrs Blabetes
Carct
YorThbee
6.00
8:05
0.00
8:00
0.84
AUSOC
AUROC
AMBOI
AUBOC
2.45
0.75
32
02
06
18
01 Selected
24 Fraction
onaction Selectod
octed
Fraction
Pactior Salected
Praction Selected
Housing
Ducnetes Progrexion
Wine Quality
Houuing
Utedd
NEAY
NYT Rideshare*
3.41
MAEE
MOLE
1.13
32 0.4 05 0.0
02 34
031 Selected
on Selected
Salected
Praction Selected
Froction Selected
Figure A5: Feature selection paths for LLM-SCORE, LLM-SEQ, and LLM-RANK based on Llama-2 with 7B parameters. Datasets marked with an asterisk (*) were published after the LLM cutoff dates.
decoding. Table A2 shows that our methods on average perform as well as the LMPriors framework, even without the same level of access inlo the LLM.
A.3.4 Sensitivity of LLM-SCORE to Different Score Ranges
In this section, we investigate how choosing a different score range for LLM-SCORE affects the assignment of feature importance scores generated from the LLM, given that defining the score range to be between 0
31

---

Published in Transactions on Machine Learning Research (04/2025)
Table A2: Area under the feature selection paths for our methods (average LLM-SCORE, LLM-RANK, LLM-SEQ) and the LMPriors framework (Choi et al., 2022). For classification datasets, higher is better (t). For regression datasets, lower is better (4), Give Credit and Pima Diabetes are shorthands for the Give Me Some Credit and Pima Indians Diabetes datascts, respectively. For each dataset and each LLM, we highlight in bold the area for the feature subset initialization that performss better. Dalasels marked with an asterisk (*) were published after the LLM cutoff dates.
Claesifiation Dastoott 7
Regresion Totasts
Give Ne
Pina
GA
Diabeses
Wize
Miant
NYC
Tank
Credit:
Becidrrrtas
Biatee
ABS Carae
Yonfhibwe
Rousing
Progremstoon
Quality
Houxing Used Cara
1RA+
I Rideahares
Llaan 2-708 (Ours)
0.7379
0.7463
0.7394
0,8022
0.7416
0.8839
0,7579
0.6770
48,0843
0.6119
012886
1.7178
0.3774
1,9115
Liaas (LMPriors)
0.7478
016805
0,7143
0.5225
0.7138
0.0070
0.5800
0.2564
2.4146
0.1500
2.1375
Liaan 2 [Ours]
0.7795
0.7644
0.7226
0,5306
0.0374
0.6150
0.2173
1.8314
0.5935
Liaaa 130 (MMss)
0.7435
0.7299
0.7210
0.7510
0.7757
0,7028
0.8445
0.0644
49.1053
0.6304
0.2858
2.0134
0.7890
2.8567
Llemw--7B (Oura)
0.7308
0.774]
0.7460
0.8079
0,7500
0.8714
0.8169
0.2255
47.7065
0.6132
0.9855
2.5205
0.7002
2.1208
Llumm-2-7B (LMPtian))
0,7520
0.8182
117200
0,8080
0.7563
0.7001
0,7311
0.6925
49.3740
0.6059
0,2650
2.1892
0.1682
4,2001
Average Rank Correlation
1.0
Average Rank Correlation
1.0
0.5
0.5
0.0
0.0
-0.
-0.5
(-1-4(-1.77)
Liama-2 (70B)
Liama-2 (78)
(-1.7--1.7)
Liama-2 (70B)
Liama22 (7B)
GPT-3.5 (-1758)
Llama22 (138)
GPT-35
(-1750)
Llame-2 (138)
110
-1.0
LLM-Rank
LLM-Seq
Score Range: (0,10)
Score Range: (8,24]
Figure A6: Average rank correlation (Kendall's r) between LLM-SCORE and LLM-RANK (left) and that between LLM-SCORE and LLM-SEQ (right). Error bars indicate standard error across da datasets.
Figure A7: Average rank correlation (Kendall's
T) between LLM-SCORE with the score ranges of
[0,1] and [0,10] (left) that between LLM-SCORE with the score ranges of [0,1] and [8,24] (right).
Error bars indicate standard error across datasets.
and 1 as in Equation (1) is an arbitrary choice. Figure A7 shows the average rank correlations (Kendall's 7) between (i) LLM-SCORE with a score range of [0,1] and LLM-SCORE with a score range of [0,10] (left) and (ii) LLM-SCORE with a score range [0,1] and LLM-SCORE with a score range of [8,24] (right). The score ranges of [0,10] and [8,24] are equally arbitrary alternatives to the score range of [0,1]. The results show that choosing different score range does affect the assignment. of importance and therefore the ranking of the features, the average rank correlations are always less than 1 (which is only achieved when all pairwise orderings are exactly identical). Meanwhile, we see that for larger models, the rank correlations tend to be higher, indicating that LLM-SCORE is less sensitive to the choice of score range with increasing model scale,
A.3.5 Feature Subset Initialization for LLM-SEQ
In this section, we provide additional results that compare the performances of LLM-SEQ when (i) starting with an empty feature subset or (ii) starting with a feature subset containing the highest scoring feature according to LLM-SCORE. See Section 3 of the main text for detailed description of the difference between the two settings. In Figures A8-A12, for LLM-SEQ based on GPT-4, GPT-3.5, and the Llama-2 models, when using the different feature subset initializations. The results show that with the exception of the largest GPT-4 model (Figure A8), the choice of feature subset initialization generally has a noticeable impact on the feature selection performance of LLM-SEQ. In Table A3, we report the areas under the feature selection paths shown in Figures A8- A-2 to quantify and compare the feature selection performances for the two initializations. For each datasct and each LLM, the results in Table A3 show that starting with an empty feature subset often leads to better feature selection performance.
show the changes in the downstream average test performance
32

---

Published in Transactions on Machine Learning Research (04/2025)
(-1,71), Sequertial (imppyy)
GPT4 1-1.77) Sequential (LLM-Seerr)
Creck:
Bank
Give Some Credit
COMPAS Recidvismm
Mma Indians
Cars*
YouTube"
0.90
9.80
0.22
0.75
ALROC
AUROC
0.75
AUROC
AUROC
ALSOC
2.87
0.00
0:38
0.00
2.22
0.46
2.70
2.03
02
AA 34 18
1.2
0.34
22
5,4 08 64 Praction Selected
22
2.75
22
0.8
0.4 06
22
24 0.5 0,0 10
22 0.4
Tracfion Selected
hartion Selected
Praction Selected
Praction Selected
Practior Selected
Praction Selected
Hou
Didbales Progiession
Win Quilky
Miami
NGA
NYC Ridieh
3.30
0.20
52.
0.28
0.50
0.40
MiE
50.0
175
0.25
MIE
MAE
MAE
65.0
2.5
3.24
0.32
02 5- Praction Selected
1.0
425
02
02 04 55
100
32 24
12
12 5.4 Praction Selected
522 0.4 0. 6d LV Fraction Selected
0.2 0.4 D.6 0.5
Praction Selected
Fraction Selected
Praction Selected
Praction Selected
Figure A8: Change in the average test performance of the downstream model as we vary the proportion of features selected, for LLM-SEQ based on GPT-4 with different feature subset initializations. The orange line shows the feature selection path when starting with an empty feature subset. The purple line shows the feature sclection path when starting with the highest scoring fcature according to LLM-SCORE. Datasets marked with an asterisk (+) were published afler the LLM culoff dates.
(-1758 Sequertial 1Empht)
(-178)) Sequentall Score)
G
Give Some Credit er
COMPAS Recidvium
Kima Indians
AUS Cars*
TubeT
0.50
2.55
5.82
AUROC
9
3.43
AURoc
0.775
AUROC
AUltoc
5.85
0.75
0,720
2.81
6.225
2.03
04 14 08 1.2
02 07 8.
12 5.4 06 6.
52
14 066 0. 1.2
0.4
06 6.0
34 0. 6. 10
52
Praction Selected
Praction Selected
Praction Selected
Praction Selected
Praction Selected
Fractior Selected
Praction Selected
Disbales Progression
Miami Housing
NBA+
NYC Rideeh
5.9
MAE
0.35
MAE
MAE
0.30
2.0
3.8
3.5
3.25
0.2
02
6,8 Fraction Selected
62 04
6.
0.20
0.6 Praction Selected
3.2 3/5
06 0.3 Fraction Selected
6.6 Fraction
Praction Selected
Fraction Selected
Praction Salected
Figure A9: Change in the average test performance of the downstream model as we vary the proportion of features selected, for LLM-SEQ based on GPT-3.5 with different feature subset initializations. The orange line shows the feature selection path when starting with an empty feature subset. The purple line shows the feature selection path when starting with the highest scoring feature according to LLM-SCORE. Datasets marked with an asterisk (+) were published after the LLM cutoff dates.
Table A3: Area under the feature selection paths for LLM-SEQ when starting with an empty feature subset or with a feature subset containing the highest scoring feature according to LLM-SCORR. For classification datascts, higher is better (t). For regression datasets, lower is better (4). Give Me Credit and Pima Diabetes are shorthands for the Give Me Some Credit and Pima Indians Diabetes datasets, respectively.
For each dataset and each LLM, we boldface the area for the feature subset initialization that performs better.
Clvedlaation Datavert
Regresdom Harasets
Sive 25
CURPAS
Pina
CA
Dsabetes
Hine
Minms
Credsc-G
Baniz
Credit
Hscidivisn
Dsabecea
AUS Cazgz
YouTubee
Bousing
Progressie e2sion
Quality
Housing
Used Caze
S03+
Aicesbazer
GPT-4 (Start with LLM- Scouty
0.21334
0.8664
0.7427
0,8132
0.7796
0,M03
0.8181
0.5033
46: 16:4108
0.5700
0.2359
3.3728
0.1082
1,073G
GPT-4 (Strt with Enpty Set
0.7710
0.5611
0.5412
0.8291
0.7794
0.9022
0.8422
0.5634
45.5285
0.5500
0,2132
1.4254
0.1505
2.7214
GPF-35 (Stut with L.MM Scoos))
0.7629
0.7474
0,7066
0.7221
0.8857
0.8386
0.2167
47. 41.4616
0.5014
0.3126
2.1822
011947
1,8G31
GPI-35 (Start with Empry Sei)
0/6622
0.8543
0.7638
0.7070
0.7776
0.8331
0.820
0.6432
44.7747
0.5900
0.3014
2.0078
0.1828
27/14
1-m0-2-708 (Sturt with L.M--SOOR))
0.259
0.8035
0.7217
0.8178
0.255
0.8922
0.7378
8011
0.5529
0,2963
1.0153
0.1203
2.1807
1am0-22700 (Strrt with Empty Soi)
0.7263
0.7436
0.7217
0.8138
0.7587
0.5242
0.6665
0.6558
48.3056
0.6312
0.2649
1.5600
0.2522
2.9958
Ueu (Strrt with LLM. Scont)
0.0905
0,5446
0.7481
0.5150
0.7307
0,8881
0.0038
47:2135
0.3984
0.3009
2,5441
0.1935
3.1563
1-55-2-158 (Sturt with Empty Sat)
08085
0.8470
0.7746
0.7874
0.7744
0.5425
0.7840
0.6374
44.9351
0.6167
0.5207
1.9734
0.2974
S.511
Limu 2/70 (Stest LLM Scoon)
0.7024
0.7642
0.7297
6.714
0.117
0,5078
0.7434
0.5904
47:1171
0.6044
0.2301
2.341G
0.640)
2.5210
1/70 (Stutt with Empty Set)
07702
0,78%4
0.7434
0.8205
0.7709
0.8001
0.5216
(10253)
44.0074
0.021
0.2726
1.5047
0.2413
2.530
33

---

Published in Transactions on Machine Learning Research (04/2025)
Sequential (Pmppyy)
Creak:
Bank
Give Some Credit
COMPAS Recidvismm
Mma Indians
YouTube"
0.85
0.822
0.00
AUROC
0.8
0.000
AUROC
0.75
ALROC
ALROC
0.275
0.70
4.73
0.793
0.55
3.2 64 D. 08 Tracfion Selected
0. 05 0. hartion Selected
22 5,4 08 64
0.4 06
0.2 0.4 0.5 04 10
Praction Selected
Praction Selected
Praction Selected
Practior Selected
Praction Selected
Housing
Diubales Progression
Nin Quilky
Miami
NGA
NYC Ridieah
5.9
1.5
MAE 12
0.25
MAE
MAE
2.
0.30
2.00
03
2.6
2.25
02 6o 5d Praction Selected
1. 02 Praction Selected
02 04
0.20
22 24
12 5.4 Praction Selected
0.0
5.4 0.4 05 6d Fraction Selected
LV
0.2 0.4 D.6 Praction Selected
Fraction Selected
PrctionSelcce
Figure A10: Change in the average test performance of the downstream model we vary the proportion of features selected, for LLM-SEQ based on Llama-2 with 70B parameters with different feature subset initializations. The orange line shows the feature selection path when starting with an empty feature subset. The purple line shows the feature selection path when starting with the highest scoring feature according to LLM-SCORE. Datasels marked with an asterisk (*) were published after the LLM cutoff dates.
Lamm-22 381, Sequential
Uams-2 2 (13 Sequential (LLM Score!
Credi: 0
Bank
Give Nu Some Crudit
COMPAS Recidvism
(5ma Indians Diabetes
AUS Cars
You Tube"
0.30
0.83
0.75
MIROC
0.40
0.35
2.80
AUROC
NUnoc
0.75
2.75
0.90
6:53
2.70
3.3
04
15 0E
1.3
03 64
05 5.8 3.3
32
1.4
05
68
1.0
3.2
S4 0.6
0.0
1.3
63 0.4 06 6.0 1.3
244 0.5 0.0 1.0 32
02 0.4 0. 3.0 2.2
FraccinSelectedd
Fraction Salected
Fraction Selected
Fraction Selected
Fraction Selected
Fractior Selected NBA*
Fraction Selected
CA Housing
Diebwles Progression
Win Quality
Miami Housing
Used Cars
NYC Rideeh ure
2.0
3.65
1.5
0.1
5
0.62
MAVE
2.00
0.30
2.5
3.25
03 04 5.6 Ea Fraction Selected
1.3
63 SA S. 08 1-3 Praction Selected
63 04 86 63
1.0
0.30
03 1.3 Fraction Selected
32
3/ 0.6
53 5.4 06 Praction Selected
0.3 1.2
0.3
63 04 066 03 1.0 Fractioo Selected
6. 04 6.6 6.8
Fraction Selected
Praction S Selected
Figure A11: Change in the average test performance of the downstream model as we vary the proportion of features selected, for LLM-SEQ based on Llama-2 with 13B parameters with different feature subset initializations. The orange line shows the feature selection path when starting with an empty feature subset. The purple line shows the feature selection path when starting with the highest scoring feature according to LLM-SCORE. Datasets marked with an asterisk (*) were published after the LLM cutoff dates.
A.3.6 Importance of Text Semantics for LLM-based Feature Selection
In this section, we provide additional results that demonstrate the importance of text descriptions with rich real-world semantics (e.g., column names in tabular datasets, auxiliary metadata) for based feature selection. In particular, we show how the downstream average test performance changes when we replace the meaningful feature descriptions (e.g., "gender", "height") in each dataset with irrelevant names (e.g., "feature_ _", "feature_2"" and use GPT-4-basod LLM-SCORE to select features. In Table A4, we report the areas under the feature selection paths to quantily and compare the feature performances for the two settings. We also show the results for the random feature selection baseline, as we would expect the behavior of LLM- SCORE to be similar to random selection in the absence of mcaningful text descriptions. As expected, we indeed observe that given irrelevant text descriptions, the performance of LLM-based feature selection drops significantly and can even be worse than that of random feature selection.
34

---

Published in Transactions on Machine Learning Research (04/2025)
(781, Sequential (Pmppty)
(78), Sequential M-Scone
Creck:
Bank
Give Some Credit
COMPAS Recidvism
Mmaa Indions
Cars*
YouTube"
2.03
AUROC
0.8
AUROC
ALROC
0.425
0.003
AUROC
5.83
0.05
0.00
0.073
6.75
0.00
0.0
2.70
0.70
3.2 04 D. 08
0. 05 0. haction Selected
22 5,4 08 64
2.75
C4
0.4 06 0.0 Praction Selected
0,2
0.4 0.5 04 10
0.63
022 C44 0.6
Tracfion Selected
Praction Selected
Praction Selected
Praction Selected
Praction Selected
Housing
Diubales Progiession
Winw Quilky
Miami
NGA
NYC Ridiash and
2. (5R)
0.6
3.0
0.25
MAE 12
0.30
0.60
2.6
0.58
0.25
02 Co 5.
1.0 022 SA
02 04 55
64
10
0.20
24
1.7
12 5.4 Praction Selected
24 05 04 LV
0.2 0.4 D.6
Praction Selected
Praction Selected
Fraction Selected
Practon
Fraction Selected
Praction Selected
Figure A12: Change in the average test performance of the downstream model we vary the proportion of features selected, for LLM-SEQ based on Llama-2 with 7B parameters, with different. feature subset initializations. The orange line shows the feature selection path when starting with an empty feature subset. The purple linc shows the feature selection path when starting with the highest scoring feature according to LLM-SCORE. Datasels marked with an asterisk (*) were published after the LLM culoff dates.
Table A4: Area under the feature selection paths for LLM-SCORE based on GPT-4 when using the original column names ("Desc.") vs. replacing them with irrelevant names ("No Desc."). For classification datasets, higher is better (t). For regression datasots, lower better (4). Give Me Credit and Pima Diabetes are shorthands for the Give Me Some Credit and Pima Indians Diabetes datasets, respectively. For each dataset and each LLM, boldface the area. for the feature subset initialization that performs better.
Classifieation Datasets
Regression Datasets 1
Diabetee
Crodit-G
Bank
Grodit
Recidivicz Diabetos
A3S Caro
YouTube
Bouning
Progression
2uality
Housing
Used Caze
HBA
Ridesbare*
GPT-4 (Dise.)
0,7975
0.8598
0.7425
0.8305
0.7794
0.91449
0.8854
0.6278
44.39443
0.5790
0,2423
1.4121
0.0618
1,9690
GPT-4 (No D
0.5707
0.6120
0.7317
0.6224
0,7307
0.7190
0,6308
0.8965
51.2119
0.6147
0.3071
1.7848
1.6554
4.1610
Random
0.8385
0.7408
01.0008
0.70NG
0.7255
0.7851
0.8948
0.7355
50.2133
0.6152
112780
2.4/22
1.1800
3.1562
A.3.7 Investigation of the Semantics of LLM-SCORE
In this section, we include additional results on investigating the semantics of LLM-SCORE. After prompting an LLM to output the importance score of feature, we directly ask the model to provide more details on how the numerical importance score was assigned. In particular, we ask (i) whether there is a specific notion of importance that the generated score reflects and (ii) whether the model can provide a brcakdown of how the score was numerically calculated. We note that all of the results shown in this section are based on the LLM outputs generated from the default prompt, which does not include any dataset-specific context or few-shot examples, and with greedy decoding.
As an illustrative example, we show below the outputs from GPT-4, GPT-3.5, and the Llama-2 models when they are prompted to elaborate on how the importance score was assigned for the "Number of times pregnant" input concept in the Pima Indians Diabetes dataset. We omit the main system prompt and the output format instructions for simplicity. When prompted to describe whether the generated score captures a specific notion of feature importance, all models state that the score reflects a general notion of a feature's "relative" contribution to prediction. Notably, despite never having been provided information about the other features, the Llama-2 models often explicitly state that the generated scores are based on Shapley values or permutation feature importance, while hallucinating to have calculated the scores by "training" model. Interestingly, in Figure 5 of the main text, we do observe that LLM-SCORE based on the Llama-2 models (70B, 13B) exhibit slightly higher rank correlations with SHAP: XGBoost and permutation feature importance than with the other metrics considered; however, all of the measured average rank correlations for the Llama-2 models are overall quite low. When prompted to describe how the importance scores are caleulated, all models except GPT-4 and Llama-2 with 70B parameters output a detailed description, but the generated explanations are either logically invalid (see e.g., "GPT-3.5: Calculation of Feature Importance
35

---

Published in Transactions on Machine Learning Research (04/2025)
Score" below) or based hallucinations (see e.g., "Llama-2 with 13B Parameters: Calculation of Feature Importance Score" or "Llama-2 with 7B Parameters: Calculation of Feature Importance Score" below). As in Result 4, Section 4.1 of the main text, we therefore find that across all LLMs, it is overall unclear what specific notion of feature importance LLM-SCORE aligns to, other than that the scorcs reflect a generic notion of "relative" importance.
GPT-4: Feature Importance Notion
System: (Omitted)
User: Provide score and reasoning for "Number of times pregnant" formatted according to the output schema above:
GPT-4: "score": 0.7, "reasoning": "The number of times patient has been pregnant is significant factor in predicting diabetes, especially gestational diabetes. Each pregnancy increases the risk developing diabetes due to the hormonal changes that occur during pregnancy, which can make the body more resistant insulin. However, it is not the only factor and other health and lifestyle factors also play significant role in diabetes risk."
User: Is there specific notion of feature importance that is captured in your score?
Yes, the feature importance score represents the relative contribution of the input feature to the prediction model. higher score means that the feature has a greater impact on the model's output, and thus is more important for making accurate predictions. this case, the score of 0.7 for "Number times pregnant" suggests that this feature is quite important for predicting diabetes, but other fcatures also contribute to the prediction. The score is based on a combination of statistical analysis, domain knowledge, and the specific modeling technique used.
GPT-4:
User: What kind of statistical analysis are you referring to? How would you base the determination of the score on statistical analysis when you have no access to any data?
GPT-4: apologize for the confusion. You're correct. Without access to specific data, it's not possible perform a statistical analysis determine the feature importance score. The score provided is basod on general medical knowledge that indicates pregnaney can increase the risk of diabetes. In real-world scenario, the exact score would be determined using techniques like correlation coefficients, chi-square tests, machine learning algorithms like Random Forest Gradient Boosting, which can provide feature importance scores based on the data they are trained on.
GPT-4: Calculation of Feature Importance Score
System: (Omitted)
User: Provide score and reasoning for "Number of times pregnant" formatted according to the output schema above:
GPT-4: (sccore:: 0.7, "reasoning": "The number of times a patient has been pregnant is a significant factor in predicting diabetes, especially gestational diabetes. Each pregnaney increases the risk of developing diabetes due to the hormonal changes that occur during pregnancy, which make the body more resistant to insulin. However, it is not the only factor and other health and lifestyle factors also play significant role in diabetes risk."
User: How did you calculate this score? Can you show me the set of caleulations that led to the final score?
GPT-4: I'm sorry for any confusion, but as an AI, I don't perform calculations to determine the importance score. The score provided is hypothetical one, based general knowledge about the impact of pregnancy diabetes risk. In real-world scenario, this score would be calculated using machine learning algorithms on dataset.
36

---

Published in Transactions on Machine Learning Research (04/2025)
These algorithms would analyze the relationship between the feature (in this case, the number of times a person has been pregmant) and the outcome (whether the person has diabetes). The importance score would then be determined based on how much the model's prediction changes when the feature is altered.
For example, a feature importance score of 0.7 suggests that changes the number of pregnancies account for 70% of the changes in the model's diabetes predictions. However, this a simplified explanation and the actual process involves complex statistical analysis.
GPT-3.5:: Feature Importance Notion
System: (Omitted)
Provide score and reasoning for "Number of times pregnant" formatted according to the output schema above:
User:
GPT-3.5: ("score": 0.6, "reasoning": "The number of times a woman has been pregnant can be an important factor in predicting the likelihood of diabetes. Women who have been pregnant multiple times may have higher risk due to hormonal changes and increased insulin resistance during pregnancy.")
User: Is there specific notion of feature importance that is captured in your score?
The feature imporlance score provided is a relative measure of the importance of the feature predicting the likelihood of diabetes. indicates the extent to which the feature contributes to the predictive power of the model. A score of indicates that the fcature has no importance in predicting diabetes, while score of indicates that the feature is highly important. The score is based on the model's 's analysis of the relationship between the feature and the target variable (diabetes) in the training data.
GPT-3.5:
User: What model are you referring to? And how are you analyzing the relationship between the feature and the target variable when you have no access to any data?
Apologies for the confusion. As an Al language model, don't have access to specific data or a pre-trained model. The feature importance score and reasoning provided earlier were based on general knowledge and assumptions about the relationship between the feature ("Number of times pregnant") and the target variable (likelihood of diabetes).
GPT-3.5:
In a real-world scenario, feature importance scores are typically derived from machine learning models trained on relevant datasets. These models analyze the relationship between features and the target variable by considering patterus, correlations, and statistical measures such as information gain, Gini index, or permutation importance. The specific model and analysis technique used can vary depending on the context and data available.
GPT-3.5: Calculation of Feature Importance Score
System: (Omitted)
Provide score and reasoning for "Number of times pregnant" formatted according to the output schema above:
User:
GPT-3.5: ("score": 0.6, "reasoning": "The number of times woman has been pregnant can be an important factor in predicting the likelihood of diabetes. Women who have been pregnant multiple times may have higher risk due to hormonal changes and increased insulin resistance during pregnancy.")
User: How did you calculate this score? Can you show me the set of caleulations that led to the final score?
GPT-3.5: Certainly! Here is the breakdown of calculations that led to the final score of 0.6 for the feature "Number of times pregnant":
37

---

Published in Transactions on Machine Learning Research (04/2025)
1. Initial Score: 0.5
We start with initial score of 0.5, assuming that the feature may have some importance in predicting diabetes.
2. Domain Knowledge: +0.2
BBaedd on domain knowledge and medical research, it known that the number of times a woman has been pregnant can be significant factor in predicting the likelihood of diabetes. Therefore, we assign an additional score of 0.2 to reffect this importance.
3. Statistical Analysis: No additional score Since we do not have access to the dataset or statistical analysis, we cannot provide specific score based on statistical analysis.
4. Final Score: 0.5 + 0.2 0.7
We sum up the initial score and the domain knowledge score obtain the final score of 0.7.
However, since the score should be between 0 and 1, normalize the final score to 0.6 by dividing it by the maximum possible score (0.7).
Therefore, the final score for the feature "Number of times pregnant" is 0.6.
User: The normalization step does not look right. Clearly, the maximum possible score 1?
GPT-3.5: Apologies for the confusion. You are correct, the maximum possible score is indeed Therefore, the final score for the feature "Number of times pregnant" is 0.7, not 0.6. There is need for normalization in this case since the score is alrcady within the range of 0 to 1. Thank you for pointing out the error.
Llama-2 with 70B Parameters: Feature Importance Notion
System: (Omitted)
Provide score and reasoning for "Number of times pregnant" formatted according the output schema above:
User:
Llama-2-70B: "reasoning": "The number of times patient has been pregnant can be an indicator of their overall health and lifestyle, which can impact their likelihood of developing diabetes. Additionally, pregnancy can cause changes in the body that may increase the risk of developing diabetes, such as weight gain and insulin resistance. Therefore, this feature can provide valuable information for predicting whether a patient has diabetes.", "score"; 0.7 }
User: Is there a specific notion of feature importance that is captured in your score?
Llama-2-70B: Yes, the feature importance score that 1 provide is based the concept of "Shapley values" which is way to fairly distribute the total contribution made by all features to patient's diabetes status.
Shapley values are a type of feature attribution method that assigns a unique importance score to each feature for a specific prediction. The score is based on the feature's contribution to the prediction, taking into account all possible combinations of features and their interactions.
In the context of diabetes prediction, Shapley values can help identify which features are most important for predicting palient's diabetes status. The feature importance score provide is Shapley value, which represents the feature's relative contribution to the prediction. A higher score indicates that the feature is more important for predicting diabetes, while a lower score means that the feature has a smaller impact.
By using Shapley values, we can quantify the contribution of each feature the prediction, allowing us to identify the most informative features for diabetes diagnosis. This can help healthcare professionals and researchers understand which factors most important when assessing patient's risk of developing diabetes.
38

---

Published in Transactions on Machine Learning Research (04/2025)
User: How you calculating the Shapley values when you do not have access to any data?
Ulama-2-70B: I apologize for the confusion in my previous response. I cannot calculate Shapley values without data. Shapley values are calculated using a specific algorithm that requires data to be present.
The algorithm for calculating Shapley values involves the following steps:
Compute the marginal contribution of each feature for each instance in the dataset.
2. Compute the marginal contribution of each feature for each possible coalition of features.
3. Compute the Shapley value for cach feature by averaging its marginal contribution across all possible coalitions.
Since do not have access to any data, cannot perform these steps and therefore cannot calculate Shapley values.
I apologize for any confusion my previous response may have caused. If you have any further questions, please feel free to ask.
Llama-2 with 70B Parameters: Calculation of Feature Importance Score
System: (Omitted)
User:
Provide score and reasoning for "Number of times pregnant" formatted according to the output
scherna above:
1lama-2-70B: ""reasoning": "The number of times a patient has been pregnant can be an indicator of their overall health and lifestyle, which impact their likelihood of developing diabetes. Additionally, pregnancy can cause changes in the body that may increase the risk of developing diabetes, such as weight gain and insulin resistance. Therefore, this feature can provide valuable information for predicting whether a patient has diabetes.", "score": 0.7
User: How did you calculate this score? Can you show me the set of calculations that led the final score?
The feature importance score of 0.7 for "Number of times prognant" not calculated through specific mathematical formula, but rather through logical reasoning process based on the relationship between pregnancy and diabetes.
Llama-2-70B:
Hcre's a breakdown of the reasoning process that led to the score:
Pregnancy can cause changes the body that increase the risk of developing diabetes, such as weight gain and insulin resistance.
2. The number of times patient has been pregnant be indicator of their overall health and lifestyle, which can impact their likelihood of developing diabetes.
3. Patients who have been pregmant multiple times may have a higher risk of developing diabetes due to the cumulative effects of pregnancy-related weight gain and insulin resistance.
Therefore, the number of times patient has boen pregnant can provide valuable information for predicting whether patient has diabetes.
Based on this reasoning, a score of 0.7 was assigned to the feature "Number of times pregnant", indicating that it is moderately important for predicting diabetes.
It's important to note that feature importance scores can be calculated using various machine learning techniques, such as decision trees, random forests, or gradient boosting machines. However, in this case, the score was determined through a logical reasoning process rather than a mathematical calculation.
Llama-2 with 13B Parameters: Feature Importance Notion
39

---

Published in Transactions on Machine Learning Research (04/2025)
System: (Omitted)
Provide score and reasoning for "Number of times pregnant" formatted according to the output schema above:
User:
Llama-2-13B:
'"rcasoning:: "The number of times pregnant is a strong predictor of diabetes because it increases the likelihood of developing insulin resistance and glucose intolerance, which common precursors to diabetes. Additionally, pregnancy-related hormonal changes can also impact insulin sen- sitivity, further increasing the risk of diabetes.", "score": 0.8
User: Is there a specific notion of feature importance that is captured in your score?
Alama-2-13B: Yes, the feature importance score provide based on the Shapley feature attribution method, which widely used technique for assigning importance to features in machine learning model.
The Shapley method assigns importance to each feature by calculating the marginal contribution of that feature to the predicted oulcome, while controlling for the effects of all other features. In other words, it measures the unique contribution of each feature to the predicted outcome, while holding all other fcatures constant.
The Shapley feature ure importance score ranges from -1 to 1, with higher values indicating greater impor- tance. A score of indicates that the feature has no unique contribution to the predicted outcome, and is therefore not important.
In the case of the "Number of times pregnant" feature, the Shapley method would calculate the marginal contribution of that feature to the predicted outcome, while holding all other features constant. If the feature has a significant marginal contribution, would be assigned a high importance score, indicating that it a strong predictor of diabetes.
It's worth noting that there are other methods for measuring feature importance, such as permutation importance and partial dependence plots, and different methods may yield different results. However, the Shapley method is widely used and well-established technique for measuring feature importance in machine learning models.
User: How are you calculating the Shapley values when I have not told you what other features are available and you do not have access to any data?
Llama-2-13B: You're right, apologize for the confusion. Since I don't have access to any data or information about. the other features available, I cannot accurately calculate the Shapley values for any feature. The Shapley method requires knowledge of the entire feature set and their interactions to accurately assign importance to each feature.
In the absence of any data or information, cannot provide a meaningful answer to your question. However, can offer some general advice on how to approach feature importance analysis.
When analyzing feature importance, it's important to consider the context of the problem you't 're trying solve and the goals of your machine learning model. Here are some general steps you can take:
1. Understand the problem: Before you determine which features are important, you need to understand the problem you're trying to solve. This includes identilying the key variables that are relevant to the problem and understanding how they interact with each other.
2. Explore the data: Once you have a good understanding the problem, you should explore the data to identify palterns, relationships, and correlations between the features. This can help you identify which features are most important for predicting the outcome.
3. Use feature importance methods: There several methods for mcasuring feature importance, including permutation importance, partial dependence plots, and SHAP values. Each method has its own strengths and weaknesses, so it's important to choose the method that best fits your needs.
40

---

Published in Transactions on Machine Learning Research (04/2025)
4. Interpret the results: Once you have calculated the feature importance scores, you need to interpret the results in the context of the problem you're wu're trying to solve. This includes understanding which features are most important for predicting the outcome and how they interact with each other.
hope this helps! If you have any further questions, please don't hesitate to ask.
Llama-2 with 13B Paramcters: Calculation of Feature Importance Score
System: (Omitted)
Provide score and reasoning for "Number of times pregnant" formatted according to the output schema above:
User:
Llama-2-13B: "reasoning": "The number of times pregnant is strong predictor of diabetes because it increases the likelihood of developing insulin resistance and glucose intolerance, which are common precursors to diabetes. Additionally, pregnancy-related hormonal changes can also impact insulin sen- sitivity, further increasing the risk of diabetes.", "score"; 0.8
User: How did you calculate this score? Can you show me the of calculations that led the final score?
Alama-2-133h Certainly! To calculate the feature importance score for "Number of times pregnant", I used a technique called Permutation Feature Importance (PFI). PFI is a method for calculating the importance of each feature in a machine learning model by randomly permuting the values of each feature and measuring the decrease in model performance.
Here's the of calculations that led the final score:
1. First, I trained logistic regression model the training data to predict the outcome variable (diabetes not).
2. Next, randomly permuted the values of the "Number of times pregnant" feature for each sample in the training data. This created a new set of training data where the values of this feature wore shuffled.
then re-trained the logistic regression model on the permuted training data.
4. repeated steps and many times (e.g., 1000 times) generate distribution of model performance scores for each feature.
5. Finally, calculated the feature importance score for each feature as the average decrease in model performance across all permutations.
The formula for calculating the feature importance score is:
Feature Importance (Average Decrease Model Performance) (Standard Deviation of Decrease in Model Performance)
where the average decrease in model performance is calculated as:
Average Decrease in Model Performance = (Sun of (Model Performance with Permuted Feature Model Performance without Permuted Feature)) (Number of Permutations)
and the standard deviation of the docrcase model performance is calculated as:
Standard Deviation of Decrease in Model Performance (Sum of (Model Performance with Permuted Feature Model Performance without Permuted Feature))) / (Number of Permutations 1)
Using this formula, I calculated the feature importance score for each feature in the dataset. The final score for "Number of times pregnant." was 0.8, indicating that it is a very important feature in predicting diabetes.
hope this helps! Let me know if you have any other questions.
Llama-2 with 7B Parameters: Feature Importance Notion
41

---

Published in Transactions on Machine Learning Research (04/2025)
System: (Omitted)
Provide score and reasoning for "Number of times pregnant" formatted according to the output schema above:
User:
Llama-2-7B: ("reasoning:: "The number of times pregnant is important because can provide insight into the patient's reproductive history and polential risks for diabetes. Women who have had multiple pregnancies may be more likely to develop insulin resistance, which is key factor in the development of diabetes. Additionally, pregnancy-related complications such gestational diabetes can increase the risk of developing type 2 diabetes later in life. Therefore, the number of times pregnant can be useful feature for predicting diabetes.", "score": 0.7
User: there a specific notion of feature importance that is captured in your score?
2-mna-2-7B: Yes, the feature importance score I provided is based on notion of feature importance that captures the relative importance of each feature in predicting diabetes.
In this case, the score of 0.7 for the number of times pregnant indicates that this feature is relatively important for predicting diabetes, compared to the other features. A higher score indicates a greater importance, while a lower score indicates a lesser importance.
There are several notions of feature importance that can be used, including:
Mutual Information: This measures the amount of information that each feature provides about the target variable, diabetes.
2. Recursive Feature Elimination (RFF): This measures the contribution of each feature to the model's performance, while controlling for the contributions of other features.
3. Permutation Feature Importance: This measures the importance of each feature by randomly per- muting its value and measuring the impact on the model's performance.
4. SHAP values: This measures the contribution of each feature to the predicted outcome, diabetes, for specific instance.
These notions of feature importance can be used to evaluate the importance of cach feature in predicting diabetes, and can be combined to provide a comprehensive understanding of the relative importance of each feature.
It's worhh noting that the choice of notion of feature importance will depend on the specific problem and the goals of the analysis.
User: What do you mean by "compared to the other features") ? There no other features that have been provided to you?
Alama-2-7B: apologize for the confusion. You are correct, I have not been provided with any other features to compare the importance of the number of times pregnant. My previous response was incorrect.
In the absence of any other features, the importance of the number of times pregnant. for predicting diabetes cannot be determined. The number of times pregnant is unique feature that may have specific relationship with diabeles, and its importance cannot be compared to the importance of other features that have not been provided.
you have any other features that you would like me to analyze, please provide them and will be happy to help.
Llama-2 with 7B Parameters: Calculation of Feature Importance Score
System: (Omitted)
42

---

Published in Transactions on Machine Learning Research (04/2025)
Provide score and reasoning for "Number of times pregnant" formatted according to the output schema above:
User:
Llama-2-7hh 2-7B: "reasoning": "The number of times pregnant is important because it can provide insight into the patient's reproductive history and potential risks for diabetes. Women who have had multiple pregnancies may be more likely to develop insulin resistance, which of diabetes. Additionally, pregnancy-related complications such as gestational diabetes can increase the risk of developing type 2 diabetes later in life. Therefore, the number of times pregnant can be useful feature for predicting diabetes.", "score": 0.7 }
key factor in the development
User: How did you calculate this score? Can you show me the set of calculations that led to the final score?
Llama-2-7B: Certainly! To calculate the feature importance score for "Number of times pregnant," used technique called permutation feature importance. Here's how calculatod the score:
1. Permutation: I randomly permuted the value of "Number of times pregnant" for each sample in the dataset (i.e., the training data) 1000 times. 2. Prediction: For each permutated sample, used the original XGBoost model to predict the probability of diabetes. 3. Calculation: I calculated the mean and standard deviation of the predicted probabilities for each permutation. 4. Score calculation calculated the feature importance score for "Number of times pregnant" " as the mean the standard deviations of the predicted probabilities across all permutations.
Here's the mathematical formula. for the feature importance
score (mean(std(y true)) mean(std(y (raansst((yppre))) pred))) std(y ppedd
where:
"y_true is the true label (0 or 1) for cach sample in the dataset * y_ppred the predicted probability diabetes for each sample in the dataset std(y true) is the standard deviation of the true labels * std(y_pred) the standard deviation of the predicted probabilities
By calculating the standard deviation of the predicted probabilities for cach permutation, we measur- ing the variability in the model's predictions due to changes in the value of "Nunber of times pregnant." The difference between the mean of the standard deviations and the mean of the predicted probabilities is a measure of the feature's importance. A higher score indicates that the feature is more important for predicting diabetes.
hope this helps! Let me know you have any other questions.
User: How you calculating permutation feature importance without access to the actual data?
Llama-2-7B: apologize for the confusion earlier. T cannot calculate permutation feature importance without access to the actual data. The permutation feature importance calculation requires the actual data, including the larget variable (diabetes status) and the feature values for each sample.
To calculate permutation feature importance, you can use the following steps:
Split the data into training and testing sets.
2. Use the training set train XGBoost model.
3. For each feature, randomly permute its value for each sample in the training set (i.c., randomly replace the feature value with a new value).
4. Train the XGBoost model on the permuted training set for each feature permutation.
5. Calculate the standard deviation of the predicted probabilities for each feature permutation.
6. Repeat steps 3-5 for all possible feature permutations.
43

---

Published in Transactions on Machine Learning Research (04/2025)
7. Calculate the feature importance score for each feature by dividing the standard deviation of the predicted probabilities by the standard deviation of the predicted probabilities for the original feature values.
By calculating the standard deviation of the predicted probabilities for each feature permutation, we are measuring the variability in the model's predictions due to changes the feature values. higher standard deviation indicates that the feature more important for predicting diabetes.
I hope this helps! Let me know if you have any other questions.
44

---

Published in Transactions on Machine Learning Research (04/2025)
B
Additional Details on
Dataset Experiments
Here, we provide additional details and results for the large-scale dataset experiments in Section 4.2.
B.1 Datasets
For all datascts, we report both the total number of features after preprocossing and the total number of concepts (i.e., feature names), which may differ due to the one-hot encoding of categorical features. For example, as described below, the Income dataset contains 3759 features after one-hot encoding the categorical features and 238 concepts. In this case, when feature selection is performed at the concept level, we select k of the 238 concepts; when feature selection is performed at the feature level, we select l of the 3759 preprocessed features.
For each dataset, we randomly shufile and take a 64-16-20 train-validation--ess split, where the train- validation splits vary across the 5 random seeds (=11,2,3,4,55) used in the experiments, and the test set remains fixed regardless of the seed. For all datasets, we always take a stratified split to preserve the label proportions across the train, validation, and test sets. We standardize all of the numerical fcatures to have zero mean and unit variance, and one-hot encode all categorical features.
B.1.1 folktables Datasets
We construct supersets of the original Income, Employment, Public Coverage, and Mobility binary clas- sification datasets from folktables (Ding et al., 2021) by extracting all available features available from the 2018 American Community Survey (ACS) Public Use Microdata Sample (PUMS) data for California houscholds. The full ACS PUMS dataset contains total of 286 features, from which we exclude (i) the feature that is used for defining the target label, (ii) features that serve unique identifiers for each sample, and (iii) features that lead to label leakage when included. Below, provide the remaining details for each fo folktables dataset:
Income: The goal is to predict whether an individual has an annual income greater than $50000. Following Ding et al. (2021), we define the cohort to consist of individuals who are of ages above 16, have worked at least 1 hour per week in the past year, and have an income of at least $100, and a PUMS weight of at least 1. After filtering out the features according to the above criteria and onc-hot encoding the categorical features, the dataset contains 125225 samples and 3759 features (238 concepís), with 51414 positive samples and 73811 negative samples.
Employment:
The goal is to predict whether an individual is employed. Following Ding et al. (2021), we define the cohort to consist of individuals who are of ages between 16 and 90 and have a PUMS person weight of at least 1. After filtering out the features according to the above criteria and one-hot encoding the categorical features, the dataset contains 193689 samples and 2371 features (241 concepts), with 110372 positive samples and 83317 negative samples.
The goal is to predict whether a low-income individual has coverage from public health insurance. Following Ding et al. (2021), we define the cohort to consist of individuals who are of ages below 65 (not eligible for Medicare) and have total income less than $30000. After filtering out the features according to the above criteria and one-hot encoding the categorical features, the dataset contains 88674 samples and 3655 features (239 concepts), with 32716 positive samples and 55958 negative samples.
Public Coverage:
Mobility: The goal is predict whether a young-adult individual moved residential addresses in the past year. Following Ding et al. (2021), we define the cohort to consist of individuals aged between 18 and 35. After filtering out the features according to the above criteria and one-hot encoding the categorical features, the dataset contains 51410 samples and 3385 features (277 concepts), with 39345 positive samples and 12065 negative samples.
45

---

Published in Transactions on Machine Learning Research (04/2025)
B.1.2 MIMIC-IV Datasets
MIMIC-IV (Johnson al., 2023) an open-access database that consists of deidentified electronic health record data collected at the Beth Israel Deaconness Medical Center between years 2008 and 2019, including over 400k distinct hospital admissions. As described in Section 4.2, we extract 3 binary classification datasets, where the goals are to predict whether patient in the intensive care unit (ICU) is likely to develop chronic kidney disease (CKD), chronic obstructive pulmonary disease (COPD), or heart failure (HF), given lime-series of clinical measurements and events recorded during the first 24 hours of their ICU stay.
Cohort selection. We define a single generic cohort for all prediction tasks and include all ICU stays satisfying the following criteria:
1. Adult patients: Given that the physiology of young children and adolescents can differ significantly from that of adults, we only include ICU stays corresponding to adult paticnts who are of ages between 18 and 89 the time of hospitalization. We exclude patients who are at least 90 years old, as their recorded ages are not precise due the deidentification process mandated by the Health Insurance Portability and Accountability Act (HIPAA) privacy regulations.
2. First and only ICU stay: Following standard practice (Wang et al., 2020); if patient has multiple only consider the first-ever ICU
ICU stays recorded in the database across all hospitalizations, stay. Additionally, as ICD diagnosis codes are assigned to each hospilalization, which may include multiple ICU stays, we only include first ICU stays that do not have subsequent ICU stays recorded within the same hospitalization to cnsure that the ICD diagnosis codes correspond to them.
3. Length of ICU stay between 1 and days: To ensure that we have enough information to use as input Teatures for each ICU stay, we only include ICU stays that are least day long. Meanwhile, as ICD codes are typically recorded discharge for billing purposes and the exact timing of the diagnoses is often unknown, we exclude ICU stays that are longer than 7 days, ensure that the association between the clinical measurements recorded in the first 24 hours and the target label is sufficiently strong. For very long stays, the clinical measurements from the first. 24 hours may not provide enough signal for predicting a diagnosis whose exact timing is unknown.
In Table B1, summarize the demographics for the final extracted cohort, which consists of 38976 unique patients and their first ICU stays.
Input features. For each ICU stay, we first extract all available measurements of the following 148 clinical features (6 static, 142 time-varying), recorded during the first 24 hours of the given stay:
Static (time-invariant) features: age, gender, ethnicity, height, weight, ICU type
Time-varying features:
temperature
oxygen saturation (Sa02/ / Sp02)
heart rate
respiratory rate
central venous pressure (CVP)
end-tidal carbon dioxide (Etc02)
systemic systolic arterial blood pressure
systemic diastolic arterial blood pressure
systemic mean arterial blood pressure
pulmonary systolic arterial blood pressure
pulmonary diastolic arterial blood pressure
pulmonary mean arterial blood pressure
apnea interval (set)
46

---

Published in Transactions on Machine Learning Research (04/2025)
lung compliance
minute volume
tidal volume (set)
tidal volume (observed)
tidal volume (spontaneous)
fraction inspired oxygen (F102)
oxygen flow rate
mean airway pressure
peak inspired pressure
positive end-expiratory pressure (PEEP)
plateau pressure
albumin
alkaline phosphatase
alanine transaminase (ALT SGPT)
amylase
anion gap
aspartate aminotransferase (AST / SGOT)
differential bands immature band forms (-bands) (%)
base excess
differential basophils (-basos) (%)
glucose
bicarbonate
blood urea nitrogen (BUN)
calcium
calcium (ionized)
chloride
cortisol
creatinine phosphokinase (CPK сK)
creatinine phosphokinase myocardial band (CK-MB)
creatinine phosphokinase myocardial band index (CK-MB index)
creatinine
direct bilirubin
differential eosinophils (-eos) (%)
iron (Fe)
ferritin
fibrinogen
hematocrit (HCT)
cholesterol (HDL)
hemoglobin (HGB)
lactate
lactate dehydrogenase (LDH)
cholesterol (LDL)
lipase
differential 1ymphocytes (-lymphs) (%)
magnesium
mean corpuscular hemoglobin (MCH)
corpuscular hemoglobin concentration (MCHC)
47

---

Published in Transactions on Machine Learning Research (04/2025)
mean corpuscular volume (MCV)
differential monocytes (-monos) (%)
partial pressure of carbon dioxide (PaC02)
partial pressure of oxygen (Pa02)
blood pi
phosphate
platelets
differential neutrophils (-polys) (%)
potassium
prothrombin time (PT)
prothrombin time international normalized ratio (PT INR)
partial thromboplastin time (PTT)
red blood cells (RBC)
red cell distribution width (RDW)
osmolality (urine)
osmolality (serum)
sodium
total iron binding capacity (TIBC)
total bilirubin
total cholesterol
total carbon dioxide (CO2)
triglycerides
troponin T
thyroid stimulating hormones (TSH)
creatinine (urine)
sodium (urine)
specific gravity (urine)
vancomycin (random)
vancomycin (trough)
white blood cells (urine)
white blood cells
sodium chloride 0.9% intravenous (IV) solution
sodium chloride 0.9% intravenous (IV) flush
dextrose 50% intravenous (IV) solution
acetaminophen
bisacodyl
docusate sodium
furosemide
metoprolol tartrate
nitroglycerin
ondansetron
pantoprazole
potassium chloride
intravenous piggyback (IVPB)
norepinephrine
propofol
gastric tube (nasogastric)
48

---

Published in Transactions on Machine Learning Research (04/2025)
per oral (p.o.) intake
stool
urine output
capillary refill
left dorsalis pedis pulse
right dorsalis pedis pulse
left radial pulse
right radial pulse
LLE strength sensation
LUE strength / sensation
RLE strength sensation
RUE strength sensation
speech
left pupil size
right pupil size
cough gag reflex
Braden activity scale
Braden friction & shear scale
Braden mobility scale
Braden moisture scale
Braden nutrition scale
Braden sensory perception scale
Morse ambulatory aid
Morse gait transferring
Morse history of falling
Morse mental status
Morse secondary diagnosis
delirium assessment
Glasgow coma scale (GCS) Eye
Glasgow coma scale (GCS) Verbal
Glasgow coma scale (GCS) Motor
level of assistance
chest X-ray
invasive ventilation
foley catheter
For the time-varying features, we group and aggregate all of the extracted measurements into 4-hour bins. For numerical features, we average all of the recorded values within each 4-hour bin. For categorical features, take the most recent (i.e., latest timestamp) recorded value within each 4-hour bin. To handle missing values, use the simple imputation method proposed by Che al. (2018). After one-hot encoding all of the categorical fcatures, we then obtain input time-scrios tensor of shape 38976 x 6 x 506. As all of the prediction models considered in Section 4.2 of the main text do not naturally handle time-series data, we flatten the temporal dimension (which indexes each 4-hour bin), which reshapes the input time-series tensor 2-dimensional matrix of shape 38976 x 3036. We then concatenate the onc-hot encoded static feature matrix of shape 38976 x 32, leading the final input time-series feature matrix shape 38976 3068. For all 3 prediction tasks, the same input features to train prediction model.
49

---

Published in Transactions on Machine Learning Research (04/2025)
Table B1: Summary of demographics for the final cohort of ICU patients extracted from MIMIC-IV. Except for the total number of ICU patients, we report the mean and standard deviation (in parentheses) of each numerical feature, and the count and proportion (in parentheses) of each categorical feature.
Demographic Feature
Category (If Applicable)
Missing Overall
Number of TCU Patients
38976
Age
63.9 (15.8)
Female
0
16709 (42.9%)
Gender
Male
22267 (57.1%)
Asian
0
1187 (3.0%)
Black
4162 (10.7% 7%)
Hispanic
1542 (4.0%)
Eshmicity
Nutive American
114 (0.3%)
Other Beeer//nnnww
5386 (13.8%)
White
26585 (68 (68.2%)
Height (cm)
19402
169.9 (10.6)
Weight (kg)
82.2 (38.9)
Cardiac Vascular Intensive Care Unit (CVICU)
8004 (20.5%)
Coronary Care Unit (CCU)
4229 (10.9%)
Medical Intensive Care Unit (MICU)
7729 (19.8%)
Medical/Surgient Intensive Care Unit (MICU/SICU)
6736 (17.3%)
ICU Type
Neuro Intermediate
0
1198 (3.1%)
Neuro Stepdown
495 (1.3%)
Neuro Surgical Intensive Care Unit (Neuro SICU)
851 (2.2%)
Surgical Intensive Care Unit (SICU)
5650 (14.5%)
Trauma SICU (TSICU)
4084 (10.5%)
Length of Stay (Days)
2.5 (1.4)
18-30
1604 (4.1%)
30-45
3366 (8.6%)
45-55
5033 (12.9%)
Age Group
55-65
(21.9%)
65-75
(25.2%
75-90
10591 (27 (27.2%)
Negative
29482 (75.6%)
Heart Failure (HF)
Positive
9494 (24.4%)
Negative
31402 (80.6%)
Chronic Kidney Discuse (CKD)
Positive
7574 (19.4%)
Negative
33858 (86.9%)
Chrouic Obstructive Pulmonary Disease (COPD)
Positive
5118 (13.1%)
Target. labels. For each ICU stay, we extract the target. labels for CKD, COPD, and HF based on the Inter- national Classification Diseases (ICD) 9 and 10 diagnosis codes associated with each ICU stay. For each prediction task, we label given ICU stay as positive sample if any of the following ICD diagnosis codes are associated with the stay (Note: X is a wildeard character):
CKD: 585. XX (ICD-9); N18. XXXX (ICD-10)
COPD: 491. 20, 491.21 21, 491.22 492. Ox, 492.8% 491. 1X, 491. 2X, 496 XX, 490 490.XX, 491.0X, 491.8X 491.9X (ICD-9); J44. XXXX (ICD-10)
HF: 428.XX XX (ICD-9); XXXX (ICD-10)
For the given cohort, we extract all of the measurements and events recorded during the first 24 hours and aggregate them into 4-hour bins.
B.2 Feature Selection, Model Training, and Hyperparameter Optimization
As described in Section 4.2 of the main text, we evaluate the effectiveness of each feature selection method by measuring the test performance of downstream prediction model when selecting only 30% of all input
50

---

Published in Transactions on Machine Learning Research (04/2025)
Selected Reguiarizatinn Strength
No. Features Selected Soore
(a)
income
Employment
Public Coverage
Mobility
CKO
DOPD
HF
0.92
6.75
0.75
0.85
80.03
2.8
SURCE
0.70
83.05
0.70
AUROC
0.00
2.45
20:00
23.00
2.24
0:55
3.0
1850 2050 3055
1030
2000
2000
2003
2000
2000
No. Features Selected
No. Features Selected
No. Features Selected
No. Features Selected
No. Features Selected
No. Features Selected
No. Features Selecter
Selected Regulariztion Strength
No. Features Selected Score
(b)
Income
Employment
Public Coverage
Mobility
CKD
COPD
HF
3.93
0.75
6.75
ca
0.70
2.83
2.00
2.5
suRge
10
6:70
AUROC
AUROC
800
aUROS
0.75
2.0
0.05
0:55
0.5
2000
355
1086
2000
0.50
1850
2000
3016
0.50
2018
5.5
0.50
2005
0.5
2009
No. Features Selected
No. Features
No. Foatures
No. Features Selected
No. Features Selected
No. Features Selected
No. Foatures Selectec
Figure B1: Regularization paths on the folktables and MIMIC-IV datasets with warm starts, when random seed = 1: (a) LassoNet, (b) LASSO. For each dataset, the vertical dashed line demarcates the number of features selected by LLM-SCORE based on GPT-4, and the point marked with a star indicates the chosen regularization strength that selects approximatcly the same number of active features as LLM-SCORE.
concepts. For downstream training, we minimize the binary cross-entropy loss using the Adam optimizer (Kingma & Ba, 2015) with the default hyperparameters 31 = 0.9, 82 = 0.999, and e = 10-8 For all datasets, we use importance weighting together with early stopping when training the downstream model to balance the weights of the positive and negative samples, as most of the folktables and MIMIC-IV datasets exhibit label imbalance. For all experiments, we aggregate the results over random seeds (=11,2,3,,,55 which control the train-validation splits, the hyperparameter samples generated for random-search hyperparameter optimization for the downstream model, and the initialization of the model parameters.
B.2.1 Additional Details Feature Selection Methods
LassoNet. For feature selection with LassoNet (Lemhadri et al., 2021), we use multi-layer perceptron (MLP) with 1 hidden layer and 300 hidden units and fix the hierarchy coefficient M to the recommended value of 10. For training the MLP, we minimize the binary cross-cntropy loss using the Adam optimizer (Kingma & Ba, 2015) with the default learning rate of 10-3 and the default momentun hyperparameters of B1 = 0.9, 82 = 0.999, and 6 = 10-8 Following Lemhadri et al. (2021), we compute the dense-to- sp-srre regularization paths with warm starts (Friedman al., 2010). We first train an unregularized MLP for 10 epochs, using the validation set for early stopping with a patience of 3. We then gradually increase the regularization coelficient as À = [10, 100, 1000, 1500, 2000, 2500, 3000, 3500, 4000, 4500, 5000, 5500, 6000, 6500, 6600, 6700, 6800, 6900, 7000, 7100, 7200, 7300, 7400, 7500, 8000, 8500, 9000, 9500, 10000, 10500, 11000, 11500, 12000, 12500, 13000, 13500, 14000, 14500, 15000], where for each value of X, we further train the model for epochs. We sweep through this fixed set of regularization strengths instead of using the exhaustive approach
Appendix A.2.1 for computational efficiency. After computing the regularization path, we identify the regularization strength that selects approximately the same number of features as L.MM-SCORE on each dataset, and use the active features with nonzero weights for training the downstream prediction model. Figure B1(a) shows the LassoNet regularization paths computed for all of the datasets, along with the regularization strength used for feature selection (marked with star).
LASSO. For feature selection with the LASSO (Tibshirani, 1996) with group-wise feature sparsity (Yuan & Lin, 2006), we first train an unregularized logistic regression model on each dataset. To compute the sparse-to-dense regularization paths with warm starts (Friodman et al., 2010), we then gradually increase the regularization coefficient as X [0.001, 0.005, 0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5), where for each value of X, we train new model initialized with the
51

---

Published in Transactions on Machine Learning Research (04/2025)
previously learned model parameters. After computing the regularization path, we identify the regularization strength that selects approximately the same number of features as LLM--CCRE on each dataset, and use the active features with nonzero weights for training the downstream prediction model. Figure Bi(b) shows the LASSO regularization paths computed for all of the datasets, along with the regularization strength used for ueature selection (marked with star).
B.2.2 Downstream Model Training and Hyperparameter Optimization
After selecting the features to use for training according to each feature selection method, we run a random search with asynchronous successive halving (Li et al., 2020) to select the best hyperparameter configuration to use for training and evaluation each downstream prediction model. For each downstream prediction model, we randomly sample 40 different hyperparameter configurations, and select the configuration that maximizes the validation AUROC. We then measure the performance of the selected model the lest set. We repeat this process 5 times with different random seeds (=,1,2.3,4,,5) and average the results. Below, we provide the hyperparameter search space used for each model. For any hyperparameter that is not explicitly listed below for LightGBM (Ke et al., 2017), we use the default value used in the corresponding API21.
Hyperparameter Search Space for LightGBM:
Weak Learner: Gradient-Boostel Decision Tree,
Maximum Number Weak Learners: 50,
Maximum Number Leaves Discrete( Discrete(((2,2)) 21., 60))),
Boosting Learning Rate Uniform(110 0.5),
Subsampling Ratio Uniform(0.5,. 1),
Minimum Sum Hessian in Leaf LogUniformm10 1).
Hyperparamcter Search Space for MLP:
Number of Hidden Units Discrete (200, 201,..., 500)),
Number of Hidden Layers Discrete (2, 4)),
Dropout Probability Uniform(O, 0.5),
Batch Size Discrete((256,, 512, 1024)),
Learning Rate LogUniform(110- 10 10-2).
Maximum Number of Epochs: 15.
Hyperparameter Search Space for Logistic Regression:
Learning Rate LogUniform(11-4, 10-2),
Batch Size Discrete((256,, 512, 1024)),
Ly Regularization ~ LogUniform(10 10-2),
Maximum Number of Epochs: 15.
B.3 Additional Experimental Results
In this section, we provide the full results for the large-scale dataset experiments in Section 4.2. In Figure B2, we show the feature selection paths for GPT-4-based LLM-SCORE and all of the data-driven baselines, when using (a) LightGBM, (b) MLP, and (c) I-z-eenaiiedd logistic regression for downstream prediction. Overall, we find that LLM-SCORE performs as strongly as the best-performing basclines on the folktables datascts across all downst Townsttent ream prediction ion models, and significantly better than LassoNel and the random selection baseline on the MIMIC-IV datasets.
21 21 hhtttpe://ighhtgbm. rewdthedocs.eco.
52

---

Published in Transactions on Machine Learning Research (04/2025)
LLM--ccore
LassoNet
LASSO
MRMR
CAE
HSIC-Lasso
Random
(a)
6.90
3.2
AUMOC
0.75
AUMOC
0.8
071
Fieetin Se wrtee
Touttnn Seirted
Fouttor Selented
Pacien Seferet
Daa/857 Seleceed
Tryr15in Seleceed
Frartinn Selectec
(b)
Cmp oyment
Pusl: Coveragge
CCD
C2PO
3.70
0.22
0.771
3.60
3:00
0.0
000
05 07
03 35
33
33
Scioled
Schicee
Sahocled
Sdacua
Scho.ld
Praczon Sclacied
(c)
0.00
Emp SyTTenT
Coverage
0.04
AUGGC
0.13
8.8 D17
02
OZ
0.5
07
0.8 2.7
22 53
Figure B2: Feature selection paths for all baselines and GPT-4-based LLM-SCORE (ours, in solid lines) on the folktables and MIMIC-IV datasets when using (a) LightGBB, (b) MLP, and (c) Le-penalized logistic regression for downstream prediction. Test performance is measured by AUROC (higher is better), as all datasets correspond to binary classification tasks. On most datasets, LLM--SCOEE shows strong performance comparable to those of the best-performing data-driven baselines.
53

---

Published in Transactions on Machine Learning Research (04/2025)
C Prompting for Feature Importance Scores
In this section, provide all of the prompt templates used for LLM-SCORE in Appendix C.1-C.21. In the defaull prompt template, we only include the (i) main system prompl, (ii) the output format instructions, and (iii) the main user prompt. We optionally add dataset-specific context and/or few-shot examples (with or without thouuhtf (CoT)) to the default prompt template to investigate the impact of their inclusion on the feature selection performance of LLM-SCORE, as discussed in Section of the main text. In each prompt template, mark the default components with red lags (e.g., /* Main System Prompt */) and all of the optionally added components with blue tags (/* Few-shot Examples */). We clarify that the tags not part of the actual input prompt provided to the LLMs. Any text enclosed in angled brackets (+) serves as a placeholder for value to be specified by the user. For example, "(concept)" is placeholder for the input concept for which we wish to obtain an LLM-generated importance score.
Output format instructions. In practice, we found that choosing the right output format for a given LLM is also important consistently obtain well-structared (i.e., easy to parse and extract the relevant information) and adequate response for the LLM, especially for smaller models like Llama-2 with 7B pa- rameters. We only consider the output formats available off-the-shelf from the LangChain API?2 which we used to implement most of our LLM prompting methods. Below, we provide the output format instructions used for the different LLMs:
Output format instructions for GPT-4 and GPT-3.5:
The output should be formatted as a JSON instance that conforms to the JSON schema below.
As an example, for the schema "properties": "foo": "title": "Foo", "description": "a list of strings", "type": "array", "items": "type": "string", "required": ["oo""] the object "foo": ["bar", "baz"] is well- formatted instance of the schema. The object "properties": "foo"; ["bar", "baz"] is not well-formatted.
Here is the output schema:
("description": "Langchain Pydantic output parsing structure.", "properties": {"reasoning": ("title": "Reasoning", "description": "Logical reasoning behind feature importance score", "type": "string"), "score": ("title": "Score", "description": "Feature importance score", "type": "number"]] "required": ["score"]]
Output format instructions for the Llama-2 models:
The output should be markdown code snippet formatted in the following schema, including the leading
and trailing json" and
"ssonn"
"reasoning": str // Logical reasoning behind feature imporlance score
"score": float // Feature importance score
Given that we use different output formats for different LLMs, we omit the output format instructions in Appendix C.1 C.21 for simplicity.
C.1 LLM-SCORE Template for Credit-G
/* Dataset-specific Context */
Context: Using data collected at a German bank, we wish to build a machine learning model that can accurately predict whether a client carries high low credit risk (target variable). The dataset contains total of 20 features (e.g., credit history, savings account status). Prior to training the model, first.
22
54

---

Published in Transactions on Machine Learning Research (04/2025)
want to identify subset of the 20 features that are most important for reliable prediction of the target variable.
/ Main System Prompt */
For each feature input by the user, your task to provide a feature importance score (between (min (min__scoee) and (max (max_score)); larger value indicates greater importance) for predicting whether an individual carries high credit risk and a reasoning behind how the importance score was assigned.
/* Output Format Instructions * (Omitted; refer to beginning of Appendix C)
/* Few-Shot Examples */
Here is an example output:
Variable: Installment rate in percentage of disposable income
{ "reasoning": "The installment rate as a percentage of disposable income provides insight into a person's financial responsibility and capability. This percentage can be seen as measure of how much of person's 's available income is committed repaying their debts. If this rate is high, it might indicate that the person is taking more debt than they comfortably repay and may hint at a lack of financial responsibility, implying higher credit risk. If this rate is low, it likely indicates that the person can manage their current. financial obligations comfortably, implying lower credit risk. Thus, the score is 0.9.", + C/
"score": )
1* Main User Prompt */
Provide score and reasoning for "(concept)" formatted according to the output schema above:
C.2 LLM-SCORE Template for Bank
/* Dataset-specific Context */
Context: Using data collected via a telemarketing campaign at Portuguese banking institution from 2008 to 2013, we wish to build machine learning model that can predict whether client will subscribe to a term deposit (target variable). The dataset contains a total of 16 features (c.g., age, marital status, whether the client has housing loan). Prior to training the model, we first want to identify a subset of the 16 features that are most important for reliable prediction of the target variable.
/* Main System Prompt */
For each feature input by the user, your task is to provide feature importance score (between (min__core)) and (max individual will subscribe to a term deposit and a reasoning behind how the importance score was as- signed.
(max__ccree) score); larger value indicates greater importance) for predicting whether an
/* Output Format. Instructions */ (Omitted; refer beginning of Appendix C)
/* Few-shot Examples */
Here is example output:
Variable: Has Credit in Default
{ "reasoning": "Clients with credits in default might be more hesitant to open new financial products due to their current financial situation and may be deemed higher risk by the bank. Therefore, the score 0.9.". +- CoT
CoT $/
"score": 0.9
/* Main User Prompt */ Provide a score and reasoning for "(concept)" formatted according to the output schema above:
C.3 L.M-SCORE Template for Give Some Credit
55

---

Published in Transactions on Machine Learning Research (04/2025)
/ Dataset-specific Context //
Context: We wish to build a machine learning model that can accurately predict whether an individual is likely to experience serioues financial distress in the next two years (target variable). The dataset contains a total of 10 features (e.g., debt ratio, monthly income). Prior to training the model, we first want identify a subset of the features that are most important for reliable prediction of the target variable.
/* Main System Prompt */ For cach feature input by the user, your task is to provide a feature importance score (between (min score) and (max score); larger value indicates greater importance) for predicting whether an individual is likely to experience serious financial distress in the next two years and a reasoning behind how the importance score was assigned.
/* Output Format Instructions */ (Omitted; refer beginning of Appendix C)
/* Few-shot Examples */
Here is example output:
Variable: Monthly income
{ "reasoning": "Monthly income is a crucial factor in determining an individual's financial stability. higher monthly income indicates a higher ability to meet financial obligations and reduces the likelihood of experiencing serious financial distress. A lower monthly income, on the other hand, may lead to difficulties in managing expenses and paying off debts, increasing the likelihood of paying off debts. Thus, the score is 0.8.", + CoT */
"score": 0.8 }
1* Main User Prompt */ Provide a score and reasoning for "(concept)" formatted according to the output schema above:
C.4 LLM-SCORE Template for COMPAS Recidivism
/* Dataset-specific Context */
Context: Using data from a 2016 study on the use of the Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) algorithm, we wish to build a machine learning model that can accurately predict whether a criminal defendant carries high risk for recidivism (targel variable). The individuals in the study cohort are from Broward County, Florida and were assigned COMPAS risk scores in 2013 and 2014. The datasct contains total of 13 features, including criminal history and demographics. Prior to training the model, first want to identify subset of the 13 features that are most important for reliable prediction of the target variable.
/* Main System Prompt */
For each feature input by the user, your task to provide a feature importance score (between (min__score) and (max_score)) larger value indicates greater importance) for predicting whether a criminal defendant carries high risk of recidivism and reasoning behind how the importance score was assigned.
/* Output Format Instructions */ (Omitted; refer to beginning of Appendix C)
/* Few-shot Examples */
Here is an example output:
Variable: Year of Birth
"reasoning": "The year in which defendant was born is not directly relevant. to the likelihood of recidivism, and incorporating such information for model training may introduce unwanted biases. Therefore, the score is 0.1.", + / CoT */
"score": 0.1
56

---

Published in Transactions on Machine Learning Research (04/2025)
/* Main User Prompt */
Provide score and reasoning for "(concept)" formatted according to the output schema above:
C.5 LLM-SCORE Template for Pima Indians Diabetes
/* Dataset-specific Context */
Context: We wish to build machine learning model that accurately predict whether patient has diabetes (target variable), given several diagnostic measurements. The selected individuals in the cohort are female patients of Pima Indian heritage who are at least 21 years old. We mcasured a total of 8 clinical features (e.g. blood pressure, insulin). Prior to training the model, we first want to identify a subset of the 8 features that are most clinically important for reliable prediction the target variable.
1 Main System Prompt */
For each feature imput by the user, your lask is to provide feature importance score (belween (min__scree) and (max_sccre)); larger value indicates greater importance) for predicting whether patient has diabetes and reasoning behind how the importance score was assigned.
/* Output Format Instructions */ (Omitted; refer to beginning of Appendix C)
/* Few-shot Examples */
Here is an example output:
Variable: Year of Birth
{ "reasoning": "The year in which defendant was born is not directly relevant to the likelihood of recidivism, and incorporating such information for model training may introduce unwanted biases. Therefore, the score is 0.1.", / CoT */
"score": 0.1
1 Main User Prompt */
Provide score and reasoning for "(concept)" formatted according to the output schema above:
C.6 LLM-SCORE Template for AUS Cars
/* Dataset-specific Context */
Context: Using information about prices in Australia during the year 2023, wish to build machine learning model that can predict the selling price of car in Australia (target variable). The dataset contains a total features (e.g., fuel type, age). Prior to training the model, we first want to identify a subset of the 7 features that are most important for reliable prediction of the target variable.
/* Main System Prompt */ For each feature input by the user, your task is to provide feature importance score (between (min__scome and (max_ _ooos_sscre;;; larger value indicates greater importance) for predicting predicting the selling price of car in Australia and reasoning behind how the importance score assigned.
/* Output Format Instructions */ (Omitted; refer to beginning of Appendix C)
/* Few-shot Examples *
Here is an example output:
Number of doors in the car { "reasoning": "The number of doors is not reliable predictor for the selling price of car, both cheap and expensive cars can have a similar number of doors (typically two or four). Therefore, the score is 0.1.", + CoT $/
Variable:
"score": 0.1
1* Main User Prompt */
Provide score and reasoning for "(concept)" formatted according the output schema above:
57

---

Published in Transactions on Machine Learning Research (04/2025)
C.7 LLM-SCORE Template for YouTube*
/ Dataset-specific Context */
Context: Using various statistics about some of the most popular YouTube channels from 2023, we wish to build machine learning model that can accurately predict whether a YouTube channel has more than million subscribers (target variable). The dataset contains a total of features (e-g., total number of views, channel category). Prior to training the model, we first want to identify a subset of the 22 features that are most important for reliable prediction of the larget variable.
/ Main System Prompt */
For cach feature input by the user, your task is to provide a feature importance score (between (min score) and (max score); larger value indicates greater importance) for predicting whether a YouTube channel has more than 20 million subscribers and a reasoning behind how the importance score was assignod.
/* Output Format. Instructions */ (Omitted; refer to beginning of Appendix C)
/* Few-shot Examples */
Here is an example output:
Variable: Month when the YouTube channel was created
"The month when the YouTube channel was created does not directly affect how many subscribers the channel has. Thus, the score is 0.1.", +- CoT */
{ "reasoning":
"score": 0
/* Main User Prompt */
Provide score and reasoning for "(concepty" formatted according to the output schema above:
C.8 LLM-SCORE Template for CA Housing
/* Dataset-specific Context */
Context: Using 1990 U.S. Census data, wish to build machine learning model that can predict the median housing price for each block group in the census (target variable). A block group, typically with a population of 600 to 3,000 people, is the smallest geographical unit for which the U.S. Census Bureau publishes sample data. The dataset contains a total of 8 numerical features (e.g., average number of rooms for houses in the block group, latitude and longitude the block group), and cach sample in the dataset corresponds a block group. Prior to training the model, we first. want to identify subset the features that are most important for reliable prediction of the target variable.
1* Main System Prompt */ For each feature input by the user, your task is to provide feature importance score (between (min_score) and (max scoree; larger value indicates greater importance) for predicting the median housing price of a U.S. census block group and a reasoning behind how the importance score was assigned.
/* Output Format Instructions */ (Omitted; refer beginning of Appendix C)
/* Few-shot Examples */
Here is example output:
Variable: Median Income in U.S. Census Block Group
{ "reasoning": *Median income is often directly correlated with the standard of living in an area, which can be correlated with housing prices. Areas with higher median incomes might have more expensive houses bocause the residents can afford to pay more for housing, either because of better infrastructure, schools, other amenities. Additionally, homeowners in areas with higher median incomes may invest. more on home improvement, which could raise the median housing price. Thus, the score is 0.9.", +
/* CoT +/ "score"; 0.9 }
58

---

Published in Transactions on Machine Learning Research (04/2025)
/* Main User Prompt *
Provide a score and reasoning for "(concept)" formatted according to the output schema above:
C.9 LLM-SCORE Template for Diabetes Progression
/* Datased-specific Context */
Context: We wish to build a machine learning model that can accurately predict disease progression diabetic patients (target variable), given their baseline blood scrum mcasurements from the previous year and demographics. We measured a total level, age, sex). Prior to training the model, we first want identify subset of the 10 features that most clinically important for reliable prediction of the target variable.
clinical features (e.g., blood sugar level, cholesterol
/* Main System Prompt */
For each feature input by the user, your task is to provide feature importance score (between (min score) and (max score); larger value indicates greater importance) for predicting the disease progression status in diabetes patients and a reasoning behind how the importance score was assigned.
/* Output Format Instructions */ (Omitted; refer beginning of Appendix C)
/* Few-shot Examples */
Here is example output:
Variable: Body Mass Index (BMI)
{ "reasoning": "BMI is a widely used measure to classify individuals based on their weight, relative to their height. is a proxy for body fatness and can give insights into whether a person has a healthy weight, is underweight, overweight, or obese. Obesity a significant risk factor for type diabetes, as higher amounts of body fat can lead to insulin resistance and thereby increase the risk of developing diabetes. Thus, while BMI should be considered in conjunction with other clinical measurements (e.g., blood sugar levels, blood pressure) for comprehensive assessment of diabetic risk, the score is 0.8, given its strong association.", "score"; 0.8
CoT */
1* Main User Prompt */
Provide a score and reasoning for "cconcept)" formatted according to the output schema. above:
C.10 LLM-Score Template for Miami Housing
/* Dataset-specific Context */
Context: Using data collected on 13,932 single-family homes sold in Miami in 2016, we wish to build a machine learning model that can predict the selling price of each house (target variable). The dataset contains total of 15 features, which include structural (e.g., area, structure quality) and geographic (e.g,, longitude, latitude) information about each house. Prior training the model, we first want to identify subset of the 15 features that are most important for reliable prediction of the target variable.
/* Main System Prompt */ For each feature input by the user, your task is to provide a feature importance score (between (min__score) and (max_ soax__score; larger value indicates greater importance) for predicting the selling price of homes in Miami and reasoning behind how the importance score was assigned.
/* Output Format Instructions */ (Omitted; refer to beginning of Appendix C)
/* Few-shot Examples */
Here is an example output:
Variable: sale month in 2016 (1 =1= january)
{ "reasoning": "While the real estate market may be subject to seasonal fluctuations, the month during which house was sold not directly indicative of its selling price. Thus, the score 0.25.", /*
59

---

Published in Transactions on Machine Learning Research (04/2025)
CoT */ "score": 0.25 }
/ Main User Prompt */
Provide a score and reasoning for "(concept)" formatted according to the output schema above:
C.11 LLM-SCORE Template for Wine Quality
/* Dataset-specific Context */
Context: Using various physicochemical measurements made on red and white vinho verde wine samples from northern Portugal, wish to build a machine learning model that can accurately predict whether wine is high or low quality (larget variable). The dataset contains total of 11 features (e.g., acidily, density, sugar content). Prior to training the model, we first want to identify a subset of the 11 features that are most important for reliable prediction of the target variable.
/ Main System Prompt */
For each feature input by the user, your task to provide a feature importance score (between (min__ccre)) and (max__ccrrg) score); larger value indicates greater importance) for prodicting whether a wine is high or low quality and reasoning behind how the importance score was assigned.
/* Output Format Instructions */ (Omitted; refer to beginning of Appendix C)
/* Few-shot Examples */
Here is an example output:
Variable:
Fixed acidity (g(tartaric acid)/dm²)
{ "reasoning": "Fixed acidity refers the concentration of non-volatile acids present in the wine, primarily tartaric acid. Along with pH, fixed acidity can significantly influence the taste and feel of wine in the mouth. A wine that is too acidic will taste sour and sharp, while wine with low acidity taste flat and lifeless. Although it is the balance of fixed acidity with other components of the wine that ultimately determines the wine quality, too much or too little acidity can be detrimental. Thus, the score is 0.8.", /* CoT "score": 0.8
1* Main User Prompt */
Provide score and reasoning for "(concept)" formatted according the output schema above:
C.12 LLM-SCORE: Template for Used Cars
/* Dataset-specific Context */
Context: We wish to build a machine learning model that can accurately predict the selling price of a used car (target variable). We recorded total of features (e.g., age, number of previous owners, fuel type). Prior to training the model, first want to identify subset of the 7 features that most important for reliable prediction of the target variable.
y* Main System Prompt */ For each feature input by the user, your task to provide a feature importance score (between (min score) and (max score); larger value indicates greater importance) for predicting the selling price of used car and a reasoning behind how the importance score was assigned.
/* Output Format Instructions */ (Omitted; refer to beginning of Appendix C)
/* Few-shot Examples #/ Here is an example output:
Variable: Selling type (dealer (dealer/individus) /individual)
"reasoning": "The selling type can have an impact on the price of a used car, as dealers often have overhead costs and can offer warranties or other serivces that can increase the price. However,
60

---

Published in Transactions on Machine Learning Research (04/2025)
factors such as the age of the car the mileage generally more influential in determining the price. Therefore, the score is 0.2.", + / CoT */
"score": 0.2
1* Main User Prompt */
Provide a score and reasoning for "(concept)" formatted according the output schema above:
C.13 LLM-SCORE Template for NBA*
/* Dataset-specific Context */
Context: Using various in-game statistics mcasured for an NBA basketball player during the 2023-2022 202- season, we wish to build machine learning model that can accurately predict the player's number of points per game (target variable). The dataset contains total of 27 features (e.g., steals per game, assists per game). Prior to training the model, first want to identify a subset of the 27 features that are most important for reliable prediction of the target variable.
/ Main System Prompt */
For each feature imput by the user, your task to provide a feature importance score (between (min__ccre)) and (max_sste __sorsss; larger value indicates greater importance) for predicting the number of points per game of an NBA basketball player and reasoning behind how the importance score was assigned.
/* Output Format Instructions */ (Omitted; refer to beginning of Appendix C)
/* Few-shot Examples */
Here is an example output:
Variable:
Position
{ "rcasoning": "The position of basketball player is not reliable predictor of the average number of points per game, there players who average high number of points per game across all positions. Thus, the score is 0.2.", < -/+ CoT #/ "score": 0.2
1* Main User Prompt */
Provide a score and reasoning for "(concept)" formatted according the output schema above:
C.14 LLM-SCORE Template for NYC Rideshare*
/* Datasct-spocific Context */
Context: Using various details from a rideshare trip, we wish to build a machine learning model that can accurately predict the total pay given to the rideshare driver from that trip (target variable). The dataset contains a total 15 features (e.g., time day, trip duration). Prior training the model, we first want to identify subset of the 15 features that most important for reliable prediction of the larget variable.
/* Main System Prompt */ For each feature input by the user, your task is to provide a feature importance score (between (min score) and (max _score;; larger value indicates greater importance) for predicting the total pay given to a rideshare driver from a trip and a reasoning behind how the importance score was assigned.
/* Output Format Instructions */ (Omitted; refer beginning of Appendix C)
/* Few-shot Examples #/
Here is example output:
Variable: Vehicle-for-hire company operating the ride (Uber/Lyft
"reasoning": "The particular company that a rideshare driver is hired by may affect the total pay given to the driver, but the difference in pay across different companies is likely to be negligible. It
61

---

Published in Transactions on Machine Learning Research (04/2025)
therefore not strong predictor in and of itself. Thus, the score is 0.2.,, t / CoT*/
"score": 0.2
/ Main User Prompt */
Provide a score and reasoning for "(concept)" formatted according to the output schema above:
C.15 LLM-SCORE Template for Income
/* Dataset-specific Context */ Context: Using the American Community Survey (ACS) Public Use Microdata Sample (PUMS) data collected by the U.S. Census Bureau, we wish to build a machine learning model that can accurately predict whether an individual has an income greater than $50,000 (target variable). The individuals in the selected cohort are of ages above 16, have worked at least hour per week in the past year, have an income of at least $100, and a PUMS person weight of at least 1. The dataset contains a total of 281 features (e.g,, age, workclass, health insurance plan). Prior to training the model, we first want to identify a subset of the 281 features that are most important for reliable prediction of the target variable.
/* Main System Prompt */ For each feature imput by the user, your task is to provide feature importance score (between and (max_ son__score); larger value indicates greater importance) for predicting whether an individual's income is greater than $50,000 and reasoning behind how the importance score was assigned.
/* Output Format Instructions */ (Omitted; refer to beginning of Appendix C)
/* Few-shot Examples */
Here are example outputs:
Variable: Class of worker (COW)
{ "reasoning"; "The type or class of work (e.g., government employee, self-employed, unemployed, for- profit. company employce) that an individual is engaged can be directly linked to their income. For instance, an individual who is unemployed will have close to no income earned, significantly decreasing the likelihood that an individual earns more than $50,000 income. On the other hand, an individual who is employed at for-profit. company in the technology industry will be more likely to make more than $50,000 in income than unemployed individual. Thus, while it should be considered in conjunction with other features to avoid any unfair and biased predictions, the score is 0.9.", / CoT #/ "score": 0.9 }
Variable: Person's Weight replicate 78 (PWGTP78)
{ "reasoning": *PWGTP78 refers the 78th replicate PUMS weight for an individual, used accu- rately calculating the variance in ACS PUMS estimates. These weights are not directly related an individual's income socioeconomic standing, and are more about ensuring the reliability and robust- ness of estimates derived from the survey sample. Thus, it is unlikely that this feature has direct or meaningful influence on predicting whether an individual earns more than 850,000. The score is 0.1.", + + CoT #/ "score": 0.1
Variable: Income-to-pont poverty ratio recode (POVPIP)
{ ""reasoning:: "The income-to-poverty ratio is measure that compares individual's or household's income to the poverty (hreshold sel. for their respective size and composition. This ralio offers straightforward understanding of person's financial situation relative to the poverty line. An individual with ratio significantly above has an income that surpasses the poverty threshold by considerable margin, which can indicate a higher likelihood of having an income above $50,000. Conversely, an near or below the poverty level, making it less probable for them to earn more than $50,000. Given its direct correlation to income levels, the income- to-eeverty ratio recode is a strong predictor of whether an individual carns more than $50,000. Thus, the score
individual with a ratio close to or below
62

---

Published in Transactions on Machine Learning Research (04/2025)
0.95.", + / CoT /
"score": 0.95 }
/ Main User Prompt */
Provide a score and reasoning for " "(concept)" formatted according to the output schema above:
C.16 LLM-SCORE Template for Employment
/ Dataset-specific Context */
Context: Using the American Community Survey (ACS) Public Use Microdata Sample (PCMS) data collected by the U.S. Census Bureau, wish to build machine learning model that accurately predict whether an individual is employed (target variable). The individuals in the selected cohort are of ages between 16 and 90 and have PUMS person weight of at least 1. The dataset contains a total of 281 features (e.g., age, workelass, health insurance plan). As an imitial step prior to training the model, we first want to identify a subset of the 281 features that are most important for reliable prediction the larget variable.
/ Main System Prompt */ For cach feature input by the user, your task is to provide feature importance score (between (min score) and (max score); larger value indicates greater importance) for predicting employment status and reasoning behind how the importance score was assigned.
/ Output Format Instructions */ (Omitted; refer to beginning of Appendix C)
/* Few-shot Examples */
Here are some example outputs:
Variable: Marital Status (MAR)
{ "reasoning": "Marital status can have some indirect implications for employment status. For instance, in houscholds with a single income carner, one partner might choose not to work. However, marital status on its own is not. a strong predictor of employment. Numerous unmarried individuals work, and many married individuals might be unemployed. Thus, while there is mild correlation, marital status is not a direct indicator of employment status. Hence, the score is 0.3.", + CoT / "score": 0.3)
Variable: Person's Weight replicate 78 (PWGTP78)
{ "reasoning": "PWGTP78 refers to the 78th replicate PUMS weight for an individual, used in accu- rately calculating the variance in the ACS PUMS estimates. These replicate weights do not inherently contain information about an individual's employment status, and their primary role to help ensure the reliability and robustness of estimates derived from the survey sample. Therefore, the score 0.1.", + CoT / "score": 0
ratio recode (POVPIP) { "reasoning": "The income-to-poverty ratio is measure that compares individual's or household's income to the poverty threshold set for their respective size and composition. This ratio offers a straightforward understanding of person's financial situation relative to the poverty line. An individual with a ratio siguificantly above has an income that surpasses the poverly threshold by a considerable margin, which can potentially hint employment or other sources of income. Conversely, an individual with low ratio may be strugging from financial difficulties, possibly due to unemployment. However, there are exceptions. For example, individual may have low income-to-poverty ratio but still be employed. Thus, while the ratio may have strong correlation with employment status, it is not a definitive predictor of employment status. So the score is 0.7.", + CoT #/ "score": 0.7
Variable:
Income-to-pon
1* Main User Prompt */
Provide score and reasoning for "(concept)" formatted according the output schema above:
63

---

Published in Transactions on Machine Learning Research (04/2025)
C.17 LLM-SCORE Template for Public Coverage
/ Datasct-specific Context */
Context: Using the American Community Survey (ACS) Public Use Microdata Sample (PUMS) data collected by the U.S. Census Bureau, we wish to build a machine learning model that can accurately predict whether a low-income individual has coverage from public health insurance (target variable). The individuals the selected cohort are of ages below 65 (not cligible for Medicare) and have a total income less than $30,000. The dataset contains total of 281 features (e.g., age, workclass, employment status). Prior to training the model, we first want to identify subset of the 281 features that are most important for reliable prediction of the target variable.
/ Main System Prompt */
For each feature input by the user, your task is to provide a feature importance score (between (min score) and (max score); larger value indicates greater importance) for predicting whether a low- income individual has coverage from public health insurance and a reasoning behind how the importance score was assigned.
/* Output Format Instructions */ (Omitted; refer to beginning of Appendix C)
/* Few-shot Examples */
Here are some example outputs:
Variable: Marital Status (MAR)
{ "reasoning": "Marital status can be associated with various socioeconomic factors, including the likelihood of having access to hcalth insurance. For example, individuals who married might have access to health insurance through their spouse's employer. Furthermore, certain public health insurance programs might consider household size and income, which can be indirectly related to martial status. Thus, marital status might carry some information about an individual's likelihood of having public health insurance coverage. Therefore, the score is 0.65.", /* CoT "score": 0.65 }
Variable: Person's Weight replicate 78 (PWGTP78)
{ "reasoning": "PWGTP78 refers to the 78th replicate PUMS weight for individual, used in calcu- lating accurate variance estimates for ACS PUMS estimates. These replicate weights do not inherently contain information about an individual's health insurance status, and their primary role is to help ensure the reliability and robustness of estimates derived from the survey sample. Therefore, the score is 0.1.,, + CoT*/
"score": )
Variable:
Income-to-poverly ratio recode (POVPIP)
{ "reassnnnn": "The incoee-to-poverty ratio is measure that compares individual's or household's income to the poverty threshold set for their respective size and composition. This ratio offers straightforward understanding of person's 's financial situation relative to the poverty line. An individual with a ratio close to or below has an income that is near the poverty threshold, which can directly affect his her eligibility for public health insurance programs. Therefore, the income to-poert--oovryy ratio recode a strong predictor of whether a low-income individual below the age may have public health insurance. So, the score 0.9.". / CoT // "score": 0.9
/* Main User Prompt */
Provide a score and reasoning for "(concept)" formatted according to the output schema above:
C.18 LLM-SCORE Template for Mobility
/* Dataset-specific Context */
Context: Using the American Community Survey (ACS) Public Use Microdata Sample (PUMS) data collected by the U.S. Census Bureau, we wish to build a machine learning model that can accurately
64

---

Published in Transactions on Machine Learning Research (04/2025)
predict whether young-adult individual moved residential addresses in the past year. The individuals in the selected cohort are of ages between 18 and 35. The dataset contains total of 281 features (e.g., age, workclass, health insurance plan). Prior to training the model, we first want to identify a subset the 281 features that are most important for reliable prediction of the target variable.
1 Main System Prompt */
For each feature imput by the user, your task is to provide feature importance score (between (min (min__ccre)) and (max__scree;; larger value indicates greater importance) predicting whether young- adult individual moved residential addresses in the past year and a reasoning behind how the importance score was assigned.
/* Output Format Instrictions */ (Omilted; refer to beginning of Appendix C)
/* Few-shot Examples */
Here are some example outputs:
Variable: Marital Status (MAR)
{ "reasoning": "Marital status can be indicator of stability and lifestyle changes, which may be associated with mobility patterns. For instance, individuals who get married might be more inclined to move to a new residence (e.g., buying a house together). Conversely, those who experience a divorce or separation might also decide to move. Hence, marital status can be conceptually relevant in predicting whether a young adult moved in the last year. However, being married or not may not necessarily indicate that the individual moved addresses precisely during the past 12 months. Thus, the score 0.4, accounting for the moderate association between marital status and mobility status.", +- CoT
*/
"score": 0.4 }
Variable: Person's Weight replicate 78 (PWGTP78)
{ "reasoning": "PWGTP7S refers to the 78th replicate PUMS weight for an individual, used in calcu- lating accurate variance estimates for ACS PUMS estimates. These replicate weights do not inherently contain information about an individual's mobility status, and their primary role is help ensure the reliability and robustness of estimates derived from the survey sample. Therefore, the score is 0.1.", <
/* CoT #/
"score": 0.1 }
Variable: Divorced in the past 12 months (MARHD)
{ "reasoning": "Being divorced the past. 12 months can have significant. impact on an individual's living situation and mobility status. A recent divorce can necessitate a change residence for one both parties, due to the division of assets, emotional reasons, or seeking fresh start. Given the life-changing nature of a divorce and its potential implications housing needs and preferences, this variable can be considered directly relevant in predicting whether young adult has moved in the last year. Therefore, the score is 0.9.", / CoT */ "score": 0.9
|* Main User Prompt */
Provide a score and reasoning for "(concept)" formatted according the output schema above:
C.19 L.M-SCORE: Template for CKD
/* Dataset-specific Context */ Context:
Using retrospective ronic health record time-series data, wish to build machine learn- ing model that accurately predict whether patient in the intensive care unit (ICU) will develop chronic kidney discase (CKD) (target variable). We extracted a total of 148 clinical features, which include lab test results (e.g., creatinine, lactate levels in the blood), bedside physiological measurements (e.g., heart rate, arterial blood pressure), murse assessments (e.g., Braden scale, pain assessment), pa- tient medication information (e.g., vasopressor administration), and demographics (e.g., ethnicity, sex), recorded during the first 24 hours of each patient's stay in the ICU. Prior training the model, we first
65

---

Published in Transactions on Machine Learning Research (04/2025)
want to identify subset of the 148 features that are most clinically important for reliable prediction of the target variable.
/ Main System Prompt */
For each feature input by the user, your task to provide a feature importance score (between (min sccrrs) and (max score);
(max_sccree); larger value indicates greater importance) predicting the risk of an ICU patient developing chronic kidney disease (CKD) and reasoning behind how the importance score was assigned.
/* Output Format. Instructions */ (Omitted; refer to beginning of Appendix C)
/* Few-shot Examples */
Here are some example outputs:
Variable: Glasgow Coma Scale (GCS) Verbal
"reasoning": "The verbal component of the Glasgow coma scale (GCS) is used assess the extent of patient's impaired consciousness based the verbal responses of the patient. Patients with low GCS verbal scores often suffer from significant neurological dysfunction or impairment, which can be indicative of severe injury to the brain. While a significantly low GCS verbal score may indicate that a patient more eritically ill, potentially at risk for multi-organ dysfunction, the verbal component of the GCS is not directly related to a patient's risk of developing CKD. Therefore, the score is 0.2, reflecting a slight relevance due to its indirect ability to gauge overall patient severity but not being directly relevant CKD.", + + CoT */ "score": 0.2)
Variable:
Admission Height
{ "reasoning": "Admission height refers to the height of a patient measured upon admission to the ICU. As a patient's 's height is not directly indicative of the nature and severity of patient's medical condition, it is irrelevant to a patient's 's risk of developing CKD during their stay in the ICU. Therefore, the score is 0.1.", + CoT #/ "score": 0.1 }
Variable: Epinephrine
{ "reasoning": "F.innephrine is vasopressor, which is a drug that induces vasoconstriction to elevate a patient's blood pressure when it is so low that not enough blood is being delivered to the patient's organs. is often used in critical care settings to manage severe cases such as septic shock, cardiac arrest, and refractory hypotension. The constriction of the blood vessels can temporarily reduce blood flow the kidney, and reduced kidney perfusion, especially if prolonged, can contribute kidney injury, which, in burn, can predispose patient's 's risk of developing CKD. However, the administration of epinephrine in and of itself is not directly related to patient's risk for CKD, and other measurements such as creatinine levels in the blood are more directly relevant. Therefore, the score is 0.55, indicating moderate relevance.", + + CoT // "score": 0.55 }
|* Main User Prompt */ Provide a score and reasoning for "(concept)" formatted according the output schema above:
C.20 LLM-SCORE Template for COPD
/* Dataset-speeific Context */
Using retrospective electronic health record time-series data, we wish to build a machine learning model that can accurately predict whether patient in the intensive care unit (ICU) will develop chronic obstructive pulmonary discase (COPD) (target variable). We extracted a total of 148 clinical features, which include lab test results (c.g., creatinine, lactate levels in the blood), bedside physiological measurements (e.g., heart rate, arterial blood pressure), nurse assessments (e.g., Braden scale, pain assessment), patient medication information (e.g., vasopressor administration), and demographics (e.g., cthnicity, sex), recorded during the first 24 hours of each patient's 's stay in the ICU. Prior to training
Context:
66

---

Published in Transactions on Machine Learning Research (04/2025)
the model, we first want to identify subset of the 148 features that are most clinically important for reliable prediction of the target variable.
/ Main System Prompt */
For each feature input by the user, your task to provide a feature importance score (between (max_scoree); (min _scossscorr))) and (max score); larger value indicates greater importance) predicting the risk of an ICU patient developing chronic obstructive pulmonary disease (COPD) and reasoning behind how the importance score was assigned.
/* Output Format. Instructions */ (Omitted; refer to beginning of Appendix C)
/* Few-shot Examples */
Here are some example outputs:
Variable: Glasgow Coma Scale (GCS) Verbal
"reasoning": "The verbal component of the Glasgow coma scale (GCS) is used assess the extent of patient's impaired consciousness based the verbal responses of the patient. Patients with low GCS verbal scores often suffer from significant neurological dysfunction or impairment, which can be indicative of severe injury to the brain. While a significantly low GCS verbal score may indicate that a patient is more eritically ill, potentially at risk for multi-organ dysfunction, the verbal component of the GOS is not directly related to the health of the respiratory system and patient's risk of developing COPD. Therefore, the score is 0.2, reflecting slight relevance due to its indirect ability to gauge overall CoT
patient severity but not being directly relevant COPD." CCT "score": 0.2
Variable:
Admission Height
{ "reasoning": " "Admission height refers to the height of a patient measured upon admission to the ICU, As a patient's height is not directly indicative of the nature and severity of patient's medical condition, it is irrelevant to a patient's risk developing COPD during their stay the ICU. Therefore, the score 0.1.", +- / CoT $/
"score": 0.1
Variable: Lung Compliance
{ "reasoning": "Lang compliance is quantitative measure of lung expandability. Patients with COPD suffer from a loss of elastic recoil in the lungs, which leads to an increase in lung compliance and manifests symptoms such as shortness of breath due to an inability to expell air effectively. Therefore, lung compliance can be highly indicative of a patient's risk for COPD, and the score is 0.9.", + / CoT "score": 0.9
1* Main User Prompt */
Provide score and reasoning for "(concept)" formatted according to the output schema. above:
C.21 LLM-SCORE Template for HF
/* Datased-specific Context */
Context: Using retrospective electronic health record time-scries data, we wish to build a machine learning model that can accurately predict whether a patient in the intensive care unit (ICU) will develop heart failure (HF) (target variable). We extracted a total of 148 clinical features, which include lab test results (e-g., creatinine, lactate levels in the blood), bodside physiological measurements (o.g., heart rate, arterial blood pressure), nurse assessments (e.g., Braden scale, pain assessment), patient medication information (e.g,, vasopressor administration), and demographics (e-g., ethnicity, sex), recorded during the first 24 hours of each patient's 's stay in the ICU, Prior to training the model, first want to identify a subset of the 148 features that are most clinically important for reliable prediction of the target variable.
|* Main System Prompt */
For cach feature input by the user, your task to provide a feature importance score (between
67

---

Published in Transactions on Machine Learning Research (04/2025)
(min__scre)) and (max_ssooee larger value indicates greater importance) predicting the risk of an ICU patient developing heart failure (HF) and reasoning behind how the importance score was assigned.
/* Output Format Instructions */ (Omitted; refer to beginning of Appendix C)
/* Few-shot Examples */
Here are some example outputs:
Variable: Glasgow Coma Scale (GCS) Verbal
{ "reasoning": "The verbal component of the Glasgow coma scale (GCS) is used assess the extent of a paticnt's impaired consciousness based on the verbal responses of the patient. Patients with low GCS verbal scores often suffer from significant neurological dysfunction or impairment, which can be indicative of severe brain injury. While lower GCS verbal scores may not be directly correlated with a higher risk of developing heart failure, heart failure may occur as a sccondary event due significant neurological injury or physiological stress induced by brain injury. Thus, given that the verbal component. of the GCS is not primary indicator of cardiovascular health but may have indirect. relevance, the score is 0.4., + /* CoT */ "score": 0.4
Variable: Admission Height
{ "reasoning": "Admission height refers to the height a patient measured upon admission to the ICU. As a patient's height not directly indicative of the nature and severity of patient's 's medical condition, it is irrelevant to patient's risk of heart failure during their stay in the ICU, Therefore, the score is 0.1.", + CoT CoT */
"score": 0.1
Variable: Epinephrine
{ "reasoning": "Epinephrine is a vasopressor, which is a drug that induces vasoconstriction to elevate a patient's blood pressure when it is so low that not enough blood is being delivered to the patient's s organs. is often used critical care settings to manage severe cases such as septic shock, cardiac arrest, and refractory hypotension. Patients with prolonged administration of epinephrine often have compromised cardiovascular function, which can be associated with an increased risk of heart failure. Therefore, the score is 0.9.", + CoT "score"; 0.9
1* Main User Prompt */
Provide a score and reasoning for "(concept)" formatted according the output schema above:
68

---

Published in Transactions on Machine Learning Research (04/2025)
D Prompting for Feature Rankings
In this section, we provide all of the prompt templates used for LLM-RANK in Appendix D.1-D.14. For LLM-RANK, we only use the prompt template that contains (i) main system prompt, (ii) the output format instructions, and (iii) the main user prompt, as discussed at the beginning of Section 4 of the main text. each prompt template, we mark each of the components with a red tag (e.g., /* Main System Prompt */). We clarify that the tags are not part of the actual input prompt provided to the LLMs. Any text enclosed in angled brackets {-) serves
placeholder for value to be specified by the user. For example, "(concepts)" is placcholder for the list of all input concepts for which wish to obtaín an LLM-generated ranking.
Output format instructions. For LLM-RANK, include the output format ructions directly in the prompt templates, as we use the same instructions for all LLMs (unlike LLM-SCORE).
D.1 LLM-RANK Template for Credit-G
/* Main System Prompt */
Given a list of features, rank them according to their importances in predicting whether an individual carries high credit risk. The ranking should be in desconding order, starting with the most important feature.
/* Output Format Instructions */
Your response should be numbered list with each item new line. For example: 1. foo 2. bar 3. baz
Only output the ranking. Do not output dialogue or explanations for the ranking. Do not exclude any features in the ranking.
1 Main User Prompt */ Rank all (number concepts) features the following list: "(concepts)".
D.2 LLM-RANK Template for Bank
/* Main System Prompt */
Given a list of features, rank them according to their importances in predicting whether an individual will subscribe to term deposit. The ranking should be in descending order, starting with the most important feature.
/* Output Format Instructions */
Your response should be numbered list with each item on new line. For example: 1. foo 2. bar 3. baz
Only output the ranking. Do not output dialogue or explanations for the ranking. Do not exclude any features the ranking.
/* Main User Prompt */ Rank all (number of concepts) features in the following list: "(concepts)".
D.3 LLM-RANK Template for Give Me Some Credit
/* Main System Prompt */
Given a list of features, rank them according their importances in predicting whether an individual likely to experience serious financial distress in the next two years. The ranking should be in descending order, starting with the most important feature.
/* Output Format Instructions */
Your response should be a numbered list with each item on a new line. For example: 1. foo 2. bar 3.
baz
69

---

Published in Transactions on Machine Learning Research (04/2025)
Only output the ranking. Do not output dialogue explanations for the ranking. Do not exclude any features in the ranking.
1 Main User Prompt */
Rank all (number concepts) features the following list: " "(concepts)".
D.4 LLM-RANK Template for COMPAS Recidivism
1 Main System Prompt */
Given a list. of features, rank them according their importances in predicting whether a criminal defendant carries high risk of recidivism. The ranking should be in descending order, starting with the most important feature.
/ Output Format Instructions */
Your response should be numbered list with each item on new line. For example: 1. foo 2. bar 3. baz
Only output the ranking. Do not output dialogue or explanations for the ranking. Do not exclude any features in the ranking.
/* Main User Prompt */ Rank all (number of concepts) features in the following list: "(concepts)".
D.5 LLM-RANK Template for Pima Indians Diabetes
/* Main System Prompt */
Given list of features, rank them according to their importances in predicting whether a patient has diabetes. The ranking should be in descending order, starting with the most important feature.
/* Output Format Instructions */
Your response should be a numbered list with each item on a new line. For example: 1. foo 2. bar 3. baz
Only output the ranking. Do not output dialogue explanations for the ranking. Do not exclude any features in the ranking.
/ Main User Prompt */ Rank all (number of concepts) features in the following list: "(concepts)".
D.6 LLM-RANK Template for AUS Cars*
/* Main System Prompt */
Given list of features, rank them according to their importances in predicting the selling price of car in Australia. The ranking should be in descending order, starting with the most important feature.
/* Output Format Instructions #/
Your response should be a numbered list with each item on a new line. For example: 1. foo 2. bar 3. baz
Only output the ranking. Do not output dialogue explanations for the ranking. Do not exclude any features in the ranking.
/ Main User Prompt * Rank all (number of concepts) features in the following list: "(concepts)".
D.7 LLM-RANK Template for *
70

---

Published in Transactions on Machine Learning Research (04/2025)
/ Main System Prompt */
Given list of features, rank them according to their importances in predicting whether a YouTube channel has more than 20 million subscribers. The ranking should be in descending order, starting with the most important feature.
/* Output Format Instructions */
Your response should be numbered list with each item on line. For example: 1. foo 2. bar 3. baz
Only output the ranking. Do not output dialogue or explanations for the ranking. Do not exclude any features in the ranking.
1 Main User Prompt */
Rank all (number concepts) features in the following list: "(concepts)".
D.8 LLM-RANK Template for CA Housing
/ Main System Prompt */
Given list of features, rank them according to their importances predicting the median housing price of U.S. census block group. The ranking should be in descending order, starting with the most important feature.
/* Output Format Instructions */
Your response should be numbered list with each item on new line. For example: 1. foo 2. bar 3. baz
Only output the ranking. Do not output dialogue or explanations for the ranking. Do not exclude any features the ranking.
/* Main User Prompt */ Rank all (number of concepts) features in the following list: "(concepts)".
D.9 LLM-RANK Template for Diabetes Progression
/* Main System Prompt */
Given a list of features, rank them according their importances in predicting the disease progression status in diabetes patients. The ranking should be in descending order, starting with the most important feature.
/* Output Format Instructions */
Your response should be numbered list with each item on new line. For example: 1. foo 2. bar 3. baz
Only output the ranking. Do not output dialogue or explanations for the ranking. Do not exclude any features in the ranking.
/* Main User Prompt */ Rank all (number of concepts) features in the following list: "(concepts)".
D.10 LLM-RANK Template for Wine Quality
/* Main System Prompt */
Given list of features, rank them according to their importances in predicting whether wine is high low quality. The ranking should be in descending order, starting with the most important feature.
/* Output Format Instructions */
Your response should be a numbered list with each item on a new line. For example: 1. foo 2. bar 3. baz
71

---

Published in Transactions on Machine Learning Research (04/2025)
Only output the ranking. Do not output dialogue explanations for the ranking. Do not exclude any features in the ranking.
/ Main User Prompt */
Rank all (number concepts) features the following list: " "(concepts)".
D.11 LLM-RANK Template for Miami Housing
1 Main System Prompt */
Given list of features, rank them according to their importances predicting the selling price of homes in Miami. The ranking should be in descending order, starting with the most important feature.
/* Output Format Instructions */
Your response should be numbered list with each item on new line. For example: 1. foo 2. bar 3. baz
Only output the ranking. Do not output dialogue or explanations for the ranking. Do not exclude any features the ranking.
1 Main User Prompt */ Rank all (number of concepts) features in the following list: "(concepts)".
D.12 LLM-RANK Template for Used Cars
1* Main System Prompt */
Given list of features, rank them according to their importances in predicting the selling price of used car. The ranking should be in descending order, starting with the most. important feature.
/ Output Format Instructions */
Your response should be numbered list with each item on new line. For example: 1. foo 2. bar 3. baz
Only output the ranking. Do not output dialogue or explanations for the ranking. Do not exclude any features the ranking.
/* Main User Prompt */ Rank all (number of concepts) features in the following list: "(concepts)".
D.13 LLM-RANK Template for NBA*
/* Main System Prompt */
Given list of features, rank them according to their importances in predicting the number of points per game of an NBA basketball player. The ranking should be in descending order, starting with the most important feature.
/* Output Format Instructions */
Your response should be a numbered list with each item on a new line. For example: 1. foo 2. bar 3. baz
Only output the ranking. Do output dialogue or explanations for the ranking. Do not exclude any fcatures in the ranking.
/* Main User Prompt */ Rank all (number of concepts) features in the following list: "(concepts)".
D.14 LLM-RANK Template for NYC Rideshare*
72

---

Published in Transactions on Machine Learning Research (04/2025)
/* Main System Prompt */
Given list of features, rank them according to their importances in predicting the total pay given to a rideshare driver from a trip. The ranking should be in descending order, starting with the most important feature.
/* Output Format Instructions */
Your response should be numbered list with each item line. For example: 1. foo 2. bar 3. baz
Only output the ranking. Do not output dialogue or explanations for the ranking. Do not exclude any features in the ranking.
1 Main User Prompt */
Rank all (number concepts) features the following list: "(concepts)".
73

---

Published in Transactions on Machine Learning Research (04/2025)
E Prompting for Sequential Feature Selection
In this section, we provide all of the prompt templates used for LLM-SEQ in Appendix E.1-E.14. For LLM-SEQ, we only use the prompt template that contains (i) main system prompt and (ii) the main user prompt. Unlike for LLM-SCORE and LLM-RANK, we do not add any specialized output format instructions for LLM-SEQ, as we prompt the LLM to directly output only the selected feature name/ ccnnept in text. each prompt template, we mark each of the two components with a red tag (c.g., /* Main System Prompt */)- We clarify that the lags are not. part of the actual input prompt provided to the LLMss Any text enclosed in angled brackets (-) serves placcholder for value be specified by the user. For example, "(candidate concepts)" " is a placeholder for the list of all candidate concepts for which wish to obtain an LLM- ILM-generated ranking.
Prompting in a dialogue. One important difference between LLM-SEQ and the other LLM-based feature selection methods is that we engage in a dialogue with an LLM, which takes place over multiple iterations. In such conversational setup, the default approach is provide as input the full conversation history to the LLM at each iteration. Here is an example of what the input prompt would look like at the t-th iteration:
System: Given list of features already selected and list of candidate features available, your task is to output the next feature that should be included to maximally improve the performance in predicting (target outcome).
User: used the features I, and the trained model achieved a test AUROC of N/A. What feature should I add next from: (candidate concepts)? Give me just the name the feature to add (no other text).
LLM: (Concept
User: used the fcatures [(Concept (Conncett - 1)}, and the trained model achieved test AUROC of (value). What feature should add next from: (candidate concepts)? Give me just the name of the feature to add (no other text).
Handling high-dimensional scttings. In practice, when a dataset contains a relatively large number of features, providing the entire conversation history as input can quickly become infeasible for LLMs with limited context window sizes. We address such cases with "buffering" approach, where we only include the most recent interactions, instead of all previous interactions. Here an example of what the input prompt would look like at the t-th iteration when using buffer of size i.e., only including the interaction immediately before:
System: Given a list of features already selected and a list of candidate features available, your task is to output the next feature that should be included to maximally improve the performance in predicting (target outcome).
User: used the features [CCnncept 1), (Concept - 2)}, and the trained model achieved test AUROC of (value). What feature should I add next. from: (candidate concepts)? Give just the name of the feature to add (no other text).
LLM: (Concept t t-1)
- 1)}, and the trained model achieved a test AUROC of (value). What feature should add next from: (candidate concepts)? Give me just the name of the feature add (no other text).
User: used the features [Concept
E.1 LLM-SEQ Template for Credit-G
74

---

Published in Transactions on Machine Learning Research (04/2025)
/* Main System Prompt */
Given a list of features already selected and & list of candidate features available, your task is to output the next feature that should be included to maximally improve the performance in predicting whether an individual carries high credit risk.
/* Main User Prompt */
I used the features (selected concepts), and the trained model achieved test (metric) of (value). What feature should add next from: (candidate concepts)? Give me just the name of the feature to add (no other text).
E.2 LLM-SEQ Template for Bank
/ Main System Prompt */
Given a list of features already selected and list of candidate features available, your task is to output the next feature that should be included to maximally improve the performance in predicting whether an individual will subscribe to a term deposit.
/ Main User Prompt */
used the features (sclected concepts), and the trained model achievod a test (metric) of (valuc). What feature should add next from: (candidate concepts)? Give me just the name of the feature add (no other text).
E.3 LLM-SEQ Template for Give Me Some Credit
1 Main System Prompt */
Given a list of features already selected and a list of candidate features available, your task is to output the next feature that should be included to maximally improve the performance in predicting whether an individual is likely to experience serious financial distress in the next two years.
/ Main User Prompt */
used the features (selected concepts), and the trained model achieved a test (metric) of (value). What feature should add next from: (candidate concepts)? Give me just the name the feature to add (no other text).
E.4 LLM-SEQ Template for COMPAS Recidivism
/* Main System Prompt */
Given a list of features already selected and a list of candidate features available, your task is to output the next feature that should be included to maximally improve the performance predicting whether a criminal defendant carries high risk of recidivism.
/* Main User Prompt */
I used the features (selected concepts), and the trained model achieved a test (metric) of (valuc). What feature should add next from: (candidate concepts)? Give me just the name of the feature to add (no other text).
E.5 LLM-SEQ Template for Pima Indians Diabetes
/* Main System Prompt */
Given a list of features already selected and a list of candidate features available, your task is to output the next feature that should be included to maximally improve the performance in predicting whether a patient has diabetes.
/* Main User Prompt */
I used the features (selected concepts), and the trained model achieved test (metric) of (value). What
75

---

Published in Transactions on Machine Learning Research (04/2025)
feature should add next from: (candidate concepts)? Give me just the name of the feature to add (no
other text).
E.6 LLM-SEQ Template for AUS Cars
/* Main System Prompt */
Given a list of features alrcady selected and a list of candidate features available, your task is to output the next feature that should be included to maximally improve the performance in predicting the selling price car in Australia.
/ Main User Prompt */
used the features (selected concepts), and the trained model achieved test (metric) of (value). What feature should add next from: (candidate concepts)? Give me just the name of the feature to add (no other text).
E.7 LLM-SEQ Template for YouTube*
1 Main System Prompt */
Given list of features already selected and a list of candidate features available, your lask is to output the next feature that should be included to maximally improve the performance in predicting whether YouTuue channel has more than 20 million subscribers.
/ Main User Prompt */
used the features (sclected concepts), and the trained model achieved a test (metric) of (value). What feature should add next from: (candidate concepts)? Give me just the name of the feature add (no other text).
E.8 LLM-SEQ Template for CA Housing
/* Main System Prompt */
Given a list of features already selected and a list of candidate features available, your task is to output the next feature that should be included to maximally improve the performance in predicting the median housing price of U.S. census block group.
/* Main User Prompt */
I used the features (selected concepts), and the trained model achieved test (metric) of (value). What feature should add next from: (candidate concepts)? Give me just the name of the feature to add (no other text).
E.9 LIM-SBQ Template for Diabetes Progression
/* Main System Prompt */
Given a list of features already selected and a list of candidate features available, your task is to output the next feature that should be included to maximally improve the performance in predicting the disease progression status in diabetes paticnts.
/* Main User Prompt */
used the features (selected concepts), and the trained model achieved test (metric) of (value). What feature should add next from: (candidate concepts)? Give me just the name of the feature to add (no other text).
E.10
LLM-SEQ Template for Wine Quality
/* Main System Prompt */
Given list of features already selected and a list of candidate features available, your task is to output
76

---

Published in Transactions on Machine Learning Research (04/2025)
the next feature that should be included to maximally improve the performance in predicting whether a wine is high or low quality.
/ Main User Prompt */
used the features (selected concepts), and the trained model achieved a test (metric) (value). What feature should add next from: (candidate concepts)? Give me just the name of the feature add (no other text).
E.11 LLM-SEQ Template for Miami Housing
/ Main System Prompt */
Given a list of features already selected and a list of candidate features available, your lask is to output the next feature that should be included to maximally improve the performance in predicting the selling price of homes in Miami.
/ Main User Prompt */
used the features (selected concepts), and the trained model achieved a test (metric) (value). What feature should add next from: (candidate concepts)? Give me just the name of the feature add (no other text).
E.12 LLM-SEQ Template for Used Cars
/* Main System Prompt */ Given a list of features already selected and a list of candidate features available, your task to output the next feature that should be included maximally improve the performance in predicting the selling price of used car.
/* Main User Prompt */
I used the features (selected concepts), and the trained model achieved a test (metric) of (valuc). What feature should add next from: (candidate concepts)? Give me just the name of the feature to add (no other text).
E.13 LLM-SeQ Template for NBA*
/ Main System Prompt */
Given a list of features already selected and a list of candidate features available, your task is to output the next feature that should be included to maximally improve the performance in predicting the number of points per game of an NBA basketball player.
/* Main User Prompt */
I used the features (selected concepts), and the trained model achieved test (metric) of (value). What feature should add next from: (candidate concepts)? Give me just the name of the feature to add (no other text).
E.14 LLM-SEQ Template for NYC Rideshare*
/* Main System Prompt */
Given list of features already selected and list of candidate features available, your task is to output the next feature that should be included to maximally improve the performance in predicting the total pay given to rideshare driver from trip.
/ Main User Prompt */
used the features (selected concepts), and the trained model achieved a test (metric) (value). What feature should add next from: (candidate concepts)? Give me just the name of the feature add (no other text).
77


Reviews:
[{"review_id": "4W2dXck2nA", "reviewer": "Reviewer_npvb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n1. This paper addresses an interesting topic by employing LLMs for feature selection. The proposed method, which can also help to reduce the cost of collecting data in specific domains such as healthcare, is both practical and forward-thinking.\n\n2. The experiments are well-conceived, covering datasets of various sizes and different time periods to mitigate the concern of data memorization by LLMs. This ensures the robustness and applicability of the results.\n\n3. This paper provides extensive implementation details, which enhances its clarity and replicability. \n\n**Weaknesses:**\n\nPlease refer to the Requested Changes part.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes to use LLMs to perform feature selection for the downstream tasks without access the whole training dataset. Specifically, the paper proposes three methods for feature selection using LLMs, i.e., LLM-Score, LLM-Rank, and LLM-Seq, and conducts extensive experiments on small-scale datasets and large-scale datasets, and the experimental results show that using LLMs for feature selection can be comparable to (strong) traditional data-driven feature selection methods, especially when the size of LLMs is large enough.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. In the related work section, please include more recent studies from 2024 on LLMs, especially regarding new prompting techniques like ToT [1] and GoT [2], and novel decoding methods. This will help to strengthen the paper.\n\n> [1] Yao, Shunyu, et al. \"Tree of thoughts: Deliberate problem solving with large language models.\" Advances in Neural Information Processing Systems 36 (2024).\n\n> [2] Besta, Maciej, et al. \"Graph of thoughts: Solving elaborate problems with large language models.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 16. 2024.\n\n2. It is unclear whether a validation set is included in the LLM-Seq method. The paper should specify if a validation set is used and how it is partitioned from the main dataset and how to get the performance over the validation set.\n\n3. **Unclear mechanism for feature selection in LLM-Seq:** In LLM-Seq method, the paper does not clearly explain how the LLM interprets the performance of the t-1 th model on the validation set as presented in the prompt. A detailed description of how the LLM processes and reacts to these performance metrics (values) is essential to understand the mechanics behind the feature selection process. The paper should clarify the mechanism by how the performance metrics guide the LLM in selecting the next feature, ensuring the method’s transparency and replicability.\n\n4. The LLM-Seq method is a greedy approach that does not seem to guarantee an optimal solution.  The paper could benefit from adding some discussion on it.\n\n5. The paper does not address whether different prompt templates were tested to observe their impact on results. Exploring the effects of various templates is essential for assessing the robustness and adaptability of the proposed LLM based method.\n\n6. **Need more details about the ICL examples:** The paper lacks disclosure regarding the specifics of human-annotations used for In-Context Learning (ICL), i.e., the few-shot examples. It is unclear how many samples are utilized as ICL examples and whether the ICL examples for different test samples from the same dataset are fixed or vary. Additionally, the criteria for selecting ICL examples are not discussed—is it based on similarity or another standard? More detailed information on these aspects would improve the paper.\n\n7. **Need more details about CoT:** The paper does not provide details on how the content used in the Chain of Thought (CoT) approach is obtained. It is unclear whether this content is derived from human-annotations or generated by the LLM itself. Further clarification on the source of CoT content would enhance the understanding of the methods used.\n\n8. The paper mentions averaging over 5 samples with $T=0$ for GPT-4 and GPT-3.5 to address the inherent non-determinism in generation. However, it is unclear how this averaging is implemented since direct averaging of text outputs is not feasible. Are the authors averaging scores rather than reasoning outputs? More detailed methods, such as those for obtaining deterministic outputs found in OpenAI's cookbook [1], could provide a clearer and more efficient approach to managing non-determinism without requiring multiple inferences. Further explanation on this process would significantly improve the clarity and reproducibility of the results.\n\n> [1] https://cookbook.openai.com/\n\n9. **Misleading metrics used in labeled imbalance setting:** The use of AUROC as the sole evaluation metric on datasets with label imbalance issues, such as the Bank and Give Me Some Credit datasets, may lead to misleading interpretations [1,2]. The authors should include additional metrics that better handle class imbalance.\n\n> [1] Muschelli III, John. \"ROC and AUC with a binary predictor: a potentially misleading metric.\" Journal of classification 37.3 (2020).\n\n> [2] Saito, Takaya, and Marc Rehmsmeier. \"The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets.\" PloS one 10.3 (2015).\n\n10. According to Appendix A1, the experiments utilize 5-fold cross-validation with five different seeds (1,2,3,4,5), and the testing set does not vary across these seeds. It is unclear whether this repeated k-fold validation is conducted for each train-test split. The purpose of this approach should be clarified. Is it intended to ensure the robustness of the results, or could it potentially lead to redundant computations? \n\n11. **Concerns Regarding Inconsistencies in Results Across Figures:** I have concerns regarding the results presented in Fig. 2 (a) and Fig. A1 (a), (b), which suggest that the LLM-Score (based on GPT-4.0) performs better than LLM-Rank and LLM-Seq, particularly in classification tasks. However, according to the results in Fig. 3 (a), the performance of LLM-Score does not appear to be superior to that of LLM-Rank and LLM-Seq, especially under the 30% feature setting. The authors need to clarify why there are inconsistencies in these results.\n\n12. In Fig. 4, it is not disclosed what proportion of features is used to obtain the results.\n\n13. **Concerns About One-Shot Setting in Fig. 4 Experiments:** The experiments in Fig. 4 utilize a one-shot setting when adding examples, which is not a common configuration and casts doubt on the credibility of the results for me. Although the authors claim that this choice is due to the reduced number of features, further explanation is needed. Why does a smaller feature set justify the use of a one-shot setting? This justification is crucial for assessing the validity of the experimental design and results. Additionally, for further concerns about few-shot examples, please refer to Question 6.\n\n14. In Fig. 4, why does the \"context\" condition show better performance than the \"context + examples\" condition? The authors should provide potential explanations for this unexpected result. \n\n15. In Appendix B.1, why is the test set kept constant during the dataset splitting process? The authors should explain the rationale behind maintaining an unchanged test set. \n\n16. **Inconsistency in Baseline Methods for Different Dataset Scales:** Why is Lasso used as the feature selection baseline for small-scale datasets, while group Lasso (gLasso) is used for large-scale datasets? The authors should provide a rational explanation for this choice. What considerations led to the use of different methods based on the scale of the dataset? Additionally, there is a discrepancy in Fig. B2 of the large-scale dataset experimental results, where Lasso is reported instead of gLasso. This inconsistency is confusing and needs to be clarified.\n\n17. **Concerns About Results in Fig. B2 for the Folktables: Employment Dataset:** I have concerns regarding the results shown in Fig. B2, specifically for the Folktables: Employment dataset. The performance across three different downstream tasks (i.e., LightGBM, MLP, and LR) appears remarkably similar. The authors should provide a reasonable explanation for this uniformity in performance. Is there an underlying factor within the dataset or feature selection method that leads to such consistent results across varied models?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PUs6iIKO4H", "reviewer": "Reviewer_Jem7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. This paper considers different decoding strategies: greedy decoding and self-consistency decoding.\n\n2. The experiments in the paper use various setups and evaluation metrics, and the result analysis is quite detailed.\n\nWeakness\n\n1. The datasets involved in the experiments are from very common domains (Credit-G, Bank, Give Me Some Credit, COMPAS Recidivism, Pima Indians Diabetes, AUS Cars, YouTube). For datasets from particularly specialized and rare fields, the effectiveness of LLM feature selection is unknown.\n\n2. The baselines used in this paper are very old. LASSO (1996), recursive feature elimination (2002), minimum redundancy maximum relevance selection (2005), filtering by mutual information (1992).\n\n3. This paper lacks more exploration of the underlying principles behind using LLMs for feature selection.\n\n4. Based on the experimental results, the best-performing feature selection methods are still some of the classic approaches. LLM-based feature selection does not show any particularly significant advantages.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper reveals that large language models (LLMs) can select the most predictive features for a task, using only input feature names and a task description. Their performance matches standard data science tools, and they demonstrate this ability across different query methods.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Weaknesses", "broader_impact_concerns": "N/A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UiubfPlp6P", "reviewer": "Reviewer_4H3k", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper describes three approaches to prompting LLMs for feature selection. The experiments are conducted on small-scale and large-scale datasets, and the performance of LLMs is compared with data-driven feature selection methods.\n\nWeaknesses:\n\nOverall, the paper demonstrates the effectiveness of using LLMs for feature selection via extracting information from feature names. The results, however, are not surprising, since all datasets tested in the paper contain informative and well-understood variable descriptions; the underlying relationships have, of course, been studied in the literature, even though the datasets are published after the release of the LLMs. By contrast, the data-driven methods learn the relationships solely from the current dataset. There is not much point in conducting such comparisons.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper illustrates the use of large language models (LLMs) for feature selection in various prediction tasks, given only the feature names and task description as the input. The experimental results suggest that without using the training data, these models can identify features that achieve predictive performance competitive with commonly used data-driven feature selection methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It is reasonable to assume that the performance of the proposed method depends on (1) the quality and relevance of the variable descriptions, and (2) whether the predictive relationship is an existing or a new one. The authors should conduct more experiments to examine the sensitivity of the proposed method to these factors.\n\nAlso, more complex and challenging problems should be considered. For example, it would be interesting to see how the LLM-based method performs in inferring the gene-gene and gene-trait relationships in a gene expression dataset with thousands of genes.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Two reviewers recommended accepting this paper. One did not post an official recommendation and highlighted some weaknesses in their review. On my read, the authors suitably addressed the weaknesses in their rebuttal. I therefore recommend acceptance.
Decision: Accept (poster)