LLM-Guided Self-Supervised Tabular Learning With Task-Specific Pre-text Tasks
Authors: Sungwon Han
Seungeon Lee
Meeyoung Cha
Sercan O Arik
Jinsung Yoon
Venue: Accepted by TMLR ()

Abstract: One of the most common approaches for self-supervised representation learning is defining pre-text tasks to learn data representations. 
Existing works determine pre-text tasks in a "task-agnostic'' way, without considering the forthcoming downstream tasks. This offers an advantage of broad applicability across tasks, but can also lead to a mismatch between task objectives, potentially degrading performance on downstream tasks. In this paper, we introduce TST-LLM, a framework that effectively reduces this mismatch when the natural language-based description of the downstream task is given without any ground-truth labels. TST-LLM instructs the LLM to use the downstream task's description and meta-information of data to discover features relevant to the target task. These discovered features are then treated as ground-truth labels to define "target-specific'' pre-text tasks. TST-LLM consistently outperforms contemporary baselines, such as STUNT and LFR, with win ratios of 95% and 81%, when applied to 22 benchmark tabular datasets, including binary and multi-class classification, and regression tasks.

Published in Transactions on Machine Learning Research (05/2025)
LLM-Guided Self-Supervised Tabular Learning With Task- Specific Pre-text Tasks
Sungwon Han
lion4 152@gmaall com
Korea Advanced Institute of Science and Technology (KAIST)
Seungeon Lee
marinnearchon/159@mmaai
Korea Advanced Institute of Science and Technology (KAIST)
Meeyoung Cha
mia.cha@mmpi-spe
Maz Planck Institute for Security and Privacy
Sercan Ö. Arik Google Cloud
soarik@ggoogle.com
Jinsung Yoon
Thasungyoon@google..om
Google Cloud AI
Reviewed on OpenReview: https: opwprevveeereeeee..e net/ forum? id= jicadoA222A
Abstract
One of the most common approaches for self-supervised representation learning is defining pre-text tasks to learn data representations. Existing works determine pre-text tasks in a "task-agnostic" way, without considering the forthcoming downstream tasks. This offers an advantage of broad applicability across tasks, but can also lead to mismatch between task objectives, potentially degrading performance on downstream tasks. In this paper, we introduce TST-LLM (Task-specific Self-supervised Tabular learning with LLMs), a framework that effectively reduces this mismatch when the natural language-based description of the downstream fask is given without any ground-truth labels. TST-LLM instructs the LLM to use the downstream task's description and intaaiinfomationn of data to discover features relevant to the target task. These discovered features are then treated ground-truth labels to define "target-specific" pre-text tasks. TST-LLM consistently outperforms contemporary baselines with win ratios 95% and 81%, when applied to 22 benchmark (abular datasets, including binary and multi-class classification, and regression tasks.
1 Introduction
Obtaining unlabeled data for machine learning is typically more scalable and cheaper than gathering labeled data real-world applications. Self-supervised representation learning, which was proposed extract useful information from unlabeled data, enhances the performance of downstream tasks by obtaining superior representations (Chen et al.) 2020 Oord al.) 2018 Tschannen et al. 2019). A common approach for self-supervised representation learning is utilizing pre-text task based the type or characteristics of the data, and then learning representations by optimizing the objective of the pre-text task (instead of the downstream lask) (Assran el al. 2022 Kim el al. 2018 Zhang et al.) 2017). For example, in compuler vision domain, such a pre-text task can be defined to estimate the degree of rotation applied to an image (Gidariss ct al. 2018). Other works have defined augmentations that do not deform the contents of images, such as horizontal flips or color jittering, learn representations that are invariant these modifications (rril
et al. 2020 Han al. 2020 Zbontar al. 2021

---

Published in Transactions on Machine Learning Research (05/2025)
Existing works
Define task-agnostic pre-text tasks
Unlabeled data
Train encoder & Apply for downstream tasks
Unlabeled data- meta-information
Define task-specific pre-text tasks
TST-LLM
Figure 1: Illustration comparing existing self-supervised representation learning methods with our approach.
These pre-text task-based methods have also been extended to tabular data specifically, efforts have been made to adapt pre-text tasks that were previously limited to images and text. Common tasks for labular data include corrupting or masking data and then recoustructing the original sample from the representation (Yoon 2020: Wu et al.] 2024), or designing augmentations suited to tabular data to perform contrastive let al.
learning tasks (Bahri et al. 2022 Somepalli et al., 2022). The inductive bias provided by the pre-text tasks plays a role in preemptively removing spurious correlations or noisy information within tabular data.
Despite the success of using pre-text (asks in tabular learning, fundamental limitation remains in the task mismatch between the pre-text tasks optimized by representation learning and the actual downstream tasks (Sui et al. 2024). This is because the pre-text tasks in representation learning are determined in a "task-agnostie" way that are oblivious to the forthcoming downstream tasks (see the upper part of Figure D Task-agnostic definitions offer the advantage of broad applicability across lasks, but also potentially risks including noisy irrelevant information or eliminating critical information for the downstream task. For example, in the diabetes classification task (Smith et al., 1988), height and weight were important correlated features for classification. However, applying augmentation like randomly masking these features in tabular data with high variable correlation can create unrealistic samples and result in the loss of crucial correlation information (Sui et al. 2024 Hajiramezanali et al. 2022). Such fallacies impair the performance of the downstream task.
Our study seeks to address the issue of the lask-objective mismatch by defining pre-text tasks in a lask-specific rather than task-agnostic manner using the natural-language based description of the downstream task. The description includes the task objective (e.g., "Does this patient have diabetes? Yes or no?") and the answer candidates (e.g., "Yes" or "No"). Using this information, we propose, TST-LLM (Task-specific self-supervised Tabular learning with LLMs), which aims to improve representation learning via L.M.ddicooeeeed features without incorporating any ground-truth labels or label statistics (sec the lower part of Figure D. By leveraging the prior knowledge of the LLM, we explore the relationship bet the lask and data features from their natural language-basee descriptions. This process aims to determine which combinations or transformations of original data fcatures can yield mcaningful information for solving the task. Then, the new features created through the LLM's prior knowledge are used to generate the ground-truth labels for the pre-text tasks that train the representation. For example, the task of predicting whether a patient has diabetes, newly discovered features such as "bmi = weight / height", based on prior knowledge, are likely to have a higher correlation with the label. Learning with these features provides additional task-relevant information, such as the importance of original features to the task and the correlation between them. Further, this enables learning representations that are better aligned with the target downstream task, as opposed to conventional pre-text tasks that are defined solely based on the data structure without considering the underlying problem.
TST-LLM consists of two main stages. Tn the first stage, target. task's textual description, meta-information of data (c.g., feature names and descriptions), and toxt-scrialized unlabcled data are used to construct prompts. Then, they are fed into an LLM to extract new task-relevant features. This process is repeated, while previously extracted features are excluded at the next iteration to ensure the diversity of feature synthesis. In the second stage, the discovered features are considered as ground-truth labels to define a pre-text task. We use supervised contrastive learning (Khosla et al. 2020) to perform multi-task learning for each label, learning useful representations. Additionally, we introduce a process for selecting a diverse feature set from the discovered features that are distinctly aligned with both the original data and each other for computational efficiency.
A key advantage of TST-LLM is its simplicity; it can be applied to any problem as long as a task description and feature descriptions are defined in natural language. We demonstrate that the features discovered
2

---

Published in Transactions on Machine Learning Research (05/2025)
by the LLM are meaningful and relevant to the actual target task. Our model consistently outperforms contemporary baselines with win ratios of 95% and 81%, when applied to 22 benchmark tabular datasets including binary and multi-class classification, and regression tasks.
The remainder of the paper is structured as follows: Section reviews related work, Section 3 presents our method, Section discusses experimental results, and Section concludes with key findings and future directions. Our code is available on Githuty
2 Related Work
Self-suporvised representation learning for tabular data. New advancements in sclf-supervised representation learning enable the discovery of meaningful representations from unlabeled data across wide
modalities, from images (Caron et al.] 2020 Wen et al. 2022 Wu et al. 2018) to texts (Gao et al. 2021
Kenton& & Toutanova 2019 Radford et al. 2019
audio (Mittal et al. 2022 Owens & Efros (2018) and most
recently to tabular data (Balestriero et al. 2023 Gharibshah & Zhu, 2022). One of the common ways of
sell-supervised learning to define a pre-text task on an unlabeled dataset to facilitate learning. According to the litcrature (Gharibshah Zhu 2022), pro-text tasks for tabular data can be broadly classified into three types. The first category, invariance learning, involves defining a positive view of a given sample and learning the representation invariance between them. The positive view of the sample can be created using weak augmentations that do not distort the original content (Bahri et al.) 2022; Somepalli et al. 2022) or by selecting samples with similar characteristies from the training data (Nam et al.) 2023b). The second category, predictive learning, includes methodologies that generate explicit, labels from the dataset and train the model to predict these labels, For example, masking or corrupting data and then using the original data for reconstruction as label (Wu et al.) 2024: Yoon et al. 2020). Some studies also proposed pre-text tasks on various publicly available benchmark datasets or synthelic datasets and then performing transfer learning
2022 The last category includes hybrid approach combining invariance and predictive learning QUear et al. 2021 Zhu et al. 2023
for downstream tasks (Hollmann et al. 2023a Wang & Sun|
All of the above methods define pre-text tasks in a task-agnostic manner, which can lead to inconsistencies with the actual objectives of the downstream tasks that can ultimately hinder performance. Our study proposes a method to effectively reduce this inconsistency by generating task-specific pre-text tasks using the description of the downstream task.
Tabular learning with large language model. LLMs can be applied to various domains by leveraging
their prior knowledge and generalizability to handle unseen tasks (Anil et al. 2023 OpenAl 2023 Recent research has explored serializing tabular data into natural sanguage-based text to solve tabular tasks (Dinh
et al. 2022 Hegselmann et al. 2023 Wang et al., 2023 Task description and meta-information of the tabular data, such as feature names and descriptions, guide the L.LM on which features to focus on order to solve the problem. ESspecally, the rich prior knowledge of LLMs boosts performance in few-shot settings, where labeled data is scarce (Hegselmann et al., 2023). Previous works (Dinh et. al., 2022 Hegselmann et al. 2023 Nam et al.] 2023a) broadly cover the case of in-context learning while calibrating the prompts without further training of the LLM, and the case of applying parameter-cfficint fine-tuning techniques like (IA)3 (Liu et al.
2022 or LoRA (Hu et al. 2021).
Our method leverages the prior knowledge and reasoning ability of LLMs to aim for task-specific pre-text task generation through the downstream task description and the meta-information of data. We only use prompt engineering, enabling operation with limited access to LLMs, similar to an API.
3 Method
Problem formulation. Let's consider tabular dataset with d-dimensional input features D {x; { Jim where x; E dd and no ground-truth labels are provided. A downstream task description in natural language, Erask, and the names and short descriptions of cach feature, Ename = {clame Do and Edese = fe 15 1 are provided. The model aims to train an encoder ʃ that maps raw input x; into a informative data
hutps///gthhabbcmm//uugggo Hatp:///w...mm//moo//////////AAa TSTTTT
3

---

Published in Transactions on Machine Learning Research (05/2025)
Task Description
LLM
Head
SCL with 91
Classifier Regressor
Does this porson
You making for the solve the tack
Downstream Tasks
more then 50,000
Unlabeled Samples
Test Samples
dolars per year?
Fri ago hours
Encoder
Encoder
Meta Information
Task Featurest...
Py edu/ age
Head
SCL with 92
hours hours to work
per wesk
Examples:
age:
Pu:edu
The possible tyge operations follows:..
Unlabeled Samples
Discovered Features
hours
age
Head
SCL with 9M
Prompt for Feature Discovery
(a) LLM-guided feature discovery
(b) Task-specific learning
(c) Apply for downstream tasks
Figure 2: Illustration of TST-LLM. (a) It utilizes the downstream task description and the meta-information of the data to discover features relevant to the task. (b) Subsequently, the discovered features are treated as ground-truth labels to perform lask-specific representation learning. (c) The trained encoder is then frozen and used to solve downstream tasks by attaching a simple Dassifier/regressse
representations ((x).. The trained encoder S is then frozen and used to solve the downstream task by attaching a simple classifier iaceeeeereegee /regressor on top of fxx.). The downstream tasks can be binary or multi-class classification and regression.
Figure and Algorithm 1 illustrate our model. TST-LLM takes in rownstream lask description and meta information to define a pre-text task that aligns with the downstream task objectives, and performs representation learning via this task. Initially, the model passes the task description Erask along with meta-information of data Ename and Edesc to the LLM to generate potentially relevant features through combinations or transformations of original features (Section 3.1 These generated features are set as target labels for the pre-text task, which are then used to train the encoder through multi-task contrastive learning (Section 3.2 Details of each stage are described below.
Algorithm 1: Algorithm for TST-LLM.
Input : Original dataset D, Large language model backbone LLM, Encoder f. Original input feature set V, Number of features to select M, Entropy threshold feur, Meta-information Erasks Ename, and Edesc-
Output : :Trained encoder f.
Donst TeaturrDiissoverry (LLM, V. Erask, Ename, Edesc) ; y FestureSelectionn(nnns D, M, fent)
// LLM-guided feature discovery // Feature selection with minimum redundancy
Define set of projection heads G {g] 1 ....g" MM
Optimize f.g using ESci-malti (Eq. 3
// Representation learning with discovered features
Freeze f for downstream tasks
3.1 LLM-Guided Feature Discovery with Task Description
The model generates data-related features from task description and meta information utilizing the prior knowl- edge and reasoning abilities of an LLM. Feature discovery process involves running multiple LLM inferences. The designed prompt consists of four main components: data description summary, operation instruction, diversity enforcement, and response instruction (see Figure and Appendix A.2 for the example prompt).
Data description summary. This component provides a basic data description for feature discovery (see blue part in Figure 3) It includes the downstream task's description Etask (e.g., "Does this person carn more than 50,000 dollars per year? Yes or no?") as well as feature names and descriptions Ename: "the hours an individual has reported to work per week"). Similar to other works (Hegselmann et al. 2023), we serialize the sample data in-context demonstration, giving hints
Edase (c.g., "hours-per perowek-er- week':
4

---

Published in Transactions on Machine Learning Research (05/2025)
You are data engineer. Given the task description and the list of features and data examples, you making new column for the data which is informative to solve the task.
Task: [Dowustream Task Description)
Features: [Feature Descriptions] Examples: [Serialized Examples]
Given a type of operations below, generate new columns which are most informative solve task using operations. Refer to the examples when generating features. Ouly use features listed in the feature description. Note that multiple operations can nested to generate a new column.
The possible type of operations is as follows:. 'Operation Descriptions)
You also have some new example features generated with these modules.
Example Features:
Index Feature_nanae name Feature_ desc
Features From Previous Trial]
You must write new feature that is different from all above examples features with respect to both names and descriptions.
Format of response for columns:
Thought 1: Any
why the following new feature would be helpful for the task
New feature Type of operation New_column name One line pscudo code for generating columns
Thought 5: .. New feature 5:
Figure Prompt for feature discovery. Text blue corresponds to data description summary part, red text to operation instruction, teal text diversity enforcement, and brown text to response instruction part.
the scale and format of the data. Given the data x, serialization is applied as:
Serialize(x, Ensme) e name is xt.
ed name is x
(1)
where the superscript represents the vector's index value.
Operation instruction. This component guides the LLM on possible operations for feature discovery (see red part in Figure 3.. It encourages the LLM search only for feasibly-generaten features, preventing erroncous behaviors (c.g., generating features that cannot be created from original data features establishing ambignous feature definitions). The operations used are follows:
Transform the feature value with one of the following operators: absolute, logarithm, square root, sigmoid, or frequency.
Transformations:
Numerical Operations: Conduct arithmetic operations from multiple numerical features.
Categorical Operations: Combine two categorical features generate new feature.
Mixed-type Operations: Combine categorical and numerical features to generate new one. For example, the model can discretize numerical feature into categorical one, allowing for categorical operations between the two features.
Diversity enforcement. Instead of concluding feature discovery with a single query, we aggregate features from multiple queries find useful features. However, we want avoid the model from discovering duplicate fcatures multiple trials. To ensure diverse scarch, provide additional instructions to prevent the LLM from selecting features identified in previous attempts (see teal part in Figure 3.. We include descriptions via one-line pseudo-code, along with feature names, to prevent the LLM from simply renaming and selecting the same features. This component is integrated in all iterations except for the initial iteration.
5

---

Published in Transactions on Machine Learning Research (05/2025)
Response instruction. This component includes instructions on how the LLM should format its response (see brown part in Figure 3) The format includes the type of operation, the feature's name, and a one-line pseudo-code necessary to regenerate the feature (e.g., "Numerical Operations capital diff Subtract capital- loss from capital-gain to get the net capital difference"). Setting the response format facilitates easier parsing later on and also gives further evidence on each feature discovery, explaining why the particular feature was selected upon response.
The output text from the LLM prompt is parsed to generate the discovered feature. The generation process is automatically carried out using the LLM, which uses one-line pseudo-code and data to generate function code for producing the feature. The prompt used for automated generation is in the Appendix A.4
3.2 Representation Learning with Discovered Features
The discovered features are semantically related to the target downstream task based on LLM's prior knowledge. We considered these features as ground-truth labels to define pre-text task aligned with the target downstream task. Although TST-LLM is agnostic to the choice of learning method we adopt supervised contrastive learning for its generalizability to downstream tasks (Graf et al. 2021 Khosla et al.) 2020). We define the projected representation of sample x; as 2, = (x,)), where f is the encoder and g is the projection head. According the literature (Khosla et al. 2020): given batched set of N, samples with a pseudo label B {x,, (i i-1 the supervised contrastive loss with a temperature 7 is defined as below. For numerical features among the discovered features, transformed them into discrete features using 1-dimensional k-means clustering with k = 10.
1 1 log exp(z; 2z/T)
LSCL
Σ TEB JEB,jA
Σ
(2)
B
keB 1 Ligh exp(z; Zz/T)
In our model, multiple features discovered, and corrospondingly, there multiple labels available for supervised contrastive learning. We utilize multi-task learning approach to train the encoder, by defining projection head for each ground-truth label and simultancously performing supervised contrastive learning for each label. Specifically, given a set of M discovered features, y {5- 1"" ........ ",M), we define a set of projection heads G = {g',9? .....M........................ "). Subsequently, the encoder f optimizes the following loss:
1
LSCL-multi
SSCL.
M
where each Lyer is a supervised contrastive loss computed with the respective projection head nd for the corresponding label set gmh
Feature selection with minimum redundancy. Multi-task learning on all features generated by the LLM can be computationally heavy. Not all features are informative, and those that closely correlate with original features tend to lose their value pre-texts. Furthermore, high correlation among the generated features could diminish the benefits of multi-task learning. To address this, we eliminate features that do not contribute meaningful information and carefully choose diverse set of features with minimal redundancy, thereby reducing computation costs (see Algorithm 2
First, we define uninformative features as those with the lowest entropy values in the distribution. For example, feature is predominantly assigned only one class across all samples (i.c., low entropy), the amount of information that can be learned from this feature also limited. After calculating the entropy of each discovered feature, those with an entropy below specific threshold (i.e., tent) are eliminated. The filtering threshold tent is set to 0.7, taking into account the entropy distribution of the entire feature set.
For the remaining features, the model selects a feature set that minimizes redundancy. The initial choice is the feature with the smallest correlation to the original data. The remaining features are then selected among the ones with the smallest correlation with the original data, including the previously added features. This
2ee the Appendix F for the comparison with alternative learning methods (including reconstruction).

---

Published in Transactions on Machine Learning Research (05/2025)
process is repeated until predetermined number of features, M, are selected. Cramer's V value (Cramér 1999 is used to measure the correlation after discretizing all numerical features in the same manner y- This approach ensures that the selected features have low correlation with the original data while maintaining diversity among the discovered features, enabling efficient multi-task learning (see Table 1 for an analysis of feature diversity). Refer to the Appendix for example Teatures generated and selected by our method.
Algorithm 2: Algorithm for feature selection with minimum redundancy.
Input Initial feature set Dinit. Number of features to select M, Original dataset D, Entropy threshold fent. Output : Selected feature set y
y +++
D Viitered
Entropy(O) 2 lant; Pinit
// Filter low entropy features
while
D < do
+ 0
for ÿ E Dateer do
6
By 5- max (Mramnerss (D, 9)):
// Compute redundancy the feature
$ $ u U( 01}
end
/* Select features with minimum redundancy */
Vxclected + 10 Φv = mins, ( (0), (0y, () E +}
10
Vittered t Vilcored
I Delected
D + DU2 Doelected
11
J. + D U D Vanlectod
12
13
end
4 Experiment
We evaluate TST-LLM across multiple tabular datasets with various downstream lasks. Through our experiments, discuss which components of the model contributed to performance enhancements and how our model operates. Due to space constraints, full results, computational cost analysis, results with alternative learning objectives, and other extra analyses can be found in the Appendix.
4.1 Performance Evaluation
Datasets. Our study used diverse range of downstream tasks in terms of size and complexity. Since our model requires task and feature descriptions, we selected total of 22 benchmark datasets along with
appropriate mota-information, reforring to previous works FeatLLM (Han et al., 2024) and TabLLM (Hegsel-
mann et al. 2023) Adult (Asuncion & Newman
2007):
Balance-scale (Siegler 1994h, Bank (Moro al.,
2014h.
Blood Yeh et al. 2009), Car (Kadra et al.
2021
Communities (Redmond) (200)) Credit-g (Kadra
et al. 2021), Diabetes (Smith al.) 1988), Eucalyptus Bulloch et al.) 1991), Forest-fires (Cortez & Morais
2008
Heart [fedesoriano] 2021). Junglechess (van Rijn & Vis 2014). Myocardial (Golovenkin et al. 2020
Tic-tac-toe (Aha) 1991). Vehicle (Mowforth & Shepherd), Bike (Panace-T] 2013). Crab (Sidhu
2021
Housing (Pace & Barry 1997). Insurance (Dattal (2020). Wine (Cortez & Reis 2009), Sequence-type and
Solution-mix. Descriptive statistics and task descriptions for each dataset are available in the Appendix A.]
and Among them, 15 datasets were used for classification problems and 7 for regression problems; Two datasets datasets-Sequennaeeeet Sequenco-type and Solution-mix-an are synthetic, ensuring they are not included in the L.L''s pre-training corpus. Evaluation results on additional datasets are included in Appendix [.]
Basclines. Our model was compared to nine basclines, all of which were trained under the same unsupervised selting as our experimental setup. (1) Raw Data: Uses the data as-is for the downstrremm task without any representation learning; (2) AutoEncoder (Baldi 2012): Utilizes a pre-text objective that projects data into emboddings and reconstructs the original data; (3) SimSiam (Chen & He (202)): Trains minimize the embedding distance between a sample and its augmented version using a siamese network structure; (4) SCARF (Bahri et. al. 2022): Employs self-supervised contrastive learning to train augmentationniivaaiann
7

---

Published in Transactions on Machine Learning Research (05/2025)
embeddings. Augmentations involve corrupting some columns of sample by drawing from their marginal
distributions; (5) STAB (Hajiranezanali et al. 2022): Similar to SimSiam but performs augmentation- free representation learning through stochastic regularization; (6) STUNT (Nam et al. 2023bb: Creates self-gencrated tasks based on clustering to facilitate learning through meta-learning: (7) LFR (Sui et al. 2024h: Iteratively learns the larget of a pre-lext lask and the encoder using a random data projector. (8) FeatLLM (Han et al. 2024): Leverages LLMs to discover new rule-based binary features, which are then utilized as representations. (9) MET (Majmundar et al. 2022): Utilizes a Mask-and-Preddic approach to uncover the latent structure within the tabular data. Implementation details for all baselines followed the original works, except that the encoder architecture was standardized. Detailed settings can be found in the Appendix C.2 We further compare our model's performance with zero-shot and in-context learning- inference in A Appendix G though these methods are not designed for unsupervised representation learning. Evaluation results on additional baselines are included in Appendix 4.2
Implementation details. TST-LLM currently employs GPT-3.5 as the LLM backbone for feature discovery but it can be combined with other LLMs. During LLM generation, the temperature was set to 0.5 and the top-p value was set to the API's default of 1. The discovery process generated five features per trial, with the number of trials set at 40. The number serialized samples included in the prompt was set to a maximum 20, as allowed by the prompt limit. The number of selected features M was set. 20. Effects of hyper-parametter M are discussed in the section 4.4 and the number of trials is addressed in the Appendix H The structure of the encoder was consistent with the baselines, configured as 2-layer MLP with 1024 dimensions, and the projection head consisted of single linear layer. Effects of the encoder size are discussed in the Appendix ! Training utilized the Adam optimizer with a learning rate of 1e-4, a batch size of 128, and 1000 training iterations. For information on computing resources and computational complexity, refer to the Appendix D
Evaluation. After training, we fixed the learned embeddings, and the evaluation is performed with two downstream task classifiers: (1) Linear model: This can be either logistic regression for classification tasks or linear regression for regression tasks. This method assesses how linearly separable the classes are in the embedding; (2) Non-parametric classifier: This involves fitting a weighted k-NN module to the downstream classification task. We run evaluations with two different settings: k =3 and 5. This method evaluates how well the embeddings form coherent local clusters. Evaluation results on additional classifiers are included in Appendix F.2 For performance metrics, AUROC is used for classification tasks (one-versus-all for multi-class scttings), and RMSE for regression tasks. Experiments were run with 3 different random sceds, and the average values were reported.
Raw Data AutaEncoder SimSiam
Raw Data AutaEncoder
SCARF
STAB
STUNT
LFR
FeatLLM
MET
Ours
SimSiam
SCARF
STAB
STUNT
LFR
Featl LM
MET
ours
Raw Data
36.36
50.00 36.36
36.3640..1
40.91. 47.62
31.82 60.00 54.55
18.18
Raw Data-
60.71
71
73.33
70.37 97 64.29 44 83
00.00 34.48
AutoEncoder 63.64
63.16
60.00
22.73
AutoEncoder 33.29
68.5
56.67
SimSiam 55.000 36.84
51.5857.14
54.55
46.67
60.00
22.73
SimSiam 14.29
31.03
33.33
32.14
24 25.00
23
51.72 23.33
SCARF 63.64436.84 68.42
75.00
59.09 61.00
SCARF 26.67
32.16
STAB 33..0928.577 42.84 22.87
25.00
54.55 54.5531.82555
53.33 66.67
9.09
STAB 29.63 51.72
87 70.00
64.29 02.35333466.0
76.67
41.38
STUNT 52.3836.36 45.45 40.91 45.45
45.45
59.09
STUNT 31.08466677
LFR 58.18 47.66 47.67 68.168811190 54.55
63.9361.90
19.05
LFR 33.71 43.39
75 00 86 63.63
36.67
77.00
FeatLLM
53.33
53.33 43.3346.67
FeatLLM -5.11 76.67
50.00
MET -4.445 00.00
40.00
23.81
33.33
33.33 40.91 38.10 46.67
22.78
MET- 10.00 76.67
MET-10.00
48.26
27.59 23.5922.3333 23.58 20.00
30.00
Ours -1.80 77.27 77.27 86.36
Ours -6552 58.62 76.67 64.29 58.62 72.41 sa
90.00
(a) Evaluation with linear model.
(b) Exaluation with non-parametric classifier.
Figure& 4: Win matrices comparing self-supervised tabular learning methods against each other with (a) linear model and (b) non-parametric classifier. Sclf-supervised tabular learning methods are aligned on the x-axis and the y-axis while the numbers represent. the winning ratio of the x-axis model against the y-axis model. Full results are reported in the Appendix K
8

---

Published in Transactions on Machine Learning Research (05/2025)
To facilitate straightforward comparison across datasets, adopted win matrix from existing litera- ture (Bahri et al. 2022). The win matrix calculates the ratio over the number of times each method outperforms another method j across the datasets, excluding ties:
EkEDatastts L|Performance(i, k) Performance(j. k)]
W
(4)
Eke kk Dutastsss
T [[Performancei, k) Performance
where Performance(i, k) denotes the performance of method on dataset k.
Results. Figure compares the performance of self-supervised baselines and TST-LLM against each other using win matrices. For all baselines, the average win ratio is 82% for the linear model and 66% for the non-parametric classifier, demonstrating TST-LLM's superiority; This gives strong evidence that task- specific pre-text tasks lead the latent representations that readily form decision boundaries for the target downstream task in the case of the lincar model. At the same time, evaluations with non-parametric classifier indicates that our pre-text lasks ellectively extract and utilize information from existing features to enable clustering.
4.2
Comparison with Conventional Baselines
In this section, additionally compare TST-LLM with 8 conventional baselines, including 2 traditional dimension reduction methods and 6 tabular Icarning methods. (1) PCA (Paarson 1901 Reduces di-
mensionality by transforming data into a set of orthogonal components; (2) I-SNE Van der Maaten & Hinton 2008: Projects data into a lover-dinensional space preserving local structure; (3) XGBoost (Chen & Guestrin 2016): a gradient-boosting algorithm optimized with parallelization and regularization for high performance. (4) Random Forest (Breiman 2001] Is an ensemble learning method that builds multiple decision trees and averages their predictions; (5) Light GBM (Ke et. al. 2017 Is a gradient-boosting algorithm that handles data using histogram-baase based learning: (6) Explainable Boosting Machine (EBM, Nori al.
(2099)): :): Is an interpretable machine learning model that uses generalized additive models with boosting: (7) TabPFN (Hollmann et al. 2023bb: Is a zero-shot tabular classification model that predicts outcomes without training by leveraging a pre-trained Transformer; (8) TP-BERTa (Yan et al. 2024): a tabular representation learning model with BERT-style pre- pre-training. We set the dimensionality for PCA to and for
t-SNE to 2, and we fine-tune TP-BERTa on each dataset prior to comparison. linear model is used as the downstream task classifier of PCA, t-SNE, and model.
Random Forest LightGBM
XGBoost
TabPFN
TP-BERTa
PCA
E-SNE
EBM
Ours
100
PCA-
40.00
33.33
27.59
60.00
t-SNE
0.00
10.00
6.67
10.00
0:00
0.00
36.67
3.33
XGBoost
60.00
90.00
48.28
34.48
40.00
39.13
79.31
33.33
Random Forest
66.67
93.33
51.72
32.14
46.67
43.48
73.33
40.00
LightGBM
72.41
67.86
EBM
44.83
27.27
90.00
36.67
TabPFN
86.36
100.00
60.87
56.52
56.52
72.73
86.96
40.91
TP-BERTa 40.00
63.33
20.69
26.67
16.67
10.00
13. 04
20.00
-20
Ours 82.76 96.67
66.67
60.00
50.00
63.33
59.09
80.00
Figure Win matrices comparing conventional baselines with our model. Tabular learning methods are aligned on the x-axis and the y-axis while the numbers represent the winning ratio of the x-axis model against the y-axis model. Full results reported in the Appendix K
Figure 5 compares TST-LLM with conventional baselines, across both the datasets used in the main paper and those introduced in Appendix F.1 For all baselines, the average win ratios are 70%, underscoring

---

Published in Transactions on Machine Learning Research (05/2025)
TST-LLM's superior performance. Notably, TST-LLM demonstrates results comparable to or exceeding those of supervised tabular learning methods, despite being an unsupervised representation learning framework.
4.3 Ablation Study
We conducted an ablation study to evaluate the contribution of each component in our model. We assessed two primary components: discovering features from the downstream task's description and training the encoder with multi-task contrastive learning. We defined the following ablations by removing modifying each component: (1) Top-1 selection: Only the top-1 feature, which has the least redundancy with the original data among the discovered features, is used; (2) Random-1 selection: Same as Top-1 selection, a single head is used for training, yet. the label used for supervised contrastive learning is randomly changed to one of the discovered features in each iteration; (3) Random feature discovery: Instead of using the LLM for feature discovery, we expand features using operations commonly employed in traditional feature engineering work (Zhang et al. 2023), and then randomly select M features. Representation learning subsequently conducted with these selected features, identical to our original model's approach; (4) Without learning: Instead of performing representation learning, features discovered through feature discovery are directly concatenated with the original data and used as is; (5) Without feature selection: All discovered features are used in representation learning without undergoing the feature selection process.
(1)
(2)
(3)
(4) (5) Ours
(1) (2) (3) (4) (5) Ours
100
100
(1)
55.56
47.37
68.18
35.00 33.33
(1)
46.43
56.67
51.72
51.72
33.33
(2)
-4444
45.00
68.18
40.00 36.10
(2)
-53.57
55.17
50,00
46.67
33.33
80
(1) Top-1 selection
(3)
52.63
55.00
68.18
31.58
36.84
(3) 43.33
44.83
44.83
40.00
31.08
(2) Random-1 selection
(4) 31.82
3182
31.82
31.82 31.82
(4) 48.28
50,00
55:17
50.00 53.33
(3) Random feature discovery
(5)
65 15.00
60.00
68.42
68.18
52.63
(5) 48.28
53.
60.00
50, 00
37.93
(4) Without learning
20
Ours
61.90 68.18
47.37
Ours 66.67 68.07
46.67
62.07
(5) Without feature selection
(a) Linear model results.
(b) Non-parametric classifier results.
Figure 6: Win matrices comparing our full model and its ablations against each other with (a) linear model
and (b) non-paraen non-paraettric classifier. The numbers represent the winning ratio.
Figure shows the degree of performance degradation in each ablation study. We find that every ablation led to a negative effect on performance, underscoring the contribution of the lested compouent. Specifically, using multiple features for multi-task learning, rather than relying on a single feature (i.e., Top-1 selection) alternating features for single-task learning (i.e., Random-1 selection), provided an ensemble effect that enhanced performance. Even without training, merely concatenating features that are relevant to the actual label facilitated the formation of effective local clusters with the parametrei classifie. By conducting training with a pre-text task, TST-LLM could further obtain embeddings that are linearly separable among the labels. In addition, selecting features docs not significantly differ in performance from using all features without feature selection, which suggests that our selection stralegy leads to efficient learning (see Section 4.4 for comparison on computational complexity of using all features).
4.4 Analysis & Discussion
How informative are the discovered fcatures for the downstream task? When training TST-LLM, we utilize the features that have been identified through the LLM. To see how well these pre-text lasks align with actual downstream tasks, we computed the average increase ratio of mutual information between the discovered features and downstream task's labels compared to the original features. The ratio is computed as a percentage for each dataset. According to Figure 7al for most datasets, the discovered features show stronger correlation with the labels than the original data. This suggests that our pre-text tasks are
10

---

Published in Transactions on Machine Learning Research (05/2025)
600
Average Performance Decrease Ratio (%)
Classification
Increase ratio (%)
500
2
Regression
100
75
50
Heart Eucalyptus Crab Housing
-2
0
Solution-ms stale
Wine
Sequence type
Bank
Blord
Forest fires
Bike
Car Credit- Communitio initie Vehlele
Adult Junglechess Myorardial Diabetes
10
20
30 All
Number of Selected Features
(a) Increase ratio of mutual information
(b) Effect of syper-parameite M
Figure 7: (a) Average increase ratio of mutual information between the discovered features and ground-truth labels compared to the original features. The ratio is reported as percentage for each dataset; (b) analysis on the number of selected features M. Average decrease ratios from our model's settings (i.c., M = =20) across both classification and regression tasks are reported.
closely aligned with the actual downstream tasks, than the models trained solely on the original data. We also observed that datasets with higher increase ratio also demonstrated greater performance improvement in our model compared to the Raw Data model (Spearman correlation 0.52). When evaluating our model compared to the raw data model over datasets with positive and negative increase ratios, the positive set showed a 25.8% greater improvement in performance (average 8.1% inerease in the positive set vs. 5.6% increase the negative set). Although some datasets showed a decrease average mutual information, most of them exhibited a high standard deviation in mutual information (see Appendix K.5 for full results), where multi-task learning using a variety of features could be helpful.
How diverse are the features used for our pre-text task? We applied two strategies to ensure the diversity of discovered features coming from the LLM and pre-text tasks for representation learning. One strategy involved adding a diversity enforcement component within the LLM's prompt avoid selecting the previously selected features, and the other aimed lo minimize redundancy among selected features. To verify the effectiveness of these methodologies on feature diversity, conducted additional experiments. We defined three ablation scenarios: (1) No diversity enforcement & No selection strategy: using features without applying the two strategies; (2) No selection strategy: using the diversity enforcement component but not conducting feature selection; (3) With entropy-based filtering: applying only rotropyba filtering as the selection strategy.
Table Ablation study results on feature diversity. Feature diversity evaluated using the average Cramer's V value across features, with the standard deviation noted. Performance change is computed as an averaged change ratio in percentage across all datasets compared to the proposed full model.
Ablation for feature diversity
Cramer's V
Performance change (%)
Time cost ratio
No diversity enforcement & No selection strategy
0.24
-0.1744.24
5.62
No selection strategy
0.13
-0.50
4.84
With entropy-basee filtering
0.09+0.05
-0.05=+0.15
2,75 7.78
Full model
0.07++0.03
0.00+10.00
1.00
We compared each ablation using three evaluation metrics. The first metric Cramer's V value (Cramér 1999 between features, where higher value indicates greater number of highly correlated features, implying lower diversity. The second metric is the percentage change in performance across all datasets compared to
11

---

Published in Transactions on Machine Learning Research (05/2025)
the proposed full model. The final metric is the time cost ratio for running the model. According to the results in Table T models with lower diversity are inefficient both in terms of performance and time cost.
Can TST-LLM be applied other LLM backbones? To verify whether our method performs well with publicly available LLM backbones beyond GPT-3.5, we applied TST-LLM to the Llama3-7b and Llama3- 70b models. Table reports the win ratio of TST-LLM compared to other baselines when applied to different LLM backbones, following the evaluation method in Figure 3-a. The results show that TST-LLM consistently outperforms existing baselines, even with different LLM backbones. Additionally, we observed a positive correlation between the size and reasoning capability of the LLM and the overall model performance. Table 3 reports the average increase ratio of MI between the discovered features and the downstream task's labels compared the original features, using each LLM backbone. The results indicate that larger LLMs with better reasoning abilities achieve higher MI increase ratios, further supporting the effectiveness of our approach.
Table Comparing TST-LLM with different LLM backbones against supersupeeeieed tabular learning baselines. Each value in the table represents the winning ratio of the row method compared to the column method.
Raw Data AutoEncoder SimSiam SCARF STAB STUNT
LFR Leat
MET
Average
Raw Data
0.00
36.36
50.00
36.36
40.91
47.62
31.82
60.00
54.55
39.74
AutoEncoder
63.64
0.00
63.16
63.16
71.43
63.64
52.38
60.00
71.43
56.54
SimSiam
50.00
36.84
0.00
31.58
57.14
54.55
52.38
46.67
60.00
43.24
SCARF
63.64
36.84
68.42
0.00
75.00
59.09
61.90
46.67
76.19
54.19
STAB
59.09
28.57
42.86
25.00
0.00
54.55
31.82
53.33
66.67
40.21
STUNT
52.38
36.36
45.45
40.91
45.45
0.00
45.45
46.67
59.09
41.31
LFR.
68.18
47.62
47.62
38.10
68.18
541.55
0.00
53.33
61.90
48.83
FeatLLM
40.00
40.00
53.33
53.33
46.67
53.33
46.67
0.00
53.33
42.96
MET
45.45
28.57
40.00
23.81
33.33
40.91
38.10
46.67
0.00
32.98
Ours (GPT3.5)
81.82
77.27
77.27
86.36
90.91
95.00
80.95
66.67
77.27
81.50
Ours (Llama3-700) 70B))
81.82
81.82
77.27
86.36
90.91
85.71
80.00
66.67
77.27
80.87
Ours (Llama3-8B)
72.73
70.00
75.00
90.00
86.36
81.82
80.00
60.00
76. 19
76.90
Table Average increase ratio of mutual information across different LLM backbones between the discovered features and ground-trth labels compared to the original features.
LLM backbone
GPT-3.5
Llama3-70B
Llama3-8B
Average increase ratio of MI (avgistee 69.991136.08
58.07+20.22 47.63++24.00
M affect the performance? TST-LLM has a pappee-rrraeter, M, which represents the number of features discovered for the pre-text. task. To investigate the impact of M on performance, we conducted experiments using M 10, 20, 30, and all features (i.e., M all) for the pretext lask. The results, presented in Figure 7b include the average decrease ratio in performance from our model's settings across all classification and regression datasets. The performance of TST-LLM is insensitive to M when M is set bigger than 10, allowing for flexibility in choosing the number of features to optimize computational efficiency. Based on our findings, we selected M = 20, which delivered the best performance without, imposing a computational burden.
Does
5 Conclusion
We introduced TST-LLM, a representation learning method that creates pre-text tasks that are tailored to downstream task objectives using an LLM. TST-LLM leverages the prior knowledge and reasoning abilities of the LLM determine how combine original data features into informative features based on natural language descriptions of downstream tasks and feature descriptions. The combined features, after undergoing
12

---

Published in Transactions on Machine Learning Research (05/2025)
feature selection process to minimize redundancy, serve as ground-truth labels for the pre-text tasks in representation learning. Extensive analysis confirms that our methodology can identify diverse and task-aligned features, and as a result consistently achieves outstanding performance across various downstream tasks.
As a potential future direction, we propose extending beyond self-supervised learning to semi-supervised learning by incorporating labeled samples. In situations where meta-information alone is insufficient define relationships between target labels or where relying on prior knowledge is challenging for specific tasks, labeled samples can serve as valuable complement. We believe that model effectively leveraging both the knowledge from LLMs and data achieve broader applicability.
Limitations and broader impact. Our method requires task meta-information for discovery, but this comes at a minimal annotation cost. In addition, our method relies on LLM for feature discovery, which may not yield optimal results for tasks that the LLM is unfamiliar with. To mitigate this, one could consider optimizing alongside traditional self-supervised representation learning objectives in the tabular domain, such as reconstruction (Yoon et al., 2020g or contrastive learning (Bahri et al.) 2022). In terms of the impact, TST-LLM facilitates easy learning through lask-aligned pre-text tasks with the desired downstream task objective, when these goals be articulated through text. This adaptability renders it suitable for variety of real-world scenarios, such as in the healtheare and financial sectors. We believe this work provides new perspective on the integration of LLMs into the tabular learning domain.
Acknowledgement
Han, Lee, and Cha were partially supported by the National Research Foundation of Korea grant (RS-2022- 00165347).
References
David Aha. Tic- Vic-Tac-Tor Toe Endgame.
UCI Machine Learning Repository, 1991.
DOL:
hupp://doi.orr//000 /tt.///ooo.r//0.2434343 111///00.224///111 CC5688..
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm technical report. arXie preprint arXiv:2305.104 10403, 2023.
Scrcan Ö Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 6679 6687, 2021.
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas, Masked siamese networks for label-efficient learning. In European Conference on Computer Vision, pp. 456-473. Springer, 2022.
Arthur Asuncion and David Newman. Uci machine learning repository, 2007.
Dara Bahri, Heinrich Jiang, Yi Tay, and Donald Metzler. Scarf: Self-supervised contrastive learning using random feature corruption. In International Conference on Learning Representations, 2022.
Pierre Baldi. Autoencoders, unsupervised learning, and deep architectures. In Proceedings of ICML workshop on unsuperoised and transfer learning, pp. 37-49. JMLR Workshop and Conference Proceedings, 2012.
Randall Balestriero, Mark Tbrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, et al. A cookbook of self-supervised learning. arXiv preprint 1xXi:22004,12100 2023.
Leo Breiman. Random forests. Machine learning, 45:5-32, 2001.
BT Bulloch et al. Eucalyptus species selection for soil conservation in seasonally dry hill country- year assessment. New Zealand journal of forestry science, 21 (1):110333 31, 1991.
13

---

Published in Transactions on Machine Learning Research (05/2025)
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912 9924, 2020.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sighdd international conference on knowledge discovery and data mining, pp. 785-794, 2016.
Ting Chen, Simon Kornblith, Mohammad Norouž,, and GeofTrey Hinton. simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1601 PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IBEE/CVF conference on compuler vision and pattern recognilion, pp. 15750-15758, 2021.
Cerdeira A-Almeida F. Matos T. Cortez, Paulo and J. Reis. Wine Quality. UCI Machine Learning Repository, 2009. DOL:
(atrg//11..44499 10.24432/C56537
Paulo Cortez and Anbal Morais. Forest Fires.
UCI Machine Learning Repository, 2008.
DOL:
Harald Cramér. Mathematical methods of statistics, volume 26. Princeton university press, 1999.
Anirban Datta. glc.com/datasett/tteettt (twrrhhhwwahhlltiissaaaccaaaaa
US Health Insurance Dataset.
Kaggle,
2020.
kag-
Tuan Dinh, Yuchen Zeng, Ruisu Zhang, Ziqian Lin, Michael Gira, Shashank Rajput, Jy-yong Sohn, Dimitris
Papailiopoulos, and Kangwook Lee. Lift: iogguae-inteefacee fine-tuning for non-language machine 2022.
learning tasks. Advances in Newral Information Processing Systems,
Hadi Fanace-T. Bike Sharing.
UCI
Machine
Learning Repository, 2013.
DOI:
htups://doi.org/ 10.24432/C5W899
fedesoriano. Heart Failure Prediction Datasct. Kaggle, 2021.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp- 6894-6910 2021.
Zhabiz Gharibshah and Xingquan Zhu. Local contrastive fcature learning for tabular data. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pp.
2022.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations, 2018.
S.E. Golovenkin, V.A. Shulman, D.A. Rossiev, P.A. Shesternya, S.Yu. Nikulina, Yu.V. Orlova, and V.F. Voino Yasenetsss.. Myocardial infarction complications. UCI Machine Learning Repository, 2020. DOL: www..oo/gr/...22////..
Florian Graf, Christoph Hofer, Mare Niethammer, and Roland Kwitt. Dissecting supervised contrastive lcarning. In International Conference on Machine Learning, pp. 3821-3830. PMLR, 2021.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et. al. Bootstrap your own latent-a new approach to self-supervised learning. Advances neural information processing systems, 33:21271-2128. 2020.
Ehsan Hajiramezanali, Nathaniel Lee Diamant, Gabriele Scalia, and Max W Shen. Stab: Self-supervised learning for tabular data. In NewrIPS 2022 First Table Representation Workshop, 2022.
14

---

Published in Transactions on Machine Learning Research (05/2025)
Sungwon Han, Sungwon Park, Sungkyu Park, Sundong Kim, and Meeyoung Cha. Mitigating embedding and class assignment mismatch in unsupervised image classification. In European Conference on Computer Vision, pp. 768 784. Springer, 2020.
Sungwon Han, Jinsung Yoon, Sercan O Arik, and Tomas Pfister. Large language models can automatically engineer features for few-shot tabular learning, arXiv preprint arXiv:2404.09491, 2024.
Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. Tabllm: Few-shot classification of tabular data with large language models. In International Conference on Artificial Intelligence and Statistics, pp. 5549-558L PMLR, 2023.
Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. Tabplu: A transformer that solves small tabular classification problems in second. In The Eleventh International Conference on Learning Representations, 2023a.
Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. Tabpfn: A transformer that solves small tabular classification problems in second. In The Eleventh International Conference on Learning Represenlations, 2023b.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
Arlind Kadra, Marius Lindauer, Frank Hutter, and Josif Grabocka. Well-tuned simple nets excel on tabular datasets. Advances in neural information processing systems, 34:23928 23941, 2021.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing syslems, 30, 2017.
Jacob Devlin Ming- Weig Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pp. 4171-4188 2019.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:18661-1867 2020.
Dahun Kim, Donghyeon Cho, Donggeun Yoo, and In So Kweon. Learning image representations by completing damaged jigsaw puzzles. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV),
pp. IEEE, 2018.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel, Few-shot parameter- -rramtterefficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950-1965 2022.
Kushal Alpesh Majmmndar, Sachin Goyal, Praneeth Netrapalli, and Prateek Jain. Met: Masked encoding for tabular data. In NeurIPS 2022 First Table Representation Workshop, 2022.
Himangi Mittal, Pedro Morgado, Unnat Jain, and Abhinav Gupta. Learning state-aware visual representations from audible interactions. Advances in Neural Information Processing Systems, 35:23765-2377 23779, 2022.
Sérgio Moro, Paulo Cortez, and Paulo Rita. A data-driven approach to predict the success of bank telemarketing. Decision Support Systems, 62:22-31, 2014.
Pete Mowforth and Barry Shepherd. Statlog (Vehicle Silhouettes). UCI Machine Learning Repository. DOL: https://doi.orr//l 10. 24432/C5HHGm
Jachyun Nam, Woomin Song, Scong Hycon Park, Jihoon Tack, Sukmin Yun, Jachyung Kim, and Jinwoo Shin. Semi-super tabular classification via in-context learning of large language models. In Workshop on Ffficcint Systems for Foundation Models ICML2023. 2023a.
15

---

Published in Transactions on Machine Learning Research (05/2025)
Jachyun Nam, Jihoon Tack, Kyungmin Lee, Hankook Lee, and Jinwoo Shin. Stunt: Few-shot tabular learning with self-generated tasks from unlabeled tables. In The Eleventh International Conference on Learning Representations, 2023b.
Harsha Nori, Samuel Jenkins, Paul Koch, and Rich Caruana. Interpretml: A unified framework for machine learning interpretability. arXiv preprint arXiv:1909.00 09223, 2019.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Reprosentation learning with contrastive predictive coding. arviv preprinl 1r 807.03745 2018.
OpenAl. Gpt-4 technical report, 2023.
Andrew Owens and Alexci A Efros. Audio-visual scene analysis with self-suporvised multisensory features. In Proceedings of the European conference on compuler vision (ECCV), pp. 631 648, 2018.
R Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Slatislics & Probability Lellers, 33(3): 291-297, 1997.
Karl Pearson. Liii on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin philosophical magazine and journal of science, 2(11): 572, 1901.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, Nyy Sutskever, Jeffrey Dean, and Sanjay Ghemawat, Language models are unsupervised multitask learners. In OSDI004: Sixth Symposium on Operaling System Design and Implementation, pp. 137 150, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.
PMLR, 2021.
Michael Redmond. Communities and Crime.
UCI Machine Learning Repository, 2009.
DOI:
https://doi.orr// 10. 244224433233333
Gursewak Singh Sidhu. Crab age prediction, 2021. URL
R. Siegler. Balance Scale. UCI Machine Learning Repository, 1994. DOL: https://doi.orr// /ttp://oorr//0.24/////C/C/
Jack W Smith, James Everhart, WC Dickson, William Knowler, and Robert Scott Johannes. Using the adap learning algorithm to forecast the onset of diabetes mellitus. In Proceedings of the annual symposium on computer application in medical care, pp. 261. American Medical Informatics Association, 1988.
Gowthami Somepalli, Avi Schwarzschild, Micah Goldblum, C Bayan Bruss, and Tom Goldstein. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. Tn NeurIPS 2022 First Table Representation Workshop, 2022.
Yi Sui, Tongzi Wu, Jesse C Cresswell, Ga Wu, George Stein, Xiao Shi Huang, Xiaochen Zhang, and Maksims Volkovs. Self-supervised representation learning from random data projectors. In The Twelfth International Conference on Learning Representations, 2024.
Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual informa- tion maximization for representation learning. In International Conference on Learning Representations, 2019.
Talip Ucar, Ehsan Hajiramezanali, and Lindsay Edwards. Subtab: Subsetting features of tabular data for self- supervised representation learning. Advances in Neural Information Processing Systems, 34:18853-18865, 2021.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Jan N van Rijn and Jonathan K Vis. Endgame analysis of dou shou qi. ICGA Journal, 37(2):120-122 124, 2014.
16

---

Published in Transactions on Machine Learning Research (05/2025)
Zifeng Wang and Jimeng Sun. Transtab: Learning transferable tabular transformers across tables. Advances in Neural Information Processing Systems, 35:2902-20) 2915, 2022.
Zifeng Wang, Chufan Gao, Cao Xiao, and Jimeng Sun. Anypredict: Foundation model for tabular prediction. arXiv preprint ar Xxx:2905.12081 2023.
Xin Wen, Bingchen Zhao, Anlin Zheng, Xiangyu Zhang, and Xiaojuan Qi. Self-supervised visual representation learning with semantic grouping. Advances in neural information processing systems, 35:16423 16438, 2022.
Jing Wu, Suiyao Chen, Qi Zhao, Renat Sergazinov, Chen Li, Shengjie Liu, Chongchao Zhao, Tianpei Xie, Hanqing Guo, Cheng Ji, et al. Switchtab: Switched autoencoders are effective tabular learners. In Proceedings of the AAAI Conference on Arlificial Intelligence, volume 38, pp. 15924 15924-15933, 2024.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pallern recognilion, pp. 3733-3333 3742, 2018.
Jiahuan Yan, Bo Zheng, Hongxia Xu, Yiheng Zhu, Danny Z Chen, Jimeng Sun, Jian Wu, and Jintai Chen. Making pro-trained language models great tabular prediction. In 12th International Conference Learning Represenlations, ICLR 2024, 2024.
I-Cheng Yeh, King-Jang Yang, and Tao-Ming Ting. Knowledge discovery on rfm model using bernoulli sequence. Expert Systems with applications, 36(3):5866-557) 5871, 2009.
Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela van der Schaar. Vime: Extending the success of self-and semi-supervised learning to tabular domain. Advances in Neural Information Processing Systems, 33:11033 11043, 2020.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International conference on machine learning, pp. 12310 12320. PMLR, 2021.
Richard Zhang, Phillip Isola, and Alexci A Efros. Split-brain autoencoders: Unsupervised learning by cross-chamnel prediction. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1058 1067, 2017.
Tianping Zhang, Zheyu Aqa Zhang, Zhiyuan Fan, Haoyan Luo, Fengyuan Liu, Qian Liu, Wei Cao, and Li Jian. Openfe: Automated feature generation with expert-level performance. In International Conference Machine Learning, pp. 41880-41901. PMLR, 2023.
Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karypis, and Mahsa Shoaran. Xtab: Cross-table pretraining for tabular transformers. ar preprint orXiv:2305.0609 2023.
17

---

Published in Transactions on Machine Learning Research (05/2025)
Appendix
A Full Prompt Examples
A.1 Task Description for Each Dataset
This section presents the downstream task descriptions corresponding the dataset used for evaluation.
TST-LLM uses these text descriptions to perform lask-relevant. feature discovery. Each description is defined
by referencing the dataset's original source previous works (Hegselmann et al. 2023 Han et al. 2024. For classification tasks, answer class candidates were provided.
Table 4: Downstream task's description of each dataset used for feature discovery.
Data
Downstream task's description
Adult
Does this person earn more than 50000 dollars per year? Yes no? Which direction does the balance scale tip to? Right, left, or balanced? Does this client subscribe to a term deposit? Yes or no? Did the person donate blood? Yes or no?
Balanco-scale
Bank
Blood
Car
How would you rate the decision to buy this car?
Unacceptable, acceptable, good or very good?
Communities
How high will the rate of violent crimes per 100K population be in this area. Low, medium, or high?
Credit-g
Does this person receive credit? Yes no? Does this patient have diabetes? Yes or no? How good is this Eucalyptus species for soil conservation in the specified location? None, low, average, good, or best? Estimate the burned area of forest fires from given information.
Diabetes
Eucalyptus
Forest-fires
Heart
Does the coronary angiography of this patient show a heart disease? Yes or no? Which player wins this two pieces endgame of Jungle Chess? Black, white or draw? Does the myocardial infarction complications data of this patient show chronic heart failure? Yes or no?
Junglechess
Myocardial
Tic-tac-toe
Will the first player (player x) win the game? Positive or negative? What kind of vehicle is the given silhouette information about? Bus, opel, saab, van? Estimate the count of total rental bikes from given information. Estimate the age of the crab from given information.
Vehicle
Bike
Crab
Housing
Estimate the house price from given information.
Insurance
Estimate the individual medical cost of this patient billed by health insurance.
Wine
Estimate the wine quality on a scale from to 10 from given information.
Sequence-type
What is the type of following sequence? Arithmetic, geometric, fibonacci, or collatz?
Solution-mix
Given the volumes and concentrations of four solutions, calculate the percent concentration of the mixed solution after mixing them.
18

---

Published in Transactions on Machine Learning Research (05/2025)
A.2 Full Prompt Example for Feature Discovery
The following is an example of prompt used for leature discovery on the Adult datasel. For the initial query in the LLM, prompt without a diversity enforcement component, as shown in Figure 8 was used as there is no information from previous itcrations. For subsequent itcrations, a prompt with a diversity enforcement component Figure 9 was used.
You data engineer. Given the task description and the list of features and data examples, you making column for the data which informative to solve the task.
Task:
Does this person earn more than 50000 dollars per year? Yes no?
Features:
age: the age of an individual (numerical variable within range [17, 90])
conive-country: country origin for an individual (categorical variable with categories [United-Stane Poland,
Hlland-Nethhrlandk))
Examples:
age is 49. workelass is Private. falwgt is 123807. education HS-grad. educational-mum is 9. marital-status is Separated. occupation is Adm-clerical. relationship is Unmarried. race is Black. gender is Female. capital-gain is O. capital-loss is 0. hours-per-weck is 40. native-country is United-States.
age is 52. workelass is Private. falwgt is 208302. education HS-grad. educational-num is 9. marital-status is -oproee- occupation Sales. relationship is Husband. is White, gender is Male. capital-gain 0. capital-loss is 0. hours-per-weck is 36. native-country is United-States.
Given type of operations below, generate using operations. Refer the examples when generating features. Only use features listed the feature description. Note that multiple operations can be nested to generate new column.
columns which the most informative solve the task
The possible type of operations as follows:
Transformations: Numerical features only. Transform the feature value with one the following operators:
absolute, logarithm, square root, sigmoid, or frequency (l.e., frequency of feature
data).
Numerical Operations: Numerical features only. Conduct arithmetic operation from multiple columns.
Mixed-type Operations: Combine categorical feature and numerical feature to generate new one.
Categorical Operations: Combine two categorical features to generate a new feature. For example, you can infer a condition to make a binary feature, indicating whether it follows the condition.
Format of response for new columns:
Thought 1:
based examples above why the following feature would be helpful for the task] New feature [Type of operation] New NNw__column columnn name One line detailed pscudo code for generating columuns
[Any
Thought 3: New feature
Answer:
Thought 1:
Figure 8:
Full prompt example for feature discovery in the Adult dataset (initial quory without diversity enforcement).
19

---

Published in Transactions on Machine Learning Research (05/2025)
You data engineer. Given the task description and the list of features and data examples, you making column for the data which is informative to solve the task.
Task: Does this person
than 50000 dollars per year? Yes no?
Features:
age: the age of an individual (numerical variable within range [17, 90])
Examples:
age is 49. workclass is Private. fnlwgt is 123807. education HS-grad. oducational-num is 9. marital-status is Separated. occupation is Adm-clerical, relationship Unmarried. race is Black. gender is Femalc. capital-gain is O. capital-osss is 0. hours-per-weck is 40. native-country is United-States.
Given type of operations below, generate 5 new columns which are the most informative to solve the task using operations. Refer to the examples when generating features. Only use features listed in the feature description. Note that multiple operations be nested to generate column.
The possible type of operations is follows:
Transformations: Numerical features only. Transform the feature value with of the following operators: absolute, logarithm, square root, sigmoid, frequency (i.e., frequency of feature the data).
Numerical Operations: Numerical features only. Conduct arithmetic operation from multiple columns.
Mixed-type Operations: Combine categorical feature numerical feature generate
one.
Categorical Operations: Combine two categorical features to generate a new feature. For example, you can infer a condition to make a binary feature, indicating whether follows the condition.
You also have some new example features generated with these modules.
Example Features:
Index Feature name
Feature desc
workclass_genne _gender Combine work kelass and gender to see if certain workelasses have gender-specific income differences education_marree
Combine education and marital-status to see if higher education levels are correlated with
higher income levels
hours per week squared Square the week- column see if there is non-linear relationship with income age age_log log Take the logarithm of age normalize the distribution
capital_ net Subtract eapital-loss from capital-gain
You must
feature that is different from all above examples features with respect both and descriptions.
Format response for columns:
Thought 1: [Any reasons based examples above why the following new feature would helpful for the task] feature [Type operation] New column name One line detailed pseudo code generating columns
Thought 3: New feature 3:
Answer:
Thought 1:
Figure 9: Full prompt example for feature discovery in the Adult dataset (subsequent query with diversity enforcement). Text colored blue represents the diversity enforcement component.
20

---

Published in Transactions on Machine Learning Research (05/2025)
A.3 Example Generation Result from Feature Discovery Prompt
Figure 10 is an example of the oulcome when a prompl for feature discovery is input into the LLM. A rationale and simple description for the implementation of each new feature are generated.
Thought 1: Based on the examples, we can see that capital-gain and capital-loss could be important factors determining if person carns more than 50000 dollars per year. People with higher capital gains are more likely earn more.
New feature 1:
Categorical Operations HighEduuation grad, set HighEducation to 1, else 0.
If education is higher than HS-
Thought 2: The occupation of an individual can also be indicative of their income level.
New feature 2:
Categorical Operations HighInconeeocupatiinn If the occupation Exec- managerial or Prof-specialty, set HighllecomeOoccpation to else 0.
Thought 3: The capital gain and capital loss could be combined to create net capital fea- ture.
New feature 3: Numerical Operations loss
NetCapital
NetCapital
= capital-gain capital-
The combination of age and hours-per-week could provide insights into the indi- vidual's work ethic and experience.
Thought 4:
New feature 4: Numerical Operations WorkEthic WorkEthic age hours-pert wers--per-wek
Thought 5: The combination marital status and relationship could indicate the individ- ual's family situation, which might affect their income.
New feature 5: Mixed-type Operations FamilyStatus Tf marital-status Married-civ-spouse and relationship is Husband or Wife, set FamilyStatus to else 0.
Figure 10: Example generation result by LLM from feature discovery prompt in the Adult dataset.
21

---

Published in Transactions on Machine Learning Research (05/2025)
A.4 Full Prompt Example for Parsing Features
Figure shows the prompl used to create converter code for generating actual features based on the new feature descriptions provided by the LLM during the feature discovery process. This prompt is input into the LLM to extract Python-based conversion code for use.
Provide me a python code for function, given description below.
Function name: column column_appender
Input: Dataframe df input
Input Features:
age: the age of an individual (numerical variable within range [17, 90])
Output: Dataframe df output.
Function description: Create new dataframe of output. Each column in df output con- tains new columns which are made based on the column description below. Be sure that the function code well matches with its feature type (i.e., numerical, categorical), and follows the below guidelines.
Code guidelines (Format Type New column name One line pscudo code):
Generate each column with one line code.
If there is no sufficient information to generate code, ignore the corresponding column.
Make sure that the resulting type column is either string (for categorical) or numeric (for continuous).
Column description:
New feature 1: Categorical Operations HighEducation HighEducation to 1, else 0.
If education is higher than HS-grad, set
New feature 2: managerial or Prof-specialty, set HighlncomeOOccupation to else O.
Categorical Operations
HiglIncomooccupation If the occupation Exec-
New feature 3: Numerical Operations loss
NetCapital
NetCapital capital-gain capital-
New feature 4: Numerical Operations
WorkEthic
WorkEthic = age hours-per-weck
New feature Mixed-type Operations FamilyStatus If marital-status Married-civ-spouse and relationship is Husband Wife, set FamilyStatus to else 0.
Figure 11: Full prompt example for parsing features in the Adult datsaet.
22

---

Published in Transactions on Machine Learning Research (05/2025)
A.5 Example Generation Result for Parsing Features Prompt
Figure 12 is example of the LLM's output for the parsing features prompt. It provides a Python code function for creating the discovered feature.
<start>
def column__aat aat_aaapppddd__ampge.e
df df_output df imput.copy( df ootpotutpiiiiaae "Icatinndccttoon]] = df leeeeeccattioooaapllllllll df [f__oupuu]" (iigaa//iiHlloooooccuaaaa ("Exec-manageriial" 'Prof-specialty"] else 0) df output["Net df outputt df _f__uuutff wwwtttttFamillllleet [aailyySaatus'] = df pntput.apply(lambl x: (x[' d'"maritalssaats"" 'Married-
if != 'HS-grad else 0) df output] patp:patioon.apllyllaambaa
x: 1 if x in
= df outputs outputfccapital-gall capptallgaain'] df output[ccapital-oss
= df output a'ag']] * df dF_outputfhours-ppoe "hoors-per-wockk]
civ-spouse' and xf {[relationship]] in ('Husband', 'Wife")) else O, axis=1)
return df output
<end>
Figure 12: Example generation results for parsing features prompt in the Adult dataset.
23

---

Published in Transactions on Machine Learning Research (05/2025)
B
Dataset Details
In our work, we use total of 22 datasets. Each dataset is assigned to either binary, multi-class classification, or regression task. The datasets were selected considering the size, variety, and types of features. Basic
information of each dataset are shown in Table 5 below. Task objectives of entire datasets are listed
Appendix
Table 5: Basic information of datasets used for evaluation.
Data
# of samples
# of features CCategorrcallNumeeei
Task
Adult
48842
14 (7/7)
Binary classífication
Balance-scale
625
4 (0/4)
Multi-class classification
Bank
45211
16 (8/8)
Binary classification
Blood
748
4 (0/4)
Binary classification
Car
1728
6 (5/1)
Multi-class classification
Communities
1994
103 (1/102)
Multi-class classification
Credit-g
1000
20 (12/8)
Binary classification
Diabetes
768
(0/8)
Binary classification
Eucalyptus
736
19 (5/14)
Multi-class classification
Forest-fires
517
12 (2/10)
Regression
Heart
918
11 (4/7)
Binary classification
Junglechess
44819
6 (0/6)
Multi-class classification
Myocardial
1700
I11 (94/17)
Binary classification
Tic-tac-toe
958
9 (9/0)
Binary-classffication
Vehicle
846
18 (0/18)
Multi-class classification
Bike
17379
12 (3/9)
Regression
Crab
3893
8 (1/7)
Regression
Housing
20640
9 (1/8)
Regression
Insurance
1338
6 (3/3)
Regression
Wine
6497
12 (1/11)
Regression
Sequencetyype
250
(0/5)
Multi-class classification
Solution-mix
300
(0/8)
Regression
24

---

Published in Transactions on Machine Learning Research (05/2025)
C Implementation Details
C.1 TST-LLM details
This section provides additional implementation details of our model. In the feature discovery process of TST- LLM, we use the GPT-3.5 model as the LLM backbone. Meta-information such as feature names and descrip- tions were included the prompt. For categorical features, a list of categories for each feature was added, and for numerical features, the min-max value statistics were included. During LLM generation, the temperature was set 0.5 and the top-p value was set to the API's default o. The discovery process generated five features per trial, with the number of trials set at 40. The number of serialized samples included in the prompt maximum of 20, allowed by the prompt limit. When the number of features in dataset exceeded 100 (e.g., communities, myocardial), and the prompt limit was reached, we resolved this by selecting a random 10 columns per query. Over 40 trials, we ensured that all features were used at. least once in the feature discovery process.
set to
After the LLM completed feature discovery, a feature set satisfying the minimum redundancy between the origi- nal data was selected for representation learning. The number of selected features M was to 20. The encoder structure for representation learning was consistent with the baselines, configured dimensions. The projection head consisted of single lincar layer, projecting 1024 to 128 dimensions. Training utilized the Adain optimizer with learning rate of 1e-4, batch size of 128, and 1000 training iteralions.
2-layer MLP with 1024
C.2 Baseline details
This section describes the implementation details of the baselines. While the implementation of the baselines followed the original works of the respective papers, the encoder used to extract representations configured uniformly for fair comparison (i.c., a 2-layer MLP with 1024 hidden dimensions). Different decoder and projector networks were used according lo each methodology.
For Autoencoder baseline, the decoder was the same 2-layer MLP with 1024 hidden dimensions as the encoder. For Siamese network-based methodologics (c.g., SimSiam, SCARF, STAB), a 2-laycr MLP with 256 hidden dimensions was used as the projector, and for SimSiam, the predictor consisted of single linear layer. STUNT, which uses prototype-basee learning, does not have separate decoder. For LFR, single linear layer predictor and 2-yayer ReLU network with 256 hidden dimensions were used as the random data projector.
For all baselines, we referred to the following links for the implementatior
http:://gthuu..om//rrrrfffallllt
httpe://gthubb.cmm/mggaraaaaaai HannesstLL
25

---

Published in Transactions on Machine Learning Research (05/2025)
D Computational Complexity
In this section, we compare the computational time required for model training. The comparison was conducted on the Adult dataset using & single A100 GPU. For our model, the computation time includes the entire process of feature discovery and selection from the LLM,, as well as training. Table t reports the total time spent for each method. We found that our model has a computational time complexity comparable to other baselines.
Table 6: Computational time complexity analysis of self-supervised representation learning methods. The total time spent (in seconds) and the ratio compared to our model are reported for each method.
Model
Time spent (second)
Time spent (ratio)
Autoencoder
520.4
1.08
SimSiam
350.7
0.73
SCARF
479.6
1.00
STAB
208.7
0.43
STUNT
608.8
1.27
LFR
470.3
0.98
LLM
682.2
1.42
TST-LLM
481.2
1.00
26

---

Published in Transactions on Machine Learning Research (05/2025)
E
Learning with Other Objectives
In our framework, we utilize supervised contrastive learning to integrate information from LLM-discovered features into embeddings, although it is not the only available approach. Therefore, in this section, we compare the performance of our framework using different loss objectives with a linear model. Figure 13 compares the performance of self-supervised baselines and TST-LLM against each other using win matrices, while our framework uses different training objectives including supervised contrastive learning (Figure 13a CLIP (Radlord al. p021p (Figure 13b reconstruction (Figure 13c and cross-entropy (Figure 13d Our framework consistently outperforms other self-supervised baselines, irrespective of the training objectives used.
AutoEncoder
Raw Data AutaEncoder Simslam
Raw Data
SimSlam
SCARF STAB STUNT
LFR
FeatLLM
MET
Ours
SCARF
STAB
STUNT
LFR
FeatLLLM
MET
Ours
100
300
Raw Data
36
50.00
36.36
40.91 47.62
31.82
60.00 54.55
18.18
AutoEncoder 643.64
63.16
1.43665
71.43
AutoEncoder
63.64
83 83-16 83
73 69.64 52. 60 00.00 71 28.57
SimSlam -50.00 36.84
32.58
2.5857
57.14
54.55
52.
06.67
00.00
22.73
SimSiam
50.00
54.55 52.38 00
80
SCARF -63.64 36.84
68.02
01.90
33.64
SCARF -33.64
42
STAB -59.09 28.57
-3.00926..8 42.00
25 .0
54.55 31.02
53.33
06.67
9.09
STAB -39.09
28.57
42.86
$4.55 31.82
53.33
66.67
13.64
STUNT -52.38 45.05
60.9144..45
45.45 59.09
STUNT
LFR -8.18.88777 58.10
08.00
54.55
53.3361.90
19.05
40
LFR -68.18
47.62
41.62
38.10
68
54.55
53.33 61.90 30.00
FeatLLM -4.00 40.00 53.33
MET -4.45 28.57
40.00
53.33466667
33.33
53.33 46.67
40.91 38.10
53.33
22.73
33.33
20
FeatLLM 40.00
MET -5.45
40.00
28.57
46.67
46.67
53.33 33.33
20
Ours $1.82 77.27 77.27 3.36660 90.90 95.00 60.90 66.67
67.27
37.27
Ours
$1.82 71.43
80
80:95
80:95
86.36
81.82
70.00
66.67
77.27
(a) Supervised Contrastive Learning
(b) CLIP
Raw Data AutoEncoder
AutoEncoder
SimSlam
SCARF
STAB
STUNT
LFR
FeatLLM
MET
Ours
Raw Data
Simslam
SCARF
STAB
STUNT
LFR
FEatLLM
MET
Ours
100
300
AutoEncoder 66364
63.16 63.16 71.43
63.64 52.39 00.0 71.43 28.57
AutoEncoder
-53.64
63 63
71.43
63 63.64 52.38 60.00
71.4
33.33
SimSiam 50.00
31.58
52.39
SimSiam
54.55
52.38 00
SCARF
68.42
61, 46.67
9.52
SCARF
88
14.29
STAB 58.09 28.57 42.86
25.0625.00
25.00
54.55 31.82
66.67
9.09
STAB 59.09 28.57
42.86 25.00
$4.55 31.82
53.33
66.67
9.09
STUNT 36.36 45.45 40.945445
STUNT
LFR 68.18 68.18 47.62
47.62 38.10.8.......5
53.33 61.90
23.81
LFR 68.18 47.64
47.62
38.10
54.55
53.33
61.90
28.57
FeatLLM -5.00 40.00 03.33333.3346.55 43.33
53.33 33.33
FeatLLM
-10.00
00.000400
40.00
53
53.33
06.33466..67
53.33 45.67
$3.33 33.33
-20
20
MET 40.00
23.31 30.13344...33.14..00
MET
38.10 46.67
Ours 77.27 71.43 80.95 $0.48 90.91 86.35
76.19 66.67
97.27
77.27
Ours $1.82 66.67
76 19
85 71 10.91 60.05 71.43 65.57
77.27
(c) Reconstruction
(d) Cross-Entron
Figure 13: Win matrices comparing self-supervised tabular learning methods against each other, while our framework uses different training objectives including (a) Supervised Contrastive Learning, (b) CLIP, (c) Reconstruction, and (d) Cross-Entropy. Self-supervised tabular learning methods are aligned on the x-axis and the y-axis while the numbers represent the winning ratio of the x-axis model against the y-axis model. Full results are in the Appendix K.7
27

---

Published in Transactions on Machine Learning Research (05/2025)
F Experiments with Additional Datasets & Baselines
F.1 Comparison on OpenML-CC18 datasets
In this section, we compare TST-LLM with other basclinc models using datascts from OpenMI-CC18. In addition to the 11 datasets analyzed in the main paper, we have incorporated 8 additional datasets from OpenML-CC18. Additional datasets include metadata about columns and tasks, which are essential for our framework. Detailed information about the additional datascts is provided Table 7
Figure 14 displays the comparison results between sell-supervised baselines and TST-LLM using win matrices the OpenML-CC18 datasets. For all baselines, the average win ratios are 83% for the linear model and 64% for the con-paramctric classifier, demonstrating TST-LLM's superior performance with the additional datasets.
Table 7: Basic information of additional OpenML-CCC8 datasets.
Data
# samples
# of features (Categorical/Numerrio)
Task
Authorship
672
69 (0/69)
Multi-class classification
Breast-w
559
9 (0/9)
Binary classification
Cme
1178
9 (3/6)
Multi-class classification
Cylinder-ban
432
39 (19/20)
Binary classification
Dmft
637
(2/2)
Multi-class classification
Ilpd
466
10 (1/9)
Binary classification
Optdigits
4496
64 (3/61)
Multi-class classification
Pcl
887
21 (0/21)
Binary classification
AutoEncoder
AutaEncoder
Raw Data
SimSiam
SCARF
STAB
STUNT
LFR
Feat LM
MET
Ours
Raw Data
SimSlam
SCARF
STAB
STUNT
LFR
FeatLLM
MET
Ours
Raw Data
52
63.16
Raw Data-
24 86.36
59.5 59.525588:
61.36 46.7
73.33
3953
AutoEncoder .4.37
AutoEncoder
SimSiam
21.05 22.22
2..18
42.11
36.84 42 47.37 35. 79
SimSiam
3..66
20.
26.09 25.00 5:56
20.45 3957
37.70 3357
SCARF
SCARF
STAB 44.44
47.37
68.42
57.89
78.95
42.11
73.68
35.75
STAB -50.000000 55.56
75.00
72.73
58.14 54.5555555211
71.74 48.89
STUNT
STUNT -44.19 46.67
84.44
66.67
52.1752.27
66.67
LFR 42..11 42.11
63.16
69
86.67
$7.80 63.16
2..05
LFR -FR-38.644 45.65
79.55
62.79
45.45
47.83
40.00 67.39 35.56
FeatLLM -3..84
47.3726.33
FeatLLM 53.3855.27
56.52
MET-a 36.84 06
32
52.63
36
26.3
47.37 52.8452.63
35.79
MET 26.67 34.78
62.22
45.67 28.26228663333 28.2633.33
32.61 43.48
18:18
Ours 56.67
00.4.4422111 78.88888755
Ours 60.47 62.22 80.43 68. 51.1152.17 66.6456552
(a) Evaluation with linear model.
(b) Evaluation with non-parametric classifier.
Figure 14: Win matrices comparing self-supervised tabular learning methods against each other with (a) linear model and (b) non-parametric classifier, using OpenMI-CC18 datasets. Self-supervised tabular learning methods are aligned the x-axis and the y-axis while the numbers represent the winning ratio of the x-axis model against the y-axis model. Full results are reported in the Appendix K
28

---

Published in Transactions on Machine Learning Research (05/2025)
F.2
Comparison with additional classifiers
In this section, we explore whether TST-LLM oflers benefits when applied to other classifiers. We evaluate two tree-based classifiers, XGBoost (Chen & Guestrin 2016) and Explainable Boosting Machine (EBM, Nori et al. (2019) alongside two deep lcarning- based classificrs, Multi-Layer Perceptrons (MLP) and TabNet (Arik & Pfister 2021
Table displays the AUC scores for these classifiers using both raw data and the embeddings generated by TST-LLM across 23 classification datasets, well as the percentage improvement attributed to our model. We can observe that TST-LLM enhances performance in all scenarios except with XGBoost. We hypothesize that this is due to our model being deep learning-based, which likely produces embeddings that facilitate smoother decision boundaries, whereas a tree-based model like XGBoost may perform better with discrete decision boundaries. The performance improvement on other classifiers demonstrates the broad applicability of TST-LLM.
Table 8: Average AUC scores with various classifiers across 23 classification datasets, using the raw data and the embeddings from our model. Full results are reported in the Appendix K
Linear Model
XGboost
EBM
MLP
TabNet
Raw Data Ours
86.25
86.95
87.35
87.01
84.08
87.74
86.51
87.81
87.02
84.16
Improvement Ratio
+1.73%
-0.51%
+0.53%
+0.01%
+0.10%
29

---

Published in Transactions on Machine Learning Research (05/2025)
G Comparison with Zero-Shot & In-Context Learning
TST-LLM performs unsupervised representation learning by targeting features discovered through the LLM. However, leveraging the high generative capabilities of LLMs, one could also consider applying zero-shot inference or in-context learning with few-shot samples for tabular tasks. Table compares the classification performance of our model with zero-shot and in-context learning approaches using the same GPT-3.5 backbone on the Adult dataset. The results demonstrate that our model significantly outperforms both zero-shot and in-context learning methods. Furchermore, observed that even when using a advanced model, zero-shot and in-context learning approaches still underperform in prediction tasks compared to method, underscoring the superiority of our approach.
Table 9: Comparison between TST-LLM and zero-shot / in-context learning-based inference the Adult dataset. Classification accuracy over the test set is reported.
GPT-335 zero-shot
GPT-3.5 4-shot
GPT-335 8-shot
GPT-3.5 16-shot
GPT-4o 16-shot
TST-LLM
65.2
75.3
76.4
80.3
82.1
91.3
30

---

Published in Transactions on Machine Learning Research (05/2025)
H Impact of the Number of Trials in Feature Discovery
In this section, we analyze the impact of the number of trials on downstream task performance when performing feature discovery through an LLM. In our current model setting, five new features are discovered per trial, and total of 40 trials are made to obtain the feature set. Figure 15 below measures the performance change ratio compared the current model as the number of trials is varied to 5, 10, 20, and 30. The results indicate that with 10 or more trials, stable performance achieved across multiple tasks.
Average Performance
Classification Regression
Decrease Ratio (%)
2
0
-2
5
10
20
30
40
Number of Trials
Figure 15: Effect of the number of trials in feature discovery the performance of downstream tasks.
31

---

Published in Transactions on Machine Learning Research (05/2025)
Impact of the Number of Layers in the Encoder
In this scction, we analyze the impact of the number of layers in the encoder on downstream task performance when training the model through supervised contrastive learning. In our default model setting, MLP that consists of two layers with ReLU activation are used. Figure T6 below measures the performance change ratio compared to the default setting as the number of layers in the encoder varied to 2, 3, 4 and, 5. The results indicate that small number of layers provides better performance, reducing the risk of overfitting.
10
Average Performance Decrease Ratio (%)
Classification Regression
-5
-10
2
3
5
Number of Layers
Figure 16: Effect of the number of layers in encoder on the performance of downstream tasks.
32

---

Published in Transactions on Machine Learning Research (05/2025)
J Qualitative Analysis
To verify whether the features discovered by the LLM align with the task definition, we selected and examined the top three discovered features for each dataset using our selection strategy (see Table 10h. We observed that the discovered features somewhat. intuitively align with the downstream task.
Table 10: Top-3 discovered features from our selection strategy for each dataset.
Data
Top-3 discovered features
Adult
age hours- Mours-per-week, educational-num / age, educatioonll-uum
age
sbs(left-weight right-weight),
Balance-scale
wbs(left-weight + left-distance right- - right distance),
(left-weight loft-distance)**22 (right-weight- right-distance)**
Bank
duration campaign, balance duration, duration/ day
Blood
(Reeeny ** 0.5) Frequenry Time, 1. / (1 + np.exp(Roxoncy Timo)),, (Time- Receney) Proquency
(miit 'low's 'mndiuum: 2, "high's 3, 'very high': 4]) doors.map( ( ('Smoree' 5, 14:s 4, '3's '2: 2/). buying. (uy( ''iihh'', 'low':1, 'medium' 12, 'very high + ma lhigh 13, 'lowk1, 'mediium 22, 'very high':43)
Car
+ doors.ma doors.mapp( 'Smmore's :4, 14:3, 22,1, '3 (3:22)) - persons.mpp( moreee, 2:1, 41:3)) lug boot hoot.mpp( l'meed'2, 'big"3. 33, (small::1]] safety map(("mddd) 22 'lo';1, (hight3)),
(maint + safety) /
Commonities
PetEmaaaan HousVacant, MedRentPetHonsshn + petWWage, agePet12:21 NuminShellerr
Credit-g
duration/ age, age duration, age / duration
Ghacose/ Age; Pregnancies Diubetes Prnitrrn, Pedireee map Podigree calut__com_()))
Diabetes
Eucalyptus
(Surv + Vig) Ht, Stem Fin- Brach Fm, Crown Fm Bruch Fm
Forest-fires
temp RH, wind temp, FFMC DMC + ISI
Heart
MMaso.)) RestingBP MaxHR, abe(RestiugBB MaxHR)
(white piecedi file white piecel) rank)/ (black pieceu file + black piecel) rank),
Junglechess
white_ _ae___eeee______ file white_ [rcitee__pecce_____111 pank,
groupby([White ((wchieepppeeeeli pleceo file', 'white pleceO rank', "black pleceO, file', "black pieceo rank) white pleceO) file"). treauffomm 'count')
Myscardial
lsg(AST BLOOD), BLOOD ROE, BLOOD.value counts(]]L BLOOD)
apply(lambda x: [xpltopllf
'x') + [VTmiddll-midddl--quurjj -- 'x') + (=(bottom-righh--saarr)
'x')),
Tic-tac-toe
apply(lamdde x: (x[booton-leff--quuee == 'o') + (xl (cfbottoon-middll--uaaee == 'o') + (x(hootom--right--nuaa == 'o').
apply[lambda x: [x] _e[to-left-squueee] [, x| ttto---ght--ssuaee]] x| <//bottom-leff-quuaet x! l'bottom-right--quaee ccouttt)
COMPACTNESS CIRCULARITY,
Vehicle
SCALED RADIUS OF GYRATION RADIUS RATIO,
PR.AXIS RECTANGULARITY CIRCULARITY
Bike
abs(temp- hum), abs(temp- stemp). hr 4 mnth
Crab
Shell Weight/ (Weight- Shucked Weight - Viscera Weight), Shacked Weight / Viscera Weight, Weight, valuee_ valtu_coouttte(
Housing
population / houscholds, total bedrooms houscholds, modian inrome / population
Insurance
age bui, absfuge bui), age / (children + 1)
Wine
sulphates- volatile acidity, citric acid rosidual sugar, fixed acidity alrohol
Number2 / Numberl Number3 Number2,
Sequenco-type
|'Numberl', 'Number2', 'Numbez3', 'Numberd', 'Number5"] (umb:err//smmxxiiis=))) % 2,
(Number2 Numberl + Number3 / Number2 Numberr / Number3 + Number5 / Xumbber)).cummmun))
Solution volume * "Solution_ __coneentrationn + Solution 2 volume * Solution 2_oomeentrattonn Solution volume
* Solution 3 woo.centraiionnn Solution_
Solution L_oncenrtttnnnn.nnnnn......
abs(Soluticn 1 concentration Solution 2 concentration) + abe(Solution 2 concentration Solution 3 concentration)
Solution-mix
abe abe(Solution___ ___oomo_tttatiinnn Solution 4 __conentration)
np. ap.loo((SShttto))) concentration + Solution concentration + Solution 3 concenttration + Solution 4_ comcentracion)
/ (Solution volume + Solution_ _______v____ volume Solution 3__olume- Solution_ Baatiinn___voli v_oolne))
33

---

Published in Transactions on Machine Learning Research (05/2025)
K Full Results
In this section, we present full results of our experiments.
K.1 Evaluation with Linear Model
Table 11: Evaluation results of self-supervised models on linear model, showing (a) AUC across 15 datasets for classification and (b) RMSE across 7 datasets for regression. Best performances are bolded, and our framework's performances, when second-best, are underlined. For FeatLLM, is not applicable to regression tasks and therefore denoted as
(a) Classification (AUC)
Doraset
Raw Dota
Antoonnnder
SimSSum
SCARE
STAB
STUNT
LFR
FeatLLM
MET
Ours
Adals
90.75=00117
91.07=0.21
80.01=0.244
90.00+0.17
90.55:40.24
01.06:0:20
91.2940.19
88.97=0.40
80.98=0-42
91.32::0.2
Relaons-ssaae
97.24++111 1.11
0.35
99.37+0.40
00.44+0-30
97.06+1.4
93.10+2.50
100.00-0.00
78.81+1
50.51++0.3
Bank
90.48:20.15
91.14:0.00
87 37.3220.11 0.16
91.7320.01 04
90.0620.21
91.10:0.3 38
91.65:20.17
88 21:00.01
90.9620.10 16
92.08:0.18
Blood
35.15113.21
T1.98=3.52
75.
74.7543.21
73.881:5.11
66. 66.65.1100
74.8522..8
Car
98. 08.96:20..8 30
99 600 23
97 09:9520000 12
99.50120.3 31
99.25:00:55
97 96.9666.2222 28
99.91-0.04
99 90 04
98. 13:01120000 06
99. 70.07320000 18
Commmnities
03.5610.85
71.7315.11
85.25100.77
Credit-g
77.89
77 77.60=5.222 26
77. 67.22776
77.12:55.46
77.26:13.18 18
75.04:14.51
75.0416.32
77.40: 1.57
75.45
78.38 4.885
43.07+4.74
82. 92.73+5558
81.96+55.97
+6.28+6.22
81.43+7.38
83 13.43-+1.00
2.02
82.56+5..1
Encalypaus
91.64:11.10
90.85 1.33
90.50:
90.44:11.23
80.6611.97
85.61:1.51
80.85:10.64
90.80=00.02
85.4611.10
91.34410.90 9.
93.10+2.11
82.79-1.00
93.07+2.333
93.15+1.58
03.15+2..22
52.38+2.70
92.60+-2.16
92 02-00-1.553
93.07+2.000
933.55+
Junglechates
80.61#0.33
89.89 09.88:0.000
86 0.09220070
88.45+0.70
92.10=0.47
01.62400.45
02.95+0.12
90 60:00.777
85, 05.46200.11 19
93.43+00.3
Myneardial
61.20+53.13
60.90-4.97
06.11+44.05
60.43+3.35
00.29+3-96
63.27+44.35
62.06+33388
07.01-11.18
68.23 2.29
63.64+3..00
Sequencettyee
92.11=2208
96. 16.37-0.77
96.34=1.117
9T. 07.36
97.16:11448
92.90:1.115
97.41=00.33
100-00=0.00
93, 03.0024:11
96.44:1
Tic-tac-toe
99.3120.600
99.8120000
96. 1.35
98.0010.67
95.9811.87
9L07733.1 14
99. 99.80100.15
100.0010.00
35.801:2.17 17
99.52.10.3
Vehiele
94.82:0.50
96.16=0.83
92.37
08.0240.52
95.3210.49
93.55.11077
96.32-0.38
89. 00.20:::.25
92.11&1.11
96.2210:25
(b) Regression (RMSE)
Datasel
Raw Data
AutoEncoder
SinSiam
SCARF
STAB
Bike
142.36-1-1.5
126.90:4-1.0 1,02
121.59-41.550
111.674-2.08
126.42
Crab
2.21+-0.05
2.12+0.03
2.12-00004
2. 12:00..03
2.15 2.15++0.02
Forest-fires
75.07=33.28
81.21=229008
82.01+227.0 45
82,87=28.115
80. 13:333
Housing
09132.79H4
11132.79:41888
$6155.13::-111
59159: 18:462 79
56941 63:5199779 79
60071 29772977999
Insurance
5930.14::273.22
4611.
(611.788:220.22
4666 25:.2922222999
4657.87 87577777
4787.108 1797110:170.666
Solution--is
0.07:00.0
0.03:00.0
0.03-10.00
0.03.10.00
0.03110.00
Wine
0.73=0.01
0.69=0.00
0.69+20.0
0.69=0001
0.71=0.00
DaLaset
STUNT
LFR
FeatLLM
MET
Ours
Bike
115.98-42.11
121.02++2.43
87. 97.83+00 83-+0.92
111.46+1.72
Crab
2.13::0.04
2.16:0.02
2.1520.04
2.1310.03
Forest-fires
77.57:1:2.56
83. 83.84:226.81
T6. 76.36-288.
83.19::22.47
Housing
66151.344::32.44
58064, 28-1:312.777 2.73
60230.45:
56069.83:1:00
Insurance
5099.0321
0099.03:1333.72
4833.99 (033.99-293.68
1311189.72
1378.1411149.88
Solution-mix
0.071:0.0
0.02:10.00
0.08
0,02:10.01
Wine
0.67:10.00
0.69:10.01
0.69=#001
0.67:10.00
34

---

Published in Transactions on Machine Learning Research (05/2025)
K.2 Evaluation with Non-Parametric Classifier
Table 12: Evaluation results of self-supervised models on Non-parametrie classifier with (a) 3 and (b) 5 clusters, showing AUC across 16 datasets for classification. Best performances are bolded, and our framework's performances, when second-best, are underlined.
(a) Clusters
Datases
Raw Data
AutoEncuder
SimnSiam
SCARF
STAB
STUNT
LFR
FeatLLM
MET
Ours
Adult
82.3110.18
81.5320.10
80.51=0.11
$1.90:00000
82.21:0.21
82 0.000.0
82:28:0.29
81.3220.17
80.81:10.29
81 10.00.000000
Balance-saale
79.47:2.44
78.001100
82.67
78.6720.40
79.73.3.20
1.00
78.18
18.80.1..70
65.8743.00
79.7310.00
Bank
89.09:00114
89. 09.19.0.03
88. 0.22
88.8520..1
09.01
58 38.9610.11
89.16:0.11
87.58200088
58.3110.22
89.36=0.226
Blood
74.44=-4.44
71.33+3.40
31.33-4.07
69.78=5.00
71 43
74.67-377
72.00=4.166
72:07=1.89 89
70.44:+2.70 78
74.33+:3.06
Car
87.09:22.09
86.12:2.5
77.17:1.16
79.00
82.01 00:01::10 5.
82 02.85111.97
29.77
01.36:0.61
38.92216.04
89. 29.10:22773
Communities
63.07=42.25
02.32+-2445
08 2.116
00:000+110 83
63 37-57-0005
62. 62.24+1110 1004
62 -0.00
64.29900.89
16. 72.-2222.8
62 29.
Crolit-g
73.00=1.550
0..73+0.76
00:00=4.44
71883+3332
72.00+3.044
71.67=44.65
71 71.50=1.000
73.00=00.00
70.50+ 1.32
11:5061.77
Diabetee
73.16++4.96
62.94+6637
00:91
70.78=3..8 M6
74.24=1.55
74:03++4406
72.51=4.12
74.03+22.75
71.43+1.72
72.73+-4.00
Encalyptus
30.23+33.19
50.00+3.96
62.03=4.00
56:08-44.22
52.25++2.06
55.233-2.06
61.82=1.42
00:594-5.1
Heart
84.60=1.66
84.96+1.991
84.42=1.13
85.33==0.54 0454
85.61=1.76
85.055+22.07
84 44.60--22445
85-14#22.4
86.14 10.144-2277
84. 70.-70+1100 00
Junglechess
75.08=00.54
74.35+0.27
77.40-0.14
12.94=0.22
74.84+0.64
73.65++0.47
73-87=0.38
12.31+0.65
12.31++0.65
75.35++0.52
Myocardial
74.40:25.63
70.9142251
73.49:22.0
75.12:11.11
13.1
71.01::3332
74.64
71.20:5..77
71.29:5.66 67
72.22::2.74
Soquenosttype
00.00=-2.00
91.33+1.11
86-00-3.46
93.33=11.1 1.15
90.67+22311
9/ 20.000-22000
93.33-2.31
84
84.67-46.11
31.333-2.31
Tic-tac-toe toe
91.15:10.52
82.29::0.90
85.07=:2.41
72.40 2.
44.90111.50
96. 1.59
77. 77.95
86, 16.11
18.11:14.37
93.92:1.55
Vehicde
39.80122.66
63.512:4411
03.5111444
(b) 5 Clusters
Datases
Raw Data
AntoEnccder
SimSiam
SCARF
STAB
STUNT
LFB
FeatLLM
MET
Ours
Adult
83.17+00.19
12.48+0.12
01.77-0.1
22.00+0.11
83.17+0.11
82.93-0.35
83.15-0.37
81. 51.65++0.22
81.53+0.34
83.22-0.32
Dalanco-scale
$2.40::2.88
$2.40 122
86.67=0.46
20.20+2.400
31.60:33220
85.07=2.81
81.30=3.33
98.00+22.83
69.87455550
$1.33::0.92
Bank
80.40+00.22
40.48+0.14
80.13+0.16
80.54-0.04
89.37-00 0.26
00.47-0.27
88-08+0.10
80.05+00117
80.25+0.06
Dlood
74.67:44.16
74.4434.07
73.56-4.70
74.44:2.78
74.67=2.91
74.22-4.34
72.80=3.67
70.67 24
71.7842.66
73.7944407
2..5.
81.721192
83.09
91.331.11
Communities
65.58=11.16 1.16
64.55:
61 0.14
60. 1.57
63.41=11.00
65.25 05.25511.0 1.67
64.91 1.15
07.229:1.24
$6.47:422555 54
65 65.584:1.44
Credit-g
71.33-1.15
71.07
Diabelte
73.38 3:25
72.51:21.32 1.32
73 73.38:55.11
16:16::2220 86
72 91:11.3335
74.03:1.62
73.38 8,00
78.70:2:113
73.16:133 20
T1. 11.05::5..52
Eucalyptus
99.86.1..87
2.56
64.26.55.40
52.70-3.10
58.03
Heart
1..69:11.91
86.23:20
85, $5.09=1.555
86.
06.41:11.44
84. 06:00.0006
85 0.80
66.59:1.06
81.25:1:1225
89.33:13 56
84. 0LL006::11 .87
Tunglechess
05.23
75.80110.43
Myocardal
75.60:12.74
75.85:211. 82
733 73791
76.33:11.11
76.57=1.82
71.40-2.54
76.09-2.00 90
17.54:1:00
75.18:11.00 1.9
76. 16.1222:.33
Sep
86:00-3.299
30:00:33:46
93.33-2311
100.00=0000
86.00+8.75
Tietac-toe
91.10:10..8 80
81.38:1:1.
87 1.97
17.26:12.0 86
90. 18:22.87
97.7 0..70
82.12 82.12=1.08
100.00:00000
90.28:1:.37
95 01..31::180 1.88
Vehiclo
72.76:00.44
71.57=2:33
74.71=3.11
07.63++5.82
62.13+5.33
76.47-3.58
35

---

Published in Transactions on Machine Learning Research (05/2025)
K.3 Ablation Study with Linear Model
Table 13: Evaluation results of ablation studies on linear model, showing (a) AUC across 15 datasets for classification and (b) RMSE across datasets for regression. Best performances bolded, and our framework's performances, when second-best, are underlined.
(a) Classification (AUC)
Dataset
Top-1 selection
Random-I selection
Random fenture discovery
Wichout learning
Without feature selection
Ours
Adult
91.27
91.27 -0133
91.38 1-.88-0..1
91.65-+0.54
91.36-+0.13
91.32-00.1
Halance-senle
99.46++0.37
90.53-0.29
99.54-+0.22
99.96-+0.07
99.64-+0.22
90.51 +0033
Bank
92.10:0.112
91.96=0.20
91.99:10.22 24
89.8310.65
91.91.1.111
92.08.0.18
Blood
74.721:2.85
74.89
74.85
4.91
75.26:122
Car
99.65-+0.22
99.48-0.40
09.68-+0.11
98.05+1..7
99.81-+0.08
09.73-+0.18
Commonittiss
85.37+11228
84.74=0.44
85 $5.36+0.77 73
84. 19:11444
555.9++1.22
86. 86.25=00.77
Credit-g
78.23#5.11
77.97=5.89
77.53
71.18:1..1
78. 131-1.446
78.38+1.85
Diabetes
82,2015.36
82:23=5.19
82.6014.81
84.3323.89
$2.19:33.00
52.5625.12
Eucalyplus
91.1610.84
90.92=1.26
91.15:10.99
Heart
93.56 1..4
93.46-1.28
93.25
92.5011.63
13.47:11.48
93.45.1
Junglochess
93.37+00.28
92.07-0.29
03.3740.21
93.57+
93.43+0.39
93.43+00.31
Myocardial
62. 02.10+22.4
63.06=1.51
62.46:22.77
63.98-42.70
62.23#3331 70
63. 03.64+33.05
Sequence-gye type
96.58.00.90
96.8011.02
96.45.1..01
83.33.1.28.8
96.47.11.10
96.441.1.01
Tic-tac-toe
90.58+0.23
-3..55+-0.53
99.52-0.51
Vehicle
95.95+0.51
95.94=0.47
96.00=+00.56
94.00=0030
96.08 96.08++0.53
96.22::0.00
(b) Regression (RMSE)
Dataset
Top-1 selection
Random-1 selection
Random feature disnovary
Without lmurning
Without feature selection
Ours
Bike
122.00+1.8
111.27+1.88
111.09+1.75
740.74+657.85 86
111.68++2.16 18
1.44++11.72
Crab
2.1210.03
2.12-10.01
2.13-0.02
6.94.6.20
2:13.00:2
2.13-0.08
83.55-23.44
4-32.62
86.
83.19-222
Housing
26162.70:247.14
36251 2411333.66
35984 821-323.1
65521 90114922.1
13864.26.190.86
58069 8011-066
Insurance
1630.87::19..11
1567.01#116.99
1587 118.18333
11100.8:41111.80
4582.000
4678.1/st 1/1:1199..88
Solution-mix
0.02++0.01
0.02-+0.00
0.02-0.00
0.02-0.01
Wine
0.68+0.00
0.68=00.00
0.68=0.00
0.90+00.0 26
0.68+:001
0.67=0.00
36

---

Published in Transactions on Machine Learning Research (05/2025)
K.4 Ablation Study with Non- parametric Classifier
Table 14: Evaluation results of ablation studies on non-parametric classifier with (a) and (b) clusters across 16 datasets for classification. Best performances bolded, and framework's performances, when second-best, are underlined.
(a) 3 Clusters
Dataset
Top-1 selection
Random-I selection
Random fenture discovery
Without learning
Without feature selection
Ours
Adult
80.47
10.08
82.64:10.30
83.07 007--0.24 0.23
83.04-10.21
81.90--0.46
Halance-senle
73.07-45.45
70.20=1.39
80.00-:1100
86. 13--0.46
80.27+00.95
79.73+-0.46
Bank
89.0810.005
89.04=0.10
88 36.91:10036
88.76:10.99
88.71:10.19
89.36:10.20
Blood
73.33-1-3.00
76.00:1
72.67:13.55
Car
82.56+99.62
89.02-2.37
82.66+6.95
81.79+5.60
83.82-43.33
89.40-+22 2.73
Commonittiss
63.32+22227
64.24=2.04
62.744+0.22 29
63. 1..2441122
64.75+11116
12.16-2339
Credit-g
70.00+2.60
71.33=2.57
72.33+1.76
72.00=5.63
71.00=22.60
71.83=1.76
Diabetes
72.9125.52
72:29=1.32
71.9144..2
7..3
72.9446.14
72.73=4.90
Eucalyplus
36.5313.96
36.53112.8
5..5
Heart
8.76
84.24 2.72
84.05
86.7810.88
64.06-12.57
84.70
Junglochess
75.46+1.21
76.45-0.75
74.91+0.63
75.76+2.05
74.15+0.52
76.35+0.52
Myocardial
75.12:44.25
73. 19.1923.32
73.67#0.84
72.46=44.35
74.15+2233
72.2242.74
Sequence-gye type
90.001:3.46
39.33.2..3
91.3312.31
73.331129.14
92.00.135
91.33.12.
Tic-tac-toe
90.80+-1.20
91.49-1.50
81.25+-5.29
91.154-12.18
93.9241.58
Vehicle
71.76+5.79
70.59=5.79
72.75:44.42
74.1240.59
70.78+1.48
74.51+2.65
(b) 5 Clusters
Dataset
Top-1 selection
Random-I selection
Random feature discovery
Without learning
Without feature selection
Ours
Adult
82.64++0.15
82.90-0.13
83.43-00335
84.03-00.1
83.83+-0.22
83.22--0.32
Halanoe-seale
81.33++3.23
82.93-2.44
82.93+1.22 1.22
86.
$1,07+22.01
81.33+00.2
Bank
89.51=0.18
89.37=0.13 13
89.21+10
89.11=0.11 18
89 03.2320.000
89.25:20066
Blood
73.7842.11
75.33=3.06
73.56:23.7
76.00=3.06
72.6742.9
7..78=1107
88.8246.79
91.04-2.00
88.54433.47
83.6245.43
89.69++0600
93.7742.29
Communities
1..4
65.33=1.43
61.41=11.25
61.24=1.13
64.83+2.03
6..58:11.43
Credit-g
73.17=1.01
73.33+2.36
73.00:22.000
71.33=5.33
71.67=3.62
13.17=3.85
Diabetes
73.5915.67
73.39=3.33
74.2414..92
72.2914.61
73.16:24.770
71.65:15.52
Eucalyplas
38.5617.22
59.46
56.76.14.11
58.1113.770
62.39.16.89
63.16:13.10
15.14-1.66
85.87
8..06+1.37
Junglechess
16.31
75.30=0.31
75.73++0.60
76.94+1.00
75.05++0.41
75.80+-0.43
Myocardial
75,85=11.67
73.91=0.72
75.36+1.92
75.36=2.90
75.60+1.11
76.12=2.33
Sequence-type
91.33=11.16
91.33=1.16
90.67+41.22
73.33:129.00 18
92.00:1:.00
92.67:3.06
Tic-tac-tee
91.6220.60
92.53=2.10
86 $6.9815.002
91.15:111.0
93.9241.99
95.31:1.88
Vehicle
72.75+22.8
73.10-3.02
14.51+1.36
70.39-44.0
73.33+1.80
76.4743.53
37

---

Published in Transactions on Machine Learning Research (05/2025)
K.5
Informativeness of Discovered Features
Table 15: Analysis of the informativeness of features discovered via LLM. The average mutual information (MT) between features and the downstream task's labols reported for each dataset. The increase ratio in MI when using discovered features compared to original features also reported, along with standard deviations.
Data
Average MI in original feat ures
Average MI in discovered features
Increase ratio (%)
Tic-tac-toe
0.010
0.076
643.6:1535..1
Solution-mix
0.064
0.386
507.311348..2
Balance-scale
0.082
0.178
117.3#229.0
Wine
0.055
0.100
81.2+112.3
Bank
0.013
0.022
65.81172.0
Blood
0.031
0.045
44.6=95.2
Sequence-type
0.345
0.459
32.9186.9
Forest-fires
0.019
0.025
32.2+131.3
Bike
0.103
0.132
27.9+155.0
Car
0.036
0.041
14.2
Credit-g
0.009
0.009
8.8+136.5
Insurance
0.364
0.395
8.4=127.7
Communities
0.089
0.095
7.1485.2
Vchicle
0.212
0.226
6.8+56.5
Adult
0.031
0.032
5.3-123.1
Junglechess
0.049
0.051
4.8.73.88
Myocardial
0.007
0.007
3.1
Diabetes
0.043
0.045
3.0471.9
Heart
0.067
0.063
-5.1481.5
Eucalyptus
0.177
0.158
-10.9-86.9
Crab
0.350
0.254
437.5+43.7
Housing
0.154
0.102
-34.0+72.2
38

---

Published in Transactions on Machine Learning Research (05/2025)
K.6 Hyperparameter Analysis M
Table 16:
Evaluation results with various hyperparameter M on linear model, showing (a) AUC across datasets for classification and (b) RMSE across datasets for regression. Best. performances are bolded.
(a) Classification (AUC)
Dataset
M 10
M 20
M 30
M All
Adult
91.33+0.16
91.32:40.12
91.31=0.14
91.27+0016
Balance-scale
99.50=0.36
09.3
99.57=0.31
99.62+0.28
Bank
02.05
92.0811.118
92.01
91.8410.06
Blood
74.95
74.8542.89
74.95+2.80
74.7243.04
Car
99.77=0.13
99.73++.18
99.80#0.13
99.78+0.14
Communities
8..30
85.25+0.67
8..70+0.88
85.25+0.89
Credit-g
79.0715.94
78.3814.85
78.6615.36
78.08144.2
Diabetes
81.86=5.34
82.5615.12
81.9925.31
82.02+4.71
Eucalyptus
91.58=0.81
91.34+0.99
91.49+0.88
91.74+0.45
Heart
93.6911.26
93.4511.60
93.5211.58
93.4211.49
Junglechess
93.2910.29
93.4311031
93.6410.36
93.45-20.33
Myocardial
61.72=2.65
63.64+3.08
61.95=3.44
61.7742.43
Sequence-type
96.50+0.92
96.44+1.01
96.69+0.97
96.61+0.91
Tic-tac-toe
99.6710.36
99.5211051
99.5910.11
99.45110.56
Vehicle
96.08=0.46
96.22+0.28
96.12=0.52
96.15+0.52
(b) Regression (RMSE)
Dataset
M = 10
M = 20
M 30
M All
Bike
112.57+1.73
111.46+1.72
110.71+2.62
111.09+1.29
Crab
2.1310.02
2.13:10.03
2.13
2.14+0.03
Forest-fires
83.33+-33.21
83.19+222.47
82.17+22.78
81.14+23.21
Housing
56266.214248644
46069.83+400699
56047.02+125.19
56056.16+240.30
Insurance
1622.891173.26
4578.141144983
4615.54+160.49
4614.741196.02
Solution-mix
0.02+0.00
0.02+0.01
0.02+0.00
0.02+00000
Wine
0.68+0.01
0.67
0.68+0000
0.68+
39

---

Published in Transactions on Machine Learning Research (05/2025)
K.7 Learning with Other Objectives
Table 17: Evaluation results with various loss objoctives on linear model, showing (a) AUC across 15 datasets for elassification and (b) RMSE across 7 datasets for regression. Best performances bolded.
(a) Classification (AUC)
Dataset
Supervised Contrastive Learning
CLIP
Reconstruction
Cross-entropy
Adult
91.32:10.12
91.29:10.12
91.36:10.18
91.26:10.13
Balance-scale
99.5110.33
99.5110.22
99.65-10.27
99.5710.32
Bank
92.08:00.18
92.06:10.15
91.95:10.28
92.19=0.28
Blood
74.85+2.88
74.54+3.117
74.78:2..6
74.96=2.98
Car
99.73:10.18
99.74:00.17
99.79100.18
99.77400.11
Communities
85.25+0.67
85.53=0.42
85.33+1064
85.33+0.66
Credit-g
78.38=4.85
78.20:5.74
78.0745.23
78.31+5.36
Diabetes
82.56=5.12
81.60=5.45
82.33+5.227
81.59+5.47
Eucalyplus
91.34110.99
91.45.1.20
91,47-1..88
91,42110.66
Heart
93.45-1.60
13.4411.45
93.4011.447
93.4011.64
Junglechess
93.43-0.31
92.70110.08
93.0810.42
93.05110.29
Myocardial
63.64=3.08
13.3343.4
60.66:1.00
61.26112.00
Sequence-type type
96.4441.001
96.6311..77
96.4111.00
96.3110.78
Tic-tac-toe
99.52=0.51
99.4911.32
99.4710.25
99.3911. 42
Vehicle
96.22=0.28
96.1210.57
96. 96.15:10.48
96. 96.05:10.58
(b) Regression (RMSE)
Dataset
Supervised Contrastive Learning
CLIP
Reconstruction
Cross-entropy
Bike
111.4611.72
112.6612.16
112.56::2.08
112.87-2216
Crab
2.13=00.3
2.12:00.2
2.12::0.03
2.12:10.02
Forest-fires
83.19:222.47
87.95:124.07
82.04+22..0
82.45:1:22.55
Housing
4.0699883::06.999
46048.85-131.0
56381.03-248..5
Insurance
4578.144149.83
4615.23
4644.91=122.84
4572.54=159.84
Solution-mix
0.02#0001
0.02
0.02+0000
0.02
Wine
0.67=0000
0.68
0.68+10.00
0.67=10.00
40

---

Published in Transactions on Machine Learning Research (05/2025)
K.8 Experiments with Additional Datasets & Baselines
K.8.1 Comparison on OpenML-CC18 datasets
Table 18: Evaluation results of self-supervised models on (a) linear model, (b) non- non-parametric classifier with 3 clusters, and (c) non-parametric classifier with 5 clusters, showing AUC across 19 OpenML-CC18 datasets. Additional datasets not included in the main paper are listed at the bottom of the table. Best performances are bolded, and ourframework's performances, when second-best, are underlined.
(a) Linear Model
Dataset
Raw Data
AutoEncoder
SimSiam
SCARF
STAB
STUNT
LFR
FeatLLM
MET
Owr
Adult
90.75.10:17
91.07200.20
89:011:022 24
90.50.10.17
90.55:10.21
91.06110.20 20
91.29:10119
68.97.00
59, 59.98.10.12
91:32:10.12
Balance-scale
07.2441.11
99.58
09.37
99.44:0.39
97.6841.46 1.46
63.1042..0
99.25-0.30
10010.00
78.8141993
09.51:0.33
Bank
00.48.10.18
91.1110.000
67.32.0.116
91.73.0..0
90.0610.25
91. 91.1010.38
91.62.0117
68 68.2110.17
90.96110.16
02.08:00118
Blood
75.15:43.21 21
74.96=3302
75. 18--88-44 42
73.92:4-1.04
74. 74.75:33
74. 74.00--.88 88
7388 11
66.05 UT
7..87=1.61
T4.85++2.66 80
Car
08.95++0.00
99.60+0.23
07.95++0.42 42
90.50+0.31
00.25+0-43
97.96+0.25
99.91 00.04
90.9+0.04
98-43+00.05
00.73+00188
Creditgg
77.80-26.44
77.00
77. 077.612:222 27
77.12:45.40 46
77 177261::3113 18
75 1.55
75.04:66.3 32
7..4+1.57
75.45+1.77
78.38-44885
Diuhates
83.07744.77
61.64+6.16
82 73+-5.88
51.96+5.97
80.33 0-9+--.2
82.21 +14004
81.43+7.38
83.43+1.94
16.15+2.02
82.16+5-12
Eucadyptus
91.64::1.10
10.85=1.33
90. 10.500:11.80
1.23
60. 90.66++1.91
65.61=1.511
89.85=00.44
00.80++0022
85, 1.14
91.34++0990
Tunglechess
50.61:10.33
89.89110.19
66. 00.02.00070
4..00.70
10
01.02:0:44
92.90.00:12
90.61.0.3
65.46.10199
0..1310.31
Tic-tac-toe
09:31:+0.00
99.84=00.88
08, 1.35
90.00++0.67
06, 00.99++1.8
94.0743.14
99 99.80:
100:00.00
86, 06.80122.177
90. 09.52:00.51
Vehicle
01.52410.50
96.162.0.83
92:374:1.39
96.02.10.52
96.52.10.49
98.55.1.00
90.3220.38
89.210..5
12.11:1:12
22.00.2..2 28
Authorship
100.00:10.00
99.90100001
99. 0..900000008
99. 99.78.10.31
99. 09.08.10.02
99. 99.99.10.01
99.9810001
99 36.30.3335 35
99. 70.70.10:21
100.00.10.00
Breast-w
19.35:10:44
13
99. 002222000. 69
10:12:11:21
99.60=0.38
16::::2
88.0120.56
98 00:00.00000
29:10:4
99.45220.38
Canc
09.52.12.88 86
72.56.3.88
63.74133.4 11
63.37.13.32
73.03.33.10 10
66, 66.22153.57
65.06:13.15
7..27=0.47
70.58:11220 102
72 72.9011:: 101::1.7..
Cylinder-baudds
84.58:1:399
72.78=3:22
49 00.64:2:444 64
76.887:::.88
01.031144
66.966::112
85.92:4.005
16.7751:16 82
60) 60.03:::1500
33..96:::322 24
Dmnft
59:35++11.12
50.57+1.08
19.86++1.64
56.61++1 .8
60-67+1.22 26
65.67+00.00
63.45+ 2.44
56.58++0.40 440
76.95+11 1 .80
59.52++0 801
Dpd
77.73=33.34
3..06==3.51
73. 1.1
76.39+4-4.14
77 77.50+3.31
74.20+-2.67
76.12:4.66
13.35++.17
58. 4.:42.8444
77.53++2.42
Optdigitt
99 89:92++0.02
00.60+00.12
84.60+0.25
07.64++0.36
90 0000022
54. 34.36+0.42
90.88+0007
50 00-36++0011
BB 60++0.004
99 18.09700.01
Pel
62.45:43.53
59.92
74.2147.02
76.48+47.59
84.47+11449
78.07:44.00
76.98 00.88=117 17.95
83.66++2.52
86. 10..8&&11.00
87.54=2.66 2.63
(b) Non-parametric Classifier with 3 Clusters
Dataset
Raw Data
AunoErroder
SamStam
SCARF
STAB
STUNT
LFR
FeatLLM
MET
Ours
Adult
12.3110,11
81.53=0.16
80.51 41
$1.90:0.099
02.211:0:4
82,20:10.30
82.28
02.2811.22 29
8L.6210.17 17
80.81:0.29
$1.90:20.46
weale
79.17:2..11
78. 18.40=1.60
82.67
78.0710..0 16
79. 79.73:13.2 23
79.20:1160
78. 1..13:2.40 44
98.50:1.70
65.87:33.03
79.73:20. 16
Bank
89.00:00.14
89.19-0.03
88:41:40:255
88 861-.13
89. 11:00000
88.96:0011
89.16:40 11
87 07.5
88.34=0022
89.36:00.28
Blood
72.11::1.11
71.33_3.00 16
71.33.16.7
69.78115.00
71.331:1.67
74.67.33.7 71
72.004:433 16
72.6741.8 89
70.11:2.78 2.88
74.33-:3.00 3.66
Car
87.09=2.00
86.42-2.37
77.17=1.16
79 79.00:00775 73
82.01+1.50 50
82.85:11997
79.77=2.08 08
81 0.
88.12226.440 46
89.40:22.75
Choiltgg
73.00+1.50
73.33+0.70
60.00+44.44
71.83+3.33 33
72.00+33.04
71.67+4.65
71.50+1.00
73.00+0.00
70.50+1.32
71. 71.83-1.700 1.76
Diabetes
73.16:24.96
72.94=4.57
69.91=7.330
10.78:33.00 3.95
74.24
74.03:24.00
72.51::4.55 12
74.00 75
11.43:1177
72.73 4.90
Eucalyptus
59.28.13.19
33.60=3.96
57.58.12.77
52 02.0031:44
56.081-1.22
52,25 06
28.28112.0
61.62.11:45
58.28115.87
60.
Juaglechess
75.06=00.54
7.235=00.27
77.40:0.14
72.34 22
74.8440.64
70.65::0.47
10.87=10.38 0.338
75.21#1.44
72.31=0.68 65
75.35
Tie- toe
91.1510.52
82.29=0.90
85.0712.41
72. 22.22.12.7
81.9011.00
96.70.11.59
17.95.2.
99.48100.00
86.1124.337
93.922 1.59
Velicle
69 89.8022..6
70.1013.55
59 2.56
68 82:55.22
68. 1.:1 80
67.59:3367
74.31444.34
66. 06.18:15:41 41
G3.51 44
74.51 71.51::2.00 2665
Authorship
90.11:00.44
94.87=3336
38,66 38.66:t-4.08
87 67.7766688 83
09.21:10.34
99.6110.34
90. 90.7315.99
89.3510088 84
93.45:20.00
98 02.882::000 0.59
Breast-w w
95.95:12.51
91. 0.00
89 89.52.12118
95.95
95.9511.49
95,95111.5
91.52:1.80 19
96.7112.02 02
96.19.10.41 41
91.76:1 19
Cine
45.42:11.55
47.68
40.34:2.556
44 1.97
46. 16.2011:207
45.54:21.04
4388112
45.25:13311
45.12:2.37
45, 3.11
Syllnde-bbudss
64.20++4.67
69.21 2.821-2.83
M1.93+0..3
72.84+1.41
78.09+5.48
60.40+2.98
17.47+3.74
71.30+1.31
74.38+4.57
74.69--4.18
Dmftt
17.71=33.08
18.33=1.30
18.33-40.36
15:00+11.30
16.67+1.00
19.3842.17
17.50+3.12
16.5620.44
15.72=2.74
17.71=2.01
Dpt
69.66-+0-49
64.30-4.04
61.97+6.066
60 60.80+11.00 1.31
67. 07-2444-.93
70.09+1.48
67.52+33.08
70.05+2.42
69.38+5.13 13
65.81 +3.08
Optdigits
97.84:10.2
9..58=0.62
12.71
77. 07.707
08
19.12:0.81
97.75+0.52
88.4810.44
90.51=2.06
97.12:0067
Pel
93.54:11.04
93.54_0.699
90.09400.78
1.89
93.0910.52
93.54110.94
92.49100.99
93.47422.25 23
93.9911.71
93.69.0..0
(c) Non- parametric Classifier with 5 Clusters
Dataset
Raw Data
AusoEacoder
SanSioan
SCARF
STAB
STUNT
LFR
Fea:LLM
MET
Ours
Adult
83.17:10.19
22.18:10.12
811711031
82.6010.11
83.1720.11
82.8810.35
83.1510.37
81.6520:22
81.58:20334
83.22:00.3
Balsnce-scale
82.40=22.88
22..0:22.12
86 66.0740.46
23-2042-40
81 60H-3-20
80.044281
81.33+3.33
98.00+2.88
60.87=5.50
81.33=0011 0.2
Bank
89.40++0.32
80.48+0..14
88.85+0..2
58. 13.13+0.16
89.54-0.04
$0.3740.26
80.47+0.27
88 18.08+00.10
89.05+00.17
80.25+0006
Blood
74.
44.44=44.07
73.56+44.73
74.44+2.78
74.67-2.91
74.224-..34
72.80-33.67
73.67=t4.24 2.24
71.78=2.62
73.78+:4.07
Car
89.69+00.83
14.90=1.01
78.13+1.77
85.07+1.00 .44
88.54+2.55
98.54+22.77
84.78+1.92
83.00=0..00
91.33+4,26
93.77+2.29
Credit-g
72.33=2..57
74.8310.29
71.1744.54
70.50:13.61
71 11.80122000 93
71.35&1.155
71.6741155
20.00::2.12
7..50=0.61
73.171:3.88
Diabetes
73.36.13.25
12.51:11.32
13.38.15.15
73. 23.16612.44
72.9 72.911133335
74.0313.62 62
73.38813.99
73.70.:4.13
79.16133.00
71. 21.6515..5 5.52
Eucalyptus
59.46:14.887
7..96
58.33-12.56
53.60113.47
$4.28:55.46
52,70:13..10 10
58 16..33:33.90 96
60. 2.67772
56.01
63.16:13
Junglechess
75.20.10.12
75.28.10.00
76.9310.57
71.0110.51
76.09110.61
75.67.10.3
75.2511..5
75.97=1.62
72.9910.51
75.801.0.00
Ti-tac-tue
96,10:20.80
$4.88:1.0
87.53:11.97
77.26:1: 46
90-45:2:2.45 87
07.74:10.60
82 82.12:t:100 08
100.00
00.28::44.37
96.31
Vehiele
72.76+1..70
75.40+0..0
60.39+0..34
70.39+33.20
72.75+0034
71.6742.23
74.71+33.11
67.63 5.82
62.13+5.33
76.47+3.58
Authorship
99.11:10.31
97.24:11.33
10.01.1.81
88. 18.15.1
99, 11.1..90..9
99.61:10.34
87. 67.97.11..55 88
88, 7..76611.84
93.45:11.08
98. 66.6220.34 .11
Hreast-w
95.46:20.41
94.76:20.41
$8.06:11.65
16.71:11.24
961-48:1.89
19::2.0:000
06.71:00.71
15.71:1101
96:19:4441
20.:29:11 1:24
Cme
45.08+1.55
47.34+1.99
40.56++1.00
46. 16.1043.44
45.42+1.11
45.31++1.00
45.65++-3.39
46 10-0044
47.12+1.22
48.02+1.90
Cylnnder-bands
62.65:2..14
68.21144.6
52.7843..34
70.9942.33
79.32=2.96
62.35:1-4.28
75.001:3.34
18.06:1196
1..9
71.91=3.25
Dirft
15.62+3.25
19.38+2.25
18.75+2..17
17.08+3.08
18.1243344
19.17+2.01
18.12444.38
16.56+2.21
17.19+0.96
17.71 1-71+++11 1.91
Dpd
66.96:15.49
65.53-14.93
62.96:5..56
69. 18.23422.6
65 08.099:2.10
72.36:3.00
08.09422.75
5..37
67.52=0.00
68. 08.06:23.44
Opidigits
97.78.10.18
95.02:1.000
33.5011.11
79.12.10.59
97. 17.48.1..0 40
79.18.10.50
07.6611..00 60
88. 88.97
90.18.11.55
97. 97.21
Pel
92.79:20.78
90,29:0..2
91.11:11.38
03.24:11.35
93. 03.248:0.78
92 26.
98. 54.50.00000
94.37-00322
9.55
95.399:00220
41

---

Published in Transactions on Machine Learning Research (05/2025)
K.8.2 Comparison with additional baselines
Table 19: Evaluation results of self-supervisod models on lincar model, showing (a) AUC across 23 datasets for classification and (b) RMSE across datasets for regression. Best performances are bolded, and ourframework's performances, when second-best, are underlined
(a) Classification (AUC)
PCA
Random
LightCDM
Adult
89.1410.16
83.6010,21
02.9510.22
9..35
92. 02.91110.15
93.2410.15
88.7310.54
92.9610.02
91.32:10.12
Balanco-seale
07.24440.91
71.20:5-.81
91.87+00.77
81.80-1.3 100-1.22
91.80-00.74
00.97:00.04
09.98-0.02
93.60+2.85
90.51+00333
Bank
86.5620.43
99.57.0.11
96.0410.110
Blood
2.66
12.18:12.91
1.766113.76
66.94 4.95
70.71:4.19
72.92:13.45
T4.57
68.56116.00
74.85.12.89
Caz
90.85 01
67.1919.52
99.922:0.11
99.65=0.21
99.97110.4
99.21:20:25
99.70:0.00
17.88:11.37
99.73:10.18
Commmnities
84.7410.52
80.0411..34
85.75.10.22
0.344
85.63.10.44
84.90110:13
85.01
65.6610.355
85.2 05..5
Credit-g
73.26 13
$6,21
5.641=5.61
78.67= 1.05
78. 10:2 61
71.27=41.76
17.25 6.00
69.55
Diahetes
80.45.12.00
81.75.14.77
Eacalyptus
68.16-1.35
50.85++11. 1.34
90.76+-0.62
00.82 0.78
00.17++0.24
91.16++0.41
93.23 0.57
60.81 +-2.3
Heart
02.39
91.9411911
92.8912.118
93.96.11.11
93.43.1 62
93.17:1:2116
93.29911.56
92.1 02.1110998
931.4..110
Junglochess
60.61:0.27
1.33
97.73=0.10
90.75=0.21
07.1
86.80-+0.23
91.75:1-0.8
05.04 1.58
913, 00.43
Myocardial
63.15:55.45 48
51.5213.67
62.0313.02
66.06-1661
63.75.10.. 88
63. 61.41333.09
70.14:2.25
63.28
611 6.1.616 61.61:33006
Sequenco-ype
69.96113.14
95.87+12.32
2.02
96.70:12.18
05.29-12.51
76.00144.0
96.44.1..0
Tic-tac-te aae-
71.8813.95
66.5516.83
99.9220.12
09.05000005
99.92-10.10
99 88:00.02
99.90 09..0000.01
98.74211.25
99.52210.51
Velicle
86.48:10.79
86.7010.63
93.10100.99
03.11
93.00.10.47
92.67.10.33
96.18:10.61
84.9012.117
26.22.15 08.22.10.28 0..28
Authomshi
100.00:10.00
99.9 00.0000.008
09.9410003
99.97=0.01
09.96.10.00
91.98110001
100.00.10.00
90.5311.67
100.00 0.000
Breast-w W
09.55:1:0.45
10.50
98.87:10.65
99.19-0.62
98.68-10.88
00. 08:0055
99.60=0.42
98.45:: 0.33
Cue
69.51422.39
60.9811.67
69. 197
68.74-1.58
70.11 1.9
73.21:13.45
74.14:22.54
55.25
72.99::1.77
Cylinder-hande
71.17:10.9
55.11
92.12:33.63
90.53 3.58
92.02.1 1.22..1.72
90.66-13.20
$3.08-10.99
$1.8811.66
$1..3611..24
Dude
68.15:101.87
59.00.8.81
5:.877422225
52.65 2.80
56.3661 80
39. 39.01120.85
58.56 38.56:11.75
16.25
59.52:0.88
Uldd
77:001:2:20
73.5215.16
75.79:14.20
44.94111.11
77.53.12.44
Optdigits
98.75:20.12
155
99.95 19.95220.02
9.09
99. 96:00660000 02
99. 09.99200001
99. 19..3320002
93.36200.67
99.97:0.01
Pel
60.96114.69
73.0418.14
86.7316663
85.85 1.96
88.019.1..00 70
84.71.13.33
91.691144
57.3328.26
87.54.12.66
(b) Regression (RMSE)
Dutuset
PCA
S-SNE
XGBoost
Random Forest
LightGBM
Bike
142,63:: 1.35
+78.41=00377
11.30=1-43
41.11+1.3 39
42, 42,90+1 1-5
Crah
2.25-+0.04
2.70+-0.07
2.33-+0.06
2.23++0.01
2,23-+0.02
Forest-fires
74.16+229.94
74.62+330.6
90. 90.70-4-41.0
$0.72428.60
77.95+426.25
Housing
702622.25.1417.55
101805.39
47142.65 (1.44..5111 1.21.8
48536.76.1310..
47589 19..99966... 596.00
Insurance
6181.51=2012.66
12061.28::34..1
50MM 126116688
4635. 0055222=187.00 933.
48226. 001:170.12
Solution-mix
0.07=0.00
0.17+0.02
0.08+00.00
0.08++0.0
0.07+00.00
Wine
0.75:0001
0.8320.01
0.71
0.60#0.01
0.6120000
Dataset
EBM
TebPEN
TP-Berta
Ours
Bike
14.53+11443
17+0.40
111.46+1.72
Crab
2.18.10.03
2.19.0..3
2.13
Forest-fires
74.36:230.47
108 69..9110212
83. $3.19:222.44 47
Housing
47782.57.11122.
49205. 93.134000000.
56069.883 18.1184466499999
Insurauce
4177.76=179.99
4519 13..00
4578. 198.144:149..8
Solution-mix
0,05:100.0
0.12:00.11
0.02:0.01
Wine
0.68:+0.00
0.71+0.00
0.67++0.00
42

---

Published in Transactions on Machine Learning Research (05/2025)
K.8.3 Comparison with Other Classifiers
Table 20: Evaluation results of various classifiers on raw data and on our model's embeddings. Bolded values indicates that our model outperformed the raw data.
Linear Model
XGBoost
RBM
MLP
TabNet
Dotasel
Adali
90.75=00117
91.32:00112
92.8640.00
09.6920007
93.21:0 15
91.11=0.12
88.0110.07
88.89=0.02
87.20:2088
87.92:0.20
Balance scale
97.21
99.5110.333 33
91.8710.27
96.86....0 03
99.9740.00
95.2112
90.9110.08
09.9
95.2143..00
96.431:1.00
Bank
90.18:0.18
92.0810.15
95. 88.1840022
90,9020.00 80
95, 03.22200.1 13
92.14:0028
90.0920.22
90.75=00.1
89.17+1.02 0.
88, 23..0000.50
Blood
75.15
74.85.2
60.6643176
67.411:667
72.92433 45
74.6913.222
73.8242
73.95:13.36
70.552:4.45
71.9513.85
Car
98.86:20.90
98.7510.18
99.9240.11
99.3920.13
98:2140:22 25
99.82::0.09
90.9920.00
100.00:0.00
100.00:0.000
99. 00.952:0.05
Communities
85.25100.6
85,75
81.0740.62
84.9011..3 33
85.5410.88
81.9741.11
12.114:11388
80.511
79.0110.73
Credit-g
77.80=6.44
78.38:14.85
79.41:5.61
75.95:45.37
77.27 4:4.7
78.15+66.36
75.90:144
77.41:: 77.41::4.445
65.31
6..40:4477
Diabetes
4.707
82.56115.1
79.14:11.92
80.95.1..00 96
83.39 69
81.6111.11
80.14.16.
75, 9..0
74.99
7..22.3.79
Encslypens
91.64:1.110
91.3440.99
90.76:0..2
89.08:1.35
91.16:0044
91.14+1.07
92.48:0.51
92.91:0.44
87.15
88,95
Heart
93,10 12
93.45.1 1.00
92.89.2.13
92.65.10.99
93.174:2 16
93.0911
92.6111
90.46.10.91
89.6110.70
Junglochess
80.61=0.33
93.43-1..31
97.73++0.10
96.6110.12
86.80++0.22
95.75+0.20
98.04:50.09
98.90:10.04
08.16:0.81
0.
Myocarcial
61.20:55.13
63.64133.00
62.03:13.02
63.08114.70
63.44::3.00
61.33
50.49122.0
62.441:11.04
60.1242.92
35.62.1.88
Sypeence-type
92.11=2.03
96.44:1.00
95.87:+2.52
98.16 1.27
95.29:42.51
98.35+1.11
9 1.26
97.18:10.52
94.45+3.02
89.314:5.26
Tio-tao-tee
00.01
99.52+0.51
00.922+0-12
00.29+0.21
90.888+00002
90.82+00.2
90.96++0.04
99.997+0.02
98.300+0030
99.71 000.71++00 +00.99
Vehicle
94.52=0.50
08..22:0.222
93.10:10.99
94.58:0.55
02.67:00.33
94.80+0.24
96.25 26
96.40:10.38
95.12::..1
90.8240074
Authorslip
100.00:0.00
100.00:0000 00
90.94:20003 03
90 00.922:0000
90.98
99.99+0.01
90.99120.01 01
99 90:00000001
98.30: 78
99. 09..62:0022 0.224
Presstww
90.55-0.44
00.45++0.38
08.87+00.65
99.27+0.60
98.18+0.55
08.97+0.911
90.10++0.66 0.64
1-+00
97.83+0.97
97.004+22000
Cuu
69.52-2.85
1.
60 00.4222.44 47
68 08.96:22.22 21
73.21 45
74.46:2.885
72.49:22
70. 70.700::0000 997
66.94:33.1 18
69. 20/233:22 2.96
Cylinder- bands
Dmff
59.35:21.12
09.52++0.80
64.87:422 225
53.21 86
50.04::0 85
60.42:+1.24
59.0992:00
68. 56:066000000 63
56. 13:22:111
53. 13.11:::2.8
Dpd
77.53.12
78.831:2.45
70.9511..1
Optdigile
29.92:0.022
99.97:t0.01
99 00..9-2200000 02
90.90:20.00 04
00..93:2-0000 01
90.997 01
90.988-20 01)
99 94:094:0000 04
98.87 20000
99 0o..65-2..11
Pel
87.5112.663
81.7113..5
51.19:7.558
77.62:1..55
62.071..225
43

---

Published in Transactions on Machine Learning Research (05/2025)
K.9 Impact of the Number of LLM Trials
Table 21:
Evaluation results with various numbers of trials on linear model, showing (a) AUC across
datasets for classification and (b) RMSE across datasets for regression. Best. performances are bolded.
(a) Classification (AUC)
Number of Trials
10
20
30
40
Adult
91.32=0.14
91.31+0.09
91.32-10.08
91.29:10.08
91.321:0.12
Balance-scale
99.67=0.20
99.66=:..22
99.58=10.30
99.53=t0.33
99.51=:0.33
Bank
91.98++0.15
92.06+10.88
91.91++.299
91.96:+0.22
92.08:10118
Blood
74.8142.84
74.9412.99
75.1412.98
74.84+2..0
74.85+2.89
Car
99.56:0.24
99.5710.22
99.7310.13
99.75-0.14
99.7310.18
Communities
85.27=0.52
85.13+0.64
85.01+0.95
85.18:0.799
85.25+0.67
Credit-g
77.55+4.774
78:2145.70
78.77=4.80
78.60=55.35
78.38+44885
Diabetes
82.07+5..48
81.9445.38
81.76++..16
81.71+5.54
82.56+5.12
Eucalyptus
91.12+00.9
91.56+0..8
91.71+0.71
91.32+1.01
91.34+0.99
Heart
93.69-11.37
93.9811.57
93.52++1.29
93.55-+1.27
93.45+1.60
Junglochess
93.45:+0228
93.7010.40
93.4810.32
93.60+:0.39
93.43+0.31
Myocardial
61.1543.08
61.6742.08
64.05-2.48
64.03+2.35
63.6413.08
Sequence-typp
96.50=0..73
96.30=1.03
96.66=1.09
96.50=t0.71
96.44+1.01
Tic-tac fic-tac-toe
99.73-0.18
99.63:10.22
99.52:1.54
99.34+0.39
99.52+0.51
Vehicle
96.17::..4
95.96
96.04+0.29
96.07=10.49
96.2210.28
(b) Regression (RMSE)
Number of Trials
5
10
30
Bike
111.39++2.14
111.60++2.09
111.10+1.76
11/438+2243
111.46++1.22
Crab
2.13++0.02
2.13+0002
2.13+-0.03
2.12--0.02
2.13+00.03
Forest-fires
82.46122..6
81.84422.78 78
81.56+23.50
81.75:121.90
19:22.47
Housing
$5959.59:11 258.338333
56001. 136.1.28:11..22
40124.43:+1002
56069. 56069.833+4006,
8314406,99
Insurance
23.2.6
4618.29+179.44
4576.85=156.01
4598. 146.246.1
4578. 149.88
Solution-man mix
0.02#1.00
0.02#0000
0.02+1000
0.02=10.0
0.02#0.01
Wine
0.68
0.68+0.01
0.68+0.00
0.68+0000
0.67+0000
44


Reviews:
[{"review_id": "lPEifjwRqS", "reviewer": "Reviewer_RxCJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- I believe this paper is tackling an important problem (identifying features from unlabeled data based on a text description), and utilizing recent LLMs to address this problem sounds like a natural solution.\n\n- The experimental evaluation is solid, mostly positive, and comes with a detailed ablation study. I have not worked on any of the baselines so I could not verify the details, but from a quick glance I have found the analysis to be rigorously conducted.\n\nWeaknesses:\n\n- The proposed approach appears to be ad-hoc, and there is no rigorous justification as to why or how it works. In more detail, the overall approach is presented as an illustrative pipeline figure (Figure 1), and there is no algorithm box, no clear description of what's the input and what's the output for this problem.\n\nThis ad-hoc nature limits the generalizability of the proposed approach.\n\n- While the improvement in the experiments appears to be convincing, I do not get a good sense of why this improves over the baselines (such as SimSiam and other SSL methods). \n\nThis limits the current contribution of the paper mostly at a data-engineering level, lacking a more solid treatment.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors consider the problem of self-supervised representation learning, in particular, focusing on utilizing the downstream task description and the meta-information of the data to discover features relevant to the task, and then, treat the discovered features as ground-truth labels to perform task-specific representation learning.\n\nProblem formulation: The authors consider an unlabeled tabular dataset with $d$-dimensional input features. Then, a downstream task description in natural language, and the names and short descriptions of each feature are provided.\n\nThe model aims to train an encoder that extracts data representations to tackle the downstream task in an unsupervised setting (i.e., no ground truth labels are provided).\n\nExperiment: The proposed approach is evaluated across multiple tabular datasets with various downstream tasks.\n- The evaluation uses a total of 22 datasets.\n\n- The baselines are all trained in an unsupervised manner as in the experimental setup, including using Raw data, AutoEncoder, SimSiam, SCARF (an augmentation-invariant embedding approach), etc. Compared to all of these baselines, the paper shows that their proposed approach can outperform them in most cases.\n\n- The implementation uses GPT-3.5 as the LLM backbone for feature discovery.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In summary, while I see potential in this paper as publication in TMLR, the current writing falls short in the bar as required by the reviewing criteria. Specifically, the following areas can be improved to move towards that:\n\n* Problem formulation is unclear: The authors need to work out a more precise problem definition. Then, the proposed approach should be laid out in more precise language.\n\nAfter that, within the problem setup, the authors need to clearly describe **under what conditions are the proposed approach expected to work well or not.**\n\n Without that it's difficult to evaluate the scope and the accuracy of the claims made by this paper.\n\n* Writing lacks clarity: While the paper is mostly carefully written, there are plenty of room for improvement, and there are gaps in the logic of the writing. In particular, I think the authors need to put in effort to improve the quality and the presentation of their work.\n\nHere're some examples (and there're more in the paper):\n\n- In the abstract, the authors introduce TST-LLM, without giving the full name of this acronym. In particular, it's not clear what TST stands for.\n\n- Similarly, the authors refer to the baselines including STUNT and LFT, but did not describe what they are in the abstract.\n\n- On page 3, the authors discuss \"in-context learning-based prompt engineering\" --- I find this phrase to be particularly confusing. In-context learning usually does not include prompt engineering --- It might also involve prompt selection, but the description here lacks precision.\n\nIn summary, I think the paper requires proofreading and appears to be fairly rough in writing.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KkIzWEDjoB", "reviewer": "Reviewer_D9b8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The introduction effectively presents the problem but lacks an overview of the document’s structure and concrete examples of applications within the experimental setting.  Overall, the paper is well-written and well-presented, with a clear methodology. However, incorporating a running example would further clarify the workflow for readers. The experimental section is thorough, covering various aspects, including ablation studies and comparisons with baselines and alternative methods. To strengthen the analysis, I recommend adding a baseline using raw data combined with traditional feature reduction techniques (e.g., PCA, t-SNE) and evaluating additional models such as decision trees and ensemble methods (Random Forest, XGBoost, LightGBM, Explainable Boosting Machine). This would help assess whether the augmented feature space benefits state-of-the-art models for tabular data. Lastly, the conclusion is quite minimal and does not discuss potential directions for future research, which would be a valuable addition to the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces TST-LLM, a novel approach to representation learning that leverages features discovered by large language models (LLMs) without relying on ground-truth labels or label statistics. This method is designed to solve task-specific pre-text tasks through a two-step process. First, the target task’s textual description, feature meta-information, and text-serialized unlabeled data are used to construct prompts, which are then fed into an LLM to extract task-relevant features. These discovered features serve as pseudo ground-truth labels, defining a pre-text task that is optimized through supervised contrastive learning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See Strengths And Weaknesses.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FFL2Zbkg7q", "reviewer": "Reviewer_zHjZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS:\n\nThe work is well-documented; figures are useful and appropriately placed. The setting of tabular classification and regression, although extremely well-studied, remains highly relevant, and the attempt to integrate LLMs into the process is timely. The authors are to be commended for including many strong baseline methods; however, that commendation comes with a significant caveat (see req'd changes).\n\nWEAKNESSES: \n\nSee below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents TST-LLM, a novel framework for tabular self-supervised representation learning. By defining pre-text tasks in a task-specific manner using natural language descriptions, TST-LLM claims to identify and combine relevant features, boosting performance on downstream tasks compared to strong baseline methods.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. An appropriate choice of baseline comparison models is challlenging for this paper, because the different components of the model are operating with different degrees of awareness of the data. While the authors present this as an unsupervised/self-supervised method, and the LLM Feature Discovery and Self-supervised Pretraining components do not rely on labels, the authors use all of the training data labels to fit the final (linear or non-parametric) classifier in the main paper experiments.\n\nHowever, the strongest baseline methods the authors have chosen to compare to include;\n\nSTUNT: Designed for few-shot learning scenarios\n\nFeatLLM: Focused on few-shot tabular learning\n\nLFR: Development was motivated by scenarios with limited or noisy labels\n\nSCARF: Development was motivated by scenarios with limited or noisy labels\n\nEssentially, there is a mismatch between the authors' experimental setting (which ultimately requires full labeled datasets and limits the choice of classifier architecture) and several of their chosen baselines (few-shot/limited data methods, many of which rely on particular classifier architectures). This could potentially paint an overly favorable picture of their method's performance.\n\nTherefore, I would like to see comparisons to general-purpose tabular learning methods, such as XGBoost, CatBoost, TabPFN, XTFormer, etc, with 30 iterations of HPO (for those methods which require it) and parameter search ranges sampled from recent literature (https://arxiv.org/abs/2305.02997), added to the main paper. The authors should feel free to use either linear or non-parametric classifier for these experiments.\n\nIn addition, I would like to see comparisons to strong self-supervised tabular baselines which are designed to work in the large-data regime without label noise, such as the recent TP-Berta (https://arxiv.org/html/2403.01841v2).\n\n2. The authors have also chosen a non-standard benchmark suite; \"a total of 22 datasets to ensure a diverse range of downstream tasks in terms of size and complexity\". This choice is insufficiently motivated by the paper. Non-standard benchmark suites make it unnecessarily difficult for readers and reviewers to understand the authors' overall contribution in the context of prior work, most of which, in this case, has consistently used OpenML-CC18. Therefore, I would ask the authors to redo their experiments using a standard benchmark, ideally OpenML-CC18, or at least a subset consistent with prior work to ensure consistency, including those requested in (1). This can be an appendix result, and the authors need not show SOTA performance.\n\nBoth (1) and (2) are critical to securing my recommendation for acceptance.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: There was a majority opinion leaning accept on this paper, and I tend to agree with those reviewers. This work is interesting and presents a well-motivated and reasonable approach to solve the problem of SSL on tabular data. The experimental analysis has become more complete with the help of constructive criticism from the reviewers, and the writing in the paper is now of sufficient quality for TMLR.

I am accepting the paper with the following revisions:
1. Please bring the results of Appendix F.2 up to the main paper.
2. Please provide better justification for the choice of your tabular benchmarks in the main paper.

As for certifications, no reviewers suggested any.
Decision: Accept (poster)