LLM-Guided Self-Supervised Tabular Learning With Task-Specific Pre-text Tasks

Authors: Sungwon Han, Seungeon Lee, Meeyoung Cha, Sercan O Arik, Jinsung Yoon
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Self-supervised learning, Representation learning, Tabular data, Large language model

Abstract:
One of the most common approaches for self-supervised representation learning is defining pre-text tasks to learn data representations. Existing works determine pre-text tasks in a "task-agnostic'' way, without considering the forthcoming downstream tasks. This offers an advantage of broad applicability across tasks, but can also lead to a mismatch between task objectives, potentially degrading performance on downstream tasks. In this paper, we introduce TST-LLM, a framework that effectively reduces this mismatch when the natural language-based description of the downstream task is given without any ground-truth labels. TST-LLM instructs the LLM to use the downstream task's description and meta-information of data to discover features relevant to the target task. These discovered features are then treated as ground-truth labels to define "target-specific'' pre-text tasks. TST-LLM consistently outperforms contemporary baselines, such as STUNT and LFR, with win ratios of 95% and 81%, when applied to 22 benchmark tabular datasets, including binary and multi-class classification, and regression tasks.

Decision: reject
Reviewer scores: 3, 3, 5, 5

Reviews and discussion:
<thread>
creation date:
1730577919184

thread_type:
official_review

<thread_object>
creator: reviewer_kgqm

<content>
summary:
This paper introduces LLMs into self-supervised learning for tabular data. The authors propose the TST-LLM framework, which uses LLMs to generate task-specific pre-training tasks based on natural language descriptions of downstream tasks. The method consists of two main steps: first using LLMs to discover relevant features based on task descriptions, then using these features as supervision signals to train representations. Experiments on 22 benchmark datasets show significant improvements over existing methods.

strengths:
1. The method is well-designed, including multiple components like feature discovery, feature selection, and multi-task learning, forming an end-to-end solution
2. Thorough experimental validation with comprehensive comparative experiments and ablation studies across multiple datasets

weaknesses:
1. Limited novelty. The core idea is essentially using LLMs for feature engineering with prompt design, then using these features as self-supervision signals. Using LLMs as feature generators has been explored in many other fields - this paper simply applies it to tabular data.

2. At the methodology level, the framework is quite straightforward and lacks deep technical innovation. The feature selection and multi-task learning components use basic methods without proposing new improvements.

3. Lacks theoretical foundation. There's no analysis of why LLM-generated features help improve performance; no guarantees on feature quality; the whole method feels more like an empirical attempt.

questions:
Please refer the weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730714387098

thread_type:
official_review

<thread_object>
creator: reviewer_cmx7

<content>
summary:
This work focuses on learning semantic representations for tabular data, specifically tailored for downstream tasks. It aims to narrow the gap between representation learning objectives and downstream task objectives. Specifically, the paper proposes a two-stage procedure: (1) applying existing models (GPT-3.5) to extract new task-related features, and (2) using pre-text training to predict the values of these new features as the target. In contrast to existing methods for learning semantic representations over tables, this work aims to solve the mismatch between pretext tasks and downstream applications. The experimental results demonstrate the superiority of the proposed method.

strengths:
- The idea of using existing models to extract new features is interesting. It can be used as feature augmentation for tabular data.

- The proposed method shows superior performance to baseline methods employed in this work.

weaknesses:
- The proposed training method tailored for specific downstream task limits its application into broader spectrum.

- Lack of comparision against existing pretraining methods on tables (e.g. TUTA, TabLLM, XTab, TabPFN). As the model is trained with the specific objectives tailored for downstream task, the learned representation in itself demonstrates better performance against other representation learning methods. Adding comparision against other existing pretraining methods will provide a thorough examination. 

- The small-scale training restricts the broader applicability of the proposed method. The training datasets are limited in comparison to existing self-supervised training approaches.

questions:
- Whether or not the author(s) will provide a comparative analysis against existing self-supervised training methods like mask-and-predict approaches?

- Does the author(s) plan to scale up the model? In the paper, the trained encoder is a two-layer MLP.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730876204229

thread_type:
official_review

<thread_object>
creator: reviewer_tghy

<content>
summary:
The paper introduces TST-LLM, a framework that improves self-supervised learning by aligning pre-text tasks with downstream tasks. It uses task descriptions and data meta-information to discover relevant features, treating them as ground-truth labels. TST-LLM outperforms methods like STUNT and LFR on 22 benchmark datasets, achieving win ratios of 95% and 81%.

strengths:
1. The method is well-explained and easy to understand. The detailed description of how features are discovered and integrated into the learning process is commendable.

2. TST-LLM shows superior performance compared to the baselines. The model utilizes features from the LLM and achieves high performance across various datasets.

weaknesses:
1. While the paper compares TST-LLM to several self-supervised learning methods, it overlooks some highly competitive supervised learning approaches (e.g., XGBoost [1], CatBoost [2], TabR [3], FT-Transformer [4], ModernNCA [5]), as well as general learning methods like TabPFN [6], Tp-Berta [7], and XTFormer [8]. These methods, along with LLM-based feature engineering techniques (CAAFE [9], OCTree [10], FeatLLM [11]), are already highly effective in tabular data tasks. A comparison to these methods would be necessary to justify the need for self-supervised learning with TST-LLM.

2. TST-LLM seems to mainly rely on features generated by LLM-based methods like CAAFE [9], OCTree [10], and FeatLLM[11] and applies self-supervised learning on them. This approach does not offer significant novelty in comparison to these existing methods, as it mainly repurposes the generated features for a different training objective.

3. The ablation study does not fully explore the potential of automatic feature engineering methods like OPENFE, which can generate a large number of features within the same time as LLM-based methods. A more fair comparison would involve using TST-LLM's feature selection approach on the most informative features generated by OPENFE, rather than randomly selecting features.

4. The method appears to be more effective on datasets with clear semantic relationships that can be easily articulated by LLMs. This limits the applicability of TST-LLM to domains where such semantic relationships are less obvious or harder to define, restricting its generalizability compared to other tabular data methods.

[1] Xgboost: A scalable tree boosting system

[2] CatBoost: unbiased boosting with categorical features

[3] TabR: Tabular Deep Learning Meets Nearest Neighbors in 2023

[4] Revisiting Deep Learning Models for Tabular Data

[5] Modern Neighborhood Components Analysis: A Deep Tabular Baseline Two Decades Later

[6] TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second

[7] Making Pre-trained Language Models Great on Tabular Prediction

[8] Cross-Table Pretraining towards a Universal Function Space for Heterogeneous Tabular Data

[9] Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering

[10] Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning

[11] Optimized feature generation for tabular data via llms with decision tree reasoning

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731230466419

thread_type:
official_review

<thread_object>
creator: reviewer_ukii

<content>
summary:
The paper presents TST-LLM, a novel framework for self-supervised representation learning tailored for tabular data. By defining pre-text tasks in a task-specific manner using natural language descriptions, TST-LLM effectively identifies and combines relevant features, leading to enhanced performance across various downstream tasks. The framework was evaluated on 22 diverse datasets, demonstrating its ability to outperform existing methods in both classification and regression tasks.

strengths:
- Task-Specific Approach: TST-LLM utilizes natural language descriptions to create pre-text tasks that align closely with downstream objectives, improving relevance and performance.

- Diverse Dataset Evaluation: The framework was tested on a wide range of datasets, ensuring robustness and generalizability across different types of tabular data.

weaknesses:
- Lack of Novelty: The idea of tailoring pre-text tasks to specific downstream objectives has been previously explored in various frameworks, making the contributions appear incremental rather than groundbreaking. Additionally, the reliance on existing techniques such as supervised contrastive learning further diminish the uniqueness of the approach, as many prior works have already demonstrated similar strategies. Overall, the integration of known methods does not provide a significant advancement in the field, leading to questions about the paper's originality.

- Complexity of Implementation: The task-specific design may require more intricate setup and understanding compared to simpler, task-agnostic approaches.

questions:
1. What does it mean to treat the discovered features as the gound-truth labels?

2. Have you tried any other open-source LLMs (e.g., llama, mistral) as the backbone models? What is the performance?
</content>
</thread_object>

</thread>


Metareview: