Tabular Transfer Learning via Prompting LLMs
Authors: Jaehyun Nam
Woomin Song
Seong Hyeon Park
Jihoon Tack
Sukmin Yun
Jaehyung Kim
Kyu Hwan Oh
Jinwoo Shin
Venue: COLM (2024)
Keywords: Tabular learning, Transfer learning, In-context learning
TL;DR: We propose a novel tabular transfer learning framework that leverages LLM's in-context learning capabilities.

Abstract: Learning with a limited number of labeled data is a central problem in real-world applications of machine learning, as it is often expensive to obtain annotations. To deal with the scarcity of labeled data, transfer learning is a conventional approach; it suggests to learn a transferable knowledge by training a neural network from multiple other sources. In this paper, we investigate transfer learning of tabular tasks, which has been less studied and successful in the literature, compared to other domains, e.g., vision and language.
This is because tables are inherently heterogeneous, i.e., they contain different columns and feature spaces, making transfer learning difficult. On the other hand, recent advances in natural language processing suggest that the label scarcity issue can be mitigated by utilizing in-context learning capability of large language models (LLMs). Inspired by this and the fact that LLMs can also process tables within a unified language space, we ask whether LLMs can be effective for tabular transfer learning, in particular, under the scenarios where the source and target datasets are of different format. As a positive answer, we propose a novel tabular transfer learning framework, coined Prompt to Transfer (P2T), that utilizes unlabeled (or heterogeneous) source data with LLMs. Specifically, P2T identifies a column feature in a source dataset that is strongly correlated with a target task feature to create examples relevant to the target task, thus creating pseudo-demonstrations for prompts. Experimental results demonstrate that P2T outperforms previous methods on various tabular learning benchmarks, showing good promise for the important, yet underexplored tabular transfer learning problem. Code is available at https://github.com/jaehyun513/P2T.

Published as a conference paper at COLM 2024
Tabular Transfer Learning via Prompting LLMs
Jaehyun Nam', Woomin Song', Seong Hyeon Park', Park',Jihooo Jihoon Tack! Sukmin Yun?, Jaehyung Kim Kyu Hwan Oh', Jinwoo Shin'
'KAIST, Haanyang Unviersity ERICA, Carneeiie Mellon University *Hyundai Motor Company (jaehyun.nan nam, jinwoos Linwoos@ekaitt @caist.ac.kr
Abstract
Learning with limited number of labeled data is a central problem in real- world applications of machine learning, as is often expensive obtain annotations. To deal with the scarcity of labeled data, transfer learning is a conventional approach; it suggests to learn a transferable knowledge by training neural network from multiple other sources. In this paper, we investigate transfer learning of tabular tasks, which has been less studied and successful in the literature, compared to other domains, e.g,, vision and language. This is because tables are inherently heterogeneous, i.e., they contain different columns and feature spaces, making transfer learning difficult. On the other hand, recent advances in natural language pro- cessing suggest that the label scarcity issue can be mitigated by utilizing in-context learning capability of large language models (LLMs). Inspired by this and the fact that LLMs can also process tables within unified language space, we ask whether LLMs can be effective for tabular transfer learning, in particular, under the scenarios where the source and target datasets are of different format. As a positive answer, we propose novel tabular transfer learning framework, coined Prompt to Transfer (P2T), that utilizes unlabeled (or heterogeneous) source data with LLMs. Specifically, P2T identifies a column feature in a source dataset that is strongly corre- lated with target task feature to create examples relevant to the target task, thus creating pseudo-demonstrattinn results demonstrate that P2T outperforms previous methods on various tabular learning benchmarks, showing good promise for the important, yet underexplored tabular transfer learning problem. Code is available at https: / thub. com/ jatp//jahhun511/31 3/P2T.
for prompts. Experimental
Introduction
Learning with limited number of labeled samples is often critical requirement in real- world machine learning applications. This limited data problem is particularly important in the tabular domain; tabular datasets often require substantial annotation efforts (e.g,, credit risk assessment; Clements et al., 2020), or it is hard to obtain new samples for emerging tasks (e.g,, identifying patients with new diseases such as COVID-19; Peplow, 2016; Zhou et al., 2020). To address this problem, various methods have been thoroughly studied in domains such as vision (Assran et al., 2021; Pham et al., 2021) and language (Chen al., 2021; Min et al., 2022). However, the research on tabular data has only recently begun to gain traction (Yoon et al., 2020; Nam al., 2023), despite its wide-ranging impact across variety of industries (Guo et al., 2017; Ulmer et al., 2020; Zhang et al., 2020).
Learning transferable knowledge by training a neural network from various sources (Chen et al., 2020; Perez et al., 2021; Lee & Shin, 2022) is common way to address this limited data problem in other domains. However, such transfer learning is challenging in the tabular domain, because the source and target datasets are often very heterogeneous (i.e., they have different columns and feature spaces; Zhu et al., 2023; Wang & Sun, 2022; Yan et al., 2024). For example, using the collected features to predict diabetes to predict whether another
1

---

Published as a conference paper at COLM 2024
e
Target Data (Labeled)
Source Data (Unlabeled)
Insulin
BMI
Age
Diabetes
Insulin
BMI
Age
130
37.9
21
No
64
33.6
22
210
42.9
36
Yes
171
34.2
33
Conventional prompting (Baseline)
Read given information and questions.
Q: If insulin 130 pU/ml, BMI 37,9, age
the patient diabetic? No
O
No X
Q: If insulin is 210 pU/ml, BMI is 42.9, age is 36, the patient diabetic? A: Yes
Q: If insulin pU/ml, BMI is 37.4, age 24, is the patient diabetic? A:
LLM
P2T (Ours)
Read given information and questions.
Q: If BMI is 37.9, age is 21, then what is the insulin level? A: 64 pU/ml
Q: If BMI 34.2, age is 33, then what is insulin level? A: 171 pU/ml
v
Q: If insulin is 130 pU/ml, BMI is 37.9. age is 21, is the patient diabetic? A: No
Q: insulin 210 pU/ml. BMI 42.9, age is 36, is the patient diabetic? Yes
LLM
Q: If insulin is 36 pU/ml, BMI is 37.4, age is 24, is the patient diabetic? A:
Figure 1: Overview. P2T creates pseudo-demonstrations to effectively transfer knowledge of source data in in-context manner. 'Insulin' is used as a prediction target for pseudo- demonstrations because has the highest correlation with the task feature 'Diabetes."
patient has breast cancer is not straightforward because the required features to predict each disease (i.e., columns in table) are very different. This leads to the question of how transferable knowledge can be extracted from various tabular data sources in unified space.
On the other hand, converting tabular data into text has recently gained attention as this conversion leads to several benefits; first, one can inject linguistic context (Dinh et al., 2022; Hegselmann et al., 2023; Manikandan et al., 2023), for example, using column name 'Age' along with its numerical value. In addition, it opens up using large language models (LLMs) for tabular learning, which have demonstrated the capability data problems with few task-specific instructions (Brown et al., 2020; Dong et al., 2022; Wei et al., 2022; Kim et al., 2024), known as prompting (or int-ontext learning; ICI.)
address limited
Contribution. Motivated by this, we ask whether LLMs can be tabular transfer modules and address limited data problems in the tabular domain. As a positive answer, we propose novel tabular transfer learning framework that utilizes LLM's ICL capabilities, coined Prompt to Transfer (P2T); see the overview in Figure 1. Our main idea is to use LLM to extract transferable knowledge from the source dataset (e.g,, unlabeled dataset) and use it as in-context samples when prompting the LLM (called
Specifically, P2T starts by prompting LLM to determine which column feature is most important for the target task based on given few-shot labeled samples (i.e., demonstrations). P2T then creates aseudo-demonstratioon lected column feature is the target and the remaining ones are input, thus ensuring high relevance to the actual target task. These pseudo-demonstran selected column feature from the remaining features and this knowledge would be useful to the original target task (e.g., knowledge to predict 'Insulin' could be useful to predict 'Diabetes)). Therefore, P2T prompts the LLM to generate the desired output through the created pseudo-demonstrations with few-shot labeled demonstrations.
that describe the prediction task where the se-
reveal how to predict the
We verify the effectiveness of P2T, by conducting comprehensive evaluations on diverse tabular learning scenarios considering different source types (i.e., unlabeled and hetero- geneous datasets) and the number of labeled samples (i.e., few and zero-shot). Our results show that P2T significantly and consistently outperforms existing methods, includ- ing self-supervised (Yoon et al., 2020) and unsupervised meta-learning (Nam et al., 2023) methods, by transferring knowledge through prompting. As LLMs recently continue to advance (e.g., GPT-4; OpenAT, 2023), improved performance is expected with future models.
2

---

Published as a conference paper at COLM 2024
2 Related work
Tabular transfer learning. Researchers have developed a number of methods to train transferable representations for tabular data. One major stream of work exploits unlabeled datasets. In this context, Yoon et al. (2020) and Ucar et al. (2021) introduced various pre-text task losses (e.g,, reconstruction loss) with the goal of self-supervised learning on tabular datasets. In addition, Nam al. (2023) proposed to address few-shot tabular learning problems. Another stream of work exploits multiple heterogeneous datasets. Zhu et al. (2023) used a federated learning approach with separate encoders for each dataset to aggregate the parameters into a single transformer. Wang & Sun (2022) and Yan et al. (2024) used tokenizers tables into a unified language space and then fine-tuned the language model. We propose transfer learning framework that leverages both unlabeled and heterogeneous sources while applying in-context transfer to enable immediate predictions (i.e,, training-free free).
unsupervised meta-learning framework
map the heterogeneous structure of
Tabular learning with large language models. Recent advances in LLMs have provided an
impetus to explore their potential for tabular learning. Dinh et al. (2022) investigated the performance of fine-tuned GPT-3 models (Brown al., 2020) on tabular data. Extending this line of research, Hegselmann et al. (2023) conducted a comprehensive analysis using the TO model (Sanh et al., 2022), leveraging the language prior in LLMs. Their analysis is extended to sample efficiency, even conducting zero-shot experiments. Recently, Manikandan et al. (2023) developed a boosting framework over prompting LLMs that uses LLMs as weak learners in tabular prediction tasks. Inspired by these preceding studies, our work proposes a method for the effective exploitation of various transfer sources an aspect overlooked in prior research. By integrating the utilization of unlabeled (or heterogeneous) data with LLMs, we aim to enhance performance in transfer learning scenarios.
In-context learning. As model and dataset sizes increase (Brown et al., 2020), LLMs have exhibited the capability for ICL (or prompting), where they draw knowledge from a handful of contextual examples. For example, Wei et al. (2022) have illustrated the competency of LLMs in solving mathematical reasoning problems via ICL. The ICL process begins by employing a small number of examples to establish a contextual framework, typically constructed using natural language templates. Following this, a query question and contextual demonstration are combined to form a prompt, which is subsequently fed to the LLMs for prediction. Notably, ICL does not necessitate parameter updates and directly carries out predictions using LLMs, enabling easy implementation for real-world applications. In our work, we delve deeper into the potential of ICL by examining its performance on transfer learning, using source data for creating effective demonstrations.
P2T: Prompt to Transfer
In this section, we propose a simple yet effective tabular transfer learning that leverages ICL, the capability of LLMs that adapt to new task using the context provided in prompt without updating model parameters. In a nutshell, our framework creates pseudo- demonstrations from transfer sources that act proxies for causal relationships between input features and labels, and then prompt the LLM to make predictions. We first briefly describe preliminaries (Section 3.1), and then the core component coined Prompt to Transfer (P2T), which creates effective pseudo- peeudd-demonstrations for ICL (Section 3.2).
3.1 Preliminaries
tabular transfer learning. We first describe the problem setup of our interest. A labeled target dataset D = ((x) y C X x y and a source dataset D = (x)) C X are given, where x is d'-dimensional, and x is df-dimensional feature which correspond to the value in the respective table columns. Also, we assume that column name sets F iff,.... .ff)) } are given well (e.g., 'x Male' is feature values of the 'fi sex' column), and flarge is the column name of the labels y of the target dataset. Labels y y are provided in the form of natural language annotations (egg. 'Non Diabetic' and
Problem setup:
3

---

Published as a conference paper at COLM 2024
'Diabetic' are labels of the Diabetes dataset). Here, the cardinality of D' is assumed to be much smaller than the source dataset, i.e., [D'] < knowledge from D to predict the accurate label of test query xlest in the target dataset.
[D']. Our goal is to effectively extract
In-context learning for tabular prediction tasks. We now describe ICL-based tabular prediction (Dinh et al., 2022). Similar to ICL in the natural language domain (Brown et al., 2020), tabular prediction via ICL is executed by constructing an input prompt comprising (i) a fask description Prask (D'), (ii) few-shot labeled demonstrations Plabel (D') (i.e., descriptions of input-output pairs of the labeled samples), and (iii) a lest query Ptest (xles)) To begin, the task description is manually crafted through a systematic procedure. This description includes prompts directing the LLM to analyze the data in a step-by-step manner, and the dataset's s details such column descriptions. Next,few-shi few-shot labeled demonstrations are created by serializing tabular data into natural language, thereby presenting the data a question format. There exist numerous design choices for tabular serialization; however, we mainly follow Dinh et al. (2022). Finally, we predict the output by prompting LLM M:
ypred (Ptaak (D') @ Plabel (D') Prest (xies)),,
where, ( is the concatenation of each prompt (see the conventional prompling in Figure 1).
3.2 In-context tabular transfer learning with P2T
We now present P2T, a novel approach to improve tabular prediction performance by creating additional pseudo-demoostindie few-shot labeled demonstrations, This is achieved by (i) first identifying the column feature in the source dataset that exhibits the highest correlation with the target ftarget and (ii) then demonstrations from the transfer source. The constructed prompt pe2r, constructing pseudo denedddddmnstttttinn with conventional prompting, is then fed into the LLM which integrates
from the source data, which serve as proxies for
M to generate the desired prediction output.
Correlation identification. P2T begins by iden- tifying the feature ft E F (the column name set of the source data) that holds the most sig- nificant correlation with the task column ftarget in the target data D'. To achieve this, we ask LLM M which feature among F is most im- portant for predicting flarget. Formally, an in- put prompt designed to identify correlations consists of two main components: (i) few--sot labeled demonstrations Plabel (D') and (ii) identi- fication instructions Pcor Pcor(Fs, ftarget ); see Figure 2. Finally, we prompt LLM M to obtain fx:
Pear: Identification instruction
Q: If insulin is 130 pU/ml, BMI is 37.9, age is 21, is the patient diabetic? A: No Q: If insulin is 210 pU/ml, BMI is 42.9, age is 36, is the patient diabetic? A: Yes
Choose the most important feature predict whether patient is diabetic.
Choices: [Insulin, BMI, Agel
Insulin
Figure 2: Example prompt for correlation identification on the Diabetes dataset.
fx = M(Paabell (D') + Pcor (Fs, .furget)).
Our choice of using LLMs over conventional methods (Chen & Guestrin, 2016; Prokhorenkova et al., 2018) is driven by the capabilities of LLMs to interpret linguistic context; their ability to leverage both the semantics of column names and the associated numeric values offers distinct advantage. For instance, considering a column like 'Age,' conventional algorithms would focus solely on the numeric values. Conversely, LLMs can utilize the semantic understanding of 'Age,' incorporating additional contextual information.
Peeuto-demonstration from the source tables. Our main idea is to create psendo- demonstrations from the unlabeled (or heterogeneous) source tables using highly correlated column feature fi as a new target feature. The rationale behind using ft stems from the intuition that predicting the most correlated column feature from the remaining features would resemble the original task of predicting the label from all the features; therefore, from source data include the useful knowledge to predict
these pseudo-demonstre the original target task. For instance, predicting 'Diabetes' from 'BMI' and 'Age' is similar to predicting 'Insulin' using the same features (Nam et al., 2023). Thus, we create psendo- demonstrations that predict the value of fx from remaining features, which we refer to as
4

---

Published as a conference paper at COLM 2024
Target dataset
Source dataset
Method
Accuracy (t)
zero-shot
68.00
Credit-R
P2T (Ours)
70.00
Adult
Electricity
P2T (Ours)
72.00
Unlabeled Adult
P2T (Ours)
74.00
zero-shot
46.00
Credit-g
Credit-A Unlabeled Credit-g
P2T (Ours)
62.00
P2T (Ours)
68.00
zero-shot
60.00
Heart-c
Diabetes
P2T (Ours)
65.00
Unlabeled Heart-c
P2T (Ours)
63.33
zero-shot
41.07
Breast
Haberman
P2T (Ours)
58.93
Unlabeled Breast
P2T (Ours)
62.50
Table 1: Test accuracy (%) on various zero-shot leamiing scenarios, Both unlabeled dataset and heterogeneous dataset improves the zero-shot test accuracy of the target dataset. Bold indicates the highest accuracy, and underlined indicates the second highest accuracy.
Ppaeddo (D*, (D*,ft) ft) See Figure for specific example of how P2T creates pseudo-demonstratiin from an unlabeled dataset. As shown in Figure 1, 'Inuslin' is selected as the most important feature fs for predicting whether a patient has diabetes (see Figure 2). Therefore, predicting insulin levels from BMI and age becomes components of preudd-demmoutrationn Finally, our P2T framework outputs prediction results follows:
Pe2r PD' Dd, D'r xiost) Prask Ppaeudo (D",fa) fi) @ (label(D')) @ Pteat (x(est),
ypred = ((pprr (D', Dd, D x)
4 Experiments
In this section, we validate the effectiveness of our proposed method for transfer learning scenarios on variety tabular datasets from the OpenML. repositories (Vanschoren al., 2014) and Kaggle. First, in Section 4.1, we verify that P2T improves zero-shot prediction performance by leveraging different types of transfer sources (i.e., unlabeled and hetero- geneous datasets). Note that zero-shot prediction is one of the big benefits of using LLMs that not possible with traditional methods like CatBoost (Prokhorenkova et al., 2018). Then, in Section 4.2, we verify that method outperforms other tabular learning methods, including unsupervised meta-learning methods (Nam et al., 2023), in the few-shot learning scenario by extracting effective knowledge from the transfer sources via prompting. Finally, in Section 4.3, we validate the effectiveness of our proposed neeudodemoonttration and that better performance can be achieved with more advanced LLM (i.e., GPTT-4; OpenAl, 2023).
Common setup and baselines. When using unlabeled data as the training source, we use 20% of the labeled dataset for test samples and convert the remaining 80% to unlabeled, except for a limited number of labeled samples. Following Nam et al. (2023), we one-hot encode categorical features for the baseline and then min-max scaling. To validate P2T, we consider supervised learning baselines such as CatBoost (Prokhorenkova et al., 2018), logistic regression (LR), and nearest neighbor classifier (kNN) that do not utilize unlabeled (or heterogeneous) data. We also consider VIME (Yoon et al., 2020), self-supervised learning baseline where the model is initially pre-trained and then evaluated using labeled samples via logistic regression. Note that we intentionally exclude methods that require careful hyperparameter tuning, e.g., MT (Tarvainen & Valpola, 2017), MPL (Pham et al., 2021), ICT (Verma et al., 2022), due to sensitivity to hyperparameters and overfitting issues; our problem settings characterized by limited labeled data are not suitable for hyperparameter tuning in real-world scenarios due to the lack of labeled validation sets. We also consider

---

Published as a conference paper at COLM 2024
Dataset
LR
kNN
CatBoost
VIME
STUNT
LIFT-ICL
P2T (Ours)
shot
Breast
61.23
61.88
57.64
57.38
53.04
66.43
68.93
TAE
37.35
37.26
34.29
37.87
36.87
30,97
43.23
Hamster
51.07
51.00
51.87
51.53
51.73
48.00
58.67
Customers
61.34
63.81
64.12
62.48
65.14
70.45
74.32
Pollution
63.67
63.67
63.58
63.33
63.00
58.33
65.00
Diabetes
57.61
58.56
58.60
56.95
61.08
62.60
68.44
Car
36.95
31.51
32.33
34.51
36.48
69.13
71.40
BTC
51.60
51.54
53.02
51.13
52.71
60.40
62.27 19.02
Haberman
52.81
52.81
52.82
51.55
53.82
60.32
61.29
Caesarian
62.50
62.50
56.63
60.38
60.06
55.00
63.75+5.28
VC
53.76
53.77
54.00
56.34
62.11
70.00
70.64+0.89
Salaries
59.52
58.18
58.45
66.55
70.26
45.53
71.06
Average
54.12
53.87
53.11
54.17
55.53
58.10
64.92
# shot = 5
Breast
61.21
62.33
57.63
60.89
61.30
67.86
72.85
TAE
43.42
44.65
39.71
42.84
40.77
35.48
45.81
Hamster
51.60
54.53
56.33
52.80
52.87
58.67
64.00
Customers
60.82
64.92
81.40
66.07
66.44
78.41
83.1810.95
Pollution
73.33
72.83
70.58
75.50
70.92
65.00
76.67 13.73
Diabetes
64.19
67.32
64.94
64.29
69.88
69.20
71.44
Car
53.29
49.62
46.96
52.37
51.73
70.81
72.08+1.66
BTC
58.03
55.71
56.43
55.83
54.11
67.73
69.33
Haberman
53.92
53.40
55.35
53.45
54.85
62.26
64.84
Caesarian
69.56
64.31
66.25
64.88
66.75
65.00
80.00
VC
61.66
61.65
68.00
62.65
66.66
70.65
70.97
Salaries
70.87
71.38
66.38
74.82
76.86
55.65
75.06+1.70
Average 60.16
60.22
60.83
60.53
61.10
63.89
70.52
Table 2: Few-shot test accuracy (%) using unlabeled samples as transfer source. shot indicates the number of labeled samples per class. For the baselines, we report the average test accuracy over 100 different seeds. We report the average accuracy and standard devia- tion over 5 different seeds for LIFT-ICL (Dinh et al., 2022) and our method, due the high cost of OpenAl API. The bold denotes the highest average score.
STUNT (Nam al., 2023), state-of-the-art few-shot tabular learning method. Finally, we consider LIFT (Dinh et al., 2022) in the ICL setting (LIFT-ICL) as representative way to exploit the power of LLM. In all experiments using LLM,, we use 5.t-3.b-
4.1 Zero-shot prediction
One of the distinct advantages of using LLMs is that they can easily obtain the desired answer in a zero-shot manner. In fact, Hegselmann et al. (2023) have shown that LLMs can perform zero-shot prediction tasks even tabular domains. In this section, we investigate whether the proposed P2T framework can improve the performance of zero-shot prediction by transferring knowledge from unlabeled and heterogeneous datasets, respectively. We emphasize that P2T can be seen zero-shot transfer module because it is based on the ICL.
As shown in Table 1, using a transfer source improves the zero-shot prediction performance. We first found that LLM (GPT-3.5 in our case) often fails to predict with zero-shot. For example, on the Credit-g dataset, which is a binary classification, LLM scores behind random guessing (see Table 1). However, using heterogeneous sources (i.e., Credit-A) or unlabeled samples of the Credit-g dataset, P2T improves zero-shot prediction accuracy by 16.0% and 24.0%, respectively (note that randomly select 30 samples from the transfer sources due

---

Published as a conference paper at COLM 2024
Number of samples from source dataset (N)
Target
Source
Method
=
2
N 4
N =
N =
10
LR
54.00
69.33
69.33
66.67
62.00
57.33
kNNt
54.00
72.00
72.00
57.33
57.33
57.33
Credit-R
CatBoost
56.00
54.67
60.00
61.33
51.33
49.33
LIFT-ICL
69.33
25.33
35.33
52.00
60.00
43.33
Adult
P2T (Ours)
74.67
75.33
76.00
77.33
79.33
80.00
LRt
54.00
54.67
50.67
50.00
37.33
60.00
kNNt
54.00
57.33
42.67
42.67
28.00
42.67
Electricity
CatBoost*
56.00
50.00
50.67
48.67
45.33
58.00
LIFT-ICL
69.33
60.67
64.67
63.33
58.67
54.00
P2T (Ours)
74.67
80.00
76.00
78.67
80.00
81.33
LRt
52.67
49.33
48.00
34.00
42.00
38.67
kNN
52.67
58.67
41.33
41.33
41.33
24.00
Credit-g
Credit-A
CatBoost*
55.33
46.67
41.33
46.67
40.67
44.00
LIFT-ICL
42.67
49.17
48.17
45.83
46.00
48.67
P2T (Ours)
55.00
54.50
58.67
59.33
59.33
60.67
Table 3: 1-shot test accuracy (%) using heterogeneous data as transfer source. We report average test accuracy over three different seeds for all methods. Experiments for non- LM baselines (t) are implemented by extending columns for the heterogeneous data with zero-padded values. Bold indicates the highest average score.
to the high cost of the OpenAl API). We also note to provide intuition that heterogeneous datasets should come from similar domains. For example, the Credit-g dataset and the Credit-A dataset are both financial datasets consisting of an individual's credit information.
4.2 Few-shot prediction
For few-shot tabular prediction, we evaluate the performance when one and five labeled samples are available per class, respectively. Firstly, we found that taking few-shot labeled tabular data, serializing it into text and prompting it to LLMs (i.e., LIFT-ICL, in Table 2) outperforms existing state-of-the-arr methods for few-shot tabular prediction (egg,, STUNT; Nam et al., 2023). Thus, using LLMs prediction models is indeed promising direction for tabular learning, and the P2T further extends this line of work by using LLMs as transfer modules via prompting. In this section, provide few-shot tabular prediction results using (i) unlabeled data and (ii) heterogeneous data as the transfer source, respectively.
Transfer learning utilizing unlabeled dataset. As demonstrated in Table 2, P2T significantly and consistently improves the few-shot prediction performance on 12 tabular datasets by utilizing unlabeled data of the same dataset transfer source (here, we use 30 unlabeled samples that are closest to each labeled sample in terms of Euclidean distance). Note that this improvement is achieved without model updates. To provide specific example, P2T sig- nificantly outperforms LIFT-ICL in 1-shot classification, raising the average performance from 58.10% 64.92%. Additionally, P2T consistently achieves superior results, yielding the highest score in all 12 datasets in the 1-shot classification problem, and in 11 out of the 12 datasets in the 5-shot scenario. These results represent an improvement of approximately 6.8% and 6.6% over the best performing baselines, respectively. The significant improvement achieved by constructing effective pseudo-demonstrationn from the unlabeled dataset, therefore extracting valuable information in an in-context manner.
Transfer learning utilizing heterogeneous dataset. We next demonstrate the effect of introducing training samples from heterogeneous data sources. First, we found that merg- ing distinct column sets from diverse sources in the tabular domain demands a heuristic process to create unified feature set. Such approach may not generalize well, and
7

---

Published as a conference paper at COLM 2024
require sophisticated designs for different data combinations. In this regard, we find our tabular serialization to be simple and effective method for combining columns from various heterogeneous sources. As tabular data is transformed into natural language, the language model can automatically understand the relations between different features from their descriptions. To investigate the effect of incorporating heterogeneous data, we consider transfer scenario where the target data should be classified given 1-shot training samples from the same dataset and N additional samples from heterogeneous source dataset that contains disparate column sets (e.g,, 'work class," "education etc. for the Adult dataset and 'loan amount,' 'credit history length, etc. for the Credit-R dataset in Table 3).
As shown in Table 3, P2T consistently benefits from heterogeneous data source. For example, 74.67%->80.0077 on the Adult dataset, when 10 additional samples from the Credit-R tasas is provided. More importantly, P2T is the only method that shows steady performance improvements as the number of heterogeneous training samples N increases, while the baselines are not able to properly learn from the additional samples, and their performance could even deteriorate compared to their 1-shot performances. Interestingly, cramming the extra columns from the heterogeneous dataset in LIFT-ICL only incurred noise to the accuracy. We attribute this to that our P2T framework enables the LLMs to exploit the deeper relationship between heterogeneous data, while the naïve concatenation without proper descriptions only perplexes the LLMs.
4.3 Ablation studies
In this section, we perform further analysis of the proposed P2T framework. (i) First, we use the column feature in the source dataset that is most relevant to the target task label as useful target for creating pseudoo-emoonstration demonstrations are really similar the real task, i.e, if we create pseuulodemonstration. with arbitrary column feature of the source data target, the created demonstrations may not be similar to the real task and hence the LLM may not convey knowledge well. (ii) Second, variety of LLMs have emerged in recent years. This raises the question of whether P2T can achieve better performance with more advanced models.
Here,
ask whether the created pseudo-
Effectiveness of identifying tar- get for pseudo-demonstration construction. In Figure 3, we val- idate the effectiveness of using the identified column feature as an useful target for creating pseudo- demonstrations. Here, we consider transfer learning scenario utiliz- ing an unlabeled dataset for 1-shot prediction. As shown in Figure using the identified target that is highly correlated with the target task consistently outperforms us- ing random targets (e.g,, using the 'Age' column as the target for the peeudo-demonsstrationns in Figure 1). Interestingly, using randomly se- lected column features as targets can perform worse than not leveraging unlabeled data (see the Haberman dataset results in Figure 3). This because, for example, the Haberman dataset, 'patient's year of operation' may not have any correlation to the target task 'patient's survival status.' This highlights that carefully constructing ppenuo--demonstrations designed to be highly relevant to the target task is key factor in enabling transfer learning via prompting.
75
LIFT-ICL
P2T (Ours) w/ random target
P2T (Ours) identified target
Accuracy (%)
55
Customers
BTC
Haberman
Dataset
Figure Ablation study that varies the column features used as targets for pseudo-demonstrations..
Qualitative analysis on selected column. In this part, we use the Breast dataset as an example to qualitatively analyze which column feature is most relevant to breast cancer. For the Breast dataset, 'deg-malig' selected as the most relevant, making it a useful target when creating pseudo-demonstrations On the other hand, the other feature 'tumor-size is
8

---

Published as a conference paper at COLM 2024
Customers
BTC
Haberman
Method
GPT-3.5
GPT-4
GPT-3.5
GPT-4
GPT-3.5
GPT-4
LIFT-ICL
70.45
88.18
60.40
61.73
60.32
67.74
P2T (Ours)
74.32
89.77
62.27
63.47
61.29
70.32
Table 4: Comparison between GPT-3.5 and GPT-4. We report 1-shot test accuracy (%) using unlabeled samples as transfer source. We report the average accuracy over 5 different seeds. The bold denotes the highest average score.
Dataset
LR
LIFT-ICL
P2T (Ours)
Dataset
CatBoost
LLM (Ours)
Laptops
7.47+2.0
4.22-11.
3.97+0.9
Customers
69.32-4.17
74.32-347
Choles. 1.56+138
0.37-034
00.0
BTC Haberman
62.00-s45
62.27-990
House
2.34
2.56-079
2.48+077
60.97
61.29-55.9
Table 5: 10-shot regression of P2T using unla- beled samples transfer source. We report the average test mean squared errors (MSEs) and standard deviation over 5 different seeds. The bold denotes the lowest average score.
Table 6: LLM's superiority for correla- tion identification. We report 1-shot test accuracy (%) using unlabeled samples as transfer source. We report the average accuracy over 5 different seeds.
also highly correlated, but LLM recommends using "eegg. In fact, both deg-mallie and 'tumor-size' are important factors in predicting the occurrence of breast cancer because higher 'deg-malig' corresponds to a more aggressive cancer, and 'tumor-size' indicates more advanced stage of cancer. However, 'deg-malig' is often considered a leading factor since it is more indicative of the outcome than size alone, as small tumor might be highly malignant and thus more dangerous than a larger but less malignant one.
Results of using an advanced LLM. All previous experiments use GPT-3.5 (i.e., gpt-3.5-turbo). A natural extension is to ask whether better performance can be achieved by P2T using a more advanced model. To verify whether P2T performs better with advanced LLMs, we use GPT-4 (i.e,, gpt-4- 1et----turb--peeee prrviiee)) in a transfer learning scenario leveraging an unlabeled dataset. capability of GPT-44 is better. With better ICL capabilities, and shows better performance than using GPT-3.5. This indicates that as LLMs continue to advance, improved performance by P2T framework is expected with future models.
can be seen from LIFT-ICL in Table 4, we first note that the ICL
P2T framework also benefits
Extending P2T to regression tasks. While our main interest in classification tasks, one way to naturally extend P2T to regression tasks is to prompt the LLM to output the numerical values of the regression task (in classification, the LLM is guided with multiple choices). Here, we evaluate P2T's capabilities in 10-shot regression tasks three tabular regression datasets from OpenML and Kaggle. As shown in Table 5, the results indicate that P2T is competitive approach for few-shot tabular regression tasks. However, the improvement is relatively small compared to classification tasks. This is because LLM often faces difficulties when dealing with numerical values, and we leave as future work to address this.
Using LLM for correlation identification. To verify that LLM is better than conventional methods for identifying the most correlated feature, we perform an ablation study using Cat- Boost for correlation identification. As shown in Table 6, using the column feature selected by CatBoost (which differs from LLM'' selection) to construct the pseudo-demonstral performs worse than using LLM. This is because (i) CatBoost often does not perform well with few-shot examples (see Table 2), and (ii) LLM, on the other hand, can understand the linguistic context of the task and thus is better able to identify correlations.
Robustness to missing values. In practice, tabular data often contains missing values for various reasons. For instance, biopsy results may not be collected for all patients due to the risks and complications involved the data collection process (Yoon et al., 2018). Conventionally, missing values are managed using imputation algorithms (Yoon et al., 2018;

---

Published as a conference paper at COLM 2024
Customers
BTC
Haberman
Method
KNN
63.81
+ 56.17
51.54
7 50.05
52.81
50.11
LR
61.34
+ 53.89
51.60
7 51.37
52.81
7
51.40
LIFT-ICL
70.45
64.70
60.40
-
56.00
60.32
60.00
P2T (Ours)
74.32
70.01
62.27
56.93
61.29
60.97
Table 7: Robustness P2T to missing values. We report 1-shot test accuracy (%) using unlabeled samples transfer source. We report the average accuracy over 5 different seeds. indicates that all features are used, and 1 indicates that 50% of the features are randomly omitted. The bold denotes the highest average score.
Yi et al., 2020) in the tabular domain, which estimates missing values from other existing information. The performance of standard machine learning methodologies largely depends imputation algorithms, as incorrectly estimated data could introduce severe noise.
In contrast, P2T naturally handles missing values by simply excluding these values from the input prompt. For example, if the 'Age' feature is missing, P2T serializes the table to prompt like "Insulin is 130, BMI 37.9. To verify the robustness of P2T to missing values, we simulate scenario where 50% of features are randomly omitted. As shown in the Table 7, P2T outperforms the baselines using simple zero imputation (Yi et al., 2020), and the non-LM baselines fitted with all features. Furthermore, even when the non-LM baseline shows significant performance degradation (e.g., on the Customers dataset), P2T shows smaller performance drop. These results highlight P2I's robustness to missing values, which we attribute to the fact the P2T is not significantly affected by inaccurate estimates that the estimation algorithm can introduce.
P2T without language descriptions. Indeed, LLMs require explicit column descriptions to effectively exploit the language prior, but infor- mative descriptions are often absent in tabular datasets (Asuncion & Newman, 2007). Con- sequently, researchers are forced to use the generic indicator, a prompt that is used to sub- stitute (or pretend as) actual column names, for instance, 'feature' (Dinh et al., 2022). In Figure 4, we present the 1-shot test results of generic model where the language descrip- tions are removed. Specifically, we employ generic indicator 'feature' and 'outuut y' for the column names. Even in this case, lever- aging unlabeled data with P2T significantly improves performance, demonstrating P2T's potential to handle all types of tabular data.
75
LIFT-ICL P2T (Ours)
70
Accuracy (%)
65
60
55
Customers
Figure 4: Transfer learning with unlabeled data without language descriptions.
Conclusion
In this paper, we introduce P2T, a novel framework for transfer tabular learning. Our approach utilizes prompting to effectively transfer knowledge from the transfer source. Through extensive experiments, we show the efficacy of P2T across various datasets. It is noteworthy that P2T does not necessitate expert-level knowledge in machine learning. This allows practitioners to adopt our framework effortlessly. We hope that our work holds promising potential in broadening the use of LLMs for tabular learning, particularly in conjunction with readily available resources such unlabeled data.
10

---

Published as a conference paper at COLM 2024
Acknowledgements and disclosure of funding
We would like to thank Seojin Kim, Subin Kim, Changyeon Kim, Sangkyung Kwak, Se- unghyuk Oh, and anonymous reviewers for their helpful feedback and discussions. This work was supported by Institute of Information & communications Technology Promotion (ITTP) grant funded by the Korea government (MSIT) N.0RSS20019-1190005 Artificial In- telligence Graduate School Program(KAIST) No.RS- RS-2022-11220955 Few-shot Learning of Casual Inference in Vision and Language for Decision Making) and Hyundai Motor Group.
References
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Armand Joulin, Nico- las Ballas, and Michael Rabbat. Semi-supervised learning of visual features by non- parametrically predicting view assignments with support samples, In IEEE International Conference on Computer Vision, 2021.
Arthur Asuncion and David Newman. UCT machine learning repository, 2007.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 2020.
Luoxin Chen, Francisco Garcia, Varun Kumar, He Xie, and Jianhua Lu. Industry scale semi-supervised learning for natural language understanding, In Anmual Conference of the North American Chapter of the Association for Computational Linguistics, 2021.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd infernational conference on knowledge discovery and data mining, 2016.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple frame- work for contrastive learning of visual representations. In International Conference on Machine Learning, 2020.
Jillian M. Clements, Di Xu, Nooshin Yousefi, and Dmitry Efimov. Sequential deep learning for credit risk monitoring with tabular financial data. arXiv preprint arXio:2012.15330, 2020.
Tuan Dinh, Yuchen Zeng, Ruisu Zhang, Ziqian Lin, Michael Gira, Shashank Rajput, Jy- yong Sohn, Dimitris Papailiopoulos, and Kangwook Lee. Lift: Language-interfaced Language-interfan fine-tuning for non-language machine learning tasks. Advances in Neural Information Processing Systems, 2022.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, lingjing Xu, and Zhifang Sui. A survey for in-context learning. arXiv preprint XrXi2:3301 xrXi::2301.0024, 2022.
Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfm: A factorization-machiie based neural network for ctr prediction. In International Joint Conferences on Artificial Intelligence, 2017.
Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. Tabllm: few-shot classification of tabular data with large language models. In International Conference on Artificial Intelligence and Statistics, 2023.
Jaehyung Kim, Jaehyun Nam, Sangwoo Mo, Jongjin Park, Sang- Woo Lee, Minjoon Seo, Jung-Woo Ha, and Jinwoo Shin. Sure: Improving open-domain question answering of L.LMs via summarized retrieval. In The Twelfth International Conference on Learning Representations, 2024.
Kyungmin Lee and Jinwoo Shin. Rényicl: Contrastive representation learning with skew rényi divergence. In Advances in Neural Information Processing Systems, 2022.
11

---

Published as a conference paper at COLM 2024
Hariharan Manikandan, Yiding Jiang, and] J Zico Kolter. Language models are weak learners. Advances in Neural Information Processing Systems, 2023.
Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Noisy channel language model prompting for few-shot text classification. In Annual Conference of the Association for Computational Linguistics, 2022.
Jaehyun Nam, Jihoon Tack, Kyungmin Lee, Hankook Lee, and Jinwoo Shin. STUNT: Few- shot tabular learning with self self-generated tasks from unlabeled tables. In International Conference on Learning Representations, 2023.
OpenAL. Gpt-4 technical report. Xrviv preprint arXiv:2303.08774, 2023.
Mark Peplow. The 100,000 genomes project. British Medical Journal, 2016.
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. In Advances in Neural Information Processing Systems, 2021.
Hieu Pham, Zihang Dai, Qizhe Xie, Minh-Thang Luong, and Quoc V. Le. Meta pseudo labels. In JEEE Conference on Computer Vision and Pattern Recognition, 2021.
Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. In Advances in Neural Information Processing Systems, 2018
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. In International Conference Learning Representations, 2022.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Advances in Neural Information Processing Systems, 2017.
Talip Ucar, Ehsan Hajiramezanali, and Lindsay Edwards. Subtab: Subsetting features of tabular data for self-supervised representation learning. In Advances in Neural Information Processing Systems, 2021.
Dennis Ulmer, Lotta Meijerink, and Giovanni Cinà. Trust issues: Uncertainty estimation does not enable reliable ood detection on medical tabular data. In Machine Learning for Health, 2020.
Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luis Torgo OpenML. ACM SIGKDD Explorations Newsletter, 2014.
Vikas Verma, Kenji Kawaguchi, Alex Lamb, Juho Kannala, Amno Solin, Yoshua Bengio, and David Lopez-Paz. Interpolation consistency training for semi-supervised learning. Neural Networks, 2022.
Zifeng Wang and Jimeng Sun. Transtab: Learning transferable tabular transformers across tables. Advances in Neural Information Processing Systems, 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022.
Jiahuan Yan, Bo Zheng, Hongxia Xu, Yiheng Zhu, Danny Chen, Jimeng Sun, Jian Wu, and Jintai Chen. Making pre-trained language models great on tabular prediction. In International Conference on Learning Representations, 2024.
Joonyoung Yi, Juhyuk Lee, Kwang Joon Kim, Sung Ju Hwang, and Eunho Yang. Why not to use zero imputation? correcting sparsity bias in training neural networks. In International Conference on Learning Representations, 2020.
12

---

Published as a conference paper at COLM 2024
Jinsung Yoon, James Jordon, and Mihaela Schaar. Gain: Missing data imputation using generative adversarial nets. International conference on machine learning, 2018.
Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela van der Schaar. Vime: Extending the success of self- and semi-supervised learning to tabular domain. In Advances in Neural Information Processing Systems, 2020.
Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. Deep learning based recommender system. Association for Computing Machinery (ACM) Computing Surveys, 2020.
Peng Zhou, Xing-Lou Yang, Xian-Guang Wang, Ben Hu, Lei Zhang, Wei Zhang, Hao-Rui Si, Yan Zhu, Bei Li, Chao-Lin Huang, Hui-Dong Chen, Jing Chen, Yun Luo, Hua Guo, Ren-Di Jiang, Mei-Qin Liu, Ying Chen, Xu-Rui Shen, Xi Wang, Xiao-Shuang Zheng, Kai Zhao, Quan-Jiao Chen, Fei Deng, Lin-Lin Liu, Bing Yan, Fa-Xian Zhan, Yan-Yi Wang, Geng-Fu Fu Xiao, and Zheng-Li Shi. A pneumonia outbreak associated with new coronavirus of probable bat origin. Nature, 2020.
Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karypis, and Mahsa Shoaran. Xtab: Cross-table pretraining for tabular transformers. In International Conference on Learning Representations, 2023.
13

---

Published as a conference paper at COLM 2024
Appendix: Tabular Transfer Learning via Prompting LLMs
A Baseline details
In this section, we provide brief explanations of the chosen baselines. For CatBoost (Prokhorenkova et al., 2018) and legistic regression, we employ the default hyperparameters as provided by the CatBoost library and the Scikit-learn library, respectively. For VIME (Yoon et al., 2020) pre-training, we adopt the hyperparameters recommended in the original paper, utilizing the Adam optimizer with a learning rate of le - and weight decay of When implementing STUNT (Nam et al., 2023), follow the unsupervised validation scheme proposed the original paper for hyperparameter search and early stopping. For LIFT-ICL (Dinh et al., 2022), provide prompt example used on the Customers dataset in Listing 1.
f 'Read a given information and questions. Think step by step, and then predict whether - its value is class1 or class2. You must choose in [class1, class2]. Class1 indicates Horeca (Hotel, Restaurant, Cafe) channel, and class2 indicates Retail channel.
The dataset consists
input variables: annual spending fresh product, annual spending on milk products, annual spending on grocery products, annual spending on frozen products, annual spending on detergents and paper products, annual spending on delicatessen products, and customer's region. The output variable is the customer's channel.
Question: If the annual spending on fresh product 583.0, annual spending on milk
products is 685.0, annual spending grocery products is 2216.0. 2, annual spending frozen products is 469.0, annual spending on detergents and paper products is 954.0, annual spending on delicatessen products 18.0, customer's region indicates Lisbon, 2 indicates Porto, and indicates Other) is 1, then what is the customer's s channel? Choose between [class1, class2]. Class indicates Horeca (Hotel, Restaurant, Cafe) channel, and class2 indicates Retail channel. Answer: class1
Question:
annual spending on fresh product 7823.0, annual spending on milk products is 6245.0, annual spending on grocery products is 6544.0, annual spending on frozen products is 4154.0, 0, annual spending detergents and paper products is 4074.0, annual spending delicatessen products is 964.0. 0, customer's region (1 indicates Lisbon, 2 indicates Porto, and indicates Other) is 3, then what is the customer's s channel? Choose between [class1, class2]. Classl indicates Horeca (Hotel, Restaurant, Cafe) channel, and class2 indicates Retail channel. Answer: class2
Question: If the annual spending fresh product is 11686.0, annual spending milk
products is 2154.0, annual spending on grocery products is 6824.0, 0, annual spending on frozen products is $527.0, annual spending on detergents and paper products is
592.0, annual spending on delicatessen products is 697.0, customer's region (1 indicates Lisbon, 2 indicates Porto, and indicates Other) is 1, then what is the customer's channel? Choose between [classl, class2]. Classl indicates Horeca (Hotel, Restaurant, Cafe) channel, and class2 indicates Retail channel. Answer:
Listing 1: Prompt for LIFT-ICL (Dinh et al., 2022) used on the Customers dataset.
14

---

Published as a conference paper at COLM 2024
B Prompt examples used in P2T
In this section, we provide examples of prompts used in P2T, specifically on the Customers dataset in scenario of transferring knowledge from the unlabeled dataset. particular, we illustrate the prompts employed to identify the column feature with the highest correlation (see Listing 2), along with the prompts used during the final inference stage (see Listing 3). For the sake of brevity and due to constraints on paper length, the prompts we provide consist of merely two unlabeled samples along with single labeled sample per class. We also provide prompt used for zero-shot prediction (see Table 1) on the Adult dataset, where the source dataset is Electricity, in Listing 4. Note that prompt used in Table 3 just requires additional labeled samples of the target dataset.
F' 'Read given information and questions. Think step by step, and then choose the most important feature to predict whether its value is class1 class2. You must choose in [annual spending on fresh product, annual spending on milk products, annual spending on grocery products, annual spending on frozen products, annual spending on detergents and paper products, annual spending on delicatessen products, and customer's region].
The dataset consists of 7 input variables: annual spending fresh product, annual spending on milk products, annual spending on grocery products, annual spending on frozen products, annual spending on detergents and paper products, annual spending on delicatessen products, and customer's s region. The output variable is: Class1 indicates Horeca (Hotel, Restaurant, Cafe) channel, and class2 indicates Retail channel.
Question: If the annual spending on fresh product is 7823.0, 8, annual spending on milk
products is 6245.0, annual spending on grocery products is 6544.0, annual spending frozen products is 4154.0, 0, annual spending detergents and paper products is 4074.0, annual spending on delicatessen products is 964.0, 0, customer's region (1 indicates Lisbon, 2 indicates Porto, and indicates Other) is 3, then what is the customer's 's channel? Choose between [class1, class2]. Class indicates Horeca (Hotel, Restaurant, Cafe) channel, and class2 indicates Retail channel. Answer: class2
Question: theannual spending on fresh product is 583.0, annual spending on milk "> products is 685.0, annual spending on grocery products is 2216.0, 0, annual spending on frozen products is 459.0, annual spending on detergents and paper products is 954.0, annual spending delicatessen products is 18.0, customer's region (1 indicates Lisbon, indicates Porto, and indicates 0ther) is 1, then what is the customer's s channel? Choose between [class1, class2]. Class1 indicates Horeca (Hotel, Restaurant, Cafe) channel, and class2 indicates Retail channel. Answer: classl
Choose the most important feature predict predict its value is classl class2. + Answer:
Listing 2: Prompt for correlation identification used on the Customers dataset.
15

---

Published as a conference paper at COLM 2024
Read given information and questions. Think step by step, and then predict whether its value is class1 class2. You must choose in [classl, class2]. Class1 indicates
Horeca (Hotel, Restaurant, Cafe) channel, and class2 indicates Retail channel.
The dataset consists of 7 input variables: annual spending on fresh product, annual spending on milk products, annual spending on grocery products, annual spending on frozen products, annual spending detergents and paper products, annual spending delicatessen products, and customer's region. The output variable is the customer's channel.
Question: If the annual spending on fresh product is 8190.0, annual spending milk
products is 6343.0, annual spending on frozen products is 1285.0, annual spending detergents and paper products is 1901.0, annual spending delicatessen products is 1780.0, 0, customer's s region (1 indicates Lisbon, indicates Porto, and indicates Other) is 3, then what is the annual spending on grocery products. Answer: 9794.0
Question: the annual spending on fresh product 3191.0, annual spending milk products is 1993.0, annual spending on frozen products is 1730. 073 annual spending on detergents and paper products is 234.0, annual spending on delicatessen products is 710.0, customer's region (1 indicates Lisbon, 2 indicates Porto, and 3 indicates Other) is 1, then what is the annual spending grocery products. Answer: 1799.0
Question: If the annual spending on fresh product 7823.0, annual spending on milk
products is 6245.0, annual spending on grocery products is 6544.0. annual spending frozen products is 4154.0, annual spending detergents and paper products is 4074.0, annual spending delicatessen products is 964.0, 00 customer's region (1 indicates Lisbon, 2 indicates Porto, and indicates Other) is 3, then what is the customer's s channel? Choose between [class1, class2]. Class1 indicates Horeca (Hotel, Restaurant, Cafe) channel, and class2 indicates Retail channel. Answer: class2
Question:
annual spending on fresh product 583.0, annual spending on milk products is 685.0, 0, annual spending on grocery products is 2216.0, annual spending on frozen products is 469.0, annual spending on detergents and paper products is 954.0, annual spending delicatessen products is 18.0, customer's region (1 indicates Lisbon, indicates Porto, and indicates Other) is 1, then what is the customer's s channel? Choose between [class1, class2]. Classl indicates Horeca (Hotel, Restaurant, Cafe) channel, and class2 indicates Retail channel. Answer: classl
Question: If the annual spending fresh product is 11686.0, annual spending milk
products is 2154.0, annual spending on grocery products is 6824.0, 0, annual spending paper products is 592.0, annual spending on delicatessen products is 697.0, customer's region (1 indicates Lisbon, 2 indicates Porto, and indicates Other) is 1, then what is the customer's channel? Choose between [classl, class2]. Classl indicates Horeca
on frozen products is 5527.0, annual spending on detergents
(Hotel, Restaurant, Cafe) channel, and class2 indicates Retail channel. Answer:
Listing 3: Prompt example of P2T on the Customers dataset.
16

---

Published as a conference paper at COLM 2024
Read given information and questions. Think step by step, and then predict whether its value is classl or class2. You must choose in [classl, class2]. Class1 indicates 'less than 50k', and class2 indicates 'more than 50k'.
The dataset consists of 14 input variables: age, workclass, fnlwgt, education, education-num, marital-status, occupation, relationship, race, sex, capi tal--ann,, capital-loss, hours-per-week, native-country. The output variable is the person annual income.
Question: When num_rooms is 1.0, num_people is 6.0, housearea is 895.91, is_ac is 0.0, is_tv is 0.0, flat is 0.0, num_children is 1.0, is_urban is 0.0, amount_paid is 222. 222.22476310000002, then what is the wwe_monthll_incom innomme Answer: 18822.41
Question:Wher When num_rooms is 3.0, num_people is 2.0, housearea is 676.98, is_ac is 0.0, is_tv is 1.0, is_flat is 0.0, num_children is 2.0, is_urban is 1.0, amount_paid is 711.565416, then what is the ave_monthly_innomee Answer:22900.
Question:When age is 53, workclass is Self-emp-not-ingg, fnlwgt is 169112 (69112.0, education is Bachelors, education-num is 13, marital-status is Married-civ-spouse occupation is Exec-managerial, relationship is Husband, race is White, sex is Male,
capital-gain is 0.0, capital-loss is 0.0, hours-per-week is 40, native-country is person's s annual income? Class1 indicates 'less than 50k', and class2 indicates 'more than 50k'. Choices: [class1, class2].? Answer:
Hungary, then what
Listing 4: Prompt used for zero-shot prediction on the Adult cataset by transferring knowl- edge from the Electricity dataset.
C Future work and limitations
Despite its superiority in transfer tabular learning, method is constrained by the prompt size limit of LLMs. For example, tabular data with a large number columns may not fully benefit from P2T, the result of serialized tabular data be excessively lengthy. Nonetheless, we hope that future advances LLMs, GPT-4 (OpenAI, 2023) for instance, will mitigate this issue by accommodating larger prompt sizes. Another potential strategy could involve utilizing LLMs to subsample critical features, thereby enabling more efficient integration of serialized tables into LLMs. Building this, hope that work will inspire researchers to further investigate the relatively under-explored problems associated with tabular learning via LLMs.
D Broader impacts
Tabular data often include privacy-sensitive or confidential features, such as social security numbers. As such, it is crucial to handle this data with care. However, P2T is also effective for managing anonymized features. For instance, our experiments indicate that even when categorical features are replaced with random alphabetical symbuls, and generic indicators are used instead of actual column names, P2T still shows competitive performance (see Figure 4). Therefore, despite potential privacy concerns related to tabular classification, techniques.
P2T shows promise for widespread use alongside
17

---

Published as a conference paper at COLM 2024
E Dataset details
In this section, we provide detailed descriptions of the considered datasets chosen from the OpenML repository (Vanschoren al., 2014) and Kaggle. We select 17 tabular datasets which have been previously used in the in-context learning experiments by Dinh et al. (2022); Manikandan et al. (2023). We provide detailed dataset description in Table 8.
Dataset
Source
# Col.
# Num.
Cat.
Classes
Heart-c
OpenML (49)
13
7
2
Breast
OpenML (13)
9
2
TAE
OpenMLL (48)
5
4
3
Hamster
OpenML (893)
0
2
Customers
OpenML. (1511)
7
1
2
Pollution
OpenML (882)
15
15
0
2
Diabetes
OpenML (37)
8
0
2
Car
OpenMLL (40975)
6
6
4
BTC
OpenMLL (1464)
4
4
0
2
Haberman
OpenML (43)
3
2
Caesarian
OpenMLL (42901)
2
VC
OpenML. (1524)
6
6
2
Adult
OpenML. (1590)
8
2
Credit-g
OpenMLL (31)
20
12
2
Credit-R
OpenML (43454)
11
3
2
Electricity
OpenMVL (43588)
8
0
2
Credit-A
OpenML (29)
15
9
2
Salaries
Kaggle
9
8
2
Table 8: Dataset description. We select 17 tabular datasets from the OpenML repository (Vanschoren et al., 2014) for evaluation. We denote OpenML dataset id in parentheses. One dataset is from Kaggle.
18


Reviews:
[{"review_id": "JxiZQCLsOP", "reviewer": "Reviewer_kaWX", "summary": "The paper presents a novel framework, Prompt to Transfer (P2T), which leverages large language models (LLMs) for transfer learning in the domain of tabular data. The authors address a significant problem in machine learning applications where labeled data is scarce, and they propose a solution that creates pseudo-demonstrations from unlabeled or heterogeneous source data to enhance the learning process. The paper claims that P2T outperforms existing methods on various tabular learning benchmarks.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the review. For additional experiments, we perform 1-shot classification using unlabeled data as the source (cf. Table 2).\n\n**How does P2T exploit the capabilities of LLMs?**\nP2T mainly consists of three components to thoroughly exploit LLM’s capabilities. First, the LLM processes heterogeneous tables within a unified language space. Second, we leverage ICL to extract transferable knowledge from the source data. Finally, we carefully design our prompts by referring to some prompting skills (e.g., using “Think step by step.”).\n\n**Result with Llama 3 70B, an open-source LLM.**\nThe table below shows the compatibility of P2T with open-source LLMs.\n\n\\begin{array}{lccc}\\hline\n&\\text{Customers}&\\text{BTC}&\\text{Haberman}\\newline\\hline\n\\text{Best non-LM baseline}&65.1&53.0&53.8\\newline\n\\textbf{P2T (Ours)}&72.1&53.6&60.0\\newline\\hline\n\\end{array}\n\n**More attempts on pseudo-demonstrations.**\nWe carefully note that we have already considered some design choices in Figure 3. Specifically, using the random column feature as the target of the pseudo-demonstrations performed worse than our design. Such a suboptimal design can make P2T perform worse than not utilizing the transfer source.\n\n**LLMs’ superiority for correlation identification.**\nWe tried CatBoost for comparison. As shown in the table below, using the column feature selected by CatBoost (which differs from LLM’s selection) to construct the pseudo-demonstration performs worse. We believe that this is because (i) CatBoost often does not perform well with few-shot examples (cf. Table 2), and (ii) LLM, on the other hand, can understand the linguistic context of the task.\n\n\\begin{array}{lccc}\\hline\n\\text{Identification}&\\text{Customers}&\\text{BTC}&\\text{Haberman}\\newline\\hline\n\\text{CatBoost}&69.3&62.0&61.0\\newline\n\\textbf{LLM (Ours)}&74.3&62.3&61.3\\newline\\hline\n\\end{array}\n\n**Number of test samples.**\nWe use 20% of the dataset as test samples (cf. Section 4). The exact values will be included in the final revision.\n\n**Using multiple datasets.**\nP2T can be extended to use multiple datasets by constructing pseudo-demonstrations from each transfer source. However, the datasets need to be closely related so that benchmarking will take time. Therefore, we will try our best to incorporate results using multiple datasets in the final revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The paper introduces a unique approach to transfer learning in tabular data by utilizing LLMs, which is a novel contribution to the field.\n2) The authors provide extensive experimental results demonstrating the effectiveness of P2T across diverse datasets, which supports the significance of their work.\n3) The method's ability to operate without the need for training makes it accessible to environments with constrained hardware resources.", "reasons_to_reject": "1) The proposed method in the paper is too simplistic and does not fully exploit the capabilities of large language models. There is a lack of exploration into the characteristics of tabular data. \n2) The method in the paper requires calling third-party APIs, which raises privacy concerns in real-world applications. It is suggested that the authors consider using open-source large language models. \n3) While the effectiveness of using Pseudo-demonstrations has been verified, there should be more attempts and comparative analyses in constructing Pseudo-demonstrations. \n4) The section on Correlation Identification needs to experiment with conventional algorithms to demonstrate the superiority of using large language models.", "questions_to_authors": "1) How many test samples are there in each dataset? \n2) The experiments in the paper only involve the case where the transfer source is a single dataset. Is it feasible to use multiple datasets?", "extra_scores": {}}, {"review_id": "Ss5kG0B1UW", "reviewer": "Reviewer_DX82", "summary": "This paper proposes a framework called \"Prompt to Transfer\" (P2T) for performing transfer learning on tabular data using large language models (LLMs) and in-context learning. The key idea is to create \"pseudo-demonstrations\" from unlabeled or heterogeneous source data that are relevant to the target task, and then prompt the LLM with these demonstrations along with a few labeled examples from the target dataset. Specifically, P2T first identifies the column feature in the source data that is most correlated with the target task label. It then constructs pseudo-demonstrations where that column is treated as the prediction target using the other columns as input features. These demonstrations capture knowledge about predicting the correlated column, which can transfer to the actual target task. Experiments on various tabular datasets show P2T consistently outperforms previous methods in few-shot and zero-shot transfer learning settings.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the review. For additional experiments, unless otherwise noted, we perform 1-shot classification using unlabeled data as the source (cf. Tab. 2).\n\n**Analysis, e.g., empirical evidence.**\nA common consensus for improving ICL is to use good demonstrations. Therefore, pseudo-demonstrations indeed resemble real causal relationships (cf. Fig. 3). Here, we assume that the features in the source must related to the target. Using unrelated data will break the assumption, e.g., using Electricity (entirely different from Heart-c) as the source degrades zero-shot accuracy on Heart-c from 60.0 to 58.3.\n\n**Data w/ many columns.**\nOne strategy is to ask the LLM to select the important features for the task and then run P2T only with them. As shown below, with half of the features (but informative), P2T outperforms the best baseline. Therefore, P2T can be scaled efficiently to datasets with many columns.\n\n\\begin{array}{lccc}\\hline\n&\\text{Customers}&\\text{BTC}&\\text{Haberman}\\newline\\hline\n\\text{Best baseline (All col.)}&70.5&60.4&60.3\\newline\n\\textbf{P2T (Half col.)}&72.7&62.1&61.0\\newline\\hline\n\\end{array}\n\n**Regression.**\nWe evaluate P2T in 10-shot regression, reporting scaled MSE. As shown below, P2T is a competitive approach for a few-shot regression. However, the improvement is relatively small compared to classification. We believe this is because LLM often suffers from dealing with numerics, leaving it as a future work to address this.\n\n\\begin{array}{lccc}\\hline\n&\\text{Laptops}&\\text{House}&\\text{Cholesterol}\\newline\\hline\n\\text{kNN}&9.3&4.2&3.1\\newline\n\\text{LR}&7.5&15.6&2.3\\newline\n\\text{LIFT}&4.2&3.7&2.6\\newline\n\\textbf{P2T}&4.0&3.4&2.5\\newline\\hline\n\\end{array}\n\n**Choice of the correlated feature.**\nWe’ve already shown that P2T outperforms other possible choices in Fig. 3. Specifically, using random features can lead to worse performance because the wrong columns may be selected.\n\n**Handling missing values.**\nWe verify P2T’s robustness to missing values, where 50% of the features are randomly omitted. Specifically, P2T naturally handles missing values by excluding them from the prompt. As shown below, P2T outperforms the baselines (using zero imputation) and shows a small performance drop when compared to LR’s 7% drop on Customers (cf. Tab. 2).\n\n\\begin{array}{lccc}\\hline\n&\\text{Customers}&\\text{BTC}&\\text{Haberman}\\newline\\hline\n\\text{LR}&53.9&51.4&51.4\\newline\n\\text{LIFT}&64.7&56.0&60.0\\newline\n\\textbf{P2T}&70.0&56.9&61.0\\newline\\hline\n\\end{array}", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Tabular data is very common across domains like healthcare, finance, etc., but tabular transfer learning has received little attention compared to areas like vision and NLP. The P2T framework is specifically designed for tabular datasets. Developing effective transfer learning methods for tabular data is a significant challenge due to heterogeneity across datasets. P2T tackles this by leveraging large language models and in-context learning in an innovative way.\n- Annotating tabular data is expensive. An effective transfer learning solution that can leverage unlabeled/heterogeneous sources can have substantial practical impact across many data-rich fields.", "reasons_to_reject": "- The paper is largely empirical in nature and lacks deeper analysis of the proposed P2T framework. Without principled justifications, the work may be perceived as somewhat ad-hoc or heuristic. \n- The experiments mainly focus on classification tasks (mostly binary classification) across a limited set of 17 tiny tabular datasets. Evaluating P2T on more datasets, especially larger-scale ones, as well as other tasks like regression could improve the generalizability of the results.", "questions_to_authors": "1. In the pseudo-demonstration construction, how sensitive is P2T's performance to the choice of the correlated column feature identified by the language model? What happens if the wrong column is selected?\n\n2. The pseudo-demonstrations are constructed based on the assumption that predicting the correlated column feature is a good proxy for the original target task. However, what theoretical justification or empirical evidence do you have that this assumption holds across different types of tabular datasets? Are there scenarios where this assumption could break down?\n\n3. The current approach relies on enumerating and scoring all columns to identify the most correlated feature. How does this scale to datasets with hundreds or thousands of columns? Have you explored more efficient strategies?\n\n4. What about handling missing values or noisy data entries commonly presented in tabular datasets? How can the framework be extended to be more robust and handle realistic data issues?", "extra_scores": {}}, {"review_id": "DnF22FUwLg", "reviewer": "Reviewer_ijuV", "summary": "This paper presents a technique called P2T for transfer learning in tabular data. The core idea of the technique is as follows: supposed you want to use LLMs for doing few or zero shot learning in a domain T where the target prediction is column t, and you have access to data from related but perhaps not exactly the same domain S. First you ask an LLM to identify the column in S that is most relevant to t; let's say s. Then you create few shot examples from S where the target prediction is column s. Each of these examples is essentially descriving the row in natural language. Additionally you can also add few examples from domain T. This makes your prompt. Using this prompt, you can ask the LLM to predict the value of t for any incoming example from T. \n\nThe main claim of the paper is that this technique improves prediction in domain T and outperforms techniques like LiftICL. In paticular the claims of the paper are\n1. When asking the LLM for most relevant target column s, and including those examples in the prompt, improve 1 shot (i.e. 1 example per example from target domain) accuracy of the model, even if S is heterogenous from T.\n2. Which column is selected for transfer from S, matters. Performance drops if you take a random column as target.\n3. Including unlabeled examples from domain T also improves the performance.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Thanks for the review. For additional experiments, unless otherwise noted, we perform 1-shot classification using unlabeled data as the source (cf. Tab. 2).\n\n**P2T on TabZilla.**\nTabZilla and ours overlap (both are subsets of OpenML). Nevertheless, we considered more datasets in TabZilla to alleviate your concerns. The final draft will include more datasets.\n\n\\begin{array}{lcc}\\hline\n&\\text{heart-h}&\\text{profb}\\newline\\hline\n\\text{CatBoost}&67.3& 50.0\\newline\n\\textbf{P2T}&75.5&54.5\\newline\\hline\n\\end{array}\n\n**Releasing prompts.**\nWe will release the prompts in the final draft (some are in Appendix A).\n\n**Using unlabeled data.**\nIn this case, P2T constructs pseudo-demonstrations using the most correlated feature (among the input features) as a target. Here, making tasks from unlabeled data that resemble the causality of real input-output pairs is non-trivial, which is our unique contribution, making it different from LIFT-ICL.\n\n**LR, kNN in Table 3.**\nTo clarify, we concatenate every column in both datasets based on the overlapping columns. Therefore, a classifier fitted on one dataset can still be meaningful on the other. Nevertheless, non-LM methods (including the suggested reference) cannot account for non-overlapping columns, while P2T can model the semantic relations between them, even if not present in the transfer source.\n\n**Ablation: Closeness of the datasets.**\nAs shown in Tab. 1, using unlabeled data (close) as a source usually performs better than using heterogeneous data (far). To further alleviate concerns about tasks being far apart, we performed a zero-shot prediction on Heart-c. As shown below, using Diabetes as the source (both are medical data) improves accuracy, while using Electricity, which is entirely different, degrades the performance. This indicates that datasets that are too far from the target task may hinder LLM from outputting correct answers.\n\n\\begin{array}{lcc}\\hline\n&\\text{Source}&\\text{Acc.}\\newline\\hline\n\\text{zero-shot}&-&60.0\\newline\n\\textbf{P2T}&\\text{Electricity}&58.3\\newline\n&\\text{Diabetes}&65.0\\newline\\hline\n\\end{array}\n\n**Fine-tune?**\nWe utilize ICL to exploit increasingly large LLMs (cf. Tab. 4). Fine-tuning may result in better performance, but it may be too expensive.\n\n**Sensitivity: Naming styles.**\nUsing clear language descriptions is important for performance. Performance drops when using generic indicators (cf. Fig. 4). However, even here, performance is improved compared to not using a transfer source.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper provides a simple and useful idea for transfer learning when using LLMs for tabular data prediction. The presented approach is simple, even incremental, but still useful. \n2. The experiments for the most part are useful, and the results for 1-shot heterogenous transfer is particularly quite nice.", "reasons_to_reject": "1. The openml data is particulatrly simplistic. Could the authors also try to replicate this work on TabZilla which seems more suitable for comparisons between various LLM techniques as well as LLM vs other ML techniques. https://openreview.net/pdf?id=CjVdXey4zT\n\n2. Given the surpising nature of the outcomes, I do want to hold the bar on release of prompts and outcomes used for comparing p2t vs lift-icl e.g. so the community can help spot check the outcomes, and since it is trivial to verify the outcomes.", "questions_to_authors": "1. I am a bit confused by the section on including unlabeled examples. In what way are they included in the prompt? Why can't they also be included in LIFT-ICL and how does P2T even differ from LIFT-ICL in this case (since there is no source domain so to speak)? The finding that including unlabeled examples in the prompt helps has been brought up elsewhere as well (https://arxiv.org/pdf/2404.11018) and seems orthogonal to the core p2t idea itself. What am I missing here?\n\n2. What does inclusiion of LR and kNN in table3 mean? is LR trained on source and target data combined and if so, how does transfer happen in this case (is it via just common regularization)? If so, it is perhaps better to follow the tehnique here: https://arxiv.org/pdf/0907.1815.\n\n3. The finding that adding heterogenous source improves the performance significantly is quite surprising. I'd highly encourage the authors to release all of their prompts for this finding for independent verification (which should be trivial).\n\n4. How does the performance change with the closeness between two datasets? What happens if the tasks are far apart?\n\n5. If I were allowed to fine tune a model on source examples, how does the performance change? \n\n6. How sensitive is this approach to different naming styles or conventions across the data?", "extra_scores": {}}, {"review_id": "6V6dQBU1QK", "reviewer": "Reviewer_VukR", "summary": "This paper proposes a novel framework, Prompt to Transfer (P2T), which utilizes large language models (LLMs) for tabular transfer learning. The methodology leverages in-context learning (ICL) by creating pseudo-demonstrations from unlabeled or heterogeneous source datasets to predict target tasks effectively. The paper is well-structured, offering a clear introduction to the problem and a detailed explanation of the approach. The experimental results show that P2T significantly outperforms existing methods across various datasets, to support the significance of the work.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the review. We respond to each of your comments one by one regarding what follows.\n\n----\n**Clarification on how the pseudo-demonstrations are generated and utilized.**\n\nWe would like to clarify how P2T constructs a pseudo-demonstration. First, P2T identifies the column feature that is most correlated to the task. Then, the pseudo-demonstration is constructed with language descriptions of input-output pairs, where the output is the column feature selected in the previous step, and the input is the remaining column features. Finally, the pseudo-demonstration is concatenated into a single prompt with a few-shot labeled demonstration, which is fed to the LLM.\n\nIn the meantime, some example prompts are provided in Appendix A, and other prompts will be released in the final version.\n\n----\n**Discussion on the selection of source and target datasets.**\n\nAs shown in Table 1, using unlabeled samples as the transfer source usually performs better than using heterogeneous datasets. This indicates that better performance can be achieved by using a dataset closer to the target dataset (unlabeled samples consist of all the column features of the target task except for the label column).\n\nTo further alleviate concerns, we performed a more in-depth case study on zero-shot classification on the Heart-c dataset (see Table 1 for experimental setup). As shown in the table below, using the Diabetes dataset as the transfer source (Heart-c and Diabetes are both medical datasets) improves the zero-shot performance, while using the Electricity dataset, which is entirely different in nature from Heart-c, degrades the performance. This indicates that datasets that are too far from the target task may hinder LLM from outputting correct answers.\n\n\\begin{array}{lcc}\\hline\n&\\text{Source}&\\text{Acc.}\\newline\\hline\n\\text{zero-shot}&-&60.0\\newline\n\\textbf{P2T (Ours)}&\\text{Electricity}&58.3\\newline\n&\\text{Diabetes}&65.0\\newline\\hline\n\\end{array}", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The P2T framework introduces a novel use of pseudo-demonstrations for tabular transfer learning, which is a significant contribution to the field of machine learning with LLMs.\n\n- The experiments conducted show that P2T consistently outperforms traditional and contemporary methods in zero-shot and few-shot learning scenarios. This is clearly demonstrated with extensive experimental validation.\n\n- The research addresses a real-world problem of data scarcity in tabular datasets, making the findings relevant for various applications, including finance and healthcare.", "reasons_to_reject": "- Some aspects of the methodology, particularly how the pseudo-demonstrations are generated and utilized, could be described more transparently to enhance the reproducibility of the results.\n\n- There is insufficient discussion on how the selection of source and target datasets might affect the generalizability of the P2T framework, which could be a concern for adopting this method in broader applications.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a simple yet effective method that uses LLMs for transfer learning on tabular tasks. The main idea is to create "pseudo-demonstrations" identify from unlabeled source data and then prompt the LLM with these demonstrations + (optionally) labeled examples from the target dataset to do the final task. Experiments on a range of tables and tasks from OpenML demonstrate the effectiveness of the proposed method.

Reasons to Accept
- Transfer learning on tabular data is important due to the scarcity and cost of labeled data. Using LLMs in the proposed manner is a plausible approach.
- Reasonably well-written paper.

Reasons to Reject
- A major criticism, as pointed out by DX82 and ijuV and I also concur, is the simplicity and small size of the tabular tasks being used. Tabular ML has come a long way and there are many datasets and tasks that have a level of complexity closer to real-world application scenarios. See TURL: Table understanding through representation learning and references therein and thereof for some pointers. As it currently stands, it's not clear how well the proposed method will generalize to real-world tables and tabular tasks that could be more complex.
Decision: Accept