Tabular Transfer Learning via Prompting LLMs

Authors: jaehyun  nam
woomin  song
seong hyeon park
jihoon  tack
sukmin  yun
jaehyung  kim
kyu hwan oh
jinwoo  shin
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-20 11:33:49

Abstract: Learning with a limited number of labeled data is a central problem in real-world applications of machine learning, as it is often expensive to obtain annotations. To deal with the scarcity of labeled data, transfer learning is a conventional approach; it suggests to learn a transferable knowledge by training a neural network from multiple other sources. In this paper, we investigate transfer learning of tabular tasks, which has been less studied and successful in the literature, compared to other domains, e.g., vision and language.
This is because tables are inherently heterogeneous, i.e., they contain different columns and feature spaces, making transfer learning difficult. On the other hand, recent advances in natural language processing suggest that the label scarcity issue can be mitigated by utilizing in-context learning capability of large language models (LLMs). Inspired by this and the fact that LLMs can also process tables within a unified language space, we ask whether LLMs can be effective for tabular transfer learning, in particular, under the scenarios where the source and target datasets are of different format. As a positive answer, we propose a novel tabular transfer learning framework, coined Prompt to Transfer (P2T), that utilizes unlabeled (or heterogeneous) source data with LLMs. Specifically, P2T identifies a column feature in a source dataset that is strongly correlated with a target task feature to create examples relevant to the target task, thus creating pseudo-demonstrations for prompts. Experimental results demonstrate that P2T outperforms previous methods on various tabular learning benchmarks, showing good promise for the important, yet underexplored tabular transfer learning problem. Code is available at https://github.com/jaehyun513/P2T.

Main paper: 
 introduction: 
 Learning with a limited number of labeled samples is often a critical requirement in realworld machine learning applications. This limited data problem is particularly important in the tabular domain; tabular datasets often require substantial annotation efforts (e.g., credit risk assessment; Clements et al., 2020) , or it is hard to obtain new samples for emerging tasks (e.g., identifying patients with new diseases such as COVID-19; Peplow, 2016; Zhou et al., 2020) . To address this problem, various methods have been thoroughly studied in domains such as vision (Assran et al., 2021; Pham et al., 2021) and language (Chen et al., 2021; Min et al., 2022) . However, the research on tabular data has only recently begun to gain traction (Yoon et al., 2020; Nam et al., 2023) , despite its wide-ranging impact across a variety of industries (Guo et al., 2017; Ulmer et al., 2020; Zhang et al., 2020) .Learning transferable knowledge by training a neural network from various sources (Chen et al., 2020; Perez et al., 2021; Lee & Shin, 2022) is a common way to address this limited data problem in other domains. However, such transfer learning is challenging in the tabular domain, because the source and target datasets are often very heterogeneous (i.e., they have different columns and feature spaces; Zhu et al., 2023; Wang & Sun, 2022; Yan et al., 2024) . For example, using the collected features to predict diabetes to predict whether another patient has breast cancer is not straightforward because the required features to predict each disease (i.e., columns in a table) are very different. This leads to the question of how transferable knowledge can be extracted from various tabular data sources in a unified space.On the other hand, converting tabular data into text has recently gained attention as this conversion leads to several benefits; first, one can inject linguistic context (Dinh et al., 2022; Hegselmann et al., 2023; Manikandan et al., 2023) , for example, using column name 'Age' along with its numerical value. In addition, it opens up using large language models (LLMs) for tabular learning, which have demonstrated the capability to address limited data problems with few task-specific instructions (Brown et al., 2020; Dong et al., 2022; Wei et al., 2022; Kim et al., 2024) , known as prompting (or in-context learning; ICL).Motivated by this, we ask whether LLMs can be tabular transfer modules and address limited data problems in the tabular domain. As a positive answer, we propose a novel tabular transfer learning framework that utilizes LLM's ICL capabilities, coined Prompt to Transfer (P2T); see the overview in Figure 1 . Our main idea is to use LLM to extract transferable knowledge from the source dataset (e.g., unlabeled dataset) and use it as in-context samples when prompting the LLM (called pseudo-demonstrations).Specifically, P2T starts by prompting LLM to determine which column feature is most important for the target task based on given few-shot labeled samples (i.e., demonstrations). P2T then creates pseudo-demonstrations that describe the prediction task where the selected column feature is the target and the remaining ones are input, thus ensuring high relevance to the actual target task. These pseudo-demonstrations reveal how to predict the selected column feature from the remaining features and this knowledge would be useful to the original target task (e.g., knowledge to predict 'Insulin' could be useful to predict 'Diabetes'). Therefore, P2T prompts the LLM to generate the desired output through the created pseudo-demonstrations with few-shot labeled demonstrations.We verify the effectiveness of P2T, by conducting comprehensive evaluations on diverse tabular learning scenarios considering different source types (i.e., unlabeled and heterogeneous datasets) and the number of labeled samples (i.e., few-shot and zero-shot). Our results show that P2T significantly and consistently outperforms existing methods, including self-supervised (Yoon et al., 2020) and unsupervised meta-learning (Nam et al., 2023) methods, by transferring knowledge through prompting. As LLMs recently continue to advance (e.g., GPT-4; OpenAI, 2023) , improved performance is expected with future models. 
related work: 
 Tabular transfer learning. Researchers have developed a number of methods to train transferable representations for tabular data. One major stream of work exploits unlabeled datasets. In this context, Yoon et al. (2020) and Ucar et al. (2021) introduced various pre-text task losses (e.g., reconstruction loss) with the goal of self-supervised learning on tabular datasets. In addition, Nam et al. (2023) proposed an unsupervised meta-learning framework to address few-shot tabular learning problems. Another stream of work exploits multiple heterogeneous datasets. Zhu et al. (2023) used a federated learning approach with separate encoders for each dataset to aggregate the parameters into a single transformer. Wang & Sun (2022) and Yan et al. (2024) used tokenizers to map the heterogeneous structure of tables into a unified language space and then fine-tuned the language model. We propose a transfer learning framework that leverages both unlabeled and heterogeneous sources while applying in-context transfer to enable immediate predictions (i.e., training-free).Tabular learning with large language models. Recent advances in LLMs have provided an impetus to explore their potential for tabular learning. Dinh et al. (2022) investigated the performance of fine-tuned GPT-3 models (Brown et al., 2020) on tabular data. Extending this line of research, Hegselmann et al. (2023) conducted a comprehensive analysis using the T0 model (Sanh et al., 2022) , leveraging the language prior in LLMs. Their analysis is extended to sample efficiency, even conducting zero-shot experiments. Recently, Manikandan et al. (2023) developed a boosting framework over prompting LLMs that uses LLMs as weak learners in tabular prediction tasks. Inspired by these preceding studies, our work proposes a method for the effective exploitation of various transfer sources -an aspect overlooked in prior research. By integrating the utilization of unlabeled (or heterogeneous) data with LLMs, we aim to enhance performance in transfer learning scenarios.In-context learning. As model and dataset sizes increase (Brown et al., 2020) , LLMs have exhibited the capability for ICL (or prompting), where they draw knowledge from a handful of contextual examples. For example, Wei et al. (2022) have illustrated the competency of LLMs in solving mathematical reasoning problems via ICL. The ICL process begins by employing a small number of examples to establish a contextual framework, typically constructed using natural language templates. Following this, a query question and a contextual demonstration are combined to form a prompt, which is subsequently fed to the LLMs for prediction. Notably, ICL does not necessitate parameter updates and directly carries out predictions using LLMs, enabling easy implementation for real-world applications. In our work, we delve deeper into the potential of ICL by examining its performance on transfer learning, using source data for creating effective demonstrations. 
p2t: prompt to transfer: 
 In this section, we propose a simple yet effective tabular transfer learning that leverages ICL, the capability of LLMs that adapt to a new task using the context provided in a prompt without updating model parameters. In a nutshell, our framework creates pseudodemonstrations from transfer sources that act as proxies for causal relationships between input features and labels, and then prompt the LLM to make predictions. We first briefly describe preliminaries (Section 3.1), and then the core component coined Prompt to Transfer (P2T), which creates effective pseudo-demonstrations for ICL (Section 3.2).Problem setup: tabular transfer learning. We first describe the problem setup of our interest. A labeled target dataset D t = {(x t i , y t i )} N t i=1 ⊆ X t × Y and a source dataset D s = {x s i } N s i=1 ⊆ X s are given, where x t is d t -dimensional, and x s is d s -dimensional feature which correspond to the value in the respective table columns. Also, we assume that column name sets F = { f 1 , • • • , f d } are given as well (e.g., 'x 1 : Male' is feature values of the ' f 1 : sex' column), and f target is the column name of the labels y of the target dataset. Labels y ∈ Y are provided in the form of natural language annotations (e.g., 'Non Diabetic' and 'Diabetic' are labels of the Diabetes dataset). Here, the cardinality of D t is assumed to be much smaller than the source dataset, i.e., |D t | ≪ |D s |. Our goal is to effectively extract knowledge from D s to predict the accurate label of test query x t test in the target dataset. In-context learning for tabular prediction tasks. We now describe ICL-based tabular prediction (Dinh et al., 2022) . Similar to ICL in the natural language domain (Brown et al., 2020) , tabular prediction via ICL is executed by constructing an input prompt comprising (i) a task description p task (D t ), (ii) few-shot labeled demonstrations p label (D t ) (i.e., descriptions of input-output pairs of the labeled samples), and (iii) a test query p test (x t test ). To begin, the task description is manually crafted through a systematic procedure. This description includes prompts directing the LLM to analyze the data in a step-by-step manner, and the dataset's details such as column descriptions. Next, few-shot labeled demonstrations are created by serializing tabular data into natural language, thereby presenting the data in a question format. There exist numerous design choices for tabular serialization; however, we mainly follow Dinh et al. (2022) . Finally, we predict the output by prompting LLM M:y pred = M(p task (D t ) ⊕ p label (D t ) ⊕ p test (x t test )), where, ⊕ is the concatenation of each prompt (see the conventional prompting in Figure 1 ).We now present P2T, a novel approach to improve tabular prediction performance by creating additional pseudo-demonstrations from the source data, which serve as proxies for few-shot labeled demonstrations. This is achieved by (i) first identifying the column feature in the source dataset that exhibits the highest correlation with the target f target and (ii) then constructing pseudo-demonstrations from the transfer source. The constructed prompt p P2T , which integrates pseudo-demonstrations with conventional prompting, is then fed into the LLM M to generate the desired prediction output.𝒑 𝐜𝐨𝐫 : Identification instruction Q: If insulin is 130 µU/ml, BMI is 37.9, age is 21, is the patient diabetic? A: No Q: If insulin is 210 µU/ml, BMI is 42.9, age is 36, is the patient diabetic? A: Yes … Choose the most important feature to predict whether a patient is diabetic. Choices: [Insulin, BMI, Age] 🤖 Insulin Correlation identification. P2T begins by identifying the feature f k ∈ F s (the column name set of the source data) that holds the most significant correlation with the task column f target in the target data D t . To achieve this, we ask LLM M which feature among F s is most important for predicting f target . Formally, an input prompt designed to identify correlations consists of two main components: (i) few-shot labeled demonstrations p label (D t ) and (ii) identification instructions p cor (F s , f target ); see Figure 2 . Finally, we prompt LLM M to obtain f k :f k = M(p label (D t ) ⊕ p cor (F s , f target )).Our choice of using LLMs over conventional methods (Chen & Guestrin, 2016; Prokhorenkova et al., 2018) is driven by the capabilities of LLMs to interpret linguistic context; their ability to leverage both the semantics of column names and the associated numeric values offers a distinct advantage. For instance, considering a column like 'Age,' conventional algorithms would focus solely on the numeric values. Conversely, LLMs can utilize the semantic understanding of 'Age,' incorporating additional contextual information.Pseudo-demonstrations from the source tables. Our main idea is to create pseudodemonstrations from the unlabeled (or heterogeneous) source tables using highly correlated column feature f k as a new target feature. The rationale behind using f k stems from the intuition that predicting the most correlated column feature from the remaining features would resemble the original task of predicting the label from all the features; therefore, these pseudo-demonstrations from source data include the useful knowledge to predict the original target task. For instance, predicting 'Diabetes' from 'BMI' and 'Age' is similar to predicting 'Insulin' using the same features (Nam et al., 2023) . Thus, we create pseudodemonstrations that predict the value of f k from remaining features, which we refer to as p pseudo (D s , f k ). See Figure 1 for a specific example of how P2T creates pseudo-demonstrations from an unlabeled dataset. As shown in Figure 1 , 'Inuslin' is selected as the most important feature f k for predicting whether a patient has diabetes (see Figure 2 ). Therefore, predicting insulin levels from BMI and age becomes components of pseudo-demonstrations. Finally, our P2T framework outputs prediction results as follows:p P2T (D t , D s , f k , x t test ) = p task (D t ) ⊕ p pseudo (D s , f k ) ⊕ p label (D t ) ⊕ p test (x t test ), y pred = M(p P2T (D t , D s , f k , x t test )). 
experiments: 
 In this section, we validate the effectiveness of our proposed method for transfer learning scenarios on a variety of tabular datasets from the OpenML repositories (Vanschoren et al., 2014) and Kaggle. First, in Section 4.1, we verify that P2T improves zero-shot prediction performance by leveraging different types of transfer sources (i.e., unlabeled and heterogeneous datasets). Note that zero-shot prediction is one of the big benefits of using LLMs that is not possible with traditional methods like CatBoost (Prokhorenkova et al., 2018) . Then, in Section 4.2, we verify that our method outperforms other tabular learning methods, including unsupervised meta-learning methods (Nam et al., 2023) , in the few-shot learning scenario by extracting effective knowledge from the transfer sources via prompting. Finally, in Section 4.3, we validate the effectiveness of our proposed pseudo-demonstration and that better performance can be achieved with a more advanced LLM (i.e., GPT-4; OpenAI, 2023) .Common setup and baselines. When using unlabeled data as the training source, we use 20% of the labeled dataset for test samples and convert the remaining 80% to unlabeled, except for a limited number of labeled samples. Following Nam et al. (2023) , we one-hot encode categorical features for the baseline and then min-max scaling. To validate P2T, we consider supervised learning baselines such as CatBoost (Prokhorenkova et al., 2018) , logistic regression (LR), and nearest neighbor classifier (kNN) that do not utilize unlabeled (or heterogeneous) data. We also consider VIME (Yoon et al., 2020) , a self-supervised learning baseline where the model is initially pre-trained and then evaluated using labeled samples via logistic regression. Note that we intentionally exclude methods that require careful hyperparameter tuning, e.g., MT (Tarvainen & Valpola, 2017) , MPL (Pham et al., 2021) , ICT (Verma et al., 2022) Table 2 : Few-shot test accuracy (%) using unlabeled samples as transfer source. # shot indicates the number of labeled samples per class. For the baselines, we report the average test accuracy over 100 different seeds. We report the average accuracy and standard deviation over 5 different seeds for LIFT-ICL (Dinh et al., 2022) and our method, due to the high cost of OpenAI API. The bold denotes the highest average score.STUNT (Nam et al., 2023) , a state-of-the-art few-shot tabular learning method. Finally, we consider LIFT (Dinh et al., 2022) in the ICL setting (LIFT-ICL) as a representative way to exploit the power of LLM. In all experiments using LLM, we use gpt-3.5-turbo.One of the distinct advantages of using LLMs is that they can easily obtain the desired answer in a zero-shot manner. In fact, Hegselmann et al. (2023) have shown that LLMs can perform zero-shot prediction tasks even in tabular domains. In this section, we investigate whether the proposed P2T framework can improve the performance of zero-shot prediction by transferring knowledge from unlabeled and heterogeneous datasets, respectively. We emphasize that P2T can be seen as a zero-shot transfer module because it is based on the ICL.As shown in Table 1 , using a transfer source improves the zero-shot prediction performance. We first found that LLM (GPT-3.5 in our case) often fails to predict with zero-shot. For example, on the Credit-g dataset, which is a binary classification, LLM scores behind random guessing (see Table 1 ). However, using heterogeneous sources (i.e., Credit-A) or unlabeled samples of the Credit-g dataset, P2T improves zero-shot prediction accuracy by 16.0% and 24.0%, respectively (note that we randomly select 30 samples from the transfer sources due Number of samples from a source dataset (N) to the high cost of the OpenAI API). We also note to provide intuition that heterogeneous datasets should come from similar domains. For example, the Credit-g dataset and the Credit-A dataset are both financial datasets consisting of an individual's credit information.Target Source Method N = 0 N = 2 N = 4 N = 6 N = 8 N =For few-shot tabular prediction, we evaluate the performance when one and five labeled samples are available per class, respectively. Firstly, we found that taking few-shot labeled tabular data, serializing it into text and prompting it to LLMs (i.e., LIFT-ICL in Table 2 ) outperforms existing state-of-the-art methods for few-shot tabular prediction (e.g., STUNT; Nam et al., 2023) . Thus, using LLMs as prediction models is indeed a promising direction for tabular learning, and the P2T further extends this line of work by using LLMs as transfer modules via prompting. In this section, we provide few-shot tabular prediction results using (i) unlabeled data and (ii) heterogeneous data as the transfer source, respectively.Transfer learning utilizing unlabeled dataset. As demonstrated in Table 2 , P2T significantly and consistently improves the few-shot prediction performance on 12 tabular datasets by utilizing unlabeled data of the same dataset as transfer source (here, we use 30 unlabeled samples that are closest to each labeled sample in terms of Euclidean distance). Note that this improvement is achieved without model updates. To provide a specific example, P2T significantly outperforms LIFT-ICL in 1-shot classification, raising the average performance from 58.10% to 64.92%. Additionally, P2T consistently achieves superior results, yielding the highest score in all 12 datasets in the 1-shot classification problem, and in 11 out of the 12 datasets in the 5-shot scenario. These results represent an improvement of approximately 6.8% and 6.6% over the best performing baselines, respectively. The significant improvement is achieved by constructing effective pseudo-demonstrations from the unlabeled dataset, therefore extracting valuable information in an in-context manner.Transfer learning utilizing heterogeneous dataset. We next demonstrate the effect of introducing training samples from heterogeneous data sources. First, we found that merging distinct column sets from diverse sources in the tabular domain demands a heuristic process to create a unified feature set. Such an approach may not generalize well, and require sophisticated designs for different data combinations. In this regard, we find our tabular serialization to be simple and effective method for combining columns from various heterogeneous sources. As tabular data is transformed into natural language, the language model can automatically understand the relations between different features from their descriptions. To investigate the effect of incorporating heterogeneous data, we consider a transfer scenario where the target data should be classified given 1-shot training samples from the same dataset and N additional samples from a heterogeneous source dataset that contains disparate column sets (e.g., 'work class,' 'education,' etc. for the Adult dataset and 'loan amount,' 'credit history length,' etc. for the Credit-R dataset in Table 3 ).As shown in Table 3 , P2T consistently benefits from heterogeneous data source. For example, 74.67%→80.00% on the Adult dataset, when 10 additional samples from the Credit-R dataset is provided. More importantly, P2T is the only method that shows steady performance improvements as the number of heterogeneous training samples N increases, while the baselines are not able to properly learn from the additional samples, and their performance could even deteriorate compared to their 1-shot performances. Interestingly, cramming the extra columns from the heterogeneous dataset in LIFT-ICL only incurred noise to the accuracy. We attribute this to that our P2T framework enables the LLMs to exploit the deeper relationship between heterogeneous data, while the naïve concatenation without proper descriptions only perplexes the LLMs.In this section, we perform further analysis of the proposed P2T framework. (i) First, we use the column feature in the source dataset that is most relevant to the target task label as useful target for creating pseudo-demonstrations. Here, we ask whether the created pseudodemonstrations are really similar to the real task, i.e., if we create pseudo-demonstrations with an arbitrary column feature of the source data as a target, the created demonstrations may not be similar to the real task and hence the LLM may not convey knowledge well. (ii) Second, a variety of LLMs have emerged in recent years. This raises the question of whether P2T can achieve better performance with more advanced models. In Figure 3 , we validate the effectiveness of using the identified column feature as an useful target for creating pseudodemonstrations. Here, we consider a transfer learning scenario utilizing an unlabeled dataset for 1-shot prediction. As shown in Figure 3 , using the identified target that is highly correlated with the target task consistently outperforms using random targets (e.g., using the 'Age' column as the target for the pseudo-demonstrations in Figure 1 ). Interestingly, using randomly selected column features as targets can perform worse than not leveraging unlabeled data (see the Haberman dataset results in Figure 3 ). This is because, for example, in the Haberman dataset, 'patient's year of operation' may not have any correlation to the target task 'patient's survival status.' This highlights that carefully constructing pseudo-demonstrations designed to be highly relevant to the target task is a key factor in enabling transfer learning via prompting.Qualitative analysis on selected column. In this part, we use the Breast dataset as an example to qualitatively analyze which column feature is most relevant to breast cancer.For the Breast dataset, 'deg-malig' is selected as the most relevant, making it a useful target when creating pseudo-demonstrations. On the other hand, the other feature 'tumor-size' isMethod GPT-3.5 GPT-4 GPT-3.5 GPT-4 GPT-3. since it is more indicative of the outcome than size alone, as a small tumor might be highly malignant and thus more dangerous than a larger but less malignant one.All previous experiments use . A natural extension is to ask whether better performance can be achieved by P2T using a more advanced model. To verify whether P2T performs better with advanced LLMs, we use GPT-4 (i.e., gpt-4-turbo-preview) in a transfer learning scenario leveraging an unlabeled dataset. As can be seen from LIFT-ICL in Table 4 , we first note that the ICL capability of GPT-4 is better. With better ICL capabilities, our P2T framework also benefits and shows better performance than using GPT-3.5. This indicates that as LLMs continue to advance, improved performance by our P2T framework is expected with future models.Extending P2T to regression tasks. While our main interest is in classification tasks, one way to naturally extend P2T to regression tasks is to prompt the LLM to output the numerical values of the regression task (in classification, the LLM is guided with multiple choices).Here, we evaluate P2T's capabilities in 10-shot regression tasks on three tabular regression datasets from OpenML and Kaggle. As shown in Table 5 , the results indicate that P2T is a competitive approach for few-shot tabular regression tasks. However, the improvement is relatively small compared to classification tasks. This is because LLM often faces difficulties when dealing with numerical values, and we leave it as a future work to address this.Using LLM for correlation identification. To verify that LLM is better than conventional methods for identifying the most correlated feature, we perform an ablation study using Cat-Boost for correlation identification. As shown in Table 6 , using the column feature selected by CatBoost (which differs from LLM's selection) to construct the pseudo-demonstration performs worse than using LLM. This is because (i) CatBoost often does not perform well with few-shot examples (see Table 2 ), and (ii) LLM, on the other hand, can understand the linguistic context of the task and thus is better able to identify correlations.Robustness to missing values. In practice, tabular data often contains missing values for various reasons. For instance, biopsy results may not be collected for all patients due to the risks and complications involved in the data collection process (Yoon et al., 2018) .Conventionally, missing values are managed using imputation algorithms (Yoon et al., 7 : Robustness of P2T to missing values. We report 1-shot test accuracy (%) using unlabeled samples as transfer source. We report the average accuracy over 5 different seeds.indicates that all features are used, and indicates that 50% of the features are randomly omitted. The bold denotes the highest average score. Yi et al., 2020) in the tabular domain, which estimates missing values from other existing information. The performance of standard machine learning methodologies largely depends on imputation algorithms, as incorrectly estimated data could introduce severe noise.In contrast, P2T naturally handles missing values by simply excluding these values from the input prompt. For example, if the 'Age' feature is missing, P2T serializes the table to prompt like "Insulin is 130, BMI 37.9." To verify the robustness of P2T to missing values, we simulate a scenario where 50% of features are randomly omitted. As shown in the Table 7 , P2T outperforms the baselines using simple zero imputation (Yi et al., 2020) , and the non-LM baselines fitted with all features. Furthermore, even when the non-LM baseline shows significant performance degradation (e.g., on the Customers dataset), P2T shows a smaller performance drop. These results highlight P2T's robustness to missing values, which we attribute to the fact the P2T is not significantly affected by inaccurate estimates that the estimation algorithm can introduce. P2T without language descriptions. Indeed, LLMs require explicit column descriptions to effectively exploit the language prior, but informative descriptions are often absent in tabular datasets (Asuncion & Newman, 2007) . Consequently, researchers are forced to use the generic indicator, a prompt that is used to substitute (or pretend as) actual column names, for instance, 'feature' (Dinh et al., 2022) . In Figure 4 , we present the 1-shot test results of generic model where the language descriptions are removed. Specifically, we employ generic indicator 'feature' and 'output y' for the column names. Even in this case, leveraging unlabeled data with P2T significantly improves performance, demonstrating P2T's potential to handle all types of tabular data. 
conclusion: 
 In this paper, we introduce P2T, a novel framework for transfer tabular learning. Our approach utilizes prompting to effectively transfer knowledge from the transfer source. Through extensive experiments, we show the efficacy of P2T across various datasets. It is noteworthy that P2T does not necessitate expert-level knowledge in machine learning. This allows practitioners to adopt our framework effortlessly. We hope that our work holds promising potential in broadening the use of LLMs for tabular learning, particularly in conjunction with readily available resources such as unlabeled data. 
 
 Appendix: In this section, we provide brief explanations of the chosen baselines. For CatBoost (Prokhorenkova et al., 2018) and logistic regression, we employ the default hyperparameters as provided by the CatBoost library and the Scikit-learn library, respectively. For VIME (Yoon et al., 2020) pre-training, we adopt the hyperparameters recommended in the original paper, utilizing the Adam optimizer with a learning rate of 1e -3 and weight decay of 1e -4. When implementing STUNT (Nam et al., 2023) , we follow the unsupervised validation scheme proposed in the original paper for hyperparameter search and early stopping. For LIFT-ICL (Dinh et al., 2022 ), we provide prompt example used on the Customers dataset in Listing 1. The dataset consists of 7 input variables: annual spending on fresh product, annual spending on milk products, annual spending on grocery products, annual spending on frozen products, annual spending on detergents and paper products, annual spending on delicatessen products, and customer's region. The output variable is the customer's channel. Listing 1: Prompt for LIFT-ICL (Dinh et al., 2022) used on the Customers dataset.In this section, we provide examples of prompts used in P2T, specifically on the Customers dataset in scenario of transferring knowledge from the unlabeled dataset. In particular, we illustrate the prompts employed to identify the column feature with the highest correlation (see Listing 2), along with the prompts used during the final inference stage (see Listing 3). For the sake of brevity and due to constraints on paper length, the prompts we provide consist of merely two unlabeled samples along with a single labeled sample per class. We also provide prompt used for zero-shot prediction (see Table 1 ) on the Adult dataset, where the source dataset is Electricity, in Listing 4. Note that prompt used in Table 3 The dataset consists of 14 input variables: age, workclass, fnlwgt, education, education-num, marital-status, occupation, relationship, race, sex, capital-gain, capital-loss, hours-per-week, native-country. The output variable is the person's annual income.→ → → Question:When num_rooms is 1.0, num_people is 6.0, housearea is 895.91, is_ac is 0.0, is_tv is 0.0, is_flat is 0.0, num_children is 1.0, is_urban is 0. Listing 4: Prompt used for zero-shot prediction on the Adult dataset by transferring knowledge from the Electricity dataset.Despite its superiority in transfer tabular learning, our method is constrained by the prompt size limit of LLMs. For example, tabular data with a large number of columns may not fully benefit from P2T, as the result of serialized tabular data can be excessively lengthy.Nonetheless, we hope that future advances in LLMs, GPT-4 (OpenAI, 2023) for instance, will mitigate this issue by accommodating larger prompt sizes. Another potential strategy could involve utilizing LLMs to subsample critical features, thereby enabling more efficient integration of serialized tables into LLMs. Building on this, we hope that our work will inspire researchers to further investigate the relatively under-explored problems associated with tabular learning via LLMs.Tabular data often include privacy-sensitive or confidential features, such as social security numbers. As such, it is crucial to handle this data with care. However, P2T is also effective for managing anonymized features. For instance, our experiments indicate that even when categorical features are replaced with random alphabetical symbols, and generic indicators are used instead of actual column names, P2T still shows competitive performance (see Figure 4 ). Therefore, despite potential privacy concerns related to tabular classification, P2T shows promise for widespread use alongside privacy-preserving techniques. 


Decision: This paper proposes a simple yet effective method that uses LLMs for transfer learning on tabular tasks. The main idea is to create "pseudo-demonstrations" identify from unlabeled source data and then prompt the LLM with these demonstrations + (optionally) labeled examples from the target dataset to do the final task. Experiments on a range of tables and tasks from OpenML demonstrate the effectiveness of the proposed method.

Reasons to Accept
- Transfer learning on tabular data is important due to the scarcity and cost of labeled data. Using LLMs in the proposed manner is a plausible approach.
- Reasonably well-written paper.

Reasons to Reject
- A major criticism, as pointed out by DX82 and ijuV and I also concur, is the simplicity and small size of the tabular tasks being used. Tabular ML has come a long way and there are many datasets and tasks that have a level of complexity closer to real-world application scenarios. See TURL: Table understanding through representation learning and references therein and thereof for some pointers. As it currently stands, it's not clear how well the proposed method will generalize to real-world tables and tabular tasks that could be more complex.

Reviews: review: paper_summary: paper_summary: The paper presents a novel framework, Prompt to Transfer (P2T), which leverages large language models (LLMs) for transfer learning in the domain of tabular data. The authors address a significant problem in machine learning applications where labeled data is scarce, and they propose a solution that creates pseudo-demonstrations from unlabeled or heterogeneous source data to enhance the learning process. The paper claims that P2T outperforms existing methods on various tabular learning benchmarks.
strength_weakness: strength_weakness: 1) The proposed method in the paper is too simplistic and does not fully exploit the capabilities of large language models. There is a lack of exploration into the characteristics of tabular data. 
2) The method in the paper requires calling third-party APIs, which raises privacy concerns in real-world applications. It is suggested that the authors consider using open-source large language models. 
3) While the effectiveness of using Pseudo-demonstrations has been verified, there should be more attempts and comparative analyses in constructing Pseudo-demonstrations. 
4) The section on Correlation Identification needs to experiment with conventional algorithms to demonstrate the superiority of using large language models.
questions: questions: 1) How many test samples are there in each dataset? 
2) The experiments in the paper only involve the case where the transfer source is a single dataset. Is it feasible to use multiple datasets?
paper_summary: paper_summary: This paper proposes a framework called "Prompt to Transfer" (P2T) for performing transfer learning on tabular data using large language models (LLMs) and in-context learning. The key idea is to create "pseudo-demonstrations" from unlabeled or heterogeneous source data that are relevant to the target task, and then prompt the LLM with these demonstrations along with a few labeled examples from the target dataset. Specifically, P2T first identifies the column feature in the source data that is most correlated with the target task label. It then constructs pseudo-demonstrations where that column is treated as the prediction target using the other columns as input features. These demonstrations capture knowledge about predicting the correlated column, which can transfer to the actual target task. Experiments on various tabular datasets show P2T consistently outperforms previous methods in few-shot and zero-shot transfer learning settings.
strength_weakness: strength_weakness: - The paper is largely empirical in nature and lacks deeper analysis of the proposed P2T framework. Without principled justifications, the work may be perceived as somewhat ad-hoc or heuristic. 
- The experiments mainly focus on classification tasks (mostly binary classification) across a limited set of 17 tiny tabular datasets. Evaluating P2T on more datasets, especially larger-scale ones, as well as other tasks like regression could improve the generalizability of the results.
questions: questions: 1. In the pseudo-demonstration construction, how sensitive is P2T's performance to the choice of the correlated column feature identified by the language model? What happens if the wrong column is selected?

2. The pseudo-demonstrations are constructed based on the assumption that predicting the correlated column feature is a good proxy for the original target task. However, what theoretical justification or empirical evidence do you have that this assumption holds across different types of tabular datasets? Are there scenarios where this assumption could break down?

3. The current approach relies on enumerating and scoring all columns to identify the most correlated feature. How does this scale to datasets with hundreds or thousands of columns? Have you explored more efficient strategies?

4. What about handling missing values or noisy data entries commonly presented in tabular datasets? How can the framework be extended to be more robust and handle realistic data issues?
paper_summary: paper_summary: This paper presents a technique called P2T for transfer learning in tabular data. The core idea of the technique is as follows: supposed you want to use LLMs for doing few or zero shot learning in a domain T where the target prediction is column t, and you have access to data from related but perhaps not exactly the same domain S. First you ask an LLM to identify the column in S that is most relevant to t; let's say s. Then you create few shot examples from S where the target prediction is column s. Each of these examples is essentially descriving the row in natural language. Additionally you can also add few examples from domain T. This makes your prompt. Using this prompt, you can ask the LLM to predict the value of t for any incoming example from T. 

The main claim of the paper is that this technique improves prediction in domain T and outperforms techniques like LiftICL. In paticular the claims of the paper are
1. When asking the LLM for most relevant target column s, and including those examples in the prompt, improve 1 shot (i.e. 1 example per example from target domain) accuracy of the model, even if S is heterogenous from T.
2. Which column is selected for transfer from S, matters. Performance drops if you take a random column as target.
3. Including unlabeled examples from domain T also improves the performance.
strength_weakness: strength_weakness: 1. The openml data is particulatrly simplistic. Could the authors also try to replicate this work on TabZilla which seems more suitable for comparisons between various LLM techniques as well as LLM vs other ML techniques. https://openreview.net/pdf?id=CjVdXey4zT

2. Given the surpising nature of the outcomes, I do want to hold the bar on release of prompts and outcomes used for comparing p2t vs lift-icl e.g. so the community can help spot check the outcomes, and since it is trivial to verify the outcomes.
questions: questions: 1. I am a bit confused by the section on including unlabeled examples. In what way are they included in the prompt? Why can't they also be included in LIFT-ICL and how does P2T even differ from LIFT-ICL in this case (since there is no source domain so to speak)? The finding that including unlabeled examples in the prompt helps has been brought up elsewhere as well (https://arxiv.org/pdf/2404.11018) and seems orthogonal to the core p2t idea itself. What am I missing here?

2. What does inclusiion of LR and kNN in table3 mean? is LR trained on source and target data combined and if so, how does transfer happen in this case (is it via just common regularization)? If so, it is perhaps better to follow the tehnique here: https://arxiv.org/pdf/0907.1815.

3. The finding that adding heterogenous source improves the performance significantly is quite surprising. I'd highly encourage the authors to release all of their prompts for this finding for independent verification (which should be trivial).

4. How does the performance change with the closeness between two datasets? What happens if the tasks are far apart?

5. If I were allowed to fine tune a model on source examples, how does the performance change? 

6. How sensitive is this approach to different naming styles or conventions across the data?
paper_summary: paper_summary: This paper proposes a novel framework, Prompt to Transfer (P2T), which utilizes large language models (LLMs) for tabular transfer learning. The methodology leverages in-context learning (ICL) by creating pseudo-demonstrations from unlabeled or heterogeneous source datasets to predict target tasks effectively. The paper is well-structured, offering a clear introduction to the problem and a detailed explanation of the approach. The experimental results show that P2T significantly outperforms existing methods across various datasets, to support the significance of the work.
strength_weakness: strength_weakness: - Some aspects of the methodology, particularly how the pseudo-demonstrations are generated and utilized, could be described more transparently to enhance the reproducibility of the results.

- There is insufficient discussion on how the selection of source and target datasets might affect the generalizability of the P2T framework, which could be a concern for adopting this method in broader applications.

Comments: title: Thanks!
thanks!
Further Clarification
somwhat helpful; rating remains unchanged
A Gentle Reminder
A Gentle Reminder
A Gentle Reminder
A Gentle Reminder
comment: Dear Reviewer ijuV,

We truly appreciate your acknowledgment of our rebuttal and overall positive review! We are happy to hear that our response helps to address your suggestions. In the final draft, we will add the clarifications, which we believe will further improve our manuscript.

If you have any further comments and suggestions, please let us know. We are committed to improving the quality of our work, and we value your feedback.

Thank you very much,

Authors
please provide these clarifications in your paper. I have upgraded my rating thanks to your clarifications.
**Further clarification: LR and kNN.**

First, we would like to clarify that we use all columns from the heterogeneous datasets for the LR and kNN. Specifically, by concatenating all columns in both datasets, the classifier trained on both the source and target dataset (as they will have the same input feature dimensions) can be applied to the few-shot target samples. All of the experiments in Table 3 assume that some sets of columns in the source and target datasets overlap, allowing a classifier that fits one dataset to make meaningful inferences on the other.

Nevertheless, to further alleviate your concern, we conducted an additional experiment using the suggested technique (i.e., FA [1]). However, such an adaptation technique also exhibited similar behavior (cf. Table 3), presumably because it is not helpful in the few-shot scenario, which is our main problem setting.

\begin{array}{lllcccccc}\hline
\text{Target}&\text{Source}&\text{Method}&N=0&N=2&N=4&N=6&N=8&N=10\newline\hline
\text{Adult}&\text{Credit-R}&\text{LR+FA [1]}&54.00&64.67&68.67&61.33&58.00&51.33\newline
&&\textbf{P2T (Ours)}&\textbf{74.67}&\textbf{75.33}&\textbf{76.00}&\textbf{77.33}&\textbf{79.33}&\textbf{80.00}\newline\hline
\text{Adult}&\text{Electricity}&\text{LR+FA [1]}&54.00&54.00&51.33&51.33&47.33&60.00\newline
&&\textbf{P2T (Ours)}&\textbf{74.67}&\textbf{80.00}&\textbf{76.00}&\textbf{78.67}&\textbf{80.00}&\textbf{81.33}\newline\hline
\text{Credit-g}&\text{Credit-A}&\text{LR+FA [1]}&52.67&50.00&48.673&34.67&46.00&47.33\newline
&&\textbf{P2T (Ours)}&\textbf{55.00}&\textbf{54.50}&\textbf{58.67}&\textbf{59.33}&\textbf{59.33}&\textbf{60.67}\newline\hline
\end{array}

[1] Hal Daume III, Frustratingly Easy Domain Adaptation, ACL 2007.

----
**Further clarification: Using unlabeled data as a transfer source.**

For further clarification, we provide an example from the Diabetes dataset, where the target task is to predict whether a patient is diabetic (i.e., the target column), and the input columns are Insulin, BMI, and Age. The unlabeled samples do not have a target column, so the pseudo-demonstration consists of insulin, BMI, and Age. For example, the pseudo-demonstration will be ‘Q: If BMI is 37.9, and Age is 21, what is Insulin level? A: 64’ (see Figure 1, and note that Insulin is most highly correlated with the target column Diabetes). Our intuition here is that knowledge of predicting Insulin levels from BMI and Age could be useful for predicting patients with Diabetes from Insulin, BMI, and Age (as these are highly related tasks).
Your clarifications do help but some things are still not clear
1. The lr/knn baselines are still not super clear and I believe they have not been optimized to the fullest extent possible for transfer learning.
2. For unlabeled examples: can you provide an example of how it is used? e.g. let's say I have columns x,y,z and target column t. For unlabeled examples, column t is empty. So now you find the most correlated column to 't' amongst x,y, and z? If so, what is this column for the datasets being discussed? The presence of such a column in features does not seem like a very likely phenomenon.
Dear Reviewer VukR,

Thank you very much again for your time and efforts in reviewing our paper.

We kindly remind that we have only three days or so in the discussion period.

We just wonder whether there is any further concern and hope to have a chance to respond before the discussion phase ends.

Many thanks,

Authors
Dear Reviewer ijuV,

Thank you very much again for your time and efforts in reviewing our paper.

We kindly remind that we have only three days or so in the discussion period.

We just wonder whether there is any further concern and hope to have a chance to respond before the discussion phase ends.

Many thanks,

Authors
Dear Reviewer DX82,

Thank you very much again for your time and efforts in reviewing our paper.

We kindly remind that we have only three days or so in the discussion period.

We just wonder whether there is any further concern and hope to have a chance to respond before the discussion phase ends.

Many thanks,

Authors
Dear Reviewer kaWX,

Thank you very much again for your time and efforts in reviewing our paper.

We kindly remind that we have only three days or so in the discussion period.

We just wonder whether there is any further concern and hope to have a chance to respond before the discussion phase ends.

Many thanks,

Authors