ProtoLLM: Training and Example-free LLMs for Few-shot Tabular Learning

Authors: Peng Wang, Dongsheng Wang, He Zhao, Hangting Ye, Dan dan Guo
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Tabular data, Few-shot learning, Large language models

Abstract:
Recent breakthroughs in large language models (LLMs) have opened the door to in-depth investigation of their potential in tabular data modeling. However, the paradigm for effectively utilizing advanced LLMs in few-shot and even unseen scenarios remains to be explored. We observed an unusual phenomenon: directly using LLMs for data augmentation or rule generation by feeding a few examples significantly degrades the reasoning ability in tabular data understanding. We identified two main obstacles behind this issue: overfitting to the examples and knowledge disruption. Specifically, the provided examples may introduce noisy patterns that interfere with the model's prior knowledge, leading to unexpected and less reliable results. To this end, we propose an example-free framework to leverage the inherent knowledge of LLMs. Our key idea is to prompt the LLM for oracle feature generation based solely on task and feature description. Without such example pollution, each output feature is treated as a standard guideline, and they together act as a prototype for each class. To transfer the LLM's knowledge to a given task, we further design an efficient fusion strategy to integrate the prototype with example features, showing impressive generalizability in the few-shot setting. Importantly, our pipeline requires no learnable variables, resulting in a desired training-free property. Extensive comparisons and ablations on multiple tabular datasets demonstrate the improvements of our simple framework.

Decision: reject
Reviewer scores: 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1729243299607

thread_type:
official_review

<thread_object>
creator: reviewer_j7jo

<content>
summary:
This paper presents ProtoLLM a novel approach to leverage LLM for supervised classification on tabular data. More precisely, the presented method focuses on few-shot and zero-shot learning for tabular data.

The authors observe that that few-shot learning of LLMs for supervised tasks on tabular data can perform poorly if the few examples fed to the model are not properly selected. The authors thus propose to generate a prototype solely based on the existing knowledge of the LLM through careful prompting, and proceed to use this prototype in a training free maner to compute a prediction for a sample of interest. Their approach can also be extended to few-shot learning by aggregating existing examples with the generated prototype (e.g. averaging the feature values). 

The authors test their approach on a tabular data benchmark composed of 10 datasets following previous work, and compare to existing methods. They show that their approach performs strongly based on the AUROC metric and also perform several ablations studies to further demonstrate the relevance of their proposed method.

strengths:
- The proposed method does not require any training and inference is perfomed only based on a distance measure between the sample of interest and the generated prototypes.
- The proposed approach is relevant for both zero-shot and few-shot learning set-ups.
- Proposed approach has (to our knowledge) significant novelty.
- The experiments are extensive and fair, and show that ProtoLLM offers strong performance on the tested benchmark. 
- Ablation study is rather exhaustive and investigates most of the relevant factors.
- Work is reproductible.

weaknesses:
- The paper is sometimes hard to follow and wordy. The overall style could be improved, and part of the paper contains either typos or odd turns of phrase (e.g. line 356:"*To validate the seasoning ability of our ProtoLLM on tabular data*", line 300: "*where $\tau$ is a temperature hype-parameter*", line 381: "*We attribute this superior to our example-free prototype generation*".).
- While the proposed approach is novel and interesting, the **contribution is relatively thin** for a conference like ICLR.
- The proposed approach requires prompting an LLM $d \times K \times C$ times, where $d$ is the number of features, $K$ the number of iteration considered to construct the prototype, and $C$ the number of classes. This might restrains this type of approach to datasets with reasonable number of features or classes. Moreover, depending on the set-up, this might be very costly for tabular data classification when considering hardware requirements to run LLMs.
- the proposed approach seems restrained to applications on which LLMs may have relevant knowledge, hence not too specialized or novel. While this out of the scope of this work, it might be interesting to investigate whether humans may generate prototypes as relevant as the LLMs on some datasets.
- The proposed approach is restrained to classification and cannot be directly adapted for regression tasks.

questions:
## Typos:
Authors might consider going over their manuscript to correct typos which occur quite frequently. In particular, citations should be placed after a word and separated with by a whitespace (e.g. line 346 "*STUNT(Nam et al., 2023),*" -> "*STUNT (Nam et al., 2023),*").


## Questions
- **(i)** The $\tau$ parameter in equation $3$ appears critical. **How is $\tau$ selected ?**
- **(ii) Have the authors investigated the impact of the number of paramaters of the base LLM on the overall performance?** While one would (of course) expect the performance to drop as the the number of parameters of the LLM decreases, it might be interesting to analyse the relationship between performance and number of parameters. In particular, if the performance drop is small between large models and smaller models (e.g. 13B parameter models), this might prove to be a significant argument in favor of ProtoLLM.
- **(iii)** As mentionned as a weaknesses of the proposed method, generating the "*golden prototypes*" requires several iterations that may cause a significant "*training*" time. The authors should consider comparing the overall time required to train the competing models vs the time required to construct the golden prototypes used in inference.
- **(iv)** In figure 5 as the reported results are averaged for different seeds, I would be curious to see the figures including the standard deviations/confidence intervals to assess statistical significance of performance gains. **Have the authors performed any statistcal tests (e.g. Wilcoxon signed-rank test) to compare performance with the other best performing alternative?**
- **(v)** As mentionned as a weakness of the proposed approach, ProtoLLM cannot directly be applied for regression. **Have the authors considered modifications/adaptation to the proposed setup to possibly perform regression tasks?** If so, could they share their opinion on the faisability?
- **(vi)** In [1], the authors analyze the impact of spurious correlation on the performance of their proposed method in comparison to existing methods. While ProtoLLM works for zero-shot classification, **have the authors analyzed the effect of including those noisy samples when generating the golden prototypes in the few-shot setting?**
- **(vii)** I might be interesting to analyze how ProtoLLM performs on datasets that have a high number of features, e.g. Arrhythmia (https://archive.ics.uci.edu/dataset/5/arrhythmia) or other tabular datasets the authors might think of.
- **(viii)** For the few-shot settings, to assess the added value of the proposed method, one may consider the following setting: For each class, construct the *prototype* using **only** the samples and excluding the prototype generated by the LLM. That way one could assess the explicit added value of the prototype in the few-shot setting as done for the zero-shot setting. 

Given the listed strength and weaknesses, as well as questions and interrogations listed above, we lean towards reject. However, we are open to discussion and would happily increase our score if the authors address our concerns and interrogations. 

[1] Sungwon Han, Jinsung Yoon, Sercan O Arik, and Tomas Pfister Large language models can automatically engineer features for few-shot tabular learning. In Forty-first International Conference on Machine Learning, 2024.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730130444644

thread_type:
official_review

<thread_object>
creator: reviewer_8ebi

<content>
summary:
Conventional machine learning models for tabular data rely on labeled samples, and performance tends to degrade when few samples are available. It has recently been argued that LLMs may learn useful information for certain tabular classification tasks during pretraining. This may make them superior classifiers in the extreme few-shot setting (which, for tabular data, could roughly be classified as fewer than 100 labeled datapoints available for training). TabLLM and FeatLLM constitute two important precursor works in this line of research.

The authors of ProtoLLM remove the few-shot examples from the prompt template and prompt LLMs to generate oracle feature values according to the LLM's inherent knowledge about the given tasks. They query LLMs feature by feature, rather than generating all features of a tabular sample simultaneously. Instead of generating a single value for each feature, they query LLMs K = 10 times for each feature across all datasets. Results are provided for zero-shot and few-shot.

The authors claim their method leads to robust and superior performance on a range of datasets.

strengths:
* The authors provide some intriguing indications that LLMs can solve certain tabular classification problems which are well specified using natural language zero-shot, simply by predicting likely feature values for each class.
* The authors include some helpful ablation studies and appendix material, including a complete results table.
* The choice of few-shot baseline methods is comprehensive and strong.

weaknesses:
MAJOR

* This paper argues for the scientific relevance of a method whose main results are demonstrated only on ChatGPT, a closed-source LLM. Closed models make bad baselines, are not reproducible, and bias future research [https://hackingsemantics.xyz/2023/closed-baselines/]. To make the claim that ProtoLLM is significant, it is necessary to show that it is doing something qualitatively different which improves results while controlling for model size, the type and amount of data, and important hyperparameters. Since we don’t have any of this information for GPT, it is not possible to claim ProtoLLM improves on prior work; apparent improvements could simply reflect a quirk of the chosen GPT checkpoint, for example. At a bare minimum, the authors should show results on a small open-source LLM such as T0 or LLAMA-3-8B in addition to GPT.
* The choice of datasets is small and non-standard; using a large established benchmark in the literature would lend more credibility to the findings [https://arxiv.org/abs/2305.02997]. Data contamination is a major concern here, as most of the datasets included are old. On the two datasets where data contamination is less likely (NHANES and Cultivars), a simple logistic regression baseline with just 64 data points beats ProtoLLM.
* Some datasets in Fig. 5 have missing baselines.
* Something is wrong with the formatting of this paper; line numbers don't align with the lines.
* There are few zero-shot baselines provided; DSPy would have been a natural choice [https://github.com/stanfordnlp/dspy]. The one zero-shot baselines that is provided (TabLLM GPT-3.5) beats ProtoLLM on 4 of 9 datasets; given that the authors argue their contributions are most significant in the zero-shot regime, this is troubling.
* TabLLM GPT-3.5 results should be made available for few shot as well as zero-shot for a fair comparison between methods.

MINOR

* Fig. 5 is hard to read; it should be larger.
* Please provide links to the datasets used in the appendix; there are many versions of these datasets in common use.
* Many typos (lines 283, 300, Table 1, Sec. 4.1 header, line 337, etc).

questions:
* How does the method deal with continuous features? Is it similar to FeatLLM?
* How many tokens are consumed for an average iteration of ProtoLLM?
* How is the distance function used in ProtoLLM? What is meant by 'logistics' in this sentence: "All we need to calculate the logistics is the Euclidean distance between the input samples and obtained prototype"?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730534196829

thread_type:
official_review

<thread_object>
creator: reviewer_toln

<content>
summary:
This paper suggests generating individual tabular feature values with LLM (which they denote as Oracle feature generation) per class, then building a prototype, which is used to build a non-parametric classifier. This prototype can be jointly used with few-shot samples by building prototypes with few-shot samples (i.e., averaged feature value of few-shot samples). The author shows that the method is effective for zero to low-shot (e.g., 64-shot) classification for tabular dataset.

strengths:
**Strength**

1. The overall performance is strong.

2. The method may offer novelty from one perspective: generating features with LLMs already exists [1], but using them additionally for building prototypes is relatively new.

**Reference**\
[1] Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning, ICML 2024

weaknesses:
**Weakness**

1. The writing lacks clarity and is not mathematically formulated. For example, there are several questions:
- What is the feature value z? This value should be mathematically defined. Currently, it appears to be the feature generated by the LLM. However, it is not clearly defined, making it difficult for readers to understand.
- In Figure 3, 4, can the authors clarify what each color represents?
- Why is the LLM-generated feature referred to as "Oracle"? "Oracle" is a term that requires careful use; I would suggest "LLM-generated feature" instead, as the LLM is not an Oracle.

2. Most of the datasets focus on binary classification, which can be considered a toy example (also, how did the authors compute AUC for the car dataset, which seems to be a multi-class classification task?). The authors should consider including more multi-class classification datasets, such as those in STUNT [1]. Extending this approach to regression tasks is also essential, as it’s a significant application area for tabular learning.

3. The experiment section could be strengthened.
- Variance should be reported for few-shot (or low-shot) learning experiments, as stability is also a crucial aspect of the work.
- Reporting full-shot accuracy and discussing limitations in this setup would add valuable insights.

4. There is a concern regarding dataset contamination and whether the method will work on truly novel datasets. While it’s acknowledged that the authors have considered two datasets new to the LLM, tabular datasets often contain novel features (e.g., from fields like factory settings or biology). It is unclear if this method would generalize well to such data.

5. There are existing works focused on feature generation that can be combined with other effective methods (e.g., XGBoost) to enhance performance further [2,3,4,5]. Typically, in this context, few-shot learning is not the primary focus, as it is somewhat constrained or limited in application.

6. Lack of comparison with LLM used Tabular learning frameworks [6,7,8].

**Summary**\
The paper has certain strengths; however, currently, the weaknesses are more prominent. In this context, I respectfully request the authors to address these issues during the rebuttal.

**Reference**\
[1] STUNT: Few-shot Tabular Learning with Self-generated Tasks from Unlabeled Tables, ICLR 2023\
[2] Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering, NeurIPS 2023\
[3] Generalized and Heuristic-Free Feature Construction for Improved Accuracy, SIAM Data Mining 2010\
[4] Learning a Data-Driven Policy Network for Pre-Training Automated Feature Engineering, ICLR 2023\
[5] OpenFE: Automated Feature Generation with Expert-level Performance, ICML 2023\
[6] LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks, NeurIPS 2022\
[7] Tabular Transfer Learning via Prompting LLMs, COLM 2024\
[8] Language models are weak learners, NeurIPS 2023

questions:
The questions are in the weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733101267629

thread_type:
public_comment

<thread_object>
creator: ~linlu_you1

<content>
title:
Issues Report

comment:
I suggest all the reviewers to consider the issues. First, I appreciate the authors for their interesting work. And I also appreciate all the reviewers for their efforts.
However, I find the reported results in this paper to be unconvincing.

1. **Prototype generation**. Since the key component of ProtoLLM are the constructed prototypes, the reasoning process of generating possible values for each class (Line 209 in revised paper) is important. However, for the example prompt shown in Fig. 3 and the corresponding response in Fig. 4, I observed significant instability in the responses generated by the LLM. Specifically, when I queried all the LLMs mentioned in the paper multiple times (K times) using the prompt from Fig. 3 as input, I was unable to replicate the response illustrated in the paper. Similar inconsistencies were also observed for the examples in Fig. 8 and 9.

2. **Reproducibility of AUC Results**. Based on this, I can not reproduce the reported AUC performance in the paper.

3. **The reported results for baseline methods exist issues**. 
There are discrepancies in the experimental setups for baseline methods. As stated in the paper's 'Implementation Details' section, the results for STUNT, In-context Learning, TABLET, TabLLM, and FeatLLM were derived from [1]. While this is acceptable, the experimental protocol described in [1] differs from that of ProtoLLM. Specifically, in [1], few-shot samples for each class are generated by randomly selecting a certain number of samples (as per utils.py in [1]'s repository). In ProtoLLM, however, the authors use the first few samples in each class, as shown in data.py. Tabular data is highly sensitive to sampling, and this discrepancy could lead to significantly different results. To verify this, I applied ProtoLLM's protocol to FeatLLM and obtained different (and higher) results than those reported in the paper. This suggests an unfair comparison, especially given that FeatLLM is a primary baseline.

4. In addition, the reported results of LogReg, XGBoost, TabPFN are diiferent from that reported in [1].

5. **Missing values**. The authors remove columns with more than 20% missing values in data.py of the given code, which is not mentioned in the paper. According to [2], handling missing values is a critical aspect of feature heterogeneity. Since ProtoLLM aims to address heterogeneous tabular features, removing such columns oversimplifies the task and undermines its generalizability in the tabular domain.

6. **Performance Claims**. Solid performance is a key strength cited by many reviewers. However, given the issues with experimental protocols and reproducibility, the reported performance metrics are not sufficiently convincing. I suggest reviewers to carefully evaluate these concerns before considering a score increase.

7. **Uncertainty about few-shot definition**. The paper does not clearly define "few-shot" in the context of tabular learning. Does it refer to the total number of shots across all classes? If so, how does ProtoLLM perform with different numbers of classes, such as 3, 5, or 7? All experiments appear to use 0, 2, 4, 6, and 8 shots, leaving this aspect unexplored. It seems that the few-shot setting is quite different from that of N-way K-shot setting.

8. **Clarity of Writing**. Despite revisions, parts of the paper remain hard to read. For instance, Line 290 mentions 'From the Bayesian perspective,' but the Bayesian context is not adequately explained.

9. **Lack of Theoretical Support**. While the paper provides intuitive explanations for its motivation, it lacks robust theoretical support. Combined with the aforementioned issues regarding experimental results, this lack of theoretical grounding diminishes the credibility of the findings. Is there any theoretical support to demonstrate the feasibility of this method or clarify the scenarios in which it is applicable?

[1] Large language models can automatically engineer features for few-shot tabular learning.

[2] Deep Neural Networks and Tabular Data: A Survey.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a new method for few-shot learning in tabular data using large language models (LLMs). Unlike existing approaches, this method avoids overfitting to the limited training examples by prompting the LLM without directly including them in the prompt. Instead, the few-shot samples are used to create "prototypes" that guide the LLM's predictions.

Strengths:
- The novel prompting strategy helps prevent overfitting to the small training sets.
- Additional experiments conducted during the rebuttal phase demonstrate the broader applicability of the method.

Weaknesses:
- The paper's writing is unclear and difficult to understand.
- The performance improvements over simpler baseline methods are minimal, despite the proposed method requiring significantly more computational resources for inference.
- The practical relevance of few-shot learning in tabular data remains unclear.

While the proposed method presents an interesting idea, the paper's weaknesses outweigh its strengths. The marginal performance gains, coupled with increased computational cost and unclear practical significance, lead me to recommend rejection.