SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hariharan Manikandan<sup>1</sup> Yiding Jiang<sup>1</sup> J Zico Kolter<sup>1,2</sup>
$^{1}$ Carnegie Mellon University  $^{2}$ Bosch Center for AI
{hmanikan, yidingji, zkolter}@cs.cmu.edu
A central notion in practical and theoretical machine learning is that of a weak learner, classifiers that achieve better-than-random performance (on any given distribution over data), even by a small margin. Such weak learners form the practical basis for canonical machine learning methods such as boosting. In this work, we illustrate that prompt-based large language models can operate effectively as said weak learners. Specifically, we illustrate the use of a large language model (LLM) as a weak learner in a boosting algorithm applied to tabular data. We show that by providing (properly sampled according to the distribution of interest) text descriptions of tabular data samples, LLMs can produce a summary of the samples that serves as a template for classification and achieves the aim of acting as a weak learner on this task. We incorporate these models into a boosting approach, which in some settings can leverage the knowledge within the LLM to outperform traditional tree-based boosting. The model outperforms both few-shot learning and occasionally even more involved fine-tuning procedures, particularly for tasks involving small numbers of data points. The results illustrate the potential for prompt-based LLMs to function not just as few-shot learners themselves, but as components of larger machine learning pipelines.
Weak learners refer to classifiers that are able to attain better performance than random chance, by some given margin, on any specified distribution over training data. One of the early breakthroughs in machine learning established that this weak learning was sufficient for arbitrarily strong classification, via an ensembling procedure [Schapire, 1990]. This approach in turn led to the development of boosting algorithms [Freund and Schapire, 1997], a class of approaches that continue to perform extremely well, particularly on tabular datasets that lack the input space regularity of vision or language tasks.
In a seemingly separate thread of research, large language models (LLMs) based on transformers [Vaswani et al., 2017] in recent years have come to dominate many natural language domains. These models are often finetuned on the data of new downstream tasks [Devlin et al., 2018, Liu et al., 2019], but in recent years have also been shown to exhibit strong performance as zero-shot or few-shot learning solely via prompting the model [Brown et al., 2020] with a piece of context string.
In this paper, we align these two threads of research and ask a simple question: can LLMs also serve as weak learners in a boosting framework, specifically on tabular data (where boosting methods are most commonly applied)? We answer this question largely in the affirmative. Specifically, we show that by appropriately converting tabular data to text form, and asking LLMs to summarize a carefully chosen set of examples from the data, we produce a summary of the examples that can serve as a template (i.e., a prompt) for a tabular data classifier, and one which typically achieves this weak learning aim. This enables us to correspondingly integrate this collection of LLM-generated weak learners into a boosting framework.
We show that the resulting approach performs well in many settings, easily outperforming zero-shot and few-shot classification, as well as "single-shot" summaries generated by the LLM. This is all done without any retraining or finetuning of the LLM itself, but rather only via prompting. Furthermore, on certain domains (particularly those with very few examples, where leveraging the prior knowledge built into LLMs would be of particular importance), we show that the approach can even outperform traditional tree-based boosting and LLM-based finetuning methods and its performance would likely improve as LLMs capabilities improve. Overall, we believe this work highlights the potential of incorporating LLMs as sub-routines of a larger machine learning system.
Deep Learning for Tabular Data. Tabular data refers to a generic data format that represents data as a collection of discrete or continuous attributes [Borisov et al., 2021]. Due to their flexibility, tabular data are ubiquitous in many ML settings. However, such flexibility comes with a cost - they lack the inherent structure found in images or text, which makes applying deep learning to them challenging. Furthermore, they are often domain-specific and may have a relatively small number of data points. As a result, traditional deep learning methods, which thrive on large datasets and high-dimensional data, have seen limited success when applied to tabular data [Gorishniy et al., 2021, Shwartz-Ziv and Armon, 2022].
Recently, however, there has been increasing interest in applying deep learning to tasks related to tables such as data integration, imputation [Narayan et al., 2022], semantic parsing, and even running SQL queries [Herzig et al., 2020, Yin et al., 2020]. Deep learning models have also been successful at learning tabular data classification by optimizing loss functions [Hollmann et al., 2022, Schäfl et al., 2022, Dinh et al., 2022]. Unlike these approaches, we study how we can use LLM for classifying tabular data without finetuning or building a new language model. Since many tabular data can be grounded in natural language, texts are in fact a natural representation for tabular data. Motivated by the observation that LLMs can convert tables to text through prompting alone [Saha et al., 2022], we utilize LLMs to do this conversion. After the conversion, our classification algorithm also interacts with existing LLMs strictly through prompts. This creates an abstraction between the underlying language model and the learning procedure which may be desirable for various applications since access to the gradients or parameter updates are not required.
Prompting Prompting [Liu et al., 2023] refers to providing initial text or instructions to guide the response of a language model. The advancements in Language Model-based Learning (LLM) have unveiled new capabilities, such as chain of thought reasoning [Wei et al., 2022], zero-shot reasoning [Kojima et al., 2022], compositional problem solving [Zhou et al., 2022a], and self-improvement [Huang et al., 2022, Ho et al., 2022, Haluptzok et al., 2022]. As a result, prompting has gained widespread application across various Natural Language Processing (NLP) tasks, including arithmetic, common reasoning [Wang et al., 2022b], among others [Brown et al., 2020]. While prompts offer flexibility, it is crucial to note that LLMs interpret them differently from humans. Therefore, the process of prompt tuning, which involves carefully engineering prompts, becomes essential for obtaining accurate and relevant outputs [Reynolds and McDonell, 2021]. At its core, prompt tuning is an optimization process that aims to find the best prompt for a certain downstream task. Though a long line of works propose gradient-guided search to optimize "continuous prompt" instead of the language tokens [Liu et al., 2021, Qin and Eisner, 2021, Lester et al., 2021, Shin et al., 2020, Rakotonirina et al., 2023, Wang et al., 2022c, Diao et al., 2023], gradient-based updates can be limiting, as LLMs become bigger and the access to these models become increasingly API-based. Our approach aligns more with discrete search methods based on the fact that LLMs can automatically generate prompts for themselves [Zhou et al., 2022b, Zhang et al., 2022, Yu et al., 2022]. Specifically, we prompt the LLM to summarize the tabular dataset. The summary in turn acts as a prompt that the LLM uses to make predictions as it encodes knowledge of the dataset. A sequence of such prompts summarizing different subsets of the data can be seen as weak learners for a boosting procedure.
Boosting Boosting [Schapire, 1990, Freund and Schapire, 1997] is a widely used technique to improve the accuracy of a model by combining weak learners (models that perform slightly better than random guessing) to make a strong learner (model with high accuracy). Common boosting algorithms include AdaBoost [Freund and Schapire, 1997], Gradient Boosting [Friedman, 2001], and Stochastic Gradient Boosting [Friedman, 2002]; the XGBoost library [Chen and Guestrin, 2016] in
Figure 1: The conversion for a data point on the Wholesale customers dataset (OpenML ID 1511).
particular is a commonly used implementation of gradient boosting. In concurrent work most relevant to ours, Hou et al. [2022] integrate LLM into AdaBoost for natural language inference (NLI) tasks, by training an MLP projection of the final hidden state of a special token. Our proposed method diverges from theirs in two key aspects. Firstly, our method avoids the overhead of learning additional parameters, is gradient-free, and does not require access to the model's internal states. Secondly, instead of storing knowledge in parameters, our approach concentrates on condensing knowledge into an intermediary representation referred to as "summary." This alternative strategy enhances interpretability and strictly learns through prompts, rendering it particularly suitable for small tabular data, where the prior knowledge in LLM can significantly benefit the learning process.
3 Summary Boosting with Language Models
We now describe the main methodology of our paper, which uses LLMs to generate weak learners, and in turn, uses these weak learners within a boosting framework. We refer to the full method as Summary Boosting, as the core learning process is one that uses a language model to create a summary of (specifically chosen) samples from the dataset; these summaries themselves function as prompts by which we can make predictions on new examples. Finally, we use boosting to construct an ensemble of these summaries that gives the overall predictions on new data points.
3.1 Data conversion
To utilize large language models (LLMs) with tabular data, it is necessary to first convert the records into natural language descriptions. We will refer to these as data descriptions. Template matching, commonly used in previous approaches [Dinh et al., 2022], inserts attribute values into predefined templates. However, this approach often produces unnatural descriptions that differ from how humans might describe the data. Depending on the dataset, designing the template by hand can also be challenging. To overcome this, we propose using LLMs as a more suitable solution.
As illustrated in Figure 1, we can get these data descriptions with little effort by zero-shot prompting the LLM with information about the dataset (which is generally available as metadata for tabular datasets) and a textual representation of the tabular record (e.g., parsed JSON). Specifically, to ensure examples can serve as both training data and query inputs, we extract the descriptions of the features and concatenate them with the target label using a separator token (refer to Appendix A.1). Interestingly, we find that descriptions generated by LLM this way often perform better than those from a template. This ablation study can be found in Section 5.
One key challenge in this process is how to encode numerical attributes effectively; naively including numerical values in the descriptions can lead to poor performance in subsequent learning tasks. To address this, we adopt a straightforward approach: we bin all numerical features into percentiles and encode them descriptively as "low," "medium," and "high." In Section 5, we compare the performance of several such approaches and discuss more examples in Appendix A.6. Overall, the data descriptions can be generated automatically with minimal manual engineering.
Figure 2: The process of generating summaries and using them to make predictions on new data. The top half describes how the weak learning hypothesis (summary) is generated. The bottom half illustrates how the summary is used to perform inference. "Best summary" is the one achieving the smallest validation error rate, or higher training error in case of a tie.
3.2 Weak learning via summarization
A typical method for performing few-shot learning with large language models (LLMs) involves providing a small number of demonstrations of the intended task as a prompt and then asking the model to generate an answer. One could, for instance in the few-shot setting, simply present the natural language descriptions above and generate predictions on new examples. However, for tabular data, there may be a larger number of data points that do not fit within the LLM context. Furthermore, we observed that increasing the number of examples in the context naively does not always improve performance (Figure 4, right bottom), and there was no obvious way to manage weighted distributions over examples as is required in boosting methods. These observations necessitate alternative approaches to weak learning via LLMs.
We propose instead that producing summaries of a collection of examples can serve as a powerful proxy for learning models based upon some number of examples, as summarization naturally encourages the extraction of representative information in the data. Concretely, given a set of data descriptions, we first perform summarization on the data by calling the LLM (e.g., by concatenating a list of examples in natural language form and appending the prompt "t1dr"). This resulting summary can be seen as a hypothesis as it provides an explanation for the data. By using the summary as a prompt, the LLM in turn uses the hypothesis to perform inference instead of the raw data description (shown in Figure 2). Since the sampled summary can sometimes be noisy, we generate a fixed number of summaries and pick the one with the smallest validation error rate. In case of a tie, we choose the one with a higher training error, i.e., a lower generalization gap (see Appendix A.2). Several methods of building such summaries are possible, but simple approaches such as the "t1dr" approach mentioned above tend to work as well as more sophisticated alternatives, as we show in Section 5. Additional considerations in designing these prompts are elaborated in Appendix A.2 and A.3.
(Weighted) Cluster Sampling. Since the context size of existing LLMs is still limited, we cannot in general fit the entire dataset into the context for summarization. Furthermore, boosting algorithms require that we provide weak learners on weighted samples of the training set, effectively guiding the boosting process to focus on "harder" examples as the boosting process continues. Thus, instead of attempting to summarize the entire dataset, we propose to use only a representative subset of the dataset. The size of this subset is governed by the maximum context size and size of the data descriptions. To select this representative subset, we use weighted stratified sampling using subpopulations defined by clusters of language embeddings of each data description. The language embeddings are sentence representations generated by GPT-3. In particular, we use hierarchical agglomerative clustering [Nielsen, 2016] to identify clusters in the embedding. This process is shown in Algorithm 1. As we will show in Section 5, this process is able to consistently produce weak
Algorithm 1 Cluster Sampling
1: Input: X, all training data; y, all training label; r, ratio of classes; p, AdaBoost weights of the current round; s, target number of samples.  $\triangleright$  r [k] is the proportion of examples in class k.  
2: S ← new empty set  
3: w ← new array with same length as X filled with -1.  $\triangleright$  w [i] is probability of sampling example i.  
4: for k = 1 to number of target classes in y do  
5: E ← GPTEmbedding(X [y == k])  $\triangleright$  E refers to the embeddings of the data descriptions  
6: C ← AgglomerativeClustering(E).  $\triangleright$  Cj is set of data indices present in the  $j^{th}$  cluster.  
7: c ← new empty array same size as C.  $\triangleright$  c [j] will store sampling probability of cluster j.  
8: for j = 1 to len(C) do  
9: c [j] ←  $\frac{\text{len}(X)}{\text{len}(C_j)}$   
10: end for  
11: for i = 1 to len(X) do  
12: w [i] ← c [j], such that, i ∈ Cj  
13: end for  
14: w ← Normalize(Normalize(w) × p)  $\triangleright$  Normalize turns weights to a probability distribution.  
15: Sample s × r [c] examples from X using categorical distribution w and append to S.  
16: end for  
17: Return S
Algorithm 2 Summary Boosting (compact version)
1: Input: X, all training data; y, all training label; T: maximum number of rounds; s: size of the sampling subset; r: ratio of classes. ▷r [k] denotes the proportion of examples in class k.  
2: h,  $\epsilon$ ,  $\alpha$  ← new empty arrays of length T.  
3: N ← len(X); K ← number of target classes in y.  
4: w ← new array of length N filled with  $\frac{1}{N}$ . ▷w is the data distribution  
5: for r = 1 to T do  
6: (Xs,ys) ← ClusterSampling(X, y, r, w, s) ▷ sample s training examples from distribution w.  
7: h [r] ← Summary(Xs, ys) ▷ h [r] is the weak learner in the current round r.  
8:  $\epsilon[\mathbf{r}]\gets \frac{\sum_{i=1}^{N} w[i] \times \mathbb{1}\{h[r](X[i]) \neq y[i]\}}{\sum_{i=1}^{N} w[i]}$  ▷  $\epsilon[\mathbf{r}]$  is the weighted error at round r.  
9: if  $\epsilon[\mathbf{r}] \geq 1 - \frac{1}{K}$  then  
10: Goto Step 6.  
11: end if  
12:  $\alpha[\mathbf{r}] \gets \log\left(\frac{1-\epsilon[\mathbf{r}]}{\epsilon[\mathbf{r}}\right) + \log(K-1) \triangleright \alpha[\mathbf{r}]$  refers to coefficient of the hypothesis at round r.  
13: for i = 1 to N do  
14: w[i] = w[i] × exp( $\alpha[\mathbf{r}]\mathbb{1}\{h[\mathbf{r}](X[i]) \neq y[i]\}$ )  
15: end for  
16: w ←Normalize(w)  
17: end for  
18: Return h,  $\alpha$
learners, and able to improve upon random guessing under the distribution of interest (denoted by the input p to the algorithm). We share more details in Appendix A.7.
3.3 Boosting
Finally, we use the AdaBoost [Freund and Schapire, 1997] algorithm to produce an ensemble with these collections of summary-based weak learners. The central idea of AdaBoost is to fit a sequence of weak learners on repeatedly modified versions of the data. The algorithm is carried out over  $T$  rounds, where the weights of the training data points are adjusted based on the training error.
Given a new data point, the predictions from classifiers from all rounds are then combined through a weighted majority vote to produce the final prediction. We use the error on a holdout validation set to determine the number of rounds  $T$ . A compact version of this process is presented in Algorithm 2. In the algorithm, the Summary method summarizes the examples in the prompt via the process discussed in Section 3.2. Each summary can be treated as a hypothesis that can classify new data.
However, unlike the summary process in Section 3.2, where we resample multiple times to find the best learner, the boosting process returns immediately when a summary with an error rate better than random guessing is found (refer Appendix A.2). We use ClusterSampling to subsample a mini-batch of examples that fit within the LLM's allowed context length. In Appendix A.10 and A.11, we provide a time complexity analysis of our method, including cost estimations for using APIs. Appendix A.8 covers the full version of our boosting procedure that works in practice.
We conduct all of our experiments with OpenAI's GPT-3 API [Brown et al., 2020] and choose a collection of 18 tabular datasets from the UCI dataset [Dua and Graff, 2017] and OpenML [Vanschoren et al., 2014]. All main experiments are done with the Curie variant of GPT-3 unless otherwise specified, which has 13B parameters<sup>1</sup>. We compare the following methods:
- Zero-shot: query the language model with the data description and ask the model to complete the answer (refer Appendix A.4).
- Few-shot: provide a few labeled data descriptions of the training data as the context and ask the model to complete the answer for a new data description. To preserve consistency, we standardize the number of fewshot examples to approximately 15 for all datasets. The setting is explained in Appendix A.5.
- Summary (ours): generate a population of summaries given a list of data descriptions with cluster sampling and pick the summary with the lowest validation error; use the best summary as the context and ask the model to complete the answer for a new data description.
- Summary Boosting (ours): use Summary as a subroutine in AdaBoost.
Furthermore, we compared Summary Boosting against popular baselines for tabular data that do not use prompting:
- KNN: first embed the data descriptions with the GPT-3 embedding  $\mathrm{API}^2$  and then use K-nearest neighbor to classify a new data description. This simple baseline demonstrates how much information can the naive representation produced by LLMs provide about the tasks.
- LIFT [Dinh et al., 2022]: Language-Interfaced Fine-Tuning (LIFT) finetunes the LM with data descriptions (without binning) and their corresponding labels in a zero-shot.
- TabPFN [Hollmann et al., 2022]: TabPFN is a transformer-based architecture that performs Bayesian inference on the entire training and test data points at the same time.
- XGBoost [Chen and Guestrin, 2016]: XGBoost (eXtreme Gradient Boosting) is a regularized gradient boosting algorithm that is widely used for tabular data.
For each method and dataset, we use a  $50/10/40$  split for train, validation, and test sets and repeat each experiment for 3 random seeds. The results are shown in Table 1 and 2.
4.1 Analysis of prompting-based methods
As a general trend from Table 1, test performance improves in the order of Zero-shot  $<$  Few-shot  $<$  Summary  $<$  Summary Boosting. Firstly, unlike most works on zero-shot reasoning with LLMs, the LLMs do not have enough prior knowledge to make the correct prediction without additional information. As a result, we observe that Zero-shot performs poorly on all of the datasets. This observation highlights the necessity of learning from the data, and unlike other tasks, the LLMs themselves do not have enough built-in knowledge to succeed at tabular data zero-shot. Since zero-shot does not have enough prior knowledge to classify tabular data, we use few-shot in-context learning (Few-shot) to see if the added information helps make better predictions. As expected, on all the datasets other than visualizing-hamster, and wholesale-customers, Few-shot consistently improves the test performance compared to Zero-shot, suggesting that this added information is crucial for LLMs to work on tabular datasets.
<sup>1</sup>We use Curie because it is more cost-effective for large-scale experiments.
$^{2}$ https://beta.openai.com/docs/guides/embeddings