Language Models are Weak Learners
Authors: Hariharan Manikandan
Yiding Jiang
J Zico Kolter
Venue: NeurIPS 2023 poster (2023)
Keywords: language model
prompting
tabular data
summarization
boosting
adaboost
TL;DR: We use summarization with language models as a weak learner for boosting algorithm.

Abstract: A central notion in practical and theoretical machine learning is that of a *weak learner*, classifiers that achieve better-than-random performance (on any given distribution over data), even by a small margin.  Such weak learners form the practical basis for canonical machine learning methods such as boosting.  In this work, we illustrate that prompt-based large language models can operate effectively as said weak learners.  Specifically, we illustrate the use of a large language model (LLM) as a weak learner in a boosting algorithm applied to tabular data.  We show that by providing (properly sampled according to the distribution of interest) text descriptions of tabular data samples, LLMs can produce a summary of the samples that serves as a template for classification, and achieves the aim of acting as a weak learner on this task.  We incorporate these models into a boosting approach, which in many settings can leverage the knowledge within the LLM to outperform traditional tree-based boosting.  The model outperforms both few-shot learning and occasionally even more involved fine-tuning procedures, particularly for some tasks involving small numbers of data points.  The results illustrate the potential for prompt-based LLMs to function not just as few-shot learners themselves, but as components of larger machine learning models.

Language models are weak learners
Hariharan Manikandan'
Yiding Jiang' J Zico Kolter 1.2
CCamegie Mellon University Bosch Center for Al (hmanikan, yidingji, zkolter) cmu. edu
Abstract
A central notion in practical and theoretical machine learning is that of weak learner, classifiers that achieve better-than-random performance (on any given distribution over data), even by a small margin. Such weak learners form the practical basis for canonical machine learning methods such as boosting. In this work, we illustrate that prompt- large language models can operate effectively said weak learners. Specifically, we illustrate the use of large language model (LLM) as a weak learner in boosting algorithm applied to tabular data. We show that by providing (properly sampled according the distribution of interest) text descriptions tabular data samples, LLMs can produce summary of the samples that serves template for classification and achieves the aim of acting weak learner on this task. We incorporate these models into a boosting approach, which in some settings can leverage the knowledge within the LLM outperform traditional tree-based boosting. The model outperforms both few-shot learning and occasionally even more involved fine-tuning procedures, particularly for tasks involving small numbers of data points. The results illustrate the potential for prompt-based LLMs function not just as few-shot learners themselves, but components of larger machine learning pipelines.
Introduction
Weak learners refer to classifiers that are able attain better performance than random chance, by some given margin, any specified distribution over training data. One of the early breakthroughs in machine learning established that this weak learning was sufficient for arbitrarily strong classification, via an ensembling procedure [Schapire, 1990]. This approach in turn led to the development of boosting algorithms [Freund and Schapire, 1997], a class of approaches that continue perform extremely well, particularly on tabular datasets that lack the input space regularity of vision or language tasks.
In a seemingly separate thread of research, large language models (LLMs) based on transform- ers [Vaswani al., 2017] in recent years have come to dominate many natural language domains. These models are often finetuned on the data of new downstream tasks [Devlin et al., 2018. Liu al., 2019], but recent years have also been shown exhibit strong performance as zero-shot or few-shot learning solely via prompting the model [Brown et al., 2020] with a piece of context string.
In this paper, align these two threads of research and ask simple question: LLMs also serve as weak learners in a boosting framework, specifically on tabular data (where boosting methods are most commonly applied)? We answer this question largely in the affirmative. Specifically, show that by appropriately converting tabular data to text form, and asking LLMs to summarize carefully chosen set of examples from the data, produce summary of the examples that can serve as template (i.c., prompt) for a tabular data classifier, and one which typically achieves this weak learning aim. This enables to correspondingly integrate this collection of LLM- generated weak learners into a boosting framework.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

We show that the resulting approach performs well in many settings, easily outperforming zero-shot and few-shot classification, as well as "single-shot" summaries generated by the LLM. This all done without any retraining finetuning of the L.M itself, but rather only via prompting. Furthermore, on certain domains (particularly those with very few examples, where leveraging the prior knowledge built into LLMs would be of particular importance), we show that the approach can even outperform traditional tree-based boosting and LLM-based finetuning methods and its performance would likely improve LLMs capabilities improve. Overall, we believe this work highlights the potential of incorporating LLMs as sub-routines of a larger machine learning system.
2 Related Works
Deep Learning for Tabular Data. Tabular data refers to a generic data format that represents data as a collection of discrete or continuous attributes [Borisov et al., 2021]. Due to their flexibility, tabular data are ubiquitous in many ML settings. However, such flexibility comes with a cost they lack the inherent structure found in images or text, which makes applying deep learning to them challenging. Furthermore, they are often fomainn-seccifi and may have a relatively small number of data points. As a result, traditional deep learning methods, which thrive on large datasets and high-dimensional data, have seen limited success when applied to tabular data [Gorishniy et al., 2021, Shwartz-Ziv and Armon, 2022].
Recently, however, there has been increasing interest in applying deep learning tasks related to tables such as data integration, imputation [Narayan et al., 2022], semantic parsing, and even running SQL queries [Herzig al., 2020, Yin al., 2020]. Deep learning models have also been successful at learning tabular data classification by optimizing loss functions [Hollmann et al., 2022, Schäfl et al., 2022. Dinh et al., 2022]. Unlike these approaches, we study how we can use LLM for classifying tabular data without finetuning or building new language model. Since many tabular data can be grounded in natural language. texts are in fact a natural representation for tabular data. Motivated by the observation that LLMs can convert tables to text through prompting alone [Saha et al., 2022], we utilize LLMs to do this conversion. After the conversion, our classification algorithm also interacts with existing LLMs strictly through prompts. This creates an abstraction between the underlying language model and the learning procedure which may be desirable for various applications since access to the gradients or parameter updates are not required.
Prompting Prompting [Liu et al., 2023] refers to providing initial text or instructions to guide the response of a language model. The advancements in Language Model-based Learning (LLM) have unveiled new capabilities, such as chain of thought reasoning [Wei et al., 2022], zero-shot reasoning [Kojima et al., 2022], compositional problem solving [Zhou et al., 2022a], and self- improvement [Huang et al., 2022, Ho et al., 2022, Haluptzok et al., 2022]. As a result, prompting has gained widespread application across various Natural Language Processing (NLP) tasks, including arithmetic, common reasoning [Wang et al., 2022b], among others [Brown et al., 2020]. While prompts offer flexibility, it is crucial to note that LLMs interpret them differently from humans. Therefore, the process of prompt tuning, which involves carefully engineering prompts, becomes essential for obtaining accurate and relevant outputs [Reynolds and McDonell, 2021]. At its core, prompt tuning is an optimization process that aims to find the best prompt for a certain downstream task. Though a long line of works propose gradient-gun guidedd search optimize "continuous prompt" instead of the language tokens [Liu et al., 2021, Qin and Eisner, 2021, Lester et al., 2021, Shin et al., 2020, Rakotonirina et al., 2023, Wang et al., 2022c, Diao et al., 2023], gradient-based updates can be limiting, as LLMS become bigger and the access to these models become increasingly API-based. Our approach aligns more with discrete search methods based on the fact that LLMs can automatically generate prompts for themselves [Zhou et al., 2022b, Zhang et al., 2022, Yu et al., 2022]. Specifically, we prompt the LLM to summarize the tabular dataset. The summary turn acts as prompt that the LLM uses to make predictions as it encodes knowledge of the dataset. A sequence of such prompts summarizing different subsets of the data can be seen as weak learners for boosting procedure.
Boosting Boosting [Schapire, 1990, Freund and Schapire, 1997] is widely used technique to improve the accuracy of model by combining weak learners (models that perform slightly better than random guessing) to make a strong learner (model with high accuracy). Common boosting algorithms include AdaBoost [Freund and Schapire, 1997], Gradient Boosting [Friedman, 2001], and Stochastic Gradient Boosting [Friedman, 2002]; the XGBoost library [Chen and Guestrin, 2016] in

---

fresh products
milk products
grocery prodlccts
Trozen products
detergents and paper products
delicatasson products
oustomer's rogion
customer's
charnel
7057.0
06
0666.0
1782.0
$293 0
1778
Gluda Labon and
Rocal
Metadaata
Fresh products:
The sxssect eefers to dennts of sipyitutt
grocery products: high
mik products medium
DItI Preprocessor
Concetanate the ground truth
on oivens product categeries
monetary (ruz)
dee calessen producti medium
devarpentt
products:
cuatomer's negann: Cutarite Luizon and Porto
Duta Conversion prompt
Data Description
(Wettdaan))
The customor's oponding on fresh products low while spording on mi products, grocery products and detergents poper productss high: Spendlngg products The cuskomeris from outsite Libon and Ponto eta Hence Ihi austomer's channel was Forol.
Hore one osample dataect.
Goal: Deser the given data words. (Datts as Teats Use desoribe
medum.
not add addilionallnn inforaaiionnnn
Figure 1: The conversion for a data point on the Wholesale customers dataset (OpenML ID 1511).
particular is a commonly used implementation of gradient boosting. In concurrent work most relevant to ours, Hou et al. [2022] integrate LLM into AdaBoost for natural language inference (NLI) tasks, by training an MLP projection of the final hidden state of special token. Our proposed method diverges from theirs in two key aspects. Firstly, our method avoids the overhead of learning additional parameters, is gradient- free, and does not require access to the model's 's internal states. Secondly, instead of storing knowledge in parameters, our approach concentrates on condensing knowledge into an intermediary representation referred to as "summary." " This alternative strategy enhances interpretability and strictly learns through prompts, rendering it particularly suitable for small tabular data, where the prior knowledge in LLM can significantly benefit the learning process.
Summary Boosting with Language Models
We now describe the main methodology of our paper, which uses LLMs to generate weak learners, and turn, uses these weak learners within a boosting framework. We refer to the full method Summary Boosting, the learning process is that uses a language model to create summary of (specifically chosen) samples from the dataset; these summaries themselves function as examples. Finally,
prompts by which an ensemble of these summaries that gives the overall predictions on new data points.
make predictions
boosting construct
3.1 Data conversion
To utilize large language models (LLMs) with tabular data, it is necessary first convert the records into natural language descriptions. We will refer to these as data descriptions. Template matching, commonly used in previous approaches [Dinh et al., 2022], inserts attribute values into predefined templates. However, this approach often produces unnatural descriptions that differ from how humans might describe the data. Depending the dataset, designing the template by hand can also be challenging. To overcome this, we propose using LLMs as more suitable solution.
As illustrated in Figure 1, we can get these data descriptions with little effort by zero-shot prompling the LLM with information about the dataset (which generally available as metadata for tabular datasets) and textual representation of the tabular record (e.g., parsed ISON). Specifically, to ensure examples can serve as both training data and query inputs, we extract the descriptions of the features and concatenate them with the target label using separator token (refer to Appendix A. Interestingly, we find that descriptions generated by LLM this way often perform better than those from template. This ablation study can be found in Section 5.
One key challenge in this process how to encode numerical attributes effectively; naively including numerical values in the descriptions can lead to poor performance in subsequent learning tasks. To address this, we adopt straightforward approach: we bin all numerical features into percentiles and encode them descriptively as "low," "medium," and "high,". : In Section 5, we compare the performance of several such approaches and discuss more examples in Appendix A.6. Overall, the data descriptions can be generated automatically with minimal manual engineering.
3

---

Records
The datane offees dieen veekume booo,
Summariiac
Cardicate simmanie
this suturne sgmnk ancoarcas milk and Adlahseen prodacts high arnourt,, une palldeeige
iigirraati unta divers pokket
troser presictt
Rarndia The Harro nocorne seneer sll spends on araatt onaal exupter del calsssn
(Rich Representatien esomples
Ports, aee flance donal
Stratited cluster campling
LLM
and tush products. They speed verylose smount mils, product, and detergents and outaie Laber Ports and nca tus cussener
Tadly Summories lee detail how can lotether cuntonr
Select best sumtira
Hypothesis generation
Inferance prompe
Qyenry
Goal: Prediciors (Metaddata)
This customer
lwah, products, and spends less on dete deserpents and paper products and delicatessen
and frozen
(Bummary)
LIM
(SErawY)
products. The oustomer ii from outaide of liston and
Now, (query)
Therefore. this customer' channel
Inference
(Horeca Ratall;
The process of generating summaries and using them make predictions on new data. The top half describes how the weak learning hypothesis (summary) is generated. The bottom half illustrates how the summary is used perform inference. "Best summary" is the one achieving the smallest validation error rate, or higher training error in case of a tie.
Figure 2:
3.2 Weak learning via summarization
A typical method for performing few-shot learning with large language models (LLMs) involves providing a small number of demonstrations of the intended task as a prompt and then asking model to generate natural language descriptions above and generate predictions on new examples. However, for tabular data, there may be a larger number of data points that do not fit within the LLM context. Furthermore, we observed that increasing the number examples the context naively does not always improve performance (Figure 4, right bottom), and there tions over examples as is required in boosting methods. These observations necessitate alternative approaches to weak learning via LLMs.
answer. One could, for instance in the few-shot setting, simply present the
obvious way to manage weighted distribu-
propose instead that producing summaries of collection of examples can serve as a powerful proxy for learning models based upon some number of examples, as summarization naturally encourages the extraction of representative information in the data. Concretely, given set of data descriptions, we first perform summarization the data by calling the LLM (e.g., by concatenating list examples in natural language form and appending the prompt "tldr"). This resulting summary can be seen as hypothesis it provides an explanation for the data. By using the summary prompt, the LLM turn uses the hypothesis to perform inference instead of the raw data description sometimes be noisy,
(shown in Figure 2). Since the sampled summary of summaries and pick the one with the smallest validation eтror rate. In case of a tie, we choose the with higher training error, i.e., lower generalization gap (see Appendix A.2). Several methods of building such summaries are possible, but simple approaches such as the "tldr" approach mentioned above tend work well more sophisticated alternatives, Additional considerations designing these prompts are elaborated in Appendix A.2 and A.3.
generate fixed number
show in Section 5.
(Weighted) Cluster Sampling. Since the context size of existing LLMs is still limited, we cannot general fit the entire dataset into the context for summarization. Furthermore, boosting algorithms require that we provide weak learners on weighted samples of the training set, effectively guiding the boosting process focus on "harder" examples as the boosting process continues. Thus, instead of attempting to summarize the entire dataset, we propose to use only representative subset of the dataset. The size of this subset is governed by the maximum context size and size of the data descriptions. To select this representative subset, we use weighted stratified sampling using subpopulations defined by clusters of language embeddings of each data description. The language embeddings are sentence representations generated by GPT-3. In particular, we use hierarchical agglomerative clustering [Nielsen, 2016] to identify clusters the embedding. This process shown in Algorithm I. As we will show in Section 5, this process is able to consistently produce weak

---

Algorithm Cluster Sampling
1: Input: X, all training data; all training label; r, ratio of classes; AdaBoost weights of the
current round; s, target number of samples.
[k] is the proportion of examples in class
2: S < new empty set
3: < new array with same length as filled with -1. i> [i] is probability of sampling example i.
4: for k = to number of target classes in do
5:
f- GPTEmbedding [y k])
refers the embeddings of the data descriptions
6: C' < eeglomerraiiveCusteringgg
Cj is set of data indices present in the j cluster.
7:
new empty array same size as C.
& c[[]] will store sampling probability of cluster j.
8:
for = to Len(C) do
9:
c[j] < (cn(Cy)
(e)(X)
10:
end for
11:
for = to len (X) do
12: [[i] < c[j]. such that, Cj
13: end for
14: Normallie(NNrmalzzzsss)) p) D. Normalize turns weights probability distribution.
15: Sample s x r[c] examples from using categorical distribution w and append to S.
16:
end for
17:
Return S
Algorithm 2 Summary Boosting (compact version)
1: Input: x, all training data; y. all training label; maximum number of rounds; s: size of the sampling subset; ratio of classes.
r [k] denotes the proportion of examples in class k.
2: h, e, a f- new empty arrays of length T.
3: N <- len(X); K <- number of target classes y.
4: w < new array of length N filled with 1 n T do
w is the data distribution
5: for r
6:
(x., + ClusterSampling(X, y, r, w, s) sample 8 training examples from distribution w. D h[r]] is the weak learner in the current round
7: h[r] +- Summary yx,
8:
e[r] D v(i] (XC=)
e[x] is the weighted error at round
9:
if e[r] > 1 1- K
then
10:
Goto Step 6.
11:
end if
12:
a[rr] + log c(x) + log(K - 1) [[r]] refers to coefficient of the hypothesis at round r.
13:
for 1 to N do
14:
w[i] = w[i] x exp(a (x(i) + y(li])
end for 16: w +- Noromalliz))ss
end for 18: Return h,
learners, and able to improve upon random guessing under the distribution of interest (denoted by the input p the algorithm). We share more details in Appendix A.7.
3.3 Boosting
Finally, use the AdaBoost [Freund and Schapire, 1997] algorithm to produce ensemble with these collections of summary-based weak learners. The central idea of AdaBoost is to a sequence of weak learners repeatedly modified versions of the data. The algorithm is carried out over T rounds, where the weights the training data points are adjusted based on the training error.
Given new data point, the predictions from classifiers from all rounds are then combined through weighted majority vote to produce final prediction. We use the error on holdout validation set to determine the number of rounds T. A compact version of this process is presented in Algorithm 2. In the algorithm, the Summary method summarizes the examples in the prompt via the process discussed in Section 3.2. Each summary be treated hypothesis that can classify new data.

---

However, unlike the summary process in Section 3.2, where we resample multiple times to find the best learner, the boosting process returns immediately when a summary with an error rate better than random guessing is found (refer Appendix A.2). We use ChusterSampling to subsample mini-batch of examples that fit within the L.MM's allowed context length. In Appendix A. and A, we provide a time complexity analysis of our method, including cost estimations for using APIs, Appendix A.8 covers the full version of our boosting procedure that works practice.
Experiments
We conduct all of our experiments with OpenAI's GPT-3 API [Brown al., 2020] and choose collection of 18 tabular datasets from the UCI dataset [Dua and Graff, 2017] and OpenML [Vanschoren al., 2014]. All main experiments are done with the Curie variant of GPT-3 unless otherwise specified, which has 13B parameters'. We compare the following methods:
Zero-shot: query the language model with the data description and ask the model to complete the answer (refer Appendix A.4).
Few-shot: provide a few labeled data descriptions of the training data the context and ask the model to complete the answer for a new data description. To preserve consistency, we standardize the number of fewshot examples approximately 15 for all datasets. The setting is explained in Appendix A.5.
Summary (ours): generate a population of summaries given a list of data descriptions with cluster sampling and pick the summary with the lowest validation error; the best summary as the context and ask the model to complete the answer for a new data description.
Summary Boosting (ours): use Summary as a subroutine AdaBoost.
Furthermore, we compared Summary Boosting against popular baselines for tabular data that do not prompting:
KNN: first embed the data descriptions with the GPT-3 embedding API and then use K- nearest neighbor classify new data description. This simple baseline demonstrates how much information can the naive representation produced by LLMs provide about the tasks.
LIFT [Dinh et al., 2022]: Language- Interfaced Fine- Fine-Tuning (LIFT) finetunes the LM with data descriptions (without binning) and their corresponding labels in zero-shot.
Language-Interfaeed
TabPFN [Hollmann et al., 2022]: TabPFN is transformer-batt architecture that performs Bayesian inference on the entire training and test data points at the same time.
XGBoost [Chen and Guestrin, 2016]: XGBoost (eXtreme Gradient Boosting) a regularized gradient boosting algorithm that is widely used for tabular data.
For each method and dataset, we use a 50/10/40 split for train, validation, and test sets and repeat each experiment for 3 random seeds The results are shown in Table and 2.
4.1 Analysis of prompting-based methods
As a general trend from Table 1, test performance improves in the order Zero-shot < Few-shot < Summary Summary Boosting. Firstly, unlike most works zero-shot reasoning with LLMs, the LLMs do not have enough prior knowledge to make the correct prediction without additional information. As result, observe that Zero-shot performs poorly on all of the datasets. This observation highlights the necessity of learning from the data, and unlike other tasks, the L.LMs themselves do not have enough built-in knowledge zero-shot does not have enough prior knowledge to classify tabular data, we use few-shot in-context learning (Few-shot) to see the added information helps make better predictions. As expected, on all the datasets other than pisualizing hamster, and wholesale-cusstomrrss Few-shot consistently improves the test performance compared to Zero-shot, suggesting that this added information is crucial for LLMs to work on tabular datasets.
succeed at tabular data zero-shot. Since
Wee use Curie because is more cost-effective for large-scale experiments. httpps:/ //beta. openai. wpwnai.com//oseeeeeppp..pppee www.ideeeeeemedddign....

---

Table 1:
Test errors for the chosen methods on all datasets ()). Data Type indicates the number and types of attributes the dataset has (c is continuous and d is discrete). Size denotes the number of data points. In square brackets (if present) next to every dataset name, we provide its acronym referred to in our main text. In the bracket next to each dataset name is either the OpenMI. ID of the dataset or reference to the dataset's s associated publication. Error represents one standard deviation.
Dataset
Data Type
Zezo-shot
Fuu-shot
Sussary
Burnary
Bocating
cacsarian [cac] (42901)
1c44
U.4255+00
0,388+
0.350+ 0.04
0.3004
iris (61)
1:04
150
0.680 0.02
0.450 0.01
0.275++ 0.07
0.193 0.08
(48)
1c43
151
0.036 0.07
0.4944 0.01
0.4744 0.02
0.4544 0.03
9:04
0.486+
0.473+
0.466+
0.370
breast cancer [bc) (13)
7c0d
0.754+ 0.02
0.516+
0.337
0.2884
visudlazing emvroonental [ve] (678)
3:04
111
0. 0.5222 0.01
0.3662 0.01
0.3044 0.02
0.268$ 0.05
analcandmm chizmydia dia [ac] (535)
2:24
100
0. 0,200++ 0.00
0,200+ 0.00
0.170 0.01
0.170= 0.01
(43571)
13:0d
178
0,820
0.674 0.02
0.475
0.320 0.01
bload trassusixe-eeetee [bxc] (1464)
4:00
U.6444 0.03
0.430+
0:25844 0.04
0.240 0.04
somerville happiness-unn survey [shs] [Koczkodaj, 2018]
0:74
143
0. 0.416+ 0.03
0.385+ 0.03
0.422+ 0.02
0.350 0.02
vehicle (54)
18:03
846
0. 1.780
0.5602 0.01
0.510: 0.02
0.410 0.04
statlgg heart [stuch] [uu and Graff, 2017]
0.6284
0.444
0.430=
verterbra column [ve] (1525)
6:04
0.714+ 0.03
0.435+
0.327 0.01
0.2624 0.01
scall (1011)
7:00
336
0.681 1 0.02
0.5622 0.01
0.4800: 0.01
0.270 0.03
habemman survival hss (43)
3:04
305
0200 0.3088
0.2622 0.01
0.277 0.01
0.200= 0.01
disbetes (dia) (37)
8:04
0. 0.4466+
0,400+
0.360
0.344
visnalizing- flaneter [bams]] (708)
5:04
0.4644::0 0:03
0.4814::0 05
0.360: 0.02
0.207 0.00
wholesale customers (wc] (1511)
6:14
440
0.364 0.01
0.347 0.01
0.349 + 0.02
0.930 0.00
Table 2: Test errors for chosen methods on all datasets (L). (acronym and notation in Table 1)
Daliset
Duta Type
Size
Summary
Boosting
LIFT
KBM
TaaPFN
Igboost
cae (42001)
1old
80
0.300 0.04
0.312 0.02
0.300 0.00
0.425. 0 07
0.4124 0.05
irik (61)
4x00
1.0
0.193 0.03
0.100 0.01
0.1061 02
0.027 0.00
0.0541 0.04
(48)
1c6d
0.454-
0.08
0.532+
0.450
0.4644
glos (41)
9c0d
214
0.370= 0 02
0.218=0.02
0.2944 0 03
0.158 0.05
0.2544 0.05
be (13)
7:5d
277
0.288 0. 02
0.31820.01
0.2771 02
0,264 0.01
0.2701 0.01
(678)
3cdd
111
0.268.1
0.450
0.308 0.01
0.370 0.04
0.279
0.170=0.0 0.01
0.180
0.1
0.1101
wine (43571)
13c0d
178
0.320= 0.01
0.085=0.03
0.214 0.05
0.040 0.01
0,040: 0.01
(146))
4c0d
748
0.240= 0.04
0.270=
0.238 D. 00
0.209 0.01
0.2104 0.01
<he [Kocskodaj, 2018]
0c7d
143
0.350 0.02
0.419=0.02
0.326 0.03
0.3921 0.00
0.4061
vehicle (54)
18:0d
0.410=0.04
0.10
0.636+
0.178+
0.260
stuhh [Dus and Grall, 2017]
6c7d
270
0.00
0.122:1 0.17
0.244 0. 0.03
0.148 0.03
0.2154:.00
ve (1554)
6:0d
310
0.26220.01
0.192=00.03
0.318 02
0.135 0.00
0.1874
acoli (10LI)
7c0d
336
0.270=
0.126=0.08
0.211 0.08
0.036 0.02
0.068+
(43)
3c0d
0.250
0.31420.03
0.2781
0.2624
0.26110.00
dia (37)
8c0d
768
0.344 0.01
0.324-0.04
0.353+ 0.02
0.238++ 0.03
0.234: 0.00
hams (708)
5c0d
73
0.207 0.00
0.334=0.08
0.528 0.02
0.328 0.01
0.411 0.01
(1511)
6c1d
440
0.330= 0.00
0.125=0.04
0.0431 0.00
0.0884 8.00
0.098 0.02
Unlike naively stacking examples inside the prompt as in Few-shot, Summary condenses knowledge from these examples and is the first important algorithmic component of our framework for creating weak learners using LLMs. We see that Summary consistently improves upon Few- shot on all the datasets other than interrman-survvvee that summarization a powerful way to improve few-shot performance and has the potential for even other tasks using L.MSs. Finally, for every dataset we tested, boosting with summarization consistently outperforms all other prompting-based approaches. This observation corroborates our hypothesis that LLMs with summarization are good candidate for creating weak learners in boosting.
and
eewal-cusstteerreeeeeeeeaeeeaeee This observation suggests
4.2 Comparison to other tabular methods
In Table 2, we also observe that LLMs have a hard time reasoning about continuous attributes without finetuning, especially on the glass, wine, iris and vehicle datasets. In particular, when the datasets have many continuous features, the performance of Summary Boosting can be considerably worse than other methods such as LIFT or Xgboost. This may be because LLMs are fairly bad at quantitive reasoning without finetuning [Lewkowycz et al., 2022], which may be overcome in future LLMs.
While KNN is a relatively simple baseline, its performance is surprisingly good at a few tasks such as schoesslle e- customers, statlog-hears ecoli and wine. This highlights that LLMs have remarkable amount of general prior knowledge about the world compared to methods like XGboost that sometimes this knowledge alone can produce good performances.
7

---

Task agnostic training
Sampling boosting
Type
Curee Davinci
cae
w/ names
Dabaett
ve
Cluster Random
glass
Template
boc
0.0
0.2
0.4
0.0
0.5
50 100
0.0
0.2
0.4
0.25
0.50
0.75
Test
Test
Number rounds
Test
Validation
Figure 3: Ablation experiments compared for the Summary method. The dataset acronyms are referred to Left: Performance of curie vs davinci. Second from left: Comparison with and without task-specific attribute names, Middle: Effect of Cluster vs. Random sampling on the number of rounds till convergence and Second from Right: their final test errors. Right: Performance of templatized vs LLM generated data descriptions.
Grouped
Detailed
Prefix
Tüdr
Examples ordering
Summary prompt Inference prompt
jao
0.0
parcent0e
ard
quardas
Comparing summary vs. few shot with support set size
tac
bc
ve
VS
Tew
6--
Toat armar
8.5
0.25
0.50
0.75
1000
0.5
1.0 0.2
0.4
0.6
0.8
Validation enror rate
Support set size
Figure 4: Additional ablations for the Summary method. (Left) Prompt design choices. The first plot shows the effect of shuffling examples vs presenting them by class. The center plot compares tldr more explicit prompt for inducing summary. The last plot compares prompts for doing inference. (Right Top) Performance of methods for discretizing continuous attributes on the Wine dataset. (Right Bottom) Performance of Few-shot and Summary function of the number of examples in the context.
Finally, we observe that Summary Boosting performs very well when the size of the dataset is very small. This makes since the strength of using LLMs weak learners is that they have large amount of generic prior about the world from pre-training. When the dataset is large, this prior knowledge might become less relevant and methods like finetuning become more competitive.
Ablations
Summarization forms the of our methodology for generating weak learners. Consequently, it becomes important to identify an ideal setting that can induce high-quality summaries. We perform ablation studies the Summary method, decide hyperparameters for getting good weak learner.
Preprocessing of continuous attributes. We tried several encoding techniques for continuous features, including binning, percentiles, and standard deviations. We chose the approach of describing them in technical language terms as well as assigning quantifiers for each level, as illustrated in Figure 4 right top. observed that binning with quantifiers such as "low". "medium", and "high" was most effective for comparing examples and generating high-quality summaries. After hyperparameter tuning, we identified that using 5 bins provides sufficient granularity to distinguish variations the continuous values. More details can be found in the Appendix A.6.

---

Does the LLM explore prior knowledge to infer? To demonstrate the LLM's utilization of prior knowledge, we conduct an ablation study by masking the attribute names and using a template "This example has features // [). f2 = (] and so on." Figure 3 (second from left) shows the result. Using true variable names the data descriptions leads to superior few-shot learning performance compared to using dummy names. This confirms that the model indeed leverages its prior knowledge of variables for predictions.
How does model size affect the performance? A natural question to ask is how the model size affects the downstream performance. We compare the Summary performances of GPT-3-davinct (175B parameters) and GPT-3-curie (13B parameters) on 5 datasets in Figure 3 (left). Surprisingly, we find that the larger model (davinci) does not consistently improve upon the smaller model. We also compare ChatGPT in A.12 and discuss the effects of RLHF [Ouyang al., 2022]. We further show that our method generalizes to LLMs other than GPT, by comparing its performance with Claude-2 in Appendix A.13).
How does the performance scale with more examples? In Figure 4 (right bottom), we study how the behavior of FFw--sof and Summary change with different support set sizes (i.c., the number of data descriptions that are summarized). Feu-shot performance reaches an optimal size around the medium context length and degrades with more examples. contrast, Summary improves with more data, which is the more desirable behavior.
Ordering of examples. Unlike conventional machine learning models, the ordering of examples in summarization affects the generation of hypotheses. There are two approaches: 1. presenting descriptions randomly (shuffled), and 2. grouping descriptions by classes (grouped). Figure (left), we compare these approaches on 4 datasets with Summary and find no significant difference. We use shuffled for all other experiments.
Different summary and inference prompts. The LLM easily generates concise summaries on standard datasets like iris using a simple "tmple "t1;dr" prompt, but requires a more explicit prompt on complex datasets like "vertebra-column". Comparing their performance in Figure 4 (left), both prompting modes are equally effective, so detailed prompts were used in all other experiments. See Appendix table 3 for the complete list of summary prompts used. In the left of Figure 4, we also compare two strategies for inference- prefix prompt (e.g. "This flower will be classified as"), and two-stage
prompting (e.g. "Let's think step by step") [Kojima al., 2022]. We observe no statistically significant difference between them'.. Since the prefix prompt more often completes the query well under lesser compute, we use prefix prompt for all the other experiments.
Texts generated from template vs. GPT. In Figure 3 (right), we see that using GPT-generated data descriptions consistently achieves better results. Refer to Appendix A.9 for examples of these templates. This may be due to the fact that the data description generated by LLMs conforms to natural text distribution more closely, which is desirable for performing inference using LLMs.
Effect of cluster sampling. Cluster sampling improves the performance of the LLM by selecting representative set of texts that generalize better, reducing validation error faster compared to random sampling during boosting. Although it may require more resampling achieve weak learner, cluster sampling converges much faster than random sampling as we see in Figure 3 middle and second from right. However, with sufficient boosting rounds, the performances of the two sampling methods are not statistically different.
6 Limitations
Although the weak learning approach we develop here shows promise, there are currently several drawbacks. Although summarization and boosting alleviate manual prompt tuning to large extent, we still had to minimally tune some parts the pipeline to get ideal performance (see Appendix table 3). Additionally, when the dataset contains many continuous attributes, there is non-trivial gap between Summary Boosting and the other methods such as XGBoost or finetuning. Finally, the
We do observe that CoT performs better with larger models.

---

max imput length of GPT-3 makes it harder to generate good summaries with just subset sampling. Eventually on larger datasets, after certain number of boosting rounds, the summaries derived from subset of examples may not further decrease the weighted error across all training examples. Rigorous techniques such as structured prompting handle this issue by rescaling attention weights [Hao et al., 2022]. We believe this issue could be solved with more powerful LLMs such as GPT-4. Furthermore, methods like ours built using LLMs inherit bias that comes from the pretraining data. Specifically, can affect the model's ability to objectively summarize the examples make predictions that are inconsistent with the biased pre-training data. We believe with better and debiased LLMs these issues can be alleviated. We want to emphasize that our experiments utilized OpenAI APIs. In the appendix A. A.14, we provide a brief discussion on the impacts commercial APIs.
Conclusion
LLMs have been widely used in recent years, not just for their generative abilities but for their ability to serve as zero- or few-shot learners with proper prompting. This work aims to situate them within another context of "simple" learning algorithms the weak learner paradigm that forms the foundation of boosting approaches. We show that leveraging the summarization capabilities of LLMs indeed leads to models that function weak learners, and which thus be integrated into boosting frameworks. "Suummay"" as an intermediate step makes reasoning easier and works inside boosting algorithms for small tabular data up to 300 data points without many continuous features. Overall, this result leads to new potential paradigms for treating the results of LLM prompting not just as individual predictors, but as part of a larger set of meta-models well.
Acknowledgements
Yiding is supported by funding from the Bosch Center of Artificial Intelligence. We would like to thank the anonymous reviewers for their valuable feedbacks.
References
V. Borisov, T. Leemann, K. Seßler, J. Haug, M. Pawelezyk, and G. Kasneci. Deep neural networks and tabular data: A survey. arXiv preprint arXiv:2 10.0888, 2021.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
T. Chen and C. Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785-794, 2016.
G. Cui, S. Hu, N. Ding, L. Huang, and Z. Liu. Prototypical verbalizer for prompt-based few-shot tuning, arXiv preprint arXiv: 2:XXi:2203.097000 2022.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.048005 2018
S. Diao, P. Wang. Y. Lin, and T. Zhang. Active prompting with chain-of th-thugh for large language models. arXiv preprint arXiv:2302.112246 2023.
T. Dinh, Y. Zeng, R. Zhang, Z. Lin, S. Rajput, M. Gira, J.-y. Sohn, D. Papailiopoulos, and K. Lee. Lift: Language-interfaeed fine-tuning for non-language machine learning tasks. arXiv preprint arXiv:2206.0655 06565, 2022.
D. Dua and C. Graff. UCI machine learning repository, 2017. URL htt.::://aacchie.....aa edu/ml.
Y. Freund and R. E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of computer and system sciences, 55(1):119-139, 1997.
H. Friedman. Greedy function approximation: gradient boosting machine. Annals of statistics, pages 1189-1232, 2001.
10

---

J. H. Friedman. Stochastic gradient boosting. Computational statistics & data analysis, 38(4): 367-378, 2002.
Y. Gorishniy, I. Rubachev, Khrulkov, and A. Babenko. Revisiting deep learning models for tabular data. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, edi- tors, Advances in Neural Information Processing Systems, volume 34, pages 18932-18943, Cur- ran Associates, Inc., 2021. URL https / http///peeeenn....//// wwweeediinnnnnniips...... /eearr/ps.ccpppppe/2////////// aBGG@81122f21199eaiaabbe222222 Paper pdf.
Haluptzok, M. Bowers, and A. Kalai. Language models can teach themselves to program better. arXiv preprint arXiv:2207. 14502, 2022.
Y. Hao, Y. Sun, L. Dong, Z. Han, Y. Gu, and F. Wei. Structured prompting: Scaling in-context learning to 1,000 examples. arXiv preprint arXiv:2212.06713, 2022.
J. Herzig, P. K. Nowak, T. Müller, F. Piccinno, and J. M. Eisenschlos. Tapas: Weakly supervised table parsing via pre-training. arXiv preprint (tt::2004.023449 2020.
N. Ho, L. Schmid, and S.-Y. Yun. Large language models are reasoning teachers. arXiv preprint arXiv:2212.10071 2022.
N. Hollmann, S. Müller, K. Eggensperger, and F. Hutter. Tabpfn: A transformer that solves small tabular classification problems in second. arXiv preprint arXiv.2207.01848, 2022.
B. Hou, O'Connor, Andreas, S. Chang. and Y. Zhang, Promptboosting: Black-box text classification with ten forward passes, arXiv preprint arXiv:2212.09225,, 2022.
S. Hu, N. Ding, H. Wang, Z. Liu, J. Li, and M. Sun. Knowledgeable wommpt-uuning Incorporating knowledge into prompt verbalizer for text classification. arXiv preprint arXiv:2108.0220 02035, 2021.
Huang, S. Gu, L. Hou, Y. Wu, X. Wang. Yu, and J. Han. Large language models can self-improve. arXiv preprint arXiv:2210.1166 2022.
W. W. Koczkodaj. Somerville happiness survey data set. https://archive.ier uci edcim/d 2018.
datasets/
Kojima, Gu, M. Reid, Matsuo, and Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916 2022.
B. Lester, R. A1-Rfou, and N. Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.0869) 2021.
A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, Ramasesh, A. Slone, C. Anil, Schlag, T. Gutman-Sole al. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206. 14858, 2022.
Liu, W. Yuan, Fu, Z. Jiang, H. Hayashi, and G. Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35, 2023.
X. Liu, Y. Zheng, Z. Du, M. Ding, Y. Qian, Z. Yang, and J. Tang. Gpt understands, too. arXiv preprint arXiv:2103.1 10385, 2021.
Y. Liu, M. Ott, N. Goyal, Du. M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoy- anov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 190. 2019.
A. Narayan, Chami, L. Orr, and C. Ré. Can foundation models wrangle your data? arXiv preprint arXiv:2205.09911, 2022.
F. Nielsen. Introduction to HPC with MPI for Data Science. Springer, 2016.
L. Ouyang, J. WU, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.

---

G. Qin and J. Eisner. Learning how to ask: Querying Ims with mixtures of soft prompts. arXiv preprint wxXi:2214.066599 2021.
N. C. Rakotonirina, R. Dessi, F. Petroni, Riedel, and M. Baroni. Can discrete information extraction prompts generalize across language models? arXiv preprini arXiv:2302.09865, 2023.
L. Reynolds and K. McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1-7, 2021.
S. Saha, X. V. Yu, M. Bansal, Pasunuru, and A. Celikyilmaz. Murmur: Modular multi-step reasoning for semi-structured data-to-text generation. arXiv preprint arXiv:2212.08607 2022.
B. Schäfl, L. Gruber, A. Bitto-Nemling, and S. Hochreiter. Hopular: Modern hopfield networks for tabular data, arXiv preprint arXiv:2206. 00664, 2022.
R. Schapire. The strength weak learnability. Machine learning, 5:197-227, 1990.
T. Shin, Y. Razeghi, R. L. Logan IV, E. Wallace, and S. Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15960 2020.
R. Shwartz-Ziv and A. Armon. Tabular data: Deep learning is not all you need. Information Fusion, 81:84-90, 2022.
Vanschoren, N. Van Rijn, B. Bischl, and L. Torgo Openml: networked science in machine learning. ACM SIGKDD Explorations Newsletter, 15(2):49-66) 60, 2014.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, t. Kaiser, and I. Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
H. Wang, C. Xu, and McAuley. Automatic multi-label prompting: Simple and interpretable few-shot classification. arXiv preprint DxXiv:2204.06305. 2022a.
X. Wang, Wei, D. Schuurmans, Q Le, Chi, and D. Zhou. Self-con coff-consistency improves chain of thought reasoning in language models, arXiv preprint arXiv:2203 11171, 2022b.
Z. Wang, Z. Zhang, C.-Y. Lee, H. Zhang, R. Sun, X. Ren, G. Su. V. Perot, J. Dy, and T. Pfister. Learning to prompt for continual learning. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 139-149, 2022c.
J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q Le. and D. Zhou. Chain of thought prompting clicits reasoning in large language models. arXiv preprint arXiv:220.111903, 2022.
Yin, G. Neubig, W.-t. Yih, and S. Riedel. Tabert: Pretraining for joint understanding of textual and tabular data. arXiv preprint arXiv:2005.08314 2020.
W. Yu, D. Iter, Wang, Y. Xu, M. Ju, S. Sanyal, Zhu, M. Zeng, and M. Jiang. Generate rather than retrieve: Large language models are strong context generators. arXiv preprint 21XX::2209.10063 2022.
Zhang, A. Zhang, M. Li, and Smola. Automatic chain of thought prompting large language models. arXiv preprint a0X:v:2210.03449, 2022.
H. Zhou, A. Nova, H. Larochelle, A. Courville, B. Neyshabur, and H. Sedghi. Teaching algorithmic reasoning via in-context learning. arXiv preprint arXiv:2211.0906. 2022a.
Y. Zhou, A. I. Muresanu, Z. Han, K. Paster, S. Pitis, H. Chan, and Ba. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.019/0 2022b.
12

---

A
Failure Modes
The design of the prompt plays pivotal role in entire process. Specifying instructions precisely can create a significant difference, whether it comes to effectively deseribing tabular data, generating reliable summaries prompting has been successful at instructing L.MSs [Lester et al., 2021], cannot be applied to our setting because our classification algorithm learns human-level summary as a prompt for classifying data, rather than a soft prompt. Instead we chose to write prompts that ask the LLM to perform these tasks. In this way, our prompting method is entirely gradient-free. Hand-written prompts also offer flexibility, aligning well with our core methodology for creating weak learner. While carefully handcrafting prompts this way might seem intensive, we will show that once we identify the ideal hyperparameter settings they can be framed with little effort.
inferring accurate predictions (shown in Figures 1 and 2). While soft
A.1 Data Conversion Challenges
Language models (LLMs) have demonstrated impressive performance on standard tasks with minimal supervision [Wang et al., 2022b, Brown et al., 2020]. However, for converting tabular data to text, there were several considerations meet the requirements of our task. As highlighted in Section 3.1, we will refer to these texts as data descriptions.
Ensuring Uniform Length Firstly, the data descriptions should not be too long or short, also be of comparable length. Excessively long descriptions limit the number of examples that can be fit inside the prompt and summarized. We observed in Figure 4 (right bottom) that the summary performance also scales with having deseriptions of approximately uniform length.
examples, so it makes sense accomodate many of them possible by
A straightforward way of achieving this uniformity would be by specifying a max word length as part of the conversion prompt itself, as "Describe in not more than 80 words". However, we found this approach can falter, sometimes leading to overly simplistic descriptions like "These are annual spendings of customer." " (in the www.lesale-uusttrree
dataset).
Consequently, we adopt more nuanced strategy by first modifying the prompt with the terms "con- cisely" and "accurately" to emphasize the brevity and preciseness of the generated descriptions (shown in Figure 1). Then, we implement resampling strategy, that generates descriptions until find- ing the one with a desired length ranging between 20 to 80 words. This process achieves consistent and uniformly long descriptions.
Including Metadata Prepending metadata to the prompt enhances the contextual awareness of the task, resulting in higher-quality descriptions (shown in Figure 1).
Separating Features from Labels In our method, the data descriptions function dual role, both as training examples and as query for inferring class labels. This suggests that, when converting data to text, the features need to be described separately from the target label as illustrated in Figure 1. The resulting strings are then concatenated to form the data description. Instead, the whole tabular record were passed to the LLM, it often produces texts that assimilate the classification label information in the meat of the description itself, rendering it difficult to extract a query for doing inference.
While one might cleverly come up with prompts that can allow the L.MM to describe the features and target label in separate sentences, describing and not reveal any information about the target task. Sometimes that can liberate the LLM to hallucinate some facts about the task and form biased data to begin with.
found it to be more sensible to supply just the features for
Natural-Sounding Descriptions While the LLM generates different-styled response every time, to explicitly ensure that the generated descriptions are not template-like by chance, add directive at the end of the prompt: "Use your creativity".. This encourages the LLM to produce more natural narratives of the record. Alternatively, setting a higher temperature during decoding achieves a similar effect.
13

---

A.2 Summarization
There are several aspects worth considering that can contribute high-quality summaries.
Sampling candidate summaries A well-crafted summary a one that captures salient information of the dataset, in a way that facilitates inferring predictions off it. However, the process of generating summary using LLM is inherently stochastic due temperature sampling, summary can be noisy. From our experiments with tuning this temperature, we found 0.80 to be ideal through Bayesian optimization. Even at this value, on average only 1 out of 3 summaries were meaningful.
result, the generated
A noisy summary can be distinguished quite easily. For instance, the vehicle dataset, the t1; ;1 prompt elicits summaries as naive as "The given data describes a bus, van, or saab silhouette." or "The data in this table identifies vehicle as bus, saab, opel, van. The compactness, circularity, distance circularity, radius ratio, hollows ratio, and symmetry are all predictive of the vehicle's 's type." which does not offer actionable insight.
This observation indicates that summaries need to be sampled quite a few times and the best one can be determined based the validation error. As result, for the Summary learning procedure in Section 3.2, we resample approximately 25 times to find a good summary. Also, given that our datasets small, it is not unusual for the summaries to have the same validation error. When tied, we pick one having a higher training error rate, i.e. lower generalization gap.
Differently, in our Summary boosting procedure explained in Section 3.3, we resample only until finding a summary whose training error is better than random guessing and return immediately.
Ordering examples inside the summarization prompt Unlike gradient descent, prompting is not robust to the presentation of the examples to the learning algorithm. While show via ablation studies in Section 5 that there is no statistically significant difference in performance between either shuffling the examples or listing them by class, can generally expect that depending on the dataset and the number of target classes, one might be preferred over the other.
For instance, in multi-class setting, listing examples by class might be more helpful in reaching a weak learner quickly. However, in two-class setting, the summary might actually benefit from randomness in the shuffled examples.
Customizing the summarization prompt The way of asking the LLM summarize examples can also give rise to good/bad summaries. For instance, one can prompt the LLM with a simple tl; or specify the task more elaborately. We will refer the latter option explicit. As we demonstrated in Figure 4 (left), both are means to the goal and do not statistically differ in terms of performance induced.
However, in our experiments on certain datasets, we would rather be incentivized choosing the explicit over the dr to attain weak learner more quickly. This choice becomes important purely for compute reasons as it will take relatively lesser resampling, while the t1; dr still works. For instance, this scenario can happen when the LLM cannot decipher what the summary is supposed to say, by just observing the examples. As examples, the t1 ; dr prompt suffices on datasets such as iris, diabetes, and wine that are commonly encountered in prediction context, whereas the LLM might not be very familar with the goals of vertebra- partebra-colume column or somerville-happ saneeesssssveyyy data, necessitating the use of the explicit prompt. For these other datasets, the fact that it is a classification problem based on some features and target classes may not be very apparent from just the examples and metadata. So, providing a directive such as "Summarize in detail how we can tell apart people with normal and abnormal vertebra-column" reduces ambiguity in the task setup and reduces probability of noisy summary.
:rr
While manual intervention is necessary, framing this prompt can be done with little effort. We provide a comprehensive list of these parameters for all datasets in Table 3.
Including Metadata Similar to data conversion, including meta-data information in the prompt offers better penetration into the world of the dataset, as result improves boosting performance.
14

---

Metadats
refors
distrbutte Includes the anpaalapendiig ng monetary certs jruv divers product catagoriia.
Inference prompt
(Metedata)
(quary)
LLM
(snswwe)
This is cusiomer who spends very ittle on freeh, milk, grocery, and frozen products, spends less on detergents procucts delicatessen
Tharafore, this customer's channal will be (Horeca Retail):
products. customer outside Lisbon Porto.
Figure 5: Steps Zeroshot prompting
A.3 Inference
Mapping labels from LLM responses Answer mapping refers the process of assigning the model's answer to a target output class. This step might be trivial when the answer is the class itself, for example when the LLM responds with "non-relapse" the breast-cancer dataset. However, in other instances, it can become tricky when the LLM's responses are "will recur" or "has higher chance of non-relapse than recurrence", requiring more complex decoding logic to identify the target class.
"recurrence" to query
Previous works have handled this problem by disguising the task as Cloze prompt and learning verbalizer, i.c. MLP projection of hidden state of the [MASK] token, that maps the predicted token to the target classes [Cui et al., 2022, Hu et al., 2021]. J. By training verbalizer, can determinsically go from token vocabulary to the label space. There also exist unsupervised statistical techniques for achieving label mapping [Wang al., 2022a].
In our method however, we strictly interact with the LLM through prompts and not access the hidden state nor gradients. As result, our inference process shown in Figure 2 focusses on inferring class label solely through prefix prompts, without relying on learning an explicit mapping. Specifically, by conditioning on suitable prefix, we constrain the LLM to return exactly the class label string. For example, the prefix "Therefore this iris flower is likely to be (sctosa, versicolor, virginica):" works for the iris dataset. key observation guiding the design of such prefix prompt the fact that specifying the output classes entices the LLM to predict from among these classes. With rather plain prefix like "Predict what will be the type of this flower.". the LLM's answer space unconstrained and might liberally go on to explain a chain reasoning such as "The flower has short petals and long sepals, hence it is versicolor, and not setosa." preventing simple keyword search for the class label.
For full list of these inference prompts, refer Table 3.
Two-step prompting, Davinei vs. Curie It worth mentioning that a two-step prompting trick, by first calling "Lets think step by step" then concatenating the response with the prefix prompt also results in accurate answers, as we have shown in Figure 4 (left). However, it could only be implemented on the larger model Davinci but not Curie which is primarily used in our experiments. Interestingly Davinci's 's chain of thought reasoming even outperforms its prefix prompt counterpart. In all our experiments with Curie however, the prefix technique works reasonably well.
The Davinci API also offers a suffix argument which can be invoked for predicting in a more natural way. For instance, for the breast-cancer dataset, the prompt can be posed with prefix "All in all, this woman is more likely to have a" and suffix of breast cancer." directly expecting the LLM to fill in with "recurrence" or "non-relapse."
A.4 Zeroshot Setting
We extend the analysis of prompting-be Zeroshot experiment. We illustrate this experimental setting in Figure 5. It only consists of the inference prompt, wherein the L.M presented with the metadata and a query. To facilitate answer mapping, the output classes are also indicated in the prompt. Unlike Summary, the zeroshot process
methods Section 4.1 by further delving into the
15

---

leat
Mecadala
This cuatomer spends medum arouns
fresh, rulk, ass deliatesson products high
wookoall disricator itindacta the anrual sponding in mondary un jrus nens produc:
products, grecery desepeons protucts.
title
This cusiomer loasted putaics usben and Ports. ane keenee this cutwier chonsells is Ricel
calaggr
Inference pronet
2500the lort spera lot
delkttooser span)
Prodiction from examples
(Metadats)
smount product,, detergents paper product.. This clent is located cutoce of inbon and Ports, ass mence customea
Stratified dusster sampling
(Rich Representative support set esamples)
Channe was honece Mitte(haaturrrr/aatt
(anawerr)
Now, (query)
Quary
Thancforo, this customer channal will ba (Horeca or Retall:
grassry, spends vary
products, spends delergents and paper peoducts deliaatesson
products. The customer from outside of Lisbon and Porto.
Figure 6: Workflow in fewshot prompting
is not stochastic there is no learning involved. For inference, the predicted tokens sampled greedily at temperature 0.
A.5 Few-shot Setting
Extending the analysis from Section 4, we explain the Fewshot setting. As illustrated in Figure it is instance of in-context learning, where the support set examples are listed inside the prompt attached with a query the end. The support set was chosen from the training set through the stratified cluster sampling outlined in Algorithm 1. This results in semantically diverse collection of examples evenly spread across the classes. Since we observe in Figure 4 (right bottom) that the Fewshot performance drops with more examples, choose approximately 15 examples to fit in the prompt.
Similar to the Summary method, this inference prompt carries meta- meta-informatte information about the dataset, and also indicates the output classes. The prompt also injects support set examples that are stringed together as we show in Figure 6. The predictions are made greedily at temperature of 0.
Again, the Fewshot method is stochastic process whose outcome depends the selection of the support set. So finding the ideal prompt. requires sampling different support sets from the training set quite few times. We perform this resampling approximately 25 times and pick the best prompt based on the validation error.
A.6 Preprocessing continuous attributes
Extending from the ablation studies in Section 5, we demonstrate Table 4, concrete examples of these encoding techniques applied continuous features. Every numerical column in the dataset was subject to the transformation independently of the rest.
We applied several encoding techniques for continuous features, including binning, percentiles, and standard deviations. Our approach involved using technical language terms to describe these ranges, such "falls in the uth bin/nth percentile deviations above/below mean". We also characterize them in a more naturalistic way by assigning quantifiers such as low, medium, and high to each level in the binning technique.
To create effective textual descriptions, we examined three high-level approaches: presenting only numerical values, 2. using solely textual encodings, and 3. concatenating both. We observed that utilizing textual encoding alone outperformed the other methods. As a result, we focused on mainly comparing textual encoding methods as shown in Figure 4 (right top). Through Bayesian optimization, we found that binning with "5" quantifiers was ideal for generating high-quality summaries.
We describe each encoding technique as follows:
Binning: It involves creating histogram with the given number of bins. As outputs, the values are directly described as "falling in the n-th bin" as illustrated the bins experiment, However, in the presence of degree quantifiers which are categorical names
16

---

assigned to the bins, these tags are used instead. We found that as opposed to calling out the bin number, describing terms of these quantifiers further aids the LLM in comparing the relative extent to which features match and improving the estimation of similarities. This led us to tune the number of bins against these degree quantifiers, selecting values in the range of 4, 5, 7, and 9 bins. The first four rows in Table 4 show how these tags get translated into the record.
Percentile: given by computing the percentile rank of a value relative to that series of values. Then, the value is described as falling in that percentile rank in words. This is closer to representation of the true numerical values per se, but helps the LLM draw comparisons on a scale of 1-000.
Standard deviations:
In this procedure, the values are segmented into six ranges based on distance from the mean, given by one/two/three standard deviations above/below the mean.
Quartiles: Here, we consider the first and third quartiles, and the median as landmarks to bucketize the values into four partitions.
Among these methods, the "5 bins with quantifiers" strikes a balance in granularity scale. is not excessively fine-grained as "percentile", nor overly abstract, as the "4-bin" approach. This balance ultimately leads to optimal performance.
A.7 Cluster Sampling components
We discuss more of the functions in Algorithm 1.
GPT-Embedding is OpenAI's text similarity model text-eeeeeeeee -ewedding--dd--Oeeee that takes max- imum input size of 8191 tokens. It returns a 1536-dimensional embedding for text. OpenAI recommends cosine distance for comparing ada embeddings in downstream tasks.
As a result, the AgglomerativeClusteriin algorithm applies hicrarchical clustering over these features using cosine distance, average linkage and heuristically selected distance threshold of 0.05. It yields a set of clusters C' and each Cj contains a list of indices of data points that belong to that cluster j.
A.8 Adaboost Optimizations
We additionally apply several run-time optimizations to the boosting algorithm described in 3.3. Thus we present its full version in Algorithm 3.
Our goal was to create high-quality summaries that dramatically reduce the validation error rate and significantly accelerate the convergence of the boosting procedure. Thus we raise the performance threshold to a notch slightly higher than random guessing probability (see Step 15 in Algorithm 3), provoking insightful summaries.
Raising the bar for a weak learner:
We resample until finding a weak learner that satisfies this threshold.
The positive quantity is hyperparameter that typically takes values 0.08 for 2-class problem and 0.16 for 3-class problem, and so on.
Although this step increases compute, it yields better weak learners and improves conver- gence overall.
Permuting the predicted class label assignments:
We hamness the potential of permuting the class assignments by exploring K! different mappings of predictions to classes using the LarmutedlabelMaapiin function in steps 11-14. This process helps us identify the mapping that minimizes the training error to the greatest extent.
By considering multiple permutations of predictions across the label space, as outlined in Steps 11-14 of Algorithm 3, we obtain a hashmap p from the Permutedl.bbelMappings function. This hashmap maps the predictions ÿ to the permuted label space. Selecting the mapping, p² that results in the lowest training error effectively diminishes the cumulative training error during boosting iterations and proves to be an effective strategy for generating strong weak learners. This technique is particularly advantageous in scenarios involving more than two classes.
17

---

Algorithm 3 Summary Boosting
1: Input: X, all training data; all training label; T: maximum number of rounds; s: size of the sampled subset.
2: h, P. &- empty array of size T. P h holds the round-wise hypotheses, P are the corresponding
label mappings, gathers the weighted train errors, and a are coefficients of the hypotheses.
3: N < len(X)
4: C <- set of target classes
5: K len(c)
6: < new array of size N filled with x'
D. is the weighted data distribution
7: for = to T do
8:
(X., y.) +- Cluster-sample examples from training distribution w.
9:
h[r] + Summary (Xa, Us)
h[]] is the weak learner in the current round > ý refers to predictions on training set E[p] will have rate of the corresponding label mapping for p in PermuteddLabelMapppnng do
5|1 < [x(]
11:
empty hashmap
12:
Clp) EN (10 end for
EE x|
14:
pz + arg min, Slp]
16:
if Elp"] > 1 - K OR AllSame(i) then Goto Step 8.
else
18:
P[r] - p": c(r] + Elp*]
end if
20:
if <[r] == 0 then
Break
22:
end if
a[r] + log 1- log(K - 1)
23:
24:
for w xi] x cxple[e(1(P(x)]MM(((X0) vl1))
to N do
25:
26:
end for < Normalize
27:
28:
end for Return a,
29:
Sanity checks: Finally. to ensure robustness of the weak learner when faced with skewed datasets, we have implemented a policy that disallows naive all-ones classifier. The condition calling AllSame in Step 15 of Algorithm performs this check.
A.9 Text Templates
In the ablation study which involves comparing the descriptions created manually by LLM, as illustrated in Figure 3 (right), we transform the descriptive attributes into the textual format by applying pre-defined template. In Table we provide examples of these templates for selected datasets.
A.10 Complexity Analysis
We provide the time complexity analysis comparing our boosting procedure to finetuning the LLM.
For finetuning, the complexity is O(TNf), where f is runtime of the LLM, T is number of epochs, N is the number of data points.
For summary boosting, the complexity is O(TRf), where f is runtime of the LLM, T is number of boosting rounds and Rt is the number of resampling per round.
Concretely, for a dataset with 175 examples, finetuning takes 20 epochs 175 examples x 7000 passes through the LLM. 2 stands for both forward and backward passes through the model.
18

---

For the same dataset boosting requires 50 rounds x 25 resampling on average = 1250 passes through the LLM.
Thus, we believe the complexity of our algorithm is at least comparable to, if not better than, that of finetuning (without considering the cost of the actual API calls).
A.11 Estimating the cost of API calls
While our method is applicable any large language model, primarily conducted experiments using GPT-3, Each API call to GPT-3 incurs a specific dollar cost.
After analyzing the running time complexity of summary boosting, which is QTTRf), we can provide a rough estimation of the cost associated with training classifier on any given dataset.
To begin, when making a call to summarize examples, the prompt is filled up to the maximum context length, which is 2048 tokens for the query prompt and completion. We'll Il refer these summary tokens as S. 2048.
Additionally, if N represents the size of the dataset and we allocate (50 10)
Now, to obtain a weak learner at boosting round r, we may need resample up to R. candidate summaries. Furthermore, we calculate the training error for each candidate summary to determine if performs better than random guessing. Once the desired weak learner found, we compute the validation error for that round only once. Therefore, each round requires querying R x (S, + 0.5. x P.) + 0.1N x P. tokens.
Considering that the maximum number of rounds is denoted as T, the total number of tokens exchanged would be T x [R (S, 0.5.V P.) + 0.1N x P.].
For instance, let's consider a dataset with 175 examples. In this case, the cost would be 30 rounds x [20 resampling (2048 summary tokens + (0.5 x 175 training examples) x 210 prediction tokens) + (0.1 175 validation examples) x 210 prediction tokens] = 12364050 tokens, which approximately costs $25 for Curie at rate of $0.002/1 tokens.
A.12 Can ChatGPT function weak learner?
One would expect that it is more advantageous to try newer LLMs such as ChatGPT that produce increasingly more human-like text and are far more sample-efficient, i.e. can summarize more examples since they come with larger context length. To investigate this, we conduct experiments by feeding ChatGPT with the same tabular data descriptions and using identical prompts to create weak learners. The results are presented in Table 6.
Surprisingly ChatGPT outperforms Curie in classifying datasets with more numerical features, such as wine, wholesale-cus cea.leeccsttterss, and iris. This observation suggests that LLMs are becoming more adept at quantitative reasoning from finetuning with more data. However, the reinforcement learning from human feedback (RLHF) [Ouyang al., 2022] poses limitation as it still ensures the generated text does not deviate too much from its prior. The generated text distribution adheres closely to the behavior programmed into the LLM induced by optimizing with such reward model. Consequently it becomes challenging to bias the LLM with adversarial examples that might occasionally emerge in the training set.
For example, ChatGPT does not mostly generalize well on datasets with medical information such as verterbra- column, breast-cances caesarian and there can be examples contrary to common medical beliefs. In these cases, the RLHF more restrictive due to its conformity to human preferences and does not neutrally summarize examples at hand from classification standpoint. However, boosting imposes a significantly higher penalty on examples that the model fails to classify correctly, causing ChatGPT to not decrease training error after a few epochs. While these models exhibit promise terms of higher-order problem-solving skills, their capabilities can also be limited by their alignment with human preferences.
ceneeceneerrr where
19

---

A.13 Others LLM providers: Claude-2
Summarizing documents and text-based reasoning are standard use cases of LLM, so in principle we expect Summary Boosting to work with other prompt- based LLMs. To validate this claim, we tested our method on other LLMs different from GPT, specifically Claude-2. The results for four datasets are given in Table 7. We see that the performance of Claude-2 is slightly worse than that of GPT-3 Curie (but better than ChatGPT). This experiment demonstrates that our method still works outside GPT-3, and is able to produce weak learners for different base LLMss. Though the scope of the experiments limited, it shows preliminary evidence that the method is robust across models. On the other hand, 2laude-2 released by Anthropic is accessed through commercial API. In section A. we discuss the limitations of using commercial APIs for summary boosting.
A.14 Implications using commercial API for LLM experiments
Our experiments are done using OpenAl's API access. We chose to use APIS instead of hosting the LLMs because it lets us test the capabilities of the models at a reasonable efficiency as boosting requires many sequential calls to the language model and we need to do it on many datasets in parallel. At the current pace of innovation in LLMs, we are cautiously optimistic that there will be software solutions (e.g., for efficient inference) that will change the circumstances. In this section, we discuss the limitations and effects of using LLMs for ML research general.
Limited Customization: Commercial APIs may not allow fine-tuning the model according to our specific needs. We might not have control over the architecture, hyperparameters, training data used for the model, which can be crucial for certain research tasks. The backend models offered by these vendors are also continuously updated either through new training methods more data. For MI. research in general, it will be good practice to mention the version of the model for replicating the experiments.
Lack of Transparency: The inner workings and architecture details of commercial APIs are often proprictary and not fully disclosed. This lack of transparency can make it challenging to understand model behavior, biases, and limitations. Specifically, it deprives us of the opportunity study the training process itself, which can be valuable for understanding how these models learn and for improving our own research methodologies. Commercial APIs may inherit biases present in the training data, potentially leading to biased unfair responses.
Data Privacy and Security: Using a commercial API involves sending the data to an external server, which might raise concerns about data privacy and security, especially if we are dealing with sensitive or confidential information. There might also be regulatory and legal considerations when passing sensitive data to these API.
20

---

Table 3:
Prompt design: Prompt parameter settings for every dataset.
Dataset
Prompting hyperparameters
metadoue This dacase cuntains information about conaarim secion results of 80 pregmant women with the must importan characterisiis of delivery problems in the medical field The goal is to prodict whether woman will undergo nommal caesarian delivery.
capsarian
clarses: [normal, caesariam
swamary direcme: Thdr
inference dizecmie: Ulenc this weman's 's delivery mode likely be tnormal or cuesarian):
merndar This is the inis datheel, perhaps the best known datahiee to be found in the pultern recogmitio literature Fisher's is classic in the field and is reforenced froquently this day. (Sea Duda & Hart, for example) The data class refers a type ins plant. from the other 2: the latter are NOT linearly sepurable from ench other. Prodicted attribute- cluss tris plant- seteso. versicolor, virginic
set 3 classes 50 instances each,
linearly separable
Lis
clarres: (eeisss, versicolor, sirginical
swamary divecite:
inforence directive: Bused on the above information, prodict this flower will be classified as seiosa, versicolor, virginica
sortudota: The data consis evaliations tesching perfonmance over three regular semesters two summer sensester of 1S1 teaching assiscant (TA) assignments at the Statisties Depirtment of the Univenityy of Wiscomin-Maa Madison The scores were divided into nughly equal-sized eotegoriis ("kow,, "mediam", and "high") to foom the class varinile. clarres: [low, medium, high)
swamory divectiv:: Thdr
inforence divectioe: Predict whather this class will acore low or medium or high:
meruudau:: This is the glass dataset from USA Forensic Scienee Service: 6 rypes of glass; defined in teams of their oxide content (i. Na, Fe. K, dte). The study of claviiication of types of glass motivated by criminologieal investigtionn the crime, classes: [building windows_ propppsssottppeeescctt. building, Bolddingwwiddwwss windowssnon, foat dloce_pooessed, vehicle_ windows__oow _eatt_oooossssse___ee processed. contiiners, rabloware, heallamps]
glass left
conccily idzutifted!
glass
susumary dievctioe: Tiddr
inforence divectioe: There passible type of bailding_ windows Ber___Booo_____e_eee hailding windows non, floal processed, vehicle windows flost procossed, containers, tableware, headlamps, Prodict which one will this sample be:
This is one of three domeins provided by the Onoology lnstitute that has repeatedly agpeared in the machine learning literature. This data set includes 201 insances of one class and 85 instances another class. The instances are desesibed by amiiuunes recurrence or celape becast cancer after their fisst time.
metaddou::
of which linear and
nominal. It contaiss infonmation ahout that had
breast sanoer
clarses: |recurrence, non relapse)
swamary dicvciive Bised on the ahove exampies, figure our under what conditions will woman have recurennce or relasee breast cancer?
inference divectiv:: Prodict whether women will hove recurnencc or non--ellapee:
the book Visualicing Data published by Hobart Press (books is bobbat..com). This data describes indicators for a pastivehueqaiive enviroument based on ozone, rudiation and temperature, closves: ipusitive, negasive)
meradaat:: This visualizing envirommental dataset, of 22 data
visualiizing environmental
directiv:: Tharr
inforence directive: There are clear signs environment being (postiive or negutive):
meradata: This chiamydia dutaset
"Asalyzing Categurical Data" by Jelfrey Simonuff, Springer Veriag, New York, 2003, contains resolts of individull that lested for chlamydia.
the
unalcundata chianydia
clarms [posifive, negative]
directive: Tharr
inference directine: Predict if person test positive or negasive for chlamydia:
Updand Sept 21, 1908 by C.Rlae. It contains results of chemical aralysis same region in huly but derived from three differest culivars. The anllysis determined the quanitties of 13 constiuenis found in each of the three types of wines.
avrtador: This the Wine recognition
wines grown
wine
elorses: 2, 3]
swmmary directme:: Using exumples and bused on cuntents consiitenns, summarize what distinguishes of type or or 3?
inforence divectiow Hence this wine will he classified as -slype
metuudau:: Data taken from the Blood Transfasion Service Center in Hain-Chu City Taiwan- - this is classifiction problem. The goal to predict whether given individual will consent or avoid donating blood. olarnes: [oonsent, swamary dievctive. Ti,rr inference directise: Therefore, this individual is likely to fnveidiconseni::
Mood--meshhriin-cceeee
metadouu: This is the Somerville Huppiness Survey Data Set. has ratings collected from survey of Somerville resideats. Frozh the responses of a resudent, the goal to prodiet whether they feel happy or unhappy about the place. clazses: junhappy, hapgy) swamary dievctiie Blaed the Somerville happiness survey, how cas we predict whether resideat happy or unhappy with their place!
somerville-lappiiosss htppppppsppssssrpyeeeoe souveey
inference dizectiie: So this resident (happy or unhappy':
meradous: This is the Stadog (Vehicle Silhouettes) Duta Set. The is to classify a given silhouctic as one of four types of vehicle- bus, sub, upel viewed from clazzes: (bus, stab, opel. van] swamary dicective: Using these examples, summarize ize how can we differentiste a silhouete is that a bus, sanb, opel
using of features extracted from the silhometle. The vehicle many different angles.
vebicle
inference directin:: Out szab, bus, van opel. this vehicle likely
meraddas This disaset is heart disease database similar a database alrewdy present in repository (Heart Disease databuses) but in a slightly different furm. has data on individuals having and not having beart discase. clazzes: [present, absent]
statlog heart
dirrctir: Differemiase people with heart diserse present from absent. inference directire: this case, heurt disease is likely to be (pessen/absenti
21

---

metadatu: This dataset comtann values for biomechumical features used classity oethopuedic patients 3 classes (normal. or spendilolythessst 2 classes (nomaal abnoemal). Biomedical medical residence period in the Group of Applied Research in OOthoquofics (GARO) of the Centr MAkedio-Chirngiie Chiruiin- de RAStadattzion des Massues, Lyon.
Dt: Heurque Mota dunigg
weerbbn
France. The patients).
clacify putients belonging
categories: Nornal patients) Abnormal (210
classes: [ohsormal, normall
dirvetive: Resed on the above eximples, summarize how will you distinguish patiens that have ncemal vx. abnomnal vertchral
inference directive: Therefore, individunl's 's vertebral column likely to be (shnormal or normal);
metadata: data contains protein localization Reforence: Knowledge Base Predicting Protein Loculization in Bikaryalic Colls', Keata Nakai & Minon Kanclisa, Genomics 14:8979911, 1992.
cooli
classes: [I. 21
summary directive: Using examples, how sell apart cells with protein localized
2
inferecee divettie:: Hence protein localization will be at
the Univensity of Chicago's Billings Hospital on the survical pasients who had undergone surgery for breast cancer. clazres: [survived, died]
mettddat:: The dataset contains cases study that conducted between 1958
Iaherman-arvical
directive: Bised these examples, figure what commenaiiies predictive patieuts surviving years and
inferesce divective So, 5 years down the line, this person (survivedidied)
This ditaset is originally from National Instituee of Dishetes and Digestive and Kidnry Disenses, The objjective is to predict based on diagnostic measurements whether patient has highllw risk of developing diabetes. clarres: [low, high) cummary
metadato:
diabetes
Besed these ccamples. distinguish patients having inference directive Bused on the reasoning, this patient likely to have (ow/high::
high disbetes.
metadato: This is visualizing-hmman humnser daiase contains data sets from book Visualizing Data published by Hoburt Press Shokkk@hhari.com It contains examples of hamster that are ill and healthy. clares: (ill, healthy]
visualizing hamster
cummary directive: Using eumples, identify predictive indicttors of ill healthy humsiers.
inference divectio:: Predict whether hamster will or healthy:
suttdott:: The set to cliests of wholesale distribator. includes the anmaal sponding monetury (tn (s) on diverse product categosies. This data gives incomaation about spending pattems and region of operations of Ketail and Horeca (HonetRestannndfiil custemers of the whv viviesule distrilutoe
wheleealle-csooee
clanses: lectall, horeca)
summary directive: Using those examples, summarize how can we Teremiiat Retail enstmmes and Horeca customers. inference divecise: Therefeer, which one of Retail or Horen this enstomer likely to be:
22

---

Table 4: Continuous variable transformations applied
example from the
wewleeale-custommen dataset. The raw tabular record is as follows: spending on fresh products: 6353.0, spending milk products: 8808.0, spending on grocery products: 7684.0, spending on frozen products: 2405.0, spending on detergents and paper products: 3516.0, spending on delicatessen products: 7844.0 and customer's region: Outside Lisbon and Porto.
Method
Data Represcntation
Examople as text
spendinn on frezhh products
spending on milk produces: very high
This customer spends low amounts on fresh products, very high amounts on milk produces, ligh amounss on groeery penducts. frosen products, detergents and paper products, and very high amounts on delicasessen products. They are located outside Lishon and Porto.
bina guantifiers
-pendinn geocery products: high
Ivery Jow, low, high,
spendingg
products: high
very high)
spending on detergents and paper products: high
-pending on dellcunessen products: very high customer's iss region: Outide Lisbon and Porto
spending on fresh products: ; medium
spending milk products: very high
This from outside Lishon and Porto spends medium fresh products, very high milk producss, high groeery produucts, high on frozen products, high on detergents and paper products, and very ligh on delicatesien producss.
bins quantifiers
spending on gracery products: high
(very low, low, medium.
spending on frozen products: high
high, very high!
spending on detergents and paper producrs high spendingg delicutessen products: very ligh
customer's is region Outside Lisbon
spending on freth products low
qpending on milk products: very high
7 bina quantifien
(extremely low, very low,
spending on geocery producte: : high
Tis customer situated outside Lisbon and Porto spends low on fresh products, very high on milk produces, high giveery products, high Frozen penducts, very high datergents and paper products, extremely high delicitessen products.
lon, medium, very hight extrossely high)
spending frozen products: high
qponding detergents and papez producrs: very high
spending on deltcatessen products: extremely high customer's it oegion : Ounside Liboo and Porto
-pendingg on fresh products: : low spendinn on milk products: estresncly high
9 bina qf
This customer spends low amounes on fresh products, extrenelly grocory
[lowest exteemely very low, low, medium,
yponding on gencery producte:
high
milk products, high productss, frozen products, detergents and paper peoducts, and highest amouns on deliculessen products. They are located ousiie I Lisbon and Porto.
spendinng on frocm products: high
high, very high.
spending on detergents and paper products: very high
extronelyy high, highest)
spending delicutessen products: highest it orgion: Outside Lisbon and Porto
spending fresh products: falls the fins bins valces.
-pending on milk products: falls in the second out of ten bass of values
-pending gancery producte: falls the first of valuss
This customer speads relotively fiesh, grocary, frozen
10 bina
spending on freze products: falls the first out of ten hirs of values.
and detergems/pager products, and more on milk and
delicatesson products. They are baved omsiid I ishon and Parto.
speading detergentt and paper products. falls in
out
bins of values
spending ow delieaiessen producte: fulls in the szcand out of hins of values
customer's 't orgion: Ounside Lishon
spending fresh products: : falls the forty first pescentile
spendlnng on milk products: falls in the eighty second
percenile
spending gencery productz: falls the sixty-fiftt
This customer his an annial spending 41s percenile fresh products, 8lnd percenile on milk products, pencentile grocery products, 63rd pencentlle on frozen producss, 72nl persenille on detergents and paper products, and 98th penseille delicatessen producss, and located ouiside of Lisbon and Porto.
percemile
Parcentill
ow froze products: falls the sixty-thind
percentle
spending detergents and paper products: falls in seventy- second percestile
spending on dilleunessnn products: falls in the ninaty eighth percentle
i' orgion: Outside Lishon
-pennding fresh products: within dev below mean value
spending on milk products: is within one sid dee above the mean value
spending grocery products: within std dev below mean value
The eustonser las anmall speoding on fresh products, milk products, grocery products,
products, detergents and puper productss, and delicstessen producss withis standard deviation of the mean, excups for delisatessen products which is two standund deviations above the mean. The eustomer is locsted ounside Lisbon and Porto.
Standard deviaticn
-ppending on frozen products: within one dev below the mean vulue
spendling dittergents and paper products is within one std dey above mean value spending on dellcatessen products: is two std dev above the mean value
region: Outside Lisbon Porto
spending fresh products between the first quartile and mediam
spending on milk products is mare than the third quartile value
spending grocery producte: between mediam third quarile
This customer spends more than the thinl quatille value milk, delicatessen and detergents paper products. customer's spending un fresh, grocery, and frozen products falls between the median and third quartle values, while the customeria located outside Lisbon and Porto.
Quartiles
spending on frozen products: between median and thurd quartile values
-peendnng detergents and paper products: is between mediam third quartile values
-spending on dielcunessen products: is more than the third quartlle value
customer's region: Outside Lisbon and Portg

---

Table 5: Templatized descriptions: Templates used to format examples for the ablation study between LLM-created data descriptions vs. template descriptions
Dataset
Deseriptive attribute values
Template
Ivery young, young. middle aged, old. very second, third, fourth,
delivery,
This (age) woman her (delivery numberr delivery and _amee) She has a Pbloodd_pvessure blood presoure and [heart problemi heart problems. eet Based on these attributes, this wuman is likely deliver /defivery_ mode)
deiiverv_tiee: [timely, peemature, latecomer] plood_pressne: presoue: [low, sormal, hight
cuearraa
heart, proble: [has, doesn't havel
_mode: [noemal, caesarian]
sepai, fengtt, petul_ Jength: [very short, sbort. medium length, long. very long)
This iris flower has isrpol Jengsh) and (sepul widlh) sepuls, also has (nvial length and (peial widht) petal. usa Hence this flower [flower_ppe)]
ini
aspal with potal widtt (very narrow, narrow. medium widh, wids, widel
flwerr jppe [setoss, versicolor, virginical
This patient las (peliic incidence) pelvic incidence, (pelvic pelvic and flambor Jondoais angle) lumbar londosis angle. (sacral stopej sactal stope, fpelvic. radiar) pelvic ratios and (prate of spondyleliettenn grade of spenlylolisheeis Add As patient's vettbxilooolua likely (resufi]
inciieiineeeece,, pelvic jilt, lumber Jondosis pagle, sacral stope, radics grade_of ooooddlplllistee (very low, low, medium, high. very high) resull: loormal. absormal)
vertebral-oouum
age: Ivery young. young, middle aged, old, very srx [male, female]
chost_pain_ pain (eeymptomatic, comaaginal pain, atypical angine, eypocal anginal
This individusl is alaa [age] [sex] with (ohest pype chest pain.
[tp] resting blood pressure and (chofentevll serum cholesterel.
ly, cholestenf, depeeziion, heart raie, mwn sasjor_sesseeee pessade (very low, low, mediam,, high. very high]
Their fasting bload sugat [furting Mood sugarf >120 mightl, feletnourdinggphi resalts] reaalts and a (hoor sate] maximumm They fexercise induved ongine) exntiaa-innued anginn, and have [st depremion] ST depression inducad extreise relative to rest. (stape it__stt_ s..emmattt slope,
stallog heart
faxsing blood sugar: [high, low]
electocudiogoaphin wttpoooooooghheressiaaee hepertrophy, noemal, baving ST-T wave shuomalitt) slope at sygmont: [fut. upsloping, downskpingg exercive inducedd ongine: (has, do not have] defect oppe:
[having left veuticolar
Their peak exaccise segment they have from mujor vesselsj number of mujor vessels, The defoct type is (defect tpp/ ven Hence heset discase is likely to be. (presence of Jeart disnove.
revesible, fised presence of heart diseave [present, shscnt]
age at Jiw__rl ot_op: (very young, young.
middle aged, old, very cid]
This puient was fage, at time of ap) at the time of operation in lyear of on). They had from pos, axillary noder number of positive asillary nodes detected. hie Therefore years down line, the puieent fmurival stanis)
ywar of op: [1964, 1962, 1965, 1959, 1958, 1960, 1966, 1961, 1967, 1963, 1909, 1968)
aubemman-survaul
mum_jor pow_por__tttlry__eee _utffary___dez: (very low, medium. high very high]
survival natls (survivod, died]
Table 6: Comparing test error rate of Summary Boosting backended by Curie and ChatGPT on all datasets (4). Refer to caption of Table 1 for the notations used.
Dataset
Data Type
Size
Curie
ChatGPT
caesarian [cae] (42901)
1c4d
80
0.300 0.04
0.406++
iris (61)
4c0d
150
0.1931 0.00
0.083: 0.01
tae (48)
1c4d
151
0.454:
0.03
0.443:
glass (41)
9c0d
214
0.37010..0
0.49211..2
breast-cancer [hc] (13)
7c5d
277
0.288+ 0.02
0.360+
visualizing-enviroomeeen [ve] (678)
3c0d
111
0.268
0.333 0.04
analcataata-chlamydia [ac] (535)
2c2d
100
0.170+
0.300=
wine (43571)
13c0d
178
0.320
0.01
0,250 0.01
comerville-happiness [btc] (1464) vehicle (54)
Blood-transfusionn
4c0d
748
0.240 0.04
0.433
[shs] [Koczkodaj, 2018]
0c7d
143
0.350 0.02
0.430+ 0.02
18c0d
846
0.410=
0.350+ 0.16
statlog-heart [stath] [Dua and Graff, 2017]
6c7d
270
0.430
0.
0.370
verterbra-column [ve] (1524)
6c0d
310
0.2622
0.66940.03
ccoli (1011)
7c0d
336
0.270+0.03
0.193
bobberman-suriivee [hs] (43)
3c0d
306
0.250+
0.415
diabetes [dia] (37)
8c0d
768
0.344+
0.297+
visualizing-haraan hamster [hams] (708)
5c0d
73
0.400+ 0.20
wholesale- customers [we] (1511)
cholesal-custommers
6c1d
440
0.3301 0.00
0.199
24

---

Table 7: Test error rate of Summary Boosting applied to Claude-2 vs. Curie (L). The experiments with Claude-2 were run using the same prompts as Curie from table 3.
Dataset
Data Type
Size
Claude-2
Curie
caesarian [cae] (42901)
1c4d
0.312 0.08
0.300 0.04
iris (61)
4c0d
150
0.200 0.02
0.193+ 0.03
breast-cancer [bc] (13)
7c5d
277
0.385 0.05
0.288+ 0.02
vertebra-column [ve] (1524)
6c0d
310
0.310: 0.03
0.2621
25


Reviews:
[{"review_id": "Wwfkn2CjHT", "reviewer": "Reviewer_inEG", "summary": "This paper proposes a novel way to use large language models (LLMs) as weak learners in boosting frameworks for tabular data. The core idea is called LLM Summary Boosting, a novel method that prompts large language models (LLMs) to create weak learners for usages within a boosting framework to make predictions on tabular data. The authors conducted thorough experiments on a variety of benchmark datasets. Compared to conventional tree-based boosting and LLM-based finetuning, this approach demonstrates superior performance in specific scenarios and remains effective with limited examples. To the best of my knowledge, the contribution of this paper is novel despite its incremental nature of contribution. I therefore recommend weak accept and am willing to hear what the authors think.", "questions": "N/A", "limitations": "The authors described the limitations of the proposed methods in section 6. The authors also provide a detailed failure modes documentation and I think it is very valuable.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "This paper is well-motivated and well-written. I also find the paper very easy to follow along with. Ideas of prompting LLMs to perform data manipulation/wrangling task aren’t new but this paper focuses on the specific usages of prompting for use with boosting scheme on tabular data. I think the contribution is clear here with the experiments.", "weaknesses": "While I find the arguments and experiments quite comprehensive, I am personally on the fence about the title, but I do not fully object to it. However, in the current form, the title seems to be a little bit overclaiming. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive views of our paper! We are glad that you found it well-written and easy to follow along with. Please find our responses to your review as follows.\n\n**Weaknesses**\n\n> 1. While I find the arguments and experiments quite comprehensive, I am personally on the fence about the title, but I do not fully object to it. However, in the current form, the title seems to be a little bit overclaiming.\n\nThank you for your suggestion! We have justified the title by showing that LLMs can be used for boosting, i.e. serve as weak learners.\n\nBy definition, a weak learner is a classifier that achieves better than random guessing performance under the distribution of interest. We believe that our tabular data classifier created through prompts was able to demonstrate this property."}, {"review_id": "oKlnCEaalP", "reviewer": "Reviewer_LBci", "summary": "This paper demonstrates that prompt-based LLMs can be used as weak learners, with applications on boosting algorithms for tabular data. By providing text descriptions of tabular data samples, the authors show that LLMs can produce a summary of the samples and use it as a template for classification that can be leveraged as weak learners for the task at hand. The proposed approach outperforms zero- and few-shot learning and, occasionally, even SOTA algorithms.", "questions": "see Weakness above", "limitations": "The main concern of this reviewer is the practical usage of Summary & Summary Boost. Without clarifying the issues and questions raised under \"Weaknesses,\" the proposed approach has very limited applicability under still-unclear circustances.", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper introduces a novel approach to using LLMs as weak learners that can be leveraged on tabular tasks via boosting. The proposed approach appears to be novel, but in order to properly judge the paper's significance, the authors should address the comments below on the quality and clarity of presentation. Compared to zero and few- shot approaches, the proposed Summary & Summary Boosting is clearly superior; however, when compared with KNN, XGBoost, and (especially) TabPFN, the practical applicability seems quite limited and not well understood. ", "weaknesses": "The paper can be significantly improved by providing a crisper, more intuitive description of the proposed approached (both Summary & Summary Boosting), and by offering an in-depth discussion of the practical impact of the empirical results in Section 4.\n\nWith respect to the presentation: Figure 2 appears to present the \"Summary\" algorithm, without Boosting. Even so, it is unclear (i) what is the output of the stratified cluster sampling (I expect it to be a subset of the input to this step), (ii) what does \"Select BEST summary\" mean - best based on what? only one summary is selected? probably not, because one would expect it to be not so much \"the best\" but rather \"the MOST USEFUL\" to classify a new, unlabeled example. Furthermore, lines 137-138 seem to imply that just one summary is chosen (per class?). Adding all these key details to Figure 2 is critical for removing the current ambiguities.\n\nThe paper should add a new figure, similar to Fig 2, that fully illustrates in detail the Summary Boost algorithm. \n\nLast but not least, the authors should also clarify & discuss the results in Tables 1 & 2, for both of which they should BOLD the best result (to improve readability). While it is obvious that both Summary & Summary Boosting outperform zero- and few- shot learners (Table 1), the picture is a lot more confusing when comparing Summary Boosting with the four strong \"baselines\" in Table 2.  First of all, the novel approach obtains the best results on only 3 or 4 of the 16 datasets. Second, often times one of the baselines outperforms the novel approach by almost an order of magnitude (eg, wc & wine). Last but not least, the two conclusions of the paper (that SB does best on datasets with few examples and worst on those with continuous attributes) are not necessarily correct: even though the best performances of SB are on datasets with 1/3/3/5 continuous attributes and 73-306 examples, it is unsafe to generalize from here. For example, iris has 4 continuous attributes and 150 examples, but SB's performance on it is abysmal (0.193 vs 0.027), even though this dataset's properties are similar in nature hams (5 & 73) or tae (1 & 151), where it does comparable/better than the competition. The authors should do an in-depth study about the suitability of the approach to various types of domains; if they need extra space, they could shrink/send-t0\\o-appendices Section 5.    \n\nOTHER COMMENTS:\n- line 87:    you should either explain what \"the special token\" does, or omit to mention it\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable suggestions for our manuscript.\n\n> 1. The paper can be significantly improved by providing a more intuitive description of the proposed approachs and by discussing the practical impact of the results in Section 4.\n\nThank you for your comments! We have brought out the intuiton behind our proposed method in multiple places leading upto Section 3.1. Lines 32-37 introduces the idea, lines 77-80 and 95-100 explain it further.\n\nA practical applicability of our work will be the fact it improves the ability of LLMs to reason on tabular data over few-shot and zero-shot. We have shown that “summary” as an intermediate step makes reasoning easier and infact can be used inside boosting models for small tabular data upto 300 data points without many continuous features.\n\n> 2. With respect to the presentation: Figure 2 appears to present the \"Summary\" algorithm, without Boosting. Even so, it is unclear (i) what is the output of the stratified cluster sampling\n\nStratified cluster sampling yields a representative subset of examples that will be injected into the prompt for summarization. Since the LLM’s context length is limited, only a few examples selected by this sampling process can be fed as input. This has been mentioned in lines 142-154 and depicted in Figure 2.\n\n> 3. What does \"Select BEST summary\" mean - best based on what? only one summary is selected? probably not, because one would expect it to be not so much \"the best\" but rather \"the MOST USEFUL\" to classify a new, unlabeled example. \n\nWe use validation set to select the best summary. The best summary is the one that achieves the least validation error rate.  We are not sure what “most useful” could mean in this case without a more concrete definition.\n\n> 4. Furthermore, lines 137-138 seem to imply that just one summary is chosen (per class?). \n\nWe would like to clarify that there is only one summary for each weak learner.\n\nFor the  \"Summary\" method compared in Section 4 - Experiments, we generate a fixed number (25) of summaries and pick the one with the smallest validation error rate. This is mentioned in lines 137-139.\n\nHowever to obtain a weak learner that is used inside \"Summary Boosting\", the procedure is slightly different. We sample summaries until we find the first summary that does better than random guessing on the training distribution in that round. This summary will be our weak learner. We refer to the lines 165-167.\n\nFurther details have been explained in Appendix A.2.\n\n> 5. Adding all these details to Figure 2 is critical for removing the current ambiguities. \n\nWe thank the reviewer for the suggestion. Figure with the boosting algorithm included will take up a lot of space. So, we found it better to describe the missing details in text and through Algorithm 2. In the revision we can refer to the text in the caption if that will be more helpful.\n\n\n> 6. The authors should also clarify & discuss the results in Tables 1 & 2, for both of which they should BOLD the best result (to improve readability). While both Summary & Summary Boosting outperform zero- and few- shot learners (Table 1), it is not true of the \"baselines\" in Table 2. \n\nWe have posted a PDF of the table with the best-performing results highlighted in bold. We refer to our overall response note to all reviewers.\n\n> 7. The two conclusions of the paper (that SB does best on datasets with few examples and worst on those with continuous attributes) are not necessarily correct: iris has 4 continuous attributes and 150 examples, but SB's performance on it is abysmal (0.193 vs 0.027), even though this dataset's properties are similar in nature hams (5 & 73) or tae (1 & 151), where it does comparable/better than the competition. \n\nOur observations are based on the majority of datasets and not any single one. It is true that the LLM does not perform well when the dataset has more continuous features as supported by its error rate on *iris, glass, wine*, and *wholesale-customer* datasets. On the other hand, our approach works best when the dataset is small, as evident in *caesarian, TA evaluations, somerville-happiness-survey, haberman-survival*, and *visualizing-hamster* datasets. This is reasonable because LLM leverages prior knowledge from pretraining, which becomes less relevant and less competitive as the dataset size increases.\n\nHowever, it's worth noting that the *visualizing-hamster* dataset is very small (< 80 points), and in this case, it appears that the XGboost overfitting makes our method seem like the best performer. It is our hypothesis that prompting is likely to not overfit on any dataset since it remains a few-shot learner.\n\nWe wish to highlight here that our method does not always show improvements over the state of the art due to some current limitations in LLM with respect to context size that prevent it from ingesting a large number of examples. This may be addressed in newer LLMs - for instance, GPT-4 has 32k context length. Additionally, even converting numerical values to discrete categories does not help address continuous features, which is a known problem in LLM that they are bad at quantitative reasoning. These issues may be resolved as LLMs get better. Also there may be ways to combine trees and LLM in a single boosting algorithm.\n\n> 8. line 87: you should either explain what \"the special token\" does, or omit to mention it\n\nBy \"special token\" we mean the [MASK] token in prompt which is commonly used for Masked Language Modeling in BERT, i.e. predicting a masked token in a sequence. We will include this change in the revision.\n\n**Limitations:**\n\n> 1. The main concern of this reviewer is the practical usage of Summary & Summary Boost. Without clarifying the issues and questions raised under \"Weaknesses,\" the proposed approach has very limited applicability under still-unclear circustances.\n\nWe refer to our explanations addressed in the previous comments. We will emphasize these aspects more in the revised paper."}, {"review_id": "dwWVnv5ysO", "reviewer": "Reviewer_NJzp", "summary": "The paper investigates the use of large language models (LLMs) as weak learners in a boosting algorithm applied to tabular data. By providing text descriptions of tabular data samples, LLMs can generate a summary that acts as a template for classification, effectively serving as a weak learner. The authors incorporate these LLM-generated weak learners into a boosting approach and demonstrate their performance superiority over few-shot learning and fine-tuning procedures, especially for tasks with limited data points.The results showcase the potential of prompt-based LLMs not only as standalone learners but also as components of larger machine learning pipelines. ", "questions": "1. The proposed method to split continuous values in tabular data into discrete attributes and generate corresponding summaries. I wonder whether using this approach to fine-tune the LLM could yield promising results. This way, it can effectively align the tabular data format with the language model while enabling the model to learn latent knowledge embedded in the data.\n2. How can the determination of the number of discrete categories (e.g., low, medium, high) for continuous numerical values in a table be made? From what I understand, your approach involves experimenting to determine the optimal number of categories.", "limitations": "Limitations are discussed in Section 6 of this paper.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The experiments are conducted on a large number of datasets and illustrate the effectiveness of the method proposed on tabular data. \n2. Clear writing and visualization.\n3. The examples in the appendix are enjoyable to read. The prompt templates in the appendix are worth learning.  ", "weaknesses": "1. The integration of multiple weak learners using ensemble learning methods, each requiring the invocation of LLM, may result in significant resource costs. In Appendix A.11, it is calculated that running a dataset of 175 instances would incur a cost of $25, which seems relatively high.\n2. The presentation of experimental results could be improved by highlighting the relevant information in the tables. Adding bold formatting to the results would make them more prominent and eye-catching.\n3. In Appendix A.12, it is mentioned that ChatGPT performs poorly on some datasets due to the utilization of RLHF. However, no experimental evidence or persuasive argument is provided to support this claim. It is also possible that the issue stems from suboptimal prompts. Also, the Figure 3 shows the results of using GPT-3, where GPT-3 does not consistenty improve upon the smaller model. GPT-3 and ChatGPT have the same pretrained data, which means the performance degradation may be influenced by the pretraining data of the model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad that you found our paper enjoyable to read and our writing lucid to follow. Thank you for your positive feedback of our work! Please find our responses to your review as follows.\n\n**Weaknesses**\n\n> 1. The integration of multiple weak learners using ensemble learning methods, each requiring the invocation of LLM, may result in significant resource costs. In Appendix A.11, it is calculated that running a dataset of 175 instances would incur a cost of $25, which seems relatively high.\n\nThank you for the suggestion! It is true that querying OpenAI for creating the prompts seem to be expensive, however in principle our approach does not need OpenAI API and can utilize open-source models hosted locally which could drastically reduce the cost.\n\nWe also show in Appendix A.10 that our method is infact more compute-efficient compared to finetuning. For a dataset of 175 points, finetuning makes 7000 calls to the LLM, while prompting needs only 1250 passes through the LLM.\n\n> 2. The presentation of experimental results could be improved by highlighting the relevant information in the tables. Adding bold formatting to the results would make them more prominent and eye-catching.\n\nWe have posted a PDF of the table with the best-performing results highlighted in bold. We refer to our overall response note to all reviewers.\n\n> 3. In Appendix A.12, it is mentioned that ChatGPT performs poorly on some datasets due to the utilization of RLHF. However, no experimental evidence or persuasive argument is provided to support this claim. Also, GPT-3 does not consistenty improve upon the smaller model and GPT-3 and ChatGPT have the same pretrained data, which means the performance degradation may be influenced by the pretraining data of the model.\n\nIn Appendix A.12, we show that the performance of ChatGPT is worse than GPT-3 Curie on many datasets, especially medical-related such as patients, diseases, etc. \n\nWe speculate that RLHF is responsible for this behavior since it is one of the main differences between ChatGPT and GPT-3 Curie. Since we do not have information about the pretraining data changes, we believe it is reasonable to infer that RLHF is a major factor. We note that we did not alter any prompts for ChatGPT and our method mainly relies on the LLM generating summary for itself which it uses to perform reasoning.\n\nWe have provided our explanations based on experiments in Table 6. We have observed that it was hard for models with RLHF to change their opinion on scenarios that sound unlikely but plausible. For example in the *somerville-happiness-survey* dataset, a resident rates many amenities as low but is overall *happy* due to other factors. ChatGPT was unable to reconcile this instance with the remaining examples. Further, we have observed that the models with RLHF often would abstain from answering (making predictions) on such examples, making the boosting algorithm less effective.\n\nAgain, we wish to highlight that this is just speculation based on our empirical observations, since we don't have access to the actual models/ pretraining data of OpenAI. This observation is important because if one wishes to use this method it would be better to use the base pretrained model rather than one that has been fine-tuned with RLHF.\n\n**Questions:**\n\n\n> 1. The proposed method to split continuous values in tabular data into discrete attributes and generate corresponding summaries. I wonder whether using this approach to fine-tune the LLM could yield promising results. This way, it can effectively align the tabular data format with the language model while enabling the model to learn latent knowledge embedded in the data.\n\nYes we have compared a similar approach called LIFT [1] in Table 2, which is comparable to XGboost on most datasets. It involves finetuning the LLM directly with plain English sentence form of the tabular records. Without needing to convert continuous values to discrete categories, this method is already competitive on datasets with many continuous features such as glass, wine, iris, vehicle, and wholesale-customers. This suggests that with finetuning, LLMs are able to handle continuous attributes better.\n\n[1] T. Dinh, Y. Zeng, R. Zhang, Z. Lin, S. Rajput, M. Gira, J.-y. Sohn, D. Papailiopoulos, and K. Lee. Lift: Language-interfaced fine-tuning for non-language machine learning tasks. arXiv preprint arXiv:2206.06565, 2022.\n\n> 2. How can the determination of the number of discrete categories (e.g., low, medium, high) for continuous numerical values in a table be made? From what I understand, your approach involves experimenting to determine the optimal number of categories.\n\nYes in Appendix A.6 and Table 4 we provide a detailed description of various techniques used for encoding numerical values into descriptive attributes. Among these strategies, we found that binning with quantifiers works best, and further the the number of bins was determined through hyperparameter search. These results are summarized in Figure 4 (right top)."}, {"review_id": "jJVWrXM19M", "reviewer": "Reviewer_gzhs", "summary": "This paper explores the concept of weak learners, which are classifiers that achieve slightly better than random performance on any given data distribution. The paper demonstrates the effective utilization of large language models (LLMs) as weak learners. The study focuses on applying a large language model to tabular data using a boosting algorithm. By providing properly sampled text descriptions of tabular data samples according to the target distribution, LLMs can generate a summary or template for classification, serving as a weak learner for the task. The paper incorporates these models into a boosting approach, which, in certain cases, outperforms traditional tree-based boosting methods by leveraging the knowledge within the LLMs. The experimental results indicate that the proposed method outperforms both few-shot learning and complex fine-tuning procedures, particularly when dealing with a limited number of data points. These findings highlight the potential of prompt-based LLMs not only as few-shot learners themselves but also as components of larger machine learning pipelines. Overall, the paper showcases the effectiveness of prompt-based LLMs as weak learners in boosting algorithms for tabular data, offering insights into how they can improve classification performance, particularly in situations with scarce data availability.", "questions": "Please see the weaknesses above.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Strengths:\n---------------\n\n1. The paper successfully brings together the concept of weak learners in boosting algorithms with the advancements in large language models (LLMs), creating a novel approach for utilizing LLMs as weak learners in tabular data classification.\n\n\n2. The paper introduces a unique approach by converting tabular data into text form and using LLMs to generate summaries or prompts. This methodology allows the LLM-generated prompts to serve as effective templates for tabular data classification without the need for retraining or fine-tuning the LLM itself.\n\n\n3. Through comprehensive evaluations, the paper demonstrates that the proposed approach outperforms alternative techniques such as zero-shot and few-shot learning. It also showcases the approach's superiority over traditional tree-based boosting and LLM-based fine-tuning methods, particularly in domains with limited examples. This performance advantage highlights the potential of LLMs as weak learners in boosting frameworks.\n\n\nOverall the paper is well-written and paves a way to utilize LLMs in boosting.", "weaknesses": "Weaknesses:\n------------------\n\n1. The paper focuses specifically on tabular data classification, which may restrict the generalizability of the proposed approach to other types of data or domains. It would be valuable to explore the performance and applicability of LLM-based weak learners in a wider range of datasets and tasks.\n\n\n2. Although LLMs have shown impressive performance in various natural language domains, they still have inherent limitations, such as sensitivity to input phrasing and potential biases in the training data. The paper would benefit from discussing and addressing these limitations to provide a more balanced perspective on the capabilities and potential drawbacks of LLM-based weak learners.\nWhat was the rationale for choosing these datasets? Are there some data settings that are favorable to summary boosting and similarly the settings that are not good for the proposed method? \n\n3. Minor comments/questions,\n\n   a)  Please make fonts bold (or use color) in the tables for the best numbers in each row. \n\n   b) Are the results reproducible? What is the cost of running the experiments?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review of our manuscript! We appreciate your recognition of our work as novel and paving a way to utilize LLMs in boosting.\n\n**Weaknesses:**\n\n> 1. The paper focuses specifically on tabular data classification, which may restrict the generalizability of the proposed approach to other types of data or domains. It would be valuable to explore the performance and applicability of LLM-based weak learners in a wider range of datasets and tasks.\n\nThank you for your suggestions! The primary goal of this paper is to test whether LLMs can be incorporated into larger ML systems like boosting. Since boosting is primarily used for tabular data, we decided to focus on tabular data. Indeed, in principle, the method can be applied to any other data in text form, such as GLUE, SQUAD and we plan to explore this in the future.\n\n> 2. The paper would benefit from discussing the limitations in LLMs such as sensitivity to input phrasing and biases in training data to provide a more balanced perspective on the capabilities and potential drawbacks of LLM-based weak learners. \n\nIndeed, LLM exhibit sensitivity to the presentation of prompts which elicit different behaviors. Although our core methodology involves using LLMs to generate prompts (both summary and inference), it still requires a modest amount of manual engineering. We have elaborately discussed failure modes in the creation of these LLM prompts in Appendix A.1-3. Moreover Table 3, lists all the prompts used in this paper, dataset-wise.\n\nThe reviewer is correct in pointing out that any biases in the dataset will also reflect in the summary produced by the LLM and this is true of ML model in general. \n\nYes our method inherits the problem of bias in LLM that comes from pretraining data. Specifically, it can affect the model’s ability to objectively summarize the examples or make predictions that are inconsistent with the biased pre-training data. We believe with better and debiased LLMs these issues can be alleviated. We will update the paper to emphasize more on this aspect.\n\n> 3. What was the rationale for choosing these datasets? Are there some data settings that are favorable to summary boosting and similarly the settings that are not good for the proposed method?\n\nThese datasets were picked to be diverse in the number of features, proportions of different feature types (continuous, categorical) and the dataset size, so we believe that we have covered some of the most common settings. These were also the datasets commonly used in related papers, including TabPFN [1] and LIFT [2]. We haven’t tested on datasets with larger numbers of data points because the context size of current LLMs are limited.  \n\nIn Section 4.2 we have mentioned that our method generally works well for small tabular data without many continuous features. It especially works well when the task benefits from background knowledge where the LLM pretraining is helpful, such as *caesarian, somerville-happiness-survey, haberman-survival* and *TA evaluations*. \n\nWhen the dataset is large, this prior knowledge might become less relevant, so methods like finetuning become more competitive. Summary boosting is also less useful on datasets with many continuous variables, such as wine, iris, glass, vehicle, even though we encode these numerical values as descriptive attributes. Quantitative reasoning is inherently a problem with LLM but it is increasingly being solved in newer models like GPT-4 and Minerva.\n\n[1] N. Hollmann, S. Müller, K. Eggensperger, and F. Hutter. Tabpfn: A transformer that solves small tabular classification problems in a second. arXiv preprint arXiv:2207.01848, 2022.\n\n[2] T. Dinh, Y. Zeng, R. Zhang, Z. Lin, S. Rajput, M. Gira, J.-y. Sohn, D. Papailiopoulos, and K. Lee. Lift: Language-interfaced fine-tuning for non-language machine learning tasks. arXiv preprint arXiv:2206.06565, 2022.\n\n**Minor comments/questions:**\n\n> a) Please make fonts bold (or use color) in the tables for the best numbers in each row.\n\nWe have posted a PDF of the table with the best-performing results highlighted in bold. We refer to our overall response note to all reviewers.\n\n> b) Are the results reproducible? What is the cost of running the experiments?\n\nYes our results are reproducible and we have included code and instructions to replicate our experiments. However, there has been evidence that the ability of the models exposed through OpenAPI APIs are changing which is outside of our control.\n\nThe cost of running the experiments is discussed in Appendix A.11."}, {"review_id": "AX8b5YXTxs", "reviewer": "Reviewer_im25", "summary": "This paper explored an interesting problem that how to apply and extend LLMs over tabular supervised learning tasks. The paper first described each tabular sample as text, and then resorted to LLM to generate the summary for a set of selected representative samples as the template, which can be viewed as a weak classifier. Finally, by integrating these induced weak classifiers followed by booting learning paradigm, a stronger boosting classifier is constructed. Experiments were conducted over several tabular classification data sets to demonstrated the effectiveness of such learning procedure.\n\nThis procedure makes full use of zero-shot/few-shot learning ability of LLMs and extends such ability to few-shot learning setting over tabular data.", "questions": "1.In Algorithm 1, the proposed Clustering sampling is based on GPT embedding to group the data. How does the clustering performance? Such as inter- intra clustering distance metrics? Or does such clustering method will group samples with different class labels together? Can we also make use of some encoder based LLMs (such as BERT, BigBird for long sequence) to perform such clustering?\n2.How to make use of meta-data of the feature to generate the text description of categorical feature values? And how to generate the summarization prompt pattern based on meta-data? Dose this pattern depend on the property of learning target? For example, binary classification vs multi-class classification.\n3.Can we make use of results of tree-based method (such as XGBoost) to guide the preprocess of continuous features? That is, making use of same bin method with XGBoost and remap each bin to a meaningful text phrase based on the feature’s meta-data? This is due to the fact that the performance of proposed method heavily depends on such discretization method.\nOne benefit of XGBoost is that it is able to perform feature selection when generate sub-trees. The irrelevant features of course have negative impact on clustering and summarization. This might be also affecting the conclusion that more examples do not means better performance in “How does the performance scale with more examples?”  ", "limitations": "1.The proposed method might difficult to be directly applied to high-dimension tabular data. \n2.The tabular data might be derived from different domains, such as biological? May be we need domain-specific fine-tuned LLMs or pretrained LM to perform clustering or summary to get optimal performance. Therefore, some conclusions especially from ablation study might be LLM depended.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.This paper extends the application scope of LLMs to traditional tabular data and sheds light on integrating LLMs to many real world machine learning systems.\n2.The paper gave some effective guidelines to transform tabular samples to text description based on LLMs and metadata automatically with minimal manual engineer.\n3.Extensive experiments are conducted to demonstrate some important impact factors on the classification performance. ", "weaknesses": "1.The current method seems not easy to apply on high-dimensional tabular data (and for high-dimensional data, we might suffer from insufficient training data, also might due to the limited text sequence handled by LLMs) or when there might be many irrelevant features (this might introduce noise in text description).\n\n2.There are generally three types of features in traditional tabular data, quantitative, ordinal and categorical. The paper talks more about numerical features. It is better to give a systematical discussion of how to generate text description over these three different types while considering the meta-data or give the guidelines to generate the prompt patterns over LLMs to get the text description.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are thankful for your positive review of our work! We are happy that you quote our experiments as extensive and as demonstrating the important aspects of our method. Please find our responses to your comments as follows.\n\n> 1. The current method seems not easy to apply on high-dimensional tabular data (and for high-dimensional data, we might suffer from insufficient training data, also might due to the limited text sequence handled by LLMs).\n\nThank you for the comments! In general, high-dimensional data usually causes problems for machine learning due to the curse of dimensionality. So, for very high-dimensional data, we could generally expect there to be low-dimensional structure in the data. In our case, one could perhaps first summarize the individual data description to get a shorter data description. As we have explained in Appendix A.1, our data-to-text conversion pipeline tries to obtain a textual description of 80 words or less, which maintains a uniform length among all the descriptions.\n\n> 2. There are generally three types of features: quantitative, ordinal and categorical in tabular data. It is better to discuss/give guidlines on how to generate text description over these three different types while considering the meta-data.\n\nCurrently, we encode only the quantitative features separately and we rely on the LLM to convert ordinal / categorical data appropriately. This is because we believe that the LLM should be able to infer what these types of data represent from the context and meta data.\n\nHowever, we could potentially prompt the model to be aware of these ordinal/categorical features explicitly. We do not investigate this direction in the current work, but we will add this for future work.\n\n**Questions:**\n> 1. In Algorithm 1, the proposed Clustering sampling is based on GPT embedding to group the data. How does this group samples with different class labels? What are distance metrics used? Can we also use some encoder based LLMs (such as BERT, BigBird for long sequence) for clustering? \n\nSpecifications of our sampling are explained in Appendix A.7. The clustering algorithm used is Agglomerative Hierarchical Clustering (AGNES) from sklearn, with cosine distance metric, average linkage and a distance threshold of 0.05. \n\nAlgorithm 2 presents our stratified cluster sampling approach. This method creates separate clusters for different class labels and independently samples from them in a stratified fashion. Specifically, for each class, we obtain samples uniformly over its respective clusters. The stratification is performed at the class-level, ensuring that the proportion of class labels in the obtained samples remains the same as in the original training dataset.\n\nYes, we can also use other language embedding techniques such as BERT for getting these embeddings.\n\n> 2. How to make use of meta-data of the feature to generate the text description of categorical feature values? And how to generate the summarization prompt pattern based on meta-data? Dose this pattern depend on the property of learning target? For example, binary classification vs multi-class classification. \n\nWe are not sure if we understand the question properly, but the meta-data is generally for the whole dataset. It is integrated into the prompts for data conversion and summarization, providing the LLM with contextual understanding of the task (refer to figures 1 and 2). Further explanation can be found in Appendix A.1-2.\n\nIt's important to note that the inclusion of meta-data is independent of the target labels. Prompts for all datasets are detailed in Table 3. The summarization process is solely guided by the summarization directive, whether it's \"Tl;dr\" (Too long; didn't read) or \"Summarize in detail.\" When making inference, the prompt does mention the target labels.\n\n> 3. Can we use XGBoost to guide the preprocessing of continuous features? That is, apply the same bin method with XGBoost and remap each bin to a meaningful text phrase based on the feature’s meta-data? The irrelevant features have a negative impact on clustering and summarization. This might be also affecting the conclusion that more examples do not means better performance in \"How does the performance scale with more examples?\"\n\nWe thank the reviewer for this suggestion. This should be a promising experiment to try. Since trees created inside XGBoost have different feature splits in each boosting round, it might be difficult to formulate how they can be effectively combined to create a general discretization method for every feature. We leave this for future work. \n\nOn the flip side, as we explore ways to improve the processing of numerical values, this problem itself might go away as newer LLMs may become better at quantitative reasoning.\n\nWe further seek one clarification from the reviewer, to understand what they meant by: \"This might be also affecting the conclusion that more examples do not means better performance in \"How does the performance scale with more examples?\"\"\n\n**Limitations:**\n\n> 1.The proposed method might difficult to be directly applied to high-dimension tabular data. \n\nWe refer to our explanations addressed in the previous comments.\n\n> 2. The tabular data might be derived from different domains, such as biological? Can we use domain-specific fine-tuned LLMs or pretrained LM to perform clustering or summary ?\n\nThe reviewer’s observation may be correct since tabular datasets can represent different domains. Indeed, we expect domain specific LLMs to perform better however we don't think is this a limitation per se. For example, in scenarios such as medical diagnosis, it can be more prudent to use a LLM pretrained on medical data which will achieve better results. For example a medical-specific LLM might perform better on these datasets - verterbra-column, breast-cancer, caesarian, blood-transfusion-center, and haberman-survival. This observation should be true of the LLM embeddings used in clustering as well."}]

Author rebuttal:
A common point shared by many reviewers was to make fonts bold in the tables 1 & 2 for the best numbers in each row. We have added bolding to highlight the best-peforming results which can be viewed in the attached file. 

We note that our method doesn't always show improvements over the state of the art due to some current limitations in LLM with respect to context size that prevent it from ingesting a large number of examples (might be solved in GPT-4 or newer models). We show that our approach works best when the dataset is small and doesn't contain many continuous features, which is reasonable since LLM exploits prior from pretraining to its advantage. In the future, one may combine this strength with other models to get the best of both worlds.

Meta-review: This paper asks the question: "Can large language models serve as weak learners?" Here, weak is used in the sense of boosting algorithms. Text descriptions of tabular data are fed into large language models (LLMs) and the resulting prompt responses are used within a boosting algorithm. This is an interesting way to combine prompts that can outperform typical few-shot learning.

The reviewers generally liked the creative combination of traditional machine learning and LLMs. This is a novel direction and, as the paper indicates, opens up interesting possibilities for incorporating LLMs into larger machine learning pipelines. The reviewers made several suggestions for improving the presentation of the paper, which the authors are strongly encouraged to include in the final version. One reviewer raised concerns regarding the cost of applying LLMs in this way, but as the authors say, this work points in a future direction that can also be explored with open-source models and the reviewer did not respond. While this approach might not be completely practical at this time, the approach seems promising and this direction should be shared with the community.
Decision: Accept (poster)