SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Test errors for the chosen methods on all datasets (↓). Data Type indicates the number and types of attributes the dataset has (c is continuous and d is discrete). Size denotes the number of data points. In square brackets (if present) next to every dataset name, we provide its acronym referred to in our main text. In the bracket next to each dataset name is either the OpenML ID of the dataset or a reference to the dataset's associated publication. Error represents one standard deviation.
Dataset	Data Type	Size	Zero-shot	Few-shot	Summary	Summary Boosting
caesarian [cae] (42901)	1c4d	80	0.425±0.04	0.388±0.02	0.350±0.04	0.300±0.04
iris (61)	4c0d	150	0.680±0.02	0.460±0.01	0.275±0.07	0.193±0.03
tae (48)	1c4d	151	0.556±0.07	0.494±0.01	0.474±0.02	0.454±0.03
glass (41)	9c0d	214	0.486±0.01	0.473±0.01	0.466±0.02	0.370±0.02
breast-cancer [bc] (13)	7c5d	277	0.754±0.02	0.516±0.02	0.337±0.02	0.288±0.02
visualizing-environmental [ve] (678)	3c0d	111	0.522±0.01	0.366±0.01	0.304±0.02	0.268±0.03
analcatdata-chlamydia [ac] (535)	2c2d	100	0.200±0.00	0.200±0.00	0.170±0.01	0.170±0.01
wine (43571)	13c0d	178	0.820±0.03	0.674±0.02	0.475±0.01	0.320±0.01
blood-transfusion-center [btc] (1464)	4c0d	748	0.544±0.01	0.430±0.00	0.258±0.04	0.240±0.04
somerville-happiness-survey [shs] [Koczvodaj, 2018]	0c7d	143	0.416±0.03	0.385±0.03	0.422±0.02	0.350±0.02
vehicle (54)	18c0d	846	0.765±0.00	0.560±0.01	0.510±0.02	0.410±0.04
statlog-heart [sth] [Dua and Graff, 2017]	6c7d	270	0.551±0.01	0.528±0.01	0.444±0.05	0.430±0.01
vertebra-column [vc] (1524)	6c0d	310	0.714±0.03	0.435±0.06	0.327±0.01	0.262±0.01
ecoli (1011)	7c0d	336	0.581±0.02	0.562±0.01	0.480±0.01	0.270±0.03
haberman-survival [hs] (43)	3c0d	306	0.308±0.02	0.262±0.01	0.277±0.01	0.250±0.01
diabetes [dia] (37)	8c0d	768	0.446±0.04	0.400±0.00	0.360±0.01	0.344±0.01
visualizing-hamster [hams] (708)	5c0d	73	0.464±0.03	0.481±0.05	0.360±0.02	0.207±0.00
wholesale-customers [wc] (1511)	6c1d	440	0.364±0.01	0.347±0.01	0.349±0.02	0.330±0.00
Table 2: Test errors for chosen methods on all datasets (↓). (acronym and notation in Table 1)
Dataset	Data Type	Size	Summary Boosting	LIFT	KNN	TabPFN	Xgboost
cae (42901)	1c4d	80	0.300±0.04	0.312±0.02	0.300±0.00	0.425±0.07	0.412±0.05
iris (61)	4c0d	150	0.193±0.03	0.100±0.01	0.106±0.02	0.027±0.00	0.054±0.04
tae (48)	1c4d	151	0.454±0.03	0.480±0.04	0.532±0.01	0.450±0.13	0.464±0.01
glass (41)	9c0d	214	0.370±0.02	0.218±0.02	0.294±0.03	0.158±0.05	0.254±0.05
bc (13)	7c5d	277	0.288±0.02	0.318±0.01	0.277±0.02	0.264±0.01	0.270±0.01
ve (678)	3c0d	111	0.268±0.03	0.430±0.04	0.308±0.01	0.370±0.04	0.279±0.02
ac (535)	2c2d	100	0.170±0.01	0.180±0.06	0.170±0.01	0.090±0.01	0.110±0.04
wine (43571)	13c0d	178	0.320±0.01	0.065±0.01	0.214±0.05	0.040±0.01	0.040±0.01
btc (1464)	4c0d	748	0.240±0.04	0.270±0.01	0.238±0.00	0.209±0.01	0.219±0.01
shs [Koczkodaj, 2018]	0c7d	143	0.350±0.02	0.419±0.02	0.326±0.03	0.392±0.00	0.406±0.00
vehicle (54)	18c0d	846	0.410±0.04	0.111±0.16	0.636±0.01	0.178±0.01	0.260±0.00
stath [Dua and Graff, 2017]	6c7d	270	0.430±0.01	0.122±0.17	0.244±0.03	0.148±0.03	0.215±0.00
vc (1524)	6c0d	310	0.262±0.01	0.192±0.03	0.318±0.02	0.135±0.00	0.187±0.04
ecoli (1011)	7c0d	336	0.270±0.03	0.126±0.03	0.211±0.03	0.036±0.02	0.066±0.01
hs (43)	3c0d	306	0.250±0.01	0.314±0.03	0.278±0.00	0.262±0.02	0.281±0.02
dia (37)	8c0d	768	0.344±0.01	0.324±0.04	0.353±0.02	0.238±0.03	0.234±0.00
hams (708)	5c0d	73	0.207±0.00	0.334±0.08	0.528±0.02	0.328±0.01	0.411±0.01
wc (1511)	6c1d	440	0.330±0.00	0.125±0.04	0.043±0.00	0.088±0.00	0.098±0.02
Unlike naively stacking examples inside the prompt as in Few-shot, Summary condenses knowledge from these examples and is the first important algorithmic component of our framework for creating weak learners using LLMs. We see that Summary consistently improves upon Few-shot on all the datasets other thanhaberman-survival and wholesale-customers.This observation suggests that summarization is a powerful way to improve few-shot performance and has the potential for even other tasks using LLMs. Finally, for every dataset we tested, boosting with summarization consistently outperforms all other prompting-based approaches. This observation corroborates our hypothesis that LLMs with summarization are a good candidate for creating weak learners in boosting.
4.2 Comparison to other tabular methods
In Table 2, we also observe that LLMs have a hard time reasoning about continuous attributes without finetuning, especially on the glass, wine, iris and vehicle datasets. In particular, when the datasets have many continuous features, the performance of Summary Boosting can be considerably worse than other methods such as LIFT or Xgboost. This may be because LLMs are fairly bad at quantitative reasoning without finetuning [Lewkowycz et al., 2022], which may be overcome in future LLMs.
While KNN is a relatively simple baseline, its performance is surprisingly good at a few tasks such as wholesale-customers, statlog-heart, ecoli and wine. This highlights that LLMs have a remarkable amount of general prior knowledge about the world compared to methods like XGboost that sometimes this knowledge alone can produce good performances.
Figure 3: Ablation experiments compared for the Summary method. The dataset acronyms are referred to 1. Left: Performance of curie vs davinci. Second from left: Comparison with and without task-specific attribute names. Middle: Effect of Cluster vs. Random sampling on the number of rounds till convergence and Second from Right: their final test errors. Right: Performance of templatized vs LLM-generated data descriptions.
Figure 4: Additional ablations for the Summary method. (Left) Prompt design choices. The first plot shows the effect of shuffling examples vs presenting them by class. The center plot compares t1dr vs a more explicit prompt for inducing summary. The last plot compares prompts for doing inference. (Right Top) Performance of methods for discretizing continuous attributes on the Wine dataset. (Right Bottom) Performance of Few-shot and Summary as a function of the number of examples in the context.
Finally, we observe that Summary Boosting performs very well when the size of the dataset is very small. This makes sense since the strength of using LLMs as weak learners is that they have a large amount of generic prior about the world from pre-training. When the dataset is large, this prior knowledge might become less relevant and methods like finetuning become more competitive.
Summarization forms the core of our methodology for generating weak learners. Consequently, it becomes important to identify an ideal setting that can induce high-quality summaries. We perform ablation studies over the Summary method, to decide hyperparameters for getting a good weak learner.
Preprocessing of continuous attributes. We tried several encoding techniques for continuous features, including binning, percentiles, and standard deviations. We chose the approach of describing them in technical language terms as well as assigning quantifiers for each level, as illustrated in Figure 4 right top. We observed that binning with quantifiers such as "low", "medium", and "high" was most effective for comparing examples and generating high-quality summaries. After hyperparameter tuning, we identified that using 5 bins provides sufficient granularity to distinguish variations in the continuous values. More details can be found in the Appendix A.6.
Does the LLM explore prior knowledge to infer? To demonstrate the LLM's utilization of prior knowledge, we conduct an ablation study by masking the attribute names and using a template "This example has features  $f1 = \{\}$ ,  $f2 = \{\}$  and so on." Figure 3 (second from left) shows the result. Using true variable names in the data descriptions leads to superior few-shot learning performance compared to using dummy names. This confirms that the model indeed leverages its prior knowledge of variables for predictions.
How does model size affect the performance? A natural question to ask is how the model size affects the downstream performance. We compare the Summary performances of GPT-3-davinci (175B parameters) and GPT-3-curie (13B parameters) on 5 datasets in Figure 3 (left). Surprisingly, we find that the larger model (davinci) does not consistently improve upon the smaller model. We also compare ChatGPT in A.12 and discuss the effects of RLHF [Ouyang et al., 2022]. We further show that our method generalizes to LLMs other than GPT, by comparing its performance with Claude-2 in Appendix A.13).
How does the performance scale with more examples? In Figure 4 (right bottom), we study how the behavior of Few-shot and Summary change with different support set sizes (i.e., the number of data descriptions that are summarized). Few-shot performance reaches an optimal size around the medium context length and degrades with more examples. In contrast, Summary improves with more data, which is the more desirable behavior.
Ordering of examples. Unlike conventional machine learning models, the ordering of examples in summarization affects the generation of hypotheses. There are two approaches: 1. presenting descriptions randomly (shuffled), and 2. grouping descriptions by classes (grouped). In Figure 4 (left), we compare these approaches on 4 datasets with Summary and find no significant difference. We use shuffled for all other experiments.
Different summary and inference prompts. The LLM easily generates concise summaries on standard datasets like iris using a simple "t1;dr" prompt, but requires a more explicit prompt on complex datasets like "vertebra-column". Comparing their performance in Figure 4 (left), both prompting modes are equally effective, so detailed prompts were used in all other experiments. See Appendix table 3 for the complete list of summary prompts used. In the left of Figure 4, we also compare two strategies for inference - prefix prompt (e.g. "This flower will be classified as"), and two-stage chain-of-thought prompting (e.g. "Let's think step by step") [Kojima et al., 2022]. We observe no statistically significant difference between them<sup>3</sup>. Since the prefix prompt more often completes the query well under lesser compute, we use prefix prompt for all the other experiments.
Texts generated from template vs. GPT. In Figure 3 (right), we see that using GPT-generated data descriptions consistently achieves better results. Refer to Appendix A.9 for examples of these templates. This may be due to the fact that the data description generated by LLMs conforms to natural text distribution more closely, which is desirable for performing inference using LLMs.
Effect of cluster sampling. Cluster sampling improves the performance of the LLM by selecting a representative set of texts that generalize better, reducing validation error faster compared to random sampling during boosting. Although it may require more resampling to achieve a weak learner, cluster sampling converges much faster than random sampling as we see in Figure 3 - middle and second from right. However, with sufficient boosting rounds, the performances of the two sampling methods are not statistically different.
Although the weak learning approach we develop here shows promise, there are currently several drawbacks. Although summarization and boosting alleviate manual prompt tuning to a large extent, we still had to minimally tune some parts of the pipeline to get ideal performance (see Appendix table 3). Additionally, when the dataset contains many continuous attributes, there is a non-trivial gap between Summary Boosting and the other methods such as XGBoost or finetuning. Finally, the
3We do observe that CoT performs better with larger models.
max input length of GPT-3 makes it harder to generate good summaries with just subset sampling. Eventually on larger datasets, after a certain number of boosting rounds, the summaries derived from a subset of examples may not further decrease the weighted error across all training examples. Rigorous techniques such as structured prompting handle this issue by rescaling attention weights [Hao et al., 2022]. We believe this issue could be solved with more powerful LLMs such as GPT-4. Furthermore, methods like ours built using LLMs inherit bias that comes from the pretraining data. Specifically, it can affect the model's ability to objectively summarize the examples or make predictions that are inconsistent with the biased pre-training data. We believe with better and debiased LLMs these issues can be alleviated. We want to emphasize that our experiments utilized OpenAI APIs. In the appendix A.14, we provide a brief discussion on the impacts of commercial APIs.
LLMs have been widely used in recent years, not just for their generative abilities but for their ability to serve as zero- or few-shot learners with proper prompting. This work aims to situate them within another context of "simple" learning algorithms – the weak learner paradigm that forms the foundation of boosting approaches. We show that leveraging the summarization capabilities of LLMs indeed leads to models that function as weak learners, and which can thus be integrated into boosting frameworks. "Summary" as an intermediate step makes reasoning easier and works inside boosting algorithms for small tabular data up to 300 data points without many continuous features. Overall, this result leads to new potential paradigms for treating the results of LLM prompting not just as individual predictors, but as part of a larger set of meta-models as well.
Yiding is supported by funding from the Bosch Center of Artificial Intelligence. We would like to thank the anonymous reviewers for their valuable feedbacks.
V. Borisov, T. Leemann, K. Seßler, J. Haug, M. Pawelczyk, and G. Kasneci. Deep neural networks and tabular data: A survey. arXiv preprint arXiv:2110.01889, 2021.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
T. Chen and C. Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm SIGkdd international conference on knowledge discovery and data mining, pages 785-794, 2016.
G. Cui, S. Hu, N. Ding, L. Huang, and Z. Liu. Prototypical verbalizer for prompt-based few-shot tuning. arXiv preprint arXiv:2203.09770, 2022.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
S. Diao, P. Wang, Y. Lin, and T. Zhang. Active prompting with chain-of-thought for large language models. arXiv preprint arXiv:2302.12246, 2023.
T. Dinh, Y. Zeng, R. Zhang, Z. Lin, S. Rajput, M. Gira, J.-y. Sohn, D. Papailiopoulos, and K. Lee. Lift: Language-interfaced fine-tuning for non-language machine learning tasks. arXiv preprint arXiv:2206.06565, 2022.
D. Dua and C. Graff. UCI machine learning repository, 2017. URL http://archive.ics.uci.edu/ml.
Y. Freund and R. E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of computer and system sciences, 55(1):119-139, 1997.
J. H. Friedman. Greedy function approximation: a gradient boosting machine. Annals of statistics, pages 1189-1232, 2001.
G. Qin and J. Eisner. Learning how to ask: Querying lms with mixtures of soft prompts. arXiv preprint arXiv:2104.06599, 2021.
N. C. Rakotonirina, R. Dessi, F. Petroni, S. Riedel, and M. Baroni. Can discrete information extraction prompts generalize across language models? arXiv preprint arXiv:2302.09865, 2023.
L. Reynolds and K. McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. In *Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems*, pages 1-7, 2021.
S. Saha, X. V. Yu, M. Bansal, R. Pasunuru, and A. Celikyilmaz. Murmur: Modular multi-step reasoning for semi-structured data-to-text generation. arXiv preprint arXiv:2212.08607, 2022.
B. Schäfl, L. Gruber, A. Bitto-Nemling, and S. Hochreiter. Hoplar: Modern hopfield networks for tabular data. arXiv preprint arXiv:2206.00664, 2022.
R. E. Schapire. The strength of weak learnability. Machine learning, 5:197-227, 1990.
T. Shin, Y. Razeghi, R. L. Logan IV, E. Wallace, and S. Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980, 2020.
R. Shwartz-Ziv and A. Armon. Tabular data: Deep learning is not all you need. Information Fusion, 81:84-90, 2022.
J. Vanschoren, J. N. Van Rijn, B. Bischl, and L. Torgo. Openml: networked science in machine learning. ACM SIGKDD Explorations Newsletter, 15(2):49-60, 2014.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
H. Wang, C. Xu, and J. McAuley. Automatic multi-label prompting: Simple and interpretable few-shot classification. arXiv preprint arXiv:2204.06305, 2022a.
X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, and D. Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022b.
Z. Wang, Z. Zhang, C.-Y. Lee, H. Zhang, R. Sun, X. Ren, G. Su, V. Perot, J. Dy, and T. Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 139-149, 2022c.
J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022.
P. Yin, G. Neubig, W.-t. Yih, and S. Riedel. Tabert: Pretraining for joint understanding of textual and tabular data. arXiv preprint arXiv:2005.08314, 2020.
W. Yu, D. Iter, S. Wang, Y. Xu, M. Ju, S. Sanyal, C. Zhu, M. Zeng, and M. Jiang. Generate rather than retrieve: Large language models are strong context generators. arXiv preprint arXiv:2209.10063, 2022.
Z. Zhang, A. Zhang, M. Li, and A. Smola. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493, 2022.
H. Zhou, A. Nova, H. Larochelle, A. Courville, B. Neyshabur, and H. Sedghi. Teaching algorithmic reasoning via in-context learning. arXiv preprint arXiv:2211.09066, 2022a.
Y. Zhou, A. I. Muresanu, Z. Han, K. Paster, S. Pitis, H. Chan, and J. Ba. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910, 2022b.
A Failure Modes
The design of the prompt plays a pivotal role in our entire process. Specifying instructions precisely can create a significant difference, whether it comes to effectively describing tabular data, generating reliable summaries or inferring accurate predictions (shown in Figures 1 and 2). While soft prompting has been successful at instructing LLMs [Lester et al., 2021], it cannot be applied to our setting because our classification algorithm learns a human-level summary as a prompt for classifying data, rather than a soft prompt. Instead we chose to write prompts that ask the LLM to perform these tasks. In this way, our prompting method is entirely gradient-free. Hand-written prompts also offer flexibility, aligning well with our core methodology for creating weak learner. While carefully handcrafting prompts this way might seem intensive, we will show that once we identify the ideal hyperparameter settings they can be framed with little effort.
A.1 Data Conversion Challenges
Language models (LLMs) have demonstrated impressive performance on standard tasks with minimal supervision [Wang et al., 2022b, Brown et al., 2020]. However, for converting tabular data to text, there were several considerations to meet the requirements of our task. As highlighted in Section 3.1, we will refer to these texts as data descriptions.
Ensuring Uniform Length Firstly, the data descriptions should not be too long or short, also be of comparable length. Excessively long descriptions limit the number of examples that can be fit inside the prompt and summarized. We observed in Figure 4 (right bottom) that the summary performance also scales with more examples, so it makes sense to accommodate as many of them as possible by having descriptions of approximately uniform length.
A straightforward way of achieving this uniformity would be by specifying a max word length as part of the conversion prompt itself, as in "Describe in not more than 80 words". However, we found this approach can falter, sometimes leading to overly simplistic descriptions like "These are annual spending of a customer." (in the wholesale-customers dataset).
Consequently, we adopt more nuanced strategy by first modifying the prompt with the terms "concisely" and "accurately" to emphasize the brevity and preciseness of the generated descriptions (shown in Figure 1). Then, we implement a resampling strategy, that generates descriptions until finding the one with a desired length ranging between 20 to 80 words. This process achieves consistent and uniformly long descriptions.
Including Metadata Prepending metadata to the prompt enhances the contextual awareness of the task, resulting in higher-quality descriptions (shown in Figure 1).
Separating Features from Labels In our method, the data descriptions function dual role, both as training examples and as query for inferring class labels. This suggests that, when converting data to text, the features need to be described separately from the target label as illustrated in Figure 1. The resulting strings are then concatenated to form the data description. Instead, if the whole tabular record were passed to the LLM, it often produces texts that assimilate the classification label information in the meat of the description itself, rendering it difficult to extract a query for doing inference.
While one might cleverly come up with prompts that can allow the LLM to describe the features and target label in separate sentences, we found it to be more sensible to supply just the features for describing and not reveal any information about the target task. Sometimes that can liberate the LLM to hallucinate some facts about the task and form biased data to begin with.
Natural-Sounding Descriptions While the LLM generates a different-styled response every time, to explicitly ensure that the generated descriptions are not template-like by chance, add a directive at the end of the prompt: "Use your creativity". This encourages the LLM to produce more natural narratives of the record. Alternatively, setting a higher temperature during decoding achieves a similar effect.
A.2 Summarization
There are several aspects worth considering that can contribute to high-quality summaries.
Sampling candidate summaries A well-crafted summary is a one that captures salient information of the dataset, in a way that facilitates inferring predictions off it. However, the process of generating summary using a LLM is inherently stochastic due to temperature sampling, as a result, the generated summary can be noisy. From our experiments with tuning this temperature, we found 0.80 to be ideal through Bayesian optimization. Even at this value, on average only 1 out of 3 summaries were meaningful.
A noisy summary can be distinguished quite easily. For instance, on the vehicle dataset, the t1;dr prompt elicits summaries as naive as "The given data describes a bus, van, or saab silhouette." or "The data in this table identifies a vehicle as a bus, saab, opel, or van. The compactness, circularity, distance circularity, radius ratio, hollows ratio, and symmetry are all predictive of the vehicle's type." which does not offer actionable insight.
This observation indicates that summaries need to be sampled quite a few times and the best one can be determined based on the validation error. As a result, for the Summary learning procedure in Section 3.2, we resample approximately 25 times to find a good summary. Also, given that our datasets are small, it is not unusual for the summaries to have the same validation error. When tied, we pick one having a higher training error rate, i.e. lower generalization gap.
Differently, in our Summary boosting procedure explained in Section 3.3, we resample only until finding a summary whose training error is better than random guessing and return immediately.