LLMs Boost the Performance of Decision Trees on Tabular Data across Sample Sizes

Authors: Bethmage Mayuka Jayawardhana, Renbo Tu, Samuel Dooley, Valeriia Cherepanova, Andrew Gordon Wilson, Frank Hutter, Colin White, Tom Goldstein, Micah Goldblum
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Tabular data, large language models, decision trees, ensembling

Abstract:
Large language models (LLMs) perform remarkably well on tabular datasets in zero- and few-shot settings, since they can extract meaning from natural language column headers that describe features and labels. In contrast to LLMs, gradient-boosted decision trees (GBDTs) must learn the relationships among columns from scratch, increasing their data requirements. Meanwhile, LLMs are not competitive with GBDTs on medium or large datasets, and their scalability is capped by their limited context lengths. In this paper, we propose LLM-Boost, a simple and lightweight approach for fusing large language models with gradient-boosted decision trees, which enables larger datasets to benefit from the natural language capabilities of LLMs than was previously shown. While matching LLMs at sufficiently small dataset sizes and GBDTs at sufficiently large sizes, LLM-Boost outperforms both standalone models on a wide range of dataset sizes in between. We demonstrate state-of-the-art performance against numerous baselines and ensembling approaches, and we also show how to fuse GBDTs with TabPFN, a recent non-LLM model for in-context learning on tabular data. We find that this combination achieves the best performance on larger datasets. We release our code at https://anonymous.4open.science/r/LLM-Boost-21DD.

Decision: reject
Reviewer scores: 5, 3, 3, 3

Reviews and discussion:
<thread>
creation date:
1730026043977

thread_type:
official_review

<thread_object>
creator: reviewer_h4av

<content>
summary:
The authors propose LLM-Boost, a fusion of LLM and decision tree algorithms. The main motivation here is the idea that GBDT is known to be the best performing predictive model in the tabular domain, but it cannot handle linguistic context, so it would be useful to be able to inject linguistic context into GBDT.

strengths:
1. Great motivation. I also agree that injecting linguistic context is important to improve tabular predictive models, and to that end, many researchers are trying to incorporate LLM, which is an important research direction in the tabular learning community.

2. Simple method. If I understood the paper clearly, the main idea of LLM-Boost is to change the first step of GBDT determination to LLM's prediction, which seems really simple and intuitive to me.

3. The authors conducted extensive experiments on a variety of datasets.

weaknesses:
1. Visualizations for the main results are too difficult to understand.

2. In some cases, the performance improvement is too small or even no improvement. While the authors claim that it is important to integrate LLM and GBDT as the dataset size increases, results such as Table 3 (Appendix) do not support this. When the training set size is at its maximum, the performance of LLM-Boost is almost identical to XGBoost.

questions:
1. Did the authors try other LLMs, such as Llama 3?

2. Why did the authors use Flan-T5 for the main experiment?

3. Can the proposed LLM-Boost be used with commercial-level LLMs like GPT-4 (which is a black-box API)?

4. Have you tried a dataset with more samples? For example, a dataset with a million samples.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730026691268

thread_type:
official_review

<thread_object>
creator: reviewer_sk2m

<content>
summary:
The paper introduces LLM-Boost, a method that combines large language models (LLMs) and gradient-boosted decision trees (GBDTs) to improve performance on tabular data. The approach leverages LLMs' ability to interpret column headers and GBDTs' efficiency on large datasets. LLMs are used to extract predictions and residuals, which are then refined by GBDT models, leading to improvements, particularly on small to medium-sized datasets.

strengths:
1. **Novelty.** Demonstrating that combining LLMs with GBDTs for tabular prediction tasks in such a simple manner can be effective is a valuable contribution.

2. **Discussion on limitations and future work.** A thorough discussion of limitations and future work is often overlooked, but it is something researchers reading the paper can greatly appreciate.

weaknesses:
1. **Computational overhead.** It is unclear how practical it is to use LLM inference for tabular prediction tasks. An alternative use of LLMs in the tabular domain, which arguably is much simpler conceptually, is to use them for automatic feature engineering and training GBDTs on the augmented set of features for prediction [1, 2].

2. **Missing experiments.** Several crucial baselines, such as comparing LLM-Boost against simply increasing the number of trees in the GBDT, are missing. Additionally, it would be useful to evaluate how the method scales to datasets with significantly larger feature sets.

---
[1] Hollmann et al. “Large language models for automated data science: Introducing caafe for context-aware automated feature engineering.” (2023).\
[2] Nam et al. “Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning.” (2024).

questions:
1. How does LLM-Boost perform when column headers are semantically ambiguous or incorrect? Would it still provide performance benefits over standalone GBDTs?

2. Why can’t we simply compare prediction accuracies to evaluate performance?

3. Typos: In Section 3.2, “replacing the first tree int he” → “replacing the first tree in the”.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730603942521

thread_type:
official_review

<thread_object>
creator: reviewer_fo2b

<content>
summary:
This paper combines both LLM and decision trees to do the classification task on the tabular data. The major experiment is to explore how LLM, decision tree and the combined method work across different data sizes.

strengths:
1. This paper combines LLM and decision trees for tabular classification tasks. It shows certain novelty in the proposed method.

weaknesses:
1. The major baseline is quite strange. It would be better to separately show how LLM and decision tree work and treat both as the baselines, rather than mix them into one baseline 'Select'.
2. The experiment is not comprehensive. This paper simply lists the performance of different data sizes. It would be better to show some error analysis and cases to indicate when the proposed method can excel.
3. The selection of the LLM (eg, Flan-T5) is not very convincing.

questions:
1. Can close-sourced LLMs such as GPT-family models perform well on such tasks?
2. What is the rationale for choosing Flan-T5 for the major experiments?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731223279345

thread_type:
official_review

<thread_object>
creator: reviewer_2c18

<content>
summary:
The paper introduces the LLM-Boost algorithm, which integrates large language models (LLMs) with gradient-boosted decision trees (GBDTs) to enhance classification performance on tabular datasets. The method involves extracting LLM logits for each row of data and using these predictions to augment the GBDT model, allowing it to learn the residuals to the true labels. The authors demonstrate that LLM-Boost outperforms traditional models and other ensemble techniques across various dataset sizes, showcasing its potential in automating predictive modeling pipelines.

strengths:
- **Enhanced Performance:** LLM-Boost shows superior classification performance compared to traditional GBDTs and other ensemble methods.
- **Model Agnostic:** The boosting mechanism can be applied to various high-performing tabular architectures beyond LLMs.
- **Efficiency in Training:** Once LLM outputs are pre-computed, the training cost aligns with that of standard GBDT training.

weaknesses:
- **Dependence on Interpretability:** The method requires interpretable text descriptors as column headers, which may necessitate prompt engineering for some datasets.
- **Pre-computation Costs:** For very large datasets, the initial cost of pre-computing LLM outputs may become significant.

questions:
1. What is the performance with large-sized datasets, instead of small and medium sized datasets?
2. line 201 "weather" -> "whether"
3. Missing inference:
- Iida, Hiroshi, Dung Thai, Varun Manjunatha, and Mohit Iyyer. "Tabbie: Pretrained representations of tabular data." arXiv preprint arXiv:2105.02584 (2021).
- Chen, Pei, Soumajyoti Sarkar, Leonard Lausen, Balasubramaniam Srinivasan, Sheng Zha, Ruihong Huang, and George Karypis. "HYTREL: Hypergraph-enhanced tabular data representation learning." Advances in Neural Information Processing Systems 36 (2024).
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents LLM-Boost, an algorithm that integrates large language models (LLMs) with gradient-boosted decision trees (GBDTs) for classification tasks on tabular data. The proposed method replaces the first tree of the GBDT ensemble with the static prediction of the LLM. The authors argue that this hybrid method can leverage the strengths of both LLMs (textual/context understanding) and GBDTs (robust performance on tabular data, especially with sufficient samples). Experiments compare the proposed method against baseline models and ensemble techniques across various dataset sizes, with the conclusion that LLM-Boost excels particularly in small to medium-sized datasets.

**Strengths:**

* "Simple and intuitive" method (h4aV), with "great motivation" (h4aV) and "valuable contribution" (Sk2m) to use a hybrid approach to combine LLMs and classical GDBT methods. 

* The proposed method shows potential of "superior classification performance compared to traditional GBDTs and other ensemble methods" (2C18), while the improvements largely come from intermediate-sized datasets (see weaknesses for details), leading to concerns around its limited practical impact.

* The authors explicitly discussed the limitations of the proposed approach, which is "often overlooked, but it is something researchers reading the paper can greatly appreciate." (Sk2m).

* "extensive experiments on a variety of datasets." – (h4aV)

**Weaknesses:**

There are two major issues of this paper:

* First of all, it is unclear whether the focus on "intermediate" dataset sizes would limit the practical impact of the paper. As noted by h4aV:

> The authors need to clearly define the scope. … now the authors claim that their main contribution is on intermediate-sized datasets – how do they define ‘intermediate-sized’ and is this really a practical situation?

while I believe this setting is important, the authors need to provide further clarification along with analysis to clearly motivate the rationale of focusing on tabular datasets with such sizes. Without detailed explantation, some claims in the paper might be confusing (h4aV: "while the authors claim that it is important to integrate LLM and GBDT as the dataset size increases, results such as Table 3 (Appendix) do not support this ...").

* Using Flan-T5 as the primary LLM for experiments "is not very convincing" (Fo2b). While the authors further provided results from Llama-8B, the worse results from this paper raises concerns on "once a better open LLM is released, LLM-Boost will not be able to benefit from it. (h4aV)". The authors shall experiment with more recently released LLMs with ~10B parameters in order to show the general applicability of LLM-Boost while studying the impact of different LLMs on the performance of the approach. Nit: such important results, including those from Llama-8B, shall be moved to the main text of the paper.