TabM: Advancing tabular deep learning with parameter-efficient ensembling

Authors: Yury Gorishniy, Akim Kotelnikov, Artem Babenko
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: tabular, tabular data, deep learning, architecture

Abstract:
Deep learning architectures for supervised learning on tabular data range from simple multilayer perceptrons (MLP) to sophisticated Transformers and retrieval-augmented methods.
This study highlights a major, yet so far overlooked opportunity for substantially improving tabular MLPs; namely, parameter-efficient ensembling -- a paradigm for imitating an ensemble of models with just one model.
We start by describing TabM -- a simple model based on MLP and BatchEnsemble (an existing technique), improved with our custom modifications.
Then, we perform a large scale evaluation of tabular DL architectures on public benchmarks in terms of both task performance and efficiency, which renders the landscape of tabular DL in a new light.
In particular, we find that TabM outperforms prior tabular DL models, while the complexity of attention- and retrieval-based methods does not pay off.
Lastly, we conduct a detailed empirical analysis, that sheds some light on the high performance of TabM.
For example, we show that parameter-efficient ensembling is not an arbitrary trick, but rather a highly effective way to reduce overfitting and improve optimization dynamics of tabular MLPs.
Overall, our work brings an impactful technique to tabular DL, analyses its behaviour, and advances the performance-efficiency tradeoff with TabM -- a simple and powerful baseline for researchers and practitioners.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1730494697554

thread_type:
official_review

<thread_object>
creator: reviewer_agcy

<content>
summary:
The paper applies the BatchEnsemble technique to MLPs for tabular data, and investigates several modifications. The results show an improvement over MLPs and several deep baseline models and GBRT on a broach benchmark of 50 datasets from the literature. The experiments show that in the first adapter layer in particular is extremely critical, and results in the majority of gains.

strengths:
The paper discusses the adoption of the BatchNorm architecture for the tabular setting, and describes several interesting ablations.
The empirical evaluation is broad and in-depth and well presented.
The empirical results are quite strong against a broad variety of baselines.
The paper is well written.

weaknesses:
- The technical novelty of the paper is somewhat small; however, this is made up for by the in-depth analysis with somewhat surprising results and the good performance of the proposed model.
- The paper doesn't describe the relationship to dropout, even though dropout was original describes as an efficient ensemble approach. Given the simple nature of TabM_mini, there seem to be some obvious parallels between drop-out and TabM_mini that I think are worth discussing. In particular, TabM_mini actually contains a dropout layer, and it would be interesting to evaluate how the two forms of ensembling complement each other.
- The comparison doesn't include TabPFN, an extremely strong baseline. McElfresh showed that even subsampling data to at most 3000 datapoints, TabPFN outperforms most other models.
- Given the focus on improving MLP performance, a comparison with regularization cocktails and "Better by Default: Strong Pre-Tuned MLPs and Boosted Trees on Tabular Data" might be relevant but not essential.
- Given the somewhat surprising result about the importance of the initial adapter, it would be great to have more ablations on the different components. In particular, it's unclear in how far the initial adapter and last layer are coupled. How does the performance change if the best performing last layer is used for all the initial adapters? I.e. are the initial layer and last layer co-adapted or do they work independently? 

## Minor comments
Figure 2 cuts off points at 8%. It seems there's a lot of points on exactly that line, which seems a bit suspicious. Are these all the outliers that are clipped to the limits of the figure? It would be great to indicate outliers in the plot.

Table 2 is far below the mention and maybe should be moved up.

Table 2, MLP on the Maps dataset is missing units in duration.

The title of section 5.3 should read "How does the performance of TabM depend on k?"

Line 257: "requires a special care" should be "requires special care"

Line 160: multiplication with 100% is not mathematically meaningful.

questions:
- Are number of layers for MLP and TabM tuned as hyper-parameters? What are typical values for each?

- Why is k not included in the hyper-parameter search for TabM?

- How do you ensure diversity for the MLPs in the ensemble in Figure 6?

- How do you initialize the last layers in TabM to ensure diversity?

- Is it possible that TabM_mini outperforms the real ensemble because of more diversity in the initialization?

- Is it correct that when doing the full ensemble, adding an adapter into each model would lead to an equivalent ensemble? I was wondering why this ablation was not performed, but I assume random initialization of weights for each member of the ensemble makes the adapters irrelevant.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730522859072

thread_type:
official_review

<thread_object>
creator: reviewer_for5

<content>
summary:
This paper proposes TabM, a deep learning model for tabular data based on MLP enhanced with BatchEnsemble, along with custom modifications designed to improve the model's efficiency and performance. The authors position TabM as an efficient alternative to more complex architectures like Transformer-based models, claiming it achieves superior performance without their added computational cost. The paper also provides a detailed analysis, suggesting that parameter-efficient ensembling can effectively address overfitting and improve optimization dynamics in tabular MLPs.

strengths:
1. The paper is well-structured and clearly written, making the methods, results, and analysis easy to understand.
2. TabM demonstrates competitive performance, positioning it as an efficient alternative for tabular data modeling.
3. By introducing BatchEnsemble techniques into tabular data modeling, the authors have adapted and enhanced a previously underutilized approach in this domain.
4. The paper provides a thorough analysis of each component's impact on model performance, contributing valuable insights.

weaknesses:
1. Although the authors mention similar methods, a more detailed comparison with other related approaches for tabular data, such as Trompt, would be helpful. Despite structural differences, both methods share commonalities in prediction, such as averaging multiple head (cycle) outputs and summing losses across heads (cycles). 
2. Comparisons with standard ensemble methods are insufficient. Incorporating results from methods like model soup could help position TabM's performance.
3. Dataset scope is somewhat limited, particularly for classification tasks. For a more comprehensive evaluation, the authors might consider incorporating the datasets from Tabzilla, which could provide richer classification benchmarks and enable a more detailed analysis, such as comparing performance across binary, multiclass, and regression tasks or evaluating model performance on datasets of varying sizes.

[1] Kuan-Yu Chen, Ping-Han Chiang, Hsin-Rung Chou, Ting-Wei Chen, Tien-Hao Chang: Trompt: Towards a Better Deep Neural Network for Tabular Data. In ICML

[2] Duncan C. McElfresh, Sujay Khandagale, Jonathan Valverde, Vishak Prasad C., Ganesh Ramakrishnan, Micah Goldblum, Colin White: When Do Neural Nets Outperform Boosted Trees on Tabular Data? In NeurIPS

questions:
1. In Table 2 (p.10), the last column of the MLP row lists "10." Could the authors confirm if it is measured in minutes, as there’s no unit mentioned?
2. In line 1027, the authors state, *For numerical features, by default, we used a slightly modified version of the quantile normalization from the Scikit-learn package, with rare exceptions when it turned out to be detrimental.* Could the authors clarify how they define “detrimental” in this context and provide further explanation?
3. Could the authors provide more detailed diversity metrics for the predictions of the 32 base models?
4. Line 431 notes, *Interestingly, the best prediction head of TabM performs no better than the plain MLP.* Would an ensemble of 32 independently and undertrained MLPs yield similarly strong performance?
5. The ranking method used is unconventional. Is this ranking method novel to this paper? Could the authors also provide results for a more traditional average rank metric to facilitate comparison, as it would be interesting to see how these two measures differ?
6. One advantage of using deep learning for tabular data is the ability to generate embeddings that are useful for downstream tasks and multimodal learning. How does TabM determine the representation of each sample? Is it based on concatenation, averaging across base models, or some other method? Additionally, could the authors provide a comparison of TabM’s embeddings with those from other deep learning approaches on specific tasks to showcase their effectiveness?

---
Lastly, if the authors can address the concerns noted in the weaknesses and questions, I will consider raising my score accordingly.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730709355040

thread_type:
official_review

<thread_object>
creator: reviewer_4una

<content>
summary:
This paper come up with an efficient ensembling method for tabular deep learning — which integrate BatchEnsemble with multilayer perceptron. This method exhibits great improvements in their benchmark.

strengths:
- [Major] This method is simple yet effective.
- [Major] The paper is easy to read.

weaknesses:
- [Medium] The proposed method is not much novel. Note that this concern is not the main reason that I chose to reject this paper. If other concerns listed can be addressed (e.g., more technical contributions, comprehensive empirical study for depper understandings), I think this paper is still very useful.
- [Major] More experiments are needed to fully understand the behavior of TabM. In the abstract, the authors mentioned “we show that parameter-efficient ensembling is not an arbitrary trick, but rather a highly effective way to reduce overfitting and improve optimization dynamics of tabular MLPs.” However, in the paper, there is no discussion why TabM helps with reducing overfitting and optimization dynamics. Of course the experiments proves this, but it seems like a general benefits of ensemble models. At least, some explanation supported by experiments is needed to fully understand this method — especially this method is already very simple and not much novel, understanding this better is important to have sufficient technical contribution.
- [Major] Baselines missing. This method is an ensemble method. At least we could compare it to other naive ensemble method. For instance, how much speed up it provides compared to normal ensemble; how robust it compared to normal ensemble; etc.
- [Major] Datasets missing. Yes this paper test on many datasets — but the more important thing is the diversity. For instance, how about high-dimensional datasets and large datasets (I know it is on Table 2, but only evaluated on two datasets and three baselines, moreover, only TabMmini has been evaluated). I suggest the authors to use TabZilla hard benchmark, and then evaluate the method’s capability under different circumstances. These kind of experiments can make the findings more interesting.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731106584325

thread_type:
official_review

<thread_object>
creator: reviewer_crvx

<content>
summary:
The authors propose TabM, a variation of an MLP with $k$ heads that uses a shared backbone, and only certain aspects of the architecture in the beginning  and in the end are specialized. The heads are trained independently. The work is compared with prior DL methods (plain and attention architectures), retrieval-based methods, and tree-based methods on a diverse collection of classification and regression tasks. Based on the results, the proposed method manages to outperform all baselines with tuned hyperparameters.

strengths:
- The proposed method achieves better performance compared to the other methods with tuned hyperparameters. 
- The set of considered baselines is extensive.
- Extensive results on classification/regression datasets. Additionally, results are provided on well-known benchmarks in the domain.
- The authors provide extensive ablations of the different components of the method. The authors additionally provide a time comparison of the different methods.

weaknesses:
- The work is difficult to read, the captions of Figure 2 and Figure 3 are long and blend with the core manuscript text, making it hard to keep track of the manuscript flow.

questions:
- Table 2, for the MLP (Maps Routing dataset) the time unit is missing. I am guessing it is 10 minutes.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732230237411

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global reply

comment:
We thank the reviewers for their feedback and suggestions. In this post, we summarize the global context for further discussion:

1. **A summary of the positive comments** from the reviews.
2. **The big picture** as an easy reference for the discussion.

The PDF received a minor update to address small issues mentioned in the reviews. Plus, we added Figure 9 -- an additional illustration of the model.

## Positive things

- **Performance/effectiveness of the proposed method**
    - `CrVx`: *"The proposed method achieves better performance compared to the other methods ..."*
    - `4UNA`: *"The method is simple yet effective"*
    - `For5`: *"TabM demonstrates competitive performance ... efficient alternative for tabular data modeling"*
    - `aGcY`: *"The empirical results are quite strong"*
- **Model design**
    - `CrVx`: *"extensive ablations ... time comparison ..."*
    - `For5`: *"thorough analysis of each component's impact on model performance, contributing valuable insights."*
    - `aGcY`: *"interesting ablations"*
- **Empirical evaluation**
    - `CrVx`: *"The set of considered baselines is extensive ... extensive results on classification/regression datasets ... Additionally, results are provided on well-known benchmarks in the domain."*
    - `aGcY`: *"The empirical evaluation is broad and in-depth ... broad variety of baselines"*
- **Presentation**
    - `4UNA`: *"The paper is easy to read"*
    - `For5`: *"The paper is well-structured and clearly written, making the methods, results, and analysis easy to understand."*
    - `aGcY`: *"The empirical evaluation is ... well presented"*

## Big picture

It seems that the reviews do not fully agree on things like technical novelty and contribution scores. To find the common ground, we briefly summarize the big picture:

- `(A)` Our study identifies an overlooked opportunity to make a significant positive impact in the field of tabular DL architectures.
- `(B)` The technical context allows us to make this impact a reality without making a conceptual breakthrough, but we still *do* need to explore and identify the essential technical elements and ideas (see the details below).
- `(C)` The significant change in the performance-efficiency-simplicity coordinates, caused by TabM, *along with the issues described in the second paragraph of Introduction*, motivates us to perform a detailed reevaluation of tabular DL architectures. This gives a fresh and informative perspective on the modern state of the field and can have its own methodological impact on future studies (see details below).
- `(D)` We mainly focus on the above points. While we provide additional analysis, that we subjectively find interesting and important, we do not aim to explore all properties related to the ensemble-like nature of TabM in this one paper.

---

Additional details on the above points:

`(A)`
- The balance of task performance, simplicity, efficiency, and scalability, provided by TabM, is _not_ available in tabular DL architectures without TabM.
- Parameter-efficient ensembles have been known for many years, yet there has been no successful adaptation of this paradigm for tabular DL. Our study identifies and closes this gap.

`(B)` Examples of technical elements:

- Efficient ensembling
    - Seeing BatchEnsemble as a set of adapters that can be turned on/off.
    - The idea of "minimal" ensembling with only _one_ adapter per model compared to O(N) adapters for methods like BatchEnsemble and FiLM-Ensemble (N is the number of modified layers).
    - The idea of "safe" initialization of BatchEnsemble, directly inspired by the previous point.
- Tabular DL
    - The finding that the very first adapter is important for tabular models.

`(C)` In the story, this point is reflected in the abstract (*"renders the landscape of tabular DL in a new light"*) and contributions (*"fresh perspective on tabular DL"*). To give an in-depth perspective on results, we experiment with what to measure and how to present it for a given set of models and datasets. Examples:

- The fact that a sophisticated model performs no better or worse than plain MLP on 25% of tasks can significantly affect its positioning. In that regard, the middle and right parts of Figure 4 allow reasoning about the *weakest* parts of performance profiles. Namely, using the lower quantiles and individual data points, it is possible to see how *reliable* a model is on "inconvenient" datasets, how frequent and significant are its *failure modes*, etc.
- Another big factor is the efficiency-related properties of models. For example, when the task performance metrics from Figure 4 are combined with efficiency metrics from Figure 5, attention-based and retrieval-based models start looking less practical.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732797514443

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
To all reviewers (new revision)

comment:
We thank the reviewers for taking part in the discussion!

**We have uploaded a new revision of the paper:**

- For the reviewer `CrVx`: we replaced the *two* model-related figures with *one* new figure, and placed it *at the top of the page*, which hopefully improves the presentation.
- For the reviewer `4UNA`, we extended the *analysis*:
    - We added Section A.2 with *analysis on the sources of TabM's power.*
    - We improved the text of the *analysis of the gradient diversity* in Section A.3 (previously, this section was located in a different place in Appendix).

P.S. The above changes could affect the figure and section enumerations used previously in the discussion.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors propose TabM an efficient ensembling architecture that uses MLPs as its backbone, where the main weight matrix of a linear layer is shared between the ensemble members, and only member-specific adapters are learned. The proposed method outperforms traditional ensembles of neural networks and it offers a faster runtime compared to the plain ensemble counterpart. 

The experimental protocol features a comparison with an extensive number of baselines/datasets where the proposed method achieves the best overall performance. However, in terms of novelty, the proposed approach is somehow limited.

Initially, the reviewers had concerns regarding the efficiency of the proposed method and the complexity of the datasets included in the experimental protocol. However, the authors provided a thorough rebuttal. After the rebuttal the reviewers agree that the proposed method has a strong performance and it includes an extensive experimental protocol, yielding valuable results that would be helpful to the community.

I agree with the reviewers that the novelty of the proposed method is limited, however, the proposed work provides valuable insights and a strong baseline in the realm of tabular data. Based on the previous points, I recommend acceptance.