Mambular: A Sequential Model for Tabular Deep Learning

Authors: Anton Frederik Thielmann, Manish Kumar, Christoph Weisser, Arik Reuter, Benjamin Säfken, Soheila Samiee
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: Tabular Deep Learning, Mamba, Sequential Models, SSM, Recurrent Neural Networks

Abstract:
The analysis of tabular data has traditionally been dominated by gradient-boosted decision trees (GBDTs), known for their proficiency with mixed categorical and numerical features. However, recent deep learning innovations are challenging this dominance. We introduce Mambular, an adaptation of the Mamba architecture optimized for tabular data. We extensively benchmark Mambular against state-of-the-art models, including neural networks and tree-based methods, and demonstrate its competitive performance across diverse datasets.
Additionally, we explore various adaptations of Mambular to understand its effectiveness for tabular data. We investigate different pooling strategies, feature interaction mechanisms, and bi-directional processing. Our analysis shows that interpreting features as a sequence and passing them through Mamba layers results in surprisingly performant models.  The results highlight Mambular’s potential as a versatile and powerful architecture for tabular data analysis, expanding the scope of deep learning applications in this domain.
 The source code is available at: https://anonymous.4open.science/r/mamba-tabular-485F/

Decision: reject
Reviewer scores: 3, 3, 8, 3

Reviews and discussion:
<thread>
creation date:
1730339014414

thread_type:
official_review

<thread_object>
creator: reviewer_na6p

<content>
summary:
The paper introduces Mambular, an adaptation of the Mamba architecture specifically designed for tabular data problems. The authors claim that their model leverages a sequential interpretation of tabular data, similar to its successful application in other domains such as text, vision, and time series. The proposed architecture incorporates pooling strategies and feature interaction mechanisms to enhance performance. The paper provides experimental results on various datasets, comparing Mambular with state-of-the-art models such as FT-Transformer, TabTransformer, XGBoost, and LightGBM. According to the authors, Mambular achieves competitive results, highlighting the feasibility of applying sequential models to tabular data.

strengths:
The paper explores the application of the Mamba architecture to tabular data, leveraging lessons from its success in handling other data types like text and time series. While not entirely novel, this extension demonstrates an attempt to generalize Mamba to a new domain.

The authors present experimental results comparing Mambular with a limited selection of state-of-the-art models, such as FT-Transformer, TabTransformer, XGBoost, and LightGBM. While the comparisons are not exhaustive, these initial results provide some insight into the model’s potential.

The architecture incorporates feature interaction mechanisms and various pooling strategies, potentially providing flexibility to handle different types of tabular data characteristics.

weaknesses:
- Lack of novelty: One of the main weaknesses is the lack of significant novelty. The concept of adapting Mamba for tabular data is not substantially different from previously proposed models like MambaTab [1]. The minor modifications presented in Mambular mainly involve hyperparameter tuning and slight architectural changes, which do not justify the need for a new model.

[1] MA Ahamed et al., MambaTab: A Plug-and-Play Model for Learning Tabular Data, MIPR, 2024.

- Misalignment with tabular data characteristics: The paper fails to convincingly justify why treating tabular data as sequential offers any distinct advantage. Unlike sequential data such as text or time series, tabular data lacks a natural ordering of features. The authors did not provide clear evidence or theoretical grounding for why a sequence-based model should work better in this context.

- Limited scope of experiments: The experimental results are based on only 12 datasets, and the comparison to state-of-the-art algorithms is not comprehensive. Given the varied nature of tabular data, a broader set of datasets and comparisons to more established tabular models would have strengthened the claims.

- Inconsistent Performance: Despite the claims of Mambular’s superiority, the reported results do not consistently show a significant performance improvement over existing methods, such as XGBoost or LightGBM.

questions:
What is the rationale behind interpreting tabular data as sequences? Could the authors provide a theoretical explanation or empirical evidence showing that treating features as a sequence offers a distinct advantage?

How do the authors address the lack of consistency in performance improvements across the datasets? Were any specific types of datasets or characteristics where Mambular performed particularly well?

Could the authors explain why comparisons were not made with a wider range of established tabular models, such as TabR[2], T2G-Former[3], and SAINT[4]?
[2] TabR: Unlocking the Power of Retrieval-Augmented Tabular Deep Learning, 2023
[3] T2G-Former: Organizing Tabular Features into Relation Graphs Promotes Heterogeneous Feature Interaction, 2023
[4] SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training, 2021
</content>
</thread_object>

</thread>
<thread>
creation date:
1730540401251

thread_type:
official_review

<thread_object>
creator: reviewer_7hmn

<content>
summary:
This paper presents Mambular, a sequential deep learning model based on the Mamba architecture, designed for tabular data tasks. The authors evaluate Mambular against state-of-the-art neural and (ensemble) tree-based models, showing that it performs as well as or better than these models across different datasets.

strengths:
- A novel sequential approach to tabular data, treating features as ordered sequences to capture feature interactions. 

- Solid benchmarks against state-of-the-art models confirm its competitive performance across diverse datasets.

- Paper is clearly written and easy to follow,  the model and its components are clearly explained, and the comprehensive ablation study provides insight into the impact of different architectural choices.

weaknesses:
- The paper would benefit from a more extensive comparison with recent deep learning baselines for tabular data. While it presents strong benchmarks, adding models like TabPFN and GANDALF could enhance the evaluation, giving a clearer picture of Mambular’s performance against the latest advancements. 
- Additionally, thorough hyperparameter tuning is necessary for many tabular learning models to ensure fair and optimal comparisons. Please refer to Questions.

questions:
- Hyperparameter Tuning and Model Performance: In Section 5, the authors claim that hyperparameter tuning does not significantly impact model ranking, citing Grinsztajn et al. (2022) and Gorishniy et al. (2021). I find this very questionable, especially given that Table 3 shows Random Forest outperforming XGBoost, which is typically strong for tabular data (Also, see Appendix tables here [1]).  
Additionally, the model’s sensitivity to kernel size in permutation experiments suggests that hyperparameter choices affect performance significantly. To address this, I suggest comparing models using an established benchmark such as TabZilla [2], which offers a diverse suite of datasets for evaluating tabular models. 

- The Mambular model’s sequential structure makes it highly sensitive to feature order, which impacts performance under certain hyperparameter settings. Although feature order dependency is a known issue in tabular data models, I recommend that the authors conduct an experiment with random feature permutations (see Definition 2 in [3]). This could enable Mambular to emphasize dependencies rather than sequence, potentially improving model's robustness.



[1] https://arxiv.org/abs/2305.13072

[2] https://github.com/naszilla/tabzilla

[3] Borisov, V., Seßler, K., Leemann, T., Pawelczyk, M., & Kasneci, G. (2022). "Language models are realistic tabular data generators." arXiv preprint arXiv:2210.06280.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730647728286

thread_type:
official_review

<thread_object>
creator: reviewer_2wj3

<content>
summary:
The authors introduced Mamba architecture to solve the problems of tabular data. They showed the effectiveness of the proposed method by comparing its performance with neural networks-based algorithms and tree-based methods. And they confirmed the superiority of a sequence and passing method by comparing various pooling methods.

strengths:
The authors proposed the Mamba architecture to solve tabular problems. It showed good performance overall, especially in classification performance.

weaknesses:
The proposed method seems to be nothing more than using the Mamba structure on tabular data. Although there are some structural suggestions, it is difficult to say that the idea is novel overall.
And when I check the experimental results, it seems difficult to say that the performance has been greatly improved.Additional discussions and experiments are needed to see what advantages can be taken advantage of by applying the Mamba structure to tabular data, what differential performance improvements can be made compared to existing methods, and what can be considered if a Mamba structure or a new structure based on Mamba is proposed in the future.

questions:
Please check the above weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731069227928

thread_type:
official_review

<thread_object>
creator: reviewer_tnbh

<content>
summary:
The authors propose Mambular, an adaption of the Mamba architecture for tabular data. The authors compare Mambular against various well-known model families in the tabular domain, in both regression and classification tasks, showcasing the competitive performance of the proposed method. The authors additionally perform a series of ablations to provide insights on several design choices.

strengths:
- The work is well written.
- The work ablates several design choices of the proposed method.
- The proposed method achieves competitive performance compared to the considered baselines. 
- The work considers both regression and classification tasks.

weaknesses:
- The related work section misses core references and can be further strengthened. [1][2]
- Line 279, I would rather the authors advocated that the results are not significant, rather than considering a 10% significance level. The standard is a 5% significance level.
- The authors do not perform hyperparameter tuning.
- It is not clear how the set of defaults for all methods is devised.
- The number of datasets considered is limited. Additionally, the authors do not use well-established benchmarks from the community. [3][4]
- A detailed analysis regarding time is not provided to have a clear understanding of the pros and cons of different methods.

[1] Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., & Gulin, A. (2018). CatBoost: unbiased boosting with categorical features. Advances in neural information processing systems, 31.

[2] Kadra, A., Lindauer, M., Hutter, F., & Grabocka, J. (2021). Well-tuned simple nets excel on tabular datasets. Advances in neural information processing systems, 34, 23928-23941.

[3] Gijsbers, P., Bueno, M. L., Coors, S., LeDell, E., Poirier, S., Thomas, J., ... & Vanschoren, J. (2024). Amlb: an automl benchmark. Journal of Machine Learning Research, 25(101), 1-65.

[4] Grinsztajn, L., Oyallon, E., & Varoquaux, G. (2022). Why do tree-based models still outperform deep learning on typical tabular data?. Advances in neural information processing systems, 35, 507-520.

questions:
- Line 47, missing related work [1].
- Line 252, missing related work [2].
- Line 264, how was the set of defaults devised? Do the authors use the defaults of the baselines from the corresponding papers? 
- Line 271, how were the datasets selected?
- In terms of baselines, why was CatBoost not included? Given that it handles categorical variables natively? It is additionally among the best performing methods from the gradient-boosted decision tree family.
- Could the authors apply significance tests over the entire suite of datasets and not on a per-dataset basis? The mean value could be used to aggregate the multiple runs on a single dataset. Additionally, could all of the methods be considered and a critical difference diagram be provided?
- Line 297, the results do not align with the referenced work, in particular in [3], (Table 1, Table 2, Figure 3) the FT-Transformer architecture lags behind the ResNet architecture in terms of performance. Additionally, as I previously mentioned, CatBoost is the top performing method, which the authors do not include in the comparisons.
- A comparison with TabPFN for datasets that fit the limitations of the method would be interesting.
- What is the runtime of the proposed method compared to the baselines? What about the inference time?

[1] Kadra, A., Lindauer, M., Hutter, F., & Grabocka, J. (2021). Well-tuned simple nets excel on tabular datasets. Advances in neural information processing systems, 34, 23928-23941.

[2] Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., & Gulin, A. (2018). CatBoost: unbiased boosting with categorical features. Advances in neural information processing systems, 31.

[3] McElfresh, D., Khandagale, S., Valverde, J., Prasad C, V., Ramakrishnan, G., Goldblum, M., & White, C. (2024). When do neural nets outperform boosted trees on tabular data?. Advances in Neural Information Processing Systems, 36.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732097269855

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
Dear Reviewers,

We would like to extend our sincere gratitude for your careful consideration of our work. After reviewing all feedback, we observed several consistent points that we would like to address in a joint response. Overall, we were somewhat surprised by the low scores, especially since the primary critique from most reviewers focuses on a single aspects of our contribution.

## Benchmarks
Nearly all reviewers have suggested expanding the benchmarks, citing studies such as [1, 2, 3]. While additional experiments would indeed strengthen any study, we respectfully urge you to consider the following points:

- The referenced studies [1, 2, 3] are large-scale benchmark studies, some involving over 150 datasets. These benchmarks require extensive hyperparameter optimization and k-fold cross-validation. Given our computational resources, reproducing such an extensive benchmark is infeasible, particularly for deep learning (DL) models on tabular data, which demand considerable computational power. In the case of [2], for example, even a WandB subscription is required. This presents a barrier in research publication, where access to substantial compute resources has become an unspoken prerequisite. We find that the pressure for "more experiments" has become a generic critique in ML research, while core aspects such as reproducibility often go overlooked. While we cannot feasibly match the requested benchmark scale, our codebase is comprehensive, and we provide detailed configuration files for all models to ensure full reproducibility.

- To still provide a robust experimental evaluation, we conducted a benchmark using datasets sourced from [OpenML](https://www.openml.org/search?type=study&study_type=task&sort=tasks_included&id=353) and presented in [4], with missing values removed and datasets with fewer than 1000 observations omitted. Currently, we have run the best models from our manuscript as well as the suggested Catboost [6] and achieved the results detailed below. Catboost does indeed outperform XGboost, contrary to our expectations given results from [3] and [5]. Catboost performs on average, over all datasets practially identical to Mambular.
  




| Model             |      BH ↓  |    CW ↓   |   FF ↓   |     GS ↓  |       HI ↓ |   K8 ↓   |     AV ↓  |      KC ↓  |     MH ↓  |       NP ↓ |      PP ↓ |   SA ↓   |   SG ↓     |       VT ↓| Rank ↓    |
|:------------------|-----------:|----------:|---------:|----------:|-----------:|---------:|----------:|-----------:|----------:|-----------:|----------:|---------:|-----------:|----------:|----------:|
| Mambular          |  **0.021** | **0.701** |  0.272   |     0.057 |  **0.595** |    0.168 |     0.018 |      0.137 |     0.085 |  **0.003** |     0.402 |**0.015** |      0.318 | **0.003** | **1.79**  |
| FTTransformer     |      0.028 |     0.701 |  0.301   |     0.205 |      0.609 |    0.451 |     0.089 |      0.149 |     0.101 |      0.009 |     0.542 |    0.033 |      0.36  |     0.045 |     4.36  |
| CatBoost          |      0.032 |     0.702 |**0.245** | **0.041** |      0.597 |**0.150** |     0.004 |  **0.110** | **0.078** |      0.005 | **0.39**  |    0.018 |  **0.297** |     0.013 | **1.79**  |
| LightGBM          |      0.048 |     0.707 |  0.263   |     0.059 |      0.599 |    0.239 |     0.024 |      0.140 |     0.091 |      0.009 |     0.452 |    0.031 |      0.302 |     0.013 |     3.26  |
| XGBoost           |      0.039 |     0.752 |  0.281   |     0.078 |      0.635 |    0.259 | **0.004** |      0.161 |     0.098 |      0.006 |     0.403 |    0.024 |      0.329 |     0.013 |     3.71  |


We plan to incorporate additional models and benchmarks gradually. However, given our computational constraints, we performed a single train-test-validation split for this evaluation. All necessary files to reproduce these benchmarks are available in our codebase.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors apply the Mamba architecture to tabular data. The proposed technique uses memory more efficiently than transformer-based models and performs about as well as the selected baselines.

However, reviewers expressed concerns that evaluation is limited. In addition, no thorough hyperparameter tuning is performed, and the number of datasets and baselines is small. The authors do not exhaustively explain why a sequential model would be a good fit for tabular data in the first place.

Overall the work lacks novelty, the analysis is not sufficiently detailed, and the experimental protocol is sub-optimal (in terms of datasets, and baselines). Although the authors added a few more results in their rebuttal, these new findings did not change the essence of the quality of the manuscript.

I suggest rejecting this submission for now and encouraging the authors to incorporate the reviewers’ feedback into a future version.