Mambular: A Sequential Model for Tabular Deep Learning

## Abstract

Abstract The analysis of tabular data has traditionally been dominated by gradient-boosted decision trees (GBDTs), known for their proficiency with mixed categorical and numerical features. However, recent deep learning innovations are challenging this dominance. We introduce Mambular, an adaptation of the Mamba architecture optimized for tabular data. We extensively benchmark Mambular against state-of-the-art models, including neural networks and tree-based methods, and demonstrate its competitive performance across diverse datasets.
Additionally, we explore various adaptations of Mambular to understand its effectiveness for tabular data. We investigate different pooling strategies, feature interaction mechanisms, and bi-directional processing. Our analysis shows that interpreting features as a sequence and passing them through Mamba layers results in surprisingly performant models. The results highlight Mambular’s potential as a versatile and powerful architecture for tabular data analysis, expanding the scope of deep learning applications in this domain.
The source code is available at https://github.com/basf/mamba-tabular .

## 1 Introduction

Gradient-boosted decision trees (GBDTs) have historically dominated the field of tabular data analysis, with widely used variants including XGBoost, LightGBM, and CatBoost. These models excel in managing the mix of categorical and numerical features typical of tabular data (Grinsztajn et al.,, ). Historically, deep learning models struggled with tabular data due to its inherent complexity and diversity, often failing to outperform GBDTs due to challenges in handling missing values, diverse feature types, and the necessity for extensive preprocessing (Borisov et al.,, ). However, recent advancements in deep learning are gradually challenging this paradigm by introducing innovative architectures that leverage advanced mechanisms to capture complex feature dependencies, promising significant improvements (Popov et al.,, ; Hollmann et al.,, ; Gorishniy et al.,, ).

One of the most effective advancements in tabular deep learning is the application of attention mechanisms in models like TabTransformer (Huang et al.,, ), FT-Transformer (Gorishniy et al.,, ) and many more (Wang and Sun,, ; ; Arik and Pfister,, ). These models leverage the attention mechanism to capture dependencies between features, offering a significant improvement over traditional approaches. FT-Transformers, in particular, have demonstrated robust performance across various tabular datasets, often surpassing the accuracy of GBDTs (McElfresh et al.,, ).

Additionally, more traditional models like Multi-Layer Perceptrons (MLPs) and ResNets have demonstrated improvements when well-designed and when the data undergoes thorough preprocessing (Gorishniy et al.,, , ). These models have benefited especially from innovations in advanced preprocessing methods that make them more competitive.

More recently, the Mamba architecture (Gu and Dao,, ) has shown promising results in textual problems. Tasks previously dominated by Transformer architectures, such as DNA modeling and large language models (LLMs), have seen improvements with the application of Mamba models (Gu and Dao,, ; Schiff et al.,, ; Zhao et al.,, ).

Several adaptations have demonstrated its versatility, such as Vision Mamba for image classification (Xu et al.,, ), video analysis (Yang et al.,, ; Yue and Li,, ) and point cloud analysis (Zhang et al.,, ; Liu et al.,, ). Additionally, the architecture has been adapted for time series problems, with notable successes reported by Patro and Agneeswaran, (), Wang et al., () and .
Mamba has also been integrated into graph learning (Behrouz and Hashemi,, ) and imitation learning (Correia and Alexandre,, ).
Further advancements have improved the language model, by e.g. incorporating attention (Lieber et al.,, ), Mixture of Experts (Pióro et al.,, ) or bi-directional sequence processing (Liang et al.,, ).

These advancements underscore Mamba’s broad applicability, making it a powerful and flexible architecture for diverse tasks and data types. Similarly to the transformer architecture, the question arises whether the Mamba architecture can also be leveraged for tabular problems.
While has already demonstrated that the Mamba architecture can be used for tabular data, a deeper analysis regarding the model architecture as well as insights into optimizations for tabular datasets is needed.

The contributions of the paper can be summarized as follows:

- I.
We present Mambular, a tabular adaptation of Mamba, and showcase the applicability of sequential models to tabular problems.
- II.
Mambular is extensively benchmarked against several other competitive neural as well as tree-based methods, showcasing that a default Mambular model performs on par with or better than tree-based models across a wide range of datasets.
- III.
We analyze the effects of bi-directional processing as well as feature interaction layers on Mambular’s performance and compare classical pooling methods.
- IV.
Lastly, we conduct a thorough analysis of the sequential nature of Mambular, exploring the effects of feature orderings in a sequential tabular model.

## 2 Methodology

For a tabular problem, let $\mathcal{D}=\{(\bm{x}^{(i)},y^{(i)})\}_{i=1}^{n}$ be the training dataset of size $n$ and let $y$ denote the target variable that can be arbitrarily distributed. Each input $\bm{x}=(x_{1},x_{2},\dots,x_{J})$ contains $J$ features (variables).
Categorical and numerical features are distinguished as $\bm{x}\equiv(\bm{x}_{cat},\bm{x}_{num})$, with the complete feature vector denoted as $\bm{x}$.
Further, let $x_{j(cat)}^{(i)}$ denote the $j$-th categorical feature of the $i$-th observation, and hence $x_{j(num)}^{(i)}$ denote the $j$-th numerical feature of the $i$-th observation.

The core of the Mambular architecture is a combination of the FT-Transformer (Gorishniy et al.,, ) and Mamba (Gu and Dao,, ).
Following classical tabular transformer architectures, the categorical features are first encoded and embedded. In contrast to classical language models, each categorical feature has its own, distinct vocabulary to avoid problems with binary or integer encoded variables. Including $<$unk$>$ tokens additionally allows to easily deal with unknown or missing categorical values during training or inference.

Numerical features are mapped to the embedding space via a simple linear layer. However, since a single linear layer does not add information beyond a linear transformation, Periodic Linear Encodings, as introduced by Gorishniy et al., (), are used for all numerical features. Thus, each numerical feature is encoded before being passed through the linear layer for rescaling. Simple decision trees are used for detecting the bin boundaries, and depending on the task, either classification or regression is employed for the target-dependent encoding function $h_{j}(\bm{x}_{j(num)},y)$. Let $b_{t}$ denote the decision boundaries from the decision trees, and the encoding function is given in Eq. 1.

PLE

$$ $z_{j(\text{num})}^{t}=\begin{cases}0&\text{if }x<b_{t-1},\\ 1&\text{if }x\geq b_{t},\\ \frac{x-b_{t-1}}{b_{t-2}-b_{t-1}}&\text{else.}\end{cases}$ (1) $$

The feature encoding and embedding generation is demonstrated in Figure 1. The created embeddings, following classical statistical literature (Hastie et al.,, ; Kneib et al.,, ) are denoted as $\mathbf{Z}$ and not $\mathbf{X}$ to clarify the difference between the embeddings and the raw features.

Figure: Figure 1: Generation of the input matrix that are fed through the Mamba blocks. The categorical features are tokenized and embedded similar to classical embeddings for language models. The numerical features are encoded and embedded via a simple linear layer. The final input matrix of the Mamba blocks are the concatenated embeddings $\mathbf{z}\in\mathbb{R}^{N\times J\times d}$ with embedding dimension $d$.

Subsequently, the embeddings are passed jointly through a stack of Mamba layers. These include one-dimensional convolutional layers as well as a state-space (SSM) model (Gu et al.,, ; Hamilton,, ) . The feature matrix before being passed through the SSM model has a shape of (Batch Size) $\times$ J $\times$ (Embedding Dimension), later referenced as $N\times J\times d$. Importantly, the sequence length in a tabular context refers to the number of variables, and hence the second dimension, $J$, corresponds to the number of features rather than to the length of, e.g., a document. Given matrices:

$$ $\mathbf{A}\in\mathbb{R}^{1\times 1\times d\times\delta},\quad\mathbf{B}\in \mathbb{R}^{N\times J\times 1\times\delta},\quad\Delta\in\mathbb{R}^{N\times J \times d\times 1},\bar{\mathbf{z}}\in\mathbb{R}^{N\times J\times d\times 1},$ $$

where $\delta$ denotes a inner dimension, similar to the feed forward dimension in Transformer architectures and $\bar{\mathbf{z}}$ has the same entries as $\mathbf{z}$, but one additional axis, the formula for updating the hidden state $\mathbf{h}_{j}\in\mathbb{R}^{N\times d\times\delta}$ is:

$$ $\mathbf{h}_{j}=\exp\left(\Delta\odot_{3}\mathbf{A}\right)_{:,j,:,:}\odot_{1,2, 3}\mathbf{h}_{j-1}+\left(\left(\Delta\odot_{1,2}\mathbf{B}\right)\odot_{1,2,3} \bar{\mathbf{z}}\right)_{:,j,:,:}.$ (2) $$

The symbol $\odot_{d}$ denotes an outer product where the multiplication is done for the $d$-th axis and parallelized wherever a singleton axis length meets an axis of length one(^1^11This corresponds to using the ordinary multiplication operator ”*” in PyTorch and relying on the default broadcasting). The exponential function is applied element-wise.
The state transition matrix $\mathbf{A}$ governs the transformation of the hidden state from the previous time step to the current one, capturing how the hidden states evolve independently of the input features. The input-feature matrix $\mathbf{B}$ maps the input features to the hidden state space, determining how each feature influences the hidden state at each step. The gating matrix $\mathbf{\Delta}$ acts as a gating mechanism, modulating the contributions of the state transition and input-feature matrices, and allowing the model to control the extent to which the previous state and the current input affect the current hidden state.

Figure: Figure 2: SSM updating step with recursive update of $h$: The hidden state is iteratively updated by going through the sequence (features) similar to a recurrent neural network. The final representation is generated as described in Equations 3-4.

In contrast to FT-Transformer (Gorishniy et al.,, ) and TabTransformer (Huang et al.,, ) Mambular truly iterates through all variables as if they are a sequence. Hence, feature interactions are detected sequentially; and the effect of feature position in a sequence is analyzed with respect to performance in section 4.
Furthermore, it should be noted that in contrast to TabPFN (Hollmann et al.,, ), Mambular does not transpose dimensions and iterates over observations. Hence, training on large datasets is possible and it can scale well to any training data size, just as Mamba (Gu and Dao,, ) does.

After stacking and further processing, the final representation, $\tilde{\textbf{x}}\in\mathbb{R}^{N\times J\times d}$ is retrieved. In truly sequential data, these are the contextualized embeddings of the input tokens, for tabular problems $\tilde{\textbf{x}}$ represents a contextualized, or feature interaction accounting variable representation, in the embedding space.
The hidden states are stacked along the sequence dimension to form:

$$ $\mathbf{H}=[\mathbf{h}_{0},\mathbf{h}_{1},\ldots,\mathbf{h}_{T-1}]\in\mathbb{R }^{N\times J\times d\times\delta}.$ $$

The final output representation $\tilde{\mathbf{x}}$ is then computed by performing matrix multiplication of the stacked hidden states with matrix $\mathbf{C}\ \in\mathbb{R}^{N\times J\times 1\times\delta}$ where the multiplication and summation is done over the last axis, and adding the vector $\mathbf{\alpha}\ \in\mathbb{R}^{1\times 1\times d}$ scaled by the input $\mathbf{z}$:

$$ $\tilde{\mathbf{x}}=\left(\mathbf{H}\cdot_{4}\mathbf{C}\right)+\left(\mathbf{ \alpha}\odot_{3}\mathbf{z}\right).$ (3) $$

More explicitly, this can be written as:

$$ $\tilde{x}_{i,j,k}=\sum_{\delta}\mathbf{H}_{i,j,k,\delta}\mathbf{C}_{i,j,1, \delta}+\mathbf{\alpha}_{1,1,k}\mathbf{z}_{i,j,k}.$ $$

where $\mathbf{C}$ and $\mathbf{\alpha}$ are learnable parameters.
For final processing, $\tilde{\mathbf{x}}$ is element-wise multiplied with $\mathbf{z}^{\prime}$, and the result is passed through a final linear layer:

$$ $\tilde{\mathbf{x}}_{\text{final}}=(\tilde{\mathbf{x}}\odot_{1,2,3}\mathbf{z^{ \prime}})\mathbf{W}_{\text{final}}+\mathbf{b}_{\text{final}}.$ (4) $$

Before it is passed to the final task specific model head, several pooling techniques along the sequence axis are possible and tested, e.g. sum pooling, average pooling or simple stacking.

We conduct several experiments with [cls] tokens, and analyze the model’s performance when only passing the [cls] token to the task-specific head. Given the recurrent update of the hidden states when iterating over the variables, variable positioning was also analyzed, since the order of the variables could affect this sequential setup.
The model is trained end-to-end by minimizing the task-specific loss, e.g., mean squared error for regression or categorical cross entropy for classification tasks. An overview of a forward pass in the model is given in Figure 2.

Figure: Figure 3: The forward pass of a single sequence in the model. After embedding the inputs, the embeddings are passed to several Mamba blocks. The tabular head is a single task specific output layer. Before being passed to the Linear Layer, the contextualized embeddings are pooled via average pooling. For bidirectional processing a second block with a flipped sequence is used and the learnable matrices are not shared between the directions.

## 3 Experiments

Mambular is benchmarked across a variety of datasets against the currently top-performing models (McElfresh et al.,, ). FT-Transformer (Gorishniy et al.,, ), TabTransformer (Huang et al.,, ), the consistently well-performing XGBoost (Grinsztajn et al.,, ; McElfresh et al.,, ), a baseline Multi-Layer Perceptron, and a ResNet are the models which contributed in this benchmark. Note, that TabPFN (Hollmann et al.,, ) is not included because it is not suited for larger datasets.

We perform 5-fold cross-validation for all datasets and report average results as well as standard deviations. For all neural models, PLE encodings (Eq. 1) with a maximum number of bins equal to the model dimension are used (128 for most models, including MLP and ResNet). All categorical features are integer-encoded. For the regression tasks, the targets are normalized. Mean Squared Error (MSE) and Area Under the Curve (AUC) statistics are reported for regression and classification tasks, respectively. TabTransformer, FT-Transformer, and Mambular use the same architectures for embeddings as well as for the task-specific head, which consists of a single output layer without activation or dropout. For FT-Transformer, the [cls] token embedding for final prediction is used since it has been shown to improve performance (; Gorishniy et al.,, ).

Several shared parameters are used for all neural models: a starting learning rate of 1e-04, weight decay of 1e-06, an early stopping patience of 15 epochs with respect to the validation loss, a maximum of 200 epochs for training, and a small learning rate decay with a factor of 0.1 with a patience of 10 epochs with respect to the validation loss. Additionally, a universal batch size of 128 is used, and the best model with respect to the validation loss is returned for testing. For TabTransformer, FT-Transformer, and Mambular, the same embedding functions are used. For the benchmarks, a vanilla Mambular architecture is used, utilizing average pooling, no feature interaction layer, and no bi-directional processing. Additionally, the columns/sequence are always sorted with numerical features first, followed by categorical features. Within these two groups, the features are sorted as they were originally provided in the dataset from the UCI Machine Learning Repository. Details on the used datasets and preprocessing can be found in Appendix A. Details on the model architectures and hyperparameters can be found in Appendix C.

#### MambaTab

In addition to these popular tabular models, we tested out the architecture proposed by . MambaTab is the first architecture to leverage Mamba blocks for tabular problems. However, the authors propose using a combined linear layer to project all inputs into a single feature representation, transforming the features into a pseudo-sequence of fixed length 1. This approach simplifies the recursive update from Eq. 2 into a matrix multiplication and makes the model resemble a ResNet due to the residual connections in the final processing. Utilizing a sequential model with a sequence length of 1 does not fully exploit the strengths of sequential processing, as it reduces the model’s capacity to capture dependencies across multiple features.

We tested the architecture proposed by and could achieve similar results for shared datasets, but overall found MambaTab to perform similar to a ResNet, aligning with expectations (see Table 2 and 23). Additionally, we experimented with transposing the axes to create an input matrix of shape (1) $\times$ (Batch Size) $\times$ (Embedding Dimension), as outlined in their implementation. While this approach draws on ideas from TabPFN (Hollmann et al.,, ), it did not lead to performance improvements in our experiments. When using PLE encodings and increasing the number of layers and dimensions compared to the default implementation from we are able to increase performance. See Appendix C for a further discussion on MambaTab.

#### Comparison to XGBoost

Comparing mambular to XGBoost, we find that with default hyperparameter settings, Mambular performs as good, if not minimally better than XGBoost. On 12 datasets it outperforms XGBoost significantly for 4 datasets at the 10% significance level, while XGboost outperforms Mambular for 2 datasets at the 10 % significance level. The $p$-values from simple t-tests over the folds are reported for each dataset and the testing is oriented on Gorishniy et al., (). When adjusting for multiple testing via Benjamini-Hochberg (Ferreira and Zwinderman,, ; Benjamini and Hochberg,, ) the Abalone results - only significant at the 10% level with standard testing - are not significant anymore. All other results remain unchanged(^2^22Due to the small sample sizes, Benjamini-Hochberg is preferred to the conservative Bonferroni adjustments (Nakagawa,, ).).

**Table 1: Comparison between Mambular and XGBoost. The left tables shows regression results with average MSE values over 5 folds. The right side shows (binary) classification results with average AUC values. Significantly better values at the 5% significance level are in green and marked bold. Significantly better values at the 10% significance level are underscored. Dataset details can be found in appendix A. $\uparrow$ depicts higher is better and vice-versa.**
| Models | DI $\downarrow$ | AB $\downarrow$ | CA $\downarrow$ | WI $\downarrow$ | PA $\downarrow$ | HS $\downarrow$ | CP $\downarrow$ | BA $\uparrow$ | AD $\uparrow$ | CH $\uparrow$ | FI $\uparrow$ | MA $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Mambular | 0.018 | 0.452 | 0.167 | 0.628 | 0.035 | 0.132 | 0.027 | 0.927 | 0.928 | 0.856 | 0.795 | 0.917 |
| XGB | 0.019 | 0.506 | 0.171 | 0.528 | 0.036 | 0.119 | 0.024 | 0.928 | 0.929 | 0.845 | 0.774 | 0.922 |
| $p$-value | 0.0079 | 0.0870 | 0.4865 | 1.3e-07 | 0.6287 | 0.3991 | 0.1999 | 0.7883 | 0.7930 | 0.0192 | 0.0120 | 0.010 |

#### Results

Generally, we can confirm the strong results from the FT-Transformer architecture as introduced by Gorishniy et al., () and validated by McElfresh et al., (). Unsurprisingly, XGboost performs comparably well on all datasets but that is slightly outperformed by the FT-Transformer. Overall, Mambular performs best on average across all datasets among evaluated models. However, Mambular similarly struggles as the other neural methods compared to XGBoost for e.g. the Wine dataset.

**Table 2: Average Rank and standard deviation of the models over all datasets. The best rank is marked in bold. MambaTab refers to the model exactly as described by . MambaTab^T refers to transposing the axis and thus iterating over batch size. MambaTab^∗ refers to our implementation. See the details on MambaTab in the paragraph above.**
| Model | Avg. Rank |
| --- | --- |
| Mambular | 2.083 $\pm$ 1.037 |
| FT-Transformer | 2.417 $\pm$ 1.256 |
| XGBoost | 3.167 $\pm$ 2.577 |
| MambaTab^∗ | 4.333 $\pm$ 1.374 |
| ResNet | 4.750 $\pm$ 1.639 |
| TabTransformer | 6.222 $\pm$ 1.618 |
| MLP | 6.500 $\pm$ 1.500 |
| MambaTab | 6.583 $\pm$ 1.801 |
| MambaTab^T | 7.917 $\pm$ 1.187 |

The detailed results for all datasets and tasks are given below (see Table 23 and 24). The results for the original MambaTab implementation, along with a discussion of our results, can be found in Appendix C. Interestingly, for the tested datasets, all neural models perform considerably worse on the Wine dataset (see Appendix for the dataset description). Similarly, XGBoost performs worse than all neural models on the Abalone and FICO datasets. Additionally, our findings suggest that both FT-Transformer and Mambular perform very well on datasets with very few categorical features (e.g., FICO, California Housing, Abalone, CPU), although leveraging embeddings and structures originally designed for discrete data inputs.

**Table 3: Benchmarking results for the regression tasks. Average mean squared error values over 5 folds and the corresponding standard deviations are reported. Smaller values are better. The best performing model is marked in bold.**
| Models | DI $\downarrow$ | AB $\downarrow$ | CA $\downarrow$ | WI $\downarrow$ | PA $\downarrow$ | HS $\downarrow$ | CP $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| XGBoost | 0.019 $\pm$ 0.000 | 0.506 $\pm$ 0.044 | 0.171 $\pm$ 0.007 | 0.528 $\pm$ 0.008 | 0.036 $\pm$ 0.004 | 0.119 $\pm$ 0.024 | 0.024 $\pm$ 0.004 |
| FT-Transformer | 0.018 $\pm$ 0.001 | 0.458 $\pm$ 0.055 | 0.169 $\pm$ 0.006 | 0.615 $\pm$ 0.012 | 0.024 $\pm$ 0.005 | 0.111 $\pm$ 0.014 | 0.024 $\pm$ 0.001 |
| MLP | 0.066 $\pm$ 0.003 | 0.462 $\pm$ 0.051 | 0.198 $\pm$ 0.011 | 0.654 $\pm$ 0.013 | 0.764 $\pm$ 0.023 | 0.147 $\pm$ 0.017 | 0.031 $\pm$ 0.001 |
| TabTransformer | 0.065 $\pm$ 0.002 | 0.472 $\pm$ 0.057 | 0.247 $\pm$ 0.013 | - | 0.135 $\pm$ 0.001 | 0.160 $\pm$ 0.028 | - |
| ResNet | 0.039 $\pm$ 0.018 | 0.455 $\pm$ 0.045 | 0.178 $\pm$ 0.006 | 0.639 $\pm$ 0.013 | 0.606 $\pm$ 0.031 | 0.141 $\pm$ 0.017 | 0.030 $\pm$ 0.002 |
| MambaTab^∗ | 0.040 $\pm$ 0.008 | 0.455 $\pm$ 0.043 | 0.180 $\pm$ 0.008 | 0.601 $\pm$ 0.010 | 0.571 $\pm$ 0.021 | 0.122 $\pm$ 0.017 | 0.030 $\pm$ 0.002 |
| Mambular | 0.018 $\pm$ 0.000 | 0.452 $\pm$ 0.043 | 0.167 $\pm$ 0.011 | 0.628 $\pm$ 0.010 | 0.035 $\pm$ 0.005 | 0.132 $\pm$ 0.020 | 0.027 $\pm$ 0.002 |

**Table 4: Benchmarking results for the classification tasks. Average AUC values over 5 folds and the corresponding standard deviations are reported. Larger values are better.**
| Models | BA $\uparrow$ | AD $\uparrow$ | CH $\uparrow$ | FI $\uparrow$ | MA $\uparrow$ |
| --- | --- | --- | --- | --- | --- |
| XGBoost | 0.928 $\pm$ 0.004 | 0.929 $\pm$ 0.002 | 0.845 $\pm$ 0.008 | 0.774 $\pm$ 0.009 | 0.922 $\pm$ 0.002 |
| FT-Transformer | 0.926 $\pm$ 0.003 | 0.926 $\pm$ 0.002 | 0.863 $\pm$ 0.007 | 0.792 $\pm$ 0.011 | 0.916 $\pm$ 0.003 |
| MLP | 0.895 $\pm$ 0.004 | 0.914 $\pm$ 0.002 | 0.840 $\pm$ 0.005 | 0.793 $\pm$ 0.011 | 0.886 $\pm$ 0.003 |
| TabTransformer | 0.921 $\pm$ 0.004 | 0.912 $\pm$ 0.002 | 0.835 $\pm$ 0.007 | - | 0.910 $\pm$ 0.002 |
| ResNet | 0.896 $\pm$ 0.006 | 0.917 $\pm$ 0.002 | 0.841 $\pm$ 0.006 | 0.793 $\pm$ 0.013 | 0.889 $\pm$ 0.003 |
| MambaTab^∗ | 0.900 $\pm$ 0.004 | 0.916 $\pm$ 0.003 | 0.846 $\pm$ 0.007 | 0.792 $\pm$ 0.011 | 0.890 $\pm$ 0.003 |
| Mambular | 0.927 $\pm$ 0.006 | 0.928 $\pm$ 0.002 | 0.856 $\pm$ 0.004 | 0.795 $\pm$ 0.011 | 0.917 $\pm$ 0.003 |

### 3.1 Distributional Regression

To further verify the applicability of Mambular to tabular problems, we conducted a small task on distributional regression (Kneib et al.,, ). Distributional regression describes regression beyond the mean, i.e., the modeling of all distributional parameters. Thus, Location Scale and Shape (LSS) models can quantify the effects of covariates on not just the mean but also on any parameter of a potentially complex distribution assumed for the responses. A major advantage of these models is their ability to identify changes in all aspects of the response distribution, such as variance, skewness, and tail probabilities, enabling the model to properly disentangling aleatoric uncertainty from epistemic uncertainty. While this has been a common standard in classical statistical approaches (Stasinopoulos and Rigby,, ), it has not yet been widely adopted by the ML community. Recent interpretable approaches (), however, have demonstrated the applicability of distributional regression in tabular deep learning. Furthermore, approaches like XGBoostLSS (März,, ; März and Kneib,, ) demonstrate that tree-based models are capable of effectively solving such tasks. Below, we show that Mambular for Location Scale and Shape (MambularLSS) outperforms XGBoostLSS in terms of Continuous Ranked Probability Score (CRPS) (Gneiting and Raftery,, ) when minimizing the negative log-likelihood while maintaining a small MSE. See appendix D.1 for a short introduction to the CRPS metric.

**Table 5: Results for distributional regression for a normal distribution for the Abalone and California Housing datasets. Significantly better models at the 5% level are marked in green. $p$-vales are 0.20 and 0.00002 respectively for Abalone and and CA housing for the CRPS metric.**
|  | AB | CA |  |  |
| --- | --- | --- | --- | --- |
|  | CRPS $\downarrow$ | MSE $\downarrow$ | CRPS $\downarrow$ | MSE $\downarrow$ |
| MambularLSS | 0.345 $\pm$ 0.016 | 0.458 | 0.201 $\pm$ 0.004 | 0.181 |
| XGBoostLSS | 0.359 $\pm$ 0.016 | 0.479 | 0.227 $\pm$ 0.005 | 0.215 |

## 4 Ablation

We test out different architectures for Mambular to analyze whether different pooling techniques or bi-directional processing can be advantageous or may decrease performance. Sum-pooling, Max-pooling, standard Average-pooling, and Last token pooling—where only the last token in the sequence is passed to the task-specific model head—are tested. Additionally, the effectiveness of a learnable interaction layer between the features in improving performance is evaluated. This layer captures and models interactions between features by learning an interaction matrix $\mathbf{W}\in\mathbb{R}^{J\times J}$ such that $\text{interactions}=\mathbf{zW}$, where $\mathbf{z}$ is the input feature matrix, before being passed through the SSM. In Transformer networks for natural language processing, pooling via [cls] token embeddings is quite common (Gorishniy et al.,, ; ). This has also been shown to be beneficial in tabular problems (). Instead of appending the [cls] token to the beginning of each sequence, it is appended to the end of each sequence. Pooling is then performed as last token pooling.

**Table 6: Mean AUC and Mean MSE for various datasets and model configurations. We test different pooling methods, bi-directional processing and a learnable interaction layer. Significantly worse results compared to the default (average pooling, no interaction and no bi-directional processing) are marked red and bold at the 5% significance level and underscored and red at the 10% significance level. All results are achieved with 5-fold cross validation with identical seeds to the main results.**
| Pooling | bi-directional | Interaction | BA $\uparrow$ | AD $\uparrow$ | AB $\downarrow$ | CA $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| Last | $\times$ | $\times$ | 0.916 $\pm$ 0.004 | 0.927 $\pm$ 0.002 | 0.449 $\pm$ 0.043 | 0.181 $\pm$ 0.012 |
| Sum | $\times$ | $\times$ | 0.925 $\pm$ 0.005 | 0.928 $\pm$ 0.002 | 0.449 $\pm$ 0.048 | 0.171 $\pm$ 0.009 |
| Max | $\times$ | $\times$ | 0.928 $\pm$ 0.004 | 0.927 $\pm$ 0.002 | 0.455 $\pm$ 0.050 | 0.172 $\pm$ 0.008 |
| Avg | $\checkmark$ | $\times$ | 0.927 $\pm$ 0.004 | 0.928 $\pm$ 0.002 | 0.450 $\pm$ 0.045 | 0.170 $\pm$ 0.010 |
| Avg | $\times$ | $\checkmark$ | 0.928 $\pm$ 0.004 | 0.928 $\pm$ 0.002 | 0.453 $\pm$ 0.046 | 0.170 $\pm$ 0.007 |
| [cls] | $\times$ | $\times$ | 0.914 $\pm$ 0.005 | 0.928 $\pm$ 0.002 | 0.478 $\pm$ 0.044 | 0.194 $\pm$ 0.018 |
| Avg | $\times$ | $\times$ | 0.927 $\pm$ 0.006 | 0.928 $\pm$ 0.002 | 0.452 $\pm$ 0.043 | 0.167 $\pm$ 0.011 |

Interestingly, we find that the vanilla architecture of average pooling, no interaction, and only one-directional processing performs consistently best among model configurations. Additionally, that last token pooling and [CLS] token pooling perform significantly worse on two of the four tested datasets. A 5-fold cross-validation is performed, using the same hyperparameters across all models The matrices $\mathbf{A},\mathbf{B}$ and $\Delta$ are not shared in bi-directional processing but each direction has its own set of learnable parameters. Thus, the bi-directional models have additional trainable parameters.

Additionally, we analyze whether the order of the sequence and the position of the variables within the sequence affect the model’s performance. For textual data, shuffling the order of words/tokens has a significant impact, and even switching out single words can lead to completely different contextualized embeddings. Since these contextualized representations are pooled and fed directly to the task-specific head in Mambular, this could also affect performance.
We test two different shuffling settings: i) shuffling the embeddings after they have passed through the embedding layer, ii) shuffling the sequence of variables before being passed through the embedding layers.

All sequences are ordered by default with numerical features first, followed by categorical features, as arranged in the datasets from the UCI Machine Learning Repository. For the ablation study, a dataset with 5,000 samples and 10 features—five numerical and five categorical—was simulated. The numerical features were generated with large correlations, including two pairs with correlations of 0.8 and 0.6, respectively. The categorical features were created with four distinct categories.
Interaction terms were included as follows: An interaction between two numerical features, an interaction between a categorical and a numerical feature, and an interaction between two categorical features. The numerical features were scaled using standard normalization before generating the target variable. The target variable was constructed to include linear effects from each feature and the specified interaction terms, with added Gaussian noise for variability. We first fit a XGBoost model for a sanity check. Subsequently, we fit Mambular with default ordering (numerical before categorical features), flipped ordering and switched categorical and numerical ordering. Subsequently, we randomly shuffled the order and fit 10 models. We find that ordering does not have an effect on this simulated data, even with these large interaction and correlation effects(^3^33See the appendix for the chosen model parameters. Since the dataset is comparably smaller, we used a smaller Mambular model. Hyperparameters such as the learning rate, batch size etc. are kept identical to the default Mambular model.).

**Table 7: Performance for different orderings of features. Numerical features are given as integer numbers, categorical features as capital letters. Feature interaction between numerical features is given in blue. Feature interaction between categorical features is denoted in green and feature interaction between a numerical and a categorical feature is given in lavender. We find that reordering the features either before or after the embedding layers does not affect performance of the model. No ordering performs significantly better or worse than the default model, while all models perform significantly better than the XGBoost model.**
| Before Embedding Layer | After Embedding Layer | Ordering |
| --- | --- | --- |
| Default | 0.918 $\pm$ 0.045 | [1 2 3 4 5 A B C D E] |
| 0.916 $\pm$ 0.043 | 0.913 $\pm$ 0.043 | [E D C B A 5 4 3 2 1] |
| 0.919 $\pm$ 0.044 | 0.914 $\pm$ 0.042 | [A B C D E 1 2 3 4 5] |
| 0.917 $\pm$ 0.043 | 0.915 $\pm$ 0.045 | [A B 2 3 1 D E 4 C 5] |
| 0.920$\pm$ 0.046 | 0.917 $\pm$ 0.045 | [D C 2 A B E 1 5 3 4] |
| 0.914 $\pm$ 0.043 | 0.914 $\pm$ 0.044 | [B 1 4 C D A 2 E 3 5] |
| 0.916 $\pm$ 0.045 | 0.914 $\pm$ 0.041 | [1 5 E B C 4 3 D 2 A] |
| 0.918 $\pm$ 0.046 | 0.914 $\pm$ 0.045 | [2 5 E B 4 A 1 3 D C] |
| 0.916 $\pm$ 0.044 | 0.915 $\pm$ 0.043 | [1 C A 2 D 4 E 3 5 B] |
| 0.917 $\pm$ 0.040 | 0.914 $\pm$ 0.043 | [A 1 4 5 2 C E B D 3] |
| 0.917 $\pm$ 0.044 | 0.922 $\pm$ 0.040 | [4 A 1 2 3 B 5 C D E] |
| 0.920 $\pm$ 0.040 | 0.913 $\pm$ 0.040 | [1 A D C B 3 E 2 5 4] |
| 0.920 $\pm$ 0.041 | 0.916 $\pm$ 0.044 | [C 5 B 2 4 A E D 3 1] |
| XGBoost | 1.096 $\pm$ 0.038 |  |

To verify these results, we tested it on 4 of the benchmark datasets. We again find, that flipping the sequence before or after the embedding layer does not have a significant impact. Additionally, we generally find that the sequence ordering does not significantly affect model performance at the 5% level for most datasets.

**Table 8: Mean AUC and Mean MSE for different feature orderings in the sequence. Flipping. the sequence does not significantly affect the performance at the 5% or 10% significance level. Significantly different values at the 5% level from the default configuration (Num$\mid$Cat) are in bold and marked red.**
| Model | BA $\uparrow$ | AD $\uparrow$ | AB $\downarrow$ | CA $\downarrow$ |
| --- | --- | --- | --- | --- |
| Num$\mid$Cat | 0.927 $\pm$ 0.006 | 0.928 $\pm$ 0.002 | 0.452 $\pm$ 0.043 | 0.167 $\pm$ 0.011 |
| Cat$\mid$Num | 0.925 $\pm$ 0.004 | 0.927 $\pm$ 0.002 | 0.454 $\pm$ 0.044 | 0.158 $\pm$ 0.007 |
| random shuffle | 0.923 $\pm$ 0.002 | 0.927 $\pm$ 0.002 | 0.457 $\pm$ 0.045 | 0.172 $\pm$ 0.070 |
| random shuffle | 0.921 $\pm$ 0.005 | 0.927 $\pm$ 0.002 | 0.459 $\pm$ 0.049 | 0.177 $\pm$ 0.010 |
| random shuffle | 0.924 $\pm$ 0.005 | 0.927 $\pm$ 0.002 | 0.453 $\pm$ 0.045 | 0.190 $\pm$ 0.010 |

However, for the california housing dataset we find one randomly shuffled sequence that does perform worse than the baseline at the 5% significance level. Analyzing the variable sequence, however, we do find a consistent, but puzzling pattern. The position of the variables Longitude and Latitude seems to directly influence model performance. However, Figure 4 shows that there are much stronger correlations between the features than the one between Longitude and Latitude. Additionally, we analyzed the effect strengths of all pairwise interactions with a simple Linear regression and XGBoost. We find, that the interaction effect between Longitude and Latitude is much less important than that between Longitude and Median Income for example. We have analyzed the California housing dataset in detail in Appendix D.

Figure: Figure 4: Correlation plot for California Housing dataset
Refer to caption: extracted/5786111/images/corr_plot.png

This finding, while only relevant for one of the four tested real world datasets, suggests that variable ordering can have an effect on model performance for Mambular. While the default ordering demonstrates strong performance, exploring alternative orderings might further enhance Mambular’s benchmark results. As expected, we do not find these results for Attention based model. See the appendix for the results for the FT-Transformer.

**Table 9: Analysis of results for CA Housing**
| Model | CA $\downarrow$ | $p$-value | Ordering |
| --- | --- | --- | --- |
| Num$\mid$Cat | 0.167 $\pm$ 0.011 | - | [LO, LA, MA, TR, TB, Po, Hh, MI, OP] |
| Cat$\mid$Num | 0.158 $\pm$ 0.007 | 0.168 | [OP, MI, Hh, Po, TB, TR, MA, LA, LO] |
|  | 0.177 $\pm$ 0.007 | 0.136 | [LO, MA, LA, TR, TB, Po, Hh, MI, OP] |
|  | 0.175 $\pm$ 0.008 | 0.240 | [LO, MA, TR, LA, TB, Po, Hh, MI, OP] |
|  | 0.194 $\pm$ 0.010 | 0.003 | [LO, MA, TR, TB, LA, Po, Hh, MI, OP] |
|  | 0.196 $\pm$ 0.011 | 0.003 | [LO, MA, TR, TB, Po, LA, Hh, MI, OP] |
|  | 0.194 $\pm$ 0.011 | 0.004 | [LO, MA, TR, TB, Po, Hh, LA, MI, OP] |
|  | 0.195 $\pm$ 0.010 | 0.004 | [LO, MA, TR, TB, Po, Hh, MI, LA, OP] |
|  | 0.194 $\pm$ 0.012 | 0.005 | [LO, MA, TR, TB, Po, Hh, MI, OP, LA] |

## 5 Limitations

The presented model is tested on a variety of datasets and benchmarked against a range of models. Note that we do not perform hyperparameter tuning, as results from Grinsztajn et al., () and Gorishniy et al., () indicate that most models already perform well without tuning. These studies suggest that while hyperparameter tuning does improve performance across all models simultaneously, the relative ranking of the models remains largely unchanged. This implies that a model performing best or worst with default configurations will likely maintain its ranking even after extensive tuning. Additionally, McElfresh et al., () found similar results, reinforcing the idea that hyperparameter tuning benefits most models to a similar degree without altering their comparative performance.

The lack of tuning leaves room for improvement across all models. However, the default configurations for the comparison models are thoroughly tested in a variety of studies, and it is expected that if any model might benefit more from hyperparameter tuning, it would be Mambular, as we lack extensive literature to guide its default settings. For the comparison models, we based our choices on literature to ensure meaningful and high-performing parameters by default. We were able to reproduce average results from studies such as Gorishniy et al., () and Grinsztajn et al., (). Additionally, critical hyperparameters such as learning rate, patience and number of epochs are shared among all models for a more consistent approach.
All hyperparameter configurations are listed in Appendix C and can also be found at [https://github.com/basf/mamba-tabular](https://github.com/basf/mamba-tabular).

## 6 Conclusion

We present Mambular, a new architecture for tabular deep learning. We show that a truly sequential model can be applied to tabular problems, offering a fresh perspective on how to interpret and handle tabular data by treating it as a sequential problem.
Our results demonstrate that a sequential model is effective for both regression and classification tasks across various datasets.
The performance of Mambular, along with its extension to MambularLSS, demonstrates its broad applicability to a range of tabular tasks.

Although Mamba is still relatively new compared to architectures like the Transformer, its rapid adoption suggests significant potential for further improvement. Advances such as those proposed by Lieber et al., () and Wang et al., () could be particularly beneficial for tabular applications. Additionally, exploring the optimal ordering of features or incorporating column-specific information through textual embeddings could further enhance performance.

Interpreting tabular data as a sequence presents significant advantages for feature incremental learning. New features can be appended directly to the sequence without the need to retrain the entire model.

## Acknowledgements

We sincerely acknowledge and appreciate the financial support provided by the Key Digital Capability (KDC) for Generative AI at BASF and the BASF Data & AI Academy, which played a critical role in facilitating this research.

Funding by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) within project 450330162 is gratefully acknowledged.

## Appendix A Datasets

All used datasets are taken from the UCI Machine Learning repository and publicly available. We drop out all missing values. For the regression tasks we standard normalize the targets.
Otherwise, preprocessing is performed as described above. Note, that before PLE encoding we scale the numerical features to be within (-1, +1).

**Table 10: The used datasets for benchmarking. All datasets are taken from the UCI Machine Learning repository. #num and #cat represent the number of numerical and categorical features respectively. The number of features thus determines for Mambular the ”sequence length”. The train, test and val numbers represent the average number of samples in the respective split for the 5-fold cross validation. Ratio marks the percentage of the dominant class for the binary classification tasks.**
| Name | Abbr. | #cat | #num | train | test | val | ratio |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | Regression Datasets |  |  |  |  |  |  |
| Diamonds | DI | 4 | 7 | 34522 | 10788 | 8630 | - |
| Abalone | AB | 1 | 8 | 2673 | 835 | 668 | - |
| California Housing | CA | 1 | 9 | 13210 | 4128 | 3302 | - |
| Wine Quality | WI | 0 | 12 | 4158 | 1299 | 1039 | - |
| Parkinsons | PA | 2 | 20 | 3760 | 1175 | 940 | - |
| House Sales | HS | 8 | 19 | 13832 | 4322 | 3458 | - |
| CPU small | CPU | 0 | 13 | 5243 | 1638 | 1310 | - |
|  | Classification Datasets |  |  |  |  |  |  |
| Bank | BA | 13 | 8 | 28935 | 9042 | 7233 | 88.3% |
| Adult | AD | 9 | 6 | 31259 | 9768 | 7814 | 76.1% |
| Churn | CH | 3 | 9 | 6400 | 2000 | 1600 | 79.6% |
| FICO | FI | 0 | 32 | 6694 | 2091 | 1673 | 53.3% |
| Marketing | MA | 15 | 8 | 27644 | 8638 | 6910 | 88.4% |

## Appendix B Results

**Table 11: Benchmarking results for the regression tasks. Average mean squared error values over 5 folds and the corresponding standard deviations are reported. Smaller values are better.**
| Models | DI $\downarrow$ | AB $\downarrow$ | CA $\downarrow$ | WI $\downarrow$ | PA $\downarrow$ | HS $\downarrow$ | CP $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| XGBoost | 0.019 $\pm$ 0.000 | 0.506 $\pm$ 0.044 | 0.171 $\pm$ 0.007 | 0.528 $\pm$ 0.008 | 0.036 $\pm$ 0.004 | 0.119 $\pm$ 0.024 | 0.024 $\pm$ 0.004 |
| FT-Transformer | 0.018 $\pm$ 0.001 | 0.458 $\pm$ 0.055 | 0.169 $\pm$ 0.006 | 0.615 $\pm$ 0.012 | 0.024 $\pm$ 0.005 | 0.111 $\pm$ 0.014 | 0.024 $\pm$ 0.001 |
| MLP | 0.066 $\pm$ 0.003 | 0.462 $\pm$ 0.051 | 0.198 $\pm$ 0.011 | 0.654 $\pm$ 0.013 | 0.764 $\pm$ 0.023 | 0.147 $\pm$ 0.017 | 0.031 $\pm$ 0.001 |
| TabTransformer | 0.065 $\pm$ 0.002 | 0.472 $\pm$ 0.057 | 0.247 $\pm$ 0.013 | - | 0.135 $\pm$ 0.001 | 0.160 $\pm$ 0.028 | - |
| ResNet | 0.039 $\pm$ 0.018 | 0.455 $\pm$ 0.045 | 0.178 $\pm$ 0.006 | 0.639 $\pm$ 0.013 | 0.606 $\pm$ 0.031 | 0.141 $\pm$ 0.017 | 0.030 $\pm$ 0.002 |
| MambaTab^∗ | 0.040 $\pm$ 0.008 | 0.455 $\pm$ 0.043 | 0.180 $\pm$ 0.008 | 0.601 $\pm$ 0.010 | 0.571 $\pm$ 0.021 | 0.122 $\pm$ 0.017 | 0.030 $\pm$ 0.002 |
| Mambular | 0.018 $\pm$ 0.000 | 0.452 $\pm$ 0.043 | 0.167 $\pm$ 0.011 | 0.628 $\pm$ 0.010 | 0.035 $\pm$ 0.005 | 0.132 $\pm$ 0.020 | 0.027 $\pm$ 0.002 |

## Appendix C Default Model Hyperparameters

In the following, we describe the default model parameters used for all the neural models. We based our choices on the literature to ensure meaningful and high-performing parameters by default. Additionally, except for MambaTab, we were able to reproduce results (on average) from popular studies, such as Gorishniy et al., () and Grinsztajn et al., (). While most larger benchmark studies perform extensive hyperparameter tuning for each dataset, analyzing these results (Grinsztajn et al.,, ; Gorishniy et al.,, ) shows that most models already perform well without tuning, as also found by McElfresh et al., (). Furthermore, the results suggest that performing hyperparameter tuning for all models does not change the ranking between the models, since most models benefit from tuning to a similar degree. Thus, we have collected informed hyperparameter defaults which we list in the following. The hyperparameters such as learning rate, patience and number of epochs are shared among all models for a more consistent approach.

**Table 12: Shared hyperparameters among all models**
| Hyperparameter | Value |
| --- | --- |
| Learning rate | 1e-04 |
| Learning rate patience | 10 |
| Weight decay | 1e-06 |
| Learning rate factor | 0.1 |
| Max Epochs | 200 |

#### MLP

As a simple baseline, we fit a simple MLP without any special architecture. However, PLE encodings are used, as they have been shown to significantly improve performance.

**Table 13: Default Hyperparameters for the MLP Model**
| Hyperparameter | Value |
| --- | --- |
| Layer sizes | (256, 128, 32) |
| Activation function | SELU |
| Dropout rate | 0.5 |
| PLE encoding dimension | 128 |

#### ResNet

A ResNet architecture for tabular data has been shown to be a sensible baseline (Gorishniy et al.,, ). Furthermore, McElfresh et al., () has validated the strong performance of ResNets compared to e.g. TabNet (Arik and Pfister,, ) or NODE (Popov et al.,, ).

**Table 14: Default Hyperparameters for the ResNet Model**
| Hyperparameter | Value |
| --- | --- |
| Layer sizes | (256, 128, 32) |
| Activation function | SELU |
| Dropout rate | 0.5 |
| Skip connections | True |
| Batch normalization | True |
| Number of blocks | 3 |
| PLE encoding dimension | 128 |

#### FT-Transformer

For the FT-Transformer architecture we orientated on the default parameters conducted by Gorishniy et al., (). We only slightly adapted them from 3 layers and an embedding dimension of 192 to 4 layers and an embedding dimension of 128 to be more consistent with the other models. However, we tested out the exact same architecture from Gorishniy et al., () and did not find any differences in performance, even a minimal (non-significant) decrease. Additionally, we found that using ReGLU instead of ReLU activation function in the transformer blocks does improve performance consistently.

**Table 15: Default Hyperparameters for the FT Transformer Model**
| Hyperparameter | Value |
| --- | --- |
| Model Dim | 128 |
| Number of layers | 4 |
| Number of attention heads | 8 |
| Attention dropout rate | 0.2 |
| Feed-forward dropout rate | 0.1 |
| Normalization method | LayerNorm |
| Embedding activation function | Identity |
| Pooling method | cls |
| Normalization first in transformer block | False |
| Use bias in linear layers | True |
| Transformer activation function | ReGLU |
| Layer normalization epsilon | 1e-05 |
| Feed-forward layer dimensionality | 256 |
| PLE encoding dimension | 128 |

#### TabTransformer

We practically used the same hyperparameter for TabTransformer as we used for Ft-Transformer. For consistency we do not use a multi-layer MLP for where the contextualized embeddings are being passed to. While this deviates from the original architecture, leaving this out ensures a more consistent comparison to FT-Transformer and Mambular since both models use a single layer after pooling. However, we used a larger feed forward dimensionality in the transformer to counteract this. Overall, our results are in line with the literature and we can validate that TabTransformer outperforms a simple MLP on average. For datasets where no categorical features are available, the TabTransformer converges to a simple MLP. Thus we left these results blank in the benchmarks.

**Table 16: Default Hyperparameters for the TabTransformer Model**
| Hyperparameter | Value |
| --- | --- |
| Model Dim | 128 |
| Number of layers | 4 |
| Number of attention heads | 8 |
| Attention dropout rate | 0.2 |
| Feed-forward dropout rate | 0.1 |
| Normalization method | LayerNorm |
| Embedding activation function | Identity |
| Pooling method | cls |
| Normalization first in transformer block | False |
| Use bias in linear layers | True |
| Transformer activation function | ReGLU |
| Layer normalization epsilon | 1e-05 |
| Feed-forward layer dimensionality | 512 |
| PLE encoding dimension | 128 |

#### MambaTab

We test out three different MambaTab architectures. Firstly, we implement the same architecture as for Mambular but instead of an embedding layer for each feature and creating a sequence of length $J$ we feed all features jointly through a single embedding layer and create a sequence of length 1. The Axis argument thus specifies over which axis the SSM model iterates. As described by the model iterates over this pseudo-sequence length of 1.

**Table 17: Default Hyperparameters for the MambaTab^∗ Model**
| Hyperparameter | Value |
| --- | --- |
| Model Dim | 64 |
| Number of layers | 4 |
| Expansion factor | 2 |
| Dimensionality of convolutional layers | 4 |
| Use bias in convolutional layers | True |
| Dropout rate | 0.0 |
| Dimensionality of the state | 128 |
| Normalization method | RMSNorm |
| Activation function | SiLU |
| PLE encoding dimension | 64 |
| Axis | 1 |

Additionally, we test out the default architecture from and hence have a super small model with only a single layer and embedding dimensionality of 32.

**Table 18: Default Hyperparameters for the MambaTab Model**
| Hyperparameter | Value |
| --- | --- |
| Model Dim | 32 |
| Number of layers | 1 |
| Expansion factor | 2 |
| Dimensionality of convolutional layers | 4 |
| Use bias in convolutional layers | True |
| Dropout rate | 0.0 |
| Dimensionality of the state | 32 |
| Normalization method | RMSNorm |
| Activation function | SiLU |
| Axis | 1 |

Lastly, we follow the Github implementation from [https://github.com/basf/mamba-tabular](https://github.com/basf/mamba-tabular) where the sequence is flipped and the SSM iterates over the number of observations instead of the pseudo-sequence length of 1.

**Table 19: Default Hyperparameters for the MambaTab^T Model**
| Hyperparameter | Value |
| --- | --- |
| Model Dim | 32 |
| Number of layers | 1 |
| Expansion factor | 2 |
| Dimensionality of convolutional layers | 4 |
| Use bias in convolutional layers | True |
| Dropout rate | 0.0 |
| Dimensionality of the state | 32 |
| Normalization method | RMSNorm |
| Activation function | SiLU |
| Axis | 0 |

#### Mambular

For Mambular we create a sensible default, following hyperparameters from the literature. We keep all hyperparameters from the Mambablocks as introduced by Gu and Dao, (). Hence we use SiLU activation and RMSNorm. WE use an expansion factor of 2 and use an embedding dimensionality of 64. The PLE encoding dimension is adapted to always match the embedding dimensionalitiy since first expanding the dimensionality in preprocessing to subsequently reduce it in the embedding layer seems counter intuitive.

**Table 20: Default Hyperparameters for the Mambular Model**
| Hyperparameter | Value |
| --- | --- |
| Model Dim | 64 |
| Number of layers | 4 |
| Expansion factor | 2 |
| Dimensionality of convolutional layers | 4 |
| Use bias in convolutional layers | True |
| Dropout rate | 0.0 |
| Dimensionality of the state | 128 |
| Normalization method | RMSNorm |
| Activation function | SiLU |
| PLE encoding dimension | 64 |

#### Model sizes

Below you find the number of trainable parameters for all models for all datasets. Note, that MambaTab^∗ and Mambular have very similar numbers of parameters since the sequence length does not have a large impact on the number of model parameters. Overall there is no correlation between model size and performance since e.g. the FT-Transformer architecture which is comparably larger to e.g. the MLP and ResNet architectures performs very well whereas the largest architecture, the TabTransformer performs worse than the smaller ResNet. Additionally, since the models have distinctively different architectures, the overall number of trainable parameters is not conclusive for training time or memory usage.

**Table 21: Number of trainable parameters for all models and datasets. Note that the number of trainable parameters is dependent on the dataset, since e.g. a larger number of variables leads to more trainable parameters in the embedding layer.**
| Dataset | AB | AD | BA | CA | CH | CP | DI | FI | HS | MA | PA | WI |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| FT-Transformer | 765k | 709k | 795k | 784k | 722k | 852k | 763k | 834k | 837k | 794k | 944k | 822k |
| MLP | 242k | 103k | 124k | 280k | 156k | 418k | 233k | 351k | 310k | 105k | 594k | 356k |
| ResNet | 261k | 123k | 144k | 299k | 176k | 437k | 253k | 371k | 330k | 125k | 614k | 375k |
| TabTransformer | 1060k | 1073k | 1149k | 1061k | 1060k | - | 1063k | - | 1100k | 1157k | 1068k | - |
| MambaTab^∗ | 331k | 318k | 316k | 335k | 321k | 352k | 328k | 358k | 339k | 312k | 373k | 348k |
| MambaTab | 13k | 14k | 14k | 13k | 13k | 14k | 13k | 14k | 14k | 14k | 14k | 14k |
| Mambular | 331k | 324k | 361k | 335k | 321k | 352k | 329k | 365k | 358k | 361k | 374k | 348k |

**Table 22: Benchmarking results for the classification tasks. Average AUC values over 5 folds and the corresponding standard deviations are reported. Larger values are better.**
| Models | BA $\uparrow$ | AD $\uparrow$ | CH $\uparrow$ | FI $\uparrow$ | MA $\uparrow$ |
| --- | --- | --- | --- | --- | --- |
| XGBoost | 0.928 $\pm$ 0.004 | 0.929 $\pm$ 0.002 | 0.845 $\pm$ 0.008 | 0.774 $\pm$ 0.009 | 0.922 $\pm$ 0.002 |
| FT-Transformer | 0.926 $\pm$ 0.003 | 0.926 $\pm$ 0.002 | 0.863 $\pm$ 0.007 | 0.792 $\pm$ 0.011 | 0.916 $\pm$ 0.003 |
| MLP | 0.895 $\pm$ 0.004 | 0.914 $\pm$ 0.002 | 0.840 $\pm$ 0.005 | 0.793 $\pm$ 0.011 | 0.886 $\pm$ 0.003 |
| TabTransformer | 0.921 $\pm$ 0.004 | 0.912 $\pm$ 0.002 | 0.835 $\pm$ 0.007 | - | 0.910 $\pm$ 0.002 |
| ResNet | 0.896 $\pm$ 0.006 | 0.917 $\pm$ 0.002 | 0.841 $\pm$ 0.006 | 0.793 $\pm$ 0.013 | 0.889 $\pm$ 0.003 |
| MambaTab^∗ | 0.900 $\pm$ 0.004 | 0.916 $\pm$ 0.003 | 0.846 $\pm$ 0.007 | 0.792 $\pm$ 0.011 | 0.890 $\pm$ 0.003 |
| Mambular | 0.927 $\pm$ 0.006 | 0.928 $\pm$ 0.002 | 0.856 $\pm$ 0.004 | 0.795 $\pm$ 0.011 | 0.917 $\pm$ 0.003 |

#### MambaTab

As described in the main part, MambaTab () significantly differs from Mambular, since it is not a sequential model. To achieve the presented results from MambaTab, we have followed the provided implementation from the authors retrieved from [https://github.com/Atik-Ahamed/MambaTab](https://github.com/Atik-Ahamed/MambaTab). It is worth noting, however, that MambaTab benchmarks the model on a lot of smaller datasets. 50% of the benchmarked datasets have not more than 1000 observations. Additionally, the provided implementation suggests, that MambaTab does indeed not iterate over a pseudo sequence length of 1, but rather over the number of observations, similar to a TabPFN (Hollmann et al.,, ). We have also tested this version, denoted as MambaTab^T but did not find that it performs better than the described version. On the Adult dataset, our achieved result of 0.901 AUC on average is very similar to the default results reported in with 0.906. The difference could be firstly due to us performing 5-fold cross validation and secondly different seeds in model initialization.

**Table 23: Benchmarking results for the regression tasks for the original MambaTab implementation provided by [https://github.com/Atik-Ahamed/MambaTab](https://github.com/Atik-Ahamed/MambaTab)**
| Models | DI $\downarrow$ | AB $\downarrow$ | CA $\downarrow$ | WI $\downarrow$ | PA $\downarrow$ | HS $\downarrow$ | CP $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| MambaTab | 0.035 $\pm$ 0.006 | 0.456 $\pm$ 0.053 | 0.272 $\pm$ 0.016 | 0.685 $\pm$ 0.015 | 0.531 $\pm$ 0.032 | 0.163 $\pm$ 0.009 | 0.030 $\pm$ 0.002 |
| MambaTab^T | 0.038 $\pm$ 0.002 | 0.468 $\pm$ 0.048 | 0.279 $\pm$ 0.010 | 0.694 $\pm$ 0.015 | 0.576 $\pm$ 0.022 | 0.179 $\pm$ 0.027 | 0.033 $\pm$ 0.002 |

**Table 24: Benchmarking results for the classification tasks. Average AUC values over 5 folds and the corresponding standard deviations are reported. Larger values are better.**
| Models | BA $\uparrow$ | AD $\uparrow$ | CH $\uparrow$ | FI $\uparrow$ | MA $\uparrow$ |
| --- | --- | --- | --- | --- | --- |
| MambaTab | 0.886 $\pm$ 0.006 | 0.901 $\pm$ 0.001 | 0.828 $\pm$ 0.005 | 0.785 $\pm$ 0.012 | 0.880 $\pm$ 0.003 |
| MambaTab^T | 0.888 $\pm$ 0.005 | 0.899 $\pm$ 0.002 | 0.815 $\pm$ 0.009 | 0.783 $\pm$ 0.012 | 0.878 $\pm$ 0.005 |

## Appendix D California Housing

Given the results for the California Housing dataset for mambular, we further analyzed the dataset.
Below are more results for Mambular with random shuffling. Again we can see the the position of Latitude and Longitude significantly impact model performance.
We generally observe, that when more than 3 variables are between these two variables, this negatively impacts performance. Having these two right after each other, independent of the order, does not significantly impact the performance. However, we do find that the best model is the one where these two variables are right at the end.

**Table 25: Analysis of results for CA Housing**
| Model | CA $\downarrow$ | $p$-value | Ordering |
| --- | --- | --- | --- |
| Num$\mid$Cat | 0.167 $\pm$ 0.011 | - | [LO, LA, MA, TR, TB, Po, Hh, MI, OP] |
| Cat$\mid$Num | 0.158 $\pm$ 0.007 | 0.168 | [OP, MI, Hh, Po, TB, TR, MA, LA, LO] |
|  | 0.174 $\pm$ 0.009 | 0.304 | [Po, Hh, MI, OP, LO, LA, MA, TR, TB] |
|  | 0.195 $\pm$ 0.012 | 0.005 | [LO, MA, TR, TB, Po, Hh, MI, OP, LA] |
|  | 0.197 $\pm$ 0.010 | 0.002 | [MA, LO, TR, TB, Po, Hh, MI, LA, OP] |
|  | 0.188 $\pm$ 0.010 | 0.014 | [MA, TR, LO, TB, Po, Hh, LA, MI, OP] |
|  | 0.178 $\pm$ 0.010 | 0.137 | [MA, LO, LA, TR, TB, Po, Hh, MI, OP] |
|  | 0.177 $\pm$ 0.008 | 0.142 | [MA, TR, TB, Po, LA, LO, Hh, MI, OP] |
|  | 0.178 $\pm$ 0.009 | 0.123 | [LA, LO, MA, TR, TB, Po, Hh, MI, OP] |
|  | 0.172 $\pm$ 0.070 | 0.420 | [Hh, TB, Po, MI, MA, OP, LA, LO, TR] |
|  | 0.177 $\pm$ 0.010 | 0.171 | [LO, Po, OP, LA, MI, MA, TR, Hh, TB] |
|  | 0.190 $\pm$ 0.010 | 0.009 | [Hh, TB, LO, MI, Po, OP, TR, MA, LA] |

To analyze the feature interaction effect between these two variables, we conducted a simple regression with pairwise feature interactions and analyzed the effect strengths. Interestingley, we find that the interaction between Longitude and Latitude is not as prominent as that between other variables.

Figure: Figure 5: Linear Regression with pairwise interaction effects on the california housing dataset.
Refer to caption: extracted/5786111/images/linreg_interaction.png

Additionally, we have fit a XGboost model and analyzed the pairwise feature importance metrics and generally find the same results as for the linear regression.

Figure: Figure 6: Pairwise feature importance statistics from a XGBoost model on the california housing dataset.
Refer to caption: extracted/5786111/images/xgboost_interaction_importance.png

### D.1 Distributional Regression

Distributional regression describes regression that goes beyond the mere prediction of mean responses. In detail, all distributional parameters, e.g., mean, variance, kurtosis, and skewness, are jointly modeled and thus allow for a comprehensive understanding of the response distribution. This approach allows for a more detailed analysis by taking into account the entire distribution shape. It helps to make more accurate predictions and gain deeper insights into the variability and asymmetry within the data. For a comprehensive overview see Kneib et al., ().

This is achieved by minimizing the negative log-likelihood via optimizing the parameters $\theta$

$$ $\mathcal{L}(\theta)=-\sum_{i=1}^{n}\log f(y_{i}\mid\mathbf{x}_{i},\theta)$ $$

For the two examples in the main part, a normal distribution is modelled and hence, the models minimize:

$$ $log\left(\mathcal{L}(\mu,\sigma^{2}|y)\right)=-\frac{n}{2}\log(2\pi\sigma^{2}) -\frac{1}{2\sigma^{2}}\sum_{i=1}^{n}(y_{i}-\mu)^{2},$ $$

where $n$ is the underlying number of observations and parameters $y\in\mathbb{R}$, location $\mu\in\mathbb{R}$ and scale $\sigma\in\mathbb{R}^{+}$.

#### CRPS

Analyzing distributional regression models also requires careful consideration of the evaluation metrics. Traditionally, mean focused models are evaluated using mean-centric metrics, e.g. MSE, AUC or Accuracy. However, a model that takes all distributional parameters into account should be evaluated on the predictive performance for all of the distributional parameters.
Following Gneiting and Raftery, (), the evaluation metric should be proper, i.e. enforce the analyst to report their true beliefs in terms of a predictive distribution. In terms of classical mean-centric metrics, e.g. MSE is proper for the mean, however, not proper for evaluating the complete distributional prediction.
We therefore rely on the Continuous Ranked Probability Score (Gneiting and Raftery,, ) for model evaluation, given by:

$$ $CRPS(F,x)=-\int_{-\infty}^{\infty}(F(y)-\mathbf{1}_{y\geq x})^{2}\,dy.$ $$

See Gneiting and Raftery, () for more details.

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 3
presentation: 3
questions: - Line 47, missing related work [1].
- Line 252, missing related work [2].
- Line 264, how was the set of defaults devised? Do the authors use the defaults of the baselines from the corresponding papers? 
- Line 271, how were the datasets selected?
- In terms of baselines, why was CatBoost not included? Given that it handles categorical variables natively? It is additionally among the best performing methods from the gradient-boosted decision tree family.
- Could the authors apply significance tests over the entire suite of datasets and not on a per-dataset basis? The mean value could be used to aggregate the multiple runs on a single dataset. Additionally, could all of the methods be considered and a critical difference diagram be provided?
- Line 297, the results do not align with the referenced work, in particular in [3], (Table 1, Table 2, Figure 3) the FT-Transformer architecture lags behind the ResNet architecture in terms of performance. Additionally, as I previously mentioned, CatBoost is the top performing method, which the authors do not include in the comparisons.
- A comparison with TabPFN for datasets that fit the limitations of the method would be interesting.
- What is the runtime of the proposed method compared to the baselines? What about the inference time?

[1] Kadra, A., Lindauer, M., Hutter, F., & Grabocka, J. (2021). Well-tuned simple nets excel on tabular datasets. Advances in neural information processing systems, 34, 23928-23941.

[2] Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., & Gulin, A. (2018). CatBoost: unbiased boosting with categorical features. Advances in neural information processing systems, 31.

[3] McElfresh, D., Khandagale, S., Valverde, J., Prasad C, V., Ramakrishnan, G., Goldblum, M., & White, C. (2024). When do neural nets outperform boosted trees on tabular data?. Advances in Neural Information Processing Systems, 36.
soundness: 3
strengths: - The work is well written.
- The work ablates several design choices of the proposed method.
- The proposed method achieves competitive performance compared to the considered baselines. 
- The work considers both regression and classification tasks.
summary: The authors propose Mambular, an adaption of the Mamba architecture for tabular data. The authors compare Mambular against various well-known model families in the tabular domain, in both regression and classification tasks, showcasing the competitive performance of the proposed method. The authors additionally perform a series of ablations to provide insights on several design choices.
weaknesses: - The related work section misses core references and can be further strengthened. [1][2]
- Line 279, I would rather the authors advocated that the results are not significant, rather than considering a 10% significance level. The standard is a 5% significance level.
- The authors do not perform hyperparameter tuning.
- It is not clear how the set of defaults for all methods is devised.
- The number of datasets considered is limited. Additionally, the authors do not use well-established benchmarks from the community. [3][4]
- A detailed analysis regarding time is not provided to have a clear understanding of the pros and cons of different methods.

[1] Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., & Gulin, A. (2018). CatBoost: unbiased boosting with categorical features. Advances in neural information processing systems, 31.

[2] Kadra, A., Lindauer, M., Hutter, F., & Grabocka, J. (2021). Well-tuned simple nets excel on tabular datasets. Advances in neural information processing systems, 34, 23928-23941.

[3] Gijsbers, P., Bueno, M. L., Coors, S., LeDell, E., Poirier, S., Thomas, J., ... & Vanschoren, J. (2024). Amlb: an automl benchmark. Journal of Machine Learning Research, 25(101), 1-65.

[4] Grinsztajn, L., Oyallon, E., & Varoquaux, G. (2022). Why do tree-based models still outperform deep learning on typical tabular data?. Advances in neural information processing systems, 35, 507-520.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: Please check the above weaknesses.
soundness: 2
strengths: The authors proposed the Mamba architecture to solve tabular problems. It showed good performance overall, especially in classification performance.
summary: The authors introduced Mamba architecture to solve the problems of tabular data. They showed the effectiveness of the proposed method by comparing its performance with neural networks-based algorithms and tree-based methods. And they confirmed the superiority of a sequence and passing method by comparing various pooling methods.
weaknesses: The proposed method seems to be nothing more than using the Mamba structure on tabular data. Although there are some structural suggestions, it is difficult to say that the idea is novel overall.
And when I check the experimental results, it seems difficult to say that the performance has been greatly improved.Additional discussions and experiments are needed to see what advantages can be taken advantage of by applying the Mamba structure to tabular data, what differential performance improvements can be made compared to existing methods, and what can be considered if a Mamba structure or a new structure based on Mamba is proposed in the future.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 4
questions: - Hyperparameter Tuning and Model Performance: In Section 5, the authors claim that hyperparameter tuning does not significantly impact model ranking, citing Grinsztajn et al. (2022) and Gorishniy et al. (2021). I find this very questionable, especially given that Table 3 shows Random Forest outperforming XGBoost, which is typically strong for tabular data (Also, see Appendix tables here [1]).  
Additionally, the model’s sensitivity to kernel size in permutation experiments suggests that hyperparameter choices affect performance significantly. To address this, I suggest comparing models using an established benchmark such as TabZilla [2], which offers a diverse suite of datasets for evaluating tabular models. 

- The Mambular model’s sequential structure makes it highly sensitive to feature order, which impacts performance under certain hyperparameter settings. Although feature order dependency is a known issue in tabular data models, I recommend that the authors conduct an experiment with random feature permutations (see Definition 2 in [3]). This could enable Mambular to emphasize dependencies rather than sequence, potentially improving model's robustness.



[1] https://arxiv.org/abs/2305.13072

[2] https://github.com/naszilla/tabzilla

[3] Borisov, V., Seßler, K., Leemann, T., Pawelczyk, M., & Kasneci, G. (2022). "Language models are realistic tabular data generators." arXiv preprint arXiv:2210.06280.
soundness: 3
strengths: - A novel sequential approach to tabular data, treating features as ordered sequences to capture feature interactions. 

- Solid benchmarks against state-of-the-art models confirm its competitive performance across diverse datasets.

- Paper is clearly written and easy to follow,  the model and its components are clearly explained, and the comprehensive ablation study provides insight into the impact of different architectural choices.
summary: This paper presents Mambular, a sequential deep learning model based on the Mamba architecture, designed for tabular data tasks. The authors evaluate Mambular against state-of-the-art neural and (ensemble) tree-based models, showing that it performs as well as or better than these models across different datasets.
weaknesses: - The paper would benefit from a more extensive comparison with recent deep learning baselines for tabular data. While it presents strong benchmarks, adding models like TabPFN and GANDALF could enhance the evaluation, giving a clearer picture of Mambular’s performance against the latest advancements. 
- Additionally, thorough hyperparameter tuning is necessary for many tabular learning models to ensure fair and optimal comparisons. Please refer to Questions.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: What is the rationale behind interpreting tabular data as sequences? Could the authors provide a theoretical explanation or empirical evidence showing that treating features as a sequence offers a distinct advantage?

How do the authors address the lack of consistency in performance improvements across the datasets? Were any specific types of datasets or characteristics where Mambular performed particularly well?

Could the authors explain why comparisons were not made with a wider range of established tabular models, such as TabR[2], T2G-Former[3], and SAINT[4]?
[2] TabR: Unlocking the Power of Retrieval-Augmented Tabular Deep Learning, 2023
[3] T2G-Former: Organizing Tabular Features into Relation Graphs Promotes Heterogeneous Feature Interaction, 2023
[4] SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training, 2021
soundness: 1
strengths: The paper explores the application of the Mamba architecture to tabular data, leveraging lessons from its success in handling other data types like text and time series. While not entirely novel, this extension demonstrates an attempt to generalize Mamba to a new domain.

The authors present experimental results comparing Mambular with a limited selection of state-of-the-art models, such as FT-Transformer, TabTransformer, XGBoost, and LightGBM. While the comparisons are not exhaustive, these initial results provide some insight into the model’s potential.

The architecture incorporates feature interaction mechanisms and various pooling strategies, potentially providing flexibility to handle different types of tabular data characteristics.
summary: The paper introduces Mambular, an adaptation of the Mamba architecture specifically designed for tabular data problems. The authors claim that their model leverages a sequential interpretation of tabular data, similar to its successful application in other domains such as text, vision, and time series. The proposed architecture incorporates pooling strategies and feature interaction mechanisms to enhance performance. The paper provides experimental results on various datasets, comparing Mambular with state-of-the-art models such as FT-Transformer, TabTransformer, XGBoost, and LightGBM. According to the authors, Mambular achieves competitive results, highlighting the feasibility of applying sequential models to tabular data.
weaknesses: - Lack of novelty: One of the main weaknesses is the lack of significant novelty. The concept of adapting Mamba for tabular data is not substantially different from previously proposed models like MambaTab [1]. The minor modifications presented in Mambular mainly involve hyperparameter tuning and slight architectural changes, which do not justify the need for a new model.

[1] MA Ahamed et al., MambaTab: A Plug-and-Play Model for Learning Tabular Data, MIPR, 2024.

- Misalignment with tabular data characteristics: The paper fails to convincingly justify why treating tabular data as sequential offers any distinct advantage. Unlike sequential data such as text or time series, tabular data lacks a natural ordering of features. The authors did not provide clear evidence or theoretical grounding for why a sequence-based model should work better in this context.

- Limited scope of experiments: The experimental results are based on only 12 datasets, and the comparison to state-of-the-art algorithms is not comprehensive. Given the varied nature of tabular data, a broader set of datasets and comparisons to more established tabular models would have strengthened the claims.

- Inconsistent Performance: Despite the claims of Mambular’s superiority, the reported results do not consistently show a significant performance improvement over existing methods, such as XGBoost or LightGBM.

Meta-review: The authors apply the Mamba architecture to tabular data. The proposed technique uses memory more efficiently than transformer-based models and performs about as well as the selected baselines.

However, reviewers expressed concerns that evaluation is limited. In addition, no thorough hyperparameter tuning is performed, and the number of datasets and baselines is small. The authors do not exhaustively explain why a sequential model would be a good fit for tabular data in the first place.

Overall the work lacks novelty, the analysis is not sufficiently detailed, and the experimental protocol is sub-optimal (in terms of datasets, and baselines). Although the authors added a few more results in their rebuttal, these new findings did not change the essence of the quality of the manuscript.

I suggest rejecting this submission for now and encouraging the authors to incorporate the reviewers’ feedback into a future version.

Decision: Reject