Multi-branch of Attention Yields Accurate Results for Tabular Data

## Abstract

Abstract Tabular data inherently exhibits significant feature heterogeneity, but existing transformer-based methods lack specialized mechanisms to handle this property. To bridge the gap, we propose MAYA, an encoder-decoder transformer-based framework. In the encoder, we design a M ulti- B ranch of A ttention (MBA) that constructs multiple parallel attention branches and averages the features at each branch, effectively fusing heterogeneous features while limiting parameter growth. Additionally, we employ collaborative learning with a dynamic consistency weight constraint to produce more robust representations. In the decoder, cross-attention is utilized to seamlessly integrate tabular data with corresponding label features. This dual-attention mechanism effectively captures both intra-instance and inter-instance interactions. We evaluate the proposed method on a wide range of datasets and compare it with other state-of-the-art transformer-based methods. Extensive experiments demonstrate that our model achieves superior performance among transformer-based methods in both tabular classification and regression tasks.

## 1 Introduction

As society advances rapidly, tabular data have become a widely used format zhou24ftta. Despite its complex and multifaceted nature, its significant economic value drives growing researcher interest in analysis and study, leading to the proposal of various solutions klambauer2017self; wang2021dcn; Popov2020Neural; bonet2024hyperfast; arik2021tabnet; ye2024modern.

Figure: Figure 1: (a) The overview of MBA block. It employs multiple parallel MHA branches to enrich feature extraction. To maintain a constant size of hidden states, features are averaged branch-wise with dynamic weights, imposing a consistency constraint and enabling collaborative branch learning. (b) The visualization of the MBA block. Each branch exhibits distinct modes of feature interaction, such as sparse, dense, and hybrid patterns.
Refer to caption: x1.png

Inspired by transformers’ success in NLP and speech, transformer-based architectures are increasingly adopted for tabular data. AutoInt song2019autoint projects numerical and categorical features into a shared low-dimensional space and employs a multi-head self-attention network with residual connections to model feature interactions, enabling efficient end-to-end training on large-scale raw data. To investigate the effectiveness of embeddings, Huang et al. huang2020tabtransformer
propose transforming the embeddings of categorical features into robust contextual representations, achieving performance comparable to tree-based methods chen2016xgboost; prokhorenkova2018catboost on some datasets. Then, FT-Transformer gorishniy2021revisiting achieves strong performance across multiple datasets, demonstrating potential for superiority. Afterwards, ExcelFormer chen2024can introduces a semi-permeable attention module that selectively lets informative features integrate data from less relevant ones.
To further improve performance,  xu2024bishop; chen2022danets enhance model competitiveness by developing feature interaction modules (intra- and inter-instance) to boost performance. Moreover,  cheng2024arithmetic enhances feature representation through arithmetic operations beyond basic interactions.

Current transformer architectures struggle to effectively process heterogeneous data chen2024can; cheng2024arithmetic; chen2023trompt. While diverse feature extraction is promising, simply increasing attention heads is inefficient. This approach leads to an expansion of feature dimensions due to the concatenation operation, consequently increasing the parameters quadratically of subsequent Feed-Forward Network (FFN) layers and compromising computational efficiency. Thus, a transformer architecture for effective diverse feature extraction from tabular data is crucial.

In this work, we propose MAYA (Mixture of Attention Yields Accurate results for tabular data), a novel encoder-decoder transformer-based framework that achieves rich feature representations while maintaining parameter efficiency. Specifically, in the encoder, we propose an innovative Multi-Branch of Attention (MBA) structure, as illustrated in Fig.[1](https://arxiv.org/html/2502.12507v3#S1.F1)(a), which implements parallelized independent attention branches within a single encoder block. This architecture enables effective feature fusion while significantly improving feature diversity for tabular data. Inspired by collaborative mechanisms ke2020guided, our framework weights multi-branch attention outputs via collaborative policy for robust representations. In addition, we cascade multiple MBA blocks, enabling the capture of high-level semantic information and enhancing the encoder’s representation capacity. The MBA’s multi-branch design achieves two key objectives: it extracts diverse features while controlling dimensionality growth through averaging operations, thereby substantially reducing the parameter burden on downstream FFN layers. Unlike traditional concatenation, this avoids quadratic parameter growth in later layers. Furthermore, the decoder’s inter-instance attention and label integration yield more reliable results.

MAYA outperforms SOTA tabular transformers across multiple datasets. Furthermore, ablation studies confirm MBA’s effectiveness, positioning MAYA as a new tabular transformer baseline. The main contributions of MAYA are:

- •
We empirically verify on several public datasets that the proposed MBA structure can effectively obtain diverse features to deal with heterogeneity in tabular data.
- •
We show that the collaborative mechanism applied in branch weight computation of MBA can improve consistency training for tabular data.
- •
We propose a transformer encoder-decoder with specialized attention mechanisms for intra- and inter-instance relationships.

## 2 Related Work

### 2.1 Attention Mechanisms

The attention mechanism was introduced to improve the performance of the encoder-decoder model for machine translation vaswani2017attention. It reveals the relative importance of each component in a sequence compared to the other components. Usually, self-attention and cross-attention are the most widely used in transformer architecture models. Self-attention captures relationships within a single input sequence, and cross-attention captures relationships between elements of different input sequences.
Somepalli et al. somepalli2021saint introduce self-attention and inter-sample attention mechanisms for the classification of tabular data. They use self-attention for intra-sample features and inter-sample attention for inter-sample relationships. In our approach, we introduce two novel attention mechanisms: the Multi-branch of Attention (MBA) for self-attention and Inter-instance Attention Incorporating with Labels (IAIL) for cross-attention, to enhance data utilization and feature representation. Compared to somepalli2021saint, the inter-instance attention mechanism of IAIL is more parameter-efficient.

### 2.2 Collaborative Learning

Collaborative learning dillenbourg1999collaborative refers to methods and environments in which learners work together on a shared task, with each individual relying on and being responsible to others. In pixel-wise tasks, collaborative policy has been used to train different subset networks and to ensemble the reliable pixels predicted by these networks. As a result, this strategy covers a broad spectrum of pixel-wise tasks without requiring structural adaptation ke2020guided. Here, we use collaborative policy to construct branch weight in MBA block during training, with the weight proportional to the difference between the branch output and the true label. This dynamic consistency constraint can provide substantial benefit to enrich feature representations.

## 3 Methods

In this section, we present a detailed introduction to the proposed MAYA, adopting a top-down approach—starting with the network’s holistic view and progressing to its fundamental components.

### 3.1 Notations

In this work, let’s consider supervised learning problems on tabular data. A typical dataset with $N$ instances can be denoted as $\mathcal{D}=\{(x_{i},y_{i})\}_{i=1}^{N}$, where $x_{i}$ represents the features and $y_{i}$ the label of the $i$-th instance, respectively. Specifically, $x_{i}$ contains both numerical features $x_{i}^{num}$ and categorical features $x_{i}^{cat}$, that is to say, $x_{i}=(x_{i}^{num},x_{i}^{cat})$. To be convenient and without loss of generality, we denote an instance with $k$ features as $x_{i}\in\mathbb{R}^{k}$. For the label $y_{i}$, three types of tasks are considered, which are binary classification $y_{i}\in\{0,1\}$, multi-class classification $y_{i}\in\{1,\dots,C\}$ and regression $y_{i}\in\mathbb{R}$.

### 3.2 Tokenizer

Built upon transformer-like architecture, MAYA needs a tokenizer to map instances into an embedding space.
For an instance $x_{i}\in\mathbb{R}^{k}$, we define the $d$-dimension embeddings obtained by tokenizer as $z_{i}$:

$$ $z_{i}={\rm Tokenizer}(x_{i}),z_{i}\in\mathbb{R}^{k\times d}$ (1) $$

The tokenizer can be implemented in various ways, while in this paper, we primarily utilize the tokenizer of gorishniy2021revisiting, which embeds numerical features through element-wise multiplication and categorical features via a lookup table.

Unlike gorishniy2021revisiting, we innovatively find that adding an activation function after the tokenizer boosts network performance. We attribute this to the Kaiming initialization of tokenizer weights/biases which pairs effectively with ReLU. We’ll explore this detail in Section [4.3](https://arxiv.org/html/2502.12507v3#S4.SS3.SSS0.Px3) via ablation study.

Before the extraction of features, we append the embedding of a special token, i.e. [CLS] token, to $z_{i}$:

$$ $z^{0}_{i}={\rm Stack}[[CLS],z_{i}],z^{0}_{i}\in\mathbb{R}^{(k+1)\times d}$ (2) $$

where the subscript “0" denotes the initial input, i.e., the input to the first layer of the entire network.

### 3.3 Encoder-Decoder Architecture

Our network adopts an encoder-decoder framework (Fig.[2](https://arxiv.org/html/2502.12507v3#S3.F2)). The encoder processes the initial input $z_{i}^{0}$ into $\hat{z}_{i}$ (i.e. the processed [CLS] token representing the instance). $\hat{z}_{i}$ is then fed into the decoder to produce $\tilde{z}_{i}$, and a predictor generates the final prediction $p$. The process is summarized as:

$$ $\hat{z}_{i}={\rm Encoder}(z_{i}^{0}),\hat{z}_{i}\in\mathbb{R}^{d}$ (3) $$

$$ $\tilde{z}_{i}={\rm Decoder}(\hat{z}_{i}),\tilde{z}_{i}\in\mathbb{R}^{d}$ (4) $$

$$ $p={\rm Predictor}(\tilde{z}_{i})$ (5) $$

Figure: Figure 2: The overall structure of MAYA. It comprises an encoder (with MBA blocks) and a decoder (with IAIL blocks), arranged in upper and lower rows respectively. Inputs are tokenized before encoder processing, and decoder’s outputs are fed into a Predictor for final predictions.
Refer to caption: x2.png

where the predictor is a simple fully-connected layer.

### 3.4 Encoder

The construction of the encoder employs a deep-wide integration strategy: “deep" via traditional block cascading, and “wide" using our novel MBA block with parallel attentions. An encoder with $L_{e}$ MBA blocks is thus structured as:

$$ $z_{i}^{\ell}={\rm MBA}^{\ell}(z_{i}^{\ell-1}),\ell\in[1,L_{e}]$ (6) $$

#### 3.4.1 Multi-branch of Attention (MBA)

The heterogeneity of tabular data makes a single feature extractor inadequate. The classical transformer’s Multi-Head Attention (MHA) focuses on diverse representation subspaces via multiple heads vaswani2017attention, suggesting enhanced feature-processing capacity with more heads. However, this approach has two drawbacks: (1) concatenating MHA head subspaces enlarges attention output hidden states, thus increasing the number of parameters in FFN and computational cost; (2) output projection of concatenated features causes mixing, reducing subspace diversity.

To tackle the issues, we introduce the Multi-branch of Attention (MBA, Fig.[1](https://arxiv.org/html/2502.12507v3#S1.F1)(a)), which employs multiple parallel MHA-based attention branches with branch-level feature averaging. This approach preserves feature extraction diversity and constant hidden state sizes, avoiding FFN parameter inflation. In addition, feature averaging serves as regularization, reducing overfitting risk and improving robustness, as well as mitigating the impact of individual MHA failures.

Specifically, the input is duplicated and fed into identical attention branches with different weights. Their outputs pass through a shared FFN and individual layer normalization, then are averaged and normalized by another layer normalization to form the MBA block’s output. Formally, the output $z_{i}^{\ell}$ of the $\ell$-th MBA block (with $n$ parallel branches) is derived as follows:

$$ $z_{i}^{\ell}={\rm LayerNorm}(\frac{1}{n}\sum_{j=1}^{n}{\rm Branch}_{j}(z_{i}^{\ell-1})+z^{\ell-1}_{i})$ (7) $$

where ${\rm Branch}_{j}(\cdot)\coloneqq{\rm LayerNorm}_{j}({\rm FFN}({\rm Attention}_{j}(\cdot)+(\cdot)))$.

#### 3.4.2 Intra-block Balance: Branch Weight of MBA

The MBA block’s output averages results from multiple attention branches, implying equal contribution from each branch—a collaborative learning paradigm. Then, we introduce a dynamic consistency constraint, proposing a balancing strategy via branch weights (Fig.[3](https://arxiv.org/html/2502.12507v3#S3.F3)(a)). During training, we attach a shared Predictor (same as in Eq.[5](https://arxiv.org/html/2502.12507v3#S3.E5)) to each attention branch. Branches with significant prediction deviations (quantified by MSE for regression or CE for classification) are assigned larger weights, which are computed via softmax over branch losses. This penalizes deviating branches, balancing their contributions. To stabilize the computation of branch weights, we apply Exponential Moving Average (EMA) for smoothing purposes. The branch weight $W_{B}$ is formulated as:

$$ $W_{B}={\rm EMA}({\rm Softmax}(s))$ (8) $$

where $W_{B}=\{w_{j}\}_{j=1}^{n}$ and $s=\{s_{j}\}_{j=1}^{n}$. $s_{j}$ is the supervised loss of ${\rm Branch}_{j}$, that is

$$ $s_{j}={\rm L}_{pred}({\rm Predictor}({\rm Branch}_{j}(z^{\ell-1})),y)$ (9) $$

where ${\rm L}_{pred}$ is MSE loss or CE loss.

By applying the branch weight, Eq.[7](https://arxiv.org/html/2502.12507v3#S3.E7) can be reformulated as

$$ $z_{i}^{\ell}={\rm LayerNorm}(\sum_{j=1}^{n}w_{j}{\rm Branch}_{j}(z_{i}^{\ell-1})+z_{i}^{\ell-1})$ (10) $$

Figure: Figure 3: (a) Branch weights are computed during training by appending a shared Predictor to each branch, calculating losses, and assigning higher weights to branches with larger losses. This dynamic constraint promotes collaborative learning. (b) The overview of IAIL block in decoder.
Refer to caption: x3.png

#### 3.4.3 Inter-block Averaging

To balance both intra-block and inter-block feature extraction efficacy, we average the outputs of $L_{e}$ stacked MBA blocks, append a LayerNorm layer, and obtain the encoder’s final output. Combining Eq.[3](https://arxiv.org/html/2502.12507v3#S3.E3) and Eq.[6](https://arxiv.org/html/2502.12507v3#S3.E6), the encoder’s output $\hat{z}_{i}$ is derived as follows:

$$ $\hat{z}_{i}={\rm LayerNorm}(\frac{1}{L_{e}}\sum_{l=1}^{L_{e}}{z_{i}^{l}})$ (11) $$

#### 3.4.4 Analysis of MBA

In this section, we present a more in-depth analysis and insights into the MBA block to validate its effectiveness. Fig.[1](https://arxiv.org/html/2502.12507v3#S1.F1)(b) illustrates the attention maps of a specific layer within MAYA, where the MBA block comprises 3 branches, each with 4 attention heads. For details on the generation of Fig.[1](https://arxiv.org/html/2502.12507v3#S1.F1)(b), please refer to the Appendix [A](https://arxiv.org/html/2502.12507v3#A1). Given the utilization of the [CLS] token, in each attention map, features that significantly impact the [CLS] token are marked by correspondingly larger values in the first row, as highlighted by the red boxes in Fig.[1](https://arxiv.org/html/2502.12507v3#S1.F1)(b). The distinct branches within MBA exhibit varied attention patterns: some branches focus solely on a select few features, resulting in sparse attention maps (e.g., branch 0, where all heads exhibit sparsity); others engage in extensive feature interactions, leading to dense attention maps (e.g., branch 2, where most heads are dense); while certain branches achieve a balance between these two extremes (as illustrated by branch 1). These diverse branches effectively enhance the diversity of attention outcomes, thereby augmenting interactions among heterogeneous features. Our experimental results further demonstrate that MBA achieves this diversity in a parameter-efficient manner. In addition, Fig.[1](https://arxiv.org/html/2502.12507v3#S1.F1)(b) displays the branch weights for each branch, and the disparities among these weights underscore the distinctiveness and diversity of the different branches.

### 3.5 Decoder

The decoder features an inter-instance cross-attention block. Leveraging the encoder’s instance-level representation extraction, we treat the instance set $\hat{Z}=\{\hat{z}_{i}\}$ as a sequence (full dataset or a batch, i.e. $|\hat{Z}|\leq N$), with each instance as a token (e.g., reshaping $[batchsize,d]$ to $[1,batchsize,d]$ in PyTorch). This reformulation enables direct use of encoder outputs as input for inter-instance attention. We integrate this mechanism with label information in a novel decoder block, combining cross-instance relational modeling and label-aware processing.

#### 3.5.1 Inter-instance Attention Incorporated with Labels

We introduce IAIL (Inter-instance Attention Incorporated with Labels), a decoder component utilizing cross-attention (Fig.[3](https://arxiv.org/html/2502.12507v3#S3.F3)(b)), which accepts multiple inputs. The $\ell$-th IAIL (denoted as ${\rm IAIL}^{\ell}$) processes $\hat{Z}^{\ell-1}=\{\hat{z}_{i}^{\ell-1}\}$ to produce $\hat{Z}^{\ell}=\{\hat{z}_{i}^{\ell}\}$, with $\hat{Z}^{0}=\{\hat{z}_{i}^{0}\}$ as initial input. To integrate label information, we employ a trainable Label Embedding Layer (LEL), projecting labels into an embedding space as:

$$ $\hat{y}_{i}={\rm LEL}(y_{i}),\hat{y}_{i}\in\mathbb{R}^{d}$ (12) $$

where, for classification tasks, LEL acts as a lookup table, while for regression, it serves as a linear mapper.
To align with feature embeddings, label embeddings are collectively denoted as $\hat{Y}=\{\hat{y}_{i}\}$.

During training and inference, IAIL’s cross-attention mechanism receives distinct inputs, as follows:

- •
Training: for the $\ell$-th IAIL, the query comes from the prior IAIL’s output, while the key uses the encoder’s raw batch outputs, and the value incorporates the corresponding batch label information:
$\hat{Z}_{batch}^{\ell-1}\rightarrow{\rm Query}^{\ell},\hat{Z}_{batch}^{0}\rightarrow{\rm Key}^{\ell},\hat{Y}_{batch}\rightarrow{\rm Value}^{\ell}$
(13)
where the subscript “batch" refers to the group of instances in a specific batch.
- •
Inference: during inference, we utilize all instances from the training set as the input for both key and value:
$\hat{z}_{i}^{\ell-1}\rightarrow{\rm Query}^{\ell},\hat{Z}_{train}^{0}\rightarrow{\rm Key}^{\ell},\hat{Y}_{train}\rightarrow{\rm Value}^{\ell}$
(14)
where the subscript “i" denotes the $i$-th test instance and “train" the training dataset.

Following gorishniy2024tabr, we find that shared query/key projection weights occasionally improve model performance, prompting us to formalize this as a dataset-tunable hyperparameter. Meanwhile, we compute similarity via L2 distance instead of scaled dot-product. This eliminates the need for multi-head attention in IAIL.

Compared to somepalli2021saint, IAIL achieves a more parameter-efficient inter-instance attention mechanism by leveraging the encoder’s output, whereby each instance is compactly represented by a single [CLS] token. Incorporating label information enables the decoder to better learn the mapping from features to predictions. In Section [4.3](https://arxiv.org/html/2502.12507v3#S4.SS3), we demonstrate the efficacy of the IAIL block through ablation studies.

## 4 Experiments

In this section, we compare the proposed MAYA with SOTA transformer-based methods on real-world classification/regression datasets to show its superiority and analyze its attributes via ablation studies.

### 4.1 Experimental Setting

##### Datasets

Our datasets are divided into two parts. For the first part (referred to as Part I in the following content), we select 14 datasets from previous studies huang2020tabtransformer; somepalli2021saint; gorishniy2021revisiting, which have been widely utilized in the literature. For the second part (referred to as Part II in the following content), we employ a benchmark proposed by grinsztajn2022why, utilizing a broader range of datasets to test the superiority of MAYA. Detailed information and statistics about both parts of the datasets can be found in Table [6](https://arxiv.org/html/2502.12507v3#A2.T6) and Table [7](https://arxiv.org/html/2502.12507v3#A2.T7) in the Appendix. For details regarding the partitioning and preprocessing of the datasets, please refer to the Appendix [B.2](https://arxiv.org/html/2502.12507v3#A2.SS2).

##### Evaluation

We primarily follow gorishniy2021revisiting for evaluation, using 15 random seeds per dataset and reporting average results along with standard deviations on test sets. Accuracy is used as metric for classification and Root Mean Squared Error (RMSE) for regression. We also report the average rank (lower is better) across datasets to assess overall performance.

##### Comparison Methods

The proposed MAYA will be benchmarked against several SOTA transformer-based methods, including AutoInt (AT) song2019autoint, TabTransformer (TT) huang2020tabtransformer, SAINT (SNT) somepalli2021saint, FT-Transformer (FTT) gorishniy2021revisiting, ExcelFormer (EF) chen2024can and AMFormer (AMF) cheng2024arithmetic.
All methods are based on their officially released implementations. Note that, for Datasets Part II, due to limitations in computational resources and time consumption, we do not compare with SAINT.

##### Implementation Details

For comparison methods, we cite results directly from papers if available; otherwise, we tune hyperparameters within their search spaces (see Appendix [D](https://arxiv.org/html/2502.12507v3#A4)). When multiple results exist, we cite the best. For our method, we use Optuna akiba2019optuna for 100 trials to find the optimal hyperparameters, applied uniformly across 15 random seeds. All hyperparameter tuning is based on training and validation sets.

### 4.2 Main Results

The comparison results between MAYA and other transformer-based methods on Datasets Part I and Part II are shown in Table[1](https://arxiv.org/html/2502.12507v3#S4.T1) and Table [2](https://arxiv.org/html/2502.12507v3#S4.T2), respectively. Here, the first five datasets are selected from each of Part I and Part II for reporting. The results of the remaining datasets, along with their standard deviations, can be found in the Appendix [C](https://arxiv.org/html/2502.12507v3#A3).
It is evident that MAYA significantly outperforms other methods in terms of the average rank, indicating that the overall performance of MAYA is currently the best among transformer-based models.
Furthermore, as evidenced by the full results on Datasets Part II (Table [10](https://arxiv.org/html/2502.12507v3#A3.T10)), MAYA demonstrates a particularly significant advantage on relatively larger datasets (where the number of training instances exceeds 9000).

**Table 1: The results of all methods across the first 5 datasets on Datasets Part I. All datasets are referred to by their abbreviations. For specific information about each dataset, please refer to Table [6](https://arxiv.org/html/2502.12507v3#A2.T6). The scientific notation next to the dataset names indicates the scale of the results. The best result for each dataset is bolded. $\uparrow$ $\thicksim$ accuracy (higher is better), $\downarrow$ $\thicksim$ RMSE (lower is better).**
| Dataset | AI | TT | SNT | FTT | EF | AMF | MAYA (ours) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| AD $\uparrow$ | 0.8576 | 0.8509 | 0.8600 | 0.8588 | 0.8594 | 0.8594 | 0.8632 |
| BA $\uparrow$ | 0.9077 | 0.8998 | 0.9075 | 0.9095 | 0.9092 | 0.9082 | 0.9100 |
| BL $\uparrow$ | 0.7985 | 0.7775 | 0.8008 | 0.7995 | 0.8018 | 0.7990 | 0.8003 |
| CA $\downarrow$ | 0.5007 | 0.5936 | 0.4680 | 0.4564 | 0.4519 | 0.4626 | 0.4373 |
| DI${}_{\times 10^{3}}$ $\downarrow$ | 0.5372 | 0.7345 | 0.5466 | 0.5334 | 0.5331 | 0.5384 | 0.5324 |
| rank | 4.5714 | 6.4286 | 4.0000 | 3.5714 | 3.6429 | 3.5000 | 2.2857 |

**Table 2: The results of all methods across the first 5 datasets on Datasets Part II. All datasets are referred to by their Dataset IDs. For specific information about each dataset, please refer to Table [7](https://arxiv.org/html/2502.12507v3#A2.T7). The scientific notation next to the dataset names indicates the scale of the results. The best result for each dataset is bolded. $\uparrow$ $\thicksim$ accuracy (higher is better), $\downarrow$ $\thicksim$ RMSE (lower is better).**
| Dataset ID | AI | TT | FTT | EF | AMF | MAYA (ours) |
| --- | --- | --- | --- | --- | --- | --- |
| 01${}_{\times 10^{-3}}$ $\downarrow$ | 0.1594 | 0.1890 | 0.1578 | 0.1583 | 0.1568 | 0.1556 |
| 02${}_{\times 10^{2}}$ $\downarrow$ | 0.5215 | 1.1276 | 0.4301 | 0.4305 | 0.4328 | 0.4324 |
| 03 $\uparrow$ | 0.7675 | 0.7363 | 0.8009 | 0.7991 | 0.8009 | 0.7959 |
| 04 $\uparrow$ | 0.8580 | 0.8134 | 0.8620 | 0.8582 | 0.8590 | 0.8586 |
| 05 $\downarrow$ | 0.1573 | 0.1732 | 0.1502 | 0.1481 | 0.1490 | 0.1456 |
| rank | 4.6818 | 5.9545 | 2.7727 | 2.9545 | 2.6818 | 1.9545 |

### 4.3 Ablation Studies

In this section, we undertake an exhaustive analysis of the design characteristics of MAYA, utilizing six diverse datasets: BA/BL/QS/SE for classification and CA/SH for regression.

##### The Properties of MBA

We first investigate the impact of the MBA block’s design details by degrading its attention mechanism to a single MHA (i.e., setting the number of parallel branches to 1). To ensure a fair comparison, we maintain the same total number of heads in MHA as in the MBA block of MAYA (i.e., $\text{num_heads}_{\text{MHA}}=\text{num_heads}_{\text{per_branch}}\times\text{num_branch}$). Two configurations are tested:

- •
MHAhidden_size: aligns the hidden state dimension of MHA with that of the MBA block in MAYA;
- •
MHAhead_dim: aligns the subspace dimension per head of MHA with that of the MBA block in MAYA.

All other architectural parameters, e.g., FFN intermediate scaling factor (i.e., intermediate$_$factor) and the number of blocks (i.e., num$_$layers) remain consistent with MAYA, while other hyperparameters are tuned as described in Section [4.1](https://arxiv.org/html/2502.12507v3#S4.SS1.SSS0.Px4). Details are provided in Appendix [E.1](https://arxiv.org/html/2502.12507v3#A5.SS1).
Both configurations increase feature richness and diversity by expanding the number of heads. However,

- •
in MHAhidden_size, the subspace dimension per head is reduced, preserving overall feature richness but sacrificing per-subspace expressiveness;
- •
in MHAhead_dim, the hidden state dimension grows, increasing FFN parameter size and computational overhead.

These comparisons highlight the MBA block’s advantage in balancing feature complexity and parameter efficiency. Additionally, we perform a MHAfree configuration by tuning all parameters except the branch count (set to 1) and conduct ablation studies on branch weights within the MBA block by removing them entirely and retuning hyperparameters.

The experimental results (Table [3](https://arxiv.org/html/2502.12507v3#S4.T3)) demonstrate that MAYA outperforms all ablation configurations of the MBA block, including the branch-weight-free variant, which also surpasses single-branch MHA settings. This suggests that MBA’s branch-level feature averaging balances parameter efficiency and feature richness, while enhancing predictive capability akin to feature pooling. Branch weights further improve performance by promoting collaborative learning among branches. In contrast, MHAhidden_size (which increases head count but reduces subspace dimensionality) yields the weakest results, indicating that such dimensionality reduction deteriorates predictive accuracy.

**Table 3: The results of ablation studies regarding the MBA block. MHA denotes the use of only one attention branch, where the subscript “hidden$_$size" indicates that the hidden$_$size of this attention branch is consistent with that of MBA in MAYA. The subscript “head$_$dim" signifies that the head$_$dim of this attention branch aligns with that of MBA in MAYA. The subscript “free" indicates that all parameters are tuned. w/o $W_{B}$ represents the removal of the branch weight from MBA. The multiples $N\times$ following the results of MHAhidden_size and MHAhead_dim indicate that the parameter count of the corresponding encoder is $N$ times that of MAYA. The computation details of $N$ are provided in Appendix [E.2](https://arxiv.org/html/2502.12507v3#A5.SS2). The best result for each dataset is in bold. $\uparrow$ $\thicksim$ accuracy (higher is better), $\downarrow$ $\thicksim$ RMSE (lower is better).**
| Dataset | MHAhidden_size | MHAhead_dim | MHAfree | w/o $W_{B}$ | MAYA |
| --- | --- | --- | --- | --- | --- |
| BA $\uparrow$ | 0.9084 (0.5$\times$) | 0.9101 (7.3$\times$) | 0.9092 | 0.9088 | 0.9100 |
| BL $\uparrow$ | 0.8000 (0.4$\times$) | 0.7995 (10.7$\times$) | 0.7969 | 0.8001 | 0.8003 |
| CA $\downarrow$ | 0.4479 (0.4$\times$) | 0.4412 (12.0$\times$) | 0.4405 | 0.4473 | 0.4373 |
| QS $\uparrow$ | 0.8578 (0.2$\times$) | 0.8559 (12.8$\times$) | 0.8288 | 0.8385 | 0.8515 |
| SE $\uparrow$ | 0.9275 (0.4$\times$) | 0.9278 (6.4$\times$) | 0.9287 | 0.9306 | 0.9299 |
| SH $\downarrow$ | 0.3161 (0.5$\times$) | 0.3166 (4.2$\times$) | 0.3143 | 0.3154 | 0.3126 |
| rank | 3.8333 | 3.1667 | 3.3333 | 3.0000 | 1.6667 |

##### The Properties of IAIL

We conduct two experiments to investigate the impact of IAIL. In the first experiment, we remove the decoder and directly connect the Predictor to the encoder, denoted as “w/o decoder". In the second experiment, we retain the decoder but do not incorporate labels within the IAIL block, using the same input for both value and key, referred to as “IAIL w/o labels". All other hyperparameters are tuned according to the description in Section [4.1](https://arxiv.org/html/2502.12507v3#S4.SS1.SSS0.Px4). The experimental results are presented in Table [4](https://arxiv.org/html/2502.12507v3#S4.T4). It is evident that removing the decoder (i.e. removing the IAIL blocks), significantly impairs the model’s predictive capability. Even without utilizing labels, integrating the decoder enhances the model’s performance. Furthermore, incorporating labels leads to a further improvement in the model’s predictive ability.

**Table 4: The results of ablation study regarding the IAIL block. The best result for each dataset is in bold. $\uparrow$ $\thicksim$ accuracy (higher is better), $\downarrow$ $\thicksim$ RMSE (lower is better).**
| Dataset | BA $\uparrow$ | BL $\uparrow$ | CA $\downarrow$ | QS $\uparrow$ | SE $\uparrow$ | SH $\downarrow$ | rank |
| --- | --- | --- | --- | --- | --- | --- | --- |
| w/o decoder | 0.9088 | 0.7980 | 0.4411 | 0.8367 | 0.9248 | 0.3133 | 3.0000 |
| IAIL w/o labels | 0.9093 | 0.7983 | 0.4388 | 0.8392 | 0.9296 | 0.3117 | 1.8333 |
| MAYA | 0.9100 | 0.8003 | 0.4373 | 0.8515 | 0.9299 | 0.3126 | 1.1667 |

**Table 5: The results of ablation study regarding the activation following the tokenizer. The best result for each dataset is in bold. $\uparrow$ $\thicksim$ accuracy (higher is better), $\downarrow$ $\thicksim$ RMSE (lower is better).**
| Dataset | BA $\uparrow$ | BL $\uparrow$ | CA $\downarrow$ | QS $\uparrow$ | SE $\uparrow$ | SH $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| w/o activation in tokenizer | 0.9094 | 0.7997 | 0.4401 | 0.8346 | 0.9275 | 0.3129 |
| MAYA | 0.9100 | 0.8003 | 0.4373 | 0.8515 | 0.9299 | 0.3126 |

##### The Influence of Activation after Tokenizer

We attempt to remove the activation function following the tokenizer, while maintaining all other configurations constant, and perform hyperparameter tuning according to the protocol outlined in Section [4.1](https://arxiv.org/html/2502.12507v3#S4.SS1.SSS0.Px4). This configuration is subsequently compared against MAYA that retains the activation function, to ascertain its impact. The results, presented in Table [5](https://arxiv.org/html/2502.12507v3#S4.T5), reveal that the incorporation of a ReLU activation function subsequent to tokenizer yields a marked enhancement in the model’s predictive accuracy.

## 5 Conclusion

This paper introduces MAYA, a novel transformer-based encoder-decoder architecture for tabular data. Its dual-attention mechanism enhances intra- and inter-instance information fusion, while the Multi-Branch Attention (MBA) block leverages multiple attention branches to extract diverse features. Experiments on public datasets demonstrate MAYA’s efficacy in tabular data processing. Ablation studies validate the effectiveness of MBA and IAIL blocks, while also demonstrating that incorporating activation functions following the tokenizer enhances the model’s predictive performance.

## References

- (1)
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama.
Optuna: A next-generation hyperparameter optimization framework.
In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2623–2631, 2019.
- (2)
Sercan Ö Arik and Tomas Pfister.
Tabnet: Attentive interpretable tabular learning.
In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 6679–6687, 2021.
- (3)
David Bonet, Daniel Mas Montserrat, Xavier Giró-i Nieto, and Alexander G Ioannidis.
Hyperfast: Instant classification for tabular data.
In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 11114–11123, 2024.
- (4)
Jintai Chen, Kuanlun Liao, Yao Wan, Danny Z Chen, and Jian Wu.
Danets: Deep abstract networks for tabular data classification and regression.
In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 3930–3938, 2022.
- (5)
Jintai Chen, Jiahuan Yan, Qiyuan Chen, Danny Z Chen, Jian Wu, and Jimeng Sun.
Can a deep learning model be a sure bet for tabular prediction?
In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 288–296, 2024.
- (6)
Kuan-Yu Chen, Ping-Han Chiang, Hsin-Rung Chou, Ting-Wei Chen, and Darby Tien-Hao Chang.
Trompt: towards a better deep neural network for tabular data.
In Proceedings of the 40th International Conference on Machine Learning, 2023.
- (7)
Tianqi Chen and Carlos Guestrin.
Xgboost: A scalable tree boosting system.
In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785–794, 2016.
- (8)
Yi Cheng, Renjun Hu, Haochao Ying, Xing Shi, Jian Wu, and Wei Lin.
Arithmetic feature interaction is necessary for deep tabular learning.
In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 11516–11524, 2024.
- (9)
Pierre Dillenbourg.
Collaborative learning: Cognitive and computational approaches. advances in learning and instruction series.
ERIC, 1999.
- (10)
Yury Gorishniy, Ivan Rubachev, Nikolay Kartashev, Daniil Shlenskii, Akim Kotelnikov, and Artem Babenko.
Tabr: Tabular deep learning meets nearest neighbors.
In The Twelfth International Conference on Learning Representations, 2024.
- (11)
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko.
Revisiting deep learning models for tabular data.
Advances in Neural Information Processing Systems, 34:18932–18943, 2021.
- (12)
Leo Grinsztajn, Edouard Oyallon, and Gael Varoquaux.
Why do tree-based models still outperform deep learning on typical tabular data?
In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
- (13)
Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin.
Tabtransformer: Tabular data modeling using contextual embeddings.
arXiv preprint arXiv:2012.06678, 2020.
- (14)
Zhanghan Ke, Di Qiu, Kaican Li, Qiong Yan, and Rynson WH Lau.
Guided collaborative training for pixel-wise semi-supervised learning.
In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIII 16, pages 429–445. Springer, 2020.
- (15)
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter.
Self-normalizing neural networks.
Advances in neural information processing systems, 30, 2017.
- (16)
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay.
Scikit-learn: Machine learning in python.
Journal of Machine Learning Research, 12(85):2825–2830, 2011.
- (17)
Sergei Popov, Stanislav Morozov, and Artem Babenko.
Neural oblivious decision ensembles for deep learning on tabular data.
In International Conference on Learning Representations, 2020.
- (18)
Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin.
Catboost: unbiased boosting with categorical features.
Advances in neural information processing systems, 31, 2018.
- (19)
Noam Shazeer.
Glu variants improve transformer.
arXiv preprint arXiv:2002.05202, 2020.
- (20)
Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein.
Saint: Improved neural networks for tabular data via row attention and contrastive pre-training.
arXiv preprint arXiv:2106.01342, 2021.
- (21)
Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang.
Autoint: Automatic feature interaction learning via self-attentive neural networks.
In Proceedings of the 28th ACM international conference on information and knowledge management, pages 1161–1170, 2019.
- (22)
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.
Llama: Open and efficient foundation language models.
arXiv preprint arXiv:2302.13971, 2023.
- (23)
A Vaswani.
Attention is all you need.
Advances in Neural Information Processing Systems, 2017.
- (24)
Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi.
Dcn v2: Improved deep & cross network and practical lessons for web-scale learning to rank systems.
In Proceedings of the web conference 2021, pages 1785–1797, 2021.
- (25)
Chenwei Xu, Yu-Chao Huang, Jerry Yao-Chieh Hu, Weijian Li, Ammar Gilani, Hsi-Sheng Goan, and Han Liu.
Bishop: Bi-directional cellular learning for tabular data with generalized sparse modern hopfield model.
In Forty-first International Conference on Machine Learning (ICML), 2024.
- (26)
Han-Jia Ye, Huai-Hong Yin, and De-Chuan Zhan.
Modern neighborhood components analysis: A deep tabular baseline two decades later.
arXiv preprint arXiv:2407.03257, 2024.
- (27)
Zhi Zhou, Yu-Kun Yang, Lan-Zhe Guo, and Yu-Feng Li.
Fully test-time adaptation for tabular data.
In Proceedings of the 39th AAAI conference on Artificial Intelligence, 2025.

## Appendix A Analysis of MBA

To obtain Fig.[1](https://arxiv.org/html/2502.12507v3#S1.F1) (b), we perform hyperparameter tuning on the shrutime (SH) dataset in accordance with the experimental configuration (Section [4.1](https://arxiv.org/html/2502.12507v3#S4.SS1)), yielding the MAYA comprising 10 layers, with 3 parallel branches per layer and 4 attention heads per branch. The 5th layer is selected for visualization purposes. Furthermore, we utilize CatBoost prokhorenkova2018catboost to rank feature importance, identifying features 10, 2, 5, and 6 as the top four most important features. These features also frequently make substantial contributions to the [CLS] token, as exemplified by the red boxes in Fig.[1](https://arxiv.org/html/2502.12507v3#S1.F1)(b). Notably, the prominent contributions of these features typically manifest across distinct branches, thereby highlighting the necessity of employing multiple branches.

## Appendix B Details on Datasets

### B.1 Information and Statistics

In this section, we provide some detailed information and statistics about the datasets used in our experiments. Specifically, Table [6](https://arxiv.org/html/2502.12507v3#A2.T6) presents the detailed information of the 14 datasets in Datasets Part I, which we summarized from previous works huang2020tabtransformer; somepalli2021saint; gorishniy2021revisiting and are widely used for evaluating the effectiveness of methods. Table [7](https://arxiv.org/html/2502.12507v3#A2.T7) provides the detailed information of all the datasets in Datasets Part II, which originate from grinsztajn2022why, and we have selected 22 datasets from it.

**Table 6: The details of Datasets Part I. “#Num" and “#Cat" denote the number of numerical and categorical features, respectively. Size denotes the number of all instances.**
| Name | Abbr | Task | Size | #Num | #Cat | #Class | URL |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Adult | AD | cls | 48842 | 6 | 8 | 2 | [adult](http://automl.chalearn.org/data) |
| Bank | BA | cls | 45211 | 7 | 9 | 2 | [bank](https://archive.ics.uci.edu/ml/datasets/Bank+Marketing) |
| Blastchar | BL | cls | 7032 | 3 | 16 | 2 | [blastchar](https://www.kaggle.com/blastchar/telco-customer-churn) |
| California Housing | CA | reg | 20640 | 8 | 0 | - | [california$_$housing](https://www.dcc.fc.up.pt/~ltorgo/Regression/cal_housing.html) |
| Diamonds | DI | reg | 53940 | 6 | 3 | - | [diamonds](https://www.kaggle.com/datasets/joebeachcapital/diamonds) |
| Helena | HE | cls | 65196 | 27 | 0 | 100 | [helena](https://www.openml.org/search?type=data&status=active&id=41169&sort=runs) |
| Income | IN | cls | 48842 | 6 | 8 | 2 | [income](https://www.kaggle.com/lodetomasi1995/income-classification) |
| Jannis | JA | cls | 83733 | 54 | 0 | 4 | [jannis](https://www.openml.org/search?type=data&status=active&id=41168&sort=runs) |
| Otto Group Products | OT | cls | 61878 | 93 | 0 | 9 | [otto$_$group$_$products](https://www.kaggle.com/c/otto-group-product-classification-challenge/data) |
| QSAR Bio | QS | cls | 1055 | 31 | 10 | 2 | [qsar$_$bio](https://archive.ics.uci.edu/dataset/254/qsar+biodegradation) |
| Seismic Bumps | SE | cls | 2584 | 14 | 4 | 2 | [seismic$_$bumps](https://archive.ics.uci.edu/dataset/266/seismic+bumps) |
| Shrutime | SH | reg | 10000 | 4 | 6 | - | [shrutime](https://www.openml.org/search?type=data&status=active&id=45062&sort=runs) |
| Spambase | SP | cls | 4601 | 57 | 0 | 2 | [spambase](https://archive.ics.uci.edu/ml/datasets/spambase) |
| Volume | VO | reg | 50993 | 53 | 0 | - | [volume](https://archive.ics.uci.edu/dataset/363/facebook+comment+volume+dataset) |

**Table 7: The details of Datasets Part II. “#Num" and “#Cat" denote the number of numerical and categorical features, respectively. #Training means the number of instances in the training set.**
| Name | Dataset ID | Task | #Training | #Num | #Cat | #Class |
| --- | --- | --- | --- | --- | --- | --- |
| Ailerons | 01 | reg | 9625 | 33 | 0 | - |
| BikeSharingDemand | 02 | reg | 10000 | 6 | 5 | - |
| KDDCup09upselling | 03 | cls | 3589 | 34 | 15 | 2 |
| MagicTelescope | 04 | cls | 9363 | 10 | 0 | 2 |
| MiamiHousing2016 | 05 | reg | 9752 | 13 | 0 | - |
| OnlineNewsPopularity | 06 | reg | 9999 | 45 | 15 | - |
| blackfriday | 07 | reg | 50000 | 4 | 5 | - |
| california | 08 | reg | 10000 | 8 | 0 | - |
| compass | 09 | cls | 10000 | 8 | 9 | 2 |
| cpuact | 10 | reg | 5734 | 21 | 0 | - |
| credit | 11 | cls | 10000 | 10 | 0 | 2 |
| electricity | 12 | cls | 10000 | 7 | 1 | 2 |
| elevators | 13 | reg | 10000 | 16 | 0 | - |
| fifa | 14 | reg | 10000 | 5 | 0 | - |
| housesales | 15 | reg | 10000 | 15 | 2 | - |
| houses | 16 | reg | 10000 | 8 | 0 | - |
| kddipumsla97-small | 17 | cls | 3631 | 20 | 0 | 2 |
| particulate-matter-ukair-2017 | 18 | reg | 50000 | 3 | 3 | - |
| phoneme | 19 | cls | 2220 | 5 | 0 | 2 |
| pol | 20 | reg | 10000 | 26 | 0 | - |
| rl | 21 | cls | 3749 | 5 | 7 | 2 |
| winequality | 22 | reg | 4547 | 11 | 0 | - |

### B.2 Preprocessing

For Datasets Part I, we generally adhere to the preprocessing methodology outlined by gorishniy2021revisiting, partitioning the dataset into training, validation, and test sets in ratios of 64%/16%/20%, respectively. Normalization is performed using quantile transformation from Scikit-learn library JMLR:v12:pedregosa11a on the datasets.

Regarding Datasets Part II, we follow the preprocessing procedures specified in grinsztajn2022why; gorishniy2024tabr. We utilize the identical partitioning ratios as those in grinsztajn2022why and truncate the training set (to 10,000 or 50,000 instances). In accordance with gorishniy2024tabr, if a dataset comprises $n$ splits (corresponding to the $n$-fold cross-validation mentioned in grinsztajn2022why), each split is treated as an independent dataset for tuning and evaluation. Subsequently, the results across these $n$ splits are averaged to derive the final outcome for the dataset, effectively averaging over $n\times 15$ seeds. The normalization method employed is identical to that used in Datasets Part I.

## Appendix C More Results

In this section, we present all the experimental results, including the average results and standard deviations of each method on Datasets Part I and Part II, as well as the average rankings. Specifically, the average results for Datasets Part I are shown in Table [8](https://arxiv.org/html/2502.12507v3#A3.T8), with the corresponding standard deviations in Table [9](https://arxiv.org/html/2502.12507v3#A3.T9). The average results for Datasets Part II are presented in Table [10](https://arxiv.org/html/2502.12507v3#A3.T10), and their standard deviations are in Table [11](https://arxiv.org/html/2502.12507v3#A3.T11).
The experimental results underscore the superiority of MAYA, particularly evident in the Datasets Part II findings (Table [10](https://arxiv.org/html/2502.12507v3#A3.T10)), where its advantages become pronounced on relatively larger-scale datasets (where the number of training instances exceeds 9000).

**Table 8: The results of all methods across 14 datasets on Datasets Part I. The upward arrow ($\uparrow$) indicates the accuracy for classification tasks (higher is better), while the downward arrow ($\downarrow$) represents the RMSE for regression tasks (lower is better). The average rank results also follow the principle that lower is better. The scientific notation next to the dataset names indicates the scale of the results. The best result for each dataset is bolded.**
| Dataset | AI | TT | SNT | FTT | EF | AMF | MAYA (ours) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| AD $\uparrow$ | 0.8576 | 0.8509 | 0.8600 | 0.8588 | 0.8594 | 0.8594 | 0.8632 |
| BA $\uparrow$ | 0.9077 | 0.8998 | 0.9075 | 0.9095 | 0.9092 | 0.9082 | 0.9100 |
| BL $\uparrow$ | 0.7985 | 0.7775 | 0.8008 | 0.7995 | 0.8018 | 0.7990 | 0.8003 |
| CA $\downarrow$ | 0.5007 | 0.5936 | 0.4680 | 0.4564 | 0.4519 | 0.4626 | 0.4373 |
| DI${}_{\times 10^{3}}$ $\downarrow$ | 0.5372 | 0.7345 | 0.5466 | 0.5334 | 0.5331 | 0.5384 | 0.5324 |
| HE $\uparrow$ | 0.3811 | 0.3589 | 0.3856 | 0.3890 | 0.3802 | 0.3882 | 0.3831 |
| IN $\uparrow$ | 0.8605 | 0.8535 | 0.8661 | 0.8633 | 0.8665 | 0.8625 | 0.8681 |
| JA $\uparrow$ | 0.7178 | 0.7084 | 0.7111 | 0.7306 | 0.7262 | 0.7281 | 0.7207 |
| OT $\uparrow$ | 0.8084 | 0.8019 | 0.8120 | 0.8082 | 0.8035 | 0.8074 | 0.7984 |
| QS $\uparrow$ | 0.8357 | 0.8461 | 0.8452 | 0.8330 | 0.8389 | 0.8537 | 0.8515 |
| SE $\uparrow$ | 0.9309 | 0.9243 | 0.9259 | 0.9275 | 0.9282 | 0.9253 | 0.9299 |
| SH $\downarrow$ | 0.3255 | 0.3464 | 0.3131 | 0.3222 | 0.3197 | 0.3149 | 0.3126 |
| SP $\uparrow$ | 0.9362 | 0.9393 | 0.9294 | 0.9424 | 0.9346 | 0.9435 | 0.9425 |
| VO${}_{\times 10^{2}}$ $\downarrow$ | 0.3987 | 0.4940 | 0.4171 | 0.4210 | 0.4253 | 0.4097 | 0.4037 |
| rank | 4.5714 | 6.4286 | 4.0000 | 3.5714 | 3.6429 | 3.5000 | 2.2857 |

**Table 9: The standard deviations of all methods across 14 datasets on Datasets Part I.**
| Dataset | AI | TT | SNT | FTT | EF | AMF | MAYA (ours) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| AD $\uparrow$ | 0.0011 | 0.0016 | 0.0020 | 0.0014 | 0.0014 | 0.0012 | 0.0017 |
| BA $\uparrow$ | 0.0016 | 0.0016 | 0.0012 | 0.0012 | 0.0012 | 0.0017 | 0.0012 |
| BL $\uparrow$ | 0.0031 | 0.0037 | 0.0025 | 0.0017 | 0.0027 | 0.0042 | 0.0021 |
| CA $\downarrow$ | 0.0034 | 0.0022 | 0.0050 | 0.0036 | 0.0049 | 0.0051 | 0.0034 |
| DI $\downarrow$ | 5.8841 | 12.317 | 2.7438 | 4.5085 | 6.8204 | 4.0224 | 6.3196 |
| HE $\uparrow$ | 0.0015 | 0.0020 | 0.0025 | 0.0014 | 0.0023 | 0.0019 | 0.0021 |
| IN $\uparrow$ | 0.0009 | 0.0013 | 0.0022 | 0.0026 | 0.0015 | 0.0017 | 0.0015 |
| JA $\uparrow$ | 0.0018 | 0.0020 | 0.0015 | 0.0020 | 0.0021 | 0.0020 | 0.0021 |
| OT $\uparrow$ | 0.0020 | 0.0021 | 0.0020 | 0.0028 | 0.0027 | 0.0029 | 0.0049 |
| QS $\uparrow$ | 0.0119 | 0.0642 | 0.0126 | 0.0159 | 0.0170 | 0.0147 | 0.0091 |
| SE $\uparrow$ | 0.0031 | 0.0023 | 0.0027 | 0.0067 | 0.0067 | 0.0028 | 0.0024 |
| SH $\downarrow$ | 0.0028 | 0.0027 | 0.0014 | 0.0044 | 0.0014 | 0.0019 | 0.0020 |
| SP $\uparrow$ | 0.0027 | 0.0030 | 0.0029 | 0.0036 | 0.0031 | 0.0037 | 0.0039 |
| VO $\downarrow$ | 0.7686 | 3.2592 | 1.6812 | 1.3830 | 0.9772 | 1.4935 | 1.1377 |

**Table 10: The results of all methods across 22 datasets on Datasets Part II. All datasets are referred to by their Dataset IDs. For specific information about each dataset, please refer to Table [7](https://arxiv.org/html/2502.12507v3#A2.T7). The upward arrow ($\uparrow$) indicates the accuracy for classification tasks (higher is better), while the downward arrow ($\downarrow$) represents the RMSE for regression tasks (lower is better). The average rank results also follow the principle that lower is better. The scientific notation next to the dataset names indicates the scale of the results. The best result for each dataset is bolded.**
| Dataset ID | AI | TT | FTT | EF | AMF | MAYA (ours) |
| --- | --- | --- | --- | --- | --- | --- |
| 01${}_{\times 10^{-3}}$ $\downarrow$ | 0.1594 | 0.1890 | 0.1578 | 0.1583 | 0.1568 | 0.1556 |
| 02${}_{\times 10^{2}}$ $\downarrow$ | 0.5215 | 1.1276 | 0.4301 | 0.4305 | 0.4328 | 0.4324 |
| 03 $\uparrow$ | 0.7675 | 0.7363 | 0.8009 | 0.7991 | 0.8009 | 0.7959 |
| 04 $\uparrow$ | 0.8580 | 0.8134 | 0.8620 | 0.8582 | 0.8590 | 0.8586 |
| 05 $\downarrow$ | 0.1573 | 0.1732 | 0.1502 | 0.1481 | 0.1490 | 0.1456 |
| 06 $\downarrow$ | 0.8653 | 0.8751 | 0.8639 | 0.8623 | 0.8646 | 0.8666 |
| 07 $\downarrow$ | 0.3698 | 0.4732 | 0.3673 | 0.3661 | 0.3666 | 0.3650 |
| 08 $\downarrow$ | 0.1441 | 0.1837 | 0.1381 | 0.1374 | 0.1386 | 0.1341 |
| 09 $\uparrow$ | 0.7618 | 0.7499 | 0.7729 | 0.7478 | 0.7627 | 0.7912 |
| 10×10 $\downarrow$ | 0.2332 | 0.5714 | 0.2219 | 0.2261 | 0.2186 | 0.2226 |
| 11 $\uparrow$ | 0.7745 | 0.7595 | 0.7743 | 0.7747 | 0.7754 | 0.7730 |
| 12 $\uparrow$ | 0.8203 | 0.8000 | 0.8339 | 0.8339 | 0.8311 | 0.8434 |
| 13${}_{\times 10^{-2}}$ $\downarrow$ | 0.1861 | 0.2802 | 0.1828 | 0.1854 | 0.1830 | 0.1795 |
| 14 $\downarrow$ | 0.7973 | 1.1280 | 0.7922 | 0.7923 | 0.7905 | 0.7873 |
| 15 $\downarrow$ | 0.1754 | 0.1999 | 0.1668 | 0.1677 | 0.1670 | 0.1653 |
| 16 $\downarrow$ | 0.2322 | 0.2926 | 0.2247 | 0.2281 | 0.2290 | 0.2269 |
| 17 $\uparrow$ | 0.8800 | 0.8771 | 0.8823 | 0.8827 | 0.8841 | 0.8832 |
| 18 $\downarrow$ | 0.3775 | 0.5275 | 0.3750 | 0.3696 | 0.3733 | 0.3658 |
| 19 $\uparrow$ | 0.8587 | 0.8207 | 0.8667 | 0.8692 | 0.8708 | 0.8675 |
| 20×10 $\downarrow$ | 0.3797 | 1.9690 | 0.2580 | 0.2824 | 0.2667 | 0.2547 |
| 21 $\uparrow$ | 0.6832 | 0.6206 | 0.7234 | 0.7358 | 0.7288 | 0.7662 |
| 22 $\downarrow$ | 0.6728 | 0.6931 | 0.6802 | 0.6794 | 0.6802 | 0.6706 |
| rank | 4.6818 | 5.9545 | 2.7727 | 2.9545 | 2.6818 | 1.9545 |

**Table 11: The standard deviations of all methods across 22 datasets on Datasets Part II.**
| Dataset ID | AI | TT | FTT | EF | AMF | MAYA (ours) |
| --- | --- | --- | --- | --- | --- | --- |
| 01 $\downarrow$ | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| 02 $\downarrow$ | 10.370 | 0.9339 | 0.4644 | 0.6977 | 0.7884 | 0.8374 |
| 03 $\uparrow$ | 0.0096 | 0.0115 | 0.0096 | 0.0137 | 0.0090 | 0.0112 |
| 04 $\uparrow$ | 0.0056 | 0.0067 | 0.0035 | 0.0047 | 0.0073 | 0.0053 |
| 05 $\downarrow$ | 0.0041 | 0.0062 | 0.0032 | 0.0030 | 0.0028 | 0.0015 |
| 06 $\downarrow$ | 0.0009 | 0.0014 | 0.0008 | 0.0015 | 0.0009 | 0.0005 |
| 07 $\downarrow$ | 0.0008 | 0.0197 | 0.0007 | 0.0008 | 0.0008 | 0.0006 |
| 08 $\downarrow$ | 0.0009 | 0.0009 | 0.0011 | 0.0015 | 0.0012 | 0.0012 |
| 09 $\uparrow$ | 0.0065 | 0.0072 | 0.0071 | 0.0076 | 0.0123 | 0.0257 |
| 10 $\downarrow$ | 0.0535 | 0.3803 | 0.0694 | 0.0694 | 0.0531 | 0.0551 |
| 11 $\uparrow$ | 0.0043 | 0.0044 | 0.0044 | 0.0045 | 0.0042 | 0.0031 |
| 12 $\uparrow$ | 0.0020 | 0.0015 | 0.0027 | 0.0030 | 0.0023 | 0.0046 |
| 13 $\downarrow$ | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| 14 $\downarrow$ | 0.7973 | 1.1280 | 0.7922 | 0.7923 | 0.0120 | 0.7873 |
| 15 $\downarrow$ | 0.0010 | 0.0008 | 0.0007 | 0.0008 | 0.0007 | 0.0005 |
| 16 $\downarrow$ | 0.0024 | 0.0021 | 0.0008 | 0.0018 | 0.0044 | 0.0018 |
| 17 $\uparrow$ | 0.0044 | 0.0051 | 0.0062 | 0.0049 | 0.0060 | 0.0050 |
| 18 $\downarrow$ | 0.0019 | 0.0003 | 0.0014 | 0.0011 | 0.0019 | 0.0009 |
| 19 $\uparrow$ | 0.0121 | 0.0094 | 0.0090 | 0.0110 | 0.0116 | 0.0128 |
| 20 $\downarrow$ | 0.1161 | 0.1567 | 0.1317 | 0.1240 | 0.1253 | 0.1241 |
| 21 $\uparrow$ | 0.0127 | 0.0159 | 0.0148 | 0.0158 | 0.0145 | 0.0106 |
| 22 $\downarrow$ | 0.0153 | 0.0166 | 0.0155 | 0.0055 | 0.0149 | 0.0250 |

## Appendix D Hyperparameter Tuning Space

Here we provide hyperparameter tuning spaces used for Optuna tuning for all methods in Section [4.2](https://arxiv.org/html/2502.12507v3#S4.SS2).

### D.1 MAYA

The hyperparameter tuning space for MAYA is presented in Table [12](https://arxiv.org/html/2502.12507v3#A4.T12).

### D.2 AutoInt

The hyperparameter tuning space for AutoInt is presented in Table [13](https://arxiv.org/html/2502.12507v3#A4.T13).

### D.3 TabTransformer

The hyperparameter tuning space for TabTransformer is presented in Table [14](https://arxiv.org/html/2502.12507v3#A4.T14).

### D.4 SAINT

The hyperparameter tuning space for SAINT is presented in Table [15](https://arxiv.org/html/2502.12507v3#A4.T15).

### D.5 FT-Transformer

The hyperparameter tuning space for FT-Transformer is presented in Table [16](https://arxiv.org/html/2502.12507v3#A4.T16).

### D.6 ExcelFormer

The hyperparameter tuning space for ExcelFormer is presented in Table [17](https://arxiv.org/html/2502.12507v3#A4.T17).

### D.7 AMFormer

The hyperparameter tuning space for AMFormer is presented in Table [18](https://arxiv.org/html/2502.12507v3#A4.T18).

**Table 12: Hyperparameter tuning space for MAYA.**
|  | Parameter | Distribution |
| --- | --- | --- |
| encoder | num$_$layers | $\mathrm{UniformInt}[1,16]$ |
| hidden$_$size | $\mathrm{UniformInt}[64,256]$ |  |
| num$_$heads | $\mathrm{Categorical}[1,4,8,32]$ |  |
| num$_$branch | $\mathrm{UniformInt}[2,8]$ |  |
| intermediate$_$factor | $\mathrm{Categorical}[0.8,1,1.3,2]$ |  |
| dropout | $\mathrm{Uniform}[0,0.3]$ |  |
| add$_$act | $\mathrm{Categorical}[\mathrm{true,false}]$ |  |
| if$_$bias | $\mathrm{Categorical}[\mathrm{true,false}]$ |  |
| act$_$type | $\mathrm{Categorical}[\mathrm{relu,prelu}]$ |  |
| decoder | num$_$decoder$_$layers | $\mathrm{UniformInt}[1,16]$ |
| decoder$_$intermediate$_$factor | $\mathrm{Categorical}[0.8,1,1.3,2]$ |  |
| decoder$_$dropout | $\mathrm{Uniform}[0,0.3]$ |  |
| decoder$_$if$_$bias | $\mathrm{Categorical}[\mathrm{true,false}]$ |  |
| decoder$_$act$_$type | $\mathrm{Categorical}[\mathrm{relu,prelu}]$ |  |
| qk$_$shared$_$weights | $\mathrm{Categorical}[\mathrm{true,false}]$ |  |
| others | batch$_$size | $\mathrm{Categorical}[\mathrm{1024,2048,4096}]$ |
| learning$_$rate | $\mathrm{LogUniform}[1e\text{-}5,5e\text{-}3]$ |  |
| weight$_$decay | $\mathrm{Uniform}[0,0.3]$ |  |

**Table 13: Hyperparameter tuning space for AutoInt.**
| Parameter | Distribution |
| --- | --- |
| n$_$layers | $\mathrm{UniformInt}[1,6]$ |
| d$_$token | $\mathrm{UniformInt}[8,64]$ |
| residual$_$dropout | $\{0,\mathrm{Uniform}[0,0.2]\}$ |
| attention$_$dropout | $\{0,\mathrm{Uniform}[0,0.5]\}$ |
| learning$_$rate | $\mathrm{LogUniform}[1e\text{-}5,1e\text{-}3]$ |
| weight$_$decay | $\mathrm{LogUniform}[1e\text{-}6,1e\text{-}3]$ |

**Table 14: Hyperparameter tuning space for TabTransformer.**
| Parameter | Distribution |
| --- | --- |
| dim | $\mathrm{Categorical}[32,64,128,256]$ |
| depth | $\mathrm{Categorical}[1,2,3,6,12]$ |
| heads | $\mathrm{Categorical}[2,4,8]$ |
| attn$_$dropout | $\mathrm{Uniform}[0,0.5]$ |
| ffn$_$dropout | $\mathrm{Uniform}[0,0.5]$ |
| learning$_$rate | $\mathrm{LogUniform}[1e\text{-}6,1e\text{-}3]$ |
| weight$_$decay | $\mathrm{LogUniform}[1e\text{-}6,1e\text{-}1]$ |

**Table 15: Hyperparameter tuning space for SAINT.**
| Parameter | Distribution |
| --- | --- |
| dim | $\mathrm{Categorical}[16,32,64]$ |
| depth | $\mathrm{Categorical}[4,6]$ |
| heads | $\mathrm{Categorical}[4,8]$ |
| attn$_$dropout | $\mathrm{Uniform}[0,0.5]$ |
| ffn$_$dropout | $\mathrm{Uniform}[0,0.5]$ |
| attn$_$type | $\{\mathrm{colrow},\mathrm{Categorical}[\mathrm{colrow,row,col}]\}$ |
| learning$_$rate | $\mathrm{LogUniform}[3e\text{-}5,1e\text{-}3]$ |
| weight$_$decay | $\{0,\mathrm{LogUniform}[1e\text{-}6,1e\text{-}4]\}$ |

**Table 16: Hyperparameter tuning space for FT-Transformer.**
| Parameter | Distribution |
| --- | --- |
| n$_$layers | $\mathrm{UniformInt}[1,4]$ |
| d$_$token | $\mathrm{UniformInt}[64,512]$ |
| residual$_$dropout | $\{0,\mathrm{Uniform}[0,0.2]\}$ |
| attention$_$dropout | $\mathrm{Uniform}[0,0.5]$ |
| ffn$_$dropout | $\mathrm{Uniform}[0,0.5]$ |
| ffn$_$factor | $\mathrm{Uniform}[\nicefrac{{2}}{{3}},\nicefrac{{8}}{{3}}]$ |
| learning$_$rate | $\mathrm{LogUniform}[1e\text{-}5,1e\text{-}3]$ |
| weight$_$decay | $\mathrm{LogUniform}[1e\text{-}6,1e\text{-}3]$ |

**Table 17: Hyperparameter tuning space for ExcelFormer.**
| Parameter | Distribution |
| --- | --- |
| n$_$layers | $\mathrm{UniformInt}[2,5]$ |
| d$_$token | $\mathrm{Categorical}[64,128,256]$ |
| n$_$heads | $\mathrm{Categorical}[4,8,16,32]$ |
| residual$_$dropout | $\mathrm{Uniform}[0,0.5]$ |
| mix$_$type | $\mathrm{Categorical}[\mathrm{none},\mathrm{feat}$_$\mathrm{mix},\mathrm{hidden}$_$\mathrm{mix}]$ |
| learning$_$rate | $\mathrm{LogUniform}[3e\text{-}5,1e\text{-}3]$ |
| weight$_$decay | $\{0,\mathrm{LogUniform}[1e\text{-}6,1e\text{-}3]\}$ |

**Table 18: Hyperparameter tuning space for AMFormer.**
| Parameter | Distribution |
| --- | --- |
| dim | $\mathrm{Categorical}[8,16,32,64,128]$ |
| depth | $\mathrm{Categorical}[1,4]$ |
| attn$_$dropout | $\mathrm{Uniform}[0,0.5]$ |
| ffn$_$dropout | $\mathrm{Uniform}[0,0.5]$ |
| num$_$special$_$tokens | $\mathrm{UniformInt}[1,4]$ |
| learning$_$rate | $\mathrm{LogUniform}[1e\text{-}5,1e\text{-}3]$ |
| weight$_$decay | $\mathrm{LogUniform}[1e\text{-}6,1e\text{-}3]$ |

## Appendix E Ablation Studies

### E.1 Implementation Details

In this section, we provide implementation details of the ablation studies on the properties of MBA block. The first two experimental configurations (i.e., MHAhidden_size and MHAhead_dim) ensure that the number of heads in MHA matches that in MBA of MAYA, which is formulated as:

$$ ${\rm{num_head}}_{\rm MHA}={\rm{num_heads}}_{per_branch}\times\rm{num_branch}$ (15) $$

For the first configuration, MHAhidden_size, we ensure that the hidden$_$size is consistent with MBA. The hidden$_$size for MHA should be:

$$ $\rm{hidden_size}_{MHA}=\rm{head_dim}_{MHA}\times\rm{num_head}_{MHA}$ (16) $$

However, there may be cases where the hidden$_$size cannot be evenly divided by the number of parallel attention branches (i.e., num$_$branch), resulting in a non-integer value for head$_$dimMHA in Eq.[16](https://arxiv.org/html/2502.12507v3#A5.E16). In such cases, we perform a ceiling operation on head$_$dimMHA, which may lead to hidden$_$sizeMHA being slightly larger than hidden$_$sizeMBA. Please refer to Table [19](https://arxiv.org/html/2502.12507v3#A5.T19) for the specific hidden sizes corresponding to each dataset.

For the second configuration, MHAhead_dim, we ensure that the subspace dimension for each head, i.e. head$_$dimMHA, remains identical with MBA, whereby the hidden$_$size for MHA is still calculated using Eq.[16](https://arxiv.org/html/2502.12507v3#A5.E16). For the specific hidden sizes corresponding to each dataset, please refer to Table [20](https://arxiv.org/html/2502.12507v3#A5.T20).

**Table 19: Details for MHAhidden_size. Notations: $\#_{branch}$ $\thicksim$ num$_$branch, $\#_{heads}$ $\thicksim$ num$_$heads, Dattn $\thicksim$ hidden$_$size, Dhead $\thicksim$ head$_$dim.**
|  | MBA in MAYA | MHAhidden_size |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| $\#_{branch}$ | $\#_{heads}$ | Dattn | $\#_{heads}$ | Dhead | Dattn |  |
| BA | 4 | 4 | 192 | $4\times 4=16$ | $192/16=12$ | $16\times 12=192$ |
| BL | 6 | 8 | 128 | $6\times 8=48$ | $\lceil 128/48\rceil=3$ | $48\times 3=144$ |
| CA | 6 | 1 | 128 | $6\times 1=6$ | $\lceil 128/6\rceil=22$ | $6\times 22=132$ |
| QS | 8 | 1 | 160 | $8\times 1=8$ | $160/8=20$ | $8\times 20=160$ |
| SE | 4 | 1 | 224 | $4\times 1=4$ | $224/4=56$ | $4\times 56=224$ |
| SH | 3 | 4 | 224 | $3\times 4=12$ | $\lceil 224/12\rceil=19$ | $12\times 19=228$ |

**Table 20: Details for MHAhead_dim. Notations: $\#_{branch}$ $\thicksim$ num$_$branch, $\#_{heads}$ $\thicksim$ num$_$heads, Dattn $\thicksim$ hidden$_$size, Dhead $\thicksim$ head$_$dim.**
|  | MBA in MAYA | MHAhead_dim |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| $\#_{branch}$ | $\#_{heads}$ | Dattn | Dhead | $\#_{heads}$ | Dhead | Dattn |  |
| BA | 4 | 4 | 192 | $192/4=48$ | $4\times 4=16$ | 48 | $16\times 48=768$ |
| BL | 6 | 8 | 128 | $128/8=16$ | $6\times 8=48$ | 16 | $48\times 16=768$ |
| CA | 6 | 1 | 128 | $128/1=128$ | $6\times 1=6$ | 128 | $6\times 128=768$ |
| QS | 8 | 1 | 160 | $160/1=160$ | $8\times 1=8$ | 160 | $8\times 160=1280$ |
| SE | 4 | 1 | 224 | $224/1=224$ | $4\times 1=4$ | 224 | $4\times 224=896$ |
| SH | 3 | 4 | 224 | $224/4=56$ | $3\times 4=12$ | 56 | $12\times 56=672$ |

### E.2 Computation of Parameter Counts in Tabel 3

In Table [3](https://arxiv.org/html/2502.12507v3#S4.T3) of the main text, we present a comparison of the parameter counts in the encoder of MHAhidden_size and MHAhead_dim with that of MAYA. Here, we provide the specific method for the calculation.

In the MBA block, each attention branch constitutes an MHA. The MHA is comprised of four linear layers, namely query, key, value, and output projection. The weights of each linear layer form a square matrix of dimensions hidden$_$size $\times$ hidden$_$size. Therefore, the number of parameters in the attention mechanism can be calculated as follows:

$$ $P_{attn}=\rm{num_branch}\times(4\times\rm{hidden_size}^{2})$ (17) $$

In the FFN, we employ ReGLU shazeer2020glu, which differs from the original transformer architecture by incorporating three linear layers. Analogous to the naming convention used in LLaMA touvron2023llama, we refer to these layers as up, gate, and down projections. Due to the presence of the intermediate$_$factor
, the matrix weights of each linear layer are of dimension intermediate$_$factor $\times$ hidden$_$size $\times$ hidden$_$size. Consequently, the number of parameters in the FFN can be calculated as follows:

$$ $P_{ffn}=3\times\rm{intermediate_factor}\times\rm{hidden_size}^{2}$ (18) $$

Combining Eq.[17](https://arxiv.org/html/2502.12507v3#A5.E17) and Eq.[18](https://arxiv.org/html/2502.12507v3#A5.E18), the number of parameters in an encoder with num$_$layers of blocks is calculated as follows:

$$ $\displaystyle P=$ $\displaystyle{\rm{num_layers}}\times(P_{attn}+P_{ffn})$ (19) $\displaystyle=$ $\displaystyle{\rm{num_layers}}\times\rm{hidden_size}^{2}$ (20) $\displaystyle\times(4\times\rm{num_branch}+3\times\rm{intermediate_factor})$ (21) $$

According to Eq.[19](https://arxiv.org/html/2502.12507v3#A5.E19), when calculating the number of parameters for the encoders of MHAhidden_size, MHAhead_dim, and MAYA, one simply substitutes the corresponding variables into the equation. Note that the num$_$layers and intermediate$_$factor for both MHA and MAYA are always kept consistent. For instance, for the BA dataset (intermediate$_$factor$=2$, num$_$layers$=15$), the number of parameters for the encoders of MHAhidden_size, MHAhead_dim, and MAYA are as follows, respectively:

$$ $P_{{\rm MHA}_{hidden_size}}=15\times 192^{2}\times(4\times 1+3\times 2)$ (22) $$

$$ $P_{{\rm MHA}_{head_dim}}=15\times 768^{2}\times(4\times 1+3\times 2)$ (23) $$

$$ $P_{{\rm MAYA}}=15\times 192^{2}\times(4\times 4+3\times 2)$ (24) $$

Therefore, the multiples presented in Table [3](https://arxiv.org/html/2502.12507v3#S4.T3) are

$$ $P_{{\rm MHA}_{hidden_size}}/P_{{\rm MAYA}}\approx 0.5$ (25) $$

$$ $P_{{\rm MHA}_{head_dim}}/P_{{\rm MAYA}}\approx 7.3$ (26) $$

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 4
presentation: 3
questions: - Why did you omit GBDTs (e.g., tuned XGBoost, CatBoost) from your experiments? These are the standard SOTA for tabular data. How does MAYA compare to them?

- Your IAIL decoder's inference mechanism is conceptually almost identical to the one in TabR, which you cite. Why was TabR omitted as a baseline, and how do you justify the novelty of IAIL given this prior work?

- The inference cost of IAIL seems prohibitive. Can you provide a practical wall-clock time and memory usage comparison for inference (on a single instance) against FT-Transformer or XGBoost on a medium-sized dataset (e.g., 100k training samples)?

- In your MBA branch weighting, you assign larger weights to branches with larger losses . This is counter-intuitive; one might expect to down-weight poorly performing branches. Can you provide more intuition for why this "penalizing" strategy is superior?
soundness: 2
strengths: - **Novel and Efficient Encoder (MBA)**: The MBA block is the paper's strongest point. It is a well-motivated and clever solution to the parameter-growth problem in standard MHA-based tabular models. The design (parallel branches with weighted averaging) is simple, effective, and well-supported by ablations and visualizations.

- **Strong Results vs. Other Transformers**: The paper is empirically rigorous within its chosen subgroup. It demonstrates SOTA performance against a comprehensive suite of other deep tabular models (AutoInt, FTT, SAINT, etc.) across 36 datasets.

- **Thorough Ablations**: The authors provide good ablation studies that validate their individual design choices for the MBA block , the IAIL decoder , and even the tokenizer's activation function.
summary: his paper proposes MAYA, an encoder-decoder Transformer architecture designed to handle feature heterogeneity in tabular data.

The main contributions are twofold:
-  Encoder: A novel Multi-Branch of Attention (MBA) block is introduced. This block runs multiple parallel MHA branches and then averages their outputs, rather than concatenating them. This is intended to capture diverse feature interactions without the quadratic parameter growth seen in standard MHA-based models. The averaging is weighted using a "collaborative learning" scheme, where weights are dynamically assigned based on each branch's prediction loss.
- Decoder: An inter-instance attention mechanism, IAIL, is used. This is a cross-attention block that, during training, uses batch label embeddings as the Value. Critically, during inference, it uses the entire training set's [CLS] tokens and label embeddings as the Key and Value .

The authors evaluate MAYA on 36 datasets (14 in Part I, 22 in Part II) and claim superior performance among transformer-based methods.
weaknesses: - **Missing Critical Baselines (GBDTs)**: This is a fatal flaw. Any paper claiming SOTA on tabular data must compare against tuned Gradient-Boosted Decision Trees (e.g., XGBoost, CatBoost). The paper only compares against other Transformers. This is not a fair or complete comparison, and the "superior performance" claim is unsubstantiated.

- **Missing Critical Baselines (TabR)**: The IAIL decoder's inference mechanism—using the entire training set as a Key/Value store —is a kNN-like retrieval method. This is extremely similar to the mechanism in TabR (Gorishniy et al., 2024). The authors cite TabR but do not compare against it. This is a major, unacceptable omission of a highly relevant SOTA model.

- **Impractical Inference** Cost: The IAIL decoder is computationally and memory-prohibitive. Requiring L2 computation against the entire training set for every test instance is not a "limitation", it is a fundamental design flaw that makes the method non-scalable and unusable in practice.

- **Failure to Address Resubmission Feedback**: Given this is a resubmission, the failure to include GBDTs and TabR as baselines is inexplicable. This is the most basic and obvious feedback any prior reviewer would have given.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See Above.
soundness: 3
strengths: - **Architectural intuition and originality**

I think the Multi-Branch of Attention (MBA) module is a well-conceived and intuitive idea. Tabular data often exhibit highly irregular and non-smooth decision boundaries—an aspect that tree-based models have long exploited through ensemble partitioning. MBA can be viewed as a soft, attention-based analogue of this principle: multiple attention branches learn complementary sub-spaces, akin to an ensemble, but within a unified Transformer framework. This design is intuitively aligned with the known structure of tabular decision boundaries.

- **Clear ablations and interpretability evidence.**

The ablation studies convincingly show how MBA and IAIL contribute independently and jointly. The accompanying visualizations help readers understand how different attention branches capture distinct feature interactions.
summary: This paper proposes MAYA, a Transformer-based architecture for tabular data that introduces two key components: the Multi-Branch of Attention (MBA) for capturing diverse intra-feature dependencies and the Inter-instance Attention Incorporated with Labels (IAIL) for modeling cross-sample relations. Empirical results show consistent improvements across various benchmarks.
weaknesses: - **Lack of comparison with TabPFN.**

The experiments are extensive but omit a comparison with TabPFN, which has recently set a strong benchmark for tabular learning. Such a comparison would be helpful to establish the true empirical strength of MAYA.

- **Insufficient analysis of data-dependent behavior.**

The paper does not examine how the multi-branch attention design performs under different data conditions. Since tabular datasets vary widely in feature type composition, noise level, and sample size, these factors could strongly influence the effectiveness of MBA. A systematic study along these axes would clarify when the proposed architecture provides genuine benefits and when it may offer limited advantage.

---

confidence: 3
contribution: 1
initial_rating: 2
presentation: 2
questions: - Can you either submit results to tabarena, for a fair comparison to state-of-the-art methods, or compare against TabPFNV2, TabDBT, RealMLP and TabM on your benchmark?

- Can you please provide training and inference time comparisons for all methods?
soundness: 2
strengths: - The paper uses a well-established benchmark suite, the Grinsztajn benchmark.
- The paper addresses heterogeneous feature distributions, a common source of low performance for neural architectures.
- The paper does extensive hyper-parameter tuning for all methods, and provides the hyper-parameter spaces that were used.
- The paper provides ablation studies for some of the architectural choices.
summary: The paper proposes a novel, transformer-based architecture for classification and regression on tabular data using an encoder/decoder framework called MAYA. The work introduces two new layer types: multi-branch attention in the encoder and Inter--instance Attention Incorporating with Labels (IAIL) in the decoder.

The paper compares classification performance on 5 regression datasets from the literature, and the Grinsztajn benchmark suite, showing the superiority of the proposed method over some previously published transformer based architectures.
weaknesses: - The paper does not compare against any state-of-the-art algorithms for regression and classification for tabular data. Limiting the comparison to "transformer-based architectures" is not a meaningful constraint to me. Why would one be interested in the best transformer-based architecture, and not in the best architecture overall? The authors also exclude state-of-the-art pre-trained transformer methods like TabPFNV2, TabDBT and TabICL. A good overview of state-of-the-art methods for tabular classification and regression can be found in the TabArena benchmark: https://huggingface.co/spaces/TabArena/leaderboard  https://huggingface.co/spaces/TabArena/leaderboard

- The only algorithm considered in the paper that had better than baseline performance on the TabZilla benchmark https://arxiv.org/pdf/2305.02997 is SAINT, and this method was not evaluated on the full benchmark.

- TabICL in particular has addressed feature heterogeneity through pre-trained column embeddings. The benefit over this approach should be clearly demonstrated in the paper.

- No comparison of training and inference times are provided in the paper.

The main question about any new architecture is "does it work". It's unclear that the proposed architecture is superior to the relatively simple TabM or RealMLP, for example.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: Given that the model does not demonstrate state-of-the-art performance and the paper lacks theoretical analysis, what is the primary intended contribution of this work? Are the methodological components (MBA, IAIL) intended to provide generalizable insights? Please clarify how this work should be positioned within the broader literature on tabular deep learning.
soundness: 3
strengths: 1. **Novel architecture.** The proposed architecture appears novel. While the proposed MBA scheme is a bit confusing, I find the IAIL interesting. It reminds me of the ModernNCA (Ye 2025) approach but adapted to the transformer architecture. 
2. **Clarity.** The paper is relatively easy to follow and is well-written. 
3. **Results.** The authors provide an ablation section, which highlights the importance of MBA and IAIL. The authors also provide a brief comparison of their method with other tabular Transformer architectures on a selected set of open-source datasets.
summary: The paper proposes a new attention mechanism tailored for tabular data. The authors first identify two drawbacks of standard multi-head attention (MHA) for heterogeneous features: (1) that increasing the number of heads ($H$) to capture feature diversity unnecessarily increases FFN parameters, and (2) that the final projection layer mixes subspaces, reducing diversity.

As a solution, they propose Multi-Branch Attention (MBA), which computes several MHA blocks in parallel and combines their outputs using a weighted average. These weights are learned by a predictor network based on the branch outputs.

Second, the paper introduces an inter-instance attention with labels (IAIL) prediction scheme. IAIL uses a decoder architecture where queries are the test instance features, while keys and values are the training batch features and label embeddings, respectively. A final MLP head forms the prediction.
weaknesses: 1. **MBA formulation, motivation and ablation.** I find the design of the MBA block confusing. First, I want to establish that in the standard Transformer implementation, the number of heads is the main driver of the diversity. The authors argue that this is inadequate for two reasons:
>  two drawbacks: (1) concatenating MHA head subspaces enlarges attention output hidden states, thus increasing the number of parameters in FFN and computational cost; (2) output projection of concatenated features causes mixing, reducing subspace diversity.

For reason (1), I feel this is incorrect usage of terminology: concatenating heads does not enlarge hidden space, it just increases the projection matrix, $W_O$, dimensions. For reason (2), I find it unsubstantiated, do you have reference works or anything to back this story?   

In general, I think the design choice of MBA is not canonical, i.e., there are multiple ways to achieve this, which makes the work less interesting. For example, from the authors' own ablation Table 3, it seems it's the introduction of the weights $W_B$ that is crucial for the performance. However, these weights could be similarly computed for the heads, and then the head outputs could be similarly combined.

2. **The IAIL block.** I think the formulation is very close to the one proposed in ModernNCA (Ye 2024). Therefore, I would want to see a close comparison and the differences to be highlighted. 

3. **Lacking experimental results.** Transformers are not SOTA in tabular, except when they are used in ICL context such as in TabPFN, TabICL, and Limix. Why pursue this direction? I would want to see a comparison of the proposed MAYA against the SOTA tabular models. For example, for California, I've seen models achieving below 0.4 in RMSE.


Decision: Reject